dynbatcher

Name	dynbatcher JSON
Version	1.0.0 JSON
	download
home_page	https://github.com/robuno/dynbatcher
Summary	A package for creation of dataloader with dynamic batch sizes in PyTorch
upload_time	2024-08-01 10:37:02
maintainer	None
docs_url	None
author	Unat Teksen
requires_python	>=3.6
license	MIT
keywords
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            # dynbatcher - Dynamic Batch Size Dataloader Generator

`dynbatcher` is a Python package designed to facilitate the creation and management of PyTorch DataLoaders with custom batch sizes and ratios. This package is especially useful for training neural networks with dynamic batch sizes. With `dynbatcher` you can divide a dataset into subsets with different batch sizes and turn it into a single Dataloader ready for training. 

## Features

- Split datasets based on custom ratios and batch sizes.
- Create DataLoaders for each subset with different batch sizes.
- Combine multiple DataLoaders into a single DataLoader.
- Plot samples for a selected batch in created Dataloader.

## Installation

Install `dynbatcher` using pip:

```bash
pip install dynbatcher
```

## Usage

### Importing the package

```
from torch.utils.data import DataLoader
import torchvision
import torchvision.transforms as transforms
import dynbatcher

# Define batch sizes and ratios
ratios = [0.5, 0.3, 0.2]              # Corresponding ratios for each subset
batch_sizes_train = [32, 64, 128]     # Corresponding batch sizes for each subset

# Add transforms and download datasets
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)
testset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)

# Set dataloaders and use DBS Merger & Generator
trainloader = dynbatcher.load_merged_trainloader(trainset, batch_sizes_train, ratios, print_info=True)
testloader = DataLoader(testset, batch_size=64, shuffle=False)

# Example: Access and print information for the selected batch, and display samples
dynbatcher.print_batch_info(trainloader, index=127, display_samples=True)
```

### Parameters

- `batch_sizes_train`: You can choose which batch sizes you want to split the dataset into.
- `ratios`: You can choose the ratio in which the data will be allocated to the batch sizes you choose for the dataset. If you do not specify a ratio, it will allocate an equal number of samples to the given batch sizes.

### Functions

- `load_merged_trainloader`: This function divides the given dataset with the batch size and ratios you choose and combines them to create a single dataloader.
- `print_batch_info`: This function displays and allows you to examine all samples for any batch you select.


### Investigating Batch Samples

<div align="center">
    <img src="https://raw.githubusercontent.com/starkslab/starkslab.github.io/main/dbstraining/static/images/batch128.png" width="250" style="vertical-align: top;" alt="Batch Size 128 Samples" />
    <img src="https://raw.githubusercontent.com/starkslab/starkslab.github.io/main/dbstraining/static/images/batch64.png" width="250" style="vertical-align: top;" alt="Batch Size 64 Samples" />
    <img src="https://raw.githubusercontent.com/starkslab/starkslab.github.io/main/dbstraining/static/images/batch32.png" width="250" style="vertical-align: top;" alt="Batch Size 32 Samples" />
</div>

## Contributing
Contributions are welcome! Please open an issue or submit a pull request on GitHub.

## License
This project is licensed under the MIT License.

## Acknowledgments
Inspired by the need for flexible and efficient DataLoader management in PyTorch.

```
This `README.md` provides an overview of your project, installation instructions, usage examples, a brief description of the key functions, and information on contributing and licensing.
```

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/robuno/dynbatcher",
    "name": "dynbatcher",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.6",
    "maintainer_email": null,
    "keywords": null,
    "author": "Unat Teksen",
    "author_email": null,
    "download_url": "https://files.pythonhosted.org/packages/1e/cb/d3c64ed3f466e4cec35744dfb45d56b5cfc1c921e49c7ec6ce168e900f1f/dynbatcher-1.0.0.tar.gz",
    "platform": null,
    "description": "# dynbatcher - Dynamic Batch Size Dataloader Generator\r\n\r\n`dynbatcher` is a Python package designed to facilitate the creation and management of PyTorch DataLoaders with custom batch sizes and ratios. This package is especially useful for training neural networks with dynamic batch sizes. With `dynbatcher` you can divide a dataset into subsets with different batch sizes and turn it into a single Dataloader ready for training. \r\n\r\n## Features\r\n\r\n- Split datasets based on custom ratios and batch sizes.\r\n- Create DataLoaders for each subset with different batch sizes.\r\n- Combine multiple DataLoaders into a single DataLoader.\r\n- Plot samples for a selected batch in created Dataloader.\r\n\r\n## Installation\r\n\r\nInstall `dynbatcher` using pip:\r\n\r\n```bash\r\npip install dynbatcher\r\n```\r\n\r\n## Usage\r\n\r\n### Importing the package\r\n\r\n```\r\nfrom torch.utils.data import DataLoader\r\nimport torchvision\r\nimport torchvision.transforms as transforms\r\nimport dynbatcher\r\n\r\n# Define batch sizes and ratios\r\nratios = [0.5, 0.3, 0.2]              # Corresponding ratios for each subset\r\nbatch_sizes_train = [32, 64, 128]     # Corresponding batch sizes for each subset\r\n\r\n# Add transforms and download datasets\r\ntransform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])\r\ntrainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)\r\ntestset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)\r\n\r\n# Set dataloaders and use DBS Merger & Generator\r\ntrainloader = dynbatcher.load_merged_trainloader(trainset, batch_sizes_train, ratios, print_info=True)\r\ntestloader = DataLoader(testset, batch_size=64, shuffle=False)\r\n\r\n# Example: Access and print information for the selected batch, and display samples\r\ndynbatcher.print_batch_info(trainloader, index=127, display_samples=True)\r\n```\r\n\r\n### Parameters\r\n\r\n- `batch_sizes_train`: You can choose which batch sizes you want to split the dataset into.\r\n- `ratios`: You can choose the ratio in which the data will be allocated to the batch sizes you choose for the dataset. If you do not specify a ratio, it will allocate an equal number of samples to the given batch sizes.\r\n\r\n### Functions\r\n\r\n- `load_merged_trainloader`: This function divides the given dataset with the batch size and ratios you choose and combines them to create a single dataloader.\r\n- `print_batch_info`: This function displays and allows you to examine all samples for any batch you select.\r\n\r\n\r\n### Investigating Batch Samples\r\n\r\n<div align=\"center\">\r\n    <img src=\"https://raw.githubusercontent.com/starkslab/starkslab.github.io/main/dbstraining/static/images/batch128.png\" width=\"250\" style=\"vertical-align: top;\" alt=\"Batch Size 128 Samples\" />\r\n    <img src=\"https://raw.githubusercontent.com/starkslab/starkslab.github.io/main/dbstraining/static/images/batch64.png\" width=\"250\" style=\"vertical-align: top;\" alt=\"Batch Size 64 Samples\" />\r\n    <img src=\"https://raw.githubusercontent.com/starkslab/starkslab.github.io/main/dbstraining/static/images/batch32.png\" width=\"250\" style=\"vertical-align: top;\" alt=\"Batch Size 32 Samples\" />\r\n</div>\r\n\r\n## Contributing\r\nContributions are welcome! Please open an issue or submit a pull request on GitHub.\r\n\r\n## License\r\nThis project is licensed under the MIT License.\r\n\r\n## Acknowledgments\r\nInspired by the need for flexible and efficient DataLoader management in PyTorch.\r\n\r\n```\r\nThis `README.md` provides an overview of your project, installation instructions, usage examples, a brief description of the key functions, and information on contributing and licensing.\r\n```\r\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "A package for creation of dataloader with dynamic batch sizes in PyTorch",
    "version": "1.0.0",
    "project_urls": {
        "Homepage": "https://github.com/robuno/dynbatcher"
    },
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "a27d25840fbc2a5e2e8f6ca53bfc613cfc100d96d204f32e726f2b630c4e666f",
                "md5": "a3daf04dcaeafddf1c483a3ea4656631",
                "sha256": "5b8e7d44bb2803b584cb5f33a13d8c8ffe726f1973461fdb6f405ea10b94f5b3"
            },
            "downloads": -1,
            "filename": "dynbatcher-1.0.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "a3daf04dcaeafddf1c483a3ea4656631",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6",
            "size": 7159,
            "upload_time": "2024-08-01T10:37:00",
            "upload_time_iso_8601": "2024-08-01T10:37:00.984359Z",
            "url": "https://files.pythonhosted.org/packages/a2/7d/25840fbc2a5e2e8f6ca53bfc613cfc100d96d204f32e726f2b630c4e666f/dynbatcher-1.0.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "1ecbd3c64ed3f466e4cec35744dfb45d56b5cfc1c921e49c7ec6ce168e900f1f",
                "md5": "3c43fd460dafa1ffc1577d198d37a56c",
                "sha256": "4bfbdc18c6237b13a118e774338cb5f341c8b2a3ab640068367c2d6bc029956d"
            },
            "downloads": -1,
            "filename": "dynbatcher-1.0.0.tar.gz",
            "has_sig": false,
            "md5_digest": "3c43fd460dafa1ffc1577d198d37a56c",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6",
            "size": 7621,
            "upload_time": "2024-08-01T10:37:02",
            "upload_time_iso_8601": "2024-08-01T10:37:02.353705Z",
            "url": "https://files.pythonhosted.org/packages/1e/cb/d3c64ed3f466e4cec35744dfb45d56b5cfc1c921e49c7ec6ce168e900f1f/dynbatcher-1.0.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-08-01 10:37:02",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "robuno",
    "github_project": "dynbatcher",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "requirements": [],
    "lcname": "dynbatcher"
}