# blended-tiling
[][#github-license]
[][#pypi-package]
[](https://zenodo.org/badge/latestdoi/503953108)
[#github-license]: https://github.com/progamergov/blended-tiling/blob/main/LICENSE
[#pypi-package]: https://pypi.org/project/blended-tiling/
This module adds support for splitting NCHW tensor inputs like images & activations into overlapping tiles of equal size, and then blending those overlapping tiles together after they have been altered. This module is also fully Autograd & JIT / TorchScript compatible.
This tiling solution is intended for situations where one wishes to render / generate outputs that are larger than what their computing device can support. Tiles can be separately rendered and periodically blended together to maintain tile feature coherence.
## Setup:
**Installation Requirements**
- Python >= 3.6
- PyTorch >= 1.6
**Installation via `pip`:**
```
pip install blended-tiling
```
**Dev / Manual install**:
```
git clone https://github.com/progamergov/blended-tiling.git
cd blended-tiling
pip install -e .
# Notebook installs also require appending to environment variables
# import sys
# sys.path.append('/content/blended-tiling')
```
## Documentation
### `TilingModule`
The base blended tiling module.
```
blended_tiling.TilingModule(tile_size=(224, 224), tile_overlap=(0.25, 0.25), base_size=(512, 512))
```
**Initialization Variables**
* `tile_size` (int or tuple of int): The size of tiles to use. A single integer to use for both the height and width dimensions, or a list / tuple of dimensions with a shape of: `[height, width]`. The chosen tile sizes should be less than or equal to the sizes of the full NCHW tensor (`base_size`).
* `tile_overlap` (int or tuple of int): The amount of overlap to use when creating tiles. A single integer to use for both the height and width dimensions, or a list / tuple of dimensions with a shape of: `[height, width]`. The chosen overlap percentages should be in the range [0.0, 0.50] (0% - 50%).
* `base_size` (int or tuple of int): The size of the NCHW tensor being split into tiles. A single integer to use for both the height and width dimensions, or a list / tuple of dimensions with a shape of: `[height, width]`.
#### Methods
**`num_tiles()`**
* Returns
* `num_tiles` (int): The number of tiles that the full image shape is divided into based on specified parameters.
**`tiling_pattern()`**
* Returns:
* `pattern` (list of int): The number of tiles per column and number of tiles per row, in the format of: `[n_tiles_per_column, n_tiles_per_row]`.
**`split_into_tiles(x)`**: Splits an NCHW image input into overlapping tiles, and then returns the tiles. The `base_size` parameter is automatically readjusted to match the input.
* Returns:
* `tiles` (torch.Tensor): A set of tiles created from the input image.
**`get_tile_masks(channels=3, device=torch.device("cpu"), dtype=torch.float)`**: Return a stack of NCHW masks corresponding to the tiles outputted by `.split_into_tiles(x)`.
* Variables:
* `channels` (int, optional): The number of channels to use for the masks. Default: 3
* `device` (torch.device, optional): The desired device to create the masks on. Default: torch.device("cpu")
* `dtype` (torch.dtype, optional): The desired dtype to create the masks with. Default: torch.float
* Returns:
* `masks` (torch.Tensor): A set of tile masks stacked across the batch dimension.
**`rebuild(tiles, border=None, colors=None)`**: Creates and returns the full image from a stack of NCHW tiles stacked across the batch dimension.
* Variables:
* `tiles` (torch.Tensor): A set of tiles that may or not be masked, stacked across the batch dimension.
* `border` (int, optional): Optionally add a border of a specified size to the edges of tiles in the full image for debugging and explainability. Set to None for no border.
* `colors` (list of float, optional): A set of floats to use for the border color, if using borders. Default is set to red unless specified.
* Returns:
* `full_image` (torch.Tensor): The full image made up of tiles merged together without any blending.
**`rebuild_with_masks(tiles, border=None, colors=None)`:** Creates and returns the full image from a stack of NCHW tiles stacked across the batch dimension, using tile blend masks.
* Variables:
* `tiles` (torch.Tensor): A set of tiles that may or not be masked, stacked across the batch dimension.
* `border` (int, optional): Optionally add a border of a specified size to the edges of tiles in the full image for debugging and explainability. Set to None for no border.
* `colors` (list of float, optional): A set of floats to use for the border color, if using borders. Default is set to red unless specified.
* Returns:
* `full_image` (torch.Tensor): The full image made up of tiles blended together using masks.
**`forward(x)`:** Takes a stack of tiles, combines them into the full image with blending masks, then splits the image back into tiles.
* Variables:
* `x` (torch.Tensor): A set of tiles to blend the overlapping regions together of.
* Returns:
* `x` (torch.Tensor): A set of tiles with overlapping regions blended together.
### Supported Tensor Types
The `TilingModule` class has been tested with and is confirmed to work with the following PyTorch [Tensor Data types / dtypes](https://pytorch.org/docs/stable/tensors.html): `torch.float32` / `torch.float`, `torch.float64` / `torch.double`, `torch.float16` / `torch.half`, & `torch.bfloat16`.
## Usage
The `TilingModule` class is pretty easy to use.
```
from blended_tiling import TilingModule
full_size = [512, 512]
tile_size = [224, 224]
tile_overlap = [0.25, 0.25] # 25% overlap on both H & W
tiling_module = TilingModule(
tile_size=tile_size,
tile_overlap=tile_overlap,
base_size=full_size,
)
# Shape of tiles expected in forward pass
input_shape = [tiling_module.num_tiles(), 3] + tile_size
# Tiles are blended together and then split apart by default
blended_tiles = tiling_module(torch.ones(input_shape))
```
Tiles can be created and then merged back into the original tensor like this:
```
full_tensor = torch.ones(1, 3, 512, 512)
tiles = tiling_module.split_into_tiles(full_tensor)
full_tensor = tiling_module.rebuild_with_masks(tiles)
```
The tile boundaries can be viewed on the full tensor like this:
```
tiles = torch.ones(9, 3, 224, 224)
full_tensor = tiling_module.rebuild_with_masks(tiles, border=2)
```
And the number of tiles and tiling pattern can be obtained like this:
```
num_tiles = tiling_module.num_tiles()
tiling_pattern = tiling_module.tiling_pattern()
print("{}x{}".format(tiling_pattern[0], tiling_pattern[1]))
```
### Custom Classes
It's also easy to modify the forward function of the tiling module:
```
from typing import Union, List, Tuple
from blended_tiling import TilingModule
class CustomTilingModule(TilingModule):
def __init__(
self,
tile_size: Union[int, List[int], Tuple[int, int]] = [224, 224],
tile_overlap: Union[float, List[float], Tuple[float, float]] = [0.25, 0.25],
base_size: Union[int, List[int], Tuple[int, int]] = [512, 512],
) -> None:
TilingModule.__init__(self, tile_size, tile_overlap, base_size)
self.custom_module = torch.nn.Identity()
def forward(self, x: torch.Tensor) -> torch.Tensor:
x = self.rebuild_with_masks(x)
x = self.custom_module(x) + 4.0
return self._get_tiles_and_coords(x)[0]
```
## Examples
To demonstrate the tile blending abilities of the `TilingModule` class, an example has been created below.
First we'll create a set of tiles & give them all unique colors for this example:
```
# Setup TilingModule instance
full_size = [768, 1014]
tile_size = [256, 448]
tile_overlap = [0.25, 0.25]
tiling_module = TilingModule(
tile_size=tile_size,
tile_overlap=tile_overlap,
base_size=full_size,
)
# Create unique colors for tiles
tile_colors = [
[0.5334, 0.0, 0.8459],
[0.0, 1.0, 0.0],
[0.0, 0.7071, 0.7071],
[0.7071, 0.7071, 0.0],
[1.0, 0.0, 0.0],
[0.8459, 0.0, 0.5334],
[0.7071, 0.0, 0.7071],
[0.0, 0.8459, 0.5334],
[0.5334, 0.8459, 0.0],
[0.0, 0.5334, 0.8459],
[0.0, 0.0, 1.0],
[0.8459, 0.5334, 0.0],
]
tile_colors = torch.as_tensor(tile_colors).view(12, 3, 1, 1)
# Create tiles
tiles = torch.ones([tiling_module.num_tiles(), 3] + tile_size)
# Color tiles
tiles = tiles * tile_colors
```
<img src="https://github.com/ProGamerGov/blended-tiling/raw/main/examples/without_masks_separate_tiles.jpg" width="500">
Next we apply the blend masks to the tiles:
```
tiles = tiles * tiling_module.get_tile_masks()
```
<img src="https://github.com/ProGamerGov/blended-tiling/raw/main/examples/with_masks_separate_tiles.jpg" width="500">
We can now combine the masked tiles into the full image:
```
# Build full tiled image
output = tiling_module.rebuild(tiles)
```
<img src="https://github.com/ProGamerGov/blended-tiling/raw/main/examples/with_masks.jpg" width="500">
We can also view the tile boundaries like so:
```
# Build full tiled image
output = tiling_module.rebuild(tiles, border=2, colors=[0,0,0])
```
<img src="https://github.com/ProGamerGov/blended-tiling/raw/main/examples/with_masks_and_borders.jpg" width="500">
We can view an animation of the tiles being added like this:
```
from torchvision.transforms import ToPILImage
tile_steps = [
tiling_module.rebuild(tiles[: i + 1]) for i in range(tiles.shape[0])
]
tile_frames = [
ToPILImage()(x[0])
for x in [torch.zeros_like(tile_steps[0])] + tile_steps + [tile_steps[-1]]
]
tile_frames[0].save(
"tiles.gif",
format="GIF",
append_images=tile_frames[1:],
save_all=True,
duration=700,
loop=0,
)
```
<img src="https://github.com/ProGamerGov/blended-tiling/raw/main/examples/with_masks.gif" width="500">
Raw data
{
"_id": null,
"home_page": "https://github.com/progamergov/blended-tiling",
"name": "blended-tiling",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": null,
"keywords": "blended-tiling, tiler, tiling, masking",
"author": "Ben Egan",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/ed/a0/867beb4490a8da97a205441317e592c0e75e26f502c54ef2b0d56120a2d8/blended_tiling-0.0.1.dev7.tar.gz",
"platform": null,
"description": "# blended-tiling\n\n[][#github-license]\n[][#pypi-package]\n[](https://zenodo.org/badge/latestdoi/503953108)\n\n[#github-license]: https://github.com/progamergov/blended-tiling/blob/main/LICENSE\n[#pypi-package]: https://pypi.org/project/blended-tiling/\n\n\nThis module adds support for splitting NCHW tensor inputs like images & activations into overlapping tiles of equal size, and then blending those overlapping tiles together after they have been altered. This module is also fully Autograd & JIT / TorchScript compatible.\n\nThis tiling solution is intended for situations where one wishes to render / generate outputs that are larger than what their computing device can support. Tiles can be separately rendered and periodically blended together to maintain tile feature coherence.\n\n## Setup:\n\n**Installation Requirements**\n- Python >= 3.6\n- PyTorch >= 1.6\n\n**Installation via `pip`:**\n\n```\npip install blended-tiling\n```\n\n**Dev / Manual install**:\n\n```\ngit clone https://github.com/progamergov/blended-tiling.git\ncd blended-tiling\npip install -e .\n\n# Notebook installs also require appending to environment variables\n# import sys\n# sys.path.append('/content/blended-tiling')\n```\n\n\n## Documentation\n\n### `TilingModule`\n\nThe base blended tiling module.\n\n```\nblended_tiling.TilingModule(tile_size=(224, 224), tile_overlap=(0.25, 0.25), base_size=(512, 512))\n```\n\n**Initialization Variables**\n\n* `tile_size` (int or tuple of int): The size of tiles to use. A single integer to use for both the height and width dimensions, or a list / tuple of dimensions with a shape of: `[height, width]`. The chosen tile sizes should be less than or equal to the sizes of the full NCHW tensor (`base_size`).\n* `tile_overlap` (int or tuple of int): The amount of overlap to use when creating tiles. A single integer to use for both the height and width dimensions, or a list / tuple of dimensions with a shape of: `[height, width]`. The chosen overlap percentages should be in the range [0.0, 0.50] (0% - 50%).\n* `base_size` (int or tuple of int): The size of the NCHW tensor being split into tiles. A single integer to use for both the height and width dimensions, or a list / tuple of dimensions with a shape of: `[height, width]`.\n\n\n#### Methods\n\n\n**`num_tiles()`**\n\n * Returns\n * `num_tiles` (int): The number of tiles that the full image shape is divided into based on specified parameters.\n\n**`tiling_pattern()`**\n\n * Returns:\n * `pattern` (list of int): The number of tiles per column and number of tiles per row, in the format of: `[n_tiles_per_column, n_tiles_per_row]`.\n\n**`split_into_tiles(x)`**: Splits an NCHW image input into overlapping tiles, and then returns the tiles. The `base_size` parameter is automatically readjusted to match the input.\n * Returns:\n * `tiles` (torch.Tensor): A set of tiles created from the input image.\n\n**`get_tile_masks(channels=3, device=torch.device(\"cpu\"), dtype=torch.float)`**: Return a stack of NCHW masks corresponding to the tiles outputted by `.split_into_tiles(x)`.\n\n * Variables:\n * `channels` (int, optional): The number of channels to use for the masks. Default: 3\n * `device` (torch.device, optional): The desired device to create the masks on. Default: torch.device(\"cpu\")\n * `dtype` (torch.dtype, optional): The desired dtype to create the masks with. Default: torch.float\n * Returns:\n * `masks` (torch.Tensor): A set of tile masks stacked across the batch dimension.\n\n\n**`rebuild(tiles, border=None, colors=None)`**: Creates and returns the full image from a stack of NCHW tiles stacked across the batch dimension.\n * Variables:\n * `tiles` (torch.Tensor): A set of tiles that may or not be masked, stacked across the batch dimension.\n * `border` (int, optional): Optionally add a border of a specified size to the edges of tiles in the full image for debugging and explainability. Set to None for no border.\n * `colors` (list of float, optional): A set of floats to use for the border color, if using borders. Default is set to red unless specified.\n * Returns:\n * `full_image` (torch.Tensor): The full image made up of tiles merged together without any blending.\n\n**`rebuild_with_masks(tiles, border=None, colors=None)`:** Creates and returns the full image from a stack of NCHW tiles stacked across the batch dimension, using tile blend masks.\n * Variables:\n * `tiles` (torch.Tensor): A set of tiles that may or not be masked, stacked across the batch dimension.\n * `border` (int, optional): Optionally add a border of a specified size to the edges of tiles in the full image for debugging and explainability. Set to None for no border.\n * `colors` (list of float, optional): A set of floats to use for the border color, if using borders. Default is set to red unless specified.\n * Returns:\n * `full_image` (torch.Tensor): The full image made up of tiles blended together using masks.\n\n**`forward(x)`:** Takes a stack of tiles, combines them into the full image with blending masks, then splits the image back into tiles.\n * Variables:\n * `x` (torch.Tensor): A set of tiles to blend the overlapping regions together of.\n * Returns:\n * `x` (torch.Tensor): A set of tiles with overlapping regions blended together.\n\n\n### Supported Tensor Types\n\nThe `TilingModule` class has been tested with and is confirmed to work with the following PyTorch [Tensor Data types / dtypes](https://pytorch.org/docs/stable/tensors.html): `torch.float32` / `torch.float`, `torch.float64` / `torch.double`, `torch.float16` / `torch.half`, & `torch.bfloat16`.\n\n\n## Usage\n\nThe `TilingModule` class is pretty easy to use.\n\n```\nfrom blended_tiling import TilingModule\n\n\nfull_size = [512, 512]\ntile_size = [224, 224]\ntile_overlap = [0.25, 0.25] # 25% overlap on both H & W\n\ntiling_module = TilingModule(\n tile_size=tile_size,\n tile_overlap=tile_overlap,\n base_size=full_size,\n)\n\n# Shape of tiles expected in forward pass\ninput_shape = [tiling_module.num_tiles(), 3] + tile_size\n\n# Tiles are blended together and then split apart by default\nblended_tiles = tiling_module(torch.ones(input_shape))\n```\n\n\nTiles can be created and then merged back into the original tensor like this:\n\n```\nfull_tensor = torch.ones(1, 3, 512, 512)\n\ntiles = tiling_module.split_into_tiles(full_tensor)\n\nfull_tensor = tiling_module.rebuild_with_masks(tiles)\n```\n\nThe tile boundaries can be viewed on the full tensor like this:\n\n```\ntiles = torch.ones(9, 3, 224, 224)\nfull_tensor = tiling_module.rebuild_with_masks(tiles, border=2)\n```\n\nAnd the number of tiles and tiling pattern can be obtained like this:\n\n```\nnum_tiles = tiling_module.num_tiles()\n\ntiling_pattern = tiling_module.tiling_pattern()\nprint(\"{}x{}\".format(tiling_pattern[0], tiling_pattern[1]))\n```\n\n\n### Custom Classes\n\nIt's also easy to modify the forward function of the tiling module:\n\n```\nfrom typing import Union, List, Tuple\nfrom blended_tiling import TilingModule\n\nclass CustomTilingModule(TilingModule):\n def __init__(\n self,\n tile_size: Union[int, List[int], Tuple[int, int]] = [224, 224],\n tile_overlap: Union[float, List[float], Tuple[float, float]] = [0.25, 0.25],\n base_size: Union[int, List[int], Tuple[int, int]] = [512, 512],\n ) -> None:\n TilingModule.__init__(self, tile_size, tile_overlap, base_size)\n self.custom_module = torch.nn.Identity()\n\n def forward(self, x: torch.Tensor) -> torch.Tensor:\n x = self.rebuild_with_masks(x)\n x = self.custom_module(x) + 4.0\n return self._get_tiles_and_coords(x)[0]\n```\n\n\n## Examples\n\n\nTo demonstrate the tile blending abilities of the `TilingModule` class, an example has been created below.\n\n\n\nFirst we'll create a set of tiles & give them all unique colors for this example:\n\n```\n# Setup TilingModule instance\nfull_size = [768, 1014]\ntile_size = [256, 448]\ntile_overlap = [0.25, 0.25]\ntiling_module = TilingModule(\n tile_size=tile_size,\n tile_overlap=tile_overlap,\n base_size=full_size,\n)\n\n# Create unique colors for tiles\ntile_colors = [\n [0.5334, 0.0, 0.8459],\n [0.0, 1.0, 0.0],\n [0.0, 0.7071, 0.7071],\n [0.7071, 0.7071, 0.0],\n [1.0, 0.0, 0.0],\n [0.8459, 0.0, 0.5334],\n [0.7071, 0.0, 0.7071],\n [0.0, 0.8459, 0.5334],\n [0.5334, 0.8459, 0.0],\n [0.0, 0.5334, 0.8459],\n [0.0, 0.0, 1.0],\n [0.8459, 0.5334, 0.0],\n]\ntile_colors = torch.as_tensor(tile_colors).view(12, 3, 1, 1)\n\n# Create tiles\ntiles = torch.ones([tiling_module.num_tiles(), 3] + tile_size)\n\n# Color tiles\ntiles = tiles * tile_colors\n```\n\n<img src=\"https://github.com/ProGamerGov/blended-tiling/raw/main/examples/without_masks_separate_tiles.jpg\" width=\"500\">\n\nNext we apply the blend masks to the tiles:\n\n```\ntiles = tiles * tiling_module.get_tile_masks()\n```\n\n<img src=\"https://github.com/ProGamerGov/blended-tiling/raw/main/examples/with_masks_separate_tiles.jpg\" width=\"500\">\n\n\nWe can now combine the masked tiles into the full image:\n\n```\n# Build full tiled image\noutput = tiling_module.rebuild(tiles)\n```\n\n<img src=\"https://github.com/ProGamerGov/blended-tiling/raw/main/examples/with_masks.jpg\" width=\"500\">\n\nWe can also view the tile boundaries like so:\n\n```\n# Build full tiled image\noutput = tiling_module.rebuild(tiles, border=2, colors=[0,0,0])\n```\n\n<img src=\"https://github.com/ProGamerGov/blended-tiling/raw/main/examples/with_masks_and_borders.jpg\" width=\"500\">\n\nWe can view an animation of the tiles being added like this:\n\n```\nfrom torchvision.transforms import ToPILImage\n\ntile_steps = [\n tiling_module.rebuild(tiles[: i + 1]) for i in range(tiles.shape[0])\n]\ntile_frames = [\n ToPILImage()(x[0])\n for x in [torch.zeros_like(tile_steps[0])] + tile_steps + [tile_steps[-1]]\n]\ntile_frames[0].save(\n \"tiles.gif\",\n format=\"GIF\",\n append_images=tile_frames[1:],\n save_all=True,\n duration=700,\n loop=0,\n)\n```\n\n<img src=\"https://github.com/ProGamerGov/blended-tiling/raw/main/examples/with_masks.gif\" width=\"500\">\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Blended tiling with PyTorch",
"version": "0.0.1.dev7",
"project_urls": {
"Homepage": "https://github.com/progamergov/blended-tiling"
},
"split_keywords": [
"blended-tiling",
" tiler",
" tiling",
" masking"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7daf05d2e1e62965f719665a8e8c9653daaab3d074416f90b206e22177950a71",
"md5": "7e111720b5c7743b314e86c097c8d7b9",
"sha256": "d16eebd461b21380b8ad8d74f94a3c4455c1dd7384f84d8cb15a3a9fafd5a173"
},
"downloads": -1,
"filename": "blended_tiling-0.0.1.dev7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7e111720b5c7743b314e86c097c8d7b9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 10409,
"upload_time": "2024-12-18T22:42:02",
"upload_time_iso_8601": "2024-12-18T22:42:02.300521Z",
"url": "https://files.pythonhosted.org/packages/7d/af/05d2e1e62965f719665a8e8c9653daaab3d074416f90b206e22177950a71/blended_tiling-0.0.1.dev7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "eda0867beb4490a8da97a205441317e592c0e75e26f502c54ef2b0d56120a2d8",
"md5": "c0baa17a7a36c37b937ffc050f30ac80",
"sha256": "04b2ddd7d54ca688823171741948f04f8180205c92e9d3e662336c1a4372c267"
},
"downloads": -1,
"filename": "blended_tiling-0.0.1.dev7.tar.gz",
"has_sig": false,
"md5_digest": "c0baa17a7a36c37b937ffc050f30ac80",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 14641,
"upload_time": "2024-12-18T22:42:05",
"upload_time_iso_8601": "2024-12-18T22:42:05.229508Z",
"url": "https://files.pythonhosted.org/packages/ed/a0/867beb4490a8da97a205441317e592c0e75e26f502c54ef2b0d56120a2d8/blended_tiling-0.0.1.dev7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-18 22:42:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "progamergov",
"github_project": "blended-tiling",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "blended-tiling"
}
JSON
