# WaveMix
[](https://paperswithcode.com/sota/image-classification-on-emnist-balanced?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/image-classification-on-emnist-byclass?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/image-classification-on-emnist-bymerge?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/image-classification-on-emnist-digits?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/image-classification-on-emnist-letters?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/image-classification-on-inat2021-mini?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/scene-classification-on-places365-standard?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/semantic-segmentation-on-cityscapes-val?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/image-classification-on-caltech-256?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/image-classification-on-places365-standard?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/image-classification-on-svhn?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/image-classification-on-tiny-imagenet-1?p=wavemix-lite-a-resource-efficient-neural)
[](https://paperswithcode.com/sota/image-classification-on-fashion-mnist?p=wavemix-lite-a-resource-efficient-neural)
## Resource-efficient Token Mixing for Images using 2D Discrete Wavelet Transform
### WaveMix Architecture
### WaveMix-Lite
We propose WaveMix– a novel neural architecture for computer vision that is resource-efficient yet generalizable and scalable. WaveMix networks achieve comparable or better accuracy than the state-of-the-art convolutional neural networks, vision transformers, and token mixers for several tasks, establishing new benchmarks for segmentation on Cityscapes; and for classification on Places-365, f ive EMNIST datasets, and iNAT-mini. Remarkably, WaveMix architectures require fewer parameters to achieve these benchmarks compared to the previous state-of-the-art. Moreover, when controlled for the number of parameters, WaveMix requires lesser GPU RAM, which translates to savings in time, cost, and energy. To achieve these gains we used multi-level two-dimensional discrete wavelet transform (2D-DWT) in WaveMix blocks, which has the following advantages: (1) It reorganizes spatial information based on three strong image priors– scale-invariance, shift-invariance, and sparseness of edges, (2) in a lossless manner without adding parameters, (3) while also reducing the spatial sizes of feature maps, which reduces the memory and time required for forward and backward passes, and (4) expanding the receptive field faster than convolutions do. The whole architecture is a stack of self-similar and resolution-preserving WaveMix blocks, which allows architectural f lexibility for various tasks and levels of resource availability.
| Task | Dataset | Metric | Value |
|-----------------------|-------------|----------|--------|
| Semantic Segmentation | Cityscapes | Single-scale mIoU | 82.70% (SOTA) |
| Image Classification | ImageNet-1k | Accuracy | 74.93% |
### Parameter Efficiency
| Task | Model | Parameters |
|------------------------------|-------------------------------------------------|------------|
| 99% Accu. in MNIST | WaveMix Lite-8/10 | 3566 |
| 90% Accu. in Fashion MNIST | WaveMix Lite-8/5 | 7156 |
| 80% Accu. in CIFAR-10 | WaveMix Lite-32/7 | 37058 |
| 90% Accu. in CIFAR-10 | WaveMix Lite-64/6 | 520106 |
The high parameter efficiency is obtained by replacing Deconvolution layers with Upsampling
This is an implementation of code from the following papers : [Openreview Paper](https://openreview.net/forum?id=tBoSm4hUWV), [ArXiv Paper 1](https://arxiv.org/abs/2203.03689), [ArXiv Paper 2](https://arxiv.org/abs/2205.14375)
## Install
```bash
$ pip install wavemix
```
## Usage
### Semantic Segmentation
```python
import torch, wavemix
from wavemix.SemSegment import WaveMix
import torch
model = WaveMix(
num_classes= 20,
depth= 16,
mult= 2,
ff_channel= 256,
final_dim= 256,
dropout= 0.5,
level=4,
stride=2
)
img = torch.randn(1, 3, 256, 256)
preds = model(img) # (1, 20, 256, 256)
```
### Image Classification
```python
import torch, wavemix
from wavemix.classification import WaveMix
import torch
model = WaveMix(
num_classes= 1000,
depth= 16,
mult= 2,
ff_channel= 192,
final_dim= 192,
dropout= 0.5,
level=3,
patch_size=4,
)
img = torch.randn(1, 3, 256, 256)
preds = model(img) # (1, 1000)
```
### Single Image Super-resolution
```python
import wavemix, torch
from wavemix.sisr import WaveMix
model = WaveMix(
depth = 4,
mult = 2,
ff_channel = 144,
final_dim = 144,
dropout = 0.5,
level=1,
)
img = torch.randn(1, 3, 256, 256)
out = model(img) # (1, 3, 512, 512)
```
### To use a single Waveblock
```python
import wavemix, torch
from wavemix import Level1Waveblock
```
## Parameters
- `num_classes`: int.
Number of classes to classify/segment.
- `depth`: int.
Number of WaveMix blocks.
- `mult`: int.
Expansion of channels in the MLP (FeedForward) layer.
- `ff_channel`: int.
No. of output channels from the MLP (FeedForward) layer.
- `final_dim`: int.
Final dimension of output tensor after initial Conv layers. Channel dimension when tensor is fed to WaveBlocks.
- `dropout`: float between `[0, 1]`, default `0.`.
Dropout rate.
- `level`: int.
Number of levels of 2D wavelet transform to be used in Waveblocks. Currently supports levels from 1 to 4.
- `stride`: int.
Stride used in the initial convolutional layers to reduce the input resolution before being fed to Waveblocks.
- `initial_conv`: str.
Deciding between strided convolution or patchifying convolutions in the intial conv layer. Used for classification. 'pachify' or 'strided'.
- `patch_size`: int.
Size of each non-overlaping patch in case of patchifying convolution. Should be a multiple of 4.
#### Cite the following papers
```
@misc{
p2022wavemix,
title={WaveMix: Multi-Resolution Token Mixing for Images},
author={Pranav Jeevan P and Amit Sethi},
year={2022},
url={https://openreview.net/forum?id=tBoSm4hUWV}
}
@misc{jeevan2022wavemix,
title={WaveMix: Resource-efficient Token Mixing for Images},
author={Pranav Jeevan and Amit Sethi},
year={2022},
eprint={2203.03689},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
@misc{jeevan2023wavemix,
title={WaveMix: A Resource-efficient Neural Network for Image Analysis},
author={Pranav Jeevan and Kavitha Viswanathan and Anandu A S and Amit Sethi},
year={2023},
eprint={2205.14375},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/pranavphoenix/WaveMix",
"name": "wavemix",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "artificial intelligence,token-mixing,image classification,semantic segmentation,image super-resolution",
"author": "Pranav Jeevan",
"author_email": "pranav13phoenix@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/7a/05/816641d46281b06d745d4d4b35cd76409dd6baca9049fb439ee6c0fd114c/wavemix-0.2.4.tar.gz",
"platform": null,
"description": "# WaveMix\r\n[](https://paperswithcode.com/sota/image-classification-on-emnist-balanced?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/image-classification-on-emnist-byclass?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/image-classification-on-emnist-bymerge?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/image-classification-on-emnist-digits?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/image-classification-on-emnist-letters?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/image-classification-on-inat2021-mini?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/scene-classification-on-places365-standard?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/semantic-segmentation-on-cityscapes-val?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/image-classification-on-caltech-256?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/image-classification-on-places365-standard?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/image-classification-on-svhn?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/image-classification-on-tiny-imagenet-1?p=wavemix-lite-a-resource-efficient-neural)\r\n[](https://paperswithcode.com/sota/image-classification-on-fashion-mnist?p=wavemix-lite-a-resource-efficient-neural)\r\n\r\n \r\n \r\n\r\n\r\n## Resource-efficient Token Mixing for Images using 2D Discrete Wavelet Transform \r\n\r\n### WaveMix Architecture\r\n\r\n\r\n### WaveMix-Lite\r\n\r\n\r\nWe propose WaveMix\u00e2\u20ac\u201c a novel neural architecture for computer vision that is resource-efficient yet generalizable and scalable. WaveMix networks achieve comparable or better accuracy than the state-of-the-art convolutional neural networks, vision transformers, and token mixers for several tasks, establishing new benchmarks for segmentation on Cityscapes; and for classification on Places-365, f ive EMNIST datasets, and iNAT-mini. Remarkably, WaveMix architectures require fewer parameters to achieve these benchmarks compared to the previous state-of-the-art. Moreover, when controlled for the number of parameters, WaveMix requires lesser GPU RAM, which translates to savings in time, cost, and energy. To achieve these gains we used multi-level two-dimensional discrete wavelet transform (2D-DWT) in WaveMix blocks, which has the following advantages: (1) It reorganizes spatial information based on three strong image priors\u00e2\u20ac\u201c scale-invariance, shift-invariance, and sparseness of edges, (2) in a lossless manner without adding parameters, (3) while also reducing the spatial sizes of feature maps, which reduces the memory and time required for forward and backward passes, and (4) expanding the receptive field faster than convolutions do. The whole architecture is a stack of self-similar and resolution-preserving WaveMix blocks, which allows architectural f lexibility for various tasks and levels of resource availability.\r\n\r\n\r\n| Task | Dataset | Metric | Value |\r\n|-----------------------|-------------|----------|--------|\r\n| Semantic Segmentation | Cityscapes | Single-scale mIoU | 82.70% (SOTA) |\r\n| Image Classification | ImageNet-1k | Accuracy | 74.93% |\r\n\r\n### Parameter Efficiency\r\n| Task | Model | Parameters |\r\n|------------------------------|-------------------------------------------------|------------|\r\n| 99% Accu. in MNIST | WaveMix Lite-8/10 | 3566 |\r\n| 90% Accu. in Fashion MNIST | WaveMix Lite-8/5 | 7156 |\r\n| 80% Accu. in CIFAR-10 | WaveMix Lite-32/7 | 37058 |\r\n| 90% Accu. in CIFAR-10 | WaveMix Lite-64/6 | 520106 | \r\n\r\nThe high parameter efficiency is obtained by replacing Deconvolution layers with Upsampling\r\n\r\nThis is an implementation of code from the following papers : [Openreview Paper](https://openreview.net/forum?id=tBoSm4hUWV), [ArXiv Paper 1](https://arxiv.org/abs/2203.03689), [ArXiv Paper 2](https://arxiv.org/abs/2205.14375)\r\n\r\n## Install\r\n\r\n```bash\r\n$ pip install wavemix\r\n```\r\n\r\n## Usage\r\n### Semantic Segmentation\r\n\r\n```python\r\nimport torch, wavemix\r\nfrom wavemix.SemSegment import WaveMix\r\nimport torch\r\n\r\nmodel = WaveMix(\r\n num_classes= 20, \r\n depth= 16,\r\n mult= 2,\r\n ff_channel= 256,\r\n final_dim= 256,\r\n dropout= 0.5,\r\n level=4,\r\n stride=2\r\n)\r\n\r\nimg = torch.randn(1, 3, 256, 256)\r\n\r\npreds = model(img) # (1, 20, 256, 256)\r\n```\r\n\r\n### Image Classification\r\n\r\n```python\r\nimport torch, wavemix\r\nfrom wavemix.classification import WaveMix\r\nimport torch\r\n\r\nmodel = WaveMix(\r\n num_classes= 1000, \r\n depth= 16,\r\n mult= 2,\r\n ff_channel= 192,\r\n final_dim= 192,\r\n dropout= 0.5,\r\n level=3,\r\n patch_size=4,\r\n)\r\nimg = torch.randn(1, 3, 256, 256)\r\n\r\npreds = model(img) # (1, 1000)\r\n```\r\n\r\n### Single Image Super-resolution\r\n\r\n```python\r\nimport wavemix, torch\r\nfrom wavemix.sisr import WaveMix\r\n\r\nmodel = WaveMix(\r\n depth = 4,\r\n mult = 2,\r\n ff_channel = 144,\r\n final_dim = 144,\r\n dropout = 0.5,\r\n level=1,\r\n)\r\n\r\nimg = torch.randn(1, 3, 256, 256)\r\nout = model(img) # (1, 3, 512, 512)\r\n```\r\n\r\n### To use a single Waveblock\r\n\r\n```python\r\nimport wavemix, torch\r\nfrom wavemix import Level1Waveblock\r\n\r\n```\r\n\r\n## Parameters\r\n\r\n- `num_classes`: int. \r\nNumber of classes to classify/segment.\r\n- `depth`: int. \r\nNumber of WaveMix blocks.\r\n- `mult`: int. \r\nExpansion of channels in the MLP (FeedForward) layer. \r\n- `ff_channel`: int. \r\nNo. of output channels from the MLP (FeedForward) layer. \r\n- `final_dim`: int. \r\nFinal dimension of output tensor after initial Conv layers. Channel dimension when tensor is fed to WaveBlocks.\r\n- `dropout`: float between `[0, 1]`, default `0.`. \r\nDropout rate. \r\n- `level`: int. \r\nNumber of levels of 2D wavelet transform to be used in Waveblocks. Currently supports levels from 1 to 4.\r\n- `stride`: int. \r\nStride used in the initial convolutional layers to reduce the input resolution before being fed to Waveblocks. \r\n- `initial_conv`: str. \r\nDeciding between strided convolution or patchifying convolutions in the intial conv layer. Used for classification. 'pachify' or 'strided'.\r\n- `patch_size`: int. \r\nSize of each non-overlaping patch in case of patchifying convolution. Should be a multiple of 4.\r\n\r\n\r\n#### Cite the following papers \r\n```\r\n@misc{\r\np2022wavemix,\r\ntitle={WaveMix: Multi-Resolution Token Mixing for Images},\r\nauthor={Pranav Jeevan P and Amit Sethi},\r\nyear={2022},\r\nurl={https://openreview.net/forum?id=tBoSm4hUWV}\r\n}\r\n\r\n@misc{jeevan2022wavemix,\r\n title={WaveMix: Resource-efficient Token Mixing for Images},\r\n author={Pranav Jeevan and Amit Sethi},\r\n year={2022},\r\n eprint={2203.03689},\r\n archivePrefix={arXiv},\r\n primaryClass={cs.CV}\r\n}\r\n\r\n@misc{jeevan2023wavemix,\r\n title={WaveMix: A Resource-efficient Neural Network for Image Analysis}, \r\n author={Pranav Jeevan and Kavitha Viswanathan and Anandu A S and Amit Sethi},\r\n year={2023},\r\n eprint={2205.14375},\r\n archivePrefix={arXiv},\r\n primaryClass={cs.CV}\r\n}\r\n\r\n```\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "WaveMix - Pytorch",
"version": "0.2.4",
"project_urls": {
"Homepage": "https://github.com/pranavphoenix/WaveMix"
},
"split_keywords": [
"artificial intelligence",
"token-mixing",
"image classification",
"semantic segmentation",
"image super-resolution"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ef94c48cc277716e532206649041e882ec359322dc9d8f04b04c9f8a6749d2f3",
"md5": "94720304d910b30948cf8ab3b2c6309b",
"sha256": "99660deafe6eb13e3a464b1d5e26a133b3accc72dae2592ecff075d59b8510b1"
},
"downloads": -1,
"filename": "wavemix-0.2.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "94720304d910b30948cf8ab3b2c6309b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 11577,
"upload_time": "2023-05-04T11:02:05",
"upload_time_iso_8601": "2023-05-04T11:02:05.108585Z",
"url": "https://files.pythonhosted.org/packages/ef/94/c48cc277716e532206649041e882ec359322dc9d8f04b04c9f8a6749d2f3/wavemix-0.2.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7a05816641d46281b06d745d4d4b35cd76409dd6baca9049fb439ee6c0fd114c",
"md5": "57f2cced12f0c40df9e091ba5ed35b97",
"sha256": "ffcfbdb15ac64075e9466c11748ff3d5c0893a56c2b27ba72dc2d7499e46099e"
},
"downloads": -1,
"filename": "wavemix-0.2.4.tar.gz",
"has_sig": false,
"md5_digest": "57f2cced12f0c40df9e091ba5ed35b97",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 13562,
"upload_time": "2023-05-04T11:02:07",
"upload_time_iso_8601": "2023-05-04T11:02:07.354531Z",
"url": "https://files.pythonhosted.org/packages/7a/05/816641d46281b06d745d4d4b35cd76409dd6baca9049fb439ee6c0fd114c/wavemix-0.2.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-04 11:02:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pranavphoenix",
"github_project": "WaveMix",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "wavemix"
}
JSON
