# EvNN: Event-based Neural Networks
EvNN is a CUDA and C++ implementation of event-based RNN layers with built-in [DropConnect](http://proceedings.mlr.press/v28/wan13.html) and [Zoneout](https://arxiv.org/abs/1606.01305) regularization. These layers are exposed through C++ and Pytorch APIs for easy integration into your own projects or machine learning frameworks. The code framework and base layers are adopted from [Haste](https://github.com/lmnt-com/haste/) Library.
## EGRU: Event-based Gated Recurrent Unit
Event based GRU was publised as a conference paper at ICLR 2023: [Efficient recurrent architectures through activity sparsity and sparse back-propagation through time](https://openreview.net/pdf?id=lJdOlWg8td) (**notable-top-25%**)
*Illustration of EGRU.
**A.** A single unit of the original GRU model adapted from [Cho et al.](#references).
**B:** EGRU unit with event generating mechanism.
**C:** Heaviside function and surrogate gradient.
**D:** Forward state dynamics for two EGRU units (*$i$ *and* $j$ *).
**E:** Activity-sparse backward dynamics for two EGRU units (*$i$ *and* $j$ *).
(Note that we only have to backpropagate through units that were active or whose state was close to the threshold at each time step.)*
Which RNN types are currently supported?
- [GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit)
What's included in this project?
- a PyTorch API (`evnn_pytorch`) for event based neural networks
## Install
Here's what you'll need to get started:
- a [CUDA Compute Capability](https://developer.nvidia.com/cuda-gpus) 3.7+ GPU (required only if using GPU)
- [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit) 10.0+ (required only if using GPU)
- [PyTorch](https://pytorch.org) 1.3+ for PyTorch integration (GPU optional)
- [BLAS](https://netlib.org/blas/) or any BLAS-like library for CPU computation.
- [Eigen 3](http://eigen.tuxfamily.org/) to build the C++ examples (optional)
Once you have the prerequisites, you can install with pip or by building the source code.
<!-- ### Using pip
```
pip install evnn_pytorch
``` -->
### Building from source
> **Note**
>
> Currenty supported only on Linux, use Docker for building on Windows.
```bash
make evnn_pytorch # Build PyTorch API
```
If you built the PyTorch API, install it with `pip`:
```bash
pip install evnn_pytorch-*.whl
```
If the CUDA Toolkit that you're building against is not in `/usr/local/cuda`, you must specify the
`$CUDA_HOME` environment variable before running make:
```bash
CUDA_HOME=/usr/local/cuda-10.2 make
```
## Performance
Code for the experiments and benchmarks presented in the paper are published in ``benchmarks`` directory.
Note that these benchmarks have additional dependencies as documented in `benchmarks/requirements.txt`
## Documentation
### PyTorch API
```python
import torch
import evnn_pytorch as evnn
# setting use_custom_cuda=False makes the model use pytorch code instead of EvNN extension
egru_layer = evnn.EGRU(input_size, hidden_size, zoneout=0.0, batch_first=True,
use_custom_cuda=True)
egru_layer.cuda()
# `x` is a CUDA tensor with shape [N,T,C]
x = torch.rand([5, 25, 128]).cuda()
y, state = egru_layer(x)
```
The PyTorch API is documented in [`docs/pytorch/evnn_pytorch.md`](docs/pytorch/evnn_pytorch.md).
## Code layout
- [`docs/pytorch/`](docs/pytorch): API reference documentation for `evnn_pytorch`
- [`frameworks/pytorch/`](frameworks/pytorch): PyTorch API and custom op code
- [`lib/`](lib): CUDA kernels and C++ API
- [`validation/`](validation): scripts and tests to validate output and gradients of RNN layers
- [`benchmarks/`](benchmarks): Experiments from ICLR 2023 Paper.
## Testing
use python unittest with this command
- Numpy is required for testing
```
python -m unittest discover -p '*_test.py' -s validation
```
> **Note**
>
> Tests will fail if you set the the dimensions (batch_size,time_steps,input_size,
hidden_size) too high, this is because floating point errors can accumulate and cause the units to generate events one timestep off. This causes the numerical tests to fail but the Neural Network training will work without any issues.
## Implementation notes
- the EGRU is based on Haste GRU which is in turn based on `1406.1078v1` (same as cuDNN) rather than `1406.1078v3`
## References
1. Nanavati, Sharvil, ‘Haste: A Fast, Simple, and Open RNN Library’, 2020 <https://github.com/lmnt-com/haste/>
1. K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar, Oct. 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1179. [URL](https://aclanthology.org/D14-1179)
<!-- if we have tables from the paper here, then add references -->
## Citing this work
To cite this work, please use the following BibTeX entry:
```
@inproceedings{
evnn2023,
title={Efficient recurrent architectures through activity sparsity and sparse back-propagation through time},
author={Anand Subramoney and Khaleelulla Khan Nazeer and Mark Sch{\"o}ne and Christian Mayr and David Kappel},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=lJdOlWg8td},
howpublished={https://github.com/KhaleelKhan/EvNN/}
}
```
## License
[Apache 2.0](LICENSE)
Raw data
{
"_id": null,
"home_page": "https://tu-dresden.de/ing/elektrotechnik/iee/hpsn",
"name": "evnn-pytorch",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "pytorch machine learning rnn lstm gru custom op",
"author": "TUD and RUB",
"author_email": "khaleelulla.khan_nazeer@tu-dresden.de",
"download_url": "https://files.pythonhosted.org/packages/f0/d8/63be06a6d4010245c486639d9121ff49abba699109733c1c71374e6da345/evnn_pytorch-0.1.0.tar.gz",
"platform": null,
"description": "# EvNN: Event-based Neural Networks\n\nEvNN is a CUDA and C++ implementation of event-based RNN layers with built-in [DropConnect](http://proceedings.mlr.press/v28/wan13.html) and [Zoneout](https://arxiv.org/abs/1606.01305) regularization. These layers are exposed through C++ and Pytorch APIs for easy integration into your own projects or machine learning frameworks. The code framework and base layers are adopted from [Haste](https://github.com/lmnt-com/haste/) Library.\n\n## EGRU: Event-based Gated Recurrent Unit\n\nEvent based GRU was publised as a conference paper at ICLR 2023: [Efficient recurrent architectures through activity sparsity and sparse back-propagation through time](https://openreview.net/pdf?id=lJdOlWg8td) (**notable-top-25%**)\n\n\n\n*Illustration of EGRU. \n**A.** A single unit of the original GRU model adapted from [Cho et al.](#references). \n**B:** EGRU unit with event generating mechanism. \n**C:** Heaviside function and surrogate gradient.\n**D:** Forward state dynamics for two EGRU units (*$i$ *and* $j$ *).\n**E:** Activity-sparse backward dynamics for two EGRU units (*$i$ *and* $j$ *). \n(Note that we only have to backpropagate through units that were active or whose state was close to the threshold at each time step.)*\n\n\n\nWhich RNN types are currently supported?\n- [GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit)\n\nWhat's included in this project?\n- a PyTorch API (`evnn_pytorch`) for event based neural networks\n\n\n## Install\nHere's what you'll need to get started:\n- a [CUDA Compute Capability](https://developer.nvidia.com/cuda-gpus) 3.7+ GPU (required only if using GPU)\n- [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit) 10.0+ (required only if using GPU)\n- [PyTorch](https://pytorch.org) 1.3+ for PyTorch integration (GPU optional)\n- [BLAS](https://netlib.org/blas/) or any BLAS-like library for CPU computation.\n- [Eigen 3](http://eigen.tuxfamily.org/) to build the C++ examples (optional)\n\nOnce you have the prerequisites, you can install with pip or by building the source code.\n\n<!-- ### Using pip\n```\npip install evnn_pytorch\n``` -->\n\n### Building from source\n> **Note**\n> \n> Currenty supported only on Linux, use Docker for building on Windows.\n\n```bash\nmake evnn_pytorch # Build PyTorch API\n```\n\nIf you built the PyTorch API, install it with `pip`:\n```bash\npip install evnn_pytorch-*.whl\n```\n\nIf the CUDA Toolkit that you're building against is not in `/usr/local/cuda`, you must specify the\n`$CUDA_HOME` environment variable before running make:\n```bash\nCUDA_HOME=/usr/local/cuda-10.2 make\n```\n\n## Performance\n\nCode for the experiments and benchmarks presented in the paper are published in ``benchmarks`` directory.\nNote that these benchmarks have additional dependencies as documented in `benchmarks/requirements.txt`\n\n## Documentation\n\n### PyTorch API\n```python\nimport torch\nimport evnn_pytorch as evnn\n\n# setting use_custom_cuda=False makes the model use pytorch code instead of EvNN extension\negru_layer = evnn.EGRU(input_size, hidden_size, zoneout=0.0, batch_first=True,\n use_custom_cuda=True)\n\negru_layer.cuda()\n\n# `x` is a CUDA tensor with shape [N,T,C]\nx = torch.rand([5, 25, 128]).cuda()\n\ny, state = egru_layer(x)\n```\n\nThe PyTorch API is documented in [`docs/pytorch/evnn_pytorch.md`](docs/pytorch/evnn_pytorch.md).\n\n## Code layout\n- [`docs/pytorch/`](docs/pytorch): API reference documentation for `evnn_pytorch`\n- [`frameworks/pytorch/`](frameworks/pytorch): PyTorch API and custom op code\n- [`lib/`](lib): CUDA kernels and C++ API\n- [`validation/`](validation): scripts and tests to validate output and gradients of RNN layers\n- [`benchmarks/`](benchmarks): Experiments from ICLR 2023 Paper.\n\n## Testing\nuse python unittest with this command\n\n- Numpy is required for testing\n\n```\npython -m unittest discover -p '*_test.py' -s validation\n```\n> **Note**\n> \n> Tests will fail if you set the the dimensions (batch_size,time_steps,input_size,\nhidden_size) too high, this is because floating point errors can accumulate and cause the units to generate events one timestep off. This causes the numerical tests to fail but the Neural Network training will work without any issues.\n\n## Implementation notes\n- the EGRU is based on Haste GRU which is in turn based on `1406.1078v1` (same as cuDNN) rather than `1406.1078v3`\n\n## References\n1. Nanavati, Sharvil, \u2018Haste: A Fast, Simple, and Open RNN Library\u2019, 2020 <https://github.com/lmnt-com/haste/>\n\n1. K. Cho, B. van Merri\u00ebnboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder\u2013decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724\u20131734, Doha, Qatar, Oct. 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1179. [URL](https://aclanthology.org/D14-1179)\n\n<!-- if we have tables from the paper here, then add references -->\n\n## Citing this work\nTo cite this work, please use the following BibTeX entry:\n```\n@inproceedings{\nevnn2023,\ntitle={Efficient recurrent architectures through activity sparsity and sparse back-propagation through time},\nauthor={Anand Subramoney and Khaleelulla Khan Nazeer and Mark Sch{\\\"o}ne and Christian Mayr and David Kappel},\nbooktitle={The Eleventh International Conference on Learning Representations },\nyear={2023},\nurl={https://openreview.net/forum?id=lJdOlWg8td},\nhowpublished={https://github.com/KhaleelKhan/EvNN/}\n}\n```\n\n## License\n[Apache 2.0](LICENSE)\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "EVNN: a torch extension for custom event based RNN models.",
"version": "0.1.0",
"split_keywords": [
"pytorch",
"machine",
"learning",
"rnn",
"lstm",
"gru",
"custom",
"op"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f0d863be06a6d4010245c486639d9121ff49abba699109733c1c71374e6da345",
"md5": "10136932786f51dcac1882dd61b972f0",
"sha256": "f1d79b714d9b5ea33040f56a62c9686d4b0dc7c0bc5ab1f3a2713c432d5808dd"
},
"downloads": -1,
"filename": "evnn_pytorch-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "10136932786f51dcac1882dd61b972f0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 27321,
"upload_time": "2023-03-18T10:45:44",
"upload_time_iso_8601": "2023-03-18T10:45:44.991521Z",
"url": "https://files.pythonhosted.org/packages/f0/d8/63be06a6d4010245c486639d9121ff49abba699109733c1c71374e6da345/evnn_pytorch-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-18 10:45:44",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "evnn-pytorch"
}
JSON