# APEC and MAPEC Activation Functions for Neural Networks
- [APEC and MAPEC Activation Functions for Neural Networks](#apec-and-mapec-activation-functions-for-neural-networks)
- [Overview](#overview)
- [Installation](#installation)
- [Usage](#usage)
- [Activation Functions](#activation-functions)
- [Mathematical Formulation](#mathematical-formulation)
- [APEC (Asymmetric Parametric Exponential Curvature)](#apec-asymmetric-parametric-exponential-curvature)
- [MAPEC (Multiplicative Asymmetric Parametric Exponential Curvature)](#mapec-multiplicative-asymmetric-parametric-exponential-curvature)
- [Evaluation](#evaluation)
- [Results](#results)
- [Contributing](#contributing)
- [License](#license)
## Overview
This repository introduces two novel activation functions, APEC (Asymmetric Parametric Exponential Curvature) and its variant MAPEC (Multiplicative APEC), designed for deep learning models to capture complex patterns with improved performance. Functions have been tested on the CIFAR-100 dataset (results included) and on some of my experimental models (results not included).
## Installation
```bash
pip install apec-afn
```
## Usage
```python
import torch
from apec import MAPEC
x = torch.randn([8])
f = MAPEC()
print(f(x))
```
## Activation Functions
- **APEC**: Offers a balance between flexibility and performance, as demonstrated by the improvement over traditional functions on CIFAR-100.
- **MAPEC**: An extension of APEC with an additional multiplicative term, allowing for an even richer model expressiveness and an observed faster convergence (up to 15%).
## Mathematical Formulation
### APEC (Asymmetric Parametric Exponential Curvature)
APEC is designed to introduce a non-linear response with an adjustable curvature, defined by:
$$f(x) = \alpha + \frac{\beta - x}{(\gamma - \exp(-x)) + \epsilon}$$
- **Initialization**: Parameters `a` and `b` are initialized with a normal distribution of zero mean and a standard deviation of 0.35. Parameter `g` is initialized with a mean of -1.375 and a standard deviation of 0.35.
- **Constraints**: The default constraints for `a`, `b`, and `g` are [-2.0, +2.0], [-2.5, +2.5], and [-2.5, -0.25], respectively.
- **Stability**: A small constant `eps` (1.0e-5) is added to prevent division by zero.
### MAPEC (Multiplicative Asymmetric Parametric Exponential Curvature)
MAPEC extends APEC by adding a multiplicative term, enhancing its flexibility:
$$f(x) = (\alpha + \frac{\beta - x}{-abs(\gamma) - \exp(-x) - \epsilon} + (x \cdot \delta)) \cdot \zeta$$
- **Initialization**: Parameters initialization values are -3.333e-2, -0.1, -2.0, +0.1 and +1.0 for alpha, beta, gamma, delta and zeta respectively.
- **Constraints**: There are no constraints on the parameters for MAPEC, allowing for a fully adaptive response.
- **Stability**: A small constant `eps` (1.0e-3) is subtracted from denominator to prevent division by zero.
These functions aim to provide enhanced flexibility and adaptability for neural networks, particularly beneficial for complex pattern recognition tasks.
## Evaluation
To evaluate a model with a specific activation function on CIFAR-100 and plot _training loss*_, use:
```bash
python scripts/eval_cifar100.py --activation APEC --plot-loss
```
_* Plotting training loss requires `self-projection` package to be installed._
## Results
Evaluation results on CIFAR-100:
| Activation | Average Loss | Accuracy |
| ---------- | ------------ | -------- |
| MAPEC 16e* | 2.2004 | 43% |
| APEC | 2.2235 | 43% |
| MAPEC 20e* | 2.2456 | 43% |
| Mish | 2.2704 | 43% |
| SELU | 2.2674 | 42% |
| PReLU | 2.2759 | 42% |
| ReLU | 2.3933 | 39% |
_* Results provided for training with MAPEC activation for 20 and 16 epochs respectively._
**APEC** leads to the best performance, closely followed by Mish and SELU.
**MAPEC** leads to the faster convergence with performance closely followed by APEC.
You could look at training loss plots [here](doc/plots.md).
## Contributing
Contributions and suggestions are welcome! Feel free to fork the repository, open issues, and submit pull requests.
## License
`APEC` is released under the MIT License. See the `LICENSE` file for more details.
Raw data
{
"_id": null,
"home_page": "https://github.com/Sombressoul/apec",
"name": "apec-afn",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10.0",
"maintainer_email": "",
"keywords": "machine learning,deep learning,pytorch layers,pytorch modules",
"author": "Konstantin Bashinskiy",
"author_email": "sombressoul@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/0a/9c/be0839e243270cbb2ae98823ccbf7b1a2d445c60f778f8bbb3742ca73bbf/apec-afn-0.1.3.tar.gz",
"platform": null,
"description": "# APEC and MAPEC Activation Functions for Neural Networks\n\n- [APEC and MAPEC Activation Functions for Neural Networks](#apec-and-mapec-activation-functions-for-neural-networks)\n - [Overview](#overview)\n - [Installation](#installation)\n - [Usage](#usage)\n - [Activation Functions](#activation-functions)\n - [Mathematical Formulation](#mathematical-formulation)\n - [APEC (Asymmetric Parametric Exponential Curvature)](#apec-asymmetric-parametric-exponential-curvature)\n - [MAPEC (Multiplicative Asymmetric Parametric Exponential Curvature)](#mapec-multiplicative-asymmetric-parametric-exponential-curvature)\n - [Evaluation](#evaluation)\n - [Results](#results)\n - [Contributing](#contributing)\n - [License](#license)\n\n## Overview\nThis repository introduces two novel activation functions, APEC (Asymmetric Parametric Exponential Curvature) and its variant MAPEC (Multiplicative APEC), designed for deep learning models to capture complex patterns with improved performance. Functions have been tested on the CIFAR-100 dataset (results included) and on some of my experimental models (results not included).\n\n## Installation\n```bash\npip install apec-afn\n```\n\n## Usage\n```python\nimport torch\nfrom apec import MAPEC\n\nx = torch.randn([8])\nf = MAPEC()\n\nprint(f(x))\n```\n\n## Activation Functions\n- **APEC**: Offers a balance between flexibility and performance, as demonstrated by the improvement over traditional functions on CIFAR-100.\n- **MAPEC**: An extension of APEC with an additional multiplicative term, allowing for an even richer model expressiveness and an observed faster convergence (up to 15%).\n\n## Mathematical Formulation\n\n### APEC (Asymmetric Parametric Exponential Curvature)\n\n\nAPEC is designed to introduce a non-linear response with an adjustable curvature, defined by:\n$$f(x) = \\alpha + \\frac{\\beta - x}{(\\gamma - \\exp(-x)) + \\epsilon}$$\n\n- **Initialization**: Parameters `a` and `b` are initialized with a normal distribution of zero mean and a standard deviation of 0.35. Parameter `g` is initialized with a mean of -1.375 and a standard deviation of 0.35.\n- **Constraints**: The default constraints for `a`, `b`, and `g` are [-2.0, +2.0], [-2.5, +2.5], and [-2.5, -0.25], respectively.\n- **Stability**: A small constant `eps` (1.0e-5) is added to prevent division by zero.\n\n### MAPEC (Multiplicative Asymmetric Parametric Exponential Curvature)\n\n\nMAPEC extends APEC by adding a multiplicative term, enhancing its flexibility:\n$$f(x) = (\\alpha + \\frac{\\beta - x}{-abs(\\gamma) - \\exp(-x) - \\epsilon} + (x \\cdot \\delta)) \\cdot \\zeta$$\n\n- **Initialization**: Parameters initialization values are -3.333e-2, -0.1, -2.0, +0.1 and +1.0 for alpha, beta, gamma, delta and zeta respectively.\n- **Constraints**: There are no constraints on the parameters for MAPEC, allowing for a fully adaptive response.\n- **Stability**: A small constant `eps` (1.0e-3) is subtracted from denominator to prevent division by zero.\n\nThese functions aim to provide enhanced flexibility and adaptability for neural networks, particularly beneficial for complex pattern recognition tasks.\n\n## Evaluation\nTo evaluate a model with a specific activation function on CIFAR-100 and plot _training loss*_, use:\n```bash\npython scripts/eval_cifar100.py --activation APEC --plot-loss\n```\n\n_* Plotting training loss requires `self-projection` package to be installed._\n\n## Results\nEvaluation results on CIFAR-100:\n\n| Activation | Average Loss | Accuracy |\n| ---------- | ------------ | -------- |\n| MAPEC 16e* | 2.2004 | 43% |\n| APEC | 2.2235 | 43% |\n| MAPEC 20e* | 2.2456 | 43% |\n| Mish | 2.2704 | 43% |\n| SELU | 2.2674 | 42% |\n| PReLU | 2.2759 | 42% |\n| ReLU | 2.3933 | 39% |\n\n_* Results provided for training with MAPEC activation for 20 and 16 epochs respectively._\n\n\n**APEC** leads to the best performance, closely followed by Mish and SELU.\n\n**MAPEC** leads to the faster convergence with performance closely followed by APEC.\n\nYou could look at training loss plots [here](doc/plots.md).\n\n## Contributing\nContributions and suggestions are welcome! Feel free to fork the repository, open issues, and submit pull requests.\n\n## License\n\n`APEC` is released under the MIT License. See the `LICENSE` file for more details.\n",
"bugtrack_url": null,
"license": "Apache",
"summary": "APEC (Asymmetric Parametric Exponential Curvature) activation function.",
"version": "0.1.3",
"project_urls": {
"Homepage": "https://github.com/Sombressoul/apec"
},
"split_keywords": [
"machine learning",
"deep learning",
"pytorch layers",
"pytorch modules"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "16926c5c99e524b6c053671bbcf7d6b8b8aa9b5b7eeab5d56413a9d26d71cf63",
"md5": "c60baa70fad52eb39e1af59539fa7e1f",
"sha256": "38e1b7206f3e8e81963502510bb3d989fb7cd5ae05167fd2b2c7ee1377df683a"
},
"downloads": -1,
"filename": "apec_afn-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c60baa70fad52eb39e1af59539fa7e1f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10.0",
"size": 5561,
"upload_time": "2024-02-06T07:39:26",
"upload_time_iso_8601": "2024-02-06T07:39:26.656229Z",
"url": "https://files.pythonhosted.org/packages/16/92/6c5c99e524b6c053671bbcf7d6b8b8aa9b5b7eeab5d56413a9d26d71cf63/apec_afn-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0a9cbe0839e243270cbb2ae98823ccbf7b1a2d445c60f778f8bbb3742ca73bbf",
"md5": "2436037b9bb340c2d4bd1eb63cf58130",
"sha256": "d1b2f4ae08a7e2c8d5beb3e23da245339f202e6c17df93a60acb24f07fe2a7fd"
},
"downloads": -1,
"filename": "apec-afn-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "2436037b9bb340c2d4bd1eb63cf58130",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10.0",
"size": 5410,
"upload_time": "2024-02-06T07:39:28",
"upload_time_iso_8601": "2024-02-06T07:39:28.423241Z",
"url": "https://files.pythonhosted.org/packages/0a/9c/be0839e243270cbb2ae98823ccbf7b1a2d445c60f778f8bbb3742ca73bbf/apec-afn-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-06 07:39:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Sombressoul",
"github_project": "apec",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "apec-afn"
}
JSON
