| Name | autolrtuner JSON |
| Version |
1.0.2
JSON |
| download |
| home_page | None |
| Summary | Heuristically optimize learning rates in neural networks through subsampling loss differentials. |
| upload_time | 2024-12-20 11:02:15 |
| maintainer | None |
| docs_url | None |
| author | Adam Wu |
| requires_python | >=3.8 |
| license | MIT License Copyright (c) [2024] [adamvvu] Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
tensorflow
hyperparameter tuning
neural networks
deep learning
learning rate
machine learning
artificial intelligence
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
numpy
pandas
scipy
matplotlib
tensorflow
tqdm
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
[](https://github.com/adamvvu/autolrtuner/actions/workflows/autolrtuner_tests.yml)
[](https://pypi.org/project/autolrtuner/)
[](https://pypi.org/project/autolrtuner/)
[](https://github.com/adamvvu/autolrtuner/blob/master/LICENSE)
Heuristically optimize learning rates in neural networks through subsampling loss differentials.
---
## Auto LR Tuner
The learning rate is often one of the most important hyperparameters when training neural networks. Adaptive gradient-based methods (e.g. ADAM), decreasing learning rates based on a validation set, and cosine annealing are common tricks done in practice to improve convergence.
This library provides a simple algorithm for automatically tuning learning rates for TensorFlow Keras models. Importantly, these methods are largely based on heuristics and my own experience training neural networks and there are no formal results.
### Algorithm Details
The main idea behind this implementation is to estimate the optimal learning rate by trying to determine the steepness of the loss surface. Intuitively, a very small learning rate leads to almost no change in losses, while an excessively large learning rate can overshoot local minima and even increase the loss.
We start with a user-specified grid $\Theta = [lr_{min}, lr_{max}]$ of potential learning rates to search over. Next, we subsample $m$ observations from the data, and evaluate $\Delta L := L_{post} - L_{pre}$ where $L_{pre}$ is the baseline loss value, and $L_{post}$ is the loss after a single backpropagation step for a given learning rate $\theta \in \Theta$. The subsampling process is repeated $M$ times to approximate $E_{M}[\Delta L]$ and construct confidence intervals based on the subsampled distribution $\hat{P}_{\Delta L, M}$.
In this interpretation of the problem, the "optimal" learning rate is the one that consistently decreases the loss across the dataset without excessively high variance. For example in the plot below, a small learning rate near-zero may be very slow to converge while a learning rate of one actually increases the loss on average. Here the optimal rate may be something close to 0.125.
### Getting Started
Install from PyPi:
`$ pip install autolrtuner`
#### Dependencies
- `python >= 3.8`
- `tensorflow >= 2.5`
- `numpy`
- `pandas`
- `scipy`
- `tqdm`
- `matplotlib` *(Optional)*
#### Example
```
from autolrtuner import AutoLRTuner
# Compiled TensorFlow Keras model
model
# Run the tuning algorithm
lrTuner = AutoLRTuner(model)
lrTuner.Tune(X, Y, num_subsamples=100, num_evals=100, batch_size=32)
# Get the optimal learning rate
bestLR = lrTuner.GetBestLR(method='mean')
# Plot the loss differentials
lrTuner.Plot()
```
Raw data
{
"_id": null,
"home_page": null,
"name": "autolrtuner",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "tensorflow, hyperparameter tuning, neural networks, deep learning, learning rate, machine learning, artificial intelligence",
"author": "Adam Wu",
"author_email": "adamwu1@outlook.com",
"download_url": "https://files.pythonhosted.org/packages/63/3f/3b8c22846f5b20423ad4dc8cf1ab6e93b68a09c5be5720bd0d9dc1a447af/autolrtuner-1.0.2.tar.gz",
"platform": null,
"description": "[](https://github.com/adamvvu/autolrtuner/actions/workflows/autolrtuner_tests.yml)\r\n[](https://pypi.org/project/autolrtuner/)\r\n[](https://pypi.org/project/autolrtuner/)\r\n[](https://github.com/adamvvu/autolrtuner/blob/master/LICENSE)\r\n\r\nHeuristically optimize learning rates in neural networks through subsampling loss differentials.\r\n\r\n---\r\n\r\n## Auto LR Tuner\r\n\r\nThe learning rate is often one of the most important hyperparameters when training neural networks. Adaptive gradient-based methods (e.g. ADAM), decreasing learning rates based on a validation set, and cosine annealing are common tricks done in practice to improve convergence.\r\n\r\nThis library provides a simple algorithm for automatically tuning learning rates for TensorFlow Keras models. Importantly, these methods are largely based on heuristics and my own experience training neural networks and there are no formal results.\r\n\r\n### Algorithm Details\r\n\r\nThe main idea behind this implementation is to estimate the optimal learning rate by trying to determine the steepness of the loss surface. Intuitively, a very small learning rate leads to almost no change in losses, while an excessively large learning rate can overshoot local minima and even increase the loss.\r\n\r\nWe start with a user-specified grid $\\Theta = [lr_{min}, lr_{max}]$ of potential learning rates to search over. Next, we subsample $m$ observations from the data, and evaluate $\\Delta L := L_{post} - L_{pre}$ where $L_{pre}$ is the baseline loss value, and $L_{post}$ is the loss after a single backpropagation step for a given learning rate $\\theta \\in \\Theta$. The subsampling process is repeated $M$ times to approximate $E_{M}[\\Delta L]$ and construct confidence intervals based on the subsampled distribution $\\hat{P}_{\\Delta L, M}$.\r\n\r\nIn this interpretation of the problem, the \"optimal\" learning rate is the one that consistently decreases the loss across the dataset without excessively high variance. For example in the plot below, a small learning rate near-zero may be very slow to converge while a learning rate of one actually increases the loss on average. Here the optimal rate may be something close to 0.125. \r\n\r\n\r\n\r\n### Getting Started\r\n\r\nInstall from PyPi:\r\n\r\n`$ pip install autolrtuner`\r\n\r\n#### Dependencies\r\n\r\n- `python >= 3.8`\r\n- `tensorflow >= 2.5`\r\n- `numpy`\r\n- `pandas`\r\n- `scipy`\r\n- `tqdm`\r\n- `matplotlib` *(Optional)*\r\n\r\n#### Example\r\n\r\n```\r\nfrom autolrtuner import AutoLRTuner\r\n\r\n# Compiled TensorFlow Keras model\r\nmodel \r\n\r\n# Run the tuning algorithm\r\nlrTuner = AutoLRTuner(model)\r\nlrTuner.Tune(X, Y, num_subsamples=100, num_evals=100, batch_size=32)\r\n\r\n# Get the optimal learning rate\r\nbestLR = lrTuner.GetBestLR(method='mean')\r\n\r\n# Plot the loss differentials\r\nlrTuner.Plot()\r\n```\r\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) [2024] [adamvvu] Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "Heuristically optimize learning rates in neural networks through subsampling loss differentials.",
"version": "1.0.2",
"project_urls": {
"documentation": "https://github.com/adamvvu/autolrtuner",
"homepage": "https://github.com/adamvvu/autolrtuner",
"repository": "https://github.com/adamvvu/autolrtuner"
},
"split_keywords": [
"tensorflow",
" hyperparameter tuning",
" neural networks",
" deep learning",
" learning rate",
" machine learning",
" artificial intelligence"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "64af901b3d0906ff4dcceaaffcad023833b5d0fa495da0341f5f73c8e75f5ee4",
"md5": "e8de7ff644c89300bcd8a4180dd0c52f",
"sha256": "ab50b3be740128a7cd226f5f4845d4ad77715623b24166edb9c550ab5c5dc1bb"
},
"downloads": -1,
"filename": "autolrtuner-1.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e8de7ff644c89300bcd8a4180dd0c52f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 7541,
"upload_time": "2024-12-20T11:02:13",
"upload_time_iso_8601": "2024-12-20T11:02:13.861857Z",
"url": "https://files.pythonhosted.org/packages/64/af/901b3d0906ff4dcceaaffcad023833b5d0fa495da0341f5f73c8e75f5ee4/autolrtuner-1.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "633f3b8c22846f5b20423ad4dc8cf1ab6e93b68a09c5be5720bd0d9dc1a447af",
"md5": "b17dfafbc540898cf7ebc291a8683d8f",
"sha256": "dad8a89e612565cb9d1629899a0636d42e4b68e42739dcc2ee5207b5886d5928"
},
"downloads": -1,
"filename": "autolrtuner-1.0.2.tar.gz",
"has_sig": false,
"md5_digest": "b17dfafbc540898cf7ebc291a8683d8f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 7480,
"upload_time": "2024-12-20T11:02:15",
"upload_time_iso_8601": "2024-12-20T11:02:15.247170Z",
"url": "https://files.pythonhosted.org/packages/63/3f/3b8c22846f5b20423ad4dc8cf1ab6e93b68a09c5be5720bd0d9dc1a447af/autolrtuner-1.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-20 11:02:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "adamvvu",
"github_project": "autolrtuner",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "numpy",
"specs": []
},
{
"name": "pandas",
"specs": []
},
{
"name": "scipy",
"specs": []
},
{
"name": "matplotlib",
"specs": []
},
{
"name": "tensorflow",
"specs": [
[
">=",
"2.5"
]
]
},
{
"name": "tqdm",
"specs": []
}
],
"lcname": "autolrtuner"
}
