| Name | evolutionary-policy-optimization JSON |
| Version |
0.2.15
JSON |
| download |
| home_page | None |
| Summary | EPO - Pytorch |
| upload_time | 2025-08-03 02:20:48 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT License
Copyright (c) 2025 Phil Wang
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE. |
| keywords |
artificial intelligence
deep learning
genetic algorithms
reinforcement learning
robotics
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
box2d-py
gymnasium
moviepy
numpy
tqdm
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<img width="450px" alt="fig1" src="https://github.com/user-attachments/assets/33bef569-e786-4f09-bdee-56bad7ea9e6d" />
## Evolutionary Policy Optimization
Pytorch implementation of [Evolutionary Policy Optimization](https://web3.arxiv.org/abs/2503.19037), from [Wang](https://www.jianrenw.com/) et al. of the Robotics Institute at Carnegie Mellon University
This paper stands out, as I have witnessed the positive effects first hand in an [exploratory project](https://github.com/lucidrains/firefly-torch) (mixing evolution with gradient based methods). Perhaps the Alexnet moment for genetic algorithms has not come to pass yet.
Besides their latent variable strategy, I'll also throw in some attempts with crossover in weight space
## Install
```bash
$ pip install evolutionary-policy-optimization
```
## Usage
```python
import torch
from evolutionary_policy_optimization import (
LatentGenePool,
Actor,
Critic
)
latent_pool = LatentGenePool(
num_latents = 128,
dim_latent = 32,
)
state = torch.randn(1, 32)
actor = Actor(dim_state = 32, dim = 256, mlp_depth = 2, num_actions = 4, dim_latent = 32)
critic = Critic(dim_state = 32, dim = 256, mlp_depth = 3, dim_latent = 32)
latent = latent_pool(latent_id = 2)
actions = actor(state, latent)
value = critic(state, latent)
# interact with environment and receive rewards, termination etc
# derive a fitness score for each gene / latent
fitness = torch.randn(128)
latent_pool.genetic_algorithm_step(fitness) # update latent genes with genetic algorithm
```
End to end learning
```python
import torch
from evolutionary_policy_optimization import (
create_agent,
EPO,
Env
)
agent = create_agent(
dim_state = 512,
num_latents = 16,
dim_latent = 32,
actor_num_actions = 5,
actor_dim = 256,
actor_mlp_depth = 2,
critic_dim = 256,
critic_mlp_depth = 3,
latent_gene_pool_kwargs = dict(
frac_natural_selected = 0.5
)
)
epo = EPO(
agent,
episodes_per_latent = 1,
max_episode_length = 10,
action_sample_temperature = 1.
)
env = Env((512,))
epo(agent, env, num_learning_cycles = 5)
# saving and loading
agent.save('./agent.pt', overwrite = True)
agent.load('./agent.pt')
```
## Contributing
At the project root, run
```bash
$ pip install '.[test]' # or `uv pip install '.[test]'`
```
Then add your tests to `tests/test_epo.py` and run
```bash
$ pytest tests/
```
That's it
## Citations
```bibtex
@inproceedings{Wang2025EvolutionaryPO,
title = {Evolutionary Policy Optimization},
author = {Jianren Wang and Yifan Su and Abhinav Gupta and Deepak Pathak},
year = {2025},
url = {https://api.semanticscholar.org/CorpusID:277313729}
}
```
```bibtex
@article{Farebrother2024StopRT,
title = {Stop Regressing: Training Value Functions via Classification for Scalable Deep RL},
author = {Jesse Farebrother and Jordi Orbay and Quan Ho Vuong and Adrien Ali Taiga and Yevgen Chebotar and Ted Xiao and Alex Irpan and Sergey Levine and Pablo Samuel Castro and Aleksandra Faust and Aviral Kumar and Rishabh Agarwal},
journal = {ArXiv},
year = {2024},
volume = {abs/2403.03950},
url = {https://api.semanticscholar.org/CorpusID:268253088}
}
```
```bibtex
@inproceedings{Khadka2018EvolutionGuidedPG,
title = {Evolution-Guided Policy Gradient in Reinforcement Learning},
author = {Shauharda Khadka and Kagan Tumer},
booktitle = {Neural Information Processing Systems},
year = {2018},
url = {https://api.semanticscholar.org/CorpusID:53096951}
}
```
```bibtex
@article{Fortunato2017NoisyNF,
title = {Noisy Networks for Exploration},
author = {Meire Fortunato and Mohammad Gheshlaghi Azar and Bilal Piot and Jacob Menick and Ian Osband and Alex Graves and Vlad Mnih and R{\'e}mi Munos and Demis Hassabis and Olivier Pietquin and Charles Blundell and Shane Legg},
journal = {ArXiv},
year = {2017},
volume = {abs/1706.10295},
url = {https://api.semanticscholar.org/CorpusID:5176587}
}
```
```bibtex
@article{Banerjee2022BoostingEI,
title = {Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel States},
author = {Chayan Banerjee and Zhiyong Chen and Nasimul Noman},
journal = {2023 62nd IEEE Conference on Decision and Control (CDC)},
year = {2022},
pages = {7009-7014},
url = {https://api.semanticscholar.org/CorpusID:252682944}
}
```
```bibtex
@article{Doerr2017FastGA,
title = {Fast genetic algorithms},
author = {Benjamin Doerr and Huu Phuoc Le and R{\'e}gis Makhmara and Ta Duy Nguyen},
journal = {Proceedings of the Genetic and Evolutionary Computation Conference},
year = {2017},
url = {https://api.semanticscholar.org/CorpusID:16196841}
}
```
```bibtex
@article{Lee2024AnalysisClippedCritic
title = {On Analysis of Clipped Critic Loss in Proximal Policy Gradient},
author = {Yongjin Lee, Moonyoung Chung},
journal = {Authorea},
year = {2024}
}
```
```bibtex
@article{Ash2019OnTD,
title = {On the Difficulty of Warm-Starting Neural Network Training},
author = {Jordan T. Ash and Ryan P. Adams},
journal = {ArXiv},
year = {2019},
volume = {abs/1910.08475},
url = {https://api.semanticscholar.org/CorpusID:204788802}
}
```
```bibtex
@inproceedings{Gerasimov2025YouDN,
title = {You Do Not Fully Utilize Transformer's Representation Capacity},
author = {Gleb Gerasimov and Yaroslav Aksenov and Nikita Balagansky and Viacheslav Sinii and Daniil Gavrilov},
year = {2025},
url = {https://api.semanticscholar.org/CorpusID:276317819}
}
```
```bibtex
@article{Lee2024SimBaSB,
title = {SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning},
author = {Hojoon Lee and Dongyoon Hwang and Donghu Kim and Hyunseung Kim and Jun Jet Tai and Kaushik Subramanian and Peter R. Wurman and Jaegul Choo and Peter Stone and Takuma Seno},
journal = {ArXiv},
year = {2024},
volume = {abs/2410.09754},
url = {https://api.semanticscholar.org/CorpusID:273346233}
}
```
```bibtex
@article{Karras2019stylegan2,
title = {Analyzing and Improving the Image Quality of {StyleGAN}},
author = {Tero Karras and Samuli Laine and Miika Aittala and Janne Hellsten and Jaakko Lehtinen and Timo Aila},
journal = {CoRR},
volume = {abs/1912.04958},
year = {2019},
}
```
```bibtex
@article{Chebykin2023ShrinkPerturbIA,
title = {Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Search},
author = {Alexander Chebykin and Arkadiy Dushatskiy and Tanja Alderliesten and Peter A. N. Bosman},
journal = {ArXiv},
year = {2023},
volume = {abs/2307.15621},
url = {https://api.semanticscholar.org/CorpusID:260316291}
}
```
```bibtex
@article{Jiang2022GeneralIR,
title = {General intelligence requires rethinking exploration},
author = {Minqi Jiang and Tim Rocktaschel and Edward Grefenstette},
journal = {Royal Society Open Science},
year = {2022},
volume = {10},
url = {https://api.semanticscholar.org/CorpusID:253523156}
}
```
```bibtex
@misc{xie2025simplepolicyoptimization,
title = {Simple Policy Optimization},
author = {Zhengpeng Xie and Qiang Zhang and Fan Yang and Marco Hutter and Renjing Xu},
year = {2025},
eprint = {2401.16025},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2401.16025},
}
```
*Evolution is cleverer than you are.* - Leslie Orgel
Raw data
{
"_id": null,
"home_page": null,
"name": "evolutionary-policy-optimization",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "artificial intelligence, deep learning, genetic algorithms, reinforcement learning, robotics",
"author": null,
"author_email": "Phil Wang <lucidrains@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/d5/fd/2eec9bef4de000ec59eae7a6abd10e13bfc16c54dcb694a66265c8076e45/evolutionary_policy_optimization-0.2.15.tar.gz",
"platform": null,
"description": "<img width=\"450px\" alt=\"fig1\" src=\"https://github.com/user-attachments/assets/33bef569-e786-4f09-bdee-56bad7ea9e6d\" />\n\n## Evolutionary Policy Optimization\n\nPytorch implementation of [Evolutionary Policy Optimization](https://web3.arxiv.org/abs/2503.19037), from [Wang](https://www.jianrenw.com/) et al. of the Robotics Institute at Carnegie Mellon University\n\nThis paper stands out, as I have witnessed the positive effects first hand in an [exploratory project](https://github.com/lucidrains/firefly-torch) (mixing evolution with gradient based methods). Perhaps the Alexnet moment for genetic algorithms has not come to pass yet.\n\nBesides their latent variable strategy, I'll also throw in some attempts with crossover in weight space\n\n## Install\n\n```bash\n$ pip install evolutionary-policy-optimization\n```\n\n## Usage\n\n```python\nimport torch\n\nfrom evolutionary_policy_optimization import (\n LatentGenePool,\n Actor,\n Critic\n)\n\nlatent_pool = LatentGenePool(\n num_latents = 128,\n dim_latent = 32,\n)\n\nstate = torch.randn(1, 32)\n\nactor = Actor(dim_state = 32, dim = 256, mlp_depth = 2, num_actions = 4, dim_latent = 32)\ncritic = Critic(dim_state = 32, dim = 256, mlp_depth = 3, dim_latent = 32)\n\nlatent = latent_pool(latent_id = 2)\n\nactions = actor(state, latent)\nvalue = critic(state, latent)\n\n# interact with environment and receive rewards, termination etc\n\n# derive a fitness score for each gene / latent\n\nfitness = torch.randn(128)\n\nlatent_pool.genetic_algorithm_step(fitness) # update latent genes with genetic algorithm\n```\n\nEnd to end learning\n\n```python\nimport torch\n\nfrom evolutionary_policy_optimization import (\n create_agent,\n EPO,\n Env\n)\n\nagent = create_agent(\n dim_state = 512,\n num_latents = 16,\n dim_latent = 32,\n actor_num_actions = 5,\n actor_dim = 256,\n actor_mlp_depth = 2,\n critic_dim = 256,\n critic_mlp_depth = 3,\n latent_gene_pool_kwargs = dict(\n frac_natural_selected = 0.5\n )\n)\n\nepo = EPO(\n agent,\n episodes_per_latent = 1,\n max_episode_length = 10,\n action_sample_temperature = 1.\n)\n\nenv = Env((512,))\n\nepo(agent, env, num_learning_cycles = 5)\n\n# saving and loading\n\nagent.save('./agent.pt', overwrite = True)\nagent.load('./agent.pt')\n```\n\n## Contributing\n\nAt the project root, run\n\n```bash\n$ pip install '.[test]' # or `uv pip install '.[test]'`\n```\n\nThen add your tests to `tests/test_epo.py` and run\n\n```bash\n$ pytest tests/\n```\n\nThat's it\n\n## Citations\n\n```bibtex\n@inproceedings{Wang2025EvolutionaryPO,\n title = {Evolutionary Policy Optimization},\n author = {Jianren Wang and Yifan Su and Abhinav Gupta and Deepak Pathak},\n year = {2025},\n url = {https://api.semanticscholar.org/CorpusID:277313729}\n}\n```\n\n```bibtex\n@article{Farebrother2024StopRT,\n title = {Stop Regressing: Training Value Functions via Classification for Scalable Deep RL},\n author = {Jesse Farebrother and Jordi Orbay and Quan Ho Vuong and Adrien Ali Taiga and Yevgen Chebotar and Ted Xiao and Alex Irpan and Sergey Levine and Pablo Samuel Castro and Aleksandra Faust and Aviral Kumar and Rishabh Agarwal},\n journal = {ArXiv},\n year = {2024},\n volume = {abs/2403.03950},\n url = {https://api.semanticscholar.org/CorpusID:268253088}\n}\n```\n\n```bibtex\n@inproceedings{Khadka2018EvolutionGuidedPG,\n title = {Evolution-Guided Policy Gradient in Reinforcement Learning},\n author = {Shauharda Khadka and Kagan Tumer},\n booktitle = {Neural Information Processing Systems},\n year = {2018},\n url = {https://api.semanticscholar.org/CorpusID:53096951}\n}\n```\n\n```bibtex\n@article{Fortunato2017NoisyNF,\n title = {Noisy Networks for Exploration},\n author = {Meire Fortunato and Mohammad Gheshlaghi Azar and Bilal Piot and Jacob Menick and Ian Osband and Alex Graves and Vlad Mnih and R{\\'e}mi Munos and Demis Hassabis and Olivier Pietquin and Charles Blundell and Shane Legg},\n journal = {ArXiv},\n year = {2017},\n volume = {abs/1706.10295},\n url = {https://api.semanticscholar.org/CorpusID:5176587}\n}\n```\n\n```bibtex\n@article{Banerjee2022BoostingEI,\n title = {Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel States},\n author = {Chayan Banerjee and Zhiyong Chen and Nasimul Noman},\n journal = {2023 62nd IEEE Conference on Decision and Control (CDC)},\n year = {2022},\n pages = {7009-7014},\n url = {https://api.semanticscholar.org/CorpusID:252682944}\n}\n```\n\n```bibtex\n@article{Doerr2017FastGA,\n title = {Fast genetic algorithms},\n author = {Benjamin Doerr and Huu Phuoc Le and R{\\'e}gis Makhmara and Ta Duy Nguyen},\n journal = {Proceedings of the Genetic and Evolutionary Computation Conference},\n year = {2017},\n url = {https://api.semanticscholar.org/CorpusID:16196841}\n}\n```\n\n```bibtex\n@article{Lee2024AnalysisClippedCritic\n title = {On Analysis of Clipped Critic Loss in Proximal Policy Gradient},\n author = {Yongjin Lee, Moonyoung Chung},\n journal = {Authorea},\n year = {2024}\n}\n```\n\n```bibtex\n@article{Ash2019OnTD,\n title = {On the Difficulty of Warm-Starting Neural Network Training},\n author = {Jordan T. Ash and Ryan P. Adams},\n journal = {ArXiv},\n year = {2019},\n volume = {abs/1910.08475},\n url = {https://api.semanticscholar.org/CorpusID:204788802}\n}\n```\n\n```bibtex\n@inproceedings{Gerasimov2025YouDN,\n title = {You Do Not Fully Utilize Transformer's Representation Capacity},\n author = {Gleb Gerasimov and Yaroslav Aksenov and Nikita Balagansky and Viacheslav Sinii and Daniil Gavrilov},\n year = {2025},\n url = {https://api.semanticscholar.org/CorpusID:276317819}\n}\n```\n\n```bibtex\n@article{Lee2024SimBaSB,\n title = {SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning},\n author = {Hojoon Lee and Dongyoon Hwang and Donghu Kim and Hyunseung Kim and Jun Jet Tai and Kaushik Subramanian and Peter R. Wurman and Jaegul Choo and Peter Stone and Takuma Seno},\n journal = {ArXiv},\n year = {2024},\n volume = {abs/2410.09754},\n url = {https://api.semanticscholar.org/CorpusID:273346233}\n}\n```\n\n```bibtex\n@article{Karras2019stylegan2,\n title = {Analyzing and Improving the Image Quality of {StyleGAN}},\n author = {Tero Karras and Samuli Laine and Miika Aittala and Janne Hellsten and Jaakko Lehtinen and Timo Aila},\n journal = {CoRR},\n volume = {abs/1912.04958},\n year = {2019},\n}\n```\n\n```bibtex\n@article{Chebykin2023ShrinkPerturbIA,\n title = {Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Search},\n author = {Alexander Chebykin and Arkadiy Dushatskiy and Tanja Alderliesten and Peter A. N. Bosman},\n journal = {ArXiv},\n year = {2023},\n volume = {abs/2307.15621},\n url = {https://api.semanticscholar.org/CorpusID:260316291}\n}\n```\n\n```bibtex\n@article{Jiang2022GeneralIR,\n title = {General intelligence requires rethinking exploration},\n author = {Minqi Jiang and Tim Rocktaschel and Edward Grefenstette},\n journal = {Royal Society Open Science},\n year = {2022},\n volume = {10},\n url = {https://api.semanticscholar.org/CorpusID:253523156}\n}\n```\n\n```bibtex\n@misc{xie2025simplepolicyoptimization,\n title = {Simple Policy Optimization}, \n author = {Zhengpeng Xie and Qiang Zhang and Fan Yang and Marco Hutter and Renjing Xu},\n year = {2025},\n eprint = {2401.16025},\n archivePrefix = {arXiv},\n primaryClass = {cs.LG},\n url = {https://arxiv.org/abs/2401.16025}, \n}\n```\n\n*Evolution is cleverer than you are.* - Leslie Orgel\n",
"bugtrack_url": null,
"license": "MIT License\n \n Copyright (c) 2025 Phil Wang\n \n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n \n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n \n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE.",
"summary": "EPO - Pytorch",
"version": "0.2.15",
"project_urls": {
"Homepage": "https://pypi.org/project/evolutionary-policy-optimization/",
"Repository": "https://github.com/lucidrains/evolutionary-policy-optimization"
},
"split_keywords": [
"artificial intelligence",
" deep learning",
" genetic algorithms",
" reinforcement learning",
" robotics"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "eb7f69d229e4f9adf9cbcecca907fb78fde2fd583fef41323617175680c3125f",
"md5": "2d248d5fb5ab8447d40e4de349f4ef77",
"sha256": "e485fc3598edad85e6488cc27f0fdeadd41516c30ff0977f91076183d329979a"
},
"downloads": -1,
"filename": "evolutionary_policy_optimization-0.2.15-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2d248d5fb5ab8447d40e4de349f4ef77",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 24347,
"upload_time": "2025-08-03T02:20:47",
"upload_time_iso_8601": "2025-08-03T02:20:47.453356Z",
"url": "https://files.pythonhosted.org/packages/eb/7f/69d229e4f9adf9cbcecca907fb78fde2fd583fef41323617175680c3125f/evolutionary_policy_optimization-0.2.15-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "d5fd2eec9bef4de000ec59eae7a6abd10e13bfc16c54dcb694a66265c8076e45",
"md5": "3ada3c83d5aff92fede8dfe195c414b4",
"sha256": "f28e0eb2929e4dd0aaff702e02b67abd258002d8a36d9cd3f467710e64abaee3"
},
"downloads": -1,
"filename": "evolutionary_policy_optimization-0.2.15.tar.gz",
"has_sig": false,
"md5_digest": "3ada3c83d5aff92fede8dfe195c414b4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 25748,
"upload_time": "2025-08-03T02:20:48",
"upload_time_iso_8601": "2025-08-03T02:20:48.732111Z",
"url": "https://files.pythonhosted.org/packages/d5/fd/2eec9bef4de000ec59eae7a6abd10e13bfc16c54dcb694a66265c8076e45/evolutionary_policy_optimization-0.2.15.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-03 02:20:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lucidrains",
"github_project": "evolutionary-policy-optimization",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "box2d-py",
"specs": []
},
{
"name": "gymnasium",
"specs": [
[
">=",
"1.0.0"
]
]
},
{
"name": "moviepy",
"specs": [
[
">=",
"1.0.3"
]
]
},
{
"name": "numpy",
"specs": [
[
">=",
"2.2.5"
]
]
},
{
"name": "tqdm",
"specs": []
}
],
"lcname": "evolutionary-policy-optimization"
}
