<p align='center'>
<a href='https://github.com/infer-actively/pymdp'>
<img src='.github/pymdp_logo_2-removebg.png' />
</a>
</p>
A Python package for simulating Active Inference agents in Markov Decision Process environments.
Please see our companion paper, published in the Journal of Open Source Software: ["pymdp: A Python library for active inference in discrete state spaces"](https://joss.theoj.org/papers/10.21105/joss.04098) for an overview of the package and its motivation. For a more in-depth, tutorial-style introduction to the package and a mathematical overview of active inference in Markov Decision Processes, see the [longer arxiv version](https://arxiv.org/abs/2201.03904) of the paper.
This package is hosted on the [`infer-actively`](https://github.com/infer-actively) GitHub organization, which was built with the intention of hosting open-source active inference and free-energy-principle related software.
Most of the low-level mathematical operations are [NumPy](https://github.com/numpy/numpy) ports of their equivalent functions from the `SPM` [implementation](https://www.fil.ion.ucl.ac.uk/spm/doc/) in MATLAB. We have benchmarked and validated most of these functions against their SPM counterparts.
## Status
[](https://pymdp-rtd.readthedocs.io/en/latest/?badge=latest)
[](https://doi.org/10.21105/joss.04098)
# ``pymdp`` in action
Here's a visualization of ``pymdp`` agents in action. One of the defining features of active inference agents is the drive to maximize "epistemic value" (i.e. curiosity). Equipped with such a drive in environments with uncertain yet disclosable hidden structure, active inference can ultimately allow agents to simultaneously learn about the environment as well as maximize reward.
The simulation below (see associated notebook [here](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/cue_chaining_demo.html)) demonstrates what might be called "epistemic chaining," where an agent (here, analogized to a mouse seeking food) forages for a chain of cues, each of which discloses the location of the subsequent cue in the chain. The final cue (here, "Cue 2") reveals the location a hidden reward. This is similar in spirit to "behavior chaining" used in operant conditioning, except that here, each successive action in the behavioral sequence doesn't need to be learned through instrumental conditioning. Rather, active inference agents will naturally forage the sequence of cues based on an intrinsic desire to disclose information. This ultimately leads the agent to the hidden reward source in the fewest number of moves as possible.
You can run the code behind simulating tasks like this one and others in the **Examples** section of the [official documentation](https://pymdp-rtd.readthedocs.io/en/stable/).
<!--
<p align="center">
<img src=".github/chained_cue_navigation_v1.gif" width="50%" height="50%"/>
<img src=".github/chained_cue_navigation_v2.gif" width="50%" height="50%"/>
</p> -->
<!--  |  -->
<table><tr>
<td>
<p align="center" style="padding: 10px">
<img src=".github/chained_cue_navigation_v1.gif" width="100%" height="50%"/>
<br>
<em style="color: grey">Cue 2 in Location 1, Reward on Top</em>
</p>
</td>
<td>
<p align="center">
<img src=".github/chained_cue_navigation_v2.gif" width="100%" height="50%"/>
<br>
<em style="color: grey">Cue 2 in Location 3, Reward on Bottom</em>
</p>
</td>
</tr></table>
## Quick-start: Installation and Usage
In order to use `pymdp` to build and develop active inference agents, we recommend installing it with the the package installer [`pip`](https://pip.pypa.io/en/stable/), which will install `pymdp` locally as well as its dependencies. This can also be done in a virtual environment (e.g. with `venv`).
When pip installing `pymdp`, use the package name `inferactively-pymdp`:
```bash
pip install inferactively-pymdp
```
Once in Python, you can then directly import `pymdp`, its sub-packages, and functions.
```bash
import pymdp
from pymdp import utils
from pymdp.agent import Agent
num_obs = [3, 5] # observation modality dimensions
num_states = [3, 2, 2] # hidden state factor dimensions
num_controls = [3, 1, 1] # control state factor dimensions
A_matrix = utils.random_A_matrix(num_obs, num_states) # create sensory likelihood (A matrix)
B_matrix = utils.random_B_matrix(num_states, num_controls) # create transition likelihood (B matrix)
C_vector = utils.obj_array_uniform(num_obs) # uniform preferences
# instantiate a quick agent using your A, B and C arrays
my_agent = Agent( A = A_matrix, B = B_matrix, C = C_vector)
# give the agent a random observation and get the optimized posterior beliefs
observation = [1, 4] # a list specifying the indices of the observation, for each observation modality
qs = my_agent.infer_states(observation) # get posterior over hidden states (a multi-factor belief)
# Do active inference
q_pi, neg_efe = my_agent.infer_policies() # return the policy posterior and return (negative) expected free energies of each policy as well
action = my_agent.sample_action() # sample an action
# ... and so on ...
```
## Getting started / introductory material
We recommend starting with the Installation/Usage section of the [official documentation](https://pymdp-rtd.readthedocs.io/en/stable/) for the repository, which provides a series of useful pedagogical notebooks for introducing you to active inference and how to build agents in `pymdp`.
For new users to `pymdp`, we specifically recommend stepping through following three Jupyter notebooks (can also be used on Google Colab):
- [`Pymdp` fundamentals](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/pymdp_fundamentals.html)
- [Active Inference from Scratch](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/active_inference_from_scratch.html)
- [The `Agent` API](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/using_the_agent_class.html)
Special thanks to [Beren Millidge](https://github.com/BerenMillidge) and [Daphne Demekas](https://github.com/daphnedemekas) for their help in prototyping earlier versions of the [Active Inference from Scratch](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/active_inference_from_scratch.html) tutorial, which were originally based on a grid world POMDP environment create by [Alec Tschantz](https://github.com/alec-tschantz).
We also have (and are continuing to build) a series of notebooks that walk through active inference agents performing different types of tasks, such as the classic [T-Maze environment](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/tmaze_demo.html) and the newer [Epistemic Chaining](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/cue_chaining_demo.html) demo.
## Contributing
This package is under active development. If you would like to contribute, please refer to [this file](CONTRIBUTING.md)
If you would like to contribute to this repo, we recommend using venv and pip
```bash
cd <path_to_repo_fork>
python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
pip install -e ./ # This will install pymdp as a local dev package
```
You should then be able to run tests locally with `pytest`
```bash
pytest test
```
## Citing `pymdp`
If you use `pymdp` in your work or research, please consider citing our [paper](https://joss.theoj.org/papers/10.21105/joss.04098) (open-access) published in the Journal of Open-Source Software:
```
@article{Heins2022,
doi = {10.21105/joss.04098},
url = {https://doi.org/10.21105/joss.04098},
year = {2022},
publisher = {The Open Journal},
volume = {7},
number = {73},
pages = {4098},
author = {Conor Heins and Beren Millidge and Daphne Demekas and Brennan Klein and Karl Friston and Iain D. Couzin and Alexander Tschantz},
title = {pymdp: A Python library for active inference in discrete state spaces},
journal = {Journal of Open Source Software}
}
```
For a more in-depth, tutorial-style introduction to the package and a mathematical overview of active inference in Markov Decision Processes, you can also consult the [longer arxiv version](https://arxiv.org/abs/2201.03904) of the paper.
## Authors
- Conor Heins [@conorheins](https://github.com/conorheins)
- Alec Tschantz [@alec-tschantz](https://github.com/alec-tschantz)
- Beren Millidge [@BerenMillidge](https://github.com/BerenMillidge)
- Brennan Klein [@jkbren](https://github.com/jkbren)
- Arun Niranjan [@Arun-Niranjan](https://github.com/Arun-Niranjan)
- Daphne Demekas [@daphnedemekas](https://github.com/daphnedemekas)
Raw data
{
"_id": null,
"home_page": "https://github.com/infer-actively/pymdp",
"name": "inferactively-pymdp",
"maintainer": "",
"docs_url": null,
"requires_python": ">3.7",
"maintainer_email": "",
"keywords": "artificial intelligence,active inference,free energy principleinformation theory,decision-making,MDP,Markov Decision Process,Bayesian inference,variational inference,reinforcement learning",
"author": "infer-actively",
"author_email": "conor.heins@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/bf/38/d19b9594cfbc2a7fffbc63202c5cc04d06ca0f77752efff96705329e1bc4/inferactively-pymdp-0.0.7.1.tar.gz",
"platform": null,
"description": "\n<p align='center'>\n <a href='https://github.com/infer-actively/pymdp'>\n <img src='.github/pymdp_logo_2-removebg.png' />\n </a> \n</p>\n\nA Python package for simulating Active Inference agents in Markov Decision Process environments.\nPlease see our companion paper, published in the Journal of Open Source Software: [\"pymdp: A Python library for active inference in discrete state spaces\"](https://joss.theoj.org/papers/10.21105/joss.04098) for an overview of the package and its motivation. For a more in-depth, tutorial-style introduction to the package and a mathematical overview of active inference in Markov Decision Processes, see the [longer arxiv version](https://arxiv.org/abs/2201.03904) of the paper.\n\nThis package is hosted on the [`infer-actively`](https://github.com/infer-actively) GitHub organization, which was built with the intention of hosting open-source active inference and free-energy-principle related software.\n\nMost of the low-level mathematical operations are [NumPy](https://github.com/numpy/numpy) ports of their equivalent functions from the `SPM` [implementation](https://www.fil.ion.ucl.ac.uk/spm/doc/) in MATLAB. We have benchmarked and validated most of these functions against their SPM counterparts.\n\n## Status\n\n\n\n[](https://pymdp-rtd.readthedocs.io/en/latest/?badge=latest)\n[](https://doi.org/10.21105/joss.04098)\n\n\n# ``pymdp`` in action\n\nHere's a visualization of ``pymdp`` agents in action. One of the defining features of active inference agents is the drive to maximize \"epistemic value\" (i.e. curiosity). Equipped with such a drive in environments with uncertain yet disclosable hidden structure, active inference can ultimately allow agents to simultaneously learn about the environment as well as maximize reward.\n\nThe simulation below (see associated notebook [here](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/cue_chaining_demo.html)) demonstrates what might be called \"epistemic chaining,\" where an agent (here, analogized to a mouse seeking food) forages for a chain of cues, each of which discloses the location of the subsequent cue in the chain. The final cue (here, \"Cue 2\") reveals the location a hidden reward. This is similar in spirit to \"behavior chaining\" used in operant conditioning, except that here, each successive action in the behavioral sequence doesn't need to be learned through instrumental conditioning. Rather, active inference agents will naturally forage the sequence of cues based on an intrinsic desire to disclose information. This ultimately leads the agent to the hidden reward source in the fewest number of moves as possible.\n\nYou can run the code behind simulating tasks like this one and others in the **Examples** section of the [official documentation](https://pymdp-rtd.readthedocs.io/en/stable/).\n\n<!-- \n<p align=\"center\">\n <img src=\".github/chained_cue_navigation_v1.gif\" width=\"50%\" height=\"50%\"/>\n <img src=\".github/chained_cue_navigation_v2.gif\" width=\"50%\" height=\"50%\"/>\n</p> -->\n\n<!--  |  -->\n\n<table><tr>\n<td> \n <p align=\"center\" style=\"padding: 10px\">\n <img src=\".github/chained_cue_navigation_v1.gif\" width=\"100%\" height=\"50%\"/>\n <br>\n <em style=\"color: grey\">Cue 2 in Location 1, Reward on Top</em>\n </p> \n</td>\n<td> \n <p align=\"center\">\n <img src=\".github/chained_cue_navigation_v2.gif\" width=\"100%\" height=\"50%\"/>\n <br>\n <em style=\"color: grey\">Cue 2 in Location 3, Reward on Bottom</em>\n </p> \n</td>\n</tr></table>\n\n## Quick-start: Installation and Usage\n\nIn order to use `pymdp` to build and develop active inference agents, we recommend installing it with the the package installer [`pip`](https://pip.pypa.io/en/stable/), which will install `pymdp` locally as well as its dependencies. This can also be done in a virtual environment (e.g. with `venv`). \n\nWhen pip installing `pymdp`, use the package name `inferactively-pymdp`:\n\n```bash\npip install inferactively-pymdp\n```\n\nOnce in Python, you can then directly import `pymdp`, its sub-packages, and functions.\n\n```bash\n\nimport pymdp\nfrom pymdp import utils\nfrom pymdp.agent import Agent\n\nnum_obs = [3, 5] # observation modality dimensions\nnum_states = [3, 2, 2] # hidden state factor dimensions\nnum_controls = [3, 1, 1] # control state factor dimensions\nA_matrix = utils.random_A_matrix(num_obs, num_states) # create sensory likelihood (A matrix)\nB_matrix = utils.random_B_matrix(num_states, num_controls) # create transition likelihood (B matrix)\n\nC_vector = utils.obj_array_uniform(num_obs) # uniform preferences\n\n# instantiate a quick agent using your A, B and C arrays\nmy_agent = Agent( A = A_matrix, B = B_matrix, C = C_vector)\n\n# give the agent a random observation and get the optimized posterior beliefs\n\nobservation = [1, 4] # a list specifying the indices of the observation, for each observation modality\n\nqs = my_agent.infer_states(observation) # get posterior over hidden states (a multi-factor belief)\n\n# Do active inference\n\nq_pi, neg_efe = my_agent.infer_policies() # return the policy posterior and return (negative) expected free energies of each policy as well\n\naction = my_agent.sample_action() # sample an action\n\n# ... and so on ...\n```\n\n## Getting started / introductory material\n\nWe recommend starting with the Installation/Usage section of the [official documentation](https://pymdp-rtd.readthedocs.io/en/stable/) for the repository, which provides a series of useful pedagogical notebooks for introducing you to active inference and how to build agents in `pymdp`.\n\nFor new users to `pymdp`, we specifically recommend stepping through following three Jupyter notebooks (can also be used on Google Colab):\n\n- [`Pymdp` fundamentals](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/pymdp_fundamentals.html)\n- [Active Inference from Scratch](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/active_inference_from_scratch.html)\n- [The `Agent` API](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/using_the_agent_class.html)\n\nSpecial thanks to [Beren Millidge](https://github.com/BerenMillidge) and [Daphne Demekas](https://github.com/daphnedemekas) for their help in prototyping earlier versions of the [Active Inference from Scratch](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/active_inference_from_scratch.html) tutorial, which were originally based on a grid world POMDP environment create by [Alec Tschantz](https://github.com/alec-tschantz).\n\nWe also have (and are continuing to build) a series of notebooks that walk through active inference agents performing different types of tasks, such as the classic [T-Maze environment](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/tmaze_demo.html) and the newer [Epistemic Chaining](https://pymdp-rtd.readthedocs.io/en/latest/notebooks/cue_chaining_demo.html) demo.\n\n## Contributing\n\nThis package is under active development. If you would like to contribute, please refer to [this file](CONTRIBUTING.md)\n\nIf you would like to contribute to this repo, we recommend using venv and pip\n```bash\ncd <path_to_repo_fork>\npython3 -m venv env\nsource env/bin/activate\npip install -r requirements.txt\npip install -e ./ # This will install pymdp as a local dev package\n```\n\nYou should then be able to run tests locally with `pytest`\n```bash\npytest test\n```\n\n## Citing `pymdp`\nIf you use `pymdp` in your work or research, please consider citing our [paper](https://joss.theoj.org/papers/10.21105/joss.04098) (open-access) published in the Journal of Open-Source Software:\n\n```\n@article{Heins2022,\n doi = {10.21105/joss.04098},\n url = {https://doi.org/10.21105/joss.04098},\n year = {2022},\n publisher = {The Open Journal},\n volume = {7},\n number = {73},\n pages = {4098},\n author = {Conor Heins and Beren Millidge and Daphne Demekas and Brennan Klein and Karl Friston and Iain D. Couzin and Alexander Tschantz},\n title = {pymdp: A Python library for active inference in discrete state spaces},\n journal = {Journal of Open Source Software}\n}\n```\n\nFor a more in-depth, tutorial-style introduction to the package and a mathematical overview of active inference in Markov Decision Processes, you can also consult the [longer arxiv version](https://arxiv.org/abs/2201.03904) of the paper.\n\n## Authors\n\n- Conor Heins [@conorheins](https://github.com/conorheins)\n- Alec Tschantz [@alec-tschantz](https://github.com/alec-tschantz)\n- Beren Millidge [@BerenMillidge](https://github.com/BerenMillidge)\n- Brennan Klein [@jkbren](https://github.com/jkbren)\n- Arun Niranjan [@Arun-Niranjan](https://github.com/Arun-Niranjan)\n- Daphne Demekas [@daphnedemekas](https://github.com/daphnedemekas)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A Python package for solving Markov Decision Processes with Active Inference",
"version": "0.0.7.1",
"split_keywords": [
"artificial intelligence",
"active inference",
"free energy principleinformation theory",
"decision-making",
"mdp",
"markov decision process",
"bayesian inference",
"variational inference",
"reinforcement learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7810726e33b6e7882d8019c2470612ace2766419b792771e6fa5fa57052adb1a",
"md5": "06254368f5c2bea12b6bdeff75e4a3f2",
"sha256": "147f5f10b5f43e369ce25de7fdbddae36069a58895e3227cc0f54659e59832f7"
},
"downloads": -1,
"filename": "inferactively_pymdp-0.0.7.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "06254368f5c2bea12b6bdeff75e4a3f2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">3.7",
"size": 58924,
"upload_time": "2023-03-25T17:50:02",
"upload_time_iso_8601": "2023-03-25T17:50:02.026678Z",
"url": "https://files.pythonhosted.org/packages/78/10/726e33b6e7882d8019c2470612ace2766419b792771e6fa5fa57052adb1a/inferactively_pymdp-0.0.7.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bf38d19b9594cfbc2a7fffbc63202c5cc04d06ca0f77752efff96705329e1bc4",
"md5": "e20f8d3215c97c9b27489e48a71ed7dc",
"sha256": "0042ccd529fe47c6604112ae2640adae13805d7d3f3d3997105606fde14cd80a"
},
"downloads": -1,
"filename": "inferactively-pymdp-0.0.7.1.tar.gz",
"has_sig": false,
"md5_digest": "e20f8d3215c97c9b27489e48a71ed7dc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">3.7",
"size": 70919,
"upload_time": "2023-03-25T17:50:04",
"upload_time_iso_8601": "2023-03-25T17:50:04.364200Z",
"url": "https://files.pythonhosted.org/packages/bf/38/d19b9594cfbc2a7fffbc63202c5cc04d06ca0f77752efff96705329e1bc4/inferactively-pymdp-0.0.7.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-25 17:50:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "infer-actively",
"github_project": "pymdp",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "attrs",
"specs": [
[
">=",
"20.3.0"
]
]
},
{
"name": "cycler",
"specs": [
[
">=",
"0.10.0"
]
]
},
{
"name": "iniconfig",
"specs": [
[
">=",
"1.1.1"
]
]
},
{
"name": "kiwisolver",
"specs": [
[
">=",
"1.3.1"
]
]
},
{
"name": "matplotlib",
"specs": [
[
">=",
"3.1.3"
]
]
},
{
"name": "nose",
"specs": [
[
">=",
"1.3.7"
]
]
},
{
"name": "numpy",
"specs": [
[
">=",
"1.19.5"
]
]
},
{
"name": "openpyxl",
"specs": [
[
">=",
"3.0.7"
]
]
},
{
"name": "packaging",
"specs": [
[
">=",
"20.8"
]
]
},
{
"name": "pandas",
"specs": [
[
">=",
"1.2.4"
]
]
},
{
"name": "Pillow",
"specs": [
[
">=",
"8.2.0"
]
]
},
{
"name": "pluggy",
"specs": [
[
">=",
"0.13.1"
]
]
},
{
"name": "py",
"specs": [
[
">=",
"1.10.0"
]
]
},
{
"name": "pyparsing",
"specs": [
[
">=",
"2.4.7"
]
]
},
{
"name": "pytest",
"specs": [
[
">=",
"6.2.1"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
">=",
"2.8.1"
]
]
},
{
"name": "pytz",
"specs": [
[
">=",
"2020.5"
]
]
},
{
"name": "scipy",
"specs": [
[
">=",
"1.6.0"
]
]
},
{
"name": "seaborn",
"specs": [
[
">=",
"0.11.1"
]
]
},
{
"name": "six",
"specs": [
[
">=",
"1.15.0"
]
]
},
{
"name": "toml",
"specs": [
[
">=",
"0.10.2"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
">=",
"3.7.4.3"
]
]
},
{
"name": "xlsxwriter",
"specs": [
[
">=",
"1.4.3"
]
]
},
{
"name": "sphinx-rtd-theme",
"specs": [
[
">=",
"0.4"
]
]
},
{
"name": "myst-nb",
"specs": [
[
">=",
"0.13.1"
]
]
},
{
"name": "autograd",
"specs": [
[
">=",
"1.3"
]
]
}
],
"lcname": "inferactively-pymdp"
}
JSON
