# Reinforcement Learning Framework
An easy-to-read Reinforcement Learning (RL) framework. Provides standardized interfaces and implementations to various Reinforcement Learning methods and environments. Also this is the main place to start your journey with Reinforcement Learning and learn from tutorials and examples.
### Main Features
- Choose from a growing number of **Gym environments** and **MLAgent environments**
- Using various Reinforcement Learning algorithms for learning, which are implemented in **Stable-Baselines 3**
- Integrate or implement own **custom environments and agents** in a standardized interface
- Upload your models to the **HuggingFace Hub**
## Set-Up
### Activate your development environment
If you are on a UNIX-based OS:
You are fine. Continue with the next step.
If you are on Windows:
Make sure to use a WSL Python interpreter as your development environment, since we require a UNIX-based system underneath Python to run a lot of the environments and algorithms.
For users using PyCharm, see https://www.jetbrains.com/help/pycharm/using-wsl-as-a-remote-interpreter.html for more information.
For users using Visual Studio Code, see https://code.visualstudio.com/docs/remote/wsl-tutorial and https://code.visualstudio.com/docs/remote/wsl for more information.
### Install all dependencies in your development environment
To set up your local development environment, please run:
```
poetry install
```
Behind the scenes, this creates a virtual environment and installs `rl_framework` along with its dependencies into a new virtualenv. Whenever you run `poetry run <command>`, that `<command>` is actually run inside the virtualenv managed by poetry.
You can now import functions and classes from the module with `import rl_framework`.
### Optional: Install FFMPEG to enable generation of videos (for upload)
The creation of videos for the functionality of creating video-replays of the agent performance on the environment requires installing the FFMPEG package on your machine.
This feature is important if you plan to upload replay videos to an experiment tracking service together with the agent itself.
The `ffmpeg` command needs to be available to invoke from the command line, since it is called from Python through a `os.system` invoke. Therefore, it is important that you install this package directly on your machine.
Please follow the guide which can be found [here](https://www.geeksforgeeks.org/how-to-install-ffmpeg-on-windows/) to install the FFMPEG library on your respective machine.
### Optional: Preparation for pushing your models to the HuggingFace Hub
- Create an account to HuggingFace and sign in. ➡ https://huggingface.co/join
- Create a new token with write role. ➡ https://huggingface.co/settings/tokens
- Store your authentication token from the Hugging Face website. ➡ `huggingface-cli login`
### Optional: Preparation for using a Unity environment (optional)
In order to use environments based on the Unity game framework, make sure to follow the installation procedures detailed in [following installation guideline provided by Unity Technologies](https://github.com/Unity-Technologies/ml-agents/blob/develop/docs/Installation.md).
In short:
- Install Unity. ➡ https://unity.com/download
- Create a new Unity project.
- Navigate to the menu `Window -> Package Manager` and install the `com.unity.ml-agents` package in Unity. ➡ https://docs.unity3d.com/Manual/upm-ui-install.html
## Getting Started
### Configuring an environment
To integrate your environment you wish to train on, you need to create an Environment class representing your problem. For this you can
- you use an existing Gym environment with [the `GymEnvironment` class](src/rl_framework/environment/gym_environment.py)
- you use an existing MLAgent environment with [the `MLAgentsEnvironment` class](src/rl_framework/environment/mlagents_environment.py)
- create a custom environment by inheriting from [the base `Environment` class](src/rl_framework/environment/base_environment.py), which specifies the required interface
### Configuring an agent
To integrate the Reinforcement Learning algorithm you wish to train an agent on your environment with, you need to create an Agent class representing your training agent. For this you can
- you use an existing Reinforcement Learning algorithm implemented in the Stable-Baselines 3 framework with [the `StableBaselinesAgent` class](src/rl_framework/agent/stable_baselines.py) (see the Example section below)
- create a custom Reinforcement Learning algorithm by inheriting from [the base `BaseAgent` class](src/rl_framework/agent/base_agent.py), which specifies the required interface
### Training
After configuring the environment and the agent, you can start training your agent on the environment.
This can be done in one line of code:
```
agent.train(environments=environments, total_timesteps=100000)
```
Independent of which environment and which agent you choose, the unified interface allows to always start the training this way.
### Evaluating
Once you trained the agent, you can evaluate the agent policy on the environment and get the average accumulated reward (and standard deviation) as evaluation metric.
This evaluation method is implemented in the [evaluate function of the agent](src/rl_framework/agent/base_agent.py) and called with one line of code:
```
agent.evaluate(evaluation_environment=environment, n_eval_episodes=100, deterministic=False)
```
### Uploading and downloading models from the HuggingFace Hub
Once you trained the agent, you can upload the agent model to the HuggingFace Hub in order to share and compare your agent to others. You can also downloaded yours or other agents from the same HuggingFace Hub and use them for solving environments or re-training.
The object which allows for this functionality is `HuggingFaceConnector`, which can be found in the [connection collection package](src/rl_framework/util/saving_and_loading/connector).
### Example
In [this example script](exploration/train_sb3_agent.py) you can see all of the above steps unified.
For a quick impression in this README, find a minimal training and evaluation example here:
```
# Create environment(s); multiple environments for parallel training
environments = [GymEnvironmentWrapper(ENV_ID) for _ in range(PARALLEL_ENVIRONMENTS)]
# Create new agent
agent = StableBaselinesAgent(
algorithm=StableBaselinesAlgorithm.PPO,
algorithm_parameters={
"policy": "MlpPolicy"
}
)
# Train agent
agent.train(environments=environments, total_timesteps=100000)
# Evaluate the model
mean_reward, std_reward = agent.evaluate(evaluation_environment=environments[0])
```
## Development
### Notebooks
You can use your module code (`src/`) in Jupyter notebooks without running into import errors by running:
```
poetry run jupyter notebook
```
or
```
poetry run jupyter-lab
```
This starts the jupyter server inside the project's virtualenv.
Assuming you already have Jupyter installed, you can make your virtual environment available as a separate kernel by running:
```
poetry add ipykernel
poetry run python -m ipykernel install --user --name="reinforcement-learning-framework"
```
Note that we mainly use notebooks for experiments, visualizations and reports. Every piece of functionality that is meant to be reused should go into module code and be imported into notebooks.
### Testing
We use `pytest` as test framework. To execute the tests, please run
```
pytest tests
```
To run the tests with coverage information, please use
```
pytest tests --cov=src --cov-report=html --cov-report=term
```
and have a look at the `htmlcov` folder, after the tests are done.
### Distribution Package
To build a distribution package (wheel), please use
```
python setup.py bdist_wheel
```
this will clean up the build folder and then run the `bdist_wheel` command.
### Contributions
Before contributing, please set up the pre-commit hooks to reduce errors and ensure consistency
```
pip install -U pre-commit
pre-commit install
```
If you run into any issues, you can remove the hooks again with `pre-commit uninstall`.
## License
© Alexander Zap
Raw data
{
"_id": null,
"home_page": "https://github.com/alexander-zap/reinforcement-learning-framework",
"name": "reinforcement-learning-framework",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "reinforcement learning, rl, imitation learning",
"author": "Alexander Zap",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/3f/aa/fd3b7b5cfc3ceeffeea61c220f0d28166310a0ec144505ff5d59debea75e/reinforcement_learning_framework-0.6.1.tar.gz",
"platform": null,
"description": "# Reinforcement Learning Framework\n\nAn easy-to-read Reinforcement Learning (RL) framework. Provides standardized interfaces and implementations to various Reinforcement Learning methods and environments. Also this is the main place to start your journey with Reinforcement Learning and learn from tutorials and examples.\n\n### Main Features\n\n- Choose from a growing number of **Gym environments** and **MLAgent environments**\n- Using various Reinforcement Learning algorithms for learning, which are implemented in **Stable-Baselines 3**\n- Integrate or implement own **custom environments and agents** in a standardized interface\n- Upload your models to the **HuggingFace Hub**\n\n## Set-Up\n\n### Activate your development environment\n\nIf you are on a UNIX-based OS:\nYou are fine. Continue with the next step.\n\nIf you are on Windows:\nMake sure to use a WSL Python interpreter as your development environment, since we require a UNIX-based system underneath Python to run a lot of the environments and algorithms.\nFor users using PyCharm, see https://www.jetbrains.com/help/pycharm/using-wsl-as-a-remote-interpreter.html for more information.\nFor users using Visual Studio Code, see https://code.visualstudio.com/docs/remote/wsl-tutorial and https://code.visualstudio.com/docs/remote/wsl for more information.\n\n### Install all dependencies in your development environment\n\nTo set up your local development environment, please run:\n\n```\npoetry install\n```\n\nBehind the scenes, this creates a virtual environment and installs `rl_framework` along with its dependencies into a new virtualenv. Whenever you run `poetry run <command>`, that `<command>` is actually run inside the virtualenv managed by poetry.\n\nYou can now import functions and classes from the module with `import rl_framework`.\n\n### Optional: Install FFMPEG to enable generation of videos (for upload)\n\nThe creation of videos for the functionality of creating video-replays of the agent performance on the environment requires installing the FFMPEG package on your machine.\nThis feature is important if you plan to upload replay videos to an experiment tracking service together with the agent itself.\nThe `ffmpeg` command needs to be available to invoke from the command line, since it is called from Python through a `os.system` invoke. Therefore, it is important that you install this package directly on your machine.\n\nPlease follow the guide which can be found [here](https://www.geeksforgeeks.org/how-to-install-ffmpeg-on-windows/) to install the FFMPEG library on your respective machine.\n\n### Optional: Preparation for pushing your models to the HuggingFace Hub\n\n- Create an account to HuggingFace and sign in. \u27a1 https://huggingface.co/join\n- Create a new token with write role. \u27a1 https://huggingface.co/settings/tokens\n- Store your authentication token from the Hugging Face website. \u27a1 `huggingface-cli login`\n\n### Optional: Preparation for using a Unity environment (optional)\n\nIn order to use environments based on the Unity game framework, make sure to follow the installation procedures detailed in [following installation guideline provided by Unity Technologies](https://github.com/Unity-Technologies/ml-agents/blob/develop/docs/Installation.md).\nIn short:\n\n- Install Unity. \u27a1 https://unity.com/download\n- Create a new Unity project.\n- Navigate to the menu `Window -> Package Manager` and install the `com.unity.ml-agents` package in Unity. \u27a1 https://docs.unity3d.com/Manual/upm-ui-install.html\n\n## Getting Started\n\n### Configuring an environment\n\nTo integrate your environment you wish to train on, you need to create an Environment class representing your problem. For this you can\n\n- you use an existing Gym environment with [the `GymEnvironment` class](src/rl_framework/environment/gym_environment.py)\n- you use an existing MLAgent environment with [the `MLAgentsEnvironment` class](src/rl_framework/environment/mlagents_environment.py)\n- create a custom environment by inheriting from [the base `Environment` class](src/rl_framework/environment/base_environment.py), which specifies the required interface\n\n### Configuring an agent\n\nTo integrate the Reinforcement Learning algorithm you wish to train an agent on your environment with, you need to create an Agent class representing your training agent. For this you can\n\n- you use an existing Reinforcement Learning algorithm implemented in the Stable-Baselines 3 framework with [the `StableBaselinesAgent` class](src/rl_framework/agent/stable_baselines.py) (see the Example section below)\n- create a custom Reinforcement Learning algorithm by inheriting from [the base `BaseAgent` class](src/rl_framework/agent/base_agent.py), which specifies the required interface\n\n### Training\n\nAfter configuring the environment and the agent, you can start training your agent on the environment.\nThis can be done in one line of code:\n\n```\nagent.train(environments=environments, total_timesteps=100000)\n```\n\nIndependent of which environment and which agent you choose, the unified interface allows to always start the training this way.\n\n### Evaluating\n\nOnce you trained the agent, you can evaluate the agent policy on the environment and get the average accumulated reward (and standard deviation) as evaluation metric.\nThis evaluation method is implemented in the [evaluate function of the agent](src/rl_framework/agent/base_agent.py) and called with one line of code:\n\n```\nagent.evaluate(evaluation_environment=environment, n_eval_episodes=100, deterministic=False)\n```\n\n### Uploading and downloading models from the HuggingFace Hub\n\nOnce you trained the agent, you can upload the agent model to the HuggingFace Hub in order to share and compare your agent to others. You can also downloaded yours or other agents from the same HuggingFace Hub and use them for solving environments or re-training.\nThe object which allows for this functionality is `HuggingFaceConnector`, which can be found in the [connection collection package](src/rl_framework/util/saving_and_loading/connector).\n\n### Example\n\nIn [this example script](exploration/train_sb3_agent.py) you can see all of the above steps unified.\n\nFor a quick impression in this README, find a minimal training and evaluation example here:\n\n```\n# Create environment(s); multiple environments for parallel training\nenvironments = [GymEnvironmentWrapper(ENV_ID) for _ in range(PARALLEL_ENVIRONMENTS)]\n\n# Create new agent\nagent = StableBaselinesAgent(\n algorithm=StableBaselinesAlgorithm.PPO,\n algorithm_parameters={\n \"policy\": \"MlpPolicy\"\n }\n)\n# Train agent\nagent.train(environments=environments, total_timesteps=100000)\n\n# Evaluate the model\nmean_reward, std_reward = agent.evaluate(evaluation_environment=environments[0])\n```\n\n## Development\n\n### Notebooks\n\nYou can use your module code (`src/`) in Jupyter notebooks without running into import errors by running:\n\n```\npoetry run jupyter notebook\n```\n\nor\n\n```\npoetry run jupyter-lab\n```\n\nThis starts the jupyter server inside the project's virtualenv.\n\nAssuming you already have Jupyter installed, you can make your virtual environment available as a separate kernel by running:\n\n```\npoetry add ipykernel\npoetry run python -m ipykernel install --user --name=\"reinforcement-learning-framework\"\n```\n\nNote that we mainly use notebooks for experiments, visualizations and reports. Every piece of functionality that is meant to be reused should go into module code and be imported into notebooks.\n\n### Testing\n\nWe use `pytest` as test framework. To execute the tests, please run\n\n```\npytest tests\n```\n\nTo run the tests with coverage information, please use\n\n```\npytest tests --cov=src --cov-report=html --cov-report=term\n```\n\nand have a look at the `htmlcov` folder, after the tests are done.\n\n### Distribution Package\n\nTo build a distribution package (wheel), please use\n\n```\npython setup.py bdist_wheel\n```\n\nthis will clean up the build folder and then run the `bdist_wheel` command.\n\n### Contributions\n\nBefore contributing, please set up the pre-commit hooks to reduce errors and ensure consistency\n\n```\npip install -U pre-commit\npre-commit install\n```\n\nIf you run into any issues, you can remove the hooks again with `pre-commit uninstall`.\n\n## License\n\n\u00a9 Alexander Zap\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "An easy-to-read Reinforcement Learning (RL) framework. Provides standardized interfaces and implementations to various Reinforcement Learning methods and environments. Also this is the main place to start your journey with Reinforcement Learning and learn from tutorials and examples.",
"version": "0.6.1",
"project_urls": {
"Homepage": "https://github.com/alexander-zap/reinforcement-learning-framework"
},
"split_keywords": [
"reinforcement learning",
" rl",
" imitation learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "de8a36e3c998896cc240c06ca6f6bf303e7fde0b9bf53421f0d1099d16c09b4e",
"md5": "6deb33f77a7ffa13e0697c5e182f41bf",
"sha256": "4f9f75249b0c17b577ea06410eae13d2ed9a1d2b296917613bdd5972e6cf36ee"
},
"downloads": -1,
"filename": "reinforcement_learning_framework-0.6.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6deb33f77a7ffa13e0697c5e182f41bf",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 44584,
"upload_time": "2024-12-19T13:39:25",
"upload_time_iso_8601": "2024-12-19T13:39:25.596137Z",
"url": "https://files.pythonhosted.org/packages/de/8a/36e3c998896cc240c06ca6f6bf303e7fde0b9bf53421f0d1099d16c09b4e/reinforcement_learning_framework-0.6.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3faafd3b7b5cfc3ceeffeea61c220f0d28166310a0ec144505ff5d59debea75e",
"md5": "31ee9b6a2fe529362b2569fa63117dc2",
"sha256": "4637e3a97b5cfcbe731279c15fee41783c47353ed77c3694abc3738eb8854d86"
},
"downloads": -1,
"filename": "reinforcement_learning_framework-0.6.1.tar.gz",
"has_sig": false,
"md5_digest": "31ee9b6a2fe529362b2569fa63117dc2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 33691,
"upload_time": "2024-12-19T13:39:27",
"upload_time_iso_8601": "2024-12-19T13:39:27.814750Z",
"url": "https://files.pythonhosted.org/packages/3f/aa/fd3b7b5cfc3ceeffeea61c220f0d28166310a0ec144505ff5d59debea75e/reinforcement_learning_framework-0.6.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-19 13:39:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "alexander-zap",
"github_project": "reinforcement-learning-framework",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "reinforcement-learning-framework"
}
JSON
