# remote-gym: Hosting Gym-environments remotely
This is a module to run Gym environments remotely, to enable splitting environment hosting and agent training into separate processes (or even separate machines).
Communication between the two processes is executed by using TLS and the gRPC protocol.
Adapted `dm_env_rpc` for `Gym.env` environments.
## Usage
### Main Features
- Use the `create_remote_environment_server` method to start a `Gym.env` environment as a remotely running environment.
- Use the `RemoteEnvironment` class to manage the connection to a remotely running environment (from `create_remote_environment_server`) and provide the standardized `Gym.env` interface to your agents through a `RemoteEnvironment` object.
- Basically: `remote-gym` is to `Gym.env` as what `dm_env_rpc` is to `dm_env`.
### Getting Started
In [this example script](exploration/start_remote_environment.py) you can see how to start a remotely running environment.
In [this accompanying script](exploration/start_environment_interaction.py) you can see how to connect to and interact with the previously started environment from a separate process.
For a quick impression in this README, find a minimal environment hosting and environment interaction example below.
First process:
```py
import logging
from remote_gym import create_remote_environment_server
server = create_remote_environment_server(
default_args={
"entrypoint": "exploration/remote_environment_entrypoint.py",
},
# IP of the machine hosting the remote environment; can also be 0.0.0.0
url=YOUR_SERVER_IP,
# port the remote environment should use on the hosting machine
port=PORT_FOR_REMOTE_ENVIRONMENT_TO_LISTEN,
# not using a tuple but setting this completely to None is also possible in case only a local connection is required
server_credentials_paths=("path/to/server.pem", "path/to/server-key.pem", "optional/path/to/ca.pem"),
)
try:
server.wait_for_termination()
except Exception as e:
server.stop(None)
logging.exception(e)
```
With an `entrypoint.py` like this:
```py
import gymnasium as gym
def create_environment(enable_rendering: bool, env_id: int, **kwargs) -> gym.Env:
return gym.make("Acrobot-v1")
```
Second process:
```py
from remote_gym import RemoteEnvironment
environment = RemoteEnvironment(
url=YOUR_SERVER_IP,
port=PORT_FOR_REMOTE_ENVIRONMENT_TO_RUN_ON,
# not using a tuple but setting this completely to None is also possible in case only a local connection is required
client_credentials_paths=("path/to/ca.pem", "optional/path/to/client.pem", "optional/path/to/client-key.pem"),
# can be set to "human" or "rgb_array" if `enable_rendering` was set to True in remote environment hosting process
render_mode=None,
)
done = False
episode_reward = 0
environment.reset()
while not done:
action = environment.action_space.sample()
_observation, reward, terminated, truncated, _info = environment.step(action)
episode_reward += reward
done = terminated or truncated
```
### Rendering
To preserve server resources and prevent network slowdowns, it is recommended to only enable rendering if required.
Renderings are automatically transferred from the remote management server to the RemoteEnvironment together with the
observation if the `render_mode` of the hosted environment is `rgb_array`.
The `render_mode` of the hosted environment should be controlled by the `entrypoint_kwargs` passed to the entrypoint.
## Set-Up
### Install all dependencies in your development environment
To set up your local development environment, please run:
```
poetry install
```
Behind the scenes, this creates a virtual environment and installs `remote_gym` along with its dependencies into a new virtualenv. Whenever you run `poetry run <command>`, that `<command>` is actually run inside the virtualenv managed by poetry.
You can now import functions and classes from the module with `import remote_gym`.
### Set-up for connecting the agent training process to remote environments running on a separate machine
Authenticating the communication channel via the connection of one machine to the other requires TLS (formerly SSL)
authentication.
This is achieved by using a [self-signed certificate](https://en.wikipedia.org/wiki/Self-signed_certificate),
meaning the certificate is not signed by a publicly trusted certificate authority (CA) but by a locally created CA.
> See https://github.com/joekottke/python-grpc-ssl for more details and a more in-depth tutorial on how to create the self-signed certificates.
All required configuration files to create a self-signed certificate chain can be found in the [ssl folder](/ssl).
1. The root certificate of the certificate authority (`ca.pem`) is created by following command:
```
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
```
1. The server certificate (`server.pem`) and respective private key (`server-key.pem`) is created by following command:
```
cfssl gencert -ca="ca.pem" -ca-key="ca-key.pem" -config="ca-config.json" server-csr.json | cfssljson -bare server
```
Make sure to add all known hostnames of the machine hosting the remote environment. You can now test, whether the
client is able to connect to the server by running both example scripts.
- [`start_remote_environment`](/exploration/start_remote_environment.py) `-u SERVER.IP.HERE -p 56765 --server_certificate path\to\server.pem --server_private_key path\to\server-key.pem`
- [`start_environment_interaction`](/exploration/start_environment_interaction.py) `-u SERVER.IP.HERE -p 56765 --root_certificate path\to\ca.pem`
If the connection is not successful and the training is not starting, you can investigate on the server
(remote environment hosting machine) which IP is unsuccessfully attempting a TLS authentication to your IP by using
the [Wireshark tool](https://www.wireshark.org/download.html) with the filter `tcp.flags.reset==1 or tls.alert_message.level`.
Afterward you can add this IP to your hostnames to the [server SSL config file](/ssl/server-csr.json).
3. Optional for client authentication on the machine connecting to the remote environment:
Create a client certificate (`client.pem`) and respective private key `client-key.pem` by running following command:
```
cfssl gencert -ca="ca.pem" -ca-key="ca-key.pem" -config="ca-config.json" client-csr.json | cfssljson -bare client
```
Then you can use all certificates and keys:
- [`start_remote_environment`](/exploration/start_remote_environment.py) `-u SERVER.IP.HERE -p 56765 --root_certificate path\to\ca.pem --server_certificate path\to\server.pem --server_private_key path\to\server-key.pem`
- [`start_environment_interaction`](/exploration/start_environment_interaction.py) `-u SERVER.IP.HERE -p 56765 --root_certificate path\to\ca.pem --client_certificate path\to\client.pem --client_private_key path\to\client-key.pem`
## Development
### Notebooks
You can use your module code (`src/`) in Jupyter notebooks without running into import errors by running:
```
poetry run jupyter notebook
```
or
```
poetry run jupyter-lab
```
This starts the jupyter server inside the project's virtualenv.
Assuming you already have Jupyter installed, you can make your virtual environment available as a separate kernel by running:
```
poetry add ipykernel
poetry run python -m ipykernel install --user --name="remote-gym"
```
Note that we mainly use notebooks for experiments, visualizations and reports. Every piece of functionality that is meant to be reused should go into module code and be imported into notebooks.
### Contributions
Before contributing, please set up the pre-commit hooks to reduce errors and ensure consistency
```
pip install -U pre-commit
pre-commit install
```
If you run into any issues, you can remove the hooks again with `pre-commit uninstall`.
## License
© Alexander Zap
Raw data
{
"_id": null,
"home_page": "https://github.com/alexander-zap/remote-gym",
"name": "remote-gym",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.11,>=3.8",
"maintainer_email": null,
"keywords": "reinforcement learning, rl, imitation learning, gym, gymnasium, dm_env",
"author": "Alexander Zap",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/05/c2/3f610441bcebe15b64a2d353aacf46b3c483d12ba9ccbb1a2e82f326c63e/remote_gym-0.6.0.tar.gz",
"platform": null,
"description": "# remote-gym: Hosting Gym-environments remotely\n\nThis is a module to run Gym environments remotely, to enable splitting environment hosting and agent training into separate processes (or even separate machines).\nCommunication between the two processes is executed by using TLS and the gRPC protocol.\n\nAdapted `dm_env_rpc` for `Gym.env` environments.\n\n## Usage\n\n### Main Features\n\n- Use the `create_remote_environment_server` method to start a `Gym.env` environment as a remotely running environment.\n- Use the `RemoteEnvironment` class to manage the connection to a remotely running environment (from `create_remote_environment_server`) and provide the standardized `Gym.env` interface to your agents through a `RemoteEnvironment` object.\n- Basically: `remote-gym` is to `Gym.env` as what `dm_env_rpc` is to `dm_env`.\n\n### Getting Started\n\nIn [this example script](exploration/start_remote_environment.py) you can see how to start a remotely running environment.\n\nIn [this accompanying script](exploration/start_environment_interaction.py) you can see how to connect to and interact with the previously started environment from a separate process.\n\nFor a quick impression in this README, find a minimal environment hosting and environment interaction example below.\n\nFirst process:\n\n```py\nimport logging\n\nfrom remote_gym import create_remote_environment_server\n\nserver = create_remote_environment_server(\n default_args={\n \"entrypoint\": \"exploration/remote_environment_entrypoint.py\",\n },\n # IP of the machine hosting the remote environment; can also be 0.0.0.0\n url=YOUR_SERVER_IP,\n # port the remote environment should use on the hosting machine\n port=PORT_FOR_REMOTE_ENVIRONMENT_TO_LISTEN,\n # not using a tuple but setting this completely to None is also possible in case only a local connection is required\n server_credentials_paths=(\"path/to/server.pem\", \"path/to/server-key.pem\", \"optional/path/to/ca.pem\"),\n)\n\ntry:\n server.wait_for_termination()\nexcept Exception as e:\n server.stop(None)\n logging.exception(e)\n```\n\nWith an `entrypoint.py` like this:\n\n```py\nimport gymnasium as gym\n\ndef create_environment(enable_rendering: bool, env_id: int, **kwargs) -> gym.Env:\n return gym.make(\"Acrobot-v1\")\n```\n\nSecond process:\n\n```py\nfrom remote_gym import RemoteEnvironment\n\nenvironment = RemoteEnvironment(\n url=YOUR_SERVER_IP,\n port=PORT_FOR_REMOTE_ENVIRONMENT_TO_RUN_ON,\n # not using a tuple but setting this completely to None is also possible in case only a local connection is required\n client_credentials_paths=(\"path/to/ca.pem\", \"optional/path/to/client.pem\", \"optional/path/to/client-key.pem\"),\n # can be set to \"human\" or \"rgb_array\" if `enable_rendering` was set to True in remote environment hosting process\n render_mode=None,\n)\n\ndone = False\nepisode_reward = 0\nenvironment.reset()\nwhile not done:\n action = environment.action_space.sample()\n _observation, reward, terminated, truncated, _info = environment.step(action)\n episode_reward += reward\n done = terminated or truncated\n```\n\n### Rendering\n\nTo preserve server resources and prevent network slowdowns, it is recommended to only enable rendering if required.\nRenderings are automatically transferred from the remote management server to the RemoteEnvironment together with the\nobservation if the `render_mode` of the hosted environment is `rgb_array`.\nThe `render_mode` of the hosted environment should be controlled by the `entrypoint_kwargs` passed to the entrypoint.\n\n## Set-Up\n\n### Install all dependencies in your development environment\n\nTo set up your local development environment, please run:\n\n```\npoetry install\n```\n\nBehind the scenes, this creates a virtual environment and installs `remote_gym` along with its dependencies into a new virtualenv. Whenever you run `poetry run <command>`, that `<command>` is actually run inside the virtualenv managed by poetry.\n\nYou can now import functions and classes from the module with `import remote_gym`.\n\n### Set-up for connecting the agent training process to remote environments running on a separate machine\n\nAuthenticating the communication channel via the connection of one machine to the other requires TLS (formerly SSL)\nauthentication.\nThis is achieved by using a [self-signed certificate](https://en.wikipedia.org/wiki/Self-signed_certificate),\nmeaning the certificate is not signed by a publicly trusted certificate authority (CA) but by a locally created CA.\n\n> See https://github.com/joekottke/python-grpc-ssl for more details and a more in-depth tutorial on how to create the self-signed certificates.\n\nAll required configuration files to create a self-signed certificate chain can be found in the [ssl folder](/ssl).\n\n1. The root certificate of the certificate authority (`ca.pem`) is created by following command:\n\n ```\n cfssl gencert -initca ca-csr.json | cfssljson -bare ca\n ```\n\n1. The server certificate (`server.pem`) and respective private key (`server-key.pem`) is created by following command:\n\n ```\n cfssl gencert -ca=\"ca.pem\" -ca-key=\"ca-key.pem\" -config=\"ca-config.json\" server-csr.json | cfssljson -bare server\n ```\n\nMake sure to add all known hostnames of the machine hosting the remote environment. You can now test, whether the\nclient is able to connect to the server by running both example scripts.\n\n- [`start_remote_environment`](/exploration/start_remote_environment.py) `-u SERVER.IP.HERE -p 56765 --server_certificate path\\to\\server.pem --server_private_key path\\to\\server-key.pem`\n- [`start_environment_interaction`](/exploration/start_environment_interaction.py) `-u SERVER.IP.HERE -p 56765 --root_certificate path\\to\\ca.pem`\n\nIf the connection is not successful and the training is not starting, you can investigate on the server\n(remote environment hosting machine) which IP is unsuccessfully attempting a TLS authentication to your IP by using\nthe [Wireshark tool](https://www.wireshark.org/download.html) with the filter `tcp.flags.reset==1 or tls.alert_message.level`.\n\nAfterward you can add this IP to your hostnames to the [server SSL config file](/ssl/server-csr.json).\n\n3. Optional for client authentication on the machine connecting to the remote environment:\n\n Create a client certificate (`client.pem`) and respective private key `client-key.pem` by running following command:\n\n ```\n cfssl gencert -ca=\"ca.pem\" -ca-key=\"ca-key.pem\" -config=\"ca-config.json\" client-csr.json | cfssljson -bare client\n ```\n\nThen you can use all certificates and keys:\n\n- [`start_remote_environment`](/exploration/start_remote_environment.py) `-u SERVER.IP.HERE -p 56765 --root_certificate path\\to\\ca.pem --server_certificate path\\to\\server.pem --server_private_key path\\to\\server-key.pem`\n- [`start_environment_interaction`](/exploration/start_environment_interaction.py) `-u SERVER.IP.HERE -p 56765 --root_certificate path\\to\\ca.pem --client_certificate path\\to\\client.pem --client_private_key path\\to\\client-key.pem`\n\n## Development\n\n### Notebooks\n\nYou can use your module code (`src/`) in Jupyter notebooks without running into import errors by running:\n\n```\npoetry run jupyter notebook\n```\n\nor\n\n```\npoetry run jupyter-lab\n```\n\nThis starts the jupyter server inside the project's virtualenv.\n\nAssuming you already have Jupyter installed, you can make your virtual environment available as a separate kernel by running:\n\n```\npoetry add ipykernel\npoetry run python -m ipykernel install --user --name=\"remote-gym\"\n```\n\nNote that we mainly use notebooks for experiments, visualizations and reports. Every piece of functionality that is meant to be reused should go into module code and be imported into notebooks.\n\n### Contributions\n\nBefore contributing, please set up the pre-commit hooks to reduce errors and ensure consistency\n\n```\npip install -U pre-commit\npre-commit install\n```\n\nIf you run into any issues, you can remove the hooks again with `pre-commit uninstall`.\n\n## License\n\n\u00a9 Alexander Zap\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "This is a module to run Gym environments remotely, to enable splitting environment hosting and agent training into separate processes (or even separate machines).",
"version": "0.6.0",
"project_urls": {
"Homepage": "https://github.com/alexander-zap/remote-gym"
},
"split_keywords": [
"reinforcement learning",
" rl",
" imitation learning",
" gym",
" gymnasium",
" dm_env"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7bfd2f9a2bfc64f138510762ea10308bc6a300a2cf825b4a6521892fa8144995",
"md5": "0c0ada00b4b0e701c0e0bb162e17e02a",
"sha256": "e32ff90bc5c023af3cbed1a2b58e5124b5761fc9e78065eeff5415f75c84d05c"
},
"downloads": -1,
"filename": "remote_gym-0.6.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0c0ada00b4b0e701c0e0bb162e17e02a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.11,>=3.8",
"size": 15978,
"upload_time": "2024-12-16T18:19:20",
"upload_time_iso_8601": "2024-12-16T18:19:20.058538Z",
"url": "https://files.pythonhosted.org/packages/7b/fd/2f9a2bfc64f138510762ea10308bc6a300a2cf825b4a6521892fa8144995/remote_gym-0.6.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "05c23f610441bcebe15b64a2d353aacf46b3c483d12ba9ccbb1a2e82f326c63e",
"md5": "f61114b3b931a4f2b146c682f71e0d9f",
"sha256": "2a7160b54499a8826d67cd543c0b5d10c0a60164b3d474bacf065d91674e1503"
},
"downloads": -1,
"filename": "remote_gym-0.6.0.tar.gz",
"has_sig": false,
"md5_digest": "f61114b3b931a4f2b146c682f71e0d9f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.11,>=3.8",
"size": 17677,
"upload_time": "2024-12-16T18:19:23",
"upload_time_iso_8601": "2024-12-16T18:19:23.248734Z",
"url": "https://files.pythonhosted.org/packages/05/c2/3f610441bcebe15b64a2d353aacf46b3c483d12ba9ccbb1a2e82f326c63e/remote_gym-0.6.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-16 18:19:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "alexander-zap",
"github_project": "remote-gym",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "remote-gym"
}
JSON
