# How to develop and build Reverb with the Docker containers
## Overview
This document covers a couple scenarios:
* <a href='#Release'>Create a Reverb release</a>
* <a href='#Develop'>Develop Reverb inside a Docker container</a>
* <a href='#builds-tips-and-hints'>Build tips and hints</a>
While there is overlap in the above scenarios, treating them separately seems
the most clear at the moment. Before you get started, setup a local variable
pointing to your local Reverb GitHub repo.
```shell
$ export REVERB_DIR=/path/to/reverb/github/repo
```
<a id='Release'></a>
## Create a stable Reverb release
There are two steps for building the Reverb package.
* Build the Docker container with the version of TensorFlow to build Reverb
against.
* Execute the build and declare any specific TensorFlow dependency for the
pip install. The dependency is only enforced if the user uses
`pip install reverb[tensorflow]`.
Execute from the root of the git repository. The end result will end up in
`$REVERB_DIR/dist`.
```shell
##################################
# Creates the Docker container.
##################################
# Builds the container with Python 3.8, 3.9, 3.10, and 3.11. Set the
# `build-arg tensorflow_pip` to the version of TensorFlow to build against.
# If building against an RC, use == rather than ~= for `tensorflow_pip`.
$ docker build --pull --no-cache \
--tag tensorflow:reverb_release \
--build-arg tensorflow_pip=tensorflow~=2.8.0 \
--build-arg python_version="python3.8 python3.9 python3.10 python3.11" \
- < "$REVERB_DIR/docker/release.dockerfile"
#################################################
# Builds Reverb against TensorFlow stable or rc.
#################################################
# Builds Reverb against most recent stable release of TensorFlow and
# requires `tensorflow~=2.8.0` if using `pip install reverb[tensorflow]`.
# Packages for Python 3.8, 3.9, 3.10, and 3.11 are created.
$ docker run --rm --mount "type=bind,src=$REVERB_DIR,dst=/tmp/reverb" \
tensorflow:reverb_release bash oss_build.sh --clean true \
--tf_dep_override "tensorflow~=2.11.0" --release --python "3.8 3.9 3.10 3.11"
# Builds Reverb against an RC of TensorFlow. `>=` and `~=` are not effective
# because pip does not recognize 2.4.0rc0 as greater than 2.3.0. RC builds need
# to have a strict dependency on the RC of TensorFlow used.
$ docker run --rm --mount "type=bind,src=$REVERB_DIR,dst=/tmp/reverb" \
tensorflow:reverb_release bash oss_build.sh --clean true \
--tf_dep_override "tensorflow==2.12.0rc0" --release --python "3.8 3.9 3.10 3.11"
# Builds a debug version of Reverb. The debug version is not labeled as debug
# as that can result in a user installing both the debug and regular packages
# making it unclear which is installed as they both have the same package
# namespace. The command below puts the .whl files in ./dist/debug/**.
# Debug builds are ~90M compared to normal builds that are closer to 7M.
$ docker run --rm --mount "type=bind,src=$REVERB_DIR,dst=/tmp/reverb" \
tensorflow:reverb_release bash oss_build.sh --clean true --debug_build true \
--output_dir /tmp/reverb/dist/debug/ --tf_dep_override "tensorflow~=2.10.0" \
--release --python "3.8 3.9 3.10 3.11"
```
<a id='Develop'></a>
## Develop Reverb inside a Docker container
1. Build the Docker container. By default the container is setup for python 3.9.
Use the `python_version` arg to configure the container with 3.8 or 3.9.
```shell
$ docker build --tag tensorflow:reverb - < "$REVERB_DIR/docker/release.dockerfile"
# Alternatively you can build the container with Python 3.8 support.
$ docker build --tag tensorflow:reverb \
--build-arg python_version=python3.8 \
- < "$REVERB_DIR/docker/release.dockerfile"
```
1. Run and enter the Docker container.
```shell
$ docker run --rm -it \
--mount "type=bind,src=$REVERB_DIR,dst=/tmp/reverb" \
--mount "type=bind,src=$HOME/.gitconfig,dst=/etc/gitconfig,ro" \
--name reverb tensorflow:reverb bash
```
1. Compile Reverb.
```shell
$ python3.8 configure.py
$ bazel build -c opt //reverb/pip_package:build_pip_package
```
1. Build the .whl file and output it to `/tmp/reverb_build/dist/`.
```shell
$ ./bazel-bin/reverb/pip_package/build_pip_package \
--dst /tmp/reverb_build/dist/
```
1. Install the .whl file.
```shell
# If multiple versions were built, pass the exact wheel to install rather than
# *.whl.
$ $PYTHON_BIN_PATH -mpip install --upgrade /tmp/reverb_build/dist/*.whl
```
<a id='#builds-tips-and-hints'></a>
## Builds Tips and Hints
### protoc / protobuf version mismatch
There is a
[check in the Reverb build process](https://github.com/deepmind/reverb/blob/master/third_party/protobuf.BUILD)
that checks if the protoc library in Tensorflow matches what Reverb is using. It
throws this error if there is a mismatch: "Please update the PROTOC_VERSION in
your WORKSPACE". Here for search reasons :-).
Tensorflow sets its version of Protobuf in this
[WORKSPACE](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/workspace2.bzl)
file. Reverb sets its version of protoc in this
[WORKSPACE](https://github.com/deepmind/reverb/blob/master/WORKSPACE) file. The
twist is the protobuf people release protoc as 21.9 and the protobuf library as
3.21.0 as seen
[here](https://github.com/protocolbuffers/protobuf/releases/tag/v21.12).
Until Feb-2023, Tensorflow was using a version of Protobuf that was ~2 years
old. In Feb 2023, TF jumped to the latest protobuf 3.21.9.
Note: While I am pretty sure Tensorflow was on 21.9 as of Feb-2023, The checker
in Reverb saw it as 21.0. 21.0 worked so I (tobyboyd) did not look into it any
farther.
Raw data
{
"_id": null,
"home_page": "https://github.com/engichang1467/reverb",
"name": "dm-reverb-macos",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3",
"maintainer_email": null,
"keywords": "tensorflow deepmind reinforcement learning machine replay jax",
"author": "Michael Chang",
"author_email": "mchang0926@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/6e/be/7d3583e4b4b4f83a11edc2b85302e263305f7b9c3789df1150f523a929e6/dm-reverb-macos-0.11.0.dev0.tar.gz",
"platform": null,
"description": "# How to develop and build Reverb with the Docker containers\n\n## Overview\n\nThis document covers a couple scenarios:\n\n * <a href='#Release'>Create a Reverb release</a>\n * <a href='#Develop'>Develop Reverb inside a Docker container</a>\n * <a href='#builds-tips-and-hints'>Build tips and hints</a>\n\nWhile there is overlap in the above scenarios, treating them separately seems\nthe most clear at the moment. Before you get started, setup a local variable\npointing to your local Reverb GitHub repo.\n\n```shell\n$ export REVERB_DIR=/path/to/reverb/github/repo\n```\n\n<a id='Release'></a>\n## Create a stable Reverb release\n\nThere are two steps for building the Reverb package.\n\n * Build the Docker container with the version of TensorFlow to build Reverb\n against.\n * Execute the build and declare any specific TensorFlow dependency for the\n pip install. The dependency is only enforced if the user uses\n `pip install reverb[tensorflow]`.\n\nExecute from the root of the git repository. The end result will end up in\n`$REVERB_DIR/dist`.\n\n```shell\n\n##################################\n# Creates the Docker container.\n##################################\n# Builds the container with Python 3.8, 3.9, 3.10, and 3.11. Set the\n# `build-arg tensorflow_pip` to the version of TensorFlow to build against.\n# If building against an RC, use == rather than ~= for `tensorflow_pip`.\n$ docker build --pull --no-cache \\\n --tag tensorflow:reverb_release \\\n --build-arg tensorflow_pip=tensorflow~=2.8.0 \\\n --build-arg python_version=\"python3.8 python3.9 python3.10 python3.11\" \\\n - < \"$REVERB_DIR/docker/release.dockerfile\"\n\n#################################################\n# Builds Reverb against TensorFlow stable or rc.\n#################################################\n\n# Builds Reverb against most recent stable release of TensorFlow and\n# requires `tensorflow~=2.8.0` if using `pip install reverb[tensorflow]`.\n# Packages for Python 3.8, 3.9, 3.10, and 3.11 are created.\n$ docker run --rm --mount \"type=bind,src=$REVERB_DIR,dst=/tmp/reverb\" \\\n tensorflow:reverb_release bash oss_build.sh --clean true \\\n --tf_dep_override \"tensorflow~=2.11.0\" --release --python \"3.8 3.9 3.10 3.11\"\n\n# Builds Reverb against an RC of TensorFlow. `>=` and `~=` are not effective\n# because pip does not recognize 2.4.0rc0 as greater than 2.3.0. RC builds need\n# to have a strict dependency on the RC of TensorFlow used.\n$ docker run --rm --mount \"type=bind,src=$REVERB_DIR,dst=/tmp/reverb\" \\\n tensorflow:reverb_release bash oss_build.sh --clean true \\\n --tf_dep_override \"tensorflow==2.12.0rc0\" --release --python \"3.8 3.9 3.10 3.11\"\n\n# Builds a debug version of Reverb. The debug version is not labeled as debug\n# as that can result in a user installing both the debug and regular packages\n# making it unclear which is installed as they both have the same package\n# namespace. The command below puts the .whl files in ./dist/debug/**.\n# Debug builds are ~90M compared to normal builds that are closer to 7M.\n$ docker run --rm --mount \"type=bind,src=$REVERB_DIR,dst=/tmp/reverb\" \\\n tensorflow:reverb_release bash oss_build.sh --clean true --debug_build true \\\n --output_dir /tmp/reverb/dist/debug/ --tf_dep_override \"tensorflow~=2.10.0\" \\\n --release --python \"3.8 3.9 3.10 3.11\"\n\n```\n\n<a id='Develop'></a>\n## Develop Reverb inside a Docker container\n\n1. Build the Docker container. By default the container is setup for python 3.9.\n Use the `python_version` arg to configure the container with 3.8 or 3.9.\n\n ```shell\n $ docker build --tag tensorflow:reverb - < \"$REVERB_DIR/docker/release.dockerfile\"\n\n # Alternatively you can build the container with Python 3.8 support.\n $ docker build --tag tensorflow:reverb \\\n --build-arg python_version=python3.8 \\\n - < \"$REVERB_DIR/docker/release.dockerfile\"\n ```\n\n1. Run and enter the Docker container.\n\n ```shell\n $ docker run --rm -it \\\n --mount \"type=bind,src=$REVERB_DIR,dst=/tmp/reverb\" \\\n --mount \"type=bind,src=$HOME/.gitconfig,dst=/etc/gitconfig,ro\" \\\n --name reverb tensorflow:reverb bash\n ```\n\n1. Compile Reverb.\n\n ```shell\n $ python3.8 configure.py\n $ bazel build -c opt //reverb/pip_package:build_pip_package\n ```\n\n1. Build the .whl file and output it to `/tmp/reverb_build/dist/`.\n\n ```shell\n $ ./bazel-bin/reverb/pip_package/build_pip_package \\\n --dst /tmp/reverb_build/dist/\n ```\n\n1. Install the .whl file.\n\n ```shell\n # If multiple versions were built, pass the exact wheel to install rather than\n # *.whl.\n $ $PYTHON_BIN_PATH -mpip install --upgrade /tmp/reverb_build/dist/*.whl\n ```\n\n<a id='#builds-tips-and-hints'></a>\n\n## Builds Tips and Hints\n\n### protoc / protobuf version mismatch\n\nThere is a\n[check in the Reverb build process](https://github.com/deepmind/reverb/blob/master/third_party/protobuf.BUILD)\nthat checks if the protoc library in Tensorflow matches what Reverb is using. It\nthrows this error if there is a mismatch: \"Please update the PROTOC_VERSION in\nyour WORKSPACE\". Here for search reasons :-).\n\nTensorflow sets its version of Protobuf in this\n[WORKSPACE](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/workspace2.bzl)\nfile. Reverb sets its version of protoc in this\n[WORKSPACE](https://github.com/deepmind/reverb/blob/master/WORKSPACE) file. The\ntwist is the protobuf people release protoc as 21.9 and the protobuf library as\n3.21.0 as seen\n[here](https://github.com/protocolbuffers/protobuf/releases/tag/v21.12).\n\nUntil Feb-2023, Tensorflow was using a version of Protobuf that was ~2 years\nold. In Feb 2023, TF jumped to the latest protobuf 3.21.9.\n\nNote: While I am pretty sure Tensorflow was on 21.9 as of Feb-2023, The checker\nin Reverb saw it as 21.0. 21.0 worked so I (tobyboyd) did not look into it any\nfarther.\n\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Reverb is an efficient and easy-to-use data storage and transport system designed for machine learning research.",
"version": "0.11.0.dev0",
"project_urls": {
"Homepage": "https://github.com/engichang1467/reverb"
},
"split_keywords": [
"tensorflow",
"deepmind",
"reinforcement",
"learning",
"machine",
"replay",
"jax"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "851bf990d3b0b2a184d4986a6ab9db5ab45810c659e427a26490b55b20feb753",
"md5": "a16e4b10d88014b650562e5dd9d954f6",
"sha256": "d4fa0ca15ae86fe922566de782c5eb7ba0afb0c557459d79cf9b0b4d29c9326e"
},
"downloads": -1,
"filename": "dm_reverb_macos-0.11.0.dev0-cp36-cp36m-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "a16e4b10d88014b650562e5dd9d954f6",
"packagetype": "bdist_wheel",
"python_version": "cp36",
"requires_python": ">=3",
"size": 3967,
"upload_time": "2024-12-08T03:09:05",
"upload_time_iso_8601": "2024-12-08T03:09:05.908620Z",
"url": "https://files.pythonhosted.org/packages/85/1b/f990d3b0b2a184d4986a6ab9db5ab45810c659e427a26490b55b20feb753/dm_reverb_macos-0.11.0.dev0-cp36-cp36m-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6ebe7d3583e4b4b4f83a11edc2b85302e263305f7b9c3789df1150f523a929e6",
"md5": "f96fa03a022c1ebef6a986e885b28c42",
"sha256": "75ce80a4f3fa4ad083aaf5d6875a9fa8cc53ae10daaa33d8c0e6cc3c98ed8918"
},
"downloads": -1,
"filename": "dm-reverb-macos-0.11.0.dev0.tar.gz",
"has_sig": false,
"md5_digest": "f96fa03a022c1ebef6a986e885b28c42",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3",
"size": 5639,
"upload_time": "2024-12-08T03:09:07",
"upload_time_iso_8601": "2024-12-08T03:09:07.784960Z",
"url": "https://files.pythonhosted.org/packages/6e/be/7d3583e4b4b4f83a11edc2b85302e263305f7b9c3789df1150f523a929e6/dm-reverb-macos-0.11.0.dev0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-08 03:09:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "engichang1467",
"github_project": "reverb",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "dm-reverb-macos"
}
JSON
