# Launchpad
[](https://badge.fury.io/py/dm-launchpad)
Launchpad is a library that simplifies writing distributed programs by
seamlessly launching them on a variety of different platforms. Switching between
local and distributed execution requires only a flag change.
Launchpad introduces a programming model that represents a distributed system
as a graph data structure (a **Program**) describing the system’s topology.
Each node in the program graph represents a service in the distributed system,
i.e. the fundamental unit of computation that we are interested in running.
As nodes are added to this graph, Launchpad constructs a handle for each of them.
A handle ultimately represents a client to the yet-to-be-constructed service.
A directed edge in the program graph, representing communication between
two services, is created when the handle associated with one node is given
to another at construction time. This edge originates from the receiving node,
indicating that the receiving node will be the one initiating communication.
This process allows Launchpad to define cross-service communication simply
by passing handles to nodes. Launchpad provides a number of node types,
including:
* **PyNode** - a simple node executing provided Python code upon entry.
It is similar to a main function, but with the distinction that
each node may be running in separate processes and on different machines.
* **CourierNode** - it enables cross-node communication. **CourierNodes** can
communicate by calling public methods on each other either synchronously
or asynchronously via futures. The underlying remote procedure calls
are handled transparently by Launchpad.
* **ReverbNode** - it exposes functionality of Reverb, an easy-to-use data
storage and transport system primarily used by RL algorithms as
an experience replay. You can read more about Reverb
[here](https://github.com/deepmind/reverb).
* **MultiThreadingColocation** - allows to colocate multiple other nodes in
a single process.
* **MultiProcessingColocation** - allows to colocate multiple other nodes as
sub processes.
Using Launchpad involves writing nodes and defining the topology of your
distributed program by passing to each node references of the other nodes that
it can communicate with. The core data structure dealing with this is called a
**Launchpad program**, which can then be executed seamlessly with a number of
supported runtimes.
## Supported launch types
Launchpad supports a number of launch types, both for running programs on
a single machine, in a distributed manner, or in a form of a test. Launch type
can be controlled by the `launch_type` argument passed to `lp.launch` method,
or specified through the `--lp_launch_type` command line flag.
Please refer to the documentation of the [LaunchType](<https://github.com/deepmind/launchpad/search?q="class%20LaunchType">)
for details.
## Table of Contents
- [Installation](#installation)
- [Quick Start](docs/get_started.md)
- [Citing Launchpad](#citing-Launchpad)
- [Acknowledgements](#acknowledgements)
- [Other resources](#other-resources)
## Installation
Please keep in mind that Launchpad is not hardened for production use, and while we
do our best to keep things in working order, things may break or segfault.
> :warning: Launchpad currently only supports Linux based OSes.
The recommended way to install Launchpad is with `pip`. We also provide
instructions to build from source using the same docker images we use for
releases.
TensorFlow can be installed separately or as part of the `pip` install.
Installing TensorFlow as part of the install ensures compatibility.
```shell
$ pip install dm-launchpad[tensorflow]
# Without Tensorflow install and version dependency check.
$ pip install dm-launchpad
```
### Nightly builds
[](https://badge.fury.io/py/dm-launchpad-nightly)
```shell
$ pip install dm-launchpad-nightly[tensorflow]
# Without Tensorflow install and version dependency check.
$ pip install dm-launchpad-nightly
```
Similarily, [Reverb](https://github.com/deepmind/reverb) can be installed
ensuring compatibility:
```shell
$ pip install dm-launchpad[reverb]
```
### Develop Launchpad inside a docker container
The most convenient way to develop Launchpad is with Docker.
This way you can compile and test Launchpad inside a container without
having to install anything on your host machine, while you can still
use your editor of choice for making code changes.
The steps are as follows.
Checkout Launchpad's source code from GitHub.
```
$ git checkout https://github.com/deepmind/launchpad.git
$ cd launchpad
```
Build the Docker container to be used for compiling and testing Launchpad.
You can specify `tensorflow_pip` parameter to set the version
of Tensorflow to build against. You can also specify which version(s) of Python
container should support. The command below enables support for Python
3.7, 3.8, 3.9 and 3.10.
```
$ docker build --tag launchpad:devel \
--build-arg tensorflow_pip=tensorflow==2.3.0 \
--build-arg python_version="3.7 3.8 3.9 3.10" - < docker/build.dockerfile
```
The next step is to enter the built Docker image, binding checked out
Launchpad's sources to /tmp/launchpad within the container.
```
$ docker run --rm --mount "type=bind,src=$PWD,dst=/tmp/launchpad" \
-it launchpad:devel bash
```
At this point you can build and install Launchpad within the container by
executing:
```
$ /tmp/launchpad/oss_build.sh
```
By default it builds Python 3.8 version, you can change that with `--python`
flag.
```
$ /tmp/launchpad/oss_build.sh --python 3.8
```
To make sure installation was successful and Launchpad works as expected, you
can run some examples provided:
```
$ python3.8 -m launchpad.examples.hello_world.launch
$ python3.8 -m launchpad.examples.consumer_producers.launch --lp_launch_type=local_mp
```
To make changes to Launchpad codebase, edit sources checked out from GitHub
directly on your host machine (outside of the Docker container). All changes are
visible inside the Docker container. To recompile just run the `oss_build.sh`
script again from the Docker container. In order to reduce compilation time of
the consecutive runs, make sure to not exit the Docker container.
## Citing Launchpad
If you use Launchpad in your work, please cite the accompanying
[technical report](https://arxiv.org/pdf/2106.04516):
```bibtex
@article{yang2021launchpad,
title={Launchpad: A Programming Model for Distributed Machine Learning
Research},
author={Fan Yang and Gabriel Barth-Maron and Piotr Stańczyk and Matthew
Hoffman and Siqi Liu and Manuel Kroiss and Aedan Pope and Alban
Rrustemi},
year={2021},
journal={arXiv preprint arXiv:2106.04516},
url={https://arxiv.org/abs/2106.04516},
}
```
## Acknowledgements
We greatly appreciate all the help from [Reverb](https://github.com/deepmind/reverb)
and [TF-Agents](https://github.com/tensorflow/agents) teams in setting
up building and testing setup for Launchpad.
## Other resources
* [FAQ](docs/faq.md)
Raw data
{
"_id": null,
"home_page": "https://github.com/deepmind/launchpad",
"name": "dm-launchpad-nightly",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3",
"maintainer_email": "",
"keywords": "deepmind reinforcement learning machine distributed",
"author": "DeepMind",
"author_email": "DeepMind <no-reply@google.com>",
"download_url": "",
"platform": null,
"description": "# Launchpad\n\n[](https://badge.fury.io/py/dm-launchpad)\n\nLaunchpad is a library that simplifies writing distributed programs by\nseamlessly launching them on a variety of different platforms. Switching between\nlocal and distributed execution requires only a flag change.\n\nLaunchpad introduces a programming model that represents a distributed system\nas a graph data structure (a **Program**) describing the system\u2019s topology.\nEach node in the program graph represents a service in the distributed system,\ni.e. the fundamental unit of computation that we are interested in running.\nAs nodes are added to this graph, Launchpad constructs a handle for each of them.\nA handle ultimately represents a client to the yet-to-be-constructed service.\nA directed edge in the program graph, representing communication between\ntwo services, is created when the handle associated with one node is given\nto another at construction time. This edge originates from the receiving node,\nindicating that the receiving node will be the one initiating communication.\nThis process allows Launchpad to define cross-service communication simply\nby passing handles to nodes. Launchpad provides a number of node types,\nincluding:\n\n* **PyNode** - a simple node executing provided Python code upon entry.\n It is similar to a main function, but with the distinction that\n each node may be running in separate processes and on different machines.\n* **CourierNode** - it enables cross-node communication. **CourierNodes** can\n communicate by calling public methods on each other either synchronously\n or asynchronously via futures. The underlying remote procedure calls\n are handled transparently by Launchpad.\n* **ReverbNode** - it exposes functionality of Reverb, an easy-to-use data\n storage and transport system primarily used by RL algorithms as\n an experience replay. You can read more about Reverb\n [here](https://github.com/deepmind/reverb).\n* **MultiThreadingColocation** - allows to colocate multiple other nodes in\n a single process.\n* **MultiProcessingColocation** - allows to colocate multiple other nodes as\n sub processes.\n\nUsing Launchpad involves writing nodes and defining the topology of your\ndistributed program by passing to each node references of the other nodes that\nit can communicate with. The core data structure dealing with this is called a\n**Launchpad program**, which can then be executed seamlessly with a number of\nsupported runtimes.\n\n## Supported launch types\nLaunchpad supports a number of launch types, both for running programs on\na single machine, in a distributed manner, or in a form of a test. Launch type\ncan be controlled by the `launch_type` argument passed to `lp.launch` method,\nor specified through the `--lp_launch_type` command line flag.\nPlease refer to the documentation of the [LaunchType](<https://github.com/deepmind/launchpad/search?q=\"class%20LaunchType\">)\nfor details.\n\n## Table of Contents\n\n- [Installation](#installation)\n- [Quick Start](docs/get_started.md)\n- [Citing Launchpad](#citing-Launchpad)\n- [Acknowledgements](#acknowledgements)\n- [Other resources](#other-resources)\n\n## Installation\n\nPlease keep in mind that Launchpad is not hardened for production use, and while we\ndo our best to keep things in working order, things may break or segfault.\n\n> :warning: Launchpad currently only supports Linux based OSes.\n\nThe recommended way to install Launchpad is with `pip`. We also provide\ninstructions to build from source using the same docker images we use for\nreleases.\n\nTensorFlow can be installed separately or as part of the `pip` install.\nInstalling TensorFlow as part of the install ensures compatibility.\n\n```shell\n$ pip install dm-launchpad[tensorflow]\n\n# Without Tensorflow install and version dependency check.\n$ pip install dm-launchpad\n```\n\n### Nightly builds\n\n[](https://badge.fury.io/py/dm-launchpad-nightly)\n\n```shell\n$ pip install dm-launchpad-nightly[tensorflow]\n\n# Without Tensorflow install and version dependency check.\n$ pip install dm-launchpad-nightly\n```\n\nSimilarily, [Reverb](https://github.com/deepmind/reverb) can be installed\nensuring compatibility:\n\n```shell\n$ pip install dm-launchpad[reverb]\n```\n\n### Develop Launchpad inside a docker container\n\nThe most convenient way to develop Launchpad is with Docker.\nThis way you can compile and test Launchpad inside a container without\nhaving to install anything on your host machine, while you can still\nuse your editor of choice for making code changes.\nThe steps are as follows.\n\nCheckout Launchpad's source code from GitHub.\n```\n$ git checkout https://github.com/deepmind/launchpad.git\n$ cd launchpad\n```\n\nBuild the Docker container to be used for compiling and testing Launchpad.\nYou can specify `tensorflow_pip` parameter to set the version\nof Tensorflow to build against. You can also specify which version(s) of Python\ncontainer should support. The command below enables support for Python\n3.7, 3.8, 3.9 and 3.10.\n```\n$ docker build --tag launchpad:devel \\\n --build-arg tensorflow_pip=tensorflow==2.3.0 \\\n --build-arg python_version=\"3.7 3.8 3.9 3.10\" - < docker/build.dockerfile\n```\n\nThe next step is to enter the built Docker image, binding checked out\nLaunchpad's sources to /tmp/launchpad within the container.\n```\n$ docker run --rm --mount \"type=bind,src=$PWD,dst=/tmp/launchpad\" \\\n -it launchpad:devel bash\n```\n\nAt this point you can build and install Launchpad within the container by\nexecuting:\n```\n$ /tmp/launchpad/oss_build.sh\n```\n\nBy default it builds Python 3.8 version, you can change that with `--python`\nflag.\n```\n$ /tmp/launchpad/oss_build.sh --python 3.8\n```\n\nTo make sure installation was successful and Launchpad works as expected, you\ncan run some examples provided:\n```\n$ python3.8 -m launchpad.examples.hello_world.launch\n$ python3.8 -m launchpad.examples.consumer_producers.launch --lp_launch_type=local_mp\n```\n\nTo make changes to Launchpad codebase, edit sources checked out from GitHub\ndirectly on your host machine (outside of the Docker container). All changes are\nvisible inside the Docker container. To recompile just run the `oss_build.sh`\nscript again from the Docker container. In order to reduce compilation time of\nthe consecutive runs, make sure to not exit the Docker container.\n\n\n## Citing Launchpad\n\nIf you use Launchpad in your work, please cite the accompanying\n[technical report](https://arxiv.org/pdf/2106.04516):\n\n```bibtex\n@article{yang2021launchpad,\n title={Launchpad: A Programming Model for Distributed Machine Learning\n Research},\n author={Fan Yang and Gabriel Barth-Maron and Piotr Sta\u0144czyk and Matthew\n Hoffman and Siqi Liu and Manuel Kroiss and Aedan Pope and Alban\n Rrustemi},\n year={2021},\n journal={arXiv preprint arXiv:2106.04516},\n url={https://arxiv.org/abs/2106.04516},\n}\n```\n\n## Acknowledgements\n\nWe greatly appreciate all the help from [Reverb](https://github.com/deepmind/reverb)\nand [TF-Agents](https://github.com/tensorflow/agents) teams in setting\nup building and testing setup for Launchpad.\n\n## Other resources\n\n* [FAQ](docs/faq.md)\n\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Launchpad is a library that simplifies writing distributed programs and seamlessly launching them on a range of supported platforms.",
"version": "0.5.2.dev20230323",
"split_keywords": [
"deepmind",
"reinforcement",
"learning",
"machine",
"distributed"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "67ce1386a6547cbb946d937e39010afdb0f8d78b5ce00ccb7827616c793d0808",
"md5": "e99debb89a8ca5f4791963625d256def",
"sha256": "5e87049710985e81be097dfa0820c84902f9f9346a1472302fa006db956fd66f"
},
"downloads": -1,
"filename": "dm_launchpad_nightly-0.5.2.dev20230323-cp310-cp310-manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "e99debb89a8ca5f4791963625d256def",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3",
"size": 14146901,
"upload_time": "2023-03-23T10:21:26",
"upload_time_iso_8601": "2023-03-23T10:21:26.868489Z",
"url": "https://files.pythonhosted.org/packages/67/ce/1386a6547cbb946d937e39010afdb0f8d78b5ce00ccb7827616c793d0808/dm_launchpad_nightly-0.5.2.dev20230323-cp310-cp310-manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a56b8185212ae3e84a209da4886f1e9341a047b766e6db1fc4dda96e649a3fb2",
"md5": "a9cfbd78d8a2a5ddcf6fab4129dc7e32",
"sha256": "e5ceae3d0f70fb05ad546270709046460dbfedaa0dc271bbaee34a5a591d1b97"
},
"downloads": -1,
"filename": "dm_launchpad_nightly-0.5.2.dev20230323-cp39-cp39-manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "a9cfbd78d8a2a5ddcf6fab4129dc7e32",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3",
"size": 14149050,
"upload_time": "2023-03-23T10:21:31",
"upload_time_iso_8601": "2023-03-23T10:21:31.073795Z",
"url": "https://files.pythonhosted.org/packages/a5/6b/8185212ae3e84a209da4886f1e9341a047b766e6db1fc4dda96e649a3fb2/dm_launchpad_nightly-0.5.2.dev20230323-cp39-cp39-manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-23 10:21:26",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "deepmind",
"github_project": "launchpad",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "dm-launchpad-nightly"
}
JSON
