# pygloo
Pygloo provides Python bindings for [gloo](https://github.com/facebookincubator/gloo).
It is implemented using [pybind11](https://github.com/pybind/pybind11).
It is currenlty used in [Ray for collective communication](https://github.com/ray-project/ray/tree/master/python/ray/util/collective) between CPUs.
## Requirements
```python
Python >= 3.6
```
## Installation
### Install From Wheels
We provide prepackaged Python wheels (`manylinux2014_x86_64`,`manylinux_2_24_x86_64`). To install from wheels:
```python
pip install pygloo
```
### Building from source
One can build pygloo from source if none of released wheels fit with the development environment.
Pygloo uses [Bazel](https://github.com/bazelbuild/bazel) to automatically manange dependencies and compilation.
To compile from source, install Bazel>=2.0.0 following the [Bazel installation guide](https://docs.bazel.build/versions/master/install.html).
After installing Bazel, build and install pygloo following this command:
```python
python setup.py install
```
## Testing
Pygloo uses [Ray](https://github.com/ray-project/ray) to create multiple, distributed processes for collective communication tests. See `tests` directory.
## Example
An example for allreduce.
```python
import os
import ray
import pygloo
import numpy as np
@ray.remote(num_cpus=1)
def test_allreduce(rank, world_size, fileStore_path):
'''
rank # Rank of this process within list of participating processes
world_size # Number of participating processes
fileStore_path # The path to create filestore
'''
context = pygloo.rendezvous.Context(rank, world_size)
# Prepare device and store for rendezvous
attr = pygloo.transport.tcp.attr("localhost")
dev = pygloo.transport.tcp.CreateDevice(attr)
fileStore = pygloo.rendezvous.FileStore(fileStore_path)
store = pygloo.rendezvous.PrefixStore(str(world_size), fileStore)
context.connectFullMesh(store, dev)
sendbuf = np.array([[1,2,3],[1,2,3]], dtype=np.float32)
recvbuf = np.zeros_like(sendbuf, dtype=np.float32)
sendptr = sendbuf.ctypes.data
recvptr = recvbuf.ctypes.data
pygloo.allreduce(context, sendptr, recvptr,
sendbuf.size, pygloo.glooDataType_t.glooFloat32,
pygloo.ReduceOp.SUM, pygloo.allreduceAlgorithm.RING)
if __name__ == "__main__":
ray.init(num_cpus=6)
world_size = 2
fileStore_path = f"{ray.worker._global_node.get_session_dir_path()}" + "/collective/gloo/rendezvous"
os.makedirs(fileStore_path)
ray.get([test_allreduce.remote(rank, world_size, fileStore_path) for rank in range(world_size)])
```
## License
Gloo is licensed under the Apache License, Version 2.0.
Raw data
{
"_id": null,
"home_page": "https://github.com/ray-project/pygloo",
"name": "pygloo-kaihsun",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "collective communication",
"author": "Ray Team",
"author_email": "ray-dev@googlegroups.com",
"download_url": null,
"platform": null,
"description": "# pygloo\n\nPygloo provides Python bindings for [gloo](https://github.com/facebookincubator/gloo).\nIt is implemented using [pybind11](https://github.com/pybind/pybind11).\n\nIt is currenlty used in [Ray for collective communication](https://github.com/ray-project/ray/tree/master/python/ray/util/collective) between CPUs.\n\n\n## Requirements\n```python\nPython >= 3.6\n```\n\n## Installation\n### Install From Wheels\nWe provide prepackaged Python wheels (`manylinux2014_x86_64`,`manylinux_2_24_x86_64`). To install from wheels:\n```python\npip install pygloo\n```\n\n### Building from source\nOne can build pygloo from source if none of released wheels fit with the development environment.\n\nPygloo uses [Bazel](https://github.com/bazelbuild/bazel) to automatically manange dependencies and compilation.\nTo compile from source, install Bazel>=2.0.0 following the [Bazel installation guide](https://docs.bazel.build/versions/master/install.html).\nAfter installing Bazel, build and install pygloo following this command:\n```python\npython setup.py install\n```\n\n## Testing\nPygloo uses [Ray](https://github.com/ray-project/ray) to create multiple, distributed processes for collective communication tests. See `tests` directory.\n\n## Example\nAn example for allreduce.\n```python\nimport os\nimport ray\nimport pygloo\nimport numpy as np\n\n@ray.remote(num_cpus=1)\ndef test_allreduce(rank, world_size, fileStore_path):\n '''\n rank # Rank of this process within list of participating processes\n world_size # Number of participating processes\n fileStore_path # The path to create filestore\n '''\n context = pygloo.rendezvous.Context(rank, world_size)\n # Prepare device and store for rendezvous\n attr = pygloo.transport.tcp.attr(\"localhost\")\n dev = pygloo.transport.tcp.CreateDevice(attr)\n fileStore = pygloo.rendezvous.FileStore(fileStore_path)\n store = pygloo.rendezvous.PrefixStore(str(world_size), fileStore)\n\n context.connectFullMesh(store, dev)\n\n sendbuf = np.array([[1,2,3],[1,2,3]], dtype=np.float32)\n recvbuf = np.zeros_like(sendbuf, dtype=np.float32)\n sendptr = sendbuf.ctypes.data\n recvptr = recvbuf.ctypes.data\n\n pygloo.allreduce(context, sendptr, recvptr,\n sendbuf.size, pygloo.glooDataType_t.glooFloat32,\n pygloo.ReduceOp.SUM, pygloo.allreduceAlgorithm.RING)\n\nif __name__ == \"__main__\":\n ray.init(num_cpus=6)\n world_size = 2\n fileStore_path = f\"{ray.worker._global_node.get_session_dir_path()}\" + \"/collective/gloo/rendezvous\"\n os.makedirs(fileStore_path)\n ray.get([test_allreduce.remote(rank, world_size, fileStore_path) for rank in range(world_size)])\n```\n\n\n## License\nGloo is licensed under the Apache License, Version 2.0.\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "A python binding for gloo",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/ray-project/pygloo"
},
"split_keywords": [
"collective",
"communication"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8906849188515f35ebdbe3187eab1c0f7ba018014689e08a1cbc7d7b3f36e1d0",
"md5": "6622119676f1a9994926843ee054e146",
"sha256": "3ab1f08a8912b4fb7d2c47bfa5e5d71d44350f59c9c8798c2d510b29efd4080b"
},
"downloads": -1,
"filename": "pygloo_kaihsun-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "6622119676f1a9994926843ee054e146",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": null,
"size": 1047589,
"upload_time": "2024-12-20T05:22:55",
"upload_time_iso_8601": "2024-12-20T05:22:55.612636Z",
"url": "https://files.pythonhosted.org/packages/89/06/849188515f35ebdbe3187eab1c0f7ba018014689e08a1cbc7d7b3f36e1d0/pygloo_kaihsun-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-20 05:22:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ray-project",
"github_project": "pygloo",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pygloo-kaihsun"
}
JSON
