| Name | unidist JSON |
| Version |
0.5.1
JSON |
| download |
| home_page | https://github.com/modin-project/unidist |
| Summary | Unified Distributed Execution |
| upload_time | 2023-11-25 21:27:58 |
| maintainer | |
| docs_url | https://pythonhosted.org/unidist/ |
| author | |
| requires_python | >=3.7.1 |
| license | Apache-2.0 |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<p align="center">
<a href="https://unidist.readthedocs.io"><img alt="" src="https://github.com/modin-project/unidist/blob/d17f0da551846277c9d56a7f5e7d8f244c09d660/docs/img/unidist-logo-simple-628x128.png?raw=true"></a>
</p>
<h2 align="center">Unified Distributed Execution</h2>
<p align="center">
<a href="https://github.com/modin-project/unidist/actions"><img src="https://github.com/modin-project/unidist/actions/workflows/ci.yml/badge.svg" align="center"></a>
<a href="https://unidist.readthedocs.io/en/latest/?badge=latest"><img alt="" src="https://readthedocs.org/projects/unidist/badge/?version=latest" align="center"></a>
<a href="https://pypi.org/project/unidist/0.5.1/"><img src="https://img.shields.io/badge/pypi-0.5.1-blue.svg" alt="PyPI version" align="center"></a>
<a href="https://pepy.tech/project/unidist"><img src="https://static.pepy.tech/personalized-badge/unidist?period=total&units=international_system&left_color=black&right_color=blue&left_text=Downloads" align="center"></a>
</p>
### What is unidist?
unidist is a framework that is intended to provide the unified API for distributed execution by supporting various performant execution backends. At the moment the following backends are supported under the hood:
* [MPI](https://www.mpi-forum.org/)
* [Dask Distributed](https://distributed.dask.org/en/latest/)
* [Ray](https://docs.ray.io/en/master/index.html)
* [Python Multiprocessing](https://docs.python.org/3/library/multiprocessing.html)
unidist is designed to work in a [task-based parallel](https://en.wikipedia.org/wiki/Task_parallelism) model.
Also, the framework provides a Python Sequential backend (`pyseq`), that can be used for debugging.
### Installation
#### Using pip
unidist can be installed with `pip` on Linux, Windows and MacOS:
```bash
pip install unidist # Install unidist with dependencies for Python Multiprocessing and Python Sequential backends
```
unidist can also be used with MPI, Dask or Ray execution backend.
If you don't have MPI, Dask or Ray installed, you will need to install unidist with one of the targets:
```bash
pip install unidist[all] # Install unidist with dependencies for all the backends
pip install unidist[mpi] # Install unidist with dependencies for MPI backend
pip install unidist[dask] # Install unidist with dependencies for Dask backend
pip install unidist[ray] # Install unidist with dependencies for Ray backend
```
unidist automatically detects which execution backends are installed and uses that for scheduling computation.
**Note:** There are different MPI implementations, each of which can be used as a backend in unidist.
Mapping `unidist[mpi]` installs `mpi4py` package, which is just a Python wrapper for MPI.
To enable unidist on MPI execution you need to have a working MPI implementation and certain software installed beforehand.
Refer to [Installation](https://mpi4py.readthedocs.io/en/latest/install.html) page of the `mpi4py` documentation for details.
Also, you can find some instructions on [MPI backend](https://unidist.readthedocs.io/en/latest/optimization_notes/mpi.html) page.
#### Using conda
For installing unidist with dependencies for MPI and Dask execution backends into a conda environment
the following command should be used:
```bash
conda install unidist-mpi unidist-dask -c conda-forge
```
All set of backends could be available in a conda environment by specifying:
```bash
conda install unidist-all -c conda-forge
```
or explicitly:
```bash
conda install unidist-mpi unidist-dask unidist-ray -c conda-forge
```
**Note:** There are different MPI implementations, each of which can be used as a backend in unidist.
By default, mapping `unidist-mpi` installs a default MPI implementation, which comes with `mpi4py` package and is ready to use.
The conda dependency solver decides on which MPI implementation is to be installed. If you want to use a specific version of MPI,
you can install the core dependencies for MPI backend and the specific version of MPI as `conda install unidist-mpi <mpi>`
as shown in the [Installation](https://mpi4py.readthedocs.io/en/latest/install.html)
page of `mpi4py` documentation. That said, it is highly encouraged to use your own MPI binaries as stated in the
[Using External MPI Libraries](https://conda-forge.org/docs/user/tipsandtricks.html#using-external-message-passing-interface-mpi-libraries)
section of the conda-forge documentation in order to get ultimate performance.
For more information refer to [Installation](https://unidist.readthedocs.io/en/latest/installation.html) section.
#### Choosing an execution backend
If you want to choose a specific execution backend to run on,
you can set the environment variable `UNIDIST_BACKEND` and unidist will do computation with that backend:
```bash
export UNIDIST_BACKEND=mpi # unidist will use MPI
export UNIDIST_BACKEND=dask # unidist will use Dask
export UNIDIST_BACKEND=ray # unidist will use Ray
```
This can also be done within a notebook/interpreter before you initialize unidist:
```python
from unidist.config import Backend
Backend.put("mpi") # unidist will use MPI
Backend.put("dask") # unidist will use Dask
Backend.put("ray") # unidist will use Ray
```
If you have installed all the execution backends and haven't specified any of the execution backends, MPI is used by default.
Currently, almost all MPI implementations require ``mpiexec`` command to be used when running an MPI program.
If you use a backend other than MPI, you run a program as a regular python script (see below).
#### Usage
```python
# script.py
import unidist
unidist.init() # MPI backend is used by default
@unidist.remote
def foo(x):
return x * x
# This will run `foo` on a pool of workers in parallel;
# `refs` will contain object references to actual data
refs = [foo.remote(i) for i in range(5)]
# To get the data call `unidist.get(...)`
print(unidist.get(refs))
```
Run the `script.py` with:
```bash
$ mpiexec -n 1 python script.py # for MPI backend
# $ python script.py # for any other supported backend
[0, 1, 4, 9, 16] # output
```
For more examples refer to [Getting Started](https://unidist.readthedocs.io/en/latest/getting_started.html) section
in our documentation.
### Powered by unidist
unidist is meant to be used not only directly by users to get better performance in their workloads,
but also be a core component of other libraries to power those with the performant execution backends.
Refer to `Libraries powered by unidist` section of [Using Unidist](https://unidist.readthedocs.io/en/latest/using_unidist/index.html) page
to get more information on which libraries have already been using unidist.
### Full Documentation
Visit the complete documentation on readthedocs: https://unidist.readthedocs.io.
Raw data
{
"_id": null,
"home_page": "https://github.com/modin-project/unidist",
"name": "unidist",
"maintainer": "",
"docs_url": "https://pythonhosted.org/unidist/",
"requires_python": ">=3.7.1",
"maintainer_email": "",
"keywords": "",
"author": "",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/a9/e9/507cc0b78fda05e6270ba5d8f8f20d212ce40e5b528d5addd4456ae33bd2/unidist-0.5.1.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <a href=\"https://unidist.readthedocs.io\"><img alt=\"\" src=\"https://github.com/modin-project/unidist/blob/d17f0da551846277c9d56a7f5e7d8f244c09d660/docs/img/unidist-logo-simple-628x128.png?raw=true\"></a>\n</p>\n<h2 align=\"center\">Unified Distributed Execution</h2>\n\n<p align=\"center\">\n<a href=\"https://github.com/modin-project/unidist/actions\"><img src=\"https://github.com/modin-project/unidist/actions/workflows/ci.yml/badge.svg\" align=\"center\"></a>\n<a href=\"https://unidist.readthedocs.io/en/latest/?badge=latest\"><img alt=\"\" src=\"https://readthedocs.org/projects/unidist/badge/?version=latest\" align=\"center\"></a>\n<a href=\"https://pypi.org/project/unidist/0.5.1/\"><img src=\"https://img.shields.io/badge/pypi-0.5.1-blue.svg\" alt=\"PyPI version\" align=\"center\"></a>\n<a href=\"https://pepy.tech/project/unidist\"><img src=\"https://static.pepy.tech/personalized-badge/unidist?period=total&units=international_system&left_color=black&right_color=blue&left_text=Downloads\" align=\"center\"></a>\n</p>\n\n### What is unidist?\n\nunidist is a framework that is intended to provide the unified API for distributed execution by supporting various performant execution backends. At the moment the following backends are supported under the hood:\n\n* [MPI](https://www.mpi-forum.org/)\n* [Dask Distributed](https://distributed.dask.org/en/latest/)\n* [Ray](https://docs.ray.io/en/master/index.html)\n* [Python Multiprocessing](https://docs.python.org/3/library/multiprocessing.html)\n\nunidist is designed to work in a [task-based parallel](https://en.wikipedia.org/wiki/Task_parallelism) model.\n\nAlso, the framework provides a Python Sequential backend (`pyseq`), that can be used for debugging.\n\n### Installation\n\n#### Using pip\n\nunidist can be installed with `pip` on Linux, Windows and MacOS:\n\n```bash\npip install unidist # Install unidist with dependencies for Python Multiprocessing and Python Sequential backends\n```\n\nunidist can also be used with MPI, Dask or Ray execution backend.\nIf you don't have MPI, Dask or Ray installed, you will need to install unidist with one of the targets:\n\n```bash\npip install unidist[all] # Install unidist with dependencies for all the backends\npip install unidist[mpi] # Install unidist with dependencies for MPI backend\npip install unidist[dask] # Install unidist with dependencies for Dask backend\npip install unidist[ray] # Install unidist with dependencies for Ray backend\n```\n\nunidist automatically detects which execution backends are installed and uses that for scheduling computation.\n\n**Note:** There are different MPI implementations, each of which can be used as a backend in unidist.\nMapping `unidist[mpi]` installs `mpi4py` package, which is just a Python wrapper for MPI.\nTo enable unidist on MPI execution you need to have a working MPI implementation and certain software installed beforehand.\nRefer to [Installation](https://mpi4py.readthedocs.io/en/latest/install.html) page of the `mpi4py` documentation for details.\nAlso, you can find some instructions on [MPI backend](https://unidist.readthedocs.io/en/latest/optimization_notes/mpi.html) page.\n\n#### Using conda\n\nFor installing unidist with dependencies for MPI and Dask execution backends into a conda environment\nthe following command should be used:\n\n```bash\nconda install unidist-mpi unidist-dask -c conda-forge\n```\n\nAll set of backends could be available in a conda environment by specifying:\n\n```bash\nconda install unidist-all -c conda-forge\n```\n\nor explicitly:\n\n```bash\nconda install unidist-mpi unidist-dask unidist-ray -c conda-forge\n```\n\n**Note:** There are different MPI implementations, each of which can be used as a backend in unidist.\nBy default, mapping `unidist-mpi` installs a default MPI implementation, which comes with `mpi4py` package and is ready to use.\nThe conda dependency solver decides on which MPI implementation is to be installed. If you want to use a specific version of MPI,\nyou can install the core dependencies for MPI backend and the specific version of MPI as `conda install unidist-mpi <mpi>`\nas shown in the [Installation](https://mpi4py.readthedocs.io/en/latest/install.html)\npage of `mpi4py` documentation. That said, it is highly encouraged to use your own MPI binaries as stated in the\n[Using External MPI Libraries](https://conda-forge.org/docs/user/tipsandtricks.html#using-external-message-passing-interface-mpi-libraries)\nsection of the conda-forge documentation in order to get ultimate performance.\n\nFor more information refer to [Installation](https://unidist.readthedocs.io/en/latest/installation.html) section.\n\n#### Choosing an execution backend\n\nIf you want to choose a specific execution backend to run on,\nyou can set the environment variable `UNIDIST_BACKEND` and unidist will do computation with that backend:\n\n```bash\nexport UNIDIST_BACKEND=mpi # unidist will use MPI\nexport UNIDIST_BACKEND=dask # unidist will use Dask\nexport UNIDIST_BACKEND=ray # unidist will use Ray\n```\n\nThis can also be done within a notebook/interpreter before you initialize unidist:\n\n```python\nfrom unidist.config import Backend\n\nBackend.put(\"mpi\") # unidist will use MPI\nBackend.put(\"dask\") # unidist will use Dask\nBackend.put(\"ray\") # unidist will use Ray\n```\n\nIf you have installed all the execution backends and haven't specified any of the execution backends, MPI is used by default.\nCurrently, almost all MPI implementations require ``mpiexec`` command to be used when running an MPI program.\nIf you use a backend other than MPI, you run a program as a regular python script (see below).\n\n#### Usage\n\n```python\n# script.py\n\nimport unidist\nunidist.init() # MPI backend is used by default\n\n@unidist.remote\ndef foo(x):\n return x * x\n\n# This will run `foo` on a pool of workers in parallel;\n# `refs` will contain object references to actual data\nrefs = [foo.remote(i) for i in range(5)]\n# To get the data call `unidist.get(...)`\nprint(unidist.get(refs))\n```\n\nRun the `script.py` with:\n\n```bash\n$ mpiexec -n 1 python script.py # for MPI backend\n# $ python script.py # for any other supported backend\n[0, 1, 4, 9, 16] # output\n```\n\nFor more examples refer to [Getting Started](https://unidist.readthedocs.io/en/latest/getting_started.html) section\nin our documentation.\n\n### Powered by unidist\n\nunidist is meant to be used not only directly by users to get better performance in their workloads,\nbut also be a core component of other libraries to power those with the performant execution backends.\nRefer to `Libraries powered by unidist` section of [Using Unidist](https://unidist.readthedocs.io/en/latest/using_unidist/index.html) page\nto get more information on which libraries have already been using unidist.\n\n### Full Documentation\n\nVisit the complete documentation on readthedocs: https://unidist.readthedocs.io.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Unified Distributed Execution",
"version": "0.5.1",
"project_urls": {
"Homepage": "https://github.com/modin-project/unidist"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f8c56423c5e140ef6b6476d663e43c941da328d5d9c592c336d8328b59f9d262",
"md5": "381b5e770b2d9e47f3885aa3baf1fe84",
"sha256": "b10d0d7bfdbda5679616adef01965216ebf46fbfb4ca1057381c740f3c6539c4"
},
"downloads": -1,
"filename": "unidist-0.5.1-py3-none-macosx_10_9_x86_64.whl",
"has_sig": false,
"md5_digest": "381b5e770b2d9e47f3885aa3baf1fe84",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7.1",
"size": 206262,
"upload_time": "2023-11-25T21:27:49",
"upload_time_iso_8601": "2023-11-25T21:27:49.398056Z",
"url": "https://files.pythonhosted.org/packages/f8/c5/6423c5e140ef6b6476d663e43c941da328d5d9c592c336d8328b59f9d262/unidist-0.5.1-py3-none-macosx_10_9_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "115888ad316954245cd203f7e6d35cab991b6b7288585e379114e0ab5c111362",
"md5": "2d8fc2e5bc8e77dfbcebc3d5f534e504",
"sha256": "d25dbc706911996419403eb2897f3a4cf313fbd5af686467ef0b9cc208201a58"
},
"downloads": -1,
"filename": "unidist-0.5.1-py3-none-manylinux1_i686.whl",
"has_sig": false,
"md5_digest": "2d8fc2e5bc8e77dfbcebc3d5f534e504",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7.1",
"size": 206259,
"upload_time": "2023-11-25T21:27:51",
"upload_time_iso_8601": "2023-11-25T21:27:51.050747Z",
"url": "https://files.pythonhosted.org/packages/11/58/88ad316954245cd203f7e6d35cab991b6b7288585e379114e0ab5c111362/unidist-0.5.1-py3-none-manylinux1_i686.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fcfd850dac733b0037ff8039f3eb1624c6b10471bea511d6b07d751450e577d4",
"md5": "fb41d4202e9a908622098074b222340f",
"sha256": "fe65e44d09a59b3d62a11f7c08708a06e1c47e16c84b2c4e4a2396663e240230"
},
"downloads": -1,
"filename": "unidist-0.5.1-py3-none-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "fb41d4202e9a908622098074b222340f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7.1",
"size": 206263,
"upload_time": "2023-11-25T21:27:52",
"upload_time_iso_8601": "2023-11-25T21:27:52.997166Z",
"url": "https://files.pythonhosted.org/packages/fc/fd/850dac733b0037ff8039f3eb1624c6b10471bea511d6b07d751450e577d4/unidist-0.5.1-py3-none-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6ad861afccaf3518763be7afba4b9fb90c200cec24d0c088fe98a54e3a67cace",
"md5": "b8f83eb257f47ee456e32664e8dd9643",
"sha256": "739e18c3b4a3c31d0ab6a5fb8a1ce2ddc51cc330524dc51b7746ba81457ce2b3"
},
"downloads": -1,
"filename": "unidist-0.5.1-py3-none-win32.whl",
"has_sig": false,
"md5_digest": "b8f83eb257f47ee456e32664e8dd9643",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7.1",
"size": 206251,
"upload_time": "2023-11-25T21:27:55",
"upload_time_iso_8601": "2023-11-25T21:27:55.026379Z",
"url": "https://files.pythonhosted.org/packages/6a/d8/61afccaf3518763be7afba4b9fb90c200cec24d0c088fe98a54e3a67cace/unidist-0.5.1-py3-none-win32.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "16a5b8635fa2ae3f71c49c96044e2a44a848a38d3e936e3f654429d7653c286b",
"md5": "52f3508a5f2c26a155223cc21baa25cd",
"sha256": "41fc5a3c9d70615d71fa173e3b749eebd0bb9061a9947acc17a2cdb5101b9da2"
},
"downloads": -1,
"filename": "unidist-0.5.1-py3-none-win_amd64.whl",
"has_sig": false,
"md5_digest": "52f3508a5f2c26a155223cc21baa25cd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7.1",
"size": 206253,
"upload_time": "2023-11-25T21:27:56",
"upload_time_iso_8601": "2023-11-25T21:27:56.491799Z",
"url": "https://files.pythonhosted.org/packages/16/a5/b8635fa2ae3f71c49c96044e2a44a848a38d3e936e3f654429d7653c286b/unidist-0.5.1-py3-none-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a9e9507cc0b78fda05e6270ba5d8f8f20d212ce40e5b528d5addd4456ae33bd2",
"md5": "8eb2e0efd8a86db9f33c96817cc91a2a",
"sha256": "f60a71a38a3e96280e8bd11f540ff6d0d9211e7551a963cb8b2da4f0af4e7fa0"
},
"downloads": -1,
"filename": "unidist-0.5.1.tar.gz",
"has_sig": false,
"md5_digest": "8eb2e0efd8a86db9f33c96817cc91a2a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7.1",
"size": 235202,
"upload_time": "2023-11-25T21:27:58",
"upload_time_iso_8601": "2023-11-25T21:27:58.234676Z",
"url": "https://files.pythonhosted.org/packages/a9/e9/507cc0b78fda05e6270ba5d8f8f20d212ce40e5b528d5addd4456ae33bd2/unidist-0.5.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-25 21:27:58",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "modin-project",
"github_project": "unidist",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "unidist"
}
