| Name | modin JSON |
| Version |
0.29.0
JSON |
| download |
| home_page | https://github.com/modin-project/modin |
| Summary | Modin: Make your pandas code run faster by changing one line of code. |
| upload_time | 2024-04-15 17:53:05 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | Apache 2 |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<p align="center"><a href="https://modin.readthedocs.io"><img width=77% alt="" src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/MODIN_ver2_hrz.png?raw=true"></a></p>
<h2 align="center">Scale your pandas workflows by changing one line of code</h2>
<div align="center">
| <h3>Dev Community & Support</h3> | <h3>Forums</h3> | <h3>Socials</h3> | <h3>Docs</h3> |
|:---: | :---: | :---: | :---: |
| [](https://join.slack.com/t/modin-project/shared_invite/zt-yvk5hr3b-f08p_ulbuRWsAfg9rMY3uA) | [](https://stackoverflow.com/questions/tagged/modin) | <a href="https://twitter.com/modin_project"><img alt="Twitter Follow" src="https://img.shields.io/twitter/follow/modin_project?style=social" height=28 align="center"></a> | <a href="https://modin.readthedocs.io/en/latest/?badge=latest"><img alt="" src="https://readthedocs.org/projects/modin/badge/?version=latest" height=28 align="center"></a> |
</div>
<p align="center">
<a href="https://pepy.tech/project/modin"><img src="https://static.pepy.tech/personalized-badge/modin?period=total&units=international_system&left_color=black&right_color=blue&left_text=Downloads" align="center"></a>
<a href="https://codecov.io/gh/modin-project/modin"><img src="https://codecov.io/gh/modin-project/modin/branch/master/graph/badge.svg" align="center"/></a>
<a href="https://github.com/modin-project/modin/actions"><img src="https://github.com/modin-project/modin/workflows/master/badge.svg" align="center"></a>
<a href="https://github.com/modin-project/modin/actions/workflows/ci.yml?query=event%3Apush"><img src="https://github.com/modin-project/modin/actions/workflows/ci.yml/badge.svg?branch=master" align="center"></a>
<a href="https://pypi.org/project/modin/"><img src="https://badge.fury.io/py/modin.svg" alt="PyPI version" align="center"></a>
<a href="https://modin.org/modin-bench/#/"><img src="https://img.shields.io/badge/benchmarked%20by-asv-blue.svg" align="center"></a>
</p>
### What is Modin?
Modin is a drop-in replacement for [pandas](https://github.com/pandas-dev/pandas). While pandas is
single-threaded, Modin lets you instantly speed up your workflows by scaling pandas so it uses all of your
cores. Modin works especially well on larger datasets, where pandas becomes painfully slow or runs
[out of memory](https://modin.readthedocs.io/en/latest/getting_started/why_modin/out_of_core.html).
Also, Modin comes with the [additional APIs](https://modin.readthedocs.io/en/latest/usage_guide/advanced_usage/index.html#additional-apis)
to improve user experience.
By simply replacing the import statement, Modin offers users effortless speed and scale for their pandas workflows:
<img src="https://github.com/modin-project/modin/raw/master/docs/img/Import.gif" style="display: block;margin-left: auto;margin-right: auto;" width="100%"></img>
In the GIFs below, Modin (left) and pandas (right) perform *the same pandas operations* on a 2GB dataset. The only difference between the two notebook examples is the import statement.
<table class="tg">
<thead>
<tr>
<th class="tg-0lax" style="text-align: center;"><img src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/MODIN_ver2_hrz.png?raw=True" height="35px"></th>
<th class="tg-0lax" style="text-align: center;"><img src="https://pandas.pydata.org/static/img/pandas.svg" height="50px"></img></th>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-0lax"><img src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Modin.gif"></img></td>
<td class="tg-0lax"><img src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Pandas.gif"></img></td>
</tr>
</tbody>
</table>
The charts below show the speedup you get by replacing pandas with Modin based on the examples above. The example notebooks can be found [here](examples/jupyter). To learn more about the speedups you could get with Modin and try out some examples on your own, check out our [10-minute quickstart guide](https://modin.readthedocs.io/en/latest/getting_started/quickstart.html) to try out some examples on your own!
<img src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Modin_Speedup.svg" style="display: block;margin-left: auto;margin-right: auto;" width="100%"></img>
### Installation
#### From PyPI
Modin can be installed with `pip` on Linux, Windows and MacOS:
```bash
pip install "modin[all]" # (Recommended) Install Modin with Ray and Dask engines.
```
If you want to install Modin with a specific engine, we recommend:
```bash
pip install "modin[ray]" # Install Modin dependencies and Ray.
pip install "modin[dask]" # Install Modin dependencies and Dask.
pip install "modin[mpi]" # Install Modin dependencies and MPI through unidist.
```
To get Modin on MPI through unidist (as of unidist 0.5.0) fully working
it is required to have a working MPI implementation installed beforehand.
Otherwise, installation of `modin[mpi]` may fail. Refer to
[Installing with pip](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-pip)
section of the unidist documentation for more details about installation.
Modin automatically detects which engine(s) you have installed and uses that for scheduling computation.
#### From conda-forge
Installing from [conda forge](https://github.com/conda-forge/modin-feedstock) using `modin-all`
will install Modin and four engines: [Ray](https://github.com/ray-project/ray), [Dask](https://github.com/dask/dask),
[MPI through unidist](https://github.com/modin-project/unidist) and [HDK](https://github.com/intel-ai/hdk).
```bash
conda install -c conda-forge modin-all
```
Each engine can also be installed individually (and also as a combination of several engines):
```bash
conda install -c conda-forge modin-ray # Install Modin dependencies and Ray.
conda install -c conda-forge modin-dask # Install Modin dependencies and Dask.
conda install -c conda-forge modin-mpi # Install Modin dependencies and MPI through unidist.
conda install -c conda-forge modin-hdk # Install Modin dependencies and HDK.
```
Refer to
[Installing with conda](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-conda)
section of the unidist documentation for more details on how to install a specific MPI implementation to run on.
To speed up conda installation we recommend using libmamba solver. To do this install it in a base environment:
```bash
conda install -n base conda-libmamba-solver
```
and then use it during istallation either like:
```bash
conda install -c conda-forge modin-ray modin-hdk --experimental-solver=libmamba
```
or starting from conda 22.11 and libmamba solver 22.12 versions:
```bash
conda install -c conda-forge modin-ray modin-hdk --solver=libmamba
```
#### Choosing a Compute Engine
If you want to choose a specific compute engine to run on, you can set the environment
variable `MODIN_ENGINE` and Modin will do computation with that engine:
```bash
export MODIN_ENGINE=ray # Modin will use Ray
export MODIN_ENGINE=dask # Modin will use Dask
export MODIN_ENGINE=unidist # Modin will use Unidist
```
If you want to choose the Unidist engine, you should set the additional environment
variable ``UNIDIST_BACKEND``. Currently, Modin only supports MPI through unidist:
```bash
export UNIDIST_BACKEND=mpi # Unidist will use MPI backend
```
This can also be done within a notebook/interpreter before you import Modin:
```python
import modin.config as modin_cfg
import unidist.config as unidist_cfg
modin_cfg.Engine.put("ray") # Modin will use Ray
modin_cfg.Engine.put("dask") # Modin will use Dask
modin_cfg.Engine.put('unidist') # Modin will use Unidist
unidist_cfg.Backend.put('mpi') # Unidist will use MPI backend
```
Check [this Modin docs section](https://modin.readthedocs.io/en/latest/development/using_hdk.html) for HDK engine setup.
_Note: You should not change the engine after your first operation with Modin as it will result in undefined behavior._
#### Which engine should I use?
On Linux, MacOS, and Windows you can install and use either Ray, Dask or MPI through unidist. There is no knowledge required
to use either of these engines as Modin abstracts away all of the complexity, so feel
free to pick either!
On Linux you also can choose [HDK](https://modin.readthedocs.io/en/latest/development/using_hdk.html), which is an experimental
engine based on [HDK](https://github.com/intel-ai/hdk) and included in the
[Intel® Distribution of Modin](https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/distribution-of-modin.html),
which is a part of [Intel® oneAPI AI Analytics Toolkit (AI Kit)](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html).
### Pandas API Coverage
<p align="center">
| pandas Object | Modin's Ray Engine Coverage | Modin's Dask Engine Coverage | Modin's Unidist Engine Coverage |
|-------------------|:------------------------------------------------------------------------------------:|:---------------:|:---------------:|
| `pd.DataFrame` | <img src=https://img.shields.io/badge/api%20coverage-90.8%25-hunter.svg> | <img src=https://img.shields.io/badge/api%20coverage-90.8%25-hunter.svg> | <img src=https://img.shields.io/badge/api%20coverage-90.8%25-hunter.svg> |
| `pd.Series` | <img src=https://img.shields.io/badge/api%20coverage-88.05%25-green.svg> | <img src=https://img.shields.io/badge/api%20coverage-88.05%25-green.svg> | <img src=https://img.shields.io/badge/api%20coverage-88.05%25-green.svg>
| `pd.read_csv` | ✅ | ✅ | ✅ |
| `pd.read_table` | ✅ | ✅ | ✅ |
| `pd.read_parquet` | ✅ | ✅ | ✅ |
| `pd.read_sql` | ✅ | ✅ | ✅ |
| `pd.read_feather` | ✅ | ✅ | ✅ |
| `pd.read_excel` | ✅ | ✅ | ✅ |
| `pd.read_json` | [✳️](https://github.com/modin-project/modin/issues/554) | [✳️](https://github.com/modin-project/modin/issues/554) | [✳️](https://github.com/modin-project/modin/issues/554) |
| `pd.read_<other>` | [✴️](https://modin.readthedocs.io/en/latest/supported_apis/io_supported.html) | [✴️](https://modin.readthedocs.io/en/latest/supported_apis/io_supported.html) | [✴️](https://modin.readthedocs.io/en/latest/supported_apis/io_supported.html) |
</p>
Some pandas APIs are easier to implement than others, so if something is missing feel
free to open an issue!
### More about Modin
For the complete documentation on Modin, visit our [ReadTheDocs](https://modin.readthedocs.io/en/latest/index.html) page.
#### Scale your pandas workflow by changing a single line of code.
_Note: In local mode (without a cluster), Modin will create and manage a local (Dask or Ray) cluster for the execution._
To use Modin, you do not need to specify how to distribute the data, or even know how many
cores your system has. In fact, you can continue using your previous
pandas notebooks while experiencing a considerable speedup from Modin, even on a single
machine. Once you've changed your import statement, you're ready to use Modin just like
you would with pandas!
#### Faster pandas, even on your laptop
<img align="right" style="display:inline;" height="350" width="300" src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/read_csv_benchmark.png?raw=true"></a>
The `modin.pandas` DataFrame is an extremely light-weight parallel DataFrame.
Modin transparently distributes the data and computation so that you can continue using the same pandas API
while working with more data faster. Because it is so light-weight,
Modin provides speed-ups of up to 4x on a laptop with 4 physical cores.
In pandas, you are only able to use one core at a time when you are doing computation of
any kind. With Modin, you are able to use all of the CPU cores on your machine. Even with a
traditionally synchronous task like `read_csv`, we see large speedups by efficiently
distributing the work across your entire machine.
```python
import modin.pandas as pd
df = pd.read_csv("my_dataset.csv")
```
#### Modin can handle the datasets that pandas can't
Often data scientists have to switch between different tools
for operating on datasets of different sizes. Processing large dataframes with pandas
is slow, and pandas does not support working with dataframes that are too large to fit
into the available memory. As a result, pandas workflows that work well
for prototyping on a few MBs of data do not scale to tens or hundreds of GBs (depending on the size
of your machine). Modin supports operating on data that does not fit in memory, so that you can comfortably
work with hundreds of GBs without worrying about substantial slowdown or memory errors.
With [cluster](https://modin.readthedocs.io/en/latest/getting_started/using_modin/using_modin_cluster.html)
and [out of core](https://modin.readthedocs.io/en/latest/getting_started/why_modin/out_of_core.html)
support, Modin is a DataFrame library with both great single-node performance and high
scalability in a cluster.
#### Modin Architecture
We designed [Modin's architecture](https://modin.readthedocs.io/en/latest/development/architecture.html)
to be modular so we can plug in different components as they develop and improve:
<img src="https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/modin_architecture.png" alt="Modin's architecture" width="75%"></img>
### Other Resources
#### Getting Started with Modin
- [Documentation](https://modin.readthedocs.io/en/latest/)
- [10-min Quickstart Guide](https://modin.readthedocs.io/en/latest/getting_started/quickstart.html)
- [Examples and Tutorials](https://modin.readthedocs.io/en/latest/getting_started/examples.html)
- [Videos and Blogposts](https://modin.readthedocs.io/en/latest/getting_started/examples.html#talks-podcasts)
- [Benchmarking Modin](https://modin.readthedocs.io/en/latest/usage_guide/benchmarking.html)
#### Modin Community
- [Slack](https://join.slack.com/t/modin-project/shared_invite/zt-yvk5hr3b-f08p_ulbuRWsAfg9rMY3uA)
- [Twitter](https://twitter.com/modin_project)
- [Mailing List](https://groups.google.com/g/modin-dev)
- [GitHub Issues](https://github.com/modin-project/modin/issues)
- [StackOverflow](https://stackoverflow.com/questions/tagged/modin)
#### Learn More about Modin
- [Frequently Asked Questions (FAQs)](https://modin.readthedocs.io/en/latest/getting_started/faq.html)
- [Troubleshooting Guide](https://modin.readthedocs.io/en/latest/getting_started/troubleshooting.html)
- [Development Guide](https://modin.readthedocs.io/en/latest/development/index.html)
- Modin is built on many years of research and development at UC Berkeley. Check out these selected papers to learn more about how Modin works:
- [Flexible Rule-Based Decomposition and Metadata Independence in Modin](https://people.eecs.berkeley.edu/~totemtang/paper/Modin.pdf) (VLDB 2021)
- [Dataframe Systems: Theory, Architecture, and Implementation](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-193.pdf) (PhD Dissertation 2021)
- [Towards Scalable Dataframe Systems](https://arxiv.org/pdf/2001.00888.pdf) (VLDB 2020)
#### Getting Involved
***`modin.pandas` is currently under active development. Requests and contributions are welcome!***
For more information on how to contribute to Modin, check out the
[Modin Contribution Guide](https://modin.readthedocs.io/en/latest/development/contributing.html).
### License
[Apache License 2.0](LICENSE)
Raw data
{
"_id": null,
"home_page": "https://github.com/modin-project/modin",
"name": "modin",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/55/87/6c4ac92b414983c3d02615452231117eb019ae5dcb170c90552e4af6b9e7/modin-0.29.0.tar.gz",
"platform": null,
"description": "<p align=\"center\"><a href=\"https://modin.readthedocs.io\"><img width=77% alt=\"\" src=\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/MODIN_ver2_hrz.png?raw=true\"></a></p>\r\n<h2 align=\"center\">Scale your pandas workflows by changing one line of code</h2>\r\n\r\n<div align=\"center\">\r\n\r\n| <h3>Dev Community & Support</h3> | <h3>Forums</h3> | <h3>Socials</h3> | <h3>Docs</h3> |\r\n|:---: | :---: | :---: | :---: |\r\n| [](https://join.slack.com/t/modin-project/shared_invite/zt-yvk5hr3b-f08p_ulbuRWsAfg9rMY3uA) | [](https://stackoverflow.com/questions/tagged/modin) | <a href=\"https://twitter.com/modin_project\"><img alt=\"Twitter Follow\" src=\"https://img.shields.io/twitter/follow/modin_project?style=social\" height=28 align=\"center\"></a> | <a href=\"https://modin.readthedocs.io/en/latest/?badge=latest\"><img alt=\"\" src=\"https://readthedocs.org/projects/modin/badge/?version=latest\" height=28 align=\"center\"></a> |\r\n\r\n</div>\r\n\r\n<p align=\"center\">\r\n<a href=\"https://pepy.tech/project/modin\"><img src=\"https://static.pepy.tech/personalized-badge/modin?period=total&units=international_system&left_color=black&right_color=blue&left_text=Downloads\" align=\"center\"></a>\r\n<a href=\"https://codecov.io/gh/modin-project/modin\"><img src=\"https://codecov.io/gh/modin-project/modin/branch/master/graph/badge.svg\" align=\"center\"/></a>\r\n<a href=\"https://github.com/modin-project/modin/actions\"><img src=\"https://github.com/modin-project/modin/workflows/master/badge.svg\" align=\"center\"></a>\r\n<a href=\"https://github.com/modin-project/modin/actions/workflows/ci.yml?query=event%3Apush\"><img src=\"https://github.com/modin-project/modin/actions/workflows/ci.yml/badge.svg?branch=master\" align=\"center\"></a>\r\n<a href=\"https://pypi.org/project/modin/\"><img src=\"https://badge.fury.io/py/modin.svg\" alt=\"PyPI version\" align=\"center\"></a>\r\n<a href=\"https://modin.org/modin-bench/#/\"><img src=\"https://img.shields.io/badge/benchmarked%20by-asv-blue.svg\" align=\"center\"></a>\r\n</p>\r\n\r\n### What is Modin?\r\n\r\nModin is a drop-in replacement for [pandas](https://github.com/pandas-dev/pandas). While pandas is\r\nsingle-threaded, Modin lets you instantly speed up your workflows by scaling pandas so it uses all of your\r\ncores. Modin works especially well on larger datasets, where pandas becomes painfully slow or runs\r\n[out of memory](https://modin.readthedocs.io/en/latest/getting_started/why_modin/out_of_core.html).\r\nAlso, Modin comes with the [additional APIs](https://modin.readthedocs.io/en/latest/usage_guide/advanced_usage/index.html#additional-apis)\r\nto improve user experience.\r\n\r\nBy simply replacing the import statement, Modin offers users effortless speed and scale for their pandas workflows:\r\n\r\n<img src=\"https://github.com/modin-project/modin/raw/master/docs/img/Import.gif\" style=\"display: block;margin-left: auto;margin-right: auto;\" width=\"100%\"></img>\r\n\r\nIn the GIFs below, Modin (left) and pandas (right) perform *the same pandas operations* on a 2GB dataset. The only difference between the two notebook examples is the import statement. \r\n\r\n<table class=\"tg\">\r\n<thead>\r\n <tr>\r\n <th class=\"tg-0lax\" style=\"text-align: center;\"><img src=\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/MODIN_ver2_hrz.png?raw=True\" height=\"35px\"></th>\r\n <th class=\"tg-0lax\" style=\"text-align: center;\"><img src=\"https://pandas.pydata.org/static/img/pandas.svg\" height=\"50px\"></img></th>\r\n </tr>\r\n</thead>\r\n<tbody>\r\n <tr>\r\n <td class=\"tg-0lax\"><img src=\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Modin.gif\"></img></td>\r\n <td class=\"tg-0lax\"><img src=\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Pandas.gif\"></img></td>\r\n </tr>\r\n</tbody>\r\n</table>\r\n\r\nThe charts below show the speedup you get by replacing pandas with Modin based on the examples above. The example notebooks can be found [here](examples/jupyter). To learn more about the speedups you could get with Modin and try out some examples on your own, check out our [10-minute quickstart guide](https://modin.readthedocs.io/en/latest/getting_started/quickstart.html) to try out some examples on your own!\r\n\r\n<img src=\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/Modin_Speedup.svg\" style=\"display: block;margin-left: auto;margin-right: auto;\" width=\"100%\"></img>\r\n\r\n### Installation\r\n\r\n#### From PyPI\r\n\r\nModin can be installed with `pip` on Linux, Windows and MacOS:\r\n\r\n```bash\r\npip install \"modin[all]\" # (Recommended) Install Modin with Ray and Dask engines.\r\n```\r\n\r\nIf you want to install Modin with a specific engine, we recommend:\r\n\r\n```bash\r\npip install \"modin[ray]\" # Install Modin dependencies and Ray.\r\npip install \"modin[dask]\" # Install Modin dependencies and Dask.\r\npip install \"modin[mpi]\" # Install Modin dependencies and MPI through unidist.\r\n```\r\n\r\nTo get Modin on MPI through unidist (as of unidist 0.5.0) fully working\r\nit is required to have a working MPI implementation installed beforehand.\r\nOtherwise, installation of `modin[mpi]` may fail. Refer to\r\n[Installing with pip](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-pip)\r\nsection of the unidist documentation for more details about installation.\r\n\r\nModin automatically detects which engine(s) you have installed and uses that for scheduling computation.\r\n\r\n#### From conda-forge\r\n\r\nInstalling from [conda forge](https://github.com/conda-forge/modin-feedstock) using `modin-all`\r\nwill install Modin and four engines: [Ray](https://github.com/ray-project/ray), [Dask](https://github.com/dask/dask),\r\n[MPI through unidist](https://github.com/modin-project/unidist) and [HDK](https://github.com/intel-ai/hdk).\r\n\r\n```bash\r\nconda install -c conda-forge modin-all\r\n```\r\n\r\nEach engine can also be installed individually (and also as a combination of several engines):\r\n\r\n```bash\r\nconda install -c conda-forge modin-ray # Install Modin dependencies and Ray.\r\nconda install -c conda-forge modin-dask # Install Modin dependencies and Dask.\r\nconda install -c conda-forge modin-mpi # Install Modin dependencies and MPI through unidist.\r\nconda install -c conda-forge modin-hdk # Install Modin dependencies and HDK.\r\n```\r\n\r\nRefer to\r\n[Installing with conda](https://unidist.readthedocs.io/en/latest/installation.html#installing-with-conda)\r\nsection of the unidist documentation for more details on how to install a specific MPI implementation to run on.\r\n\r\nTo speed up conda installation we recommend using libmamba solver. To do this install it in a base environment:\r\n\r\n```bash\r\nconda install -n base conda-libmamba-solver\r\n```\r\n\r\nand then use it during istallation either like:\r\n\r\n```bash\r\nconda install -c conda-forge modin-ray modin-hdk --experimental-solver=libmamba\r\n```\r\n\r\nor starting from conda 22.11 and libmamba solver 22.12 versions:\r\n\r\n```bash\r\nconda install -c conda-forge modin-ray modin-hdk --solver=libmamba\r\n```\r\n\r\n#### Choosing a Compute Engine\r\n\r\nIf you want to choose a specific compute engine to run on, you can set the environment\r\nvariable `MODIN_ENGINE` and Modin will do computation with that engine:\r\n\r\n```bash\r\nexport MODIN_ENGINE=ray # Modin will use Ray\r\nexport MODIN_ENGINE=dask # Modin will use Dask\r\nexport MODIN_ENGINE=unidist # Modin will use Unidist\r\n```\r\n\r\nIf you want to choose the Unidist engine, you should set the additional environment \r\nvariable ``UNIDIST_BACKEND``. Currently, Modin only supports MPI through unidist:\r\n\r\n```bash\r\nexport UNIDIST_BACKEND=mpi # Unidist will use MPI backend\r\n```\r\n\r\nThis can also be done within a notebook/interpreter before you import Modin:\r\n\r\n```python\r\nimport modin.config as modin_cfg\r\nimport unidist.config as unidist_cfg\r\n\r\nmodin_cfg.Engine.put(\"ray\") # Modin will use Ray\r\nmodin_cfg.Engine.put(\"dask\") # Modin will use Dask\r\n\r\nmodin_cfg.Engine.put('unidist') # Modin will use Unidist\r\nunidist_cfg.Backend.put('mpi') # Unidist will use MPI backend\r\n```\r\n\r\nCheck [this Modin docs section](https://modin.readthedocs.io/en/latest/development/using_hdk.html) for HDK engine setup.\r\n\r\n_Note: You should not change the engine after your first operation with Modin as it will result in undefined behavior._\r\n\r\n#### Which engine should I use?\r\n\r\nOn Linux, MacOS, and Windows you can install and use either Ray, Dask or MPI through unidist. There is no knowledge required\r\nto use either of these engines as Modin abstracts away all of the complexity, so feel\r\nfree to pick either!\r\n\r\nOn Linux you also can choose [HDK](https://modin.readthedocs.io/en/latest/development/using_hdk.html), which is an experimental\r\nengine based on [HDK](https://github.com/intel-ai/hdk) and included in the\r\n[Intel\u00ae Distribution of Modin](https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/distribution-of-modin.html),\r\nwhich is a part of [Intel\u00ae oneAPI AI Analytics Toolkit (AI Kit)](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html).\r\n\r\n### Pandas API Coverage\r\n\r\n<p align=\"center\">\r\n\r\n| pandas Object | Modin's Ray Engine Coverage | Modin's Dask Engine Coverage | Modin's Unidist Engine Coverage |\r\n|-------------------|:------------------------------------------------------------------------------------:|:---------------:|:---------------:|\r\n| `pd.DataFrame` | <img src=https://img.shields.io/badge/api%20coverage-90.8%25-hunter.svg> | <img src=https://img.shields.io/badge/api%20coverage-90.8%25-hunter.svg> | <img src=https://img.shields.io/badge/api%20coverage-90.8%25-hunter.svg> |\r\n| `pd.Series` | <img src=https://img.shields.io/badge/api%20coverage-88.05%25-green.svg> | <img src=https://img.shields.io/badge/api%20coverage-88.05%25-green.svg> | <img src=https://img.shields.io/badge/api%20coverage-88.05%25-green.svg> \r\n| `pd.read_csv` | \u2705 | \u2705 | \u2705 |\r\n| `pd.read_table` | \u2705 | \u2705 | \u2705 |\r\n| `pd.read_parquet` | \u2705 | \u2705 | \u2705 |\r\n| `pd.read_sql` | \u2705 | \u2705 | \u2705 |\r\n| `pd.read_feather` | \u2705 | \u2705 | \u2705 |\r\n| `pd.read_excel` | \u2705 | \u2705 | \u2705 |\r\n| `pd.read_json` | [\u2733\ufe0f](https://github.com/modin-project/modin/issues/554) | [\u2733\ufe0f](https://github.com/modin-project/modin/issues/554) | [\u2733\ufe0f](https://github.com/modin-project/modin/issues/554) |\r\n| `pd.read_<other>` | [\u2734\ufe0f](https://modin.readthedocs.io/en/latest/supported_apis/io_supported.html) | [\u2734\ufe0f](https://modin.readthedocs.io/en/latest/supported_apis/io_supported.html) | [\u2734\ufe0f](https://modin.readthedocs.io/en/latest/supported_apis/io_supported.html) |\r\n\r\n</p>\r\nSome pandas APIs are easier to implement than others, so if something is missing feel\r\nfree to open an issue!\r\n\r\n### More about Modin\r\n\r\nFor the complete documentation on Modin, visit our [ReadTheDocs](https://modin.readthedocs.io/en/latest/index.html) page.\r\n\r\n#### Scale your pandas workflow by changing a single line of code.\r\n\r\n_Note: In local mode (without a cluster), Modin will create and manage a local (Dask or Ray) cluster for the execution._\r\n\r\nTo use Modin, you do not need to specify how to distribute the data, or even know how many\r\ncores your system has. In fact, you can continue using your previous\r\npandas notebooks while experiencing a considerable speedup from Modin, even on a single\r\nmachine. Once you've changed your import statement, you're ready to use Modin just like\r\nyou would with pandas!\r\n\r\n#### Faster pandas, even on your laptop\r\n\r\n<img align=\"right\" style=\"display:inline;\" height=\"350\" width=\"300\" src=\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/read_csv_benchmark.png?raw=true\"></a>\r\n\r\nThe `modin.pandas` DataFrame is an extremely light-weight parallel DataFrame.\r\nModin transparently distributes the data and computation so that you can continue using the same pandas API\r\nwhile working with more data faster. Because it is so light-weight,\r\nModin provides speed-ups of up to 4x on a laptop with 4 physical cores.\r\n\r\nIn pandas, you are only able to use one core at a time when you are doing computation of\r\nany kind. With Modin, you are able to use all of the CPU cores on your machine. Even with a\r\ntraditionally synchronous task like `read_csv`, we see large speedups by efficiently\r\ndistributing the work across your entire machine.\r\n\r\n```python\r\nimport modin.pandas as pd\r\n\r\ndf = pd.read_csv(\"my_dataset.csv\")\r\n```\r\n\r\n#### Modin can handle the datasets that pandas can't \r\n\r\nOften data scientists have to switch between different tools\r\nfor operating on datasets of different sizes. Processing large dataframes with pandas\r\nis slow, and pandas does not support working with dataframes that are too large to fit\r\ninto the available memory. As a result, pandas workflows that work well\r\nfor prototyping on a few MBs of data do not scale to tens or hundreds of GBs (depending on the size\r\nof your machine). Modin supports operating on data that does not fit in memory, so that you can comfortably\r\nwork with hundreds of GBs without worrying about substantial slowdown or memory errors.\r\nWith [cluster](https://modin.readthedocs.io/en/latest/getting_started/using_modin/using_modin_cluster.html)\r\nand [out of core](https://modin.readthedocs.io/en/latest/getting_started/why_modin/out_of_core.html)\r\nsupport, Modin is a DataFrame library with both great single-node performance and high\r\nscalability in a cluster.\r\n\r\n#### Modin Architecture\r\n\r\nWe designed [Modin's architecture](https://modin.readthedocs.io/en/latest/development/architecture.html)\r\nto be modular so we can plug in different components as they develop and improve:\r\n\r\n<img src=\"https://github.com/modin-project/modin/raw/7c009c747caa90554607e30b9ac2bd1b190b8c7d/docs/img/modin_architecture.png\" alt=\"Modin's architecture\" width=\"75%\"></img>\r\n\r\n### Other Resources\r\n\r\n#### Getting Started with Modin\r\n\r\n- [Documentation](https://modin.readthedocs.io/en/latest/)\r\n- [10-min Quickstart Guide](https://modin.readthedocs.io/en/latest/getting_started/quickstart.html)\r\n- [Examples and Tutorials](https://modin.readthedocs.io/en/latest/getting_started/examples.html)\r\n- [Videos and Blogposts](https://modin.readthedocs.io/en/latest/getting_started/examples.html#talks-podcasts)\r\n- [Benchmarking Modin](https://modin.readthedocs.io/en/latest/usage_guide/benchmarking.html)\r\n\r\n#### Modin Community\r\n\r\n- [Slack](https://join.slack.com/t/modin-project/shared_invite/zt-yvk5hr3b-f08p_ulbuRWsAfg9rMY3uA)\r\n- [Twitter](https://twitter.com/modin_project)\r\n- [Mailing List](https://groups.google.com/g/modin-dev)\r\n- [GitHub Issues](https://github.com/modin-project/modin/issues)\r\n- [StackOverflow](https://stackoverflow.com/questions/tagged/modin)\r\n\r\n#### Learn More about Modin\r\n\r\n- [Frequently Asked Questions (FAQs)](https://modin.readthedocs.io/en/latest/getting_started/faq.html)\r\n- [Troubleshooting Guide](https://modin.readthedocs.io/en/latest/getting_started/troubleshooting.html)\r\n- [Development Guide](https://modin.readthedocs.io/en/latest/development/index.html)\r\n- Modin is built on many years of research and development at UC Berkeley. Check out these selected papers to learn more about how Modin works:\r\n - [Flexible Rule-Based Decomposition and Metadata Independence in Modin](https://people.eecs.berkeley.edu/~totemtang/paper/Modin.pdf) (VLDB 2021)\r\n - [Dataframe Systems: Theory, Architecture, and Implementation](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-193.pdf) (PhD Dissertation 2021)\r\n - [Towards Scalable Dataframe Systems](https://arxiv.org/pdf/2001.00888.pdf) (VLDB 2020)\r\n\r\n#### Getting Involved\r\n\r\n***`modin.pandas` is currently under active development. Requests and contributions are welcome!***\r\n\r\nFor more information on how to contribute to Modin, check out the\r\n[Modin Contribution Guide](https://modin.readthedocs.io/en/latest/development/contributing.html).\r\n\r\n### License\r\n\r\n[Apache License 2.0](LICENSE)\r\n",
"bugtrack_url": null,
"license": "Apache 2",
"summary": "Modin: Make your pandas code run faster by changing one line of code.",
"version": "0.29.0",
"project_urls": {
"Homepage": "https://github.com/modin-project/modin"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "46367ea4e0d179feaf58321c61c1c48419c93ff3586462520108aeaab6de5dc3",
"md5": "e449c3bcb7326d73253153bf7ad50f01",
"sha256": "0b4e9ca4948e6fc93bcca942aec1db58d4224fac77e5f6daf3870b05bcbb8181"
},
"downloads": -1,
"filename": "modin-0.29.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e449c3bcb7326d73253153bf7ad50f01",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 1182772,
"upload_time": "2024-04-15T17:53:02",
"upload_time_iso_8601": "2024-04-15T17:53:02.450769Z",
"url": "https://files.pythonhosted.org/packages/46/36/7ea4e0d179feaf58321c61c1c48419c93ff3586462520108aeaab6de5dc3/modin-0.29.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "55876c4ac92b414983c3d02615452231117eb019ae5dcb170c90552e4af6b9e7",
"md5": "a2209edf2c0da70de8d2ea74c97eb84a",
"sha256": "d40394e48eceff9103a2fc7ad736d37a917b9312fc1e660c92d102bd398dc6f9"
},
"downloads": -1,
"filename": "modin-0.29.0.tar.gz",
"has_sig": false,
"md5_digest": "a2209edf2c0da70de8d2ea74c97eb84a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 910811,
"upload_time": "2024-04-15T17:53:05",
"upload_time_iso_8601": "2024-04-15T17:53:05.274972Z",
"url": "https://files.pythonhosted.org/packages/55/87/6c4ac92b414983c3d02615452231117eb019ae5dcb170c90552e4af6b9e7/modin-0.29.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-15 17:53:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "modin-project",
"github_project": "modin",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "modin"
}
