# Prefect Multiprocess Task Runner
<p align="center">
<a href="https://pypi.python.org/pypi/prefect-multiprocess/" alt="PyPI version">
<img alt="PyPI" src="https://img.shields.io/pypi/v/prefect-multiprocess?color=0052FF&labelColor=090422"></a>
<a href="https://github.com/rpeden/prefect-multiprocess/" alt="Stars">
<img src="https://img.shields.io/github/stars/rpeden/prefect-multiprocess?color=0052FF&labelColor=090422" /></a>
<a href="https://github.com/rpeden/prefect-multiprocess/pulse" alt="Activity">
<img src="https://img.shields.io/github/commit-activity/m/rpeden/prefect-multiprocess?color=0052FF&labelColor=090422" /></a>
<br>
</p>
## Welcome!
This Prefect collection contains a multiprocess task runner. It is ideal for running CPU-intensive Prefect tasks in parallel. It is useful in scenarios where you want to spread computation across multiple CPU cores on a single machine without adding heavy dependencies like Dask. This package does not require any extra dependencies beyond what Prefect already installs.
## Getting Started
Install the package by running:
```
pip install prefect-multiprocess
```
Then, use the task runner in your Prefect flows. The task runner only accepts one parameter: `processes`, which controls the number of worker processes to start for running tasks. If not provided, it defaults to the number of CPUs on the host machine.
Examples:
Create one process for every CPU/CPU core
```python
from prefect import flow
from prefect_multiprocess.task_runners import MultiprocessTaskRunner
@flow(task_runner=MultiprocessTaskRunner())
def my_flow():
...
```
Customizing the number of processes:
```python
@flow(task_runner=MultiprocessTaskRunner(processes=4))
def my_flow():
...
```
### Python setup
Requires an installation of Python 3.8+.
We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.
These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to the [Prefect documentation](https://docs.prefect.io/).
## Limitations
`MultiprocessTaskRunner` uses `cloudpickle` to serialize tasks and return values, so task parameters and returns need to be values that `cloudpickle` can handle.
## Resources
If you encounter any bugs while using `prefect-multiprocess`, feel free to open an issue in the [prefect-multiprocess](https://github.com/rpeden/prefect-multiprocess) repository.
Feel free to ⭐️ or watch [`prefect-multiprocess`](https://github.com/rpeden/prefect-multiprocess) for updates too!
## Development
If you'd like to install a version of `prefect-multiprocess` for development, clone the repository and perform an editable install with `pip`:
```bash
git clone https://github.com/rpeden/prefect-multiprocess.git
cd prefect-multiprocess/
pip install -e ".[dev]"
# Install linting pre-commit hooks
pre-commit install
```
Raw data
{
"_id": null,
"home_page": "https://github.com/rpeden/prefect-multiprocess",
"name": "prefect-multiprocess",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "prefect",
"author": "Ryan Peden",
"author_email": "ryan@rpeden.com",
"download_url": "https://files.pythonhosted.org/packages/d0/24/442efae392bc41284dba864ffce78e23c24124b2d621e7d6698a6f0c1a9c/prefect-multiprocess-0.1.2.tar.gz",
"platform": null,
"description": "# Prefect Multiprocess Task Runner\n\n<p align=\"center\">\n <a href=\"https://pypi.python.org/pypi/prefect-multiprocess/\" alt=\"PyPI version\">\n <img alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/prefect-multiprocess?color=0052FF&labelColor=090422\"></a> \n <a href=\"https://github.com/rpeden/prefect-multiprocess/\" alt=\"Stars\">\n <img src=\"https://img.shields.io/github/stars/rpeden/prefect-multiprocess?color=0052FF&labelColor=090422\" /></a> \n <a href=\"https://github.com/rpeden/prefect-multiprocess/pulse\" alt=\"Activity\">\n <img src=\"https://img.shields.io/github/commit-activity/m/rpeden/prefect-multiprocess?color=0052FF&labelColor=090422\" /></a>\n <br>\n</p>\n\n## Welcome!\n\nThis Prefect collection contains a multiprocess task runner. It is ideal for running CPU-intensive Prefect tasks in parallel. It is useful in scenarios where you want to spread computation across multiple CPU cores on a single machine without adding heavy dependencies like Dask. This package does not require any extra dependencies beyond what Prefect already installs.\n\n## Getting Started\n\nInstall the package by running:\n```\npip install prefect-multiprocess\n```\n\nThen, use the task runner in your Prefect flows. The task runner only accepts one parameter: `processes`, which controls the number of worker processes to start for running tasks. If not provided, it defaults to the number of CPUs on the host machine.\n\nExamples:\n\nCreate one process for every CPU/CPU core\n```python\nfrom prefect import flow\nfrom prefect_multiprocess.task_runners import MultiprocessTaskRunner\n\n\n@flow(task_runner=MultiprocessTaskRunner())\ndef my_flow():\n ...\n```\n\nCustomizing the number of processes:\n```python\n@flow(task_runner=MultiprocessTaskRunner(processes=4))\ndef my_flow():\n ...\n```\n\n### Python setup\n\nRequires an installation of Python 3.8+.\n\nWe recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.\n\nThese tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to the [Prefect documentation](https://docs.prefect.io/).\n\n## Limitations\n\n`MultiprocessTaskRunner` uses `cloudpickle` to serialize tasks and return values, so task parameters and returns need to be values that `cloudpickle` can handle.\n\n## Resources\n\nIf you encounter any bugs while using `prefect-multiprocess`, feel free to open an issue in the [prefect-multiprocess](https://github.com/rpeden/prefect-multiprocess) repository.\n\nFeel free to \u2b50\ufe0f or watch [`prefect-multiprocess`](https://github.com/rpeden/prefect-multiprocess) for updates too!\n\n## Development\n\nIf you'd like to install a version of `prefect-multiprocess` for development, clone the repository and perform an editable install with `pip`:\n\n```bash\ngit clone https://github.com/rpeden/prefect-multiprocess.git\n\ncd prefect-multiprocess/\n\npip install -e \".[dev]\"\n\n# Install linting pre-commit hooks\npre-commit install\n```\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "A multiprocess task runner for Prefect 2.8.3+.",
"version": "0.1.2",
"project_urls": {
"Homepage": "https://github.com/rpeden/prefect-multiprocess"
},
"split_keywords": [
"prefect"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ddbf249d2af47750477ae60e1fab24d8129db6760ca5a5acb1d75bbc976aa409",
"md5": "a7ba48432b4ae630e6e2a100027405d2",
"sha256": "fa4f3d390c7651de45547ade2c5829d3c73dc03cbbcbc27e9599ecd310a90292"
},
"downloads": -1,
"filename": "prefect_multiprocess-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a7ba48432b4ae630e6e2a100027405d2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 13147,
"upload_time": "2023-08-22T15:44:49",
"upload_time_iso_8601": "2023-08-22T15:44:49.676864Z",
"url": "https://files.pythonhosted.org/packages/dd/bf/249d2af47750477ae60e1fab24d8129db6760ca5a5acb1d75bbc976aa409/prefect_multiprocess-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d024442efae392bc41284dba864ffce78e23c24124b2d621e7d6698a6f0c1a9c",
"md5": "1b5ec29419c1396f29decf0738755d3b",
"sha256": "9f21b10c1ef04c147f9b4c3aa1c32b218b84007609f91f16d9cfc7fcb6924591"
},
"downloads": -1,
"filename": "prefect-multiprocess-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "1b5ec29419c1396f29decf0738755d3b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 31734,
"upload_time": "2023-08-22T15:44:51",
"upload_time_iso_8601": "2023-08-22T15:44:51.239734Z",
"url": "https://files.pythonhosted.org/packages/d0/24/442efae392bc41284dba864ffce78e23c24124b2d621e7d6698a6f0c1a9c/prefect-multiprocess-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-22 15:44:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "rpeden",
"github_project": "prefect-multiprocess",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "prefect-multiprocess"
}
JSON
