| Name | nest-joblib JSON |
| Version |
0.1.7
JSON |
| download |
| home_page | https://github.com/34j/nest-joblib |
| Summary | Patch joblib to allow nested parallelism |
| upload_time | 2024-11-15 16:03:06 |
| maintainer | None |
| docs_url | None |
| author | 34j |
| requires_python | <4.0,>=3.8 |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# nest-joblib
<p align="center">
<a href="https://github.com/34j/nest-joblib/actions/workflows/ci.yml?query=branch%3Amain">
<img src="https://img.shields.io/github/actions/workflow/status/34j/nest-joblib/ci.yml?branch=main&label=CI&logo=github&style=flat-square" alt="CI Status" >
</a>
<a href="https://nest-joblib.readthedocs.io">
<img src="https://img.shields.io/readthedocs/nest-joblib.svg?logo=read-the-docs&logoColor=fff&style=flat-square" alt="Documentation Status">
</a>
<a href="https://codecov.io/gh/34j/nest-joblib">
<img src="https://img.shields.io/codecov/c/github/34j/nest-joblib.svg?logo=codecov&logoColor=fff&style=flat-square" alt="Test coverage percentage">
</a>
</p>
<p align="center">
<a href="https://python-poetry.org/">
<img src="https://img.shields.io/badge/packaging-poetry-299bd7?style=flat-square&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA4AAAASCAYAAABrXO8xAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAJJSURBVHgBfZLPa1NBEMe/s7tNXoxW1KJQKaUHkXhQvHgW6UHQQ09CBS/6V3hKc/AP8CqCrUcpmop3Cx48eDB4yEECjVQrlZb80CRN8t6OM/teagVxYZi38+Yz853dJbzoMV3MM8cJUcLMSUKIE8AzQ2PieZzFxEJOHMOgMQQ+dUgSAckNXhapU/NMhDSWLs1B24A8sO1xrN4NECkcAC9ASkiIJc6k5TRiUDPhnyMMdhKc+Zx19l6SgyeW76BEONY9exVQMzKExGKwwPsCzza7KGSSWRWEQhyEaDXp6ZHEr416ygbiKYOd7TEWvvcQIeusHYMJGhTwF9y7sGnSwaWyFAiyoxzqW0PM/RjghPxF2pWReAowTEXnDh0xgcLs8l2YQmOrj3N7ByiqEoH0cARs4u78WgAVkoEDIDoOi3AkcLOHU60RIg5wC4ZuTC7FaHKQm8Hq1fQuSOBvX/sodmNJSB5geaF5CPIkUeecdMxieoRO5jz9bheL6/tXjrwCyX/UYBUcjCaWHljx1xiX6z9xEjkYAzbGVnB8pvLmyXm9ep+W8CmsSHQQY77Zx1zboxAV0w7ybMhQmfqdmmw3nEp1I0Z+FGO6M8LZdoyZnuzzBdjISicKRnpxzI9fPb+0oYXsNdyi+d3h9bm9MWYHFtPeIZfLwzmFDKy1ai3p+PDls1Llz4yyFpferxjnyjJDSEy9CaCx5m2cJPerq6Xm34eTrZt3PqxYO1XOwDYZrFlH1fWnpU38Y9HRze3lj0vOujZcXKuuXm3jP+s3KbZVra7y2EAAAAAASUVORK5CYII=" alt="Poetry">
</a>
<a href="https://github.com/ambv/black">
<img src="https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square" alt="black">
</a>
<a href="https://github.com/pre-commit/pre-commit">
<img src="https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white&style=flat-square" alt="pre-commit">
</a>
</p>
<p align="center">
<a href="https://pypi.org/project/nest-joblib/">
<img src="https://img.shields.io/pypi/v/nest-joblib.svg?logo=python&logoColor=fff&style=flat-square" alt="PyPI Version">
</a>
<img src="https://img.shields.io/pypi/pyversions/nest-joblib.svg?style=flat-square&logo=python&logoColor=fff" alt="Supported Python versions">
<img src="https://img.shields.io/pypi/l/nest-joblib.svg?style=flat-square" alt="License">
</p>
Patch joblib to allow nested parallelism.
## Installation
Install this via pip (or your favourite package manager):
```shell
pip install nest-joblib
```
## Usage
```python
from nest_joblib import apply
apply()
```
With the above code, LokyBackend supports nested-parallelism.
## Advanced Usage
The following joblib specification of not doing nested-parallelism may be inefficient in an environment with sufficient memory.
`joblib/_parallel_backends.py`:
```python
def get_nested_backend(self):
"""Backend instance to be used by nested Parallel calls.
By default a thread-based backend is used for the first level of
nesting. Beyond, switch to sequential backend to avoid spawning too
many threads on the host.
"""
nesting_level = getattr(self, 'nesting_level', 0) + 1
if nesting_level > 1:
return SequentialBackend(nesting_level=nesting_level), None
else:
return ThreadingBackend(nesting_level=nesting_level), None
```
After calling `nest_joblib.apply()`, when `joblib.parallel.register_parallel_backend(name, backend)` is called, a subclass of `backend` with modified `get_nested_backend` is dynamically generated and registered with the name `f"nested-{name}"`.
```python
from joblib.parallel import parallel_backend
from nest_joblib import apply
from ray.util.joblib import register_ray
# use LokyBackend
apply()
# use DaskDistributedBackend
apply()
parallel_backend("nested-dask")
# use RayBackend
apply()
register_ray()
parallel_backend("nested-ray")
# use custom backend
from joblib.parallel import LokyBackend
apply()
class MyBackend(LokyBackend):
pass
register_parallel_backend("custom", MyBackend)
parallel_backend("nested-custom")
```
## Contributors ✨
Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
<!-- prettier-ignore-start -->
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->
<table>
<tbody>
<tr>
<td align="center" valign="top" width="14.28%"><a href="https://github.com/34j"><img src="https://avatars.githubusercontent.com/u/55338215?v=4?s=80" width="80px;" alt="34j"/><br /><sub><b>34j</b></sub></a><br /><a href="https://github.com/34j/nest-joblib/commits?author=34j" title="Code">💻</a> <a href="#ideas-34j" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/34j/nest-joblib/commits?author=34j" title="Documentation">📖</a></td>
</tr>
</tbody>
</table>
<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->
<!-- ALL-CONTRIBUTORS-LIST:END -->
<!-- prettier-ignore-end -->
This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!
Raw data
{
"_id": null,
"home_page": "https://github.com/34j/nest-joblib",
"name": "nest-joblib",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": null,
"author": "34j",
"author_email": "34j.95a2p@simplelogin.com",
"download_url": "https://files.pythonhosted.org/packages/e0/1c/5eed2ecbe4d52e879ad041517e485404bffe37a822076dddf6d15fb6a9d8/nest_joblib-0.1.7.tar.gz",
"platform": null,
"description": "# nest-joblib\n\n<p align=\"center\">\n <a href=\"https://github.com/34j/nest-joblib/actions/workflows/ci.yml?query=branch%3Amain\">\n <img src=\"https://img.shields.io/github/actions/workflow/status/34j/nest-joblib/ci.yml?branch=main&label=CI&logo=github&style=flat-square\" alt=\"CI Status\" >\n </a>\n <a href=\"https://nest-joblib.readthedocs.io\">\n <img src=\"https://img.shields.io/readthedocs/nest-joblib.svg?logo=read-the-docs&logoColor=fff&style=flat-square\" alt=\"Documentation Status\">\n </a>\n <a href=\"https://codecov.io/gh/34j/nest-joblib\">\n <img src=\"https://img.shields.io/codecov/c/github/34j/nest-joblib.svg?logo=codecov&logoColor=fff&style=flat-square\" alt=\"Test coverage percentage\">\n </a>\n</p>\n<p align=\"center\">\n <a href=\"https://python-poetry.org/\">\n <img src=\"https://img.shields.io/badge/packaging-poetry-299bd7?style=flat-square&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAA4AAAASCAYAAABrXO8xAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAJJSURBVHgBfZLPa1NBEMe/s7tNXoxW1KJQKaUHkXhQvHgW6UHQQ09CBS/6V3hKc/AP8CqCrUcpmop3Cx48eDB4yEECjVQrlZb80CRN8t6OM/teagVxYZi38+Yz853dJbzoMV3MM8cJUcLMSUKIE8AzQ2PieZzFxEJOHMOgMQQ+dUgSAckNXhapU/NMhDSWLs1B24A8sO1xrN4NECkcAC9ASkiIJc6k5TRiUDPhnyMMdhKc+Zx19l6SgyeW76BEONY9exVQMzKExGKwwPsCzza7KGSSWRWEQhyEaDXp6ZHEr416ygbiKYOd7TEWvvcQIeusHYMJGhTwF9y7sGnSwaWyFAiyoxzqW0PM/RjghPxF2pWReAowTEXnDh0xgcLs8l2YQmOrj3N7ByiqEoH0cARs4u78WgAVkoEDIDoOi3AkcLOHU60RIg5wC4ZuTC7FaHKQm8Hq1fQuSOBvX/sodmNJSB5geaF5CPIkUeecdMxieoRO5jz9bheL6/tXjrwCyX/UYBUcjCaWHljx1xiX6z9xEjkYAzbGVnB8pvLmyXm9ep+W8CmsSHQQY77Zx1zboxAV0w7ybMhQmfqdmmw3nEp1I0Z+FGO6M8LZdoyZnuzzBdjISicKRnpxzI9fPb+0oYXsNdyi+d3h9bm9MWYHFtPeIZfLwzmFDKy1ai3p+PDls1Llz4yyFpferxjnyjJDSEy9CaCx5m2cJPerq6Xm34eTrZt3PqxYO1XOwDYZrFlH1fWnpU38Y9HRze3lj0vOujZcXKuuXm3jP+s3KbZVra7y2EAAAAAASUVORK5CYII=\" alt=\"Poetry\">\n </a>\n <a href=\"https://github.com/ambv/black\">\n <img src=\"https://img.shields.io/badge/code%20style-black-000000.svg?style=flat-square\" alt=\"black\">\n </a>\n <a href=\"https://github.com/pre-commit/pre-commit\">\n <img src=\"https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white&style=flat-square\" alt=\"pre-commit\">\n </a>\n</p>\n<p align=\"center\">\n <a href=\"https://pypi.org/project/nest-joblib/\">\n <img src=\"https://img.shields.io/pypi/v/nest-joblib.svg?logo=python&logoColor=fff&style=flat-square\" alt=\"PyPI Version\">\n </a>\n <img src=\"https://img.shields.io/pypi/pyversions/nest-joblib.svg?style=flat-square&logo=python&logoColor=fff\" alt=\"Supported Python versions\">\n <img src=\"https://img.shields.io/pypi/l/nest-joblib.svg?style=flat-square\" alt=\"License\">\n</p>\n\nPatch joblib to allow nested parallelism.\n\n## Installation\n\nInstall this via pip (or your favourite package manager):\n\n```shell\npip install nest-joblib\n```\n\n## Usage\n\n```python\nfrom nest_joblib import apply\n\napply()\n```\n\nWith the above code, LokyBackend supports nested-parallelism.\n\n## Advanced Usage\n\nThe following joblib specification of not doing nested-parallelism may be inefficient in an environment with sufficient memory.\n\n`joblib/_parallel_backends.py`:\n\n```python\n def get_nested_backend(self):\n \"\"\"Backend instance to be used by nested Parallel calls.\n\n By default a thread-based backend is used for the first level of\n nesting. Beyond, switch to sequential backend to avoid spawning too\n many threads on the host.\n \"\"\"\n nesting_level = getattr(self, 'nesting_level', 0) + 1\n if nesting_level > 1:\n return SequentialBackend(nesting_level=nesting_level), None\n else:\n return ThreadingBackend(nesting_level=nesting_level), None\n```\n\nAfter calling `nest_joblib.apply()`, when `joblib.parallel.register_parallel_backend(name, backend)` is called, a subclass of `backend` with modified `get_nested_backend` is dynamically generated and registered with the name `f\"nested-{name}\"`.\n\n```python\nfrom joblib.parallel import parallel_backend\nfrom nest_joblib import apply\nfrom ray.util.joblib import register_ray\n\n# use LokyBackend\napply()\n\n# use DaskDistributedBackend\napply()\nparallel_backend(\"nested-dask\")\n\n# use RayBackend\napply()\nregister_ray()\nparallel_backend(\"nested-ray\")\n\n# use custom backend\nfrom joblib.parallel import LokyBackend\napply()\nclass MyBackend(LokyBackend):\n pass\nregister_parallel_backend(\"custom\", MyBackend)\nparallel_backend(\"nested-custom\")\n```\n\n## Contributors \u2728\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n<!-- prettier-ignore-start -->\n<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->\n<!-- prettier-ignore-start -->\n<!-- markdownlint-disable -->\n<table>\n <tbody>\n <tr>\n <td align=\"center\" valign=\"top\" width=\"14.28%\"><a href=\"https://github.com/34j\"><img src=\"https://avatars.githubusercontent.com/u/55338215?v=4?s=80\" width=\"80px;\" alt=\"34j\"/><br /><sub><b>34j</b></sub></a><br /><a href=\"https://github.com/34j/nest-joblib/commits?author=34j\" title=\"Code\">\ud83d\udcbb</a> <a href=\"#ideas-34j\" title=\"Ideas, Planning, & Feedback\">\ud83e\udd14</a> <a href=\"https://github.com/34j/nest-joblib/commits?author=34j\" title=\"Documentation\">\ud83d\udcd6</a></td>\n </tr>\n </tbody>\n</table>\n\n<!-- markdownlint-restore -->\n<!-- prettier-ignore-end -->\n\n<!-- ALL-CONTRIBUTORS-LIST:END -->\n<!-- prettier-ignore-end -->\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Patch joblib to allow nested parallelism",
"version": "0.1.7",
"project_urls": {
"Bug Tracker": "https://github.com/34j/nest-joblib/issues",
"Changelog": "https://github.com/34j/nest-joblib/blob/main/CHANGELOG.md",
"Documentation": "https://nest-joblib.readthedocs.io",
"Homepage": "https://github.com/34j/nest-joblib",
"Repository": "https://github.com/34j/nest-joblib"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "553a829061f4b8b8f84169ce1e4387d161e2babf518bfc3fad8747ceb9d60342",
"md5": "f0d05cc4cb57bf13daa192e8a2de3fff",
"sha256": "5e33450b5a5a26edfb8a200da345c015bc86d7a67b362830d0999c831420616b"
},
"downloads": -1,
"filename": "nest_joblib-0.1.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f0d05cc4cb57bf13daa192e8a2de3fff",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 6583,
"upload_time": "2024-11-15T16:03:05",
"upload_time_iso_8601": "2024-11-15T16:03:05.193577Z",
"url": "https://files.pythonhosted.org/packages/55/3a/829061f4b8b8f84169ce1e4387d161e2babf518bfc3fad8747ceb9d60342/nest_joblib-0.1.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e01c5eed2ecbe4d52e879ad041517e485404bffe37a822076dddf6d15fb6a9d8",
"md5": "c4b16d34c02b2a526d2f02ccd78a4eec",
"sha256": "f4418292857b94e7c9fe6a19154e965774d4e62357b679f83c33a86eaaf87c51"
},
"downloads": -1,
"filename": "nest_joblib-0.1.7.tar.gz",
"has_sig": false,
"md5_digest": "c4b16d34c02b2a526d2f02ccd78a4eec",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 6236,
"upload_time": "2024-11-15T16:03:06",
"upload_time_iso_8601": "2024-11-15T16:03:06.830253Z",
"url": "https://files.pythonhosted.org/packages/e0/1c/5eed2ecbe4d52e879ad041517e485404bffe37a822076dddf6d15fb6a9d8/nest_joblib-0.1.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-15 16:03:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "34j",
"github_project": "nest-joblib",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "nest-joblib"
}
