# tqdm_pathos
Progress bars for multiprocessing with pathos
Wrappers based on [parmap](https://github.com/zeehio/parmap) for multiprocessing with [pathos](https://pathos.readthedocs.io/en/latest/pathos.html#module-pathos.pools) and progress bar completion with [tqdm](https://tqdm.github.io/). Following parmap, multiprocessing is extended to functions of multiple iterables, arguments, and keyword arguments.
While parmap includes these extensions and a progress bar, it is built on the default [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) library. The multiprocessing/pools module in pathos includes enhanced serialization to allow multiprocessing of, e.g., lambda functions, class methods, etc.
## Installation
You can `pip install tqdm-pathos`.
## Usage
A pool with an automatically detected number of cores is set up by default. To choose the number of cores, use the `n_cpus` kwarg.
Alternatively, an existing pool can be used by passing it to the `pool` kwarg.
Extra `kwargs` can be passed to the `tqdm` progress bar using the `tqdm_kwargs` dictionary argument, e.g., `tqdm_kwargs = {'desc': 'pbar description'}`.
Function of a single iterable:
```
f = lambda x: x**2
iterable = [1, 2, 3]
# Serial
y = [f(x) for x in iterable]
# Parallel
y = tqdm_pathos.map(f, iterable)
```
Function of a single iterable, with non-iterable args and kwargs:
```
def f(x, a, b=0):
return x**2 * a + b
iterable = [1, 2, 3]
a = 1
b = 0
# Serial
y = [f(x, a, b=b) for x in iterable]
# Parallel
y = tqdm_pathos.map(f, iterable, a, b=b)
```
Function of multiple iterables:
```
f = lambda x, y: x * y
iterable1 = [1, 2, 3]
iterable2 = [4, 5, 6]
# Serial
z = [f(x, y) for x, y in zip(iterable1, iterable2)]
# Parallel
z = tqdm_pathos.starmap(f, zip(iterable1, iterable2))
```
Function of multiple iterables, with non-iterable args and kwargs:
```
def f(x, y, a, b=0):
return x * y * a + b
iterable1 = [1, 2, 3]
iterable2 = [4, 5, 6]
a = 1
b = 0
# Serial
z = [f(x, y, a, b=b) for x, y in zip(iterable1, iterable2)]
# Parallel
z = tqdm_pathos.starmap(f, zip(iterable1, iterable2), a, b=b)
```
Raw data
{
"_id": null,
"home_page": "https://github.com/mdmould/tqdm_pathos",
"name": "tqdm-pathos",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Matthew Mould",
"author_email": "mattdmould@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/f6/8b/8ef162e336841283ed60e097a9834f27176afd74f85b677523cfb8a50243/tqdm_pathos-0.3.tar.gz",
"platform": null,
"description": "# tqdm_pathos\nProgress bars for multiprocessing with pathos\n\nWrappers based on [parmap](https://github.com/zeehio/parmap) for multiprocessing with [pathos](https://pathos.readthedocs.io/en/latest/pathos.html#module-pathos.pools) and progress bar completion with [tqdm](https://tqdm.github.io/). Following parmap, multiprocessing is extended to functions of multiple iterables, arguments, and keyword arguments.\n\nWhile parmap includes these extensions and a progress bar, it is built on the default [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) library. The multiprocessing/pools module in pathos includes enhanced serialization to allow multiprocessing of, e.g., lambda functions, class methods, etc.\n\n## Installation\n\nYou can `pip install tqdm-pathos`.\n\n## Usage\n\nA pool with an automatically detected number of cores is set up by default. To choose the number of cores, use the `n_cpus` kwarg.\nAlternatively, an existing pool can be used by passing it to the `pool` kwarg.\nExtra `kwargs` can be passed to the `tqdm` progress bar using the `tqdm_kwargs` dictionary argument, e.g., `tqdm_kwargs = {'desc': 'pbar description'}`.\n\nFunction of a single iterable:\n```\nf = lambda x: x**2\niterable = [1, 2, 3]\n\n# Serial\ny = [f(x) for x in iterable]\n\n# Parallel\ny = tqdm_pathos.map(f, iterable)\n```\n\nFunction of a single iterable, with non-iterable args and kwargs:\n```\ndef f(x, a, b=0):\n return x**2 * a + b\niterable = [1, 2, 3]\na = 1\nb = 0\n \n# Serial\ny = [f(x, a, b=b) for x in iterable]\n\n# Parallel\ny = tqdm_pathos.map(f, iterable, a, b=b)\n```\n\nFunction of multiple iterables:\n```\nf = lambda x, y: x * y\niterable1 = [1, 2, 3]\niterable2 = [4, 5, 6]\n\n# Serial\nz = [f(x, y) for x, y in zip(iterable1, iterable2)]\n\n# Parallel\nz = tqdm_pathos.starmap(f, zip(iterable1, iterable2))\n```\n\nFunction of multiple iterables, with non-iterable args and kwargs:\n```\ndef f(x, y, a, b=0):\n return x * y * a + b\niterable1 = [1, 2, 3]\niterable2 = [4, 5, 6]\na = 1\nb = 0\n\n# Serial\nz = [f(x, y, a, b=b) for x, y in zip(iterable1, iterable2)]\n\n# Parallel\nz = tqdm_pathos.starmap(f, zip(iterable1, iterable2), a, b=b)\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "tqdm_pathos",
"version": "0.3",
"project_urls": {
"Homepage": "https://github.com/mdmould/tqdm_pathos"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "270f548a6f7f34627b28c99df5a3e1b25ed295695632a908c239bb5e492fbf6f",
"md5": "00b63b8c98abdf17cbdf2ee60815abac",
"sha256": "504ec12596ee5aae509b09ff6a27edbb9822fe3bc798197e27725dde4a05e34b"
},
"downloads": -1,
"filename": "tqdm_pathos-0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "00b63b8c98abdf17cbdf2ee60815abac",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 7130,
"upload_time": "2023-08-25T07:48:25",
"upload_time_iso_8601": "2023-08-25T07:48:25.575797Z",
"url": "https://files.pythonhosted.org/packages/27/0f/548a6f7f34627b28c99df5a3e1b25ed295695632a908c239bb5e492fbf6f/tqdm_pathos-0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f68b8ef162e336841283ed60e097a9834f27176afd74f85b677523cfb8a50243",
"md5": "98dcfd65ffafffeff06c33b50c5824fd",
"sha256": "398c0189af5564c93da34ca6001c128ea4aca235e6a3383dc0c7de5cfded310a"
},
"downloads": -1,
"filename": "tqdm_pathos-0.3.tar.gz",
"has_sig": false,
"md5_digest": "98dcfd65ffafffeff06c33b50c5824fd",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6928,
"upload_time": "2023-08-25T07:48:27",
"upload_time_iso_8601": "2023-08-25T07:48:27.063774Z",
"url": "https://files.pythonhosted.org/packages/f6/8b/8ef162e336841283ed60e097a9834f27176afd74f85b677523cfb8a50243/tqdm_pathos-0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-25 07:48:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mdmould",
"github_project": "tqdm_pathos",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "tqdm-pathos"
}
JSON
