# p_tqdm
[](https://badge.fury.io/py/p-tqdm)
[](https://badge.fury.io/py/p-tqdm)
[](https://pepy.tech/project/p_tqdm)
[](https://github.com/swansonk14/p_tqdm)
[](https://codecov.io/gh/swansonk14/p_tqdm)
[](https://github.com/swansonk14/p_tqdm/blob/main/LICENSE.txt)
`p_tqdm` makes parallel processing with progress bars easy.
`p_tqdm` is a wrapper around [pathos.multiprocessing](https://github.com/uqfoundation/pathos/blob/master/pathos/multiprocessing.py) and [tqdm](https://github.com/tqdm/tqdm). Unlike Python's default multiprocessing library, pathos provides a more flexible parallel map which can apply almost any type of function, including lambda functions, nested functions, and class methods, and can easily handle functions with multiple arguments. tqdm is applied on top of pathos's parallel map and displays a progress bar including an estimated time to completion.
## Installation
```pip install p_tqdm```
## Example
Let's say you want to add two lists element by element. Without any parallelism, this can be done easily with a Python `map`.
```python
l1 = ['1', '2', '3']
l2 = ['a', 'b', 'c']
def add(a, b):
return a + b
added = map(add, l1, l2)
# added == ['1a', '2b', '3c']
```
But if the lists are much larger or the computation is more intense, parallelism becomes a necessity. However, the syntax is often cumbersome. `p_tqdm` makes it easy and adds a progress bar too.
```python
from p_tqdm import p_map
added = p_map(add, l1, l2)
# added == ['1a', '2b', '3c']
```
```
0%| | 0/3 [00:00<?, ?it/s]
33%|████████████ | 1/3 [00:01<00:02, 1.00s/it]
66%|████████████████████████ | 2/3 [00:02<00:01, 1.00s/it]
100%|████████████████████████████████████| 3/3 [00:03<00:00, 1.00s/it]
```
## p_tqdm functions
### Parallel maps
* [p_map](#p_map) - parallel ordered map
* [p_imap](#p_imap) - iterator for parallel ordered map
* [p_umap](#p_umap) - parallel unordered map
* [p_uimap](#p_uimap) - iterator for parallel unordered map
### Sequential maps
* [t_map](#t_map) - sequential ordered map
* [t_imap](#t_imap) - iterator for sequential ordered map
### p_map
Performs an ordered map in parallel.
```python
from p_tqdm import p_map
def add(a, b):
return a + b
added = p_map(add, ['1', '2', '3'], ['a', 'b', 'c'])
# added = ['1a', '2b', '3c']
```
### p_imap
Returns an iterator for an ordered map in parallel.
```python
from p_tqdm import p_imap
def add(a, b):
return a + b
iterator = p_imap(add, ['1', '2', '3'], ['a', 'b', 'c'])
for result in iterator:
print(result) # prints '1a', '2b', '3c'
```
### p_umap
Performs an unordered map in parallel.
```python
from p_tqdm import p_umap
def add(a, b):
return a + b
added = p_umap(add, ['1', '2', '3'], ['a', 'b', 'c'])
# added is an array with '1a', '2b', '3c' in any order
```
### p_uimap
Returns an iterator for an unordered map in parallel.
```python
from p_tqdm import p_uimap
def add(a, b):
return a + b
iterator = p_uimap(add, ['1', '2', '3'], ['a', 'b', 'c'])
for result in iterator:
print(result) # prints '1a', '2b', '3c' in any order
```
### t_map
Performs an ordered map sequentially.
```python
from p_tqdm import t_map
def add(a, b):
return a + b
added = t_map(add, ['1', '2', '3'], ['a', 'b', 'c'])
# added == ['1a', '2b', '3c']
```
### t_imap
Returns an iterator for an ordered map to be performed sequentially.
```python
from p_tqdm import p_imap
def add(a, b):
return a + b
iterator = t_imap(add, ['1', '2', '3'], ['a', 'b', 'c'])
for result in iterator:
print(result) # prints '1a', '2b', '3c'
```
## Shared properties
### Arguments
All `p_tqdm` functions accept any number of iterables as input, as long as the number of iterables matches the number of arguments of the function.
To repeat a non-iterable argument along with the iterables, use Python's [partial](https://docs.python.org/3/library/functools.html#functools.partial) from the [functools](https://docs.python.org/3/library/functools.html) library. See the example below.
```python
from functools import partial
l1 = ['1', '2', '3']
l2 = ['a', 'b', 'c']
def add(a, b, c=''):
return a + b + c
added = p_map(partial(add, c='!'), l1, l2)
# added == ['1a!', '2b!', '3c!']
```
### CPUs
All the parallel `p_tqdm` functions can be passed the keyword `num_cpus` to indicate how many CPUs to use. The default is all CPUs. `num_cpus` can either be an integer to indicate the exact number of CPUs to use or a float to indicate the proportion of CPUs to use.
Note that the parallel Pool objects used by `p_tqdm` are automatically closed when the map finishes processing.
### tqdm instance
All the parallel `p_tqdm` functions can be passed the keyword `tqdm` to choose a specific flavor of tqdm. By default, this value is taken from `tqdm.auto`. The `tqdm` parameter can be used pass `p_tqdm` output to `tqdm.gui`, `tqdm.tk` or any customized subclass of `tqdm`.
Raw data
{
"_id": null,
"home_page": "https://github.com/swansonk14/p_tqdm",
"name": "p-tqdm",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "tqdm,progress bar,parallel",
"author": "Kyle Swanson",
"author_email": "swansonk.14@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/90/77/c4ac58f96154d1760a06f7e569c79b923d0da452011e25bbf03d1cb7658f/p_tqdm-1.4.0.tar.gz",
"platform": null,
"description": "# p_tqdm\n\n[](https://badge.fury.io/py/p-tqdm)\n[](https://badge.fury.io/py/p-tqdm)\n[](https://pepy.tech/project/p_tqdm)\n[](https://github.com/swansonk14/p_tqdm)\n[](https://codecov.io/gh/swansonk14/p_tqdm)\n[](https://github.com/swansonk14/p_tqdm/blob/main/LICENSE.txt)\n\n`p_tqdm` makes parallel processing with progress bars easy.\n\n`p_tqdm` is a wrapper around [pathos.multiprocessing](https://github.com/uqfoundation/pathos/blob/master/pathos/multiprocessing.py) and [tqdm](https://github.com/tqdm/tqdm). Unlike Python's default multiprocessing library, pathos provides a more flexible parallel map which can apply almost any type of function, including lambda functions, nested functions, and class methods, and can easily handle functions with multiple arguments. tqdm is applied on top of pathos's parallel map and displays a progress bar including an estimated time to completion.\n\n## Installation\n\n```pip install p_tqdm```\n\n## Example\n\nLet's say you want to add two lists element by element. Without any parallelism, this can be done easily with a Python `map`.\n\n```python\nl1 = ['1', '2', '3']\nl2 = ['a', 'b', 'c']\n\ndef add(a, b):\n return a + b\n \nadded = map(add, l1, l2)\n# added == ['1a', '2b', '3c']\n```\n\nBut if the lists are much larger or the computation is more intense, parallelism becomes a necessity. However, the syntax is often cumbersome. `p_tqdm` makes it easy and adds a progress bar too.\n\n```python\nfrom p_tqdm import p_map\n\nadded = p_map(add, l1, l2)\n# added == ['1a', '2b', '3c']\n```\n\n```\n 0%| | 0/3 [00:00<?, ?it/s]\n 33%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 1/3 [00:01<00:02, 1.00s/it]\n 66%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588 | 2/3 [00:02<00:01, 1.00s/it]\n100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 3/3 [00:03<00:00, 1.00s/it]\n```\n\n## p_tqdm functions\n\n### Parallel maps\n\n* [p_map](#p_map) - parallel ordered map\n* [p_imap](#p_imap) - iterator for parallel ordered map\n* [p_umap](#p_umap) - parallel unordered map\n* [p_uimap](#p_uimap) - iterator for parallel unordered map\n\n### Sequential maps\n* [t_map](#t_map) - sequential ordered map\n* [t_imap](#t_imap) - iterator for sequential ordered map\n\n### p_map\n\nPerforms an ordered map in parallel.\n\n```python\nfrom p_tqdm import p_map\n\ndef add(a, b):\n return a + b\n\nadded = p_map(add, ['1', '2', '3'], ['a', 'b', 'c'])\n# added = ['1a', '2b', '3c']\n```\n\n### p_imap\n\nReturns an iterator for an ordered map in parallel.\n\n```python\nfrom p_tqdm import p_imap\n\ndef add(a, b):\n return a + b\n\niterator = p_imap(add, ['1', '2', '3'], ['a', 'b', 'c'])\n\nfor result in iterator:\n print(result) # prints '1a', '2b', '3c'\n```\n\n### p_umap\n\nPerforms an unordered map in parallel.\n\n```python\nfrom p_tqdm import p_umap\n\ndef add(a, b):\n return a + b\n\nadded = p_umap(add, ['1', '2', '3'], ['a', 'b', 'c'])\n# added is an array with '1a', '2b', '3c' in any order\n```\n\n### p_uimap\n\nReturns an iterator for an unordered map in parallel.\n\n```python\nfrom p_tqdm import p_uimap\n\ndef add(a, b):\n return a + b\n\niterator = p_uimap(add, ['1', '2', '3'], ['a', 'b', 'c'])\n\nfor result in iterator:\n print(result) # prints '1a', '2b', '3c' in any order\n```\n\n### t_map\n\nPerforms an ordered map sequentially.\n\n```python\nfrom p_tqdm import t_map\n\ndef add(a, b):\n return a + b\n\nadded = t_map(add, ['1', '2', '3'], ['a', 'b', 'c'])\n# added == ['1a', '2b', '3c']\n```\n\n### t_imap\n\nReturns an iterator for an ordered map to be performed sequentially.\n\n```python\nfrom p_tqdm import p_imap\n\ndef add(a, b):\n return a + b\n\niterator = t_imap(add, ['1', '2', '3'], ['a', 'b', 'c'])\n\nfor result in iterator:\n print(result) # prints '1a', '2b', '3c'\n```\n\n## Shared properties\n\n### Arguments\n\nAll `p_tqdm` functions accept any number of iterables as input, as long as the number of iterables matches the number of arguments of the function.\n\nTo repeat a non-iterable argument along with the iterables, use Python's [partial](https://docs.python.org/3/library/functools.html#functools.partial) from the [functools](https://docs.python.org/3/library/functools.html) library. See the example below.\n\n```python\nfrom functools import partial\n\nl1 = ['1', '2', '3']\nl2 = ['a', 'b', 'c']\n\ndef add(a, b, c=''):\n return a + b + c\n\nadded = p_map(partial(add, c='!'), l1, l2)\n# added == ['1a!', '2b!', '3c!']\n```\n\n### CPUs\n\nAll the parallel `p_tqdm` functions can be passed the keyword `num_cpus` to indicate how many CPUs to use. The default is all CPUs. `num_cpus` can either be an integer to indicate the exact number of CPUs to use or a float to indicate the proportion of CPUs to use.\n\nNote that the parallel Pool objects used by `p_tqdm` are automatically closed when the map finishes processing.\n\n### tqdm instance\n\nAll the parallel `p_tqdm` functions can be passed the keyword `tqdm` to choose a specific flavor of tqdm. By default, this value is taken from `tqdm.auto`. The `tqdm` parameter can be used pass `p_tqdm` output to `tqdm.gui`, `tqdm.tk` or any customized subclass of `tqdm`.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Parallel processing with progress bars",
"version": "1.4.0",
"project_urls": {
"Download": "https://github.com/swansonk14/p_tqdm/v_1.3.3.tar.gz",
"Homepage": "https://github.com/swansonk14/p_tqdm"
},
"split_keywords": [
"tqdm",
"progress bar",
"parallel"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9077c4ac58f96154d1760a06f7e569c79b923d0da452011e25bbf03d1cb7658f",
"md5": "c5a86a88528076e912025453406a7c5f",
"sha256": "0d12aa223443f387244cce5e74a0aecc472d1f98189d58ccdfcda86772772bfc"
},
"downloads": -1,
"filename": "p_tqdm-1.4.0.tar.gz",
"has_sig": false,
"md5_digest": "c5a86a88528076e912025453406a7c5f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 5241,
"upload_time": "2022-08-27T18:34:36",
"upload_time_iso_8601": "2022-08-27T18:34:36.361295Z",
"url": "https://files.pythonhosted.org/packages/90/77/c4ac58f96154d1760a06f7e569c79b923d0da452011e25bbf03d1cb7658f/p_tqdm-1.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-08-27 18:34:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "swansonk14",
"github_project": "p_tqdm",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "p-tqdm"
}
JSON
