| Name | reporting-pool JSON |
| Version |
0.1.4
JSON |
| download |
| home_page | None |
| Summary | A wrapper around multiprocessing.Pool that keeps track of the completion and exceptions. |
| upload_time | 2024-09-10 18:54:28 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.7 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Reporting Pool
A simple wrapper around Python multiprocessing Pool that allows tracking of progress and storage of exceptions.
While starmap runs, a report is printed that shows % of jobs completed, time elapsed, estimated remaining time and a list of jobs statuses. Status can be:
Q -- queued,
R -- running,
S -- success,
F -- failed (in case track_failures is enabled).
The wrapper runs [multiprocessing.pool.starmap](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.starmap) on the `func` and `p_args` (see [Examples](#Examples)).
Enabling `track_failures` allows native tracking of failed function runs. For other arguments, see help(ReportingPool).
For many people, a [tqdm tracking of parallel processes](https://leimao.github.io/blog/Python-tqdm-Multiprocessing/) would be a simpler solution. The Reporting Pool is useful when each process prints a lot of information that cannot be suppressed, so a high-frequency reporting is useful, and can raise exceptions for unpredictable reasons breaking program flow.
Suggestions and contributions are welcome.
## Getting Started
This project's code is available on [GitHub](https://github.com/nishbo/reporting_pool).
### Prerequisites
Software:
- Python 3+ (tested on 3.7, 3.8, 3.11, 3.12)
### Installation
Reporting pool can now be installed from pypi:
```
py -m pip install reporting_pool
```
#### Install from source
1. Download the [repository](https://github.com/nishbo/reporting_pool) or clone it using git: `git clone https://github.com/nishbo/reporting_pool.git`.
2. Open Terminal, Command Line, or the desired Anaconda environment in the project folder.
3. Run `python setup.py install`.
## Examples
Example of a normal use:
```
from reporting_pool import ReportingPool
def _reporting_pool_test_func_wof(v):
time.sleep(0.25)
return v**2
p_args = [[v] for v in range(40)]
pool = ReportingPool(_reporting_pool_test_func_wof, p_args, processes=8,
report_on_change=True)
res = pool.start()
print(res)
```
Output:
```
Completed 0.00% of jobs. Time elapsed: 0.00 s, remaining: NaN s. List: QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.
Completed 20.00% of jobs. Time elapsed: 0.28 s, remaining: 1.13 s. List: SRSRSRSRSRSRSRSRQQQQQQQQQQQQQQQQQQQQQQQQ.
Completed 25.00% of jobs. Time elapsed: 0.53 s, remaining: 1.59 s. List: SSSSSRSRSRSRSRSRRQRQQQQQQQQQQQQQQQQQQQQQ.
Completed 40.00% of jobs. Time elapsed: 0.56 s, remaining: 0.84 s. List: SSSSSSSSSSSSSSSSRQRQRQRQRQRQRQRQQQQQQQQQ.
Completed 42.50% of jobs. Time elapsed: 0.78 s, remaining: 1.06 s. List: SSSSSSSSSSSSSSSSSRRQRQRQRQRQRQRQQQQQQQQQ.
Completed 60.00% of jobs. Time elapsed: 0.81 s, remaining: 0.54 s. List: SSSSSSSSSSSSSSSSSRSRSRSRSRSRSRSRQQQQQQQQ.
Completed 65.00% of jobs. Time elapsed: 1.06 s, remaining: 0.57 s. List: SSSSSSSSSSSSSSSSSSSSSRSRSRSRSRSRRQRQQQQQ.
Completed 80.00% of jobs. Time elapsed: 1.09 s, remaining: 0.27 s. List: SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSRQRQRQRQ.
Completed 82.50% of jobs. Time elapsed: 1.31 s, remaining: 0.28 s. List: SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSRRQRQRQ.
Completed 90.00% of jobs. Time elapsed: 1.34 s, remaining: 0.15 s. List: SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSRSRSRSR.
Completed 95.00% of jobs. Time elapsed: 1.59 s, remaining: 0.08 s. List: SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSRSR.
Reporting pool finished after 1.6205 s.
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961, 1024, 1089, 1156, 1225, 1296, 1369, 1444, 1521]
```
Example where the function can throw exceptions:
```
from reporting_pool import ReportingPool
# a function that throws an error
def _reporting_pool_test_func_wf(v):
time.sleep(0.25)
if v % 6 == 0:
raise ValueError()
return v**2
p_args = [[v] for v in range(40)]
pool = ReportingPool(_reporting_pool_test_func_wf, p_args,
report_rate=20, track_failures=True)
res = pool.start()
print(res)
```
Output:
```
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961, 1024, 1089, 1156, 1225, 1296, 1369, 1444, 1521]
Completed 0.00% of jobs. Time elapsed: 0.00 s, remaining: NaN s. List: QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.
Completed 0.00% of jobs. Time elapsed: 0.07 s, remaining: NaN s. List: RRRRRRRRRRRRQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.
Completed 0.00% of jobs. Time elapsed: 0.13 s, remaining: NaN s. List: RRRRRRRRRRRRQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.
Completed 0.00% of jobs. Time elapsed: 0.19 s, remaining: NaN s. List: RRRRRRRRRRRRQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.
Completed 0.00% of jobs. Time elapsed: 0.26 s, remaining: NaN s. List: RRRRRRRRRRRRQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.
Completed 30.00% of jobs. Time elapsed: 0.32 s, remaining: 0.75 s. List: FSSSSSFSSSSSRRRRRRRRRRRRQQQQQQQQQQQQQQQQ.
Completed 30.00% of jobs. Time elapsed: 0.38 s, remaining: 0.89 s. List: FSSSSSFSSSSSRRRRRRRRRRRRQQQQQQQQQQQQQQQQ.
Completed 30.00% of jobs. Time elapsed: 0.44 s, remaining: 1.04 s. List: FSSSSSFSSSSSRRRRRRRRRRRRQQQQQQQQQQQQQQQQ.
Completed 30.00% of jobs. Time elapsed: 0.51 s, remaining: 1.19 s. List: FSSSSSFSSSSSRRRRRRRRRRRRQQQQQQQQQQQQQQQQ.
Completed 60.00% of jobs. Time elapsed: 0.57 s, remaining: 0.38 s. List: FSSSSSFSSSSSFSSSSSFSSSSSRRRRRRRRRRRRQQQQ.
Completed 60.00% of jobs. Time elapsed: 0.63 s, remaining: 0.42 s. List: FSSSSSFSSSSSFSSSSSFSSSSSRRRRRRRRRRRRQQQQ.
Completed 60.00% of jobs. Time elapsed: 0.70 s, remaining: 0.47 s. List: FSSSSSFSSSSSFSSSSSFSSSSSRRRRRRRRRRRRQQQQ.
Completed 60.00% of jobs. Time elapsed: 0.76 s, remaining: 0.51 s. List: FSSSSSFSSSSSFSSSSSFSSSSSRRRRRRRRRRRRQQQQ.
Completed 77.50% of jobs. Time elapsed: 0.83 s, remaining: 0.24 s. List: FSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFRRRRRRRRR.
Completed 90.00% of jobs. Time elapsed: 0.89 s, remaining: 0.10 s. List: FSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFSSSSSRRRR.
Completed 90.00% of jobs. Time elapsed: 0.95 s, remaining: 0.11 s. List: FSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFSSSSSRRRR.
Completed 90.00% of jobs. Time elapsed: 1.01 s, remaining: 0.11 s. List: FSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFSSSSSRRRR.
Completed 95.00% of jobs. Time elapsed: 1.08 s, remaining: 0.06 s. List: FSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFSRR.
Reporting pool finished after 1.1385 s.
Jobs 0, 6, 12, 18, 24, 30, 36 were not finished correctly.
[None, 1, 4, 9, 16, 25, None, 49, 64, 81, 100, 121, None, 169, 196, 225, 256, 289, None, 361, 400, 441, 484, 529, None, 625, 676, 729, 784, 841, None, 961, 1024, 1089, 1156, 1225, None, 1369, 1444, 1521]
```
## Authors
- [**Anton Sobinov**](https://github.com/nishbo)
##### Keywords
python, multiprocessing, pool, starmap
Raw data
{
"_id": null,
"home_page": null,
"name": "reporting-pool",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "Anton Sobinov <an.sobinov@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/f2/9b/6386f2fe8e6a0ec1627fa0f66cd255e6f9be21ee8a45d9c2e1ce31c3fa58/reporting_pool-0.1.4.tar.gz",
"platform": null,
"description": "# Reporting Pool\r\n\r\nA simple wrapper around Python multiprocessing Pool that allows tracking of progress and storage of exceptions. \r\n\r\nWhile starmap runs, a report is printed that shows % of jobs completed, time elapsed, estimated remaining time and a list of jobs statuses. Status can be: \r\n Q -- queued,\r\n R -- running,\r\n S -- success,\r\n F -- failed (in case track_failures is enabled).\r\n\r\nThe wrapper runs [multiprocessing.pool.starmap](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.starmap) on the `func` and `p_args` (see [Examples](#Examples)). \r\n\r\nEnabling `track_failures` allows native tracking of failed function runs. For other arguments, see help(ReportingPool).\r\n\r\nFor many people, a [tqdm tracking of parallel processes](https://leimao.github.io/blog/Python-tqdm-Multiprocessing/) would be a simpler solution. The Reporting Pool is useful when each process prints a lot of information that cannot be suppressed, so a high-frequency reporting is useful, and can raise exceptions for unpredictable reasons breaking program flow.\r\n\r\nSuggestions and contributions are welcome.\r\n\r\n## Getting Started\r\n\r\nThis project's code is available on [GitHub](https://github.com/nishbo/reporting_pool).\r\n\r\n### Prerequisites\r\n\r\nSoftware:\r\n- Python 3+ (tested on 3.7, 3.8, 3.11, 3.12)\r\n\r\n### Installation\r\n\r\nReporting pool can now be installed from pypi:\r\n\r\n```\r\npy -m pip install reporting_pool\r\n```\r\n\r\n#### Install from source\r\n\r\n1. Download the [repository](https://github.com/nishbo/reporting_pool) or clone it using git: `git clone https://github.com/nishbo/reporting_pool.git`.\r\n2. Open Terminal, Command Line, or the desired Anaconda environment in the project folder.\r\n3. Run `python setup.py install`.\r\n\r\n## Examples\r\n\r\nExample of a normal use:\r\n\r\n```\r\nfrom reporting_pool import ReportingPool\r\n\r\ndef _reporting_pool_test_func_wof(v):\r\n time.sleep(0.25)\r\n return v**2\r\n\r\np_args = [[v] for v in range(40)]\r\n\r\npool = ReportingPool(_reporting_pool_test_func_wof, p_args, processes=8,\r\n report_on_change=True)\r\nres = pool.start()\r\n\r\nprint(res)\r\n```\r\n\r\nOutput:\r\n```\r\nCompleted 0.00% of jobs. Time elapsed: 0.00 s, remaining: NaN s. List: QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.\r\nCompleted 20.00% of jobs. Time elapsed: 0.28 s, remaining: 1.13 s. List: SRSRSRSRSRSRSRSRQQQQQQQQQQQQQQQQQQQQQQQQ.\r\nCompleted 25.00% of jobs. Time elapsed: 0.53 s, remaining: 1.59 s. List: SSSSSRSRSRSRSRSRRQRQQQQQQQQQQQQQQQQQQQQQ.\r\nCompleted 40.00% of jobs. Time elapsed: 0.56 s, remaining: 0.84 s. List: SSSSSSSSSSSSSSSSRQRQRQRQRQRQRQRQQQQQQQQQ.\r\nCompleted 42.50% of jobs. Time elapsed: 0.78 s, remaining: 1.06 s. List: SSSSSSSSSSSSSSSSSRRQRQRQRQRQRQRQQQQQQQQQ.\r\nCompleted 60.00% of jobs. Time elapsed: 0.81 s, remaining: 0.54 s. List: SSSSSSSSSSSSSSSSSRSRSRSRSRSRSRSRQQQQQQQQ.\r\nCompleted 65.00% of jobs. Time elapsed: 1.06 s, remaining: 0.57 s. List: SSSSSSSSSSSSSSSSSSSSSRSRSRSRSRSRRQRQQQQQ.\r\nCompleted 80.00% of jobs. Time elapsed: 1.09 s, remaining: 0.27 s. List: SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSRQRQRQRQ.\r\nCompleted 82.50% of jobs. Time elapsed: 1.31 s, remaining: 0.28 s. List: SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSRRQRQRQ.\r\nCompleted 90.00% of jobs. Time elapsed: 1.34 s, remaining: 0.15 s. List: SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSRSRSRSR.\r\nCompleted 95.00% of jobs. Time elapsed: 1.59 s, remaining: 0.08 s. List: SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSRSR.\r\nReporting pool finished after 1.6205 s.\r\n[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961, 1024, 1089, 1156, 1225, 1296, 1369, 1444, 1521]\r\n```\r\n\r\nExample where the function can throw exceptions:\r\n\r\n```\r\nfrom reporting_pool import ReportingPool\r\n\r\n# a function that throws an error\r\ndef _reporting_pool_test_func_wf(v):\r\n time.sleep(0.25)\r\n if v % 6 == 0:\r\n raise ValueError()\r\n return v**2\r\n\r\np_args = [[v] for v in range(40)]\r\n\r\npool = ReportingPool(_reporting_pool_test_func_wf, p_args,\r\n report_rate=20, track_failures=True)\r\nres = pool.start()\r\n\r\nprint(res)\r\n```\r\n\r\nOutput:\r\n```\r\n[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961, 1024, 1089, 1156, 1225, 1296, 1369, 1444, 1521]\r\nCompleted 0.00% of jobs. Time elapsed: 0.00 s, remaining: NaN s. List: QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.\r\nCompleted 0.00% of jobs. Time elapsed: 0.07 s, remaining: NaN s. List: RRRRRRRRRRRRQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.\r\nCompleted 0.00% of jobs. Time elapsed: 0.13 s, remaining: NaN s. List: RRRRRRRRRRRRQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.\r\nCompleted 0.00% of jobs. Time elapsed: 0.19 s, remaining: NaN s. List: RRRRRRRRRRRRQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.\r\nCompleted 0.00% of jobs. Time elapsed: 0.26 s, remaining: NaN s. List: RRRRRRRRRRRRQQQQQQQQQQQQQQQQQQQQQQQQQQQQ.\r\nCompleted 30.00% of jobs. Time elapsed: 0.32 s, remaining: 0.75 s. List: FSSSSSFSSSSSRRRRRRRRRRRRQQQQQQQQQQQQQQQQ.\r\nCompleted 30.00% of jobs. Time elapsed: 0.38 s, remaining: 0.89 s. List: FSSSSSFSSSSSRRRRRRRRRRRRQQQQQQQQQQQQQQQQ.\r\nCompleted 30.00% of jobs. Time elapsed: 0.44 s, remaining: 1.04 s. List: FSSSSSFSSSSSRRRRRRRRRRRRQQQQQQQQQQQQQQQQ.\r\nCompleted 30.00% of jobs. Time elapsed: 0.51 s, remaining: 1.19 s. List: FSSSSSFSSSSSRRRRRRRRRRRRQQQQQQQQQQQQQQQQ.\r\nCompleted 60.00% of jobs. Time elapsed: 0.57 s, remaining: 0.38 s. List: FSSSSSFSSSSSFSSSSSFSSSSSRRRRRRRRRRRRQQQQ.\r\nCompleted 60.00% of jobs. Time elapsed: 0.63 s, remaining: 0.42 s. List: FSSSSSFSSSSSFSSSSSFSSSSSRRRRRRRRRRRRQQQQ.\r\nCompleted 60.00% of jobs. Time elapsed: 0.70 s, remaining: 0.47 s. List: FSSSSSFSSSSSFSSSSSFSSSSSRRRRRRRRRRRRQQQQ.\r\nCompleted 60.00% of jobs. Time elapsed: 0.76 s, remaining: 0.51 s. List: FSSSSSFSSSSSFSSSSSFSSSSSRRRRRRRRRRRRQQQQ.\r\nCompleted 77.50% of jobs. Time elapsed: 0.83 s, remaining: 0.24 s. List: FSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFRRRRRRRRR.\r\nCompleted 90.00% of jobs. Time elapsed: 0.89 s, remaining: 0.10 s. List: FSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFSSSSSRRRR.\r\nCompleted 90.00% of jobs. Time elapsed: 0.95 s, remaining: 0.11 s. List: FSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFSSSSSRRRR.\r\nCompleted 90.00% of jobs. Time elapsed: 1.01 s, remaining: 0.11 s. List: FSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFSSSSSRRRR.\r\nCompleted 95.00% of jobs. Time elapsed: 1.08 s, remaining: 0.06 s. List: FSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFSSSSSFSRR.\r\nReporting pool finished after 1.1385 s.\r\nJobs 0, 6, 12, 18, 24, 30, 36 were not finished correctly.\r\n[None, 1, 4, 9, 16, 25, None, 49, 64, 81, 100, 121, None, 169, 196, 225, 256, 289, None, 361, 400, 441, 484, 529, None, 625, 676, 729, 784, 841, None, 961, 1024, 1089, 1156, 1225, None, 1369, 1444, 1521]\r\n```\r\n\r\n## Authors\r\n\r\n- [**Anton Sobinov**](https://github.com/nishbo)\r\n\r\n\r\n##### Keywords\r\npython, multiprocessing, pool, starmap\r\n",
"bugtrack_url": null,
"license": null,
"summary": "A wrapper around multiprocessing.Pool that keeps track of the completion and exceptions.",
"version": "0.1.4",
"project_urls": {
"Homepage": "https://github.com/nishbo/reporting_pool"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "62f8b92db299d6889e475575890825415a4e0044d24977c5e012607852e090b2",
"md5": "bf9f90a7a970e1aa35edb86448cb0c5e",
"sha256": "99b64ff6826f44074fe81b5074371d0a216b01162664f5ed1100e488149bedce"
},
"downloads": -1,
"filename": "reporting_pool-0.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "bf9f90a7a970e1aa35edb86448cb0c5e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 6854,
"upload_time": "2024-09-10T18:54:27",
"upload_time_iso_8601": "2024-09-10T18:54:27.283486Z",
"url": "https://files.pythonhosted.org/packages/62/f8/b92db299d6889e475575890825415a4e0044d24977c5e012607852e090b2/reporting_pool-0.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f29b6386f2fe8e6a0ec1627fa0f66cd255e6f9be21ee8a45d9c2e1ce31c3fa58",
"md5": "3f2401ff4bf65b1552aaa9348905fc0e",
"sha256": "df0da63c683520a0d184d2ef199ed3de6afc272b32a2c6058bd8bb3e96d1d1a4"
},
"downloads": -1,
"filename": "reporting_pool-0.1.4.tar.gz",
"has_sig": false,
"md5_digest": "3f2401ff4bf65b1552aaa9348905fc0e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6350,
"upload_time": "2024-09-10T18:54:28",
"upload_time_iso_8601": "2024-09-10T18:54:28.308478Z",
"url": "https://files.pythonhosted.org/packages/f2/9b/6386f2fe8e6a0ec1627fa0f66cd255e6f9be21ee8a45d9c2e1ce31c3fa58/reporting_pool-0.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-10 18:54:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nishbo",
"github_project": "reporting_pool",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "reporting-pool"
}
