| Name | parmap JSON |
| Version |
1.7.0
JSON |
| download |
| home_page | |
| Summary | map and starmap implementations passing additional arguments and parallelizing if possible |
| upload_time | 2023-09-09 17:36:02 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | |
| license | |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
|
parmap
======
.. image:: https://github.com/zeehio/parmap/actions/workflows/test.yml/badge.svg
:target: https://github.com/zeehio/parmap/actions/workflows/test.yml
.. image:: https://img.shields.io/conda/vn/conda-forge/parmap.svg
:target: https://anaconda.org/conda-forge/parmap
:alt: conda-forge version
.. image:: https://readthedocs.org/projects/parmap/badge/?version=latest
:target: https://readthedocs.org/projects/parmap/?badge=latest
:alt: Documentation Status
.. image:: https://codecov.io/github/zeehio/parmap/coverage.svg?branch=main
:target: https://codecov.io/github/zeehio/parmap?branch=main
.. image:: https://codeclimate.com/github/zeehio/parmap/badges/gpa.svg
:target: https://codeclimate.com/github/zeehio/parmap
:alt: Code Climate
This small python module implements four functions: ``map`` and
``starmap``, and their async versions ``map_async`` and ``starmap_async``.
What does parmap offer?
-----------------------
- Provide an easy to use syntax for both ``map`` and ``starmap``.
- Parallelize transparently whenever possible.
- Pass additional positional and keyword arguments to parallelized functions.
- Show a progress bar (requires `tqdm` as optional package)
Installation:
-------------
::
pip install tqdm # for progress bar support
pip install parmap
Usage:
------
Here are some examples with some unparallelized code parallelized with
parmap:
Simple parallelization example:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
import parmap
# You want to do:
mylist = [1,2,3]
argument1 = 3.14
argument2 = True
y = [myfunction(x, argument1, mykeyword=argument2) for x in mylist]
# In parallel:
y = parmap.map(myfunction, mylist, argument1, mykeyword=argument2)
Show a progress bar:
~~~~~~~~~~~~~~~~~~~~~
Requires ``pip install tqdm``
::
# You want to do:
y = [myfunction(x) for x in mylist]
# In parallel, with a progress bar
y = parmap.map(myfunction, mylist, pm_pbar=True)
# Passing extra options to the tqdm progress bar
y = parmap.map(myfunction, mylist, pm_pbar={"desc": "Example"})
Passing multiple arguments:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
# You want to do:
z = [myfunction(x, y, argument1, argument2, mykey=argument3) for (x,y) in mylist]
# In parallel:
z = parmap.starmap(myfunction, mylist, argument1, argument2, mykey=argument3)
# You want to do:
listx = [1, 2, 3, 4, 5, 6]
listy = [2, 3, 4, 5, 6, 7]
param = 3.14
param2 = 42
listz = []
for (x, y) in zip(listx, listy):
listz.append(myfunction(x, y, param1, param2))
# In parallel:
listz = parmap.starmap(myfunction, zip(listx, listy), param1, param2)
Advanced: Multiple parallel tasks running in parallel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this example, Task1 uses 5 cores, while Task2 uses 3 cores. Both tasks start
to compute simultaneously, and we print a message as soon as any of the tasks
finishes, retreiving the result.
::
import parmap
def task1(item):
return 2*item
def task2(item):
return 2*item + 1
items1 = range(500000)
items2 = range(500)
with parmap.map_async(task1, items1, pm_processes=5) as result1:
with parmap.map_async(task2, items2, pm_processes=3) as result2:
data_task1 = None
data_task2 = None
task1_working = True
task2_working = True
while task1_working or task2_working:
result1.wait(0.1)
if task1_working and result1.ready():
print("Task 1 has finished!")
data_task1 = result1.get()
task1_working = False
result2.wait(0.1)
if task2_working and result2.ready():
print("Task 2 has finished!")
data_task2 = result2.get()
task2_working = False
#Further work with data_task1 or data_task2
map and starmap already exist. Why reinvent the wheel?
---------------------------------------------------------
The existing functions have some usability limitations:
- The built-in python function ``map`` [#builtin-map]_
is not able to parallelize.
- ``multiprocessing.Pool().map`` [#multiproc-map]_
does not allow any additional argument to the mapped function.
- ``multiprocessing.Pool().starmap`` allows passing multiple arguments,
but in order to pass a constant argument to the mapped function you
will need to convert it to an iterator using
``itertools.repeat(your_parameter)`` [#itertools-repeat]_
``parmap`` aims to overcome this limitations in the simplest possible way.
Additional features in parmap:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Create a pool for parallel computation automatically if possible.
- ``parmap.map(..., ..., pm_parallel=False)`` # disables parallelization
- ``parmap.map(..., ..., pm_processes=4)`` # use 4 parallel processes
- ``parmap.map(..., ..., pm_pbar=True)`` # show a progress bar (requires tqdm)
- ``parmap.map(..., ..., pm_pool=multiprocessing.Pool())`` # use an existing
pool, in this case parmap will not close the pool.
- ``parmap.map(..., ..., pm_chunksize=3)`` # size of chunks (see
multiprocessing.Pool().map)
Limitations:
-------------
``parmap.map()`` and ``parmap.starmap()`` (and their async versions) have their own
arguments (``pm_parallel``, ``pm_pbar``...). Those arguments are never passed
to the underlying function. In the following example, ``myfun`` will receive
``myargument``, but not ``pm_parallel``. Do not write functions that require
keyword arguments starting with ``pm_``, as ``parmap`` may need them in the future.
::
parmap.map(myfun, mylist, pm_parallel=True, myargument=False)
Additionally, there are other keyword arguments that should be avoided in the
functions you write, because of parmap backwards compatibility reasons. The list
of conflicting arguments is: ``parallel``, ``chunksize``, ``pool``,
``processes``, ``callback``, ``error_callback`` and ``parmap_progress``.
Acknowledgments:
----------------
This package started after `this question <https://stackoverflow.com/q/5442910/446149>`__,
when I offered this `answer <http://stackoverflow.com/a/21292849/446149>`__,
taking the suggestions of J.F. Sebastian for his `answer <http://stackoverflow.com/a/5443941/446149>`__
Known works using parmap
---------------------------
- Davide Gerosa, Michael Kesden, "PRECESSION. Dynamics of spinning black-hole
binaries with python." `arXiv:1605.01067 <https://arxiv.org/abs/1605.01067>`__, 2016
- Thibault de Boissiere, `Implementation of Deep learning papers <https://github.com/tdeboissiere/DeepLearningImplementations>`__, 2017
- Wasserstein Generative Adversarial Networks `arXiv:1701.07875 <https://arxiv.org/abs/1701.07875>`__
- pix2pix `arXiv:1611.07004 <https://arxiv.org/abs/1611.07004>`__
- Improved Techniques for Training Generative Adversarial Networks `arXiv:1606.03498 <https://arxiv.org/abs/1606.03498>`__
- Colorful Image Colorization `arXiv:1603.08511 <https://arxiv.org/abs/1603.08511>`__
- Deep Feature Interpolation for Image Content Changes `arXiv:1611.05507 <https://arxiv.org/abs/1611.05507>`__
- InfoGAN `arXiv:1606.03657 <https://arxiv.org/abs/1606.03657>`__
- Geoscience Australia, `SIFRA, a System for Infrastructure Facility Resilience Analysis <https://github.com/GeoscienceAustralia/sifra>`__, 2017
- André F. Rendeiro, Christian Schmidl, Jonathan C. Strefford, Renata Walewska, Zadie Davis, Matthias Farlik, David Oscier, Christoph Bock "Chromatin accessibility maps of chronic lymphocytic leukemia identify subtype-specific epigenome signatures and transcription regulatory networks" Nat. Commun. 7:11938 doi: 10.1038/ncomms11938 (2016). `Paper <https://doi.org/10.5281/zenodo.231352>`__, `Code <https://github.com/epigen/cll-chromatin>`__
References
-----------
.. [#builtin-map] http://docs.python.org/dev/library/functions.html#map
.. [#multiproc-starmap] http://docs.python.org/dev/library/multiprocessing.html#multiprocessing.pool.Pool.starmap
.. [#multiproc-map] http://docs.python.org/dev/library/multiprocessing.html#multiprocessing.pool.Pool.map
.. [#itertools-repeat] http://docs.python.org/dev/library/itertools.html#itertools.repeat
Raw data
{
"_id": null,
"home_page": "",
"name": "parmap",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "",
"author_email": "Sergio Oller <sergioller@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/6a/a7/440ce4b53a4918773c65077ea95136890c1037adfd87065fbb2c757ea381/parmap-1.7.0.tar.gz",
"platform": null,
"description": "parmap\n======\n\n.. image:: https://github.com/zeehio/parmap/actions/workflows/test.yml/badge.svg\n :target: https://github.com/zeehio/parmap/actions/workflows/test.yml\n\n.. image:: https://img.shields.io/conda/vn/conda-forge/parmap.svg\n :target: https://anaconda.org/conda-forge/parmap\n :alt: conda-forge version\n\n.. image:: https://readthedocs.org/projects/parmap/badge/?version=latest\n :target: https://readthedocs.org/projects/parmap/?badge=latest\n :alt: Documentation Status\n\n.. image:: https://codecov.io/github/zeehio/parmap/coverage.svg?branch=main\n :target: https://codecov.io/github/zeehio/parmap?branch=main\n\n.. image:: https://codeclimate.com/github/zeehio/parmap/badges/gpa.svg\n :target: https://codeclimate.com/github/zeehio/parmap\n :alt: Code Climate\n\n\nThis small python module implements four functions: ``map`` and\n``starmap``, and their async versions ``map_async`` and ``starmap_async``.\n\nWhat does parmap offer?\n-----------------------\n\n- Provide an easy to use syntax for both ``map`` and ``starmap``.\n- Parallelize transparently whenever possible.\n- Pass additional positional and keyword arguments to parallelized functions.\n- Show a progress bar (requires `tqdm` as optional package)\n\nInstallation:\n-------------\n\n::\n\n \u00a0pip install tqdm # for progress bar support\n pip install parmap\n\n\nUsage:\n------\n\nHere are some examples with some unparallelized code parallelized with\nparmap:\n\nSimple parallelization example:\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n import parmap\n # You want to do:\n mylist = [1,2,3]\n argument1 = 3.14\n argument2 = True\n y = [myfunction(x, argument1, mykeyword=argument2) for x in mylist]\n # In parallel:\n y = parmap.map(myfunction, mylist, argument1, mykeyword=argument2)\n\n\nShow a progress bar:\n~~~~~~~~~~~~~~~~~~~~~\n\nRequires ``pip install tqdm``\n\n::\n\n # You want to do:\n y = [myfunction(x) for x in mylist]\n # In parallel, with a progress bar\n y = parmap.map(myfunction, mylist, pm_pbar=True)\n # Passing extra options to the tqdm progress bar\n y = parmap.map(myfunction, mylist, pm_pbar={\"desc\": \"Example\"})\n\n\nPassing multiple arguments:\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n::\n\n # You want to do:\n z = [myfunction(x, y, argument1, argument2, mykey=argument3) for (x,y) in mylist]\n # In parallel:\n z = parmap.starmap(myfunction, mylist, argument1, argument2, mykey=argument3)\n\n # You want to do:\n listx = [1, 2, 3, 4, 5, 6]\n listy = [2, 3, 4, 5, 6, 7]\n param = 3.14\n param2 = 42\n listz = []\n for (x, y) in zip(listx, listy):\n listz.append(myfunction(x, y, param1, param2))\n # In parallel:\n listz = parmap.starmap(myfunction, zip(listx, listy), param1, param2)\n\n\nAdvanced: Multiple parallel tasks running in parallel\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn this example, Task1 uses 5 cores, while Task2 uses 3 cores. Both tasks start\nto compute simultaneously, and we print a message as soon as any of the tasks\nfinishes, retreiving the result.\n\n::\n\n import parmap\n def task1(item):\n return 2*item\n\n def task2(item):\n return 2*item + 1\n\n items1 = range(500000)\n items2 = range(500)\n\n with parmap.map_async(task1, items1, pm_processes=5) as result1:\n with parmap.map_async(task2, items2, pm_processes=3) as result2:\n data_task1 = None\n data_task2 = None\n task1_working = True\n task2_working = True\n while task1_working or task2_working:\n result1.wait(0.1)\n if task1_working and result1.ready():\n print(\"Task 1 has finished!\")\n data_task1 = result1.get()\n task1_working = False\n result2.wait(0.1)\n if task2_working and result2.ready():\n print(\"Task 2 has finished!\")\n data_task2 = result2.get()\n task2_working = False\n #Further work with data_task1 or data_task2\n\n\nmap and starmap already exist. Why reinvent the wheel?\n---------------------------------------------------------\n\nThe existing functions have some usability limitations:\n\n- The built-in python function ``map`` [#builtin-map]_\n is not able to parallelize.\n- ``multiprocessing.Pool().map`` [#multiproc-map]_\n does not allow any additional argument to the mapped function.\n- ``multiprocessing.Pool().starmap`` allows passing multiple arguments,\n but in order to pass a constant argument to the mapped function you\n will need to convert it to an iterator using\n ``itertools.repeat(your_parameter)`` [#itertools-repeat]_\n\n``parmap`` aims to overcome this limitations in the simplest possible way.\n\nAdditional features in parmap:\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n- Create a pool for parallel computation automatically if possible.\n- ``parmap.map(..., ..., pm_parallel=False)`` # disables parallelization\n- ``parmap.map(..., ..., pm_processes=4)`` # use 4 parallel processes\n- ``parmap.map(..., ..., pm_pbar=True)`` # show a progress bar (requires tqdm)\n- ``parmap.map(..., ..., pm_pool=multiprocessing.Pool())`` # use an existing\n pool, in this case parmap will not close the pool.\n- ``parmap.map(..., ..., pm_chunksize=3)`` # size of chunks (see\n multiprocessing.Pool().map)\n\nLimitations:\n-------------\n\n``parmap.map()`` and ``parmap.starmap()`` (and their async versions) have their own \narguments (``pm_parallel``, ``pm_pbar``...). Those arguments are never passed\nto the underlying function. In the following example, ``myfun`` will receive \n``myargument``, but not ``pm_parallel``. Do not write functions that require\nkeyword arguments starting with ``pm_``, as ``parmap`` may need them in the future.\n\n::\n\n parmap.map(myfun, mylist, pm_parallel=True, myargument=False)\n\nAdditionally, there are other keyword arguments that should be avoided in the\nfunctions you write, because of parmap backwards compatibility reasons. The list\nof conflicting arguments is: ``parallel``, ``chunksize``, ``pool``,\n``processes``, ``callback``, ``error_callback`` and ``parmap_progress``.\n\n\n\nAcknowledgments:\n----------------\n\nThis package started after `this question <https://stackoverflow.com/q/5442910/446149>`__, \nwhen I offered this `answer <http://stackoverflow.com/a/21292849/446149>`__, \ntaking the suggestions of J.F. Sebastian for his `answer <http://stackoverflow.com/a/5443941/446149>`__\n\nKnown works using parmap\n---------------------------\n\n- Davide Gerosa, Michael Kesden, \"PRECESSION. Dynamics of spinning black-hole\n binaries with python.\" `arXiv:1605.01067 <https://arxiv.org/abs/1605.01067>`__, 2016\n- Thibault de Boissiere, `Implementation of Deep learning papers <https://github.com/tdeboissiere/DeepLearningImplementations>`__, 2017\n - Wasserstein Generative Adversarial Networks `arXiv:1701.07875 <https://arxiv.org/abs/1701.07875>`__\n - pix2pix `arXiv:1611.07004 <https://arxiv.org/abs/1611.07004>`__\n - Improved Techniques for Training Generative Adversarial Networks `arXiv:1606.03498 <https://arxiv.org/abs/1606.03498>`__\n - Colorful Image Colorization `arXiv:1603.08511 <https://arxiv.org/abs/1603.08511>`__\n - Deep Feature Interpolation for Image Content Changes `arXiv:1611.05507 <https://arxiv.org/abs/1611.05507>`__\n - InfoGAN `arXiv:1606.03657 <https://arxiv.org/abs/1606.03657>`__\n- Geoscience Australia, `SIFRA, a System for Infrastructure Facility Resilience Analysis <https://github.com/GeoscienceAustralia/sifra>`__, 2017\n- Andr\u00e9 F. Rendeiro, Christian Schmidl, Jonathan C. Strefford, Renata Walewska, Zadie Davis, Matthias Farlik, David Oscier, Christoph Bock \"Chromatin accessibility maps of chronic lymphocytic leukemia identify subtype-specific epigenome signatures and transcription regulatory networks\" Nat. Commun. 7:11938 doi: 10.1038/ncomms11938 (2016). `Paper <https://doi.org/10.5281/zenodo.231352>`__, `Code <https://github.com/epigen/cll-chromatin>`__\n\n\nReferences\n-----------\n\n.. [#builtin-map] http://docs.python.org/dev/library/functions.html#map\n.. [#multiproc-starmap] http://docs.python.org/dev/library/multiprocessing.html#multiprocessing.pool.Pool.starmap\n.. [#multiproc-map] http://docs.python.org/dev/library/multiprocessing.html#multiprocessing.pool.Pool.map\n.. [#itertools-repeat] http://docs.python.org/dev/library/itertools.html#itertools.repeat\n\n",
"bugtrack_url": null,
"license": "",
"summary": "map and starmap implementations passing additional arguments and parallelizing if possible",
"version": "1.7.0",
"project_urls": {
"Bug Tracker": "https://github.com/zeehio/parmap/issues",
"Homepage": "https://github.com/zeehio/parmap"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2d39f820a78508ec19de9209eaaf5d1d4722c96a500de8e8d52c50ef1eca0da6",
"md5": "f6de29b6a5877861045d8544246033cb",
"sha256": "4953c7092442dec9560f9b25f9ff184006acd467980c00ab798f1644d432a595"
},
"downloads": -1,
"filename": "parmap-1.7.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "f6de29b6a5877861045d8544246033cb",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 32869,
"upload_time": "2023-09-09T17:35:59",
"upload_time_iso_8601": "2023-09-09T17:35:59.847671Z",
"url": "https://files.pythonhosted.org/packages/2d/39/f820a78508ec19de9209eaaf5d1d4722c96a500de8e8d52c50ef1eca0da6/parmap-1.7.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6aa7440ce4b53a4918773c65077ea95136890c1037adfd87065fbb2c757ea381",
"md5": "8e1d6005081d5debf507ba9fd1be500a",
"sha256": "77c45210617c3c084e073d61e3a8111e398bbc606d4e3ec03f9a0c3aadc3f47b"
},
"downloads": -1,
"filename": "parmap-1.7.0.tar.gz",
"has_sig": false,
"md5_digest": "8e1d6005081d5debf507ba9fd1be500a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 22608,
"upload_time": "2023-09-09T17:36:02",
"upload_time_iso_8601": "2023-09-09T17:36:02.614827Z",
"url": "https://files.pythonhosted.org/packages/6a/a7/440ce4b53a4918773c65077ea95136890c1037adfd87065fbb2c757ea381/parmap-1.7.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-09 17:36:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "zeehio",
"github_project": "parmap",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "parmap"
}
