deco3
=====
Decorated Concurrency.
A simplified parallel computing model for Python. DECO automatically
parallelizes Python programs, and requires minimal modifications to
existing serial programs.
Overview
========
|package_bold| is a strict fork of Alex Sherman's `deco package <deco_>`_
with a fix allowing to work with Python3 or higher and with a little code
reformatting and minor improvements.
`PyPI record`_.
`Documentation`_.
Overview below is a copy from the original `deco website <deco_>`_
(with only the necessary changes regarding |package|).
Documentation
-------------
You can reference the `Wiki on Github <deco_wiki_>`_
for slightly more in-depth documentation.
General Usage
-------------
Using DECO is as simple as finding, or creating, two functions in your
Python program. The first function is the one we want to run in
parallel, and is decorated with ``@concurrent``. The second function is
the function which calls the ``@concurrent`` function and is decorated
with ``@synchronized``. Decorating the second function is optional, but
provides some very cool benefits. Let's take a look at an example.
.. code:: python
@concurrent # We add this for the concurrent function
def process_lat_lon(lat, lon, data):
#Does some work which takes a while
return result
@synchronized # And we add this for the function which calls the concurrent function
def process_data_set(data):
results = defaultdict(dict)
for lat in range(...):
for lon in range(...):
results[lat][lon] = process_lat_lon(lat, lon, data)
return results
That's it, two lines of changes is all we need in order to parallelize
this program. Now this program will make use of all the cores on the
machine it's running on, allowing it to run significantly faster.
What it does
------------
- The ``@concurrent`` decorator uses multiprocessing.pool to parallelize
calls to the target function
- Indexed based mutation of function arguments is handled automatically,
which pool cannot do
- The ``@synchronized`` decorator automatically inserts synchronization
events
- It also automatically refactors assignments of the results of
``@concurrent`` function calls to happen during synchronization events
Limitations
-----------
- The ``@concurrent`` decorator will only speed up functions that take
longer than ~1ms
- If they take less time your code will run slower!
- By default, ``@concurrent`` function arguments/return values must be
pickleable for use with ``multiprocessing``
- The ``@synchronized`` decorator only works on 'simple' functions, make
sure the function meets the following criteria
- Only calls, or assigns the result of ``@concurrent`` functions to
indexable objects such as:
- concurrent(...)
- result[key] = concurrent(...)
- Never indirectly reads objects that get assigned to by calls of the
``@concurrent`` function
How it works
------------
For an in depth discussion of the mechanisms at work, we wrote a paper
for a class which `can be found here <decorated_concurrency_>`_.
As an overview, DECO is mainly just a smart wrapper for Python's
multiprocessing.pool. When ``@concurrent`` is applied to a function it
replaces it with calls to pool.apply_async. Additionally when arguments
are passed to pool.apply_async, DECO replaces any index mutable objects
with proxies, allowing it to detect and synchronize mutations of these
objects. The results of these calls can then be obtained by calling
wait() on the concurrent function, invoking a synchronization event.
These events can be placed automatically in your code by using the
``@synchronized`` decorator on functions that call ``@concurrent``
functions. Additionally while using ``@synchronized``, you can directly
assign the result of concurrent function calls to index mutable objects.
These assignments get refactored by DECO to automatically occur during
the next synchronization event. All of this means that in many cases,
parallel programming using DECO appears exactly the same as simpler
serial programming.
Installation
============
Prerequisites:
+ Python 3.10 or higher
* https://www.python.org/
+ pip and setuptools
* https://pypi.org/project/pip/
* https://pypi.org/project/setuptools/
To install run:
.. parsed-literal::
python -m pip install --upgrade |package|
Development
===========
Prerequisites:
+ Development is strictly based on *tox*. To install it run::
python -m pip install --upgrade tox
Visit `Development page`_.
Installation from sources:
clone the sources:
.. parsed-literal::
git clone |respository| |package|
and run:
.. parsed-literal::
python -m pip install ./|package|
or on development mode:
.. parsed-literal::
python -m pip install --editable ./|package|
License
=======
| |copyright|
| Copyright (c) 2016 Alex Sherman
| Licensed under the MIT License
| https://opensource.org/license/mit
| Please refer to the accompanying LICENSE file.
Authors
=======
* Alex Sherman <asherman1024@gmail.com>
* Adam Karpierz <adam@karpierz.net>
.. |package| replace:: deco3
.. |package_bold| replace:: **deco3**
.. |copyright| replace:: Copyright (c) 2025-2025 Adam Karpierz
.. |respository| replace:: https://github.com/karpierz/deco3.git
.. _Development page: https://github.com/karpierz/deco3
.. _PyPI record: https://pypi.org/project/deco3/
.. _Documentation: https://deco3.readthedocs.io/
.. _deco: https://pypi.org/project/deco/
.. _deco_wiki: https://github.com/alex-sherman/deco/wiki
.. _decorated_concurrency: _static/Decorated_Concurrency.pdf
Changelog
=========
0.9.0 (2025-08-20)
------------------
- 100% code coverage.
- Making the package typed.
- General improvements and cleanup.
- Setup (dependencies) update.
0.8.3 (2025-06-11)
------------------
- Setup (dependencies) update.
0.8.2 (2025-05-15)
------------------
- The distribution is now created using 'build' instead of 'setuptools'.
- Setup (dependencies) update (due to regressions in tox and setuptools).
0.8.1 (2025-05-04)
------------------
- Setup (dependencies) update.
0.8.0 (2025-04-28)
------------------
- Add support for Python 3.14
- Drop support for Python 3.9 (due to compatibility issues).
- Update readthedocs's python to version 3.13
- Update tox's base_python to version 3.13
- Setup (dependencies) update.
0.7.0 (2025-04-27)
------------------
- 100% code linting.
- Add support for PyPy 3.10 and 3.11
- Add/improve support for Python >= 3.9, <= 3.13
- Drop support for Python <= 3.8
- Drop support for Python 2.
- Copyright year update.
- | Tox configuration is now in native (toml) format (as part of
| pyproject.toml) and now based on tox >= 4.0.
- Setup update. Currently based on pyproject.toml.
- Source distribution (\*.tar.gz now) is compliant with PEP-0625.
- Creating a fork of Alex Sherman's *deco* package with a fix allowing
to work with Python3 or higher and newest versions of pip/setuptools.
- Minor improvements and cleanup..
Above are changes of the original (v.0.6.3) *deco*:
0.6.3 (2025-04-24)
------------------
- Initial commit.
Raw data
{
"_id": null,
"home_page": null,
"name": "deco3",
"maintainer": "Adam Karpierz",
"docs_url": null,
"requires_python": "<4.0.0,>=3.10.0",
"maintainer_email": "adam@karpierz.net",
"keywords": "deco, deco3, concurrency, multiprocessing, multitasking",
"author": "Alex Sherman, Adam Karpierz",
"author_email": "adam@karpierz.net",
"download_url": "https://files.pythonhosted.org/packages/b6/7c/a847c25f602652526a988bf82f48acec33fab113e156ba26a5d130808e2b/deco3-0.9.0.tar.gz",
"platform": "any",
"description": "deco3\r\n=====\r\n\r\nDecorated Concurrency.\r\n\r\nA simplified parallel computing model for Python. DECO automatically\r\nparallelizes Python programs, and requires minimal modifications to\r\nexisting serial programs.\r\n\r\nOverview\r\n========\r\n\r\n|package_bold| is a strict fork of Alex Sherman's `deco package <deco_>`_\r\nwith a fix allowing to work with Python3 or higher and with a little code\r\nreformatting and minor improvements.\r\n\r\n`PyPI record`_.\r\n\r\n`Documentation`_.\r\n\r\nOverview below is a copy from the original `deco website <deco_>`_\r\n(with only the necessary changes regarding |package|).\r\n\r\nDocumentation\r\n-------------\r\n\r\nYou can reference the `Wiki on Github <deco_wiki_>`_\r\nfor slightly more in-depth documentation.\r\n\r\nGeneral Usage\r\n-------------\r\n\r\nUsing DECO is as simple as finding, or creating, two functions in your\r\nPython program. The first function is the one we want to run in\r\nparallel, and is decorated with ``@concurrent``. The second function is\r\nthe function which calls the ``@concurrent`` function and is decorated\r\nwith ``@synchronized``. Decorating the second function is optional, but\r\nprovides some very cool benefits. Let's take a look at an example.\r\n\r\n.. code:: python\r\n\r\n @concurrent # We add this for the concurrent function\r\n def process_lat_lon(lat, lon, data):\r\n #Does some work which takes a while\r\n return result\r\n\r\n @synchronized # And we add this for the function which calls the concurrent function\r\n def process_data_set(data):\r\n results = defaultdict(dict)\r\n for lat in range(...):\r\n for lon in range(...):\r\n results[lat][lon] = process_lat_lon(lat, lon, data)\r\n return results\r\n\r\nThat's it, two lines of changes is all we need in order to parallelize\r\nthis program. Now this program will make use of all the cores on the\r\nmachine it's running on, allowing it to run significantly faster.\r\n\r\nWhat it does\r\n------------\r\n\r\n- The ``@concurrent`` decorator uses multiprocessing.pool to parallelize\r\n calls to the target function\r\n- Indexed based mutation of function arguments is handled automatically,\r\n which pool cannot do\r\n- The ``@synchronized`` decorator automatically inserts synchronization\r\n events\r\n- It also automatically refactors assignments of the results of\r\n ``@concurrent`` function calls to happen during synchronization events\r\n\r\nLimitations\r\n-----------\r\n\r\n- The ``@concurrent`` decorator will only speed up functions that take\r\n longer than ~1ms\r\n\r\n - If they take less time your code will run slower!\r\n\r\n- By default, ``@concurrent`` function arguments/return values must be\r\n pickleable for use with ``multiprocessing``\r\n- The ``@synchronized`` decorator only works on 'simple' functions, make\r\n sure the function meets the following criteria\r\n\r\n - Only calls, or assigns the result of ``@concurrent`` functions to\r\n indexable objects such as:\r\n\r\n - concurrent(...)\r\n - result[key] = concurrent(...)\r\n\r\n - Never indirectly reads objects that get assigned to by calls of the\r\n ``@concurrent`` function\r\n\r\nHow it works\r\n------------\r\n\r\nFor an in depth discussion of the mechanisms at work, we wrote a paper\r\nfor a class which `can be found here <decorated_concurrency_>`_.\r\n\r\nAs an overview, DECO is mainly just a smart wrapper for Python's\r\nmultiprocessing.pool. When ``@concurrent`` is applied to a function it\r\nreplaces it with calls to pool.apply_async. Additionally when arguments\r\nare passed to pool.apply_async, DECO replaces any index mutable objects\r\nwith proxies, allowing it to detect and synchronize mutations of these\r\nobjects. The results of these calls can then be obtained by calling\r\nwait() on the concurrent function, invoking a synchronization event.\r\nThese events can be placed automatically in your code by using the\r\n``@synchronized`` decorator on functions that call ``@concurrent``\r\nfunctions. Additionally while using ``@synchronized``, you can directly\r\nassign the result of concurrent function calls to index mutable objects.\r\nThese assignments get refactored by DECO to automatically occur during\r\nthe next synchronization event. All of this means that in many cases,\r\nparallel programming using DECO appears exactly the same as simpler\r\nserial programming.\r\n\r\nInstallation\r\n============\r\n\r\nPrerequisites:\r\n\r\n+ Python 3.10 or higher\r\n\r\n * https://www.python.org/\r\n\r\n+ pip and setuptools\r\n\r\n * https://pypi.org/project/pip/\r\n * https://pypi.org/project/setuptools/\r\n\r\nTo install run:\r\n\r\n .. parsed-literal::\r\n\r\n python -m pip install --upgrade |package|\r\n\r\nDevelopment\r\n===========\r\n\r\nPrerequisites:\r\n\r\n+ Development is strictly based on *tox*. To install it run::\r\n\r\n python -m pip install --upgrade tox\r\n\r\nVisit `Development page`_.\r\n\r\nInstallation from sources:\r\n\r\nclone the sources:\r\n\r\n .. parsed-literal::\r\n\r\n git clone |respository| |package|\r\n\r\nand run:\r\n\r\n .. parsed-literal::\r\n\r\n python -m pip install ./|package|\r\n\r\nor on development mode:\r\n\r\n .. parsed-literal::\r\n\r\n python -m pip install --editable ./|package|\r\n\r\nLicense\r\n=======\r\n\r\n | |copyright|\r\n | Copyright (c) 2016 Alex Sherman\r\n | Licensed under the MIT License\r\n | https://opensource.org/license/mit\r\n | Please refer to the accompanying LICENSE file.\r\n\r\nAuthors\r\n=======\r\n\r\n* Alex Sherman <asherman1024@gmail.com>\r\n* Adam Karpierz <adam@karpierz.net>\r\n\r\n.. |package| replace:: deco3\r\n.. |package_bold| replace:: **deco3**\r\n.. |copyright| replace:: Copyright (c) 2025-2025 Adam Karpierz\r\n.. |respository| replace:: https://github.com/karpierz/deco3.git\r\n.. _Development page: https://github.com/karpierz/deco3\r\n.. _PyPI record: https://pypi.org/project/deco3/\r\n.. _Documentation: https://deco3.readthedocs.io/\r\n.. _deco: https://pypi.org/project/deco/\r\n.. _deco_wiki: https://github.com/alex-sherman/deco/wiki\r\n.. _decorated_concurrency: _static/Decorated_Concurrency.pdf\r\n\r\nChangelog\r\n=========\r\n\r\n0.9.0 (2025-08-20)\r\n------------------\r\n- 100% code coverage.\r\n- Making the package typed.\r\n- General improvements and cleanup.\r\n- Setup (dependencies) update.\r\n\r\n0.8.3 (2025-06-11)\r\n------------------\r\n- Setup (dependencies) update.\r\n\r\n0.8.2 (2025-05-15)\r\n------------------\r\n- The distribution is now created using 'build' instead of 'setuptools'.\r\n- Setup (dependencies) update (due to regressions in tox and setuptools).\r\n\r\n0.8.1 (2025-05-04)\r\n------------------\r\n- Setup (dependencies) update.\r\n\r\n0.8.0 (2025-04-28)\r\n------------------\r\n- Add support for Python 3.14\r\n- Drop support for Python 3.9 (due to compatibility issues).\r\n- Update readthedocs's python to version 3.13\r\n- Update tox's base_python to version 3.13\r\n- Setup (dependencies) update.\r\n\r\n0.7.0 (2025-04-27)\r\n------------------\r\n- 100% code linting.\r\n- Add support for PyPy 3.10 and 3.11\r\n- Add/improve support for Python >= 3.9, <= 3.13\r\n- Drop support for Python <= 3.8\r\n- Drop support for Python 2.\r\n- Copyright year update.\r\n- | Tox configuration is now in native (toml) format (as part of\r\n | pyproject.toml) and now based on tox >= 4.0.\r\n- Setup update. Currently based on pyproject.toml.\r\n- Source distribution (\\*.tar.gz now) is compliant with PEP-0625.\r\n- Creating a fork of Alex Sherman's *deco* package with a fix allowing\r\n to work with Python3 or higher and newest versions of pip/setuptools.\r\n- Minor improvements and cleanup..\r\n\r\nAbove are changes of the original (v.0.6.3) *deco*:\r\n\r\n0.6.3 (2025-04-24)\r\n------------------\r\n- Initial commit.\r\n",
"bugtrack_url": null,
"license": null,
"summary": "A decorator for concurrency.",
"version": "0.9.0",
"project_urls": {
"Documentation": "https://deco3.readthedocs.io/",
"Download": "https://pypi.org/project/deco3/",
"Homepage": "https://pypi.org/project/deco3/",
"Issues": "https://github.com/karpierz/deco3/issues",
"Source": "https://github.com/karpierz/deco3"
},
"split_keywords": [
"deco",
" deco3",
" concurrency",
" multiprocessing",
" multitasking"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "c484b532de04c0c441d9ce2abe7b2bdf10f018025d240a9984edd3fd8e373130",
"md5": "5c5538ddcffe5e6795aa84543118e43f",
"sha256": "32a59546801b9b78efef6b85e331a9842e5d3080fd136298db18ba0b17df351a"
},
"downloads": -1,
"filename": "deco3-0.9.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5c5538ddcffe5e6795aa84543118e43f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0.0,>=3.10.0",
"size": 11392,
"upload_time": "2025-08-22T22:29:34",
"upload_time_iso_8601": "2025-08-22T22:29:34.273703Z",
"url": "https://files.pythonhosted.org/packages/c4/84/b532de04c0c441d9ce2abe7b2bdf10f018025d240a9984edd3fd8e373130/deco3-0.9.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "b67ca847c25f602652526a988bf82f48acec33fab113e156ba26a5d130808e2b",
"md5": "f75b0f9f30d7bd9719e9d6b5065b2406",
"sha256": "f13deee774cfefae5d96acafc6a69dd8e1f331d8004b7d26b3c824845efc8658"
},
"downloads": -1,
"filename": "deco3-0.9.0.tar.gz",
"has_sig": false,
"md5_digest": "f75b0f9f30d7bd9719e9d6b5065b2406",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0.0,>=3.10.0",
"size": 199719,
"upload_time": "2025-08-22T22:29:35",
"upload_time_iso_8601": "2025-08-22T22:29:35.911507Z",
"url": "https://files.pythonhosted.org/packages/b6/7c/a847c25f602652526a988bf82f48acec33fab113e156ba26a5d130808e2b/deco3-0.9.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-22 22:29:35",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "karpierz",
"github_project": "deco3",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "deco3"
}
JSON
