| Name | Jug JSON |
| Version |
2.3.1
JSON |
| download |
| home_page | https://jug.readthedocs.io |
| Summary | A Task Based Parallelization Framework |
| upload_time | 2023-11-05 04:19:29 |
| maintainer | |
| docs_url | https://pythonhosted.org/Jug/ |
| author | Luis Pedro Coelho |
| requires_python | |
| license | MIT |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
===========================================
Jug: A Task-Based Parallelization Framework
===========================================
Jug allows you to write code that is broken up into
tasks and run different tasks on different processors.
.. image:: https://github.com/luispedro/jug/actions/workflows/python-package.yml/badge.svg
:target: https://github.com/luispedro/jug/actions/workflows/python-package.yml
.. image:: https://zenodo.org/badge/205237.svg
:target: https://zenodo.org/badge/latestdoi/205237
.. image:: https://anaconda.org/conda-forge/jug/badges/installer/conda.svg
:target: https://anaconda.org/conda-forge/jug
.. image:: https://static.pepy.tech/personalized-badge/jug?period=total&units=international_system&left_color=black&right_color=blue&left_text=Downloads
:target: https://pepy.tech/project/jug
.. image:: https://img.shields.io/badge/CITATION-doi.org%2F10.5334%2Fjors.161-green.svg
:target: https://doi.org/10.5334/jors.161
It uses the filesystem to communicate between processes and
works correctly over NFS, so you can coordinate processes on
different machines.
Jug is a pure Python implementation and should work on any platform.
Python versions 3.5 and above are supported.
*Website*: `http://luispedro.org/software/jug <http://luispedro.org/software/jug>`__
*Documentation*: `https://jug.readthedocs.org/ <https://jug.readthedocs.org/>`__
*Video*: On `vimeo <http://vimeo.com/8972696>`__ or `showmedo
<http://showmedo.com/videotutorials/video?name=9750000;fromSeriesID=975>`__
*Mailing List*: `https://groups.google.com/group/jug-users
<https://groups.google.com/group/jug-users>`__
Testimonials
------------
"I've been using jug with great success to distribute the running of a
reasonably large set of parameter combinations" - Andreas Longva
Install
-------
You can install Jug with pip::
pip install Jug
or use, if you are using `conda <https://anaconda.org/>`__, you can install jug
from `conda-forge <https://conda-forge.github.io/>`__ using the following
commands::
conda config --add channels conda-forge
conda install jug
Citation
--------
If you use Jug to generate results for a scientific publication, please cite
Coelho, L.P., (2017). Jug: Software for Parallel Reproducible Computation in
Python. Journal of Open Research Software. 5(1), p.30.
https://doi.org/10.5334/jors.161
Short Example
-------------
Here is a one minute example. Save the following to a file called ``primes.py``
(if you have installed jug, you can obtain a slightly longer version of this
example by running ``jug demo`` on the command line)::
from jug import TaskGenerator
from time import sleep
@TaskGenerator
def is_prime(n):
sleep(1.)
for j in range(2,n-1):
if (n % j) == 0:
return False
return True
primes100 = [is_prime(n) for n in range(2,101)]
This is a brute-force way to find all the prime numbers up to 100. Of course,
this is only for didactical purposes, normally you would use a better method.
Similarly, the ``sleep`` function is so that it does not run too fast. Still,
it illustrates the basic functionality of Jug for embarassingly parallel
problems.
Type ``jug status primes.py`` to get::
Task name Waiting Ready Finished Running
----------------------------------------------------------------------
primes.is_prime 0 99 0 0
......................................................................
Total: 0 99 0 0
This tells you that you have 99 tasks called ``primes.is_prime`` ready to run.
So run ``jug execute primes.py &``. You can even run multiple instances in the
background (if you have multiple cores, for example). After starting 4
instances and waiting a few seconds, you can check the status again (with ``jug
status primes.py``)::
Task name Waiting Ready Finished Running
----------------------------------------------------------------------
primes.is_prime 0 63 32 4
......................................................................
Total: 0 63 32 4
Now you have 32 tasks finished, 4 running, and 63 still ready. Eventually, they
will all finish and you can inspect the results with ``jug shell primes.py``.
This will give you an ``ipython`` shell. The `primes100` variable is available,
but it is an ugly list of `jug.Task` objects. To get the actual value, you call
the `value` function::
In [1]: primes100 = value(primes100)
In [2]: primes100[:10]
Out[2]: [True, True, False, True, False, True, False, False, False, True]
What's New
----------
Version 2.3.1 (*5 November 2023*)
- Update for Python 3.12
Version 2.3.0 (*25 June 2023*)
- jug shell: Add ``get_filtered_tasks()``
- jug: Fix ``jug --version`` (which had been broken in the refactoring to use subcommands)
- jug shell: Fix message in jug shell when there are no dependencies (it would repeatedly print the message stating *this will only be run once*)
- jug pack: Make it much faster to invalidate elements
- file_store: ensure that the temporary directory exists
Version 2.2.3 (*26 May 2023*)
- Fix ``jug shell`` for newer versions of IPython
Version 2.2.2 (*19 July 2022*)
- Fix ``jug cleanup`` when packs are used (``jug pack``)
Version 2.2.1 (*19 May 2022*)
- Fix bug with ``jug cleanup`` and the redis backend (`#86 <https://github.com/luispedro/jug/issues/86>`__)
Version 2.2.0 (*3 May 2022*)
- Add ``jug pack`` subcommand
- Make ``get_tasks()`` return a copy of the tasks inside ``jug shell``
- Remove ``six`` dependency
Version 2.1.1 (*18 March 2021*)
- Include requirements files in distribution
Version 2.1.0 (*18 March 2021*)
- Improvements to webstatus (by Robert Denham)
- Removed Python 2.7 support
- Fix output encoding for Python 3.8
- Fix bug mixing ``mapreduce()`` & ``status --cache``
- Make block_access (used in ``mapreduce()``) much faster (20x)
- Fix important redis bug
- More precise output in ``cleanup`` command
Version 2.0.2 (Thu Jun 11 2020)
- Fix command line argument parsing
Version 2.0.1 (Thu Jun 11 2020)
- Fix handling of ``JUG_EXIT_IF_FILE_EXISTS`` environmental variable
- Fix passing an argument to ``jug.main()`` function
- Extend ``--pdb`` to exceptions raised while importing the jugfile (issue #79)
version **2.0.0** (Fri Feb 21 2020)
- jug.backend.base_store has 1 new method 'listlocks'
- jug.backend.base_lock has 2 new methods 'fail' and 'is_failed'
- Add 'jug execute --keep-failed' to preserve locks on failing tasks.
- Add 'jug cleanup --failed-only' to remove locks from failed tasks
- 'jug status' and 'jug graph' now display failed tasks
- Check environmental exit variables by default (suggested by Renato Alves, issue #66)
- Fix 'jug sleep-until' in the presence of barrier() (issue #71)
For older version see ``ChangeLog`` file or the `full history
<https://jug.readthedocs.io/en/latest/history.html>`__.
Raw data
{
"_id": null,
"home_page": "https://jug.readthedocs.io",
"name": "Jug",
"maintainer": "",
"docs_url": "https://pythonhosted.org/Jug/",
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Luis Pedro Coelho",
"author_email": "luis@luispedro.org",
"download_url": "https://files.pythonhosted.org/packages/51/e0/c5116694d10e5168b7290495d38895b4ee03b26b1cd877fe65f51e725d06/Jug-2.3.1.tar.gz",
"platform": "Any",
"description": "===========================================\nJug: A Task-Based Parallelization Framework\n===========================================\n\nJug allows you to write code that is broken up into\ntasks and run different tasks on different processors.\n\n.. image:: https://github.com/luispedro/jug/actions/workflows/python-package.yml/badge.svg\n :target: https://github.com/luispedro/jug/actions/workflows/python-package.yml\n\n.. image:: https://zenodo.org/badge/205237.svg\n :target: https://zenodo.org/badge/latestdoi/205237\n\n.. image:: https://anaconda.org/conda-forge/jug/badges/installer/conda.svg\n :target: https://anaconda.org/conda-forge/jug\n\n.. image:: https://static.pepy.tech/personalized-badge/jug?period=total&units=international_system&left_color=black&right_color=blue&left_text=Downloads\n :target: https://pepy.tech/project/jug\n\n.. image:: https://img.shields.io/badge/CITATION-doi.org%2F10.5334%2Fjors.161-green.svg\n :target: https://doi.org/10.5334/jors.161\n\n\nIt uses the filesystem to communicate between processes and\nworks correctly over NFS, so you can coordinate processes on\ndifferent machines.\n\nJug is a pure Python implementation and should work on any platform.\n\nPython versions 3.5 and above are supported.\n\n*Website*: `http://luispedro.org/software/jug <http://luispedro.org/software/jug>`__\n\n*Documentation*: `https://jug.readthedocs.org/ <https://jug.readthedocs.org/>`__\n\n*Video*: On `vimeo <http://vimeo.com/8972696>`__ or `showmedo\n<http://showmedo.com/videotutorials/video?name=9750000;fromSeriesID=975>`__\n\n*Mailing List*: `https://groups.google.com/group/jug-users\n<https://groups.google.com/group/jug-users>`__\n\nTestimonials\n------------\n\n\"I've been using jug with great success to distribute the running of a\nreasonably large set of parameter combinations\" - Andreas Longva\n\n\nInstall\n-------\n\nYou can install Jug with pip::\n\n pip install Jug\n\nor use, if you are using `conda <https://anaconda.org/>`__, you can install jug\nfrom `conda-forge <https://conda-forge.github.io/>`__ using the following\ncommands::\n\n conda config --add channels conda-forge\n conda install jug\n\nCitation\n--------\n\nIf you use Jug to generate results for a scientific publication, please cite\n\n Coelho, L.P., (2017). Jug: Software for Parallel Reproducible Computation in\n Python. Journal of Open Research Software. 5(1), p.30.\n\n https://doi.org/10.5334/jors.161\n\n\nShort Example\n-------------\n\nHere is a one minute example. Save the following to a file called ``primes.py``\n(if you have installed jug, you can obtain a slightly longer version of this\nexample by running ``jug demo`` on the command line)::\n\n from jug import TaskGenerator\n from time import sleep\n\n @TaskGenerator\n def is_prime(n):\n sleep(1.)\n for j in range(2,n-1):\n if (n % j) == 0:\n return False\n return True\n\n primes100 = [is_prime(n) for n in range(2,101)]\n\nThis is a brute-force way to find all the prime numbers up to 100. Of course,\nthis is only for didactical purposes, normally you would use a better method.\nSimilarly, the ``sleep`` function is so that it does not run too fast. Still,\nit illustrates the basic functionality of Jug for embarassingly parallel\nproblems.\n\nType ``jug status primes.py`` to get::\n\n Task name Waiting Ready Finished Running\n ----------------------------------------------------------------------\n primes.is_prime 0 99 0 0\n ......................................................................\n Total: 0 99 0 0\n\n\nThis tells you that you have 99 tasks called ``primes.is_prime`` ready to run.\nSo run ``jug execute primes.py &``. You can even run multiple instances in the\nbackground (if you have multiple cores, for example). After starting 4\ninstances and waiting a few seconds, you can check the status again (with ``jug\nstatus primes.py``)::\n\n Task name Waiting Ready Finished Running\n ----------------------------------------------------------------------\n primes.is_prime 0 63 32 4\n ......................................................................\n Total: 0 63 32 4\n\n\nNow you have 32 tasks finished, 4 running, and 63 still ready. Eventually, they\nwill all finish and you can inspect the results with ``jug shell primes.py``.\nThis will give you an ``ipython`` shell. The `primes100` variable is available,\nbut it is an ugly list of `jug.Task` objects. To get the actual value, you call\nthe `value` function::\n\n In [1]: primes100 = value(primes100)\n\n In [2]: primes100[:10]\n Out[2]: [True, True, False, True, False, True, False, False, False, True]\n\nWhat's New\n----------\n\nVersion 2.3.1 (*5 November 2023*)\n\n- Update for Python 3.12\n\nVersion 2.3.0 (*25 June 2023*)\n\n- jug shell: Add ``get_filtered_tasks()``\n- jug: Fix ``jug --version`` (which had been broken in the refactoring to use subcommands)\n- jug shell: Fix message in jug shell when there are no dependencies (it would repeatedly print the message stating *this will only be run once*)\n- jug pack: Make it much faster to invalidate elements\n- file_store: ensure that the temporary directory exists\n\nVersion 2.2.3 (*26 May 2023*)\n- Fix ``jug shell`` for newer versions of IPython\n\nVersion 2.2.2 (*19 July 2022*)\n- Fix ``jug cleanup`` when packs are used (``jug pack``)\n\nVersion 2.2.1 (*19 May 2022*)\n- Fix bug with ``jug cleanup`` and the redis backend (`#86 <https://github.com/luispedro/jug/issues/86>`__)\n\nVersion 2.2.0 (*3 May 2022*)\n\n- Add ``jug pack`` subcommand\n- Make ``get_tasks()`` return a copy of the tasks inside ``jug shell``\n- Remove ``six`` dependency\n\nVersion 2.1.1 (*18 March 2021*)\n\n- Include requirements files in distribution\n\nVersion 2.1.0 (*18 March 2021*)\n\n- Improvements to webstatus (by Robert Denham)\n- Removed Python 2.7 support\n- Fix output encoding for Python 3.8\n- Fix bug mixing ``mapreduce()`` & ``status --cache``\n- Make block_access (used in ``mapreduce()``) much faster (20x)\n- Fix important redis bug\n- More precise output in ``cleanup`` command\n\nVersion 2.0.2 (Thu Jun 11 2020)\n\n- Fix command line argument parsing\n\nVersion 2.0.1 (Thu Jun 11 2020)\n\n- Fix handling of ``JUG_EXIT_IF_FILE_EXISTS`` environmental variable\n- Fix passing an argument to ``jug.main()`` function\n- Extend ``--pdb`` to exceptions raised while importing the jugfile (issue #79)\n\nversion **2.0.0** (Fri Feb 21 2020)\n\n- jug.backend.base_store has 1 new method 'listlocks'\n- jug.backend.base_lock has 2 new methods 'fail' and 'is_failed'\n- Add 'jug execute --keep-failed' to preserve locks on failing tasks.\n- Add 'jug cleanup --failed-only' to remove locks from failed tasks\n- 'jug status' and 'jug graph' now display failed tasks\n- Check environmental exit variables by default (suggested by Renato Alves, issue #66)\n- Fix 'jug sleep-until' in the presence of barrier() (issue #71)\n\nFor older version see ``ChangeLog`` file or the `full history\n<https://jug.readthedocs.io/en/latest/history.html>`__.\n\n\n\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A Task Based Parallelization Framework",
"version": "2.3.1",
"project_urls": {
"Homepage": "https://jug.readthedocs.io"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3e3a0a06f2eb9aff7faf02e10f0c85347523ce6ca520cad08f5a5bb93a609d38",
"md5": "0af9d8b60132a66dc8724d36273cb938",
"sha256": "e42375802429a183fe8b9d6e1fcd920b9d94b7c71f0f37e1bc5dcf8575eb4f22"
},
"downloads": -1,
"filename": "Jug-2.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0af9d8b60132a66dc8724d36273cb938",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 116097,
"upload_time": "2023-11-05T04:19:27",
"upload_time_iso_8601": "2023-11-05T04:19:27.025368Z",
"url": "https://files.pythonhosted.org/packages/3e/3a/0a06f2eb9aff7faf02e10f0c85347523ce6ca520cad08f5a5bb93a609d38/Jug-2.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "51e0c5116694d10e5168b7290495d38895b4ee03b26b1cd877fe65f51e725d06",
"md5": "aee88d356a0024271867ae199af7ddd4",
"sha256": "6364d6a898bb1a3996505a5ed5bd79d0d830470f552a10a4e44a0de4d0dc3d75"
},
"downloads": -1,
"filename": "Jug-2.3.1.tar.gz",
"has_sig": false,
"md5_digest": "aee88d356a0024271867ae199af7ddd4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 69720,
"upload_time": "2023-11-05T04:19:29",
"upload_time_iso_8601": "2023-11-05T04:19:29.548776Z",
"url": "https://files.pythonhosted.org/packages/51/e0/c5116694d10e5168b7290495d38895b4ee03b26b1cd877fe65f51e725d06/Jug-2.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-05 04:19:29",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "jug"
}
