<!--
Copyright (c) 2023 Ernst Strüngmann Institute (ESI) for Neuroscience
in Cooperation with Max Planck Society
SPDX-License-Identifier: CC-BY-NC-SA-1.0
-->
# ACME: Asynchronous Computing Made ESI
[](https://anaconda.org/conda-forge/esi-acme)
[](https://badge.fury.io/py/esi-acme)
[](https://github.com/esi-neuroscience/acme/blob/main/LICENSE)
[](https://github.dev/esi-neuroscience/acme)
[](https://bestpractices.coreinfrastructure.org/projects/7144)
main: [](https://github.com/esi-neuroscience/acme/actions/workflows/tests_workflow.yml)
[](https://codecov.io/gh/esi-neuroscience/acme)
dev: [](https://github.com/esi-neuroscience/acme/actions/workflows/tests_workflow.yml)
[](https://codecov.io/gh/esi-neuroscience/acme)
## Table of Contents
1. [Summary](#summary)
1. [Installation](#installation)
1. [Usage](#usage)
- [Basic Examples](#basic-examples)
- [Intermediate Examples](#intermediate-examples)
- [Advanced Use](#advanced-use)
1. [Handling Results](#handling-results)
- [Load Results From Files](#load-results-from-files)
- [Collect Results in Single HDF5 Dataset](#collect-results-in-single-hdf5-dataset)
- [Collect Results in Local Memory](#collect-results-in-local-memory)
1. [Debugging](#debugging)
1. [Documentation and Contact](#documentation-and-contact)
- [Resources](#resources)
## Summary
The objective of ACME (pronounced *"ak-mee"*) is to provide easy-to-use
wrappers for calling Python functions concurrently ("embarassingly parallel workloads").
ACME is developed at the
[Ernst Strüngmann Institute (ESI) gGmbH for Neuroscience in Cooperation with Max Planck Society](https://www.esi-frankfurt.de/>)
and released free of charge under the
[BSD 3-Clause "New" or "Revised" License](https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22)).
ACME relies heavily on the concurrent processing library [dask](https://docs.dask.org/en/latest/>)
and was primarily designed to facilitate the use of [SLURM](https://slurm.schedmd.com/documentation.html)
on the ESI HPC cluster (although other HPC infrastructure running SLURM can be
leveraged as well). Local multi-processing hardware (i.e., multi-core CPUs)
is fully supported too. ACME is itself used as the parallelization engine of [SyNCoPy](http://www.syncopy.org/).
## Installation
ACME can be installed with `pip`
```shell
pip install esi-acme
```
or via `conda`
```shell
conda install -c conda-forge esi-acme
```
To get the latest development version, simply clone our GitHub repository:
```shell
git clone https://github.com/esi-neuroscience/acme.git
cd acme/
pip install -e .
```
## Usage
### Basic Examples
Simplest use, everything is done automatically.
```python
from acme import ParallelMap
def f(x, y, z=3):
return (x + y) * z
with ParallelMap(f, [2, 4, 6, 8], 4) as pmap:
pmap.compute()
```
See also our [Quickstart Guide](https://esi-acme.readthedocs.io/en/latest/quickstart.html).
### Intermediate Examples
Set number of function calls via `n_inputs`
```python
import numpy as np
from acme import ParallelMap
def f(x, y, z=3, w=np.zeros((3, 1)), **kwargs):
return (sum(x) + y) * z * w.max()
pmap = ParallelMap(f, [2, 4, 6, 8], [2, 2], z=np.array([1, 2]), w=np.ones((8, 1)), n_inputs=2)
with pmap as p:
p.compute()
```
More details in
[Override Automatic Input Argument Distribution](https://esi-acme.readthedocs.io/en/latest/userguide.html#override-automatic-input-argument-distribution)
### Advanced Use
Allocate custom `client` object and recycle it for several computations
(use `slurm_cluster_setup` on non-ESI HPC infrastructure or `local_cluster_setup`
when working on your local machine)
```python
import numpy as np
from acme import ParallelMap, esi_cluster_setup
def f(x, y, z=3, w=np.zeros((3, 1)), **kwargs):
return (sum(x) + y) * z * w.max()
def g(x, y, z=3, w=np.zeros((3, 1)), **kwargs):
return (max(x) + y) * z * w.sum()
n_workers = 200
client = esi_cluster_setup(partition="8GBXS", n_workers=n_workers)
x = [2, 4, 6, 8]
z = range(n_workers)
w = np.ones((8, 1))
pmap = ParallelMap(f, x, np.random.rand(n_workers), z=z, w=w, n_inputs=n_workers)
with pmap as p:
p.compute()
pmap = ParallelMap(g, x, np.random.rand(n_workers), z=z, w=w, n_inputs=n_workers)
with pmap as p:
p.compute()
```
For more information see [Reuse Worker Clients](https://esi-acme.readthedocs.io/en/latest/userguide.html#reuse-worker-clients)
## Handling Results
### Load Results From Files
By default, results are saved to disk in HDF5 format and can be accessed using
the `results_container` attribute of `ParallelMap`:
```python
def f(x, y, z=3):
return (x + y) * z
with ParallelMap(f, [2, 4, 6, 8], 4) as pmap:
filenames = pmap.compute()
```
Example loading code:
```python
import h5py
import numpy as np
out = np.zeros((4,))
with h5py.File(pmap.results_container, "r") as h5f:
for k, key in enumerate(h5f.keys()):
out[k] = h5f[key]["result_0"][()]
```
See also [Where Are My Results?](https://esi-acme.readthedocs.io/en/latest/userguide.html#where-are-my-results)
### Collect Results in Single HDF5 Dataset
If possible, results can be slotted into a single HDF5 dataset:
```python
def f(x, y, z=3):
return (x + y) * z
with ParallelMap(f, [2, 4, 6, 8], 4, result_shape=(None,)) as pmap:
pmap.compute()
```
Example loading code:
```python
import h5py
with h5py.File(pmap.results_container, "r") as h5f:
out = h5f["result_0"][()] # returns a NumPy array of shape (4,)
```
More examples can be found in
[Collect Results in Single Dataset](https://esi-acme.readthedocs.io/en/latest/userguide.html#collect-results-in-single-dataset)
### Collect Results in Local Memory
This is possible but not recommended.
```python
def f(x, y, z=3):
return (x + y) * z
with ParallelMap(f, [2, 4, 6, 8], 4, write_worker_results=False) as pmap:
result = pmap.compute() # returns a 4-element list
```
Alternatively, create an in-memory NumPy array
```python
with ParallelMap(f, [2, 4, 6, 8], 4, write_worker_results=False, result_shape=(None,)) as pmap:
result = pmap.compute() # returns a NumPy array of shape (4,)
```
## Debugging
Use the `debug` keyword to perform all function calls in the local thread of
the active Python interpreter
```python
def f(x, y, z=3):
return (x + y) * z
with ParallelMap(f, [2, 4, 6, 8], 4, z=None) as pmap:
results = pmap.compute(debug=True)
```
This way tools like `pdb` or ``%debug`` IPython magics can be used.
More information can be found in the [FAQ](https://esi-acme.readthedocs.io/en/latest/troubleshooting_faq.html).
## Documentation and Contact
To report bugs or ask questions please use our
[GitHub issue tracker](https://github.com/esi-neuroscience/acme/issues).
More usage details and background information is available in our
[online documentation](https://esi-acme.readthedocs.io).
### Resources
- [ACME Presentation](https://pantaray.github.io/RSE23-talk/)
at [deRSE23 - Conference for Research Software Engineering in Germany](https://de-rse23.sciencesconf.org/)
- [ACME Demo](https://esi-neuroscience.github.io/acme-demo/) presented at the 4th annual Data Scientist Community Meeting
- [ACME Tutorials](https://esi-acme.readthedocs.io/en/latest/tutorials.html)
- [ACME FAQ](https://esi-acme.readthedocs.io/en/latest/troubleshooting_faq.html)
Raw data
{
"_id": null,
"home_page": "https://esi-acme.readthedocs.io/en/latest/",
"name": "esi-acme",
"maintainer": "",
"docs_url": null,
"requires_python": "<3.13,>=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Ernst Str\u00fcngmann Institute (ESI) for Neuroscience in Cooperation with Max Planck Society",
"author_email": "it@esi-frankfurt.de",
"download_url": "https://files.pythonhosted.org/packages/81/fc/1a4d607ad58eb0f81aca7d1d4526a8ab2dc674157ba8cac7c1df7532d240/esi-acme-2023.12.tar.gz",
"platform": null,
"description": "<!--\nCopyright (c) 2023 Ernst Str\u00fcngmann Institute (ESI) for Neuroscience\nin Cooperation with Max Planck Society\nSPDX-License-Identifier: CC-BY-NC-SA-1.0\n-->\n\n\n\n# ACME: Asynchronous Computing Made ESI\n\n[](https://anaconda.org/conda-forge/esi-acme)\n[](https://badge.fury.io/py/esi-acme)\n[](https://github.com/esi-neuroscience/acme/blob/main/LICENSE)\n[](https://github.dev/esi-neuroscience/acme)\n[](https://bestpractices.coreinfrastructure.org/projects/7144)\n\nmain: [](https://github.com/esi-neuroscience/acme/actions/workflows/tests_workflow.yml)\n[](https://codecov.io/gh/esi-neuroscience/acme)\n\ndev: [](https://github.com/esi-neuroscience/acme/actions/workflows/tests_workflow.yml)\n[](https://codecov.io/gh/esi-neuroscience/acme)\n\n## Table of Contents\n\n1. [Summary](#summary)\n1. [Installation](#installation)\n1. [Usage](#usage)\n - [Basic Examples](#basic-examples)\n - [Intermediate Examples](#intermediate-examples)\n - [Advanced Use](#advanced-use)\n1. [Handling Results](#handling-results)\n - [Load Results From Files](#load-results-from-files)\n - [Collect Results in Single HDF5 Dataset](#collect-results-in-single-hdf5-dataset)\n - [Collect Results in Local Memory](#collect-results-in-local-memory)\n1. [Debugging](#debugging)\n1. [Documentation and Contact](#documentation-and-contact)\n - [Resources](#resources)\n\n## Summary\n\nThe objective of ACME (pronounced *\"ak-mee\"*) is to provide easy-to-use\nwrappers for calling Python functions concurrently (\"embarassingly parallel workloads\").\nACME is developed at the\n[Ernst Str\u00fcngmann Institute (ESI) gGmbH for Neuroscience in Cooperation with Max Planck Society](https://www.esi-frankfurt.de/>)\nand released free of charge under the\n[BSD 3-Clause \"New\" or \"Revised\" License](https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22)).\nACME relies heavily on the concurrent processing library [dask](https://docs.dask.org/en/latest/>)\nand was primarily designed to facilitate the use of [SLURM](https://slurm.schedmd.com/documentation.html)\non the ESI HPC cluster (although other HPC infrastructure running SLURM can be\nleveraged as well). Local multi-processing hardware (i.e., multi-core CPUs)\nis fully supported too. ACME is itself used as the parallelization engine of [SyNCoPy](http://www.syncopy.org/).\n\n\n\n## Installation\n\nACME can be installed with `pip`\n\n```shell\npip install esi-acme\n```\n\nor via `conda`\n\n```shell\nconda install -c conda-forge esi-acme\n```\n\nTo get the latest development version, simply clone our GitHub repository:\n\n```shell\ngit clone https://github.com/esi-neuroscience/acme.git\ncd acme/\npip install -e .\n```\n\n## Usage\n\n### Basic Examples\n\nSimplest use, everything is done automatically.\n\n```python\nfrom acme import ParallelMap\n\ndef f(x, y, z=3):\n return (x + y) * z\n\nwith ParallelMap(f, [2, 4, 6, 8], 4) as pmap:\n pmap.compute()\n```\n\nSee also our [Quickstart Guide](https://esi-acme.readthedocs.io/en/latest/quickstart.html).\n\n### Intermediate Examples\n\nSet number of function calls via `n_inputs`\n\n```python\nimport numpy as np\nfrom acme import ParallelMap\n\ndef f(x, y, z=3, w=np.zeros((3, 1)), **kwargs):\n return (sum(x) + y) * z * w.max()\n\npmap = ParallelMap(f, [2, 4, 6, 8], [2, 2], z=np.array([1, 2]), w=np.ones((8, 1)), n_inputs=2)\n\nwith pmap as p:\n p.compute()\n```\n\nMore details in\n[Override Automatic Input Argument Distribution](https://esi-acme.readthedocs.io/en/latest/userguide.html#override-automatic-input-argument-distribution)\n\n### Advanced Use\n\nAllocate custom `client` object and recycle it for several computations\n(use `slurm_cluster_setup` on non-ESI HPC infrastructure or `local_cluster_setup`\nwhen working on your local machine)\n\n```python\nimport numpy as np\nfrom acme import ParallelMap, esi_cluster_setup\n\ndef f(x, y, z=3, w=np.zeros((3, 1)), **kwargs):\n return (sum(x) + y) * z * w.max()\n\ndef g(x, y, z=3, w=np.zeros((3, 1)), **kwargs):\n return (max(x) + y) * z * w.sum()\n\nn_workers = 200\nclient = esi_cluster_setup(partition=\"8GBXS\", n_workers=n_workers)\n\nx = [2, 4, 6, 8]\nz = range(n_workers)\nw = np.ones((8, 1))\n\npmap = ParallelMap(f, x, np.random.rand(n_workers), z=z, w=w, n_inputs=n_workers)\nwith pmap as p:\n p.compute()\n\npmap = ParallelMap(g, x, np.random.rand(n_workers), z=z, w=w, n_inputs=n_workers)\nwith pmap as p:\n p.compute()\n```\n\nFor more information see [Reuse Worker Clients](https://esi-acme.readthedocs.io/en/latest/userguide.html#reuse-worker-clients)\n\n## Handling Results\n\n### Load Results From Files\n\nBy default, results are saved to disk in HDF5 format and can be accessed using\nthe `results_container` attribute of `ParallelMap`:\n\n```python\ndef f(x, y, z=3):\n return (x + y) * z\n\nwith ParallelMap(f, [2, 4, 6, 8], 4) as pmap:\n filenames = pmap.compute()\n```\n\nExample loading code:\n\n```python\nimport h5py\nimport numpy as np\nout = np.zeros((4,))\n\nwith h5py.File(pmap.results_container, \"r\") as h5f:\n for k, key in enumerate(h5f.keys()):\n out[k] = h5f[key][\"result_0\"][()]\n```\n\nSee also [Where Are My Results?](https://esi-acme.readthedocs.io/en/latest/userguide.html#where-are-my-results)\n\n### Collect Results in Single HDF5 Dataset\n\nIf possible, results can be slotted into a single HDF5 dataset:\n\n```python\ndef f(x, y, z=3):\n return (x + y) * z\n\nwith ParallelMap(f, [2, 4, 6, 8], 4, result_shape=(None,)) as pmap:\n pmap.compute()\n```\n\nExample loading code:\n\n```python\nimport h5py\n\nwith h5py.File(pmap.results_container, \"r\") as h5f:\n out = h5f[\"result_0\"][()] # returns a NumPy array of shape (4,)\n```\n\nMore examples can be found in\n[Collect Results in Single Dataset](https://esi-acme.readthedocs.io/en/latest/userguide.html#collect-results-in-single-dataset)\n\n### Collect Results in Local Memory\n\nThis is possible but not recommended.\n\n```python\ndef f(x, y, z=3):\n return (x + y) * z\n\nwith ParallelMap(f, [2, 4, 6, 8], 4, write_worker_results=False) as pmap:\n result = pmap.compute() # returns a 4-element list\n```\n\nAlternatively, create an in-memory NumPy array\n\n```python\nwith ParallelMap(f, [2, 4, 6, 8], 4, write_worker_results=False, result_shape=(None,)) as pmap:\n result = pmap.compute() # returns a NumPy array of shape (4,)\n```\n\n## Debugging\n\nUse the `debug` keyword to perform all function calls in the local thread of\nthe active Python interpreter\n\n```python\ndef f(x, y, z=3):\n return (x + y) * z\n\nwith ParallelMap(f, [2, 4, 6, 8], 4, z=None) as pmap:\n results = pmap.compute(debug=True)\n```\n\nThis way tools like `pdb` or ``%debug`` IPython magics can be used.\nMore information can be found in the [FAQ](https://esi-acme.readthedocs.io/en/latest/troubleshooting_faq.html).\n\n## Documentation and Contact\n\nTo report bugs or ask questions please use our\n[GitHub issue tracker](https://github.com/esi-neuroscience/acme/issues).\nMore usage details and background information is available in our\n[online documentation](https://esi-acme.readthedocs.io).\n\n### Resources\n\n- [ACME Presentation](https://pantaray.github.io/RSE23-talk/)\n at [deRSE23 - Conference for Research Software Engineering in Germany](https://de-rse23.sciencesconf.org/)\n- [ACME Demo](https://esi-neuroscience.github.io/acme-demo/) presented at the 4th annual Data Scientist Community Meeting\n- [ACME Tutorials](https://esi-acme.readthedocs.io/en/latest/tutorials.html)\n- [ACME FAQ](https://esi-acme.readthedocs.io/en/latest/troubleshooting_faq.html)\n",
"bugtrack_url": null,
"license": "BSD-3",
"summary": "Asynchronous Computing Made ESI",
"version": "2023.12",
"project_urls": {
"Bug Tracker": "https://github.com/esi-neuroscience/acme/issues",
"Documentation": "https://esi-acme.readthedocs.io/en/latest/",
"Homepage": "https://esi-acme.readthedocs.io/en/latest/",
"Source Code": "https://github.com/esi-neuroscience/acme"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "92f0087d9a18b6e0cc9bad2106cb78a10565ab681742aaa2a8ae74c84471ea76",
"md5": "97d306ec0c854fdd3925b12f72ab19e7",
"sha256": "f1f7ca0b58b8a1f9f7b28ed4aedd082863645b7be607e39912ff68e6706d1717"
},
"downloads": -1,
"filename": "esi_acme-2023.12-py3-none-any.whl",
"has_sig": false,
"md5_digest": "97d306ec0c854fdd3925b12f72ab19e7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.7",
"size": 71485,
"upload_time": "2023-12-07T15:24:34",
"upload_time_iso_8601": "2023-12-07T15:24:34.692992Z",
"url": "https://files.pythonhosted.org/packages/92/f0/087d9a18b6e0cc9bad2106cb78a10565ab681742aaa2a8ae74c84471ea76/esi_acme-2023.12-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "81fc1a4d607ad58eb0f81aca7d1d4526a8ab2dc674157ba8cac7c1df7532d240",
"md5": "ef7fa7d6fddc19100d2da1889594e878",
"sha256": "10f140ff169b96565cbb824982123e71df707792e37e0329e2acdeeb3171a1d8"
},
"downloads": -1,
"filename": "esi-acme-2023.12.tar.gz",
"has_sig": false,
"md5_digest": "ef7fa7d6fddc19100d2da1889594e878",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.7",
"size": 61056,
"upload_time": "2023-12-07T15:24:37",
"upload_time_iso_8601": "2023-12-07T15:24:37.092660Z",
"url": "https://files.pythonhosted.org/packages/81/fc/1a4d607ad58eb0f81aca7d1d4526a8ab2dc674157ba8cac7c1df7532d240/esi-acme-2023.12.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-07 15:24:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "esi-neuroscience",
"github_project": "acme",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "esi-acme"
}
JSON
