# shared-ndarray2 <!-- omit in toc -->
`SharedNDArray` encapsulates a NumPy `ndarray` interface for using shared memory in
multiprocessing, using `multiprocessing.shared_memory` in Python 3.8+.
- [Quick Start](#quick-start)
- [Requirements](#requirements)
- [Similar Projects](#similar-projects)
- [Usage](#usage)
- [Creation](#creation)
- [`SharedNDArray()`](#sharedndarray)
- [`SharedNDArray.from_shape()` or `shared_ndarray.from_shape()`](#sharedndarrayfrom_shape-or-shared_ndarrayfrom_shape)
- [`SharedNDArray.from_array()` or `shared_ndarray.from_array()`](#sharedndarrayfrom_array-or-shared_ndarrayfrom_array)
- [Using like `np.ndarray`](#using-like-npndarray)
- [Releasing Shared Memory](#releasing-shared-memory)
- [`.lock` attribute](#lock-attribute)
- [Typed SharedNDArray](#typed-sharedndarray)
## Quick Start
```python
import multiprocessing
from multiprocessing.managers import SharedMemoryManager
import numpy as np
from shared_ndarray2 import SharedNDArray
def process_data(arr: SharedNDArray):
# Work with data
arr[:] += 1
with SharedMemoryManager() as mem_mgr:
arr = SharedNDArray.from_array(mem_mgr, np.arange(1024))
p = multiprocessing.Process(target=process_data, args=(arr,))
p.start()
p.join()
assert np.all(arr[:] == np.arange(1, 1025))
```
## Requirements
- Python 3.8+
- NumPy 1.21+
## Similar Projects
- [SharedArray](https://pypi.org/project/SharedArray/) - POSIX-only. Quite a different
paradigm, uses pre-Python 3.8 memory-sharing constructs, requires building a C module
with `gcc`.
- [shared-ndarray](https://pypi.org/project/shared-ndarray/) - POSIX-only. Similar (uses
NumPy ndarray `buffer` arg), uses pre-Python 3.8 memory-sharing constructs (requires
`posix_ipc`).
## Usage
### Creation
There are three methods for constructing a `SharedNDArray`.
#### `SharedNDArray()`
To create a `SharedNDArray` object from existing shared memory that represents a NumPy
array, use the regular constructor providing _shape_ and _dtype_, either with an existing
`multiprocessing.SharedMemory` object or the name of one:
```python
shm = SharedMemory(create=True, size=1024)
arr = SharedNDArray(shm, (1024,), np.uint8)
# -or-
arr = SharedNDArray(shm.name, (1024,), np.uint8)
```
#### `SharedNDArray.from_shape()` or `shared_ndarray.from_shape()`
This method allocates shared memory managed by a SharedMemoryManager to represent a NumPy
`ndarray` with some _shape_ and _dtype_.
```python
with SharedMemoryManager as mem_mgr:
arr = SharedNDArray.from_shape(mem_mgr, (3, 1024, 1024), dtype=np.uint16)
# ... Use arr with e.g. multiprocessing.Pool or multiprocess.Process
# ... Be sure process instances join/terminate before exiting SharedMemoryManager context manager
```
<div style="font-size: small; margin-left: 4em">
Note: _`shared_ndarray.from_shape()` is a standlone function and correctly types the
`SharedNDArray`, whereas the classmethod might (in mypy) or might not (in pyright)_
</div>
#### `SharedNDArray.from_array()` or `shared_ndarray.from_array()`
This method allocates shared memory managed by a SharedMemoryManager to represent a some
provided NumPy `ndarray` and copies that ndarray into the shared memory
```python
x = np.arange(100.0).reshape(2, 2, 25)
with SharedMemoryManager as mem_mgr:
arr = SharedNDArray.from_array(mem_mgr, x)
assert np.all(arr[:] == x[:])
# ... Use arr as above...
```
<div style="font-size: small; margin-left: 4em">
Note: _`shared_ndarray.from_array()` is a standlone function and correctly types the
`SharedNDArray`, whereas the classmethod might (in mypy) or might not (in pyright)_
</div>
### Using like `np.ndarray`
The point of `SharedNDArray` is to remove the boilerplate of creating shared memory,
passing around shapes and dtypes and reconstructing `np.ndarray` objects. `SharedNDArray`
does this last step with its `.get()` method, which creates a `np.ndarray` on-the-fly
using the shared memory buffer. The `__getitem__()` and `__setitem__()` methods use the
`.get()` method to get the np.ndarray to access the data, so multi-dimensional indexing
and slicing work the same as with an `ndarray`. Other `np.ndarray` methods are not
directly implemented but may be accessed by first calling `.get()`, e.g.
`arr.get().mean()`.
### Releasing Shared Memory
`SharedNDArray` implements a `__del__()` method that calls the
[`.close()`](https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory.close)
method on the `SharedMemory` when the instance is destroyed (i.e. at process exit). When
the shared memory is unlinked in the parent process (either manually with
[`shm.unlink()`](https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory.unlink)
or by exiting a `SharedMemoryManager` context manager) the shared_memory is properly
released. However if a sub-process is not joined or terminated before the shared memory is
unlinked a warning will be emitted about "`leaked shared_memory objects to clean up at
shutdown`".
### `.lock` attribute
The `__init__()`, `from_shape()`, and `from_array()` methods may be given a `lock=True`
argument that will also create a `multiprocessing.Lock` object and include it in the
`SharedNDArray`, accesible as the `.lock` attribute. It should be noted, however, that it
doesn't work well to pass a `multiprocessing.Lock` as an argument to a
`multiprocessing.Pool` function, for reasons described
[here](https://stackoverflow.com/questions/25557686/python-sharing-a-lock-between-processes#comment72803059_25558333).
Thus by default `.lock` is set to `None`.
### Typed SharedNDArray
`SharedNDArray` is able to be typed with NumPy types. When using the `from_array()`
constructor, it is also able to inherit the type of the `ndarray` if it is typed using
`numpy.typing.NDArray` (new in NumPy 1.21). Typing information does not pass through with
slicing (`__getitem__`), however.
```python
x: npt.NDArray[np.int_] = np.arange(1024)
arr = SharedNDArray(mem_mgr, x) # type of x is SharedNDArray[int_]
arr2 = arr[:] # arr2 is typing.Any
```
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/osu-nrsg/shared-ndarray2",
"name": "shared-ndarray2",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": "numpy, shared_memory, ndarray",
"author": "Randall Pittman",
"author_email": "randallpittman@outlook.com",
"download_url": "https://files.pythonhosted.org/packages/71/92/5a4cb880a09ad0d906b107227591e90c626dd45b1800db49b48f4ffa614b/shared_ndarray2-2.0.1.tar.gz",
"platform": null,
"description": "# shared-ndarray2 <!-- omit in toc -->\n\n`SharedNDArray` encapsulates a NumPy `ndarray` interface for using shared memory in\nmultiprocessing, using `multiprocessing.shared_memory` in Python 3.8+.\n\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Similar Projects](#similar-projects)\n- [Usage](#usage)\n - [Creation](#creation)\n - [`SharedNDArray()`](#sharedndarray)\n - [`SharedNDArray.from_shape()` or `shared_ndarray.from_shape()`](#sharedndarrayfrom_shape-or-shared_ndarrayfrom_shape)\n - [`SharedNDArray.from_array()` or `shared_ndarray.from_array()`](#sharedndarrayfrom_array-or-shared_ndarrayfrom_array)\n - [Using like `np.ndarray`](#using-like-npndarray)\n - [Releasing Shared Memory](#releasing-shared-memory)\n - [`.lock` attribute](#lock-attribute)\n - [Typed SharedNDArray](#typed-sharedndarray)\n\n## Quick Start\n\n```python\nimport multiprocessing\nfrom multiprocessing.managers import SharedMemoryManager\n\nimport numpy as np\n\nfrom shared_ndarray2 import SharedNDArray\n\n\ndef process_data(arr: SharedNDArray):\n # Work with data\n arr[:] += 1\n\n\nwith SharedMemoryManager() as mem_mgr:\n arr = SharedNDArray.from_array(mem_mgr, np.arange(1024))\n p = multiprocessing.Process(target=process_data, args=(arr,))\n p.start()\n p.join()\n assert np.all(arr[:] == np.arange(1, 1025))\n```\n\n## Requirements\n\n- Python 3.8+\n- NumPy 1.21+\n\n## Similar Projects\n\n- [SharedArray](https://pypi.org/project/SharedArray/) - POSIX-only. Quite a different\n paradigm, uses pre-Python 3.8 memory-sharing constructs, requires building a C module\n with `gcc`.\n- [shared-ndarray](https://pypi.org/project/shared-ndarray/) - POSIX-only. Similar (uses\n NumPy ndarray `buffer` arg), uses pre-Python 3.8 memory-sharing constructs (requires\n `posix_ipc`).\n\n## Usage\n\n### Creation\n\nThere are three methods for constructing a `SharedNDArray`.\n\n#### `SharedNDArray()`\n\nTo create a `SharedNDArray` object from existing shared memory that represents a NumPy\narray, use the regular constructor providing _shape_ and _dtype_, either with an existing\n`multiprocessing.SharedMemory` object or the name of one:\n\n```python\nshm = SharedMemory(create=True, size=1024)\narr = SharedNDArray(shm, (1024,), np.uint8)\n# -or-\narr = SharedNDArray(shm.name, (1024,), np.uint8)\n```\n\n#### `SharedNDArray.from_shape()` or `shared_ndarray.from_shape()`\n\nThis method allocates shared memory managed by a SharedMemoryManager to represent a NumPy\n`ndarray` with some _shape_ and _dtype_.\n\n```python\nwith SharedMemoryManager as mem_mgr:\n arr = SharedNDArray.from_shape(mem_mgr, (3, 1024, 1024), dtype=np.uint16)\n # ... Use arr with e.g. multiprocessing.Pool or multiprocess.Process\n # ... Be sure process instances join/terminate before exiting SharedMemoryManager context manager\n```\n\n<div style=\"font-size: small; margin-left: 4em\">\n\nNote: _`shared_ndarray.from_shape()` is a standlone function and correctly types the\n`SharedNDArray`, whereas the classmethod might (in mypy) or might not (in pyright)_\n\n</div>\n\n#### `SharedNDArray.from_array()` or `shared_ndarray.from_array()`\n\nThis method allocates shared memory managed by a SharedMemoryManager to represent a some\nprovided NumPy `ndarray` and copies that ndarray into the shared memory\n\n```python\nx = np.arange(100.0).reshape(2, 2, 25)\nwith SharedMemoryManager as mem_mgr:\n arr = SharedNDArray.from_array(mem_mgr, x)\n assert np.all(arr[:] == x[:])\n # ... Use arr as above...\n```\n\n<div style=\"font-size: small; margin-left: 4em\">\n\nNote: _`shared_ndarray.from_array()` is a standlone function and correctly types the\n`SharedNDArray`, whereas the classmethod might (in mypy) or might not (in pyright)_\n\n</div>\n\n### Using like `np.ndarray`\n\nThe point of `SharedNDArray` is to remove the boilerplate of creating shared memory,\npassing around shapes and dtypes and reconstructing `np.ndarray` objects. `SharedNDArray`\ndoes this last step with its `.get()` method, which creates a `np.ndarray` on-the-fly\nusing the shared memory buffer. The `__getitem__()` and `__setitem__()` methods use the\n`.get()` method to get the np.ndarray to access the data, so multi-dimensional indexing\nand slicing work the same as with an `ndarray`. Other `np.ndarray` methods are not\ndirectly implemented but may be accessed by first calling `.get()`, e.g.\n`arr.get().mean()`.\n\n### Releasing Shared Memory\n\n`SharedNDArray` implements a `__del__()` method that calls the\n[`.close()`](https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory.close)\nmethod on the `SharedMemory` when the instance is destroyed (i.e. at process exit). When\nthe shared memory is unlinked in the parent process (either manually with\n[`shm.unlink()`](https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory.unlink)\nor by exiting a `SharedMemoryManager` context manager) the shared_memory is properly\nreleased. However if a sub-process is not joined or terminated before the shared memory is\nunlinked a warning will be emitted about \"`leaked shared_memory objects to clean up at\nshutdown`\".\n\n### `.lock` attribute\n\nThe `__init__()`, `from_shape()`, and `from_array()` methods may be given a `lock=True`\nargument that will also create a `multiprocessing.Lock` object and include it in the\n`SharedNDArray`, accesible as the `.lock` attribute. It should be noted, however, that it\ndoesn't work well to pass a `multiprocessing.Lock` as an argument to a\n`multiprocessing.Pool` function, for reasons described\n[here](https://stackoverflow.com/questions/25557686/python-sharing-a-lock-between-processes#comment72803059_25558333).\nThus by default `.lock` is set to `None`.\n\n### Typed SharedNDArray\n\n`SharedNDArray` is able to be typed with NumPy types. When using the `from_array()`\nconstructor, it is also able to inherit the type of the `ndarray` if it is typed using\n`numpy.typing.NDArray` (new in NumPy 1.21). Typing information does not pass through with\nslicing (`__getitem__`), however.\n\n```python\nx: npt.NDArray[np.int_] = np.arange(1024)\narr = SharedNDArray(mem_mgr, x) # type of x is SharedNDArray[int_]\narr2 = arr[:] # arr2 is typing.Any\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Interface for NumPy ndarray using multiprocessing SharedMemory",
"version": "2.0.1",
"project_urls": {
"Homepage": "https://gitlab.com/osu-nrsg/shared-ndarray2",
"Repository": "https://gitlab.com/osu-nrsg/shared-ndarray2"
},
"split_keywords": [
"numpy",
" shared_memory",
" ndarray"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "29df96edf07fff44be15042a24d1a616532f28c5f7d6febb4ccd703dac42d874",
"md5": "948536e6aae9b21df6ec18189d900dd0",
"sha256": "d18dac0a558ae27836b250bceaee5561877c75f348907c0ab98aa25cba339727"
},
"downloads": -1,
"filename": "shared_ndarray2-2.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "948536e6aae9b21df6ec18189d900dd0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 8893,
"upload_time": "2024-05-16T23:11:36",
"upload_time_iso_8601": "2024-05-16T23:11:36.483243Z",
"url": "https://files.pythonhosted.org/packages/29/df/96edf07fff44be15042a24d1a616532f28c5f7d6febb4ccd703dac42d874/shared_ndarray2-2.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "71925a4cb880a09ad0d906b107227591e90c626dd45b1800db49b48f4ffa614b",
"md5": "0813f91a227f3bdafe0ad3fb34a7ff28",
"sha256": "3f85f266220640f7067b029bd31b42a11f7cfd88cd4ce05c94d9ce3a68d413e3"
},
"downloads": -1,
"filename": "shared_ndarray2-2.0.1.tar.gz",
"has_sig": false,
"md5_digest": "0813f91a227f3bdafe0ad3fb34a7ff28",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 8336,
"upload_time": "2024-05-16T23:11:38",
"upload_time_iso_8601": "2024-05-16T23:11:38.113563Z",
"url": "https://files.pythonhosted.org/packages/71/92/5a4cb880a09ad0d906b107227591e90c626dd45b1800db49b48f4ffa614b/shared_ndarray2-2.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-16 23:11:38",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "osu-nrsg",
"gitlab_project": "shared-ndarray2",
"lcname": "shared-ndarray2"
}
JSON