# Shared Memory Dict
A very simple [shared memory](https://docs.python.org/3/library/multiprocessing.shared_memory.html) dict implementation.
**Requires**: Python >= 3.8
```python
>>> # In the first Python interactive shell
>> from shared_memory_dict import SharedMemoryDict
>> smd = SharedMemoryDict(name='tokens', size=1024)
>> smd['some-key'] = 'some-value-with-any-type'
>> smd['some-key']
'some-value-with-any-type'
>>> # In either the same shell or a new Python shell on the same machine
>> existing_smd = SharedMemoryDict(name='tokens', size=1024)
>>> existing_smd['some-key']
'some-value-with-any-type'
>>> existing_smd['new-key'] = 'some-value-with-any-type'
>>> # Back in the first Python interactive shell, smd reflects this change
>> smd['new-key']
'some-value-with-any-type'
>>> # Clean up from within the second Python shell
>>> existing_smd.shm.close() # or "del existing_smd"
>>> # Clean up from within the first Python shell
>>> smd.shm.close()
>>> smd.shm.unlink() # Free and release the shared memory block at the very end
>>> del smd # use of smd after call unlink() is unsupported
```
> The arg `name` defines the location of the memory block, so if you want to share the memory between process use the same name.
> The size (in bytes) occupied by the contents of the dictionary depends on the serialization used in storage. By default pickle is used.
## Installation
Using `pip`:
```shell
pip install shared-memory-dict
```
## Locks
To use [multiprocessing.Lock](https://docs.python.org/3.8/library/multiprocessing.html#multiprocessing.Lock) on write operations of shared memory dict set environment variable `SHARED_MEMORY_USE_LOCK=1`.
## Serialization
We use [pickle](https://docs.python.org/3/library/pickle.html) as default to read and write the data into the shared memory block.
You can create a custom serializer by implementing the `dumps` and `loads` methods.
Custom serializers should raise `SerializationError` if the serialization fails and `DeserializationError` if the deserialization fails. Both are defined in the `shared_memory_dict.serializers` module.
An example of a JSON serializer extracted from serializers module:
```python
NULL_BYTE: Final = b"\x00"
class JSONSerializer:
def dumps(self, obj: dict) -> bytes:
try:
return json.dumps(obj).encode() + NULL_BYTE
except (ValueError, TypeError):
raise SerializationError(obj)
def loads(self, data: bytes) -> dict:
data = data.split(NULL_BYTE, 1)[0]
try:
return json.loads(data)
except json.JSONDecodeError:
raise DeserializationError(data)
```
Note: A null byte is used to separate the dictionary contents from the bytes that are in memory.
To use the custom serializer you must set it when creating a new shared memory dict instance:
```python
>>> smd = SharedMemoryDict(name='tokens', size=1024, serializer=JSONSerializer())
```
### Caveat
The pickle module is not secure. Only unpickle data you trust.
See more [here](https://docs.python.org/3/library/pickle.html).
## Django Cache Implementation
There's a [Django Cache Implementation](https://docs.djangoproject.com/en/3.0/topics/cache/) with Shared Memory Dict:
```python
# settings/base.py
CACHES = {
'default': {
'BACKEND': 'shared_memory_dict.caches.django.SharedMemoryCache',
'LOCATION': 'memory',
'OPTIONS': {'MEMORY_BLOCK_SIZE': 1024}
}
}
```
**Install with**: `pip install "shared-memory-dict[django]"`
### Caveat
With Django cache implementation the keys only expire when they're read. Be careful with memory usage
## AioCache Backend
There's also a [AioCache Backend Implementation](https://aiocache.readthedocs.io/en/latest/caches.html) with Shared Memory Dict:
```python
From aiocache import caches
caches.set_config({
'default': {
'cache': 'shared_memory_dict.caches.aiocache.SharedMemoryCache',
'size': 1024,
},
})
```
> This implementation is very based on aiocache [SimpleMemoryCache](https://aiocache.readthedocs.io/en/latest/caches.html#simplememorycache)
**Install with**: `pip install "shared-memory-dict[aiocache]"`
Raw data
{
"_id": null,
"home_page": "https://github.com/luizalabs/shared-memory-dict",
"name": "shared-memory-dict",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "django,aiocache,shared memory",
"author": "Arquitetura LuizaLabs",
"author_email": "arquitetura@luizalabs.com",
"download_url": "https://files.pythonhosted.org/packages/34/0a/16c63e478658ce9ccb9e5d04b7650347a1e00f4db6da11487fdeb45b6b94/shared-memory-dict-0.7.2.tar.gz",
"platform": "",
"description": "# Shared Memory Dict\n\nA very simple [shared memory](https://docs.python.org/3/library/multiprocessing.shared_memory.html) dict implementation.\n\n**Requires**: Python >= 3.8\n\n```python\n>>> # In the first Python interactive shell\n>> from shared_memory_dict import SharedMemoryDict\n>> smd = SharedMemoryDict(name='tokens', size=1024)\n>> smd['some-key'] = 'some-value-with-any-type'\n>> smd['some-key']\n'some-value-with-any-type'\n\n>>> # In either the same shell or a new Python shell on the same machine\n>> existing_smd = SharedMemoryDict(name='tokens', size=1024)\n>>> existing_smd['some-key']\n'some-value-with-any-type'\n>>> existing_smd['new-key'] = 'some-value-with-any-type'\n\n\n>>> # Back in the first Python interactive shell, smd reflects this change\n>> smd['new-key']\n'some-value-with-any-type'\n\n>>> # Clean up from within the second Python shell\n>>> existing_smd.shm.close() # or \"del existing_smd\"\n\n>>> # Clean up from within the first Python shell\n>>> smd.shm.close()\n>>> smd.shm.unlink() # Free and release the shared memory block at the very end\n>>> del smd # use of smd after call unlink() is unsupported\n```\n\n> The arg `name` defines the location of the memory block, so if you want to share the memory between process use the same name.\n> The size (in bytes) occupied by the contents of the dictionary depends on the serialization used in storage. By default pickle is used.\n\n## Installation\n\nUsing `pip`:\n\n```shell\npip install shared-memory-dict\n```\n\n## Locks\n\nTo use [multiprocessing.Lock](https://docs.python.org/3.8/library/multiprocessing.html#multiprocessing.Lock) on write operations of shared memory dict set environment variable `SHARED_MEMORY_USE_LOCK=1`.\n\n## Serialization\n\nWe use [pickle](https://docs.python.org/3/library/pickle.html) as default to read and write the data into the shared memory block.\n\nYou can create a custom serializer by implementing the `dumps` and `loads` methods.\n\nCustom serializers should raise `SerializationError` if the serialization fails and `DeserializationError` if the deserialization fails. Both are defined in the `shared_memory_dict.serializers` module.\n\nAn example of a JSON serializer extracted from serializers module:\n\n```python\nNULL_BYTE: Final = b\"\\x00\"\n\n\nclass JSONSerializer:\n def dumps(self, obj: dict) -> bytes:\n try:\n return json.dumps(obj).encode() + NULL_BYTE\n except (ValueError, TypeError):\n raise SerializationError(obj)\n\n def loads(self, data: bytes) -> dict:\n data = data.split(NULL_BYTE, 1)[0]\n try:\n return json.loads(data)\n except json.JSONDecodeError:\n raise DeserializationError(data)\n\n```\n\nNote: A null byte is used to separate the dictionary contents from the bytes that are in memory.\n\nTo use the custom serializer you must set it when creating a new shared memory dict instance:\n\n```python\n>>> smd = SharedMemoryDict(name='tokens', size=1024, serializer=JSONSerializer())\n```\n\n### Caveat\n\nThe pickle module is not secure. Only unpickle data you trust.\n\nSee more [here](https://docs.python.org/3/library/pickle.html).\n\n## Django Cache Implementation\n\nThere's a [Django Cache Implementation](https://docs.djangoproject.com/en/3.0/topics/cache/) with Shared Memory Dict:\n\n```python\n# settings/base.py\nCACHES = {\n 'default': {\n 'BACKEND': 'shared_memory_dict.caches.django.SharedMemoryCache',\n 'LOCATION': 'memory',\n 'OPTIONS': {'MEMORY_BLOCK_SIZE': 1024}\n }\n}\n```\n\n**Install with**: `pip install \"shared-memory-dict[django]\"`\n\n### Caveat\n\nWith Django cache implementation the keys only expire when they're read. Be careful with memory usage\n\n## AioCache Backend\n\nThere's also a [AioCache Backend Implementation](https://aiocache.readthedocs.io/en/latest/caches.html) with Shared Memory Dict:\n\n```python\nFrom aiocache import caches\n\ncaches.set_config({\n 'default': {\n 'cache': 'shared_memory_dict.caches.aiocache.SharedMemoryCache',\n 'size': 1024,\n },\n})\n```\n\n> This implementation is very based on aiocache [SimpleMemoryCache](https://aiocache.readthedocs.io/en/latest/caches.html#simplememorycache)\n\n**Install with**: `pip install \"shared-memory-dict[aiocache]\"`\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A very simple shared memory dict implementation",
"version": "0.7.2",
"project_urls": {
"Homepage": "https://github.com/luizalabs/shared-memory-dict",
"Repository": "https://github.com/luizalabs/shared-memory-dict"
},
"split_keywords": [
"django",
"aiocache",
"shared memory"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0ffc8147ce28a7e16029bc6432f81a461575ea7e7d1b852a75e2acdb0011ce41",
"md5": "53004321f8604b90e30f36711816968d",
"sha256": "b1d5a40a7a40e764187ae3a95c6096a4c4a8b27625408c63950d9db2b768506b"
},
"downloads": -1,
"filename": "shared_memory_dict-0.7.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "53004321f8604b90e30f36711816968d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 10407,
"upload_time": "2022-01-12T23:50:23",
"upload_time_iso_8601": "2022-01-12T23:50:23.558015Z",
"url": "https://files.pythonhosted.org/packages/0f/fc/8147ce28a7e16029bc6432f81a461575ea7e7d1b852a75e2acdb0011ce41/shared_memory_dict-0.7.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "340a16c63e478658ce9ccb9e5d04b7650347a1e00f4db6da11487fdeb45b6b94",
"md5": "fee3a656e7fda58d4a96797a0a32d43a",
"sha256": "8294257e1a2466afb5e19a878a8c446325b8913d57d6d9578ccba905564a974d"
},
"downloads": -1,
"filename": "shared-memory-dict-0.7.2.tar.gz",
"has_sig": false,
"md5_digest": "fee3a656e7fda58d4a96797a0a32d43a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 9347,
"upload_time": "2022-01-12T23:50:22",
"upload_time_iso_8601": "2022-01-12T23:50:22.407726Z",
"url": "https://files.pythonhosted.org/packages/34/0a/16c63e478658ce9ccb9e5d04b7650347a1e00f4db6da11487fdeb45b6b94/shared-memory-dict-0.7.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-01-12 23:50:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "luizalabs",
"github_project": "shared-memory-dict",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "shared-memory-dict"
}
JSON
