Chest
=====
|Build Status| |Coverage Status| |Version Status| |Downloads|
A dictionary that spills to disk.
Chest acts likes a dictionary but it can write its contents to disk. This is
useful in the following two occasions:
1. Chest can hold datasets that are larger than memory
2. Chest persists and so can be saved and loaded for later use
Related Projects
----------------
The standard library ``shelve`` is an alternative out-of-core dictionary.
``Chest`` offers the following benefits over shelve_:
1. Chest supports any hashable key (not just strings)
2. Chest supports pluggable serialization and file saving schemes
Alternatively one might consider a traditional key-value store database like
Redis_.
Shove_ is another excellent alternative with support for a variety of stores.
How it works
------------
Chest stores data in two locations
1. An in-memory dictionary
2. On the filesystem in a directory owned by the chest
As a user adds contents to the chest the in-memory dictionary fills up. When
a chest stores more data in memory than desired (see ``available_memory=``
keyword argument) it writes the larger contents of the chest to disk as pickle
files (the choice of ``pickle`` is configurable). When a user asks for a value
chest checks the in-memory store, then checks on-disk and loads the value into
memory if necessary, pushing other values to disk.
Chest is a simple project. It was intended to provide a simple interface to
assist in the storage and retrieval of numpy arrays. However it's design and
implementation are agnostic to this case and so could be used in a variety of
other situations.
With minimal work chest could be extended to serve as a communication point
between multiple processes.
Known Failings
--------------
Chest was designed to hold a moderate amount of largish numpy arrays. It
doesn't handle the very many small key-value pairs usecase (though could with
small effort). In particular chest has the following deficiencies
1. Chest is not multi-process safe. We should institute a file lock at least
around the ``.keys`` file.
2. Chest does not support mutation of variables on disk.
LICENSE
-------
New BSD. See License_
Install
-------
``chest`` is available through ``conda``::
conda install chest
``chest`` is on the Python Package Index (PyPI)::
pip install chest
Example
-------
.. code:: python
>>> from chest import Chest
>>> c = Chest()
>>> # Acts like a normal dictionary
>>> c['x'] = [1, 2, 3]
>>> c['x']
[1, 2, 3]
>>> # Data persists to local files
>>> c.flush()
>>> import os
>>> os.listdir(c.path)
['.keys', 'x']
>>> # These files hold pickled results
>>> import pickle
>>> pickle.load(open(c.key_to_filename('x')))
[1, 2, 3]
>>> # Though one normally accesses these files with chest itself
>>> c2 = Chest(path=c.path)
>>> c2.keys()
['x']
>>> c2['x']
[1, 2, 3]
>>> # Chest is configurable, so one can use json, etc. instead of pickle
>>> import json
>>> c = Chest(path='my-chest', dump=json.dump, load=json.load)
>>> c['x'] = [1, 2, 3]
>>> c.flush()
>>> json.load(open(c.key_to_filename('x')))
[1, 2, 3]
Dependencies
------------
``Chest`` supports Python 2.6+ and Python 3.2+ with a common codebase.
It currently depends on the ``heapdict`` library.
It is a light weight dependency.
.. _shelve: https://docs.python.org/3/library/shelve.html
.. _Shove: https://pypi.python.org/pypi/shove/0.5.6
.. _License: https://github.com/ContinuumIO/chest/blob/master/LICENSE.txt
.. _Redis: http://redis.io/
.. |Build Status| image:: https://travis-ci.org/ContinuumIO/chest.png
:target: https://travis-ci.org/ContinuumIO/chest
.. |Coverage Status| image:: https://coveralls.io/repos/mrocklin/chest/badge.png
:target: https://coveralls.io/r/mrocklin/chest
.. |Version Status| image:: https://pypip.in/v/chest/badge.png
:target: https://pypi.python.org/pypi/chest/
.. |Downloads| image:: https://pypip.in/d/chest/badge.png
:target: https://pypi.python.org/pypi/chest/
Raw data
{
"_id": null,
"home_page": "https://github.com/ContinuumIO/chest",
"name": "chest",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "dictionary out-of-core",
"author": "Matthew Rocklin",
"author_email": "mrocklin@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/18/66/b883b9a26cd2f777dd04b7eedc842d31ea1567b7709b049d46eca418501e/chest-0.2.3.tar.gz",
"platform": "UNKNOWN",
"description": "Chest\n=====\n\n|Build Status| |Coverage Status| |Version Status| |Downloads|\n\nA dictionary that spills to disk.\n\nChest acts likes a dictionary but it can write its contents to disk. This is\nuseful in the following two occasions:\n\n1. Chest can hold datasets that are larger than memory\n2. Chest persists and so can be saved and loaded for later use\n\nRelated Projects\n----------------\n\nThe standard library ``shelve`` is an alternative out-of-core dictionary.\n``Chest`` offers the following benefits over shelve_:\n\n1. Chest supports any hashable key (not just strings)\n2. Chest supports pluggable serialization and file saving schemes\n\nAlternatively one might consider a traditional key-value store database like\nRedis_.\n\nShove_ is another excellent alternative with support for a variety of stores.\n\n\nHow it works\n------------\n\nChest stores data in two locations\n\n1. An in-memory dictionary\n2. On the filesystem in a directory owned by the chest\n\nAs a user adds contents to the chest the in-memory dictionary fills up. When\na chest stores more data in memory than desired (see ``available_memory=``\nkeyword argument) it writes the larger contents of the chest to disk as pickle\nfiles (the choice of ``pickle`` is configurable). When a user asks for a value\nchest checks the in-memory store, then checks on-disk and loads the value into\nmemory if necessary, pushing other values to disk.\n\nChest is a simple project. It was intended to provide a simple interface to\nassist in the storage and retrieval of numpy arrays. However it's design and\nimplementation are agnostic to this case and so could be used in a variety of\nother situations.\n\nWith minimal work chest could be extended to serve as a communication point\nbetween multiple processes.\n\n\nKnown Failings\n--------------\n\nChest was designed to hold a moderate amount of largish numpy arrays. It\ndoesn't handle the very many small key-value pairs usecase (though could with\nsmall effort). In particular chest has the following deficiencies\n\n1. Chest is not multi-process safe. We should institute a file lock at least\n around the ``.keys`` file.\n2. Chest does not support mutation of variables on disk.\n\n\nLICENSE\n-------\n\nNew BSD. See License_\n\n\nInstall\n-------\n\n``chest`` is available through ``conda``::\n\n conda install chest\n\n``chest`` is on the Python Package Index (PyPI)::\n\n pip install chest\n\n\nExample\n-------\n\n.. code:: python\n\n >>> from chest import Chest\n >>> c = Chest()\n\n >>> # Acts like a normal dictionary\n >>> c['x'] = [1, 2, 3]\n >>> c['x']\n [1, 2, 3]\n\n >>> # Data persists to local files\n >>> c.flush()\n >>> import os\n >>> os.listdir(c.path)\n ['.keys', 'x']\n\n >>> # These files hold pickled results\n >>> import pickle\n >>> pickle.load(open(c.key_to_filename('x')))\n [1, 2, 3]\n\n >>> # Though one normally accesses these files with chest itself\n >>> c2 = Chest(path=c.path)\n >>> c2.keys()\n ['x']\n >>> c2['x']\n [1, 2, 3]\n\n >>> # Chest is configurable, so one can use json, etc. instead of pickle\n >>> import json\n >>> c = Chest(path='my-chest', dump=json.dump, load=json.load)\n >>> c['x'] = [1, 2, 3]\n >>> c.flush()\n\n >>> json.load(open(c.key_to_filename('x')))\n [1, 2, 3]\n\n\nDependencies\n------------\n\n``Chest`` supports Python 2.6+ and Python 3.2+ with a common codebase.\n\nIt currently depends on the ``heapdict`` library.\n\nIt is a light weight dependency.\n\n\n.. _shelve: https://docs.python.org/3/library/shelve.html\n.. _Shove: https://pypi.python.org/pypi/shove/0.5.6\n.. _License: https://github.com/ContinuumIO/chest/blob/master/LICENSE.txt\n.. _Redis: http://redis.io/\n.. |Build Status| image:: https://travis-ci.org/ContinuumIO/chest.png\n :target: https://travis-ci.org/ContinuumIO/chest\n.. |Coverage Status| image:: https://coveralls.io/repos/mrocklin/chest/badge.png\n :target: https://coveralls.io/r/mrocklin/chest\n.. |Version Status| image:: https://pypip.in/v/chest/badge.png\n :target: https://pypi.python.org/pypi/chest/\n.. |Downloads| image:: https://pypip.in/d/chest/badge.png\n :target: https://pypi.python.org/pypi/chest/",
"bugtrack_url": null,
"license": "BSD",
"summary": "Simple on-disk dictionary",
"version": "0.2.3",
"split_keywords": [
"dictionary",
"out-of-core"
],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "1f5fc22d0caad0fc63e6bd68cc3ca873",
"sha256": "f2d1030d4720fd4c0cb258c3383d4bab764cfe441bab1366a0d186b0baf4f4d6"
},
"downloads": -1,
"filename": "chest-0.2.3.tar.gz",
"has_sig": false,
"md5_digest": "1f5fc22d0caad0fc63e6bd68cc3ca873",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 9614,
"upload_time": "2015-07-06T18:42:29",
"upload_time_iso_8601": "2015-07-06T18:42:29.826124Z",
"url": "https://files.pythonhosted.org/packages/18/66/b883b9a26cd2f777dd04b7eedc842d31ea1567b7709b049d46eca418501e/chest-0.2.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2015-07-06 18:42:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "ContinuumIO",
"github_project": "chest",
"travis_ci": true,
"coveralls": true,
"github_actions": false,
"requirements": [
{
"name": "heapdict",
"specs": []
}
],
"lcname": "chest"
}
JSON
