| Name | daodb JSON |
| Version |
0.0.6
JSON |
| download |
| home_page | https://declassed.art/repository/daodb |
| Summary | Declassed Append-Only Database |
| upload_time | 2023-03-18 19:32:52 |
| maintainer | |
| docs_url | None |
| author | AXY |
| requires_python | >=3.6 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
======
DAO DB
======
Declassed Append-Only Database
---------
Example 1
---------
.. code:: python
from daodb import DaoDB
with DaoDB('test', 'w') as db:
db.append(b'foo')
db.append(b'bar')
db.append(b'baz')
for i, record in enumerate(db):
print(i, record)
Result::
0 b'foo'
1 b'bar'
2 b'baz'
---------
Example 2
---------
.. code:: python
from daodb import JsonDaoDB
with JsonDaoDB('test.json', 'w') as db:
db.append({'foo': 'bar'})
db.append({'bar': 'foo'})
for i, record in enumerate(db):
print(i, record)
Result::
0 {'foo': 'bar'}
1 {'bar': 'foo'}
-----------
Compression
-----------
Gzip and LZMA are included in the base package. LZ4, Brotli, and Snappy
depend on `lz4 <https://pypi.org/project/lz4/>`_, `brotli <https://pypi.org/project/Brotli/>`_,
and `python-snappy <https://pypi.org/project/python-snappy/>`_ packages respectively.
All DAO DB extensions are mix-ins. Here's how to define databases that store JSON with compression:
.. code:: python
from daodb import DaoDB, GzipMixin, LzmaMixin
from daodb.lz4 import Lz4Mixin
from daodb.brotli import BrotliMixin
from daodb.snappy import SnappyMixin
class GzipDaoDB(JsonMixin, GzipMixin, DaoDB):
pass
class Lz4DaoDB(JsonMixin, Lz4Mixin, DaoDB):
pass
class LzmaDaoDB(JsonMixin, LzmaMixin, DaoDB):
pass
class BrotliDaoDB(JsonMixin, BrotliMixin, DaoDB):
pass
class SnappyDaoDB(JsonMixin, SnappyMixin, DaoDB):
pass
------------------
Database structure
------------------
* data file
* index file
Index file contains positions of records except the first one which is always zero,
and the position for new record.
Thus, the number of items in the index file equals to the number of records.
Record id is implicit, it's the index of the record.
Thus, to get a record by id, read offsets from the index file and then read the record from data file.
---------
Algorithm
---------
Write:
* lock index file exclusively
* get position for the new record from the index file
* append new record to the data file
* write new size of data file to the index file
* release file lock
Read item by id:
* Given that id is the index of record, seek to id * 16 in the index file.
* Read the position of the record and the position of the next record from the index file, 16 bytes total.
* Read the record from the data file, the size of record is calculated as a difference between positions.
No lock is necessary for read operation.
That's because append is atomic and the data written to the index file (16 bytes)
will never be split across pages to make this bug take into effect: https://bugzilla.kernel.org/show_bug.cgi?id=55651
Raw data
{
"_id": null,
"home_page": "https://declassed.art/repository/daodb",
"name": "daodb",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "",
"author": "AXY",
"author_email": "axy@declassed.art",
"download_url": "https://files.pythonhosted.org/packages/59/05/7b6e151341d9c5c36a58390a09ad962820dacfc2c8aa95e2b32c53c4a404/daodb-0.0.6.tar.gz",
"platform": null,
"description": "======\nDAO DB\n======\n\nDeclassed Append-Only Database\n\n---------\nExample 1\n---------\n\n.. code:: python\n\n from daodb import DaoDB\n\n with DaoDB('test', 'w') as db:\n db.append(b'foo')\n db.append(b'bar')\n db.append(b'baz')\n for i, record in enumerate(db):\n print(i, record)\n\nResult::\n\n 0 b'foo'\n 1 b'bar'\n 2 b'baz'\n\n\n---------\nExample 2\n---------\n\n.. code:: python\n\n from daodb import JsonDaoDB\n\n with JsonDaoDB('test.json', 'w') as db:\n db.append({'foo': 'bar'})\n db.append({'bar': 'foo'})\n for i, record in enumerate(db):\n print(i, record)\n\nResult::\n\n 0 {'foo': 'bar'}\n 1 {'bar': 'foo'}\n\n-----------\nCompression\n-----------\n\nGzip and LZMA are included in the base package. LZ4, Brotli, and Snappy\ndepend on `lz4 <https://pypi.org/project/lz4/>`_, `brotli <https://pypi.org/project/Brotli/>`_,\nand `python-snappy <https://pypi.org/project/python-snappy/>`_ packages respectively.\n\nAll DAO DB extensions are mix-ins. Here's how to define databases that store JSON with compression:\n\n.. code:: python\n\n from daodb import DaoDB, GzipMixin, LzmaMixin\n from daodb.lz4 import Lz4Mixin\n from daodb.brotli import BrotliMixin\n from daodb.snappy import SnappyMixin\n\n class GzipDaoDB(JsonMixin, GzipMixin, DaoDB):\n pass\n\n class Lz4DaoDB(JsonMixin, Lz4Mixin, DaoDB):\n pass\n\n class LzmaDaoDB(JsonMixin, LzmaMixin, DaoDB):\n pass\n\n class BrotliDaoDB(JsonMixin, BrotliMixin, DaoDB):\n pass\n\n class SnappyDaoDB(JsonMixin, SnappyMixin, DaoDB):\n pass\n\n------------------\nDatabase structure\n------------------\n\n* data file\n* index file\n\nIndex file contains positions of records except the first one which is always zero,\nand the position for new record.\nThus, the number of items in the index file equals to the number of records.\n\nRecord id is implicit, it's the index of the record.\nThus, to get a record by id, read offsets from the index file and then read the record from data file.\n\n---------\nAlgorithm\n---------\n\nWrite:\n\n* lock index file exclusively\n* get position for the new record from the index file\n* append new record to the data file\n* write new size of data file to the index file\n* release file lock\n\nRead item by id:\n\n* Given that id is the index of record, seek to id * 16 in the index file.\n* Read the position of the record and the position of the next record from the index file, 16 bytes total.\n* Read the record from the data file, the size of record is calculated as a difference between positions.\n\nNo lock is necessary for read operation.\nThat's because append is atomic and the data written to the index file (16 bytes)\nwill never be split across pages to make this bug take into effect: https://bugzilla.kernel.org/show_bug.cgi?id=55651\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Declassed Append-Only Database",
"version": "0.0.6",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "edaea1e75f2478494edc50a87cdf04e67a5ba3852137d0aa0ede30d459119888",
"md5": "a1da2648acb1e34976ba64a740fa23ed",
"sha256": "31b1628ab1f9ee740ecb0cabe43aa77777d01509bd000857cffd2e03599763da"
},
"downloads": -1,
"filename": "daodb-0.0.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a1da2648acb1e34976ba64a740fa23ed",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 8563,
"upload_time": "2023-03-18T19:32:50",
"upload_time_iso_8601": "2023-03-18T19:32:50.098062Z",
"url": "https://files.pythonhosted.org/packages/ed/ae/a1e75f2478494edc50a87cdf04e67a5ba3852137d0aa0ede30d459119888/daodb-0.0.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "59057b6e151341d9c5c36a58390a09ad962820dacfc2c8aa95e2b32c53c4a404",
"md5": "56be17cb9a166bb73021ca5a6a6ce1af",
"sha256": "82d497379806650b25274a8c3f0d0b7d651383ce9108a295f6c311794fc4ae20"
},
"downloads": -1,
"filename": "daodb-0.0.6.tar.gz",
"has_sig": false,
"md5_digest": "56be17cb9a166bb73021ca5a6a6ce1af",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 7516,
"upload_time": "2023-03-18T19:32:52",
"upload_time_iso_8601": "2023-03-18T19:32:52.100047Z",
"url": "https://files.pythonhosted.org/packages/59/05/7b6e151341d9c5c36a58390a09ad962820dacfc2c8aa95e2b32c53c4a404/daodb-0.0.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-18 19:32:52",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "daodb"
}
