# AmoreDB
Simple append-only database for Python with rich record formats and compression.
## For impatients
```bash
pip install amoredb
```
### Example 1
```python
import asyncio
from amoredb import AmoreDB
async def main():
async with AmoreDB('test', 'w') as db:
await db.append(b'foo')
await db.append(b'bar')
await db.append(b'baz')
async for record in db:
print(record)
asyncio.run(main())
```
Result:
```
b'foo'
b'bar'
b'baz'
```
### Example 2
```python
import asyncio
from amoredb.json import JsonAmoreDB
async def main():
async with JsonAmoreDB('test.json', 'w') as db:
await db.append({'foo': 'bar'})
await db.append({'bar': 'foo'})
async for record in db:
print(record)
asyncio.run(main())
```
Result:
```
{'foo': 'bar'}
{'bar': 'foo'}
```
## Record formats
The basic format for database records is bytes object. Subclasses may support other formats,
as demonstrated above in the Example 2. AmoreDB provides support for the following formats:
* JSON: `JsonMixin`, `JsonAmoreDB` from [amoredb.json](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/json.py)
* strings: `StrMixin`, `StrAmoreDB` from [amoredb.str](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/str.py)
* structures: `StructMixin`, `StructAmoreDB` from [amoredb.struct](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/struct.py)
* BSON: `BsonMixin`, `BsonAmoreDB` from [amoredb.bson](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/bson.py), requires [simple_bson](https://pypi.org/project/simple-bson/) package
Records are converted to the binary data by mixins and AmoreDB provides
predefined classes, such, for example, as
```python
class JsonAmoreDB(JsonMixin, AmoreDB):
pass
```
## Record compression
Similar to record format conversion, compression is implemented by mix-ins.
AmoreDB provides a few for the following formats:
* gzip: `GzipMixin` from [amoredb.gzip](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/gzip.py)
* lzma: `LzmaMixin` from [amoredb.lzma](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/lzma.py)
* lz4: `Lz4Mixin` from [amoredb.lzma](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/lz4.py), requires [lz4](https://pypi.org/project/lz4/) package
* brotli: `BrotliMixin` from [amoredb.brotli](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/brotli.py), requires [brotli](https://pypi.org/project/Brotli/) package
* snappy: `SnappyMixin` from [amoredb.snappy](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/snappy.py), requires [python-snappy](https://pypi.org/project/python-snappy/) package
There are no predefined classes for compression, it's up to end users to define ones for their needs.
For example,
```python
from amoredb import AmoreDB
from amoredb.json import JsonMixin
from amoredb.gzip import GzipMixin
class MyDB(JsonMixin, GzipMixin, AmoreDB):
pass
async with MyDB('test.json.gz', 'w') as db:
await db.append({'foo': 'bar'})
await db.append({'bar': 'foo'})
async for record in db:
print(record)
```
## Record transformation pipeline
Records in AmoreDB are processed by the following methods:
```python
def record_to_raw_data(self, record_data):
# do custom conversion here
# ...
# call base method
return super().record_to_raw_data(record_data)
def record_from_raw_data(self, record_data):
# do custom conversion here
# ...
# call base method
return super().record_from_raw_data(record_data)
```
Mix-ins override these methods and to make pipeline working, mix-ins should be defined in the right order.
As we have seen above,
```python
class MyDB(JsonMixin, GzipMixin, AmoreDB):
pass
```
`GzipMixin` is placed in between, because compression takes place after converting record from JSON to binary data
and before writing this data to file. Same for opposite direction.
## Database structure
The database consists of data file and index file. Optional metadata file in JSON format may contain
the structure of database class.
Index file contains positions of records except the first one which is always zero.
The first element in index file is the offset of the next record.
Thus, the number of items in the index file equals to the number of records.
Record id is implicit, it is the index of the record.
Thus, to get a record by id, read its offset from the index file and then read the record from data file.
Raw data
{
"_id": null,
"home_page": "https://github.com/amateur80lvl/amoredb",
"name": "amoredb",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "database",
"author": "amateur80lvl",
"author_email": "amateur80lvl@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/8b/9b/deb0ceac39185bae1757b5773a5225fd7f57b8c91869e647f8b25e7fd15a/amoredb-0.0.2.tar.gz",
"platform": null,
"description": "# AmoreDB\n\nSimple append-only database for Python with rich record formats and compression.\n\n## For impatients\n\n```bash\npip install amoredb\n```\n\n### Example 1\n\n```python\nimport asyncio\nfrom amoredb import AmoreDB\n\nasync def main():\n async with AmoreDB('test', 'w') as db:\n await db.append(b'foo')\n await db.append(b'bar')\n await db.append(b'baz')\n async for record in db:\n print(record)\n\nasyncio.run(main())\n```\n\nResult:\n\n```\nb'foo'\nb'bar'\nb'baz'\n```\n\n### Example 2\n\n```python\nimport asyncio\nfrom amoredb.json import JsonAmoreDB\n\nasync def main():\n async with JsonAmoreDB('test.json', 'w') as db:\n await db.append({'foo': 'bar'})\n await db.append({'bar': 'foo'})\n async for record in db:\n print(record)\n\nasyncio.run(main())\n```\n\nResult:\n\n```\n{'foo': 'bar'}\n{'bar': 'foo'}\n```\n\n## Record formats\n\nThe basic format for database records is bytes object. Subclasses may support other formats,\nas demonstrated above in the Example 2. AmoreDB provides support for the following formats:\n\n* JSON: `JsonMixin`, `JsonAmoreDB` from [amoredb.json](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/json.py)\n* strings: `StrMixin`, `StrAmoreDB` from [amoredb.str](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/str.py)\n* structures: `StructMixin`, `StructAmoreDB` from [amoredb.struct](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/struct.py)\n* BSON: `BsonMixin`, `BsonAmoreDB` from [amoredb.bson](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/bson.py), requires [simple_bson](https://pypi.org/project/simple-bson/) package\n\nRecords are converted to the binary data by mixins and AmoreDB provides\npredefined classes, such, for example, as\n\n```python\nclass JsonAmoreDB(JsonMixin, AmoreDB):\n pass\n```\n\n## Record compression\n\nSimilar to record format conversion, compression is implemented by mix-ins.\nAmoreDB provides a few for the following formats:\n\n* gzip: `GzipMixin` from [amoredb.gzip](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/gzip.py)\n* lzma: `LzmaMixin` from [amoredb.lzma](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/lzma.py)\n* lz4: `Lz4Mixin` from [amoredb.lzma](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/lz4.py), requires [lz4](https://pypi.org/project/lz4/) package\n* brotli: `BrotliMixin` from [amoredb.brotli](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/brotli.py), requires [brotli](https://pypi.org/project/Brotli/) package\n* snappy: `SnappyMixin` from [amoredb.snappy](https://github.com/amateur80lvl/amoredb/blob/main/amoredb/snappy.py), requires [python-snappy](https://pypi.org/project/python-snappy/) package\n\nThere are no predefined classes for compression, it's up to end users to define ones for their needs.\nFor example,\n\n```python\nfrom amoredb import AmoreDB\nfrom amoredb.json import JsonMixin\nfrom amoredb.gzip import GzipMixin\n\nclass MyDB(JsonMixin, GzipMixin, AmoreDB):\n pass\n\nasync with MyDB('test.json.gz', 'w') as db:\n await db.append({'foo': 'bar'})\n await db.append({'bar': 'foo'})\n async for record in db:\n print(record)\n```\n\n## Record transformation pipeline\n\nRecords in AmoreDB are processed by the following methods:\n\n```python\n def record_to_raw_data(self, record_data):\n # do custom conversion here\n # ...\n # call base method\n return super().record_to_raw_data(record_data)\n\n def record_from_raw_data(self, record_data):\n # do custom conversion here\n # ...\n # call base method\n return super().record_from_raw_data(record_data)\n```\n\nMix-ins override these methods and to make pipeline working, mix-ins should be defined in the right order.\nAs we have seen above,\n\n```python\nclass MyDB(JsonMixin, GzipMixin, AmoreDB):\n pass\n```\n\n`GzipMixin` is placed in between, because compression takes place after converting record from JSON to binary data\nand before writing this data to file. Same for opposite direction.\n\n\n## Database structure\n\nThe database consists of data file and index file. Optional metadata file in JSON format may contain\nthe structure of database class.\n\nIndex file contains positions of records except the first one which is always zero.\nThe first element in index file is the offset of the next record.\nThus, the number of items in the index file equals to the number of records.\n\nRecord id is implicit, it is the index of the record.\nThus, to get a record by id, read its offset from the index file and then read the record from data file.\n",
"bugtrack_url": null,
"license": "",
"summary": "Simple append-only database with rich record formats and compression",
"version": "0.0.2",
"project_urls": {
"Homepage": "https://github.com/amateur80lvl/amoredb"
},
"split_keywords": [
"database"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ba081c5914ea8280adc757dd534e730e51d13e874effc521663381d0babefd79",
"md5": "0aeb321927492182f0cf2eee1e88265c",
"sha256": "2bb4a7b505b3ebe1028d172b607cbef9c134e555e7c113628f58845f1d0b6763"
},
"downloads": -1,
"filename": "amoredb-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0aeb321927492182f0cf2eee1e88265c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 13783,
"upload_time": "2023-11-11T16:28:43",
"upload_time_iso_8601": "2023-11-11T16:28:43.344316Z",
"url": "https://files.pythonhosted.org/packages/ba/08/1c5914ea8280adc757dd534e730e51d13e874effc521663381d0babefd79/amoredb-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8b9bdeb0ceac39185bae1757b5773a5225fd7f57b8c91869e647f8b25e7fd15a",
"md5": "096e4db1a64265b2075196ab715f7c63",
"sha256": "9625af2250d0751d81818a4dbf8ccb8fcaa93992abb749399bb3f7d251690fb6"
},
"downloads": -1,
"filename": "amoredb-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "096e4db1a64265b2075196ab715f7c63",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11805,
"upload_time": "2023-11-11T16:28:44",
"upload_time_iso_8601": "2023-11-11T16:28:44.930669Z",
"url": "https://files.pythonhosted.org/packages/8b/9b/deb0ceac39185bae1757b5773a5225fd7f57b8c91869e647f8b25e7fd15a/amoredb-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-11 16:28:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "amateur80lvl",
"github_project": "amoredb",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "amoredb"
}
JSON
