<img src="./artwork/logo.png" alt="logo" style="zoom:50%;" />
# What's This?
An asynchronous version of `TinyDB` with extended capabilities.
Almost every method is asynchronous. And it's based on `TinyDB 4.7.0+`.
Unlike `TinyDB` which has a minimal core, `Async-TinyDB` is designed to have max flexibility and performance.
# Incompatible Changes
* **Asynchronous**: Say goodbye to blocking IO. **Don't forget to `await` async methods**!
* **Drop support**: Only supports Python 3.10+.
* **`ujson`:** Using `ujson` instead of `json`. Some arguments aren't compatible with `json`[^1]
* **[Dev-Changes](#dev-changes)**: Changes that only matter to developers (Who customise `Storage`, `Query`, etc).
* **[Miscellaneous](#misc)**: Differences that only matter in edge cases.
# New Features
* **Event Hooks**: You can now use event hooks to hook into an operation. See [Event Hooks](./docs/EventHooks.md) for more details.
* **Redesigned ID & Doc Class**: You can [replace](#replacing-id-&-document-class) and [customise them](#customise-id-class) easily.
* **DB-level Caching**: This significantly improves the performance of all operations. However, it may cause dirty reads with some types of storage [^disable-db-level].
* **Built-in `Modifier`**: Use `Modifier` to easily [compress](./docs/Modifier.md#Compression), [encrypt](#encryption) and [extend types](./docs/Modifier.md#Conversion) of your database. Sure you can do much more than these. _(See [Modifier](./docs/Modifier.md))_
* **Isolation Level**: Performance or ACID? It's up to you[^isolevel].
* **Atomic Write**: Shipped with `JSONStorage`
* **Batch Search By IDs**: `search` method now takes an extra `doc_ids` argument (works like an additional condition)
# How to use it?
## Installation
* Minimum: `pip install async-tinydb`
* Encryption: `pip install async-tinydb[encryption]`
* Compression: `pip install async-tinydb[compression]`
* Full: `pip install async-tinydb[all]`
## Importing
```Python
import asynctinydb
```
## Usage
Read the [original `TinyDB` documents](https://tinydb.readthedocs.org). Insert an `await` in front of async methods.
Notice that some codes are still blocking, for example, when calling `len()` on `TinyDB` or `Table` Objects.
That's it.
******
## Documents For Advanced Usage
* [Modifier](./docs/Modifier.md)
* [Event Hooks](./docs/EventHooks.md)
## Replacing ID & Document Class
**NOTICE: Mixing classes in one table may cause errors!**
When a table exists in a file, `Async-TinyDB` won't determine its classes by itself, it is your duty to make sure classes are matching.
### ID Classes
* `IncreID`: Default ID class, mimics the behaviours of the original `int` ID but requires much fewer IO operations.
* `UUID`: Uses `uuid.UUID`[^uuid-version].
### Document Class
* `Document`: Default document class, uses `dict`under the bonet.
```Python
from asynctinydb import TinyDB, UUID, IncreID, Document
db = TinyDB("database.db")
# Setting ID class to `UUID`, document class to `Document`
tab = db.table("table1", document_id_class=UUID, document_class=Document)
```
_See [Customisation](#customise-id-class) for more details_
## Encryption
Currently only supports AES-GCM encryption.
There are two ways to use encryption:
### 1. Use `EncryptedJSONStorage` directly
```Python
from asynctinydb import EncryptedJSONStorage, TinyDB
async def main():
db = TinyDB("db.json", key="your key goes here", storage=EncryptedJSONStorage)
```
### 2. Use `Modifier` class
_See [Encryption](./docs/Modifier.md#Encryption)_
## Isolation Level
When operating the TinyDB concurrently, there might be racing conditions.
Set a higher isolation level to mitigate this problem.
```Python
db.isolevel = 1
```
`isolevel`:
0. No isolation, best performance.
1. Serialised(Atomic) CRUD operations. (Also ensures thread safety) (default)
2. Deepcopy documents on insertion and retrieving. (**CR**UD) (Ensures `Index` & `Query Cache` consistency)
## DB-level caching
DB-level caching improves performance dramatically.
However, this may cause data inconsistency between `Storage` and `TinyDB` if the file that `Storage` referred to is been shared.
To disable it:
```Python
db = TinyDB("./path", no_dbcache=True)
```
# Example Codes:
## Simple One
```Python
import asyncio
from asynctinydb import TinyDB, Query
async def main():
db = TinyDB('test.json')
await db.insert({"answer": 42})
print(await db.search(Query().answer == 42)) # >>> [{'answer': 42}]
asyncio.run(main())
```
## Event Hooks Example
```Python
async def main():
db = TinyDB('test.json')
@db.storage.on.write.pre
async def mul(ev: str, s: Storage, data: dict):
data["_default"]["1"]['answer'] *= 2 # directly manipulate on data
@db.storage.on.write.post
async def _print(ev, s, anystr):
print(anystr) # print json dumped string
await db.insert({"answer": 21}) # insert() will trigger both write events
await db.close()
# Reload
db = TinyDB('test.json')
print(await db.search(Query().answer == 42)) # >>> [{'answer': 42}]
```
## Customise ID Class
Inherit from `BaseID` and implement the following methods, and then you are good to go.
```Python
from asynctinydb import BaseID
class MyID(BaseID):
def __init__(self, value: Any):
"""
You should be able to convert str into MyID instance if you want to use JSONStorage.
"""
def __str__(self) -> str:
"""
Optional.
It should be implemented if you want to use JSONStorage.
"""
def __hash__(self) -> int:
...
def __eq__(self, other: object) -> bool:
...
@classmethod
def next_id(cls, table: Table, keys) -> MyID:
"""
It should return a unique ID.
"""
@classmethod
def mark_existed(cls, table: Table, new_id):
"""
Marks an ID as existing; the same ID shouldn't be generated by next_id.
"""
@classmethod
def clear_cache(cls, table: Table):
"""
Clear cache of existing IDs, if such cache exists.
"""
```
## Customise Document Class
```Python
from asynctinydb import BaseDocument
class MyDoc(BaseDocument):
"""
I am too lazy to write those necessary methods.
"""
```
Anyways, a BaseDocument class looks like this:
```Python
class BaseDocument(Mapping[IDVar, Any]):
@property
@abstractmethod
def doc_id(self) -> IDVar:
raise NotImplementedError()
@doc_id.setter
def doc_id(self, value: IDVar):
raise NotImplementedError()
```
Make sure you have implemented all the methods required by `BaseDocument` class.
# Dev-Changes
* Storage `closed` property: Original `TinyDB` won't raise exceptions when operating on a closed file. Now the property `closed` of `Storage` classes is required to be implemented[^why-closed][^operating-on-closed].
* Storage data converting: The responsibility of converting the data to the correct type is transferred to the Storage[^2]
* `is_cacheable` method in `QueryInstance` is changed to `cacheable` property and will be deprecated.
# Misc
* Lazy-load: File loading & dirs creating are delayed to the first IO operation.
* `CachingMiddleWare`: `WRITE_CACHE_SIZE` is now instance-specific.
Example: `TinyDB("test.db", storage=CachingMiddleWare(JSONStorage, 1024))`
* `search` accepts optional `cond`, returns all docs if no arguments are provided
* `get` and `contains` raises `ValueError` instead of `RuntimeError` when `cond` and `doc_id` are both `None`
* `LRUCache` stores `tuple`s of ids instead of `list`s of docs
* `search` and `get` treat `doc_id` and `doc_ids` as extra conditions instead of ignoring conditions when IDs are provided. That is to say, when `cond` and `doc_id(s)` are passed, they return docs satisfies `cond` and is in `doc_id(s)`.
[^1]: Why not `orjson`? Because `ujson` is fast enough and has more features.
[^2]: e.g. `JSONStorage` needs to convert the keys to `str` by itself.
[^UUID-version]:Currently using UUID4
[^disable-db-level]: See [DB-level caching](#db-level-caching) to learn how to disable this feature if it causes dirty reads.
[^isolevel]: See [isolevel](#isolation-level)
[^why-closed]: This is for `Middleware` classes to reliably determine whether the `Storage` is closed, so they can raise `IOError`
[^operating-on-closed]: An `IOError` should be raised when operating on closed storage.
Raw data
{
"_id": null,
"home_page": "https://github.com/VermiIIi0n/async-tinydb",
"name": "async-tinydb",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10,<4.0",
"maintainer_email": "",
"keywords": "database,nosql,async",
"author": "VermiIIi0n",
"author_email": "dungeon.behind0t@icloud.com",
"download_url": "https://files.pythonhosted.org/packages/10/75/f988e79d33102a6321d660fa0ee91f1a9d3526b53112b2c37b5252919cae/async_tinydb-1.7.3.tar.gz",
"platform": null,
"description": "<img src=\"./artwork/logo.png\" alt=\"logo\" style=\"zoom:50%;\" />\n\n# What's This?\n\nAn asynchronous version of `TinyDB` with extended capabilities.\n\nAlmost every method is asynchronous. And it's based on `TinyDB 4.7.0+`. \n\nUnlike `TinyDB` which has a minimal core, `Async-TinyDB` is designed to have max flexibility and performance.\n\n# Incompatible Changes\n\n* **Asynchronous**: Say goodbye to blocking IO. **Don't forget to `await` async methods**!\n\n* **Drop support**: Only supports Python 3.10+.\n\n* **`ujson`:** Using `ujson` instead of `json`. Some arguments aren't compatible with `json`[^1]\n\n* **[Dev-Changes](#dev-changes)**: Changes that only matter to developers (Who customise `Storage`, `Query`, etc).\n\n* **[Miscellaneous](#misc)**: Differences that only matter in edge cases.\n\n# New Features\n\n* **Event Hooks**: You can now use event hooks to hook into an operation. See [Event Hooks](./docs/EventHooks.md) for more details.\n\n* **Redesigned ID & Doc Class**: You can [replace](#replacing-id-&-document-class) and [customise them](#customise-id-class) easily.\n \n* **DB-level Caching**: This significantly improves the performance of all operations. However, it may cause dirty reads with some types of storage [^disable-db-level]. \n\n* **Built-in `Modifier`**: Use `Modifier` to easily [compress](./docs/Modifier.md#Compression), [encrypt](#encryption) and [extend types](./docs/Modifier.md#Conversion) of your database. Sure you can do much more than these. _(See [Modifier](./docs/Modifier.md))_\n\n* **Isolation Level**: Performance or ACID? It's up to you[^isolevel].\n\n* **Atomic Write**: Shipped with `JSONStorage`\n\n* **Batch Search By IDs**: `search` method now takes an extra `doc_ids` argument (works like an additional condition)\n\n# How to use it?\n\n## Installation\n\n* Minimum: `pip install async-tinydb`\n* Encryption: `pip install async-tinydb[encryption]`\n* Compression: `pip install async-tinydb[compression]`\n* Full: `pip install async-tinydb[all]`\n\n## Importing\n\n```Python\nimport asynctinydb\n```\n\n## Usage\n\nRead the [original `TinyDB` documents](https://tinydb.readthedocs.org). Insert an `await` in front of async methods. \n\nNotice that some codes are still blocking, for example, when calling `len()` on `TinyDB` or `Table` Objects.\n\nThat's it.\n\n******\n\n## Documents For Advanced Usage\n\n* [Modifier](./docs/Modifier.md)\n* [Event Hooks](./docs/EventHooks.md)\n\n## Replacing ID & Document Class\n\n**NOTICE: Mixing classes in one table may cause errors!**\n\nWhen a table exists in a file, `Async-TinyDB` won't determine its classes by itself, it is your duty to make sure classes are matching.\n\n### ID Classes\n\n* `IncreID`: Default ID class, mimics the behaviours of the original `int` ID but requires much fewer IO operations.\n* `UUID`: Uses `uuid.UUID`[^uuid-version].\n\n### Document Class\n\n* `Document`: Default document class, uses `dict`under the bonet.\n\n```Python\nfrom asynctinydb import TinyDB, UUID, IncreID, Document\n\ndb = TinyDB(\"database.db\")\n\n# Setting ID class to `UUID`, document class to `Document`\ntab = db.table(\"table1\", document_id_class=UUID, document_class=Document)\n```\n\n_See [Customisation](#customise-id-class) for more details_\n\n## Encryption\n\nCurrently only supports AES-GCM encryption.\n\nThere are two ways to use encryption:\n\n### 1. Use `EncryptedJSONStorage` directly\n\n```Python\nfrom asynctinydb import EncryptedJSONStorage, TinyDB\n\nasync def main():\n db = TinyDB(\"db.json\", key=\"your key goes here\", storage=EncryptedJSONStorage)\n\n```\n\n### 2. Use `Modifier` class\n\n_See [Encryption](./docs/Modifier.md#Encryption)_\n\n## Isolation Level\n\nWhen operating the TinyDB concurrently, there might be racing conditions.\n\nSet a higher isolation level to mitigate this problem.\n\n```Python\ndb.isolevel = 1\n```\n\n`isolevel`:\n\n0. No isolation, best performance.\n1. Serialised(Atomic) CRUD operations. (Also ensures thread safety) (default)\n2. Deepcopy documents on insertion and retrieving. (**CR**UD) (Ensures `Index` & `Query Cache` consistency)\n\n\n\n## DB-level caching\n\nDB-level caching improves performance dramatically.\n\nHowever, this may cause data inconsistency between `Storage` and `TinyDB` if the file that `Storage` referred to is been shared.\n\nTo disable it:\n\n```Python\ndb = TinyDB(\"./path\", no_dbcache=True)\n```\n\n# Example Codes:\n\n## Simple One\n\n```Python\nimport asyncio\nfrom asynctinydb import TinyDB, Query\n\nasync def main():\n db = TinyDB('test.json')\n await db.insert({\"answer\": 42})\n print(await db.search(Query().answer == 42)) # >>> [{'answer': 42}] \n\nasyncio.run(main())\n```\n## Event Hooks Example\n\n```Python\nasync def main():\n db = TinyDB('test.json')\n\n @db.storage.on.write.pre\n async def mul(ev: str, s: Storage, data: dict):\n data[\"_default\"][\"1\"]['answer'] *= 2 # directly manipulate on data\n\n @db.storage.on.write.post\n async def _print(ev, s, anystr):\n \tprint(anystr) # print json dumped string\n \n await db.insert({\"answer\": 21}) # insert() will trigger both write events\n await db.close()\n # Reload\n db = TinyDB('test.json')\n print(await db.search(Query().answer == 42)) # >>> [{'answer': 42}] \n```\n\n## Customise ID Class\n\nInherit from `BaseID` and implement the following methods, and then you are good to go.\n\n```Python\nfrom asynctinydb import BaseID\n\nclass MyID(BaseID):\n def __init__(self, value: Any):\n \"\"\"\n You should be able to convert str into MyID instance if you want to use JSONStorage.\n \"\"\"\n\n def __str__(self) -> str:\n \"\"\"\n Optional.\n It should be implemented if you want to use JSONStorage.\n \"\"\"\n\n def __hash__(self) -> int:\n ...\n\n def __eq__(self, other: object) -> bool:\n ...\n\n @classmethod\n def next_id(cls, table: Table, keys) -> MyID:\n \"\"\"\n It should return a unique ID.\n \"\"\"\n\n @classmethod\n def mark_existed(cls, table: Table, new_id):\n \"\"\"\n Marks an ID as existing; the same ID shouldn't be generated by next_id.\n \"\"\"\n\n @classmethod\n def clear_cache(cls, table: Table):\n \"\"\"\n Clear cache of existing IDs, if such cache exists.\n \"\"\"\n```\n\n## Customise Document Class\n\n```Python\nfrom asynctinydb import BaseDocument\n\nclass MyDoc(BaseDocument):\n \"\"\"\n I am too lazy to write those necessary methods.\n \"\"\"\n```\n\nAnyways, a BaseDocument class looks like this:\n\n```Python\nclass BaseDocument(Mapping[IDVar, Any]):\n @property\n @abstractmethod\n def doc_id(self) -> IDVar:\n raise NotImplementedError()\n\n @doc_id.setter\n def doc_id(self, value: IDVar):\n raise NotImplementedError()\n```\n\nMake sure you have implemented all the methods required by `BaseDocument` class.\n\n# Dev-Changes\n\n* Storage `closed` property: Original `TinyDB` won't raise exceptions when operating on a closed file. Now the property `closed` of `Storage` classes is required to be implemented[^why-closed][^operating-on-closed].\n* Storage data converting: The responsibility of converting the data to the correct type is transferred to the Storage[^2]\n* `is_cacheable` method in `QueryInstance` is changed to `cacheable` property and will be deprecated.\n\n# Misc\n\n* Lazy-load: File loading & dirs creating are delayed to the first IO operation.\n* `CachingMiddleWare`: `WRITE_CACHE_SIZE` is now instance-specific. \n Example: `TinyDB(\"test.db\", storage=CachingMiddleWare(JSONStorage, 1024))`\n* `search` accepts optional `cond`, returns all docs if no arguments are provided\n* `get` and `contains` raises `ValueError` instead of `RuntimeError` when `cond` and `doc_id` are both `None`\n* `LRUCache` stores `tuple`s of ids instead of `list`s of docs\n* `search` and `get` treat `doc_id` and `doc_ids` as extra conditions instead of ignoring conditions when IDs are provided. That is to say, when `cond` and `doc_id(s)` are passed, they return docs satisfies `cond` and is in `doc_id(s)`.\n\n\n\n[^1]: Why not `orjson`? Because `ujson` is fast enough and has more features.\n[^2]: e.g. `JSONStorage` needs to convert the keys to `str` by itself.\n[^UUID-version]:Currently using UUID4\n[^disable-db-level]: See [DB-level caching](#db-level-caching) to learn how to disable this feature if it causes dirty reads.\n[^isolevel]: See [isolevel](#isolation-level)\n[^why-closed]: This is for `Middleware` classes to reliably determine whether the `Storage` is closed, so they can raise `IOError`\n[^operating-on-closed]: An `IOError` should be raised when operating on closed storage.\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Yet Another Async TinyDB",
"version": "1.7.3",
"project_urls": {
"Homepage": "https://github.com/VermiIIi0n/async-tinydb",
"Issues": "https://github.com/VermiIIi0n/async-tinydb/issues"
},
"split_keywords": [
"database",
"nosql",
"async"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "402f1446ce9245d3d924f64601dbd8cfbf440939a516065a30948773d297700c",
"md5": "3947a3efa28217c0aebd2dc099036a3f",
"sha256": "0a55c59cce44487849c787a2830f5a84263cbc306c6cdd58810258493d6b8ef2"
},
"downloads": -1,
"filename": "async_tinydb-1.7.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3947a3efa28217c0aebd2dc099036a3f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10,<4.0",
"size": 37822,
"upload_time": "2024-03-19T03:03:26",
"upload_time_iso_8601": "2024-03-19T03:03:26.379759Z",
"url": "https://files.pythonhosted.org/packages/40/2f/1446ce9245d3d924f64601dbd8cfbf440939a516065a30948773d297700c/async_tinydb-1.7.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1075f988e79d33102a6321d660fa0ee91f1a9d3526b53112b2c37b5252919cae",
"md5": "e29c4e04f3a5c91a8c388f9201fed080",
"sha256": "e20d8d4aafe3321b38365026226eca0f89170828691ebccfe4ee00f1dd2d02ad"
},
"downloads": -1,
"filename": "async_tinydb-1.7.3.tar.gz",
"has_sig": false,
"md5_digest": "e29c4e04f3a5c91a8c388f9201fed080",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10,<4.0",
"size": 36815,
"upload_time": "2024-03-19T03:03:28",
"upload_time_iso_8601": "2024-03-19T03:03:28.295505Z",
"url": "https://files.pythonhosted.org/packages/10/75/f988e79d33102a6321d660fa0ee91f1a9d3526b53112b2c37b5252919cae/async_tinydb-1.7.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-19 03:03:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "VermiIIi0n",
"github_project": "async-tinydb",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "async-tinydb"
}
JSON
