| Name | pyodb JSON |
| Version |
0.1.4
JSON |
| download |
| home_page | |
| Summary | Python Object DataBase (PyODB) is an ORM library aiming to be as simple to use as possible. |
| upload_time | 2023-08-15 16:36:27 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.10 |
| license | MIT License Copyright (c) 2023 Nikolas Teuschl Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
cache
caching
database
datacache
orm
sqlite
sqlite3
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
|
# PyODB
[](https://codecov.io/gh/NeoSecundus/PyODB)
<img src="docs/img/Logo.svg" style="margin: auto; width: 20vh" />
Python Object DataBase (PyODB) is a SQLite3 ORM library aiming to be as simple to use as possible.
This library is supposed to be used for testing, in small projects, when wanting to export complex
python data in a well structured format or for local inter-process data caching.
**Basics:**
- PyODB uses the python built-in SQLite3 implementation as database and has no external dependencies.
- PyODB can take any (non-primitive) type you pass into it, extracts its members and creates a
database schema representing the type. Sub-types of the main type are also extracted recursively.
- Saves instances of known types into the database and loads them using basic filtering options.
- Also provides a caching module 'PyODBCache' which may be used for inter-process data caching.
> When using multiprocessing excessively race conditions may occur.
- All data (cache & non-cache) can persist or be deleted upon closing of the process.
1. [Setup](#setup)
2. [Basic Usage](#basic-usage)
1. [More in-depth examples](#more-in-depth-examples)
3. [Limitations](#limitations)
4. [How it works](#how-it-works)
1. [Converting Primitive Types](#converting-primitive-types)
2. [Converting Custom Types](#converting-custom-types)
1. [Simple custom type](#simple-custom-type)
2. [Custom types contained in a collection or dict](#custom-types-contained-in-a-collection-or-dict)
3. [Dynamic type definitions](#dynamic-type-definitions)
## Setup
There are two main ways to install the package. The simplest version is using pip:
```bash
pip install pyodb
```
Since there are no external dependencies this should work without problem.
Another way is to install the package from source. To do this you have to pull or download this
repository and move into the package folder. Once inside the folder you can install the package:
```bash
pip install .
```
## Basic Usage
A very basic usage example is using the library to save a simple custom type and load it elsewhere.
To do this a custom class with some members is needed. We will use the example class below:
```python
class MyType:
some_data: list[str]
some_number: int | None
def __init__(self, number: int):
self.some_data = ["Hello", "World"]
self.some_number = number
def __repr__(self) -> str:
return f"MyType Number: {self.some_number}"
```
Next you can simply import pyodb and add MyType and try saving some instances:
```python
from pyodb import PyODB
# Create PyODB instance with persistent data and type-definitions
pyodb = PyODB(persistent=True)
# Add type and save some instances
pyodb.add_type(MyType)
pyodb.save(MyType(1))
pyodb.save_multiple([MyType(2), MyType(3), MyType(4), MyType(5)])
```
Now let's say you need the data in another process or in another python program altogether.
Create a new instance - types are re-loaded from the database since persistent was set to True
> This time, when the process exits, the data will be deleted because persistent is False by default
```python
pyodb = PyODB()
```
Now you can get a Selector instance which is used to select and filter data loaded from the database.
```python
select = pyodb.select(MyType)
## filter loaded instances by some_number > 2
select.gt(some_number = 2)
## Load results into result
result = select.all()
print(result)
```
The same select can also be written as a one-liner.
```python
result = pyodb.select(MyType).gt(some_number = 2).all()
print(result)
```
You can easily delete some saved entries with the Deleter using filters like the Selector.
```python
deleted = pyodb.delete(MyType).gt(some_number = 2).commit()
print(f"Deleted {deleted} entries")
# Count remaining entries
count = pyodb.select(MyType).count()
print(f"{count} entries remaining")
```
You can also scrap all data from the database keeping the table definitions using `clear()`.
```python
pyodb.clear()
```
Known types can also be shown and removed.
Removing types does have some restrictions. For more information look at the [PyODB Examples](./docs/PyODBExamples.md).
```python
print(pyodb.known_types)
pyodb.remove_type(MyType)
print(pyodb.known_types)
```
As you can see loading and removing data is quite simple. Basic filtering is also possible in both
select and delete.
### More in-depth examples
Below are links to two documents containing more comprehensive examples of different functions.
They also contain best practices regarding performance and some possible error cases.
For PyODB examples please refer to [PyODB examples](./docs/PyODBExamples.md)
For PyODBCache examples please refer to [Caching examples](./docs/PyODBCacheExamples.md)
## Limitations
What this library can store:
- Primitive Datatypes (int, float, str, bool)
- Lists
- Dictionaries
- Custom Types
What this library cannot store:
- Fields containing functions (maybe in the future...)
- Fields that are dependent on external states or state changes
**Important Notice:**
Storing of types imported from external libraries should be avoided.
Depending of how deep an library type is nested one single datapoint could lead to the creation of
dozens of tables. One for every nested sub-type. Additionally there may be some fields containing
stateful connections or other illegal members.
As example -> Storing a `PoolManager` from `urllib3` would result in these tables:
- PoolManager
- Url
- ProxyConfig
- SSLContext
- SSLObject
- SSLSocket
- SSLSession
- ...
- ConnectionPool
- LifoQueue
- Queue
- ...
This is still a very basic example. As a rule of thumb; The higher level the library the deeper the
objects are nested on average.
To prevent this the **max recursion depth** should not be set above 3. Using more than this is
discouraged because of performance reasons as well. You can also exclude certain members by
defining the `__odb_members__` dict containing all wanted class members.
## How it works
When a python class is added to the PyODB schema the classes are converted to SQL Tables and
inserted into the database schema.
> IMPORTANT: If a class changes between executions the table retains the old definition.
> It is advisable to reset the database manually in case it was persisted.
The UML Diagram below shows what such a conversion looks like:
### Converting Primitive Types
Primitive python datatypes are simply converted to SQL datatypes.
| SQL Datatype | Python Datatypes |
|-----------------|--------------|
| INTEGER | int |
| REAL | float |
| NUMERIC | bool |
| TEXT | str |
| BLOB (pickled) | bytes, list, dict, tuple, set |
### Converting Custom Types
There is a certain amount of scenarios when converting custom types. Below is a list of
all handled scenarios and the corresponding rules that deal with them. All of these scenarios are
also depicted in the example diagram above.
Henceforth Origin references the original class whose type is parsed into a SQL Table.
#### Simple custom type
When a simple custom type is contained by the Origin a table is created for the sub-type as
well. The sub-type's table uses a `_parent_` column which references the `_id_` of the
Origin and a `_parent_table_` column referencing the type/table-name of the Origin.
When an Origin instance is inserted the sub-type instance is saved in it's own table. The table
which the instance is saved in is then referenced by the Origin's sub-type column. As example:
Here are the columns of an imaginary origin type:
|Column Name| Type | Content |
|-----------|------|---------|
| \_id_ | TEXT | OriginID |
| subtype | TEXT | MySubType |
And here are the columns of the imaginary sub-type:
|Column Name| Type | Content |
|-----------|------|---------|
| \_parent_ | TEXT | OriginID |
| \_parent_table | TEXT | Origin |
#### Custom types contained in a collection or dict
Because of performance considerations and to keep my sanity all collection types and dicts are
converted to `bytes` objects via pickle and then saved to the Database as BLOBs.
Collections and dicts can therefore not be loaded by other programs besides python unlike all other
datatypes.
### Dynamic type definitions
Dynamic type definitions (Union) of custom types can also be saved by PyODB.
Only primitive datatypes must be unambiguous.
The only exception is None. Data may always be defined with None as an option.
**This is allowed:**
```python
class Person:
name: str
dob: int
info: str | None
card: CreditCard | Bankcard
```
**This is not:**
```python
class Square:
length: int | float
```
**Important Notice:**
A table will be created for every possible Datatype!
Raw data
{
"_id": null,
"home_page": "",
"name": "pyodb",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "cache,caching,database,datacache,orm,sqlite,sqlite3",
"author": "",
"author_email": "Nikolas Teuschl <nikolas.teuschl@alpha-origin.biz>",
"download_url": "https://files.pythonhosted.org/packages/cc/fb/49150e223ac3042ef63d73d095dc683023d161fe6f47b7b4f65c863c2dcc/pyodb-0.1.4.tar.gz",
"platform": null,
"description": "# PyODB\n\n\n[](https://codecov.io/gh/NeoSecundus/PyODB)\n\n\n\n\n\n<img src=\"docs/img/Logo.svg\" style=\"margin: auto; width: 20vh\" />\n\nPython Object DataBase (PyODB) is a SQLite3 ORM library aiming to be as simple to use as possible.\nThis library is supposed to be used for testing, in small projects, when wanting to export complex\npython data in a well structured format or for local inter-process data caching.\n\n**Basics:**\n\n- PyODB uses the python built-in SQLite3 implementation as database and has no external dependencies.\n- PyODB can take any (non-primitive) type you pass into it, extracts its members and creates a\n database schema representing the type. Sub-types of the main type are also extracted recursively.\n- Saves instances of known types into the database and loads them using basic filtering options.\n- Also provides a caching module 'PyODBCache' which may be used for inter-process data caching.\n > When using multiprocessing excessively race conditions may occur.\n- All data (cache & non-cache) can persist or be deleted upon closing of the process.\n\n1. [Setup](#setup)\n2. [Basic Usage](#basic-usage)\n 1. [More in-depth examples](#more-in-depth-examples)\n3. [Limitations](#limitations)\n4. [How it works](#how-it-works)\n 1. [Converting Primitive Types](#converting-primitive-types)\n 2. [Converting Custom Types](#converting-custom-types)\n 1. [Simple custom type](#simple-custom-type)\n 2. [Custom types contained in a collection or dict](#custom-types-contained-in-a-collection-or-dict)\n 3. [Dynamic type definitions](#dynamic-type-definitions)\n\n## Setup\n\nThere are two main ways to install the package. The simplest version is using pip:\n\n```bash\npip install pyodb\n```\n\nSince there are no external dependencies this should work without problem.\n\nAnother way is to install the package from source. To do this you have to pull or download this\nrepository and move into the package folder. Once inside the folder you can install the package:\n\n```bash\npip install .\n```\n\n## Basic Usage\n\nA very basic usage example is using the library to save a simple custom type and load it elsewhere.\n\nTo do this a custom class with some members is needed. We will use the example class below:\n\n```python\nclass MyType:\n some_data: list[str]\n some_number: int | None\n\n def __init__(self, number: int):\n self.some_data = [\"Hello\", \"World\"]\n self.some_number = number\n\n def __repr__(self) -> str:\n return f\"MyType Number: {self.some_number}\"\n```\n\nNext you can simply import pyodb and add MyType and try saving some instances:\n\n```python\nfrom pyodb import PyODB\n\n# Create PyODB instance with persistent data and type-definitions\npyodb = PyODB(persistent=True)\n\n# Add type and save some instances\npyodb.add_type(MyType)\npyodb.save(MyType(1))\npyodb.save_multiple([MyType(2), MyType(3), MyType(4), MyType(5)])\n```\n\nNow let's say you need the data in another process or in another python program altogether.\n\nCreate a new instance - types are re-loaded from the database since persistent was set to True\n> This time, when the process exits, the data will be deleted because persistent is False by default\n\n```python\npyodb = PyODB()\n```\n\nNow you can get a Selector instance which is used to select and filter data loaded from the database.\n\n```python\nselect = pyodb.select(MyType)\n\n## filter loaded instances by some_number > 2\nselect.gt(some_number = 2)\n\n## Load results into result\nresult = select.all()\nprint(result)\n\n```\n\nThe same select can also be written as a one-liner.\n\n```python\nresult = pyodb.select(MyType).gt(some_number = 2).all()\nprint(result)\n```\n\nYou can easily delete some saved entries with the Deleter using filters like the Selector.\n\n\n```python\ndeleted = pyodb.delete(MyType).gt(some_number = 2).commit()\nprint(f\"Deleted {deleted} entries\")\n\n# Count remaining entries\ncount = pyodb.select(MyType).count()\nprint(f\"{count} entries remaining\")\n```\n\nYou can also scrap all data from the database keeping the table definitions using `clear()`.\n\n```python\npyodb.clear()\n```\n\nKnown types can also be shown and removed.\nRemoving types does have some restrictions. For more information look at the [PyODB Examples](./docs/PyODBExamples.md).\n\n```python\nprint(pyodb.known_types)\npyodb.remove_type(MyType)\nprint(pyodb.known_types)\n```\n\nAs you can see loading and removing data is quite simple. Basic filtering is also possible in both\nselect and delete.\n\n### More in-depth examples\n\nBelow are links to two documents containing more comprehensive examples of different functions.\nThey also contain best practices regarding performance and some possible error cases.\n\nFor PyODB examples please refer to [PyODB examples](./docs/PyODBExamples.md)\n\nFor PyODBCache examples please refer to [Caching examples](./docs/PyODBCacheExamples.md)\n\n## Limitations\n\nWhat this library can store:\n\n- Primitive Datatypes (int, float, str, bool)\n- Lists\n- Dictionaries\n- Custom Types\n\nWhat this library cannot store:\n\n- Fields containing functions (maybe in the future...)\n- Fields that are dependent on external states or state changes\n\n**Important Notice:**\n\nStoring of types imported from external libraries should be avoided.\nDepending of how deep an library type is nested one single datapoint could lead to the creation of\ndozens of tables. One for every nested sub-type. Additionally there may be some fields containing\nstateful connections or other illegal members.\n\nAs example -> Storing a `PoolManager` from `urllib3` would result in these tables:\n- PoolManager\n- Url\n- ProxyConfig\n - SSLContext\n - SSLObject\n - SSLSocket\n - SSLSession\n - ...\n- ConnectionPool\n - LifoQueue\n - Queue\n - ...\n\nThis is still a very basic example. As a rule of thumb; The higher level the library the deeper the\nobjects are nested on average.\n\nTo prevent this the **max recursion depth** should not be set above 3. Using more than this is\ndiscouraged because of performance reasons as well. You can also exclude certain members by\ndefining the `__odb_members__` dict containing all wanted class members.\n\n## How it works\n\nWhen a python class is added to the PyODB schema the classes are converted to SQL Tables and\ninserted into the database schema.\n\n> IMPORTANT: If a class changes between executions the table retains the old definition.\n> It is advisable to reset the database manually in case it was persisted.\n\nThe UML Diagram below shows what such a conversion looks like:\n\n\n\n### Converting Primitive Types\n\nPrimitive python datatypes are simply converted to SQL datatypes.\n\n| SQL Datatype | Python Datatypes |\n|-----------------|--------------|\n| INTEGER | int |\n| REAL | float |\n| NUMERIC | bool |\n| TEXT | str |\n| BLOB (pickled) | bytes, list, dict, tuple, set |\n\n### Converting Custom Types\n\nThere is a certain amount of scenarios when converting custom types. Below is a list of\nall handled scenarios and the corresponding rules that deal with them. All of these scenarios are\nalso depicted in the example diagram above.\n\nHenceforth Origin references the original class whose type is parsed into a SQL Table.\n\n#### Simple custom type\n\nWhen a simple custom type is contained by the Origin a table is created for the sub-type as\nwell. The sub-type's table uses a `_parent_` column which references the `_id_` of the\nOrigin and a `_parent_table_` column referencing the type/table-name of the Origin.\n\nWhen an Origin instance is inserted the sub-type instance is saved in it's own table. The table\nwhich the instance is saved in is then referenced by the Origin's sub-type column. As example:\n\nHere are the columns of an imaginary origin type:\n\n|Column Name| Type | Content |\n|-----------|------|---------|\n| \\_id_ | TEXT | OriginID |\n| subtype | TEXT | MySubType |\n\nAnd here are the columns of the imaginary sub-type:\n\n|Column Name| Type | Content |\n|-----------|------|---------|\n| \\_parent_ | TEXT | OriginID |\n| \\_parent_table | TEXT | Origin |\n\n#### Custom types contained in a collection or dict\n\nBecause of performance considerations and to keep my sanity all collection types and dicts are\nconverted to `bytes` objects via pickle and then saved to the Database as BLOBs.\n\nCollections and dicts can therefore not be loaded by other programs besides python unlike all other\ndatatypes.\n\n### Dynamic type definitions\n\nDynamic type definitions (Union) of custom types can also be saved by PyODB.\nOnly primitive datatypes must be unambiguous.\n\nThe only exception is None. Data may always be defined with None as an option.\n\n**This is allowed:**\n\n```python\nclass Person:\n name: str\n dob: int\n info: str | None\n card: CreditCard | Bankcard\n```\n\n**This is not:**\n\n```python\nclass Square:\n length: int | float\n```\n\n**Important Notice:**\nA table will be created for every possible Datatype!\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2023 Nikolas Teuschl Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "Python Object DataBase (PyODB) is an ORM library aiming to be as simple to use as possible.",
"version": "0.1.4",
"project_urls": {
"Bug Tracker": "https://github.com/NeoSecundus/PyODB/issues",
"Changelog": "https://github.com/NeoSecundus/PyODB/blob/main/CHANGELOG.md",
"Homepage": "https://github.com/NeoSecundus/PyODB"
},
"split_keywords": [
"cache",
"caching",
"database",
"datacache",
"orm",
"sqlite",
"sqlite3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a686cc0d10f3826e80511a8d48d1fd18d4ea6393a081f7c5798a63436d661197",
"md5": "f46c6701b79720fdfa06bce3ebd2d23b",
"sha256": "132f04bdb73ada2d756286a1ed3b8ea48262c2862c566da85853fe964a14ff96"
},
"downloads": -1,
"filename": "pyodb-0.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f46c6701b79720fdfa06bce3ebd2d23b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 24589,
"upload_time": "2023-08-15T16:36:25",
"upload_time_iso_8601": "2023-08-15T16:36:25.074839Z",
"url": "https://files.pythonhosted.org/packages/a6/86/cc0d10f3826e80511a8d48d1fd18d4ea6393a081f7c5798a63436d661197/pyodb-0.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ccfb49150e223ac3042ef63d73d095dc683023d161fe6f47b7b4f65c863c2dcc",
"md5": "2f579716febe60a49c2fc52fc5730fff",
"sha256": "3905bde34dfaacd26c63c44f561b7faab7e655776baa586fb849feb00da1a9c5"
},
"downloads": -1,
"filename": "pyodb-0.1.4.tar.gz",
"has_sig": false,
"md5_digest": "2f579716febe60a49c2fc52fc5730fff",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 45446,
"upload_time": "2023-08-15T16:36:27",
"upload_time_iso_8601": "2023-08-15T16:36:27.230606Z",
"url": "https://files.pythonhosted.org/packages/cc/fb/49150e223ac3042ef63d73d095dc683023d161fe6f47b7b4f65c863c2dcc/pyodb-0.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-15 16:36:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "NeoSecundus",
"github_project": "PyODB",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "pyodb"
}
