jinbase


Namejinbase JSON
Version 0.0.5 PyPI version JSON
download
home_pagehttps://github.com/pyrustic/jinbase
SummaryMulti-model transactional embedded database
upload_time2024-12-10 20:54:32
maintainerPyrustic Architect
docs_urlNone
authorPyrustic Architect
requires_python>=3.5
licenseMIT
keywords pyrustic
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![PyPI package version](https://img.shields.io/pypi/v/jinbase)](https://pypi.org/project/jinbase)
[![Downloads](https://static.pepy.tech/badge/jinbase)](https://pepy.tech/project/jinbase)


<!-- Cover -->
<div align="center">
    <img src="https://raw.githubusercontent.com/pyrustic/misc/master/assets/jinbase/cover.jpg" alt="Cover image" width="800">
    <p align="center">
        <a href="https://commons.wikimedia.org/wiki/File:Rudolf_Reschreiter_Blick_von_der_H%C3%B6llentalangerh%C3%BCtte_zum_H%C3%B6llentalgletscher_und_den_Riffelwandspitzen_1921.jpg">Rudolf Reschreiter</a>, Public domain, via Wikimedia Commons
    </p>
</div>

<!-- Intro Text -->
# Jinbase
<b>Multi-model transactional embedded database</b>


## Table of contents
- [Overview](#overview)
    - [Multiple data models coexisting in a single embedded database](#multiple-data-models-coexisting-in-a-single-embedded-database)
    - [Support for transactions and complex data of arbitrary size](#support-for-transactions-and-complex-data-of-arbitrary-size)
    - [Bulk and partial access to records from byte-level to field-level](#bulk-and-partial-access-to-records-from-byte-level-to-field-level)
    - [Highly configurable database and timestamped records](#highly-configurable-database-and-timestamped-records)
- [Why use Jinbase ?](#why-use-jinbase-)
- [Data models and their corresponding storage interfaces](#data-models-and-their-corresponding-storage-interfaces)
    - [Kv](#kv)
    - [Depot](#depot)
    - [Queue](#queue)
    - [Stack](#stack)
    - [Relational](#relational)
- [The unified BLOB interface](#the-unified-blob-interface)
- [Command line interface](#command-line-interface)
- [Related projects](#related-projects)
- [Testing and contributing](#testing-and-contributing)
- [Installation](#installation)


# Overview
**Jinbase** (pronounced as **/ˈdʒɪnˌbeɪs/**) is a multi-model [transactional](https://en.wikipedia.org/wiki/Database_transaction) [embedded database](https://en.wikipedia.org/wiki/Embedded_database) that uses [SQLite](https://www.sqlite.org/) as storage engine. Its reference implementation is an eponymous lightweight [Python](https://www.python.org/) library available on [PyPI](#installation).


## Multiple data models coexisting in a single embedded database
A single Jinbase database supports **key-value**, **depot**, **queue**, **stack**, and **relational** data models. While a Jinbase file can be populated with multi-model data, depending on the needs, it is quite possible to dedicate a database file to a given model.

For each of the first four data models, there is a programmatic interface accessible via an eponymous property of a Jinbase instance.

## Support for transactions and complex data of arbitrary size
Having SQLite as the storage engine allows Jinbase to benefit from [transactions](https://www.sqlite.org/lang_transaction.html). Jinbase ensures that at the top level, reads and writes on key-value, depot, queue and stack stores are transactional. For user convenience, context managers are exposed to create transactions of different modes.

When for a write operation, the user submits data, whether it is a dictionary, string or integer, Jinbase serializes (except binary data), chunks and stores the data iteratively with the [Paradict](https://github.com/pyrustic/paradict) compact binary data format. This then allows for the smooth storage of complex data of [arbitrary size](https://www.sqlite.org/limits.html) with Jinbase.

## Bulk and partial access to records from byte-level to field-level
Jinbase not only offers bulk access to records, but also two levels of partial access granularity.

SQLite has an impressive capability which is [incremental I/O](https://sqlite.org/c3ref/blob_open.html) for [BLOBs](https://www.sqlite.org/datatype3.html). While this capability is designed to target an individual BLOB column in a row, Jinbase extends this so that for each record, incremental reads cover all chunks as if they were a single [unified BLOB](#the-unified-blob-interface).

For [dictionary](https://en.wikipedia.org/wiki/Associative_array) records only, Jinbase automatically creates and maintains a lightweight index consisting of pointers to root fields, which then allows extracting from an arbitrary record the contents of a field automatically deserialized before being returned.

## Highly configurable database and timestamped records
Jinbase exposes a database connection object to the underlying SQLite storage engine, allowing for sophisticated configuration. The [Paradict](https://github.com/pyrustic/paradict) binary data format used for serializing records also allows for customizing data types via a `paradict.TypeRef` object.

Each record stored in a key-value, depot, queue, or stack store is automatically timestamped. This allows the user to provide a `time_range` tuple when querying records. The precision of the timestamp, which defaults to milliseconds, can also be configured.



# Why use Jinbase ?
Jinbase implements persistence for familiar data models whose stores coexist in a single file with an intuitive programmatic interface. Supported [data types](https://github.com/pyrustic/paradict) range from simple to complex and of [arbitrary size](https://www.sqlite.org/limits.html).

For convenience, all Jinbase-related tables are prefixed with `jinbase_`, allowing the user to define their own tables and interact with them as they would with a regular SQLite database. 

Thanks to its multi-model coexistence capability, Jinbase can be used to open legacy SQLite databases to add four useful data models (key-value, depot, queue, and stack).

All this makes Jinbase relevant from prototype to production stages of software development of various sizes and scopes. 

Following are few of the most obvious use cases:

- Storing user preferences
- Persisting session data before exit
- Order-based processing of data streams
- Exposing data for other processes
- Upgrading legacy SQLite files with new data models
- Bespoke data persistence solution

# Data models and their corresponding storage interfaces
Following subsections discuss data models and their corresponding storage interfaces.

## Kv
The [key-value](https://en.wikipedia.org/wiki/Key%E2%80%93value_database) data model associates to a string or an integer key, a value that is serializable with the [Paradict](https://github.com/pyrustic/paradict) binary data format. 

String keys can be searched with a [glob](https://en.wikipedia.org/wiki/Glob_(programming)) pattern and integer keys can be searched within a range of numbers. Since records are automatically timestamped, a `time_range` tuple can be provided by the user to search for them as well as keys.

Example:

```python
import os.path
from datetime import datetime
from jinbase import Jinbase, JINBASE_HOME

user_data = {"id": 42, "name": "alex", "created_at": datetime.now(),
             "photo": b'\x45\xA6\x42\xDF\x69',
             "books": {"sci-fi": ["book 1", "book 2"],
                       "thriller": ["book 3", ["book4"]]}}

db_filename = os.path.join(JINBASE_HOME, "test.db")

with Jinbase(db_filename) as db:
  # set 'user'
  kv_store = db.kv
  kv_store.set("user", user_data)  # returns a UID

  # get 'user'
  data = kv_store.get("user")
  assert data == user_data

  # count total records and bytes
  print(kv_store.count_records())
  print(kv_store.count_bytes("user"))

  # list keys (the time_range is optional)
  time_range = ("2024-11-20 10:00:00Z", "2035-11-20 10:00:00Z")
  print(tuple(kv_store.keys(time_range=time_range)))

  # find string keys with a glob pattern
  print(tuple(kv_store.str_keys(glob="use*")))

  # load the 'books' field (partial access)
  books = kv_store.load_field("user", "books")
  assert books == user_data["books"]

  # iterate (descending order)
  for key, value in kv_store.iterate(asc=False):
    pass
```
> Check out the API reference for the [key-value store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/kv/class-Kv.md).


## Depot
The depot data model shares similarities with the [List](https://en.wikipedia.org/wiki/List_(abstract_data_type)) data structure. An
unique identifier (UID) is automatically assigned to a record appended to the store. This record can be retrieved later either by its unique identifier or by its [0-based](https://en.wikipedia.org/wiki/Zero-based_numbering) position in the store.

Example:

```python
import os.path
from datetime import datetime
from jinbase import Jinbase, JINBASE_HOME

user_data = {"id": 42, "name": "alex", "created_at": datetime.now(),
             "photo": b'\x45\xA6\x42\xDF\x69',
             "books": {"sci-fi": ["book 1", "book 2"],
                       "thriller": ["book 3", ["book4"]]}}

db_filename = os.path.join(JINBASE_HOME, "test.db")

with Jinbase(db_filename) as db:
    # append 'user_data' to the depot
    depot_store = db.depot
    uid = depot_store.append(user_data)

    # get 'user_data'
    data = depot_store.get(uid)
    assert data == user_data

    # get the record at position 0 in the depot
    print(depot_store.uid(0))  # prints the UID
    # get the position of a record in the depot
    print(depot_store.position(uid))  # prints the position

    # count total records and bytes
    print(depot_store.count_records())
    print(depot_store.count_bytes(uid))

    # list UIDs (unique identifiers)
    time_range = ("2024-11-20 10:00:00Z", "2035-11-20 10:00:00Z")
    print(tuple(depot_store.uids(time_range=time_range)))

    # load the 'books' field (partial access)
    books = depot_store.load_field(uid, "books")
    assert books == user_data["books"]

    # iterate (descending order)
    for uid, data in depot_store.iterate(asc=False):
        pass
```

> Check out the API reference for the [depot store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/depot/class-Depot.md).

## Queue
The [queue](https://en.wikipedia.org/wiki/Queue_(abstract_data_type)) data model like other stores, is transactional. While this store provides methods to enqueue and dequeue records, there is also `peek_xxx` methods to look at the record at the front or the back of the queue, that is, read it without dequeuing.

Example:
 
```python
import os.path
from datetime import datetime
from jinbase import Jinbase, JINBASE_HOME

user_data = {"id": 42, "name": "alex", "created_at": datetime.now(),
             "photo": b'\x45\xA6\x42\xDF\x69',
             "books": {"sci-fi": ["book 1", "book 2"],
                       "thriller": ["book 3", ["book4"]]}}

db_filename = os.path.join(JINBASE_HOME, "test.db")

with Jinbase(db_filename) as db:
    # enqueue 'user_data'
    queue_store = db.queue
    queue_store.enqueue(user_data)  # returns a UID

    # peek
    data1 = queue_store.peek_front()
    data2 = queue_store.peek_back()
    assert data1 == data2 == user_data

    # dequeue
    data = queue_store.dequeue()
    assert data == user_data

    # we could have dequeued the record inside a transaction
    # to ensure that its processing completed successfully
    # (if it fails, an automatic rollback is performed)
    with db.write_transaction():
        data = queue_store.dequeue()
        # from here, process the data
        ...
```

> Check out the API reference for the [queue store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/queue/class-Queue.md).

## Stack
The [stack](https://en.wikipedia.org/wiki/Stack_(abstract_data_type)) data model like other stores, is transactional. While this store provides methods to push and pop records, there is also a `peek` method to look at the record on top of the stack, that is, read it without popping it from the stack.

```python
import os.path
from datetime import datetime
from jinbase import Jinbase, JINBASE_HOME

user_data = {"id": 42, "name": "alex", "created_at": datetime.now(),
             "photo": b'\x45\xA6\x42\xDF\x69',
             "books": {"sci-fi": ["book 1", "book 2"],
                       "thriller": ["book 3", ["book4"]]}}

db_filename = os.path.join(JINBASE_HOME, "test.db")

with Jinbase(db_filename) as db:
    # push 'user_data' on top of the stack
    stack_store = db.stack
    stack_store.push(user_data)

    # peek
    data = stack_store.peek()
    assert data == user_data

    # pop 'user_data'
    data = stack_store.pop()
    assert data == user_data

    # we could have popped the record inside a transaction
    # to ensure that its processing completed successfully
    # (if it fails, an automatic rollback is performed)
    with db.write_transaction():
        data = stack_store.pop()
        # from here, process the data
        ...
```

> Check out the API reference for the [stack store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/stack/class-Stack.md).

## Relational
As Jinbase uses [SQLite](https://en.wikipedia.org/wiki/SQLite) as its storage engine, it de facto supports the [relational](https://en.wikipedia.org/wiki/Relational_model) data model for which it exposes an interface, [LiteDBC](https://github.com/pyrustic/litedbc), for querying SQLite.

LiteDBC is an SQL interface compliant with the DB-API 2.0 specification described by [PEP 249](https://peps.python.org/pep-0249/), itself wrapping Python's [sqlite3](https://docs.python.org/3/library/sqlite3.html) module for a more intuitive interface and multithreading support by default.

Example:

```python
import os.path
from jinbase import Jinbase, JINBASE_HOME

db_filename = os.path.join(JINBASE_HOME, "test.db")

with Jinbase(db_filename) as db:
    lite_dbc = db.dbc
    with lite_dbc.transaction() as cursor:
        # query the table names that exist in this database
        query = ("SELECT name FROM sqlite_master "
                 "WHERE type='table' AND name NOT LIKE 'sqlite_%'")
        cursor.execute(query)
        # Although the Cursor object already has the traditional "fetchone",
        # "fetchmany" and "fetchall" methods, LiteDBC adds a new lazy "fetch"
        # method for intuitive iteration.
        # Note that 'fetch()' accepts 'limit' and 'buffer_size' as arguments.
        for row in cursor.fetch():
            table_name = row[0]
            print(table_name)

```

> Check out [LiteDBC](https://github.com/pyrustic/litedbc).

# The unified BLOB interface
When for a write operation, the user submits data, whether it is a dictionary, string or integer, Jinbase serializes (except binary data), chunks and stores the data iteratively with the [Paradict](https://github.com/pyrustic/paradict) compact binary data format. Under the hood, these chunks are actually stored as SQLite Binary Large Objects ([BLOBs](https://www.sqlite.org/datatype3.html)). 

SQLite has an impressive capability which is [incremental I/O](https://sqlite.org/c3ref/blob_open.html) for BLOBs. While this capability is designed to target an individual BLOB column in a row, Jinbase extends it to enable incremental reads of record chunks as if they form a single [unified BLOB](#the-unified-blob-interface).

Example:

```python
import os.path
from jinbase import Jinbase, JINBASE_HOME

db_filename = os.path.join(JINBASE_HOME, "test.db")
CHUNK_SIZE = 1  # 1 byte, thus a 5-byte input will have 5 chunks

# The 'chunk_size' can be defined only once when Jinbase creates or opens
# the database for first time. New values for 'chunk_size' will be ignored.
# So, for this example to work, ensure that the 'db_filename' is nonexistent.
with Jinbase(db_filename, chunk_size=CHUNK_SIZE) as db:
    # some binary data
    USER_DATA = b'\x20\x55\xA9\xBC\x69\x42\xD1'  # seven bytes !
    # set the data
    kv_store = db.kv
    kv_store.set("user", USER_DATA)
    # count chunks
    n_chunks = kv_store.count_chunks("user")
    assert n_chunks == len(USER_DATA)  # seven bytes !

    # access the unified blob interface for incremental reads
    with kv_store.open_blob("user") as blob:
        # read the entire unified blob
        data = blob.read()
        assert data == USER_DATA
        assert blob.tell() == len(USER_DATA)  # cursor position
        assert blob.read() == b''
        # move the cursor back to the beginning of the blob
        blob.seek(0)
        # read the first byte
        assert blob.read(1) == bytes([USER_DATA[0]]) 
        # read the last byte
        assert blob[-1] == bytes([USER_DATA[-1]])
        # read a slice
        slice_obj = slice(2, 5)
        assert blob[slice_obj] == USER_DATA[slice_obj]
```

> The unified BLOB interface for incremental reads will only work on Python >=3.11

# Command line interface
Not yet implemented.

# Related projects
- [LiteDBC](https://github.com/pyrustic/litedbc): Lite database connector
- [Paradict](https://github.com/pyrustic/paradict): Streamable multi-format serialization with schema 
- [Asyncpal](https://github.com/pyrustic/asyncpal): Preemptive concurrency and parallelism for sporadic workloads 
- [KvF](https://github.com/pyrustic/kvf): The key-value file format with sections 

# Testing and contributing
Feel free to **open an issue** to report a bug, suggest some changes, show some useful code snippets, or discuss anything related to this project. You can also directly email [me](https://pyrustic.github.io/#contact).

## Setup your development environment
Following are instructions to setup your development environment

```bash
# create and activate a virtual environment
python -m venv venv
source venv/bin/activate

# clone the project then change into its directory
git clone https://github.com/pyrustic/jinbase.git
cd jinbase

# install the package locally (editable mode)
pip install -e .

# run tests
python -m tests

# deactivate the virtual environment
deactivate
```

<p align="right"><a href="#readme">Back to top</a></p>

# Installation
**Jinbase** is **cross-platform**. It is built on [Ubuntu](https://ubuntu.com/download/desktop) and should work on **Python 3.8** or **newer**.

## Create and activate a virtual environment
```bash
python -m venv venv
source venv/bin/activate
```

## Install for the first time

```bash
pip install jinbase
```

## Upgrade the package
```bash
pip install jinbase --upgrade --upgrade-strategy eager
```

## Deactivate the virtual environment
```bash
deactivate
```

<p align="right"><a href="#readme">Back to top</a></p>

# About the author
Hello world, I'm Alex (😎️), a tech enthusiast and the architect of [Pyrustic](https://pyrustic.github.io) ! Feel free to get in touch with [me](https://pyrustic.github.io/#contact) !

<br>
<br>
<br>

[Back to top](#readme)

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/pyrustic/jinbase",
    "name": "jinbase",
    "maintainer": "Pyrustic Architect",
    "docs_url": null,
    "requires_python": ">=3.5",
    "maintainer_email": "rusticalex@yahoo.com",
    "keywords": "pyrustic",
    "author": "Pyrustic Architect",
    "author_email": "rusticalex@yahoo.com",
    "download_url": "https://files.pythonhosted.org/packages/03/d8/0aa5ed0ca3b92604645b1c47ae8c6a4987b55e1f56ff908d52604a038be8/jinbase-0.0.5.tar.gz",
    "platform": null,
    "description": "[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![PyPI package version](https://img.shields.io/pypi/v/jinbase)](https://pypi.org/project/jinbase)\n[![Downloads](https://static.pepy.tech/badge/jinbase)](https://pepy.tech/project/jinbase)\n\n\n<!-- Cover -->\n<div align=\"center\">\n    <img src=\"https://raw.githubusercontent.com/pyrustic/misc/master/assets/jinbase/cover.jpg\" alt=\"Cover image\" width=\"800\">\n    <p align=\"center\">\n        <a href=\"https://commons.wikimedia.org/wiki/File:Rudolf_Reschreiter_Blick_von_der_H%C3%B6llentalangerh%C3%BCtte_zum_H%C3%B6llentalgletscher_und_den_Riffelwandspitzen_1921.jpg\">Rudolf Reschreiter</a>, Public domain, via Wikimedia Commons\n    </p>\n</div>\n\n<!-- Intro Text -->\n# Jinbase\n<b>Multi-model transactional embedded database</b>\n\n\n## Table of contents\n- [Overview](#overview)\n    - [Multiple data models coexisting in a single embedded database](#multiple-data-models-coexisting-in-a-single-embedded-database)\n    - [Support for transactions and complex data of arbitrary size](#support-for-transactions-and-complex-data-of-arbitrary-size)\n    - [Bulk and partial access to records from byte-level to field-level](#bulk-and-partial-access-to-records-from-byte-level-to-field-level)\n    - [Highly configurable database and timestamped records](#highly-configurable-database-and-timestamped-records)\n- [Why use Jinbase ?](#why-use-jinbase-)\n- [Data models and their corresponding storage interfaces](#data-models-and-their-corresponding-storage-interfaces)\n    - [Kv](#kv)\n    - [Depot](#depot)\n    - [Queue](#queue)\n    - [Stack](#stack)\n    - [Relational](#relational)\n- [The unified BLOB interface](#the-unified-blob-interface)\n- [Command line interface](#command-line-interface)\n- [Related projects](#related-projects)\n- [Testing and contributing](#testing-and-contributing)\n- [Installation](#installation)\n\n\n# Overview\n**Jinbase** (pronounced as **/\u02c8d\u0292\u026an\u02ccbe\u026as/**) is a multi-model [transactional](https://en.wikipedia.org/wiki/Database_transaction) [embedded database](https://en.wikipedia.org/wiki/Embedded_database) that uses [SQLite](https://www.sqlite.org/) as storage engine. Its reference implementation is an eponymous lightweight [Python](https://www.python.org/) library available on [PyPI](#installation).\n\n\n## Multiple data models coexisting in a single embedded database\nA single Jinbase database supports **key-value**, **depot**, **queue**, **stack**, and **relational** data models. While a Jinbase file can be populated with multi-model data, depending on the needs, it is quite possible to dedicate a database file to a given model.\n\nFor each of the first four data models, there is a programmatic interface accessible via an eponymous property of a Jinbase instance.\n\n## Support for transactions and complex data of arbitrary size\nHaving SQLite as the storage engine allows Jinbase to benefit from [transactions](https://www.sqlite.org/lang_transaction.html). Jinbase ensures that at the top level, reads and writes on key-value, depot, queue and stack stores are transactional. For user convenience, context managers are exposed to create transactions of different modes.\n\nWhen for a write operation, the user submits data, whether it is a dictionary, string or integer, Jinbase serializes (except binary data), chunks and stores the data iteratively with the [Paradict](https://github.com/pyrustic/paradict) compact binary data format. This then allows for the smooth storage of complex data of [arbitrary size](https://www.sqlite.org/limits.html) with Jinbase.\n\n## Bulk and partial access to records from byte-level to field-level\nJinbase not only offers bulk access to records, but also two levels of partial access granularity.\n\nSQLite has an impressive capability which is [incremental I/O](https://sqlite.org/c3ref/blob_open.html) for [BLOBs](https://www.sqlite.org/datatype3.html). While this capability is designed to target an individual BLOB column in a row, Jinbase extends this so that for each record, incremental reads cover all chunks as if they were a single [unified BLOB](#the-unified-blob-interface).\n\nFor [dictionary](https://en.wikipedia.org/wiki/Associative_array) records only, Jinbase automatically creates and maintains a lightweight index consisting of pointers to root fields, which then allows extracting from an arbitrary record the contents of a field automatically deserialized before being returned.\n\n## Highly configurable database and timestamped records\nJinbase exposes a database connection object to the underlying SQLite storage engine, allowing for sophisticated configuration. The [Paradict](https://github.com/pyrustic/paradict) binary data format used for serializing records also allows for customizing data types via a `paradict.TypeRef` object.\n\nEach record stored in a key-value, depot, queue, or stack store is automatically timestamped. This allows the user to provide a `time_range` tuple when querying records. The precision of the timestamp, which defaults to milliseconds, can also be configured.\n\n\n\n# Why use Jinbase ?\nJinbase implements persistence for familiar data models whose stores coexist in a single file with an intuitive programmatic interface. Supported [data types](https://github.com/pyrustic/paradict) range from simple to complex and of [arbitrary size](https://www.sqlite.org/limits.html).\n\nFor convenience, all Jinbase-related tables are prefixed with `jinbase_`, allowing the user to define their own tables and interact with them as they would with a regular SQLite database. \n\nThanks to its multi-model coexistence capability, Jinbase can be used to open legacy SQLite databases to add four useful data models (key-value, depot, queue, and stack).\n\nAll this makes Jinbase relevant from prototype to production stages of software development of various sizes and scopes. \n\nFollowing are few of the most obvious use cases:\n\n- Storing user preferences\n- Persisting session data before exit\n- Order-based processing of data streams\n- Exposing data for other processes\n- Upgrading legacy SQLite files with new data models\n- Bespoke data persistence solution\n\n# Data models and their corresponding storage interfaces\nFollowing subsections discuss data models and their corresponding storage interfaces.\n\n## Kv\nThe [key-value](https://en.wikipedia.org/wiki/Key%E2%80%93value_database) data model associates to a string or an integer key, a value that is serializable with the [Paradict](https://github.com/pyrustic/paradict) binary data format. \n\nString keys can be searched with a [glob](https://en.wikipedia.org/wiki/Glob_(programming)) pattern and integer keys can be searched within a range of numbers. Since records are automatically timestamped, a `time_range` tuple can be provided by the user to search for them as well as keys.\n\nExample:\n\n```python\nimport os.path\nfrom datetime import datetime\nfrom jinbase import Jinbase, JINBASE_HOME\n\nuser_data = {\"id\": 42, \"name\": \"alex\", \"created_at\": datetime.now(),\n             \"photo\": b'\\x45\\xA6\\x42\\xDF\\x69',\n             \"books\": {\"sci-fi\": [\"book 1\", \"book 2\"],\n                       \"thriller\": [\"book 3\", [\"book4\"]]}}\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\n\nwith Jinbase(db_filename) as db:\n  # set 'user'\n  kv_store = db.kv\n  kv_store.set(\"user\", user_data)  # returns a UID\n\n  # get 'user'\n  data = kv_store.get(\"user\")\n  assert data == user_data\n\n  # count total records and bytes\n  print(kv_store.count_records())\n  print(kv_store.count_bytes(\"user\"))\n\n  # list keys (the time_range is optional)\n  time_range = (\"2024-11-20 10:00:00Z\", \"2035-11-20 10:00:00Z\")\n  print(tuple(kv_store.keys(time_range=time_range)))\n\n  # find string keys with a glob pattern\n  print(tuple(kv_store.str_keys(glob=\"use*\")))\n\n  # load the 'books' field (partial access)\n  books = kv_store.load_field(\"user\", \"books\")\n  assert books == user_data[\"books\"]\n\n  # iterate (descending order)\n  for key, value in kv_store.iterate(asc=False):\n    pass\n```\n> Check out the API reference for the [key-value store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/kv/class-Kv.md).\n\n\n## Depot\nThe depot data model shares similarities with the [List](https://en.wikipedia.org/wiki/List_(abstract_data_type)) data structure. An\nunique identifier (UID) is automatically assigned to a record appended to the store. This record can be retrieved later either by its unique identifier or by its [0-based](https://en.wikipedia.org/wiki/Zero-based_numbering) position in the store.\n\nExample:\n\n```python\nimport os.path\nfrom datetime import datetime\nfrom jinbase import Jinbase, JINBASE_HOME\n\nuser_data = {\"id\": 42, \"name\": \"alex\", \"created_at\": datetime.now(),\n             \"photo\": b'\\x45\\xA6\\x42\\xDF\\x69',\n             \"books\": {\"sci-fi\": [\"book 1\", \"book 2\"],\n                       \"thriller\": [\"book 3\", [\"book4\"]]}}\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\n\nwith Jinbase(db_filename) as db:\n    # append 'user_data' to the depot\n    depot_store = db.depot\n    uid = depot_store.append(user_data)\n\n    # get 'user_data'\n    data = depot_store.get(uid)\n    assert data == user_data\n\n    # get the record at position 0 in the depot\n    print(depot_store.uid(0))  # prints the UID\n    # get the position of a record in the depot\n    print(depot_store.position(uid))  # prints the position\n\n    # count total records and bytes\n    print(depot_store.count_records())\n    print(depot_store.count_bytes(uid))\n\n    # list UIDs (unique identifiers)\n    time_range = (\"2024-11-20 10:00:00Z\", \"2035-11-20 10:00:00Z\")\n    print(tuple(depot_store.uids(time_range=time_range)))\n\n    # load the 'books' field (partial access)\n    books = depot_store.load_field(uid, \"books\")\n    assert books == user_data[\"books\"]\n\n    # iterate (descending order)\n    for uid, data in depot_store.iterate(asc=False):\n        pass\n```\n\n> Check out the API reference for the [depot store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/depot/class-Depot.md).\n\n## Queue\nThe [queue](https://en.wikipedia.org/wiki/Queue_(abstract_data_type)) data model like other stores, is transactional. While this store provides methods to enqueue and dequeue records, there is also `peek_xxx` methods to look at the record at the front or the back of the queue, that is, read it without dequeuing.\n\nExample:\n \n```python\nimport os.path\nfrom datetime import datetime\nfrom jinbase import Jinbase, JINBASE_HOME\n\nuser_data = {\"id\": 42, \"name\": \"alex\", \"created_at\": datetime.now(),\n             \"photo\": b'\\x45\\xA6\\x42\\xDF\\x69',\n             \"books\": {\"sci-fi\": [\"book 1\", \"book 2\"],\n                       \"thriller\": [\"book 3\", [\"book4\"]]}}\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\n\nwith Jinbase(db_filename) as db:\n    # enqueue 'user_data'\n    queue_store = db.queue\n    queue_store.enqueue(user_data)  # returns a UID\n\n    # peek\n    data1 = queue_store.peek_front()\n    data2 = queue_store.peek_back()\n    assert data1 == data2 == user_data\n\n    # dequeue\n    data = queue_store.dequeue()\n    assert data == user_data\n\n    # we could have dequeued the record inside a transaction\n    # to ensure that its processing completed successfully\n    # (if it fails, an automatic rollback is performed)\n    with db.write_transaction():\n        data = queue_store.dequeue()\n        # from here, process the data\n        ...\n```\n\n> Check out the API reference for the [queue store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/queue/class-Queue.md).\n\n## Stack\nThe [stack](https://en.wikipedia.org/wiki/Stack_(abstract_data_type)) data model like other stores, is transactional. While this store provides methods to push and pop records, there is also a `peek` method to look at the record on top of the stack, that is, read it without popping it from the stack.\n\n```python\nimport os.path\nfrom datetime import datetime\nfrom jinbase import Jinbase, JINBASE_HOME\n\nuser_data = {\"id\": 42, \"name\": \"alex\", \"created_at\": datetime.now(),\n             \"photo\": b'\\x45\\xA6\\x42\\xDF\\x69',\n             \"books\": {\"sci-fi\": [\"book 1\", \"book 2\"],\n                       \"thriller\": [\"book 3\", [\"book4\"]]}}\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\n\nwith Jinbase(db_filename) as db:\n    # push 'user_data' on top of the stack\n    stack_store = db.stack\n    stack_store.push(user_data)\n\n    # peek\n    data = stack_store.peek()\n    assert data == user_data\n\n    # pop 'user_data'\n    data = stack_store.pop()\n    assert data == user_data\n\n    # we could have popped the record inside a transaction\n    # to ensure that its processing completed successfully\n    # (if it fails, an automatic rollback is performed)\n    with db.write_transaction():\n        data = stack_store.pop()\n        # from here, process the data\n        ...\n```\n\n> Check out the API reference for the [stack store](https://github.com/pyrustic/jinbase/blob/master/docs/api/modules/jinbase/store/stack/class-Stack.md).\n\n## Relational\nAs Jinbase uses [SQLite](https://en.wikipedia.org/wiki/SQLite) as its storage engine, it de facto supports the [relational](https://en.wikipedia.org/wiki/Relational_model) data model for which it exposes an interface, [LiteDBC](https://github.com/pyrustic/litedbc), for querying SQLite.\n\nLiteDBC is an SQL interface compliant with the DB-API 2.0 specification described by [PEP 249](https://peps.python.org/pep-0249/), itself wrapping Python's [sqlite3](https://docs.python.org/3/library/sqlite3.html) module for a more intuitive interface and multithreading support by default.\n\nExample:\n\n```python\nimport os.path\nfrom jinbase import Jinbase, JINBASE_HOME\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\n\nwith Jinbase(db_filename) as db:\n    lite_dbc = db.dbc\n    with lite_dbc.transaction() as cursor:\n        # query the table names that exist in this database\n        query = (\"SELECT name FROM sqlite_master \"\n                 \"WHERE type='table' AND name NOT LIKE 'sqlite_%'\")\n        cursor.execute(query)\n        # Although the Cursor object already has the traditional \"fetchone\",\n        # \"fetchmany\" and \"fetchall\" methods, LiteDBC adds a new lazy \"fetch\"\n        # method for intuitive iteration.\n        # Note that 'fetch()' accepts 'limit' and 'buffer_size' as arguments.\n        for row in cursor.fetch():\n            table_name = row[0]\n            print(table_name)\n\n```\n\n> Check out [LiteDBC](https://github.com/pyrustic/litedbc).\n\n# The unified BLOB interface\nWhen for a write operation, the user submits data, whether it is a dictionary, string or integer, Jinbase serializes (except binary data), chunks and stores the data iteratively with the [Paradict](https://github.com/pyrustic/paradict) compact binary data format. Under the hood, these chunks are actually stored as SQLite Binary Large Objects ([BLOBs](https://www.sqlite.org/datatype3.html)). \n\nSQLite has an impressive capability which is [incremental I/O](https://sqlite.org/c3ref/blob_open.html) for BLOBs. While this capability is designed to target an individual BLOB column in a row, Jinbase extends it to enable incremental reads of record chunks as if they form a single [unified BLOB](#the-unified-blob-interface).\n\nExample:\n\n```python\nimport os.path\nfrom jinbase import Jinbase, JINBASE_HOME\n\ndb_filename = os.path.join(JINBASE_HOME, \"test.db\")\nCHUNK_SIZE = 1  # 1 byte, thus a 5-byte input will have 5 chunks\n\n# The 'chunk_size' can be defined only once when Jinbase creates or opens\n# the database for first time. New values for 'chunk_size' will be ignored.\n# So, for this example to work, ensure that the 'db_filename' is nonexistent.\nwith Jinbase(db_filename, chunk_size=CHUNK_SIZE) as db:\n    # some binary data\n    USER_DATA = b'\\x20\\x55\\xA9\\xBC\\x69\\x42\\xD1'  # seven bytes !\n    # set the data\n    kv_store = db.kv\n    kv_store.set(\"user\", USER_DATA)\n    # count chunks\n    n_chunks = kv_store.count_chunks(\"user\")\n    assert n_chunks == len(USER_DATA)  # seven bytes !\n\n    # access the unified blob interface for incremental reads\n    with kv_store.open_blob(\"user\") as blob:\n        # read the entire unified blob\n        data = blob.read()\n        assert data == USER_DATA\n        assert blob.tell() == len(USER_DATA)  # cursor position\n        assert blob.read() == b''\n        # move the cursor back to the beginning of the blob\n        blob.seek(0)\n        # read the first byte\n        assert blob.read(1) == bytes([USER_DATA[0]]) \n        # read the last byte\n        assert blob[-1] == bytes([USER_DATA[-1]])\n        # read a slice\n        slice_obj = slice(2, 5)\n        assert blob[slice_obj] == USER_DATA[slice_obj]\n```\n\n> The unified BLOB interface for incremental reads will only work on Python >=3.11\n\n# Command line interface\nNot yet implemented.\n\n# Related projects\n- [LiteDBC](https://github.com/pyrustic/litedbc): Lite database connector\n- [Paradict](https://github.com/pyrustic/paradict): Streamable multi-format serialization with schema \n- [Asyncpal](https://github.com/pyrustic/asyncpal): Preemptive concurrency and parallelism for sporadic workloads \n- [KvF](https://github.com/pyrustic/kvf): The key-value file format with sections \n\n# Testing and contributing\nFeel free to **open an issue** to report a bug, suggest some changes, show some useful code snippets, or discuss anything related to this project. You can also directly email [me](https://pyrustic.github.io/#contact).\n\n## Setup your development environment\nFollowing are instructions to setup your development environment\n\n```bash\n# create and activate a virtual environment\npython -m venv venv\nsource venv/bin/activate\n\n# clone the project then change into its directory\ngit clone https://github.com/pyrustic/jinbase.git\ncd jinbase\n\n# install the package locally (editable mode)\npip install -e .\n\n# run tests\npython -m tests\n\n# deactivate the virtual environment\ndeactivate\n```\n\n<p align=\"right\"><a href=\"#readme\">Back to top</a></p>\n\n# Installation\n**Jinbase** is **cross-platform**. It is built on [Ubuntu](https://ubuntu.com/download/desktop) and should work on **Python 3.8** or **newer**.\n\n## Create and activate a virtual environment\n```bash\npython -m venv venv\nsource venv/bin/activate\n```\n\n## Install for the first time\n\n```bash\npip install jinbase\n```\n\n## Upgrade the package\n```bash\npip install jinbase --upgrade --upgrade-strategy eager\n```\n\n## Deactivate the virtual environment\n```bash\ndeactivate\n```\n\n<p align=\"right\"><a href=\"#readme\">Back to top</a></p>\n\n# About the author\nHello world, I'm Alex (\ud83d\ude0e\ufe0f), a tech enthusiast and the architect of [Pyrustic](https://pyrustic.github.io) ! Feel free to get in touch with [me](https://pyrustic.github.io/#contact) !\n\n<br>\n<br>\n<br>\n\n[Back to top](#readme)\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Multi-model transactional embedded database",
    "version": "0.0.5",
    "project_urls": {
        "Homepage": "https://github.com/pyrustic/jinbase"
    },
    "split_keywords": [
        "pyrustic"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "a7b8a94bd44cfb47776d52ee74999488ec1badc8e7c3e944cf6c535069caec51",
                "md5": "e50a8e4ee3d39335432c27107a9786b6",
                "sha256": "bb1d1799015b4ae2279342686edb9d86957148e9c297ed72e72732660c26f44d"
            },
            "downloads": -1,
            "filename": "jinbase-0.0.5-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "e50a8e4ee3d39335432c27107a9786b6",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.5",
            "size": 27246,
            "upload_time": "2024-12-10T20:54:28",
            "upload_time_iso_8601": "2024-12-10T20:54:28.529275Z",
            "url": "https://files.pythonhosted.org/packages/a7/b8/a94bd44cfb47776d52ee74999488ec1badc8e7c3e944cf6c535069caec51/jinbase-0.0.5-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "03d80aa5ed0ca3b92604645b1c47ae8c6a4987b55e1f56ff908d52604a038be8",
                "md5": "c2704ec1f9df9e4a5a0dc45042c08f6a",
                "sha256": "2f47dca30f4568f49326d832952a8f5d3ed5e7328f3f2b9357ff042fb9a8806e"
            },
            "downloads": -1,
            "filename": "jinbase-0.0.5.tar.gz",
            "has_sig": false,
            "md5_digest": "c2704ec1f9df9e4a5a0dc45042c08f6a",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.5",
            "size": 38027,
            "upload_time": "2024-12-10T20:54:32",
            "upload_time_iso_8601": "2024-12-10T20:54:32.181761Z",
            "url": "https://files.pythonhosted.org/packages/03/d8/0aa5ed0ca3b92604645b1c47ae8c6a4987b55e1f56ff908d52604a038be8/jinbase-0.0.5.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-12-10 20:54:32",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "pyrustic",
    "github_project": "jinbase",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "lcname": "jinbase"
}
        
Elapsed time: 0.36625s