[](https://pypi.org/project/pyrex-rocksdb/)
[](https://img.shields.io/pypi/pyversions/pyrex-rocksdb/)
# Installation
# pyrex-rocksdb
A python wrapper for the original (C++) version of RocksDB.
## Installation
For linux systems, wheels are provided and can be installed from pypi using:
```bash
pip install pyrex-rocksdb
```
For Windows and MacOS I have built an earlier version of the library.
I will re-build once I include certain other important features in the API that are not yet implemented.
## Motivation
This library is intended for providing a fast, write-optimized, in-process key value (KV) store in python. Therefore the "big brothers" of the database are the likes of MongoDB and Cassandra. The difference is that you don't need a separate server to run this (hence "in-process") and it is designed to be fairly portable.
RocksDB, which is the underlying storage engine of this database, is an LSM-tree engine. An LSM-tree is different from the ballanced tree index databases (e.g., [B-tree](https://en.wikipedia.org/wiki/B-tree)/ and [B+tree](https://en.wikipedia.org/wiki/B%2B_tree) databases). LSM-tree databases offer very high write throughputs and better space efficiency. See more about the motivation for LSM-tree databases (and RocksDB in particular) in [this talk](https://www.youtube.com/watch?v=V_C-T5S-w8g).
### LSM-tree + SSTable engine basics
To understand where `pyrex` provides efficiency gains, it is important to understand some basics about the underlying `RocksDB` engine.
RocksDB and LevelDB are **key-value stores** with a **Log-Structured Merge-tree (LSM-tree)** architecture.
The key components of LSM-tree architectures are
* A **MemTable** that stores in-memory sorted data
* A set of **Sorted-String tables (SSTables)** which are immutable sorted files on disk where data from the MemTable is flushed
* The process of **Compaction**, which is a background process that merges the SSTables to remove redundant data and keep read performance high.
In such databases, fast writes create many small, sorted data files called SSTables. To prevent reads from slowing down by checking too many files, a background process called compaction merges these SSTables together. This process organizes the data into levels, where newer, overlapping files sit in Level 0 and are progressively merged into higher levels (Level 1, Level 2, etc.). Each higher level contains larger, non-overlapping files, which ensures that finding a key remains efficient and old data is purged to save space. There are several optimizations and configurations possible for these processes (configurability and "pluggability" are commonly cited RocksDB advantages).
However the main big advantage of RocksDB over LevelDB is its **multi-threaded compaction support** (LevelDB supports only single threaded compaction, which comes with significant performance limitations).
There are several other configurability advantages RocksDB offers over LevelDB. For a more elaborate enumaration of RocksDB advantages please refer to the [RocksDB wiki](https://github.com/facebook/rocksdb/wiki/Features-Not-in-LevelDB).
Not all are currently supported by the `pyrex` API, but I'm working on supporting more of them. Feel free to open an issue if there is a feature you want to see (or open a pull request).
## Example usage:
Here is a simple example showing the usage of put/get in the DB:
```python
import pyrex
import os
import shutil
DB_PATH = "./test_rocksdb_minimal"
with pyrex.PyRocksDB(DB_PATH) as db:
db.put(b"my_key", b"my_value")
retrieved_value = db.get(b"my_key")
print(f"Retrieved: {retrieved_value.decode()}") # Output: Retrieved: my_value
```
for more examples check the relevant folder and the documentation.
## Installation
<details>
<summary>Note on CICD</summary>
The wheels provided are not completely platform-independent at the moment.
I heavily rely on github actions to develop since I don't own mac or windows machines.
The CICD workflow for package builds is under development A windows/macos/linux build was successful, but further development is needed.
</details>
## Benchmarks
`Pyrex` was benchmarked against [plyvel](https://github.com/wbolster/plyvel) and [lmdb](https://github.com/jnwatson/py-lmdb/) (which is based on a B+tree -- based architecture and relies on OS's block cache).
Initial benchmarks are promissing and to be reported soon.
Raw data
{
"_id": null,
"home_page": null,
"name": "pyrex-rocksdb",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "rocksdb, database, key-value, pybind11",
"author": null,
"author_email": "Charilaos Mylonas <mylonas.charilaos@gmail.com>",
"download_url": null,
"platform": null,
"description": "\n[](https://pypi.org/project/pyrex-rocksdb/)\n[](https://img.shields.io/pypi/pyversions/pyrex-rocksdb/)\n\n\n\n\n# Installation\n\n\n# pyrex-rocksdb\nA python wrapper for the original (C++) version of RocksDB.\n\n## Installation\n\nFor linux systems, wheels are provided and can be installed from pypi using:\n\n```bash\npip install pyrex-rocksdb\n```\n\nFor Windows and MacOS I have built an earlier version of the library.\nI will re-build once I include certain other important features in the API that are not yet implemented.\n\n\n\n## Motivation\n\nThis library is intended for providing a fast, write-optimized, in-process key value (KV) store in python. Therefore the \"big brothers\" of the database are the likes of MongoDB and Cassandra. The difference is that you don't need a separate server to run this (hence \"in-process\") and it is designed to be fairly portable. \n\nRocksDB, which is the underlying storage engine of this database, is an LSM-tree engine. An LSM-tree is different from the ballanced tree index databases (e.g., [B-tree](https://en.wikipedia.org/wiki/B-tree)/ and [B+tree](https://en.wikipedia.org/wiki/B%2B_tree) databases). LSM-tree databases offer very high write throughputs and better space efficiency. See more about the motivation for LSM-tree databases (and RocksDB in particular) in [this talk](https://www.youtube.com/watch?v=V_C-T5S-w8g).\n\n### LSM-tree + SSTable engine basics\nTo understand where `pyrex` provides efficiency gains, it is important to understand some basics about the underlying `RocksDB` engine. \n\nRocksDB and LevelDB are **key-value stores** with a **Log-Structured Merge-tree (LSM-tree)** architecture. \n\nThe key components of LSM-tree architectures are \n* A **MemTable** that stores in-memory sorted data\n* A set of **Sorted-String tables (SSTables)** which are immutable sorted files on disk where data from the MemTable is flushed\n* The process of **Compaction**, which is a background process that merges the SSTables to remove redundant data and keep read performance high.\n\nIn such databases, fast writes create many small, sorted data files called SSTables. To prevent reads from slowing down by checking too many files, a background process called compaction merges these SSTables together. This process organizes the data into levels, where newer, overlapping files sit in Level 0 and are progressively merged into higher levels (Level 1, Level 2, etc.). Each higher level contains larger, non-overlapping files, which ensures that finding a key remains efficient and old data is purged to save space. There are several optimizations and configurations possible for these processes (configurability and \"pluggability\" are commonly cited RocksDB advantages). \n\nHowever the main big advantage of RocksDB over LevelDB is its **multi-threaded compaction support** (LevelDB supports only single threaded compaction, which comes with significant performance limitations). \nThere are several other configurability advantages RocksDB offers over LevelDB. For a more elaborate enumaration of RocksDB advantages please refer to the [RocksDB wiki](https://github.com/facebook/rocksdb/wiki/Features-Not-in-LevelDB). \n\nNot all are currently supported by the `pyrex` API, but I'm working on supporting more of them. Feel free to open an issue if there is a feature you want to see (or open a pull request).\n\n\n## Example usage:\n\nHere is a simple example showing the usage of put/get in the DB:\n\n```python\nimport pyrex\nimport os\nimport shutil\n\nDB_PATH = \"./test_rocksdb_minimal\"\n\nwith pyrex.PyRocksDB(DB_PATH) as db:\n db.put(b\"my_key\", b\"my_value\")\n retrieved_value = db.get(b\"my_key\")\n\nprint(f\"Retrieved: {retrieved_value.decode()}\") # Output: Retrieved: my_value\n\n```\n\nfor more examples check the relevant folder and the documentation.\n\n## Installation\n\n<details>\n <summary>Note on CICD</summary>\nThe wheels provided are not completely platform-independent at the moment. \nI heavily rely on github actions to develop since I don't own mac or windows machines.\nThe CICD workflow for package builds is under development A windows/macos/linux build was successful, but further development is needed.\n</details>\n\n## Benchmarks\n\n`Pyrex` was benchmarked against [plyvel](https://github.com/wbolster/plyvel) and [lmdb](https://github.com/jnwatson/py-lmdb/) (which is based on a B+tree -- based architecture and relies on OS's block cache). \n\nInitial benchmarks are promissing and to be reported soon.\n\n\n",
"bugtrack_url": null,
"license": null,
"summary": "A fast RocksDB wrapper for Python using pybind11.",
"version": "0.1.4",
"project_urls": {
"Homepage": "https://github.com/mylonasc/pyrex",
"Repository": "https://github.com/mylonasc/pyrex"
},
"split_keywords": [
"rocksdb",
" database",
" key-value",
" pybind11"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "baffe8163b8307f3821f7ec6db985bba5c0a495782a9c5479b7828d1d5575d9d",
"md5": "a3c370ebdd38ae480756ba6016022792",
"sha256": "330a865c5808501a95ae52fa6589c24beccbe4430a90518e5437dbb4ec318550"
},
"downloads": -1,
"filename": "pyrex_rocksdb-0.1.4-cp310-cp310-macosx_14_0_arm64.whl",
"has_sig": false,
"md5_digest": "a3c370ebdd38ae480756ba6016022792",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.8",
"size": 3910502,
"upload_time": "2025-08-03T21:38:11",
"upload_time_iso_8601": "2025-08-03T21:38:11.701238Z",
"url": "https://files.pythonhosted.org/packages/ba/ff/e8163b8307f3821f7ec6db985bba5c0a495782a9c5479b7828d1d5575d9d/pyrex_rocksdb-0.1.4-cp310-cp310-macosx_14_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "766deff6c307b02fbd321fcac2946f2a7654cf94caf21b1c7f8aca3b01d7e1ab",
"md5": "ce4cdaccf40060cf2e49724b7eb74b29",
"sha256": "b16fed2dad8990d279ccdd9a7e26beb8798eb16b4c71d9b5d947914c777578f3"
},
"downloads": -1,
"filename": "pyrex_rocksdb-0.1.4-cp311-cp311-macosx_14_0_arm64.whl",
"has_sig": false,
"md5_digest": "ce4cdaccf40060cf2e49724b7eb74b29",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.8",
"size": 3911732,
"upload_time": "2025-08-03T21:38:13",
"upload_time_iso_8601": "2025-08-03T21:38:13.724324Z",
"url": "https://files.pythonhosted.org/packages/76/6d/eff6c307b02fbd321fcac2946f2a7654cf94caf21b1c7f8aca3b01d7e1ab/pyrex_rocksdb-0.1.4-cp311-cp311-macosx_14_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "b2f664f5680ca1810289fbb4941921f89b92396d062c39ef88dc7f72f4fa37a5",
"md5": "b89bde8bc27d89d289fb9487bff9eb4b",
"sha256": "415695e697ee253b3e72af6e4062d81d6782bcf81c1b13fcf4401e8266ea5d78"
},
"downloads": -1,
"filename": "pyrex_rocksdb-0.1.4-cp312-cp312-macosx_14_0_arm64.whl",
"has_sig": false,
"md5_digest": "b89bde8bc27d89d289fb9487bff9eb4b",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.8",
"size": 3915423,
"upload_time": "2025-08-03T21:38:15",
"upload_time_iso_8601": "2025-08-03T21:38:15.500342Z",
"url": "https://files.pythonhosted.org/packages/b2/f6/64f5680ca1810289fbb4941921f89b92396d062c39ef88dc7f72f4fa37a5/pyrex_rocksdb-0.1.4-cp312-cp312-macosx_14_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "55f5f9a10c3375c10dd7036b324b6bca93a30c2c4074192e22f258aa3e54e4ca",
"md5": "f14225a25eaea79f907d105909d3b3c3",
"sha256": "deec9d0c3b0308666ead02ac70dd0568519e7a35bc32337046a8d017cce51ed9"
},
"downloads": -1,
"filename": "pyrex_rocksdb-0.1.4-cp39-cp39-macosx_14_0_arm64.whl",
"has_sig": false,
"md5_digest": "f14225a25eaea79f907d105909d3b3c3",
"packagetype": "bdist_wheel",
"python_version": "cp39",
"requires_python": ">=3.8",
"size": 3910631,
"upload_time": "2025-08-03T21:38:16",
"upload_time_iso_8601": "2025-08-03T21:38:16.809624Z",
"url": "https://files.pythonhosted.org/packages/55/f5/f9a10c3375c10dd7036b324b6bca93a30c2c4074192e22f258aa3e54e4ca/pyrex_rocksdb-0.1.4-cp39-cp39-macosx_14_0_arm64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-03 21:38:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mylonasc",
"github_project": "pyrex",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pyrex-rocksdb"
}
JSON
