# nosqlpy - Async, NoSQL, MQL-style embedded document store for Python
<p align="center">
<img src="logo.png" alt="nosqlpy logo" width="250">
</p>
⚠️ Warning: This is the alpha version
**nosqlpy** is a lightweight, async-first, file-backed document database for Python that speaks a **MongoDB-like Query Language (MQL)**.
It’s designed for small-to-medium apps, dev tooling, prototyping, CLI tools, and apps that want a simple embedded DB with **async/await**, durability options, indexes, and a clean API - without running a separate database server.
- ✅ Async-first
- ✅ Mongo-like queries
- ✅ Append-only op-log
- ✅ Secondary equality indexes
- ✅ Compaction & durability modes
---
# Table of contents
- [nosqlpy - Async, NoSQL, MQL-style embedded document store for Python](#nosqlpy---async-nosql-mql-style-embedded-document-store-for-python)
- [Table of contents](#table-of-contents)
- [Why nosqlpy?](#why-nosqlpy)
- [Key features](#key-features)
- [Installation](#installation)
- [Quickstart](#quickstart)
- [Core concepts \& API reference](#core-concepts--api-reference)
- [Database \& Collection](#database--collection)
- [Common operations](#common-operations)
- [List of operations](#list-of-operations)
- [Examples](#examples)
- [Query language (MQL subset)](#query-language-mql-subset)
- [Update operators](#update-operators)
- [Indexing](#indexing)
- [Compaction \& durability](#compaction--durability)
- [Performance tips](#performance-tips)
- [License](#license)
- [Keywords](#keywords)
---
# Why nosqlpy?
Many small Python apps need an embedded document store that:
* Is **async-native** (fits FastAPI / aiohttp / asyncio apps)
* Uses a **familiar query language** (MongoDB-style filters)
* Provides **durable writes** and **safe recovery**
* Enables **fast reads** using secondary indexes
Most small DB options are either synchronous (TinyDB), or require wrapping sync code into thread pools. `nosqlpy` is built async-first and offers MQL-style queries, an append-only op-log for safe durability, optional indexes, and compaction for production-ish workloads - all in a single small dependency set.
---
# Key features
* Async API (`async/await`) - designed for asyncio apps
* MongoDB-style query language (subset): `$and`, `$or`, `$not`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`, simple dot-notation for nested fields
* Update operators: `$set`, `$unset`, `$inc`, `$push`, `$pull`
* Append-only operation log (oplog) with replayable history
* Durability modes: `fast`, `safe`, `durable` (fsync)
* Secondary equality indexes (create and use for fast `field == value`)
* Compaction (background-friendly pattern) to shrink logs and produce snapshots
* Bulk ops: `insert_many`, `update_many`, `delete_many`
* find support: `projection`, `sort`, `skip`, `limit`, `find_one`
* TinyDB migration helper / compatibility shim (to ease switching)
---
# Installation
```bash
# install from PyPI (when published)
pip install nosqlpy
# or during development (from repo)
git clone https://github.com/viehoang/nosqlpy.git
cd nosqlpy
pip install -e ".[dev]"
```
Dependencies: `aiofiles` (async file I/O). Dev/test extras include `pytest`, `pytest-asyncio`.
---
# Quickstart
```python
import asyncio
from nosqlpy import Database
async def main():
db = Database("data_dir", durability="safe")
users = await db.collection("users")
# insert
uid = await users.insert_one({
"name": "alice",
"age": 30})
print("inserted id:", uid)
# find (Mongo-like query)
results = await users.find(
{"age": {"$gte": 25},
"name": "alice"})
print("found:", results)
# update operators
await users.update_many(
{"name": "alice"},
{"$inc": {"age": 1},
"$push": {"tags": "admin"}})
# create index on email for fast equality lookups
await users.create_index("email")
# compact (shrink the op-log)
await users.compact()
asyncio.run(main())
```
---
# Core concepts & API reference
> **Collection** - a single named dataset backed by an append-oplog file.
>
> **Document** - a JSON-like dict with an `_id` field (string) as the primary key.
>
> **Op-log** - append-only lines describing `insert`, `update`, `replace`, `delete` operations.
All calls are `async`.
## Database & Collection
```python
from nosqlpy import Database
db = Database("my_data_dir", durability="safe")
users = await db.collection("users")
```
## Common operations
### List of operations
| Operator | Description |
|----------|-------------|
| `open` | Asynchronously open and load the collection. |
| `insert_one` | Insert a single document into the collection. |
| `insert_many` | Insert multiple documents into the collection. |
| `find` | Find documents matching the query, with options for projection, sort, skip, and limit. |
| `find_one` | Find a single document matching the query. |
| `count` | Count the number of documents matching the query. |
| `delete_one` | Delete a single document matching the query. |
| `delete_many` | Delete multiple documents matching the query. |
| `update_one` | Update a single document matching the query, with optional upsert. |
| `update_many` | Update multiple documents matching the query, with optional upsert (upserts one if no matches). |
| `replace_one` | Replace a single document matching the query, with optional upsert. |
| `create_index` | Create a secondary hash index on a field. |
| `drop_index` | Drop the index on a field. |
| `compact` | Compact the op-log by replacing it with current state as inserts. |
### Examples
```python
# insert one
_id = await users.insert_one({"name": "Bob", "age": 22})
# insert many
ids = await users.insert_many([{"name":"A"},{"name":"B"}])
# find (MQL filter)
docs = await users.find(
{"age": {"$gte": 18}},
projection=["name","age"],
sort=[("age", -1)],
skip=0,
limit=50)
# find one
doc = await users.find_one({"name": "Bob"})
# count
n = await users.count({"age": {"$gte": 30}})
# update many (supports $set/$inc/$push/$pull)
updated = await users.update_many(
{"name": "Bob"},
{"$inc": {"age": 1}})
# replace one
ok = await users.replace_one(
{"name": "Bob"},
{"name": "Robert", "age": 23})
# delete many
deleted = await users.delete_many({"age": {"$lt": 18}})
# compact (rewrite op-log as current-state snapshot)
await users.compact()
```
## Query language (MQL subset)
Supported query operators (subset):
* Comparison: `$eq` (or plain value), `$ne`, `$gt`, `$gte`, `$lt`, `$lte`
* Membership: `$in`, `$nin`
* Existence: `$exists`
* Logical: `$and`, `$or`, `$not`
* Dot-notation: `"user.age"` for nested fields
Examples:
```python
# age >= 30 AND (country == "US" OR country == "JP")
q = {"$and": [{"age": {"$gte": 30}}, {"$or": [{"country": "US"}, {"country": "JP"}]}]}
# membership
q2 = {"status": {"$in": ["active", "pending"]}}
# nested field
q3 = {"profile.email": {"$exists": True}}
```
## Update operators
Supported update operators:
* `$set`: set field(s) to value
* `$unset`: remove field(s)
* `$inc`: increment numeric field
* `$push`: append to array field
* `$pull`: remove value(s) from array field
Example:
```python
await users.update_many(
{"_id": some_id},
{
"$set": {"name": "Alice"},
"$inc": {"score": 10}
})
```
## Indexing
Equality secondary index (hash-based) to accelerate queries like `{ "email": "a@x.com" }`:
```python
await users.create_index("email")
```
Notes:
* Indexes are in-memory by default and rebuilt on create; consider snapshotting indexes for very large DBs (TODO/roadmap).
* Planner currently optimizes single-field equality queries; range indexes are a planned feature.
## Compaction & durability
* **Op-log** (append-only) is crash friendly: each write is a single line append.
* Durability modes:
* `fast`: minimal overhead, no explicit flush (best throughput, less durable)
* `safe`: flush after write (`file.flush()`) - good compromise
* `durable`: `fsync()` after each op (strong durability, slower)
Compaction rewrites the current state as a compact snapshot (series of `insert` ops) and atomically replaces the op-log. For large DBs we recommend:
* Run `compact()` occasionally (or use background segmented compaction; see Roadmap).
* Use `durable` for critical writes, `safe` for normal persistence.
---
# Performance tips
* Use indexes for frequent equality lookups.
* Use `durability="safe"` for most apps; switch to `durable` only if you need fsync-level guarantees on every op.
* For very large data (>100k docs), enable segmented compaction or index snapshots (planned).
---
# License
`nosqlpy` is released under the **MIT License**. See `LICENSE` for details.
---
# Keywords
`nosqlpy`, `nosqlite`, `nosql lite`, `async nosql`, `python async database`, `embedded document store`, `mongo query language python`, `mql python`, `asyncio database`, `append log database`, `python nosql lite`, `tinydb alternative`, `aiosqlite alternative`, `lightweight mongodb`, `file-backed document store`
Raw data
{
"_id": null,
"home_page": null,
"name": "nosqlpy",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "nosqlpy, nosqlite, nosql, async nosql, python nosql, async database, document-store, database, mongodb, mql, query-language, asyncio, lightweight",
"author": "Hoang V. Nguyen",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/ed/b5/f3cd5b272c680bb51133263d08ce2c69ce1f80f8ae905abde786445f31bd/nosqlpy-0.1.1.tar.gz",
"platform": null,
"description": "# nosqlpy - Async, NoSQL, MQL-style embedded document store for Python\r\n\r\n<p align=\"center\">\r\n <img src=\"logo.png\" alt=\"nosqlpy logo\" width=\"250\">\r\n</p>\r\n\r\n\u26a0\ufe0f Warning: This is the alpha version\r\n\r\n**nosqlpy** is a lightweight, async-first, file-backed document database for Python that speaks a **MongoDB-like Query Language (MQL)**.\r\nIt\u2019s designed for small-to-medium apps, dev tooling, prototyping, CLI tools, and apps that want a simple embedded DB with **async/await**, durability options, indexes, and a clean API - without running a separate database server.\r\n\r\n- \u2705 Async-first \r\n- \u2705 Mongo-like queries \r\n- \u2705 Append-only op-log \r\n- \u2705 Secondary equality indexes \r\n- \u2705 Compaction & durability modes\r\n\r\n---\r\n\r\n# Table of contents\r\n\r\n- [nosqlpy - Async, NoSQL, MQL-style embedded document store for Python](#nosqlpy---async-nosql-mql-style-embedded-document-store-for-python)\r\n- [Table of contents](#table-of-contents)\r\n- [Why nosqlpy?](#why-nosqlpy)\r\n- [Key features](#key-features)\r\n- [Installation](#installation)\r\n- [Quickstart](#quickstart)\r\n- [Core concepts \\& API reference](#core-concepts--api-reference)\r\n - [Database \\& Collection](#database--collection)\r\n - [Common operations](#common-operations)\r\n - [List of operations](#list-of-operations)\r\n - [Examples](#examples)\r\n - [Query language (MQL subset)](#query-language-mql-subset)\r\n - [Update operators](#update-operators)\r\n - [Indexing](#indexing)\r\n - [Compaction \\& durability](#compaction--durability)\r\n- [Performance tips](#performance-tips)\r\n- [License](#license)\r\n- [Keywords](#keywords)\r\n\r\n---\r\n\r\n# Why nosqlpy?\r\n\r\nMany small Python apps need an embedded document store that:\r\n\r\n* Is **async-native** (fits FastAPI / aiohttp / asyncio apps)\r\n* Uses a **familiar query language** (MongoDB-style filters)\r\n* Provides **durable writes** and **safe recovery**\r\n* Enables **fast reads** using secondary indexes\r\n\r\nMost small DB options are either synchronous (TinyDB), or require wrapping sync code into thread pools. `nosqlpy` is built async-first and offers MQL-style queries, an append-only op-log for safe durability, optional indexes, and compaction for production-ish workloads - all in a single small dependency set.\r\n\r\n---\r\n\r\n# Key features\r\n\r\n* Async API (`async/await`) - designed for asyncio apps\r\n* MongoDB-style query language (subset): `$and`, `$or`, `$not`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`, simple dot-notation for nested fields\r\n* Update operators: `$set`, `$unset`, `$inc`, `$push`, `$pull`\r\n* Append-only operation log (oplog) with replayable history\r\n* Durability modes: `fast`, `safe`, `durable` (fsync)\r\n* Secondary equality indexes (create and use for fast `field == value`)\r\n* Compaction (background-friendly pattern) to shrink logs and produce snapshots\r\n* Bulk ops: `insert_many`, `update_many`, `delete_many`\r\n* find support: `projection`, `sort`, `skip`, `limit`, `find_one`\r\n* TinyDB migration helper / compatibility shim (to ease switching)\r\n\r\n---\r\n\r\n# Installation\r\n\r\n```bash\r\n# install from PyPI (when published)\r\npip install nosqlpy\r\n\r\n# or during development (from repo)\r\ngit clone https://github.com/viehoang/nosqlpy.git\r\ncd nosqlpy\r\npip install -e \".[dev]\"\r\n```\r\n\r\nDependencies: `aiofiles` (async file I/O). Dev/test extras include `pytest`, `pytest-asyncio`.\r\n\r\n---\r\n\r\n# Quickstart\r\n\r\n```python\r\nimport asyncio\r\nfrom nosqlpy import Database\r\n\r\nasync def main():\r\n db = Database(\"data_dir\", durability=\"safe\")\r\n users = await db.collection(\"users\")\r\n\r\n # insert\r\n uid = await users.insert_one({\r\n \"name\": \"alice\", \r\n \"age\": 30})\r\n \r\n print(\"inserted id:\", uid)\r\n\r\n # find (Mongo-like query)\r\n results = await users.find(\r\n {\"age\": {\"$gte\": 25}, \r\n \"name\": \"alice\"})\r\n print(\"found:\", results)\r\n\r\n # update operators\r\n await users.update_many(\r\n {\"name\": \"alice\"}, \r\n {\"$inc\": {\"age\": 1}, \r\n \"$push\": {\"tags\": \"admin\"}})\r\n\r\n # create index on email for fast equality lookups\r\n await users.create_index(\"email\")\r\n\r\n # compact (shrink the op-log)\r\n await users.compact()\r\n\r\nasyncio.run(main())\r\n```\r\n\r\n---\r\n\r\n# Core concepts & API reference\r\n\r\n> **Collection** - a single named dataset backed by an append-oplog file.\r\n> \r\n> **Document** - a JSON-like dict with an `_id` field (string) as the primary key.\r\n>\r\n> **Op-log** - append-only lines describing `insert`, `update`, `replace`, `delete` operations.\r\n\r\nAll calls are `async`.\r\n\r\n## Database & Collection\r\n\r\n```python\r\nfrom nosqlpy import Database\r\ndb = Database(\"my_data_dir\", durability=\"safe\")\r\nusers = await db.collection(\"users\")\r\n```\r\n\r\n## Common operations\r\n\r\n### List of operations\r\n\r\n| Operator | Description |\r\n|----------|-------------|\r\n| `open` | Asynchronously open and load the collection. |\r\n| `insert_one` | Insert a single document into the collection. |\r\n| `insert_many` | Insert multiple documents into the collection. |\r\n| `find` | Find documents matching the query, with options for projection, sort, skip, and limit. |\r\n| `find_one` | Find a single document matching the query. |\r\n| `count` | Count the number of documents matching the query. |\r\n| `delete_one` | Delete a single document matching the query. |\r\n| `delete_many` | Delete multiple documents matching the query. |\r\n| `update_one` | Update a single document matching the query, with optional upsert. |\r\n| `update_many` | Update multiple documents matching the query, with optional upsert (upserts one if no matches). |\r\n| `replace_one` | Replace a single document matching the query, with optional upsert. |\r\n| `create_index` | Create a secondary hash index on a field. |\r\n| `drop_index` | Drop the index on a field. |\r\n| `compact` | Compact the op-log by replacing it with current state as inserts. |\r\n\r\n### Examples\r\n\r\n```python\r\n# insert one\r\n_id = await users.insert_one({\"name\": \"Bob\", \"age\": 22})\r\n\r\n# insert many\r\nids = await users.insert_many([{\"name\":\"A\"},{\"name\":\"B\"}])\r\n\r\n# find (MQL filter)\r\ndocs = await users.find(\r\n {\"age\": {\"$gte\": 18}}, \r\n projection=[\"name\",\"age\"], \r\n sort=[(\"age\", -1)], \r\n skip=0, \r\n limit=50)\r\n\r\n# find one\r\ndoc = await users.find_one({\"name\": \"Bob\"})\r\n\r\n# count\r\nn = await users.count({\"age\": {\"$gte\": 30}})\r\n\r\n# update many (supports $set/$inc/$push/$pull)\r\nupdated = await users.update_many(\r\n {\"name\": \"Bob\"}, \r\n {\"$inc\": {\"age\": 1}})\r\n\r\n# replace one\r\nok = await users.replace_one(\r\n {\"name\": \"Bob\"}, \r\n {\"name\": \"Robert\", \"age\": 23})\r\n\r\n# delete many\r\ndeleted = await users.delete_many({\"age\": {\"$lt\": 18}})\r\n\r\n# compact (rewrite op-log as current-state snapshot)\r\nawait users.compact()\r\n```\r\n\r\n## Query language (MQL subset)\r\n\r\nSupported query operators (subset):\r\n\r\n* Comparison: `$eq` (or plain value), `$ne`, `$gt`, `$gte`, `$lt`, `$lte`\r\n* Membership: `$in`, `$nin`\r\n* Existence: `$exists`\r\n* Logical: `$and`, `$or`, `$not`\r\n* Dot-notation: `\"user.age\"` for nested fields\r\n\r\nExamples:\r\n\r\n```python\r\n# age >= 30 AND (country == \"US\" OR country == \"JP\")\r\nq = {\"$and\": [{\"age\": {\"$gte\": 30}}, {\"$or\": [{\"country\": \"US\"}, {\"country\": \"JP\"}]}]}\r\n\r\n# membership\r\nq2 = {\"status\": {\"$in\": [\"active\", \"pending\"]}}\r\n\r\n# nested field\r\nq3 = {\"profile.email\": {\"$exists\": True}}\r\n```\r\n\r\n## Update operators\r\n\r\nSupported update operators:\r\n\r\n* `$set`: set field(s) to value\r\n* `$unset`: remove field(s)\r\n* `$inc`: increment numeric field\r\n* `$push`: append to array field\r\n* `$pull`: remove value(s) from array field\r\n\r\nExample:\r\n\r\n```python\r\nawait users.update_many(\r\n {\"_id\": some_id}, \r\n {\r\n \"$set\": {\"name\": \"Alice\"}, \r\n \"$inc\": {\"score\": 10}\r\n })\r\n```\r\n\r\n## Indexing\r\n\r\nEquality secondary index (hash-based) to accelerate queries like `{ \"email\": \"a@x.com\" }`:\r\n\r\n```python\r\nawait users.create_index(\"email\")\r\n```\r\n\r\nNotes:\r\n\r\n* Indexes are in-memory by default and rebuilt on create; consider snapshotting indexes for very large DBs (TODO/roadmap).\r\n* Planner currently optimizes single-field equality queries; range indexes are a planned feature.\r\n\r\n## Compaction & durability\r\n\r\n* **Op-log** (append-only) is crash friendly: each write is a single line append.\r\n* Durability modes:\r\n\r\n * `fast`: minimal overhead, no explicit flush (best throughput, less durable)\r\n * `safe`: flush after write (`file.flush()`) - good compromise\r\n * `durable`: `fsync()` after each op (strong durability, slower)\r\n\r\nCompaction rewrites the current state as a compact snapshot (series of `insert` ops) and atomically replaces the op-log. For large DBs we recommend:\r\n\r\n* Run `compact()` occasionally (or use background segmented compaction; see Roadmap).\r\n* Use `durable` for critical writes, `safe` for normal persistence.\r\n\r\n---\r\n\r\n# Performance tips\r\n\r\n* Use indexes for frequent equality lookups.\r\n* Use `durability=\"safe\"` for most apps; switch to `durable` only if you need fsync-level guarantees on every op.\r\n* For very large data (>100k docs), enable segmented compaction or index snapshots (planned).\r\n\r\n---\r\n\r\n# License\r\n\r\n`nosqlpy` is released under the **MIT License**. See `LICENSE` for details.\r\n\r\n---\r\n\r\n# Keywords\r\n\r\n`nosqlpy`, `nosqlite`, `nosql lite`, `async nosql`, `python async database`, `embedded document store`, `mongo query language python`, `mql python`, `asyncio database`, `append log database`, `python nosql lite`, `tinydb alternative`, `aiosqlite alternative`, `lightweight mongodb`, `file-backed document store`\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A lightweight NoSQL-like database in Python",
"version": "0.1.1",
"project_urls": {
"Homepage": "https://github.com/viehoang/nosqlpy"
},
"split_keywords": [
"nosqlpy",
" nosqlite",
" nosql",
" async nosql",
" python nosql",
" async database",
" document-store",
" database",
" mongodb",
" mql",
" query-language",
" asyncio",
" lightweight"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "598810345ad138246ee23e5aa3c819f812dd05142ea3617bc59448a7dceb46bf",
"md5": "e03322db6e05ac8ff140bed808db3460",
"sha256": "67af607379277bb85c7ef75c53400a7cbabff2466095fa44dfe1d378a1345fcb"
},
"downloads": -1,
"filename": "nosqlpy-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e03322db6e05ac8ff140bed808db3460",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 4914,
"upload_time": "2025-10-10T07:48:21",
"upload_time_iso_8601": "2025-10-10T07:48:21.584827Z",
"url": "https://files.pythonhosted.org/packages/59/88/10345ad138246ee23e5aa3c819f812dd05142ea3617bc59448a7dceb46bf/nosqlpy-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "edb5f3cd5b272c680bb51133263d08ce2c69ce1f80f8ae905abde786445f31bd",
"md5": "6b3117232d4fc482f8b05bf55a80a426",
"sha256": "df7a5736e8b9f4062d9553600f011bf89eace34c3d7ab69b1b8d9985de45ca58"
},
"downloads": -1,
"filename": "nosqlpy-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "6b3117232d4fc482f8b05bf55a80a426",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 5396,
"upload_time": "2025-10-10T07:48:22",
"upload_time_iso_8601": "2025-10-10T07:48:22.757631Z",
"url": "https://files.pythonhosted.org/packages/ed/b5/f3cd5b272c680bb51133263d08ce2c69ce1f80f8ae905abde786445f31bd/nosqlpy-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-10 07:48:22",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "viehoang",
"github_project": "nosqlpy",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "nosqlpy"
}
JSON
