# Django EOSE
**Django Encrypted Object Search Engine**
An efficient search engine for encrypted fields in Django querysets, with support for parallel processing, smart batching, and result caching.
`django-eose` is ideal for scenarios where you need to search data that is encrypted in the database, providing high performance even on large datasets.
---
## Key Features
- **Parallel Search:** supports execution using processes, threads, or synchronous mode.
- **Smart Batching:** batch size automatically adapts to available memory.
- **Related Field Search:** search can be performed on fields of related objects, e.g., `order__client`.
- **Result Caching:** frequently searched results are cached for faster repeated queries.
- **Optimized for Large Datasets:** particularly useful for large datasets and encrypted fields.
---
## Installation
Install easily via `pip`:
```bash
pip install django-eose
```
## Requirements
- Python 3.10 or higher
- Django 5.2.5
- sqlparse==0.5.3
- asgiref==3.9.1
- psutil==7.0.0
- cffi==1.17.1
- cryptography==45.0.6
- pycparser==2.22
## Model Configuration
There are two ways to use django-eose. The first is to let django-eose handle the decryption itself using Fernet, which is much faster. The second is to create Getters in Django that already return the decrypted values. If you are going to use the first option, skip this step.
Example model using Fernet encryption:
```python
from django.db import models
from cryptography.fernet import Fernet
AES_KEY = b"<your_key_here>"
class Client(models.Model):
_encrypted_name = models.BinaryField()
_encrypted_email = models.BinaryField()
# Method to decrypt
def _decrypt_field(self, encrypted_value):
return Fernet(AES_KEY).decrypt(encrypted_value).decode()
# Method to encrypt
def _encrypt_field(self, value):
return Fernet(AES_KEY).encrypt(value.encode())
# Creates properties that handle encryption/decryption
@staticmethod
def _property(field_name):
def getter(self):
return self._decrypt_field(getattr(self, field_name))
def setter(self, value):
setattr(self, field_name, self._encrypt_field(value))
return property(getter, setter)
# Fields accessible as normal attributes
name = _property('_encrypted_name')
email = _property('_encrypted_email')
```
⚠️ You interact with name and email like regular fields, while encryption/decryption happens transparently.
## Usage
Add AES_PASSWORD to your .env file:
```bash
AES_PASSWORD=your-password-here
```
To decrypt data in django-eose and speed up the process, use:
```python
from django_eose import search_queryset
from orders.models import OrderItem
# Example: search for "john" in related client fields
results = search_queryset(
search="john",
queryset=OrderItem.objects.all(),
related_field="order__client",
fields=("_encrypted_name", "_encrypted_email"),
executor="processes",
max_batch_size=1_000_000,
decrypt=True
) # returns queryset.filter(pk__in=matched_ids)
```
To use the decrypted data returned by getters in Django, use:
```python
from django_eose import search_queryset
from orders.models import OrderItem
# Example: search for "john" in related client fields
results = search_queryset(
search="john",
queryset=OrderItem.objects.all(),
related_field="order__client",
fields=("name", "email"),
only_fields=("_encrypted_name", "_encrypted_email"),
executor="processes",
max_batch_size=1_000_000
) # returns queryset.filter(pk__in=matched_ids)
```
## `search_queryset` Parameters
- search: search term (case-insensitive).
- queryset: Django queryset to search in.
- related_field: path to a related object using __ notation. (optional)
- fields: fields of the object to inspect.
- only_fields: fields to load with .only() for optimization (optional).
- executor: "processes", "threads", or "sync" (default: "processes").
- cache_timeout: cache duration in seconds (default: 600).
- imap_chunksize: chunk size per worker (default: 10240).
- memory_fraction: fraction of available memory for batching (default: 0.60).
- avg_obj_size_bytes: estimated average object size in bytes (optional).
- max_workers: maximum number of parallel workers (optional).
- max_batch_size: maximum number of objects per batch. (default: 1_000_000)
- decrypt: decrypt data using Fernet. (ignores the only_fields parameter)
Refer to `search_queryset` for full parameter details.
## Default Settings
`django-eose` defines default settings in `django_eose/settings.py`:
- MEMORY_FRACTION
- IMAP_CHUNKSIZE
- EXECUTOR
- CACHE_TIMEOUT
- AVG_OBJ_SIZE_FALLBACK
- MIN_BATCH_SIZE
- MAX_BATCH_SIZE
## License
MIT © 2025 Paulo Otávio Castoldi
## Links
[Source](https://github.com/paulootaviodev/django-eose)
Raw data
{
"_id": null,
"home_page": null,
"name": "django-eose",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "django, search, multiprocessing, decrypted, cache",
"author": null,
"author_email": "Paulo Ot\u00e1vio Castoldi <paulootavio343@outlook.com>",
"download_url": "https://files.pythonhosted.org/packages/df/d9/7d88d04ce018655c5dfa3708206c4eba6cc8ab2880095db3e51039f27eca/django_eose-0.3.2.tar.gz",
"platform": null,
"description": "# Django EOSE\n\n**Django Encrypted Object Search Engine** \nAn efficient search engine for encrypted fields in Django querysets, with support for parallel processing, smart batching, and result caching.\n\n`django-eose` is ideal for scenarios where you need to search data that is encrypted in the database, providing high performance even on large datasets.\n\n---\n\n## Key Features\n\n- **Parallel Search:** supports execution using processes, threads, or synchronous mode. \n- **Smart Batching:** batch size automatically adapts to available memory. \n- **Related Field Search:** search can be performed on fields of related objects, e.g., `order__client`. \n- **Result Caching:** frequently searched results are cached for faster repeated queries. \n- **Optimized for Large Datasets:** particularly useful for large datasets and encrypted fields.\n\n---\n\n## Installation\n\nInstall easily via `pip`:\n\n```bash\npip install django-eose\n```\n\n## Requirements\n\n- Python 3.10 or higher\n- Django 5.2.5\n- sqlparse==0.5.3\n- asgiref==3.9.1\n- psutil==7.0.0\n- cffi==1.17.1\n- cryptography==45.0.6\n- pycparser==2.22\n\n## Model Configuration\n\nThere are two ways to use django-eose. The first is to let django-eose handle the decryption itself using Fernet, which is much faster. The second is to create Getters in Django that already return the decrypted values. If you are going to use the first option, skip this step.\n\nExample model using Fernet encryption:\n\n```python\nfrom django.db import models\nfrom cryptography.fernet import Fernet\n\nAES_KEY = b\"<your_key_here>\"\n\nclass Client(models.Model):\n _encrypted_name = models.BinaryField()\n _encrypted_email = models.BinaryField()\n\n # Method to decrypt\n def _decrypt_field(self, encrypted_value):\n return Fernet(AES_KEY).decrypt(encrypted_value).decode()\n\n # Method to encrypt\n def _encrypt_field(self, value):\n return Fernet(AES_KEY).encrypt(value.encode())\n \n # Creates properties that handle encryption/decryption\n @staticmethod\n def _property(field_name):\n def getter(self):\n return self._decrypt_field(getattr(self, field_name))\n \n def setter(self, value):\n setattr(self, field_name, self._encrypt_field(value))\n\n return property(getter, setter)\n \n # Fields accessible as normal attributes\n name = _property('_encrypted_name')\n email = _property('_encrypted_email')\n\n```\n\u26a0\ufe0f You interact with name and email like regular fields, while encryption/decryption happens transparently.\n\n## Usage\n\nAdd AES_PASSWORD to your .env file:\n```bash\nAES_PASSWORD=your-password-here\n```\n\nTo decrypt data in django-eose and speed up the process, use:\n\n```python\nfrom django_eose import search_queryset\nfrom orders.models import OrderItem\n\n# Example: search for \"john\" in related client fields\nresults = search_queryset(\n search=\"john\",\n queryset=OrderItem.objects.all(),\n related_field=\"order__client\",\n fields=(\"_encrypted_name\", \"_encrypted_email\"),\n executor=\"processes\",\n max_batch_size=1_000_000,\n decrypt=True\n) # returns queryset.filter(pk__in=matched_ids)\n```\n\nTo use the decrypted data returned by getters in Django, use:\n\n\n```python\nfrom django_eose import search_queryset\nfrom orders.models import OrderItem\n\n# Example: search for \"john\" in related client fields\nresults = search_queryset(\n search=\"john\",\n queryset=OrderItem.objects.all(),\n related_field=\"order__client\",\n fields=(\"name\", \"email\"),\n only_fields=(\"_encrypted_name\", \"_encrypted_email\"),\n executor=\"processes\",\n max_batch_size=1_000_000\n) # returns queryset.filter(pk__in=matched_ids)\n```\n\n## `search_queryset` Parameters\n\n- search: search term (case-insensitive).\n- queryset: Django queryset to search in.\n- related_field: path to a related object using __ notation. (optional)\n- fields: fields of the object to inspect.\n- only_fields: fields to load with .only() for optimization (optional).\n- executor: \"processes\", \"threads\", or \"sync\" (default: \"processes\").\n- cache_timeout: cache duration in seconds (default: 600).\n- imap_chunksize: chunk size per worker (default: 10240).\n- memory_fraction: fraction of available memory for batching (default: 0.60).\n- avg_obj_size_bytes: estimated average object size in bytes (optional).\n- max_workers: maximum number of parallel workers (optional).\n- max_batch_size: maximum number of objects per batch. (default: 1_000_000)\n- decrypt: decrypt data using Fernet. (ignores the only_fields parameter)\n\nRefer to `search_queryset` for full parameter details.\n\n## Default Settings\n\n`django-eose` defines default settings in `django_eose/settings.py`:\n\n- MEMORY_FRACTION\n- IMAP_CHUNKSIZE\n- EXECUTOR\n- CACHE_TIMEOUT\n- AVG_OBJ_SIZE_FALLBACK\n- MIN_BATCH_SIZE\n- MAX_BATCH_SIZE\n\n## License\n\nMIT \u00a9 2025 Paulo Ot\u00e1vio Castoldi\n\n## Links\n\n[Source](https://github.com/paulootaviodev/django-eose)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Django Encrypted Object Search Engine.",
"version": "0.3.2",
"project_urls": {
"Homepage": "https://github.com/paulootaviodev/django-eose"
},
"split_keywords": [
"django",
" search",
" multiprocessing",
" decrypted",
" cache"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "90568295648f8726d34ef418f7d03edafa1ad1623091ef6f9d57c0e2d8da0fba",
"md5": "c601a431ec4fc752dc6342966c2174c1",
"sha256": "5b1e750ecceceb5fc7e3079081831ef7db629ba232b3f5e8281169db526ecd5b"
},
"downloads": -1,
"filename": "django_eose-0.3.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c601a431ec4fc752dc6342966c2174c1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 10473,
"upload_time": "2025-09-13T22:21:58",
"upload_time_iso_8601": "2025-09-13T22:21:58.093440Z",
"url": "https://files.pythonhosted.org/packages/90/56/8295648f8726d34ef418f7d03edafa1ad1623091ef6f9d57c0e2d8da0fba/django_eose-0.3.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "dfd97d88d04ce018655c5dfa3708206c4eba6cc8ab2880095db3e51039f27eca",
"md5": "c842a3b371944136a0f38e24dc5913b9",
"sha256": "9a287458a0031f33fb5fa85c6b11a5e7e5d1626509455e92871c37f2ef976e2c"
},
"downloads": -1,
"filename": "django_eose-0.3.2.tar.gz",
"has_sig": false,
"md5_digest": "c842a3b371944136a0f38e24dc5913b9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 10396,
"upload_time": "2025-09-13T22:21:59",
"upload_time_iso_8601": "2025-09-13T22:21:59.516118Z",
"url": "https://files.pythonhosted.org/packages/df/d9/7d88d04ce018655c5dfa3708206c4eba6cc8ab2880095db3e51039f27eca/django_eose-0.3.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-13 22:21:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "paulootaviodev",
"github_project": "django-eose",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "asgiref",
"specs": [
[
"==",
"3.9.1"
]
]
},
{
"name": "cffi",
"specs": [
[
"==",
"2.0.0"
]
]
},
{
"name": "cryptography",
"specs": [
[
"==",
"45.0.7"
]
]
},
{
"name": "Django",
"specs": [
[
"==",
"5.2.6"
]
]
},
{
"name": "psutil",
"specs": [
[
"==",
"7.0.0"
]
]
},
{
"name": "pycparser",
"specs": [
[
"==",
"2.23"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
"==",
"1.1.1"
]
]
},
{
"name": "setuptools",
"specs": [
[
"==",
"80.9.0"
]
]
},
{
"name": "sqlparse",
"specs": [
[
"==",
"0.5.3"
]
]
},
{
"name": "wheel",
"specs": [
[
"==",
"0.45.1"
]
]
}
],
"lcname": "django-eose"
}
JSON
