# mongoengine-plus
[](https://codecov.io/gh/cuenca-mx/mongoengine-plus)
[](https://github.com/cuenca-mx/mongoengine-plus/actions?query=workflow%3Atest)
Extra field types, function helpers and asyncio support for [mongoengine](https://github.com/MongoEngine/mongoengine)
## Installation
```
pip install mongoengine-plus
```
## Testing
### Requirements
**Docker**
Make sure you have Docker installed, as the unit tests require a real Mongo database instance.
It is required in order to test Client-Side Field Level Encryption (CSFLE) not supported by mongomock.
Don't worry about setting up Mongo manually; the test suite will handle that for you.
**Self-signed certificates**
To test CSFLE, you need to mock a KMS provider that supports HTTPS requests. We use [moto](https://github.com/getmoto/moto)
as a KMS mock server running on localhost. This allows us to set up the required SSL certificates
so that the `pymongo.encryption` classes don't raise any complaints. The current testing certificates
are located in `tests/localhost.*`. The configuration file for the certificates is `/tests/cert.conf`.
If you need to create new self-signed certificates, you can follow these steps:
```bash
cd tests
openssl genrsa -out localhost.key 2048
openssl req -new -key localhost.key -out localhost.csr -config cert.conf
# creates a certificate valid for 1 year
openssl x509 -req -days 365 -in localhost.csr -signkey localhost.key -out localhost.crt -extensions v3_utils -extfile cert.conf
```
### Clone, install requirements and run tests
```bash
git clone git@github.com:cuenca-mx/mongoengine-plus.git
cd mongoengine-plus
make install-test
# Since we're using a self-signed certificate with the moto_server
# for testing we need to configure the localhost certificate so
# ClientEncryption object can connect to the mock KMS instance
export SSL_CERT_FILE=$(pwd)/tests/localhost.crt
make test
unset SSL_CERT_FILE
```
## Mongoengine + Asyncio
### Asynchronous Documents operation
Mongoengine-plus brings the power of asynchronous programming to your document operations.
We've introduced an `AsyncDocument` class that extends the standard `Document` class,
allowing you to perform operations like `save()`, `delete()` and `reload()` asynchronously.
To get started, simply inherit your document classes from `AsyncDocument` instead of `Document`.
You'll then have access to async counterparts of the familiar methods:
- `async_save()`: Save your document asynchronously.
- `async_delete()`: Delete your document asynchronously.
- `async_reload()`: Reload your document asynchronously.
These async methods maintain the same signature as their synchronous counterparts,
so you can seamlessly transition your existing codebase to take advantage
of asynchronous operations.
```python
from mongoengine import StringField
from mongoengine_plus.aio import AsyncDocument
class User(AsyncDocument):
name = StringField(required=True)
async def main():
user = User(name='Jane')
# Asynchronously save the user document
await user.async_save()
# Asynchronously update the user document
user.name = 'John'
await user.async_save()
# Asynchronously reload the user document
await user.async_reload()
print(f"Reloaded user: {user.name}")
# Asynchronously delete the user document
await user.async_delete()
if __name__ == "__main__":
import asyncio
asyncio.run(main())
```
### Asynchronous Querying the database
You can do asynchronous queries to your document collections.
We've introduced an `AsyncQuerySet` class that [extends](https://docs.mongoengine.org/guide/querying.html#custom-querysets)
the standard `QuerySet`, providing you with async versions of familiar
query operations.
You don't need to interact with `AsyncQuerySet` directly.
It's already configured as the `queryset_class` for the `AsyncDocument` class.
Once you inherit your document classes from `AsyncDocument`,
the `objects` property will give you access to asynchronous query methods:
- `async_first()`: Retrieve the first document asynchronously.
- `async_get()`: Retrieve a specific document asynchronously.
- `async_count()`: Count the number of documents asynchronously.
- `async_update()`: Update documents asynchronously.
- `async_insert()`: Insert documents asynchronously.
- `async_delete()`: Delete documents asynchronously.
```python
async def get_first_user():
user = await User.objects.async_first()
if user:
print(f"First user: {user.name}")
else:
print("No users found")
async def get_user_by_name(name):
user = await User.objects(name=name).async_get()
print(f"User with name '{name}': {user.name}")
async def count_users():
count = await User.objects.async_count()
print(f"Number of users: {count}")
async def update_user_name(old_name, new_name):
updated = await User.objects(name=old_name).async_update(name=new_name)
print(f"Updated {updated} user(s) from '{old_name}' to '{new_name}'")
async def insert_users(user_names):
users = [User(name=name) for name in user_names]
await User.objects.async_insert(users)
async def delete_users(name):
await User.objects(name=name).async_delete()
async def main():
await insert_users(["Jane", "John"])
await get_first_user()
await get_user_by_name("John")
await count_users()
await update_user_name("John", "Johnny")
await get_user_by_name("Johnny")
await delete_users("Johnny")
if __name__ == "__main__":
import asyncio
asyncio.run(main())
```
To iterate over the result set of a query asynchronously,
you can use the `async_to_list()` method:
```python
users = await User.objects(name='Jane').async_to_list()
async for user in users:
# Process each user
...
```
We recommend using `async_to_list()` for small result sets.
## Client-side Field Level Encryption
Mongoengine-plus introduces a new field type called `EncryptedStringField` that implements
Client-side Field Level Encryption ([CSFLE](https://www.mongodb.com/docs/manual/core/csfle/))
using [pymongo](https://pymongo.readthedocs.io/en/stable/examples/encryption.html) encryption classes.
This feature allows explicit data encryption before sending it over the network to MongoDB,
and automatic data decryption after reading from MongoDB. It supports both synchronous
and asynchronous operations. Currently, the `EncryptedStringField` implementation supports
the AWS KMS service as the Key Management Service (KMS) provider.
```python
from mongoengine import Document, StringField
from mongoengine_plus.types import EncryptedStringField
from pymongo.encryption import Algorithm
class User(Document):
id = StringField(primary_key=True)
ssn = EncryptedStringField(
algorithm=Algorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Deterministic
)
user = User(id='US1', ssn='12345')
user.save()
print(user.ssn) # Output: '12345'
user_ = User.objects.get(id='US1')
print(user_.ssn) # Output: '12345'
```
There are a few steps before you can start using `EncryptedStringField`.
### 1. Create a Data Encryption Key (DEK)
Before using `EncryptedStringField`, you'll need to create a Data Encryption Key (DEK)
for encrypting and decrypting your data. The DEK should follow the recommended
requirements described in the official MongoDB documentation on [Keys and Key Vaults](https://www.mongodb.com/docs/manual/core/csfle/fundamentals/keys-key-vaults/#std-label-csfle-reference-keys-key-vaults).
We've provided a helper method to create your DEK easily.
```python
from mongoengine import connect
from mongoengine_plus.types.encrypted_string.base import create_data_key
connect(host='mongo://localhost:27017/db')
create_data_key(
kms_provider=dict(
aws=dict(
accessKeyId='your-aws-key-id',
secretAccessKey='your-aws-secret-access-key'
)
),
key_namespace='encryption.__keyVault',
key_arn='arn:aws:kms:us-east-1:111122223333:key/your-key-id',
key_name='my_key_name',
kms_connection_url='https://kms.us-east-1.amazonaws.com',
kms_region_name='us-east-1',
)
```
You'll need to execute this step only once during the project setup. Ensure that your
MongoDB user has the necessary permissions for collection and index creation, and
access to the AWS KMS key.
### 2. Configure `EncryptedStringField`
Since `EncryptedStringField` needs to read the DEK from your MongoDB instance and access the
KMS key for encryption/decryption, you'll need to configure it as follows. This
configuration might be in your `__init__.py` file and should be executed once.
```python
from mongoengine import Document, StringField
from mongoengine_plus.types import EncryptedStringField
from pymongo.encryption import Algorithm
EncryptedStringField.configure_aws_kms(
'encryption.__keyVault',
'my_key_name',
'your-aws-key-id',
'your-aws-secret-access-key',
'us-east-1',
)
class User(Document):
id = StringField(primary_key=True)
ssn = EncryptedStringField(
algorithm=Algorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Deterministic
)
```
Now you are ready to go!
### 3. Optimize KMS requests (optional)
There's a caveat in the `EncryptedStringField` implementation. Every time `EncryptedStringField` needs
to encrypt or decrypt data, it uses the `pymongo.encryption.ClientEncryption`,
which makes a request to the AWS KMS service endpoint. This can potentially slow down
the performance of reading and writing encrypted data to MongoDB. As a workaround,
we've created a function that patches this behavior and caches the data key.
```python
from mongoengine_plus.types.encrypted_string import cache_kms_data_key
cache_kms_data_key(
'encryption.__keyVault',
'my_key_name',
'your-aws-key-id',
'your-aws-secret-access-key',
'us-east-1',
'https://kms.us-east-1.amazonaws.com',
)
```
You should execute this function once before making any database write or read operations,
perhaps in your `__init__.py` file. It will retrieve the KMS key and cache it for
subsequent requests.
Raw data
{
"_id": null,
"home_page": "https://github.com/cuenca-mx/mongoengine-plus",
"name": "mongoengine-plus",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Cuenca",
"author_email": "dev@cuenca.com",
"download_url": "https://files.pythonhosted.org/packages/c3/6d/4125b8e2df22787d88eba1b230c90a93ad36b8fd9b6d3365133582b422f6/mongoengine_plus-1.0.0.tar.gz",
"platform": null,
"description": "# mongoengine-plus\n[](https://codecov.io/gh/cuenca-mx/mongoengine-plus)\n[](https://github.com/cuenca-mx/mongoengine-plus/actions?query=workflow%3Atest)\n\nExtra field types, function helpers and asyncio support for [mongoengine](https://github.com/MongoEngine/mongoengine)\n\n## Installation\n```\npip install mongoengine-plus\n```\n\n## Testing\n\n### Requirements\n**Docker**\n\nMake sure you have Docker installed, as the unit tests require a real Mongo database instance.\nIt is required in order to test Client-Side Field Level Encryption (CSFLE) not supported by mongomock.\nDon't worry about setting up Mongo manually; the test suite will handle that for you.\n\n**Self-signed certificates**\n\nTo test CSFLE, you need to mock a KMS provider that supports HTTPS requests. We use [moto](https://github.com/getmoto/moto) \nas a KMS mock server running on localhost. This allows us to set up the required SSL certificates \nso that the `pymongo.encryption` classes don't raise any complaints. The current testing certificates \nare located in `tests/localhost.*`. The configuration file for the certificates is `/tests/cert.conf`.\n\nIf you need to create new self-signed certificates, you can follow these steps:\n\n```bash\ncd tests\nopenssl genrsa -out localhost.key 2048\nopenssl req -new -key localhost.key -out localhost.csr -config cert.conf\n# creates a certificate valid for 1 year\nopenssl x509 -req -days 365 -in localhost.csr -signkey localhost.key -out localhost.crt -extensions v3_utils -extfile cert.conf\n```\n\n### Clone, install requirements and run tests\n```bash\ngit clone git@github.com:cuenca-mx/mongoengine-plus.git\ncd mongoengine-plus\nmake install-test\n# Since we're using a self-signed certificate with the moto_server\n# for testing we need to configure the localhost certificate so \n# ClientEncryption object can connect to the mock KMS instance\nexport SSL_CERT_FILE=$(pwd)/tests/localhost.crt\nmake test\nunset SSL_CERT_FILE\n```\n\n## Mongoengine + Asyncio\n\n### Asynchronous Documents operation\n\nMongoengine-plus brings the power of asynchronous programming to your document operations.\nWe've introduced an `AsyncDocument` class that extends the standard `Document` class, \nallowing you to perform operations like `save()`, `delete()` and `reload()` asynchronously.\n\nTo get started, simply inherit your document classes from `AsyncDocument` instead of `Document`. \nYou'll then have access to async counterparts of the familiar methods:\n\n- `async_save()`: Save your document asynchronously.\n- `async_delete()`: Delete your document asynchronously.\n- `async_reload()`: Reload your document asynchronously.\n\nThese async methods maintain the same signature as their synchronous counterparts, \nso you can seamlessly transition your existing codebase to take advantage \nof asynchronous operations.\n\n```python\nfrom mongoengine import StringField\nfrom mongoengine_plus.aio import AsyncDocument\n\nclass User(AsyncDocument):\n name = StringField(required=True)\n\nasync def main():\n user = User(name='Jane')\n # Asynchronously save the user document\n await user.async_save() \n \n # Asynchronously update the user document\n user.name = 'John'\n await user.async_save() \n \n # Asynchronously reload the user document\n await user.async_reload() \n print(f\"Reloaded user: {user.name}\")\n \n # Asynchronously delete the user document\n await user.async_delete() \n\nif __name__ == \"__main__\":\n import asyncio\n asyncio.run(main())\n```\n\n### Asynchronous Querying the database\n\nYou can do asynchronous queries to your document collections. \nWe've introduced an `AsyncQuerySet` class that [extends](https://docs.mongoengine.org/guide/querying.html#custom-querysets) \nthe standard `QuerySet`, providing you with async versions of familiar\nquery operations.\n\nYou don't need to interact with `AsyncQuerySet` directly. \nIt's already configured as the `queryset_class` for the `AsyncDocument` class. \nOnce you inherit your document classes from `AsyncDocument`, \nthe `objects` property will give you access to asynchronous query methods:\n\n- `async_first()`: Retrieve the first document asynchronously.\n- `async_get()`: Retrieve a specific document asynchronously.\n- `async_count()`: Count the number of documents asynchronously.\n- `async_update()`: Update documents asynchronously.\n- `async_insert()`: Insert documents asynchronously.\n- `async_delete()`: Delete documents asynchronously.\n\n```python\nasync def get_first_user():\n user = await User.objects.async_first()\n if user:\n print(f\"First user: {user.name}\")\n else:\n print(\"No users found\")\n\nasync def get_user_by_name(name):\n user = await User.objects(name=name).async_get()\n print(f\"User with name '{name}': {user.name}\")\n\nasync def count_users():\n count = await User.objects.async_count()\n print(f\"Number of users: {count}\")\n\nasync def update_user_name(old_name, new_name):\n updated = await User.objects(name=old_name).async_update(name=new_name)\n print(f\"Updated {updated} user(s) from '{old_name}' to '{new_name}'\")\n\nasync def insert_users(user_names):\n users = [User(name=name) for name in user_names]\n await User.objects.async_insert(users)\n \nasync def delete_users(name):\n await User.objects(name=name).async_delete()\n\nasync def main():\n await insert_users([\"Jane\", \"John\"])\n await get_first_user()\n await get_user_by_name(\"John\")\n await count_users()\n await update_user_name(\"John\", \"Johnny\")\n await get_user_by_name(\"Johnny\")\n await delete_users(\"Johnny\")\n\nif __name__ == \"__main__\":\n import asyncio\n asyncio.run(main())\n```\n\nTo iterate over the result set of a query asynchronously, \nyou can use the `async_to_list()` method:\n\n```python\nusers = await User.objects(name='Jane').async_to_list()\nasync for user in users:\n # Process each user \n ... \n```\n\nWe recommend using `async_to_list()` for small result sets. \n\n\n## Client-side Field Level Encryption\n\nMongoengine-plus introduces a new field type called `EncryptedStringField` that implements\nClient-side Field Level Encryption ([CSFLE](https://www.mongodb.com/docs/manual/core/csfle/))\nusing [pymongo](https://pymongo.readthedocs.io/en/stable/examples/encryption.html) encryption classes.\nThis feature allows explicit data encryption before sending it over the network to MongoDB,\nand automatic data decryption after reading from MongoDB. It supports both synchronous\nand asynchronous operations. Currently, the `EncryptedStringField` implementation supports\nthe AWS KMS service as the Key Management Service (KMS) provider.\n\n```python\nfrom mongoengine import Document, StringField\nfrom mongoengine_plus.types import EncryptedStringField\nfrom pymongo.encryption import Algorithm\n\n\nclass User(Document):\n id = StringField(primary_key=True)\n ssn = EncryptedStringField(\n algorithm=Algorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Deterministic\n )\n\n\nuser = User(id='US1', ssn='12345')\nuser.save()\nprint(user.ssn) # Output: '12345'\n\nuser_ = User.objects.get(id='US1')\nprint(user_.ssn) # Output: '12345'\n\n```\n\nThere are a few steps before you can start using `EncryptedStringField`. \n\n### 1. Create a Data Encryption Key (DEK)\n\nBefore using `EncryptedStringField`, you'll need to create a Data Encryption Key (DEK) \nfor encrypting and decrypting your data. The DEK should follow the recommended \nrequirements described in the official MongoDB documentation on [Keys and Key Vaults](https://www.mongodb.com/docs/manual/core/csfle/fundamentals/keys-key-vaults/#std-label-csfle-reference-keys-key-vaults).\nWe've provided a helper method to create your DEK easily.\n\n```python\nfrom mongoengine import connect\nfrom mongoengine_plus.types.encrypted_string.base import create_data_key\n\nconnect(host='mongo://localhost:27017/db')\n\ncreate_data_key(\n kms_provider=dict(\n aws=dict(\n accessKeyId='your-aws-key-id',\n secretAccessKey='your-aws-secret-access-key'\n )\n ),\n key_namespace='encryption.__keyVault',\n key_arn='arn:aws:kms:us-east-1:111122223333:key/your-key-id',\n key_name='my_key_name',\n kms_connection_url='https://kms.us-east-1.amazonaws.com',\n kms_region_name='us-east-1',\n)\n```\n\nYou'll need to execute this step only once during the project setup. Ensure that your \nMongoDB user has the necessary permissions for collection and index creation, and \naccess to the AWS KMS key.\n\n### 2. Configure `EncryptedStringField`\n\nSince `EncryptedStringField` needs to read the DEK from your MongoDB instance and access the \nKMS key for encryption/decryption, you'll need to configure it as follows. This \nconfiguration might be in your `__init__.py` file and should be executed once.\n\n```python\nfrom mongoengine import Document, StringField\nfrom mongoengine_plus.types import EncryptedStringField\nfrom pymongo.encryption import Algorithm\n\nEncryptedStringField.configure_aws_kms(\n 'encryption.__keyVault',\n 'my_key_name',\n 'your-aws-key-id',\n 'your-aws-secret-access-key',\n 'us-east-1',\n)\n\n\nclass User(Document):\n id = StringField(primary_key=True)\n ssn = EncryptedStringField(\n algorithm=Algorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Deterministic\n )\n```\n\nNow you are ready to go!\n\n### 3. Optimize KMS requests (optional)\n\nThere's a caveat in the `EncryptedStringField` implementation. Every time `EncryptedStringField` needs\nto encrypt or decrypt data, it uses the `pymongo.encryption.ClientEncryption`,\nwhich makes a request to the AWS KMS service endpoint. This can potentially slow down\nthe performance of reading and writing encrypted data to MongoDB. As a workaround,\nwe've created a function that patches this behavior and caches the data key.\n\n```python\nfrom mongoengine_plus.types.encrypted_string import cache_kms_data_key\n\n\ncache_kms_data_key(\n 'encryption.__keyVault',\n 'my_key_name',\n 'your-aws-key-id',\n 'your-aws-secret-access-key',\n 'us-east-1',\n 'https://kms.us-east-1.amazonaws.com',\n)\n```\n\nYou should execute this function once before making any database write or read operations,\nperhaps in your `__init__.py` file. It will retrieve the KMS key and cache it for\nsubsequent requests.\n",
"bugtrack_url": null,
"license": null,
"summary": "Extras for mongoengine",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/cuenca-mx/mongoengine-plus"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2bb413f54d63a8b9241bd7097da87fd4ae7c42e0917335422782d64c9aa448a6",
"md5": "0d6a7811042292d66cf7973ba38921d5",
"sha256": "2fa21052139b2bbb3479a942c8dfb4b633cbd53ba041c7b02cef215ab5649c59"
},
"downloads": -1,
"filename": "mongoengine_plus-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0d6a7811042292d66cf7973ba38921d5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 21577,
"upload_time": "2025-01-15T17:35:19",
"upload_time_iso_8601": "2025-01-15T17:35:19.123985Z",
"url": "https://files.pythonhosted.org/packages/2b/b4/13f54d63a8b9241bd7097da87fd4ae7c42e0917335422782d64c9aa448a6/mongoengine_plus-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c36d4125b8e2df22787d88eba1b230c90a93ad36b8fd9b6d3365133582b422f6",
"md5": "37ee227e3b8443d9bab78cfd362ca1a0",
"sha256": "5d330996dfbd235848796722563cb2f753bbdda7151eec205bcc5fcda5b4d2a7"
},
"downloads": -1,
"filename": "mongoengine_plus-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "37ee227e3b8443d9bab78cfd362ca1a0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 19339,
"upload_time": "2025-01-15T17:35:24",
"upload_time_iso_8601": "2025-01-15T17:35:24.315414Z",
"url": "https://files.pythonhosted.org/packages/c3/6d/4125b8e2df22787d88eba1b230c90a93ad36b8fd9b6d3365133582b422f6/mongoengine_plus-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-15 17:35:24",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cuenca-mx",
"github_project": "mongoengine-plus",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "blinker",
"specs": [
[
"==",
"1.9.0"
]
]
},
{
"name": "dnspython",
"specs": [
[
"==",
"2.7.0"
]
]
},
{
"name": "mongoengine",
"specs": [
[
"==",
"0.29.1"
]
]
},
{
"name": "pymongo",
"specs": [
[
"==",
"3.13.0"
]
]
},
{
"name": "pymongocrypt",
"specs": [
[
"==",
"1.12.2"
]
]
},
{
"name": "boto3",
"specs": [
[
"==",
"1.35.95"
]
]
}
],
"lcname": "mongoengine-plus"
}
JSON
