# Django SQLite EFS Backend
**django_sqlite_efs** is a custom database backend for Django, designed to work with **SQLite** on **AWS Lambda** using **Amazon EFS** (Elastic File System) and **Amazon DynamoDB**. This backend provides a solution to protect SQLite databases from corruption caused by concurrent writes in network-attached storage environments that lack proper file locking mechanisms for SQLite.
## Features
- Implements a distributed locking mechanism using **DynamoDB** to prevent concurrent write access to the SQLite database.
- Designed for environments like **AWS Lambda** where multiple instances may attempt to access the SQLite database simultaneously.
- Protects SQLite databases from corruption due to the limitations of EFS's advisory locking.
- Uses **Amazon DynamoDB** to coordinate database locks and ensure safe write operations.
## Requirements
- **Python**: 3.11 or higher
- **Django**: 5.1 or higher
- **SQLite**: No additional installation required (built-in with Python)
- **AWS Services**:
- **AWS Lambda**: For running the code.
- **Amazon EFS**: For storing the SQLite database.
- **Amazon DynamoDB**: For distributed locking to prevent concurrent writes.
## Installation
To install `django-sqlite-efs`, simply use pip:
```bash
pip install django-sqlite-efs
```
## DynamoDB Configuration
Create a DynamoDB table to support distributed locking. The table should have the following schema:
- **Primary Key**: `pk` (Type: String)
- **Optional**: Configure expiration for the `expires_at` field for automatic cleanup of expired locks.
### Example DynamoDB Table:
| Attribute Name | Type |
|----------------|---------|
| pk | String |
| lock_id | String |
| expires_at | Number |
Configuring `expires_at` with a TTL (Time-to-Live) policy is recommended for automatic removal of expired locks.
## Configuration
In your Django project, update the `settings.py` file to use the custom **database backend** provided by `django-sqlite-efs`:
```python
DATABASES = {
'default': {
'ENGINE': 'django_sqlite_efs',
'NAME': 'path_to_your_sqlite_db_file',
"OPTIONS": {
# `timeout` - number of seconds to wait for lock acquisition.
# It must be at least several seconds less than the timeout of
# your Lambda function. Default and minimum value is 3.
"timeout": timeout
# Setting `init_commands` is not recommended because it overrides
# default commands, which may lead to unexpected behavior.
}
}
}
```
Additionally, configure the following settings in `settings.py` or as environment variables:
- **SQLITE_LOCK_MAX_ATTEMPTS**: Maximum number of retries for acquiring a lock before raising an error (default: 10).
- **SQLITE_LOCK_EXPIRATION**: Lock expiration time in seconds (should be at least equal to or greater than the Lambda function's timeout).
- **SQLITE_LOCK_DYNAMODB_TABLE**: The name of the DynamoDB table used for locking.
### AWS Configuration
Ensure that your AWS credentials are correctly configured via environment variables or IAM roles. The package uses `boto3` to interact with DynamoDB. The Lambda function must have `PutItem` and `DeleteItem` permissions on the DynamoDB table.
## How It Works
SQLite uses file-based locking to prevent concurrent writes, but this is unreliable on **Amazon EFS** because EFS employs advisory locks. Advisory locks do not prevent processes from writing to a locked file if they have adequate permissions.
The **django_sqlite_efs backend** mitigates this by using **Amazon DynamoDB** to manage database locks:
- For each write operation (e.g., `INSERT`, `UPDATE`, `DELETE`), the backend attempts to acquire a lock in DynamoDB.
- All write operations lock the database for both reads and writes until the operation completes.
- If a lock cannot be acquired, the backend retries multiple times using exponential backoff.
- The lock is released when the write operation completes.
- Read-only queries (`SELECT`) do not acquire a lock, allowing for concurrent read access without blocking.
## Limitations
1. **Concurrent Writes**: This backend **does not** support concurrent write operations. During any write operation, the database is locked, blocking all reads and writes until the operation completes.
2. **High Latency**:
- **Read Latency**: Even for read-only requests, the latency for a typical Lambda execution with a Django app interacting with the database is over 100-150 ms due to the overhead of interacting with EFS.
- **Write Latency**: Write operations have a latency of 300 ms or more during a typical Lambda execution.
This solution is designed for environments where write operations are infrequent.
## License
This project is licensed under the **MIT License**. See the [LICENSE](LICENSE) file for more details.
Raw data
{
"_id": null,
"home_page": "https://github.com/efficient-solutions/django-sqlite-efs",
"name": "django-sqlite-efs",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "Django, SQLite, AWS, Amazon EFS, AWS Lambda, Amazon DynamoDB, Serverless",
"author": "Efficient Solutions LLC",
"author_email": "contact@efficient.solutions",
"download_url": "https://files.pythonhosted.org/packages/92/69/c1556514e06ef238d039770ebf748b5f615b5387b345e29e8c488ee862fd/django-sqlite-efs-0.1.0.tar.gz",
"platform": null,
"description": "# Django SQLite EFS Backend\n\n**django_sqlite_efs** is a custom database backend for Django, designed to work with **SQLite** on **AWS Lambda** using **Amazon EFS** (Elastic File System) and **Amazon DynamoDB**. This backend provides a solution to protect SQLite databases from corruption caused by concurrent writes in network-attached storage environments that lack proper file locking mechanisms for SQLite.\n\n## Features\n\n- Implements a distributed locking mechanism using **DynamoDB** to prevent concurrent write access to the SQLite database.\n- Designed for environments like **AWS Lambda** where multiple instances may attempt to access the SQLite database simultaneously.\n- Protects SQLite databases from corruption due to the limitations of EFS's advisory locking.\n- Uses **Amazon DynamoDB** to coordinate database locks and ensure safe write operations.\n\n## Requirements\n\n- **Python**: 3.11 or higher\n- **Django**: 5.1 or higher\n- **SQLite**: No additional installation required (built-in with Python)\n- **AWS Services**:\n - **AWS Lambda**: For running the code.\n - **Amazon EFS**: For storing the SQLite database.\n - **Amazon DynamoDB**: For distributed locking to prevent concurrent writes.\n\n## Installation\n\nTo install `django-sqlite-efs`, simply use pip:\n\n```bash\npip install django-sqlite-efs\n```\n\n## DynamoDB Configuration\n\nCreate a DynamoDB table to support distributed locking. The table should have the following schema:\n\n- **Primary Key**: `pk` (Type: String)\n- **Optional**: Configure expiration for the `expires_at` field for automatic cleanup of expired locks.\n\n### Example DynamoDB Table:\n\n| Attribute Name | Type |\n|----------------|---------|\n| pk | String |\n| lock_id | String |\n| expires_at | Number |\n\nConfiguring `expires_at` with a TTL (Time-to-Live) policy is recommended for automatic removal of expired locks.\n\n## Configuration\n\nIn your Django project, update the `settings.py` file to use the custom **database backend** provided by `django-sqlite-efs`:\n\n```python\nDATABASES = {\n 'default': {\n 'ENGINE': 'django_sqlite_efs',\n 'NAME': 'path_to_your_sqlite_db_file',\n \"OPTIONS\": {\n # `timeout` - number of seconds to wait for lock acquisition.\n # It must be at least several seconds less than the timeout of\n # your Lambda function. Default and minimum value is 3.\n \"timeout\": timeout\n # Setting `init_commands` is not recommended because it overrides\n # default commands, which may lead to unexpected behavior.\n }\n }\n}\n```\n\nAdditionally, configure the following settings in `settings.py` or as environment variables:\n\n- **SQLITE_LOCK_MAX_ATTEMPTS**: Maximum number of retries for acquiring a lock before raising an error (default: 10).\n- **SQLITE_LOCK_EXPIRATION**: Lock expiration time in seconds (should be at least equal to or greater than the Lambda function's timeout).\n- **SQLITE_LOCK_DYNAMODB_TABLE**: The name of the DynamoDB table used for locking.\n\n### AWS Configuration\n\nEnsure that your AWS credentials are correctly configured via environment variables or IAM roles. The package uses `boto3` to interact with DynamoDB. The Lambda function must have `PutItem` and `DeleteItem` permissions on the DynamoDB table.\n\n## How It Works\n\nSQLite uses file-based locking to prevent concurrent writes, but this is unreliable on **Amazon EFS** because EFS employs advisory locks. Advisory locks do not prevent processes from writing to a locked file if they have adequate permissions.\n\nThe **django_sqlite_efs backend** mitigates this by using **Amazon DynamoDB** to manage database locks:\n\n- For each write operation (e.g., `INSERT`, `UPDATE`, `DELETE`), the backend attempts to acquire a lock in DynamoDB. \n- All write operations lock the database for both reads and writes until the operation completes.\n- If a lock cannot be acquired, the backend retries multiple times using exponential backoff.\n- The lock is released when the write operation completes.\n- Read-only queries (`SELECT`) do not acquire a lock, allowing for concurrent read access without blocking.\n\n## Limitations\n\n1. **Concurrent Writes**: This backend **does not** support concurrent write operations. During any write operation, the database is locked, blocking all reads and writes until the operation completes.\n \n2. **High Latency**:\n - **Read Latency**: Even for read-only requests, the latency for a typical Lambda execution with a Django app interacting with the database is over 100-150 ms due to the overhead of interacting with EFS.\n - **Write Latency**: Write operations have a latency of 300 ms or more during a typical Lambda execution.\n\nThis solution is designed for environments where write operations are infrequent.\n\n## License\n\nThis project is licensed under the **MIT License**. See the [LICENSE](LICENSE) file for more details.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Django database backend for SQLite on Amazon EFS",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/efficient-solutions/django-sqlite-efs"
},
"split_keywords": [
"django",
" sqlite",
" aws",
" amazon efs",
" aws lambda",
" amazon dynamodb",
" serverless"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "99084f340a03ce9c700731d28f7ef734b97d48f0ce942c50afd1995f9c12b247",
"md5": "01066b4f4da83cb8ae0b2623d611bde0",
"sha256": "06c2111b225273ac7d0d48909f8e5d8ccac413d928c0d88acf96316eb489e2aa"
},
"downloads": -1,
"filename": "django_sqlite_efs-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "01066b4f4da83cb8ae0b2623d611bde0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 12116,
"upload_time": "2024-09-10T22:31:05",
"upload_time_iso_8601": "2024-09-10T22:31:05.718812Z",
"url": "https://files.pythonhosted.org/packages/99/08/4f340a03ce9c700731d28f7ef734b97d48f0ce942c50afd1995f9c12b247/django_sqlite_efs-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9269c1556514e06ef238d039770ebf748b5f615b5387b345e29e8c488ee862fd",
"md5": "a345ac5dc169f9b229f2ef281a89a1c8",
"sha256": "87b013c8ccea56b88fc770f8c877ebbdc587500b22940ea114b3702506a79273"
},
"downloads": -1,
"filename": "django-sqlite-efs-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "a345ac5dc169f9b229f2ef281a89a1c8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 15106,
"upload_time": "2024-09-10T22:31:07",
"upload_time_iso_8601": "2024-09-10T22:31:07.443516Z",
"url": "https://files.pythonhosted.org/packages/92/69/c1556514e06ef238d039770ebf748b5f615b5387b345e29e8c488ee862fd/django-sqlite-efs-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-10 22:31:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "efficient-solutions",
"github_project": "django-sqlite-efs",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "Django",
"specs": [
[
"<",
"5.2"
],
[
">=",
"5.1"
]
]
},
{
"name": "boto3",
"specs": [
[
"<",
"1.36"
],
[
">=",
"1.35"
]
]
},
{
"name": "pytest",
"specs": [
[
">=",
"8.3"
],
[
"<",
"8.4"
]
]
},
{
"name": "pytest-django",
"specs": [
[
">=",
"4.9"
],
[
"<",
"4.10"
]
]
},
{
"name": "moto",
"specs": [
[
"<",
"5.1"
],
[
">=",
"5.0"
]
]
},
{
"name": "twine",
"specs": [
[
"<",
"5.2"
],
[
">=",
"5.1"
]
]
}
],
"lcname": "django-sqlite-efs"
}
JSON
