# Django DRF Extensions
Advanced operation extensions for Django REST Framework providing intelligent sync/async routing with a clean, unified API design.
## Installation
```bash
pip install django-drf-extensions
```
### Requirements
- Python 3.11+
- Django 4.0+
- Django REST Framework 3.14+
- Celery 5.2+
- Redis 4.3+
- django-redis 5.2+
## Quick Setup
1. Add to your `INSTALLED_APPS`:
```python
INSTALLED_APPS = [
# ... your other apps
'rest_framework',
'django_drf_extensions',
]
```
2. Configure Redis cache:
```python
CACHES = {
'default': {
'BACKEND': 'django_redis.cache.RedisCache',
'LOCATION': 'redis://127.0.0.1:6379/1',
'OPTIONS': {
'CLIENT_CLASS': 'django_redis.client.DefaultClient',
}
}
}
```
3. Configure Celery:
```python
# settings.py
CELERY_BROKER_URL = 'redis://127.0.0.1:6379/0'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'
```
## Overview
This package extends Django REST Framework with a unified mixin that intelligently routes between synchronous and asynchronous processing based on your data size and requirements.
### The Extension System
1. **Enhanced Standard Endpoints**: Smart sync operations for immediate results
2. **Bulk Endpoints**: Dedicated async endpoints for large dataset processing
3. **Intelligent Routing**: Automatic sync/async decision based on dataset size
4. **Progress Tracking**: Redis-based monitoring for async operations
5. **Status Management**: Comprehensive operation status and results
## Features
- ✅ **Unified API Design**: Single mixin provides both sync and async capabilities
- ✅ **Smart Standard Endpoints**: Enhanced ViewSet methods with intelligent array handling
- ✅ **Dedicated Bulk Endpoints**: Clean `/bulk/` endpoints for async operations
- ✅ **Immediate Results**: Synchronous operations for small datasets with instant feedback
- ✅ **Scalable Processing**: Asynchronous operations for large datasets without blocking
- ✅ **Real-time Monitoring**: Live progress tracking and detailed status reporting
- ✅ **Comprehensive Error Handling**: Detailed validation and error reporting per item
- ✅ **Result Persistence**: Automatic caching of results for fast retrieval
- ✅ **Full Validation**: Complete DRF serializer validation ensuring data integrity
- ✅ **Transaction Safety**: Atomic database operations with rollback on failures
## Package Philosophy
This package provides a modern approach to scalable operations by offering:
1. **Clean API Design**: Enhances existing endpoints rather than creating parallel ones
2. **Intelligent Processing**: Automatically determines optimal processing method
3. **Unified Architecture**: Single mixin extends your ViewSets without complexity
4. **Production-Ready**: Built-in monitoring, error handling, and progress tracking
## Usage
### Adding Extensions to a ViewSet
```python
from rest_framework import viewsets
from django_drf_extensions.mixins import OperationsMixin
class ContractViewSet(OperationsMixin, viewsets.ModelViewSet):
"""
Enhanced ViewSet with intelligent operation routing.
Provides:
- Standard CRUD operations
- Smart sync operations for small datasets
- Bulk async operations for large datasets
"""
queryset = Contract.objects.all()
serializer_class = ContractSerializer
```
Your ViewSet now provides these endpoints:
```bash
# Standard ModelViewSet endpoints (enhanced for arrays)
GET /api/contracts/ # List (enhanced with ?ids= support)
POST /api/contracts/ # Create (enhanced with array + ?unique_fields= support)
GET /api/contracts/{id}/ # Retrieve single
PATCH /api/contracts/ # Update (enhanced with array + ?unique_fields= support)
PUT /api/contracts/ # Replace (enhanced with array + ?unique_fields= support)
DELETE /api/contracts/{id}/ # Delete single
# Bulk endpoints for async processing
GET /api/contracts/bulk/ # Async retrieve multiple
POST /api/contracts/bulk/ # Async create multiple
PATCH /api/contracts/bulk/ # Async update/upsert multiple
PUT /api/contracts/bulk/ # Async replace/upsert multiple
DELETE /api/contracts/bulk/ # Async delete multiple
# Operation monitoring
GET /api/operations/{task_id}/status/ # Check async status
```
## API Design
### Enhanced Standard Endpoints (Sync Operations)
Smart enhancements to standard ViewSet methods for immediate results with small datasets:
#### Multi-Get
```bash
# Small dataset - immediate results
GET /api/contracts/?ids=1,2,3,4,5
# Response (immediate)
{
"count": 5,
"results": [...],
"is_sync": true
}
```
#### Sync Upsert
```bash
# Small dataset - immediate upsert
POST /api/contracts/?unique_fields=contract_number,year
Content-Type: application/json
[
{"contract_number": "C001", "year": 2024, "amount": 1000},
{"contract_number": "C002", "year": 2024, "amount": 2000}
]
# Response (immediate)
{
"message": "Upsert completed successfully",
"total_items": 2,
"created_count": 1,
"updated_count": 1,
"created_ids": [123],
"updated_ids": [124],
"is_sync": true
}
```
### Bulk Endpoints (Async Operations)
Dedicated endpoints for large dataset processing via background tasks:
#### Async Multi-Get
```bash
# Large dataset - background processing
GET /api/contracts/bulk/?ids=1,2,3,...,1000
# Response (task started)
{
"message": "Bulk get task started for 1000 items",
"task_id": "abc123-def456",
"status_url": "/api/operations/abc123-def456/status/",
"is_async": true
}
```
#### Async Upsert
```bash
# Large dataset - background processing
PATCH /api/contracts/bulk/?unique_fields=contract_number,year
Content-Type: application/json
[
{"contract_number": "C001", "year": 2024, "amount": 1000},
... // 500+ items
]
# Response (task started)
{
"message": "Bulk upsert task started for 500 items",
"task_id": "def456-ghi789",
"unique_fields": ["contract_number", "year"],
"status_url": "/api/operations/def456-ghi789/status/"
}
```
## Operation Types
### Sync Operations (Standard Endpoints)
- **Best for**: ≤50-100 items, immediate results needed
- **Use cases**: Real-time user interactions, form submissions, small API integrations
- **Response**: Immediate data or results
- **Endpoints**: Enhanced standard ViewSet methods
### Async Operations (Bulk Endpoints)
- **Best for**: >100 items, can wait for results
- **Use cases**: Data imports, batch processing, CSV uploads
- **Response**: Task ID for monitoring
- **Endpoints**: Dedicated `/bulk/` endpoints
## Configuration
### Custom Settings
```python
# Operation Settings
DRF_EXT_CHUNK_SIZE = 100 # Items per processing chunk
DRF_EXT_MAX_RECORDS = 10000 # Maximum records per operation
DRF_EXT_BATCH_SIZE = 1000 # Database batch size
DRF_EXT_CACHE_TIMEOUT = 86400 # Cache timeout (24 hours)
DRF_EXT_PROGRESS_UPDATE_INTERVAL = 10 # Progress update frequency
# Sync Operation Limits
DRF_EXT_SYNC_MAX_ITEMS = 50 # Max items for sync operations
DRF_EXT_SYNC_MULTI_GET_MAX = 100 # Max items for sync multi-get
```
## Example Usage
### Basic Contract Management
```python
# Small dataset - sync operations (immediate results)
curl -X POST "/api/contracts/?unique_fields=contract_number" \
-H "Content-Type: application/json" \
-d '[
{"contract_number": "C001", "amount": 1000},
{"contract_number": "C002", "amount": 2000}
]'
# Large dataset - async operations (background processing)
curl -X POST "/api/contracts/bulk/" \
-H "Content-Type: application/json" \
-d '[...500 contracts...]'
# Check async status
curl "/api/operations/{task_id}/status/"
```
### Migration from Previous Versions
If you're coming from older versions:
```python
# Old (separate mixins)
class ContractViewSet(SyncUpsertMixin, AsyncOperationsMixin, viewsets.ModelViewSet):
queryset = Contract.objects.all()
serializer_class = ContractSerializer
# New (unified mixin)
class ContractViewSet(OperationsMixin, viewsets.ModelViewSet):
queryset = Contract.objects.all()
serializer_class = ContractSerializer
```
### Endpoint Changes
| Operation | Old Endpoints | New Endpoints |
|-----------|---------------|---------------|
| Sync Upsert | `POST /upsert/` | `POST /?unique_fields=...` |
| Async Create | `POST /operations/` | `POST /bulk/` |
| Async Update | `PATCH /operations/` | `PATCH /bulk/` |
| Async Delete | `DELETE /operations/` | `DELETE /bulk/` |
## Error Handling
The system provides comprehensive error handling:
- **Validation Errors**: Field-level validation using DRF serializers
- **Size Limits**: Automatic routing suggestion for oversized sync requests
- **Database Errors**: Transaction rollback on failures
- **Task Failures**: Detailed error reporting in async task status
## Performance Considerations
- **Database Efficiency**: Uses optimized database operations for all bulk processing
- **Memory Management**: Processes large datasets in configurable chunks
- **Intelligent Routing**: Automatic sync/async decision based on dataset size
- **Progress Tracking**: Redis-based monitoring without database overhead
- **Result Caching**: Efficient caching of async operation results
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## License
This project is licensed under the MIT License.
Raw data
{
"_id": null,
"home_page": "https://github.com/AugendLimited/django-drf-extensions",
"name": "django-drf-extensions",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.11",
"maintainer_email": null,
"keywords": "django, drf, rest-framework, celery, redis, async, upsert, sync, operations, extensions",
"author": "Konrad Beck",
"author_email": "konrad.beck@merchantcapital.co.za",
"download_url": "https://files.pythonhosted.org/packages/59/c7/a35791953be3e2b173914cddd46721ce91f02e9be7c9003af7d2bb4940f9/django_drf_extensions-0.0.9.tar.gz",
"platform": null,
"description": "# Django DRF Extensions\n\nAdvanced operation extensions for Django REST Framework providing intelligent sync/async routing with a clean, unified API design.\n\n## Installation\n\n```bash\npip install django-drf-extensions\n```\n\n### Requirements\n\n- Python 3.11+\n- Django 4.0+\n- Django REST Framework 3.14+\n- Celery 5.2+\n- Redis 4.3+\n- django-redis 5.2+\n\n## Quick Setup\n\n1. Add to your `INSTALLED_APPS`:\n```python\nINSTALLED_APPS = [\n # ... your other apps\n 'rest_framework',\n 'django_drf_extensions',\n]\n```\n\n2. Configure Redis cache:\n```python\nCACHES = {\n 'default': {\n 'BACKEND': 'django_redis.cache.RedisCache',\n 'LOCATION': 'redis://127.0.0.1:6379/1',\n 'OPTIONS': {\n 'CLIENT_CLASS': 'django_redis.client.DefaultClient',\n }\n }\n}\n```\n\n3. Configure Celery:\n```python\n# settings.py\nCELERY_BROKER_URL = 'redis://127.0.0.1:6379/0'\nCELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'\n```\n\n## Overview\n\nThis package extends Django REST Framework with a unified mixin that intelligently routes between synchronous and asynchronous processing based on your data size and requirements.\n\n### The Extension System\n\n1. **Enhanced Standard Endpoints**: Smart sync operations for immediate results\n2. **Bulk Endpoints**: Dedicated async endpoints for large dataset processing \n3. **Intelligent Routing**: Automatic sync/async decision based on dataset size\n4. **Progress Tracking**: Redis-based monitoring for async operations\n5. **Status Management**: Comprehensive operation status and results\n\n## Features\n\n- \u2705 **Unified API Design**: Single mixin provides both sync and async capabilities\n- \u2705 **Smart Standard Endpoints**: Enhanced ViewSet methods with intelligent array handling\n- \u2705 **Dedicated Bulk Endpoints**: Clean `/bulk/` endpoints for async operations\n- \u2705 **Immediate Results**: Synchronous operations for small datasets with instant feedback\n- \u2705 **Scalable Processing**: Asynchronous operations for large datasets without blocking\n- \u2705 **Real-time Monitoring**: Live progress tracking and detailed status reporting\n- \u2705 **Comprehensive Error Handling**: Detailed validation and error reporting per item\n- \u2705 **Result Persistence**: Automatic caching of results for fast retrieval\n- \u2705 **Full Validation**: Complete DRF serializer validation ensuring data integrity\n- \u2705 **Transaction Safety**: Atomic database operations with rollback on failures\n\n## Package Philosophy\n\nThis package provides a modern approach to scalable operations by offering:\n\n1. **Clean API Design**: Enhances existing endpoints rather than creating parallel ones\n2. **Intelligent Processing**: Automatically determines optimal processing method\n3. **Unified Architecture**: Single mixin extends your ViewSets without complexity\n4. **Production-Ready**: Built-in monitoring, error handling, and progress tracking\n\n## Usage\n\n### Adding Extensions to a ViewSet\n\n```python\nfrom rest_framework import viewsets\nfrom django_drf_extensions.mixins import OperationsMixin\n\nclass ContractViewSet(OperationsMixin, viewsets.ModelViewSet):\n \"\"\"\n Enhanced ViewSet with intelligent operation routing.\n \n Provides:\n - Standard CRUD operations\n - Smart sync operations for small datasets\n - Bulk async operations for large datasets\n \"\"\"\n queryset = Contract.objects.all()\n serializer_class = ContractSerializer\n```\n\nYour ViewSet now provides these endpoints:\n\n```bash\n# Standard ModelViewSet endpoints (enhanced for arrays)\nGET /api/contracts/ # List (enhanced with ?ids= support)\nPOST /api/contracts/ # Create (enhanced with array + ?unique_fields= support)\nGET /api/contracts/{id}/ # Retrieve single\nPATCH /api/contracts/ # Update (enhanced with array + ?unique_fields= support)\nPUT /api/contracts/ # Replace (enhanced with array + ?unique_fields= support)\nDELETE /api/contracts/{id}/ # Delete single\n\n# Bulk endpoints for async processing\nGET /api/contracts/bulk/ # Async retrieve multiple\nPOST /api/contracts/bulk/ # Async create multiple \nPATCH /api/contracts/bulk/ # Async update/upsert multiple\nPUT /api/contracts/bulk/ # Async replace/upsert multiple\nDELETE /api/contracts/bulk/ # Async delete multiple\n\n# Operation monitoring\nGET /api/operations/{task_id}/status/ # Check async status\n```\n\n## API Design\n\n### Enhanced Standard Endpoints (Sync Operations)\n\nSmart enhancements to standard ViewSet methods for immediate results with small datasets:\n\n#### Multi-Get\n```bash\n# Small dataset - immediate results\nGET /api/contracts/?ids=1,2,3,4,5\n\n# Response (immediate)\n{\n \"count\": 5,\n \"results\": [...],\n \"is_sync\": true\n}\n```\n\n#### Sync Upsert\n```bash\n# Small dataset - immediate upsert\nPOST /api/contracts/?unique_fields=contract_number,year\nContent-Type: application/json\n[\n {\"contract_number\": \"C001\", \"year\": 2024, \"amount\": 1000},\n {\"contract_number\": \"C002\", \"year\": 2024, \"amount\": 2000}\n]\n\n# Response (immediate)\n{\n \"message\": \"Upsert completed successfully\",\n \"total_items\": 2,\n \"created_count\": 1,\n \"updated_count\": 1,\n \"created_ids\": [123],\n \"updated_ids\": [124],\n \"is_sync\": true\n}\n```\n\n### Bulk Endpoints (Async Operations)\n\nDedicated endpoints for large dataset processing via background tasks:\n\n#### Async Multi-Get\n```bash\n# Large dataset - background processing\nGET /api/contracts/bulk/?ids=1,2,3,...,1000\n\n# Response (task started)\n{\n \"message\": \"Bulk get task started for 1000 items\",\n \"task_id\": \"abc123-def456\",\n \"status_url\": \"/api/operations/abc123-def456/status/\",\n \"is_async\": true\n}\n```\n\n#### Async Upsert\n```bash\n# Large dataset - background processing\nPATCH /api/contracts/bulk/?unique_fields=contract_number,year\nContent-Type: application/json\n[\n {\"contract_number\": \"C001\", \"year\": 2024, \"amount\": 1000},\n ... // 500+ items\n]\n\n# Response (task started)\n{\n \"message\": \"Bulk upsert task started for 500 items\",\n \"task_id\": \"def456-ghi789\",\n \"unique_fields\": [\"contract_number\", \"year\"],\n \"status_url\": \"/api/operations/def456-ghi789/status/\"\n}\n```\n\n## Operation Types\n\n### Sync Operations (Standard Endpoints)\n- **Best for**: \u226450-100 items, immediate results needed\n- **Use cases**: Real-time user interactions, form submissions, small API integrations\n- **Response**: Immediate data or results\n- **Endpoints**: Enhanced standard ViewSet methods\n\n### Async Operations (Bulk Endpoints) \n- **Best for**: >100 items, can wait for results\n- **Use cases**: Data imports, batch processing, CSV uploads\n- **Response**: Task ID for monitoring\n- **Endpoints**: Dedicated `/bulk/` endpoints\n\n## Configuration\n\n### Custom Settings\n\n```python\n# Operation Settings\nDRF_EXT_CHUNK_SIZE = 100 # Items per processing chunk\nDRF_EXT_MAX_RECORDS = 10000 # Maximum records per operation\nDRF_EXT_BATCH_SIZE = 1000 # Database batch size\nDRF_EXT_CACHE_TIMEOUT = 86400 # Cache timeout (24 hours)\nDRF_EXT_PROGRESS_UPDATE_INTERVAL = 10 # Progress update frequency\n\n# Sync Operation Limits\nDRF_EXT_SYNC_MAX_ITEMS = 50 # Max items for sync operations\nDRF_EXT_SYNC_MULTI_GET_MAX = 100 # Max items for sync multi-get\n```\n\n## Example Usage\n\n### Basic Contract Management\n\n```python\n# Small dataset - sync operations (immediate results)\ncurl -X POST \"/api/contracts/?unique_fields=contract_number\" \\\n -H \"Content-Type: application/json\" \\\n -d '[\n {\"contract_number\": \"C001\", \"amount\": 1000},\n {\"contract_number\": \"C002\", \"amount\": 2000}\n ]'\n\n# Large dataset - async operations (background processing)\ncurl -X POST \"/api/contracts/bulk/\" \\\n -H \"Content-Type: application/json\" \\\n -d '[...500 contracts...]'\n\n# Check async status\ncurl \"/api/operations/{task_id}/status/\"\n```\n\n### Migration from Previous Versions\n\nIf you're coming from older versions:\n\n```python\n# Old (separate mixins)\nclass ContractViewSet(SyncUpsertMixin, AsyncOperationsMixin, viewsets.ModelViewSet):\n queryset = Contract.objects.all()\n serializer_class = ContractSerializer\n\n# New (unified mixin)\nclass ContractViewSet(OperationsMixin, viewsets.ModelViewSet):\n queryset = Contract.objects.all()\n serializer_class = ContractSerializer\n```\n\n### Endpoint Changes\n\n| Operation | Old Endpoints | New Endpoints |\n|-----------|---------------|---------------|\n| Sync Upsert | `POST /upsert/` | `POST /?unique_fields=...` |\n| Async Create | `POST /operations/` | `POST /bulk/` |\n| Async Update | `PATCH /operations/` | `PATCH /bulk/` |\n| Async Delete | `DELETE /operations/` | `DELETE /bulk/` |\n\n## Error Handling\n\nThe system provides comprehensive error handling:\n\n- **Validation Errors**: Field-level validation using DRF serializers\n- **Size Limits**: Automatic routing suggestion for oversized sync requests\n- **Database Errors**: Transaction rollback on failures\n- **Task Failures**: Detailed error reporting in async task status\n\n## Performance Considerations\n\n- **Database Efficiency**: Uses optimized database operations for all bulk processing\n- **Memory Management**: Processes large datasets in configurable chunks\n- **Intelligent Routing**: Automatic sync/async decision based on dataset size\n- **Progress Tracking**: Redis-based monitoring without database overhead\n- **Result Caching**: Efficient caching of async operation results\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Django REST Framework extensions for efficient async and synchronous operations with Celery and Redis",
"version": "0.0.9",
"project_urls": {
"Homepage": "https://github.com/AugendLimited/django-drf-extensions",
"Repository": "https://github.com/AugendLimited/django-drf-extensions"
},
"split_keywords": [
"django",
" drf",
" rest-framework",
" celery",
" redis",
" async",
" upsert",
" sync",
" operations",
" extensions"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "03e2570b86a9686c337db7951c0188d81f23fdd6880fd437ebe9e92be9f50231",
"md5": "1cca78a56817ea338e4fd4473be23a1e",
"sha256": "b3b5b6e8d72f72655e62441f5685dd7899b55b35be122f17d31db1eefe4ba3cd"
},
"downloads": -1,
"filename": "django_drf_extensions-0.0.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1cca78a56817ea338e4fd4473be23a1e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.11",
"size": 23014,
"upload_time": "2025-07-18T10:59:29",
"upload_time_iso_8601": "2025-07-18T10:59:29.607227Z",
"url": "https://files.pythonhosted.org/packages/03/e2/570b86a9686c337db7951c0188d81f23fdd6880fd437ebe9e92be9f50231/django_drf_extensions-0.0.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "59c7a35791953be3e2b173914cddd46721ce91f02e9be7c9003af7d2bb4940f9",
"md5": "8feaf5693b3513d96b390658be899c3b",
"sha256": "e7fc6271862e3468d7ac501d7d5e248e6204b9f8288f6de7475ff5d94e1230ca"
},
"downloads": -1,
"filename": "django_drf_extensions-0.0.9.tar.gz",
"has_sig": false,
"md5_digest": "8feaf5693b3513d96b390658be899c3b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.11",
"size": 20658,
"upload_time": "2025-07-18T10:59:31",
"upload_time_iso_8601": "2025-07-18T10:59:31.022503Z",
"url": "https://files.pythonhosted.org/packages/59/c7/a35791953be3e2b173914cddd46721ce91f02e9be7c9003af7d2bb4940f9/django_drf_extensions-0.0.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-18 10:59:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "AugendLimited",
"github_project": "django-drf-extensions",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "django-drf-extensions"
}