django-drf-extensions


Namedjango-drf-extensions JSON
Version 0.0.9 PyPI version JSON
download
home_pagehttps://github.com/AugendLimited/django-drf-extensions
SummaryDjango REST Framework extensions for efficient async and synchronous operations with Celery and Redis
upload_time2025-07-18 10:59:31
maintainerNone
docs_urlNone
authorKonrad Beck
requires_python<4.0,>=3.11
licenseMIT
keywords django drf rest-framework celery redis async upsert sync operations extensions
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # Django DRF Extensions

Advanced operation extensions for Django REST Framework providing intelligent sync/async routing with a clean, unified API design.

## Installation

```bash
pip install django-drf-extensions
```

### Requirements

- Python 3.11+
- Django 4.0+
- Django REST Framework 3.14+
- Celery 5.2+
- Redis 4.3+
- django-redis 5.2+

## Quick Setup

1. Add to your `INSTALLED_APPS`:
```python
INSTALLED_APPS = [
    # ... your other apps
    'rest_framework',
    'django_drf_extensions',
]
```

2. Configure Redis cache:
```python
CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379/1',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
        }
    }
}
```

3. Configure Celery:
```python
# settings.py
CELERY_BROKER_URL = 'redis://127.0.0.1:6379/0'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'
```

## Overview

This package extends Django REST Framework with a unified mixin that intelligently routes between synchronous and asynchronous processing based on your data size and requirements.

### The Extension System

1. **Enhanced Standard Endpoints**: Smart sync operations for immediate results
2. **Bulk Endpoints**: Dedicated async endpoints for large dataset processing  
3. **Intelligent Routing**: Automatic sync/async decision based on dataset size
4. **Progress Tracking**: Redis-based monitoring for async operations
5. **Status Management**: Comprehensive operation status and results

## Features

- ✅ **Unified API Design**: Single mixin provides both sync and async capabilities
- ✅ **Smart Standard Endpoints**: Enhanced ViewSet methods with intelligent array handling
- ✅ **Dedicated Bulk Endpoints**: Clean `/bulk/` endpoints for async operations
- ✅ **Immediate Results**: Synchronous operations for small datasets with instant feedback
- ✅ **Scalable Processing**: Asynchronous operations for large datasets without blocking
- ✅ **Real-time Monitoring**: Live progress tracking and detailed status reporting
- ✅ **Comprehensive Error Handling**: Detailed validation and error reporting per item
- ✅ **Result Persistence**: Automatic caching of results for fast retrieval
- ✅ **Full Validation**: Complete DRF serializer validation ensuring data integrity
- ✅ **Transaction Safety**: Atomic database operations with rollback on failures

## Package Philosophy

This package provides a modern approach to scalable operations by offering:

1. **Clean API Design**: Enhances existing endpoints rather than creating parallel ones
2. **Intelligent Processing**: Automatically determines optimal processing method
3. **Unified Architecture**: Single mixin extends your ViewSets without complexity
4. **Production-Ready**: Built-in monitoring, error handling, and progress tracking

## Usage

### Adding Extensions to a ViewSet

```python
from rest_framework import viewsets
from django_drf_extensions.mixins import OperationsMixin

class ContractViewSet(OperationsMixin, viewsets.ModelViewSet):
    """
    Enhanced ViewSet with intelligent operation routing.
    
    Provides:
    - Standard CRUD operations
    - Smart sync operations for small datasets
    - Bulk async operations for large datasets
    """
    queryset = Contract.objects.all()
    serializer_class = ContractSerializer
```

Your ViewSet now provides these endpoints:

```bash
# Standard ModelViewSet endpoints (enhanced for arrays)
GET    /api/contracts/                    # List (enhanced with ?ids= support)
POST   /api/contracts/                    # Create (enhanced with array + ?unique_fields= support)
GET    /api/contracts/{id}/               # Retrieve single
PATCH  /api/contracts/                    # Update (enhanced with array + ?unique_fields= support)
PUT    /api/contracts/                    # Replace (enhanced with array + ?unique_fields= support)
DELETE /api/contracts/{id}/               # Delete single

# Bulk endpoints for async processing
GET    /api/contracts/bulk/               # Async retrieve multiple
POST   /api/contracts/bulk/               # Async create multiple  
PATCH  /api/contracts/bulk/               # Async update/upsert multiple
PUT    /api/contracts/bulk/               # Async replace/upsert multiple
DELETE /api/contracts/bulk/               # Async delete multiple

# Operation monitoring
GET    /api/operations/{task_id}/status/  # Check async status
```

## API Design

### Enhanced Standard Endpoints (Sync Operations)

Smart enhancements to standard ViewSet methods for immediate results with small datasets:

#### Multi-Get
```bash
# Small dataset - immediate results
GET /api/contracts/?ids=1,2,3,4,5

# Response (immediate)
{
  "count": 5,
  "results": [...],
  "is_sync": true
}
```

#### Sync Upsert
```bash
# Small dataset - immediate upsert
POST /api/contracts/?unique_fields=contract_number,year
Content-Type: application/json
[
  {"contract_number": "C001", "year": 2024, "amount": 1000},
  {"contract_number": "C002", "year": 2024, "amount": 2000}
]

# Response (immediate)
{
  "message": "Upsert completed successfully",
  "total_items": 2,
  "created_count": 1,
  "updated_count": 1,
  "created_ids": [123],
  "updated_ids": [124],
  "is_sync": true
}
```

### Bulk Endpoints (Async Operations)

Dedicated endpoints for large dataset processing via background tasks:

#### Async Multi-Get
```bash
# Large dataset - background processing
GET /api/contracts/bulk/?ids=1,2,3,...,1000

# Response (task started)
{
  "message": "Bulk get task started for 1000 items",
  "task_id": "abc123-def456",
  "status_url": "/api/operations/abc123-def456/status/",
  "is_async": true
}
```

#### Async Upsert
```bash
# Large dataset - background processing
PATCH /api/contracts/bulk/?unique_fields=contract_number,year
Content-Type: application/json
[
  {"contract_number": "C001", "year": 2024, "amount": 1000},
  ... // 500+ items
]

# Response (task started)
{
  "message": "Bulk upsert task started for 500 items",
  "task_id": "def456-ghi789",
  "unique_fields": ["contract_number", "year"],
  "status_url": "/api/operations/def456-ghi789/status/"
}
```

## Operation Types

### Sync Operations (Standard Endpoints)
- **Best for**: ≤50-100 items, immediate results needed
- **Use cases**: Real-time user interactions, form submissions, small API integrations
- **Response**: Immediate data or results
- **Endpoints**: Enhanced standard ViewSet methods

### Async Operations (Bulk Endpoints)  
- **Best for**: >100 items, can wait for results
- **Use cases**: Data imports, batch processing, CSV uploads
- **Response**: Task ID for monitoring
- **Endpoints**: Dedicated `/bulk/` endpoints

## Configuration

### Custom Settings

```python
# Operation Settings
DRF_EXT_CHUNK_SIZE = 100                    # Items per processing chunk
DRF_EXT_MAX_RECORDS = 10000                 # Maximum records per operation
DRF_EXT_BATCH_SIZE = 1000                   # Database batch size
DRF_EXT_CACHE_TIMEOUT = 86400               # Cache timeout (24 hours)
DRF_EXT_PROGRESS_UPDATE_INTERVAL = 10       # Progress update frequency

# Sync Operation Limits
DRF_EXT_SYNC_MAX_ITEMS = 50                 # Max items for sync operations
DRF_EXT_SYNC_MULTI_GET_MAX = 100            # Max items for sync multi-get
```

## Example Usage

### Basic Contract Management

```python
# Small dataset - sync operations (immediate results)
curl -X POST "/api/contracts/?unique_fields=contract_number" \
  -H "Content-Type: application/json" \
  -d '[
    {"contract_number": "C001", "amount": 1000},
    {"contract_number": "C002", "amount": 2000}
  ]'

# Large dataset - async operations (background processing)
curl -X POST "/api/contracts/bulk/" \
  -H "Content-Type: application/json" \
  -d '[...500 contracts...]'

# Check async status
curl "/api/operations/{task_id}/status/"
```

### Migration from Previous Versions

If you're coming from older versions:

```python
# Old (separate mixins)
class ContractViewSet(SyncUpsertMixin, AsyncOperationsMixin, viewsets.ModelViewSet):
    queryset = Contract.objects.all()
    serializer_class = ContractSerializer

# New (unified mixin)
class ContractViewSet(OperationsMixin, viewsets.ModelViewSet):
    queryset = Contract.objects.all()
    serializer_class = ContractSerializer
```

### Endpoint Changes

| Operation | Old Endpoints | New Endpoints |
|-----------|---------------|---------------|
| Sync Upsert | `POST /upsert/` | `POST /?unique_fields=...` |
| Async Create | `POST /operations/` | `POST /bulk/` |
| Async Update | `PATCH /operations/` | `PATCH /bulk/` |
| Async Delete | `DELETE /operations/` | `DELETE /bulk/` |

## Error Handling

The system provides comprehensive error handling:

- **Validation Errors**: Field-level validation using DRF serializers
- **Size Limits**: Automatic routing suggestion for oversized sync requests
- **Database Errors**: Transaction rollback on failures
- **Task Failures**: Detailed error reporting in async task status

## Performance Considerations

- **Database Efficiency**: Uses optimized database operations for all bulk processing
- **Memory Management**: Processes large datasets in configurable chunks
- **Intelligent Routing**: Automatic sync/async decision based on dataset size
- **Progress Tracking**: Redis-based monitoring without database overhead
- **Result Caching**: Efficient caching of async operation results

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

This project is licensed under the MIT License.

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/AugendLimited/django-drf-extensions",
    "name": "django-drf-extensions",
    "maintainer": null,
    "docs_url": null,
    "requires_python": "<4.0,>=3.11",
    "maintainer_email": null,
    "keywords": "django, drf, rest-framework, celery, redis, async, upsert, sync, operations, extensions",
    "author": "Konrad Beck",
    "author_email": "konrad.beck@merchantcapital.co.za",
    "download_url": "https://files.pythonhosted.org/packages/59/c7/a35791953be3e2b173914cddd46721ce91f02e9be7c9003af7d2bb4940f9/django_drf_extensions-0.0.9.tar.gz",
    "platform": null,
    "description": "# Django DRF Extensions\n\nAdvanced operation extensions for Django REST Framework providing intelligent sync/async routing with a clean, unified API design.\n\n## Installation\n\n```bash\npip install django-drf-extensions\n```\n\n### Requirements\n\n- Python 3.11+\n- Django 4.0+\n- Django REST Framework 3.14+\n- Celery 5.2+\n- Redis 4.3+\n- django-redis 5.2+\n\n## Quick Setup\n\n1. Add to your `INSTALLED_APPS`:\n```python\nINSTALLED_APPS = [\n    # ... your other apps\n    'rest_framework',\n    'django_drf_extensions',\n]\n```\n\n2. Configure Redis cache:\n```python\nCACHES = {\n    'default': {\n        'BACKEND': 'django_redis.cache.RedisCache',\n        'LOCATION': 'redis://127.0.0.1:6379/1',\n        'OPTIONS': {\n            'CLIENT_CLASS': 'django_redis.client.DefaultClient',\n        }\n    }\n}\n```\n\n3. Configure Celery:\n```python\n# settings.py\nCELERY_BROKER_URL = 'redis://127.0.0.1:6379/0'\nCELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'\n```\n\n## Overview\n\nThis package extends Django REST Framework with a unified mixin that intelligently routes between synchronous and asynchronous processing based on your data size and requirements.\n\n### The Extension System\n\n1. **Enhanced Standard Endpoints**: Smart sync operations for immediate results\n2. **Bulk Endpoints**: Dedicated async endpoints for large dataset processing  \n3. **Intelligent Routing**: Automatic sync/async decision based on dataset size\n4. **Progress Tracking**: Redis-based monitoring for async operations\n5. **Status Management**: Comprehensive operation status and results\n\n## Features\n\n- \u2705 **Unified API Design**: Single mixin provides both sync and async capabilities\n- \u2705 **Smart Standard Endpoints**: Enhanced ViewSet methods with intelligent array handling\n- \u2705 **Dedicated Bulk Endpoints**: Clean `/bulk/` endpoints for async operations\n- \u2705 **Immediate Results**: Synchronous operations for small datasets with instant feedback\n- \u2705 **Scalable Processing**: Asynchronous operations for large datasets without blocking\n- \u2705 **Real-time Monitoring**: Live progress tracking and detailed status reporting\n- \u2705 **Comprehensive Error Handling**: Detailed validation and error reporting per item\n- \u2705 **Result Persistence**: Automatic caching of results for fast retrieval\n- \u2705 **Full Validation**: Complete DRF serializer validation ensuring data integrity\n- \u2705 **Transaction Safety**: Atomic database operations with rollback on failures\n\n## Package Philosophy\n\nThis package provides a modern approach to scalable operations by offering:\n\n1. **Clean API Design**: Enhances existing endpoints rather than creating parallel ones\n2. **Intelligent Processing**: Automatically determines optimal processing method\n3. **Unified Architecture**: Single mixin extends your ViewSets without complexity\n4. **Production-Ready**: Built-in monitoring, error handling, and progress tracking\n\n## Usage\n\n### Adding Extensions to a ViewSet\n\n```python\nfrom rest_framework import viewsets\nfrom django_drf_extensions.mixins import OperationsMixin\n\nclass ContractViewSet(OperationsMixin, viewsets.ModelViewSet):\n    \"\"\"\n    Enhanced ViewSet with intelligent operation routing.\n    \n    Provides:\n    - Standard CRUD operations\n    - Smart sync operations for small datasets\n    - Bulk async operations for large datasets\n    \"\"\"\n    queryset = Contract.objects.all()\n    serializer_class = ContractSerializer\n```\n\nYour ViewSet now provides these endpoints:\n\n```bash\n# Standard ModelViewSet endpoints (enhanced for arrays)\nGET    /api/contracts/                    # List (enhanced with ?ids= support)\nPOST   /api/contracts/                    # Create (enhanced with array + ?unique_fields= support)\nGET    /api/contracts/{id}/               # Retrieve single\nPATCH  /api/contracts/                    # Update (enhanced with array + ?unique_fields= support)\nPUT    /api/contracts/                    # Replace (enhanced with array + ?unique_fields= support)\nDELETE /api/contracts/{id}/               # Delete single\n\n# Bulk endpoints for async processing\nGET    /api/contracts/bulk/               # Async retrieve multiple\nPOST   /api/contracts/bulk/               # Async create multiple  \nPATCH  /api/contracts/bulk/               # Async update/upsert multiple\nPUT    /api/contracts/bulk/               # Async replace/upsert multiple\nDELETE /api/contracts/bulk/               # Async delete multiple\n\n# Operation monitoring\nGET    /api/operations/{task_id}/status/  # Check async status\n```\n\n## API Design\n\n### Enhanced Standard Endpoints (Sync Operations)\n\nSmart enhancements to standard ViewSet methods for immediate results with small datasets:\n\n#### Multi-Get\n```bash\n# Small dataset - immediate results\nGET /api/contracts/?ids=1,2,3,4,5\n\n# Response (immediate)\n{\n  \"count\": 5,\n  \"results\": [...],\n  \"is_sync\": true\n}\n```\n\n#### Sync Upsert\n```bash\n# Small dataset - immediate upsert\nPOST /api/contracts/?unique_fields=contract_number,year\nContent-Type: application/json\n[\n  {\"contract_number\": \"C001\", \"year\": 2024, \"amount\": 1000},\n  {\"contract_number\": \"C002\", \"year\": 2024, \"amount\": 2000}\n]\n\n# Response (immediate)\n{\n  \"message\": \"Upsert completed successfully\",\n  \"total_items\": 2,\n  \"created_count\": 1,\n  \"updated_count\": 1,\n  \"created_ids\": [123],\n  \"updated_ids\": [124],\n  \"is_sync\": true\n}\n```\n\n### Bulk Endpoints (Async Operations)\n\nDedicated endpoints for large dataset processing via background tasks:\n\n#### Async Multi-Get\n```bash\n# Large dataset - background processing\nGET /api/contracts/bulk/?ids=1,2,3,...,1000\n\n# Response (task started)\n{\n  \"message\": \"Bulk get task started for 1000 items\",\n  \"task_id\": \"abc123-def456\",\n  \"status_url\": \"/api/operations/abc123-def456/status/\",\n  \"is_async\": true\n}\n```\n\n#### Async Upsert\n```bash\n# Large dataset - background processing\nPATCH /api/contracts/bulk/?unique_fields=contract_number,year\nContent-Type: application/json\n[\n  {\"contract_number\": \"C001\", \"year\": 2024, \"amount\": 1000},\n  ... // 500+ items\n]\n\n# Response (task started)\n{\n  \"message\": \"Bulk upsert task started for 500 items\",\n  \"task_id\": \"def456-ghi789\",\n  \"unique_fields\": [\"contract_number\", \"year\"],\n  \"status_url\": \"/api/operations/def456-ghi789/status/\"\n}\n```\n\n## Operation Types\n\n### Sync Operations (Standard Endpoints)\n- **Best for**: \u226450-100 items, immediate results needed\n- **Use cases**: Real-time user interactions, form submissions, small API integrations\n- **Response**: Immediate data or results\n- **Endpoints**: Enhanced standard ViewSet methods\n\n### Async Operations (Bulk Endpoints)  \n- **Best for**: >100 items, can wait for results\n- **Use cases**: Data imports, batch processing, CSV uploads\n- **Response**: Task ID for monitoring\n- **Endpoints**: Dedicated `/bulk/` endpoints\n\n## Configuration\n\n### Custom Settings\n\n```python\n# Operation Settings\nDRF_EXT_CHUNK_SIZE = 100                    # Items per processing chunk\nDRF_EXT_MAX_RECORDS = 10000                 # Maximum records per operation\nDRF_EXT_BATCH_SIZE = 1000                   # Database batch size\nDRF_EXT_CACHE_TIMEOUT = 86400               # Cache timeout (24 hours)\nDRF_EXT_PROGRESS_UPDATE_INTERVAL = 10       # Progress update frequency\n\n# Sync Operation Limits\nDRF_EXT_SYNC_MAX_ITEMS = 50                 # Max items for sync operations\nDRF_EXT_SYNC_MULTI_GET_MAX = 100            # Max items for sync multi-get\n```\n\n## Example Usage\n\n### Basic Contract Management\n\n```python\n# Small dataset - sync operations (immediate results)\ncurl -X POST \"/api/contracts/?unique_fields=contract_number\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '[\n    {\"contract_number\": \"C001\", \"amount\": 1000},\n    {\"contract_number\": \"C002\", \"amount\": 2000}\n  ]'\n\n# Large dataset - async operations (background processing)\ncurl -X POST \"/api/contracts/bulk/\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '[...500 contracts...]'\n\n# Check async status\ncurl \"/api/operations/{task_id}/status/\"\n```\n\n### Migration from Previous Versions\n\nIf you're coming from older versions:\n\n```python\n# Old (separate mixins)\nclass ContractViewSet(SyncUpsertMixin, AsyncOperationsMixin, viewsets.ModelViewSet):\n    queryset = Contract.objects.all()\n    serializer_class = ContractSerializer\n\n# New (unified mixin)\nclass ContractViewSet(OperationsMixin, viewsets.ModelViewSet):\n    queryset = Contract.objects.all()\n    serializer_class = ContractSerializer\n```\n\n### Endpoint Changes\n\n| Operation | Old Endpoints | New Endpoints |\n|-----------|---------------|---------------|\n| Sync Upsert | `POST /upsert/` | `POST /?unique_fields=...` |\n| Async Create | `POST /operations/` | `POST /bulk/` |\n| Async Update | `PATCH /operations/` | `PATCH /bulk/` |\n| Async Delete | `DELETE /operations/` | `DELETE /bulk/` |\n\n## Error Handling\n\nThe system provides comprehensive error handling:\n\n- **Validation Errors**: Field-level validation using DRF serializers\n- **Size Limits**: Automatic routing suggestion for oversized sync requests\n- **Database Errors**: Transaction rollback on failures\n- **Task Failures**: Detailed error reporting in async task status\n\n## Performance Considerations\n\n- **Database Efficiency**: Uses optimized database operations for all bulk processing\n- **Memory Management**: Processes large datasets in configurable chunks\n- **Intelligent Routing**: Automatic sync/async decision based on dataset size\n- **Progress Tracking**: Redis-based monitoring without database overhead\n- **Result Caching**: Efficient caching of async operation results\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License.\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Django REST Framework extensions for efficient async and synchronous operations with Celery and Redis",
    "version": "0.0.9",
    "project_urls": {
        "Homepage": "https://github.com/AugendLimited/django-drf-extensions",
        "Repository": "https://github.com/AugendLimited/django-drf-extensions"
    },
    "split_keywords": [
        "django",
        " drf",
        " rest-framework",
        " celery",
        " redis",
        " async",
        " upsert",
        " sync",
        " operations",
        " extensions"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "03e2570b86a9686c337db7951c0188d81f23fdd6880fd437ebe9e92be9f50231",
                "md5": "1cca78a56817ea338e4fd4473be23a1e",
                "sha256": "b3b5b6e8d72f72655e62441f5685dd7899b55b35be122f17d31db1eefe4ba3cd"
            },
            "downloads": -1,
            "filename": "django_drf_extensions-0.0.9-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "1cca78a56817ea338e4fd4473be23a1e",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": "<4.0,>=3.11",
            "size": 23014,
            "upload_time": "2025-07-18T10:59:29",
            "upload_time_iso_8601": "2025-07-18T10:59:29.607227Z",
            "url": "https://files.pythonhosted.org/packages/03/e2/570b86a9686c337db7951c0188d81f23fdd6880fd437ebe9e92be9f50231/django_drf_extensions-0.0.9-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "59c7a35791953be3e2b173914cddd46721ce91f02e9be7c9003af7d2bb4940f9",
                "md5": "8feaf5693b3513d96b390658be899c3b",
                "sha256": "e7fc6271862e3468d7ac501d7d5e248e6204b9f8288f6de7475ff5d94e1230ca"
            },
            "downloads": -1,
            "filename": "django_drf_extensions-0.0.9.tar.gz",
            "has_sig": false,
            "md5_digest": "8feaf5693b3513d96b390658be899c3b",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": "<4.0,>=3.11",
            "size": 20658,
            "upload_time": "2025-07-18T10:59:31",
            "upload_time_iso_8601": "2025-07-18T10:59:31.022503Z",
            "url": "https://files.pythonhosted.org/packages/59/c7/a35791953be3e2b173914cddd46721ce91f02e9be7c9003af7d2bb4940f9/django_drf_extensions-0.0.9.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-07-18 10:59:31",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "AugendLimited",
    "github_project": "django-drf-extensions",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "lcname": "django-drf-extensions"
}
        
Elapsed time: 2.27924s