# Django Bulk DRF
Asynchronous bulk operations for Django REST Framework using Celery workers and Redis for progress tracking.
## Installation
```bash
pip install django-bulk-drf
```
### Requirements
- Python 3.11+
- Django 4.0+
- Django REST Framework 3.14+
- Celery 5.2+
- Redis 4.3+
- django-redis 5.2+
## Quick Setup
1. Add to your `INSTALLED_APPS`:
```python
INSTALLED_APPS = [
# ... your other apps
'rest_framework',
'django_bulk_drf',
]
```
2. Configure Redis cache:
```python
CACHES = {
'default': {
'BACKEND': 'django_redis.cache.RedisCache',
'LOCATION': 'redis://127.0.0.1:6379/1',
'OPTIONS': {
'CLIENT_CLASS': 'django_redis.client.DefaultClient',
}
}
}
```
3. Configure Celery:
```python
# settings.py
CELERY_BROKER_URL = 'redis://127.0.0.1:6379/0'
CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'
```
This implementation provides asynchronous bulk operations for your Django REST Framework API endpoints using Celery workers and Redis for progress tracking.
## Overview
The bulk operations system consists of:
1. **Bulk Processing Tasks** (`django_bulk_drf.bulk_processing`) - Celery tasks for handling bulk operations
2. **Bulk Mixins** (`django_bulk_drf.bulk_mixins`) - DRF ViewSet mixins to add bulk endpoints
3. **Redis Cache** (`django_bulk_drf.bulk_cache`) - Progress tracking and result caching
4. **Status Views** (`django_bulk_drf.bulk_views`) - API endpoints to check task status
## Features
- ✅ **Asynchronous Processing**: Long-running bulk operations don't block the API
- ✅ **Progress Tracking**: Real-time progress updates via Redis
- ✅ **Error Handling**: Detailed error reporting for failed items
- ✅ **Result Caching**: Final results cached in Redis for 24 hours
- ✅ **Validation**: Full DRF serializer validation for all items
- ✅ **Atomic Operations**: Database transactions ensure data consistency
- ✅ **Unified Endpoint**: Single `/bulk` endpoint supports both JSON and CSV via Content-Type detection
- ✅ **RESTful Design**: Uses HTTP methods (GET, POST, PATCH, PUT, DELETE) for different operations
## Content-Type Detection
The system automatically detects the input format based on the HTTP `Content-Type` header:
- **`Content-Type: application/json`** → JSON data processing
- **`Content-Type: multipart/form-data`** → CSV file upload processing
This means you can use the same `/bulk` endpoint for both JSON and CSV operations, making the API clean and RESTful.
## Available Operations
### JSON-based Operations
### 1. Bulk Retrieve
- **Endpoint**: `GET /api/{model}/bulk/?ids=1,2,3`
- **Method**: GET
- **Input**: Query parameters (`ids`) or request body (complex filters)
- **Output**: Serialized data (direct) or Task ID for large results
### 2. Bulk Create
- **Endpoint**: `POST /api/{model}/bulk/`
- **Method**: POST
- **Input**: Array of objects to create
- **Output**: Task ID and status URL
### 3. Bulk Update (Partial)
- **Endpoint**: `PATCH /api/{model}/bulk/`
- **Method**: PATCH
- **Input**: Array of objects with `id` and partial update data
- **Output**: Task ID and status URL
### 4. Bulk Replace (Full Update)
- **Endpoint**: `PUT /api/{model}/bulk/`
- **Method**: PUT
- **Input**: Array of complete objects with `id` and all required fields
- **Output**: Task ID and status URL
### 5. Bulk Delete
- **Endpoint**: `DELETE /api/{model}/bulk/`
- **Method**: DELETE
- **Input**: Array of IDs to delete
- **Output**: Task ID and status URL
### 6. Bulk Upsert (Insert or Update)
- **Endpoint**: `PATCH /api/{model}/bulk/` or `PUT /api/{model}/bulk/`
- **Method**: PATCH or PUT
- **Input**: Object with `data` array, `unique_fields`, and optional `update_fields`
- **Output**: Task ID and status URL
- **Description**: Similar to Django's `bulk_create` with `update_conflicts=True`. Integrated into existing PATCH/PUT endpoints.
### CSV-based Operations (Salesforce-style)
All CSV operations use the same `/bulk` endpoint with `Content-Type: multipart/form-data`:
### 7. CSV Bulk Create
- **Endpoint**: `POST /api/{model}/bulk/`
- **Method**: POST
- **Content-Type**: `multipart/form-data`
- **Input**: CSV file upload with headers matching model fields
- **Output**: Task ID and status URL
### 8. CSV Bulk Update (Partial)
- **Endpoint**: `PATCH /api/{model}/bulk/`
- **Method**: PATCH
- **Content-Type**: `multipart/form-data`
- **Input**: CSV file with `id` column and fields to update
- **Output**: Task ID and status URL
### 9. CSV Bulk Replace (Full Update)
- **Endpoint**: `PUT /api/{model}/bulk/`
- **Method**: PUT
- **Content-Type**: `multipart/form-data`
- **Input**: CSV file with `id` column and all required fields
- **Output**: Task ID and status URL
### 10. CSV Bulk Delete
- **Endpoint**: `DELETE /api/{model}/bulk/`
- **Method**: DELETE
- **Content-Type**: `multipart/form-data`
- **Input**: CSV file with `id` column containing IDs to delete
- **Output**: Task ID and status URL
### 11. CSV Bulk Upsert
- **Endpoint**: `PATCH /api/{model}/bulk/` or `PUT /api/{model}/bulk/`
- **Method**: PATCH or PUT
- **Content-Type**: `multipart/form-data`
- **Input**: CSV file with headers matching model fields + form fields for `unique_fields` and optional `update_fields`
- **Output**: Task ID and status URL
- **Description**: Similar to Django's `bulk_create` with `update_conflicts=True`. Integrated into existing PATCH/PUT endpoints.
### 12. Status Tracking
- **Endpoint**: `GET /api/bulk-operations/{task_id}/status/`
- **Output**: Task status, progress, and results
## HTTP Method Differences
- **GET**: Retrieve multiple records by IDs or complex queries
- **POST**: Creates new records (all fields required based on your model)
- **PATCH**: Partial updates - only include fields you want to change (requires `id`)
- **PUT**: Full replacement - all required fields must be provided (requires `id`)
- **DELETE**: Removes records (provide array of IDs)
- **PATCH/PUT /bulk/**: Partial/full updates or upsert records based on unique constraints (when `data`, `unique_fields` provided)
## Usage
### Adding Bulk Operations to a ViewSet
```python
from django_bulk_drf.bulk_mixins import BulkOperationsMixin
class FinancialTransactionViewSet(BulkOperationsMixin, viewsets.ModelViewSet):
queryset = FinancialTransaction.objects.all()
serializer_class = FinancialTransactionSerializer
```
### OpenAPI/Swagger Documentation
The bulk operations now include comprehensive OpenAPI schema definitions for Swagger documentation. To enable this:
1. **Install drf-spectacular** (optional dependency):
```bash
pip install drf-spectacular
```
2. **Add to INSTALLED_APPS**:
```python
INSTALLED_APPS = [
# ... your other apps
'drf_spectacular',
]
```
3. **Configure DRF settings**:
```python
REST_FRAMEWORK = {
'DEFAULT_SCHEMA_CLASS': 'drf_spectacular.openapi.AutoSchema',
}
SPECTACULAR_SETTINGS = {
'TITLE': 'Your API',
'DESCRIPTION': 'Your API description',
'VERSION': '1.0.0',
}
```
4. **Add URL patterns**:
```python
from drf_spectacular.views import SpectacularAPIView, SpectacularSwaggerView
urlpatterns = [
# ... your other URLs
path('api/schema/', SpectacularAPIView.as_view(), name='schema'),
path('api/docs/', SpectacularSwaggerView.as_view(url_name='schema'), name='swagger-ui'),
]
```
The bulk endpoints will now show proper OpenAPI documentation with:
- ✅ **Query parameters** for bulk GET operations (e.g., `?ids=1,2,3`)
- ✅ **Request body schemas** for complex queries and bulk operations
- ✅ **Array payloads** correctly specified for all bulk operations
- ✅ **Request/response examples** for each operation type
- ✅ **Proper schema references** for your model fields
- ✅ **CSV upload support** documented for file operations
- ✅ **Multiple content types** (JSON and CSV) properly documented
**Note**: If `drf-spectacular` is not installed, the mixins will work normally but without enhanced OpenAPI documentation.
### Example API Calls
#### Bulk Retrieve (Simple ID-based)
```bash
# Small result set - returns data directly
curl "http://localhost:8000/api/financial-transactions/bulk/?ids=1,2,3,4,5"
```
#### Bulk Retrieve (Large ID-based - Async)
```bash
# Large result set - returns task ID
curl "http://localhost:8000/api/financial-transactions/bulk/?ids=1,2,3,4,5,6,7,8,...,150"
```
#### Bulk Retrieve (Complex Query)
```bash
# Complex filtering via request body
curl -X GET http://localhost:8000/api/financial-transactions/bulk/ \\
-H "Content-Type: application/json" \\
-d '{
"filters": {
"amount": {"gte": 100, "lte": 1000},
"datetime": {"gte": "2025-01-01"},
"financial_account": 1
}
}'
```
#### Bulk Create
```bash
curl -X POST http://localhost:8000/api/financial-transactions/bulk/ \\
-H "Content-Type: application/json" \\
-d '[
{
"amount": "100.50",
"description": "Transaction 1",
"datetime": "2025-01-01T10:00:00Z",
"financial_account": 1,
"classification_status": 1
},
{
"amount": "-25.75",
"description": "Transaction 2",
"datetime": "2025-01-01T11:00:00Z",
"financial_account": 1,
"classification_status": 1
}
]'
```
**Response:**
```json
{
"message": "Bulk create task started for 2 items",
"task_id": "abc123-def456-ghi789",
"total_items": 2,
"status_url": "/api/bulk-operations/abc123-def456-ghi789/status/"
}
```
#### Bulk Update (Partial)
```bash
curl -X PATCH http://localhost:8000/api/financial-transactions/bulk/ \\
-H "Content-Type: application/json" \\
-d '[
{
"id": 1,
"amount": "150.00",
"description": "Updated transaction 1"
},
{
"id": 2,
"description": "Updated transaction 2"
}
]'
```
#### Bulk Replace (Full Update)
```bash
curl -X PUT http://localhost:8000/api/financial-transactions/bulk/ \\
-H "Content-Type: application/json" \\
-d '[
{
"id": 1,
"amount": "200.00",
"description": "Completely replaced transaction 1",
"datetime": "2025-01-01T15:00:00Z",
"financial_account": 1,
"classification_status": 2
},
{
"id": 2,
"amount": "75.50",
"description": "Completely replaced transaction 2",
"datetime": "2025-01-01T16:00:00Z",
"financial_account": 1,
"classification_status": 1
}
]'
```
#### Bulk Delete
```bash
curl -X DELETE http://localhost:8000/api/financial-transactions/bulk/ \\
-H "Content-Type: application/json" \\
-d '[1, 2, 3, 4, 5]'
```
#### Bulk Upsert (Insert or Update) - Salesforce Style
```bash
# Salesforce-style: unique_fields in query params, simple array payload
curl -X PATCH "http://localhost:8000/api/financial-transactions/bulk/?unique_fields=financial_account,datetime" \
-H "Content-Type: application/json" \
-d '[
{
"amount": "100.50",
"description": "Upsert transaction 1",
"datetime": "2025-01-01T10:00:00Z",
"financial_account": 1,
"classification_status": 1
},
{
"amount": "200.75",
"description": "Upsert transaction 1 (updated)",
"datetime": "2025-01-01T10:00:00Z",
"financial_account": 1,
"classification_status": 2
}
]'
```
#### Single Object Upsert (Salesforce Style)
```bash
# Single object upsert (not array)
curl -X PATCH "http://localhost:8000/api/financial-transactions/bulk/?unique_fields=financial_account,datetime" \
-H "Content-Type: application/json" \
-d '{
"amount": "100.50",
"description": "Single upsert transaction",
"datetime": "2025-01-01T10:00:00Z",
"financial_account": 1,
"classification_status": 1
}'
```
#### Bulk Upsert (Legacy Style)
```bash
# Legacy style: structured body (backward compatibility)
curl -X PATCH http://localhost:8000/api/financial-transactions/bulk/ \
-H "Content-Type: application/json" \
-d '{
"data": [
{
"amount": "100.50",
"description": "Upsert transaction 1",
"datetime": "2025-01-01T10:00:00Z",
"financial_account": 1,
"classification_status": 1
}
],
"unique_fields": ["financial_account", "datetime"]
}'
```
**Response:**
```json
{
"message": "Bulk upsert task started for 2 items",
"task_id": "abc123-def456-ghi789",
"total_items": 2,
"status_url": "/api/bulk-operations/abc123-def456-ghi789/status/"
}
```
#### Check Status
```bash
curl http://localhost:8000/api/bulk-operations/abc123-def456-ghi789/status/
```
**Response:**
```json
{
"task_id": "abc123-def456-ghi789",
"state": "SUCCESS",
"result": {
"task_id": "abc123-def456-ghi789",
"total_items": 2,
"operation_type": "bulk_create",
"success_count": 2,
"error_count": 0,
"errors": [],
"created_ids": [10, 11],
"updated_ids": [],
"deleted_ids": []
},
"progress": {
"current": 2,
"total": 2,
"percentage": 100.0,
"message": "Creating instances in database..."
},
"status": "Task completed successfully"
}
```
### CSV Upload Examples
#### CSV Bulk Create
```bash
# Upload CSV file for bulk creation
curl -X POST http://localhost:8000/api/financial-transactions/bulk/ \\
-H "Authorization: Bearer your-token" \\
-F "file=@transactions.csv"
```
**Sample CSV (transactions.csv):**
```csv
amount,description,datetime,financial_account,classification_status
100.50,"Transaction 1","2025-01-01T10:00:00Z",1,1
-25.75,"Transaction 2","2025-01-01T11:00:00Z",1,1
500.00,"Transaction 3","2025-01-01T12:00:00Z",2,2
```
#### CSV Bulk Update
```bash
# Upload CSV file for bulk updates
curl -X PATCH http://localhost:8000/api/financial-transactions/bulk/ \\
-H "Authorization: Bearer your-token" \\
-F "file=@updates.csv"
```
**Sample CSV (updates.csv):**
```csv
id,amount,description
1,150.00,"Updated Transaction 1"
2,,"Updated Transaction 2"
3,75.50,
```
#### CSV Bulk Delete
```bash
# Upload CSV file for bulk deletion
curl -X DELETE http://localhost:8000/api/financial-transactions/bulk/ \\
-H "Authorization: Bearer your-token" \\
-F "file=@delete_ids.csv"
```
**Sample CSV (delete_ids.csv):**
```csv
id
1
2
3
4
5
```
### CSV Format Requirements
1. **File encoding**: UTF-8 (supports BOM)
2. **File extension**: Must be `.csv`
3. **Headers**: First row must contain field names
4. **File size limit**: 10MB (configurable via `csv_max_file_size` attribute)
5. **Required fields**:
- **Create**: All required model fields
- **Update/Replace**: `id` column + fields to update
- **Delete**: `id` column only
### CSV Benefits
- ✅ **Easy to create**: Use Excel, Google Sheets, or any CSV editor
- ✅ **Memory efficient**: Streamed processing for large files
- ✅ **Human readable**: Easy to review and edit data
- ✅ **Widely supported**: Standard format across platforms
- ✅ **Compact**: Smaller than equivalent JSON for large datasets
## Bulk GET Response Formats
### Small Result Sets (< 100 records)
Returns data immediately:
```json
{
"count": 5,
"results": [
{"id": 1, "amount": "100.50", "description": "Transaction 1"},
{"id": 2, "amount": "75.00", "description": "Transaction 2"}
],
"is_async": false
}
```
### Large Result Sets (≥ 100 records)
Returns task ID for async processing:
```json
{
"message": "Bulk get task started for 250 IDs",
"task_id": "abc123-def456-ghi789",
"total_items": 250,
"status_url": "/api/bulk-operations/abc123-def456-ghi789/status/",
"is_async": true
}
```
### Complex Query Filters
You can use Django ORM-style filters in the request body:
```json
{
"filters": {
"amount": {"gte": 100, "lte": 1000}, // amount >= 100 AND amount <= 1000
"datetime": {"gte": "2025-01-01"}, // datetime >= 2025-01-01
"financial_account": 1, // financial_account = 1
"description": {"icontains": "payment"} // description contains "payment" (case-insensitive)
}
}
```
Supported lookup types: `exact`, `gte`, `lte`, `gt`, `lt`, `in`, `icontains`, `startswith`, `endswith`, etc.
## Task States
- **PENDING**: Task is waiting to be executed
- **PROGRESS**: Task is currently running (includes progress data)
- **SUCCESS**: Task completed successfully
- **FAILURE**: Task failed with an error
## Progress Tracking
Progress is tracked in Redis and updated every 10 items processed. The progress object includes:
```json
{
"current": 50,
"total": 100,
"percentage": 50.0,
"message": "Validated 50/100 items"
}
```
## Error Handling
Individual item errors are captured and included in the result:
```json
{
"errors": [
{
"index": 5,
"error": "amount: This field is required.",
"data": {"description": "Missing amount"}
}
]
}
```
## Configuration
### Redis Settings
Make sure your Django settings include Redis configuration:
```python
# Redis cache for bulk operations
CACHES = {
'default': {
'BACKEND': 'django_redis.cache.RedisCache',
'LOCATION': REDIS_URL,
'OPTIONS': {
'CLIENT_CLASS': 'django_redis.client.DefaultClient',
}
}
}
```
### Celery Settings
Your Celery configuration should include:
```python
# Celery settings for bulk operations
CELERY_TASK_TIME_LIMIT = 5 * 60 # 5 minutes
CELERY_TASK_SOFT_TIME_LIMIT = 60 # 1 minute
CELERY_WORKER_SEND_TASK_EVENTS = True
CELERY_TASK_SEND_SENT_EVENT = True
```
## Starting Workers
To process bulk operations, start Celery workers:
```bash
# Start Celery worker
celery -A config.celery_app worker -l info
# Start Celery beat (for periodic tasks)
celery -A config.celery_app beat -l info
# Start Flower (monitoring - optional)
celery -A config.celery_app flower
```
## Performance Considerations
1. **Batch Size**: Large arrays are processed in chunks to avoid memory issues
2. **Database Connections**: Use connection pooling for high-volume operations
3. **Redis Memory**: Monitor Redis memory usage for large result sets
4. **Worker Scaling**: Scale Celery workers based on load
## Monitoring
- Use Flower for Celery task monitoring: `http://localhost:5555`
- Monitor Redis usage with `redis-cli info memory`
- Check Django logs for task execution details
- Use the status endpoint for real-time progress tracking
## Security Considerations
1. **Authentication**: Ensure bulk endpoints require proper authentication
2. **Rate Limiting**: Implement rate limiting for bulk operations
3. **Input Validation**: All input is validated through DRF serializers
4. **Permission Checks**: Add custom permission classes as needed
## Extending the System
### Custom Bulk Operations
You can create custom bulk operations by:
1. Creating new Celery tasks in `bulk_processing.py`
2. Adding new action methods to the mixins
3. Updating the status view if needed
### Custom Progress Tracking
Override the progress tracking by extending `BulkOperationCache`:
```python
from django_bulk_drf.bulk_cache import BulkOperationCache
class CustomBulkCache(BulkOperationCache):
@classmethod
def set_custom_metric(cls, task_id: str, metric_data: dict):
# Custom metric tracking
pass
```
This bulk operations system provides a robust, scalable solution for handling large data operations asynchronously while keeping your API responsive and providing real-time feedback to users.
Raw data
{
"_id": null,
"home_page": "https://github.com/AugendLimited/django-bulk-drf",
"name": "django-bulk-drf",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.11",
"maintainer_email": null,
"keywords": "django, bulk, drf, rest-framework, celery, redis, async",
"author": "Konrad Beck",
"author_email": "konrad.beck@merchantcapital.co.za",
"download_url": "https://files.pythonhosted.org/packages/6d/a6/16f41611bb8e47270f502066ecdab1e463a8f7744c7fbef3ed9d715e039e/django_bulk_drf-0.1.16.tar.gz",
"platform": null,
"description": "# Django Bulk DRF\n\nAsynchronous bulk operations for Django REST Framework using Celery workers and Redis for progress tracking.\n\n## Installation\n\n```bash\npip install django-bulk-drf\n```\n\n### Requirements\n\n- Python 3.11+\n- Django 4.0+\n- Django REST Framework 3.14+\n- Celery 5.2+\n- Redis 4.3+\n- django-redis 5.2+\n\n## Quick Setup\n\n1. Add to your `INSTALLED_APPS`:\n```python\nINSTALLED_APPS = [\n # ... your other apps\n 'rest_framework',\n 'django_bulk_drf',\n]\n```\n\n2. Configure Redis cache:\n```python\nCACHES = {\n 'default': {\n 'BACKEND': 'django_redis.cache.RedisCache',\n 'LOCATION': 'redis://127.0.0.1:6379/1',\n 'OPTIONS': {\n 'CLIENT_CLASS': 'django_redis.client.DefaultClient',\n }\n }\n}\n```\n\n3. Configure Celery:\n```python\n# settings.py\nCELERY_BROKER_URL = 'redis://127.0.0.1:6379/0'\nCELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'\n```\n\nThis implementation provides asynchronous bulk operations for your Django REST Framework API endpoints using Celery workers and Redis for progress tracking.\n\n## Overview\n\nThe bulk operations system consists of:\n\n1. **Bulk Processing Tasks** (`django_bulk_drf.bulk_processing`) - Celery tasks for handling bulk operations\n2. **Bulk Mixins** (`django_bulk_drf.bulk_mixins`) - DRF ViewSet mixins to add bulk endpoints\n3. **Redis Cache** (`django_bulk_drf.bulk_cache`) - Progress tracking and result caching\n4. **Status Views** (`django_bulk_drf.bulk_views`) - API endpoints to check task status\n\n## Features\n\n- \u2705 **Asynchronous Processing**: Long-running bulk operations don't block the API\n- \u2705 **Progress Tracking**: Real-time progress updates via Redis\n- \u2705 **Error Handling**: Detailed error reporting for failed items\n- \u2705 **Result Caching**: Final results cached in Redis for 24 hours\n- \u2705 **Validation**: Full DRF serializer validation for all items\n- \u2705 **Atomic Operations**: Database transactions ensure data consistency\n- \u2705 **Unified Endpoint**: Single `/bulk` endpoint supports both JSON and CSV via Content-Type detection\n- \u2705 **RESTful Design**: Uses HTTP methods (GET, POST, PATCH, PUT, DELETE) for different operations\n\n## Content-Type Detection\n\nThe system automatically detects the input format based on the HTTP `Content-Type` header:\n\n- **`Content-Type: application/json`** \u2192 JSON data processing\n- **`Content-Type: multipart/form-data`** \u2192 CSV file upload processing\n\nThis means you can use the same `/bulk` endpoint for both JSON and CSV operations, making the API clean and RESTful.\n\n## Available Operations\n\n### JSON-based Operations\n\n### 1. Bulk Retrieve\n- **Endpoint**: `GET /api/{model}/bulk/?ids=1,2,3`\n- **Method**: GET\n- **Input**: Query parameters (`ids`) or request body (complex filters)\n- **Output**: Serialized data (direct) or Task ID for large results\n\n### 2. Bulk Create\n- **Endpoint**: `POST /api/{model}/bulk/`\n- **Method**: POST\n- **Input**: Array of objects to create\n- **Output**: Task ID and status URL\n\n### 3. Bulk Update (Partial)\n- **Endpoint**: `PATCH /api/{model}/bulk/`\n- **Method**: PATCH\n- **Input**: Array of objects with `id` and partial update data\n- **Output**: Task ID and status URL\n\n### 4. Bulk Replace (Full Update)\n- **Endpoint**: `PUT /api/{model}/bulk/`\n- **Method**: PUT\n- **Input**: Array of complete objects with `id` and all required fields\n- **Output**: Task ID and status URL\n\n### 5. Bulk Delete\n- **Endpoint**: `DELETE /api/{model}/bulk/`\n- **Method**: DELETE\n- **Input**: Array of IDs to delete\n- **Output**: Task ID and status URL\n\n### 6. Bulk Upsert (Insert or Update)\n- **Endpoint**: `PATCH /api/{model}/bulk/` or `PUT /api/{model}/bulk/`\n- **Method**: PATCH or PUT\n- **Input**: Object with `data` array, `unique_fields`, and optional `update_fields`\n- **Output**: Task ID and status URL\n- **Description**: Similar to Django's `bulk_create` with `update_conflicts=True`. Integrated into existing PATCH/PUT endpoints.\n\n### CSV-based Operations (Salesforce-style)\n\nAll CSV operations use the same `/bulk` endpoint with `Content-Type: multipart/form-data`:\n\n### 7. CSV Bulk Create\n- **Endpoint**: `POST /api/{model}/bulk/`\n- **Method**: POST\n- **Content-Type**: `multipart/form-data`\n- **Input**: CSV file upload with headers matching model fields\n- **Output**: Task ID and status URL\n\n### 8. CSV Bulk Update (Partial)\n- **Endpoint**: `PATCH /api/{model}/bulk/`\n- **Method**: PATCH\n- **Content-Type**: `multipart/form-data`\n- **Input**: CSV file with `id` column and fields to update\n- **Output**: Task ID and status URL\n\n### 9. CSV Bulk Replace (Full Update)\n- **Endpoint**: `PUT /api/{model}/bulk/`\n- **Method**: PUT\n- **Content-Type**: `multipart/form-data`\n- **Input**: CSV file with `id` column and all required fields\n- **Output**: Task ID and status URL\n\n### 10. CSV Bulk Delete\n- **Endpoint**: `DELETE /api/{model}/bulk/`\n- **Method**: DELETE\n- **Content-Type**: `multipart/form-data`\n- **Input**: CSV file with `id` column containing IDs to delete\n- **Output**: Task ID and status URL\n\n### 11. CSV Bulk Upsert\n- **Endpoint**: `PATCH /api/{model}/bulk/` or `PUT /api/{model}/bulk/`\n- **Method**: PATCH or PUT\n- **Content-Type**: `multipart/form-data`\n- **Input**: CSV file with headers matching model fields + form fields for `unique_fields` and optional `update_fields`\n- **Output**: Task ID and status URL\n- **Description**: Similar to Django's `bulk_create` with `update_conflicts=True`. Integrated into existing PATCH/PUT endpoints.\n\n### 12. Status Tracking\n- **Endpoint**: `GET /api/bulk-operations/{task_id}/status/`\n- **Output**: Task status, progress, and results\n\n## HTTP Method Differences\n\n- **GET**: Retrieve multiple records by IDs or complex queries\n- **POST**: Creates new records (all fields required based on your model)\n- **PATCH**: Partial updates - only include fields you want to change (requires `id`)\n- **PUT**: Full replacement - all required fields must be provided (requires `id`) \n- **DELETE**: Removes records (provide array of IDs)\n- **PATCH/PUT /bulk/**: Partial/full updates or upsert records based on unique constraints (when `data`, `unique_fields` provided)\n\n## Usage\n\n### Adding Bulk Operations to a ViewSet\n\n```python\nfrom django_bulk_drf.bulk_mixins import BulkOperationsMixin\n\nclass FinancialTransactionViewSet(BulkOperationsMixin, viewsets.ModelViewSet):\n queryset = FinancialTransaction.objects.all()\n serializer_class = FinancialTransactionSerializer\n```\n\n### OpenAPI/Swagger Documentation\n\nThe bulk operations now include comprehensive OpenAPI schema definitions for Swagger documentation. To enable this:\n\n1. **Install drf-spectacular** (optional dependency):\n```bash\npip install drf-spectacular\n```\n\n2. **Add to INSTALLED_APPS**:\n```python\nINSTALLED_APPS = [\n # ... your other apps\n 'drf_spectacular',\n]\n```\n\n3. **Configure DRF settings**:\n```python\nREST_FRAMEWORK = {\n 'DEFAULT_SCHEMA_CLASS': 'drf_spectacular.openapi.AutoSchema',\n}\n\nSPECTACULAR_SETTINGS = {\n 'TITLE': 'Your API',\n 'DESCRIPTION': 'Your API description',\n 'VERSION': '1.0.0',\n}\n```\n\n4. **Add URL patterns**:\n```python\nfrom drf_spectacular.views import SpectacularAPIView, SpectacularSwaggerView\n\nurlpatterns = [\n # ... your other URLs\n path('api/schema/', SpectacularAPIView.as_view(), name='schema'),\n path('api/docs/', SpectacularSwaggerView.as_view(url_name='schema'), name='swagger-ui'),\n]\n```\n\nThe bulk endpoints will now show proper OpenAPI documentation with:\n- \u2705 **Query parameters** for bulk GET operations (e.g., `?ids=1,2,3`)\n- \u2705 **Request body schemas** for complex queries and bulk operations\n- \u2705 **Array payloads** correctly specified for all bulk operations\n- \u2705 **Request/response examples** for each operation type\n- \u2705 **Proper schema references** for your model fields\n- \u2705 **CSV upload support** documented for file operations\n- \u2705 **Multiple content types** (JSON and CSV) properly documented\n\n**Note**: If `drf-spectacular` is not installed, the mixins will work normally but without enhanced OpenAPI documentation.\n\n### Example API Calls\n\n#### Bulk Retrieve (Simple ID-based)\n```bash\n# Small result set - returns data directly\ncurl \"http://localhost:8000/api/financial-transactions/bulk/?ids=1,2,3,4,5\"\n```\n\n#### Bulk Retrieve (Large ID-based - Async)\n```bash\n# Large result set - returns task ID\ncurl \"http://localhost:8000/api/financial-transactions/bulk/?ids=1,2,3,4,5,6,7,8,...,150\"\n```\n\n#### Bulk Retrieve (Complex Query)\n```bash\n# Complex filtering via request body\ncurl -X GET http://localhost:8000/api/financial-transactions/bulk/ \\\\\n -H \"Content-Type: application/json\" \\\\\n -d '{\n \"filters\": {\n \"amount\": {\"gte\": 100, \"lte\": 1000},\n \"datetime\": {\"gte\": \"2025-01-01\"},\n \"financial_account\": 1\n }\n }'\n```\n\n#### Bulk Create\n```bash\ncurl -X POST http://localhost:8000/api/financial-transactions/bulk/ \\\\\n -H \"Content-Type: application/json\" \\\\\n -d '[\n {\n \"amount\": \"100.50\",\n \"description\": \"Transaction 1\",\n \"datetime\": \"2025-01-01T10:00:00Z\",\n \"financial_account\": 1,\n \"classification_status\": 1\n },\n {\n \"amount\": \"-25.75\", \n \"description\": \"Transaction 2\",\n \"datetime\": \"2025-01-01T11:00:00Z\",\n \"financial_account\": 1,\n \"classification_status\": 1\n }\n ]'\n```\n\n**Response:**\n```json\n{\n \"message\": \"Bulk create task started for 2 items\",\n \"task_id\": \"abc123-def456-ghi789\",\n \"total_items\": 2,\n \"status_url\": \"/api/bulk-operations/abc123-def456-ghi789/status/\"\n}\n```\n\n#### Bulk Update (Partial)\n```bash\ncurl -X PATCH http://localhost:8000/api/financial-transactions/bulk/ \\\\\n -H \"Content-Type: application/json\" \\\\\n -d '[\n {\n \"id\": 1,\n \"amount\": \"150.00\",\n \"description\": \"Updated transaction 1\"\n },\n {\n \"id\": 2,\n \"description\": \"Updated transaction 2\"\n }\n ]'\n```\n\n#### Bulk Replace (Full Update)\n```bash\ncurl -X PUT http://localhost:8000/api/financial-transactions/bulk/ \\\\\n -H \"Content-Type: application/json\" \\\\\n -d '[\n {\n \"id\": 1,\n \"amount\": \"200.00\",\n \"description\": \"Completely replaced transaction 1\",\n \"datetime\": \"2025-01-01T15:00:00Z\",\n \"financial_account\": 1,\n \"classification_status\": 2\n },\n {\n \"id\": 2,\n \"amount\": \"75.50\",\n \"description\": \"Completely replaced transaction 2\",\n \"datetime\": \"2025-01-01T16:00:00Z\",\n \"financial_account\": 1,\n \"classification_status\": 1\n }\n ]'\n```\n\n#### Bulk Delete\n```bash\ncurl -X DELETE http://localhost:8000/api/financial-transactions/bulk/ \\\\\n -H \"Content-Type: application/json\" \\\\\n -d '[1, 2, 3, 4, 5]'\n```\n\n#### Bulk Upsert (Insert or Update) - Salesforce Style\n```bash\n# Salesforce-style: unique_fields in query params, simple array payload\ncurl -X PATCH \"http://localhost:8000/api/financial-transactions/bulk/?unique_fields=financial_account,datetime\" \\\n -H \"Content-Type: application/json\" \\\n -d '[\n {\n \"amount\": \"100.50\",\n \"description\": \"Upsert transaction 1\",\n \"datetime\": \"2025-01-01T10:00:00Z\",\n \"financial_account\": 1,\n \"classification_status\": 1\n },\n {\n \"amount\": \"200.75\",\n \"description\": \"Upsert transaction 1 (updated)\",\n \"datetime\": \"2025-01-01T10:00:00Z\",\n \"financial_account\": 1,\n \"classification_status\": 2\n }\n ]'\n```\n\n#### Single Object Upsert (Salesforce Style)\n```bash\n# Single object upsert (not array)\ncurl -X PATCH \"http://localhost:8000/api/financial-transactions/bulk/?unique_fields=financial_account,datetime\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"amount\": \"100.50\",\n \"description\": \"Single upsert transaction\",\n \"datetime\": \"2025-01-01T10:00:00Z\",\n \"financial_account\": 1,\n \"classification_status\": 1\n }'\n```\n\n#### Bulk Upsert (Legacy Style)\n```bash\n# Legacy style: structured body (backward compatibility)\ncurl -X PATCH http://localhost:8000/api/financial-transactions/bulk/ \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"data\": [\n {\n \"amount\": \"100.50\",\n \"description\": \"Upsert transaction 1\",\n \"datetime\": \"2025-01-01T10:00:00Z\",\n \"financial_account\": 1,\n \"classification_status\": 1\n }\n ],\n \"unique_fields\": [\"financial_account\", \"datetime\"]\n }'\n```\n\n**Response:**\n```json\n{\n \"message\": \"Bulk upsert task started for 2 items\",\n \"task_id\": \"abc123-def456-ghi789\",\n \"total_items\": 2,\n \"status_url\": \"/api/bulk-operations/abc123-def456-ghi789/status/\"\n}\n```\n\n#### Check Status\n```bash\ncurl http://localhost:8000/api/bulk-operations/abc123-def456-ghi789/status/\n```\n\n**Response:**\n```json\n{\n \"task_id\": \"abc123-def456-ghi789\",\n \"state\": \"SUCCESS\",\n \"result\": {\n \"task_id\": \"abc123-def456-ghi789\",\n \"total_items\": 2,\n \"operation_type\": \"bulk_create\",\n \"success_count\": 2,\n \"error_count\": 0,\n \"errors\": [],\n \"created_ids\": [10, 11],\n \"updated_ids\": [],\n \"deleted_ids\": []\n },\n \"progress\": {\n \"current\": 2,\n \"total\": 2,\n \"percentage\": 100.0,\n \"message\": \"Creating instances in database...\"\n },\n \"status\": \"Task completed successfully\"\n}\n```\n\n### CSV Upload Examples\n\n#### CSV Bulk Create\n```bash\n# Upload CSV file for bulk creation\ncurl -X POST http://localhost:8000/api/financial-transactions/bulk/ \\\\\n -H \"Authorization: Bearer your-token\" \\\\\n -F \"file=@transactions.csv\"\n```\n\n**Sample CSV (transactions.csv):**\n```csv\namount,description,datetime,financial_account,classification_status\n100.50,\"Transaction 1\",\"2025-01-01T10:00:00Z\",1,1\n-25.75,\"Transaction 2\",\"2025-01-01T11:00:00Z\",1,1\n500.00,\"Transaction 3\",\"2025-01-01T12:00:00Z\",2,2\n```\n\n#### CSV Bulk Update\n```bash\n# Upload CSV file for bulk updates\ncurl -X PATCH http://localhost:8000/api/financial-transactions/bulk/ \\\\\n -H \"Authorization: Bearer your-token\" \\\\\n -F \"file=@updates.csv\"\n```\n\n**Sample CSV (updates.csv):**\n```csv\nid,amount,description\n1,150.00,\"Updated Transaction 1\"\n2,,\"Updated Transaction 2\"\n3,75.50,\n```\n\n#### CSV Bulk Delete\n```bash\n# Upload CSV file for bulk deletion\ncurl -X DELETE http://localhost:8000/api/financial-transactions/bulk/ \\\\\n -H \"Authorization: Bearer your-token\" \\\\\n -F \"file=@delete_ids.csv\"\n```\n\n**Sample CSV (delete_ids.csv):**\n```csv\nid\n1\n2\n3\n4\n5\n```\n\n### CSV Format Requirements\n\n1. **File encoding**: UTF-8 (supports BOM)\n2. **File extension**: Must be `.csv`\n3. **Headers**: First row must contain field names\n4. **File size limit**: 10MB (configurable via `csv_max_file_size` attribute)\n5. **Required fields**:\n - **Create**: All required model fields\n - **Update/Replace**: `id` column + fields to update\n - **Delete**: `id` column only\n\n### CSV Benefits\n\n- \u2705 **Easy to create**: Use Excel, Google Sheets, or any CSV editor\n- \u2705 **Memory efficient**: Streamed processing for large files\n- \u2705 **Human readable**: Easy to review and edit data\n- \u2705 **Widely supported**: Standard format across platforms\n- \u2705 **Compact**: Smaller than equivalent JSON for large datasets\n\n## Bulk GET Response Formats\n\n### Small Result Sets (< 100 records)\nReturns data immediately:\n```json\n{\n \"count\": 5,\n \"results\": [\n {\"id\": 1, \"amount\": \"100.50\", \"description\": \"Transaction 1\"},\n {\"id\": 2, \"amount\": \"75.00\", \"description\": \"Transaction 2\"}\n ],\n \"is_async\": false\n}\n```\n\n### Large Result Sets (\u2265 100 records)\nReturns task ID for async processing:\n```json\n{\n \"message\": \"Bulk get task started for 250 IDs\",\n \"task_id\": \"abc123-def456-ghi789\",\n \"total_items\": 250,\n \"status_url\": \"/api/bulk-operations/abc123-def456-ghi789/status/\",\n \"is_async\": true\n}\n```\n\n### Complex Query Filters\n\nYou can use Django ORM-style filters in the request body:\n\n```json\n{\n \"filters\": {\n \"amount\": {\"gte\": 100, \"lte\": 1000}, // amount >= 100 AND amount <= 1000\n \"datetime\": {\"gte\": \"2025-01-01\"}, // datetime >= 2025-01-01\n \"financial_account\": 1, // financial_account = 1\n \"description\": {\"icontains\": \"payment\"} // description contains \"payment\" (case-insensitive)\n }\n}\n```\n\nSupported lookup types: `exact`, `gte`, `lte`, `gt`, `lt`, `in`, `icontains`, `startswith`, `endswith`, etc.\n\n## Task States\n\n- **PENDING**: Task is waiting to be executed\n- **PROGRESS**: Task is currently running (includes progress data)\n- **SUCCESS**: Task completed successfully\n- **FAILURE**: Task failed with an error\n\n## Progress Tracking\n\nProgress is tracked in Redis and updated every 10 items processed. The progress object includes:\n\n```json\n{\n \"current\": 50,\n \"total\": 100,\n \"percentage\": 50.0,\n \"message\": \"Validated 50/100 items\"\n}\n```\n\n## Error Handling\n\nIndividual item errors are captured and included in the result:\n\n```json\n{\n \"errors\": [\n {\n \"index\": 5,\n \"error\": \"amount: This field is required.\",\n \"data\": {\"description\": \"Missing amount\"}\n }\n ]\n}\n```\n\n## Configuration\n\n### Redis Settings\nMake sure your Django settings include Redis configuration:\n\n```python\n# Redis cache for bulk operations\nCACHES = {\n 'default': {\n 'BACKEND': 'django_redis.cache.RedisCache',\n 'LOCATION': REDIS_URL,\n 'OPTIONS': {\n 'CLIENT_CLASS': 'django_redis.client.DefaultClient',\n }\n }\n}\n```\n\n### Celery Settings\nYour Celery configuration should include:\n\n```python\n# Celery settings for bulk operations\nCELERY_TASK_TIME_LIMIT = 5 * 60 # 5 minutes\nCELERY_TASK_SOFT_TIME_LIMIT = 60 # 1 minute\nCELERY_WORKER_SEND_TASK_EVENTS = True\nCELERY_TASK_SEND_SENT_EVENT = True\n```\n\n## Starting Workers\n\nTo process bulk operations, start Celery workers:\n\n```bash\n# Start Celery worker\ncelery -A config.celery_app worker -l info\n\n# Start Celery beat (for periodic tasks)\ncelery -A config.celery_app beat -l info\n\n# Start Flower (monitoring - optional)\ncelery -A config.celery_app flower\n```\n\n## Performance Considerations\n\n1. **Batch Size**: Large arrays are processed in chunks to avoid memory issues\n2. **Database Connections**: Use connection pooling for high-volume operations\n3. **Redis Memory**: Monitor Redis memory usage for large result sets\n4. **Worker Scaling**: Scale Celery workers based on load\n\n## Monitoring\n\n- Use Flower for Celery task monitoring: `http://localhost:5555`\n- Monitor Redis usage with `redis-cli info memory`\n- Check Django logs for task execution details\n- Use the status endpoint for real-time progress tracking\n\n## Security Considerations\n\n1. **Authentication**: Ensure bulk endpoints require proper authentication\n2. **Rate Limiting**: Implement rate limiting for bulk operations\n3. **Input Validation**: All input is validated through DRF serializers\n4. **Permission Checks**: Add custom permission classes as needed\n\n## Extending the System\n\n### Custom Bulk Operations\n\nYou can create custom bulk operations by:\n\n1. Creating new Celery tasks in `bulk_processing.py`\n2. Adding new action methods to the mixins\n3. Updating the status view if needed\n\n### Custom Progress Tracking\n\nOverride the progress tracking by extending `BulkOperationCache`:\n\n```python\nfrom django_bulk_drf.bulk_cache import BulkOperationCache\n\nclass CustomBulkCache(BulkOperationCache):\n @classmethod\n def set_custom_metric(cls, task_id: str, metric_data: dict):\n # Custom metric tracking\n pass\n```\n\nThis bulk operations system provides a robust, scalable solution for handling large data operations asynchronously while keeping your API responsive and providing real-time feedback to users.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Django REST Framework mixins for asynchronous bulk operations with Celery and Redis",
"version": "0.1.16",
"project_urls": {
"Homepage": "https://github.com/AugendLimited/django-bulk-drf",
"Repository": "https://github.com/AugendLimited/django-bulk-drf"
},
"split_keywords": [
"django",
" bulk",
" drf",
" rest-framework",
" celery",
" redis",
" async"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d847535c5e9e1e35823a81c47fea5b030b1f572ec49925b63ef4e3ac4f8bf3ab",
"md5": "5418fcda1b42681eac4b21c57110a460",
"sha256": "af9245c619abe32e49a85336e2a66d345c810f5f937d188a4e04c85435dadfa2"
},
"downloads": -1,
"filename": "django_bulk_drf-0.1.16-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5418fcda1b42681eac4b21c57110a460",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.11",
"size": 35068,
"upload_time": "2025-07-17T13:25:58",
"upload_time_iso_8601": "2025-07-17T13:25:58.424793Z",
"url": "https://files.pythonhosted.org/packages/d8/47/535c5e9e1e35823a81c47fea5b030b1f572ec49925b63ef4e3ac4f8bf3ab/django_bulk_drf-0.1.16-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6da616f41611bb8e47270f502066ecdab1e463a8f7744c7fbef3ed9d715e039e",
"md5": "b0b442007f458e82cdbfbf58193554f4",
"sha256": "f300bcab682934b1eb900439e9d1b0b43d4fbc1a615e2633104e4f8efb7defb6"
},
"downloads": -1,
"filename": "django_bulk_drf-0.1.16.tar.gz",
"has_sig": false,
"md5_digest": "b0b442007f458e82cdbfbf58193554f4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.11",
"size": 32167,
"upload_time": "2025-07-17T13:25:59",
"upload_time_iso_8601": "2025-07-17T13:25:59.835021Z",
"url": "https://files.pythonhosted.org/packages/6d/a6/16f41611bb8e47270f502066ecdab1e463a8f7744c7fbef3ed9d715e039e/django_bulk_drf-0.1.16.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-17 13:25:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "AugendLimited",
"github_project": "django-bulk-drf",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "django-bulk-drf"
}
JSON
