sb-sync

Name	sb-sync JSON
Version	2.0.2 JSON
	download
home_page	https://github.com/TheSocialBytes/sync
Summary	Django package for data synchronization with PUSH/PULL APIs
upload_time	2025-08-08 06:38:22
maintainer	None
docs_url	None
author	TheSocialBytes
requires_python	>=3.8
license	MIT
keywords	django synchronization api multi-tenant audit-trails
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            # SB Sync - Django Data Synchronization Package

A robust Django package for data synchronization with PUSH/PULL APIs, featuring JWT authentication, comprehensive logging, performance optimizations, and **multi-tenant, role-based access control** for any Django application.

## 🚀 Features

- **PUSH/PULL API Endpoints**: Bidirectional data synchronization
- **🌐 Web-Based Configuration Interface**: Visual management of permissions and settings
- **📋 Audit Trails**: Complete change history tracking with Django Simple History
- **JWT Authentication**: Secure token-based authentication
- **Multi-Tenant Support**: Organization-based data isolation
- **Role-Based Access Control**: Granular permissions per user role
- **Comprehensive Logging**: Detailed sync operation tracking
- **Performance Optimizations**: Bulk operations and caching
- **Health Monitoring**: Built-in health check endpoints
- **Background Tasks**: Celery integration for maintenance tasks
- **Data Validation**: Automatic model structure validation
- **Error Handling**: Robust error handling and reporting
- **🔄 Auto-Migration System**: Automatic version detection and schema migration

## 📋 Requirements

- Python >= 3.8
- Django >= 3.2, < 5.3 (supports Django 3.2, 4.0, 4.1, 4.2, 5.0, 5.1, 5.2)
- Django REST Framework >= 3.14.0
- PyJWT >= 2.6.0
- Django Simple History >= 3.4.0 (for audit trails)
- Celery (for background tasks)

## 🔧 Django Compatibility

This package is designed to work with a wide range of Django versions:

| Django Version | Status | Support Level |
|----------------|--------|---------------|
| 3.2.x | ✅ Supported | LTS (Long Term Support) |
| 4.0.x | ✅ Supported | Standard |
| 4.1.x | ✅ Supported | Standard |
| 4.2.x | ✅ Supported | LTS (Long Term Support) |
| 5.0.x | ✅ Supported | Standard |
| 5.1.x | ✅ Supported | Standard |
| 5.2.x | ✅ Supported | Standard |
| 5.3.x | ❌ Not yet | Future versions |

**Note**: The package is tested against Django LTS versions and the latest stable releases. For production use, we recommend using Django LTS versions (3.2.x, 4.2.x) for maximum stability.

## 🛠️ Installation

1. Install the package:
```bash
pip install sb-sync
```

2. Add to your Django settings:
```python
INSTALLED_APPS = [
    # ... other apps
    'sb_sync',
]

# Optional settings
SB_SYNC_LOG_DIR = 'logs'  # Directory for sync logs
```

3. Run migrations:
```bash
python manage.py migrate
```

4. Access the web configuration interface:
```bash
# Navigate to: http://your-domain/api/sync/config/
# Login with admin credentials (staff members only)
```

5. Setup audit trails (optional but recommended):
```bash
# Setup Django Simple History for audit trails
python manage.py setup_audit_trails --action setup

# Check audit trails status
python manage.py setup_audit_trails --action check

# Cleanup old audit trail records
python manage.py setup_audit_trails --action cleanup
```

## 🔧 Configuration

### Required Settings

Add to your Django settings:

```python
# JWT Settings
SECRET_KEY = 'your-secret-key'

# Celery Settings (for background tasks)
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

# Optional: Custom log directory
SB_SYNC_LOG_DIR = 'logs'
```

### Model Discovery Configuration

The package automatically discovers Django models for synchronization with advanced filtering options.

#### Basic Configuration

```python
from sb_sync.config import SyncConfig

# Include specific apps
SyncConfig.set_config('MODEL_DISCOVERY', 'INCLUDE_APPS', ['myapp', 'ecommerce'])

# Exclude specific models
SyncConfig.set_config('MODEL_DISCOVERY', 'EXCLUDE_MODELS', ['myapp.LogModel', 'ecommerce.CacheModel'])

# Enable/disable auto discovery
SyncConfig.set_config('MODEL_DISCOVERY', 'AUTO_DISCOVER_MODELS', True)
```

#### Advanced Configuration Options

##### Model Type Filtering
```python
# Exclude abstract and proxy models
SyncConfig.set_config('MODEL_DISCOVERY', 'EXCLUDE_ABSTRACT_MODELS', True)
SyncConfig.set_config('MODEL_DISCOVERY', 'EXCLUDE_PROXY_MODELS', True)

# Include only concrete models
SyncConfig.set_config('MODEL_DISCOVERY', 'INCLUDE_MODEL_TYPES', ['concrete'])
```

##### Pattern-Based Filtering
```python
# Include models matching patterns
SyncConfig.set_config('MODEL_DISCOVERY', 'INCLUDE_MODEL_PATTERNS', [
    r'^myapp\.',  # All models from myapp
    r'^ecommerce\.Product.*$'  # Product models from ecommerce
])

# Exclude models matching patterns
SyncConfig.set_config('MODEL_DISCOVERY', 'EXCLUDE_MODEL_PATTERNS', [
    r'^.*\.Historical.*$',  # Exclude historical models
    r'^.*\.Log$',           # Exclude log models
    r'^.*\.Cache$',         # Exclude cache models
])
```

##### Field-Based Filtering
```python
# Exclude models with specific fields
SyncConfig.set_config('MODEL_DISCOVERY', 'EXCLUDE_MODELS_WITH_FIELDS', [
    'created_at',  # Exclude models with created_at field
    'updated_at',  # Exclude models with updated_at field
    'deleted_at',  # Exclude soft-delete models
])

# Require models to have specific fields
SyncConfig.set_config('MODEL_DISCOVERY', 'REQUIRE_MODELS_WITH_FIELDS', [
    'id'  # Only include models with 'id' field
])
```

##### App-Specific Exclusions
```python
# Exclude specific models from specific apps
SyncConfig.set_config('MODEL_DISCOVERY', 'APP_SPECIFIC_EXCLUSIONS', {
    'auth': ['Group', 'Permission'],  # Exclude Group and Permission from auth
    'admin': ['LogEntry'],            # Exclude LogEntry from admin
})
```

##### Performance and Caching
```python
# Enable discovery caching
SyncConfig.set_config('MODEL_DISCOVERY', 'ENABLE_DISCOVERY_CACHING', True)
SyncConfig.set_config('MODEL_DISCOVERY', 'DISCOVERY_CACHE_TIMEOUT', 3600)  # 1 hour

# Limit models per app
SyncConfig.set_config('MODEL_DISCOVERY', 'MAX_MODELS_PER_APP', 100)
```

#### Management Command

Use the management command to configure model discovery:

```bash
# List current configuration
python manage.py configure_model_discovery --list-current

# Set include apps
python manage.py configure_model_discovery --set-include-apps myapp ecommerce

# Set exclude patterns
python manage.py configure_model_discovery --set-exclude-patterns ".*Log" ".*Cache"

# Set field-based exclusions
python manage.py configure_model_discovery --set-exclude-fields created_at updated_at

# Test discovery with current settings
python manage.py configure_model_discovery --test-discovery

# Reset to defaults
python manage.py configure_model_discovery --reset-to-defaults
```

### 🔄 Auto-Migration System

The package includes an intelligent auto-migration system that automatically detects version gaps and handles schema migrations gracefully.

#### Features

- **Automatic Version Detection**: Detects current database schema version
- **Schema Change Detection**: Identifies required migrations
- **Data Preservation**: Safely migrates data between versions
- **Graceful Error Handling**: Comprehensive error recovery
- **Rollback Capabilities**: Support for migration rollbacks
- **Management Commands**: Command-line migration control
- **Startup Auto-Migration**: Automatic migration on app startup

#### Migration Scenarios Handled

- **v1.x → v2.x**: Organization-based to Site-based migration
- **Fresh Installations**: New database setup
- **Schema Updates**: Field and table structure changes
- **Data Preservation**: Maintains existing data during migration
- **Error Recovery**: Handles migration failures gracefully

#### Management Commands

```bash
# Check migration status (dry run)
python manage.py auto_migrate --dry-run --verbose

# Force migration (even if not needed)
python manage.py auto_migrate --force

# Verbose migration with detailed output
python manage.py auto_migrate --verbose
```

#### Automatic Startup Migration

The auto-migration system runs automatically when the Django app starts:

```python
# In your Django app's apps.py
class SbSyncConfig(AppConfig):
    def ready(self):
        # Auto-migration runs automatically
        from .migration_utils import setup_auto_migration
        setup_auto_migration()
```

#### Migration Detection

The system detects migration needs by checking:

- Database schema version
- Table structure differences
- Foreign key relationships
- Data integrity requirements

#### Example Migration Flow

```python
# 1. Version Detection
current_version = detector.detect_current_version()  # Returns '1.x', '2.x', or 'fresh'

# 2. Schema Change Detection
schema_changes = detector.detect_schema_changes()  # Identifies required changes

# 3. Migration Execution
if migrator.needs_migration():
    success = migrator.auto_migrate()  # Performs migration
```

#### Error Handling

The auto-migration system includes comprehensive error handling:

- **Database Connection Issues**: Graceful handling of connection failures
- **Schema Conflicts**: Resolution of table structure conflicts
- **Data Integrity**: Validation of migrated data
- **Rollback Support**: Ability to revert failed migrations

#### Configuration

```python
# Disable auto-migration (if needed)
SB_SYNC_AUTO_MIGRATION = False

# Custom migration timeout
SB_SYNC_MIGRATION_TIMEOUT = 300  # 5 minutes

# Enable verbose migration logging
SB_SYNC_MIGRATION_VERBOSE = True
```

#### Model Discovery Settings

##### Basic Settings
- `AUTO_DISCOVER_MODELS`: Enable/disable automatic model discovery
- `INCLUDE_APPS`: List of apps whose models will be synced (empty = all apps)
- `EXCLUDE_MODELS`: List of specific models to exclude from sync
- `INCLUDE_CUSTOM_MODELS`: Include custom models in discovery
- `MODEL_PREFIX`: Prefix for model names
- `MODEL_SUFFIX`: Suffix for model names
- `MODEL_NAMESPACE`: Namespace for model names

##### Advanced Settings
- `EXCLUDE_ABSTRACT_MODELS`: Exclude abstract models
- `EXCLUDE_PROXY_MODELS`: Exclude proxy models
- `EXCLUDE_HISTORICAL_MODELS`: Exclude historical models (simple_history)
- `EXCLUDE_MANAGER_MODELS`: Exclude models with custom managers
- `INCLUDE_MODEL_PATTERNS`: Regex patterns for models to include
- `EXCLUDE_MODEL_PATTERNS`: Regex patterns for models to exclude
- `EXCLUDE_MODELS_WITH_FIELDS`: Field names that will exclude models
- `REQUIRE_MODELS_WITH_FIELDS`: Field names that models must have
- `APP_SPECIFIC_EXCLUSIONS`: Per-app exclusion rules
- `INCLUDE_MODEL_TYPES`: Types of models to include ('concrete', 'abstract', 'proxy')
- `ENABLE_DISCOVERY_CACHING`: Enable discovery result caching
- `DISCOVERY_CACHE_TIMEOUT`: Cache timeout in seconds
- `MAX_MODELS_PER_APP`: Maximum models per app
- `VALIDATE_MODEL_ACCESS`: Validate that models can be accessed
- `CHECK_MODEL_PERMISSIONS`: Check if current user can access models
- `SAFE_DISCOVERY_MODE`: Only discover models that are safe to sync

### URL Configuration

Include the sync URLs in your main `urls.py`:

```python
from django.urls import path, include

urlpatterns = [
    # ... other URLs
    path('api/sync/', include('sb_sync.urls')),
]
```

## 🏢 Multi-Tenant Setup

### 1. Create Organizations

```bash
# Create organizations
python manage.py setup_organizations --action create_org --org-name "Acme Corporation" --org-slug acme-corp
python manage.py setup_organizations --action create_org --org-name "Global Retail" --org-slug global-retail
python manage.py setup_organizations --action create_org --org-name "City University" --org-slug city-university
```

### 2. Create Django Groups

```bash
# Create common groups for the application
python manage.py setup_organizations --action create_groups
```

### 3. Add Users to Organizations

```bash
# Add users with specific groups
python manage.py setup_organizations --action add_user --username john_manager --org-slug acme-corp --group-name Managers
python manage.py setup_organizations --action add_user --username mary_sales --org-slug global-retail --group-name Sales
python manage.py setup_organizations --action add_user --username bob_analyst --org-slug city-university --group-name Analysts
```

### 4. Setup Complete Example System

```bash
# Setup all example organizations with groups and permissions
python manage.py setup_organizations --action setup_example
```

## 👥 User Groups and Permissions

### Using Django Groups

The system uses Django's built-in `auth.Group` system for role-based access control. This makes it completely generic and reusable across any Django project.

### Available Groups

The system comes with common groups that can be customized for any domain:

1. **Administrators** - Full access to all data (create, read, update, delete)
2. **Managers** - Can push/pull data, create, update (no delete)
3. **Users** - Can push/pull data, create, update (no delete)
4. **Analysts** - Can only pull data (read-only access)
5. **Sales** - Can push/pull sales-related data
6. **Support** - Can push/pull support-related data
7. **Read Only** - Can only pull data, no push access

### Permission Matrix

| Group | Push | Pull |
|-------|------|------|
| Administrators | ✅ Yes | ✅ Yes |
| Managers | ✅ Yes | ✅ Yes |
| Users | ✅ Yes | ✅ Yes |
| Analysts | ❌ No | ✅ Yes |
| Sales | ✅ Yes | ✅ Yes |
| Support | ✅ Yes | ✅ Yes |
| Read Only | ❌ No | ✅ Yes |

## 🌐 Web-Based Configuration Interface

The SB Sync package includes a comprehensive web-based configuration interface that allows you to manage model permissions, monitor sync operations, and configure model discovery through an intuitive web interface.

### 🚀 Interface Features

- **📊 Dashboard**: Overview statistics and quick actions
- **🔐 Permission Matrix**: Visual matrix interface for managing model permissions
- **🔍 Model Discovery**: Configure which models are available for sync
- **📋 Sync Logs**: View operation history and performance data
- **📈 Performance Metrics**: Interactive charts and detailed metrics

### 🛠️ Accessing the Interface

1. **Navigate to the configuration dashboard**:
   ```
   http://your-domain/api/sync/config/
   ```

2. **Login with admin credentials** (staff members only)

3. **Use the sidebar navigation** to access different sections

### 🔐 Permission Matrix

The permission matrix provides a visual interface for managing model permissions:

- **Visual Matrix**: See all models vs permissions in a matrix format
- **Checkbox Controls**: Grant or revoke permissions with simple checkboxes
- **Organization Selection**: Switch between different organizations
- **Real-time Updates**: Changes are saved immediately via AJAX
- **Bulk Operations**: Select all/deselect all functionality

**URL**: `/api/sync/config/permissions/`

### 🔍 Model Discovery Configuration

Configure which models are discovered and available for sync:

- **Auto Discovery Settings**: Enable/disable automatic model discovery
- **App Filtering**: Include specific apps or all apps
- **Model Exclusion**: Exclude specific models from sync operations
- **Live Preview**: See which models are discovered in real-time

**URL**: `/api/sync/config/model-discovery/`

### 📋 Sync Logs

Monitor sync operations and performance:

- **Operation History**: View all sync operations with timestamps
- **Performance Data**: See processing times and record counts
- **Export Functionality**: Download logs as CSV
- **Auto-refresh**: Logs update automatically every 30 seconds

**URL**: `/api/sync/config/logs/`

### 📈 Performance Metrics

Track system performance and optimization:

- **Performance Charts**: Visual charts showing processing time trends
- **Detailed Metrics**: View batch sizes, memory usage, and query counts
- **Performance Analysis**: Automatic suggestions for optimization
- **Export Data**: Download performance data as CSV

**URL**: `/api/sync/config/metrics/`

### 📋 Audit Trails

Track all changes to sync system with comprehensive audit trails:

- **Complete History**: Track all changes to sync models (create, update, delete)
- **User Attribution**: See who made each change and when
- **Field-Level Changes**: View exactly what fields were modified
- **Filtering & Search**: Filter by model type, user, date range
- **Export Capabilities**: Download audit trail data as CSV
- **Real-time Updates**: Auto-refreshing audit trail display

**URL**: `/api/sync/config/audit-trails/`

### 🎨 Interface Features

- **Modern Design**: Bootstrap 5 with gradient backgrounds and smooth animations
- **Responsive Design**: Works on all devices (mobile, tablet, desktop)
- **Real-time Updates**: AJAX-powered with immediate feedback
- **Security**: Staff-only access with CSRF protection
- **Performance**: Cached data and optimized queries

## 📡 API Endpoints

### Authentication

**POST** `/api/sync/auth/token/`

Get JWT token for authentication:
```json
{
    "username": "your_username",
    "password": "your_password"
}
```

Response (now includes organization info):
```json
{
    "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9...",
    "user": {
        "id": 1,
        "username": "dr_smith",
        "email": "dr.smith@citygeneral.com"
    },
    "organizations": [
        {
            "id": 1,
            "name": "Acme Corporation",
            "slug": "acme-corp",
            "group": "Managers"
        }
    ]
}
```

### PUSH API

**POST** `/api/sync/push/`

Push data to Django models (with multi-tenant permissions):

```json
{
    "data": [
        {
            "_model": "myapp.Customer",
            "name": "John Doe",
            "age": 45,
            "department": "SALES",
            "assigned_manager_id": 1
        }
    ]
}
```

Headers:
```
Authorization: Bearer <your_jwt_token>
```

Response:
```json
{
    "status": "success",
    "success_count": 1,
    "error_count": 0,
    "processed_models": {
        "myapp.Customer": {
            "created": 1,
            "updated": 0
        }
    },
    "processing_time": 0.045
}
```

### PULL API

**POST** `/api/sync/pull/`

Pull data from Django models (with role-based filtering):

```json
{
    "models": {
        "myapp.Customer": "2024-01-14T10:00:00Z",
        "myapp.Order": "2024-01-14T10:00:00Z"
    },
    "batch_size": 100
}
```

Headers:
```
Authorization: Bearer <your_jwt_token>
```

Response:
```json
{
    "data": [
        {
            "_model": "myapp.Customer",
            "id": 1,
            "name": "John Doe",
            "age": 45,
            "department": "SALES",
            "assigned_manager_id": 1,
            "organization": 1
        }
    ],
    "metadata": {
        "myapp.Customer": {
            "count": 1,
            "last_sync": "2024-01-15T10:30:00Z",
            "user_last_sync": "2024-01-14T10:00:00Z"
        }
    },
    "batch_info": {
        "batch_size": 100,
        "total_records": 1
    }
}
```

### Performance Monitoring

**GET** `/api/sync/performance/`

Get performance statistics:

```json
{
    "performance_stats": {
        "total_operations": 150,
        "average_processing_time": 0.045,
        "cache_hit_rate": 0.85
    },
    "current_memory_usage": 245.6,
    "cache_stats": {
        "cache_hits": 1200,
        "cache_misses": 200
    },
    "optimization_suggestions": [
        "High memory usage detected. Consider reducing batch sizes."
    ]
}
```

### Health Check

**GET** `/api/sync/health/`

Check system health:

```json
{
    "status": "healthy",
    "timestamp": "2024-01-01T12:00:00Z",
    "checks": {
        "database": "healthy",
        "cache": "healthy",
        "logging": "healthy"
    }
}
```

## 🗄️ Database Models

### Core Models

#### SyncLog

Tracks all synchronization operations:

```python
class SyncLog(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE, db_index=True)
    operation = models.CharField(max_length=10, choices=[('PUSH', 'Push'), ('PULL', 'Pull')], db_index=True)
    status = models.CharField(max_length=10, choices=[('SUCCESS', 'Success'), ('ERROR', 'Error'), ('WARNING', 'Warning')], db_index=True)
    model_name = models.CharField(max_length=100, blank=True, db_index=True)
    object_count = models.IntegerField(default=0)
    error_message = models.TextField(blank=True)
    request_data = models.JSONField(blank=True, null=True)
    timestamp = models.DateTimeField(default=timezone.now, db_index=True)
    processing_time = models.FloatField(default=0.0)
```

#### SyncMetadata

Tracks last sync timestamps for models:

```python
class SyncMetadata(models.Model):
    model_name = models.CharField(max_length=100, unique=True, db_index=True)
    last_sync = models.DateTimeField(default=timezone.now, db_index=True)
    total_synced = models.BigIntegerField(default=0)
```

#### PerformanceMetrics

Tracks performance metrics for optimization:

```python
class PerformanceMetrics(models.Model):
    operation_type = models.CharField(max_length=20, db_index=True)
    model_name = models.CharField(max_length=100, db_index=True)
    batch_size = models.IntegerField()
    processing_time = models.FloatField()
    memory_usage = models.FloatField(null=True, blank=True)
    query_count = models.IntegerField()
    timestamp = models.DateTimeField(default=timezone.now, db_index=True)
```

### Multi-Tenant Models

#### Organization

Represents organizations, companies, or institutions:

```python
class Organization(models.Model):
    name = models.CharField(max_length=200, unique=True)
    slug = models.CharField(max_length=50, unique=True, db_index=True)
    description = models.TextField(blank=True)
    is_active = models.BooleanField(default=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)
```

#### UserOrganization

Links users to organizations with roles:

```python
class UserOrganization(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE, db_index=True)
    organization = models.ForeignKey(Organization, on_delete=models.CASCADE, db_index=True)
    role = models.CharField(max_length=50, choices=[
        ('ADMIN', 'Administrator'),
        ('MANAGER', 'Manager'),
        ('USER', 'User'),
        ('ANALYST', 'Analyst'),
        ('SALES', 'Sales'),
        ('SUPPORT', 'Support'),
        ('READ_ONLY', 'Read Only'),
    ], db_index=True)
    is_active = models.BooleanField(default=True)
    created_at = models.DateTimeField(auto_now_add=True)
```

#### ModelPermission

Defines which models each group can access:

```python
class ModelPermission(models.Model):
    organization = models.ForeignKey(Organization, on_delete=models.CASCADE, db_index=True)
    group = models.ForeignKey(Group, on_delete=models.CASCADE, db_index=True, help_text="Django auth group")
    model_name = models.CharField(max_length=100, db_index=True)
    can_push = models.BooleanField(default=False)
    can_pull = models.BooleanField(default=False)
    filters = models.JSONField(blank=True, null=True)  # Custom filters for data access
```

#### UserSyncMetadata

Tracks last sync timestamps for models per user/organization:

```python
class UserSyncMetadata(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE, db_index=True)
    organization = models.ForeignKey(Organization, on_delete=models.CASCADE, db_index=True)
    model_name = models.CharField(max_length=100, db_index=True)
    last_sync = models.DateTimeField(default=timezone.now, db_index=True)
    total_synced = models.BigIntegerField(default=0)
```

#### DataFilter

Custom data filters for role-based access:

```python
class DataFilter(models.Model):
    organization = models.ForeignKey(Organization, on_delete=models.CASCADE, db_index=True)
    role = models.CharField(max_length=50, db_index=True)
    model_name = models.CharField(max_length=100, db_index=True)
    filter_name = models.CharField(max_length=100)
    filter_condition = models.JSONField()  # e.g., {"field": "department", "operator": "exact", "value": "CARDIOLOGY"}
    is_active = models.BooleanField(default=True)
    created_at = models.DateTimeField(auto_now_add=True)
```

## 🔄 Background Tasks

### Celery Configuration

Add to your project's `__init__.py`:

```python
from .celery import app as celery_app
__all__ = ('celery_app',)
```

### Available Tasks

1. **cleanup_old_sync_logs**: Removes sync logs older than 90 days
2. **generate_sync_report**: Generates daily sync statistics
3. **optimize_database_tables**: Analyzes and optimizes database tables
4. **bulk_sync_operation**: Processes bulk sync operations asynchronously
5. **cache_warmup**: Warms up cache with frequently accessed data
6. **memory_optimization**: Performs garbage collection and cache cleanup
7. **performance_analysis**: Analyzes performance and provides recommendations

### Running Celery

```bash
# Start Celery worker
celery -A your_project worker -l info

# Start Celery beat (for scheduled tasks)
celery -A your_project beat -l info
```

## 🛡️ Security

- **JWT Authentication**: All API endpoints require valid JWT tokens
- **Token Expiration**: Tokens expire after 7 days by default
- **Multi-Tenant Isolation**: Complete data isolation between organizations
- **Role-Based Access Control**: Granular permissions per user role
- **Data Filtering**: Role-specific data access filters
- **User Validation**: All operations are tied to authenticated users
- **Input Validation**: Comprehensive data validation against Django models

## 📊 Monitoring & Logging

### Log Files

Sync operations are logged to `logs/sb_sync.log` with daily rotation and 30-day retention.

### Log Format

```
2024-01-01 12:00:00 - sb_sync - INFO - PUSH request from user john_manager in Acme Corporation: {"data": [...]}
```

### Health Monitoring

Use the health check endpoint to monitor:
- Database connectivity
- Cache functionality
- Log directory accessibility
- Memory usage
- Performance metrics

## ⚡ Performance Optimizations

### Bulk Operations

The package includes optimized bulk operations for high-volume data processing:

```python
from sb_sync.optimizations import BulkOperations

# Bulk create/update with batching
results = BulkOperations.bulk_create_or_update(
    model_class=YourModel,
    data_list=large_dataset,
    batch_size=1000
)
```

### Model Metadata Caching

Model field information is cached for improved performance:

```python
from sb_sync.optimizations import ModelMetadataCache

# Get cached model fields
fields = ModelMetadataCache.get_model_fields('app.ModelName')
```

### Query Optimization

```python
from sb_sync.optimizations import QueryOptimizer

# Get optimized sync logs
logs = QueryOptimizer.get_optimized_sync_logs(user, organization)

# Count queries with decorator
@QueryOptimizer.count_queries
def my_function():
    # Your code here
    pass
```

### Memory Optimization

```python
from sb_sync.optimizations import MemoryOptimizer

# Monitor memory usage
memory_usage = MemoryOptimizer.get_memory_usage()

# Memory monitoring decorator
@MemoryOptimizer.monitor_memory
def my_function():
    # Your code here
    pass
```

### Cache Optimization

```python
from sb_sync.optimizations import CacheOptimizer

# Get or set cache
data = CacheOptimizer.get_or_set_cache('key', lambda: expensive_operation())

# Cache model data
CacheOptimizer.cache_model_data('key', data, timeout=300)
```

## 🔧 Management Commands

### Model Discovery and Default Models

The system automatically discovers all Django models in your application and makes them available for push/pull operations by default. Only specific apps and models are excluded via configuration.

```bash
# Show model discovery summary
python manage.py show_models --action summary

# Show all discovered models
python manage.py show_models --action all

# Show enabled models only
python manage.py show_models --action enabled

# Show default models for push/pull operations
python manage.py show_models --action default

# Show detailed model information
python manage.py show_models --action details --verbose

# Show models from specific app
python manage.py show_models --action enabled --app-label myapp
```

**Default Behavior:**
- ✅ **Auto-discovers all Django models** in your application
- ✅ **`INCLUDE_APPS`**: List of apps whose models will be synced (empty = all apps)
- ✅ **`EXCLUDE_MODELS`**: Models within those included apps that will be excluded
- ✅ **Makes all discovered models available** for push/pull operations
- ✅ **No configuration required** - works out of the box

**Configuration Options:**
- `AUTO_DISCOVER_MODELS`: Enable/disable automatic model discovery
- `INCLUDE_APPS`: List of apps whose models will be synced (empty = all apps)
- `EXCLUDE_MODELS`: Models within included apps that will be excluded from sync
- `INCLUDE_CUSTOM_MODELS`: Include custom models from your apps

### Cleanup Sync Logs

```bash
python manage.py cleanup_sync_logs
```

### Performance Optimization

```bash
# Analyze performance
python manage.py optimize_performance --action analyze --days 7

# Optimize performance
python manage.py optimize_performance --action optimize --force

# Cleanup old data
python manage.py optimize_performance --action cleanup

# Monitor resources
python manage.py optimize_performance --action monitor
```

### Configuration Management

```bash
# Show current configuration
python manage.py manage_config --action show

# Export configuration
python manage.py manage_config --action export --file config.json --format json

# Import configuration
python manage.py manage_config --action import --file config.json --format json

# Validate configuration
python manage.py manage_config --action validate

# Get configuration summary
python manage.py manage_config --action summary

# Reset to defaults
python manage.py manage_config --action reset --force
```

### Organization Setup

```bash
# Create organization
python manage.py setup_organizations --action create_org --org-name "Acme Corporation" --org-slug acme-corp

# Add user to organization
python manage.py setup_organizations --action add_user --username john_manager --org-slug acme-corp --group-name Managers

# Set permissions from config file
python manage.py setup_organizations --action set_permissions --org-slug acme-corp --config-file permissions.json

# Setup complete example system
python manage.py setup_organizations --action setup_example
```

### Audit Trails Management

```bash
# Setup audit trails for all sync models
python manage.py setup_audit_trails --action setup

# Check audit trails status
python manage.py setup_audit_trails --action check

# Cleanup old audit trail records
python manage.py setup_audit_trails --action cleanup

# Setup audit trails for specific model
python manage.py setup_audit_trails --action setup --model sb_sync.Organization

# Check specific model audit trails
python manage.py setup_audit_trails --action check --model sb_sync.ModelPermission
```

### Dynamic Permission Configuration

```bash
# Discover all models in your project
python manage.py dynamic_permissions --action discover

# Generate permission configuration
python manage.py dynamic_permissions --action generate --org-slug acme-corp --permission-template read_write

# Apply permission configuration
python manage.py dynamic_permissions --action apply --org-slug acme-corp --config-file permissions.json

# Export current permissions
python manage.py dynamic_permissions --action export --org-slug acme-corp --output-file current_permissions.json

# Validate configuration file
python manage.py dynamic_permissions --action validate --config-file permissions.json

# Show available templates
python manage.py dynamic_permissions --action template
```

## 🧪 Testing

### Example Usage

```python
import requests
import json

# Get authentication token
auth_response = requests.post('http://localhost:8000/api/sync/auth/token/', {
    'username': 'dr_smith',
    'password': 'password'
})
token = auth_response.json()['token']

# Push data (with multi-tenant permissions)
headers = {'Authorization': f'Bearer {token}'}
push_data = {
    'data': [
        {
            '_model': 'myapp.Customer',
            'name': 'John Doe',
            'department': 'SALES',
            'assigned_manager_id': 1
        }
    ]
}
response = requests.post('http://localhost:8000/api/sync/push/', 
                        json=push_data, headers=headers)
print(response.json())

# Pull data (with role-based filtering)
pull_data = {
    'models': {
        'myapp.Customer': '2024-01-14T10:00:00Z',
        'myapp.Order': '2024-01-14T10:00:00Z'
    },
    'batch_size': 100
}
response = requests.post('http://localhost:8000/api/sync/pull/', 
                        json=pull_data, headers=headers)
print(response.json())
```

## 📝 Error Handling

The package provides comprehensive error handling:

- **Validation Errors**: Detailed field validation messages
- **Model Errors**: Clear error messages for missing or invalid models
- **Authentication Errors**: Proper JWT token validation
- **Permission Errors**: Multi-tenant and role-based access control errors
- **Database Errors**: Transaction rollback on errors
- **Partial Success Handling**: Graceful handling of partial failures

## 🏢 Multi-Tenant Use Case

### Scenario: Multiple Organizations

The system supports complex multi-tenant scenarios with multiple organizations:

```python
# Organization A (Acme Corp) - Manager Smith
POST /api/sync/push/
{
    "data": [
        {
            "_model": "myapp.Customer",
            "name": "John Doe",
            "department": "SALES",
            "assigned_manager_id": 1
        }
    ]
}
# ✅ Success - Manager Smith has permission

# Organization B (Global Retail) - Sales Wilson  
POST /api/sync/pull/
{
    "models": {"myapp.Customer": "2024-01-14T10:00:00Z"}
}
# ✅ Success - Sales Wilson gets only their assigned customers

# Organization C (City University) - Analyst Garcia
POST /api/sync/push/
{
    "data": [
        {
            "_model": "myapp.Report",
            "customer_id": 5,
            "report_type": "ANALYSIS",
            "results": "Positive"
        }
    ]
}
# ✅ Success - Analyst has permission for reports
```

### Key Benefits for Multi-Tenant Applications:

1. **🔒 Data Isolation**: Each organization only sees their own data
2. **👥 Role-Based Access**: Different roles have appropriate permissions
3. **📊 Granular Control**: Filter by department, assigned staff, etc.
4. **🔄 Per-User Sync Tracking**: Each user has their own sync history
5. **⚡ Performance**: Optimized for high-volume data processing
6. **🛡️ Security**: Meets enterprise data privacy requirements

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

## 📄 License

This project is licensed under the MIT License.

## 🆘 Support

For issues and questions:
- Check the health endpoint: `/api/sync/health/`
- Review sync logs in `logs/sb_sync.log`
- Monitor Celery task logs for background operations
- Check performance metrics: `/api/sync/performance/`

## �� Changelog

### **v2.0.0** (Latest) - 2025-08-07
**🚀 Major Release: Django Sites Integration & UI Improvements**
- **🔄 Architecture Change**: Replaced custom Organization model with Django's built-in Sites framework
- **🎨 UI Enhancement**: Updated all interfaces to use user-friendly "Organization" terminology
- **🔧 Technical Improvements**: 
  - Renamed `UserOrganization` to `UserSite` for better Django Sites integration
  - Updated all models to use `site` instead of `organization` fields
  - Simplified permission system with push/pull permissions only
  - Enhanced admin interface with proper field mappings
  - Fixed all indentation and configuration errors
- **📱 User Experience**: 
  - "Organization Selector" instead of "Site Selector" in permission matrix
  - "Organizations Overview" in configuration dashboard
  - Professional admin interface with organization actions
  - Consistent terminology across all interfaces
- **⚡ Performance**: 
  - Optimized database queries with proper select_related
  - Enhanced caching mechanisms
  - Improved bulk operations for permissions
- **🔒 Security**: 
  - Maintained all security features with Django Sites
  - Enhanced permission checking with proper site isolation
- **📊 Monitoring**: 
  - Added comprehensive audit trails
  - Enhanced logging and error handling
  - Improved performance metrics tracking

### **v1.9.2** - 2025-08-06
**🔧 Permission System Fixes**
- **Fixed**: Pull API permission issues causing "you don't have permission" errors
- **Fixed**: Removed invalid `is_active` filter from ModelPermission queries
- **Added**: Proper superuser handling in permission system
- **Enhanced**: Group resolution for admin users and superusers
- **Added**: Comprehensive debugging logs for permission troubleshooting
- **Improved**: Error handling and logging for permission checks
- **Fixed**: Admin users with proper group assignments now have correct pull/push permissions

### **v1.9.1** - 2025-08-06
**🎨 Improved Permission Matrix UI**
- **Enhanced**: Split push and pull permissions into separate columns
- **Added**: Individual "select all" checkboxes for push and pull permissions per group
- **Improved**: Better visual organization with dedicated columns for each permission type
- **Enhanced**: More intuitive interface for permission management
- **Updated**: JavaScript logic to handle separate column states independently
- **Improved**: User experience with clearer permission type separation
- **Maintained**: All existing functionality (auto-save, debouncing, bulk operations)

### **v1.9.0** - 2025-08-06
**🔧 Simplified Permission System**
- **Removed**: Delete, read, create, and update permissions from the system
- **Simplified**: Permission system now only includes push and pull permissions
- **Updated**: ModelPermission model to only have can_push and can_pull fields
- **Updated**: Permission matrix interface to show only push/pull permissions
- **Updated**: Admin interface to reflect simplified permission structure
- **Updated**: All views and templates to work with simplified permissions
- **Added**: Database migration to remove old permission fields
- **Improved**: Cleaner and more focused permission management

### **v1.8.0** - 2025-08-06
**🔧 Template Tag Fix & Version Display**
- **Fixed**: Template tag loading error for get_version in base.html
- **Added**: {% load sb_sync_extras %} to properly load custom template tags
- **Fixed**: Version display in configuration interface sidebar
- **Improved**: Template error handling and tag registration
- **Tested**: Template rendering without errors

### **v1.7.0** - 2025-08-06
**🚀 Persistent Configuration Storage**
- **Added**: SyncConfiguration model for persistent database storage
- **Implemented**: Database-backed configuration with JSONField support
- **Added**: History tracking for configuration changes with Django Simple History
- **Enhanced**: get_config() and set_config() methods with database fallback
- **Fixed**: Configuration persistence across server restarts
- **Added**: Graceful fallback to in-memory storage if database unavailable
- **Improved**: Model discovery with persistent INCLUDE_APPS configuration
- **Added**: Proper indexing and constraints for configuration table
- **Tested**: Configuration persistence and model discovery functionality

### **v1.6.1** - 2025-08-06
**🔧 Template Tag Fix & Bug Resolution**
- **Fixed**: AttributeError in permission matrix template ('bool' object has no attribute 'get')
- **Enhanced**: lookup filter in sb_sync_extras.py to handle multiple data types
- **Added**: Support for boolean values, dictionary-like objects, lists, and object attributes
- **Improved**: Error handling for template tag operations
- **Tested**: Permission matrix endpoint accessibility and functionality

### **v1.6.0** - 2025-08-06
**🔧 Model Discovery Logic Enhancement**
- **Fixed**: Model discovery logic to exclude Django built-in apps when INCLUDE_APPS is empty
- **Fixed**: Excluded sb-sync app itself and its dependencies from model discovery
- **Enhanced**: get_all_models() and is_model_enabled() methods with proper exclusion logic
- **Added**: Comprehensive list of excluded apps (Django built-ins, sb-sync, dependencies)
- **Fixed**: Missing dependencies (psutil, django-simple-history) installation issues
- **Updated**: URL configuration to properly expose model discovery endpoint
- **Tested**: Model discovery logic with custom test apps
- **Improved**: Server startup reliability and dependency management

### **v1.5.3** - 2025-08-06
**🔧 Package Installation Fixes**
- **Fixed**: Resolved pip installation issues with proper package structure
- **Fixed**: Moved all files to `sb_sync/` directory for correct Python package layout
- **Fixed**: Cleaned up excessive dependencies in `setup.py`
- **Fixed**: Removed invalid entry points that caused installation errors
- **Added**: `MANIFEST.in` file for proper package data inclusion
- **Updated**: Project URLs to point to correct GitHub repository
- **Tested**: Package installation and import functionality

### **v1.5.2** - 2025-08-06
**🔧 Django 5.2.x Compatibility & Installation Fixes**
- **Added**: Support for Django 5.2.x
- **Updated**: Django version constraint from `<5.1` to `<5.3`
- **Added**: Django 5.1 and 5.2 framework classifiers
- **Fixed**: Package structure for proper pip installation
- **Added**: Comprehensive Django compatibility documentation

### **v1.5.1** - 2025-08-06
**🔧 Django Compatibility Enhancement**
- **Added**: Support for Django 5.1.x and 5.2.x
- **Updated**: Django version requirements to support broader range
- **Added**: Django compatibility table in documentation
- **Fixed**: Version constraints in setup.py

### **v1.5.0** - 2024-01-XX
**🌐 Web-Based Configuration Interface & Audit Trails**
- **Added**: Complete web-based configuration interface with dashboard
- **Added**: Visual permission matrix for model permissions management
- **Added**: Model discovery configuration UI
- **Added**: Sync logs viewer with real-time updates
- **Added**: Performance metrics visualization with charts
- **Added**: Django Simple History integration for comprehensive audit trails
- **Added**: Audit trails management command (`setup_audit_trails`)
- **Added**: Custom template tags for enhanced UI functionality
- **Enhanced**: Admin interface with historical records support
- **Updated**: All sync models with audit trail capabilities
- **Added**: Bootstrap 5, Font Awesome, and Chart.js for modern UI

### **v1.4.0** - 2024-01-XX
**🧹 Package Streamlining & Code Organization**
- **Removed**: Dynamic data sources functionality for focused scope
- **Improved**: Code organization and structure
- **Enhanced**: Documentation clarity and completeness
- **Streamlined**: Package for Django model synchronization focus
- **Cleaned**: Codebase organization and removed unused components

### **v1.3.0** - 2024-01-XX
**🔍 Model Discovery & Configuration Enhancement**
- **Added**: Include-based model discovery system
- **Enhanced**: Configuration system with better flexibility
- **Improved**: Test coverage across all components
- **Updated**: Comprehensive documentation cleanup
- **Added**: Model introspection capabilities
- **Enhanced**: Error handling and validation

### **v1.2.0** - 2024-01-XX
**🚀 Performance & Multi-Tenant Features**
- **Added**: Multi-tenant support with organization-based isolation
- **Added**: Role-based access control (RBAC)
- **Enhanced**: Performance optimizations with bulk operations
- **Added**: Comprehensive health monitoring
- **Improved**: Background task processing with Celery
- **Added**: Advanced caching mechanisms

### **v1.1.0** - 2024-01-XX
**⚡ Performance & Error Handling**
- **Enhanced**: Configuration system flexibility
- **Added**: Performance optimizations for large datasets
- **Improved**: Error handling and reporting mechanisms
- **Added**: Advanced logging capabilities
- **Enhanced**: Data validation processes

### **v1.0.0** - 2024-01-XX
**🎉 Initial Release**
- **Added**: PUSH/PULL API endpoints for bidirectional sync
- **Added**: JWT authentication system
- **Added**: Comprehensive logging infrastructure
- **Added**: Basic model synchronization capabilities
- **Added**: Health check endpoints
- **Added**: Initial documentation and setup guides

---

## 🔄 Migration Guide

### **Upgrading to v1.5.x**

If you're upgrading from an earlier version:

1. **Update your installation**:
   ```bash
   pip install --upgrade sb-sync
   ```

2. **Run new migrations**:
   ```bash
   python manage.py migrate
   ```

3. **Setup audit trails** (recommended):
   ```bash
   python manage.py setup_audit_trails --action setup
   ```

4. **Access new web interface**:
   - Navigate to: `http://your-domain/api/sync/config/`
   - Login with admin/staff credentials

### **Breaking Changes**

- **v1.4.0**: Removed dynamic data sources - migrate to model-based sync
- **v1.5.0**: Added new dependencies (`django-simple-history`) - run `pip install --upgrade sb-sync`

### **New Features Guide**

- **Web Interface**: Access at `/api/sync/config/` for visual management
- **Audit Trails**: Use management command to setup and manage historical records
- **Django 5.2.x**: Full compatibility with latest Django versions 

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/TheSocialBytes/sync",
    "name": "sb-sync",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.8",
    "maintainer_email": "TheSocialBytes <info@thesocialbytes.com>",
    "keywords": "django, synchronization, api, multi-tenant, audit-trails",
    "author": "TheSocialBytes",
    "author_email": "TheSocialBytes <info@thesocialbytes.com>",
    "download_url": "https://files.pythonhosted.org/packages/25/f9/bca8b99c738be2ba5e5a8eb16f5d8d7612864d2e9b1ccaa47dc29459b25d/sb_sync-2.0.2.tar.gz",
    "platform": null,
    "description": "# SB Sync - Django Data Synchronization Package\n\nA robust Django package for data synchronization with PUSH/PULL APIs, featuring JWT authentication, comprehensive logging, performance optimizations, and **multi-tenant, role-based access control** for any Django application.\n\n## \ud83d\ude80 Features\n\n- **PUSH/PULL API Endpoints**: Bidirectional data synchronization\n- **\ud83c\udf10 Web-Based Configuration Interface**: Visual management of permissions and settings\n- **\ud83d\udccb Audit Trails**: Complete change history tracking with Django Simple History\n- **JWT Authentication**: Secure token-based authentication\n- **Multi-Tenant Support**: Organization-based data isolation\n- **Role-Based Access Control**: Granular permissions per user role\n- **Comprehensive Logging**: Detailed sync operation tracking\n- **Performance Optimizations**: Bulk operations and caching\n- **Health Monitoring**: Built-in health check endpoints\n- **Background Tasks**: Celery integration for maintenance tasks\n- **Data Validation**: Automatic model structure validation\n- **Error Handling**: Robust error handling and reporting\n- **\ud83d\udd04 Auto-Migration System**: Automatic version detection and schema migration\n\n## \ud83d\udccb Requirements\n\n- Python >= 3.8\n- Django >= 3.2, < 5.3 (supports Django 3.2, 4.0, 4.1, 4.2, 5.0, 5.1, 5.2)\n- Django REST Framework >= 3.14.0\n- PyJWT >= 2.6.0\n- Django Simple History >= 3.4.0 (for audit trails)\n- Celery (for background tasks)\n\n## \ud83d\udd27 Django Compatibility\n\nThis package is designed to work with a wide range of Django versions:\n\n| Django Version | Status | Support Level |\n|----------------|--------|---------------|\n| 3.2.x | \u2705 Supported | LTS (Long Term Support) |\n| 4.0.x | \u2705 Supported | Standard |\n| 4.1.x | \u2705 Supported | Standard |\n| 4.2.x | \u2705 Supported | LTS (Long Term Support) |\n| 5.0.x | \u2705 Supported | Standard |\n| 5.1.x | \u2705 Supported | Standard |\n| 5.2.x | \u2705 Supported | Standard |\n| 5.3.x | \u274c Not yet | Future versions |\n\n**Note**: The package is tested against Django LTS versions and the latest stable releases. For production use, we recommend using Django LTS versions (3.2.x, 4.2.x) for maximum stability.\n\n## \ud83d\udee0\ufe0f Installation\n\n1. Install the package:\n```bash\npip install sb-sync\n```\n\n2. Add to your Django settings:\n```python\nINSTALLED_APPS = [\n    # ... other apps\n    'sb_sync',\n]\n\n# Optional settings\nSB_SYNC_LOG_DIR = 'logs'  # Directory for sync logs\n```\n\n3. Run migrations:\n```bash\npython manage.py migrate\n```\n\n4. Access the web configuration interface:\n```bash\n# Navigate to: http://your-domain/api/sync/config/\n# Login with admin credentials (staff members only)\n```\n\n5. Setup audit trails (optional but recommended):\n```bash\n# Setup Django Simple History for audit trails\npython manage.py setup_audit_trails --action setup\n\n# Check audit trails status\npython manage.py setup_audit_trails --action check\n\n# Cleanup old audit trail records\npython manage.py setup_audit_trails --action cleanup\n```\n\n## \ud83d\udd27 Configuration\n\n### Required Settings\n\nAdd to your Django settings:\n\n```python\n# JWT Settings\nSECRET_KEY = 'your-secret-key'\n\n# Celery Settings (for background tasks)\nCELERY_BROKER_URL = 'redis://localhost:6379/0'\nCELERY_RESULT_BACKEND = 'redis://localhost:6379/0'\n\n# Optional: Custom log directory\nSB_SYNC_LOG_DIR = 'logs'\n```\n\n### Model Discovery Configuration\n\nThe package automatically discovers Django models for synchronization with advanced filtering options.\n\n#### Basic Configuration\n\n```python\nfrom sb_sync.config import SyncConfig\n\n# Include specific apps\nSyncConfig.set_config('MODEL_DISCOVERY', 'INCLUDE_APPS', ['myapp', 'ecommerce'])\n\n# Exclude specific models\nSyncConfig.set_config('MODEL_DISCOVERY', 'EXCLUDE_MODELS', ['myapp.LogModel', 'ecommerce.CacheModel'])\n\n# Enable/disable auto discovery\nSyncConfig.set_config('MODEL_DISCOVERY', 'AUTO_DISCOVER_MODELS', True)\n```\n\n#### Advanced Configuration Options\n\n##### Model Type Filtering\n```python\n# Exclude abstract and proxy models\nSyncConfig.set_config('MODEL_DISCOVERY', 'EXCLUDE_ABSTRACT_MODELS', True)\nSyncConfig.set_config('MODEL_DISCOVERY', 'EXCLUDE_PROXY_MODELS', True)\n\n# Include only concrete models\nSyncConfig.set_config('MODEL_DISCOVERY', 'INCLUDE_MODEL_TYPES', ['concrete'])\n```\n\n##### Pattern-Based Filtering\n```python\n# Include models matching patterns\nSyncConfig.set_config('MODEL_DISCOVERY', 'INCLUDE_MODEL_PATTERNS', [\n    r'^myapp\\.',  # All models from myapp\n    r'^ecommerce\\.Product.*$'  # Product models from ecommerce\n])\n\n# Exclude models matching patterns\nSyncConfig.set_config('MODEL_DISCOVERY', 'EXCLUDE_MODEL_PATTERNS', [\n    r'^.*\\.Historical.*$',  # Exclude historical models\n    r'^.*\\.Log$',           # Exclude log models\n    r'^.*\\.Cache$',         # Exclude cache models\n])\n```\n\n##### Field-Based Filtering\n```python\n# Exclude models with specific fields\nSyncConfig.set_config('MODEL_DISCOVERY', 'EXCLUDE_MODELS_WITH_FIELDS', [\n    'created_at',  # Exclude models with created_at field\n    'updated_at',  # Exclude models with updated_at field\n    'deleted_at',  # Exclude soft-delete models\n])\n\n# Require models to have specific fields\nSyncConfig.set_config('MODEL_DISCOVERY', 'REQUIRE_MODELS_WITH_FIELDS', [\n    'id'  # Only include models with 'id' field\n])\n```\n\n##### App-Specific Exclusions\n```python\n# Exclude specific models from specific apps\nSyncConfig.set_config('MODEL_DISCOVERY', 'APP_SPECIFIC_EXCLUSIONS', {\n    'auth': ['Group', 'Permission'],  # Exclude Group and Permission from auth\n    'admin': ['LogEntry'],            # Exclude LogEntry from admin\n})\n```\n\n##### Performance and Caching\n```python\n# Enable discovery caching\nSyncConfig.set_config('MODEL_DISCOVERY', 'ENABLE_DISCOVERY_CACHING', True)\nSyncConfig.set_config('MODEL_DISCOVERY', 'DISCOVERY_CACHE_TIMEOUT', 3600)  # 1 hour\n\n# Limit models per app\nSyncConfig.set_config('MODEL_DISCOVERY', 'MAX_MODELS_PER_APP', 100)\n```\n\n#### Management Command\n\nUse the management command to configure model discovery:\n\n```bash\n# List current configuration\npython manage.py configure_model_discovery --list-current\n\n# Set include apps\npython manage.py configure_model_discovery --set-include-apps myapp ecommerce\n\n# Set exclude patterns\npython manage.py configure_model_discovery --set-exclude-patterns \".*Log\" \".*Cache\"\n\n# Set field-based exclusions\npython manage.py configure_model_discovery --set-exclude-fields created_at updated_at\n\n# Test discovery with current settings\npython manage.py configure_model_discovery --test-discovery\n\n# Reset to defaults\npython manage.py configure_model_discovery --reset-to-defaults\n```\n\n### \ud83d\udd04 Auto-Migration System\n\nThe package includes an intelligent auto-migration system that automatically detects version gaps and handles schema migrations gracefully.\n\n#### Features\n\n- **Automatic Version Detection**: Detects current database schema version\n- **Schema Change Detection**: Identifies required migrations\n- **Data Preservation**: Safely migrates data between versions\n- **Graceful Error Handling**: Comprehensive error recovery\n- **Rollback Capabilities**: Support for migration rollbacks\n- **Management Commands**: Command-line migration control\n- **Startup Auto-Migration**: Automatic migration on app startup\n\n#### Migration Scenarios Handled\n\n- **v1.x \u2192 v2.x**: Organization-based to Site-based migration\n- **Fresh Installations**: New database setup\n- **Schema Updates**: Field and table structure changes\n- **Data Preservation**: Maintains existing data during migration\n- **Error Recovery**: Handles migration failures gracefully\n\n#### Management Commands\n\n```bash\n# Check migration status (dry run)\npython manage.py auto_migrate --dry-run --verbose\n\n# Force migration (even if not needed)\npython manage.py auto_migrate --force\n\n# Verbose migration with detailed output\npython manage.py auto_migrate --verbose\n```\n\n#### Automatic Startup Migration\n\nThe auto-migration system runs automatically when the Django app starts:\n\n```python\n# In your Django app's apps.py\nclass SbSyncConfig(AppConfig):\n    def ready(self):\n        # Auto-migration runs automatically\n        from .migration_utils import setup_auto_migration\n        setup_auto_migration()\n```\n\n#### Migration Detection\n\nThe system detects migration needs by checking:\n\n- Database schema version\n- Table structure differences\n- Foreign key relationships\n- Data integrity requirements\n\n#### Example Migration Flow\n\n```python\n# 1. Version Detection\ncurrent_version = detector.detect_current_version()  # Returns '1.x', '2.x', or 'fresh'\n\n# 2. Schema Change Detection\nschema_changes = detector.detect_schema_changes()  # Identifies required changes\n\n# 3. Migration Execution\nif migrator.needs_migration():\n    success = migrator.auto_migrate()  # Performs migration\n```\n\n#### Error Handling\n\nThe auto-migration system includes comprehensive error handling:\n\n- **Database Connection Issues**: Graceful handling of connection failures\n- **Schema Conflicts**: Resolution of table structure conflicts\n- **Data Integrity**: Validation of migrated data\n- **Rollback Support**: Ability to revert failed migrations\n\n#### Configuration\n\n```python\n# Disable auto-migration (if needed)\nSB_SYNC_AUTO_MIGRATION = False\n\n# Custom migration timeout\nSB_SYNC_MIGRATION_TIMEOUT = 300  # 5 minutes\n\n# Enable verbose migration logging\nSB_SYNC_MIGRATION_VERBOSE = True\n```\n\n#### Model Discovery Settings\n\n##### Basic Settings\n- `AUTO_DISCOVER_MODELS`: Enable/disable automatic model discovery\n- `INCLUDE_APPS`: List of apps whose models will be synced (empty = all apps)\n- `EXCLUDE_MODELS`: List of specific models to exclude from sync\n- `INCLUDE_CUSTOM_MODELS`: Include custom models in discovery\n- `MODEL_PREFIX`: Prefix for model names\n- `MODEL_SUFFIX`: Suffix for model names\n- `MODEL_NAMESPACE`: Namespace for model names\n\n##### Advanced Settings\n- `EXCLUDE_ABSTRACT_MODELS`: Exclude abstract models\n- `EXCLUDE_PROXY_MODELS`: Exclude proxy models\n- `EXCLUDE_HISTORICAL_MODELS`: Exclude historical models (simple_history)\n- `EXCLUDE_MANAGER_MODELS`: Exclude models with custom managers\n- `INCLUDE_MODEL_PATTERNS`: Regex patterns for models to include\n- `EXCLUDE_MODEL_PATTERNS`: Regex patterns for models to exclude\n- `EXCLUDE_MODELS_WITH_FIELDS`: Field names that will exclude models\n- `REQUIRE_MODELS_WITH_FIELDS`: Field names that models must have\n- `APP_SPECIFIC_EXCLUSIONS`: Per-app exclusion rules\n- `INCLUDE_MODEL_TYPES`: Types of models to include ('concrete', 'abstract', 'proxy')\n- `ENABLE_DISCOVERY_CACHING`: Enable discovery result caching\n- `DISCOVERY_CACHE_TIMEOUT`: Cache timeout in seconds\n- `MAX_MODELS_PER_APP`: Maximum models per app\n- `VALIDATE_MODEL_ACCESS`: Validate that models can be accessed\n- `CHECK_MODEL_PERMISSIONS`: Check if current user can access models\n- `SAFE_DISCOVERY_MODE`: Only discover models that are safe to sync\n\n### URL Configuration\n\nInclude the sync URLs in your main `urls.py`:\n\n```python\nfrom django.urls import path, include\n\nurlpatterns = [\n    # ... other URLs\n    path('api/sync/', include('sb_sync.urls')),\n]\n```\n\n## \ud83c\udfe2 Multi-Tenant Setup\n\n### 1. Create Organizations\n\n```bash\n# Create organizations\npython manage.py setup_organizations --action create_org --org-name \"Acme Corporation\" --org-slug acme-corp\npython manage.py setup_organizations --action create_org --org-name \"Global Retail\" --org-slug global-retail\npython manage.py setup_organizations --action create_org --org-name \"City University\" --org-slug city-university\n```\n\n### 2. Create Django Groups\n\n```bash\n# Create common groups for the application\npython manage.py setup_organizations --action create_groups\n```\n\n### 3. Add Users to Organizations\n\n```bash\n# Add users with specific groups\npython manage.py setup_organizations --action add_user --username john_manager --org-slug acme-corp --group-name Managers\npython manage.py setup_organizations --action add_user --username mary_sales --org-slug global-retail --group-name Sales\npython manage.py setup_organizations --action add_user --username bob_analyst --org-slug city-university --group-name Analysts\n```\n\n### 4. Setup Complete Example System\n\n```bash\n# Setup all example organizations with groups and permissions\npython manage.py setup_organizations --action setup_example\n```\n\n## \ud83d\udc65 User Groups and Permissions\n\n### Using Django Groups\n\nThe system uses Django's built-in `auth.Group` system for role-based access control. This makes it completely generic and reusable across any Django project.\n\n### Available Groups\n\nThe system comes with common groups that can be customized for any domain:\n\n1. **Administrators** - Full access to all data (create, read, update, delete)\n2. **Managers** - Can push/pull data, create, update (no delete)\n3. **Users** - Can push/pull data, create, update (no delete)\n4. **Analysts** - Can only pull data (read-only access)\n5. **Sales** - Can push/pull sales-related data\n6. **Support** - Can push/pull support-related data\n7. **Read Only** - Can only pull data, no push access\n\n### Permission Matrix\n\n| Group | Push | Pull |\n|-------|------|------|\n| Administrators | \u2705 Yes | \u2705 Yes |\n| Managers | \u2705 Yes | \u2705 Yes |\n| Users | \u2705 Yes | \u2705 Yes |\n| Analysts | \u274c No | \u2705 Yes |\n| Sales | \u2705 Yes | \u2705 Yes |\n| Support | \u2705 Yes | \u2705 Yes |\n| Read Only | \u274c No | \u2705 Yes |\n\n## \ud83c\udf10 Web-Based Configuration Interface\n\nThe SB Sync package includes a comprehensive web-based configuration interface that allows you to manage model permissions, monitor sync operations, and configure model discovery through an intuitive web interface.\n\n### \ud83d\ude80 Interface Features\n\n- **\ud83d\udcca Dashboard**: Overview statistics and quick actions\n- **\ud83d\udd10 Permission Matrix**: Visual matrix interface for managing model permissions\n- **\ud83d\udd0d Model Discovery**: Configure which models are available for sync\n- **\ud83d\udccb Sync Logs**: View operation history and performance data\n- **\ud83d\udcc8 Performance Metrics**: Interactive charts and detailed metrics\n\n### \ud83d\udee0\ufe0f Accessing the Interface\n\n1. **Navigate to the configuration dashboard**:\n   ```\n   http://your-domain/api/sync/config/\n   ```\n\n2. **Login with admin credentials** (staff members only)\n\n3. **Use the sidebar navigation** to access different sections\n\n### \ud83d\udd10 Permission Matrix\n\nThe permission matrix provides a visual interface for managing model permissions:\n\n- **Visual Matrix**: See all models vs permissions in a matrix format\n- **Checkbox Controls**: Grant or revoke permissions with simple checkboxes\n- **Organization Selection**: Switch between different organizations\n- **Real-time Updates**: Changes are saved immediately via AJAX\n- **Bulk Operations**: Select all/deselect all functionality\n\n**URL**: `/api/sync/config/permissions/`\n\n### \ud83d\udd0d Model Discovery Configuration\n\nConfigure which models are discovered and available for sync:\n\n- **Auto Discovery Settings**: Enable/disable automatic model discovery\n- **App Filtering**: Include specific apps or all apps\n- **Model Exclusion**: Exclude specific models from sync operations\n- **Live Preview**: See which models are discovered in real-time\n\n**URL**: `/api/sync/config/model-discovery/`\n\n### \ud83d\udccb Sync Logs\n\nMonitor sync operations and performance:\n\n- **Operation History**: View all sync operations with timestamps\n- **Performance Data**: See processing times and record counts\n- **Export Functionality**: Download logs as CSV\n- **Auto-refresh**: Logs update automatically every 30 seconds\n\n**URL**: `/api/sync/config/logs/`\n\n### \ud83d\udcc8 Performance Metrics\n\nTrack system performance and optimization:\n\n- **Performance Charts**: Visual charts showing processing time trends\n- **Detailed Metrics**: View batch sizes, memory usage, and query counts\n- **Performance Analysis**: Automatic suggestions for optimization\n- **Export Data**: Download performance data as CSV\n\n**URL**: `/api/sync/config/metrics/`\n\n### \ud83d\udccb Audit Trails\n\nTrack all changes to sync system with comprehensive audit trails:\n\n- **Complete History**: Track all changes to sync models (create, update, delete)\n- **User Attribution**: See who made each change and when\n- **Field-Level Changes**: View exactly what fields were modified\n- **Filtering & Search**: Filter by model type, user, date range\n- **Export Capabilities**: Download audit trail data as CSV\n- **Real-time Updates**: Auto-refreshing audit trail display\n\n**URL**: `/api/sync/config/audit-trails/`\n\n### \ud83c\udfa8 Interface Features\n\n- **Modern Design**: Bootstrap 5 with gradient backgrounds and smooth animations\n- **Responsive Design**: Works on all devices (mobile, tablet, desktop)\n- **Real-time Updates**: AJAX-powered with immediate feedback\n- **Security**: Staff-only access with CSRF protection\n- **Performance**: Cached data and optimized queries\n\n## \ud83d\udce1 API Endpoints\n\n### Authentication\n\n**POST** `/api/sync/auth/token/`\n\nGet JWT token for authentication:\n```json\n{\n    \"username\": \"your_username\",\n    \"password\": \"your_password\"\n}\n```\n\nResponse (now includes organization info):\n```json\n{\n    \"token\": \"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9...\",\n    \"user\": {\n        \"id\": 1,\n        \"username\": \"dr_smith\",\n        \"email\": \"dr.smith@citygeneral.com\"\n    },\n    \"organizations\": [\n        {\n            \"id\": 1,\n            \"name\": \"Acme Corporation\",\n            \"slug\": \"acme-corp\",\n            \"group\": \"Managers\"\n        }\n    ]\n}\n```\n\n### PUSH API\n\n**POST** `/api/sync/push/`\n\nPush data to Django models (with multi-tenant permissions):\n\n```json\n{\n    \"data\": [\n        {\n            \"_model\": \"myapp.Customer\",\n            \"name\": \"John Doe\",\n            \"age\": 45,\n            \"department\": \"SALES\",\n            \"assigned_manager_id\": 1\n        }\n    ]\n}\n```\n\nHeaders:\n```\nAuthorization: Bearer <your_jwt_token>\n```\n\nResponse:\n```json\n{\n    \"status\": \"success\",\n    \"success_count\": 1,\n    \"error_count\": 0,\n    \"processed_models\": {\n        \"myapp.Customer\": {\n            \"created\": 1,\n            \"updated\": 0\n        }\n    },\n    \"processing_time\": 0.045\n}\n```\n\n### PULL API\n\n**POST** `/api/sync/pull/`\n\nPull data from Django models (with role-based filtering):\n\n```json\n{\n    \"models\": {\n        \"myapp.Customer\": \"2024-01-14T10:00:00Z\",\n        \"myapp.Order\": \"2024-01-14T10:00:00Z\"\n    },\n    \"batch_size\": 100\n}\n```\n\nHeaders:\n```\nAuthorization: Bearer <your_jwt_token>\n```\n\nResponse:\n```json\n{\n    \"data\": [\n        {\n            \"_model\": \"myapp.Customer\",\n            \"id\": 1,\n            \"name\": \"John Doe\",\n            \"age\": 45,\n            \"department\": \"SALES\",\n            \"assigned_manager_id\": 1,\n            \"organization\": 1\n        }\n    ],\n    \"metadata\": {\n        \"myapp.Customer\": {\n            \"count\": 1,\n            \"last_sync\": \"2024-01-15T10:30:00Z\",\n            \"user_last_sync\": \"2024-01-14T10:00:00Z\"\n        }\n    },\n    \"batch_info\": {\n        \"batch_size\": 100,\n        \"total_records\": 1\n    }\n}\n```\n\n### Performance Monitoring\n\n**GET** `/api/sync/performance/`\n\nGet performance statistics:\n\n```json\n{\n    \"performance_stats\": {\n        \"total_operations\": 150,\n        \"average_processing_time\": 0.045,\n        \"cache_hit_rate\": 0.85\n    },\n    \"current_memory_usage\": 245.6,\n    \"cache_stats\": {\n        \"cache_hits\": 1200,\n        \"cache_misses\": 200\n    },\n    \"optimization_suggestions\": [\n        \"High memory usage detected. Consider reducing batch sizes.\"\n    ]\n}\n```\n\n### Health Check\n\n**GET** `/api/sync/health/`\n\nCheck system health:\n\n```json\n{\n    \"status\": \"healthy\",\n    \"timestamp\": \"2024-01-01T12:00:00Z\",\n    \"checks\": {\n        \"database\": \"healthy\",\n        \"cache\": \"healthy\",\n        \"logging\": \"healthy\"\n    }\n}\n```\n\n## \ud83d\uddc4\ufe0f Database Models\n\n### Core Models\n\n#### SyncLog\n\nTracks all synchronization operations:\n\n```python\nclass SyncLog(models.Model):\n    user = models.ForeignKey(User, on_delete=models.CASCADE, db_index=True)\n    operation = models.CharField(max_length=10, choices=[('PUSH', 'Push'), ('PULL', 'Pull')], db_index=True)\n    status = models.CharField(max_length=10, choices=[('SUCCESS', 'Success'), ('ERROR', 'Error'), ('WARNING', 'Warning')], db_index=True)\n    model_name = models.CharField(max_length=100, blank=True, db_index=True)\n    object_count = models.IntegerField(default=0)\n    error_message = models.TextField(blank=True)\n    request_data = models.JSONField(blank=True, null=True)\n    timestamp = models.DateTimeField(default=timezone.now, db_index=True)\n    processing_time = models.FloatField(default=0.0)\n```\n\n#### SyncMetadata\n\nTracks last sync timestamps for models:\n\n```python\nclass SyncMetadata(models.Model):\n    model_name = models.CharField(max_length=100, unique=True, db_index=True)\n    last_sync = models.DateTimeField(default=timezone.now, db_index=True)\n    total_synced = models.BigIntegerField(default=0)\n```\n\n#### PerformanceMetrics\n\nTracks performance metrics for optimization:\n\n```python\nclass PerformanceMetrics(models.Model):\n    operation_type = models.CharField(max_length=20, db_index=True)\n    model_name = models.CharField(max_length=100, db_index=True)\n    batch_size = models.IntegerField()\n    processing_time = models.FloatField()\n    memory_usage = models.FloatField(null=True, blank=True)\n    query_count = models.IntegerField()\n    timestamp = models.DateTimeField(default=timezone.now, db_index=True)\n```\n\n### Multi-Tenant Models\n\n#### Organization\n\nRepresents organizations, companies, or institutions:\n\n```python\nclass Organization(models.Model):\n    name = models.CharField(max_length=200, unique=True)\n    slug = models.CharField(max_length=50, unique=True, db_index=True)\n    description = models.TextField(blank=True)\n    is_active = models.BooleanField(default=True)\n    created_at = models.DateTimeField(auto_now_add=True)\n    updated_at = models.DateTimeField(auto_now=True)\n```\n\n#### UserOrganization\n\nLinks users to organizations with roles:\n\n```python\nclass UserOrganization(models.Model):\n    user = models.ForeignKey(User, on_delete=models.CASCADE, db_index=True)\n    organization = models.ForeignKey(Organization, on_delete=models.CASCADE, db_index=True)\n    role = models.CharField(max_length=50, choices=[\n        ('ADMIN', 'Administrator'),\n        ('MANAGER', 'Manager'),\n        ('USER', 'User'),\n        ('ANALYST', 'Analyst'),\n        ('SALES', 'Sales'),\n        ('SUPPORT', 'Support'),\n        ('READ_ONLY', 'Read Only'),\n    ], db_index=True)\n    is_active = models.BooleanField(default=True)\n    created_at = models.DateTimeField(auto_now_add=True)\n```\n\n#### ModelPermission\n\nDefines which models each group can access:\n\n```python\nclass ModelPermission(models.Model):\n    organization = models.ForeignKey(Organization, on_delete=models.CASCADE, db_index=True)\n    group = models.ForeignKey(Group, on_delete=models.CASCADE, db_index=True, help_text=\"Django auth group\")\n    model_name = models.CharField(max_length=100, db_index=True)\n    can_push = models.BooleanField(default=False)\n    can_pull = models.BooleanField(default=False)\n    filters = models.JSONField(blank=True, null=True)  # Custom filters for data access\n```\n\n#### UserSyncMetadata\n\nTracks last sync timestamps for models per user/organization:\n\n```python\nclass UserSyncMetadata(models.Model):\n    user = models.ForeignKey(User, on_delete=models.CASCADE, db_index=True)\n    organization = models.ForeignKey(Organization, on_delete=models.CASCADE, db_index=True)\n    model_name = models.CharField(max_length=100, db_index=True)\n    last_sync = models.DateTimeField(default=timezone.now, db_index=True)\n    total_synced = models.BigIntegerField(default=0)\n```\n\n#### DataFilter\n\nCustom data filters for role-based access:\n\n```python\nclass DataFilter(models.Model):\n    organization = models.ForeignKey(Organization, on_delete=models.CASCADE, db_index=True)\n    role = models.CharField(max_length=50, db_index=True)\n    model_name = models.CharField(max_length=100, db_index=True)\n    filter_name = models.CharField(max_length=100)\n    filter_condition = models.JSONField()  # e.g., {\"field\": \"department\", \"operator\": \"exact\", \"value\": \"CARDIOLOGY\"}\n    is_active = models.BooleanField(default=True)\n    created_at = models.DateTimeField(auto_now_add=True)\n```\n\n## \ud83d\udd04 Background Tasks\n\n### Celery Configuration\n\nAdd to your project's `__init__.py`:\n\n```python\nfrom .celery import app as celery_app\n__all__ = ('celery_app',)\n```\n\n### Available Tasks\n\n1. **cleanup_old_sync_logs**: Removes sync logs older than 90 days\n2. **generate_sync_report**: Generates daily sync statistics\n3. **optimize_database_tables**: Analyzes and optimizes database tables\n4. **bulk_sync_operation**: Processes bulk sync operations asynchronously\n5. **cache_warmup**: Warms up cache with frequently accessed data\n6. **memory_optimization**: Performs garbage collection and cache cleanup\n7. **performance_analysis**: Analyzes performance and provides recommendations\n\n### Running Celery\n\n```bash\n# Start Celery worker\ncelery -A your_project worker -l info\n\n# Start Celery beat (for scheduled tasks)\ncelery -A your_project beat -l info\n```\n\n## \ud83d\udee1\ufe0f Security\n\n- **JWT Authentication**: All API endpoints require valid JWT tokens\n- **Token Expiration**: Tokens expire after 7 days by default\n- **Multi-Tenant Isolation**: Complete data isolation between organizations\n- **Role-Based Access Control**: Granular permissions per user role\n- **Data Filtering**: Role-specific data access filters\n- **User Validation**: All operations are tied to authenticated users\n- **Input Validation**: Comprehensive data validation against Django models\n\n## \ud83d\udcca Monitoring & Logging\n\n### Log Files\n\nSync operations are logged to `logs/sb_sync.log` with daily rotation and 30-day retention.\n\n### Log Format\n\n```\n2024-01-01 12:00:00 - sb_sync - INFO - PUSH request from user john_manager in Acme Corporation: {\"data\": [...]}\n```\n\n### Health Monitoring\n\nUse the health check endpoint to monitor:\n- Database connectivity\n- Cache functionality\n- Log directory accessibility\n- Memory usage\n- Performance metrics\n\n## \u26a1 Performance Optimizations\n\n### Bulk Operations\n\nThe package includes optimized bulk operations for high-volume data processing:\n\n```python\nfrom sb_sync.optimizations import BulkOperations\n\n# Bulk create/update with batching\nresults = BulkOperations.bulk_create_or_update(\n    model_class=YourModel,\n    data_list=large_dataset,\n    batch_size=1000\n)\n```\n\n### Model Metadata Caching\n\nModel field information is cached for improved performance:\n\n```python\nfrom sb_sync.optimizations import ModelMetadataCache\n\n# Get cached model fields\nfields = ModelMetadataCache.get_model_fields('app.ModelName')\n```\n\n### Query Optimization\n\n```python\nfrom sb_sync.optimizations import QueryOptimizer\n\n# Get optimized sync logs\nlogs = QueryOptimizer.get_optimized_sync_logs(user, organization)\n\n# Count queries with decorator\n@QueryOptimizer.count_queries\ndef my_function():\n    # Your code here\n    pass\n```\n\n### Memory Optimization\n\n```python\nfrom sb_sync.optimizations import MemoryOptimizer\n\n# Monitor memory usage\nmemory_usage = MemoryOptimizer.get_memory_usage()\n\n# Memory monitoring decorator\n@MemoryOptimizer.monitor_memory\ndef my_function():\n    # Your code here\n    pass\n```\n\n### Cache Optimization\n\n```python\nfrom sb_sync.optimizations import CacheOptimizer\n\n# Get or set cache\ndata = CacheOptimizer.get_or_set_cache('key', lambda: expensive_operation())\n\n# Cache model data\nCacheOptimizer.cache_model_data('key', data, timeout=300)\n```\n\n## \ud83d\udd27 Management Commands\n\n### Model Discovery and Default Models\n\nThe system automatically discovers all Django models in your application and makes them available for push/pull operations by default. Only specific apps and models are excluded via configuration.\n\n```bash\n# Show model discovery summary\npython manage.py show_models --action summary\n\n# Show all discovered models\npython manage.py show_models --action all\n\n# Show enabled models only\npython manage.py show_models --action enabled\n\n# Show default models for push/pull operations\npython manage.py show_models --action default\n\n# Show detailed model information\npython manage.py show_models --action details --verbose\n\n# Show models from specific app\npython manage.py show_models --action enabled --app-label myapp\n```\n\n**Default Behavior:**\n- \u2705 **Auto-discovers all Django models** in your application\n- \u2705 **`INCLUDE_APPS`**: List of apps whose models will be synced (empty = all apps)\n- \u2705 **`EXCLUDE_MODELS`**: Models within those included apps that will be excluded\n- \u2705 **Makes all discovered models available** for push/pull operations\n- \u2705 **No configuration required** - works out of the box\n\n**Configuration Options:**\n- `AUTO_DISCOVER_MODELS`: Enable/disable automatic model discovery\n- `INCLUDE_APPS`: List of apps whose models will be synced (empty = all apps)\n- `EXCLUDE_MODELS`: Models within included apps that will be excluded from sync\n- `INCLUDE_CUSTOM_MODELS`: Include custom models from your apps\n\n### Cleanup Sync Logs\n\n```bash\npython manage.py cleanup_sync_logs\n```\n\n### Performance Optimization\n\n```bash\n# Analyze performance\npython manage.py optimize_performance --action analyze --days 7\n\n# Optimize performance\npython manage.py optimize_performance --action optimize --force\n\n# Cleanup old data\npython manage.py optimize_performance --action cleanup\n\n# Monitor resources\npython manage.py optimize_performance --action monitor\n```\n\n### Configuration Management\n\n```bash\n# Show current configuration\npython manage.py manage_config --action show\n\n# Export configuration\npython manage.py manage_config --action export --file config.json --format json\n\n# Import configuration\npython manage.py manage_config --action import --file config.json --format json\n\n# Validate configuration\npython manage.py manage_config --action validate\n\n# Get configuration summary\npython manage.py manage_config --action summary\n\n# Reset to defaults\npython manage.py manage_config --action reset --force\n```\n\n### Organization Setup\n\n```bash\n# Create organization\npython manage.py setup_organizations --action create_org --org-name \"Acme Corporation\" --org-slug acme-corp\n\n# Add user to organization\npython manage.py setup_organizations --action add_user --username john_manager --org-slug acme-corp --group-name Managers\n\n# Set permissions from config file\npython manage.py setup_organizations --action set_permissions --org-slug acme-corp --config-file permissions.json\n\n# Setup complete example system\npython manage.py setup_organizations --action setup_example\n```\n\n### Audit Trails Management\n\n```bash\n# Setup audit trails for all sync models\npython manage.py setup_audit_trails --action setup\n\n# Check audit trails status\npython manage.py setup_audit_trails --action check\n\n# Cleanup old audit trail records\npython manage.py setup_audit_trails --action cleanup\n\n# Setup audit trails for specific model\npython manage.py setup_audit_trails --action setup --model sb_sync.Organization\n\n# Check specific model audit trails\npython manage.py setup_audit_trails --action check --model sb_sync.ModelPermission\n```\n\n### Dynamic Permission Configuration\n\n```bash\n# Discover all models in your project\npython manage.py dynamic_permissions --action discover\n\n# Generate permission configuration\npython manage.py dynamic_permissions --action generate --org-slug acme-corp --permission-template read_write\n\n# Apply permission configuration\npython manage.py dynamic_permissions --action apply --org-slug acme-corp --config-file permissions.json\n\n# Export current permissions\npython manage.py dynamic_permissions --action export --org-slug acme-corp --output-file current_permissions.json\n\n# Validate configuration file\npython manage.py dynamic_permissions --action validate --config-file permissions.json\n\n# Show available templates\npython manage.py dynamic_permissions --action template\n```\n\n## \ud83e\uddea Testing\n\n### Example Usage\n\n```python\nimport requests\nimport json\n\n# Get authentication token\nauth_response = requests.post('http://localhost:8000/api/sync/auth/token/', {\n    'username': 'dr_smith',\n    'password': 'password'\n})\ntoken = auth_response.json()['token']\n\n# Push data (with multi-tenant permissions)\nheaders = {'Authorization': f'Bearer {token}'}\npush_data = {\n    'data': [\n        {\n            '_model': 'myapp.Customer',\n            'name': 'John Doe',\n            'department': 'SALES',\n            'assigned_manager_id': 1\n        }\n    ]\n}\nresponse = requests.post('http://localhost:8000/api/sync/push/', \n                        json=push_data, headers=headers)\nprint(response.json())\n\n# Pull data (with role-based filtering)\npull_data = {\n    'models': {\n        'myapp.Customer': '2024-01-14T10:00:00Z',\n        'myapp.Order': '2024-01-14T10:00:00Z'\n    },\n    'batch_size': 100\n}\nresponse = requests.post('http://localhost:8000/api/sync/pull/', \n                        json=pull_data, headers=headers)\nprint(response.json())\n```\n\n## \ud83d\udcdd Error Handling\n\nThe package provides comprehensive error handling:\n\n- **Validation Errors**: Detailed field validation messages\n- **Model Errors**: Clear error messages for missing or invalid models\n- **Authentication Errors**: Proper JWT token validation\n- **Permission Errors**: Multi-tenant and role-based access control errors\n- **Database Errors**: Transaction rollback on errors\n- **Partial Success Handling**: Graceful handling of partial failures\n\n## \ud83c\udfe2 Multi-Tenant Use Case\n\n### Scenario: Multiple Organizations\n\nThe system supports complex multi-tenant scenarios with multiple organizations:\n\n```python\n# Organization A (Acme Corp) - Manager Smith\nPOST /api/sync/push/\n{\n    \"data\": [\n        {\n            \"_model\": \"myapp.Customer\",\n            \"name\": \"John Doe\",\n            \"department\": \"SALES\",\n            \"assigned_manager_id\": 1\n        }\n    ]\n}\n# \u2705 Success - Manager Smith has permission\n\n# Organization B (Global Retail) - Sales Wilson  \nPOST /api/sync/pull/\n{\n    \"models\": {\"myapp.Customer\": \"2024-01-14T10:00:00Z\"}\n}\n# \u2705 Success - Sales Wilson gets only their assigned customers\n\n# Organization C (City University) - Analyst Garcia\nPOST /api/sync/push/\n{\n    \"data\": [\n        {\n            \"_model\": \"myapp.Report\",\n            \"customer_id\": 5,\n            \"report_type\": \"ANALYSIS\",\n            \"results\": \"Positive\"\n        }\n    ]\n}\n# \u2705 Success - Analyst has permission for reports\n```\n\n### Key Benefits for Multi-Tenant Applications:\n\n1. **\ud83d\udd12 Data Isolation**: Each organization only sees their own data\n2. **\ud83d\udc65 Role-Based Access**: Different roles have appropriate permissions\n3. **\ud83d\udcca Granular Control**: Filter by department, assigned staff, etc.\n4. **\ud83d\udd04 Per-User Sync Tracking**: Each user has their own sync history\n5. **\u26a1 Performance**: Optimized for high-volume data processing\n6. **\ud83d\udee1\ufe0f Security**: Meets enterprise data privacy requirements\n\n## \ud83e\udd1d Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests\n5. Submit a pull request\n\n## \ud83d\udcc4 License\n\nThis project is licensed under the MIT License.\n\n## \ud83c\udd98 Support\n\nFor issues and questions:\n- Check the health endpoint: `/api/sync/health/`\n- Review sync logs in `logs/sb_sync.log`\n- Monitor Celery task logs for background operations\n- Check performance metrics: `/api/sync/performance/`\n\n## \ufffd\ufffd Changelog\n\n### **v2.0.0** (Latest) - 2025-08-07\n**\ud83d\ude80 Major Release: Django Sites Integration & UI Improvements**\n- **\ud83d\udd04 Architecture Change**: Replaced custom Organization model with Django's built-in Sites framework\n- **\ud83c\udfa8 UI Enhancement**: Updated all interfaces to use user-friendly \"Organization\" terminology\n- **\ud83d\udd27 Technical Improvements**: \n  - Renamed `UserOrganization` to `UserSite` for better Django Sites integration\n  - Updated all models to use `site` instead of `organization` fields\n  - Simplified permission system with push/pull permissions only\n  - Enhanced admin interface with proper field mappings\n  - Fixed all indentation and configuration errors\n- **\ud83d\udcf1 User Experience**: \n  - \"Organization Selector\" instead of \"Site Selector\" in permission matrix\n  - \"Organizations Overview\" in configuration dashboard\n  - Professional admin interface with organization actions\n  - Consistent terminology across all interfaces\n- **\u26a1 Performance**: \n  - Optimized database queries with proper select_related\n  - Enhanced caching mechanisms\n  - Improved bulk operations for permissions\n- **\ud83d\udd12 Security**: \n  - Maintained all security features with Django Sites\n  - Enhanced permission checking with proper site isolation\n- **\ud83d\udcca Monitoring**: \n  - Added comprehensive audit trails\n  - Enhanced logging and error handling\n  - Improved performance metrics tracking\n\n### **v1.9.2** - 2025-08-06\n**\ud83d\udd27 Permission System Fixes**\n- **Fixed**: Pull API permission issues causing \"you don't have permission\" errors\n- **Fixed**: Removed invalid `is_active` filter from ModelPermission queries\n- **Added**: Proper superuser handling in permission system\n- **Enhanced**: Group resolution for admin users and superusers\n- **Added**: Comprehensive debugging logs for permission troubleshooting\n- **Improved**: Error handling and logging for permission checks\n- **Fixed**: Admin users with proper group assignments now have correct pull/push permissions\n\n### **v1.9.1** - 2025-08-06\n**\ud83c\udfa8 Improved Permission Matrix UI**\n- **Enhanced**: Split push and pull permissions into separate columns\n- **Added**: Individual \"select all\" checkboxes for push and pull permissions per group\n- **Improved**: Better visual organization with dedicated columns for each permission type\n- **Enhanced**: More intuitive interface for permission management\n- **Updated**: JavaScript logic to handle separate column states independently\n- **Improved**: User experience with clearer permission type separation\n- **Maintained**: All existing functionality (auto-save, debouncing, bulk operations)\n\n### **v1.9.0** - 2025-08-06\n**\ud83d\udd27 Simplified Permission System**\n- **Removed**: Delete, read, create, and update permissions from the system\n- **Simplified**: Permission system now only includes push and pull permissions\n- **Updated**: ModelPermission model to only have can_push and can_pull fields\n- **Updated**: Permission matrix interface to show only push/pull permissions\n- **Updated**: Admin interface to reflect simplified permission structure\n- **Updated**: All views and templates to work with simplified permissions\n- **Added**: Database migration to remove old permission fields\n- **Improved**: Cleaner and more focused permission management\n\n### **v1.8.0** - 2025-08-06\n**\ud83d\udd27 Template Tag Fix & Version Display**\n- **Fixed**: Template tag loading error for get_version in base.html\n- **Added**: {% load sb_sync_extras %} to properly load custom template tags\n- **Fixed**: Version display in configuration interface sidebar\n- **Improved**: Template error handling and tag registration\n- **Tested**: Template rendering without errors\n\n### **v1.7.0** - 2025-08-06\n**\ud83d\ude80 Persistent Configuration Storage**\n- **Added**: SyncConfiguration model for persistent database storage\n- **Implemented**: Database-backed configuration with JSONField support\n- **Added**: History tracking for configuration changes with Django Simple History\n- **Enhanced**: get_config() and set_config() methods with database fallback\n- **Fixed**: Configuration persistence across server restarts\n- **Added**: Graceful fallback to in-memory storage if database unavailable\n- **Improved**: Model discovery with persistent INCLUDE_APPS configuration\n- **Added**: Proper indexing and constraints for configuration table\n- **Tested**: Configuration persistence and model discovery functionality\n\n### **v1.6.1** - 2025-08-06\n**\ud83d\udd27 Template Tag Fix & Bug Resolution**\n- **Fixed**: AttributeError in permission matrix template ('bool' object has no attribute 'get')\n- **Enhanced**: lookup filter in sb_sync_extras.py to handle multiple data types\n- **Added**: Support for boolean values, dictionary-like objects, lists, and object attributes\n- **Improved**: Error handling for template tag operations\n- **Tested**: Permission matrix endpoint accessibility and functionality\n\n### **v1.6.0** - 2025-08-06\n**\ud83d\udd27 Model Discovery Logic Enhancement**\n- **Fixed**: Model discovery logic to exclude Django built-in apps when INCLUDE_APPS is empty\n- **Fixed**: Excluded sb-sync app itself and its dependencies from model discovery\n- **Enhanced**: get_all_models() and is_model_enabled() methods with proper exclusion logic\n- **Added**: Comprehensive list of excluded apps (Django built-ins, sb-sync, dependencies)\n- **Fixed**: Missing dependencies (psutil, django-simple-history) installation issues\n- **Updated**: URL configuration to properly expose model discovery endpoint\n- **Tested**: Model discovery logic with custom test apps\n- **Improved**: Server startup reliability and dependency management\n\n### **v1.5.3** - 2025-08-06\n**\ud83d\udd27 Package Installation Fixes**\n- **Fixed**: Resolved pip installation issues with proper package structure\n- **Fixed**: Moved all files to `sb_sync/` directory for correct Python package layout\n- **Fixed**: Cleaned up excessive dependencies in `setup.py`\n- **Fixed**: Removed invalid entry points that caused installation errors\n- **Added**: `MANIFEST.in` file for proper package data inclusion\n- **Updated**: Project URLs to point to correct GitHub repository\n- **Tested**: Package installation and import functionality\n\n### **v1.5.2** - 2025-08-06\n**\ud83d\udd27 Django 5.2.x Compatibility & Installation Fixes**\n- **Added**: Support for Django 5.2.x\n- **Updated**: Django version constraint from `<5.1` to `<5.3`\n- **Added**: Django 5.1 and 5.2 framework classifiers\n- **Fixed**: Package structure for proper pip installation\n- **Added**: Comprehensive Django compatibility documentation\n\n### **v1.5.1** - 2025-08-06\n**\ud83d\udd27 Django Compatibility Enhancement**\n- **Added**: Support for Django 5.1.x and 5.2.x\n- **Updated**: Django version requirements to support broader range\n- **Added**: Django compatibility table in documentation\n- **Fixed**: Version constraints in setup.py\n\n### **v1.5.0** - 2024-01-XX\n**\ud83c\udf10 Web-Based Configuration Interface & Audit Trails**\n- **Added**: Complete web-based configuration interface with dashboard\n- **Added**: Visual permission matrix for model permissions management\n- **Added**: Model discovery configuration UI\n- **Added**: Sync logs viewer with real-time updates\n- **Added**: Performance metrics visualization with charts\n- **Added**: Django Simple History integration for comprehensive audit trails\n- **Added**: Audit trails management command (`setup_audit_trails`)\n- **Added**: Custom template tags for enhanced UI functionality\n- **Enhanced**: Admin interface with historical records support\n- **Updated**: All sync models with audit trail capabilities\n- **Added**: Bootstrap 5, Font Awesome, and Chart.js for modern UI\n\n### **v1.4.0** - 2024-01-XX\n**\ud83e\uddf9 Package Streamlining & Code Organization**\n- **Removed**: Dynamic data sources functionality for focused scope\n- **Improved**: Code organization and structure\n- **Enhanced**: Documentation clarity and completeness\n- **Streamlined**: Package for Django model synchronization focus\n- **Cleaned**: Codebase organization and removed unused components\n\n### **v1.3.0** - 2024-01-XX\n**\ud83d\udd0d Model Discovery & Configuration Enhancement**\n- **Added**: Include-based model discovery system\n- **Enhanced**: Configuration system with better flexibility\n- **Improved**: Test coverage across all components\n- **Updated**: Comprehensive documentation cleanup\n- **Added**: Model introspection capabilities\n- **Enhanced**: Error handling and validation\n\n### **v1.2.0** - 2024-01-XX\n**\ud83d\ude80 Performance & Multi-Tenant Features**\n- **Added**: Multi-tenant support with organization-based isolation\n- **Added**: Role-based access control (RBAC)\n- **Enhanced**: Performance optimizations with bulk operations\n- **Added**: Comprehensive health monitoring\n- **Improved**: Background task processing with Celery\n- **Added**: Advanced caching mechanisms\n\n### **v1.1.0** - 2024-01-XX\n**\u26a1 Performance & Error Handling**\n- **Enhanced**: Configuration system flexibility\n- **Added**: Performance optimizations for large datasets\n- **Improved**: Error handling and reporting mechanisms\n- **Added**: Advanced logging capabilities\n- **Enhanced**: Data validation processes\n\n### **v1.0.0** - 2024-01-XX\n**\ud83c\udf89 Initial Release**\n- **Added**: PUSH/PULL API endpoints for bidirectional sync\n- **Added**: JWT authentication system\n- **Added**: Comprehensive logging infrastructure\n- **Added**: Basic model synchronization capabilities\n- **Added**: Health check endpoints\n- **Added**: Initial documentation and setup guides\n\n---\n\n## \ud83d\udd04 Migration Guide\n\n### **Upgrading to v1.5.x**\n\nIf you're upgrading from an earlier version:\n\n1. **Update your installation**:\n   ```bash\n   pip install --upgrade sb-sync\n   ```\n\n2. **Run new migrations**:\n   ```bash\n   python manage.py migrate\n   ```\n\n3. **Setup audit trails** (recommended):\n   ```bash\n   python manage.py setup_audit_trails --action setup\n   ```\n\n4. **Access new web interface**:\n   - Navigate to: `http://your-domain/api/sync/config/`\n   - Login with admin/staff credentials\n\n### **Breaking Changes**\n\n- **v1.4.0**: Removed dynamic data sources - migrate to model-based sync\n- **v1.5.0**: Added new dependencies (`django-simple-history`) - run `pip install --upgrade sb-sync`\n\n### **New Features Guide**\n\n- **Web Interface**: Access at `/api/sync/config/` for visual management\n- **Audit Trails**: Use management command to setup and manage historical records\n- **Django 5.2.x**: Full compatibility with latest Django versions \n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Django package for data synchronization with PUSH/PULL APIs",
    "version": "2.0.2",
    "project_urls": {
        "Bug Tracker": "https://github.com/TheSocialBytes/sync/issues",
        "Changelog": "https://github.com/TheSocialBytes/sync/blob/main/README.md#-changelog",
        "Documentation": "https://github.com/TheSocialBytes/sync/blob/main/README.md",
        "Homepage": "https://github.com/TheSocialBytes/sync",
        "Repository": "https://github.com/TheSocialBytes/sync"
    },
    "split_keywords": [
        "django",
        " synchronization",
        " api",
        " multi-tenant",
        " audit-trails"
    ],
    "urls": [
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "988102c6781cd1ca2941b27c8b99c82414314cdae87e75c56f447f6d1cfa3726",
                "md5": "4c297ac68c1fde59b8d587d9128df7ed",
                "sha256": "e6eda88ce53da239a82beb4b46e86f2d7e8f3ce45aba6d3397db19ba25b1dda8"
            },
            "downloads": -1,
            "filename": "sb_sync-2.0.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "4c297ac68c1fde59b8d587d9128df7ed",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8",
            "size": 110076,
            "upload_time": "2025-08-08T06:38:20",
            "upload_time_iso_8601": "2025-08-08T06:38:20.663344Z",
            "url": "https://files.pythonhosted.org/packages/98/81/02c6781cd1ca2941b27c8b99c82414314cdae87e75c56f447f6d1cfa3726/sb_sync-2.0.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": null,
            "digests": {
                "blake2b_256": "25f9bca8b99c738be2ba5e5a8eb16f5d8d7612864d2e9b1ccaa47dc29459b25d",
                "md5": "81980b6a8fddaf3deffe6727ef9cb96f",
                "sha256": "a9004dad4801cd19a5468bdd1c8df38e60fc6e352be03fc9b80d8ffc16ea7e52"
            },
            "downloads": -1,
            "filename": "sb_sync-2.0.2.tar.gz",
            "has_sig": false,
            "md5_digest": "81980b6a8fddaf3deffe6727ef9cb96f",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8",
            "size": 120801,
            "upload_time": "2025-08-08T06:38:22",
            "upload_time_iso_8601": "2025-08-08T06:38:22.387115Z",
            "url": "https://files.pythonhosted.org/packages/25/f9/bca8b99c738be2ba5e5a8eb16f5d8d7612864d2e9b1ccaa47dc29459b25d/sb_sync-2.0.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-08-08 06:38:22",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "TheSocialBytes",
    "github_project": "sync",
    "github_not_found": true,
    "lcname": "sb-sync"
}