# CloudBurst Fargate - Serverless Video Processing
[](https://pypi.org/project/cloudburst-fargate/)
[](https://www.python.org/downloads/)
[](https://opensource.org/licenses/MIT)
[](https://aws.amazon.com/fargate/)
My second open source project, now powered by **AWS ECS Fargate**! π
**Author**: Leo Wang ([leowang.net](https://leowang.net))
**Email**: me@leowang.net
**License**: MIT
> **π Related Projects**:
> - **Original CloudBurst (EC2)**: https://github.com/preangelleo/cloudburst
> - **Video Generation API**: https://github.com/preangelleo/video-generation-docker
> - **δΈζζζ‘£**: [README_CN.md](./README_CN.md)
## What is this?
A production-ready Python framework that uses **AWS ECS Fargate** for serverless, on-demand video generation with **parallel processing capabilities**.
**Core Value**: When your application needs to generate videos (using our [Video Generation API](https://github.com/preangelleo/video-generation-docker)), this framework:
- π Starts Fargate containers in **30 seconds** (vs 2+ minutes for EC2)
- β‘ **Parallel processing**: Handle multiple scenes across concurrent containers
- π¬ Processes your video generation requests with zero infrastructure management
- π₯ Downloads completed videos automatically with "process one β download one" efficiency
- π Containers terminate automatically after processing
- π° Pay-per-second billing with **no idle costs**
**Perfect for**: Production applications that need scalable serverless video processing without the complexity of managing EC2 instances.
## π¦ Installation
### Install from PyPI
```bash
pip install cloudburst-fargate
```
### Install from GitHub
```bash
pip install git+https://github.com/preangelleo/cloudburst-fargate.git
```
### Install from Source
```bash
git clone https://github.com/preangelleo/cloudburst-fargate.git
cd cloudburst-fargate
pip install -e .
```
## π CloudBurst Evolution: EC2 β Fargate
| Feature | CloudBurst EC2 (v1) | **CloudBurst Fargate (v2)** |
|---------|---------------------|------------------------------|
| **Startup Time** | ~75 seconds | **~30 seconds** β‘ |
| **Infrastructure** | Manage EC2 instances | **Fully serverless** π― |
| **Parallel Processing** | Single instance only | **Multiple concurrent tasks** π |
| **Availability** | Subject to quota limits | **Near 100% availability** β
|
| **Scaling** | Limited by EC2 capacity | **Unlimited concurrent tasks** π |
| **Cost Model** | Per-minute billing | **Per-second billing** π° |
| **Idle Costs** | Risk of forgotten instances | **Zero idle costs** π₯ |
## π Quick Start
### 1. Install Package
```bash
pip install cloudburst-fargate
```
### 2. Setup AWS Permissions (CRITICAL)
CloudBurst Fargate requires specific IAM permissions to manage ECS tasks, access VPC resources, and handle container operations. You'll need to add permissions **4 times** during setup:
#### Required IAM Permissions
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecs:RunTask",
"ecs:StopTask",
"ecs:DescribeTasks",
"ecs:DescribeClusters",
"ecs:ListTasks",
"ecs:ListTagsForResource",
"ecs:TagResource"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:DescribeVpcs"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams"
],
"Resource": "arn:aws:logs:*:*:log-group:/ecs/cloudburst*"
},
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": "arn:aws:iam::*:role/ecsTaskExecutionRole"
}
]
}
```
#### Step-by-Step Permission Setup
1. **First Permission**: ECS Task Management
```bash
# Add ECS permissions for running and managing Fargate tasks
aws iam attach-user-policy --user-name YOUR_USER --policy-arn arn:aws:iam::aws:policy/AmazonECS_FullAccess
```
2. **Second Permission**: VPC and Network Access
```bash
# Add EC2 permissions for VPC, subnets, and security groups
aws iam attach-user-policy --user-name YOUR_USER --policy-arn arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess
```
3. **Third Permission**: CloudWatch Logs
```bash
# Add CloudWatch permissions for container logging
aws iam attach-user-policy --user-name YOUR_USER --policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess
```
4. **Fourth Permission**: IAM Role Passing
```bash
# Add permission to pass execution roles to ECS tasks
aws iam put-user-policy --user-name YOUR_USER --policy-name ECSTaskRolePass --policy-document file://pass-role-policy.json
```
### 3. Setup Environment
```bash
# Copy and customize configuration
cp .env.example .env
# Edit .env with your AWS credentials and VPC settings:
# - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (with permissions above)
# - AWS_SUBNET_ID (your VPC subnet with internet access)
# - AWS_SECURITY_GROUP_ID (allows port 5000 and outbound HTTPS)
```
### 4. Test Your Setup
```python
from cloudburst_fargate import FargateOperationV1
# Quick single-scene test
processor = FargateOperationV1(config_priority=1)
scenes = [{
"scene_name": "test_scene",
"image_path": "path/to/image.png",
"audio_path": "path/to/audio.mp3",
"subtitle_path": "path/to/subtitle.srt" # Optional
}]
result = processor.execute_batch(scenes, language="english", enable_zoom=True)
print(f"β
Generated {result['successful_scenes']} videos")
```
### 5. Parallel Processing (Production Ready!)
```python
from cloudburst_fargate.fargate_operation import execute_parallel_batches
# Process multiple scenes across parallel Fargate containers
scenes = [
{"scene_name": "scene_001", "image_path": "...", "audio_path": "..."},
{"scene_name": "scene_002", "image_path": "...", "audio_path": "..."},
{"scene_name": "scene_003", "image_path": "...", "audio_path": "..."},
{"scene_name": "scene_004", "image_path": "...", "audio_path": "..."}
]
# Automatically distribute across 2 parallel tasks (2 scenes each)
result = execute_parallel_batches(
scenes=scenes,
scenes_per_batch=2, # 2 scenes per Fargate container
max_parallel_tasks=2, # 2 concurrent containers
language="english",
enable_zoom=True,
config_priority=1, # CPU configuration (1-5, default: 4)
watermark_path=None, # Optional watermark image
is_portrait=False, # Portrait mode (default: False)
saving_dir="./output", # Output directory
background_box=True, # Subtitle background (default: True)
background_opacity=0.2 # Background transparency 0-1 (default: 0.2)
)
print(f"π Efficiency: {result['efficiency']['speedup_factor']:.2f}x speedup")
print(f"π° Total cost: ${result['total_cost_usd']:.4f}")
print(f"π {len(result['downloaded_files'])} videos downloaded")
```
## β‘ Parallel Processing Architecture
CloudBurst Fargate v2 introduces **true parallel processing**:
### Architecture Benefits
- **Concurrent Tasks**: Multiple Fargate containers running simultaneously
- **Intelligent Distribution**: Scenes automatically distributed across tasks
- **Efficient Workflow**: Each task processes scenes β downloads β terminates
- **Cost Optimized**: Pay only for actual processing time across all containers
### Example: 4 Scenes, 2 Tasks
```
Task 1: Start β Process scene_001 β Download β Process scene_002 β Download β Terminate
Task 2: Start β Process scene_003 β Download β Process scene_004 β Download β Terminate
Result: 1.8x speedup, all videos downloaded automatically
```
## π Fargate Configuration Options
Choose the right performance level for your workload:
```python
# Economy: 1 vCPU, 2GB RAM (~$0.044/hour) - Light workloads
processor = FargateOperationV1(config_priority=5)
# Standard: 2 vCPU, 4GB RAM (~$0.088/hour) - Most common choice
processor = FargateOperationV1(config_priority=1) # Default
# High Performance: 4 vCPU, 8GB RAM (~$0.175/hour) - Heavy scenes
processor = FargateOperationV1(config_priority=2)
# Ultra Performance: 8 vCPU, 16GB RAM (~$0.351/hour) - Maximum speed
processor = FargateOperationV1(config_priority=3)
# Maximum Performance: 16 vCPU, 32GB RAM (~$0.702/hour) - Enterprise
processor = FargateOperationV1(config_priority=4)
```
## π¬ Complete Example (Production Ready)
See [`example_usage.py`](./example_usage.py) for comprehensive examples including:
- All CPU configuration options
- Complete API parameter reference
- Single scene processing
- Batch processing examples
- Parallel processing configurations
- Cost optimization strategies
```python
# Quick parallel processing example
from cloudburst_fargate import FargateOperationV1
from cloudburst_fargate.fargate_operation import execute_parallel_batches
result = execute_parallel_batches(
scenes=your_scenes,
scenes_per_batch=3, # Scenes per container
max_parallel_tasks=4, # Concurrent containers
language="chinese", # or "english"
enable_zoom=True, # Add zoom effects
config_priority=2, # High performance config (1-5)
min_scenes_per_batch=5, # Min scenes to justify startup (default: 5)
watermark_path=None, # Optional watermark
is_portrait=False, # Portrait video mode
saving_dir="./videos", # Output directory
background_box=True, # Show subtitle background
background_opacity=0.2 # Subtitle transparency
)
# Automatic results:
# β
All videos processed and downloaded
# π° Optimal cost distribution across parallel tasks
# π Detailed efficiency and timing metrics
```
## π‘ Key Advantages
### 1. **True Serverless with Parallel Scale**
- **Per-second billing** from container start to finish
- **Multiple concurrent containers** for faster processing
- No risk of forgotten running instances
- Automatic cleanup guaranteed
### 2. **Zero Infrastructure Management**
- No EC2 instances to monitor
- No SSH keys or security patches
- AWS handles all infrastructure and scaling
### 3. **Production Performance**
- **30-second startup** vs 75+ seconds for EC2
- **Parallel processing** across multiple containers
- Intelligent scene distribution and load balancing
- Consistent performance (no "noisy neighbor" issues)
### 4. **Enterprise Ready**
- Built-in high availability and auto-retry
- Integrated with AWS CloudWatch logging
- VPC networking support
- Cost tracking and optimization
## π° Cost Comparison
**Example: Processing 8 video scenes**
| Approach | Configuration | Time | Cost | Efficiency |
|----------|---------------|------|------|------------|
| **Sequential (Single Task)** | 2 vCPU | 16 min | $0.024 | 1.0x |
| **π Parallel (4 Tasks Γ 2 Scenes)** | 2 vCPU each | 9 min | $0.026 | **1.8x faster** |
| **24/7 GPU Server** | Always on | - | ~$500/month | - |
**Key Insight**: Minimal cost increase (8%) for 80% time reduction!
## π§ Advanced Features
### Intelligent Scene Distribution
The framework automatically:
- Distributes scenes evenly across parallel tasks
- Handles remainder scenes when batch sizes don't divide evenly
- Optimizes for cost vs speed based on your configuration
### Real-time Monitoring
```python
# Built-in cost tracking and performance metrics
result = execute_parallel_batches(scenes=scenes, ...)
print(f"Tasks used: {result['tasks_used']}")
print(f"Processing efficiency: {result['efficiency']['processing_efficiency']:.1f}%")
print(f"Speedup factor: {result['efficiency']['speedup_factor']:.2f}x")
print(f"Cost per scene: ${result['total_cost_usd']/len(scenes):.4f}")
```
### Flexible Configuration
```python
# Environment variables or .env file
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_SUBNET_ID=subnet-xxxxxxxxx
AWS_SECURITY_GROUP_ID=sg-xxxxxxxxx
ECS_CLUSTER_NAME=cloudburst-cluster
ECS_TASK_DEFINITION=cloudburst-task
```
## π οΈ File Structure
After cleanup, the project structure is:
```
cloudburst_fargate/
βββ fargate_operation_v1.py # Core Fargate operations and parallel processing
βββ example_usage.py # Complete usage examples and API reference
βββ README.md # This file
βββ README_CN.md # Chinese documentation
βββ requirements.txt # Python dependencies
βββ .env.example # Environment template
βββ Docs/ # Technical documentation
βββ backup_test_files/ # Test files (Git ignored)
```
## π§ Troubleshooting
### Common Issues
**Task fails to start:**
- Check subnet and security group IDs in .env
- Ensure subnet has internet access (public subnet or NAT gateway)
- Verify AWS credentials with correct permissions
**Network errors:**
- Security group must allow outbound HTTPS (port 443) for Docker pulls
- Security group must allow inbound TCP port 5000 for API access
**Permission errors:**
- Verify AWS credentials: `aws sts get-caller-identity`
- Required IAM permissions: ECS, ECR, CloudWatch, EC2 (for VPC)
### Debug Mode
```python
# Enable detailed AWS logging
import logging
logging.basicConfig(level=logging.DEBUG)
# Or check CloudWatch logs: /ecs/cloudburst
```
### Task Monitoring and Management (New in v2)
CloudBurst Fargate now includes advanced task monitoring and cleanup capabilities to ensure reliable production operations:
#### List Running Tasks
```python
from fargate_operation_v1 import FargateOperationV1
# Initialize the operation
fargate_op = FargateOperationV1()
# List all running Fargate tasks created by animagent
running_tasks = fargate_op.list_running_tasks(filter_animagent_only=True)
for task in running_tasks:
print(f"Task: {task['task_arn']}")
print(f"Status: {task['status']}")
print(f"Started: {task['started_at']}")
print(f"Public IP: {task['public_ip']}")
print(f"Tags: {task['tags']}")
```
#### Cleanup Stale Tasks
```python
# Cleanup all animagent-created tasks (double security mechanism)
cleanup_result = fargate_op.cleanup_all_tasks(
reason="Scheduled cleanup",
filter_animagent_only=True # Only cleanup tasks tagged with CreatedBy=animagent
)
print(f"Cleanup result: {cleanup_result['message']}")
print(f"Tasks terminated: {cleanup_result['terminated_count']}")
print(f"Failed cleanups: {cleanup_result['failed_count']}")
```
#### Task Identification
All tasks created by CloudBurst Fargate are automatically tagged for easy identification:
- `CreatedBy`: `animagent` - Identifies tasks created by this framework
- `Purpose`: `video-generation` - Marks the task purpose
- `Scene`: Scene name being processed
- `Language`: Processing language (english/chinese)
This tagging system ensures that cleanup operations only affect tasks created by your application, preventing interference with other services using the same ECS cluster.
## π API Reference: execute_parallel_batches()
### Complete Parameter List
```python
execute_parallel_batches(
scenes: List[Dict], # Required: List of scene dictionaries
scenes_per_batch: int = 10, # Scenes per Fargate container
max_parallel_tasks: int = 10, # Maximum concurrent containers
language: str = "chinese", # Language: "chinese" or "english"
enable_zoom: bool = True, # Enable zoom in/out effects
config_priority: int = 4, # CPU config (1-5, see table below)
min_scenes_per_batch: int = 5, # Minimum scenes to justify container startup
watermark_path: str = None, # Optional watermark image path
is_portrait: bool = False, # Portrait video mode
saving_dir: str = None, # Output directory (default: ./cloudburst_fargate_results/)
background_box: bool = True, # Show subtitle background
background_opacity: float = 0.2 # Background transparency (0=opaque, 1=transparent)
) -> Dict
```
### Scene Dictionary Format
Each scene in the `scenes` list must contain:
```python
{
"scene_name": "unique_name", # Required: Unique identifier for the scene
"image_path": "path/to/image", # Required: Path to image file
"audio_path": "path/to/audio", # Required: Path to audio file
"subtitle_path": "path/to/srt" # Optional: Path to subtitle file
}
```
### CPU Configuration Priority
| Priority | vCPU | Memory | Name | Cost/Hour | Best For |
|----------|------|--------|------|-----------|----------|
| 1 | 2 | 4GB | STANDARD | $0.088 | Most tasks |
| 2 | 4 | 8GB | HIGH_PERFORMANCE | $0.175 | Faster processing |
| 3 | 8 | 16GB | ULTRA_PERFORMANCE | $0.351 | Very fast |
| 4 | 16 | 32GB | MAXIMUM_PERFORMANCE | $0.702 | Fastest (default) |
| 5 | 1 | 2GB | ECONOMY | $0.044 | Cost-sensitive |
### Return Value Structure
```python
{
"success": bool, # Overall success status
"total_scenes": int, # Total number of input scenes
"successful_scenes": int, # Successfully processed scenes
"failed_scenes": int, # Failed scenes count
"total_cost_usd": float, # Total cost in USD
"total_duration": float, # Total time in seconds
"downloaded_files": List[str], # Paths to downloaded videos
"task_results": List[Dict], # Individual task results
"tasks_used": int, # Number of Fargate tasks used
"efficiency": {
"speedup_factor": float, # Speedup vs sequential processing
"processing_efficiency": float, # Percentage of time spent processing
"cost_per_scene": float # Average cost per scene
}
}
```
### Smart Distribution Examples
```python
# Example 1: Even distribution
# 50 scenes, batch=10, max_tasks=10 β 5 tasks Γ 10 scenes each
# Example 2: Redistribution for efficiency
# 120 scenes, batch=10, max_tasks=10 β 10 tasks Γ 12 scenes each
# Example 3: Handling remainders
# 101 scenes, batch=10, max_tasks=10 β 9 tasks Γ 10 scenes + 1 task Γ 11 scenes
```
## π― Roadmap
- [x] **β
Parallel Processing**: Multiple concurrent Fargate tasks
- [ ] **Fargate Spot**: 70% cost reduction for non-critical workloads
- [ ] **Auto-scaling**: Dynamic resource allocation based on queue size
- [ ] **S3 Integration**: Direct file transfer without local downloads
- [ ] **Webhook Support**: Real-time notifications when processing completes
- [ ] **GPU Support**: Fargate GPU instances for AI-intensive workloads
## π License
MIT License - Same as the original CloudBurst project
---
**From single-task processing to parallel serverless scale - CloudBurst Fargate is production ready!** π
*Stop managing infrastructure, start processing videos at scale.*
Raw data
{
"_id": null,
"home_page": "https://github.com/preangelleo/cloudburst-fargate",
"name": "cloudburst-fargate",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": null,
"author": "Leo Wang",
"author_email": "Leo Wang <me@leowang.net>",
"download_url": "https://files.pythonhosted.org/packages/8f/9f/1821a667374836e2718ba6e49c6b0e4b68e21daf4568c25e478ebecca0d3/cloudburst_fargate-1.0.1.tar.gz",
"platform": null,
"description": "# CloudBurst Fargate - Serverless Video Processing\n\n[](https://pypi.org/project/cloudburst-fargate/)\n[](https://www.python.org/downloads/)\n[](https://opensource.org/licenses/MIT)\n[](https://aws.amazon.com/fargate/)\n\nMy second open source project, now powered by **AWS ECS Fargate**! \ud83d\ude80\n\n**Author**: Leo Wang ([leowang.net](https://leowang.net)) \n**Email**: me@leowang.net \n**License**: MIT\n\n> **\ud83d\udcda Related Projects**: \n> - **Original CloudBurst (EC2)**: https://github.com/preangelleo/cloudburst\n> - **Video Generation API**: https://github.com/preangelleo/video-generation-docker\n> - **\u4e2d\u6587\u6587\u6863**: [README_CN.md](./README_CN.md)\n\n## What is this?\n\nA production-ready Python framework that uses **AWS ECS Fargate** for serverless, on-demand video generation with **parallel processing capabilities**.\n\n**Core Value**: When your application needs to generate videos (using our [Video Generation API](https://github.com/preangelleo/video-generation-docker)), this framework:\n- \ud83d\ude80 Starts Fargate containers in **30 seconds** (vs 2+ minutes for EC2)\n- \u26a1 **Parallel processing**: Handle multiple scenes across concurrent containers\n- \ud83c\udfac Processes your video generation requests with zero infrastructure management\n- \ud83d\udce5 Downloads completed videos automatically with \"process one \u2192 download one\" efficiency\n- \ud83d\uded1 Containers terminate automatically after processing\n- \ud83d\udcb0 Pay-per-second billing with **no idle costs**\n\n**Perfect for**: Production applications that need scalable serverless video processing without the complexity of managing EC2 instances.\n\n## \ud83d\udce6 Installation\n\n### Install from PyPI\n```bash\npip install cloudburst-fargate\n```\n\n### Install from GitHub\n```bash\npip install git+https://github.com/preangelleo/cloudburst-fargate.git\n```\n\n### Install from Source\n```bash\ngit clone https://github.com/preangelleo/cloudburst-fargate.git\ncd cloudburst-fargate\npip install -e .\n```\n\n## \ud83c\udd9a CloudBurst Evolution: EC2 \u2192 Fargate\n\n| Feature | CloudBurst EC2 (v1) | **CloudBurst Fargate (v2)** |\n|---------|---------------------|------------------------------|\n| **Startup Time** | ~75 seconds | **~30 seconds** \u26a1 |\n| **Infrastructure** | Manage EC2 instances | **Fully serverless** \ud83c\udfaf |\n| **Parallel Processing** | Single instance only | **Multiple concurrent tasks** \ud83d\udd04 |\n| **Availability** | Subject to quota limits | **Near 100% availability** \u2705 |\n| **Scaling** | Limited by EC2 capacity | **Unlimited concurrent tasks** \ud83d\udcc8 |\n| **Cost Model** | Per-minute billing | **Per-second billing** \ud83d\udcb0 |\n| **Idle Costs** | Risk of forgotten instances | **Zero idle costs** \ud83d\udd25 |\n\n## \ud83d\ude80 Quick Start\n\n### 1. Install Package\n```bash\npip install cloudburst-fargate\n```\n\n### 2. Setup AWS Permissions (CRITICAL)\n\nCloudBurst Fargate requires specific IAM permissions to manage ECS tasks, access VPC resources, and handle container operations. You'll need to add permissions **4 times** during setup:\n\n#### Required IAM Permissions\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ecs:RunTask\",\n \"ecs:StopTask\", \n \"ecs:DescribeTasks\",\n \"ecs:DescribeClusters\",\n \"ecs:ListTasks\",\n \"ecs:ListTagsForResource\",\n \"ecs:TagResource\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ec2:DescribeNetworkInterfaces\",\n \"ec2:DescribeSubnets\",\n \"ec2:DescribeSecurityGroups\",\n \"ec2:AuthorizeSecurityGroupIngress\",\n \"ec2:DescribeVpcs\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"logs:CreateLogGroup\",\n \"logs:CreateLogStream\",\n \"logs:PutLogEvents\",\n \"logs:DescribeLogGroups\",\n \"logs:DescribeLogStreams\"\n ],\n \"Resource\": \"arn:aws:logs:*:*:log-group:/ecs/cloudburst*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"iam:PassRole\"\n ],\n \"Resource\": \"arn:aws:iam::*:role/ecsTaskExecutionRole\"\n }\n ]\n}\n```\n\n#### Step-by-Step Permission Setup\n1. **First Permission**: ECS Task Management\n ```bash\n # Add ECS permissions for running and managing Fargate tasks\n aws iam attach-user-policy --user-name YOUR_USER --policy-arn arn:aws:iam::aws:policy/AmazonECS_FullAccess\n ```\n\n2. **Second Permission**: VPC and Network Access \n ```bash\n # Add EC2 permissions for VPC, subnets, and security groups\n aws iam attach-user-policy --user-name YOUR_USER --policy-arn arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess\n ```\n\n3. **Third Permission**: CloudWatch Logs\n ```bash\n # Add CloudWatch permissions for container logging\n aws iam attach-user-policy --user-name YOUR_USER --policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess\n ```\n\n4. **Fourth Permission**: IAM Role Passing\n ```bash\n # Add permission to pass execution roles to ECS tasks\n aws iam put-user-policy --user-name YOUR_USER --policy-name ECSTaskRolePass --policy-document file://pass-role-policy.json\n ```\n\n### 3. Setup Environment\n```bash\n# Copy and customize configuration\ncp .env.example .env\n\n# Edit .env with your AWS credentials and VPC settings:\n# - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (with permissions above)\n# - AWS_SUBNET_ID (your VPC subnet with internet access) \n# - AWS_SECURITY_GROUP_ID (allows port 5000 and outbound HTTPS)\n```\n\n### 4. Test Your Setup\n```python\nfrom cloudburst_fargate import FargateOperationV1\n\n# Quick single-scene test\nprocessor = FargateOperationV1(config_priority=1)\nscenes = [{\n \"scene_name\": \"test_scene\",\n \"image_path\": \"path/to/image.png\",\n \"audio_path\": \"path/to/audio.mp3\",\n \"subtitle_path\": \"path/to/subtitle.srt\" # Optional\n}]\n\nresult = processor.execute_batch(scenes, language=\"english\", enable_zoom=True)\nprint(f\"\u2705 Generated {result['successful_scenes']} videos\")\n```\n\n### 5. Parallel Processing (Production Ready!)\n```python\nfrom cloudburst_fargate.fargate_operation import execute_parallel_batches\n\n# Process multiple scenes across parallel Fargate containers\nscenes = [\n {\"scene_name\": \"scene_001\", \"image_path\": \"...\", \"audio_path\": \"...\"},\n {\"scene_name\": \"scene_002\", \"image_path\": \"...\", \"audio_path\": \"...\"},\n {\"scene_name\": \"scene_003\", \"image_path\": \"...\", \"audio_path\": \"...\"},\n {\"scene_name\": \"scene_004\", \"image_path\": \"...\", \"audio_path\": \"...\"}\n]\n\n# Automatically distribute across 2 parallel tasks (2 scenes each)\nresult = execute_parallel_batches(\n scenes=scenes,\n scenes_per_batch=2, # 2 scenes per Fargate container\n max_parallel_tasks=2, # 2 concurrent containers\n language=\"english\",\n enable_zoom=True,\n config_priority=1, # CPU configuration (1-5, default: 4)\n watermark_path=None, # Optional watermark image\n is_portrait=False, # Portrait mode (default: False)\n saving_dir=\"./output\", # Output directory\n background_box=True, # Subtitle background (default: True)\n background_opacity=0.2 # Background transparency 0-1 (default: 0.2)\n)\n\nprint(f\"\ud83d\ude80 Efficiency: {result['efficiency']['speedup_factor']:.2f}x speedup\")\nprint(f\"\ud83d\udcb0 Total cost: ${result['total_cost_usd']:.4f}\")\nprint(f\"\ud83d\udcc1 {len(result['downloaded_files'])} videos downloaded\")\n```\n\n## \u26a1 Parallel Processing Architecture\n\nCloudBurst Fargate v2 introduces **true parallel processing**:\n\n### Architecture Benefits\n- **Concurrent Tasks**: Multiple Fargate containers running simultaneously\n- **Intelligent Distribution**: Scenes automatically distributed across tasks\n- **Efficient Workflow**: Each task processes scenes \u2192 downloads \u2192 terminates\n- **Cost Optimized**: Pay only for actual processing time across all containers\n\n### Example: 4 Scenes, 2 Tasks\n```\nTask 1: Start \u2192 Process scene_001 \u2192 Download \u2192 Process scene_002 \u2192 Download \u2192 Terminate\nTask 2: Start \u2192 Process scene_003 \u2192 Download \u2192 Process scene_004 \u2192 Download \u2192 Terminate\n\nResult: 1.8x speedup, all videos downloaded automatically\n```\n\n## \ud83d\udcca Fargate Configuration Options\n\nChoose the right performance level for your workload:\n\n```python\n# Economy: 1 vCPU, 2GB RAM (~$0.044/hour) - Light workloads\nprocessor = FargateOperationV1(config_priority=5)\n\n# Standard: 2 vCPU, 4GB RAM (~$0.088/hour) - Most common choice\nprocessor = FargateOperationV1(config_priority=1) # Default\n\n# High Performance: 4 vCPU, 8GB RAM (~$0.175/hour) - Heavy scenes\nprocessor = FargateOperationV1(config_priority=2)\n\n# Ultra Performance: 8 vCPU, 16GB RAM (~$0.351/hour) - Maximum speed\nprocessor = FargateOperationV1(config_priority=3)\n\n# Maximum Performance: 16 vCPU, 32GB RAM (~$0.702/hour) - Enterprise\nprocessor = FargateOperationV1(config_priority=4)\n```\n\n## \ud83c\udfac Complete Example (Production Ready)\n\nSee [`example_usage.py`](./example_usage.py) for comprehensive examples including:\n- All CPU configuration options\n- Complete API parameter reference\n- Single scene processing\n- Batch processing examples\n- Parallel processing configurations\n- Cost optimization strategies\n\n```python\n# Quick parallel processing example\nfrom cloudburst_fargate import FargateOperationV1\nfrom cloudburst_fargate.fargate_operation import execute_parallel_batches\n\nresult = execute_parallel_batches(\n scenes=your_scenes,\n scenes_per_batch=3, # Scenes per container\n max_parallel_tasks=4, # Concurrent containers \n language=\"chinese\", # or \"english\"\n enable_zoom=True, # Add zoom effects\n config_priority=2, # High performance config (1-5)\n min_scenes_per_batch=5, # Min scenes to justify startup (default: 5)\n watermark_path=None, # Optional watermark\n is_portrait=False, # Portrait video mode\n saving_dir=\"./videos\", # Output directory\n background_box=True, # Show subtitle background\n background_opacity=0.2 # Subtitle transparency\n)\n\n# Automatic results:\n# \u2705 All videos processed and downloaded\n# \ud83d\udcb0 Optimal cost distribution across parallel tasks\n# \ud83d\udcc8 Detailed efficiency and timing metrics\n```\n\n## \ud83d\udca1 Key Advantages\n\n### 1. **True Serverless with Parallel Scale**\n- **Per-second billing** from container start to finish\n- **Multiple concurrent containers** for faster processing\n- No risk of forgotten running instances\n- Automatic cleanup guaranteed\n\n### 2. **Zero Infrastructure Management**\n- No EC2 instances to monitor\n- No SSH keys or security patches\n- AWS handles all infrastructure and scaling\n\n### 3. **Production Performance**\n- **30-second startup** vs 75+ seconds for EC2\n- **Parallel processing** across multiple containers\n- Intelligent scene distribution and load balancing\n- Consistent performance (no \"noisy neighbor\" issues)\n\n### 4. **Enterprise Ready**\n- Built-in high availability and auto-retry\n- Integrated with AWS CloudWatch logging\n- VPC networking support\n- Cost tracking and optimization\n\n## \ud83d\udcb0 Cost Comparison\n\n**Example: Processing 8 video scenes**\n\n| Approach | Configuration | Time | Cost | Efficiency |\n|----------|---------------|------|------|------------|\n| **Sequential (Single Task)** | 2 vCPU | 16 min | $0.024 | 1.0x |\n| **\ud83c\udfc6 Parallel (4 Tasks \u00d7 2 Scenes)** | 2 vCPU each | 9 min | $0.026 | **1.8x faster** |\n| **24/7 GPU Server** | Always on | - | ~$500/month | - |\n\n**Key Insight**: Minimal cost increase (8%) for 80% time reduction!\n\n## \ud83d\udd27 Advanced Features\n\n### Intelligent Scene Distribution\nThe framework automatically:\n- Distributes scenes evenly across parallel tasks\n- Handles remainder scenes when batch sizes don't divide evenly\n- Optimizes for cost vs speed based on your configuration\n\n### Real-time Monitoring\n```python\n# Built-in cost tracking and performance metrics\nresult = execute_parallel_batches(scenes=scenes, ...)\n\nprint(f\"Tasks used: {result['tasks_used']}\")\nprint(f\"Processing efficiency: {result['efficiency']['processing_efficiency']:.1f}%\")\nprint(f\"Speedup factor: {result['efficiency']['speedup_factor']:.2f}x\")\nprint(f\"Cost per scene: ${result['total_cost_usd']/len(scenes):.4f}\")\n```\n\n### Flexible Configuration\n```python\n# Environment variables or .env file\nAWS_ACCESS_KEY_ID=your_access_key\nAWS_SECRET_ACCESS_KEY=your_secret_key\nAWS_SUBNET_ID=subnet-xxxxxxxxx\nAWS_SECURITY_GROUP_ID=sg-xxxxxxxxx\nECS_CLUSTER_NAME=cloudburst-cluster\nECS_TASK_DEFINITION=cloudburst-task\n```\n\n## \ud83d\udee0\ufe0f File Structure\n\nAfter cleanup, the project structure is:\n```\ncloudburst_fargate/\n\u251c\u2500\u2500 fargate_operation_v1.py # Core Fargate operations and parallel processing\n\u251c\u2500\u2500 example_usage.py # Complete usage examples and API reference\n\u251c\u2500\u2500 README.md # This file\n\u251c\u2500\u2500 README_CN.md # Chinese documentation\n\u251c\u2500\u2500 requirements.txt # Python dependencies\n\u251c\u2500\u2500 .env.example # Environment template\n\u251c\u2500\u2500 Docs/ # Technical documentation\n\u2514\u2500\u2500 backup_test_files/ # Test files (Git ignored)\n```\n\n## \ud83d\udd27 Troubleshooting\n\n### Common Issues\n\n**Task fails to start:**\n- Check subnet and security group IDs in .env\n- Ensure subnet has internet access (public subnet or NAT gateway)\n- Verify AWS credentials with correct permissions\n\n**Network errors:**\n- Security group must allow outbound HTTPS (port 443) for Docker pulls\n- Security group must allow inbound TCP port 5000 for API access\n\n**Permission errors:**\n- Verify AWS credentials: `aws sts get-caller-identity`\n- Required IAM permissions: ECS, ECR, CloudWatch, EC2 (for VPC)\n\n### Debug Mode\n```python\n# Enable detailed AWS logging\nimport logging\nlogging.basicConfig(level=logging.DEBUG)\n\n# Or check CloudWatch logs: /ecs/cloudburst\n```\n\n### Task Monitoring and Management (New in v2)\nCloudBurst Fargate now includes advanced task monitoring and cleanup capabilities to ensure reliable production operations:\n\n#### List Running Tasks\n```python\nfrom fargate_operation_v1 import FargateOperationV1\n\n# Initialize the operation\nfargate_op = FargateOperationV1()\n\n# List all running Fargate tasks created by animagent\nrunning_tasks = fargate_op.list_running_tasks(filter_animagent_only=True)\n\nfor task in running_tasks:\n print(f\"Task: {task['task_arn']}\")\n print(f\"Status: {task['status']}\")\n print(f\"Started: {task['started_at']}\")\n print(f\"Public IP: {task['public_ip']}\")\n print(f\"Tags: {task['tags']}\")\n```\n\n#### Cleanup Stale Tasks\n```python\n# Cleanup all animagent-created tasks (double security mechanism)\ncleanup_result = fargate_op.cleanup_all_tasks(\n reason=\"Scheduled cleanup\",\n filter_animagent_only=True # Only cleanup tasks tagged with CreatedBy=animagent\n)\n\nprint(f\"Cleanup result: {cleanup_result['message']}\")\nprint(f\"Tasks terminated: {cleanup_result['terminated_count']}\")\nprint(f\"Failed cleanups: {cleanup_result['failed_count']}\")\n```\n\n#### Task Identification\nAll tasks created by CloudBurst Fargate are automatically tagged for easy identification:\n- `CreatedBy`: `animagent` - Identifies tasks created by this framework\n- `Purpose`: `video-generation` - Marks the task purpose\n- `Scene`: Scene name being processed\n- `Language`: Processing language (english/chinese)\n\nThis tagging system ensures that cleanup operations only affect tasks created by your application, preventing interference with other services using the same ECS cluster.\n\n## \ud83d\udcda API Reference: execute_parallel_batches()\n\n### Complete Parameter List\n\n```python\nexecute_parallel_batches(\n scenes: List[Dict], # Required: List of scene dictionaries\n scenes_per_batch: int = 10, # Scenes per Fargate container\n max_parallel_tasks: int = 10, # Maximum concurrent containers\n language: str = \"chinese\", # Language: \"chinese\" or \"english\"\n enable_zoom: bool = True, # Enable zoom in/out effects\n config_priority: int = 4, # CPU config (1-5, see table below)\n min_scenes_per_batch: int = 5, # Minimum scenes to justify container startup\n watermark_path: str = None, # Optional watermark image path\n is_portrait: bool = False, # Portrait video mode\n saving_dir: str = None, # Output directory (default: ./cloudburst_fargate_results/)\n background_box: bool = True, # Show subtitle background\n background_opacity: float = 0.2 # Background transparency (0=opaque, 1=transparent)\n) -> Dict\n```\n\n### Scene Dictionary Format\n\nEach scene in the `scenes` list must contain:\n\n```python\n{\n \"scene_name\": \"unique_name\", # Required: Unique identifier for the scene\n \"image_path\": \"path/to/image\", # Required: Path to image file\n \"audio_path\": \"path/to/audio\", # Required: Path to audio file\n \"subtitle_path\": \"path/to/srt\" # Optional: Path to subtitle file\n}\n```\n\n### CPU Configuration Priority\n\n| Priority | vCPU | Memory | Name | Cost/Hour | Best For |\n|----------|------|--------|------|-----------|----------|\n| 1 | 2 | 4GB | STANDARD | $0.088 | Most tasks |\n| 2 | 4 | 8GB | HIGH_PERFORMANCE | $0.175 | Faster processing |\n| 3 | 8 | 16GB | ULTRA_PERFORMANCE | $0.351 | Very fast |\n| 4 | 16 | 32GB | MAXIMUM_PERFORMANCE | $0.702 | Fastest (default) |\n| 5 | 1 | 2GB | ECONOMY | $0.044 | Cost-sensitive |\n\n### Return Value Structure\n\n```python\n{\n \"success\": bool, # Overall success status\n \"total_scenes\": int, # Total number of input scenes\n \"successful_scenes\": int, # Successfully processed scenes\n \"failed_scenes\": int, # Failed scenes count\n \"total_cost_usd\": float, # Total cost in USD\n \"total_duration\": float, # Total time in seconds\n \"downloaded_files\": List[str], # Paths to downloaded videos\n \"task_results\": List[Dict], # Individual task results\n \"tasks_used\": int, # Number of Fargate tasks used\n \"efficiency\": {\n \"speedup_factor\": float, # Speedup vs sequential processing\n \"processing_efficiency\": float, # Percentage of time spent processing\n \"cost_per_scene\": float # Average cost per scene\n }\n}\n```\n\n### Smart Distribution Examples\n\n```python\n# Example 1: Even distribution\n# 50 scenes, batch=10, max_tasks=10 \u2192 5 tasks \u00d7 10 scenes each\n\n# Example 2: Redistribution for efficiency \n# 120 scenes, batch=10, max_tasks=10 \u2192 10 tasks \u00d7 12 scenes each\n\n# Example 3: Handling remainders\n# 101 scenes, batch=10, max_tasks=10 \u2192 9 tasks \u00d7 10 scenes + 1 task \u00d7 11 scenes\n```\n\n## \ud83c\udfaf Roadmap\n\n- [x] **\u2705 Parallel Processing**: Multiple concurrent Fargate tasks\n- [ ] **Fargate Spot**: 70% cost reduction for non-critical workloads \n- [ ] **Auto-scaling**: Dynamic resource allocation based on queue size\n- [ ] **S3 Integration**: Direct file transfer without local downloads\n- [ ] **Webhook Support**: Real-time notifications when processing completes\n- [ ] **GPU Support**: Fargate GPU instances for AI-intensive workloads\n\n## \ud83d\udcc4 License\n\nMIT License - Same as the original CloudBurst project\n\n---\n\n**From single-task processing to parallel serverless scale - CloudBurst Fargate is production ready!** \ud83d\ude80\n\n*Stop managing infrastructure, start processing videos at scale.*\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Serverless video processing using AWS ECS Fargate",
"version": "1.0.1",
"project_urls": {
"Bug Tracker": "https://github.com/preangelleo/cloudburst-fargate/issues",
"Documentation": "https://github.com/preangelleo/cloudburst-fargate/blob/main/README.md",
"Homepage": "https://github.com/preangelleo/cloudburst-fargate",
"Repository": "https://github.com/preangelleo/cloudburst-fargate.git"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "656c32d8c19a7c475f69c33ed3fe9bd71c294d26986c36d1eb709a146e6a3329",
"md5": "aab2cc831fd63e869b03d239a5ef10d7",
"sha256": "83f8d6d89ba5d0fc5b1f019a3a6171acbe60a45e4c2123e08e7c2447b177117d"
},
"downloads": -1,
"filename": "cloudburst_fargate-1.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "aab2cc831fd63e869b03d239a5ef10d7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 24929,
"upload_time": "2025-08-08T13:27:12",
"upload_time_iso_8601": "2025-08-08T13:27:12.219964Z",
"url": "https://files.pythonhosted.org/packages/65/6c/32d8c19a7c475f69c33ed3fe9bd71c294d26986c36d1eb709a146e6a3329/cloudburst_fargate-1.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "8f9f1821a667374836e2718ba6e49c6b0e4b68e21daf4568c25e478ebecca0d3",
"md5": "38aff01777ac346132ec27a338b38858",
"sha256": "bd997813785bfb3660f0f41a03c15cf47b0ca43ae99daf2271e5f04830773dcc"
},
"downloads": -1,
"filename": "cloudburst_fargate-1.0.1.tar.gz",
"has_sig": false,
"md5_digest": "38aff01777ac346132ec27a338b38858",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 82849,
"upload_time": "2025-08-08T13:27:13",
"upload_time_iso_8601": "2025-08-08T13:27:13.572722Z",
"url": "https://files.pythonhosted.org/packages/8f/9f/1821a667374836e2718ba6e49c6b0e4b68e21daf4568c25e478ebecca0d3/cloudburst_fargate-1.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-08 13:27:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "preangelleo",
"github_project": "cloudburst-fargate",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "boto3",
"specs": [
[
">=",
"1.26.0"
]
]
},
{
"name": "requests",
"specs": [
[
">=",
"2.28.0"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
">=",
"0.19.0"
]
]
},
{
"name": "colorama",
"specs": [
[
">=",
"0.4.4"
]
]
}
],
"lcname": "cloudburst-fargate"
}
JSON
