# MultiAgents Framework
A hybrid event-driven orchestration framework for building scalable, fault-tolerant distributed systems using a combination of orchestration and choreography patterns.
## Features
- **Hybrid Architecture**: Combines centralized orchestration with decoupled event-driven communication
- **DSPy Integration**: Built-in support for LLM-powered workers using DSPy
- **Comprehensive Monitoring**: Complete observability with event tracking, worker performance monitoring, and structured logging
- **Saga Pattern**: Built-in support for distributed transactions with compensation actions
- **Developer-Friendly**: Simple decorator-based API for creating workers
- **Scalable**: Horizontally scalable components with Redis backend
## Quick Start
### Prerequisites
1. **Python 3.8+**
2. **Redis Server** (for event bus and state storage)
Start Redis:
```bash
# macOS with Homebrew
brew services start redis
# Ubuntu/Debian
sudo systemctl start redis
# Docker
docker run -d -p 6379:6379 redis:alpine
```
### Installation
1. Clone and navigate to the project:
```bash
git clone <repository-url>
cd multiagents
```
2. Create virtual environment:
```bash
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
```
3. Install dependencies:
```bash
pip install -r requirements.txt
```
### Running Examples
**Option 1: Use the example runner (recommended)**
```bash
python run_examples.py
```
**Option 2: Run examples directly**
```bash
# Simple workflow
python -c "
import sys
sys.path.insert(0, '.')
import asyncio
from examples.simple_workflow import main
asyncio.run(main())
"
# E-commerce example with monitoring
python -c "
import sys
sys.path.insert(0, '.')
import asyncio
from examples.ecommerce_order.main import main
asyncio.run(main())
"
# Comprehensive monitoring demo
python -c "
import sys
sys.path.insert(0, '.')
import asyncio
from examples.monitoring_example import main
asyncio.run(main())
"
```
## Framework Overview
### Core Components
1. **Orchestrator**: Manages workflow state and coordinates activities
2. **Workers**: Stateless task executors created with simple decorators
3. **Event Bus**: Decoupled communication layer (Redis Pub/Sub)
4. **Monitoring**: Comprehensive observability and debugging
### Basic Usage
```python
from multiagents import (
Orchestrator, WorkflowBuilder, WorkerManager, worker
)
from multiagents.event_bus.redis_bus import RedisEventBus
# Define workers
@worker("process_data")
async def process_data_worker(context):
data = context["input_data"]
# Process the data
return {"processed": data, "timestamp": "2024-01-01T00:00:00Z"}
# Create workflow
workflow = (WorkflowBuilder("data_processing")
.add_step("process", "process_data")
.build())
# Set up framework
event_bus = RedisEventBus()
worker_manager = WorkerManager(event_bus)
orchestrator = Orchestrator(workflow, event_bus)
# Register worker and start
worker_manager.register(process_data_worker)
await event_bus.start()
await worker_manager.start()
await orchestrator.start()
# Execute workflow
transaction_id = await orchestrator.execute_workflow(
"data_processing",
{"input_data": "example data"}
)
```
### Monitoring & Observability
The framework includes comprehensive monitoring:
```python
from multiagents.monitoring import MonitoringConfig, EventMonitor, WorkerMonitor
# Setup monitoring
config = MonitoringConfig()
logger = config.create_logger()
event_monitor = EventMonitor(logger=logger)
worker_monitor = WorkerMonitor(logger=logger)
# Integrate with framework
event_bus = RedisEventBus(event_monitor=event_monitor, logger=logger)
worker_manager = WorkerManager(event_bus, worker_monitor=worker_monitor, logger=logger)
```
**Monitoring Features:**
- **Event Lifecycle Tracking**: Complete event journey from dispatch to completion
- **Worker Performance**: Success rates, processing times, health monitoring
- **Structured Logging**: JSON logs with automatic rotation
- **Error Tracking**: Detailed error context and failure patterns
- **Real-time Metrics**: Performance dashboards and alerting
**Configuration (monitoring.yaml):**
```yaml
logging:
default_logger: "file"
level: "INFO"
file_path: "./logs/multiagents.log"
event_monitoring:
enabled: true
trace_retention_hours: 24
worker_monitoring:
enabled: true
health_check_interval_seconds: 30
```
## Examples
### 1. Simple Workflow
Basic workflow demonstrating core framework features.
### 2. E-commerce Order Processing
Complete order processing pipeline with:
- Order validation
- Inventory checking with compensation
- Payment processing with refund compensation
- DSPy-powered order confirmation generation
- Fulfillment and customer notification
### 3. Monitoring Demonstration
Comprehensive monitoring example showing:
- Event lifecycle tracking
- Worker performance monitoring
- Error handling and recovery
- System metrics collection
## Development
### Project Structure
```
multiagents/
├── orchestrator/ # Workflow orchestration
├── worker_sdk/ # Worker development SDK
├── event_bus/ # Event bus implementations
├── monitoring/ # Observability system
├── core/ # Core utilities
└── examples/ # Example implementations
```
### Key Design Principles
- **SOLID Principles**: Clean, maintainable architecture
- **Event-Driven**: Fully decoupled communication
- **Fault Tolerance**: Built-in compensation and rollback
- **Observability**: Comprehensive monitoring and debugging
- **Developer Experience**: Simple, intuitive APIs
### Common Commands
```bash
# Install dependencies
pip install -r requirements.txt
# Run examples
python run_examples.py
# Run tests (when available)
pytest
# Format code
black .
ruff check --fix .
# Type checking
mypy multiagents/
```
## Architecture
The framework follows a hybrid orchestration/choreography pattern:
- **Orchestration**: Centralized workflow management with state machine
- **Choreography**: Decoupled event-driven communication between components
- **Saga Pattern**: Distributed transaction management with compensations
- **Event Sourcing**: Complete audit trail of all activities
This approach provides the benefits of both patterns:
- **Centralized Logic**: Easy to understand and debug workflows
- **Decoupled Components**: Scalable and resilient architecture
- **Fault Tolerance**: Automatic compensation and recovery
- **Observability**: Complete visibility into system behavior
## Contributing
1. Follow SOLID principles and clean code practices
2. Add comprehensive monitoring to all components
3. Include tests for new functionality
4. Update documentation and examples
## License
[License information to be added]
Raw data
{
"_id": null,
"home_page": "https://github.com/xavierau/multiagents",
"name": "multiagents-framework",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "orchestration, workflow, event-driven, saga, dspy, llm",
"author": "MultiAgents Team",
"author_email": "MultiAgents Team <info@multiagents.dev>",
"download_url": "https://files.pythonhosted.org/packages/1a/b7/9c30c36b2ec73f31d5e0b811a1967596b3b17193f7bde4e5f6f98bc74a74/multiagents_framework-0.1.0.tar.gz",
"platform": null,
"description": "# MultiAgents Framework\n\nA hybrid event-driven orchestration framework for building scalable, fault-tolerant distributed systems using a combination of orchestration and choreography patterns.\n\n## Features\n\n- **Hybrid Architecture**: Combines centralized orchestration with decoupled event-driven communication\n- **DSPy Integration**: Built-in support for LLM-powered workers using DSPy\n- **Comprehensive Monitoring**: Complete observability with event tracking, worker performance monitoring, and structured logging\n- **Saga Pattern**: Built-in support for distributed transactions with compensation actions\n- **Developer-Friendly**: Simple decorator-based API for creating workers\n- **Scalable**: Horizontally scalable components with Redis backend\n\n## Quick Start\n\n### Prerequisites\n\n1. **Python 3.8+**\n2. **Redis Server** (for event bus and state storage)\n\nStart Redis:\n```bash\n# macOS with Homebrew\nbrew services start redis\n\n# Ubuntu/Debian\nsudo systemctl start redis\n\n# Docker\ndocker run -d -p 6379:6379 redis:alpine\n```\n\n### Installation\n\n1. Clone and navigate to the project:\n```bash\ngit clone <repository-url>\ncd multiagents\n```\n\n2. Create virtual environment:\n```bash\npython -m venv .venv\nsource .venv/bin/activate # On Windows: .venv\\Scripts\\activate\n```\n\n3. Install dependencies:\n```bash\npip install -r requirements.txt\n```\n\n### Running Examples\n\n**Option 1: Use the example runner (recommended)**\n```bash\npython run_examples.py\n```\n\n**Option 2: Run examples directly**\n```bash\n# Simple workflow\npython -c \"\nimport sys\nsys.path.insert(0, '.')\nimport asyncio\nfrom examples.simple_workflow import main\nasyncio.run(main())\n\"\n\n# E-commerce example with monitoring\npython -c \"\nimport sys\nsys.path.insert(0, '.')\nimport asyncio\nfrom examples.ecommerce_order.main import main\nasyncio.run(main())\n\"\n\n# Comprehensive monitoring demo\npython -c \"\nimport sys\nsys.path.insert(0, '.')\nimport asyncio\nfrom examples.monitoring_example import main\nasyncio.run(main())\n\"\n```\n\n## Framework Overview\n\n### Core Components\n\n1. **Orchestrator**: Manages workflow state and coordinates activities\n2. **Workers**: Stateless task executors created with simple decorators\n3. **Event Bus**: Decoupled communication layer (Redis Pub/Sub)\n4. **Monitoring**: Comprehensive observability and debugging\n\n### Basic Usage\n\n```python\nfrom multiagents import (\n Orchestrator, WorkflowBuilder, WorkerManager, worker\n)\nfrom multiagents.event_bus.redis_bus import RedisEventBus\n\n# Define workers\n@worker(\"process_data\")\nasync def process_data_worker(context):\n data = context[\"input_data\"]\n # Process the data\n return {\"processed\": data, \"timestamp\": \"2024-01-01T00:00:00Z\"}\n\n# Create workflow\nworkflow = (WorkflowBuilder(\"data_processing\")\n .add_step(\"process\", \"process_data\")\n .build())\n\n# Set up framework\nevent_bus = RedisEventBus()\nworker_manager = WorkerManager(event_bus)\norchestrator = Orchestrator(workflow, event_bus)\n\n# Register worker and start\nworker_manager.register(process_data_worker)\nawait event_bus.start()\nawait worker_manager.start()\nawait orchestrator.start()\n\n# Execute workflow\ntransaction_id = await orchestrator.execute_workflow(\n \"data_processing\",\n {\"input_data\": \"example data\"}\n)\n```\n\n### Monitoring & Observability\n\nThe framework includes comprehensive monitoring:\n\n```python\nfrom multiagents.monitoring import MonitoringConfig, EventMonitor, WorkerMonitor\n\n# Setup monitoring\nconfig = MonitoringConfig()\nlogger = config.create_logger()\nevent_monitor = EventMonitor(logger=logger)\nworker_monitor = WorkerMonitor(logger=logger)\n\n# Integrate with framework\nevent_bus = RedisEventBus(event_monitor=event_monitor, logger=logger)\nworker_manager = WorkerManager(event_bus, worker_monitor=worker_monitor, logger=logger)\n```\n\n**Monitoring Features:**\n- **Event Lifecycle Tracking**: Complete event journey from dispatch to completion\n- **Worker Performance**: Success rates, processing times, health monitoring\n- **Structured Logging**: JSON logs with automatic rotation\n- **Error Tracking**: Detailed error context and failure patterns\n- **Real-time Metrics**: Performance dashboards and alerting\n\n**Configuration (monitoring.yaml):**\n```yaml\nlogging:\n default_logger: \"file\"\n level: \"INFO\"\n file_path: \"./logs/multiagents.log\"\n\nevent_monitoring:\n enabled: true\n trace_retention_hours: 24\n\nworker_monitoring:\n enabled: true\n health_check_interval_seconds: 30\n```\n\n## Examples\n\n### 1. Simple Workflow\nBasic workflow demonstrating core framework features.\n\n### 2. E-commerce Order Processing\nComplete order processing pipeline with:\n- Order validation\n- Inventory checking with compensation\n- Payment processing with refund compensation\n- DSPy-powered order confirmation generation\n- Fulfillment and customer notification\n\n### 3. Monitoring Demonstration\nComprehensive monitoring example showing:\n- Event lifecycle tracking\n- Worker performance monitoring\n- Error handling and recovery\n- System metrics collection\n\n## Development\n\n### Project Structure\n```\nmultiagents/\n\u251c\u2500\u2500 orchestrator/ # Workflow orchestration\n\u251c\u2500\u2500 worker_sdk/ # Worker development SDK\n\u251c\u2500\u2500 event_bus/ # Event bus implementations\n\u251c\u2500\u2500 monitoring/ # Observability system\n\u251c\u2500\u2500 core/ # Core utilities\n\u2514\u2500\u2500 examples/ # Example implementations\n```\n\n### Key Design Principles\n- **SOLID Principles**: Clean, maintainable architecture\n- **Event-Driven**: Fully decoupled communication\n- **Fault Tolerance**: Built-in compensation and rollback\n- **Observability**: Comprehensive monitoring and debugging\n- **Developer Experience**: Simple, intuitive APIs\n\n### Common Commands\n```bash\n# Install dependencies\npip install -r requirements.txt\n\n# Run examples\npython run_examples.py\n\n# Run tests (when available)\npytest\n\n# Format code\nblack .\nruff check --fix .\n\n# Type checking\nmypy multiagents/\n```\n\n## Architecture\n\nThe framework follows a hybrid orchestration/choreography pattern:\n\n- **Orchestration**: Centralized workflow management with state machine\n- **Choreography**: Decoupled event-driven communication between components\n- **Saga Pattern**: Distributed transaction management with compensations\n- **Event Sourcing**: Complete audit trail of all activities\n\nThis approach provides the benefits of both patterns:\n- **Centralized Logic**: Easy to understand and debug workflows\n- **Decoupled Components**: Scalable and resilient architecture\n- **Fault Tolerance**: Automatic compensation and recovery\n- **Observability**: Complete visibility into system behavior\n\n## Contributing\n\n1. Follow SOLID principles and clean code practices\n2. Add comprehensive monitoring to all components\n3. Include tests for new functionality\n4. Update documentation and examples\n\n## License\n\n[License information to be added]\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Hybrid Event-Driven Orchestration Framework",
"version": "0.1.0",
"project_urls": {
"Changelog": "https://github.com/xavierau/multiagents/blob/main/CHANGELOG.md",
"Documentation": "https://github.com/xavierau/multiagents/tree/main/docs",
"Homepage": "https://github.com/xavierau/multiagents",
"Issues": "https://github.com/xavierau/multiagents/issues",
"Repository": "https://github.com/xavierau/multiagents"
},
"split_keywords": [
"orchestration",
" workflow",
" event-driven",
" saga",
" dspy",
" llm"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "3156015bece85d5e0267a765d949d5d62dca94d8b29f4c8f4f4d00407a37d5ed",
"md5": "062eaa2867d02b746608743f42cdf215",
"sha256": "04d422bc93998d318f90599e14ff23e76005ee448b88ba20e91c9fc3835c23a8"
},
"downloads": -1,
"filename": "multiagents_framework-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "062eaa2867d02b746608743f42cdf215",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 75926,
"upload_time": "2025-08-04T18:38:16",
"upload_time_iso_8601": "2025-08-04T18:38:16.539315Z",
"url": "https://files.pythonhosted.org/packages/31/56/015bece85d5e0267a765d949d5d62dca94d8b29f4c8f4f4d00407a37d5ed/multiagents_framework-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "1ab79c30c36b2ec73f31d5e0b811a1967596b3b17193f7bde4e5f6f98bc74a74",
"md5": "32cad7aa1b5c47e2c2af4b73c66ae600",
"sha256": "ee3c76fee8573de837586ad527676566c6d39d70b47bb57d55d6a52a8078e679"
},
"downloads": -1,
"filename": "multiagents_framework-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "32cad7aa1b5c47e2c2af4b73c66ae600",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 335717,
"upload_time": "2025-08-04T18:38:18",
"upload_time_iso_8601": "2025-08-04T18:38:18.696179Z",
"url": "https://files.pythonhosted.org/packages/1a/b7/9c30c36b2ec73f31d5e0b811a1967596b3b17193f7bde4e5f6f98bc74a74/multiagents_framework-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-04 18:38:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "xavierau",
"github_project": "multiagents",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "redis",
"specs": [
[
">=",
"5.0.0"
]
]
},
{
"name": "pydantic",
"specs": [
[
">=",
"2.0.0"
]
]
},
{
"name": "asyncio-redis",
"specs": [
[
">=",
"0.16.1"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
">=",
"1.0.0"
]
]
},
{
"name": "dspy-ai",
"specs": [
[
">=",
"2.0.0"
]
]
},
{
"name": "pytest",
"specs": [
[
">=",
"7.0.0"
]
]
},
{
"name": "pytest-asyncio",
"specs": [
[
">=",
"0.21.0"
]
]
},
{
"name": "pytest-cov",
"specs": [
[
">=",
"4.0.0"
]
]
},
{
"name": "black",
"specs": [
[
">=",
"23.0.0"
]
]
},
{
"name": "ruff",
"specs": [
[
">=",
"0.1.0"
]
]
},
{
"name": "structlog",
"specs": [
[
">=",
"23.0.0"
]
]
},
{
"name": "opentelemetry-api",
"specs": [
[
">=",
"1.20.0"
]
]
},
{
"name": "opentelemetry-sdk",
"specs": [
[
">=",
"1.20.0"
]
]
},
{
"name": "mypy",
"specs": [
[
">=",
"1.0.0"
]
]
},
{
"name": "types-redis",
"specs": [
[
">=",
"4.0.0"
]
]
},
{
"name": "PyYAML",
"specs": [
[
">=",
"6.0.0"
]
]
},
{
"name": "psutil",
"specs": [
[
">=",
"5.9.0"
]
]
},
{
"name": "aiohttp",
"specs": [
[
">=",
"3.8.0"
]
]
}
],
"lcname": "multiagents-framework"
}
JSON
