# ReasonFlow
ReasonFlow is a powerful workflow orchestration framework designed for building and managing complex AI/ML pipelines with advanced observability and tracking capabilities.
## Features
- **Workflow Orchestration**
- Task dependency management
- Parallel execution support
- Error handling and retries
- State management
- Dynamic task configuration
- **Advanced Observability**
- Real-time task tracking
- Metrics collection
- Event logging
- Alert management
- Health monitoring
- **Integrations**
- Multiple LLM providers (OpenAI, Ollama, Groq)
- Vector databases (FAISS, Milvus, Pinecone)
- Monitoring systems (Prometheus, Kafka)
- Alert systems (Slack, Email)
- **RAG Capabilities**
- Document processing
- Vector storage
- Semantic search
- Context management
## Installation
```bash
# Clone the repository
git clone https://github.com/yourusername/reasonflow.git
cd reasonflow
# Create and activate virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
```
## Dependencies
### Core Dependencies
```
reasontrack>=0.1.0
reasonchain>=0.1.0
python-dotenv>=1.0.0
```
### Observability Dependencies
```
prometheus-client>=0.17.1
opentelemetry-api>=1.20.0
opentelemetry-sdk>=1.20.0
opentelemetry-exporter-otlp>=1.20.0
influxdb-client>=1.38.0
elasticsearch>=8.10.1
elasticsearch-async>=6.2.0
confluent-kafka>=2.2.0
redis>=5.0.1
aioredis>=2.0.1
aiosmtplib>=2.0.0
```
### AI/ML Dependencies
```
sentence-transformers
faiss-cpu # or faiss-gpu for GPU support
torch
transformers
```
## Configuration
### Environment Variables
Create a `.env` file in the root directory:
```env
OPENAI_API_KEY=your_openai_api_key
GROQ_API_KEY=your_groq_api_key
SLACK_WEBHOOK_URL=your_slack_webhook_url
ALERT_EMAIL=your_alert_email
ALERT_EMAIL_PASSWORD=your_email_password
ALERT_RECIPIENTS=recipient1@example.com,recipient2@example.com
```
### ReasonTrack Configuration
Create `config/reasontrack.ini`:
```ini
[event_manager]
backend = kafka
broker_url = localhost:9092
topic_prefix = reasonflow_events_
client_id = reasonflow
batch_size = 100
flush_interval = 10
[metrics_collector]
backend = prometheus
pushgateway_url = localhost:9091
job_name = reasonflow_metrics
push_interval = 15
[alert_manager]
storage_path = alerts
retention_days = 30
severity_levels = INFO,WARNING,ERROR,CRITICAL
[alert_manager.slack]
webhook_url = ${SLACK_WEBHOOK_URL}
[alert_manager.email]
smtp_host = smtp.gmail.com
smtp_port = 587
username = ${ALERT_EMAIL}
password = ${ALERT_EMAIL_PASSWORD}
from_address = ${ALERT_EMAIL}
to_addresses = ${ALERT_RECIPIENTS}
use_tls = true
[state_manager]
storage_path = workflow_states
backend = memory
prefix = reasonflow_state_
ttl = 3600
[telemetry]
service_name = reasonflow
endpoint = localhost:4317
enable_metrics = true
enable_tracing = true
[logging]
level = INFO
format = %%(asctime)s - %%(name)s - %%(levelname)s - %%(message)s
file = logs/reasontrack.log
```
## Usage
### Basic Example
```python
from reasonflow.orchestrator.workflow_builder import WorkflowBuilder
from reasonflow.tasks.task_manager import TaskManager
from reasonchain.memory import SharedMemory
# Initialize components
shared_memory = SharedMemory()
task_manager = TaskManager(shared_memory=shared_memory)
workflow_builder = WorkflowBuilder(
task_manager=task_manager,
tracker_type="basic"
)
# Define workflow
workflow_config = {
"tasks": {
"task1": {
"type": "llm",
"config": {
"agent": llm_agent,
"params": {"prompt": "Your prompt here"}
}
}
}
}
# Create and execute workflow
workflow_id = workflow_builder.create_workflow(workflow_config)
results = workflow_builder.execute_workflow(workflow_id)
```
### Advanced Example with Tracking
See `example/1_advance_tracking_example.py` for a complete example demonstrating:
- RAG integration
- Multiple LLM providers
- Advanced tracking
- Metrics collection
- Alert management
## Directory Structure
```
reasonflow/
├── config/
│ └── reasontrack.ini
├── example/
│ ├── 1_advance_tracking_example.py
│ └── 2_tracking_example.py
├── reasonflow/
│ ├── agents/
│ ├── integrations/
│ ├── observability/
│ ├── orchestrator/
│ └── tasks/
├── tests/
├── .env
├── README.md
└── requirements.txt
```
## Contributing
1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
## License
This project is licensed under the MIT License - see the LICENSE file for details.
## Acknowledgments
- ReasonTrack for observability components
- ReasonChain for memory management
- All the amazing open-source libraries that make this possible
Raw data
{
"_id": null,
"home_page": "https://github.com/sunnybedi990/reasonflow",
"name": "reasonflow",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "workflow, orchestration, llm, rag, observability, ai, ml",
"author": "Baljindersingh Bedi",
"author_email": "Baljindersingh Bedi <baljinder@reasonchain.ai>",
"download_url": "https://files.pythonhosted.org/packages/7f/9c/1b0c63cc12895d22691e1f11141af47e818360b582891c7605cbaabe8377/reasonflow-0.2.0.tar.gz",
"platform": null,
"description": "# ReasonFlow\n\nReasonFlow is a powerful workflow orchestration framework designed for building and managing complex AI/ML pipelines with advanced observability and tracking capabilities.\n\n## Features\n\n- **Workflow Orchestration**\n - Task dependency management\n - Parallel execution support\n - Error handling and retries\n - State management\n - Dynamic task configuration\n\n- **Advanced Observability**\n - Real-time task tracking\n - Metrics collection\n - Event logging\n - Alert management\n - Health monitoring\n\n- **Integrations**\n - Multiple LLM providers (OpenAI, Ollama, Groq)\n - Vector databases (FAISS, Milvus, Pinecone)\n - Monitoring systems (Prometheus, Kafka)\n - Alert systems (Slack, Email)\n\n- **RAG Capabilities**\n - Document processing\n - Vector storage\n - Semantic search\n - Context management\n\n## Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/yourusername/reasonflow.git\ncd reasonflow\n\n# Create and activate virtual environment\npython -m venv venv\nsource venv/bin/activate # On Windows: venv\\Scripts\\activate\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n## Dependencies\n\n### Core Dependencies\n```\nreasontrack>=0.1.0\nreasonchain>=0.1.0\npython-dotenv>=1.0.0\n```\n\n### Observability Dependencies\n```\nprometheus-client>=0.17.1\nopentelemetry-api>=1.20.0\nopentelemetry-sdk>=1.20.0\nopentelemetry-exporter-otlp>=1.20.0\ninfluxdb-client>=1.38.0\nelasticsearch>=8.10.1\nelasticsearch-async>=6.2.0\nconfluent-kafka>=2.2.0\nredis>=5.0.1\naioredis>=2.0.1\naiosmtplib>=2.0.0\n```\n\n### AI/ML Dependencies\n```\nsentence-transformers\nfaiss-cpu # or faiss-gpu for GPU support\ntorch\ntransformers\n```\n\n## Configuration\n\n### Environment Variables\nCreate a `.env` file in the root directory:\n```env\nOPENAI_API_KEY=your_openai_api_key\nGROQ_API_KEY=your_groq_api_key\nSLACK_WEBHOOK_URL=your_slack_webhook_url\nALERT_EMAIL=your_alert_email\nALERT_EMAIL_PASSWORD=your_email_password\nALERT_RECIPIENTS=recipient1@example.com,recipient2@example.com\n```\n\n### ReasonTrack Configuration\nCreate `config/reasontrack.ini`:\n```ini\n[event_manager]\nbackend = kafka\nbroker_url = localhost:9092\ntopic_prefix = reasonflow_events_\nclient_id = reasonflow\nbatch_size = 100\nflush_interval = 10\n\n[metrics_collector]\nbackend = prometheus\npushgateway_url = localhost:9091\njob_name = reasonflow_metrics\npush_interval = 15\n\n[alert_manager]\nstorage_path = alerts\nretention_days = 30\nseverity_levels = INFO,WARNING,ERROR,CRITICAL\n\n[alert_manager.slack]\nwebhook_url = ${SLACK_WEBHOOK_URL}\n\n[alert_manager.email]\nsmtp_host = smtp.gmail.com\nsmtp_port = 587\nusername = ${ALERT_EMAIL}\npassword = ${ALERT_EMAIL_PASSWORD}\nfrom_address = ${ALERT_EMAIL}\nto_addresses = ${ALERT_RECIPIENTS}\nuse_tls = true\n\n[state_manager]\nstorage_path = workflow_states\nbackend = memory\nprefix = reasonflow_state_\nttl = 3600\n\n[telemetry]\nservice_name = reasonflow\nendpoint = localhost:4317\nenable_metrics = true\nenable_tracing = true\n\n[logging]\nlevel = INFO\nformat = %%(asctime)s - %%(name)s - %%(levelname)s - %%(message)s\nfile = logs/reasontrack.log\n```\n\n## Usage\n\n### Basic Example\n```python\nfrom reasonflow.orchestrator.workflow_builder import WorkflowBuilder\nfrom reasonflow.tasks.task_manager import TaskManager\nfrom reasonchain.memory import SharedMemory\n\n# Initialize components\nshared_memory = SharedMemory()\ntask_manager = TaskManager(shared_memory=shared_memory)\nworkflow_builder = WorkflowBuilder(\n task_manager=task_manager,\n tracker_type=\"basic\"\n)\n\n# Define workflow\nworkflow_config = {\n \"tasks\": {\n \"task1\": {\n \"type\": \"llm\",\n \"config\": {\n \"agent\": llm_agent,\n \"params\": {\"prompt\": \"Your prompt here\"}\n }\n }\n }\n}\n\n# Create and execute workflow\nworkflow_id = workflow_builder.create_workflow(workflow_config)\nresults = workflow_builder.execute_workflow(workflow_id)\n```\n\n### Advanced Example with Tracking\nSee `example/1_advance_tracking_example.py` for a complete example demonstrating:\n- RAG integration\n- Multiple LLM providers\n- Advanced tracking\n- Metrics collection\n- Alert management\n\n## Directory Structure\n```\nreasonflow/\n\u251c\u2500\u2500 config/\n\u2502 \u2514\u2500\u2500 reasontrack.ini\n\u251c\u2500\u2500 example/\n\u2502 \u251c\u2500\u2500 1_advance_tracking_example.py\n\u2502 \u2514\u2500\u2500 2_tracking_example.py\n\u251c\u2500\u2500 reasonflow/\n\u2502 \u251c\u2500\u2500 agents/\n\u2502 \u251c\u2500\u2500 integrations/\n\u2502 \u251c\u2500\u2500 observability/\n\u2502 \u251c\u2500\u2500 orchestrator/\n\u2502 \u2514\u2500\u2500 tasks/\n\u251c\u2500\u2500 tests/\n\u251c\u2500\u2500 .env\n\u251c\u2500\u2500 README.md\n\u2514\u2500\u2500 requirements.txt\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add some amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n## Acknowledgments\n\n- ReasonTrack for observability components\n- ReasonChain for memory management\n- All the amazing open-source libraries that make this possible\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A powerful workflow orchestration framework for AI/ML pipelines with advanced observability",
"version": "0.2.0",
"project_urls": {
"Documentation": "https://reasonflow.readthedocs.io",
"Homepage": "https://github.com/sunnybedi990/reasonflow",
"Issues": "https://github.com/sunnybedi990/reasonflow/issues",
"Repository": "https://github.com/sunnybedi990/reasonflow.git"
},
"split_keywords": [
"workflow",
" orchestration",
" llm",
" rag",
" observability",
" ai",
" ml"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4558bc7fa149eaf312b66deb138c7f6b2349938598075fddd6aca96d57841f7d",
"md5": "7318d12e627873518f91dcd3cce1980a",
"sha256": "a19b2ba8fc9a0307b3fc024fc17b71354c3ce479a54efabe9e3bc8ddca9490a2"
},
"downloads": -1,
"filename": "reasonflow-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7318d12e627873518f91dcd3cce1980a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 39569,
"upload_time": "2024-12-27T03:23:40",
"upload_time_iso_8601": "2024-12-27T03:23:40.867461Z",
"url": "https://files.pythonhosted.org/packages/45/58/bc7fa149eaf312b66deb138c7f6b2349938598075fddd6aca96d57841f7d/reasonflow-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7f9c1b0c63cc12895d22691e1f11141af47e818360b582891c7605cbaabe8377",
"md5": "be21e132a683e20ab3256fd456546ef8",
"sha256": "c01348cd6e7fe157a955d2542961521d8fe19c78940559457104b34871b41ca6"
},
"downloads": -1,
"filename": "reasonflow-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "be21e132a683e20ab3256fd456546ef8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 42854,
"upload_time": "2024-12-27T03:23:47",
"upload_time_iso_8601": "2024-12-27T03:23:47.037258Z",
"url": "https://files.pythonhosted.org/packages/7f/9c/1b0c63cc12895d22691e1f11141af47e818360b582891c7605cbaabe8377/reasonflow-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-27 03:23:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "sunnybedi990",
"github_project": "reasonflow",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "reasonchain",
"specs": [
[
">=",
"0.1.0"
]
]
},
{
"name": "networkx",
"specs": [
[
">=",
"2.8.0"
]
]
},
{
"name": "pyyaml",
"specs": [
[
">=",
"6.0"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
">=",
"1.0.0"
]
]
},
{
"name": "fastapi",
"specs": [
[
">=",
"0.68.0"
]
]
},
{
"name": "uvicorn",
"specs": [
[
">=",
"0.15.0"
]
]
},
{
"name": "sentence-transformers",
"specs": [
[
">=",
"2.2.0"
]
]
},
{
"name": "faiss-cpu",
"specs": [
[
">=",
"1.7.0"
]
]
},
{
"name": "pinecone-client",
"specs": [
[
">=",
"2.2.1"
]
]
},
{
"name": "pymilvus",
"specs": [
[
">=",
"2.2.8"
]
]
},
{
"name": "qdrant-client",
"specs": [
[
">=",
"1.1.1"
]
]
},
{
"name": "weaviate-client",
"specs": [
[
">=",
"3.15.0"
]
]
},
{
"name": "openai",
"specs": [
[
">=",
"1.0.0"
]
]
},
{
"name": "groq",
"specs": [
[
">=",
"0.4.0"
]
]
},
{
"name": "ollama",
"specs": [
[
">=",
"0.1.0"
]
]
},
{
"name": "anthropic",
"specs": [
[
">=",
"0.5.0"
]
]
},
{
"name": "pdfplumber",
"specs": [
[
">=",
"0.10.0"
]
]
},
{
"name": "camelot-py",
"specs": [
[
">=",
"0.11.0"
]
]
},
{
"name": "ghostscript",
"specs": [
[
">=",
"0.7"
]
]
},
{
"name": "PyMuPDF",
"specs": [
[
">=",
"1.20.0"
]
]
},
{
"name": "PyPDF2",
"specs": [
[
">=",
"3.0.0"
]
]
},
{
"name": "firebase-admin",
"specs": [
[
">=",
"5.0.0"
]
]
},
{
"name": "boto3",
"specs": [
[
">=",
"1.26.0"
]
]
},
{
"name": "minio",
"specs": [
[
">=",
"7.1.0"
]
]
},
{
"name": "psycopg2-binary",
"specs": [
[
">=",
"2.9.5"
]
]
},
{
"name": "sqlalchemy",
"specs": [
[
">=",
"1.4.0"
]
]
},
{
"name": "tqdm",
"specs": [
[
">=",
"4.65.0"
]
]
},
{
"name": "requests",
"specs": [
[
">=",
"2.28.0"
]
]
},
{
"name": "keyring",
"specs": [
[
">=",
"24.0.0"
]
]
},
{
"name": "numpy",
"specs": [
[
">=",
"1.21.0"
]
]
},
{
"name": "pandas",
"specs": [
[
">=",
"1.3.0"
]
]
},
{
"name": "pytest",
"specs": [
[
">=",
"7.0"
]
]
},
{
"name": "pytest-cov",
"specs": [
[
">=",
"4.0.0"
]
]
},
{
"name": "black",
"specs": [
[
">=",
"22.0"
]
]
},
{
"name": "isort",
"specs": [
[
">=",
"5.0"
]
]
},
{
"name": "mypy",
"specs": [
[
">=",
"1.0"
]
]
},
{
"name": "flake8",
"specs": [
[
">=",
"4.0"
]
]
},
{
"name": "pre-commit",
"specs": [
[
">=",
"3.0.0"
]
]
},
{
"name": "sphinx",
"specs": [
[
">=",
"4.0.0"
]
]
},
{
"name": "sphinx-rtd-theme",
"specs": [
[
">=",
"1.0.0"
]
]
},
{
"name": "myst-parser",
"specs": [
[
">=",
"0.18.0"
]
]
}
],
"lcname": "reasonflow"
}
JSON
