# Strataregula - DOE Runner
[](https://github.com/strataregula/strataregula-doe-runner/actions/workflows/test.yml)
[](https://codecov.io/gh/strataregula/strataregula-doe-runner)
[](https://pypi.org/project/strataregula-doe-runner/)
[](https://www.python.org/)
[](LICENSE)
[](https://github.com/strataregula/strataregula)
Batch experiment orchestrator for Strataregula ecosystem.
## What is DOE Runner?
**DOE Runner is NOT a Design of Experiments tool.** It is a **batch experiment orchestrator** that:
- ๐ Executes predefined cases from `cases.csv`
- ๐ Manages execution order and parallelism
- ๐ Collects standardized metrics to `metrics.csv`
- ๐ Provides deterministic, reproducible results
- ๐พ Caches execution results with `case_hash`
- ๐ Generates comprehensive run logs (JST timezone)
**Definition (1 line)**: cases.csvใงๅฎ็พฉใใใฑใผใน้ๅใใๆฑบๅฎ่ซ็ใซไธๆฌๅฎ่กใใๆจๆบๅใใใmetrics.csv๏ผ๏ผRunใญใฐ๏ผใซ้็ดใใ"ใใใๅฎ้จใชใผใฑในใใฌใผใฟ"
## Quick Start
```bash
# Install
pip install strataregula-doe-runner
# Execute cases
srd run --cases cases.csv --out metrics.csv
# With parallel execution
srd run --cases cases.csv --out metrics.csv --max-workers 4
# Force re-execution (ignore cache)
srd run --cases cases.csv --out metrics.csv --force
# Dry run (validation only)
srd run --cases cases.csv --dry-run
```
## Exit Codes
| Code | Meaning |
|------|---------|
| 0 | All cases executed successfully, thresholds met |
| 2 | Execution completed but threshold violations detected |
| 3 | I/O error, invalid configuration, or execution failure |
## Cases CSV Contract
### Required Columns
- `case_id`: Unique case identifier
- `backend`: Execution backend (`shell`/`dummy`/`simroute`)
- `cmd_template`: Command template with `{placeholders}`
- `timeout_s`: Timeout in seconds
### Recommended Columns
- `seed`: Random seed for reproducibility
- `retries`: Number of retry attempts
- `resource_group`: Resource grouping for execution
### Metrics Expectation Columns (Optional)
- `expected_p95`, `expected_p99`: Expected metric values
- `threshold_p95`, `threshold_p99`: Threshold limits
### Example cases.csv
```csv
case_id,backend,cmd_template,timeout_s,seed,retries,resource_group,expected_p95,threshold_p95
test-01,shell,"echo 'p95=0.05 p99=0.08 throughput_rps=1000'",10,42,2,default,0.05,0.06
test-02,simroute,"python -m cli_run --seed {seed}",120,123,3,compute,0.10,0.12
```
## Metrics CSV Output
### Required Columns (Fixed Order)
```csv
case_id,status,run_seconds,p95,p99,throughput_rps,errors,ts_start,ts_end
```
### Status Values
- `OK`: Successful execution
- `FAIL`: Execution failed
- `TIMEOUT`: Execution timed out
### Additional Columns
- Performance: `cpu_util`, `mem_peak_mb`, `queue_depth_p95`
- Parameters: `param_*` (input case parameters)
- Extensions: `ext_*`, `tag_*`, `note_*`
## Features
### ๐ Deterministic Execution
- Same input always produces same output
- Fixed column ordering in CSV files
- Consistent number formatting
- LF line endings enforced
### ๐พ Intelligent Caching
- `case_hash` calculated from deterministic case elements
- Automatic cache hit detection
- `--force` flag to override cache
### ๐ Threshold Validation
- Automatic metric threshold checking
- Exit code 2 for threshold violations
- Detailed violation reporting
### ๐ง Multiple Backends
- `shell`: Execute shell commands
- `dummy`: Testing/simulation backend
- `simroute`: Integration with world-simulation (extras dependency)
### ๐ Rich Logging
- JST timezone run logs
- Execution summaries with statistics
- Artifact tracking
- Error reporting
## Environment Variables
- `RUN_LOG_DIR`: Directory for run logs (default: `docs/run`)
- `RUN_LOG_WRITE_COMPAT`: Compatibility mode flag (default: `0`)
## Integration with Strataregula
This plugin integrates with Strataregula through entry points:
```toml
[project.entry-points."strataregula.plugins"]
"doe_runner" = "strataregula_doe_runner.plugin:DOERunnerPlugin"
```
When installed, it's automatically discoverable by the Strataregula framework.
### Plugin Commands
- `execute_cases`: Run cases from CSV
- `validate_cases`: Validate case format
- `get_progress`: Check execution progress
- `export_results`: Export execution results
## Development
```bash
# Clone repository
git clone https://github.com/strataregula/strataregula-doe-runner
cd strataregula-doe-runner
# Install development dependencies
pip install -e ".[dev]"
# Run tests
pytest
# Run with coverage
pytest --cov=strataregula_doe_runner
# Format code
black src tests
isort src tests
# Type checking
mypy src
```
## Examples
See the `examples/` directory for:
- `simple/`: Basic cases.csv example
- `complex/`: Advanced configuration with all features
- `simroute/`: Integration with world-simulation
## License
Apache License 2.0 - see [LICENSE](LICENSE) file.
## Contributing
1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request
## Support
- ๐ [Documentation](https://strataregula-doe-runner.readthedocs.io/)
- ๐ [Issue Tracker](https://github.com/strataregula/strataregula-doe-runner/issues)
- ๐ฌ [Discussions](https://github.com/strataregula/strataregula-doe-runner/discussions)
Raw data
{
"_id": null,
"home_page": null,
"name": "strataregula-doe-runner",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "batch, csv, experiment, metrics, orchestrator, strataregula",
"author": null,
"author_email": "Strataregula Team <team@strataregula.com>",
"download_url": "https://files.pythonhosted.org/packages/fa/41/738568373545f04ade383e274f287753f29c1796af011337eaa48444bdb7/strataregula_doe_runner-0.1.0.tar.gz",
"platform": null,
"description": "# Strataregula - DOE Runner\n\n[](https://github.com/strataregula/strataregula-doe-runner/actions/workflows/test.yml)\n[](https://codecov.io/gh/strataregula/strataregula-doe-runner)\n[](https://pypi.org/project/strataregula-doe-runner/)\n[](https://www.python.org/)\n[](LICENSE)\n[](https://github.com/strataregula/strataregula)\n\nBatch experiment orchestrator for Strataregula ecosystem.\n\n## What is DOE Runner?\n\n**DOE Runner is NOT a Design of Experiments tool.** It is a **batch experiment orchestrator** that:\n\n- \ud83d\udccb Executes predefined cases from `cases.csv`\n- \ud83d\udd00 Manages execution order and parallelism \n- \ud83d\udcca Collects standardized metrics to `metrics.csv`\n- \ud83d\udd12 Provides deterministic, reproducible results\n- \ud83d\udcbe Caches execution results with `case_hash`\n- \ud83d\udcdd Generates comprehensive run logs (JST timezone)\n\n**Definition (1 line)**: cases.csv\u3067\u5b9a\u7fa9\u3057\u305f\u30b1\u30fc\u30b9\u96c6\u5408\u3092\u3001\u6c7a\u5b9a\u8ad6\u7684\u306b\u4e00\u62ec\u5b9f\u884c\u3057\u3001\u6a19\u6e96\u5316\u3055\u308c\u305fmetrics.csv\uff08\uff0bRun\u30ed\u30b0\uff09\u306b\u96c6\u7d04\u3059\u308b\"\u30d0\u30c3\u30c1\u5b9f\u9a13\u30aa\u30fc\u30b1\u30b9\u30c8\u30ec\u30fc\u30bf\"\n\n## Quick Start\n\n```bash\n# Install\npip install strataregula-doe-runner\n\n# Execute cases\nsrd run --cases cases.csv --out metrics.csv\n\n# With parallel execution \nsrd run --cases cases.csv --out metrics.csv --max-workers 4\n\n# Force re-execution (ignore cache)\nsrd run --cases cases.csv --out metrics.csv --force\n\n# Dry run (validation only)\nsrd run --cases cases.csv --dry-run\n```\n\n## Exit Codes\n\n| Code | Meaning |\n|------|---------|\n| 0 | All cases executed successfully, thresholds met |\n| 2 | Execution completed but threshold violations detected |\n| 3 | I/O error, invalid configuration, or execution failure |\n\n## Cases CSV Contract\n\n### Required Columns\n- `case_id`: Unique case identifier\n- `backend`: Execution backend (`shell`/`dummy`/`simroute`)\n- `cmd_template`: Command template with `{placeholders}`\n- `timeout_s`: Timeout in seconds\n\n### Recommended Columns \n- `seed`: Random seed for reproducibility\n- `retries`: Number of retry attempts\n- `resource_group`: Resource grouping for execution\n\n### Metrics Expectation Columns (Optional)\n- `expected_p95`, `expected_p99`: Expected metric values\n- `threshold_p95`, `threshold_p99`: Threshold limits\n\n### Example cases.csv\n```csv\ncase_id,backend,cmd_template,timeout_s,seed,retries,resource_group,expected_p95,threshold_p95\ntest-01,shell,\"echo 'p95=0.05 p99=0.08 throughput_rps=1000'\",10,42,2,default,0.05,0.06\ntest-02,simroute,\"python -m cli_run --seed {seed}\",120,123,3,compute,0.10,0.12\n```\n\n## Metrics CSV Output\n\n### Required Columns (Fixed Order)\n```csv\ncase_id,status,run_seconds,p95,p99,throughput_rps,errors,ts_start,ts_end\n```\n\n### Status Values\n- `OK`: Successful execution\n- `FAIL`: Execution failed\n- `TIMEOUT`: Execution timed out\n\n### Additional Columns\n- Performance: `cpu_util`, `mem_peak_mb`, `queue_depth_p95`\n- Parameters: `param_*` (input case parameters)\n- Extensions: `ext_*`, `tag_*`, `note_*`\n\n## Features\n\n### \ud83d\udd04 Deterministic Execution\n- Same input always produces same output\n- Fixed column ordering in CSV files\n- Consistent number formatting\n- LF line endings enforced\n\n### \ud83d\udcbe Intelligent Caching\n- `case_hash` calculated from deterministic case elements\n- Automatic cache hit detection\n- `--force` flag to override cache\n\n### \ud83d\udcca Threshold Validation\n- Automatic metric threshold checking\n- Exit code 2 for threshold violations\n- Detailed violation reporting\n\n### \ud83d\udd27 Multiple Backends\n- `shell`: Execute shell commands\n- `dummy`: Testing/simulation backend\n- `simroute`: Integration with world-simulation (extras dependency)\n\n### \ud83d\udcdd Rich Logging\n- JST timezone run logs\n- Execution summaries with statistics\n- Artifact tracking\n- Error reporting\n\n## Environment Variables\n\n- `RUN_LOG_DIR`: Directory for run logs (default: `docs/run`)\n- `RUN_LOG_WRITE_COMPAT`: Compatibility mode flag (default: `0`)\n\n## Integration with Strataregula\n\nThis plugin integrates with Strataregula through entry points:\n\n```toml\n[project.entry-points.\"strataregula.plugins\"]\n\"doe_runner\" = \"strataregula_doe_runner.plugin:DOERunnerPlugin\"\n```\n\nWhen installed, it's automatically discoverable by the Strataregula framework.\n\n### Plugin Commands\n- `execute_cases`: Run cases from CSV\n- `validate_cases`: Validate case format \n- `get_progress`: Check execution progress\n- `export_results`: Export execution results\n\n## Development\n\n```bash\n# Clone repository\ngit clone https://github.com/strataregula/strataregula-doe-runner\ncd strataregula-doe-runner\n\n# Install development dependencies\npip install -e \".[dev]\"\n\n# Run tests\npytest\n\n# Run with coverage\npytest --cov=strataregula_doe_runner\n\n# Format code\nblack src tests\nisort src tests\n\n# Type checking\nmypy src\n```\n\n## Examples\n\nSee the `examples/` directory for:\n- `simple/`: Basic cases.csv example\n- `complex/`: Advanced configuration with all features\n- `simroute/`: Integration with world-simulation\n\n## License\n\nApache License 2.0 - see [LICENSE](LICENSE) file.\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Add tests for new functionality\n4. Ensure all tests pass\n5. Submit a pull request\n\n## Support\n\n- \ud83d\udcda [Documentation](https://strataregula-doe-runner.readthedocs.io/)\n- \ud83d\udc1b [Issue Tracker](https://github.com/strataregula/strataregula-doe-runner/issues)\n- \ud83d\udcac [Discussions](https://github.com/strataregula/strataregula-doe-runner/discussions)",
"bugtrack_url": null,
"license": null,
"summary": "Batch experiment orchestrator for Strataregula ecosystem",
"version": "0.1.0",
"project_urls": {
"Documentation": "https://strataregula-doe-runner.readthedocs.io/",
"Homepage": "https://github.com/strataregula/strataregula-doe-runner",
"Issues": "https://github.com/strataregula/strataregula-doe-runner/issues",
"Repository": "https://github.com/strataregula/strataregula-doe-runner.git"
},
"split_keywords": [
"batch",
" csv",
" experiment",
" metrics",
" orchestrator",
" strataregula"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "418f67a4cb49d69d4371e8378bb4ece564d993d52452e25a3d41cdf3dfa6f230",
"md5": "d414c076615f3e23e6b249b9b0738f14",
"sha256": "acd94c851dfdaffee604701c316c0c988ca96900da500c9253fec38aae0bf679"
},
"downloads": -1,
"filename": "strataregula_doe_runner-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d414c076615f3e23e6b249b9b0738f14",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 28517,
"upload_time": "2025-08-25T21:21:01",
"upload_time_iso_8601": "2025-08-25T21:21:01.637846Z",
"url": "https://files.pythonhosted.org/packages/41/8f/67a4cb49d69d4371e8378bb4ece564d993d52452e25a3d41cdf3dfa6f230/strataregula_doe_runner-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "fa41738568373545f04ade383e274f287753f29c1796af011337eaa48444bdb7",
"md5": "0e16f7f8b33b8696dc577907cec2ed8a",
"sha256": "bfdcc19881099080af3039556b26a4ce27d3ecb93814c7d01d8dca3e229c52f6"
},
"downloads": -1,
"filename": "strataregula_doe_runner-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "0e16f7f8b33b8696dc577907cec2ed8a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 25104,
"upload_time": "2025-08-25T21:21:02",
"upload_time_iso_8601": "2025-08-25T21:21:02.917158Z",
"url": "https://files.pythonhosted.org/packages/fa/41/738568373545f04ade383e274f287753f29c1796af011337eaa48444bdb7/strataregula_doe_runner-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-25 21:21:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "strataregula",
"github_project": "strataregula-doe-runner",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "strataregula-doe-runner"
}
JSON
