# IDOL - Incremental DAG Optimization for Learning
[](https://github.com/dongahn/auto-taskopt/actions/workflows/ci.yml)
[](https://www.python.org/downloads/)
[](https://opensource.org/licenses/MIT)
[](https://github.com/astral-sh/ruff)
Transform debug traces into validated golden datasets for ML training pipelines.
## Overview
IDOL is a CLI tool that processes debug trace JSON files to create high-quality labeled datasets through:
1. **Automatic candidate generation** using task-specific heuristics
2. **Human review interface** for label validation and correction
3. **Dataset freezing** with validation and optional train/holdout splits
## Features
- 🎯 **Deterministic ID generation** ensures idempotent operations
- 📝 **JSONL format** enables git-friendly diffs and streaming processing
- ⚡ **Memory-efficient** parsing of large JSON files
- 🔍 **8 task-specific heuristics** for automatic labeling
- ✅ **Interactive review** with accept/negate/edit options
- ❄️ **Dataset validation** before freezing
- 📊 **Train/holdout splits** for ML evaluation
## Installation
```bash
# Clone the repository
git clone <repository-url>
cd auto-taskopt
# Install dependencies
pip install -e .
# Or install from requirements
pip install -r requirements.txt
```
## Quick Start
```bash
# 1. Harvest candidates from debug traces
idol harvest examples/20250912_121325/
# 2. Review candidates for a specific task
idol review --task gpu_hw_analysis
# 3. Freeze validated dataset
idol freeze --task gpu_hw_analysis
# Check status
idol status
```
## Usage
### Harvest: Generate Candidates
Process debug trace JSON files to extract task executions and generate candidates:
```bash
# Process a single file
idol harvest trace.json
# Process all JSON files in a directory
idol harvest examples/
# Verbose output
idol harvest examples/ -v
```
The harvest command:
- Parses debug traces to extract task executions
- Generates stable SHA1-based IDs
- Applies task-specific heuristics for auto-labeling
- Saves candidates to `rca_gold/candidates/<task>.jsonl`
### Review: Validate Labels
Interactively review and correct auto-generated labels:
```bash
# Review all pending candidates for a task
idol review --task gpu_hw_analysis
# Review only first 10 items
idol review --task logs_analysis --max-items 10
```
Review options:
- `[a]` Accept auto label
- `[n]` Negate (mark as 'no_issue')
- `[e]` Edit (provide custom JSON)
- `[s]` Skip
- `[q]` Quit
### Freeze: Create Final Datasets
Merge candidates and overrides into validated datasets:
```bash
# Freeze a specific task
idol freeze --task gpu_hw_analysis
# Freeze all tasks
idol freeze
# Create train/holdout split (80/20)
idol freeze --task final_analysis --holdout
# Skip validation
idol freeze --task network_analysis --no-validate
```
Output files:
- `rca_gold/frozen/<task>.json` - Main dataset
- `rca_gold/frozen/<task>.holdout.json` - Holdout set (if requested)
### Additional Commands
```bash
# Show current status
idol status
# Clean all generated data
idol clean
```
## Supported Tasks
IDOL supports 8 task types with specialized heuristics:
1. **job_info** - Job success/failure status
2. **gpu_hw_analysis** - GPU hardware failure detection (XIDs/SXIDs)
3. **logs_analysis** - Software errors and numerical instability
4. **health_checks** - System health status
5. **scheduler_analysis** - Scheduler termination analysis
6. **network_analysis** - Network/InfiniBand issues
7. **storage_analysis** - Storage I/O and metadata spikes
8. **final_analysis** - Root cause determination with confidence
## Data Format
### Candidate Record (JSONL)
```json
{
"id": "abc123def456",
"task": "gpu_hw_analysis",
"input": {
"job_id": 4824587,
"attempt": 0,
"key_findings": "XID 48 hardware failure on node-001",
"tool_calls": [{"name": "job_summary", "args": {}}]
},
"auto": {
"result": "confirmed_hw_failure",
"evidence": "XIDs: [48]"
},
"status": "pending"
}
```
### Frozen Dataset (JSON)
```json
[
{
"id": "abc123def456",
"input": {
"job_id": 4824587,
"attempt": 0,
"key_findings": "...",
"tool_calls": []
},
"gold": {
"result": "confirmed_hw_failure",
"evidence": "Manual verification"
}
}
]
```
## Project Structure
```
auto-taskopt/
├── src/idol/
│ ├── cli.py # CLI commands
│ ├── harvester.py # Candidate generation
│ ├── reviewer.py # Human review interface
│ ├── freezer.py # Dataset finalization
│ ├── heuristics.py # Task-specific rules
│ ├── models.py # Data models
│ └── utils.py # Utilities
├── tests/ # Test suite
├── examples/ # Sample debug traces
└── rca_gold/ # Generated data
├── candidates/ # JSONL candidates
├── overrides/ # Human corrections
└── frozen/ # Final datasets
```
## Development
### Running Tests
```bash
# Run all tests
pytest tests/ -v
# Run with coverage
pytest tests/ --cov=src/idol --cov-report=term-missing
# Run specific test module
pytest tests/test_heuristics.py -v
```
### Code Quality
```bash
# Lint code
ruff check src/ --fix
# Type checking
mypy src/idol/ --strict
```
## Performance
- **Harvest**: Process 100 traces in <30 seconds
- **Review**: Display each candidate in <100ms
- **Freeze**: Validate 10,000 records in <5 seconds
## Contributing
1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request
## License
MIT License - see LICENSE file for details
Raw data
{
"_id": null,
"home_page": null,
"name": "idol-cli",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "machine-learning, data-labeling, golden-dataset, cli",
"author": null,
"author_email": "IDOL Team <idol@example.com>",
"download_url": "https://files.pythonhosted.org/packages/8e/94/e63b03bbf23c1fddf5fa6eaa2527701ceb4315aa8ce1cc2298b62baa52c7/idol_cli-0.1.0.tar.gz",
"platform": null,
"description": "# IDOL - Incremental DAG Optimization for Learning\n\n[](https://github.com/dongahn/auto-taskopt/actions/workflows/ci.yml)\n[](https://www.python.org/downloads/)\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/astral-sh/ruff)\n\nTransform debug traces into validated golden datasets for ML training pipelines.\n\n## Overview\n\nIDOL is a CLI tool that processes debug trace JSON files to create high-quality labeled datasets through:\n1. **Automatic candidate generation** using task-specific heuristics\n2. **Human review interface** for label validation and correction \n3. **Dataset freezing** with validation and optional train/holdout splits\n\n## Features\n\n- \ud83c\udfaf **Deterministic ID generation** ensures idempotent operations\n- \ud83d\udcdd **JSONL format** enables git-friendly diffs and streaming processing\n- \u26a1 **Memory-efficient** parsing of large JSON files\n- \ud83d\udd0d **8 task-specific heuristics** for automatic labeling\n- \u2705 **Interactive review** with accept/negate/edit options\n- \u2744\ufe0f **Dataset validation** before freezing\n- \ud83d\udcca **Train/holdout splits** for ML evaluation\n\n## Installation\n\n```bash\n# Clone the repository\ngit clone <repository-url>\ncd auto-taskopt\n\n# Install dependencies\npip install -e .\n\n# Or install from requirements\npip install -r requirements.txt\n```\n\n## Quick Start\n\n```bash\n# 1. Harvest candidates from debug traces\nidol harvest examples/20250912_121325/\n\n# 2. Review candidates for a specific task\nidol review --task gpu_hw_analysis\n\n# 3. Freeze validated dataset\nidol freeze --task gpu_hw_analysis\n\n# Check status\nidol status\n```\n\n## Usage\n\n### Harvest: Generate Candidates\n\nProcess debug trace JSON files to extract task executions and generate candidates:\n\n```bash\n# Process a single file\nidol harvest trace.json\n\n# Process all JSON files in a directory\nidol harvest examples/\n\n# Verbose output\nidol harvest examples/ -v\n```\n\nThe harvest command:\n- Parses debug traces to extract task executions\n- Generates stable SHA1-based IDs\n- Applies task-specific heuristics for auto-labeling\n- Saves candidates to `rca_gold/candidates/<task>.jsonl`\n\n### Review: Validate Labels\n\nInteractively review and correct auto-generated labels:\n\n```bash\n# Review all pending candidates for a task\nidol review --task gpu_hw_analysis\n\n# Review only first 10 items\nidol review --task logs_analysis --max-items 10\n```\n\nReview options:\n- `[a]` Accept auto label\n- `[n]` Negate (mark as 'no_issue')\n- `[e]` Edit (provide custom JSON)\n- `[s]` Skip\n- `[q]` Quit\n\n### Freeze: Create Final Datasets\n\nMerge candidates and overrides into validated datasets:\n\n```bash\n# Freeze a specific task\nidol freeze --task gpu_hw_analysis\n\n# Freeze all tasks\nidol freeze\n\n# Create train/holdout split (80/20)\nidol freeze --task final_analysis --holdout\n\n# Skip validation\nidol freeze --task network_analysis --no-validate\n```\n\nOutput files:\n- `rca_gold/frozen/<task>.json` - Main dataset\n- `rca_gold/frozen/<task>.holdout.json` - Holdout set (if requested)\n\n### Additional Commands\n\n```bash\n# Show current status\nidol status\n\n# Clean all generated data\nidol clean\n```\n\n## Supported Tasks\n\nIDOL supports 8 task types with specialized heuristics:\n\n1. **job_info** - Job success/failure status\n2. **gpu_hw_analysis** - GPU hardware failure detection (XIDs/SXIDs)\n3. **logs_analysis** - Software errors and numerical instability\n4. **health_checks** - System health status\n5. **scheduler_analysis** - Scheduler termination analysis\n6. **network_analysis** - Network/InfiniBand issues\n7. **storage_analysis** - Storage I/O and metadata spikes\n8. **final_analysis** - Root cause determination with confidence\n\n## Data Format\n\n### Candidate Record (JSONL)\n```json\n{\n \"id\": \"abc123def456\",\n \"task\": \"gpu_hw_analysis\",\n \"input\": {\n \"job_id\": 4824587,\n \"attempt\": 0,\n \"key_findings\": \"XID 48 hardware failure on node-001\",\n \"tool_calls\": [{\"name\": \"job_summary\", \"args\": {}}]\n },\n \"auto\": {\n \"result\": \"confirmed_hw_failure\",\n \"evidence\": \"XIDs: [48]\"\n },\n \"status\": \"pending\"\n}\n```\n\n### Frozen Dataset (JSON)\n```json\n[\n {\n \"id\": \"abc123def456\",\n \"input\": {\n \"job_id\": 4824587,\n \"attempt\": 0,\n \"key_findings\": \"...\",\n \"tool_calls\": []\n },\n \"gold\": {\n \"result\": \"confirmed_hw_failure\",\n \"evidence\": \"Manual verification\"\n }\n }\n]\n```\n\n## Project Structure\n\n```\nauto-taskopt/\n\u251c\u2500\u2500 src/idol/\n\u2502 \u251c\u2500\u2500 cli.py # CLI commands\n\u2502 \u251c\u2500\u2500 harvester.py # Candidate generation\n\u2502 \u251c\u2500\u2500 reviewer.py # Human review interface\n\u2502 \u251c\u2500\u2500 freezer.py # Dataset finalization\n\u2502 \u251c\u2500\u2500 heuristics.py # Task-specific rules\n\u2502 \u251c\u2500\u2500 models.py # Data models\n\u2502 \u2514\u2500\u2500 utils.py # Utilities\n\u251c\u2500\u2500 tests/ # Test suite\n\u251c\u2500\u2500 examples/ # Sample debug traces\n\u2514\u2500\u2500 rca_gold/ # Generated data\n \u251c\u2500\u2500 candidates/ # JSONL candidates\n \u251c\u2500\u2500 overrides/ # Human corrections\n \u2514\u2500\u2500 frozen/ # Final datasets\n```\n\n## Development\n\n### Running Tests\n\n```bash\n# Run all tests\npytest tests/ -v\n\n# Run with coverage\npytest tests/ --cov=src/idol --cov-report=term-missing\n\n# Run specific test module\npytest tests/test_heuristics.py -v\n```\n\n### Code Quality\n\n```bash\n# Lint code\nruff check src/ --fix\n\n# Type checking\nmypy src/idol/ --strict\n```\n\n## Performance\n\n- **Harvest**: Process 100 traces in <30 seconds\n- **Review**: Display each candidate in <100ms\n- **Freeze**: Validate 10,000 records in <5 seconds\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Add tests for new functionality\n4. Ensure all tests pass\n5. Submit a pull request\n\n## License\n\nMIT License - see LICENSE file for details\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "IDOL - Incremental DAG Optimization for Learning CLI",
"version": "0.1.0",
"project_urls": {
"Bug Reports": "https://github.com/dongahn/auto-taskopt/issues",
"Homepage": "https://github.com/dongahn/auto-taskopt",
"Source": "https://github.com/dongahn/auto-taskopt"
},
"split_keywords": [
"machine-learning",
" data-labeling",
" golden-dataset",
" cli"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "23da1f0cc2dc197440addd9308716cb94ce1839f34c44b2128309a2ab8ee35b1",
"md5": "f001a346e0a403a0c476c6177fdae379",
"sha256": "9261ea8b6981cf7c91b359aced3b20884fed6de747d5de9346e5d4df830f93fa"
},
"downloads": -1,
"filename": "idol_cli-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f001a346e0a403a0c476c6177fdae379",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 20553,
"upload_time": "2025-09-12T23:17:02",
"upload_time_iso_8601": "2025-09-12T23:17:02.865034Z",
"url": "https://files.pythonhosted.org/packages/23/da/1f0cc2dc197440addd9308716cb94ce1839f34c44b2128309a2ab8ee35b1/idol_cli-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "8e94e63b03bbf23c1fddf5fa6eaa2527701ceb4315aa8ce1cc2298b62baa52c7",
"md5": "94d8a1383a2395ec32d3a9b99bb1861b",
"sha256": "08b5e282fc711b55b33d71c4aebce84ae25c5a452d6e63b88d6ae04e0a5e0729"
},
"downloads": -1,
"filename": "idol_cli-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "94d8a1383a2395ec32d3a9b99bb1861b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 24703,
"upload_time": "2025-09-12T23:17:03",
"upload_time_iso_8601": "2025-09-12T23:17:03.860785Z",
"url": "https://files.pythonhosted.org/packages/8e/94/e63b03bbf23c1fddf5fa6eaa2527701ceb4315aa8ce1cc2298b62baa52c7/idol_cli-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-12 23:17:03",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dongahn",
"github_project": "auto-taskopt",
"github_not_found": true,
"lcname": "idol-cli"
}
JSON
