# AI-First Toolkit: LLM-Powered Automation
[](https://badge.fury.io/py/llm-ci-runner) [](https://github.com/Nantero1/ai-first-devops-toolkit/actions/workflows/ci.yml) [](https://github.com/Nantero1/ai-first-devops-toolkit/actions/workflows/unit-tests.yml) [](https://htmlpreview.github.io/?https://github.com/Nantero1/ai-first-devops-toolkit/blob/python-coverage-comment-action-data/htmlcov/index.html) [](https://github.com/Nantero1/ai-first-devops-toolkit/actions/workflows/github-code-scanning/codeql)
> **🚀 The Future of DevOps is AI-First**
> This toolkit represents a step toward [AI-First DevOps](https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html) - where intelligent automation handles the entire development lifecycle. Built for teams ready to embrace the exponential productivity gains of AI-powered development. Please read [the blog post](https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html) for more details on the motivation.
## TLDR: What This Tool Does
**Purpose**: Zero-friction LLM integration for pipelines with **100% guaranteed schema compliance**. This is your foundation for AI-first integration practices.
**Perfect For**:
- 🤖 **AI-Generated Code Reviews**: Automated PR analysis with structured findings
- 📝 **Intelligent Documentation**: Generate changelogs, release notes, and docs automatically
- 🔍 **Security Analysis**: AI-powered vulnerability detection with structured reports
- 🎯 **Quality Gates**: Enforce standards through AI-driven validation
- 🚀 **Autonomous Development**: Enable AI agents to make decisions in your pipelines
- 🎯 **JIRA Ticket Updates**: Update JIRA tickets based on LLM output
- 🔗 **Unlimited Integration Possibilities**: Chain it multiple times and use as glue code in your tool stack
---
### Simple structured output example
```bash
# Install and use immediately
pip install llm-ci-runner
llm-ci-runner --input-file examples/02-devops/pr-description/input.json --schema-file examples/02-devops/pr-description/schema.json
```
## The AI-First Development Revolution
This toolkit embodies the principles outlined in [Building AI-First DevOps](https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html):
| Traditional DevOps | AI-First DevOps (This Tool) |
|-------------------|----------------------------|
| Manual code reviews | 🤖 AI-powered reviews with structured findings |
| Human-written documentation | 📝 AI-generated docs with guaranteed consistency |
| Reactive security scanning | 🔍 Proactive AI security analysis |
| Manual quality gates | 🎯 AI-driven validation with schema enforcement |
| Linear productivity | 📈 Exponential gains through intelligent automation |
## Features
- 🎯 **100% Schema Enforcement**: Your pipeline never gets invalid data. Token-level schema enforcement with guaranteed compliance
- 🔄 **Resilient execution**: Retries with exponential back-off and jitter plus a clear exception hierarchy keep transient cloud faults from breaking your CI.
- 🚀 **Zero-Friction CLI**: Single script, minimal configuration for pipeline integration and automation
- 🔐 **Enterprise Security**: Azure RBAC via DefaultAzureCredential with fallback to API Key
- 📦 **CI-friendly CLI**: Stateless command that reads JSON/YAML, writes JSON/YAML, and exits with proper codes
- 🎨 **Beautiful Logging**: Rich console output with timestamps and colors
- 📁 **File-based I/O**: CI/CD friendly with JSON/YAML input/output
- 📋 **Template-Driven Workflows**: Handlebars and Jinja2 templates with YAML variables for dynamic prompt generation
- 📄 **YAML Support**: Use YAML for schemas, input files, and output files - more readable than JSON
- 🔧 **Simple & Extensible**: Easy to understand and modify for your specific needs
- 🤖 **Semantic Kernel foundation**: async, service-oriented design ready for skills, memories, orchestration, and future model upgrades
- 📚 **Documentation**: Comprehensive documentation for all features and usage examples. Use your semantic kernel skills to extend the functionality.
- 🧑⚖️ **Acceptance Tests**: pytest framework with the LLM-as-Judge pattern for quality gates. Test your scripts before you run them in production.
- 💰 **Coming soon**: token usage and cost estimation appended to each result for budgeting and optimisation
## 🚀 The Only Enterprise AI DevOps Tool That Delivers RBAC Security, Robustness and Simplicity
**LLM-CI-Runner stands alone in the market** as the only tool combining **100% schema enforcement**, **enterprise RBAC authentication**, and robust **Semantic Kernel integration with templates** in a single CLI solution. **No other tool delivers all three critical enterprise requirements together**.
## Installation
```bash
pip install llm-ci-runner
```
That's it! No complex setup, no dependency management - just install and use. Perfect for CI/CD pipelines and local development.
## Quick Start
### 1. Install from PyPI
```bash
pip install llm-ci-runner
```
### 2. Set Environment Variables
**Azure OpenAI (Priority 1):**
```bash
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_MODEL="gpt-4.1-nano" # or any other GPT deployment name
export AZURE_OPENAI_API_VERSION="2024-12-01-preview" # Optional
```
**OpenAI (Fallback):**
```bash
export OPENAI_API_KEY="your-very-secret-api-key"
export OPENAI_CHAT_MODEL_ID="gpt-4.1-nano" # or any OpenAI model
export OPENAI_ORG_ID="org-your-org-id" # Optional
```
**Authentication Options:**
- **Azure RBAC (Recommended)**: Uses `DefaultAzureCredential` for Azure RBAC authentication - no API key needed! See [Microsoft Docs](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for setup.
- **Azure API Key**: Set `AZURE_OPENAI_API_KEY` environment variable if not using RBAC.
- **OpenAI API Key**: Required for OpenAI fallback when Azure is not configured.
**Priority**: Azure OpenAI takes priority when both Azure and OpenAI environment variables are present.
### 3a. Basic Usage
```bash
# Simple chat example
llm-ci-runner --input-file examples/01-basic/simple-chat/input.json
# With structured output schema
llm-ci-runner \
--input-file examples/01-basic/sentiment-analysis/input.json \
--schema-file examples/01-basic/sentiment-analysis/schema.json
# Custom output file
llm-ci-runner \
--input-file examples/02-devops/pr-description/input.json \
--schema-file examples/02-devops/pr-description/schema.json \
--output-file pr-analysis.json
# YAML input files (alternative to JSON)
llm-ci-runner \
--input-file config.yaml \
--schema-file schema.yaml \
--output-file result.yaml
```
### 3b. Template-Based Workflows
**Dynamic prompt generation with YAML, Handlebars or Jinja2 templates:**
```bash
# Handlebars template example
llm-ci-runner \
--template-file examples/05-templates/handlebars-template/template.hbs \
--template-vars examples/05-templates/handlebars-template/template-vars.yaml \
--schema-file examples/05-templates/handlebars-template/schema.yaml \
--output-file handlebars-result.yaml
# Or using Jinja2 templates
llm-ci-runner \
--template-file examples/05-templates/jinja2-template/template.j2 \
--template-vars examples/05-templates/jinja2-template/template-vars.yaml \
--schema-file examples/05-templates/jinja2-template/schema.yaml \
--output-file jinja2-result.yaml
```
For more examples see the [examples directory](https://github.com/Nantero1/ai-first-devops-toolkit/tree/main/examples/05-templates).
**Benefits of Template Approach:**
- 🎯 **Reusable Templates**: Create once, use across multiple scenarios
- 📝 **YAML Configuration**: More readable than JSON for complex setups
- 🔄 **Dynamic Content**: Variables and conditional rendering
- 🚀 **CI/CD Ready**: Perfect for parameterized pipeline workflows
### 4. Development Setup (Optional)
For contributors or advanced users who want to modify the source:
```bash
# Install UV if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh
# Clone and install for development
git clone https://github.com/Nantero1/ai-first-devops-toolkit.git
cd ai-first-devops-toolkit
uv sync
# Run from source
uv run llm-ci-runner --input-file examples/01-basic/simple-chat/input.json
```
## Real-World Examples
You can explore the **[examples directory](https://github.com/Nantero1/ai-first-devops-toolkit/tree/main/examples)** for a complete collection of self-contained examples organized by category.
For comprehensive real-world CI/CD scenarios, see **[examples/uv-usage-example.md](https://github.com/Nantero1/ai-first-devops-toolkit/blob/main/examples/uv-usage-example.md)**. Some possibilities:
1. 🤖 AI-generated PR review – automated pull request analysis with structured review findings
2. 📝 Commit summarizer – convert commit logs into concise release notes
3. 🔍 Vulnerability scanner – map code vulnerabilities to OWASP standards with actionable remediation
4. 🎯 Quality gate enforcer – validate build artifacts against schema-defined quality criteria
5. 🏦 Loan application analyzer – transform free-text loan applications into Basel-III risk-model inputs
6. 💼 Consulting report generator – convert meeting notes into itemized Statement of Work deliverables
7. 🏛️ Legal contract parser – extract clauses and compute risk scores from contract documents
8. 📝 Court opinion digest – summarize judicial opinions into structured precedent and citation graphs
9. 🏥 Patient intake processor – build HL7/FHIR-compliant patient records from free-form intake forms
10. 📈 Earnings call analyzer – convert transcripts into KPI dashboards for financial performance review
11. 🔍 Code-review bot – scan commits and PRs to produce OWASP-mapped vulnerability reports
12. 🎯 Incident post-mortem summarizer – generate structured root cause analysis and corrective action plans
13. 📊 Regulatory compliance reporter – synthesize regulatory texts into structured compliance checklists
14. 💼 Financial audit note handler – convert audit commentary into ledger-ready journal entries
15. 🔍 Vulnerability scanner – map code vulnerabilities to OWASP standards with actionable remediation
16. 🎯 Quality gate enforcer – validate build artifacts against schema-defined quality criteria
17. 🏦 Loan application analyzer – transform free-text loan applications into Basel-III risk-model inputs
18. 💼 Consulting report generator – convert meeting notes into itemized Statement of Work deliverables
19. 🏛️ Legal contract parser – extract clauses and compute risk scores from contract documents
20. 📝 Court opinion digest – summarize judicial opinions into structured precedent and citation graphs
21. 🏥 Patient intake processor – build HL7/FHIR-compliant patient records from free-form intake forms
22. 📈 Earnings call analyzer – convert transcripts into KPI dashboards for financial performance review
23. 🔍 Code-review bot – scan commits and PRs to produce OWASP-mapped vulnerability reports
24. 🎯 Incident post-mortem summarizer – generate structured root cause analysis and corrective action plans
25. 📊 Regulatory compliance reporter – synthesize regulatory texts into structured compliance checklists
26. 💼 Financial audit note handler – convert audit commentary into ledger-ready journal entries
27. 🔧 Technical review assistant – output structured code review reports with clear action items
28. 🏥 Doctor dictation converter – transform verbal notes into ICD-10 coded encounter records
29. 🏛️ Legal discovery summarizer – extract key issues and risks from large document sets
30. 📁 Manufacturing defect analyzer – build 8D corrective-action records from production issue notes
31. 💹 Budget variance analyzer – summarize financial reports into detailed KPI and variance analyses
32. 🖥️ Ticket triage assistant – prioritize technical support tickets with automated incident classification
33. 🏦 Compliance transformer – create structured Basel reports from raw regulatory text
34. 📊 Credit risk evaluator – convert customer feedback into quantifiable risk scores
35. 💰 Investor memo summarizer – distill strategic memos into pitch-deck bullet points
36. 🛡️ Cyber threat mapper – translate security alerts into MITRE ATT&CK mapped incident reports
37. 👷 Equipment maintenance scheduler – analyze sensor logs to generate predictive maintenance reports
38. 🫀 Health history compiler – produce structured patient histories from narrative medical notes
39. 🛑 Safety inspection checker – transform inspection narratives into OSHA citation checklists
40. 🏥 Radiology result formatter – convert radiology reports into SNOMED-coded JSON outputs
41. 📝 Insurance claim analyzer – structure claim narratives into automated claim assessments
42. 💼 Contract review summarizer – extract risk factors and key dates from legal contracts
43. 🔍 Fraud detector – transform analyst notes into SAR (Suspicious Activity Report) JSON objects
44. 🏛️ Policy impact assessor – convert policy proposals into stakeholder impact matrices
45. 🏭 Production incident reporter – build actionable recovery plans from factory incident logs
46. 📝 Documentation updater – generate schema-compliant technical documentation automatically
47. 🔄 API diff analyzer – produce backward-compatibility risk reports from API specification changes
48. 📊 Financial forecaster – summarize financial reports into structured cash-flow and projection objects
49. 🔧 Deployment log analyzer – convert rollout logs into performance and downtime metrics
50. 🛒 E-commerce sentiment analyzer – tag customer reviews with sentiment and key product features
51. 🎙️ Meeting minute extractor – transform recorded meetings into action items and follow-up tasks
52. 📝 Sprint retrospective summarizer – generate improvement plans from agile team discussions
53. 🏥 Clinical trial data packager – automatically structure clinical notes for FDA-submission
54. 🏢 Employee feedback analyzer – convert free-text feedback into HR insights and action checklists
55. 🛠️ Process efficiency reporter – output production logs into structured performance metrics
56. 🏛️ Legal bill auditor – transform billing details into itemized expense and compliance reports
57. 📦 Automated inventory trigger – build reordering reports from warehouse inventory logs
58. 🧾 Receipt processor – convert OCR receipts into ledger-ready accounting entries
59. 🏦 Mortgage eligibility assessor – analyze mortgage applications to generate risk and eligibility scores
60. 🚧 Infrastructure incident analyst – summarize log files into detailed RCAs and incident timelines
61. 🏛️ Regulatory update tracker – generate structured compliance action items from updated guidelines
62. 📝 Board meeting summarizer – extract key decisions and action items from meeting transcripts
63. 🔍 Vulnerability risk assessor – create remediation plans by mapping findings to risk frameworks
64. 💼 Legal email analyzer – extract key issues and deadlines from email threads for legal review
65. 🏥 Prescription manager – transform handwritten prescription notes into structured medication lists
66. 🖥️ Git log analyzer – generate detailed changelogs from version control commit histories
67. 📋 SOP generator – create standard operating procedures with checklist items from process descriptions
68. 🎯 PR triage tool – score and tag pull requests by urgency and impact automatically
69. 🏦 Audit finding summarizer – convert audit observations into structured compliance and risk reports
70. 📈 Market trend analyzer – synthesize marketing data into structured trend forecasting objects
71. 🧑💼 Proposal evaluator – produce structured scoring and evaluation criteria from project proposals
72. 🏢 Operations dashboard creator – translate facility logs into productivity and efficiency metrics
73. 🏥 Lab result organizer – build structured diagnostic tables from laboratory results
74. 💡 Innovation evaluator – compile ideation logs into cost-benefit structured analyses
75. 🏛️ Judicial ruling summarizer – generate concise, structured digests from court rulings
76. 🔧 Commit changelog generator – extract impactful changes from commit logs for release summaries
77. 🏭 Production yield analyzer – produce reports on output statistics and downtime from factory logs
78. 💳 Fraud alert generator – transform risk signals into automated CVSS-scored alerts
79. 📝 Regulatory filing assistant – structure raw regulatory data for seamless filing and compliance tracking
80. 👩⚕️ Clinical observation compiler – convert medical research notes into structured clinical data entries
81. 🚀 Deployment success reporter – summarize production rollouts with performance metrics and KPIs
82. 🏦 Mortgage risk evaluator – process mortgage files into detailed risk scoring and eligibility summaries
83. 💼 Contract amendment monitor – track version changes and compliance updates in amended contracts
84. 🏥 Vital signs monitor – generate alert reports from patient vital signs and anomaly detection
85. 🔐 IT security auditor – convert access logs into structured audit and compliance reports
86. 🚧 Incident ticket classifier – generate detailed RCA reports and automated ticket categorizations
87. 🏛️ Governance mapper – produce structured mappings of internal policies to regulatory frameworks
88. 🏢 Onboarding compliance checker – convert training logs into automated compliance and checklist trackers
89. 📝 Data breach notifier – build structured breach incident reports with remediation plans
90. 🏦 Teller performance analyzer – transform shift logs into performance and error analysis reports
91. 💼 Contract risk assessor – generate automated legal risk memos from detailed contract reviews
92. 🛠️ Bug report classifier – categorize issue reports by severity and produce remediation plans
93. 🏥 Appointment summarizer – convert appointment notes into structured follow-up recommendations
94. 🔄 Data migration manifest – output ETL mapping details into a structured transformation record
95. 🚀 Post-release analyst – synthesize customer feedback into performance improvement metrics
96. 🏭 Equipment efficiency evaluator – analyze production logs to predict maintenance needs and cost analysis
97. 🕵️ Fraud case reporter – compile investigative notes into structured fraud case summaries
98. 🏛️ Compliance checklist generator – map internal controls to GDPR or other frameworks in structured reports
99. 👨💻 Diff summarizer – automatically generate summaries of code differences for peer review
100. 📄 Patent claim comparator – produce novelty and prior art comparison tables from patent texts
101. 🔍 Cyber incident analyzer – structure incident narratives into threat intelligence and remediation guides
102. 🛡️ Security audit mapper – create control maps aligned with NIST frameworks from audit notes
103. 🏦 Portfolio risk analyzer – transform investment notes into performance and risk metric summaries
104. 📊 Stress test reporter – compile financial stress test scenarios into structured risk reports
105. 📝 Meeting action tracker – extract decisions and assign tasks from meeting minutes
106. 🛠️ DevOps runbook creator – produce actionable standard operating procedures from runbook logs
107. 🚚 Supply chain optimizer – generate delay forecasts and automated inventory suggestions from logistics notes
108. ⚙️ Process improvement recommender – convert operational logs into structured efficiency recommendations
109. 👮 Compliance reporter – map internal governance policies to GDPR and similar frameworks
110. 🌐 API performance optimizer – analyze API usage logs to generate optimization and performance metrics
111. 🛠️ Legacy system analyzer – assess legacy code bases and produce migration impact reports
112. 🧩 Unstructured anything → your bespoke schema-validated JSON
## Input Formats
### Traditional JSON Input
```json
{
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Your task description here"
}
],
"context": {
"session_id": "optional-session-id",
"metadata": {
"any": "additional context"
}
}
}
```
### YAML Input
```yaml
messages:
- role: system
content: "You are a helpful assistant."
- role: user
content: "Your task description here"
context:
session_id: "optional-session-id"
metadata:
any: "additional context"
```
### Template-Based Input
**Handlebars Template** (`template.hbs`):
```handlebars
<message role="system">
You are an expert {{expertise.domain}} engineer.
Focus on {{expertise.focus_areas}}.
</message>
<message role="user">
Analyze this {{task.type}}:
{{#each task.items}}
- {{this}}
{{/each}}
Requirements: {{task.requirements}}
</message>
```
**Jinja2 Template** (`template.j2`):
```jinja2
<message role="system">
You are an expert {{expertise.domain}} engineer.
Focus on {{expertise.focus_areas}}.
</message>
<message role="user">
Analyze this {{task.type}}:
{% for item in task.items %}
- {{item}}
{% endfor %}
Requirements: {{task.requirements}}
</message>
```
**Template Variables** (`vars.yaml`):
```yaml
expertise:
domain: "DevOps"
focus_areas: "security, performance, maintainability"
task:
type: "pull request"
items:
- "Changed authentication logic"
- "Updated database queries"
- "Added input validation"
requirements: "Focus on security vulnerabilities"
```
## Structured Outputs with 100% Schema Enforcement
When you provide a `--schema-file`, the runner guarantees perfect schema compliance:
```bash
llm-ci-runner \
--input-file examples/01-basic/sentiment-analysis/input.json \
--schema-file examples/01-basic/sentiment-analysis/schema.json
```
**Note**: Output defaults to `result.json`. Use `--output-file custom-name.json` for custom output files.
**Supported Schema Features**:
✅ String constraints (enum, minLength, maxLength, pattern)
✅ Numeric constraints (minimum, maximum, multipleOf)
✅ Array constraints (minItems, maxItems, items type)
✅ Required fields enforced at generation time
✅ Type validation (string, number, integer, boolean, array)
## CI/CD Integration
### GitHub Actions Example
```yaml
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install LLM CI Runner
run: pip install llm-ci-runner
- name: Generate PR Review with Templates
run: |
llm-ci-runner \
--template-file .github/templates/pr-review.j2 \
--template-vars pr-context.yaml \
--schema-file .github/schemas/pr-review.yaml \
--output-file pr-analysis.yaml
env:
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
AZURE_OPENAI_MODEL: ${{ secrets.AZURE_OPENAI_MODEL }}
```
For complete CI/CD examples, see **[examples/uv-usage-example.md](https://github.com/Nantero1/ai-first-devops-toolkit/blob/main/examples/uv-usage-example.md)**.
## Authentication
**Azure OpenAI**: Uses Azure's `DefaultAzureCredential` supporting:
- Environment variables (local development)
- Managed Identity (recommended for Azure CI/CD)
- Azure CLI (local development)
- Service Principal (non-Azure CI/CD)
**OpenAI**: Uses API key authentication with optional organization ID.
## Testing
We maintain comprehensive test coverage with **100% success rate**:
```bash
# For package users - install test dependencies
pip install llm-ci-runner[dev]
# For development - install from source with test dependencies
uv sync --group dev
# Run specific test categories
pytest tests/unit/ -v # 70 unit tests
pytest tests/integration/ -v # End-to-end examples
pytest acceptance/ -v # LLM-as-judge evaluation
# Or with uv for development
uv run pytest tests/unit/ -v
uv run pytest tests/integration/ -v
uv run pytest acceptance/ -v
```
## Architecture
Built on **Microsoft Semantic Kernel** for:
- Enterprise-ready Azure OpenAI and OpenAI integration
- Future-proof model compatibility
- **100% Schema Enforcement**: KernelBaseModel integration with token-level constraints
- **Dynamic Model Creation**: Runtime JSON schema → Pydantic model conversion
- **Azure RBAC**: Azure RBAC via DefaultAzureCredential
- **Automatic Fallback**: Azure-first priority with OpenAI fallback
## The AI-First Development Journey
This toolkit is your first step toward [AI-First DevOps](https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html). As you integrate AI into your development workflows, you'll experience:
1. **🚀 Exponential Productivity**: AI handles routine tasks while you focus on architecture
2. **🎯 Guaranteed Quality**: Schema enforcement eliminates validation errors
3. **🤖 Autonomous Operations**: AI agents make decisions in your pipelines
4. **📈 Continuous Improvement**: Every interaction improves your AI system
**The future belongs to teams that master AI-first principles.** This toolkit gives you the foundation to start that journey today.
## License
MIT License - See [LICENSE](https://github.com/Nantero1/ai-first-devops-toolkit/blob/main/LICENSE) file for details. Copyright (c) 2025, Benjamin Linnik.
## Support
**🐛 Found a bug? 💡 Have a question? 📚 Need help?**
**GitHub is your primary destination for all support:**
- **📋 Issues & Bug Reports**: [Create an issue](https://github.com/Nantero1/ai-first-devops-toolkit/issues)
- **📖 Documentation**: [Browse examples](https://github.com/Nantero1/ai-first-devops-toolkit/tree/main/examples)
- **🔧 Source Code**: [View source](https://github.com/Nantero1/ai-first-devops-toolkit)
**Before opening an issue, please:**
1. ✅ Check the [examples directory](https://github.com/Nantero1/ai-first-devops-toolkit/tree/main/examples) for solutions
2. ✅ Review the error logs (beautiful output with Rich!)
3. ✅ Validate your Azure authentication and permissions
4. ✅ Ensure your input JSON follows the required format
5. ✅ Search existing [issues](https://github.com/Nantero1/ai-first-devops-toolkit/issues) for similar problems
**Quick Links:**
- 🚀 [Getting Started Guide](https://github.com/Nantero1/ai-first-devops-toolkit#quick-start)
- 📚 [Complete Examples](https://github.com/Nantero1/ai-first-devops-toolkit/tree/main/examples)
- 🔧 [CI/CD Integration](https://github.com/Nantero1/ai-first-devops-toolkit#cicd-integration)
- 🎯 [Use Cases](https://github.com/Nantero1/ai-first-devops-toolkit#use-cases)
---
*Ready to embrace the AI-First future? Start with this toolkit and build your path to exponential productivity. Learn more about the AI-First DevOps revolution in [Building AI-First DevOps](https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html).*
Raw data
{
"_id": null,
"home_page": null,
"name": "llm-ci-runner",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "ai, automation, azure-openai, ci-cd, devops, llm, semantic-kernel",
"author": null,
"author_email": "Benjamin Linnik <Benjamin@Linnik.IT>",
"download_url": "https://files.pythonhosted.org/packages/b1/1f/5b9f0257a0840ac7e4c8e38b4fe161bdf7610cc14421f75dceabe5896d57/llm_ci_runner-1.3.1.tar.gz",
"platform": null,
"description": "# AI-First Toolkit: LLM-Powered Automation\n\n[](https://badge.fury.io/py/llm-ci-runner) [](https://github.com/Nantero1/ai-first-devops-toolkit/actions/workflows/ci.yml) [](https://github.com/Nantero1/ai-first-devops-toolkit/actions/workflows/unit-tests.yml) [](https://htmlpreview.github.io/?https://github.com/Nantero1/ai-first-devops-toolkit/blob/python-coverage-comment-action-data/htmlcov/index.html) [](https://github.com/Nantero1/ai-first-devops-toolkit/actions/workflows/github-code-scanning/codeql)\n\n> **\ud83d\ude80 The Future of DevOps is AI-First** \n> This toolkit represents a step toward [AI-First DevOps](https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html) - where intelligent automation handles the entire development lifecycle. Built for teams ready to embrace the exponential productivity gains of AI-powered development. Please read [the blog post](https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html) for more details on the motivation.\n\n## TLDR: What This Tool Does\n\n**Purpose**: Zero-friction LLM integration for pipelines with **100% guaranteed schema compliance**. This is your foundation for AI-first integration practices.\n\n**Perfect For**:\n- \ud83e\udd16 **AI-Generated Code Reviews**: Automated PR analysis with structured findings\n- \ud83d\udcdd **Intelligent Documentation**: Generate changelogs, release notes, and docs automatically \n- \ud83d\udd0d **Security Analysis**: AI-powered vulnerability detection with structured reports\n- \ud83c\udfaf **Quality Gates**: Enforce standards through AI-driven validation\n- \ud83d\ude80 **Autonomous Development**: Enable AI agents to make decisions in your pipelines\n- \ud83c\udfaf **JIRA Ticket Updates**: Update JIRA tickets based on LLM output\n- \ud83d\udd17 **Unlimited Integration Possibilities**: Chain it multiple times and use as glue code in your tool stack\n---\n\n### Simple structured output example\n\n```bash\n# Install and use immediately\npip install llm-ci-runner\nllm-ci-runner --input-file examples/02-devops/pr-description/input.json --schema-file examples/02-devops/pr-description/schema.json\n```\n\n\n## The AI-First Development Revolution\n\nThis toolkit embodies the principles outlined in [Building AI-First DevOps](https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html):\n\n| Traditional DevOps | AI-First DevOps (This Tool) |\n|-------------------|----------------------------|\n| Manual code reviews | \ud83e\udd16 AI-powered reviews with structured findings |\n| Human-written documentation | \ud83d\udcdd AI-generated docs with guaranteed consistency |\n| Reactive security scanning | \ud83d\udd0d Proactive AI security analysis |\n| Manual quality gates | \ud83c\udfaf AI-driven validation with schema enforcement |\n| Linear productivity | \ud83d\udcc8 Exponential gains through intelligent automation |\n\n## Features\n\n- \ud83c\udfaf **100% Schema Enforcement**: Your pipeline never gets invalid data. Token-level schema enforcement with guaranteed compliance\n- \ud83d\udd04 **Resilient execution**: Retries with exponential back-off and jitter plus a clear exception hierarchy keep transient cloud faults from breaking your CI.\n- \ud83d\ude80 **Zero-Friction CLI**: Single script, minimal configuration for pipeline integration and automation\n- \ud83d\udd10 **Enterprise Security**: Azure RBAC via DefaultAzureCredential with fallback to API Key\n- \ud83d\udce6 **CI-friendly CLI**: Stateless command that reads JSON/YAML, writes JSON/YAML, and exits with proper codes\n- \ud83c\udfa8 **Beautiful Logging**: Rich console output with timestamps and colors\n- \ud83d\udcc1 **File-based I/O**: CI/CD friendly with JSON/YAML input/output\n- \ud83d\udccb **Template-Driven Workflows**: Handlebars and Jinja2 templates with YAML variables for dynamic prompt generation\n- \ud83d\udcc4 **YAML Support**: Use YAML for schemas, input files, and output files - more readable than JSON\n- \ud83d\udd27 **Simple & Extensible**: Easy to understand and modify for your specific needs\n- \ud83e\udd16 **Semantic Kernel foundation**: async, service-oriented design ready for skills, memories, orchestration, and future model upgrades\n- \ud83d\udcda **Documentation**: Comprehensive documentation for all features and usage examples. Use your semantic kernel skills to extend the functionality.\n- \ud83e\uddd1\u200d\u2696\ufe0f **Acceptance Tests**: pytest framework with the LLM-as-Judge pattern for quality gates. Test your scripts before you run them in production.\n- \ud83d\udcb0 **Coming soon**: token usage and cost estimation appended to each result for budgeting and optimisation\n\n## \ud83d\ude80 The Only Enterprise AI DevOps Tool That Delivers RBAC Security, Robustness and Simplicity\n\n**LLM-CI-Runner stands alone in the market** as the only tool combining **100% schema enforcement**, **enterprise RBAC authentication**, and robust **Semantic Kernel integration with templates** in a single CLI solution. **No other tool delivers all three critical enterprise requirements together**.\n\n## Installation\n\n```bash\npip install llm-ci-runner\n```\n\nThat's it! No complex setup, no dependency management - just install and use. Perfect for CI/CD pipelines and local development.\n\n## Quick Start\n\n### 1. Install from PyPI\n\n```bash\npip install llm-ci-runner\n```\n\n### 2. Set Environment Variables\n\n**Azure OpenAI (Priority 1):**\n```bash\nexport AZURE_OPENAI_ENDPOINT=\"https://your-resource.openai.azure.com/\"\nexport AZURE_OPENAI_MODEL=\"gpt-4.1-nano\" # or any other GPT deployment name\nexport AZURE_OPENAI_API_VERSION=\"2024-12-01-preview\" # Optional\n```\n\n**OpenAI (Fallback):**\n```bash\nexport OPENAI_API_KEY=\"your-very-secret-api-key\"\nexport OPENAI_CHAT_MODEL_ID=\"gpt-4.1-nano\" # or any OpenAI model\nexport OPENAI_ORG_ID=\"org-your-org-id\" # Optional\n```\n\n**Authentication Options:**\n- **Azure RBAC (Recommended)**: Uses `DefaultAzureCredential` for Azure RBAC authentication - no API key needed! See [Microsoft Docs](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for setup.\n- **Azure API Key**: Set `AZURE_OPENAI_API_KEY` environment variable if not using RBAC.\n- **OpenAI API Key**: Required for OpenAI fallback when Azure is not configured.\n\n**Priority**: Azure OpenAI takes priority when both Azure and OpenAI environment variables are present.\n\n### 3a. Basic Usage\n\n```bash\n# Simple chat example\nllm-ci-runner --input-file examples/01-basic/simple-chat/input.json\n\n# With structured output schema\nllm-ci-runner \\\n --input-file examples/01-basic/sentiment-analysis/input.json \\\n --schema-file examples/01-basic/sentiment-analysis/schema.json\n\n# Custom output file\nllm-ci-runner \\\n --input-file examples/02-devops/pr-description/input.json \\\n --schema-file examples/02-devops/pr-description/schema.json \\\n --output-file pr-analysis.json\n\n# YAML input files (alternative to JSON)\nllm-ci-runner \\\n --input-file config.yaml \\\n --schema-file schema.yaml \\\n --output-file result.yaml\n```\n\n### 3b. Template-Based Workflows\n\n**Dynamic prompt generation with YAML, Handlebars or Jinja2 templates:**\n\n```bash\n# Handlebars template example\nllm-ci-runner \\\n --template-file examples/05-templates/handlebars-template/template.hbs \\\n --template-vars examples/05-templates/handlebars-template/template-vars.yaml \\\n --schema-file examples/05-templates/handlebars-template/schema.yaml \\\n --output-file handlebars-result.yaml\n \n# Or using Jinja2 templates\nllm-ci-runner \\\n --template-file examples/05-templates/jinja2-template/template.j2 \\\n --template-vars examples/05-templates/jinja2-template/template-vars.yaml \\\n --schema-file examples/05-templates/jinja2-template/schema.yaml \\\n --output-file jinja2-result.yaml\n```\n\nFor more examples see the [examples directory](https://github.com/Nantero1/ai-first-devops-toolkit/tree/main/examples/05-templates).\n\n**Benefits of Template Approach:**\n- \ud83c\udfaf **Reusable Templates**: Create once, use across multiple scenarios\n- \ud83d\udcdd **YAML Configuration**: More readable than JSON for complex setups\n- \ud83d\udd04 **Dynamic Content**: Variables and conditional rendering\n- \ud83d\ude80 **CI/CD Ready**: Perfect for parameterized pipeline workflows\n\n### 4. Development Setup (Optional)\n\nFor contributors or advanced users who want to modify the source:\n\n```bash\n# Install UV if you haven't already\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n\n# Clone and install for development\ngit clone https://github.com/Nantero1/ai-first-devops-toolkit.git\ncd ai-first-devops-toolkit\nuv sync\n\n# Run from source\nuv run llm-ci-runner --input-file examples/01-basic/simple-chat/input.json\n```\n\n## Real-World Examples\n\nYou can explore the **[examples directory](https://github.com/Nantero1/ai-first-devops-toolkit/tree/main/examples)** for a complete collection of self-contained examples organized by category.\n\nFor comprehensive real-world CI/CD scenarios, see **[examples/uv-usage-example.md](https://github.com/Nantero1/ai-first-devops-toolkit/blob/main/examples/uv-usage-example.md)**. Some possibilities:\n\n1. \ud83e\udd16 AI-generated PR review \u2013 automated pull request analysis with structured review findings\n2. \ud83d\udcdd Commit summarizer \u2013 convert commit logs into concise release notes\n3. \ud83d\udd0d Vulnerability scanner \u2013 map code vulnerabilities to OWASP standards with actionable remediation\n4. \ud83c\udfaf Quality gate enforcer \u2013 validate build artifacts against schema-defined quality criteria\n5. \ud83c\udfe6 Loan application analyzer \u2013 transform free-text loan applications into Basel-III risk-model inputs\n6. \ud83d\udcbc Consulting report generator \u2013 convert meeting notes into itemized Statement of Work deliverables\n7. \ud83c\udfdb\ufe0f Legal contract parser \u2013 extract clauses and compute risk scores from contract documents\n8. \ud83d\udcdd Court opinion digest \u2013 summarize judicial opinions into structured precedent and citation graphs\n9. \ud83c\udfe5 Patient intake processor \u2013 build HL7/FHIR-compliant patient records from free-form intake forms\n10. \ud83d\udcc8 Earnings call analyzer \u2013 convert transcripts into KPI dashboards for financial performance review\n11. \ud83d\udd0d Code-review bot \u2013 scan commits and PRs to produce OWASP-mapped vulnerability reports\n12. \ud83c\udfaf Incident post-mortem summarizer \u2013 generate structured root cause analysis and corrective action plans\n13. \ud83d\udcca Regulatory compliance reporter \u2013 synthesize regulatory texts into structured compliance checklists\n14. \ud83d\udcbc Financial audit note handler \u2013 convert audit commentary into ledger-ready journal entries\n15. \ud83d\udd0d Vulnerability scanner \u2013 map code vulnerabilities to OWASP standards with actionable remediation\n16. \ud83c\udfaf Quality gate enforcer \u2013 validate build artifacts against schema-defined quality criteria\n17. \ud83c\udfe6 Loan application analyzer \u2013 transform free-text loan applications into Basel-III risk-model inputs\n18. \ud83d\udcbc Consulting report generator \u2013 convert meeting notes into itemized Statement of Work deliverables\n19. \ud83c\udfdb\ufe0f Legal contract parser \u2013 extract clauses and compute risk scores from contract documents\n20. \ud83d\udcdd Court opinion digest \u2013 summarize judicial opinions into structured precedent and citation graphs\n21. \ud83c\udfe5 Patient intake processor \u2013 build HL7/FHIR-compliant patient records from free-form intake forms\n22. \ud83d\udcc8 Earnings call analyzer \u2013 convert transcripts into KPI dashboards for financial performance review\n23. \ud83d\udd0d Code-review bot \u2013 scan commits and PRs to produce OWASP-mapped vulnerability reports\n24. \ud83c\udfaf Incident post-mortem summarizer \u2013 generate structured root cause analysis and corrective action plans\n25. \ud83d\udcca Regulatory compliance reporter \u2013 synthesize regulatory texts into structured compliance checklists\n26. \ud83d\udcbc Financial audit note handler \u2013 convert audit commentary into ledger-ready journal entries\n27. \ud83d\udd27 Technical review assistant \u2013 output structured code review reports with clear action items\n28. \ud83c\udfe5 Doctor dictation converter \u2013 transform verbal notes into ICD-10 coded encounter records\n29. \ud83c\udfdb\ufe0f Legal discovery summarizer \u2013 extract key issues and risks from large document sets\n30. \ud83d\udcc1 Manufacturing defect analyzer \u2013 build 8D corrective-action records from production issue notes\n31. \ud83d\udcb9 Budget variance analyzer \u2013 summarize financial reports into detailed KPI and variance analyses\n32. \ud83d\udda5\ufe0f Ticket triage assistant \u2013 prioritize technical support tickets with automated incident classification\n33. \ud83c\udfe6 Compliance transformer \u2013 create structured Basel reports from raw regulatory text\n34. \ud83d\udcca Credit risk evaluator \u2013 convert customer feedback into quantifiable risk scores\n35. \ud83d\udcb0 Investor memo summarizer \u2013 distill strategic memos into pitch-deck bullet points\n36. \ud83d\udee1\ufe0f Cyber threat mapper \u2013 translate security alerts into MITRE ATT&CK mapped incident reports\n37. \ud83d\udc77 Equipment maintenance scheduler \u2013 analyze sensor logs to generate predictive maintenance reports\n38. \ud83e\udec0 Health history compiler \u2013 produce structured patient histories from narrative medical notes\n39. \ud83d\uded1 Safety inspection checker \u2013 transform inspection narratives into OSHA citation checklists\n40. \ud83c\udfe5 Radiology result formatter \u2013 convert radiology reports into SNOMED-coded JSON outputs\n41. \ud83d\udcdd Insurance claim analyzer \u2013 structure claim narratives into automated claim assessments\n42. \ud83d\udcbc Contract review summarizer \u2013 extract risk factors and key dates from legal contracts\n43. \ud83d\udd0d Fraud detector \u2013 transform analyst notes into SAR (Suspicious Activity Report) JSON objects\n44. \ud83c\udfdb\ufe0f Policy impact assessor \u2013 convert policy proposals into stakeholder impact matrices\n45. \ud83c\udfed Production incident reporter \u2013 build actionable recovery plans from factory incident logs\n46. \ud83d\udcdd Documentation updater \u2013 generate schema-compliant technical documentation automatically\n47. \ud83d\udd04 API diff analyzer \u2013 produce backward-compatibility risk reports from API specification changes\n48. \ud83d\udcca Financial forecaster \u2013 summarize financial reports into structured cash-flow and projection objects\n49. \ud83d\udd27 Deployment log analyzer \u2013 convert rollout logs into performance and downtime metrics\n50. \ud83d\uded2 E-commerce sentiment analyzer \u2013 tag customer reviews with sentiment and key product features\n51. \ud83c\udf99\ufe0f Meeting minute extractor \u2013 transform recorded meetings into action items and follow-up tasks\n52. \ud83d\udcdd Sprint retrospective summarizer \u2013 generate improvement plans from agile team discussions\n53. \ud83c\udfe5 Clinical trial data packager \u2013 automatically structure clinical notes for FDA-submission\n54. \ud83c\udfe2 Employee feedback analyzer \u2013 convert free-text feedback into HR insights and action checklists\n55. \ud83d\udee0\ufe0f Process efficiency reporter \u2013 output production logs into structured performance metrics\n56. \ud83c\udfdb\ufe0f Legal bill auditor \u2013 transform billing details into itemized expense and compliance reports\n57. \ud83d\udce6 Automated inventory trigger \u2013 build reordering reports from warehouse inventory logs\n58. \ud83e\uddfe Receipt processor \u2013 convert OCR receipts into ledger-ready accounting entries\n59. \ud83c\udfe6 Mortgage eligibility assessor \u2013 analyze mortgage applications to generate risk and eligibility scores\n60. \ud83d\udea7 Infrastructure incident analyst \u2013 summarize log files into detailed RCAs and incident timelines\n61. \ud83c\udfdb\ufe0f Regulatory update tracker \u2013 generate structured compliance action items from updated guidelines\n62. \ud83d\udcdd Board meeting summarizer \u2013 extract key decisions and action items from meeting transcripts\n63. \ud83d\udd0d Vulnerability risk assessor \u2013 create remediation plans by mapping findings to risk frameworks\n64. \ud83d\udcbc Legal email analyzer \u2013 extract key issues and deadlines from email threads for legal review\n65. \ud83c\udfe5 Prescription manager \u2013 transform handwritten prescription notes into structured medication lists\n66. \ud83d\udda5\ufe0f Git log analyzer \u2013 generate detailed changelogs from version control commit histories\n67. \ud83d\udccb SOP generator \u2013 create standard operating procedures with checklist items from process descriptions\n68. \ud83c\udfaf PR triage tool \u2013 score and tag pull requests by urgency and impact automatically\n69. \ud83c\udfe6 Audit finding summarizer \u2013 convert audit observations into structured compliance and risk reports\n70. \ud83d\udcc8 Market trend analyzer \u2013 synthesize marketing data into structured trend forecasting objects\n71. \ud83e\uddd1\u200d\ud83d\udcbc Proposal evaluator \u2013 produce structured scoring and evaluation criteria from project proposals\n72. \ud83c\udfe2 Operations dashboard creator \u2013 translate facility logs into productivity and efficiency metrics\n73. \ud83c\udfe5 Lab result organizer \u2013 build structured diagnostic tables from laboratory results\n74. \ud83d\udca1 Innovation evaluator \u2013 compile ideation logs into cost-benefit structured analyses\n75. \ud83c\udfdb\ufe0f Judicial ruling summarizer \u2013 generate concise, structured digests from court rulings\n76. \ud83d\udd27 Commit changelog generator \u2013 extract impactful changes from commit logs for release summaries\n77. \ud83c\udfed Production yield analyzer \u2013 produce reports on output statistics and downtime from factory logs\n78. \ud83d\udcb3 Fraud alert generator \u2013 transform risk signals into automated CVSS-scored alerts\n79. \ud83d\udcdd Regulatory filing assistant \u2013 structure raw regulatory data for seamless filing and compliance tracking\n80. \ud83d\udc69\u200d\u2695\ufe0f Clinical observation compiler \u2013 convert medical research notes into structured clinical data entries\n81. \ud83d\ude80 Deployment success reporter \u2013 summarize production rollouts with performance metrics and KPIs\n82. \ud83c\udfe6 Mortgage risk evaluator \u2013 process mortgage files into detailed risk scoring and eligibility summaries\n83. \ud83d\udcbc Contract amendment monitor \u2013 track version changes and compliance updates in amended contracts\n84. \ud83c\udfe5 Vital signs monitor \u2013 generate alert reports from patient vital signs and anomaly detection\n85. \ud83d\udd10 IT security auditor \u2013 convert access logs into structured audit and compliance reports\n86. \ud83d\udea7 Incident ticket classifier \u2013 generate detailed RCA reports and automated ticket categorizations\n87. \ud83c\udfdb\ufe0f Governance mapper \u2013 produce structured mappings of internal policies to regulatory frameworks\n88. \ud83c\udfe2 Onboarding compliance checker \u2013 convert training logs into automated compliance and checklist trackers\n89. \ud83d\udcdd Data breach notifier \u2013 build structured breach incident reports with remediation plans\n90. \ud83c\udfe6 Teller performance analyzer \u2013 transform shift logs into performance and error analysis reports\n91. \ud83d\udcbc Contract risk assessor \u2013 generate automated legal risk memos from detailed contract reviews\n92. \ud83d\udee0\ufe0f Bug report classifier \u2013 categorize issue reports by severity and produce remediation plans\n93. \ud83c\udfe5 Appointment summarizer \u2013 convert appointment notes into structured follow-up recommendations\n94. \ud83d\udd04 Data migration manifest \u2013 output ETL mapping details into a structured transformation record\n95. \ud83d\ude80 Post-release analyst \u2013 synthesize customer feedback into performance improvement metrics\n96. \ud83c\udfed Equipment efficiency evaluator \u2013 analyze production logs to predict maintenance needs and cost analysis\n97. \ud83d\udd75\ufe0f Fraud case reporter \u2013 compile investigative notes into structured fraud case summaries\n98. \ud83c\udfdb\ufe0f Compliance checklist generator \u2013 map internal controls to GDPR or other frameworks in structured reports\n99. \ud83d\udc68\u200d\ud83d\udcbb Diff summarizer \u2013 automatically generate summaries of code differences for peer review\n100. \ud83d\udcc4 Patent claim comparator \u2013 produce novelty and prior art comparison tables from patent texts\n101. \ud83d\udd0d Cyber incident analyzer \u2013 structure incident narratives into threat intelligence and remediation guides\n102. \ud83d\udee1\ufe0f Security audit mapper \u2013 create control maps aligned with NIST frameworks from audit notes\n103. \ud83c\udfe6 Portfolio risk analyzer \u2013 transform investment notes into performance and risk metric summaries\n104. \ud83d\udcca Stress test reporter \u2013 compile financial stress test scenarios into structured risk reports\n105. \ud83d\udcdd Meeting action tracker \u2013 extract decisions and assign tasks from meeting minutes\n106. \ud83d\udee0\ufe0f DevOps runbook creator \u2013 produce actionable standard operating procedures from runbook logs\n107. \ud83d\ude9a Supply chain optimizer \u2013 generate delay forecasts and automated inventory suggestions from logistics notes\n108. \u2699\ufe0f Process improvement recommender \u2013 convert operational logs into structured efficiency recommendations\n109. \ud83d\udc6e Compliance reporter \u2013 map internal governance policies to GDPR and similar frameworks\n110. \ud83c\udf10 API performance optimizer \u2013 analyze API usage logs to generate optimization and performance metrics\n111. \ud83d\udee0\ufe0f Legacy system analyzer \u2013 assess legacy code bases and produce migration impact reports\n112. \ud83e\udde9 Unstructured anything \u2192 your bespoke schema-validated JSON\n\n## Input Formats\n\n### Traditional JSON Input\n\n```json\n{\n \"messages\": [\n {\n \"role\": \"system\",\n \"content\": \"You are a helpful assistant.\"\n },\n {\n \"role\": \"user\", \n \"content\": \"Your task description here\"\n }\n ],\n \"context\": {\n \"session_id\": \"optional-session-id\",\n \"metadata\": {\n \"any\": \"additional context\"\n }\n }\n}\n```\n\n### YAML Input\n\n```yaml\nmessages:\n - role: system\n content: \"You are a helpful assistant.\"\n - role: user\n content: \"Your task description here\"\ncontext:\n session_id: \"optional-session-id\"\n metadata:\n any: \"additional context\"\n```\n\n### Template-Based Input\n\n**Handlebars Template** (`template.hbs`):\n```handlebars\n<message role=\"system\">\nYou are an expert {{expertise.domain}} engineer.\nFocus on {{expertise.focus_areas}}.\n</message>\n\n<message role=\"user\">\nAnalyze this {{task.type}}:\n\n{{#each task.items}}\n- {{this}}\n{{/each}}\n\nRequirements: {{task.requirements}}\n</message>\n```\n\n**Jinja2 Template** (`template.j2`):\n```jinja2\n<message role=\"system\">\nYou are an expert {{expertise.domain}} engineer.\nFocus on {{expertise.focus_areas}}.\n</message>\n\n<message role=\"user\">\nAnalyze this {{task.type}}:\n\n{% for item in task.items %}\n- {{item}}\n{% endfor %}\n\nRequirements: {{task.requirements}}\n</message>\n```\n\n**Template Variables** (`vars.yaml`):\n```yaml\nexpertise:\n domain: \"DevOps\"\n focus_areas: \"security, performance, maintainability\"\ntask:\n type: \"pull request\"\n items:\n - \"Changed authentication logic\"\n - \"Updated database queries\"\n - \"Added input validation\"\n requirements: \"Focus on security vulnerabilities\"\n```\n\n## Structured Outputs with 100% Schema Enforcement\n\nWhen you provide a `--schema-file`, the runner guarantees perfect schema compliance:\n\n```bash\nllm-ci-runner \\\n --input-file examples/01-basic/sentiment-analysis/input.json \\\n --schema-file examples/01-basic/sentiment-analysis/schema.json\n```\n\n**Note**: Output defaults to `result.json`. Use `--output-file custom-name.json` for custom output files.\n\n**Supported Schema Features**:\n\u2705 String constraints (enum, minLength, maxLength, pattern) \n\u2705 Numeric constraints (minimum, maximum, multipleOf) \n\u2705 Array constraints (minItems, maxItems, items type) \n\u2705 Required fields enforced at generation time \n\u2705 Type validation (string, number, integer, boolean, array) \n\n## CI/CD Integration\n\n### GitHub Actions Example\n\n```yaml\n- name: Setup Python\n uses: actions/setup-python@v5\n with:\n python-version: '3.12'\n\n- name: Install LLM CI Runner\n run: pip install llm-ci-runner\n\n- name: Generate PR Review with Templates\n run: |\n llm-ci-runner \\\n --template-file .github/templates/pr-review.j2 \\\n --template-vars pr-context.yaml \\\n --schema-file .github/schemas/pr-review.yaml \\\n --output-file pr-analysis.yaml\n env:\n AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}\n AZURE_OPENAI_MODEL: ${{ secrets.AZURE_OPENAI_MODEL }}\n```\n\nFor complete CI/CD examples, see **[examples/uv-usage-example.md](https://github.com/Nantero1/ai-first-devops-toolkit/blob/main/examples/uv-usage-example.md)**.\n\n## Authentication\n\n**Azure OpenAI**: Uses Azure's `DefaultAzureCredential` supporting:\n- Environment variables (local development)\n- Managed Identity (recommended for Azure CI/CD)\n- Azure CLI (local development)\n- Service Principal (non-Azure CI/CD)\n\n**OpenAI**: Uses API key authentication with optional organization ID.\n\n## Testing\n\nWe maintain comprehensive test coverage with **100% success rate**:\n\n```bash\n# For package users - install test dependencies\npip install llm-ci-runner[dev]\n\n# For development - install from source with test dependencies\nuv sync --group dev\n\n# Run specific test categories\npytest tests/unit/ -v # 70 unit tests\npytest tests/integration/ -v # End-to-end examples\npytest acceptance/ -v # LLM-as-judge evaluation\n\n# Or with uv for development\nuv run pytest tests/unit/ -v\nuv run pytest tests/integration/ -v\nuv run pytest acceptance/ -v\n```\n\n## Architecture\n\nBuilt on **Microsoft Semantic Kernel** for:\n- Enterprise-ready Azure OpenAI and OpenAI integration\n- Future-proof model compatibility\n- **100% Schema Enforcement**: KernelBaseModel integration with token-level constraints\n- **Dynamic Model Creation**: Runtime JSON schema \u2192 Pydantic model conversion\n- **Azure RBAC**: Azure RBAC via DefaultAzureCredential\n- **Automatic Fallback**: Azure-first priority with OpenAI fallback\n\n## The AI-First Development Journey\n\nThis toolkit is your first step toward [AI-First DevOps](https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html). As you integrate AI into your development workflows, you'll experience:\n\n1. **\ud83d\ude80 Exponential Productivity**: AI handles routine tasks while you focus on architecture\n2. **\ud83c\udfaf Guaranteed Quality**: Schema enforcement eliminates validation errors\n3. **\ud83e\udd16 Autonomous Operations**: AI agents make decisions in your pipelines\n4. **\ud83d\udcc8 Continuous Improvement**: Every interaction improves your AI system\n\n**The future belongs to teams that master AI-first principles.** This toolkit gives you the foundation to start that journey today.\n\n## License\n\nMIT License - See [LICENSE](https://github.com/Nantero1/ai-first-devops-toolkit/blob/main/LICENSE) file for details. Copyright (c) 2025, Benjamin Linnik.\n\n## Support\n\n**\ud83d\udc1b Found a bug? \ud83d\udca1 Have a question? \ud83d\udcda Need help?**\n\n**GitHub is your primary destination for all support:**\n\n- **\ud83d\udccb Issues & Bug Reports**: [Create an issue](https://github.com/Nantero1/ai-first-devops-toolkit/issues)\n- **\ud83d\udcd6 Documentation**: [Browse examples](https://github.com/Nantero1/ai-first-devops-toolkit/tree/main/examples)\n- **\ud83d\udd27 Source Code**: [View source](https://github.com/Nantero1/ai-first-devops-toolkit)\n\n**Before opening an issue, please:**\n1. \u2705 Check the [examples directory](https://github.com/Nantero1/ai-first-devops-toolkit/tree/main/examples) for solutions\n2. \u2705 Review the error logs (beautiful output with Rich!)\n3. \u2705 Validate your Azure authentication and permissions\n4. \u2705 Ensure your input JSON follows the required format\n5. \u2705 Search existing [issues](https://github.com/Nantero1/ai-first-devops-toolkit/issues) for similar problems\n\n**Quick Links:**\n- \ud83d\ude80 [Getting Started Guide](https://github.com/Nantero1/ai-first-devops-toolkit#quick-start)\n- \ud83d\udcda [Complete Examples](https://github.com/Nantero1/ai-first-devops-toolkit/tree/main/examples)\n- \ud83d\udd27 [CI/CD Integration](https://github.com/Nantero1/ai-first-devops-toolkit#cicd-integration)\n- \ud83c\udfaf [Use Cases](https://github.com/Nantero1/ai-first-devops-toolkit#use-cases)\n\n---\n\n*Ready to embrace the AI-First future? Start with this toolkit and build your path to exponential productivity. Learn more about the AI-First DevOps revolution in [Building AI-First DevOps](https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html).*\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A simple CI/CD utility for running LLM tasks with Semantic Kernel",
"version": "1.3.1",
"project_urls": {
"Bug Tracker": "https://github.com/Nantero1/ai-first-devops-toolkit/issues",
"Changelog": "https://github.com/Nantero1/ai-first-devops-toolkit/releases",
"Documentation": "https://github.com/Nantero1/ai-first-devops-toolkit",
"Homepage": "https://technologyworkroom.blogspot.com/2025/06/building-ai-first-devops.html",
"Repository": "https://github.com/Nantero1/ai-first-devops-toolkit",
"Source": "https://github.com/Nantero1/ai-first-devops-toolkit"
},
"split_keywords": [
"ai",
" automation",
" azure-openai",
" ci-cd",
" devops",
" llm",
" semantic-kernel"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "27a25eeaf9277582fdcec8a55b47eba7221c652c83758e79fa2a8ed22bbc40ce",
"md5": "32b0c48845d4b59550f5d16dfbdd7ae9",
"sha256": "77b66b83bf8488f677a82b90fa3cdee1f48e2cc02a0dce5792d04060b8f76576"
},
"downloads": -1,
"filename": "llm_ci_runner-1.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "32b0c48845d4b59550f5d16dfbdd7ae9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 30050,
"upload_time": "2025-07-12T20:50:30",
"upload_time_iso_8601": "2025-07-12T20:50:30.043887Z",
"url": "https://files.pythonhosted.org/packages/27/a2/5eeaf9277582fdcec8a55b47eba7221c652c83758e79fa2a8ed22bbc40ce/llm_ci_runner-1.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "b11f5b9f0257a0840ac7e4c8e38b4fe161bdf7610cc14421f75dceabe5896d57",
"md5": "e0d28f49fac22795581843d9b0ec0d9c",
"sha256": "a4b14b43711dd184f240ab57a99dbfbb8a61dc1a6e3b88f4fcc569b8a5c6ac5c"
},
"downloads": -1,
"filename": "llm_ci_runner-1.3.1.tar.gz",
"has_sig": false,
"md5_digest": "e0d28f49fac22795581843d9b0ec0d9c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 822409,
"upload_time": "2025-07-12T20:50:31",
"upload_time_iso_8601": "2025-07-12T20:50:31.587307Z",
"url": "https://files.pythonhosted.org/packages/b1/1f/5b9f0257a0840ac7e4c8e38b4fe161bdf7610cc14421f75dceabe5896d57/llm_ci_runner-1.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-12 20:50:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Nantero1",
"github_project": "ai-first-devops-toolkit",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "llm-ci-runner"
}
JSON
