# validated-llm
- [](https://badge.fury.io/py/validated-llm)
- [](https://pypi.org/project/validated-llm/)
- [](https://opensource.org/licenses/MIT)
- LLM output validation with retry loops - ensure your language model responses meet requirements.
## Overview
`validated-llm` provides a robust framework for validating language model outputs with automatic retry mechanisms. It's designed for applications where you need reliable, structured responses from LLMs.
### Why validated-llm?
Unlike other solutions, validated-llm offers:
- **Clean separation of concerns** - Validators independent from LLM interaction
- **Framework agnostic** - Works with any LLM (OpenAI, Anthropic, Ollama)
- **Flexible validation** - Not tied to specific schema formats
- **Comprehensive debugging** - Detailed execution logs and attempt tracking
- **Simple API** - Just `execute()` with template, validator, and data
## Key Features
- Automatic Retry Logic: Handles failed validations with configurable retry attempts
- 16 Built-in Validators: JSON Schema, XML, YAML, Email, Phone, URL, Markdown, DateTime, Range, Regex, SQL, Syntax, Style, Test, Composite, and Documentation validators
- Enhanced JSON Detection: Detects nested objects, arrays of objects, and complex JSON structures with intelligent validator selection
- Code Generation & Validation: Multi-language code generation (Python, JavaScript, TypeScript, Go, Rust, Java) with syntax validation
- Template Library: 29 pre-built templates across 6 categories for common prompt patterns
- Prompt Migration Tools: Convert existing prompts to validated tasks with batch processing
- Task-Based Architecture: Organize validation logic into reusable task classes
- Langchain Integration: Full converter for migrating Langchain prompts to validated-llm
- Config File Support: Project-level configuration with `.validated-llm.yml`
- Comprehensive CLI Tools: Interactive template browsing, batch conversion, and analysis
## Quick Start
### Installation
`pip install validated-llm`
### Basic Usage
```python
from validated_llm import ValidationLoop
from validated_llm.validators import JSONSchemaValidator
# Define your validation schema
schema = {
"type": "object",
"properties": {
"scenes": {
"type": "array",
"items": {
"type": "object",
"properties": {
"title": {"type": "string"},
"description": {"type": "string"},
"characters": {"type": "array", "items": {"type": "string"}}
},
"required": ["title", "description", "characters"]
}
}
},
"required": ["scenes"]
}
# Create validation loop with built-in LLM provider
validator = ValidationLoop(
vendor="openai", # or "ollama", "anthropic"
model="gpt-4o",
api_key="your-openai-api-key",
max_retries=3,
temperature=0.7
)
# Execute with validation
result = validator.execute(
prompt_template="Convert this story into 3-5 scenes: {story}. Return as JSON with scenes array.",
validator=JSONSchemaValidator(schema),
input_data={"story": "Once upon a time..."}
)
print(result['output']) # Validated JSON response
```
### Built-in Tasks
The package includes comprehensive pre-built validation tasks:
#### JSON Generation with Schema Detection
```python
from validated_llm.tasks import JSONGenerationTask
# Automatic schema detection from examples
task = JSONGenerationTask(
schema={
"type": "object",
"properties": {
"scenes": {
"type": "array",
"items": {"type": "object"}
}
},
"required": ["scenes"]
}
)
```
#### Code Generation with Multi-Language Support
```python
from validated_llm.tasks import FunctionGenerationTask
task = FunctionGenerationTask(
language="python",
function_name="binary_search",
requirements="Implement binary search algorithm with proper error handling"
)
```
#### CSV Generation
```python
from validated_llm.tasks import CSVGenerationTask
task = CSVGenerationTask(
required_columns=["name", "age", "role"],
min_rows=3
)
```
#### Documentation Generation
```python
from validated_llm.tasks import APIDocumentationTask
task = APIDocumentationTask(
api_type="REST",
include_examples=True,
validate_completeness=True
)
```
## Architecture
### Core Components
- ValidationLoop: Main orchestrator that handles the retry logic
- BaseTask: Abstract base class for creating validation tasks
- BaseValidator: Pluggable validation system for different response types
### Creating Custom Tasks
```python
class CustomTask(BaseTask):
def get_prompt(self, input_data: str) -> str:
# Return the prompt for the LLM
return f"Process this data: {input_data}"
def validate_response(self, response: str) -> bool:
# Return True if response is valid
return len(response) > 10
def parse_response(self, response: str) -> dict:
# Optional: transform the response
return {"processed": response}
```
## CLI Tools
### Prompt to Task Converter
Convert existing prompts into validated task classes with enhanced JSON schema detection:
```bash
pip install validated-llm # Install the package
validated-llm-prompt2task prompt.txt # Convert a prompt file to a task
validated-llm-prompt2task batch prompts_directory/ # Batch convert multiple prompts with parallel processing
validated-llm-prompt2task prompt.txt --interactive # Interactive mode with validator selection
validated-llm-prompt2task prompt.txt --template api_doc # Use templates for consistent patterns
validated-llm-prompt2task prompt.txt --analyze-only # Analyze prompt without generating code
```
The tool will:
- Enhanced JSON Detection: Automatically detect nested objects, arrays of objects, and complex JSON structures
- Smart Validator Selection: Choose between JSONValidator and JSONSchemaValidator based on complexity
- Template Integration: Apply 29 pre-built templates for common use cases
- Batch Processing: Convert entire directories with parallel processing and progress tracking
- Format Detection: Detect JSON, CSV, text, lists, code, and documentation formats
- Generate Complete Tasks: Create task classes with appropriate validators and documentation
### Template Library
Browse and use pre-built templates:
```bash
validated-llm-templates list # Browse available templates
validated-llm-templates list --category "api" # Search templates by category
validated-llm-templates show product_catalog_json # Show template details
validated-llm-templates use business_email # Use a template interactively
```
### Configuration Management
Manage project-level settings:
```bash
validated-llm-config init # Initialize project configuration
validated-llm-config validate # Validate configuration file
validated-llm-config show # Show current configuration
```
### Plugin Management
Manage validator plugins for custom validation logic:
```bash
validated-llm-plugin list # List available plugins
validated-llm-plugin info credit_card_validator # Show detailed plugin information
validated-llm-plugin test credit_card_validator --args '{"strict_mode": true}' # Test a plugin
validated-llm-plugin discover ./custom_plugins/ # Discover plugins from a directory
validated-llm-plugin paths # Show plugin search paths
validated-llm-plugin validate-plugin my_validator # Validate plugin meets requirements
```
## Configuration
### LLM Provider Setup
```python
# OpenAI (default)
validator = ValidationLoop(
model="gpt-4",
api_key="your-api-key"
)
# Custom endpoint (e.g., local LLM)
validator = ValidationLoop(
model="llama2",
base_url="http://localhost:11434/v1/",
api_key="not-needed"
)
```
### Retry Configuration
```python
validator = ValidationLoop(
model="gpt-4",
max_retries=5, # Maximum retry attempts
temperature=0.7, # LLM temperature
timeout=30, # Request timeout in seconds
backoff_factor=1.5 # Exponential backoff multiplier
)
```
## Advanced Usage
### Custom Validators
```python
from validated_llm import BaseValidator
class SchemaValidator(BaseValidator):
def init(self, schema: dict):
self.schema = schema
def validate(self, response: str) -> tuple[bool, str]:
try:
data = json.loads(response)
# Validate against schema
return True, "Valid JSON"
except Exception as e:
return False, f"Invalid: {e}"
```
### Error Handling
```python
from validated_llm.exceptions import ValidationError, MaxRetriesExceeded
try:
result = validator.run_task(task, input_data)
except MaxRetriesExceeded:
print("Failed after maximum retries")
except ValidationError as e:
print(f"Validation failed: {e}")
```
## Testing
- Run the test suite: `poetry run pytest`
- With coverage: `poetry run pytest --cov=src tests/`
## Contributing
- Fork the repository
- Create a feature branch: `git checkout -b feature-name`
- Make changes and add tests
- Run tests: `poetry run pytest`
- Format code: `poetry run black . && poetry run isort .`
- Submit a pull request
## License
MIT License - see [LICENSE](LICENSE) file for details.
## Documentation
Comprehensive guides and documentation:
- [Cookbook](docs/COOKBOOK.md) - Practical examples and common patterns
- [Best Practices](docs/BEST_PRACTICES.md) - Production-ready patterns and optimization
- [Plugin Development](docs/PLUGIN_DEVELOPMENT.md) - Complete guide to creating custom validators
- [Plugin System](docs/PLUGIN_SYSTEM.md) - Architecture and usage of the plugin system
## Examples
See the `examples/` directory for more detailed usage examples:
- `basic_validation.py` - Simple validation example
- `custom_task.py` - Creating custom validation tasks
- `multiple_providers.py` - Using different LLM providers
- `story_to_scenes.py` - Real-world story processing example
- `validation_patterns.py` - Common validation patterns and use cases
Raw data
{
"_id": null,
"home_page": "https://github.com/byte-pipe/validated-llm",
"name": "validated-llm",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.11",
"maintainer_email": null,
"keywords": "llm, validation, ai, language-model, retry-loop, self-correcting, reliability, openai, anthropic, ollama",
"author": "validated-llm contributors",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/bc/43/91f9edcffda06d7299a6dffcc26937d286d8d3ef68f75be5985883916e26/validated_llm-0.1.17.tar.gz",
"platform": null,
"description": "# validated-llm\n\n- [](https://badge.fury.io/py/validated-llm)\n- [](https://pypi.org/project/validated-llm/)\n- [](https://opensource.org/licenses/MIT)\n- LLM output validation with retry loops - ensure your language model responses meet requirements.\n\n## Overview\n\n`validated-llm` provides a robust framework for validating language model outputs with automatic retry mechanisms. It's designed for applications where you need reliable, structured responses from LLMs.\n\n### Why validated-llm?\n\nUnlike other solutions, validated-llm offers:\n- **Clean separation of concerns** - Validators independent from LLM interaction\n- **Framework agnostic** - Works with any LLM (OpenAI, Anthropic, Ollama)\n- **Flexible validation** - Not tied to specific schema formats\n- **Comprehensive debugging** - Detailed execution logs and attempt tracking\n- **Simple API** - Just `execute()` with template, validator, and data\n\n## Key Features\n\n- Automatic Retry Logic: Handles failed validations with configurable retry attempts\n- 16 Built-in Validators: JSON Schema, XML, YAML, Email, Phone, URL, Markdown, DateTime, Range, Regex, SQL, Syntax, Style, Test, Composite, and Documentation validators\n- Enhanced JSON Detection: Detects nested objects, arrays of objects, and complex JSON structures with intelligent validator selection\n- Code Generation & Validation: Multi-language code generation (Python, JavaScript, TypeScript, Go, Rust, Java) with syntax validation\n- Template Library: 29 pre-built templates across 6 categories for common prompt patterns\n- Prompt Migration Tools: Convert existing prompts to validated tasks with batch processing\n- Task-Based Architecture: Organize validation logic into reusable task classes\n- Langchain Integration: Full converter for migrating Langchain prompts to validated-llm\n- Config File Support: Project-level configuration with `.validated-llm.yml`\n- Comprehensive CLI Tools: Interactive template browsing, batch conversion, and analysis\n\n## Quick Start\n\n### Installation\n\n`pip install validated-llm`\n\n### Basic Usage\n\n```python\nfrom validated_llm import ValidationLoop\nfrom validated_llm.validators import JSONSchemaValidator\n\n# Define your validation schema\nschema = {\n \"type\": \"object\",\n \"properties\": {\n \"scenes\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\"type\": \"string\"},\n \"description\": {\"type\": \"string\"},\n \"characters\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}}\n },\n \"required\": [\"title\", \"description\", \"characters\"]\n }\n }\n },\n \"required\": [\"scenes\"]\n}\n\n# Create validation loop with built-in LLM provider\nvalidator = ValidationLoop(\n vendor=\"openai\", # or \"ollama\", \"anthropic\"\n model=\"gpt-4o\",\n api_key=\"your-openai-api-key\",\n max_retries=3,\n temperature=0.7\n)\n\n# Execute with validation\nresult = validator.execute(\n prompt_template=\"Convert this story into 3-5 scenes: {story}. Return as JSON with scenes array.\",\n validator=JSONSchemaValidator(schema),\n input_data={\"story\": \"Once upon a time...\"}\n)\n\nprint(result['output']) # Validated JSON response\n```\n\n### Built-in Tasks\n\nThe package includes comprehensive pre-built validation tasks:\n\n#### JSON Generation with Schema Detection\n\n```python\nfrom validated_llm.tasks import JSONGenerationTask\n# Automatic schema detection from examples\ntask = JSONGenerationTask(\n schema={\n \"type\": \"object\",\n \"properties\": {\n \"scenes\": {\n \"type\": \"array\",\n \"items\": {\"type\": \"object\"}\n }\n },\n \"required\": [\"scenes\"]\n }\n)\n```\n\n#### Code Generation with Multi-Language Support\n\n```python\nfrom validated_llm.tasks import FunctionGenerationTask\ntask = FunctionGenerationTask(\n language=\"python\",\n function_name=\"binary_search\",\n requirements=\"Implement binary search algorithm with proper error handling\"\n)\n```\n\n#### CSV Generation\n\n```python\nfrom validated_llm.tasks import CSVGenerationTask\ntask = CSVGenerationTask(\n required_columns=[\"name\", \"age\", \"role\"],\n min_rows=3\n)\n```\n\n#### Documentation Generation\n\n```python\nfrom validated_llm.tasks import APIDocumentationTask\ntask = APIDocumentationTask(\n api_type=\"REST\",\n include_examples=True,\n validate_completeness=True\n)\n```\n\n## Architecture\n\n### Core Components\n\n- ValidationLoop: Main orchestrator that handles the retry logic\n- BaseTask: Abstract base class for creating validation tasks\n- BaseValidator: Pluggable validation system for different response types\n\n### Creating Custom Tasks\n\n```python\nclass CustomTask(BaseTask):\n def get_prompt(self, input_data: str) -> str:\n # Return the prompt for the LLM\n return f\"Process this data: {input_data}\"\n def validate_response(self, response: str) -> bool:\n # Return True if response is valid\n return len(response) > 10\n def parse_response(self, response: str) -> dict:\n # Optional: transform the response\n return {\"processed\": response}\n```\n\n## CLI Tools\n\n### Prompt to Task Converter\n\nConvert existing prompts into validated task classes with enhanced JSON schema detection:\n\n```bash\npip install validated-llm # Install the package\nvalidated-llm-prompt2task prompt.txt # Convert a prompt file to a task\nvalidated-llm-prompt2task batch prompts_directory/ # Batch convert multiple prompts with parallel processing\nvalidated-llm-prompt2task prompt.txt --interactive # Interactive mode with validator selection\nvalidated-llm-prompt2task prompt.txt --template api_doc # Use templates for consistent patterns\nvalidated-llm-prompt2task prompt.txt --analyze-only # Analyze prompt without generating code\n```\n\nThe tool will:\n\n- Enhanced JSON Detection: Automatically detect nested objects, arrays of objects, and complex JSON structures\n- Smart Validator Selection: Choose between JSONValidator and JSONSchemaValidator based on complexity\n- Template Integration: Apply 29 pre-built templates for common use cases\n- Batch Processing: Convert entire directories with parallel processing and progress tracking\n- Format Detection: Detect JSON, CSV, text, lists, code, and documentation formats\n- Generate Complete Tasks: Create task classes with appropriate validators and documentation\n\n### Template Library\n\nBrowse and use pre-built templates:\n\n```bash\nvalidated-llm-templates list # Browse available templates\nvalidated-llm-templates list --category \"api\" # Search templates by category\nvalidated-llm-templates show product_catalog_json # Show template details\nvalidated-llm-templates use business_email # Use a template interactively\n```\n\n### Configuration Management\n\nManage project-level settings:\n\n```bash\nvalidated-llm-config init # Initialize project configuration\nvalidated-llm-config validate # Validate configuration file\nvalidated-llm-config show # Show current configuration\n```\n\n### Plugin Management\n\nManage validator plugins for custom validation logic:\n\n```bash\nvalidated-llm-plugin list # List available plugins\nvalidated-llm-plugin info credit_card_validator # Show detailed plugin information\nvalidated-llm-plugin test credit_card_validator --args '{\"strict_mode\": true}' # Test a plugin\nvalidated-llm-plugin discover ./custom_plugins/ # Discover plugins from a directory\nvalidated-llm-plugin paths # Show plugin search paths\nvalidated-llm-plugin validate-plugin my_validator # Validate plugin meets requirements\n```\n\n## Configuration\n\n### LLM Provider Setup\n\n```python\n# OpenAI (default)\nvalidator = ValidationLoop(\n model=\"gpt-4\",\n api_key=\"your-api-key\"\n)\n# Custom endpoint (e.g., local LLM)\nvalidator = ValidationLoop(\n model=\"llama2\",\n base_url=\"http://localhost:11434/v1/\",\n api_key=\"not-needed\"\n)\n```\n\n### Retry Configuration\n\n```python\nvalidator = ValidationLoop(\n model=\"gpt-4\",\n max_retries=5, # Maximum retry attempts\n temperature=0.7, # LLM temperature\n timeout=30, # Request timeout in seconds\n backoff_factor=1.5 # Exponential backoff multiplier\n)\n```\n\n## Advanced Usage\n\n### Custom Validators\n\n```python\nfrom validated_llm import BaseValidator\nclass SchemaValidator(BaseValidator):\n def init(self, schema: dict):\n self.schema = schema\n def validate(self, response: str) -> tuple[bool, str]:\n try:\n data = json.loads(response)\n # Validate against schema\n return True, \"Valid JSON\"\n except Exception as e:\n return False, f\"Invalid: {e}\"\n```\n\n### Error Handling\n\n```python\nfrom validated_llm.exceptions import ValidationError, MaxRetriesExceeded\ntry:\n result = validator.run_task(task, input_data)\nexcept MaxRetriesExceeded:\n print(\"Failed after maximum retries\")\nexcept ValidationError as e:\n print(f\"Validation failed: {e}\")\n```\n\n## Testing\n\n- Run the test suite: `poetry run pytest`\n- With coverage: `poetry run pytest --cov=src tests/`\n\n## Contributing\n\n- Fork the repository\n- Create a feature branch: `git checkout -b feature-name`\n- Make changes and add tests\n- Run tests: `poetry run pytest`\n- Format code: `poetry run black . && poetry run isort .`\n- Submit a pull request\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## Documentation\n\nComprehensive guides and documentation:\n\n- [Cookbook](docs/COOKBOOK.md) - Practical examples and common patterns\n- [Best Practices](docs/BEST_PRACTICES.md) - Production-ready patterns and optimization\n- [Plugin Development](docs/PLUGIN_DEVELOPMENT.md) - Complete guide to creating custom validators\n- [Plugin System](docs/PLUGIN_SYSTEM.md) - Architecture and usage of the plugin system\n\n## Examples\n\nSee the `examples/` directory for more detailed usage examples:\n\n- `basic_validation.py` - Simple validation example\n- `custom_task.py` - Creating custom validation tasks\n- `multiple_providers.py` - Using different LLM providers\n- `story_to_scenes.py` - Real-world story processing example\n- `validation_patterns.py` - Common validation patterns and use cases\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "LLM output validation with retry loops - ensure your language model responses meet requirements",
"version": "0.1.17",
"project_urls": {
"Documentation": "https://github.com/byte-pipe/validated-llm/blob/main/README.md",
"Homepage": "https://github.com/byte-pipe/validated-llm",
"Repository": "https://github.com/byte-pipe/validated-llm"
},
"split_keywords": [
"llm",
" validation",
" ai",
" language-model",
" retry-loop",
" self-correcting",
" reliability",
" openai",
" anthropic",
" ollama"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "47198037208ab1494fea8f6af35d36b0b147e738bd399f8155e093f09526fad3",
"md5": "93e3449f6b2721a1b07e0b9694bbb0e7",
"sha256": "b8b94ae9e8aac0f84f1ac40bea9e997ecc8bec076b380f0b8bdcfd8f903663d3"
},
"downloads": -1,
"filename": "validated_llm-0.1.17-py3-none-any.whl",
"has_sig": false,
"md5_digest": "93e3449f6b2721a1b07e0b9694bbb0e7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.11",
"size": 249367,
"upload_time": "2025-09-08T11:18:17",
"upload_time_iso_8601": "2025-09-08T11:18:17.561384Z",
"url": "https://files.pythonhosted.org/packages/47/19/8037208ab1494fea8f6af35d36b0b147e738bd399f8155e093f09526fad3/validated_llm-0.1.17-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "bc4391f9edcffda06d7299a6dffcc26937d286d8d3ef68f75be5985883916e26",
"md5": "61d14f1ef40870b958146012c1e8e3c5",
"sha256": "7d91aa6da8de8f2d48b7db708ed97373b65934bb3d2e4d7033e18ea7b7f1ab31"
},
"downloads": -1,
"filename": "validated_llm-0.1.17.tar.gz",
"has_sig": false,
"md5_digest": "61d14f1ef40870b958146012c1e8e3c5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.11",
"size": 194304,
"upload_time": "2025-09-08T11:18:19",
"upload_time_iso_8601": "2025-09-08T11:18:19.044771Z",
"url": "https://files.pythonhosted.org/packages/bc/43/91f9edcffda06d7299a6dffcc26937d286d8d3ef68f75be5985883916e26/validated_llm-0.1.17.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-08 11:18:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "byte-pipe",
"github_project": "validated-llm",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "validated-llm"
}
JSON
