| Name | vetting-python JSON |
| Version |
0.1.0
JSON |
| download |
| home_page | None |
| Summary | A Python implementation of the VETTING (Verification and Evaluation Tool for Targeting Invalid Narrative Generation) framework for LLM safety and educational applications |
| upload_time | 2025-08-01 04:24:02 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | MIT License
Copyright (c) 2025 Hongming (Chip) Li
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
|
| keywords |
llm
ai-safety
education
chatbot
verification
dual-llm
prompt-engineering
educational-technology
machine-learning
natural-language-processing
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# VETTING Framework - Python Implementation
A Python implementation of the VETTING (Verification and Evaluation Tool for Targeting Invalid Narrative Generation) framework for LLM safety and educational applications.
**Developed by [VIABLE Lab](https://www.viablelab.org/) at the University of Florida.**
## Overview
The VETTING framework implements a dual-LLM architecture that separates conversational logic from policy enforcement, preventing prompt injection attacks on safety rules and enabling verifiable policy compliance. This architectural approach is particularly effective for educational applications where you need to guide learning without revealing direct answers.
### Key Features
- **🛡️ Architectural Policy Isolation**: Complete separation between user interaction (Chat-Layer) and policy enforcement (Verification-Layer)
- **🔄 Iterative Verification Loop**: Automatic refinement when responses don't meet verification criteria
- **🏫 Educational Focus**: Specialized support for tutoring and homework help scenarios
- **🌐 Multi-Provider Support**: Works with OpenAI, Anthropic Claude, and Google Gemini
- **💰 Cost Tracking**: Comprehensive cost monitoring and analysis
- **⚙️ Flexible Configuration**: Environment variables, config files, or programmatic setup
- **🔍 Safety Features**: Built-in safety prefix detection and content filtering
## Architecture
```
┌─────────────────┐ ┌─────────────────┐
│ User Input │ │ Chat-Layer │
│ │───▶│ (LLM-A) │
│ │ │ │
└─────────────────┘ └─────────┬───────┘
│
▼
┌─────────────────┐
│ Verification- │
│ Layer │◀─── Confidential
│ (LLM-B) │ Policy
└─────────┬───────┘
│
▼
┌─────────────────┐
│ Pass/Fail + │
│ Feedback │
└─────────────────┘
```
## Installation
### From PyPI (once published):
```bash
pip install vetting-python
```
### From Source:
```bash
git clone https://github.com/hichipli/vetting-python.git
cd vetting-python
pip install -e .
```
### Dependencies
```bash
pip install aiohttp pydantic dataclasses-json
```
Optional dependencies:
```bash
pip install PyYAML # For YAML configuration files
```
## Quick Start
### 1. Set up your API keys
```bash
export OPENAI_API_KEY="your-openai-api-key"
export ANTHROPIC_API_KEY="your-claude-api-key" # Optional
export GOOGLE_API_KEY="your-gemini-api-key" # Optional
```
### 2. Basic Chat Mode
```python
import asyncio
from vetting_python import VettingFramework, VettingConfig, ChatMessage, OpenAIProvider
async def basic_example():
# Setup provider
provider = OpenAIProvider(api_key="your-api-key")
# Create framework
framework = VettingFramework(chat_provider=provider)
# Simple chat configuration
config = VettingConfig(
mode="chat",
chat_model={"model_id": "gpt-4o-mini", "temperature": 0.7}
)
# Create conversation
messages = [ChatMessage("user", "Explain photosynthesis in simple terms.")]
# Process
response = await framework.process(messages, config)
print(f"Response: {response.content}")
print(f"Cost: ${response.total_cost:.4f}")
# Run the example
asyncio.run(basic_example())
```
### 3. Educational Vetting Mode
```python
import asyncio
from vetting_python import VettingFramework, OpenAIProvider
from vetting_python.config import VettingConfigBuilder
async def educational_example():
provider = OpenAIProvider(api_key="your-api-key")
framework = VettingFramework(chat_provider=provider)
# Educational configuration with answer key
config = (VettingConfigBuilder()
.vetting_mode()
.chat_model("gpt-4o-mini")
.verification_model("gpt-4o-mini")
.add_context_item(
question_text="What is the capital of France?",
correct_answer="Paris",
key_concepts=["Paris", "France", "capital city"]
)
.build())
# Student asks directly for the answer
messages = [ChatMessage("user", "What is the capital of France? I need this for homework.")]
# Process with verification
response = await framework.process(messages, config)
print(f"Response: {response.content}")
print(f"Verification passed: {response.verification_passed}")
print(f"Attempts made: {response.attempt_count}")
asyncio.run(educational_example())
```
## Configuration
### Environment Variables
The framework supports comprehensive configuration through environment variables:
```bash
# API Keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GOOGLE_API_KEY="..."
# Default Models
export VETTING_DEFAULT_CHAT_MODEL="gpt-4o-mini"
export VETTING_DEFAULT_VERIFICATION_MODEL="gpt-4o-mini"
export VETTING_DEFAULT_PROVIDER="openai"
# Generation Parameters
export VETTING_TEMPERATURE_CHAT="0.7"
export VETTING_TEMPERATURE_VERIFICATION="0.1"
export VETTING_MAX_TOKENS_CHAT="1024"
export VETTING_MAX_TOKENS_VERIFICATION="512"
export VETTING_MAX_ATTEMPTS="3"
# Features
export VETTING_ENABLE_SAFETY_PREFIX="true"
export VETTING_ENABLE_EDUCATIONAL_RULES="true"
export VETTING_ENABLE_COST_TRACKING="true"
# Logging
export VETTING_LOG_LEVEL="INFO"
export VETTING_LOG_REQUESTS="false"
```
### Configuration Files
You can also use JSON or YAML configuration files:
```json
{
"providers": {
"openai": {
"provider_type": "openai",
"api_key": "your-key",
"timeout": 60,
"max_retries": 3
}
},
"default_provider": "openai",
"default_chat_model": "gpt-4o-mini",
"default_verification_model": "gpt-4o-mini",
"enable_safety_prefix": true,
"enable_educational_rules": true
}
```
```python
from vetting_python.config import VettingSettings
# Load from file
settings = VettingSettings.from_file("config.json")
# Load from environment
settings = VettingSettings.from_env()
# Create provider and framework
provider = settings.get_provider_instance("openai")
framework = VettingFramework(chat_provider=provider)
```
## Advanced Usage
### Multi-Provider Setup
```python
from vetting_python import OpenAIProvider, ClaudeProvider
# Use different providers for chat and verification
chat_provider = OpenAIProvider(api_key="openai-key")
verification_provider = ClaudeProvider(api_key="claude-key")
framework = VettingFramework(
chat_provider=chat_provider,
verification_provider=verification_provider
)
config = VettingConfig(
mode="vetting",
chat_model={"model_id": "gpt-4o-mini"},
verification_model={"model_id": "claude-3-haiku"}
)
```
### Cost Tracking
```python
from vetting_python.utils import CostTracker
# Setup cost tracking
cost_tracker = CostTracker(enable_persistence=True)
# After processing requests
cost_tracker.track_response(response, "openai", provider, provider)
# Get cost summary
summary = cost_tracker.get_summary()
print(f"Total cost: ${summary.total_cost:.4f}")
print(f"Total tokens: {summary.total_tokens}")
# Print detailed breakdown
cost_tracker.print_summary()
```
### Complex Educational Scenarios
```python
config = (VettingConfigBuilder()
.vetting_mode()
.chat_model("gpt-4o-mini", temperature=0.8)
.verification_model("gpt-4o-mini", temperature=0.1)
.chat_system_prompt(
"You are a Socratic tutor. Guide students through discovery "
"rather than giving direct answers. Always end with a question."
)
# Multiple context items
.add_context_item(
question_text="What is photosynthesis?",
subject="Biology",
correct_answer="The process by which plants convert light energy into chemical energy",
key_concepts=["photosynthesis", "chlorophyll", "glucose", "oxygen"],
explanation="Plants use sunlight, CO2, and water to produce glucose and oxygen"
)
.add_context_item(
question_text="What gas do plants absorb during photosynthesis?",
subject="Biology",
correct_answer="Carbon dioxide",
key_concepts=["carbon dioxide", "CO2"]
)
.safety_features(enable_educational_rules=True)
.session_info(session_id="tutoring_001", user_id="student_123")
.build())
```
### Validation and Error Handling
```python
from vetting_python.utils import ValidationUtils
# Validate configuration
validation = ValidationUtils.validate_vetting_config(config)
if not validation["valid"]:
print(f"Config errors: {validation['issues']}")
# Validate messages
validation = ValidationUtils.validate_messages(messages)
if validation["warnings"]:
print(f"Message warnings: {validation['warnings']}")
# Validate API key format
validation = ValidationUtils.validate_api_key(api_key, "openai")
if not validation["valid"]:
print(f"API key issues: {validation['issues']}")
```
## API Reference
### Core Classes
#### `VettingFramework`
The main framework class that orchestrates the dual-LLM architecture.
```python
VettingFramework(
chat_provider: Provider,
verification_provider: Optional[Provider] = None
)
```
**Methods:**
- `async process(messages: List[ChatMessage], config: VettingConfig) -> VettingResponse`
#### `VettingConfig`
Configuration object for the vetting process.
```python
VettingConfig(
mode: Literal["chat", "vetting"] = "vetting",
chat_model: ModelConfig,
verification_model: Optional[ModelConfig] = None,
max_attempts: int = 3,
chat_system_prompt: Optional[str] = None,
verification_system_prompt: Optional[str] = None,
context_items: Optional[List[ContextItem]] = None,
session_id: Optional[str] = None,
user_id: Optional[str] = None,
enable_safety_prefix: bool = True,
enable_educational_rules: bool = True
)
```
#### `VettingResponse`
Response object containing the result and metadata.
```python
@dataclass
class VettingResponse:
content: str
mode: Literal["chat", "vetting"]
requires_attention: bool = False
verification_passed: Optional[bool] = None
attempt_count: int = 1
stop_reason: Optional[StopReason] = None
attempts: Optional[List[AttemptDetail]] = None
chat_usage: Optional[Usage] = None
verification_usage: Optional[Usage] = None
total_usage: Optional[Usage] = None
total_cost: float = 0.0
processing_time_ms: Optional[float] = None
# ... additional metadata fields
```
### Configuration Builder
The `VettingConfigBuilder` provides a fluent API for building configurations:
```python
config = (VettingConfigBuilder()
.vetting_mode() # or .chat_mode()
.chat_model("gpt-4o-mini", temperature=0.7, max_tokens=1024)
.verification_model("gpt-4o-mini", temperature=0.1, max_tokens=512)
.max_attempts(3)
.add_context_item(
question_text="What is X?",
correct_answer="Y",
key_concepts=["concept1", "concept2"]
)
.safety_features(enable_safety_prefix=True, enable_educational_rules=True)
.session_info(session_id="session_123", user_id="user_456")
.build())
```
### Providers
#### `OpenAIProvider`
```python
OpenAIProvider(
api_key: str,
base_url: str = "https://api.openai.com/v1",
max_retries: int = 3,
timeout: int = 60,
organization: Optional[str] = None
)
```
**Supported Models (2025 Pricing):**
- `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`
- `gpt-4o`, `gpt-4o-mini`
- Aliases: `gpt-4o-latest` → `gpt-4o`
#### `ClaudeProvider`
```python
ClaudeProvider(
api_key: str,
base_url: str = "https://api.anthropic.com",
max_retries: int = 3,
timeout: int = 60
)
```
**Supported Models (2025 Pricing):**
- `claude-sonnet-4`, `claude-sonnet-3.7`, `claude-sonnet-3.5`
- Aliases: `claude-4` → `claude-sonnet-4`
#### `GeminiProvider`
```python
GeminiProvider(
api_key: str,
base_url: str = "https://generativelanguage.googleapis.com",
max_retries: int = 3,
timeout: int = 60
)
```
**Supported Models (2025 Pricing):**
- `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`
- `gemini-2.0-flash`, `gemini-2.0-flash-lite`
- Aliases: `gemini-2.5` → `gemini-2.5-pro`
## Use Cases
### 1. Educational Tutoring
Perfect for homework help platforms where you want to guide learning without giving away answers:
```python
# Student asks: "What is the quadratic formula?"
# Instead of giving the formula directly, VETTING guides:
# "Great question! Let's think about this step by step.
# What do you know about quadratic equations? What form do they take?"
```
### 2. Assessment Integrity
Maintain assessment integrity while still providing help:
```python
# During an exam, student asks for direct answer
# VETTING detects this violates policy and provides guidance instead:
# "I can't give you the direct answer, but I can help you think through
# the problem. What approach would you take to solve this type of question?"
```
### 3. Content Safety
Prevent harmful or inappropriate responses while maintaining helpful interaction:
```python
# User asks about dangerous activities
# VETTING detects safety concern and responds appropriately:
# "[REQUIRES_ATTENTION] I understand you're curious, but I can't provide
# information that could be harmful. Instead, let me suggest some safe
# alternatives..."
```
### 4. Corporate Training
Ensure training materials adhere to company policies and learning objectives:
```python
# Training scenario with specific learning outcomes
# VETTING ensures responses align with corporate training goals
# while preventing disclosure of confidential information
```
## Best Practices
### 1. Configuration Management
- Use environment variables for API keys and basic settings
- Use configuration files for complex setups
- Validate configurations before use
- Keep verification model parameters more conservative (lower temperature)
### 2. Cost Management
- Enable cost tracking in production
- Monitor usage patterns and optimize model selection
- Use cheaper models for verification when possible
- Set up cost alerts for production systems
### 3. Educational Applications
- Design clear learning objectives for context items
- Use specific key concepts to avoid revealing
- Set appropriate maximum attempts (2-3 for homework, 1 for assessments)
- Always include explanations in answer keys for better verification
### 4. Error Handling
- Always validate inputs before processing
- Implement proper retry logic for provider failures
- Log verification failures for analysis
- Have fallback responses for system errors
### 5. Production Deployment
- Use connection pooling for high-throughput applications
- Implement proper monitoring and alerting
- Cache provider instances to avoid recreation overhead
- Set up log aggregation for debugging
## Examples
The `vetting_python/examples/` directory contains comprehensive examples:
- `basic_usage.py` - Getting started examples
- `advanced_usage.py` - Complex scenarios and custom providers
- `integration_patterns.py` - Web API and platform integration examples
Run the examples:
```bash
cd vetting_python/examples
python basic_usage.py
python advanced_usage.py
python integration_patterns.py
```
## Contributing
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
### Development Setup
```bash
git clone https://github.com/your-org/vetting-python.git
cd vetting-python
pip install -e ".[dev]"
```
### Running Tests
```bash
pytest tests/
```
### Code Style
```bash
black vetting_python/
isort vetting_python/
mypy vetting_python/
```
## Research Citation
If you use VETTING in your research, please cite our paper (citation will be updated upon publication):
```bibtex
@misc{vetting2025,
title={VETTING: Verification and Evaluation Tool for Targeting Invalid Narrative Generation},
author={VETTING Research Team},
year={2025},
note={Available at: https://github.com/hichipli/vetting-python}
}
```
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## Support
- 📚 Documentation: [README.md](https://github.com/hichipli/vetting-python#readme)
- 🐛 Issues: [GitHub Issues](https://github.com/hichipli/vetting-python/issues)
- 💬 Discussions: [GitHub Discussions](https://github.com/hichipli/vetting-python/discussions)
- 🌐 Research Lab: [VIABLE Lab](https://www.viablelab.org/)
- 📧 Contact: [Contact Form](https://www.viablelab.org/contact) or hli3@ufl.edu
## Changelog
### v0.1.0 (2025-07-31)
- ✅ Dual-LLM architecture implementation
- ✅ OpenAI, Claude, and Gemini provider support
- ✅ Educational vetting capabilities
- ✅ Cost tracking and monitoring
- ✅ Comprehensive configuration system
- ✅ Safety feature integration
- ✅ Example applications and documentation
---
Built with ❤️ for safer and more effective AI interactions in education and beyond.
Raw data
{
"_id": null,
"home_page": null,
"name": "vetting-python",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "llm, ai-safety, education, chatbot, verification, dual-llm, prompt-engineering, educational-technology, machine-learning, natural-language-processing",
"author": null,
"author_email": "VETTING Research Team <hli3@ufl.edu>",
"download_url": "https://files.pythonhosted.org/packages/4f/3b/a79ed18d090310f4e088ded4e889548b4aa5d064b2365fa1ff95b27091e3/vetting_python-0.1.0.tar.gz",
"platform": null,
"description": "# VETTING Framework - Python Implementation\n\nA Python implementation of the VETTING (Verification and Evaluation Tool for Targeting Invalid Narrative Generation) framework for LLM safety and educational applications.\n\n**Developed by [VIABLE Lab](https://www.viablelab.org/) at the University of Florida.**\n\n## Overview\n\nThe VETTING framework implements a dual-LLM architecture that separates conversational logic from policy enforcement, preventing prompt injection attacks on safety rules and enabling verifiable policy compliance. This architectural approach is particularly effective for educational applications where you need to guide learning without revealing direct answers.\n\n### Key Features\n\n- **\ud83d\udee1\ufe0f Architectural Policy Isolation**: Complete separation between user interaction (Chat-Layer) and policy enforcement (Verification-Layer)\n- **\ud83d\udd04 Iterative Verification Loop**: Automatic refinement when responses don't meet verification criteria\n- **\ud83c\udfeb Educational Focus**: Specialized support for tutoring and homework help scenarios\n- **\ud83c\udf10 Multi-Provider Support**: Works with OpenAI, Anthropic Claude, and Google Gemini\n- **\ud83d\udcb0 Cost Tracking**: Comprehensive cost monitoring and analysis\n- **\u2699\ufe0f Flexible Configuration**: Environment variables, config files, or programmatic setup\n- **\ud83d\udd0d Safety Features**: Built-in safety prefix detection and content filtering\n\n## Architecture\n\n```\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 User Input \u2502 \u2502 Chat-Layer \u2502\n\u2502 \u2502\u2500\u2500\u2500\u25b6\u2502 (LLM-A) \u2502\n\u2502 \u2502 \u2502 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Verification- \u2502\n \u2502 Layer \u2502\u25c0\u2500\u2500\u2500 Confidential\n \u2502 (LLM-B) \u2502 Policy\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u25bc\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Pass/Fail + \u2502\n \u2502 Feedback \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n## Installation\n\n### From PyPI (once published):\n\n```bash\npip install vetting-python\n```\n\n### From Source:\n\n```bash\ngit clone https://github.com/hichipli/vetting-python.git\ncd vetting-python\npip install -e .\n```\n\n### Dependencies\n\n```bash\npip install aiohttp pydantic dataclasses-json\n```\n\nOptional dependencies:\n```bash\npip install PyYAML # For YAML configuration files\n```\n\n## Quick Start\n\n### 1. Set up your API keys\n\n```bash\nexport OPENAI_API_KEY=\"your-openai-api-key\"\nexport ANTHROPIC_API_KEY=\"your-claude-api-key\" # Optional\nexport GOOGLE_API_KEY=\"your-gemini-api-key\" # Optional\n```\n\n### 2. Basic Chat Mode\n\n```python\nimport asyncio\nfrom vetting_python import VettingFramework, VettingConfig, ChatMessage, OpenAIProvider\n\nasync def basic_example():\n # Setup provider\n provider = OpenAIProvider(api_key=\"your-api-key\")\n \n # Create framework\n framework = VettingFramework(chat_provider=provider)\n \n # Simple chat configuration\n config = VettingConfig(\n mode=\"chat\",\n chat_model={\"model_id\": \"gpt-4o-mini\", \"temperature\": 0.7}\n )\n \n # Create conversation\n messages = [ChatMessage(\"user\", \"Explain photosynthesis in simple terms.\")]\n \n # Process\n response = await framework.process(messages, config)\n print(f\"Response: {response.content}\")\n print(f\"Cost: ${response.total_cost:.4f}\")\n\n# Run the example\nasyncio.run(basic_example())\n```\n\n### 3. Educational Vetting Mode\n\n```python\nimport asyncio\nfrom vetting_python import VettingFramework, OpenAIProvider\nfrom vetting_python.config import VettingConfigBuilder\n\nasync def educational_example():\n provider = OpenAIProvider(api_key=\"your-api-key\")\n framework = VettingFramework(chat_provider=provider)\n \n # Educational configuration with answer key\n config = (VettingConfigBuilder()\n .vetting_mode()\n .chat_model(\"gpt-4o-mini\")\n .verification_model(\"gpt-4o-mini\")\n .add_context_item(\n question_text=\"What is the capital of France?\",\n correct_answer=\"Paris\",\n key_concepts=[\"Paris\", \"France\", \"capital city\"]\n )\n .build())\n \n # Student asks directly for the answer\n messages = [ChatMessage(\"user\", \"What is the capital of France? I need this for homework.\")]\n \n # Process with verification\n response = await framework.process(messages, config)\n \n print(f\"Response: {response.content}\")\n print(f\"Verification passed: {response.verification_passed}\")\n print(f\"Attempts made: {response.attempt_count}\")\n\nasyncio.run(educational_example())\n```\n\n## Configuration\n\n### Environment Variables\n\nThe framework supports comprehensive configuration through environment variables:\n\n```bash\n# API Keys\nexport OPENAI_API_KEY=\"sk-...\"\nexport ANTHROPIC_API_KEY=\"sk-ant-...\"\nexport GOOGLE_API_KEY=\"...\"\n\n# Default Models\nexport VETTING_DEFAULT_CHAT_MODEL=\"gpt-4o-mini\"\nexport VETTING_DEFAULT_VERIFICATION_MODEL=\"gpt-4o-mini\"\nexport VETTING_DEFAULT_PROVIDER=\"openai\"\n\n# Generation Parameters\nexport VETTING_TEMPERATURE_CHAT=\"0.7\"\nexport VETTING_TEMPERATURE_VERIFICATION=\"0.1\"\nexport VETTING_MAX_TOKENS_CHAT=\"1024\"\nexport VETTING_MAX_TOKENS_VERIFICATION=\"512\"\nexport VETTING_MAX_ATTEMPTS=\"3\"\n\n# Features\nexport VETTING_ENABLE_SAFETY_PREFIX=\"true\"\nexport VETTING_ENABLE_EDUCATIONAL_RULES=\"true\"\nexport VETTING_ENABLE_COST_TRACKING=\"true\"\n\n# Logging\nexport VETTING_LOG_LEVEL=\"INFO\"\nexport VETTING_LOG_REQUESTS=\"false\"\n```\n\n### Configuration Files\n\nYou can also use JSON or YAML configuration files:\n\n```json\n{\n \"providers\": {\n \"openai\": {\n \"provider_type\": \"openai\",\n \"api_key\": \"your-key\",\n \"timeout\": 60,\n \"max_retries\": 3\n }\n },\n \"default_provider\": \"openai\",\n \"default_chat_model\": \"gpt-4o-mini\",\n \"default_verification_model\": \"gpt-4o-mini\",\n \"enable_safety_prefix\": true,\n \"enable_educational_rules\": true\n}\n```\n\n```python\nfrom vetting_python.config import VettingSettings\n\n# Load from file\nsettings = VettingSettings.from_file(\"config.json\")\n\n# Load from environment\nsettings = VettingSettings.from_env()\n\n# Create provider and framework\nprovider = settings.get_provider_instance(\"openai\")\nframework = VettingFramework(chat_provider=provider)\n```\n\n## Advanced Usage\n\n### Multi-Provider Setup\n\n```python\nfrom vetting_python import OpenAIProvider, ClaudeProvider\n\n# Use different providers for chat and verification\nchat_provider = OpenAIProvider(api_key=\"openai-key\")\nverification_provider = ClaudeProvider(api_key=\"claude-key\")\n\nframework = VettingFramework(\n chat_provider=chat_provider,\n verification_provider=verification_provider\n)\n\nconfig = VettingConfig(\n mode=\"vetting\",\n chat_model={\"model_id\": \"gpt-4o-mini\"},\n verification_model={\"model_id\": \"claude-3-haiku\"}\n)\n```\n\n### Cost Tracking\n\n```python\nfrom vetting_python.utils import CostTracker\n\n# Setup cost tracking\ncost_tracker = CostTracker(enable_persistence=True)\n\n# After processing requests\ncost_tracker.track_response(response, \"openai\", provider, provider)\n\n# Get cost summary\nsummary = cost_tracker.get_summary()\nprint(f\"Total cost: ${summary.total_cost:.4f}\")\nprint(f\"Total tokens: {summary.total_tokens}\")\n\n# Print detailed breakdown\ncost_tracker.print_summary()\n```\n\n### Complex Educational Scenarios\n\n```python\nconfig = (VettingConfigBuilder()\n .vetting_mode()\n .chat_model(\"gpt-4o-mini\", temperature=0.8)\n .verification_model(\"gpt-4o-mini\", temperature=0.1)\n .chat_system_prompt(\n \"You are a Socratic tutor. Guide students through discovery \"\n \"rather than giving direct answers. Always end with a question.\"\n )\n # Multiple context items\n .add_context_item(\n question_text=\"What is photosynthesis?\",\n subject=\"Biology\",\n correct_answer=\"The process by which plants convert light energy into chemical energy\",\n key_concepts=[\"photosynthesis\", \"chlorophyll\", \"glucose\", \"oxygen\"],\n explanation=\"Plants use sunlight, CO2, and water to produce glucose and oxygen\"\n )\n .add_context_item(\n question_text=\"What gas do plants absorb during photosynthesis?\",\n subject=\"Biology\",\n correct_answer=\"Carbon dioxide\",\n key_concepts=[\"carbon dioxide\", \"CO2\"]\n )\n .safety_features(enable_educational_rules=True)\n .session_info(session_id=\"tutoring_001\", user_id=\"student_123\")\n .build())\n```\n\n### Validation and Error Handling\n\n```python\nfrom vetting_python.utils import ValidationUtils\n\n# Validate configuration\nvalidation = ValidationUtils.validate_vetting_config(config)\nif not validation[\"valid\"]:\n print(f\"Config errors: {validation['issues']}\")\n\n# Validate messages\nvalidation = ValidationUtils.validate_messages(messages)\nif validation[\"warnings\"]:\n print(f\"Message warnings: {validation['warnings']}\")\n\n# Validate API key format\nvalidation = ValidationUtils.validate_api_key(api_key, \"openai\")\nif not validation[\"valid\"]:\n print(f\"API key issues: {validation['issues']}\")\n```\n\n## API Reference\n\n### Core Classes\n\n#### `VettingFramework`\n\nThe main framework class that orchestrates the dual-LLM architecture.\n\n```python\nVettingFramework(\n chat_provider: Provider,\n verification_provider: Optional[Provider] = None\n)\n```\n\n**Methods:**\n- `async process(messages: List[ChatMessage], config: VettingConfig) -> VettingResponse`\n\n#### `VettingConfig`\n\nConfiguration object for the vetting process.\n\n```python\nVettingConfig(\n mode: Literal[\"chat\", \"vetting\"] = \"vetting\",\n chat_model: ModelConfig,\n verification_model: Optional[ModelConfig] = None,\n max_attempts: int = 3,\n chat_system_prompt: Optional[str] = None,\n verification_system_prompt: Optional[str] = None,\n context_items: Optional[List[ContextItem]] = None,\n session_id: Optional[str] = None,\n user_id: Optional[str] = None,\n enable_safety_prefix: bool = True,\n enable_educational_rules: bool = True\n)\n```\n\n#### `VettingResponse`\n\nResponse object containing the result and metadata.\n\n```python\n@dataclass\nclass VettingResponse:\n content: str\n mode: Literal[\"chat\", \"vetting\"]\n requires_attention: bool = False\n verification_passed: Optional[bool] = None\n attempt_count: int = 1\n stop_reason: Optional[StopReason] = None\n attempts: Optional[List[AttemptDetail]] = None\n chat_usage: Optional[Usage] = None\n verification_usage: Optional[Usage] = None\n total_usage: Optional[Usage] = None\n total_cost: float = 0.0\n processing_time_ms: Optional[float] = None\n # ... additional metadata fields\n```\n\n### Configuration Builder\n\nThe `VettingConfigBuilder` provides a fluent API for building configurations:\n\n```python\nconfig = (VettingConfigBuilder()\n .vetting_mode() # or .chat_mode()\n .chat_model(\"gpt-4o-mini\", temperature=0.7, max_tokens=1024)\n .verification_model(\"gpt-4o-mini\", temperature=0.1, max_tokens=512)\n .max_attempts(3)\n .add_context_item(\n question_text=\"What is X?\",\n correct_answer=\"Y\",\n key_concepts=[\"concept1\", \"concept2\"]\n )\n .safety_features(enable_safety_prefix=True, enable_educational_rules=True)\n .session_info(session_id=\"session_123\", user_id=\"user_456\")\n .build())\n```\n\n### Providers\n\n#### `OpenAIProvider`\n\n```python\nOpenAIProvider(\n api_key: str,\n base_url: str = \"https://api.openai.com/v1\",\n max_retries: int = 3,\n timeout: int = 60,\n organization: Optional[str] = None\n)\n```\n\n**Supported Models (2025 Pricing):**\n- `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`\n- `gpt-4o`, `gpt-4o-mini`\n- Aliases: `gpt-4o-latest` \u2192 `gpt-4o`\n\n#### `ClaudeProvider`\n\n```python\nClaudeProvider(\n api_key: str,\n base_url: str = \"https://api.anthropic.com\",\n max_retries: int = 3,\n timeout: int = 60\n)\n```\n\n**Supported Models (2025 Pricing):**\n- `claude-sonnet-4`, `claude-sonnet-3.7`, `claude-sonnet-3.5`\n- Aliases: `claude-4` \u2192 `claude-sonnet-4`\n\n#### `GeminiProvider`\n\n```python\nGeminiProvider(\n api_key: str,\n base_url: str = \"https://generativelanguage.googleapis.com\",\n max_retries: int = 3,\n timeout: int = 60\n)\n```\n\n**Supported Models (2025 Pricing):**\n- `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`\n- `gemini-2.0-flash`, `gemini-2.0-flash-lite`\n- Aliases: `gemini-2.5` \u2192 `gemini-2.5-pro`\n\n## Use Cases\n\n### 1. Educational Tutoring\n\nPerfect for homework help platforms where you want to guide learning without giving away answers:\n\n```python\n# Student asks: \"What is the quadratic formula?\"\n# Instead of giving the formula directly, VETTING guides:\n# \"Great question! Let's think about this step by step. \n# What do you know about quadratic equations? What form do they take?\"\n```\n\n### 2. Assessment Integrity\n\nMaintain assessment integrity while still providing help:\n\n```python\n# During an exam, student asks for direct answer\n# VETTING detects this violates policy and provides guidance instead:\n# \"I can't give you the direct answer, but I can help you think through \n# the problem. What approach would you take to solve this type of question?\"\n```\n\n### 3. Content Safety\n\nPrevent harmful or inappropriate responses while maintaining helpful interaction:\n\n```python\n# User asks about dangerous activities\n# VETTING detects safety concern and responds appropriately:\n# \"[REQUIRES_ATTENTION] I understand you're curious, but I can't provide \n# information that could be harmful. Instead, let me suggest some safe \n# alternatives...\"\n```\n\n### 4. Corporate Training\n\nEnsure training materials adhere to company policies and learning objectives:\n\n```python\n# Training scenario with specific learning outcomes\n# VETTING ensures responses align with corporate training goals\n# while preventing disclosure of confidential information\n```\n\n## Best Practices\n\n### 1. Configuration Management\n\n- Use environment variables for API keys and basic settings\n- Use configuration files for complex setups\n- Validate configurations before use\n- Keep verification model parameters more conservative (lower temperature)\n\n### 2. Cost Management\n\n- Enable cost tracking in production\n- Monitor usage patterns and optimize model selection\n- Use cheaper models for verification when possible\n- Set up cost alerts for production systems\n\n### 3. Educational Applications\n\n- Design clear learning objectives for context items\n- Use specific key concepts to avoid revealing\n- Set appropriate maximum attempts (2-3 for homework, 1 for assessments)\n- Always include explanations in answer keys for better verification\n\n### 4. Error Handling\n\n- Always validate inputs before processing\n- Implement proper retry logic for provider failures\n- Log verification failures for analysis\n- Have fallback responses for system errors\n\n### 5. Production Deployment\n\n- Use connection pooling for high-throughput applications\n- Implement proper monitoring and alerting\n- Cache provider instances to avoid recreation overhead\n- Set up log aggregation for debugging\n\n## Examples\n\nThe `vetting_python/examples/` directory contains comprehensive examples:\n\n- `basic_usage.py` - Getting started examples\n- `advanced_usage.py` - Complex scenarios and custom providers\n- `integration_patterns.py` - Web API and platform integration examples\n\nRun the examples:\n\n```bash\ncd vetting_python/examples\npython basic_usage.py\npython advanced_usage.py\npython integration_patterns.py\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n### Development Setup\n\n```bash\ngit clone https://github.com/your-org/vetting-python.git\ncd vetting-python\npip install -e \".[dev]\"\n```\n\n### Running Tests\n\n```bash\npytest tests/\n```\n\n### Code Style\n\n```bash\nblack vetting_python/\nisort vetting_python/\nmypy vetting_python/\n```\n\n## Research Citation\n\nIf you use VETTING in your research, please cite our paper (citation will be updated upon publication):\n\n```bibtex\n@misc{vetting2025,\n title={VETTING: Verification and Evaluation Tool for Targeting Invalid Narrative Generation},\n author={VETTING Research Team},\n year={2025},\n note={Available at: https://github.com/hichipli/vetting-python}\n}\n```\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Support\n\n- \ud83d\udcda Documentation: [README.md](https://github.com/hichipli/vetting-python#readme)\n- \ud83d\udc1b Issues: [GitHub Issues](https://github.com/hichipli/vetting-python/issues)\n- \ud83d\udcac Discussions: [GitHub Discussions](https://github.com/hichipli/vetting-python/discussions)\n- \ud83c\udf10 Research Lab: [VIABLE Lab](https://www.viablelab.org/)\n- \ud83d\udce7 Contact: [Contact Form](https://www.viablelab.org/contact) or hli3@ufl.edu\n\n## Changelog\n\n### v0.1.0 (2025-07-31)\n\n- \u2705 Dual-LLM architecture implementation\n- \u2705 OpenAI, Claude, and Gemini provider support\n- \u2705 Educational vetting capabilities\n- \u2705 Cost tracking and monitoring\n- \u2705 Comprehensive configuration system\n- \u2705 Safety feature integration\n- \u2705 Example applications and documentation\n\n---\n\nBuilt with \u2764\ufe0f for safer and more effective AI interactions in education and beyond.\n",
"bugtrack_url": null,
"license": "MIT License\n \n Copyright (c) 2025 Hongming (Chip) Li\n \n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n \n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n \n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE.\n ",
"summary": "A Python implementation of the VETTING (Verification and Evaluation Tool for Targeting Invalid Narrative Generation) framework for LLM safety and educational applications",
"version": "0.1.0",
"project_urls": {
"Changelog": "https://github.com/hichipli/vetting-python/blob/main/CHANGELOG.md",
"Documentation": "https://github.com/hichipli/vetting-python#readme",
"Homepage": "https://github.com/hichipli/vetting-python",
"Issues": "https://github.com/hichipli/vetting-python/issues",
"Repository": "https://github.com/hichipli/vetting-python.git"
},
"split_keywords": [
"llm",
" ai-safety",
" education",
" chatbot",
" verification",
" dual-llm",
" prompt-engineering",
" educational-technology",
" machine-learning",
" natural-language-processing"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "4fb63eeb5674a6aaffcb5c701d6eefbd2b5a17791eb5bffe584a4a661464abb8",
"md5": "6eb26b32449cc0adfd54eb1773602e20",
"sha256": "a29060d27c9fb310b5788142724c4aa9956e1b457620eedc233617500160697c"
},
"downloads": -1,
"filename": "vetting_python-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6eb26b32449cc0adfd54eb1773602e20",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 61675,
"upload_time": "2025-08-01T04:24:00",
"upload_time_iso_8601": "2025-08-01T04:24:00.760191Z",
"url": "https://files.pythonhosted.org/packages/4f/b6/3eeb5674a6aaffcb5c701d6eefbd2b5a17791eb5bffe584a4a661464abb8/vetting_python-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "4f3ba79ed18d090310f4e088ded4e889548b4aa5d064b2365fa1ff95b27091e3",
"md5": "479bc30a3fafd1b91a4f710222308336",
"sha256": "0c7a43320bbc3785048965a98ffb3b1a86f1356897ec5b5298ebdc6a4109762e"
},
"downloads": -1,
"filename": "vetting_python-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "479bc30a3fafd1b91a4f710222308336",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 57435,
"upload_time": "2025-08-01T04:24:02",
"upload_time_iso_8601": "2025-08-01T04:24:02.339062Z",
"url": "https://files.pythonhosted.org/packages/4f/3b/a79ed18d090310f4e088ded4e889548b4aa5d064b2365fa1ff95b27091e3/vetting_python-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-01 04:24:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "hichipli",
"github_project": "vetting-python",
"github_not_found": true,
"lcname": "vetting-python"
}
