# Promptron
A Python package for generating evaluation datasets using Large Language Models (LLMs). Promptron helps you create structured question datasets for testing and evaluating LLM applications through code-only API.
## Features
- **Multi-Provider Support**: Works with Ollama (local), OpenAI, and Anthropic with auto-detection
- **LLM-Powered Generation**: Automatically generates questions using your chosen LLM provider
- **5 Evaluation Categories**: Pre-configured templates for comprehensive LLM evaluation
- **Enhanced Config Fields**: Support for context, audience, examples, and keywords for better accuracy
- **Code-Only API**: Simple Python functions for automation and integration
- **Flexible Configuration**: Use YAML config file or pass prompts directly (simple or enhanced)
- **Backward Compatible**: Simple configs still work; enhanced fields are optional
- **Structured Output**: Generates JSON datasets ready for evaluation pipelines
## Installation
### Prerequisites
- Python 3.8 or higher
- One of the following:
- **Ollama** (local): [Install Ollama](https://ollama.ai/) and download a model (e.g., `ollama pull llama3:latest`)
- **OpenAI**: API key from [OpenAI](https://platform.openai.com/)
- **Anthropic**: API key from [Anthropic](https://www.anthropic.com/)
### Install from PyPI
**Base installation (Ollama only):**
```bash
pip install promptron
```
**With OpenAI support:**
```bash
pip install promptron[openai]
```
**With Anthropic support:**
```bash
pip install promptron[anthropic]
```
**With all providers:**
```bash
pip install promptron[openai,anthropic]
```
## Quick Start
### 1. Install Promptron
```bash
pip install promptron
```
### 2. Ensure Ollama is Running
```bash
ollama serve
```
### 3. Download a Model (default: llama3:latest)
```bash
ollama pull llama3:latest
```
**Optional:** Use a different model by setting environment variable:
```bash
export PROMPTRON_MODEL=llama3.2:latest
```
### 4. Choose Your Usage Method
Promptron supports two ways to provide prompts:
**Method 1: Direct Prompts Array (Simplest)**
```python
from promptron import generate_prompts
generate_prompts(
prompts=[
{"category": "default", "topic": "openshift", "count": 5}
]
)
```
**Method 2: Config File (Recommended for Projects)**
Create `config.yml` manually in your project directory:
**Simple config (backward compatible):**
```yaml
prompts:
- category: "default"
topic: "openshift"
count: 5
- category: "red_teaming"
topic: "kubernetes"
count: 3
```
**Enhanced config (for better accuracy):**
```yaml
prompts:
- category: "default"
topic: "kubernetes networking"
count: 5
context: |
Kubernetes networking involves Services, Ingress, NetworkPolicies, and CNI plugins.
Focus on practical scenarios like service discovery, load balancing, and network isolation.
audience: "intermediate" # beginner/intermediate/expert
examples:
- "How do I configure a Service to expose pods to external traffic?"
- "What is the difference between ClusterIP and NodePort service types?"
keywords:
- "services"
- "ingress"
- "network policies"
```
Then use it:
```python
from promptron import generate_prompts
generate_prompts(config_file="./config.yml")
```
### 5. Generate Questions
**Option A: Using prompts array directly (no config file needed)**
```python
from promptron import generate_prompts
generate_prompts(
prompts=[
{"category": "default", "topic": "openshift", "count": 5}
],
artifacts_location="./artifacts",
single_file=False
)
```
**Option B: Using config file (create config.yml manually first)**
```python
from promptron import generate_prompts
# Uses config.yml you created manually
generate_prompts(
config_file="./config.yml",
artifacts_location="./artifacts",
single_file=False
)
```
**Option C: Using LLMConfig class (programmatic approach)**
**Ollama (Local):**
```python
from promptron import generate_prompts, LLMConfig
class MyLLMConfig(LLMConfig):
name = "llama3:latest" # Provider auto-detected from "llama"
# url defaults to http://localhost:11434
generate_prompts(
config_file="./config.yml",
llm_config=MyLLMConfig
)
```
**OpenAI:**
```python
from promptron import generate_prompts, LLMConfig
class MyLLMConfig(LLMConfig):
name = "gpt-4" # Provider auto-detected from "gpt-"
api_key = "sk-..." # Required for OpenAI
generate_prompts(
config_file="./config.yml",
llm_config=MyLLMConfig
)
```
**Anthropic:**
```python
from promptron import generate_prompts, LLMConfig
class MyLLMConfig(LLMConfig):
name = "claude-3-opus-20240229" # Provider auto-detected from "claude-"
api_key = "sk-ant-..." # Required for Anthropic
generate_prompts(
config_file="./config.yml",
llm_config=MyLLMConfig
)
```
## Recommended Categories
Promptron provides 5 recommended categories for comprehensive LLM evaluation:
1. **"default"** - Standard, straightforward questions for baseline evaluation
2. **"red_teaming"** - Adversarial, tricky, or misleading questions to test robustness and safety
3. **"out_of_scope"** - Questions outside the domain to test boundary handling
4. **"edge_cases"** - Unusual, extreme, or corner-case scenarios to test edge case handling
5. **"reasoning"** - Multi-step, complex, analytical questions to test reasoning depth
**Note:** Using categories outside these recommended ones may reduce prompt accuracy. The system will fallback to the "default" template with a warning.
## Usage Examples
### Method 1: Direct Prompts Array (Simplest - No Config File)
```python
from promptron import generate_prompts
# No config file needed - pass prompts directly
generate_prompts(
prompts=[
{"category": "default", "topic": "openshift", "count": 5},
{"category": "red_teaming", "topic": "kubernetes", "count": 3}
],
artifacts_location="./output",
single_file=True,
output_format="jsonl"
)
```
### Method 2: Using config.yml File with .env
**First, create `config.yml` manually:**
```yaml
prompts:
- category: "default"
topic: "openshift"
count: 5
context: "OpenShift is a Kubernetes-based container platform. Focus on pod management and resource limits."
audience: "intermediate"
- category: "red_teaming"
topic: "kubernetes"
count: 3
context: "Focus on security vulnerabilities and misconfigurations in Kubernetes."
```
**Then use it:**
```python
from promptron import generate_prompts
# Reads LLM config from .env file
generate_prompts(
config_file="./config.yml",
artifacts_location="./output",
single_file=True,
output_format="jsonl"
)
```
### Method 3: Using config.yml File with LLMConfig
**First, create `config.yml` manually (same as Method 2).**
**Then use it with LLMConfig:**
```python
from promptron import generate_prompts, LLMConfig
from dotenv import load_dotenv
import os
load_dotenv()
class MyLLMConfig(LLMConfig):
name = os.getenv("PROMPTRON_MODEL", "llama3:latest")
provider = os.getenv("PROMPTRON_PROVIDER", "ollama")
url = os.getenv("PROMPTRON_BASE_URL", "http://localhost:11434")
generate_prompts(
config_file="./config.yml",
artifacts_location="./output",
llm_config=MyLLMConfig
)
```
### Method 4: Direct Prompts with LLMConfig (No YAML File)
```python
from promptron import generate_prompts, LLMConfig
from dotenv import load_dotenv
import os
load_dotenv()
class MyLLMConfig(LLMConfig):
name = os.getenv("PROMPTRON_MODEL", "llama3:latest")
provider = os.getenv("PROMPTRON_PROVIDER", "ollama")
url = os.getenv("PROMPTRON_BASE_URL", "http://localhost:11434")
# Pass prompts directly
generate_prompts(
prompts=[
{"category": "default", "topic": "openshift", "count": 5},
{"category": "red_teaming", "topic": "kubernetes", "count": 3}
],
artifacts_location="./artifacts",
llm_config=MyLLMConfig
)
```
### Complete Workflow Example
**Workflow A: Using .env file**
```python
from promptron import generate_prompts
# 1. Create config.yml manually (see structure in README)
# 2. Create .env file with your LLM settings (optional, can use LLMConfig instead)
# 3. Generate questions (reads from .env automatically if not using LLMConfig)
generate_prompts(
config_file="./config.yml",
artifacts_location="./evaluation_data",
single_file=True
)
```
**Workflow B: Using LLMConfig class**
```python
from promptron import generate_prompts, LLMConfig
from dotenv import load_dotenv
import os
# 1. Create config.yml manually (see structure in README)
# 2. Load .env (user creates this)
load_dotenv()
# 3. Create LLMConfig class (user writes this)
class MyLLMConfig(LLMConfig):
name = os.getenv("PROMPTRON_MODEL", "llama3:latest")
provider = os.getenv("PROMPTRON_PROVIDER", "ollama")
url = os.getenv("PROMPTRON_BASE_URL", "http://localhost:11434")
# 4. Edit config.yml with your prompts
# 5. Generate questions with LLMConfig
generate_prompts(
config_file="./config.yml",
artifacts_location="./evaluation_data",
llm_config=MyLLMConfig,
single_file=True
)
```
## API Reference
### `LLMConfig`
Base class for LLM configuration. Supports **Ollama**, **OpenAI**, and **Anthropic** with auto-detection.
**Attributes:**
- `name` (str, required): Model name (e.g., "llama3:latest", "gpt-4", "claude-3-opus-20240229")
- `provider` (str, optional): LLM provider - "ollama", "openai", or "anthropic" (auto-detected if not set)
- `api_key` (str, optional): API key for OpenAI/Anthropic (required for those providers)
- `url` (str, optional): Base URL for Ollama (defaults to http://localhost:11434)
**Auto-detection rules:**
- Model name contains "llama" or "ollama" → provider = "ollama"
- Model name starts with "gpt-" or "text-" → provider = "openai"
- Model name starts with "claude-" → provider = "anthropic"
**Example (Ollama - auto-detected):**
```python
from promptron import LLMConfig
class MyLLMConfig(LLMConfig):
name = "llama3:latest" # Provider auto-detected
# url defaults to http://localhost:11434
```
**Example (OpenAI - auto-detected):**
```python
from promptron import LLMConfig
class MyLLMConfig(LLMConfig):
name = "gpt-4" # Provider auto-detected
api_key = "sk-..." # Required
```
**Example (Anthropic - auto-detected):**
```python
from promptron import LLMConfig
class MyLLMConfig(LLMConfig):
name = "claude-3-opus-20240229" # Provider auto-detected
api_key = "sk-ant-..." # Required
```
**Example (reading from .env):**
```python
from promptron import LLMConfig
from dotenv import load_dotenv
import os
load_dotenv()
class MyLLMConfig(LLMConfig):
name = os.getenv("PROMPTRON_MODEL", "llama3:latest")
provider = os.getenv("PROMPTRON_PROVIDER", None) # Auto-detect if None
api_key = os.getenv("PROMPTRON_API_KEY", None)
url = os.getenv("PROMPTRON_BASE_URL", None) # Optional for Ollama
```
### `generate_prompts(prompts=None, config_file=None, artifacts_location="./artifacts", single_file=False, output_format="evaluation", llm_config=None)`
Generate questions using the LLM service.
**Parameters:**
- `prompts` (list, optional): List of prompt configs. Each dict: `{"category": str, "topic": str, "count": int}`.
Enhanced fields supported: `context`, `audience`, `examples`, `keywords`.
Either `prompts` or `config_file` must be provided.
- `config_file` (str, optional): Path to config.yml file (must exist, created manually by user).
Supports both simple and enhanced config structures.
Either `prompts` or `config_file` must be provided.
- `artifacts_location` (str): Directory to save output files (default: "./artifacts")
- `single_file` (bool): If True, create one file with all categories. If False, separate file per category.
- `output_format` (str): Output format - 'evaluation', 'jsonl', 'simple', 'openai', 'anthropic', 'plain'
- `llm_config` (LLMConfig class, optional): LLM configuration class. If provided, overrides .env file settings.
**Raises:**
- `ValueError`: If both prompts and config_file are None, or if config contains dummy/example values
- `FileNotFoundError`: If config_file is provided but file doesn't exist
**LLM Configuration Priority:**
1. `llm_config` parameter (if provided)
2. `.env` file (if exists)
3. Defaults (ollama, llama3:latest, http://localhost:11434)
**Supported Providers:**
- **Ollama** (local, no API key needed) - Install: `pip install promptron`
- **OpenAI** (requires API key) - Install: `pip install promptron[openai]`
- **Anthropic** (requires API key) - Install: `pip install promptron[anthropic]`
**Example:**
```python
from promptron import generate_prompts, LLMConfig
from dotenv import load_dotenv
import os
load_dotenv()
class MyLLMConfig(LLMConfig):
name = os.getenv("PROMPTRON_MODEL", "llama3:latest")
provider = os.getenv("PROMPTRON_PROVIDER", "ollama")
url = os.getenv("PROMPTRON_BASE_URL", "http://localhost:11434")
# Using LLMConfig
generate_prompts(
config_file="./config.yml",
artifacts_location="./output",
llm_config=MyLLMConfig
)
# Or using .env file (no LLMConfig needed)
generate_prompts(
config_file="./config.yml",
artifacts_location="./output"
)
```
## Output Formats
### 1. Evaluation Format (default)
Best for tracking answers from multiple LLMs:
```json
{
"categories": [
{
"category": "default",
"prompts": [
{
"topic": "openshift",
"questions": [
{"user_question": "How do I configure pod resource limits?"},
{"user_question": "What is the difference between requests and limits?"}
]
}
]
}
]
}
```
When `single_file=False`, each category gets its own file: `artifacts/default.json`, `artifacts/red_teaming.json`, etc.
### 2. JSONL Format
Perfect for batch processing:
```jsonl
{"prompt": "How do I configure pod resource limits?", "topic": "openshift", "category": "default"}
{"prompt": "What is the difference between requests and limits?", "topic": "openshift", "category": "default"}
```
### 3. Simple JSON Format
Clean array format:
```json
[
{"question": "How do I configure pod resource limits?", "topic": "openshift", "category": "default"},
{"question": "What is the difference between requests and limits?", "topic": "openshift", "category": "default"}
]
```
### 4. OpenAI API Format
Ready to send to OpenAI:
```json
[
{
"messages": [{"role": "user", "content": "How do I configure pod resource limits?"}],
"metadata": {"topic": "openshift", "category": "default"}
}
]
```
### 5. Anthropic API Format
Ready to send to Anthropic:
```json
[
{
"messages": [{"role": "user", "content": "How do I configure pod resource limits?"}],
"metadata": {"topic": "openshift", "category": "default"}
}
]
```
### 6. Plain Text Format
Simple text file:
```
# Category: default
## Topic: openshift
How do I configure pod resource limits?
What is the difference between requests and limits?
```
## Output Structure
### When `single_file=True`:
One file (`artifacts/questions.json`) with all categories:
```json
{
"categories": [
{
"category": "default",
"prompts": [
{
"topic": "openshift",
"questions": [{"user_question": "..."}, ...]
},
{
"topic": "kubernetes",
"questions": [{"user_question": "..."}, ...]
}
]
},
{
"category": "red_teaming",
"prompts": [...]
}
]
}
```
### When `single_file=False`:
Separate file per category (`artifacts/default.json`, `artifacts/red_teaming.json`, etc.):
**File: `artifacts/default.json`**
```json
{
"category": "default",
"prompts": [
{
"topic": "openshift",
"questions": [{"user_question": "..."}, ...]
}
]
}
```
## Configuration
### config.yml Structure
**Simple Structure (Backward Compatible):**
```yaml
prompts:
- category: "default" # Required: One of 5 recommended categories
topic: "openshift" # Required: User-defined topic (anything)
count: 5 # Required: Number of questions to generate
- category: "red_teaming"
topic: "kubernetes"
count: 3
```
**Enhanced Structure (For Better Accuracy):**
```yaml
# Global template variables (optional)
template_config:
# Any custom variables available in all templates
domain: "cloud-native"
prompts:
- category: "default"
topic: "kubernetes networking"
count: 5
# Context: Background about the topic (highly recommended)
# Helps the LLM generate more accurate, relevant questions
context: |
Kubernetes networking involves Services, Ingress, NetworkPolicies, and CNI plugins.
Focus on practical scenarios like service discovery, load balancing, and network isolation.
# Audience: Target skill level (optional)
# Options: "beginner", "intermediate", "expert"
# Default: "intermediate" if not specified
audience: "intermediate"
# Examples: Show the LLM what good questions look like (optional)
# Provide 2-3 example questions to guide the generation
examples:
- "How do I configure a Service to expose pods to external traffic?"
- "What is the difference between ClusterIP and NodePort service types?"
# Keywords: Important terms to include (optional)
# Helps ensure questions cover key concepts
keywords:
- "services"
- "ingress"
- "network policies"
- "CNI"
```
**Field Descriptions:**
- **Required Fields:**
- `category`: One of 5 recommended categories (`default`, `red_teaming`, `out_of_scope`, `edge_cases`, `reasoning`)
- `topic`: User-defined topic (any string)
- `count`: Number of questions to generate (integer)
- **Optional Enhanced Fields (for better accuracy):**
- `context`: Background information about the topic. Most important field for improving accuracy.
- `audience`: Target skill level - `"beginner"`, `"intermediate"`, or `"expert"` (default: `"intermediate"`)
- `examples`: List of 2-3 example questions to guide the LLM's generation style
- `keywords`: List of important terms/concepts that should be covered in the questions
**Note:** All enhanced fields are optional. Simple configs (just category, topic, count) still work perfectly. Enhanced fields help generate more accurate and relevant questions.
### LLM Configuration (.env file)
Create a `.env` file in your project directory (or copy from `.env.example`):
```bash
# LLM Provider (currently only 'ollama' is supported)
PROMPTRON_PROVIDER=ollama
# Model name (for Ollama: e.g., llama3:latest, llama3.2:latest)
PROMPTRON_MODEL=llama3:latest
# Ollama base URL (optional, defaults to http://localhost:11434)
PROMPTRON_BASE_URL=http://localhost:11434
```
**Note:** Currently only Ollama (local) is supported. Support for OpenAI, Anthropic, and other providers will be added in future versions.
**Using environment variables directly:**
```bash
export PROMPTRON_PROVIDER=ollama
export PROMPTRON_MODEL=llama3.2:latest
export PROMPTRON_BASE_URL=http://localhost:11434
```
## Requirements
- `langchain-ollama>=0.1.0`
- `pyyaml>=6.0`
- `python-dotenv>=1.0.0` (for automatic .env file loading)
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## Author
**Hit Shiroya**
- Email: 24.hiit@gmail.com
## License
MIT License
## Support
For issues and questions, please open an issue on the GitHub repository.
Raw data
{
"_id": null,
"home_page": null,
"name": "promptron",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "llm, prompt-generation, dataset, evaluation, ollama",
"author": null,
"author_email": "Hit Shiroya <24.hiit@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/22/32/008ba2ba7e074b43d1dbc3ce09ec1ccb1ba2aa3830f17d7b7a0f71babcbb/promptron-1.0.1.tar.gz",
"platform": null,
"description": "# Promptron\n\nA Python package for generating evaluation datasets using Large Language Models (LLMs). Promptron helps you create structured question datasets for testing and evaluating LLM applications through code-only API.\n\n## Features\n\n- **Multi-Provider Support**: Works with Ollama (local), OpenAI, and Anthropic with auto-detection\n- **LLM-Powered Generation**: Automatically generates questions using your chosen LLM provider\n- **5 Evaluation Categories**: Pre-configured templates for comprehensive LLM evaluation\n- **Enhanced Config Fields**: Support for context, audience, examples, and keywords for better accuracy\n- **Code-Only API**: Simple Python functions for automation and integration\n- **Flexible Configuration**: Use YAML config file or pass prompts directly (simple or enhanced)\n- **Backward Compatible**: Simple configs still work; enhanced fields are optional\n- **Structured Output**: Generates JSON datasets ready for evaluation pipelines\n\n## Installation\n\n### Prerequisites\n\n- Python 3.8 or higher\n- One of the following:\n - **Ollama** (local): [Install Ollama](https://ollama.ai/) and download a model (e.g., `ollama pull llama3:latest`)\n - **OpenAI**: API key from [OpenAI](https://platform.openai.com/)\n - **Anthropic**: API key from [Anthropic](https://www.anthropic.com/)\n\n### Install from PyPI\n\n**Base installation (Ollama only):**\n```bash\npip install promptron\n```\n\n**With OpenAI support:**\n```bash\npip install promptron[openai]\n```\n\n**With Anthropic support:**\n```bash\npip install promptron[anthropic]\n```\n\n**With all providers:**\n```bash\npip install promptron[openai,anthropic]\n```\n\n\n## Quick Start\n\n### 1. Install Promptron\n\n```bash\npip install promptron\n```\n\n### 2. Ensure Ollama is Running\n\n```bash\nollama serve\n```\n\n### 3. Download a Model (default: llama3:latest)\n\n```bash\nollama pull llama3:latest\n```\n\n**Optional:** Use a different model by setting environment variable:\n```bash\nexport PROMPTRON_MODEL=llama3.2:latest\n```\n\n### 4. Choose Your Usage Method\n\nPromptron supports two ways to provide prompts:\n\n**Method 1: Direct Prompts Array (Simplest)**\n```python\nfrom promptron import generate_prompts\n\ngenerate_prompts(\n prompts=[\n {\"category\": \"default\", \"topic\": \"openshift\", \"count\": 5}\n ]\n)\n```\n\n**Method 2: Config File (Recommended for Projects)**\n\nCreate `config.yml` manually in your project directory:\n\n**Simple config (backward compatible):**\n```yaml\nprompts:\n - category: \"default\"\n topic: \"openshift\"\n count: 5\n \n - category: \"red_teaming\"\n topic: \"kubernetes\"\n count: 3\n```\n\n**Enhanced config (for better accuracy):**\n```yaml\nprompts:\n - category: \"default\"\n topic: \"kubernetes networking\"\n count: 5\n context: |\n Kubernetes networking involves Services, Ingress, NetworkPolicies, and CNI plugins.\n Focus on practical scenarios like service discovery, load balancing, and network isolation.\n audience: \"intermediate\" # beginner/intermediate/expert\n examples:\n - \"How do I configure a Service to expose pods to external traffic?\"\n - \"What is the difference between ClusterIP and NodePort service types?\"\n keywords:\n - \"services\"\n - \"ingress\"\n - \"network policies\"\n```\n\nThen use it:\n```python\nfrom promptron import generate_prompts\n\ngenerate_prompts(config_file=\"./config.yml\")\n```\n\n### 5. Generate Questions\n\n**Option A: Using prompts array directly (no config file needed)**\n```python\nfrom promptron import generate_prompts\n\ngenerate_prompts(\n prompts=[\n {\"category\": \"default\", \"topic\": \"openshift\", \"count\": 5}\n ],\n artifacts_location=\"./artifacts\",\n single_file=False\n)\n```\n\n**Option B: Using config file (create config.yml manually first)**\n```python\nfrom promptron import generate_prompts\n\n# Uses config.yml you created manually\ngenerate_prompts(\n config_file=\"./config.yml\",\n artifacts_location=\"./artifacts\",\n single_file=False\n)\n```\n\n**Option C: Using LLMConfig class (programmatic approach)**\n\n**Ollama (Local):**\n```python\nfrom promptron import generate_prompts, LLMConfig\n\nclass MyLLMConfig(LLMConfig):\n name = \"llama3:latest\" # Provider auto-detected from \"llama\"\n # url defaults to http://localhost:11434\n\ngenerate_prompts(\n config_file=\"./config.yml\",\n llm_config=MyLLMConfig\n)\n```\n\n**OpenAI:**\n```python\nfrom promptron import generate_prompts, LLMConfig\n\nclass MyLLMConfig(LLMConfig):\n name = \"gpt-4\" # Provider auto-detected from \"gpt-\"\n api_key = \"sk-...\" # Required for OpenAI\n\ngenerate_prompts(\n config_file=\"./config.yml\",\n llm_config=MyLLMConfig\n)\n```\n\n**Anthropic:**\n```python\nfrom promptron import generate_prompts, LLMConfig\n\nclass MyLLMConfig(LLMConfig):\n name = \"claude-3-opus-20240229\" # Provider auto-detected from \"claude-\"\n api_key = \"sk-ant-...\" # Required for Anthropic\n\ngenerate_prompts(\n config_file=\"./config.yml\",\n llm_config=MyLLMConfig\n)\n```\n\n## Recommended Categories\n\nPromptron provides 5 recommended categories for comprehensive LLM evaluation:\n\n1. **\"default\"** - Standard, straightforward questions for baseline evaluation\n2. **\"red_teaming\"** - Adversarial, tricky, or misleading questions to test robustness and safety\n3. **\"out_of_scope\"** - Questions outside the domain to test boundary handling\n4. **\"edge_cases\"** - Unusual, extreme, or corner-case scenarios to test edge case handling\n5. **\"reasoning\"** - Multi-step, complex, analytical questions to test reasoning depth\n\n**Note:** Using categories outside these recommended ones may reduce prompt accuracy. The system will fallback to the \"default\" template with a warning.\n\n## Usage Examples\n\n### Method 1: Direct Prompts Array (Simplest - No Config File)\n\n```python\nfrom promptron import generate_prompts\n\n# No config file needed - pass prompts directly\ngenerate_prompts(\n prompts=[\n {\"category\": \"default\", \"topic\": \"openshift\", \"count\": 5},\n {\"category\": \"red_teaming\", \"topic\": \"kubernetes\", \"count\": 3}\n ],\n artifacts_location=\"./output\",\n single_file=True,\n output_format=\"jsonl\"\n)\n```\n\n### Method 2: Using config.yml File with .env\n\n**First, create `config.yml` manually:**\n```yaml\nprompts:\n - category: \"default\"\n topic: \"openshift\"\n count: 5\n context: \"OpenShift is a Kubernetes-based container platform. Focus on pod management and resource limits.\"\n audience: \"intermediate\"\n - category: \"red_teaming\"\n topic: \"kubernetes\"\n count: 3\n context: \"Focus on security vulnerabilities and misconfigurations in Kubernetes.\"\n```\n\n**Then use it:**\n```python\nfrom promptron import generate_prompts\n\n# Reads LLM config from .env file\ngenerate_prompts(\n config_file=\"./config.yml\",\n artifacts_location=\"./output\",\n single_file=True,\n output_format=\"jsonl\"\n)\n```\n\n### Method 3: Using config.yml File with LLMConfig\n\n**First, create `config.yml` manually (same as Method 2).**\n\n**Then use it with LLMConfig:**\n```python\nfrom promptron import generate_prompts, LLMConfig\nfrom dotenv import load_dotenv\nimport os\n\nload_dotenv()\n\nclass MyLLMConfig(LLMConfig):\n name = os.getenv(\"PROMPTRON_MODEL\", \"llama3:latest\")\n provider = os.getenv(\"PROMPTRON_PROVIDER\", \"ollama\")\n url = os.getenv(\"PROMPTRON_BASE_URL\", \"http://localhost:11434\")\n\ngenerate_prompts(\n config_file=\"./config.yml\",\n artifacts_location=\"./output\",\n llm_config=MyLLMConfig\n)\n```\n\n### Method 4: Direct Prompts with LLMConfig (No YAML File)\n\n```python\nfrom promptron import generate_prompts, LLMConfig\nfrom dotenv import load_dotenv\nimport os\n\nload_dotenv()\n\nclass MyLLMConfig(LLMConfig):\n name = os.getenv(\"PROMPTRON_MODEL\", \"llama3:latest\")\n provider = os.getenv(\"PROMPTRON_PROVIDER\", \"ollama\")\n url = os.getenv(\"PROMPTRON_BASE_URL\", \"http://localhost:11434\")\n\n# Pass prompts directly\ngenerate_prompts(\n prompts=[\n {\"category\": \"default\", \"topic\": \"openshift\", \"count\": 5},\n {\"category\": \"red_teaming\", \"topic\": \"kubernetes\", \"count\": 3}\n ],\n artifacts_location=\"./artifacts\",\n llm_config=MyLLMConfig\n)\n```\n\n### Complete Workflow Example\n\n**Workflow A: Using .env file**\n```python\nfrom promptron import generate_prompts\n\n# 1. Create config.yml manually (see structure in README)\n\n# 2. Create .env file with your LLM settings (optional, can use LLMConfig instead)\n\n# 3. Generate questions (reads from .env automatically if not using LLMConfig)\ngenerate_prompts(\n config_file=\"./config.yml\",\n artifacts_location=\"./evaluation_data\",\n single_file=True\n)\n```\n\n**Workflow B: Using LLMConfig class**\n```python\nfrom promptron import generate_prompts, LLMConfig\nfrom dotenv import load_dotenv\nimport os\n\n# 1. Create config.yml manually (see structure in README)\n\n# 2. Load .env (user creates this)\nload_dotenv()\n\n# 3. Create LLMConfig class (user writes this)\nclass MyLLMConfig(LLMConfig):\n name = os.getenv(\"PROMPTRON_MODEL\", \"llama3:latest\")\n provider = os.getenv(\"PROMPTRON_PROVIDER\", \"ollama\")\n url = os.getenv(\"PROMPTRON_BASE_URL\", \"http://localhost:11434\")\n\n# 4. Edit config.yml with your prompts\n\n# 5. Generate questions with LLMConfig\ngenerate_prompts(\n config_file=\"./config.yml\",\n artifacts_location=\"./evaluation_data\",\n llm_config=MyLLMConfig,\n single_file=True\n)\n```\n\n## API Reference\n\n### `LLMConfig`\n\nBase class for LLM configuration. Supports **Ollama**, **OpenAI**, and **Anthropic** with auto-detection.\n\n**Attributes:**\n- `name` (str, required): Model name (e.g., \"llama3:latest\", \"gpt-4\", \"claude-3-opus-20240229\")\n- `provider` (str, optional): LLM provider - \"ollama\", \"openai\", or \"anthropic\" (auto-detected if not set)\n- `api_key` (str, optional): API key for OpenAI/Anthropic (required for those providers)\n- `url` (str, optional): Base URL for Ollama (defaults to http://localhost:11434)\n\n**Auto-detection rules:**\n- Model name contains \"llama\" or \"ollama\" \u2192 provider = \"ollama\"\n- Model name starts with \"gpt-\" or \"text-\" \u2192 provider = \"openai\"\n- Model name starts with \"claude-\" \u2192 provider = \"anthropic\"\n\n**Example (Ollama - auto-detected):**\n```python\nfrom promptron import LLMConfig\n\nclass MyLLMConfig(LLMConfig):\n name = \"llama3:latest\" # Provider auto-detected\n # url defaults to http://localhost:11434\n```\n\n**Example (OpenAI - auto-detected):**\n```python\nfrom promptron import LLMConfig\n\nclass MyLLMConfig(LLMConfig):\n name = \"gpt-4\" # Provider auto-detected\n api_key = \"sk-...\" # Required\n```\n\n**Example (Anthropic - auto-detected):**\n```python\nfrom promptron import LLMConfig\n\nclass MyLLMConfig(LLMConfig):\n name = \"claude-3-opus-20240229\" # Provider auto-detected\n api_key = \"sk-ant-...\" # Required\n```\n\n**Example (reading from .env):**\n```python\nfrom promptron import LLMConfig\nfrom dotenv import load_dotenv\nimport os\n\nload_dotenv()\n\nclass MyLLMConfig(LLMConfig):\n name = os.getenv(\"PROMPTRON_MODEL\", \"llama3:latest\")\n provider = os.getenv(\"PROMPTRON_PROVIDER\", None) # Auto-detect if None\n api_key = os.getenv(\"PROMPTRON_API_KEY\", None)\n url = os.getenv(\"PROMPTRON_BASE_URL\", None) # Optional for Ollama\n```\n\n### `generate_prompts(prompts=None, config_file=None, artifacts_location=\"./artifacts\", single_file=False, output_format=\"evaluation\", llm_config=None)`\n\nGenerate questions using the LLM service.\n\n**Parameters:**\n- `prompts` (list, optional): List of prompt configs. Each dict: `{\"category\": str, \"topic\": str, \"count\": int}`. \n Enhanced fields supported: `context`, `audience`, `examples`, `keywords`.\n Either `prompts` or `config_file` must be provided.\n- `config_file` (str, optional): Path to config.yml file (must exist, created manually by user).\n Supports both simple and enhanced config structures.\n Either `prompts` or `config_file` must be provided.\n- `artifacts_location` (str): Directory to save output files (default: \"./artifacts\")\n- `single_file` (bool): If True, create one file with all categories. If False, separate file per category.\n- `output_format` (str): Output format - 'evaluation', 'jsonl', 'simple', 'openai', 'anthropic', 'plain'\n- `llm_config` (LLMConfig class, optional): LLM configuration class. If provided, overrides .env file settings.\n\n**Raises:**\n- `ValueError`: If both prompts and config_file are None, or if config contains dummy/example values\n- `FileNotFoundError`: If config_file is provided but file doesn't exist\n\n**LLM Configuration Priority:**\n1. `llm_config` parameter (if provided)\n2. `.env` file (if exists)\n3. Defaults (ollama, llama3:latest, http://localhost:11434)\n\n**Supported Providers:**\n- **Ollama** (local, no API key needed) - Install: `pip install promptron`\n- **OpenAI** (requires API key) - Install: `pip install promptron[openai]`\n- **Anthropic** (requires API key) - Install: `pip install promptron[anthropic]`\n\n**Example:**\n```python\nfrom promptron import generate_prompts, LLMConfig\nfrom dotenv import load_dotenv\nimport os\n\nload_dotenv()\n\nclass MyLLMConfig(LLMConfig):\n name = os.getenv(\"PROMPTRON_MODEL\", \"llama3:latest\")\n provider = os.getenv(\"PROMPTRON_PROVIDER\", \"ollama\")\n url = os.getenv(\"PROMPTRON_BASE_URL\", \"http://localhost:11434\")\n\n# Using LLMConfig\ngenerate_prompts(\n config_file=\"./config.yml\",\n artifacts_location=\"./output\",\n llm_config=MyLLMConfig\n)\n\n# Or using .env file (no LLMConfig needed)\ngenerate_prompts(\n config_file=\"./config.yml\",\n artifacts_location=\"./output\"\n)\n```\n\n## Output Formats\n\n### 1. Evaluation Format (default)\n\nBest for tracking answers from multiple LLMs:\n\n```json\n{\n \"categories\": [\n {\n \"category\": \"default\",\n \"prompts\": [\n {\n \"topic\": \"openshift\",\n \"questions\": [\n {\"user_question\": \"How do I configure pod resource limits?\"},\n {\"user_question\": \"What is the difference between requests and limits?\"}\n ]\n }\n ]\n }\n ]\n}\n```\n\nWhen `single_file=False`, each category gets its own file: `artifacts/default.json`, `artifacts/red_teaming.json`, etc.\n\n### 2. JSONL Format\n\nPerfect for batch processing:\n\n```jsonl\n{\"prompt\": \"How do I configure pod resource limits?\", \"topic\": \"openshift\", \"category\": \"default\"}\n{\"prompt\": \"What is the difference between requests and limits?\", \"topic\": \"openshift\", \"category\": \"default\"}\n```\n\n### 3. Simple JSON Format\n\nClean array format:\n\n```json\n[\n {\"question\": \"How do I configure pod resource limits?\", \"topic\": \"openshift\", \"category\": \"default\"},\n {\"question\": \"What is the difference between requests and limits?\", \"topic\": \"openshift\", \"category\": \"default\"}\n]\n```\n\n### 4. OpenAI API Format\n\nReady to send to OpenAI:\n\n```json\n[\n {\n \"messages\": [{\"role\": \"user\", \"content\": \"How do I configure pod resource limits?\"}],\n \"metadata\": {\"topic\": \"openshift\", \"category\": \"default\"}\n }\n]\n```\n\n### 5. Anthropic API Format\n\nReady to send to Anthropic:\n\n```json\n[\n {\n \"messages\": [{\"role\": \"user\", \"content\": \"How do I configure pod resource limits?\"}],\n \"metadata\": {\"topic\": \"openshift\", \"category\": \"default\"}\n }\n]\n```\n\n### 6. Plain Text Format\n\nSimple text file:\n\n```\n# Category: default\n\n## Topic: openshift\n\nHow do I configure pod resource limits?\nWhat is the difference between requests and limits?\n```\n\n## Output Structure\n\n### When `single_file=True`:\n\nOne file (`artifacts/questions.json`) with all categories:\n\n```json\n{\n \"categories\": [\n {\n \"category\": \"default\",\n \"prompts\": [\n {\n \"topic\": \"openshift\",\n \"questions\": [{\"user_question\": \"...\"}, ...]\n },\n {\n \"topic\": \"kubernetes\",\n \"questions\": [{\"user_question\": \"...\"}, ...]\n }\n ]\n },\n {\n \"category\": \"red_teaming\",\n \"prompts\": [...]\n }\n ]\n}\n```\n\n### When `single_file=False`:\n\nSeparate file per category (`artifacts/default.json`, `artifacts/red_teaming.json`, etc.):\n\n**File: `artifacts/default.json`**\n```json\n{\n \"category\": \"default\",\n \"prompts\": [\n {\n \"topic\": \"openshift\",\n \"questions\": [{\"user_question\": \"...\"}, ...]\n }\n ]\n}\n```\n\n## Configuration\n\n### config.yml Structure\n\n**Simple Structure (Backward Compatible):**\n```yaml\nprompts:\n - category: \"default\" # Required: One of 5 recommended categories\n topic: \"openshift\" # Required: User-defined topic (anything)\n count: 5 # Required: Number of questions to generate\n \n - category: \"red_teaming\"\n topic: \"kubernetes\"\n count: 3\n```\n\n**Enhanced Structure (For Better Accuracy):**\n```yaml\n# Global template variables (optional)\ntemplate_config:\n # Any custom variables available in all templates\n domain: \"cloud-native\"\n\nprompts:\n - category: \"default\"\n topic: \"kubernetes networking\"\n count: 5\n \n # Context: Background about the topic (highly recommended)\n # Helps the LLM generate more accurate, relevant questions\n context: |\n Kubernetes networking involves Services, Ingress, NetworkPolicies, and CNI plugins.\n Focus on practical scenarios like service discovery, load balancing, and network isolation.\n \n # Audience: Target skill level (optional)\n # Options: \"beginner\", \"intermediate\", \"expert\"\n # Default: \"intermediate\" if not specified\n audience: \"intermediate\"\n \n # Examples: Show the LLM what good questions look like (optional)\n # Provide 2-3 example questions to guide the generation\n examples:\n - \"How do I configure a Service to expose pods to external traffic?\"\n - \"What is the difference between ClusterIP and NodePort service types?\"\n \n # Keywords: Important terms to include (optional)\n # Helps ensure questions cover key concepts\n keywords:\n - \"services\"\n - \"ingress\"\n - \"network policies\"\n - \"CNI\"\n```\n\n**Field Descriptions:**\n- **Required Fields:**\n - `category`: One of 5 recommended categories (`default`, `red_teaming`, `out_of_scope`, `edge_cases`, `reasoning`)\n - `topic`: User-defined topic (any string)\n - `count`: Number of questions to generate (integer)\n\n- **Optional Enhanced Fields (for better accuracy):**\n - `context`: Background information about the topic. Most important field for improving accuracy.\n - `audience`: Target skill level - `\"beginner\"`, `\"intermediate\"`, or `\"expert\"` (default: `\"intermediate\"`)\n - `examples`: List of 2-3 example questions to guide the LLM's generation style\n - `keywords`: List of important terms/concepts that should be covered in the questions\n\n**Note:** All enhanced fields are optional. Simple configs (just category, topic, count) still work perfectly. Enhanced fields help generate more accurate and relevant questions.\n\n### LLM Configuration (.env file)\n\nCreate a `.env` file in your project directory (or copy from `.env.example`):\n\n```bash\n# LLM Provider (currently only 'ollama' is supported)\nPROMPTRON_PROVIDER=ollama\n\n# Model name (for Ollama: e.g., llama3:latest, llama3.2:latest)\nPROMPTRON_MODEL=llama3:latest\n\n# Ollama base URL (optional, defaults to http://localhost:11434)\nPROMPTRON_BASE_URL=http://localhost:11434\n```\n\n**Note:** Currently only Ollama (local) is supported. Support for OpenAI, Anthropic, and other providers will be added in future versions.\n\n**Using environment variables directly:**\n```bash\nexport PROMPTRON_PROVIDER=ollama\nexport PROMPTRON_MODEL=llama3.2:latest\nexport PROMPTRON_BASE_URL=http://localhost:11434\n```\n\n## Requirements\n\n- `langchain-ollama>=0.1.0`\n- `pyyaml>=6.0`\n- `python-dotenv>=1.0.0` (for automatic .env file loading)\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## Author\n\n**Hit Shiroya**\n\n- Email: 24.hiit@gmail.com\n\n## License\n\nMIT License\n\n## Support\n\nFor issues and questions, please open an issue on the GitHub repository.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A Python package for generating evaluation datasets using LLMs",
"version": "1.0.1",
"project_urls": null,
"split_keywords": [
"llm",
" prompt-generation",
" dataset",
" evaluation",
" ollama"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "287034cbd6d8f4c2e10d717239467b19baab3cf023ec087abc965d286f97948d",
"md5": "336127b337a982269017aadb5585c659",
"sha256": "6d0584458ed0f11763860b2978a79759c690b5b34380da008d4b833413594993"
},
"downloads": -1,
"filename": "promptron-1.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "336127b337a982269017aadb5585c659",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 20997,
"upload_time": "2025-11-12T06:01:04",
"upload_time_iso_8601": "2025-11-12T06:01:04.976645Z",
"url": "https://files.pythonhosted.org/packages/28/70/34cbd6d8f4c2e10d717239467b19baab3cf023ec087abc965d286f97948d/promptron-1.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "2232008ba2ba7e074b43d1dbc3ce09ec1ccb1ba2aa3830f17d7b7a0f71babcbb",
"md5": "d3ea146e2c87ed6d0222c8427a04283c",
"sha256": "225146164d9438bb85b9e3d25e15353fb5bcb49438b80a5d8bafc79bfb9818ab"
},
"downloads": -1,
"filename": "promptron-1.0.1.tar.gz",
"has_sig": false,
"md5_digest": "d3ea146e2c87ed6d0222c8427a04283c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 22741,
"upload_time": "2025-11-12T06:01:06",
"upload_time_iso_8601": "2025-11-12T06:01:06.439098Z",
"url": "https://files.pythonhosted.org/packages/22/32/008ba2ba7e074b43d1dbc3ce09ec1ccb1ba2aa3830f17d7b7a0f71babcbb/promptron-1.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-11-12 06:01:06",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "promptron"
}
JSON