<div align="center">
<img src="logo.png" alt="PromptSuite Logo" width="120">
<h1>PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation</h1>
</div>
A tool that creates multi-prompt datasets from single-prompt datasets using templates with variation specifications.
## Overview
PromptSuite transforms your single-prompt datasets into rich multi-prompt datasets by applying various types of variations specified in your templates. It supports HuggingFace-compatible datasets and provides both a command-line interface and a modern web UI.
## 📚 Documentation
- 📖 **[Complete API Guide](docs/api-guide.md)** - Python API reference and examples
- 🏗️ **[Developer Documentation](docs/dev/)** - For contributors and developers
## Installation
### From PyPI (Recommended)
```bash
pip install promptsuite
```
### From GitHub (Latest)
```bash
pip install git+https://github.com/eliyahabba/PromptSuite.git
```
### From Source
```bash
git clone https://github.com/eliyahabba/PromptSuite.git
cd PromptSuite
pip install -e .
```
## Quick Start
### Command Line Interface
```bash
promptsuite --template '{"instruction": "{instruction}: {text}", "text": ["paraphrase_with_llm"], "gold": "label"}' \
--data data.csv --max-variations-per-row 50
```
### Streamlit Interface
Launch the modern Streamlit interface for an intuitive experience:
```bash
# If installed via pip
promptsuite-ui
# From project root (development)
python src/promptsuite/ui/main.py
```
The web UI provides:
- 📁 **Step 1**: Upload data or use sample datasets
- 🔧 **Step 2**: Build templates with smart suggestions
- ⚡ **Step 3**: Generate variations with real-time progress and export results
### Python API
```python
from promptsuite import PromptSuite
import pandas as pd
# Initialize
ps = PromptSuite()
# Load data
data = [{"question": "What is 2+2?", "answer": "4"}]
ps.load_dataframe(pd.DataFrame(data))
# Configure template
template = {
'instruction': 'Please answer the following questions.',
'prompt format': 'Q: {question}\nA: {answer}',
'question': ['typos and noise'],
}
ps.set_template(template)
# Generate variations
ps.configure(max_rows=2, variations_per_field=3)
variations = ps.generate(verbose=True)
# Export results
ps.export("output.json", format="json")
```
For more detailed examples of API usage, refer to the `examples/` directory.
## 📖 Code Examples
### Sentiment Analysis
```python
import pandas as pd
from promptsuite import PromptSuite
data = pd.DataFrame({
'text': ['I love this movie!', 'This book is terrible.'],
'label': ['positive', 'negative']
})
template = {
'instruction': 'Classify the sentiment',
'instruction_variations': ['paraphrase_with_llm'],
'prompt format': f"Text: {text}\nSentiment: {label}",
'text': ['typos and noise'],
}
ps = PromptSuite()
ps.load_dataframe(data)
ps.set_template(template)
ps.configure(
variations_per_field=3,
max_variations_per_row=2,
random_seed=42,
api_platform="TogetherAI", # or "OpenAI", "Anthropic", "Google", "Cohere"
model_name="meta-llama/Llama-3.3-70B-Instruct-Turbo-Free"
)
variations = ps.generate(verbose=True)
```
### Question Answering with Few-shot
```python
template = {
'instruction': 'Answer the question:\nQuestion: {question}\nAnswer: {answer}',
'instruction_variations': ['paraphrase_with_llm'],
'question': ['semantic'],
'gold': 'answer',
'few_shot': {
'count': 2,
'format': 'same_examples__synchronized_order_variations',
'split': 'train'
}
}
ps = PromptSuite()
ps.load_dataframe(qa_data)
ps.set_template(template)
ps.configure(
variations_per_field=2,
api_platform="TogetherAI", # or "OpenAI", "Anthropic", "Google", "Cohere"
model_name="meta-llama/Llama-3.3-70B-Instruct-Turbo-Free"
)
variations = ps.generate(verbose=True)
```
### Multiple Choice with Few-shot
```python
import pandas as pd
from promptsuite import PromptSuite
data = pd.DataFrame({
'question': [
'What is the largest planet in our solar system?',
'Which chemical element has the symbol O?',
'What is the fastest land animal?',
'What is the smallest prime number?',
'Which continent is known as the "Dark Continent"?'
],
'options': [
'Earth, Jupiter, Mars, Venus',
'Oxygen, Gold, Silver, Iron',
'Lion, Cheetah, Horse, Leopard',
'1, 2, 3, 0',
'Asia, Africa, Europe, Australia'
],
'answer': [1, 0, 1, 1, 1],
'subject': ['Astronomy', 'Chemistry', 'Biology', 'Mathematics', 'Geography']
})
template = {
'prompt format': 'Question: {question}\nOptions: {options}\nAnswer:',
'prompt format variations': ['format structure'],
'options': ['shuffle', 'enumerate'],
'gold': {
'field': 'answer',
'type': 'index',
'options_field': 'options'
},
'few_shot': {
'count': 2,
'format': 'same_examples__synchronized_order_variations',
'split': 'train'
}
}
ps = PromptSuite()
ps.load_dataframe(data)
ps.set_template(template)
ps.configure(max_rows=5, variations_per_field=1)
variations = ps.generate(verbose=True)
for v in variations:
print(v['prompt'])
```
### Example Output Format
A typical output from `ps.generate()` or the exported JSON file looks like this (for a multiple choice template):
```json
[
{
"prompt": "Answer the following multiple choice question:\nQuestion: What is 2+2?\nOptions: 3, 4, 5, 6\nAnswer:",
"original_row_index": 1,
"variation_count": 1,
"template_config": {
"instruction": "Answer the following multiple choice question:\nQuestion: {question}\nOptions: {options}\nAnswer: {answer}",
"options": ["shuffle"],
"gold": {
"field": "answer",
"type": "index",
"options_field": "options"
},
"few_shot": {
"count": 1,
"format": "same_examples__synchronized_order_variations",
"split": "train"
}
},
"field_values": {
"options": "3, 4, 5, 6"
},
"gold_updates": {
"answer": "1"
},
"conversation": [
{
"role": "user",
"content": "Answer the following multiple choice question:\nQuestion: What is 2+2?\nOptions: 3, 4, 5, 6\nAnswer:"
},
{
"role": "assistant",
"content": "1"
},
{
"role": "user",
"content": "Answer the following multiple choice question:\nQuestion: What is the capital of France?\nOptions: London, Berlin, Paris, Madrid\nAnswer:"
}
]
}
]
```
## 📖 Detailed Guide
### Data Loading
```python
# CSV
ps.load_csv('data.csv')
# JSON
ps.load_json('data.json')
# HuggingFace
ps.load_dataset('squad', split='train[:100]')
# DataFrame
ps.load_dataframe(df)
```
### Generation Options
```python
ps.configure(
max_rows=10, # How many data rows to use
variations_per_field=3, # Variations per field (default: 3)
max_variations_per_row=50, # Cap on total variations per row
random_seed=42, # For reproducibility
api_platform="TogetherAI", # or "OpenAI", "Anthropic", "Google", "Cohere"
model_name="meta-llama/Llama-3.3-70B-Instruct-Turbo-Free"
)
```
### Export Formats
```python
# JSON - Full data with metadata
ps.export("output.json", format="json")
# CSV - Flattened for spreadsheets
ps.export("output.csv", format="csv")
# TXT - Plain prompts only
ps.export("output.txt", format="txt")
```
## Web UI Interface
PromptSuite 2.0 includes a modern, interactive web interface built with **Streamlit**.
The UI guides you through a simple 3-step workflow:
1. **Upload Data**: Load your dataset (CSV/JSON) or use built-in samples. Preview and validate your data before continuing.
2. **Build Template**: Create or select a prompt template, with smart suggestions based on your data. See a live preview of your template.
3. **Generate & Export**: Configure generation settings, run the variation process, and export your results in various formats.
The Streamlit UI is the easiest way to explore, test, and generate prompt variations visually.
## 🤖 Supported AI Platforms
PromptSuite supports multiple AI platforms with automatic dependency detection:
### Core Platforms (Always Available)
- **TogetherAI**: Open-source models (Llama, Mistral, etc.)
- **OpenAI**: GPT models (GPT-4, GPT-3.5, etc.)
### Extended Platforms (Optional Dependencies)
- **Anthropic**: Claude models (claude-3-haiku, claude-3-sonnet, claude-3-opus)
- **Google**: Gemini models (gemini-1.5-flash, gemini-1.5-pro)
- **Cohere**: Command models (command-r-plus, command-r)
### Installation
Install optional platform dependencies:
```bash
pip install -r requirements-optional.txt
```
Or install specific platforms:
```bash
pip install anthropic # For Anthropic/Claude
pip install google-generativeai # For Google/Gemini
pip install cohere # For Cohere
```
### API Keys
Set environment variables for the platforms you want to use:
```bash
export TOGETHER_API_KEY="your_together_key"
export OPENAI_API_KEY="your_openai_key"
export ANTHROPIC_API_KEY="your_anthropic_key"
export GOOGLE_API_KEY="your_google_key"
export COHERE_API_KEY="your_cohere_key"
```
### Platform Selection
```python
# Automatic platform detection
from promptsuite.shared.model_client import get_supported_platforms, is_platform_available
available_platforms = [p for p in get_supported_platforms() if is_platform_available(p)]
print(f"Available platforms: {available_platforms}")
# Use different platforms
ps.configure(api_platform="OpenAI", model_name="gpt-4o-mini")
ps.configure(api_platform="Anthropic", model_name="claude-3-haiku-20240307")
ps.configure(api_platform="Google", model_name="gemini-1.5-flash")
```
### Adding New Platforms
See [Platform Integration Guide](docs/platform-integration-guide.md) for instructions on adding support for additional AI platforms.
## 🔧 Advanced Features
### Performance Optimization
PromptSuite automatically optimizes performance by pre-generating variations for shared fields:
- **Instruction variations** (`instruction variations`) are generated once and reused across all data rows
- **Prompt format variations** (`prompt format variations`) are generated once and reused across all data rows
This optimization is especially important for LLM-based augmenters like `paraphrase_with_llm` that would otherwise run the same API calls repeatedly for identical text.
### Gold Field Configuration
**Simple format** (for text answers):
```python
'gold': 'answer' # Just the column name
```
**Advanced format** (for index-based answers):
```python
'gold': {
'field': 'answer',
'type': 'index', # Answer is an index
'options_field': 'options' # Column with the options
}
```
### Few-Shot Configuration
Few-shot examples can be configured with different sampling strategies. The format names clearly indicate what varies across data rows and variations:
**Format naming convention: `<examples_strategy>__<order_strategy>`**
- **Examples strategy**: Whether examples are the same or different across data rows
- **Order strategy**: Whether the order of examples varies across variations
| Format | Description | Use Case |
|--------|-------------|----------|
| `same_examples__no_variations` | Same examples for all rows, no variations (single variation per row) | When you want consistent, predictable examples |
| `same_examples__synchronized_order_variations` | Same examples for all rows, synchronized order variations across all rows | When you want consistent examples but test different orderings |
| `different_examples__same_shuffling_order_across_rows` | Different examples per row, same shuffling order across rows | When you want unique examples per question but consistent ordering patterns |
| `different_examples__different_order_per_variation` | Different examples and different order per variation | When you want maximum variety and different examples per question |
**Examples:**
```python
# same_examples__no_variations
# Row 1: [Example A, Example B]
# Row 2: [Example A, Example B] # Same examples, no variations
# same_examples__synchronized_order_variations
# Row 1, Variation 1: [Example A, Example B]
# Row 1, Variation 2: [Example B, Example A]
# Row 2, Variation 1: [Example A, Example B] # Same order as Row 1, Variation 1
# Row 2, Variation 2: [Example B, Example A] # Same order as Row 1, Variation 2
# different_examples__same_shuffling_order_across_rows
# Row 1, Variation 1: [Example A, Example B]
# Row 1, Variation 2: [Example A, Example B] # Same examples for this row
# Row 2, Variation 1: [Example C, Example D] # Different examples
# Row 2, Variation 2: [Example C, Example D] # Same examples for this row
# different_examples__different_order_per_variation
# Row 1, Variation 1: [Example A, Example B]
# Row 1, Variation 2: [Example C, Example D] # Different examples
# Row 2, Variation 1: [Example E, Example F] # Different examples
# Row 2, Variation 2: [Example G, Example H] # Different examples
```
**Example:**
```python
"few_shot": {
"count": 2, # Number of examples to use
"format": "same_examples__synchronized_order_variations", # Sampling strategy
"split": "train", # Use only training data for examples
"filter_by": "category", # NEW: Optional column name to filter few-shot examples by
"fallback_strategy": "global" # NEW: What to do if not enough examples in filtered category ('global' or 'strict')
}
```
**Few-Shot Filtering with `filter_by` and `fallback_strategy`:**
This new feature allows you to control which few-shot examples are selected based on metadata columns (e.g., 'category', 'difficulty').
- `filter_by`: Specify a column name in your dataset (e.g., `"category"`, `"subject"`) to ensure that few-shot examples are chosen from the same category as the current data row.
- `fallback_strategy`: Defines the behavior when there aren't enough few-shot examples within the filtered category:
* `"global"`: (Default) If there aren't enough examples in the specified `filter_by` category, the system will sample the remaining required examples from the *entire* dataset (globally), ignoring the category filter for those additional examples. This ensures that the requested `count` of few-shot examples is always met.
* `"strict"`: If there aren't enough examples in the specified `filter_by` category, the system will *only* use the examples available within that category. It will not pull examples from other categories, and if the `count` cannot be met from the category, it will raise an `FewShotDataInsufficientError`. This is useful for strict domain-specific few-shot requirements.
This feature is useful when you want each test question to have unique few-shot examples for context, but don't need multiple variations of the few-shot examples themselves.
## Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request
## License
MIT License - see LICENSE file for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "promptsuite",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "ai, prompts, datasets, nlp, machine-learning",
"author": "Eliya Habba",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/71/22/17a6049e55a84081919da6c3d675b1c201ce2c1cb43c35107d74022aa96a/promptsuite-3.0.7.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n <img src=\"logo.png\" alt=\"PromptSuite Logo\" width=\"120\">\n <h1>PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation</h1>\n</div>\nA tool that creates multi-prompt datasets from single-prompt datasets using templates with variation specifications.\n\n## Overview\n\nPromptSuite transforms your single-prompt datasets into rich multi-prompt datasets by applying various types of variations specified in your templates. It supports HuggingFace-compatible datasets and provides both a command-line interface and a modern web UI.\n\n## \ud83d\udcda Documentation\n\n- \ud83d\udcd6 **[Complete API Guide](docs/api-guide.md)** - Python API reference and examples\n- \ud83c\udfd7\ufe0f **[Developer Documentation](docs/dev/)** - For contributors and developers\n\n## Installation\n\n### From PyPI (Recommended)\n\n```bash\npip install promptsuite\n```\n\n### From GitHub (Latest)\n\n```bash\npip install git+https://github.com/eliyahabba/PromptSuite.git\n```\n\n### From Source\n\n```bash\ngit clone https://github.com/eliyahabba/PromptSuite.git\ncd PromptSuite\npip install -e .\n```\n\n## Quick Start\n### Command Line Interface\n\n```bash\npromptsuite --template '{\"instruction\": \"{instruction}: {text}\", \"text\": [\"paraphrase_with_llm\"], \"gold\": \"label\"}' \\\n --data data.csv --max-variations-per-row 50\n```\n### Streamlit Interface\n\nLaunch the modern Streamlit interface for an intuitive experience:\n\n```bash\n# If installed via pip\npromptsuite-ui\n\n# From project root (development)\npython src/promptsuite/ui/main.py\n```\n\nThe web UI provides:\n- \ud83d\udcc1 **Step 1**: Upload data or use sample datasets\n- \ud83d\udd27 **Step 2**: Build templates with smart suggestions\n- \u26a1 **Step 3**: Generate variations with real-time progress and export results\n\n\n### Python API\n\n```python\nfrom promptsuite import PromptSuite\nimport pandas as pd\n\n# Initialize\nps = PromptSuite()\n\n# Load data\ndata = [{\"question\": \"What is 2+2?\", \"answer\": \"4\"}]\nps.load_dataframe(pd.DataFrame(data))\n\n# Configure template\ntemplate = {\n 'instruction': 'Please answer the following questions.',\n 'prompt format': 'Q: {question}\\nA: {answer}',\n 'question': ['typos and noise'],\n}\nps.set_template(template)\n\n# Generate variations\nps.configure(max_rows=2, variations_per_field=3)\nvariations = ps.generate(verbose=True)\n\n# Export results\nps.export(\"output.json\", format=\"json\")\n```\n\nFor more detailed examples of API usage, refer to the `examples/` directory.\n\n## \ud83d\udcd6 Code Examples\n\n### Sentiment Analysis\n\n```python\nimport pandas as pd\nfrom promptsuite import PromptSuite\n\ndata = pd.DataFrame({\n 'text': ['I love this movie!', 'This book is terrible.'],\n 'label': ['positive', 'negative']\n})\n\ntemplate = {\n 'instruction': 'Classify the sentiment',\n 'instruction_variations': ['paraphrase_with_llm'],\n 'prompt format': f\"Text: {text}\\nSentiment: {label}\",\n 'text': ['typos and noise'],\n}\n\nps = PromptSuite()\nps.load_dataframe(data)\nps.set_template(template)\nps.configure(\n variations_per_field=3,\n max_variations_per_row=2,\n random_seed=42,\n api_platform=\"TogetherAI\", # or \"OpenAI\", \"Anthropic\", \"Google\", \"Cohere\"\n model_name=\"meta-llama/Llama-3.3-70B-Instruct-Turbo-Free\"\n)\nvariations = ps.generate(verbose=True)\n```\n\n### Question Answering with Few-shot\n\n```python\ntemplate = {\n 'instruction': 'Answer the question:\\nQuestion: {question}\\nAnswer: {answer}',\n 'instruction_variations': ['paraphrase_with_llm'],\n 'question': ['semantic'],\n 'gold': 'answer',\n 'few_shot': {\n 'count': 2,\n 'format': 'same_examples__synchronized_order_variations',\n 'split': 'train'\n }\n}\n\nps = PromptSuite()\nps.load_dataframe(qa_data)\nps.set_template(template)\nps.configure(\n variations_per_field=2,\n api_platform=\"TogetherAI\", # or \"OpenAI\", \"Anthropic\", \"Google\", \"Cohere\"\n model_name=\"meta-llama/Llama-3.3-70B-Instruct-Turbo-Free\"\n)\nvariations = ps.generate(verbose=True)\n```\n\n### Multiple Choice with Few-shot\n\n```python\nimport pandas as pd\nfrom promptsuite import PromptSuite\n\ndata = pd.DataFrame({\n 'question': [\n 'What is the largest planet in our solar system?',\n 'Which chemical element has the symbol O?',\n 'What is the fastest land animal?',\n 'What is the smallest prime number?',\n 'Which continent is known as the \"Dark Continent\"?'\n ],\n 'options': [\n 'Earth, Jupiter, Mars, Venus',\n 'Oxygen, Gold, Silver, Iron',\n 'Lion, Cheetah, Horse, Leopard',\n '1, 2, 3, 0',\n 'Asia, Africa, Europe, Australia'\n ],\n 'answer': [1, 0, 1, 1, 1],\n 'subject': ['Astronomy', 'Chemistry', 'Biology', 'Mathematics', 'Geography']\n})\n\ntemplate = {\n 'prompt format': 'Question: {question}\\nOptions: {options}\\nAnswer:',\n 'prompt format variations': ['format structure'],\n 'options': ['shuffle', 'enumerate'],\n 'gold': {\n 'field': 'answer',\n 'type': 'index',\n 'options_field': 'options'\n },\n 'few_shot': {\n 'count': 2,\n 'format': 'same_examples__synchronized_order_variations',\n 'split': 'train'\n }\n}\n\nps = PromptSuite()\nps.load_dataframe(data)\nps.set_template(template)\nps.configure(max_rows=5, variations_per_field=1)\nvariations = ps.generate(verbose=True)\nfor v in variations:\n print(v['prompt'])\n```\n\n\n\n### Example Output Format\n\nA typical output from `ps.generate()` or the exported JSON file looks like this (for a multiple choice template):\n\n```json\n[\n {\n \"prompt\": \"Answer the following multiple choice question:\\nQuestion: What is 2+2?\\nOptions: 3, 4, 5, 6\\nAnswer:\",\n \"original_row_index\": 1,\n \"variation_count\": 1,\n \"template_config\": {\n \"instruction\": \"Answer the following multiple choice question:\\nQuestion: {question}\\nOptions: {options}\\nAnswer: {answer}\",\n \"options\": [\"shuffle\"],\n \"gold\": {\n \"field\": \"answer\",\n \"type\": \"index\",\n \"options_field\": \"options\"\n },\n \"few_shot\": {\n \"count\": 1,\n \"format\": \"same_examples__synchronized_order_variations\",\n \"split\": \"train\"\n }\n },\n \"field_values\": {\n \"options\": \"3, 4, 5, 6\"\n },\n \"gold_updates\": {\n \"answer\": \"1\"\n },\n \"conversation\": [\n {\n \"role\": \"user\",\n \"content\": \"Answer the following multiple choice question:\\nQuestion: What is 2+2?\\nOptions: 3, 4, 5, 6\\nAnswer:\"\n },\n {\n \"role\": \"assistant\",\n \"content\": \"1\"\n },\n {\n \"role\": \"user\",\n \"content\": \"Answer the following multiple choice question:\\nQuestion: What is the capital of France?\\nOptions: London, Berlin, Paris, Madrid\\nAnswer:\"\n }\n ]\n }\n]\n\n```\n## \ud83d\udcd6 Detailed Guide\n\n### Data Loading\n```python\n# CSV\nps.load_csv('data.csv')\n\n# JSON\nps.load_json('data.json')\n\n# HuggingFace\nps.load_dataset('squad', split='train[:100]')\n\n# DataFrame\nps.load_dataframe(df)\n```\n\n### Generation Options\n```python\nps.configure(\n max_rows=10, # How many data rows to use\n variations_per_field=3, # Variations per field (default: 3)\n max_variations_per_row=50, # Cap on total variations per row\n random_seed=42, # For reproducibility\n api_platform=\"TogetherAI\", # or \"OpenAI\", \"Anthropic\", \"Google\", \"Cohere\"\n model_name=\"meta-llama/Llama-3.3-70B-Instruct-Turbo-Free\"\n)\n```\n\n### Export Formats\n```python\n# JSON - Full data with metadata\nps.export(\"output.json\", format=\"json\")\n\n# CSV - Flattened for spreadsheets\nps.export(\"output.csv\", format=\"csv\")\n\n# TXT - Plain prompts only\nps.export(\"output.txt\", format=\"txt\")\n```\n\n## Web UI Interface\n\nPromptSuite 2.0 includes a modern, interactive web interface built with **Streamlit**.\n\nThe UI guides you through a simple 3-step workflow:\n\n1. **Upload Data**: Load your dataset (CSV/JSON) or use built-in samples. Preview and validate your data before continuing.\n2. **Build Template**: Create or select a prompt template, with smart suggestions based on your data. See a live preview of your template.\n3. **Generate & Export**: Configure generation settings, run the variation process, and export your results in various formats.\n\nThe Streamlit UI is the easiest way to explore, test, and generate prompt variations visually.\n\n## \ud83e\udd16 Supported AI Platforms\n\nPromptSuite supports multiple AI platforms with automatic dependency detection:\n\n### Core Platforms (Always Available)\n- **TogetherAI**: Open-source models (Llama, Mistral, etc.)\n- **OpenAI**: GPT models (GPT-4, GPT-3.5, etc.)\n\n### Extended Platforms (Optional Dependencies)\n- **Anthropic**: Claude models (claude-3-haiku, claude-3-sonnet, claude-3-opus)\n- **Google**: Gemini models (gemini-1.5-flash, gemini-1.5-pro)\n- **Cohere**: Command models (command-r-plus, command-r)\n\n### Installation\n\nInstall optional platform dependencies:\n```bash\npip install -r requirements-optional.txt\n```\n\nOr install specific platforms:\n```bash\npip install anthropic # For Anthropic/Claude\npip install google-generativeai # For Google/Gemini\npip install cohere # For Cohere\n```\n\n### API Keys\n\nSet environment variables for the platforms you want to use:\n```bash\nexport TOGETHER_API_KEY=\"your_together_key\"\nexport OPENAI_API_KEY=\"your_openai_key\"\nexport ANTHROPIC_API_KEY=\"your_anthropic_key\"\nexport GOOGLE_API_KEY=\"your_google_key\"\nexport COHERE_API_KEY=\"your_cohere_key\"\n```\n\n### Platform Selection\n\n```python\n# Automatic platform detection\nfrom promptsuite.shared.model_client import get_supported_platforms, is_platform_available\n\navailable_platforms = [p for p in get_supported_platforms() if is_platform_available(p)]\nprint(f\"Available platforms: {available_platforms}\")\n\n# Use different platforms\nps.configure(api_platform=\"OpenAI\", model_name=\"gpt-4o-mini\")\nps.configure(api_platform=\"Anthropic\", model_name=\"claude-3-haiku-20240307\")\nps.configure(api_platform=\"Google\", model_name=\"gemini-1.5-flash\")\n```\n\n### Adding New Platforms\n\nSee [Platform Integration Guide](docs/platform-integration-guide.md) for instructions on adding support for additional AI platforms.\n\n## \ud83d\udd27 Advanced Features\n\n### Performance Optimization\n\nPromptSuite automatically optimizes performance by pre-generating variations for shared fields:\n\n- **Instruction variations** (`instruction variations`) are generated once and reused across all data rows\n- **Prompt format variations** (`prompt format variations`) are generated once and reused across all data rows\n\nThis optimization is especially important for LLM-based augmenters like `paraphrase_with_llm` that would otherwise run the same API calls repeatedly for identical text.\n\n### Gold Field Configuration\n\n**Simple format** (for text answers):\n```python\n'gold': 'answer' # Just the column name\n```\n\n**Advanced format** (for index-based answers):\n```python\n'gold': {\n 'field': 'answer',\n 'type': 'index', # Answer is an index\n 'options_field': 'options' # Column with the options\n}\n```\n\n### Few-Shot Configuration\n\nFew-shot examples can be configured with different sampling strategies. The format names clearly indicate what varies across data rows and variations:\n\n**Format naming convention: `<examples_strategy>__<order_strategy>`**\n\n- **Examples strategy**: Whether examples are the same or different across data rows\n- **Order strategy**: Whether the order of examples varies across variations\n\n| Format | Description | Use Case |\n|--------|-------------|----------|\n| `same_examples__no_variations` | Same examples for all rows, no variations (single variation per row) | When you want consistent, predictable examples |\n| `same_examples__synchronized_order_variations` | Same examples for all rows, synchronized order variations across all rows | When you want consistent examples but test different orderings |\n| `different_examples__same_shuffling_order_across_rows` | Different examples per row, same shuffling order across rows | When you want unique examples per question but consistent ordering patterns |\n| `different_examples__different_order_per_variation` | Different examples and different order per variation | When you want maximum variety and different examples per question |\n\n**Examples:**\n\n```python\n# same_examples__no_variations\n# Row 1: [Example A, Example B]\n# Row 2: [Example A, Example B] # Same examples, no variations\n\n# same_examples__synchronized_order_variations \n# Row 1, Variation 1: [Example A, Example B]\n# Row 1, Variation 2: [Example B, Example A]\n# Row 2, Variation 1: [Example A, Example B] # Same order as Row 1, Variation 1\n# Row 2, Variation 2: [Example B, Example A] # Same order as Row 1, Variation 2\n\n# different_examples__same_shuffling_order_across_rows\n# Row 1, Variation 1: [Example A, Example B]\n# Row 1, Variation 2: [Example A, Example B] # Same examples for this row\n# Row 2, Variation 1: [Example C, Example D] # Different examples\n# Row 2, Variation 2: [Example C, Example D] # Same examples for this row\n\n# different_examples__different_order_per_variation\n# Row 1, Variation 1: [Example A, Example B]\n# Row 1, Variation 2: [Example C, Example D] # Different examples\n# Row 2, Variation 1: [Example E, Example F] # Different examples\n# Row 2, Variation 2: [Example G, Example H] # Different examples\n```\n\n**Example:**\n```python\n\"few_shot\": {\n \"count\": 2, # Number of examples to use\n \"format\": \"same_examples__synchronized_order_variations\", # Sampling strategy\n \"split\": \"train\", # Use only training data for examples\n \"filter_by\": \"category\", # NEW: Optional column name to filter few-shot examples by\n \"fallback_strategy\": \"global\" # NEW: What to do if not enough examples in filtered category ('global' or 'strict')\n}\n```\n\n**Few-Shot Filtering with `filter_by` and `fallback_strategy`:**\n\nThis new feature allows you to control which few-shot examples are selected based on metadata columns (e.g., 'category', 'difficulty').\n\n- `filter_by`: Specify a column name in your dataset (e.g., `\"category\"`, `\"subject\"`) to ensure that few-shot examples are chosen from the same category as the current data row.\n- `fallback_strategy`: Defines the behavior when there aren't enough few-shot examples within the filtered category:\n * `\"global\"`: (Default) If there aren't enough examples in the specified `filter_by` category, the system will sample the remaining required examples from the *entire* dataset (globally), ignoring the category filter for those additional examples. This ensures that the requested `count` of few-shot examples is always met.\n * `\"strict\"`: If there aren't enough examples in the specified `filter_by` category, the system will *only* use the examples available within that category. It will not pull examples from other categories, and if the `count` cannot be met from the category, it will raise an `FewShotDataInsufficientError`. This is useful for strict domain-specific few-shot requirements.\n\nThis feature is useful when you want each test question to have unique few-shot examples for context, but don't need multiple variations of the few-shot examples themselves.\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests\n5. Submit a pull request\n\n## License\n\nMIT License - see LICENSE file for details. \n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A tool that creates multi-prompt datasets from single-prompt datasets using templates",
"version": "3.0.7",
"project_urls": {
"Documentation": "https://github.com/eliyahabba/PromptSuite/blob/main/README.md",
"Homepage": "https://github.com/eliyahabba/PromptSuite",
"Issues": "https://github.com/eliyahabba/PromptSuite/issues",
"Repository": "https://github.com/eliyahabba/PromptSuite"
},
"split_keywords": [
"ai",
" prompts",
" datasets",
" nlp",
" machine-learning"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "4989aea1bdc265f5c8447e2df0a243888face6c8ce10e12f8db27ed913ca6d13",
"md5": "a41ea59bc65b9e5797c5c183de99f3eb",
"sha256": "2af34d783c4b52a8fb1c2119c8577c466833fd5a91d8f1afa7bf7b466071224c"
},
"downloads": -1,
"filename": "promptsuite-3.0.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a41ea59bc65b9e5797c5c183de99f3eb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 111165,
"upload_time": "2025-07-08T12:26:53",
"upload_time_iso_8601": "2025-07-08T12:26:53.934854Z",
"url": "https://files.pythonhosted.org/packages/49/89/aea1bdc265f5c8447e2df0a243888face6c8ce10e12f8db27ed913ca6d13/promptsuite-3.0.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "712217a6049e55a84081919da6c3d675b1c201ce2c1cb43c35107d74022aa96a",
"md5": "4aed8088f9246a1025ce8c81579b38c6",
"sha256": "c8a5b725fb71de3c797a0baa30f25edf45f05a9c1fb2f4eb658a8236ecc21aa8"
},
"downloads": -1,
"filename": "promptsuite-3.0.7.tar.gz",
"has_sig": false,
"md5_digest": "4aed8088f9246a1025ce8c81579b38c6",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 1685110,
"upload_time": "2025-07-08T12:26:55",
"upload_time_iso_8601": "2025-07-08T12:26:55.802888Z",
"url": "https://files.pythonhosted.org/packages/71/22/17a6049e55a84081919da6c3d675b1c201ce2c1cb43c35107d74022aa96a/promptsuite-3.0.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-08 12:26:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "eliyahabba",
"github_project": "PromptSuite",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "pandas",
"specs": [
[
">=",
"1.3.0"
]
]
},
{
"name": "datasets",
"specs": [
[
"~=",
"2.21.0"
]
]
},
{
"name": "click",
"specs": [
[
">=",
"8.0.0"
]
]
},
{
"name": "pyyaml",
"specs": [
[
">=",
"6.0"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
">=",
"0.19.0"
]
]
},
{
"name": "together",
"specs": [
[
">=",
"0.2.0"
]
]
},
{
"name": "numpy",
"specs": [
[
">=",
"1.21.0"
]
]
},
{
"name": "streamlit",
"specs": [
[
">=",
"1.45.1"
]
]
},
{
"name": "dotenv",
"specs": [
[
">=",
"0.9.9"
]
]
},
{
"name": "tqdm",
"specs": [
[
">=",
"4.67.1"
]
]
},
{
"name": "openai",
"specs": [
[
">=",
"1.91.0"
]
]
},
{
"name": "matplotlib",
"specs": [
[
">=",
"3.5.0"
]
]
},
{
"name": "anthropic",
"specs": [
[
"~=",
"0.57.1"
]
]
},
{
"name": "evaluate",
"specs": [
[
"~=",
"0.4.4"
]
]
},
{
"name": "scikit-learn",
"specs": [
[
"~=",
"1.6.1"
]
]
},
{
"name": "protobuf",
"specs": [
[
"~=",
"5.29.5"
]
]
},
{
"name": "cohere",
"specs": [
[
"~=",
"5.15.0"
]
]
}
],
"lcname": "promptsuite"
}
JSON
