# Prompture
`Prompture` is an API-first library for requesting structured **JSON** output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.
## ✨ Features
- ✅ **Structured Output**: Request models to return JSON only
- ✅ **Validation**: Automatic validation with `jsonschema`
- ✅ **Multi-driver**: Run the same specification against multiple drivers (OpenAI, Ollama, Claude, Azure, HTTP, mock)
- ✅ **Reports**: Generate JSON reports with results
- ✅ **Usage Tracking**: **NEW** - Automatic token and cost monitoring for all calls
## 🆕 Token and Cost Tracking (New)
Starting with this version, `extract_and_jsonify` and `ask_for_json` automatically include token usage and cost information:
```python
from prompture import extract_and_jsonify
from prompture.drivers import OllamaDriver
driver = OllamaDriver(endpoint="http://localhost:11434/api/generate", model="gemma3")
result = extract_and_jsonify(driver, "Text to process", json_schema)
# Now returns both the response and usage information
json_output = result["json_string"]
usage = result["usage"]
print(f"Tokens used: {usage['total_tokens']}")
print(f"Cost: ${usage['cost']:.6f}")
```
### Return Structure
The main functions now return:
```python
{
"json_string": str, # The original JSON string
"json_object": dict, # The parsed JSON object
"usage": {
"prompt_tokens": int,
"completion_tokens": int,
"total_tokens": int,
"cost": float # Cost in USD (0.0 for free models)
}
}
```
### Supported Drivers
- **OllamaDriver**: Cost = $0.00 (free local models)
- **OpenAIDriver**: Cost automatically calculated based on the model
- **ClaudeDriver**: Cost automatically calculated based on the model
- **AzureDriver**: Cost automatically calculated based on the model
## Batch Running and Testing Prompts
`run_suite_from_spec` enables you to define and run test suites against multiple models using a specification file. This powerful feature allows you to systematically test and compare different models using a consistent set of prompts and validation criteria. Here's how it works:
```python
from prompture import run_suite_from_spec
from prompture.drivers import MockDriver
spec = {
"meta": {"project": "test"},
"models": [{"id": "mock1", "driver": "mock", "options": {}}],
"tests": [
{
"id": "t1",
"prompt_template": "Extract user info: '{text}'",
"inputs": [{"text": "Juan is 28 and lives in Miami. He likes basketball and coding."}],
"schema": {"type": "object", "required": ["name", "interests"]}
}
]
}
drivers = {"mock": MockDriver()}
report = run_suite_from_spec(spec, drivers)
print(report)
```
The generated report includes comprehensive results for each test, model, and input combination:
- Validation status for each response
- Usage statistics (tokens, costs) per model
- Execution times
- Generated JSON responses
## Quick Usage (example):
```py
from prompture import run_suite_from_spec, drivers
spec = { ... }
report = run_suite_from_spec(spec, drivers={"mock": drivers.MockDriver()})
print(report)
```
## Ollama Model Comparison Example
This example demonstrates how to compare different Ollama models using a specific script located at `examples/ollama_models_comparison.py`.
| Model | Success | Prompt | Completion | Total | Fields | Validation | Name | Price | Variants | Screen Size | Warranty | Is New |
|------------------|---------|--------|------------|-------|--------|------------|---------------------|----------|----------|-------------|----------|--------|
| gpt-oss:20b | True | 801 | 945 | 1746 | 8 | ✓ | GalaxyFold Ultra | 1299.99 | 9 | 6.9 | 3 | True |
| deepseek-r1:latest | True | 757 | 679 | 1436 | 8 | ✗ | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | None | True |
| llama3.1:8b | True | 746 | 256 | 1002 | 8 | ✓ | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | 3 | True |
| gemma3:latest | True | 857 | 315 | 1172 | 8 | ✗ | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | None | True |
| qwen2.5:1.5b | True | 784 | 236 | 1020 | 8 | ✓ | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | 3 | True |
| qwen2.5:3b | True | 784 | 273 | 1057 | 9 | ✓ | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | 3 | True |
| mistral:latest | True | 928 | 337 | 1265 | 8 | ✓ | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | 3 | True |
> **Successful models (7):** gpt-oss:20b, deepseek-r1:latest, llama3.1:8b, gemma3:latest, qwen2.5:1.5b, qwen2.5:3b, mistral:latest
You can run this comparison yourself with:
`python examples/ollama_models_comparison.py`
This example script compares multiple Ollama models on a complex task of extracting structured information from a smartphone description using a detailed JSON schema. The purpose of this example is to illustrate how `Prompture` can be used to test and compare different models on the same structured output task, showing their success rates, token usage, and validation results.
**Location:** `examples/ollama_models_comparison.py`
Raw data
{
"_id": null,
"home_page": "https://github.com/jhd3197/prompture",
"name": "prompture",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Juan Denis",
"author_email": "juan@vene.co",
"download_url": "https://files.pythonhosted.org/packages/97/a1/270a8d969f7002e366d7a21cd9d7123123280f2e6d486f4c450029259f35/prompture-0.0.14.tar.gz",
"platform": null,
"description": "# Prompture\n\n`Prompture` is an API-first library for requesting structured **JSON** output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.\n\n## \u2728 Features\n\n- \u2705 **Structured Output**: Request models to return JSON only\n- \u2705 **Validation**: Automatic validation with `jsonschema`\n- \u2705 **Multi-driver**: Run the same specification against multiple drivers (OpenAI, Ollama, Claude, Azure, HTTP, mock)\n- \u2705 **Reports**: Generate JSON reports with results\n- \u2705 **Usage Tracking**: **NEW** - Automatic token and cost monitoring for all calls\n\n## \ud83c\udd95 Token and Cost Tracking (New)\n\nStarting with this version, `extract_and_jsonify` and `ask_for_json` automatically include token usage and cost information:\n\n```python\nfrom prompture import extract_and_jsonify\nfrom prompture.drivers import OllamaDriver\n\ndriver = OllamaDriver(endpoint=\"http://localhost:11434/api/generate\", model=\"gemma3\")\nresult = extract_and_jsonify(driver, \"Text to process\", json_schema)\n\n# Now returns both the response and usage information\njson_output = result[\"json_string\"]\nusage = result[\"usage\"]\n\nprint(f\"Tokens used: {usage['total_tokens']}\")\nprint(f\"Cost: ${usage['cost']:.6f}\")\n```\n\n### Return Structure\n\nThe main functions now return:\n```python\n{\n \"json_string\": str, # The original JSON string\n \"json_object\": dict, # The parsed JSON object\n \"usage\": {\n \"prompt_tokens\": int,\n \"completion_tokens\": int,\n \"total_tokens\": int,\n \"cost\": float # Cost in USD (0.0 for free models)\n }\n}\n```\n\n### Supported Drivers\n\n- **OllamaDriver**: Cost = $0.00 (free local models)\n- **OpenAIDriver**: Cost automatically calculated based on the model\n- **ClaudeDriver**: Cost automatically calculated based on the model\n- **AzureDriver**: Cost automatically calculated based on the model\n\n## Batch Running and Testing Prompts\n\n`run_suite_from_spec` enables you to define and run test suites against multiple models using a specification file. This powerful feature allows you to systematically test and compare different models using a consistent set of prompts and validation criteria. Here's how it works:\n\n```python\nfrom prompture import run_suite_from_spec\nfrom prompture.drivers import MockDriver\n\nspec = {\n \"meta\": {\"project\": \"test\"},\n \"models\": [{\"id\": \"mock1\", \"driver\": \"mock\", \"options\": {}}],\n \"tests\": [\n {\n \"id\": \"t1\",\n \"prompt_template\": \"Extract user info: '{text}'\",\n \"inputs\": [{\"text\": \"Juan is 28 and lives in Miami. He likes basketball and coding.\"}],\n \"schema\": {\"type\": \"object\", \"required\": [\"name\", \"interests\"]}\n }\n ]\n}\ndrivers = {\"mock\": MockDriver()}\nreport = run_suite_from_spec(spec, drivers)\nprint(report)\n```\n\nThe generated report includes comprehensive results for each test, model, and input combination:\n- Validation status for each response\n- Usage statistics (tokens, costs) per model\n- Execution times\n- Generated JSON responses\n\n## Quick Usage (example):\n\n```py\nfrom prompture import run_suite_from_spec, drivers\nspec = { ... }\nreport = run_suite_from_spec(spec, drivers={\"mock\": drivers.MockDriver()})\nprint(report)\n```\n\n## Ollama Model Comparison Example\n\nThis example demonstrates how to compare different Ollama models using a specific script located at `examples/ollama_models_comparison.py`.\n\n| Model | Success | Prompt | Completion | Total | Fields | Validation | Name | Price | Variants | Screen Size | Warranty | Is New |\n|------------------|---------|--------|------------|-------|--------|------------|---------------------|----------|----------|-------------|----------|--------|\n| gpt-oss:20b | True | 801 | 945 | 1746 | 8 | \u2713 | GalaxyFold Ultra | 1299.99 | 9 | 6.9 | 3 | True |\n| deepseek-r1:latest | True | 757 | 679 | 1436 | 8 | \u2717 | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | None | True |\n| llama3.1:8b | True | 746 | 256 | 1002 | 8 | \u2713 | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | 3 | True |\n| gemma3:latest | True | 857 | 315 | 1172 | 8 | \u2717 | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | None | True |\n| qwen2.5:1.5b | True | 784 | 236 | 1020 | 8 | \u2713 | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | 3 | True |\n| qwen2.5:3b | True | 784 | 273 | 1057 | 9 | \u2713 | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | 3 | True |\n| mistral:latest | True | 928 | 337 | 1265 | 8 | \u2713 | GalaxyFold Ultra | 1299.99 | 3 | 6.9 | 3 | True |\n\n> **Successful models (7):** gpt-oss:20b, deepseek-r1:latest, llama3.1:8b, gemma3:latest, qwen2.5:1.5b, qwen2.5:3b, mistral:latest\n\nYou can run this comparison yourself with:\n`python examples/ollama_models_comparison.py`\n\nThis example script compares multiple Ollama models on a complex task of extracting structured information from a smartphone description using a detailed JSON schema. The purpose of this example is to illustrate how `Prompture` can be used to test and compare different models on the same structured output task, showing their success rates, token usage, and validation results.\n\n**Location:** `examples/ollama_models_comparison.py`\n",
"bugtrack_url": null,
"license": null,
"summary": "Ask LLMs to return structured JSON and run cross-model tests. API-first.",
"version": "0.0.14",
"project_urls": {
"Homepage": "https://github.com/jhd3197/prompture"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "17d61c1cd9c054dd684ddccde4ca984974bfa0bec560360d10085a9359c97684",
"md5": "266e95fbc8ae397fa6773ab7cbe1254d",
"sha256": "ab7cd4364f1f54ac6330d932438ad9a7f621bc9c3bae11b9cbb98c274b6041e5"
},
"downloads": -1,
"filename": "prompture-0.0.14-py3-none-any.whl",
"has_sig": false,
"md5_digest": "266e95fbc8ae397fa6773ab7cbe1254d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 17103,
"upload_time": "2025-09-12T22:11:52",
"upload_time_iso_8601": "2025-09-12T22:11:52.997235Z",
"url": "https://files.pythonhosted.org/packages/17/d6/1c1cd9c054dd684ddccde4ca984974bfa0bec560360d10085a9359c97684/prompture-0.0.14-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "97a1270a8d969f7002e366d7a21cd9d7123123280f2e6d486f4c450029259f35",
"md5": "5a2bd03f64c09b0e9d627a7f32c82270",
"sha256": "9a5a255863c538fb3e4be3aa0928559be569cf0ce010ce2d75f77b6f7cabab63"
},
"downloads": -1,
"filename": "prompture-0.0.14.tar.gz",
"has_sig": false,
"md5_digest": "5a2bd03f64c09b0e9d627a7f32c82270",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 33246,
"upload_time": "2025-09-12T22:11:55",
"upload_time_iso_8601": "2025-09-12T22:11:55.950351Z",
"url": "https://files.pythonhosted.org/packages/97/a1/270a8d969f7002e366d7a21cd9d7123123280f2e6d486f4c450029259f35/prompture-0.0.14.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-12 22:11:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "jhd3197",
"github_project": "prompture",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "requests",
"specs": [
[
">=",
"2.28"
]
]
},
{
"name": "jsonschema",
"specs": [
[
">=",
"4.0"
]
]
},
{
"name": "pydantic",
"specs": [
[
">=",
"1.10"
]
]
},
{
"name": "pydantic-settings",
"specs": [
[
">=",
"2.0"
]
]
},
{
"name": "click",
"specs": [
[
">=",
"8.0"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
">=",
"0.19.0"
]
]
}
],
"lcname": "prompture"
}
JSON
