<div align="center">
<img src="docs/resources/image.png" alt="Automatic Goggles Logo" width="200"/>
# Automatic Goggles
[](https://badge.fury.io/py/automatic-goggles)
[](https://www.python.org/downloads/)
[](https://opensource.org/licenses/MIT)
[](https://pepy.tech/project/automatic-goggles)
**Post-Call Analysis & Conversational Evaluation**
*Extract structured fields from transcripts with confidence scores and evaluate conversation quality using Assertive LLM-as-a-Judge*
[Automatic-Goggles?](#-what-is-automatic-goggles) | [Installation](#-installation) | [Quick Start](#-examples) | [Features](#-features) | [Use-Cases](#-use-cases)
</div>
---
## 🎯 What is Automatic Goggles?
**Automatic Goggles** is a lightweight, production-ready Python package for analyzing conversation transcripts. It provides two core capabilities:
### 1. 🔍 Field Extraction
Extract structured data (names, emails, dates, custom fields) from transcripts with confidence scores
<div align="center">
<img src="docs/resources/fields_extraction.jpg" alt="Field Extraction" width="600"/>
</div>
### 2. ⚖️ Conversation Evaluation
Assess conversation quality against custom assertions using LLM-as-a-Judge
<div align="center">
<img src="docs/resources/asserts_evaluation.jpg" alt="Asserts Evaluation" width="600"/>
</div>
Built for **voice agent post-call analysis**, customer support quality assurance, and conversational AI evaluation.
---
## 🚀 Installation
```bash
pip install automatic-goggles
```
Requires Python 3.8+
## 🚀 Examples
### Field Extraction in 30 Seconds
```python
from transtype import TranscriptProcessor
fields = [
{
"field_name": "agent_email",
"field_type": "string",
"format_example": "agent@company.com",
"field_description": "The agent's email address for follow-up communication"
},
{
"field_name": "agent_name",
"field_type": "string",
"format_example": "John Doe",
"field_description": "Full name of the customer service agent"
}
]
processor = TranscriptProcessor(
api_key="your-openai-api-key",
fields=fields,
include_reasoning=True
)
conversation = {
"messages": [
{"role": "assistant", "content": "My name is Sarah Chen, you can reach me at sarah@example.com"},
{"role": "user", "content": "Thanks, I'll email you"}
]
}
result = processor.process(conversation)
```
**Output:**
```json
{
"fields": [
{
"field_name": "agent_email",
"field_value": "sarah@example.com",
"field_confidence": 0.92,
"field_reason": "Email explicitly mentioned by agent"
},
{
"field_name": "agent_name",
"field_value": "Sarah Chen",
"field_confidence": 0.95,
"field_reason": "Agent introduced herself by name"
}
]
}
```
### Conversation Evaluation in 30 Seconds
```python
from transtype import AssertsEvaluator
evaluator = AssertsEvaluator(
api_key="your-openai-api-key",
evaluation_steps=[
"Did the agent greet the customer politely?",
"Did the agent ask clarifying questions?",
"Did the agent resolve the customer's issue?",
"Did the agent offer additional help?"
],
threshold=0.7 # Pass threshold (0-1)
)
conversation = {
"messages": [
{"role": "user", "content": "My internet isn't working"},
{"role": "assistant", "content": "Good morning! I'm sorry to hear that. When did this issue start?"},
{"role": "user", "content": "This morning"},
{"role": "assistant", "content": "Let me help you troubleshoot. Can you check if your router is powered on?"}
]
}
result = evaluator.evaluate(conversation)
```
**Output:**
```json
{
"result": {
"score": 0.88,
"success": true,
"reason": "Agent demonstrated professionalism, asked clarifying questions, and initiated troubleshooting"
}
}
```
---
## 🔥 Features
### Field Extraction
- ✅ **Confidence-Weighted Scoring** - Log probability-based confidence scores (0-1)
- ✅ **Contextual Descriptions** - Improve accuracy with detailed field descriptions
- ✅ **Flexible Reasoning** - Toggle explanations on/off for performance/cost optimization
- ✅ **Multi-Field Support** - Extract multiple fields in one pass
- ✅ **Format Examples** - Guide extraction with format examples
### Conversation Evaluation
- ✅ **LLM-as-a-Judge** - Research-backed evaluation using GPT models
- ✅ **Custom Assertions** - Define your own quality criteria
- ✅ **Weighted Scoring** - Confidence-weighted scores
- ✅ **Pass/Fail Thresholds** - Configurable success criteria
- ✅ **Multi-Turn Support** - Evaluate entire conversations
### Technical Highlights
- ⚡ **Fast & Cost-Effective** - Optional reasoning for performance tuning
- 🧪 **Production-Ready** - Confidence scores for reliability filtering
- 📊 **Transparent** - Get reasoning explanations for every extraction/evaluation
---
## 💡 Use Cases
| Use Case | Description |
|----------|-------------|
| **Voice Agent Post-Call Analysis** | Extract key information (phone numbers, appointment dates, action items) after customer calls |
| **Quality Assurance** | Evaluate if agents followed scripts, were polite, and resolved issues |
| **Compliance Monitoring** | Verify agents disclosed required information (privacy policies, terms) |
| **Training & Coaching** | Identify coaching opportunities by evaluating agent performance against best practices |
| **Customer Insights** | Extract sentiment, pain points, and feature requests from support transcripts |
---
## 🛠️ Field Definition Schema
Each field requires:
| Property | Required | Description |
|----------|----------|-------------|
| `field_name` | ✅ Yes | Unique identifier for the field |
| `field_type` | ✅ Yes | Data type (currently supports `"string"`) |
| `format_example` | ✅ Yes | Example of expected format (e.g., `"(555) 123-4567"`) |
| `field_description` | ✅ Yes | Detailed context to guide extraction accuracy |
**Example:**
```python
{
"field_name": "customer_phone",
"field_type": "string",
"format_example": "(555) 123-4567",
"field_description": "The customer's primary phone number for callbacks. Look for 10-digit numbers in various formats."
}
```
---
## 🤝 Contributing
We welcome contributions! Here's how you can help:
1. **Report Bugs** - Open an issue on [GitHub](https://github.com/ashishorkalra/automatic-goggles-public/issues)
2. **Feature Requests** - Suggest new features via issues
3. **Pull Requests** - Submit PRs for bug fixes or features
4. **Documentation** - Improve docs or add examples
---
## 📝 License
MIT License - see [LICENSE](LICENSE) for details
---
## 🙏 Acknowledgments
- Inspired by [DeepEval's ConversationalGEval](https://github.com/confident-ai/deepeval) for confidence-weighted scoring
- Built with [DSPy](https://github.com/stanfordnlp/dspy) for robust LLM orchestration
- Powered by [OpenAI](https://openai.com/) language models
---
## 📬 Contact & Support
- **GitHub Issues**: [Report bugs or request features](https://github.com/ashishorkalra/automatic-goggles-public/issues)
- **Email**: ashishorkalra@gmail.com
- **Package**: [PyPI - automatic-goggles](https://pypi.org/project/automatic-goggles/)
---
<div align="center">
**Built with ❤️ by developers, for developers**
[⭐ Star on GitHub](https://github.com/ashishorkalra/automatic-goggles-public) | [📦 Install Now](https://pypi.org/project/automatic-goggles/)
</div>
Raw data
{
"_id": null,
"home_page": "https://github.com/ashishorkalra/automatic-goggles",
"name": "automatic-goggles",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.9",
"maintainer_email": null,
"keywords": "transcript, processing, field extraction, AI, natural language processing",
"author": "Ashish Kalra",
"author_email": "Your Name <your.email@example.com>",
"download_url": "https://files.pythonhosted.org/packages/24/b8/4876f63e082cc515fe0c5d11477d0f0d383b8cc96507d13f09060a7d7cbd/automatic_goggles-0.8.0.tar.gz",
"platform": null,
"description": "<div align=\"center\">\n\n<img src=\"docs/resources/image.png\" alt=\"Automatic Goggles Logo\" width=\"200\"/>\n\n# Automatic Goggles\n\n[](https://badge.fury.io/py/automatic-goggles)\n[](https://www.python.org/downloads/)\n[](https://opensource.org/licenses/MIT)\n[](https://pepy.tech/project/automatic-goggles)\n\n**Post-Call Analysis & Conversational Evaluation**\n\n*Extract structured fields from transcripts with confidence scores and evaluate conversation quality using Assertive LLM-as-a-Judge*\n\n[Automatic-Goggles?](#-what-is-automatic-goggles) | [Installation](#-installation) | [Quick Start](#-examples) | [Features](#-features) | [Use-Cases](#-use-cases)\n\n</div>\n\n---\n\n## \ud83c\udfaf What is Automatic Goggles?\n\n**Automatic Goggles** is a lightweight, production-ready Python package for analyzing conversation transcripts. It provides two core capabilities:\n\n### 1. \ud83d\udd0d Field Extraction\nExtract structured data (names, emails, dates, custom fields) from transcripts with confidence scores\n\n<div align=\"center\">\n<img src=\"docs/resources/fields_extraction.jpg\" alt=\"Field Extraction\" width=\"600\"/>\n</div>\n\n### 2. \u2696\ufe0f Conversation Evaluation\nAssess conversation quality against custom assertions using LLM-as-a-Judge\n\n<div align=\"center\">\n<img src=\"docs/resources/asserts_evaluation.jpg\" alt=\"Asserts Evaluation\" width=\"600\"/>\n</div>\n\nBuilt for **voice agent post-call analysis**, customer support quality assurance, and conversational AI evaluation.\n\n---\n\n## \ud83d\ude80 Installation\n\n```bash\npip install automatic-goggles\n```\n\nRequires Python 3.8+\n\n## \ud83d\ude80 Examples\n\n### Field Extraction in 30 Seconds\n\n```python\nfrom transtype import TranscriptProcessor\n\nfields = [\n {\n \"field_name\": \"agent_email\",\n \"field_type\": \"string\",\n \"format_example\": \"agent@company.com\",\n \"field_description\": \"The agent's email address for follow-up communication\"\n },\n {\n \"field_name\": \"agent_name\",\n \"field_type\": \"string\",\n \"format_example\": \"John Doe\",\n \"field_description\": \"Full name of the customer service agent\"\n }\n]\n\nprocessor = TranscriptProcessor(\n api_key=\"your-openai-api-key\",\n fields=fields,\n include_reasoning=True\n)\n\nconversation = {\n \"messages\": [\n {\"role\": \"assistant\", \"content\": \"My name is Sarah Chen, you can reach me at sarah@example.com\"},\n {\"role\": \"user\", \"content\": \"Thanks, I'll email you\"}\n ]\n}\n\nresult = processor.process(conversation)\n```\n\n**Output:**\n```json\n{\n \"fields\": [\n {\n \"field_name\": \"agent_email\",\n \"field_value\": \"sarah@example.com\",\n \"field_confidence\": 0.92,\n \"field_reason\": \"Email explicitly mentioned by agent\"\n },\n {\n \"field_name\": \"agent_name\",\n \"field_value\": \"Sarah Chen\",\n \"field_confidence\": 0.95,\n \"field_reason\": \"Agent introduced herself by name\"\n }\n ]\n}\n```\n\n### Conversation Evaluation in 30 Seconds\n\n```python\nfrom transtype import AssertsEvaluator\n\nevaluator = AssertsEvaluator(\n api_key=\"your-openai-api-key\",\n evaluation_steps=[\n \"Did the agent greet the customer politely?\",\n \"Did the agent ask clarifying questions?\",\n \"Did the agent resolve the customer's issue?\",\n \"Did the agent offer additional help?\"\n ],\n threshold=0.7 # Pass threshold (0-1)\n)\n\nconversation = {\n \"messages\": [\n {\"role\": \"user\", \"content\": \"My internet isn't working\"},\n {\"role\": \"assistant\", \"content\": \"Good morning! I'm sorry to hear that. When did this issue start?\"},\n {\"role\": \"user\", \"content\": \"This morning\"},\n {\"role\": \"assistant\", \"content\": \"Let me help you troubleshoot. Can you check if your router is powered on?\"}\n ]\n}\n\nresult = evaluator.evaluate(conversation)\n```\n\n**Output:**\n```json\n{\n \"result\": {\n \"score\": 0.88,\n \"success\": true,\n \"reason\": \"Agent demonstrated professionalism, asked clarifying questions, and initiated troubleshooting\"\n }\n}\n```\n\n---\n\n## \ud83d\udd25 Features\n\n### Field Extraction\n- \u2705 **Confidence-Weighted Scoring** - Log probability-based confidence scores (0-1)\n- \u2705 **Contextual Descriptions** - Improve accuracy with detailed field descriptions\n- \u2705 **Flexible Reasoning** - Toggle explanations on/off for performance/cost optimization\n- \u2705 **Multi-Field Support** - Extract multiple fields in one pass\n- \u2705 **Format Examples** - Guide extraction with format examples\n\n### Conversation Evaluation\n- \u2705 **LLM-as-a-Judge** - Research-backed evaluation using GPT models\n- \u2705 **Custom Assertions** - Define your own quality criteria\n- \u2705 **Weighted Scoring** - Confidence-weighted scores\n- \u2705 **Pass/Fail Thresholds** - Configurable success criteria\n- \u2705 **Multi-Turn Support** - Evaluate entire conversations\n\n### Technical Highlights\n- \u26a1 **Fast & Cost-Effective** - Optional reasoning for performance tuning\n- \ud83e\uddea **Production-Ready** - Confidence scores for reliability filtering\n- \ud83d\udcca **Transparent** - Get reasoning explanations for every extraction/evaluation\n\n---\n\n## \ud83d\udca1 Use Cases\n\n| Use Case | Description |\n|----------|-------------|\n| **Voice Agent Post-Call Analysis** | Extract key information (phone numbers, appointment dates, action items) after customer calls |\n| **Quality Assurance** | Evaluate if agents followed scripts, were polite, and resolved issues |\n| **Compliance Monitoring** | Verify agents disclosed required information (privacy policies, terms) |\n| **Training & Coaching** | Identify coaching opportunities by evaluating agent performance against best practices |\n| **Customer Insights** | Extract sentiment, pain points, and feature requests from support transcripts |\n\n---\n\n## \ud83d\udee0\ufe0f Field Definition Schema\n\nEach field requires:\n\n| Property | Required | Description |\n|----------|----------|-------------|\n| `field_name` | \u2705 Yes | Unique identifier for the field |\n| `field_type` | \u2705 Yes | Data type (currently supports `\"string\"`) |\n| `format_example` | \u2705 Yes | Example of expected format (e.g., `\"(555) 123-4567\"`) |\n| `field_description` | \u2705 Yes | Detailed context to guide extraction accuracy |\n\n**Example:**\n```python\n{\n \"field_name\": \"customer_phone\",\n \"field_type\": \"string\",\n \"format_example\": \"(555) 123-4567\",\n \"field_description\": \"The customer's primary phone number for callbacks. Look for 10-digit numbers in various formats.\"\n}\n```\n\n---\n\n## \ud83e\udd1d Contributing\n\nWe welcome contributions! Here's how you can help:\n\n1. **Report Bugs** - Open an issue on [GitHub](https://github.com/ashishorkalra/automatic-goggles-public/issues)\n2. **Feature Requests** - Suggest new features via issues\n3. **Pull Requests** - Submit PRs for bug fixes or features\n4. **Documentation** - Improve docs or add examples\n\n---\n\n## \ud83d\udcdd License\n\nMIT License - see [LICENSE](LICENSE) for details\n\n---\n\n## \ud83d\ude4f Acknowledgments\n\n- Inspired by [DeepEval's ConversationalGEval](https://github.com/confident-ai/deepeval) for confidence-weighted scoring\n- Built with [DSPy](https://github.com/stanfordnlp/dspy) for robust LLM orchestration\n- Powered by [OpenAI](https://openai.com/) language models\n\n---\n\n## \ud83d\udcec Contact & Support\n\n- **GitHub Issues**: [Report bugs or request features](https://github.com/ashishorkalra/automatic-goggles-public/issues)\n- **Email**: ashishorkalra@gmail.com\n- **Package**: [PyPI - automatic-goggles](https://pypi.org/project/automatic-goggles/)\n\n---\n\n<div align=\"center\">\n\n**Built with \u2764\ufe0f by developers, for developers**\n\n[\u2b50 Star on GitHub](https://github.com/ashishorkalra/automatic-goggles-public) | [\ud83d\udce6 Install Now](https://pypi.org/project/automatic-goggles/)\n\n</div>\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A package for extracting structured fields from call transcripts and evaluating conversation quality with confidence scores",
"version": "0.8.0",
"project_urls": {
"Bug Reports": "https://github.com/ashishorkalra/automatic-goggles-public/issues",
"Documentation": "https://github.com/ashishorkalra/automatic-goggles-public#readme",
"Homepage": "https://github.com/ashishorkalra/automatic-goggles-public",
"Repository": "https://github.com/ashishorkalra/automatic-goggles-public"
},
"split_keywords": [
"transcript",
" processing",
" field extraction",
" ai",
" natural language processing"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "2e5fbfadb8d202fdb147862b92f4896da275ed486aa39faf4c6e1be7b7ad35ef",
"md5": "2ea1946e3f0976aba80c7e080b48175b",
"sha256": "54998e1900f44f3d067ceca695be34b4e3f326a1e2f5636ed23eb18667014f5c"
},
"downloads": -1,
"filename": "automatic_goggles-0.8.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2ea1946e3f0976aba80c7e080b48175b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.9",
"size": 10974,
"upload_time": "2025-11-02T13:03:34",
"upload_time_iso_8601": "2025-11-02T13:03:34.674710Z",
"url": "https://files.pythonhosted.org/packages/2e/5f/bfadb8d202fdb147862b92f4896da275ed486aa39faf4c6e1be7b7ad35ef/automatic_goggles-0.8.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "24b84876f63e082cc515fe0c5d11477d0f0d383b8cc96507d13f09060a7d7cbd",
"md5": "dcc369354abb003b5ec540b9e14f19ea",
"sha256": "672899ddfcf8967e92c2e1598952f6c31af8bac258b239d95f25a5020ef0c5ad"
},
"downloads": -1,
"filename": "automatic_goggles-0.8.0.tar.gz",
"has_sig": false,
"md5_digest": "dcc369354abb003b5ec540b9e14f19ea",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.9",
"size": 11156,
"upload_time": "2025-11-02T13:03:36",
"upload_time_iso_8601": "2025-11-02T13:03:36.116390Z",
"url": "https://files.pythonhosted.org/packages/24/b8/4876f63e082cc515fe0c5d11477d0f0d383b8cc96507d13f09060a7d7cbd/automatic_goggles-0.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-11-02 13:03:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ashishorkalra",
"github_project": "automatic-goggles",
"github_not_found": true,
"lcname": "automatic-goggles"
}
JSON
