# ContextGem: Effortless LLM extraction from documents
[](https://github.com/shcherbak-ai/contextgem/actions/workflows/ci-tests.yml)
[](https://github.com/shcherbak-ai/contextgem/actions)
[](https://github.com/shcherbak-ai/contextgem/actions/workflows/docs.yml)
[](https://shcherbak-ai.github.io/contextgem/)
[](https://opensource.org/licenses/Apache-2.0)
[](https://www.python.org/downloads/)
[](https://github.com/shcherbak-ai/contextgem/actions/workflows/codeql.yml)
[](https://github.com/shcherbak-ai/contextgem/actions/workflows/bandit-security.yml)
[](https://github.com/psf/black)
[](https://pycqa.github.io/isort/)
[](https://pydantic.dev)
[](https://python-poetry.org/)
[](https://github.com/pre-commit/pre-commit)
[](CODE_OF_CONDUCT.md)
[](https://deepwiki.com/shcherbak-ai/contextgem)
[](https://github.com/shcherbak-ai/contextgem/commits/main)
<img src="https://contextgem.dev/_static/tab_solid.png" alt="ContextGem: 2nd Product of the week" width="250">
<br/><br/>
ContextGem is a free, open-source LLM framework that makes it radically easier to extract structured data and insights from documents — with minimal code.
---
## 💎 Why ContextGem?
Most popular LLM frameworks for extracting structured data from documents require extensive boilerplate code to extract even basic information. This significantly increases development time and complexity.
ContextGem addresses this challenge by providing a flexible, intuitive framework that extracts structured data and insights from documents with minimal effort. Complex, most time-consuming parts are handled with **powerful abstractions**, eliminating boilerplate code and reducing development overhead.
📖 Read more on the project [motivation](https://contextgem.dev/motivation.html) in the documentation.
## ⭐ Key features
<table>
<thead>
<tr style="text-align: left; opacity: 0.8;">
<th style="width: 75%">Built-in abstractions</th>
<th style="width: 10%"><strong>ContextGem</strong></th>
<th style="width: 15%">Other LLM frameworks*</th>
</tr>
</thead>
<tbody>
<tr>
<td>
Automated dynamic prompts
</td>
<td>🟢</td>
<td>◯</td>
</tr>
<tr>
<td>
Automated data modelling and validators
</td>
<td>🟢</td>
<td>◯</td>
</tr>
<tr>
<td>
Precise granular reference mapping (paragraphs & sentences)
</td>
<td>🟢</td>
<td>◯</td>
</tr>
<tr>
<td>
Justifications (reasoning backing the extraction)
</td>
<td>🟢</td>
<td>◯</td>
</tr>
<tr>
<td>
Neural segmentation (SaT)
</td>
<td>🟢</td>
<td>◯</td>
</tr>
<tr>
<td>
Multilingual support (I/O without prompting)
</td>
<td>🟢</td>
<td>◯</td>
</tr>
<tr>
<td>
Single, unified extraction pipeline (declarative, reusable, fully serializable)
</td>
<td>🟢</td>
<td>🟡</td>
</tr>
<tr>
<td>
Grouped LLMs with role-specific tasks
</td>
<td>🟢</td>
<td>🟡</td>
</tr>
<tr>
<td>
Nested context extraction
</td>
<td>🟢</td>
<td>🟡</td>
</tr>
<tr>
<td>
Unified, fully serializable results storage model (document)
</td>
<td>🟢</td>
<td>🟡</td>
</tr>
<tr>
<td>
Extraction task calibration with examples
</td>
<td>🟢</td>
<td>🟡</td>
</tr>
<tr>
<td>
Built-in concurrent I/O processing
</td>
<td>🟢</td>
<td>🟡</td>
</tr>
<tr>
<td>
Automated usage & costs tracking
</td>
<td>🟢</td>
<td>🟡</td>
</tr>
<tr>
<td>
Fallback and retry logic
</td>
<td>🟢</td>
<td>🟢</td>
</tr>
<tr>
<td>
Multiple LLM providers
</td>
<td>🟢</td>
<td>🟢</td>
</tr>
</tbody>
</table>
🟢 - fully supported - no additional setup required<br>
🟡 - partially supported - requires additional setup<br>
◯ - not supported - requires custom logic
\* See [descriptions](https://contextgem.dev/motivation.html#the-contextgem-solution) of ContextGem abstractions and [comparisons](https://contextgem.dev/vs_other_frameworks.html) of specific implementation examples using ContextGem and other popular open-source LLM frameworks.
## 💡 What you can build
With **minimal code**, you can:
- **Extract structured data** from documents (text, images)
- **Identify and analyze key aspects** (topics, themes, categories) within documents ([learn more](https://contextgem.dev/aspects/aspects.html))
- **Extract specific concepts** (entities, facts, conclusions, assessments) from documents ([learn more](https://contextgem.dev/concepts/supported_concepts.html))
- **Build complex extraction workflows** through a simple, intuitive API
- **Create multi-level extraction pipelines** (aspects containing concepts, hierarchical aspects)
<br/>
## 📦 Installation
```bash
pip install -U contextgem
```
> **⚡ v0.5.0+**: ContextGem now installs 7.5x faster with minimal dependencies (no torch/transformers required), making it easier to integrate into existing ML environments.
## 🚀 Quick start
```python
# Quick Start Example - Extracting anomalies from a document, with source references and justifications
import os
from contextgem import Document, DocumentLLM, StringConcept
# Sample document text (shortened for brevity)
doc = Document(
raw_text=(
"Consultancy Agreement\n"
"This agreement between Company A (Supplier) and Company B (Customer)...\n"
"The term of the agreement is 1 year from the Effective Date...\n"
"The Supplier shall provide consultancy services as described in Annex 2...\n"
"The Customer shall pay the Supplier within 30 calendar days of receiving an invoice...\n"
"The purple elephant danced gracefully on the moon while eating ice cream.\n" # 💎 anomaly
"Time-traveling dinosaurs will review all deliverables before acceptance.\n" # 💎 another anomaly
"This agreement is governed by the laws of Norway...\n"
),
)
# Attach a document-level concept
doc.concepts = [
StringConcept(
name="Anomalies", # in longer contexts, this concept is hard to capture with RAG
description="Anomalies in the document",
add_references=True,
reference_depth="sentences",
add_justifications=True,
justification_depth="brief",
# see the docs for more configuration options
)
# add more concepts to the document, if needed
# see the docs for available concepts: StringConcept, JsonObjectConcept, etc.
]
# Or use `doc.add_concepts([...])`
# Define an LLM for extracting information from the document
llm = DocumentLLM(
model="openai/gpt-4o-mini", # or another provider/LLM
api_key=os.environ.get(
"CONTEXTGEM_OPENAI_API_KEY"
), # your API key for the LLM provider
# see the docs for more configuration options
)
# Extract information from the document
doc = llm.extract_all(doc) # or use async version `await llm.extract_all_async(doc)`
# Access extracted information in the document object
anomalies_concept = doc.concepts[0]
# or `doc.get_concept_by_name("Anomalies")`
for item in anomalies_concept.extracted_items:
print(f"Anomaly:")
print(f" {item.value}")
print(f"Justification:")
print(f" {item.justification}")
print("Reference paragraphs:")
for p in item.reference_paragraphs:
print(f" - {p.raw_text}")
print("Reference sentences:")
for s in item.reference_sentences:
print(f" - {s.raw_text}")
print()
```
[](https://colab.research.google.com/github/shcherbak-ai/contextgem/blob/main/dev/notebooks/readme/quickstart_concept.ipynb)
---
### 📚 More Examples
**Basic usage:**
- [Aspect Extraction from Document](https://contextgem.dev/quickstart.html#aspect-extraction-from-document)
- [Extracting Aspect with Sub-Aspects](https://contextgem.dev/quickstart.html#extracting-aspect-with-sub-aspects)
- [Concept Extraction from Aspect](https://contextgem.dev/quickstart.html#concept-extraction-from-aspect)
- [Concept Extraction from Document (text)](https://contextgem.dev/quickstart.html#concept-extraction-from-document-text)
- [Concept Extraction from Document (vision)](https://contextgem.dev/quickstart.html#concept-extraction-from-document-vision)
- [LLM chat interface](https://contextgem.dev/quickstart.html#lightweight-llm-chat-interface)
**Advanced usage:**
- [Extracting Aspects Containing Concepts](https://contextgem.dev/advanced_usage.html#extracting-aspects-with-concepts)
- [Extracting Aspects and Concepts from a Document](https://contextgem.dev/advanced_usage.html#extracting-aspects-and-concepts-from-a-document)
- [Using a Multi-LLM Pipeline to Extract Data from Several Documents](https://contextgem.dev/advanced_usage.html#using-a-multi-llm-pipeline-to-extract-data-from-several-documents)
## 🔄 Document converters
To create a ContextGem document for LLM analysis, you can either pass raw text directly, or use built-in converters that handle various file formats.
### 📄 DOCX converter
ContextGem provides built-in converter to easily transform DOCX files into LLM-ready data.
- **Comprehensive extraction of document elements**: paragraphs, headings, lists, tables, comments, footnotes, textboxes, headers/footers, links, embedded images, and inline formatting
- **Document structure preservation** with rich metadata for improved LLM analysis
- **Built-in converter** that directly processes Word XML
```python
# Using ContextGem's DocxConverter
from contextgem import DocxConverter
converter = DocxConverter()
# Convert a DOCX file to an LLM-ready ContextGem Document
# from path
document = converter.convert("path/to/document.docx")
# or from file object
with open("path/to/document.docx", "rb") as docx_file_object:
document = converter.convert(docx_file_object)
# Perform data extraction on the resulting Document object
# document.add_aspects(...)
# document.add_concepts(...)
# llm.extract_all(document)
# You can also use DocxConverter instance as a standalone text extractor
docx_text = converter.convert_to_text_format(
"path/to/document.docx",
output_format="markdown", # or "raw"
)
```
📖 Learn more about [DOCX converter features](https://contextgem.dev/converters/docx.html) in the documentation.
## 🎯 Focused document analysis
ContextGem leverages LLMs' long context windows to deliver superior extraction accuracy from individual documents. Unlike RAG approaches that often [struggle with complex concepts and nuanced insights](https://www.linkedin.com/pulse/raging-contracts-pitfalls-rag-contract-review-shcherbak-ai-ptg3f), ContextGem capitalizes on continuously expanding context capacity, evolving LLM capabilities, and decreasing costs. This focused approach enables direct information extraction from complete documents, eliminating retrieval inconsistencies while optimizing for in-depth single-document analysis. While this delivers higher accuracy for individual documents, ContextGem does not currently support cross-document querying or corpus-wide retrieval - for these use cases, modern RAG systems (e.g., LlamaIndex, Haystack) remain more appropriate.
📖 Read more on [how ContextGem works](https://contextgem.dev/how_it_works.html) in the documentation.
## 🤖 Supported LLMs
ContextGem supports both cloud-based and local LLMs through [LiteLLM](https://github.com/BerriAI/litellm) integration:
- **Cloud LLMs**: OpenAI, Anthropic, Google, Azure OpenAI, xAI, and more
- **Local LLMs**: Run models locally using providers like Ollama, LM Studio, etc.
- **Model Architectures**: Works with both reasoning/CoT-capable (e.g. o4-mini) and non-reasoning models (e.g. gpt-4.1)
- **Simple API**: Unified interface for all LLMs with easy provider switching
> **💡 Model Selection Note:** For reliable structured extraction, we recommend using models with performance equivalent to or exceeding `gpt-4o-mini`. Smaller models (such as 8B parameter models) may struggle with ContextGem's detailed extraction instructions. If you encounter issues with smaller models, see our [troubleshooting guide](https://contextgem.dev/optimizations/optimization_small_llm_troubleshooting.html) for potential solutions.
📖 Learn more about [supported LLM providers and models](https://contextgem.dev/llms/supported_llms.html), how to [configure LLMs](https://contextgem.dev/llms/llm_config.html), and [LLM extraction methods](https://contextgem.dev/llms/llm_extraction_methods.html) in the documentation.
## ⚡ Optimizations
ContextGem documentation offers guidance on optimization strategies to maximize performance, minimize costs, and enhance extraction accuracy:
- [Optimizing for Accuracy](https://contextgem.dev/optimizations/optimization_accuracy.html)
- [Optimizing for Speed](https://contextgem.dev/optimizations/optimization_speed.html)
- [Optimizing for Cost](https://contextgem.dev/optimizations/optimization_cost.html)
- [Dealing with Long Documents](https://contextgem.dev/optimizations/optimization_long_docs.html)
- [Choosing the Right LLM(s)](https://contextgem.dev/optimizations/optimization_choosing_llm.html)
- [Troubleshooting Issues with Small Models](https://contextgem.dev/optimizations/optimization_small_llm_troubleshooting.html)
## 💾 Serializing results
ContextGem allows you to save and load Document objects, pipelines, and LLM configurations with built-in serialization methods:
- Save processed documents to avoid repeating expensive LLM calls
- Transfer extraction results between systems
- Persist pipeline and LLM configurations for later reuse
📖 Learn more about [serialization options](https://contextgem.dev/serialization.html) in the documentation.
## 📚 Documentation
📖 **Full documentation:** [contextgem.dev](https://contextgem.dev)
📄 **Raw documentation for LLMs:** Available at [`docs/docs-raw-for-llm.txt`](https://github.com/shcherbak-ai/contextgem/blob/main/docs/docs-raw-for-llm.txt) - automatically generated, optimized for LLM ingestion.
🤖 **AI-powered code exploration:** [DeepWiki](https://deepwiki.com/shcherbak-ai/contextgem) provides visual architecture maps and natural language Q&A for the codebase.
📈 **Change history:** See the [CHANGELOG](https://github.com/shcherbak-ai/contextgem/blob/main/CHANGELOG.md) for version history, improvements, and bug fixes.
## 💬 Community
🐛 **Found a bug or have a feature request?** [Open an issue](https://github.com/shcherbak-ai/contextgem/issues/new) on GitHub.
💭 **Need help or want to discuss?** Start a thread in [GitHub Discussions](https://github.com/shcherbak-ai/contextgem/discussions/new/).
## 🤝 Contributing
We welcome contributions from the community - whether it's fixing a typo or developing a completely new feature!
📋 **Get started:** Check out our [Contributor Guidelines](https://github.com/shcherbak-ai/contextgem/blob/main/CONTRIBUTING.md).
## 🔐 Security
This project is automatically scanned for security vulnerabilities using multiple security tools:
- **[CodeQL](https://codeql.github.com/)** - GitHub's semantic code analysis engine for vulnerability detection
- **[Bandit](https://github.com/PyCQA/bandit)** - Python security linter for common security issues
- **[Snyk](https://snyk.io)** - Dependency vulnerability monitoring (used as needed)
🛡️ **Security policy:** See [SECURITY](https://github.com/shcherbak-ai/contextgem/blob/main/SECURITY.md) file for details.
## 💖 Acknowledgements
ContextGem relies on these excellent open-source packages:
- [aiolimiter](https://github.com/mjpieters/aiolimiter): Powerful rate limiting for async operations
- [Jinja2](https://github.com/pallets/jinja): Fast, expressive template engine that powers our dynamic prompt rendering
- [litellm](https://github.com/BerriAI/litellm): Unified interface to multiple LLM providers with seamless provider switching
- [loguru](https://github.com/Delgan/loguru): Simple yet powerful logging that enhances debugging and observability
- [lxml](https://github.com/lxml/lxml): High-performance XML processing library for parsing DOCX document structure
- [pydantic](https://github.com/pydantic/pydantic): The gold standard for data validation
- [python-ulid](https://github.com/mdomke/python-ulid): Efficient ULID generation
- [wtpsplit-lite](https://github.com/superlinear-ai/wtpsplit-lite): Lightweight version of [wtpsplit](https://github.com/segment-any-text/wtpsplit) for state-of-the-art text segmentation using wtpsplit's SaT models
## 🌱 Support the project
ContextGem is just getting started, and your support means the world to us!
⭐ **Star the project** if you find ContextGem useful
📢 **Share it** with others who might benefit
🔧 **Contribute** with feedback, issues, or code improvements
Your engagement is what makes this project grow!
## 📄 License & Contact
**License:** Apache 2.0 License - see the [LICENSE](https://github.com/shcherbak-ai/contextgem/blob/main/LICENSE) and [NOTICE](https://github.com/shcherbak-ai/contextgem/blob/main/NOTICE) files for details.
**Copyright:** © 2025 [Shcherbak AI AS](https://shcherbak.ai), an AI engineering company building tools for AI/ML/NLP developers.
**Connect:** [LinkedIn](https://www.linkedin.com/in/sergii-shcherbak-10068866/) or [X](https://x.com/seshch) for questions or collaboration ideas.
Built with ❤️ in Oslo, Norway.
Raw data
{
"_id": null,
"home_page": "https://github.com/shcherbak-ai/contextgem",
"name": "contextgem",
"maintainer": "shcherbak-ai",
"docs_url": null,
"requires_python": "<3.14,>=3.10",
"maintainer_email": "sergii@shcherbak.ai",
"keywords": "artificial-intelligence, aspect-extraction, automated-prompting, concept-extraction, content-extraction, context-aware, contextgem, contract-analysis, contract-automation, contract-intelligence, contract-management, contract-parsing, contract-review, data-extraction, document, document-analysis, document-extraction, document-intelligence, document-parsing, document-pipeline, document-processing, document-qa, document-understanding, docx, entity-extraction, extraction-justifications, extraction-pipeline, fintech, generative-ai, information-extraction, insights-extraction, knowledge-extraction, large-language-models, legaltech, llm, llm-extraction, llm-framework, llm-library, llm-reasoning, low-code, machine-learning, multilingual, multimodal, neural-segmentation, nlp, no-prompt-engineering, prompt-free, question-answering, reference-mapping, semantic-analysis, structured-data, structured-data-extraction, text-analysis, text-processing, topic-extraction, unstructured-data, zero-shot",
"author": "shcherbak-ai",
"author_email": "sergii@shcherbak.ai",
"download_url": "https://files.pythonhosted.org/packages/e8/95/228b0e8487a265d911ef6d06a13eb1ae2aa65ed6657697e2bd8cf1be0046/contextgem-0.11.1.tar.gz",
"platform": null,
"description": "![ContextGem](https://contextgem.dev/_static/contextgem_readme_header.png \"ContextGem - Effortless LLM extraction from documents\")\n\n# ContextGem: Effortless LLM extraction from documents\n\n[](https://github.com/shcherbak-ai/contextgem/actions/workflows/ci-tests.yml)\n[](https://github.com/shcherbak-ai/contextgem/actions)\n[](https://github.com/shcherbak-ai/contextgem/actions/workflows/docs.yml)\n[](https://shcherbak-ai.github.io/contextgem/)\n[](https://opensource.org/licenses/Apache-2.0)\n\n[](https://www.python.org/downloads/)\n[](https://github.com/shcherbak-ai/contextgem/actions/workflows/codeql.yml)\n[](https://github.com/shcherbak-ai/contextgem/actions/workflows/bandit-security.yml)\n[](https://github.com/psf/black)\n[](https://pycqa.github.io/isort/)\n[](https://pydantic.dev)\n[](https://python-poetry.org/)\n[](https://github.com/pre-commit/pre-commit)\n[](CODE_OF_CONDUCT.md)\n[](https://deepwiki.com/shcherbak-ai/contextgem)\n[](https://github.com/shcherbak-ai/contextgem/commits/main)\n\n<img src=\"https://contextgem.dev/_static/tab_solid.png\" alt=\"ContextGem: 2nd Product of the week\" width=\"250\">\n<br/><br/>\n\nContextGem is a free, open-source LLM framework that makes it radically easier to extract structured data and insights from documents \u2014 with minimal code.\n\n---\n\n\n## \ud83d\udc8e Why ContextGem?\n\nMost popular LLM frameworks for extracting structured data from documents require extensive boilerplate code to extract even basic information. This significantly increases development time and complexity.\n\nContextGem addresses this challenge by providing a flexible, intuitive framework that extracts structured data and insights from documents with minimal effort. Complex, most time-consuming parts are handled with **powerful abstractions**, eliminating boilerplate code and reducing development overhead.\n\n\ud83d\udcd6 Read more on the project [motivation](https://contextgem.dev/motivation.html) in the documentation.\n\n\n## \u2b50 Key features\n\n<table>\n <thead>\n <tr style=\"text-align: left; opacity: 0.8;\">\n <th style=\"width: 75%\">Built-in abstractions</th>\n <th style=\"width: 10%\"><strong>ContextGem</strong></th>\n <th style=\"width: 15%\">Other LLM frameworks*</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <td>\n Automated dynamic prompts\n </td>\n <td>\ud83d\udfe2</td>\n <td>\u25ef</td>\n </tr>\n <tr>\n <td>\n Automated data modelling and validators\n </td>\n <td>\ud83d\udfe2</td>\n <td>\u25ef</td>\n </tr>\n <tr>\n <td>\n Precise granular reference mapping (paragraphs & sentences)\n </td>\n <td>\ud83d\udfe2</td>\n <td>\u25ef</td>\n </tr>\n <tr>\n <td>\n Justifications (reasoning backing the extraction)\n </td>\n <td>\ud83d\udfe2</td>\n <td>\u25ef</td>\n </tr>\n <tr>\n <td>\n Neural segmentation (SaT)\n </td>\n <td>\ud83d\udfe2</td>\n <td>\u25ef</td>\n </tr>\n <tr>\n <td>\n Multilingual support (I/O without prompting)\n </td>\n <td>\ud83d\udfe2</td>\n <td>\u25ef</td>\n </tr>\n <tr>\n <td>\n Single, unified extraction pipeline (declarative, reusable, fully serializable)\n </td>\n <td>\ud83d\udfe2</td>\n <td>\ud83d\udfe1</td>\n </tr>\n <tr>\n <td>\n Grouped LLMs with role-specific tasks\n </td>\n <td>\ud83d\udfe2</td>\n <td>\ud83d\udfe1</td>\n </tr>\n <tr>\n <td>\n Nested context extraction\n </td>\n <td>\ud83d\udfe2</td>\n <td>\ud83d\udfe1</td>\n </tr>\n <tr>\n <td>\n Unified, fully serializable results storage model (document)\n </td>\n <td>\ud83d\udfe2</td>\n <td>\ud83d\udfe1</td>\n </tr>\n <tr>\n <td>\n Extraction task calibration with examples\n </td>\n <td>\ud83d\udfe2</td>\n <td>\ud83d\udfe1</td>\n </tr>\n <tr>\n <td>\n Built-in concurrent I/O processing\n </td>\n <td>\ud83d\udfe2</td>\n <td>\ud83d\udfe1</td>\n </tr>\n <tr>\n <td>\n Automated usage & costs tracking\n </td>\n <td>\ud83d\udfe2</td>\n <td>\ud83d\udfe1</td>\n </tr>\n <tr>\n <td>\n Fallback and retry logic\n </td>\n <td>\ud83d\udfe2</td>\n <td>\ud83d\udfe2</td>\n </tr>\n <tr>\n <td>\n Multiple LLM providers\n </td>\n <td>\ud83d\udfe2</td>\n <td>\ud83d\udfe2</td>\n </tr>\n </tbody>\n</table>\n\n\ud83d\udfe2 - fully supported - no additional setup required<br>\n\ud83d\udfe1 - partially supported - requires additional setup<br>\n\u25ef - not supported - requires custom logic\n\n\\* See [descriptions](https://contextgem.dev/motivation.html#the-contextgem-solution) of ContextGem abstractions and [comparisons](https://contextgem.dev/vs_other_frameworks.html) of specific implementation examples using ContextGem and other popular open-source LLM frameworks.\n\n## \ud83d\udca1 What you can build\n\nWith **minimal code**, you can:\n\n- **Extract structured data** from documents (text, images)\n- **Identify and analyze key aspects** (topics, themes, categories) within documents ([learn more](https://contextgem.dev/aspects/aspects.html))\n- **Extract specific concepts** (entities, facts, conclusions, assessments) from documents ([learn more](https://contextgem.dev/concepts/supported_concepts.html))\n- **Build complex extraction workflows** through a simple, intuitive API\n- **Create multi-level extraction pipelines** (aspects containing concepts, hierarchical aspects)\n\n<br/>\n\n![ContextGem extraction example](https://contextgem.dev/_static/readme_code_snippet.png \"ContextGem extraction example\")\n\n\n## \ud83d\udce6 Installation\n\n```bash\npip install -U contextgem\n```\n\n> **\u26a1 v0.5.0+**: ContextGem now installs 7.5x faster with minimal dependencies (no torch/transformers required), making it easier to integrate into existing ML environments.\n\n\n## \ud83d\ude80 Quick start\n\n```python\n# Quick Start Example - Extracting anomalies from a document, with source references and justifications\n\nimport os\n\nfrom contextgem import Document, DocumentLLM, StringConcept\n\n# Sample document text (shortened for brevity)\ndoc = Document(\n raw_text=(\n \"Consultancy Agreement\\n\"\n \"This agreement between Company A (Supplier) and Company B (Customer)...\\n\"\n \"The term of the agreement is 1 year from the Effective Date...\\n\"\n \"The Supplier shall provide consultancy services as described in Annex 2...\\n\"\n \"The Customer shall pay the Supplier within 30 calendar days of receiving an invoice...\\n\"\n \"The purple elephant danced gracefully on the moon while eating ice cream.\\n\" # \ud83d\udc8e anomaly\n \"Time-traveling dinosaurs will review all deliverables before acceptance.\\n\" # \ud83d\udc8e another anomaly\n \"This agreement is governed by the laws of Norway...\\n\"\n ),\n)\n\n# Attach a document-level concept\ndoc.concepts = [\n StringConcept(\n name=\"Anomalies\", # in longer contexts, this concept is hard to capture with RAG\n description=\"Anomalies in the document\",\n add_references=True,\n reference_depth=\"sentences\",\n add_justifications=True,\n justification_depth=\"brief\",\n # see the docs for more configuration options\n )\n # add more concepts to the document, if needed\n # see the docs for available concepts: StringConcept, JsonObjectConcept, etc.\n]\n# Or use `doc.add_concepts([...])`\n\n# Define an LLM for extracting information from the document\nllm = DocumentLLM(\n model=\"openai/gpt-4o-mini\", # or another provider/LLM\n api_key=os.environ.get(\n \"CONTEXTGEM_OPENAI_API_KEY\"\n ), # your API key for the LLM provider\n # see the docs for more configuration options\n)\n\n# Extract information from the document\ndoc = llm.extract_all(doc) # or use async version `await llm.extract_all_async(doc)`\n\n# Access extracted information in the document object\nanomalies_concept = doc.concepts[0]\n# or `doc.get_concept_by_name(\"Anomalies\")`\nfor item in anomalies_concept.extracted_items:\n print(f\"Anomaly:\")\n print(f\" {item.value}\")\n print(f\"Justification:\")\n print(f\" {item.justification}\")\n print(\"Reference paragraphs:\")\n for p in item.reference_paragraphs:\n print(f\" - {p.raw_text}\")\n print(\"Reference sentences:\")\n for s in item.reference_sentences:\n print(f\" - {s.raw_text}\")\n print()\n\n```\n[](https://colab.research.google.com/github/shcherbak-ai/contextgem/blob/main/dev/notebooks/readme/quickstart_concept.ipynb)\n\n---\n\n### \ud83d\udcda More Examples\n\n**Basic usage:**\n- [Aspect Extraction from Document](https://contextgem.dev/quickstart.html#aspect-extraction-from-document)\n- [Extracting Aspect with Sub-Aspects](https://contextgem.dev/quickstart.html#extracting-aspect-with-sub-aspects)\n- [Concept Extraction from Aspect](https://contextgem.dev/quickstart.html#concept-extraction-from-aspect)\n- [Concept Extraction from Document (text)](https://contextgem.dev/quickstart.html#concept-extraction-from-document-text)\n- [Concept Extraction from Document (vision)](https://contextgem.dev/quickstart.html#concept-extraction-from-document-vision)\n- [LLM chat interface](https://contextgem.dev/quickstart.html#lightweight-llm-chat-interface)\n\n**Advanced usage:**\n- [Extracting Aspects Containing Concepts](https://contextgem.dev/advanced_usage.html#extracting-aspects-with-concepts)\n- [Extracting Aspects and Concepts from a Document](https://contextgem.dev/advanced_usage.html#extracting-aspects-and-concepts-from-a-document)\n- [Using a Multi-LLM Pipeline to Extract Data from Several Documents](https://contextgem.dev/advanced_usage.html#using-a-multi-llm-pipeline-to-extract-data-from-several-documents)\n\n\n## \ud83d\udd04 Document converters\n\nTo create a ContextGem document for LLM analysis, you can either pass raw text directly, or use built-in converters that handle various file formats.\n\n### \ud83d\udcc4 DOCX converter\n\nContextGem provides built-in converter to easily transform DOCX files into LLM-ready data.\n\n- **Comprehensive extraction of document elements**: paragraphs, headings, lists, tables, comments, footnotes, textboxes, headers/footers, links, embedded images, and inline formatting\n- **Document structure preservation** with rich metadata for improved LLM analysis\n- **Built-in converter** that directly processes Word XML\n\n```python\n# Using ContextGem's DocxConverter\n\nfrom contextgem import DocxConverter\n\nconverter = DocxConverter()\n\n# Convert a DOCX file to an LLM-ready ContextGem Document\n# from path\ndocument = converter.convert(\"path/to/document.docx\")\n# or from file object\nwith open(\"path/to/document.docx\", \"rb\") as docx_file_object:\n document = converter.convert(docx_file_object)\n\n# Perform data extraction on the resulting Document object\n# document.add_aspects(...)\n# document.add_concepts(...)\n# llm.extract_all(document)\n\n# You can also use DocxConverter instance as a standalone text extractor\ndocx_text = converter.convert_to_text_format(\n \"path/to/document.docx\",\n output_format=\"markdown\", # or \"raw\"\n)\n\n```\n\n\ud83d\udcd6 Learn more about [DOCX converter features](https://contextgem.dev/converters/docx.html) in the documentation.\n\n## \ud83c\udfaf Focused document analysis\n\nContextGem leverages LLMs' long context windows to deliver superior extraction accuracy from individual documents. Unlike RAG approaches that often [struggle with complex concepts and nuanced insights](https://www.linkedin.com/pulse/raging-contracts-pitfalls-rag-contract-review-shcherbak-ai-ptg3f), ContextGem capitalizes on continuously expanding context capacity, evolving LLM capabilities, and decreasing costs. This focused approach enables direct information extraction from complete documents, eliminating retrieval inconsistencies while optimizing for in-depth single-document analysis. While this delivers higher accuracy for individual documents, ContextGem does not currently support cross-document querying or corpus-wide retrieval - for these use cases, modern RAG systems (e.g., LlamaIndex, Haystack) remain more appropriate.\n\n\ud83d\udcd6 Read more on [how ContextGem works](https://contextgem.dev/how_it_works.html) in the documentation.\n\n## \ud83e\udd16 Supported LLMs\n\nContextGem supports both cloud-based and local LLMs through [LiteLLM](https://github.com/BerriAI/litellm) integration:\n- **Cloud LLMs**: OpenAI, Anthropic, Google, Azure OpenAI, xAI, and more\n- **Local LLMs**: Run models locally using providers like Ollama, LM Studio, etc.\n- **Model Architectures**: Works with both reasoning/CoT-capable (e.g. o4-mini) and non-reasoning models (e.g. gpt-4.1)\n- **Simple API**: Unified interface for all LLMs with easy provider switching\n\n> **\ud83d\udca1 Model Selection Note:** For reliable structured extraction, we recommend using models with performance equivalent to or exceeding `gpt-4o-mini`. Smaller models (such as 8B parameter models) may struggle with ContextGem's detailed extraction instructions. If you encounter issues with smaller models, see our [troubleshooting guide](https://contextgem.dev/optimizations/optimization_small_llm_troubleshooting.html) for potential solutions.\n\n\ud83d\udcd6 Learn more about [supported LLM providers and models](https://contextgem.dev/llms/supported_llms.html), how to [configure LLMs](https://contextgem.dev/llms/llm_config.html), and [LLM extraction methods](https://contextgem.dev/llms/llm_extraction_methods.html) in the documentation.\n\n## \u26a1 Optimizations\n\nContextGem documentation offers guidance on optimization strategies to maximize performance, minimize costs, and enhance extraction accuracy:\n\n- [Optimizing for Accuracy](https://contextgem.dev/optimizations/optimization_accuracy.html)\n- [Optimizing for Speed](https://contextgem.dev/optimizations/optimization_speed.html)\n- [Optimizing for Cost](https://contextgem.dev/optimizations/optimization_cost.html)\n- [Dealing with Long Documents](https://contextgem.dev/optimizations/optimization_long_docs.html)\n- [Choosing the Right LLM(s)](https://contextgem.dev/optimizations/optimization_choosing_llm.html)\n- [Troubleshooting Issues with Small Models](https://contextgem.dev/optimizations/optimization_small_llm_troubleshooting.html)\n\n\n## \ud83d\udcbe Serializing results\n\nContextGem allows you to save and load Document objects, pipelines, and LLM configurations with built-in serialization methods:\n\n- Save processed documents to avoid repeating expensive LLM calls\n- Transfer extraction results between systems\n- Persist pipeline and LLM configurations for later reuse\n\n\ud83d\udcd6 Learn more about [serialization options](https://contextgem.dev/serialization.html) in the documentation.\n\n## \ud83d\udcda Documentation\n\n\ud83d\udcd6 **Full documentation:** [contextgem.dev](https://contextgem.dev)\n\n\ud83d\udcc4 **Raw documentation for LLMs:** Available at [`docs/docs-raw-for-llm.txt`](https://github.com/shcherbak-ai/contextgem/blob/main/docs/docs-raw-for-llm.txt) - automatically generated, optimized for LLM ingestion.\n\n\ud83e\udd16 **AI-powered code exploration:** [DeepWiki](https://deepwiki.com/shcherbak-ai/contextgem) provides visual architecture maps and natural language Q&A for the codebase.\n\n\ud83d\udcc8 **Change history:** See the [CHANGELOG](https://github.com/shcherbak-ai/contextgem/blob/main/CHANGELOG.md) for version history, improvements, and bug fixes.\n\n## \ud83d\udcac Community\n\n\ud83d\udc1b **Found a bug or have a feature request?** [Open an issue](https://github.com/shcherbak-ai/contextgem/issues/new) on GitHub.\n\n\ud83d\udcad **Need help or want to discuss?** Start a thread in [GitHub Discussions](https://github.com/shcherbak-ai/contextgem/discussions/new/).\n\n## \ud83e\udd1d Contributing\n\nWe welcome contributions from the community - whether it's fixing a typo or developing a completely new feature! \n\n\ud83d\udccb **Get started:** Check out our [Contributor Guidelines](https://github.com/shcherbak-ai/contextgem/blob/main/CONTRIBUTING.md).\n\n## \ud83d\udd10 Security\n\nThis project is automatically scanned for security vulnerabilities using multiple security tools:\n\n- **[CodeQL](https://codeql.github.com/)** - GitHub's semantic code analysis engine for vulnerability detection\n- **[Bandit](https://github.com/PyCQA/bandit)** - Python security linter for common security issues \n- **[Snyk](https://snyk.io)** - Dependency vulnerability monitoring (used as needed)\n\n\ud83d\udee1\ufe0f **Security policy:** See [SECURITY](https://github.com/shcherbak-ai/contextgem/blob/main/SECURITY.md) file for details.\n\n## \ud83d\udc96 Acknowledgements\n\nContextGem relies on these excellent open-source packages:\n\n- [aiolimiter](https://github.com/mjpieters/aiolimiter): Powerful rate limiting for async operations\n- [Jinja2](https://github.com/pallets/jinja): Fast, expressive template engine that powers our dynamic prompt rendering\n- [litellm](https://github.com/BerriAI/litellm): Unified interface to multiple LLM providers with seamless provider switching\n- [loguru](https://github.com/Delgan/loguru): Simple yet powerful logging that enhances debugging and observability\n- [lxml](https://github.com/lxml/lxml): High-performance XML processing library for parsing DOCX document structure\n- [pydantic](https://github.com/pydantic/pydantic): The gold standard for data validation\n- [python-ulid](https://github.com/mdomke/python-ulid): Efficient ULID generation\n- [wtpsplit-lite](https://github.com/superlinear-ai/wtpsplit-lite): Lightweight version of [wtpsplit](https://github.com/segment-any-text/wtpsplit) for state-of-the-art text segmentation using wtpsplit's SaT models\n\n\n## \ud83c\udf31 Support the project\n\nContextGem is just getting started, and your support means the world to us! \n\n\u2b50 **Star the project** if you find ContextGem useful \n\ud83d\udce2 **Share it** with others who might benefit \n\ud83d\udd27 **Contribute** with feedback, issues, or code improvements\n\nYour engagement is what makes this project grow!\n\n## \ud83d\udcc4 License & Contact\n\n**License:** Apache 2.0 License - see the [LICENSE](https://github.com/shcherbak-ai/contextgem/blob/main/LICENSE) and [NOTICE](https://github.com/shcherbak-ai/contextgem/blob/main/NOTICE) files for details.\n\n**Copyright:** \u00a9 2025 [Shcherbak AI AS](https://shcherbak.ai), an AI engineering company building tools for AI/ML/NLP developers.\n\n**Connect:** [LinkedIn](https://www.linkedin.com/in/sergii-shcherbak-10068866/) or [X](https://x.com/seshch) for questions or collaboration ideas.\n\nBuilt with \u2764\ufe0f in Oslo, Norway.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Effortless LLM extraction from documents",
"version": "0.11.1",
"project_urls": {
"Bug Tracker": "https://github.com/shcherbak-ai/contextgem/issues",
"Documentation": "https://github.com/shcherbak-ai/contextgem/blob/main/README.md",
"Homepage": "https://github.com/shcherbak-ai/contextgem",
"Repository": "https://github.com/shcherbak-ai/contextgem.git"
},
"split_keywords": [
"artificial-intelligence",
" aspect-extraction",
" automated-prompting",
" concept-extraction",
" content-extraction",
" context-aware",
" contextgem",
" contract-analysis",
" contract-automation",
" contract-intelligence",
" contract-management",
" contract-parsing",
" contract-review",
" data-extraction",
" document",
" document-analysis",
" document-extraction",
" document-intelligence",
" document-parsing",
" document-pipeline",
" document-processing",
" document-qa",
" document-understanding",
" docx",
" entity-extraction",
" extraction-justifications",
" extraction-pipeline",
" fintech",
" generative-ai",
" information-extraction",
" insights-extraction",
" knowledge-extraction",
" large-language-models",
" legaltech",
" llm",
" llm-extraction",
" llm-framework",
" llm-library",
" llm-reasoning",
" low-code",
" machine-learning",
" multilingual",
" multimodal",
" neural-segmentation",
" nlp",
" no-prompt-engineering",
" prompt-free",
" question-answering",
" reference-mapping",
" semantic-analysis",
" structured-data",
" structured-data-extraction",
" text-analysis",
" text-processing",
" topic-extraction",
" unstructured-data",
" zero-shot"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0013644d35d9096095defeb2cff93deeb5205cf2cd81b785de0ab34c7c44dcf8",
"md5": "770725fbdf1467979a3e6b810ccd9e83",
"sha256": "454e7a3a42339bb6e74a6acb049bc8be3e554e2784ce05e5e40de1e0d5c706de"
},
"downloads": -1,
"filename": "contextgem-0.11.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "770725fbdf1467979a3e6b810ccd9e83",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.14,>=3.10",
"size": 187647,
"upload_time": "2025-07-11T16:24:04",
"upload_time_iso_8601": "2025-07-11T16:24:04.433392Z",
"url": "https://files.pythonhosted.org/packages/00/13/644d35d9096095defeb2cff93deeb5205cf2cd81b785de0ab34c7c44dcf8/contextgem-0.11.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e895228b0e8487a265d911ef6d06a13eb1ae2aa65ed6657697e2bd8cf1be0046",
"md5": "706fb3a28bfaea37ad10052a5604f205",
"sha256": "418af6bd558b049ed3116739986596492102d6cf40a361b28017374ccdc94c29"
},
"downloads": -1,
"filename": "contextgem-0.11.1.tar.gz",
"has_sig": false,
"md5_digest": "706fb3a28bfaea37ad10052a5604f205",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.14,>=3.10",
"size": 143465,
"upload_time": "2025-07-11T16:24:06",
"upload_time_iso_8601": "2025-07-11T16:24:06.030973Z",
"url": "https://files.pythonhosted.org/packages/e8/95/228b0e8487a265d911ef6d06a13eb1ae2aa65ed6657697e2bd8cf1be0046/contextgem-0.11.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-11 16:24:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "shcherbak-ai",
"github_project": "contextgem",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "contextgem"
}
