# IndoxMiner
[](https://badge.fury.io/py/indoxminer)
[](https://opensource.org/licenses/AGPL)
IndoxMiner is a powerful Python library that leverages Large Language Models (LLMs) to extract structured information from unstructured data sources including text, PDFs, and images. Using a flexible schema-based approach, it enables precise data extraction, validation, and transformation, making it ideal for automating document processing workflows.
## 🚀 Key Features
- **Multi-Format Support**: Extract data from text, PDFs, images, and scanned documents
- **Schema-Based Extraction**: Define custom schemas to specify exactly what data to extract
- **LLM Integration**: Seamless integration with OpenAI models for intelligent extraction
- **Validation & Type Safety**: Built-in validation rules and type-safe field definitions
- **Flexible Output**: Export to JSON, pandas DataFrames, or custom formats
- **Async Support**: Built for scalability with asynchronous processing capabilities
- **OCR Integration**: Multiple OCR engine options for image-based text extraction
- **High-Resolution Support**: Enhanced processing for high-quality PDFs
- **Error Handling**: Comprehensive error handling and validation reporting
## 📦 Installation
```bash
pip install indoxminer
```
## 🎯 Quick Start
### Basic Text Extraction
```python
from indoxminer import ExtractorSchema, Field, FieldType, ValidationRule, Extractor, OpenAi
# Initialize OpenAI extractor
llm_extractor = OpenAi(
api_key="your-api-key",
model="gpt-4-mini"
)
# Define extraction schema
schema = ExtractorSchema(
fields=[
Field(
name="product_name",
description="Product name",
field_type=FieldType.STRING,
rules=ValidationRule(min_length=2)
),
Field(
name="price",
description="Price in USD",
field_type=FieldType.FLOAT,
rules=ValidationRule(min_value=0)
)
]
)
# Create extractor and process text
extractor = Extractor(llm=llm_extractor, schema=schema)
text = """
MacBook Pro 16-inch with M2 chip
Price: $2,399.99
In stock: Yes
"""
# Extract and convert to DataFrame
result = await extractor.extract(text)
df = extractor.to_dataframe(result)
```
### PDF Processing
```python
from indoxminer import DocumentProcessor, ProcessingConfig
# Initialize processor with custom config
processor = DocumentProcessor(
files=["invoice.pdf"],
config=ProcessingConfig(
hi_res_pdf=True,
chunk_size=1000
)
)
# Process document
documents = processor.process()
# Extract structured data
schema = ExtractorSchema(
fields=[
Field(
name="bill_to",
description="Billing address",
field_type=FieldType.STRING
),
Field(
name="invoice_date",
description="Invoice date",
field_type=FieldType.DATE
),
Field(
name="total_amount",
description="Total amount in USD",
field_type=FieldType.FLOAT
)
]
)
results = await extractor.extract(documents)
```
### Image Processing with OCR
```python
# Configure OCR-enabled processor
config = ProcessingConfig(
ocr_enabled=True,
ocr_engine="easyocr", # or "tesseract", "paddle"
language="en"
)
processor = DocumentProcessor(
files=["receipt.jpg"],
config=config
)
# Process image and extract text
documents = processor.process()
```
## 🔧 Core Components
### ExtractorSchema
Defines the structure of data to be extracted:
- Field definitions
- Validation rules
- Output format specifications
```python
schema = ExtractorSchema(
fields=[...],
output_format="json"
)
```
### Field Types
Supported field types:
- `STRING`: Text data
- `INTEGER`: Whole numbers
- `FLOAT`: Decimal numbers
- `DATE`: Date values
- `BOOLEAN`: True/False values
- `LIST`: Arrays of values
- `DICT`: Nested objects
### Validation Rules
Available validation options:
- `min_length`/`max_length`: String length constraints
- `min_value`/`max_value`: Numeric bounds
- `pattern`: Regex patterns
- `required`: Required fields
- `custom`: Custom validation functions
## ⚙️ Configuration Options
### ProcessingConfig
```python
config = ProcessingConfig(
hi_res_pdf=True, # High-resolution PDF processing
ocr_enabled=True, # Enable OCR
ocr_engine="tesseract", # OCR engine selection
chunk_size=1000, # Text chunk size
language="en", # Processing language
max_threads=4 # Parallel processing threads
)
```
## 🔍 Error Handling
IndoxMiner provides detailed error reporting:
```python
results = await extractor.extract(documents)
if not results.is_valid:
for chunk_idx, errors in results.validation_errors.items():
print(f"Errors in chunk {chunk_idx}:")
for error in errors:
print(f"- {error.field}: {error.message}")
# Access valid results
valid_data = results.get_valid_results()
```
## 🤝 Contributing
We welcome contributions! To contribute:
1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Open a Pull Request
Please read our [Contributing Guidelines](CONTRIBUTING.md) for more details.
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## 🆘 Support
- **Documentation**: [Full documentation](https://indoxminer.readthedocs.io/)
- **Issues**: [GitHub Issues](https://github.com/username/indoxminer/issues)
- **Discussions**: [GitHub Discussions](https://github.com/username/indoxminer/discussions)
## 🌟 Star History
[](https://star-history.com/#username/indoxminer&Date)
Raw data
{
"_id": null,
"home_page": "https://github.com/osllmai/inDox",
"name": "indoxMiner",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "document-processing, information-extraction, llm, openai, pdf, text-processing, natural-language-processing",
"author": "nerdstudio",
"author_email": "ashkan@nematifamilyfundation.onmicrosoft.com",
"download_url": "https://files.pythonhosted.org/packages/80/91/b0c94bfa6c3838208f332dd8828b22fc30b8c037357d1220476814bc8494/indoxminer-0.0.12.tar.gz",
"platform": null,
"description": "# IndoxMiner\r\n\r\n[](https://badge.fury.io/py/indoxminer)\r\n[](https://opensource.org/licenses/AGPL)\r\n\r\nIndoxMiner is a powerful Python library that leverages Large Language Models (LLMs) to extract structured information from unstructured data sources including text, PDFs, and images. Using a flexible schema-based approach, it enables precise data extraction, validation, and transformation, making it ideal for automating document processing workflows.\r\n\r\n## \ud83d\ude80 Key Features\r\n\r\n- **Multi-Format Support**: Extract data from text, PDFs, images, and scanned documents\r\n- **Schema-Based Extraction**: Define custom schemas to specify exactly what data to extract\r\n- **LLM Integration**: Seamless integration with OpenAI models for intelligent extraction\r\n- **Validation & Type Safety**: Built-in validation rules and type-safe field definitions\r\n- **Flexible Output**: Export to JSON, pandas DataFrames, or custom formats\r\n- **Async Support**: Built for scalability with asynchronous processing capabilities\r\n- **OCR Integration**: Multiple OCR engine options for image-based text extraction\r\n- **High-Resolution Support**: Enhanced processing for high-quality PDFs\r\n- **Error Handling**: Comprehensive error handling and validation reporting\r\n\r\n## \ud83d\udce6 Installation\r\n\r\n```bash\r\npip install indoxminer\r\n```\r\n\r\n## \ud83c\udfaf Quick Start\r\n\r\n### Basic Text Extraction\r\n\r\n```python\r\nfrom indoxminer import ExtractorSchema, Field, FieldType, ValidationRule, Extractor, OpenAi\r\n\r\n# Initialize OpenAI extractor\r\nllm_extractor = OpenAi(\r\n api_key=\"your-api-key\",\r\n model=\"gpt-4-mini\"\r\n)\r\n\r\n# Define extraction schema\r\nschema = ExtractorSchema(\r\n fields=[\r\n Field(\r\n name=\"product_name\",\r\n description=\"Product name\",\r\n field_type=FieldType.STRING,\r\n rules=ValidationRule(min_length=2)\r\n ),\r\n Field(\r\n name=\"price\",\r\n description=\"Price in USD\",\r\n field_type=FieldType.FLOAT,\r\n rules=ValidationRule(min_value=0)\r\n )\r\n ]\r\n)\r\n\r\n# Create extractor and process text\r\nextractor = Extractor(llm=llm_extractor, schema=schema)\r\ntext = \"\"\"\r\nMacBook Pro 16-inch with M2 chip\r\nPrice: $2,399.99\r\nIn stock: Yes\r\n\"\"\"\r\n\r\n# Extract and convert to DataFrame\r\nresult = await extractor.extract(text)\r\ndf = extractor.to_dataframe(result)\r\n```\r\n\r\n### PDF Processing\r\n\r\n```python\r\nfrom indoxminer import DocumentProcessor, ProcessingConfig\r\n\r\n# Initialize processor with custom config\r\nprocessor = DocumentProcessor(\r\n files=[\"invoice.pdf\"],\r\n config=ProcessingConfig(\r\n hi_res_pdf=True,\r\n chunk_size=1000\r\n )\r\n)\r\n\r\n# Process document\r\ndocuments = processor.process()\r\n\r\n# Extract structured data\r\nschema = ExtractorSchema(\r\n fields=[\r\n Field(\r\n name=\"bill_to\",\r\n description=\"Billing address\",\r\n field_type=FieldType.STRING\r\n ),\r\n Field(\r\n name=\"invoice_date\",\r\n description=\"Invoice date\",\r\n field_type=FieldType.DATE\r\n ),\r\n Field(\r\n name=\"total_amount\",\r\n description=\"Total amount in USD\",\r\n field_type=FieldType.FLOAT\r\n )\r\n ]\r\n)\r\n\r\nresults = await extractor.extract(documents)\r\n```\r\n\r\n### Image Processing with OCR\r\n\r\n```python\r\n# Configure OCR-enabled processor\r\nconfig = ProcessingConfig(\r\n ocr_enabled=True,\r\n ocr_engine=\"easyocr\", # or \"tesseract\", \"paddle\"\r\n language=\"en\"\r\n)\r\n\r\nprocessor = DocumentProcessor(\r\n files=[\"receipt.jpg\"],\r\n config=config\r\n)\r\n\r\n# Process image and extract text\r\ndocuments = processor.process()\r\n```\r\n\r\n## \ud83d\udd27 Core Components\r\n\r\n### ExtractorSchema\r\n\r\nDefines the structure of data to be extracted:\r\n\r\n- Field definitions\r\n- Validation rules\r\n- Output format specifications\r\n\r\n```python\r\nschema = ExtractorSchema(\r\n fields=[...],\r\n output_format=\"json\"\r\n)\r\n```\r\n\r\n### Field Types\r\n\r\nSupported field types:\r\n\r\n- `STRING`: Text data\r\n- `INTEGER`: Whole numbers\r\n- `FLOAT`: Decimal numbers\r\n- `DATE`: Date values\r\n- `BOOLEAN`: True/False values\r\n- `LIST`: Arrays of values\r\n- `DICT`: Nested objects\r\n\r\n### Validation Rules\r\n\r\nAvailable validation options:\r\n\r\n- `min_length`/`max_length`: String length constraints\r\n- `min_value`/`max_value`: Numeric bounds\r\n- `pattern`: Regex patterns\r\n- `required`: Required fields\r\n- `custom`: Custom validation functions\r\n\r\n## \u2699\ufe0f Configuration Options\r\n\r\n### ProcessingConfig\r\n\r\n```python\r\nconfig = ProcessingConfig(\r\n hi_res_pdf=True, # High-resolution PDF processing\r\n ocr_enabled=True, # Enable OCR\r\n ocr_engine=\"tesseract\", # OCR engine selection\r\n chunk_size=1000, # Text chunk size\r\n language=\"en\", # Processing language\r\n max_threads=4 # Parallel processing threads\r\n)\r\n```\r\n\r\n## \ud83d\udd0d Error Handling\r\n\r\nIndoxMiner provides detailed error reporting:\r\n\r\n```python\r\nresults = await extractor.extract(documents)\r\n\r\nif not results.is_valid:\r\n for chunk_idx, errors in results.validation_errors.items():\r\n print(f\"Errors in chunk {chunk_idx}:\")\r\n for error in errors:\r\n print(f\"- {error.field}: {error.message}\")\r\n\r\n# Access valid results\r\nvalid_data = results.get_valid_results()\r\n```\r\n\r\n## \ud83e\udd1d Contributing\r\n\r\nWe welcome contributions! To contribute:\r\n\r\n1. Fork the repository\r\n2. Create a feature branch\r\n3. Commit your changes\r\n4. Push to the branch\r\n5. Open a Pull Request\r\n\r\nPlease read our [Contributing Guidelines](CONTRIBUTING.md) for more details.\r\n\r\n## \ud83d\udcc4 License\r\n\r\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\r\n\r\n## \ud83c\udd98 Support\r\n\r\n- **Documentation**: [Full documentation](https://indoxminer.readthedocs.io/)\r\n- **Issues**: [GitHub Issues](https://github.com/username/indoxminer/issues)\r\n- **Discussions**: [GitHub Discussions](https://github.com/username/indoxminer/discussions)\r\n\r\n## \ud83c\udf1f Star History\r\n\r\n[](https://star-history.com/#username/indoxminer&Date)\r\n",
"bugtrack_url": null,
"license": "AGPL-3.0",
"summary": "Indox Data Extraction",
"version": "0.0.12",
"project_urls": {
"Homepage": "https://github.com/osllmai/inDox"
},
"split_keywords": [
"document-processing",
" information-extraction",
" llm",
" openai",
" pdf",
" text-processing",
" natural-language-processing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6063015e48781c09868316ae7fb386c64d228c31856dcbcf71f97f916de6e4d5",
"md5": "fae9c00818e2c5166469ef217d8db2cc",
"sha256": "5b4f0fc9ce739342133b62aa8df7744e09fc49f112fac955387c9f5087eb7d2f"
},
"downloads": -1,
"filename": "indoxMiner-0.0.12-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fae9c00818e2c5166469ef217d8db2cc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 45807,
"upload_time": "2024-11-16T13:51:53",
"upload_time_iso_8601": "2024-11-16T13:51:53.473063Z",
"url": "https://files.pythonhosted.org/packages/60/63/015e48781c09868316ae7fb386c64d228c31856dcbcf71f97f916de6e4d5/indoxMiner-0.0.12-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8091b0c94bfa6c3838208f332dd8828b22fc30b8c037357d1220476814bc8494",
"md5": "1c54fccef8972fa65b588ec9f2b9d9e8",
"sha256": "d8dd268a0cf0cd3645078ce98d010aadccdd8f20d10478becaadab8368245165"
},
"downloads": -1,
"filename": "indoxminer-0.0.12.tar.gz",
"has_sig": false,
"md5_digest": "1c54fccef8972fa65b588ec9f2b9d9e8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 44969,
"upload_time": "2024-11-16T13:51:55",
"upload_time_iso_8601": "2024-11-16T13:51:55.570603Z",
"url": "https://files.pythonhosted.org/packages/80/91/b0c94bfa6c3838208f332dd8828b22fc30b8c037357d1220476814bc8494/indoxminer-0.0.12.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-16 13:51:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "osllmai",
"github_project": "inDox",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "latex2markdown",
"specs": [
[
"==",
"0.2.1"
]
]
},
{
"name": "loguru",
"specs": [
[
"==",
"0.7.2"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"2.0.0"
]
]
},
{
"name": "pandas",
"specs": [
[
"==",
"2.0.3"
]
]
},
{
"name": "protobuf",
"specs": [
[
"==",
"5.27.2"
]
]
},
{
"name": "pydantic",
"specs": [
[
"==",
"2.8.2"
]
]
},
{
"name": "PyPDF2",
"specs": [
[
"==",
"3.0.1"
]
]
},
{
"name": "python-dotenv",
"specs": [
[
"==",
"1.0.1"
]
]
},
{
"name": "Requests",
"specs": [
[
"==",
"2.32.3"
]
]
},
{
"name": "setuptools",
"specs": [
[
"==",
"69.1.1"
]
]
},
{
"name": "tenacity",
"specs": [
[
"==",
"8.2.3"
]
]
},
{
"name": "tiktoken",
"specs": [
[
"==",
"0.6.0"
]
]
},
{
"name": "tokenizers",
"specs": [
[
"==",
"0.15.2"
]
]
},
{
"name": "umap_learn",
"specs": [
[
"==",
"0.5.6"
]
]
},
{
"name": "unstructured",
"specs": [
[
"==",
"0.15.8"
]
]
},
{
"name": "nltk",
"specs": [
[
"==",
"3.9.1"
]
]
},
{
"name": "pillow_heif",
"specs": [
[
"==",
"0.18.0"
]
]
}
],
"lcname": "indoxminer"
}
JSON
