# Evaluatr
<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
[](https://pypi.org/project/evaluatr/)
[](LICENSE)
[](https://franckalbinet.github.io/evaluatr/)
## What is Evaluatr?
`Evaluatr` is an AI-powered system designed to automate the complex task
of mapping evaluation reports against structured frameworks. Initially
developed for [IOM (International Organization for
Migration)](https://www.iom.int) evaluation reports and the [Strategic
Results Framework (SRF)](https://srf.iom.int), it transforms a
traditionally manual, time-intensive process into an intelligent,
interpretable workflow.
The system maps evaluation reports—often 150+ pages of heterogeneous
content—against hierarchical frameworks like the SRF, which contains
objectives, enablers, and cross-cutting priorities, each with specific
outcomes, outputs, and indicators. `Evaluatr` targets the output level
for optimal granularity and connects to broader frameworks like the
[Sustainable Development Goals (SDGs)](https://sdgs.un.org) for
interoperability.
Beyond automation, `Evaluatr` prioritizes **interpretability and
human-AI collaboration**. IOM evaluators can understand the mapping
process, audit AI decisions, perform error analysis, build training
datasets over time, and create robust evaluation pipelines—ensuring the
AI system aligns with business needs through actionable, transparent,
auditable methodology.
## The Challenge We Solve
IOM evaluators possess deep expertise in mapping evaluation reports
against frameworks like the Strategic Results Framework (SRF), but face
significant operational challenges when processing reports that often
exceed 150 pages of diverse content across multiple projects and
contexts.
The core challenges are:
- **Time-intensive process**: Hundreds of staff-hours required per
comprehensive mapping exercise
- **Individual consistency**: Even expert evaluators may categorize the
same content differently across sessions
- **Cross-evaluator consistency**: Different evaluators may interpret
and map identical content to different framework outputs
- **Scale vs. thoroughness**: Growing volume of evaluation reports
creates pressure to choose between speed and comprehensive analysis
IOM needs a solution that leverages evaluators’ expertise while
addressing these operational bottlenecks—accelerating the mapping
process while maintaining the consistency and thoroughness that manual
review currently struggles to achieve at scale.
## Key Features
### 1. Document Preparation Pipeline ✅ **Available**
- **Repository Processing**: Read and preprocess IOM evaluation report
repositories with standardized outputs
- **Automated Downloads**: Batch download of evaluation documents from
diverse sources
- **OCR Processing**: Convert scanned PDFs to searchable text using
Optical Character Recognition (OCR) technology
- **Content Enrichment**: Fix OCR-corrupted headings and enrich
documents with AI-generated image descriptions for high-quality input
data
### 2. Intelligent Mapping 🚧 **In Development**
- **Agentic Framework Mapping**: Use DSPy-powered agents for traceable,
interpretable mapping of reports against evaluation frameworks like
the IOM Strategic Results Framework (SRF)
- **Command-line Interface**: Streamlined pipeline execution through
easy-to-use CLI tools
### 3. Knowledge Synthesis 📋 **Planned**
- **Knowledge Cards**: Generate structured summaries for downstream AI
tasks like proposal writing and synthesis
## ️ Installation & Setup
### From PyPI (Recommended)
``` bash
pip install evaluatr
```
### From GitHub
``` bash
pip install git+https://github.com/franckalbinet/evaluatr.git
```
### Development Installation
``` bash
# Clone the repository
git clone https://github.com/franckalbinet/evaluatr.git
cd evaluatr
# Install in development mode
pip install -e .
# Make changes in nbs/ directory, then compile:
nbdev_prepare
```
> [!NOTE]
>
> This project uses [nbdev](https://nbdev.fast.ai) for literate
> programming - see the Development section for more details.
### Environment Configuration
Create a `.env` file in your project root with your API keys:
``` bash
MISTRAL_API_KEY="your_mistral_api_key"
GEMINI_API_KEY="your_gemini_api_key"
```
**Note**: Evaluatr uses `llmlite` and `dspy` for LLM interactions,
giving you flexibility to use any compatible language model provider
beyond the examples above.
## Quick Start
### Reading an IOM Evaluation Repository
``` python
from evaluatr.readers import IOMRepoReader
# Initialize reader with your Excel file
reader = IOMRepoReader('files/test/eval_repo_iom.xlsx')
# Process the repository
evaluations = reader()
# Each evaluation is a standardized dictionary
for eval in evaluations[:3]: # Show first 3
print(f"ID: {eval['id']}")
print(f"Title: {eval['meta']['Title']}")
print(f"Documents: {len(eval['docs'])}")
print("---")
```
ID: 1a57974ab89d7280988aa6b706147ce1
Title: EX-POST EVALUATION OF THE PROJECT: NIGERIA: STRENGTHENING REINTEGRATION FOR RETURNEES (SRARP) - PHASE II
Documents: 2
---
ID: c660e774d14854e20dc74457712b50ec
Title: FINAL EVALUATION OF THE PROJECT: STRENGTHEN BORDER MANAGEMENT AND SECURITY IN MALI AND NIGER THROUGH CAPACITY BUILDING OF BORDER AUTHORITIES AND ENHANCED DIALOGUE WITH BORDER COMMUNITIES
Documents: 2
---
ID: 2cae361c6779b561af07200e3d4e4051
Title: Final Evaluation of the project "SUPPORTING THE IMPLEMENTATION OF AN E RESIDENCE PLATFORM IN CABO VERDE"
Documents: 2
---
Exporting it to JSON:
reader.to_json('processed_evaluations.json')
### Downloading evaluation documents
``` python
from evaluatr.downloaders import download_docs
from pathlib import Path
fname = 'files/test/evaluations.json'
base_dir = Path("files/test/pdf_library")
download_docs(fname, base_dir=base_dir, n_workers=0, overwrite=True)
```
(#24) ['Downloaded Internal%20Evaluation_NG20P0516_MAY_2023_FINAL_Abderrahim%20EL%20MOULAT.pdf','Downloaded RR0163_Evaluation%20Brief_MAY_%202023_Abderrahim%20EL%20MOULAT.pdf','Downloaded IB0238_Evaluation%20Brief_FEB_%202023_Abderrahim%20EL%20MOULAT.pdf','Downloaded Internal%20Evaluation_IB0238__FEB_2023_FINAL%20RE_Abderrahim%20EL%20MOULAT.pdf','Downloaded IB0053_Evaluation%20Brief_SEP_%202022_Abderrahim%20EL%20MOULAT.pdf','Downloaded Internal%20Evaluation_IB0053_OCT_2022_FINAL_Abderrahim%20EL%20MOULAT_0.pdf','Downloaded Internal%20Evaluation_NC0030_JUNE_2022_FINAL_Abderrahim%20EL%20MOULAT_0.pdf','Downloaded NC0030_Evaluation%20Brief_June%202022_Abderrahim%20EL%20MOULAT.pdf','Downloaded CD0015_Evaluation%20Brief_May%202022_Abderrahim%20EL%20MOULAT.pdf','Downloaded Projet%20CD0015_Final%20Evaluation%20Report_May_202_Abderrahim%20EL%20MOULAT.pdf','Downloaded Internal%20Evaluation_Retour%20Vert_JUL_2021_Fina_Abderrahim%20EL%20MOULAT.pdf','Downloaded NC0012_Evaluation%20Brief_JUL%202021_Abderrahim%20EL%20MOULAT.pdf','Downloaded Nigeria%20GIZ%20Internal%20Evaluation_JANUARY_2021__Abderrahim%20EL%20MOULAT.pdf','Downloaded Nigeria%20GIZ%20Project_Evaluation%20Brief_JAN%202021_Abderrahim%20EL%20MOULAT_0.pdf','Downloaded Evaluation%20Brief_ARCO_Shiraz%20JERBI.pdF','Downloaded Final%20evaluation%20report_ARCO_Shiraz%20JERBI_1.pdf','Downloaded Management%20Response%20Matrix_ARCO_Shiraz%20JERBI.pdf','Downloaded IOM%20MANAGEMENT%20RESPONSE%20MATRIX.pdf','Downloaded IOM%20Niger%20-%20MIRAA%20III%20-%20Final%20Evaluation%20Report%20%28003%29.pdf','Downloaded CE.0369%20-%20IDEE%20-%20ANNEXE%201%20-%20Rapport%20Recherche_Joanie%20DUROCHER_0.pdf'...]
### OCR Processing
Convert PDF evaluation reports into structured markdown files with
extracted images:
``` python
from evaluatr.ocr import process_single_evaluation_batch
from pathlib import Path
# Process a single evaluation report
report_path = Path("path/to/your/evaluation_report_folder")
output_dir = Path("md_library")
process_single_evaluation_batch(report_path, output_dir)
```
**Output Structure:**
md_library/
├── evaluation_id/
│ ├── page_1.md
│ ├── page_2.md
│ └── img/
│ ├── img-0.jpeg
│ └── img-1.jpeg
**Example markdown page with image reference as generated by Mistral
OCR:**
``` markdown
The evaluation followed the Organisation of Economic Cooperation and Development/Development Assistance Committee (OECD/DAC) evaluation criteria and quality standards. The evaluation ...
FIGURE 2. OECD/DAC CRITERIA FOR EVALUATIONS
Each evaluation question includes the main data collection ...
```
#### Batch OCR Processing
Process multiple evaluation reports efficiently using Mistral’s batch
OCR API:
``` python
from evaluatr.ocr import process_all_reports_batch
from pathlib import Path
# Get all evaluation report directories
reports_dir = Path("path/to/all/evaluation_reports")
report_folders = [d for d in reports_dir.iterdir() if d.is_dir()]
# Process all reports using batch OCR for efficiency
process_all_reports_batch(report_folders, md_library_path="md_library")
```
**Benefits of batch processing:** - Significantly faster than processing
PDFs individually - Cost-effective through Mistral’s batch API pricing
(expect \$0.5 per 1,000 pages) - Automatic job monitoring and result
retrieval
### Document Enrichment
While Mistral OCR excels at text extraction, it often struggles with
heading hierarchy detection, producing inconsistent markdown levels that
break document structure. Clean, properly nested headings are crucial
for agentic AI systems to retrieve content hierarchically—mimicking how
experienced evaluation analysts navigate reports by section and
subsection (as you’ll see in the upcoming `mappr` module). Additionally,
evaluation reports contain rich visual evidence through charts, graphs,
and diagrams that standard OCR simply references as image links. The
enrichr module addresses these “garbage in, garbage out” challenges by
fixing structural issues and converting visual content into searchable,
AI-readable descriptions.
``` python
from evaluatr.enrichr import fix_doc_hdgs, enrich_images
from pathlib import Path
# Fix heading hierarchy in OCR'd document
doc_path = Path("md_library/evaluation_id")
fix_doc_hdgs(doc_path)
# Enrich images with descriptive text
pages_dir = doc_path / "enhanced"
img_dir = doc_path / "img"
enrich_images(pages_dir, img_dir)
```
## Documentation
- **Full Documentation**: [GitHub
Pages](https://fr.anckalbi.net/evalstack/)
- **API Reference**: Available in the documentation
- **Examples**: See the `nbs/` directory for Jupyter notebooks
## Contributing
### Development Philosophy
Evaluatr is built using [**nbdev**](https://nbdev.fast.ai), a literate
programming framework that allows us to develop code, documentation, and
tests together in Jupyter notebooks. This approach offers several
advantages:
- **Documentation-driven development**: Code and explanations live
side-by-side, ensuring documentation stays current
- **Reproducible research**: Each module’s development process is fully
transparent and reproducible
- **Collaborative friendly**: Notebooks make it easier for domain
experts to understand and contribute to the codebase
**fastcore** provides the foundational utilities that power this
approach, offering enhanced Python functionality and seamless
integration between notebooks and production code.
### Development Setup
We welcome contributions! Here’s how you can help:
1. **Fork** the repository
``` bash
# Install development dependencies
pip install -e .
```
2. **Create** a feature branch
(`git checkout -b feature/amazing-feature`)
3. **Make** your changes in the `nbs/` directory
4. **Compile** with `nbdev_prepare`
5. **Commit** your changes (`git commit -m 'Add amazing feature'`)
6. **Push** to the branch (`git push origin feature/amazing-feature`)
7. **Open** a Pull Request
## License
This project is licensed under the MIT License - see the
[LICENSE](LICENSE) file for details.
## Dependencies
`Evaluatr` is built on these key Python packages:
- **fastcore** & **pandas** - Core data processing and utilities
- **mistralai** & **litellm** - AI/LLM integration for OCR and
enrichment
- **dspy** & **toolslm** - Structured AI programming and tool
integration
## Support
- **Issues**: [GitHub
Issues](https://github.com/franckalbinet/evalstack/issues)
- **Discussions**: [GitHub
Discussions](https://github.com/franckalbinet/evalstack/discussions)
Raw data
{
"_id": null,
"home_page": "https://github.com/franckalbinet/evaluatr",
"name": "evaluatr",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "nbdev jupyter notebook python",
"author": "Franck Albinet",
"author_email": "franckalbinet@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/a8/d3/c7b8c32a459a5e9ef917e2a7ea0f4a18ba418bd28bc8fdbcb0032a9d7240/evaluatr-0.5.0.tar.gz",
"platform": null,
"description": "# Evaluatr\n\n\n<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->\n\n[](https://pypi.org/project/evaluatr/)\n[](LICENSE)\n[](https://franckalbinet.github.io/evaluatr/)\n\n## What is Evaluatr?\n\n`Evaluatr` is an AI-powered system designed to automate the complex task\nof mapping evaluation reports against structured frameworks. Initially\ndeveloped for [IOM (International Organization for\nMigration)](https://www.iom.int) evaluation reports and the [Strategic\nResults Framework (SRF)](https://srf.iom.int), it transforms a\ntraditionally manual, time-intensive process into an intelligent,\ninterpretable workflow.\n\nThe system maps evaluation reports\u2014often 150+ pages of heterogeneous\ncontent\u2014against hierarchical frameworks like the SRF, which contains\nobjectives, enablers, and cross-cutting priorities, each with specific\noutcomes, outputs, and indicators. `Evaluatr` targets the output level\nfor optimal granularity and connects to broader frameworks like the\n[Sustainable Development Goals (SDGs)](https://sdgs.un.org) for\ninteroperability.\n\nBeyond automation, `Evaluatr` prioritizes **interpretability and\nhuman-AI collaboration**. IOM evaluators can understand the mapping\nprocess, audit AI decisions, perform error analysis, build training\ndatasets over time, and create robust evaluation pipelines\u2014ensuring the\nAI system aligns with business needs through actionable, transparent,\nauditable methodology.\n\n## The Challenge We Solve\n\nIOM evaluators possess deep expertise in mapping evaluation reports\nagainst frameworks like the Strategic Results Framework (SRF), but face\nsignificant operational challenges when processing reports that often\nexceed 150 pages of diverse content across multiple projects and\ncontexts.\n\nThe core challenges are:\n\n- **Time-intensive process**: Hundreds of staff-hours required per\n comprehensive mapping exercise\n- **Individual consistency**: Even expert evaluators may categorize the\n same content differently across sessions\n- **Cross-evaluator consistency**: Different evaluators may interpret\n and map identical content to different framework outputs\n- **Scale vs.\u00a0thoroughness**: Growing volume of evaluation reports\n creates pressure to choose between speed and comprehensive analysis\n\nIOM needs a solution that leverages evaluators\u2019 expertise while\naddressing these operational bottlenecks\u2014accelerating the mapping\nprocess while maintaining the consistency and thoroughness that manual\nreview currently struggles to achieve at scale.\n\n## Key Features\n\n### 1. Document Preparation Pipeline \u2705 **Available**\n\n- **Repository Processing**: Read and preprocess IOM evaluation report\n repositories with standardized outputs\n- **Automated Downloads**: Batch download of evaluation documents from\n diverse sources\n- **OCR Processing**: Convert scanned PDFs to searchable text using\n Optical Character Recognition (OCR) technology\n- **Content Enrichment**: Fix OCR-corrupted headings and enrich\n documents with AI-generated image descriptions for high-quality input\n data\n\n### 2. Intelligent Mapping \ud83d\udea7 **In Development**\n\n- **Agentic Framework Mapping**: Use DSPy-powered agents for traceable,\n interpretable mapping of reports against evaluation frameworks like\n the IOM Strategic Results Framework (SRF)\n- **Command-line Interface**: Streamlined pipeline execution through\n easy-to-use CLI tools\n\n### 3. Knowledge Synthesis \ud83d\udccb **Planned**\n\n- **Knowledge Cards**: Generate structured summaries for downstream AI\n tasks like proposal writing and synthesis\n\n## \ufe0f Installation & Setup\n\n### From PyPI (Recommended)\n\n``` bash\npip install evaluatr\n```\n\n### From GitHub\n\n``` bash\npip install git+https://github.com/franckalbinet/evaluatr.git\n```\n\n### Development Installation\n\n``` bash\n# Clone the repository\ngit clone https://github.com/franckalbinet/evaluatr.git\ncd evaluatr\n\n# Install in development mode\npip install -e .\n\n# Make changes in nbs/ directory, then compile:\nnbdev_prepare\n```\n\n> [!NOTE]\n>\n> This project uses [nbdev](https://nbdev.fast.ai) for literate\n> programming - see the Development section for more details.\n\n### Environment Configuration\n\nCreate a `.env` file in your project root with your API keys:\n\n``` bash\nMISTRAL_API_KEY=\"your_mistral_api_key\"\nGEMINI_API_KEY=\"your_gemini_api_key\"\n```\n\n**Note**: Evaluatr uses `llmlite` and `dspy` for LLM interactions,\ngiving you flexibility to use any compatible language model provider\nbeyond the examples above.\n\n## Quick Start\n\n### Reading an IOM Evaluation Repository\n\n``` python\nfrom evaluatr.readers import IOMRepoReader\n\n# Initialize reader with your Excel file\nreader = IOMRepoReader('files/test/eval_repo_iom.xlsx')\n\n# Process the repository\nevaluations = reader()\n\n# Each evaluation is a standardized dictionary\nfor eval in evaluations[:3]: # Show first 3\n print(f\"ID: {eval['id']}\")\n print(f\"Title: {eval['meta']['Title']}\")\n print(f\"Documents: {len(eval['docs'])}\")\n print(\"---\")\n```\n\n ID: 1a57974ab89d7280988aa6b706147ce1\n Title: EX-POST EVALUATION OF THE PROJECT: NIGERIA: STRENGTHENING REINTEGRATION FOR RETURNEES (SRARP) - PHASE II\n Documents: 2\n ---\n ID: c660e774d14854e20dc74457712b50ec\n Title: FINAL EVALUATION OF THE PROJECT: STRENGTHEN BORDER MANAGEMENT AND SECURITY IN MALI AND NIGER THROUGH CAPACITY BUILDING OF BORDER AUTHORITIES AND ENHANCED DIALOGUE WITH BORDER COMMUNITIES\n Documents: 2\n ---\n ID: 2cae361c6779b561af07200e3d4e4051\n Title: Final Evaluation of the project \"SUPPORTING THE IMPLEMENTATION OF AN E RESIDENCE PLATFORM IN CABO VERDE\"\n Documents: 2\n ---\n\nExporting it to JSON:\n\n reader.to_json('processed_evaluations.json')\n\n### Downloading evaluation documents\n\n``` python\nfrom evaluatr.downloaders import download_docs\nfrom pathlib import Path\n\nfname = 'files/test/evaluations.json'\nbase_dir = Path(\"files/test/pdf_library\")\ndownload_docs(fname, base_dir=base_dir, n_workers=0, overwrite=True)\n```\n\n (#24) ['Downloaded Internal%20Evaluation_NG20P0516_MAY_2023_FINAL_Abderrahim%20EL%20MOULAT.pdf','Downloaded RR0163_Evaluation%20Brief_MAY_%202023_Abderrahim%20EL%20MOULAT.pdf','Downloaded IB0238_Evaluation%20Brief_FEB_%202023_Abderrahim%20EL%20MOULAT.pdf','Downloaded Internal%20Evaluation_IB0238__FEB_2023_FINAL%20RE_Abderrahim%20EL%20MOULAT.pdf','Downloaded IB0053_Evaluation%20Brief_SEP_%202022_Abderrahim%20EL%20MOULAT.pdf','Downloaded Internal%20Evaluation_IB0053_OCT_2022_FINAL_Abderrahim%20EL%20MOULAT_0.pdf','Downloaded Internal%20Evaluation_NC0030_JUNE_2022_FINAL_Abderrahim%20EL%20MOULAT_0.pdf','Downloaded NC0030_Evaluation%20Brief_June%202022_Abderrahim%20EL%20MOULAT.pdf','Downloaded CD0015_Evaluation%20Brief_May%202022_Abderrahim%20EL%20MOULAT.pdf','Downloaded Projet%20CD0015_Final%20Evaluation%20Report_May_202_Abderrahim%20EL%20MOULAT.pdf','Downloaded Internal%20Evaluation_Retour%20Vert_JUL_2021_Fina_Abderrahim%20EL%20MOULAT.pdf','Downloaded NC0012_Evaluation%20Brief_JUL%202021_Abderrahim%20EL%20MOULAT.pdf','Downloaded Nigeria%20GIZ%20Internal%20Evaluation_JANUARY_2021__Abderrahim%20EL%20MOULAT.pdf','Downloaded Nigeria%20GIZ%20Project_Evaluation%20Brief_JAN%202021_Abderrahim%20EL%20MOULAT_0.pdf','Downloaded Evaluation%20Brief_ARCO_Shiraz%20JERBI.pdF','Downloaded Final%20evaluation%20report_ARCO_Shiraz%20JERBI_1.pdf','Downloaded Management%20Response%20Matrix_ARCO_Shiraz%20JERBI.pdf','Downloaded IOM%20MANAGEMENT%20RESPONSE%20MATRIX.pdf','Downloaded IOM%20Niger%20-%20MIRAA%20III%20-%20Final%20Evaluation%20Report%20%28003%29.pdf','Downloaded CE.0369%20-%20IDEE%20-%20ANNEXE%201%20-%20Rapport%20Recherche_Joanie%20DUROCHER_0.pdf'...]\n\n### OCR Processing\n\nConvert PDF evaluation reports into structured markdown files with\nextracted images:\n\n``` python\nfrom evaluatr.ocr import process_single_evaluation_batch\nfrom pathlib import Path\n\n# Process a single evaluation report\nreport_path = Path(\"path/to/your/evaluation_report_folder\")\noutput_dir = Path(\"md_library\")\n\nprocess_single_evaluation_batch(report_path, output_dir)\n```\n\n**Output Structure:**\n\n md_library/\n \u251c\u2500\u2500 evaluation_id/\n \u2502 \u251c\u2500\u2500 page_1.md\n \u2502 \u251c\u2500\u2500 page_2.md\n \u2502 \u2514\u2500\u2500 img/\n \u2502 \u251c\u2500\u2500 img-0.jpeg\n \u2502 \u2514\u2500\u2500 img-1.jpeg\n\n**Example markdown page with image reference as generated by Mistral\nOCR:**\n\n``` markdown\nThe evaluation followed the Organisation of Economic Cooperation and Development/Development Assistance Committee (OECD/DAC) evaluation criteria and quality standards. The evaluation ...\n\nFIGURE 2. OECD/DAC CRITERIA FOR EVALUATIONS\n\n\nEach evaluation question includes the main data collection ...\n```\n\n#### Batch OCR Processing\n\nProcess multiple evaluation reports efficiently using Mistral\u2019s batch\nOCR API:\n\n``` python\nfrom evaluatr.ocr import process_all_reports_batch\nfrom pathlib import Path\n\n# Get all evaluation report directories\nreports_dir = Path(\"path/to/all/evaluation_reports\")\nreport_folders = [d for d in reports_dir.iterdir() if d.is_dir()]\n\n# Process all reports using batch OCR for efficiency\nprocess_all_reports_batch(report_folders, md_library_path=\"md_library\")\n```\n\n**Benefits of batch processing:** - Significantly faster than processing\nPDFs individually - Cost-effective through Mistral\u2019s batch API pricing\n(expect \\$0.5 per 1,000 pages) - Automatic job monitoring and result\nretrieval\n\n### Document Enrichment\n\nWhile Mistral OCR excels at text extraction, it often struggles with\nheading hierarchy detection, producing inconsistent markdown levels that\nbreak document structure. Clean, properly nested headings are crucial\nfor agentic AI systems to retrieve content hierarchically\u2014mimicking how\nexperienced evaluation analysts navigate reports by section and\nsubsection (as you\u2019ll see in the upcoming `mappr` module). Additionally,\nevaluation reports contain rich visual evidence through charts, graphs,\nand diagrams that standard OCR simply references as image links. The\nenrichr module addresses these \u201cgarbage in, garbage out\u201d challenges by\nfixing structural issues and converting visual content into searchable,\nAI-readable descriptions.\n\n``` python\nfrom evaluatr.enrichr import fix_doc_hdgs, enrich_images\nfrom pathlib import Path\n\n# Fix heading hierarchy in OCR'd document\ndoc_path = Path(\"md_library/evaluation_id\")\nfix_doc_hdgs(doc_path)\n\n# Enrich images with descriptive text\npages_dir = doc_path / \"enhanced\"\nimg_dir = doc_path / \"img\"\nenrich_images(pages_dir, img_dir)\n```\n\n## Documentation\n\n- **Full Documentation**: [GitHub\n Pages](https://fr.anckalbi.net/evalstack/)\n- **API Reference**: Available in the documentation\n- **Examples**: See the `nbs/` directory for Jupyter notebooks\n\n## Contributing\n\n### Development Philosophy\n\nEvaluatr is built using [**nbdev**](https://nbdev.fast.ai), a literate\nprogramming framework that allows us to develop code, documentation, and\ntests together in Jupyter notebooks. This approach offers several\nadvantages:\n\n- **Documentation-driven development**: Code and explanations live\n side-by-side, ensuring documentation stays current\n- **Reproducible research**: Each module\u2019s development process is fully\n transparent and reproducible\n- **Collaborative friendly**: Notebooks make it easier for domain\n experts to understand and contribute to the codebase\n\n**fastcore** provides the foundational utilities that power this\napproach, offering enhanced Python functionality and seamless\nintegration between notebooks and production code.\n\n### Development Setup\n\nWe welcome contributions! Here\u2019s how you can help:\n\n1. **Fork** the repository\n\n``` bash\n# Install development dependencies\npip install -e .\n```\n\n2. **Create** a feature branch\n (`git checkout -b feature/amazing-feature`)\n3. **Make** your changes in the `nbs/` directory\n4. **Compile** with `nbdev_prepare`\n5. **Commit** your changes (`git commit -m 'Add amazing feature'`)\n6. **Push** to the branch (`git push origin feature/amazing-feature`)\n7. **Open** a Pull Request\n\n## License\n\nThis project is licensed under the MIT License - see the\n[LICENSE](LICENSE) file for details.\n\n## Dependencies\n\n`Evaluatr` is built on these key Python packages:\n\n- **fastcore** & **pandas** - Core data processing and utilities\n- **mistralai** & **litellm** - AI/LLM integration for OCR and\n enrichment\n- **dspy** & **toolslm** - Structured AI programming and tool\n integration\n\n## Support\n\n- **Issues**: [GitHub\n Issues](https://github.com/franckalbinet/evalstack/issues)\n- **Discussions**: [GitHub\n Discussions](https://github.com/franckalbinet/evalstack/discussions)\n",
"bugtrack_url": null,
"license": "Apache Software License 2.0",
"summary": "Streamline policy evaluation workflows with AI-driven analysis and evaluation framework-agnostic processing",
"version": "0.5.0",
"project_urls": {
"Homepage": "https://github.com/franckalbinet/evaluatr"
},
"split_keywords": [
"nbdev",
"jupyter",
"notebook",
"python"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "a55461b7d8ff3f29cd3ba5225dab923a353c87e79cee1d3874f4d04946e92812",
"md5": "fe736c15facbb17bb67a20f1feae411b",
"sha256": "ce3619baa3c8984daeac1709395c2c2adf75e9dc87805a902032c6d5fc850943"
},
"downloads": -1,
"filename": "evaluatr-0.5.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fe736c15facbb17bb67a20f1feae411b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 41516,
"upload_time": "2025-10-06T07:14:35",
"upload_time_iso_8601": "2025-10-06T07:14:35.021877Z",
"url": "https://files.pythonhosted.org/packages/a5/54/61b7d8ff3f29cd3ba5225dab923a353c87e79cee1d3874f4d04946e92812/evaluatr-0.5.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "a8d3c7b8c32a459a5e9ef917e2a7ea0f4a18ba418bd28bc8fdbcb0032a9d7240",
"md5": "0c0e57e97aed80e598dd87a8ccc2c784",
"sha256": "8b67ea506d01bc9762a06e7e9b900da32b1adeef0df3d2376fb68393b916377c"
},
"downloads": -1,
"filename": "evaluatr-0.5.0.tar.gz",
"has_sig": false,
"md5_digest": "0c0e57e97aed80e598dd87a8ccc2c784",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 42507,
"upload_time": "2025-10-06T07:14:37",
"upload_time_iso_8601": "2025-10-06T07:14:37.382510Z",
"url": "https://files.pythonhosted.org/packages/a8/d3/c7b8c32a459a5e9ef917e2a7ea0f4a18ba418bd28bc8fdbcb0032a9d7240/evaluatr-0.5.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-06 07:14:37",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "franckalbinet",
"github_project": "evaluatr",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "evaluatr"
}
JSON
