---
title: 'pydoxtools (Python Library)'
library name: pydoxtools
keywords: [ pydoxtools, AI, AI-Composition, ETL, pipelines, knowledge graphs ]
---
🎩✨📄 pydoxtools (Python Library) 🎩✨📄
================================================================================
[](https://pypi.python.org/pypi/pydoxtools)
[](https://pydoxtools.xyntopia.com)
[](https://github.com/xyntopia/pydoxtools/discussions)
[](https://github.com/xyntopia/pydoxtools/stargazers)
***
[Documentation](https://pydoxtools.xyntopia.com)
*If you have any problems or questions, please create a github [issue](https://github.com/Xyntopia/pydoxtools/issues).
So that other poeple who might want to use it can see the potential solution!*
## Summary
Pydoxtools is a library that provides a sophisticated interface for reading and
writing documents, designed to work with AI models such as GPT, LLama2, and
a variety of models on Huggingface. It offers functionalities such as:b
- Pipeline management
- Integration with AI (LLMs and more) models
- low-resource (PDF) table extraction without configuration and expensive
layout detection algorithms!
- Knowledge base extraction as a one-liner
- Document analysis and question-answering
- Support for most of todays document formats
- Vector index Creation
- Entity, address identification and more
- List and keyword extraction
- Data normalization, translation, and cleaning
The library allows for the creation of complex extraction pipelines
for batch-processing of documents by defining them as a lazily-executed graph.
## Installation
### Installing from GitHub
While pydoxtools can already be installed through pip, due to the
many updates coming in right now, it is currently recommended to use
the latest version from GitHub as follows:
pip install -U "pydoxtools[etl,inference] @ git+https://github.com/xyntopia/pydoxtools.git"
### Installing from PyPI
Pydoxtools can also be installed through pip, which will become the recommended
method once it becomes more stable:
pip install -U pydoxtools[etl,inference]
For loading additional file formats (docx, odt, epub), OCR and other options,
check out the additional > [Installation Options](#installation-options) <.
## 🚀 Teaser 🚀
Experience a new level of convenience and efficiency in handling
documents with Pydoxtools, and reimagine your data extraction pipelines!
In this teaser, we'll demonstrate how to create a document, extract
tables, and ask questions using AI models:
```python
import pydoxtools as pdx
# Create a document from various sources: file, string, bytestring, file-like object, or URL
doc = pdx.Document("https://www.raspberrypi.org/app/uploads/2012/12/quick-start-guide-v1.1.pdf")
# List available extraction functions
print(doc.x_funcs)
# get all tables from a single document:
print(doc.tables)
# Extract the first 20 tables that we can find in a directory (this might take a while,
# make sure, to only choose a small directory for testing purposes)
docs = pdx.DocumentBag("./my_directory_with_documents", forgiving_extracts=True)
print(docs.bag_apply(["tables_df", "filename"]).take(20))
# Ask a question about the documents using a local Q&A model
print(doc.answers(["how much ram does it have?"]))
# Or only ask about the documents tables (or any other extracted information):
print(doc.answers(["how much ram does it have?"], "tables"))
# To use ChatGPT for question-answering, set the API key as an environment variable:
# OPENAI_API_KEY="sk ...."
# Then, ask questions about the document using ChatGPT
print(doc.chat_answers(["What is the target group of this document?"])[0].content)
print(doc.chat_answers(["Answer if a 5-year old would be able to follow these instructions?"])[0].content)
```
With Pydoxtools, you can easily access and process your documents, perform various extractions,
and utilize AI models for more advanced analysis.
### Supported File Formats
Pydoxtools already supports loading from a large variety of different sources:
- Documents from URLs,
- pdf, html, docx, doc, odt, markdwn, rtf, epub, mediawiki
- everything supported by [pandoc](https://pandoc.org/),
- images (png, jpg, bmp, tiff etc...),
- And some "native-python" dataformats: PIL.Image.Image, <class 'dict'>, <class 'list'>
- data formats: yaml (json in progress)
- And more!
## Some Features in More Detail
### Large Pipelines
Pydoxtools' main feature is the ability to mix LLMs and other
AI models in large, composable, and customizable pipelines.
Using pipelines comes with the slight disadvantage that it
can be more challenging to add type hints to the code.
However, using pipelines decouples all parts of your code,
allowing all operators to work independently. This
makes it easy to run the pipeline in a distributed setting for big data
and enables easy, lazy evaluation. Additionally,
mixing different LLM logics together becomes much easier.
Check out how Pydoxtools' `Document` class mixes pipelines for each individual file type:
- Every node in an ellipse can be called as an attribute of the document-analysis pipeline.
- Every execution path is lazily executed throughout the entire graph.
- Every node is cached by default (but can be turned off).
- Every piece of this pipeline can be replaced by a customized version.
As an example, consider this pipeline for *.png images from the repository,
which includes OCR, keyword extraction, vectorization, and more:
Pipelines can be mixed, partially overwritten, and extended, giving you a lot of possibilities
to extend and adapt the functionality for your specific use case.
To learn more about Pydoxtools' large pipelines feature, please refer to
the [documentation](http://pydoxtools.xyntopia.com/reference/#pydoxtools.document.Document).
#### Pipeline Configuration
Pipelines can be configured. For example the local model used for
question answering can be selected like this:
```python
doc = Document(fobj="./data/PFR-PR23_BAT-110__V1.00_.pdf")
.config(qam_model_id='bert-large-uncased-whole-word-masking-finetuned-squad')
```
where "qam_model_id" can be any model from huggingface for question answering.
You can get a list of configuration options like this:
```python
doc.configuration
# >> will give you something like this:
# {'spacy_model_size': 'md',
# 'spacy_model': 'auto',
# 'use_clean_text_for_spacy': True,
# 'coreference_method': 'fast',
# 'graph_debug_context_size': 0,
# 'vectorizer_model': 'sentence-transformers/all-MiniLM-L6-v2',
# 'vectorizer_only_tokenizer': False,
# 'vectorizer_overlap_ratio': 0.1,
# 'min_size_text_segment': 256,
# 'max_size_text_segment': 512,
# 'text_segment_overlap': 0.3,
# 'max_text_segment_num': 100,
# 'top_k_text_rank_keywords': 5,
# 'top_k_text_rank_sentences': 5,
# 'summarizer_model': 'sshleifer/distilbart-cnn-12-6',
# 'summarizer_token_overlap': 50,
# 'summarizer_max_text_len': 200,
# 'qam_model_id': 'deepset/minilm-uncased-squad2',
# 'chat_model_id': 'gpt-3.5-turbo',
# 'image_dpi': 216,
# 'ocr_lang': 'auto',
# 'ocr_on': True}
```
FOr more information check the documenation:
### PDF Table Extraction Algorithms
The library features its own sophisticated Table extraction algorithm which is benchmarked
against a large pdf table dataset. In contrast to how most "classical" table extraction
algorithms work, it doesn't require:
- extensive configuration
- no expensive deep neural networks for table area recognition which need a GPU and
a lot of memory/CPU requirements
This makes it possible to run analysis on PDF files with pydoxtools on CPU with
very limited resources!
### TODO: Describe more of the features here...
## Use Cases
- create new documents from unstructured information
- analyze documents using any model from huggingface
- analyze documents using a custom model
- download a pdf from URL
- generate document keywords
- extract tables
- download document from URL "manually" and then feed to document
- extract addresses
- extract addresses and use this information for the qam
- ingest documents into a vector db
## Installation Options
If you simply want to get going, you can install the following libraries on your system
which will do evrything for you:
```bash
sudo apt-get install tesseract-ocr tesseract-ocr-deu tesseract-ocr-fra tesseract-ocr-eng tesseract-ocr-spa \
poppler-utils graphviz graphviz-dev \
sudo apt-get install pandoc
# OR (for getting the newest version with all features)
# cd /tmp
# wget https://github.com/jgm/pandoc/releases/download/2.19.2/pandoc-2.19.2-1-amd64.deb
# dpkg -i pandoc-2.19.2-1-amd64.deb
```
Below are some explanation what the different
### Supporting \*.docx, \*.odt, \*.epub
In order to be able to load docx, odt and rtf files, you have to install pandoc.
Right now, the python pandoc library does not work with pandoc version > 3.0.0. It
is therefore recommended to install a version from here for your OS:
https://github.com/jgm/pandoc/releases/tag/2.19.2
### Image OCR Support
Pydoxtools can automatically analyze images as well, makin use of
[OCR](https://en.wikipedia.org/wiki/Optical_character_recognition).
In order to be able to use this, install tesseract on your system:
Under linux this looks like the following:
```bash
sudo apt-get update && sudo apt-get tesseract-ocr
# install tesseract languages
# Display a list of all Tesseract language packs:
# apt-cache search tesseract-ocr
# install all languages:
# sudo apt install tesseract-ocr-*
# install only german, french, english, spanish language packs
sudo apt install tesseract-ocr-deu tesseract-ocr-fra tesseract-ocr-eng tesseract-ocr-spa
```
### pdf image rendering
For pdf rendering, Pydoxtools makes use of a library "poppler" which
needs to be installed on your system. Under linux, this looks like the following:
```bash
sudo apt-get install poppler-utils
```
### Graphviz
For visualizing the document logic, you need to install graphviz on your system.
Under linux, this looks like the following:
```bash
sudo apt-get install graphviz graphviz-dev
```
## Development
--> see [](DEVELOPMENT)
## License
This project is licensed under the terms of [MIT](LICENSE) license.
You can check the compatibility using the following tool in a venv environment in a production
setting:
pip install pip-licenses
pip-licenses | grep -Ev 'MIT License|BSD License|Apache Software License|Python Software Foundation License|Apache 2.0|MIT|Apache License 2.0|hnswlib|Pillow|new BSD|BSD'
### Dependencies
Here is a list of Libraries, that this project is based on:
[list](poetry.lock)
### Changelog
[changelog](CHANGELOG.md)
Raw data
{
"_id": null,
"home_page": "https://pydoxtools.xyntopia.com",
"name": "pydoxtools",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "AI,document-analysis,LLM,NLP,ML",
"author": "thomas meschede",
"author_email": "yeusblender@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/1f/b3/75ed1679d548839411a486ffde3a066771010f3ae1b4fa49fa9f0402887a/pydoxtools-0.8.0.tar.gz",
"platform": null,
"description": "---\ntitle: 'pydoxtools (Python Library)'\nlibrary name: pydoxtools\nkeywords: [ pydoxtools, AI, AI-Composition, ETL, pipelines, knowledge graphs ]\n---\n\n\n\ud83c\udfa9\u2728\ud83d\udcc4 pydoxtools (Python Library) \ud83c\udfa9\u2728\ud83d\udcc4\n================================================================================\n\n\n[](https://pypi.python.org/pypi/pydoxtools)\n[](https://pydoxtools.xyntopia.com)\n[](https://github.com/xyntopia/pydoxtools/discussions)\n[](https://github.com/xyntopia/pydoxtools/stargazers)\n\n***\n\n[Documentation](https://pydoxtools.xyntopia.com)\n\n*If you have any problems or questions, please create a github [issue](https://github.com/Xyntopia/pydoxtools/issues).\nSo that other poeple who might want to use it can see the potential solution!*\n\n## Summary\n\nPydoxtools is a library that provides a sophisticated interface for reading and\nwriting documents, designed to work with AI models such as GPT, LLama2, and\na variety of models on Huggingface. It offers functionalities such as:b\n\n- Pipeline management\n- Integration with AI (LLMs and more) models\n- low-resource (PDF) table extraction without configuration and expensive\n layout detection algorithms!\n- Knowledge base extraction as a one-liner\n- Document analysis and question-answering\n- Support for most of todays document formats\n- Vector index Creation\n- Entity, address identification and more\n- List and keyword extraction\n- Data normalization, translation, and cleaning\n\nThe library allows for the creation of complex extraction pipelines\nfor batch-processing of documents by defining them as a lazily-executed graph.\n\n## Installation\n\n### Installing from GitHub\n\nWhile pydoxtools can already be installed through pip, due to the\nmany updates coming in right now, it is currently recommended to use\nthe latest version from GitHub as follows:\n\n pip install -U \"pydoxtools[etl,inference] @ git+https://github.com/xyntopia/pydoxtools.git\"\n\n### Installing from PyPI\n\nPydoxtools can also be installed through pip, which will become the recommended\nmethod once it becomes more stable:\n\n pip install -U pydoxtools[etl,inference]\n\nFor loading additional file formats (docx, odt, epub), OCR and other options,\ncheck out the additional > [Installation Options](#installation-options) <.\n\n## \ud83d\ude80 Teaser \ud83d\ude80\n\nExperience a new level of convenience and efficiency in handling\ndocuments with Pydoxtools, and reimagine your data extraction pipelines!\n\nIn this teaser, we'll demonstrate how to create a document, extract\ntables, and ask questions using AI models:\n\n```python\nimport pydoxtools as pdx\n\n# Create a document from various sources: file, string, bytestring, file-like object, or URL\ndoc = pdx.Document(\"https://www.raspberrypi.org/app/uploads/2012/12/quick-start-guide-v1.1.pdf\")\n\n# List available extraction functions\nprint(doc.x_funcs)\n\n# get all tables from a single document:\nprint(doc.tables)\n\n# Extract the first 20 tables that we can find in a directory (this might take a while,\n# make sure, to only choose a small directory for testing purposes)\ndocs = pdx.DocumentBag(\"./my_directory_with_documents\", forgiving_extracts=True)\nprint(docs.bag_apply([\"tables_df\", \"filename\"]).take(20))\n\n# Ask a question about the documents using a local Q&A model\nprint(doc.answers([\"how much ram does it have?\"]))\n# Or only ask about the documents tables (or any other extracted information):\nprint(doc.answers([\"how much ram does it have?\"], \"tables\"))\n\n# To use ChatGPT for question-answering, set the API key as an environment variable:\n# OPENAI_API_KEY=\"sk ....\"\n# Then, ask questions about the document using ChatGPT\nprint(doc.chat_answers([\"What is the target group of this document?\"])[0].content)\nprint(doc.chat_answers([\"Answer if a 5-year old would be able to follow these instructions?\"])[0].content)\n```\n\nWith Pydoxtools, you can easily access and process your documents, perform various extractions,\nand utilize AI models for more advanced analysis.\n\n### Supported File Formats\n\nPydoxtools already supports loading from a large variety of different sources:\n\n- Documents from URLs,\n- pdf, html, docx, doc, odt, markdwn, rtf, epub, mediawiki\n- everything supported by [pandoc](https://pandoc.org/),\n- images (png, jpg, bmp, tiff etc...),\n- And some \"native-python\" dataformats: PIL.Image.Image, <class 'dict'>, <class 'list'>\n- data formats: yaml (json in progress)\n- And more!\n\n## Some Features in More Detail\n\n### Large Pipelines\n\nPydoxtools' main feature is the ability to mix LLMs and other\nAI models in large, composable, and customizable pipelines.\nUsing pipelines comes with the slight disadvantage that it\ncan be more challenging to add type hints to the code.\nHowever, using pipelines decouples all parts of your code,\nallowing all operators to work independently. This\nmakes it easy to run the pipeline in a distributed setting for big data\nand enables easy, lazy evaluation. Additionally,\nmixing different LLM logics together becomes much easier.\n\nCheck out how Pydoxtools' `Document` class mixes pipelines for each individual file type:\n\n- Every node in an ellipse can be called as an attribute of the document-analysis pipeline.\n- Every execution path is lazily executed throughout the entire graph.\n- Every node is cached by default (but can be turned off).\n- Every piece of this pipeline can be replaced by a customized version.\n\nAs an example, consider this pipeline for *.png images from the repository,\nwhich includes OCR, keyword extraction, vectorization, and more:\n\n\n\nPipelines can be mixed, partially overwritten, and extended, giving you a lot of possibilities\nto extend and adapt the functionality for your specific use case.\n\nTo learn more about Pydoxtools' large pipelines feature, please refer to\nthe [documentation](http://pydoxtools.xyntopia.com/reference/#pydoxtools.document.Document).\n\n#### Pipeline Configuration\n\nPipelines can be configured. For example the local model used for\nquestion answering can be selected like this:\n\n```python\ndoc = Document(fobj=\"./data/PFR-PR23_BAT-110__V1.00_.pdf\")\n.config(qam_model_id='bert-large-uncased-whole-word-masking-finetuned-squad')\n```\n\nwhere \"qam_model_id\" can be any model from huggingface for question answering.\n\nYou can get a list of configuration options like this:\n\n```python\ndoc.configuration\n\n# >> will give you something like this:\n# {'spacy_model_size': 'md',\n# 'spacy_model': 'auto',\n# 'use_clean_text_for_spacy': True,\n# 'coreference_method': 'fast',\n# 'graph_debug_context_size': 0,\n# 'vectorizer_model': 'sentence-transformers/all-MiniLM-L6-v2',\n# 'vectorizer_only_tokenizer': False,\n# 'vectorizer_overlap_ratio': 0.1,\n# 'min_size_text_segment': 256,\n# 'max_size_text_segment': 512,\n# 'text_segment_overlap': 0.3,\n# 'max_text_segment_num': 100,\n# 'top_k_text_rank_keywords': 5,\n# 'top_k_text_rank_sentences': 5,\n# 'summarizer_model': 'sshleifer/distilbart-cnn-12-6',\n# 'summarizer_token_overlap': 50,\n# 'summarizer_max_text_len': 200,\n# 'qam_model_id': 'deepset/minilm-uncased-squad2',\n# 'chat_model_id': 'gpt-3.5-turbo',\n# 'image_dpi': 216,\n# 'ocr_lang': 'auto',\n# 'ocr_on': True}\n```\n\nFOr more information check the documenation:\n\n### PDF Table Extraction Algorithms\n\nThe library features its own sophisticated Table extraction algorithm which is benchmarked\nagainst a large pdf table dataset. In contrast to how most \"classical\" table extraction\nalgorithms work, it doesn't require:\n\n- extensive configuration\n- no expensive deep neural networks for table area recognition which need a GPU and\n a lot of memory/CPU requirements\n\nThis makes it possible to run analysis on PDF files with pydoxtools on CPU with\nvery limited resources!\n\n### TODO: Describe more of the features here...\n\n## Use Cases\n\n- create new documents from unstructured information\n- analyze documents using any model from huggingface\n- analyze documents using a custom model\n- download a pdf from URL\n- generate document keywords\n- extract tables\n- download document from URL \"manually\" and then feed to document\n- extract addresses\n- extract addresses and use this information for the qam\n- ingest documents into a vector db\n\n## Installation Options\n\nIf you simply want to get going, you can install the following libraries on your system\nwhich will do evrything for you:\n\n```bash\nsudo apt-get install tesseract-ocr tesseract-ocr-deu tesseract-ocr-fra tesseract-ocr-eng tesseract-ocr-spa \\\n poppler-utils graphviz graphviz-dev \\\nsudo apt-get install pandoc\n# OR (for getting the newest version with all features)\n# cd /tmp\n# wget https://github.com/jgm/pandoc/releases/download/2.19.2/pandoc-2.19.2-1-amd64.deb\n# dpkg -i pandoc-2.19.2-1-amd64.deb\n```\n\nBelow are some explanation what the different\n\n### Supporting \\*.docx, \\*.odt, \\*.epub\n\nIn order to be able to load docx, odt and rtf files, you have to install pandoc.\nRight now, the python pandoc library does not work with pandoc version > 3.0.0. It\nis therefore recommended to install a version from here for your OS:\n\nhttps://github.com/jgm/pandoc/releases/tag/2.19.2\n\n### Image OCR Support\n\nPydoxtools can automatically analyze images as well, makin use of\n[OCR](https://en.wikipedia.org/wiki/Optical_character_recognition).\nIn order to be able to use this, install tesseract on your system:\n\nUnder linux this looks like the following:\n\n```bash\nsudo apt-get update && sudo apt-get tesseract-ocr\n# install tesseract languages \n# Display a list of all Tesseract language packs:\n# apt-cache search tesseract-ocr\n# install all languages:\n# sudo apt install tesseract-ocr-*\n# install only german, french, english, spanish language packs\nsudo apt install tesseract-ocr-deu tesseract-ocr-fra tesseract-ocr-eng tesseract-ocr-spa\n```\n\n### pdf image rendering\n\nFor pdf rendering, Pydoxtools makes use of a library \"poppler\" which\nneeds to be installed on your system. Under linux, this looks like the following:\n\n```bash\nsudo apt-get install poppler-utils\n```\n\n### Graphviz\n\nFor visualizing the document logic, you need to install graphviz on your system.\nUnder linux, this looks like the following:\n\n```bash\nsudo apt-get install graphviz graphviz-dev\n```\n\n## Development\n\n--> see [](DEVELOPMENT)\n\n## License\n\nThis project is licensed under the terms of [MIT](LICENSE) license.\n\nYou can check the compatibility using the following tool in a venv environment in a production\nsetting:\n\n pip install pip-licenses\n pip-licenses | grep -Ev 'MIT License|BSD License|Apache Software License|Python Software Foundation License|Apache 2.0|MIT|Apache License 2.0|hnswlib|Pillow|new BSD|BSD'\n\n### Dependencies\n\nHere is a list of Libraries, that this project is based on:\n\n[list](poetry.lock)\n\n### Changelog\n\n[changelog](CHANGELOG.md)",
"bugtrack_url": null,
"license": "MIT",
"summary": "This library contains a set of tools in order to extract and synthesize structured information from documents",
"version": "0.8.0",
"project_urls": {
"Documentation": "https://pydoxtools.xyntopia.com",
"Homepage": "https://pydoxtools.xyntopia.com",
"Repository": "https://github.com/xyntopia/pydoxtools"
},
"split_keywords": [
"ai",
"document-analysis",
"llm",
"nlp",
"ml"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "174b9fbe89bc702b0445eb7c4aba3ba4e5762339ff691bd2b3e6d37ce0a25dd6",
"md5": "a79e8cd0fe4f3e3e066c5e308d88f88e",
"sha256": "91a9a3f9f095d7c2efcd91d8a403acefbd488d6bf17b1f04e79c6aa2bfb56f7c"
},
"downloads": -1,
"filename": "pydoxtools-0.8.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a79e8cd0fe4f3e3e066c5e308d88f88e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 155867,
"upload_time": "2024-01-28T21:47:02",
"upload_time_iso_8601": "2024-01-28T21:47:02.894828Z",
"url": "https://files.pythonhosted.org/packages/17/4b/9fbe89bc702b0445eb7c4aba3ba4e5762339ff691bd2b3e6d37ce0a25dd6/pydoxtools-0.8.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1fb375ed1679d548839411a486ffde3a066771010f3ae1b4fa49fa9f0402887a",
"md5": "76287f0ba389fc7ac90eff829f214b90",
"sha256": "7ce93d16a1e2ebbaa79d971d6980374925446a8264b8e803a7510684ef2a23c9"
},
"downloads": -1,
"filename": "pydoxtools-0.8.0.tar.gz",
"has_sig": false,
"md5_digest": "76287f0ba389fc7ac90eff829f214b90",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 145034,
"upload_time": "2024-01-28T21:47:04",
"upload_time_iso_8601": "2024-01-28T21:47:04.634122Z",
"url": "https://files.pythonhosted.org/packages/1f/b3/75ed1679d548839411a486ffde3a066771010f3ae1b4fa49fa9f0402887a/pydoxtools-0.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-28 21:47:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "xyntopia",
"github_project": "pydoxtools",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pydoxtools"
}
JSON
