<p align="center">
<a href="https://github.com/ds4sd/docling">
<img loading="lazy" alt="Docling" src="https://github.com/DS4SD/docling/raw/main/logo.png" width="150" />
</a>
</p>
# Docling
[](https://arxiv.org/abs/2408.09869)
[](https://pypi.org/project/docling/)
[](https://python-poetry.org/)
[](https://github.com/psf/black)
[](https://pycqa.github.io/isort/)
[](https://pydantic.dev)
[](https://github.com/pre-commit/pre-commit)
[](https://opensource.org/licenses/MIT)
Docling bundles PDF document conversion to JSON and Markdown in an easy, self-contained package.
## Features
* ⚡ Converts any PDF document to JSON or Markdown format, stable and lightning fast
* 📑 Understands detailed page layout, reading order and recovers table structures
* 📝 Extracts metadata from the document, such as title, authors, references and language
* 🔍 Optionally applies OCR (use with scanned PDFs)
* 🤖 Integrates easily with LLM app / RAG frameworks like 🦙 LlamaIndex and 🦜🔗 LangChain
## Installation
To use Docling, simply install `docling` from your package manager, e.g. pip:
```bash
pip install docling
```
> [!NOTE]
> Works on macOS and Linux environments. Windows platforms are currently not tested.
### Use alternative PyTorch distributions
The Docling models depend on the [PyTorch](https://pytorch.org/) library.
Depending on your architecture, you might want to use a different distribution of `torch`.
For example, you might want support for different accelerator or for a cpu-only version.
All the different ways for installing `torch` are listed on their website <https://pytorch.org/>.
One common situation is the installation on Linux systems with cpu-only support.
In this case, we suggest the installation of Docling with the following options
```bash
# Example for installing on the Linux cpu-only version
pip install docling --extra-index-url https://download.pytorch.org/whl/cpu
```
### Development setup
To develop for Docling, you need Python 3.10 / 3.11 / 3.12 and Poetry. You can then install from your local clone's root dir:
```bash
poetry install --all-extras
```
## Usage
### Convert a single document
To convert invidual PDF documents, use `convert_single()`, for example:
```python
from docling.document_converter import DocumentConverter
source = "https://arxiv.org/pdf/2408.09869" # PDF path or URL
converter = DocumentConverter()
result = converter.convert_single(source)
print(result.render_as_markdown()) # output: "## Docling Technical Report[...]"
```
### Convert a batch of documents
For an example of batch-converting documents, see [batch_convert.py](https://github.com/DS4SD/docling/blob/main/examples/batch_convert.py).
From a local repo clone, you can run it with:
```
python examples/batch_convert.py
```
The output of the above command will be written to `./scratch`.
### Adjust pipeline features
The example file [custom_convert.py](https://github.com/DS4SD/docling/blob/main/examples/custom_convert.py) contains multiple ways
one can adjust the conversion pipeline and features.
#### Control pipeline options
You can control if table structure recognition or OCR should be performed by arguments passed to `DocumentConverter`:
```python
doc_converter = DocumentConverter(
artifacts_path=artifacts_path,
pipeline_options=PipelineOptions(
do_table_structure=False, # controls if table structure is recovered
do_ocr=True, # controls if OCR is applied (ignores programmatic content)
),
)
```
#### Control table extraction options
You can control if table structure recognition should map the recognized structure back to PDF cells (default) or use text cells from the structure prediction itself.
This can improve output quality if you find that multiple columns in extracted tables are erroneously merged into one.
```python
pipeline_options = PipelineOptions(do_table_structure=True)
pipeline_options.table_structure_options.do_cell_matching = False # uses text cells predicted from table structure model
doc_converter = DocumentConverter(
artifacts_path=artifacts_path,
pipeline_options=pipeline_options,
)
```
### Impose limits on the document size
You can limit the file size and number of pages which should be allowed to process per document:
```python
conv_input = DocumentConversionInput.from_paths(
paths=[Path("./test/data/2206.01062.pdf")],
limits=DocumentLimits(max_num_pages=100, max_file_size=20971520)
)
```
### Convert from binary PDF streams
You can convert PDFs from a binary stream instead of from the filesystem as follows:
```python
buf = BytesIO(your_binary_stream)
docs = [DocumentStream(filename="my_doc.pdf", stream=buf)]
conv_input = DocumentConversionInput.from_streams(docs)
results = doc_converter.convert(conv_input)
```
### Limit resource usage
You can limit the CPU threads used by Docling by setting the environment variable `OMP_NUM_THREADS` accordingly. The default setting is using 4 CPU threads.
### RAG
Check out the following examples showcasing RAG using Docling with standard LLM application frameworks:
- [Basic RAG pipeline with 🦙 LlamaIndex](https://github.com/DS4SD/docling/tree/main/examples/rag_llamaindex.ipynb)
- [Basic RAG pipeline with 🦜🔗 LangChain](https://github.com/DS4SD/docling/tree/main/examples/rag_langchain.ipynb)
## Technical report
For more details on Docling's inner workings, check out the [Docling Technical Report](https://arxiv.org/abs/2408.09869).
## Contributing
Please read [Contributing to Docling](https://github.com/DS4SD/docling/blob/main/CONTRIBUTING.md) for details.
## References
If you use Docling in your projects, please consider citing the following:
```bib
@techreport{Docling,
author = {Deep Search Team},
month = {8},
title = {Docling Technical Report},
url = {https://arxiv.org/abs/2408.09869},
eprint = {2408.09869},
doi = {10.48550/arXiv.2408.09869},
version = {1.0.0},
year = {2024}
}
```
## License
The Docling codebase is under MIT license.
For individual model usage, please refer to the model licenses found in the original packages.
Raw data
{
"_id": null,
"home_page": "https://github.com/DS4SD/docling",
"name": "docling",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "docling, convert, document, pdf, layout model, segmentation, table structure, table former",
"author": "Christoph Auer",
"author_email": "cau@zurich.ibm.com",
"download_url": "https://files.pythonhosted.org/packages/c3/4f/39615d5843ffdd1a5b26dbc6e759fd3efe272c3cba31506644571ab67913/docling-1.12.1.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <a href=\"https://github.com/ds4sd/docling\">\n <img loading=\"lazy\" alt=\"Docling\" src=\"https://github.com/DS4SD/docling/raw/main/logo.png\" width=\"150\" />\n </a>\n</p>\n\n# Docling\n\n[](https://arxiv.org/abs/2408.09869)\n[](https://pypi.org/project/docling/)\n\n[](https://python-poetry.org/)\n[](https://github.com/psf/black)\n[](https://pycqa.github.io/isort/)\n[](https://pydantic.dev)\n[](https://github.com/pre-commit/pre-commit)\n[](https://opensource.org/licenses/MIT)\n\nDocling bundles PDF document conversion to JSON and Markdown in an easy, self-contained package.\n\n## Features\n* \u26a1 Converts any PDF document to JSON or Markdown format, stable and lightning fast\n* \ud83d\udcd1 Understands detailed page layout, reading order and recovers table structures\n* \ud83d\udcdd Extracts metadata from the document, such as title, authors, references and language\n* \ud83d\udd0d Optionally applies OCR (use with scanned PDFs)\n* \ud83e\udd16 Integrates easily with LLM app / RAG frameworks like \ud83e\udd99 LlamaIndex and \ud83e\udd9c\ud83d\udd17 LangChain\n\n## Installation\n\nTo use Docling, simply install `docling` from your package manager, e.g. pip:\n```bash\npip install docling\n```\n\n> [!NOTE]\n> Works on macOS and Linux environments. Windows platforms are currently not tested.\n\n\n### Use alternative PyTorch distributions\n\nThe Docling models depend on the [PyTorch](https://pytorch.org/) library.\nDepending on your architecture, you might want to use a different distribution of `torch`.\nFor example, you might want support for different accelerator or for a cpu-only version.\nAll the different ways for installing `torch` are listed on their website <https://pytorch.org/>.\n\nOne common situation is the installation on Linux systems with cpu-only support.\nIn this case, we suggest the installation of Docling with the following options\n\n```bash\n# Example for installing on the Linux cpu-only version\npip install docling --extra-index-url https://download.pytorch.org/whl/cpu\n```\n\n\n### Development setup\n\nTo develop for Docling, you need Python 3.10 / 3.11 / 3.12 and Poetry. You can then install from your local clone's root dir:\n```bash\npoetry install --all-extras\n```\n\n## Usage\n\n### Convert a single document\n\nTo convert invidual PDF documents, use `convert_single()`, for example:\n```python\nfrom docling.document_converter import DocumentConverter\n\nsource = \"https://arxiv.org/pdf/2408.09869\" # PDF path or URL\nconverter = DocumentConverter()\nresult = converter.convert_single(source)\nprint(result.render_as_markdown()) # output: \"## Docling Technical Report[...]\"\n```\n\n### Convert a batch of documents\n\nFor an example of batch-converting documents, see [batch_convert.py](https://github.com/DS4SD/docling/blob/main/examples/batch_convert.py).\n\nFrom a local repo clone, you can run it with:\n\n```\npython examples/batch_convert.py\n```\nThe output of the above command will be written to `./scratch`.\n\n### Adjust pipeline features\n\nThe example file [custom_convert.py](https://github.com/DS4SD/docling/blob/main/examples/custom_convert.py) contains multiple ways\none can adjust the conversion pipeline and features.\n\n\n#### Control pipeline options\n\nYou can control if table structure recognition or OCR should be performed by arguments passed to `DocumentConverter`:\n```python\ndoc_converter = DocumentConverter(\n artifacts_path=artifacts_path,\n pipeline_options=PipelineOptions(\n do_table_structure=False, # controls if table structure is recovered\n do_ocr=True, # controls if OCR is applied (ignores programmatic content)\n ),\n)\n```\n\n#### Control table extraction options\n\nYou can control if table structure recognition should map the recognized structure back to PDF cells (default) or use text cells from the structure prediction itself.\nThis can improve output quality if you find that multiple columns in extracted tables are erroneously merged into one.\n\n\n```python\npipeline_options = PipelineOptions(do_table_structure=True)\npipeline_options.table_structure_options.do_cell_matching = False # uses text cells predicted from table structure model\n\ndoc_converter = DocumentConverter(\n artifacts_path=artifacts_path,\n pipeline_options=pipeline_options,\n)\n```\n\n### Impose limits on the document size\n\nYou can limit the file size and number of pages which should be allowed to process per document:\n```python\nconv_input = DocumentConversionInput.from_paths(\n paths=[Path(\"./test/data/2206.01062.pdf\")],\n limits=DocumentLimits(max_num_pages=100, max_file_size=20971520)\n)\n```\n\n### Convert from binary PDF streams\n\nYou can convert PDFs from a binary stream instead of from the filesystem as follows:\n```python\nbuf = BytesIO(your_binary_stream)\ndocs = [DocumentStream(filename=\"my_doc.pdf\", stream=buf)]\nconv_input = DocumentConversionInput.from_streams(docs)\nresults = doc_converter.convert(conv_input)\n```\n### Limit resource usage\n\nYou can limit the CPU threads used by Docling by setting the environment variable `OMP_NUM_THREADS` accordingly. The default setting is using 4 CPU threads.\n\n### RAG\nCheck out the following examples showcasing RAG using Docling with standard LLM application frameworks:\n- [Basic RAG pipeline with \ud83e\udd99 LlamaIndex](https://github.com/DS4SD/docling/tree/main/examples/rag_llamaindex.ipynb)\n- [Basic RAG pipeline with \ud83e\udd9c\ud83d\udd17 LangChain](https://github.com/DS4SD/docling/tree/main/examples/rag_langchain.ipynb)\n\n## Technical report\n\nFor more details on Docling's inner workings, check out the [Docling Technical Report](https://arxiv.org/abs/2408.09869).\n\n## Contributing\n\nPlease read [Contributing to Docling](https://github.com/DS4SD/docling/blob/main/CONTRIBUTING.md) for details.\n\n\n## References\n\nIf you use Docling in your projects, please consider citing the following:\n\n```bib\n@techreport{Docling,\n author = {Deep Search Team},\n month = {8},\n title = {Docling Technical Report},\n url = {https://arxiv.org/abs/2408.09869},\n eprint = {2408.09869},\n doi = {10.48550/arXiv.2408.09869},\n version = {1.0.0},\n year = {2024}\n}\n```\n\n## License\n\nThe Docling codebase is under MIT license.\nFor individual model usage, please refer to the model licenses found in the original packages.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Docling PDF conversion package",
"version": "1.12.1",
"project_urls": {
"Homepage": "https://github.com/DS4SD/docling",
"Repository": "https://github.com/DS4SD/docling"
},
"split_keywords": [
"docling",
" convert",
" document",
" pdf",
" layout model",
" segmentation",
" table structure",
" table former"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5c30e6771696248b36f36ed20048e98ee24431e084834db89bc9741bfcabb20d",
"md5": "747808fd1b823f8e54b1d9d586d5a88f",
"sha256": "63a7fa29942c71c9f4d9dbbcca9f32f3e538d16879fc478c009e1e8dfc3428cc"
},
"downloads": -1,
"filename": "docling-1.12.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "747808fd1b823f8e54b1d9d586d5a88f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 46284,
"upload_time": "2024-09-16T10:58:57",
"upload_time_iso_8601": "2024-09-16T10:58:57.724232Z",
"url": "https://files.pythonhosted.org/packages/5c/30/e6771696248b36f36ed20048e98ee24431e084834db89bc9741bfcabb20d/docling-1.12.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c34f39615d5843ffdd1a5b26dbc6e759fd3efe272c3cba31506644571ab67913",
"md5": "fc056d9362072ace20f25c216e9f8ba8",
"sha256": "1009cc4c4f1a9e3df323449e836ad29dbd3551563244d339cf461fb4166c6136"
},
"downloads": -1,
"filename": "docling-1.12.1.tar.gz",
"has_sig": false,
"md5_digest": "fc056d9362072ace20f25c216e9f8ba8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 39848,
"upload_time": "2024-09-16T10:58:59",
"upload_time_iso_8601": "2024-09-16T10:58:59.405406Z",
"url": "https://files.pythonhosted.org/packages/c3/4f/39615d5843ffdd1a5b26dbc6e759fd3efe272c3cba31506644571ab67913/docling-1.12.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-16 10:58:59",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "DS4SD",
"github_project": "docling",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "docling"
}
JSON
