# Using PyMuPDF as Data Feeder in LLM / RAG Applications
This package converts the pages of a PDF to text in Markdown format using [PyMuPDF](https://pypi.org/project/PyMuPDF/).
Standard text and tables are detected, brought in the right reading sequence and then together converted to GitHub-compatible Markdown text.
Header lines are identified via the font size and appropriately prefixed with one or more '#' tags.
Bold, italic, mono-spaced text and code blocks are detected and formatted accordingly. Similar applies to ordered and unordered lists.
By default, all document pages are processed. If desired, a subset of pages can be specified by providing a list of 0-based page numbers.
# Installation
```bash
$ pip install -U pymupdf4llm
```
> This command will automatically install [PyMuPDF](https://github.com/pymupdf/PyMuPDF) if required.
Then in your script do:
```python
import pymupdf4llm
md_text = pymupdf4llm.to_markdown("input.pdf")
# now work with the markdown text, e.g. store as a UTF8-encoded file
import pathlib
pathlib.Path("output.md").write_bytes(md_text.encode())
```
Instead of the filename string as above, one can also provide a PyMuPDF `Document`. By default, all pages in the PDF will be processed. If desired, the parameter `pages=[...]` can be used to provide a list of zero-based page numbers to consider.
**Feature Overview:**
* Support for pages with **_multiple text columns_**.
* Support for **_image and vector graphics extraction_**:
1. Specify `pymupdf4llm.to_markdown("input.pdf", write_images=True)`. Default is `False`.
2. Each image or vector graphic on the page will be extracted and stored as an image named `"input.pdf-pno-index.extension"` in a folder of your choice. The image `extension` can be chosen to represent a PyMuPDF-supported image format (for instance "png" or "jpg"), `pno` is the 0-based page number and `index` is some sequence number.
3. The image files will have width and height equal to the values on the page. The desired resolution can be chosen via parameter `dpi` (default: `dpi=150`).
4. Any text contained in the images or graphics will be extracted and **also become visible as part of the generated image**. This behavior can be changed via `force_text=False` (text only apears as part of the image).
* Support for **page chunks**: Instead of returning one large string for the whole document, a list of dictionaries can be generated: one for each page. Specify `data = pymupdf4llm.to_markdown("input.pdf", page_chunks=True)`. Then, for instance the first item, `data[0]` will contain a dictionary for the first page with the text and some metadata.
* As a first example for directly supporting LLM / RAG consumers, this version can output **LlamaIndex documents**:
```python
import pymupdf4llm
md_read = LlamaMarkdownReader()
data = md_read.load_data("input.pdf")
# The result 'data' is of type List[LlamaIndexDocument]
# Every list item contains metadata and the markdown text of 1 page.
```
* A LlamaIndex document essentially corresponds to Python dictionary, where the markdown text of the page is one of the dictionary values. For instance the text of the first page is the the value of `data[0].to_dict().["text"]`.
* For details, please consult LlamaIndex documentation.
* Upon creation of the `LlamaMarkdownReader` all necessary LlamaIndex-related imports are executed. Required related package installations must have been done independently and will not be checked during pymupdf4llm installation.
Raw data
{
"_id": null,
"home_page": "https://github.com/pymupdf/RAG",
"name": "pymupdf4llm",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Artifex",
"author_email": "support@artifex.com",
"download_url": "https://files.pythonhosted.org/packages/2e/3c/1a530a410bdf76d83289bf30b3b86236d338b3f5f21842790c2cf7e9c1f6/pymupdf4llm-0.0.17.tar.gz",
"platform": null,
"description": "# Using PyMuPDF as Data Feeder in LLM / RAG Applications\r\n\r\nThis package converts the pages of a PDF to text in Markdown format using [PyMuPDF](https://pypi.org/project/PyMuPDF/).\r\n\r\nStandard text and tables are detected, brought in the right reading sequence and then together converted to GitHub-compatible Markdown text.\r\n\r\nHeader lines are identified via the font size and appropriately prefixed with one or more '#' tags.\r\n\r\nBold, italic, mono-spaced text and code blocks are detected and formatted accordingly. Similar applies to ordered and unordered lists.\r\n\r\nBy default, all document pages are processed. If desired, a subset of pages can be specified by providing a list of 0-based page numbers.\r\n\r\n\r\n# Installation\r\n\r\n```bash\r\n$ pip install -U pymupdf4llm\r\n```\r\n\r\n> This command will automatically install [PyMuPDF](https://github.com/pymupdf/PyMuPDF) if required.\r\n\r\nThen in your script do:\r\n\r\n```python\r\nimport pymupdf4llm\r\n\r\nmd_text = pymupdf4llm.to_markdown(\"input.pdf\")\r\n\r\n# now work with the markdown text, e.g. store as a UTF8-encoded file\r\nimport pathlib\r\npathlib.Path(\"output.md\").write_bytes(md_text.encode())\r\n```\r\n\r\nInstead of the filename string as above, one can also provide a PyMuPDF `Document`. By default, all pages in the PDF will be processed. If desired, the parameter `pages=[...]` can be used to provide a list of zero-based page numbers to consider.\r\n\r\n**Feature Overview:**\r\n\r\n* Support for pages with **_multiple text columns_**.\r\n* Support for **_image and vector graphics extraction_**:\r\n\r\n 1. Specify `pymupdf4llm.to_markdown(\"input.pdf\", write_images=True)`. Default is `False`.\r\n 2. Each image or vector graphic on the page will be extracted and stored as an image named `\"input.pdf-pno-index.extension\"` in a folder of your choice. The image `extension` can be chosen to represent a PyMuPDF-supported image format (for instance \"png\" or \"jpg\"), `pno` is the 0-based page number and `index` is some sequence number.\r\n 3. The image files will have width and height equal to the values on the page. The desired resolution can be chosen via parameter `dpi` (default: `dpi=150`).\r\n 4. Any text contained in the images or graphics will be extracted and **also become visible as part of the generated image**. This behavior can be changed via `force_text=False` (text only apears as part of the image).\r\n\r\n* Support for **page chunks**: Instead of returning one large string for the whole document, a list of dictionaries can be generated: one for each page. Specify `data = pymupdf4llm.to_markdown(\"input.pdf\", page_chunks=True)`. Then, for instance the first item, `data[0]` will contain a dictionary for the first page with the text and some metadata.\r\n\r\n* As a first example for directly supporting LLM / RAG consumers, this version can output **LlamaIndex documents**:\r\n\r\n ```python\r\n import pymupdf4llm\r\n \r\n md_read = LlamaMarkdownReader()\r\n data = md_read.load_data(\"input.pdf\")\r\n\r\n # The result 'data' is of type List[LlamaIndexDocument]\r\n # Every list item contains metadata and the markdown text of 1 page.\r\n ```\r\n\r\n * A LlamaIndex document essentially corresponds to Python dictionary, where the markdown text of the page is one of the dictionary values. For instance the text of the first page is the the value of `data[0].to_dict().[\"text\"]`.\r\n * For details, please consult LlamaIndex documentation.\r\n * Upon creation of the `LlamaMarkdownReader` all necessary LlamaIndex-related imports are executed. Required related package installations must have been done independently and will not be checked during pymupdf4llm installation.\r\n",
"bugtrack_url": null,
"license": "GNU AFFERO GPL 3.0",
"summary": "PyMuPDF Utilities for LLM/RAG",
"version": "0.0.17",
"project_urls": {
"Homepage": "https://github.com/pymupdf/RAG"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "aeaf1576ecfc8a62d31c0c8b34b856e52f6b05f1d76546dbac0e1d037f044a9e",
"md5": "211352ae67375461ab19e7cb0765236b",
"sha256": "26de9996945f15e3ca507908f80dc18a959f5b5214bb2e302c7f7034089665a0"
},
"downloads": -1,
"filename": "pymupdf4llm-0.0.17-py3-none-any.whl",
"has_sig": false,
"md5_digest": "211352ae67375461ab19e7cb0765236b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 26190,
"upload_time": "2024-09-21T18:40:03",
"upload_time_iso_8601": "2024-09-21T18:40:03.097801Z",
"url": "https://files.pythonhosted.org/packages/ae/af/1576ecfc8a62d31c0c8b34b856e52f6b05f1d76546dbac0e1d037f044a9e/pymupdf4llm-0.0.17-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2e3c1a530a410bdf76d83289bf30b3b86236d338b3f5f21842790c2cf7e9c1f6",
"md5": "fa3852b887cc9e22b90656ec2b4db426",
"sha256": "27287ef9fe0217cf37841a3ef2bcf70da2553c43d95ea39b664a6de6485678c3"
},
"downloads": -1,
"filename": "pymupdf4llm-0.0.17.tar.gz",
"has_sig": false,
"md5_digest": "fa3852b887cc9e22b90656ec2b4db426",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 25180,
"upload_time": "2024-09-21T18:40:01",
"upload_time_iso_8601": "2024-09-21T18:40:01.033228Z",
"url": "https://files.pythonhosted.org/packages/2e/3c/1a530a410bdf76d83289bf30b3b86236d338b3f5f21842790c2cf7e9c1f6/pymupdf4llm-0.0.17.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-21 18:40:01",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pymupdf",
"github_project": "RAG",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pymupdf4llm"
}
JSON
