# Preprocess Loader
[Preprocess](https://preprocess.co) is an API service that splits any kind of document into optimal chunks of text for use in language model tasks.
Given documents in input `Preprocess` splits them into chunks of text that respect the layout and semantics of the original document.
We split the content by taking into account sections, paragraphs, lists, images, data tables, text tables, and slides, and following the content semantics for long texts.
We support PDFs, Microsoft Office documents (Word, PowerPoint, Excel), OpenOffice documents (ods, odt, odp), HTML content (web pages, articles, emails), and plain text.
This loader integrates with the `Preprocess` API library to provide document conversion and chunking or to load already chunked files inside LlamaIndex.
## Requirements
Install the Python `Preprocess` library if it is not already present:
```
pip install pypreprocess
```
## Usage
To use this loader, you need to pass the `Preprocess API Key`.
When initializing `PreprocessReader`, you should pass your `API Key`, if you don't have it yet, please ask for one at [support@preprocess.co](mailto:support@preprocess.co). Without an `API Key`, the loader will raise an error.
To chunk a file pass a valid filepath and the reader will start converting and chunking it.
`Preprocess` will chunk your files by applying an internal `Splitter`. For this reason, you should not parse the document into nodes using a `Splitter` or applying a `Splitter` while transforming documents in your `IngestionPipeline`.
If you want to handle the nodes directly:
```python
from llama_index import VectorStoreIndex
from llama_index import download_loader
PreprocessReader = download_loader("PreprocessReader")
# pass a filepath and get the chunks as nodes
loader = PreprocessReader(
api_key="your-api-key", filepath="valid/path/to/file"
)
nodes = loader.get_nodes()
# import the nodes in a Vector Store with your configuration
index = VectorStoreIndex(nodes)
query_engine = index.as_query_engine()
```
By default load_data() returns a document for each chunk, remember to not apply any splitting to these documents
```python
from llama_index import VectorStoreIndex
from llama_index import download_loader
PreprocessReader = download_loader("PreprocessReader")
# pass a filepath and get the chunks as nodes
loader = PreprocessReader(
api_key="your-api-key", filepath="valid/path/to/file"
)
documents = loader.load_data()
# don't apply any Splitter parser to documents
# if you have an ingestion pipeline you should not apply a Splitter in the transformations
# import the documents in a Vector Store, if you set the service_context parameter remember to avoid including a splitter
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
```
If you want to return only the extracted text and handle it with custom pipelines set `return_whole_document = True`
```python
# pass a filepath and get the chunks as nodes
loader = PreprocessReader(
api_key="your-api-key", filepath="valid/path/to/file"
)
document = loader.load_data(return_whole_document=True)
```
If you want to load already chunked files you can do it via `process_id` passing it to the reader.
```python
# pass a process_id obtained from a previous instance and get the chunks as one string inside a Document
loader = PreprocessReader(api_key="your-api-key", process_id="your-process-id")
```
This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.
## Other info
`PreprocessReader` is based on `pypreprocess` from [Preprocess](https://github.com/preprocess-co/pypreprocess) library.
For more information or other integration needs please check the [documentation](https://github.com/preprocess-co/pypreprocess).
Raw data
{
"_id": null,
"home_page": "",
"name": "llama-index-readers-preprocess",
"maintainer": "preprocess",
"docs_url": null,
"requires_python": ">=3.8.1,<3.12",
"maintainer_email": "",
"keywords": "chunk,chunking,documents,preprocess",
"author": "Your Name",
"author_email": "you@example.com",
"download_url": "https://files.pythonhosted.org/packages/01/7c/d97f2c7a1311e3004129452351131895a54d80b3acac5998115544b3acdf/llama_index_readers_preprocess-0.1.2.tar.gz",
"platform": null,
"description": "# Preprocess Loader\n\n[Preprocess](https://preprocess.co) is an API service that splits any kind of document into optimal chunks of text for use in language model tasks.\nGiven documents in input `Preprocess` splits them into chunks of text that respect the layout and semantics of the original document.\nWe split the content by taking into account sections, paragraphs, lists, images, data tables, text tables, and slides, and following the content semantics for long texts.\nWe support PDFs, Microsoft Office documents (Word, PowerPoint, Excel), OpenOffice documents (ods, odt, odp), HTML content (web pages, articles, emails), and plain text.\n\nThis loader integrates with the `Preprocess` API library to provide document conversion and chunking or to load already chunked files inside LlamaIndex.\n\n## Requirements\n\nInstall the Python `Preprocess` library if it is not already present:\n\n```\npip install pypreprocess\n```\n\n## Usage\n\nTo use this loader, you need to pass the `Preprocess API Key`.\nWhen initializing `PreprocessReader`, you should pass your `API Key`, if you don't have it yet, please ask for one at [support@preprocess.co](mailto:support@preprocess.co). Without an `API Key`, the loader will raise an error.\n\nTo chunk a file pass a valid filepath and the reader will start converting and chunking it.\n`Preprocess` will chunk your files by applying an internal `Splitter`. For this reason, you should not parse the document into nodes using a `Splitter` or applying a `Splitter` while transforming documents in your `IngestionPipeline`.\n\nIf you want to handle the nodes directly:\n\n```python\nfrom llama_index import VectorStoreIndex\nfrom llama_index import download_loader\n\nPreprocessReader = download_loader(\"PreprocessReader\")\n\n# pass a filepath and get the chunks as nodes\nloader = PreprocessReader(\n api_key=\"your-api-key\", filepath=\"valid/path/to/file\"\n)\nnodes = loader.get_nodes()\n\n# import the nodes in a Vector Store with your configuration\nindex = VectorStoreIndex(nodes)\nquery_engine = index.as_query_engine()\n```\n\nBy default load_data() returns a document for each chunk, remember to not apply any splitting to these documents\n\n```python\nfrom llama_index import VectorStoreIndex\nfrom llama_index import download_loader\n\nPreprocessReader = download_loader(\"PreprocessReader\")\n\n# pass a filepath and get the chunks as nodes\nloader = PreprocessReader(\n api_key=\"your-api-key\", filepath=\"valid/path/to/file\"\n)\ndocuments = loader.load_data()\n\n# don't apply any Splitter parser to documents\n# if you have an ingestion pipeline you should not apply a Splitter in the transformations\n# import the documents in a Vector Store, if you set the service_context parameter remember to avoid including a splitter\nindex = VectorStoreIndex.from_documents(documents)\nquery_engine = index.as_query_engine()\n```\n\nIf you want to return only the extracted text and handle it with custom pipelines set `return_whole_document = True`\n\n```python\n# pass a filepath and get the chunks as nodes\nloader = PreprocessReader(\n api_key=\"your-api-key\", filepath=\"valid/path/to/file\"\n)\ndocument = loader.load_data(return_whole_document=True)\n```\n\nIf you want to load already chunked files you can do it via `process_id` passing it to the reader.\n\n```python\n# pass a process_id obtained from a previous instance and get the chunks as one string inside a Document\nloader = PreprocessReader(api_key=\"your-api-key\", process_id=\"your-process-id\")\n```\n\nThis loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent. See [here](https://github.com/emptycrown/llama-hub/tree/main) for examples.\n\n## Other info\n\n`PreprocessReader` is based on `pypreprocess` from [Preprocess](https://github.com/preprocess-co/pypreprocess) library.\nFor more information or other integration needs please check the [documentation](https://github.com/preprocess-co/pypreprocess).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "llama-index readers preprocess integration",
"version": "0.1.2",
"project_urls": null,
"split_keywords": [
"chunk",
"chunking",
"documents",
"preprocess"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5f46b99833dcc7a12e7c02de6f6e40aa1f23e6020794d4dc22ad1820041465fa",
"md5": "cba56e7f15fcca265430facac35501a3",
"sha256": "25c73e4a38f6b2cb491053959f7b301a2dbf0865208514e831595ad3abd7d012"
},
"downloads": -1,
"filename": "llama_index_readers_preprocess-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cba56e7f15fcca265430facac35501a3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<3.12",
"size": 4596,
"upload_time": "2024-02-13T21:37:47",
"upload_time_iso_8601": "2024-02-13T21:37:47.173626Z",
"url": "https://files.pythonhosted.org/packages/5f/46/b99833dcc7a12e7c02de6f6e40aa1f23e6020794d4dc22ad1820041465fa/llama_index_readers_preprocess-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "017cd97f2c7a1311e3004129452351131895a54d80b3acac5998115544b3acdf",
"md5": "25e25ece7fc51dac359d8feb7c61cda7",
"sha256": "d83f7940b92af3c86275170b05108323b06162690e0a3c0452d1817cd80f9aba"
},
"downloads": -1,
"filename": "llama_index_readers_preprocess-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "25e25ece7fc51dac359d8feb7c61cda7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<3.12",
"size": 4176,
"upload_time": "2024-02-13T21:37:48",
"upload_time_iso_8601": "2024-02-13T21:37:48.774388Z",
"url": "https://files.pythonhosted.org/packages/01/7c/d97f2c7a1311e3004129452351131895a54d80b3acac5998115544b3acdf/llama_index_readers_preprocess-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-13 21:37:48",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "llama-index-readers-preprocess"
}
JSON