<h3 align="center">
<img
src="https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/img/unstructured_logo.png"
height="200"
>
</h3>
<div align="center">
<a href="https://github.com/Unstructured-IO/unstructured/blob/main/LICENSE.md"></a>
<a href="https://pypi.python.org/pypi/unstructured/"></a>
<a href="https://GitHub.com/unstructured-io/unstructured/graphs/contributors"></a>
<a href="https://github.com/Unstructured-IO/unstructured/blob/main/CODE_OF_CONDUCT.md"> </a>
<a href="https://GitHub.com/unstructured-io/unstructured/releases"></a>
<a href="https://pypi.python.org/pypi/unstructured/"></a>
[](https://pepy.tech/project/unstructured)
[](https://pepy.tech/project/unstructured)
<a
href="https://www.phorm.ai/query?projectId=34efc517-2201-4376-af43-40c4b9da3dc5">
<img src="https://img.shields.io/badge/Phorm-Ask_AI-%23F2777A.svg?&logo=data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNSIgaGVpZ2h0PSI0IiBmaWxsPSJub25lIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogIDxwYXRoIGQ9Ik00LjQzIDEuODgyYTEuNDQgMS40NCAwIDAgMS0uMDk4LjQyNmMtLjA1LjEyMy0uMTE1LjIzLS4xOTIuMzIyLS4wNzUuMDktLjE2LjE2NS0uMjU1LjIyNmExLjM1MyAxLjM1MyAwIDAgMS0uNTk1LjIxMmMtLjA5OS4wMTItLjE5Mi4wMTQtLjI3OS4wMDZsLTEuNTkzLS4xNHYtLjQwNmgxLjY1OGMuMDkuMDAxLjE3LS4xNjkuMjQ2LS4xOTFhLjYwMy42MDMgMCAwIDAgLjItLjEwNi41MjkuNTI5IDAgMCAwIC4xMzgtLjE3LjY1NC42NTQgMCAwIDAgLjA2NS0uMjRsLjAyOC0uMzJhLjkzLjkzIDAgMCAwLS4wMzYtLjI0OS41NjcuNTY3IDAgMCAwLS4xMDMtLjIuNTAyLjUwMiAwIDAgMC0uMTY4LS4xMzguNjA4LjYwOCAwIDAgMC0uMjQtLjA2N0wyLjQzNy43MjkgMS42MjUuNjcxYS4zMjIuMzIyIDAgMCAwLS4yMzIuMDU4LjM3NS4zNzUgMCAwIDAtLjExNi4yMzJsLS4xMTYgMS40NS0uMDU4LjY5Ny0uMDU4Ljc1NEwuNzA1IDRsLS4zNTctLjA3OUwuNjAyLjkwNkMuNjE3LjcyNi42NjMuNTc0LjczOS40NTRhLjk1OC45NTggMCAwIDEgLjI3NC0uMjg1Ljk3MS45NzEgMCAwIDEgLjMzNy0uMTRjLjExOS0uMDI2LjIyNy0uMDM0LjMyNS0uMDI2TDMuMjMyLjE2Yy4xNTkuMDE0LjMzNi4wMy40NTkuMDgyYTEuMTczIDEuMTczIDAgMCAxIC41NDUuNDQ3Yy4wNi4wOTQuMTA5LjE5Mi4xNDQuMjkzYTEuMzkyIDEuMzkyIDAgMCAxIC4wNzguNThsLS4wMjkuMzJaIiBmaWxsPSIjRjI3NzdBIi8+CiAgPHBhdGggZD0iTTQuMDgyIDIuMDA3YTEuNDU1IDEuNDU1IDAgMCAxLS4wOTguNDI3Yy0uMDUuMTI0LS4xMTQuMjMyLS4xOTIuMzI0YTEuMTMgMS4xMyAwIDAgMS0uMjU0LjIyNyAxLjM1MyAxLjM1MyAwIDAgMS0uNTk1LjIxNGMtLjEuMDEyLS4xOTMuMDE0LS4yOC4wMDZsLTEuNTYtLjEwOC4wMzQtLjQwNi4wMy0uMzQ4IDEuNTU5LjE1NGMuMDkgMCAuMTczLS4wMS4yNDgtLjAzM2EuNjAzLjYwMyAwIDAgMCAuMi0uMTA2LjUzMi41MzIgMCAwIDAgLjEzOS0uMTcyLjY2LjY2IDAgMCAwIC4wNjQtLjI0MWwuMDI5LS4zMjFhLjk0Ljk0IDAgMCAwLS4wMzYtLjI1LjU3LjU3IDAgMCAwLS4xMDMtLjIwMi41MDIuNTAyIDAgMCAwLS4xNjgtLjEzOC42MDUuNjA1IDAgMCAwLS4yNC0uMDY3TDEuMjczLjgyN2MtLjA5NC0uMDA4LS4xNjguMDEtLjIyMS4wNTUtLjA1My4wNDUtLjA4NC4xMTQtLjA5Mi4yMDZMLjcwNSA0IDAgMy45MzhsLjI1NS0yLjkxMUExLjAxIDEuMDEgMCAwIDEgLjM5My41NzIuOTYyLjk2MiAwIDAgMSAuNjY2LjI4NmEuOTcuOTcgMCAwIDEgLjMzOC0uMTRDMS4xMjIuMTIgMS4yMy4xMSAxLjMyOC4xMTlsMS41OTMuMTRjLjE2LjAxNC4zLjA0Ny40MjMuMWExLjE3IDEuMTcgMCAwIDEgLjU0NS40NDhjLjA2MS4wOTUuMTA5LjE5My4xNDQuMjk1YTEuNDA2IDEuNDA2IDAgMCAxIC4wNzcuNTgzbC0uMDI4LjMyMloiIGZpbGw9IndoaXRlIi8+CiAgPHBhdGggZD0iTTQuMDgyIDIuMDA3YTEuNDU1IDEuNDU1IDAgMCAxLS4wOTguNDI3Yy0uMDUuMTI0LS4xMTQuMjMyLS4xOTIuMzI0YTEuMTMgMS4xMyAwIDAgMS0uMjU0LjIyNyAxLjM1MyAxLjM1MyAwIDAgMS0uNTk1LjIxNGMtLjEuMDEyLS4xOTMuMDE0LS4yOC4wMDZsLTEuNTYtLjEwOC4wMzQtLjQwNi4wMy0uMzQ4IDEuNTU5LjE1NGMuMDkgMCAuMTczLS4wMS4yNDgtLjAzM2EuNjAzLjYwMyAwIDAgMCAuMi0uMTA2LjUzMi41MzIgMCAwIDAgLjEzOS0uMTcyLjY2LjY2IDAgMCAwIC4wNjQtLjI0MWwuMDI5LS4zMjFhLjk0Ljk0IDAgMCAwLS4wMzYtLjI1LjU3LjU3IDAgMCAwLS4xMDMtLjIwMi41MDIuNTAyIDAgMCAwLS4xNjgtLjEzOC42MDUuNjA1IDAgMCAwLS4yNC0uMDY3TDEuMjczLjgyN2MtLjA5NC0uMDA4LS4xNjguMDEtLjIyMS4wNTUtLjA1My4wNDUtLjA4NC4xMTQtLjA5Mi4yMDZMLjcwNSA0IDAgMy45MzhsLjI1NS0yLjkxMUExLjAxIDEuMDEgMCAwIDEgLjM5My41NzIuOTYyLjk2MiAwIDAgMSAuNjY2LjI4NmEuOTcuOTcgMCAwIDEgLjMzOC0uMTRDMS4xMjIuMTIgMS4yMy4xMSAxLjMyOC4xMTlsMS41OTMuMTRjLjE2LjAxNC4zLjA0Ny40MjMuMWExLjE3IDEuMTcgMCAwIDEgLjU0NS40NDhjLjA2MS4wOTUuMTA5LjE5My4xNDQuMjk1YTEuNDA2IDEuNDA2IDAgMCAxIC4wNzcuNTgzbC0uMDI4LjMyMloiIGZpbGw9IndoaXRlIi8+Cjwvc3ZnPgo=" />
</a>
</div>
<div>
<p align="center">
<a
href="https://short.unstructured.io/pzw05l7">
<img src="https://img.shields.io/badge/JOIN US ON SLACK-4A154B?style=for-the-badge&logo=slack&logoColor=white" />
</a>
<a href="https://www.linkedin.com/company/unstructuredio/">
<img src="https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white" />
</a>
</div>
<h2 align="center">
<p>Open-Source Pre-Processing Tools for Unstructured Data</p>
</h2>
The `unstructured` library provides open-source components for ingesting and pre-processing images and text documents, such as PDFs, HTML, Word docs, and [many more](https://unstructured-io.github.io/unstructured/core.html#partitioning). The use cases of `unstructured` revolve around streamlining and optimizing the data processing workflow for LLMs. `unstructured` modular functions and connectors form a cohesive system that simplifies data ingestion and pre-processing, making it adaptable to different platforms and efficient in transforming unstructured data into structured outputs.
<h3 align="center">
<p>API Announcement!</p>
</h3>
We are thrilled to announce our newly launched [Unstructured API](https://unstructured-io.github.io/unstructured/api.html), providing the Unstructured capabilities from `unstructured` as an API. Check out the [`unstructured-api` GitHub repository](https://github.com/Unstructured-IO/unstructured-api) to start making API calls. You’ll also find instructions about how to host your own API version.
While access to the hosted Unstructured API will remain free, API Keys are required to make requests. To prevent disruption, get yours [here](https://unstructured.io/api-key) and start using it today! Check out the [`unstructured-api` README](https://github.com/Unstructured-IO/unstructured-api#--) to start making API calls.</p>
#### :rocket: Beta Feature: Chipper Model
We are releasing the beta version of our Chipper model to deliver superior performance when processing high-resolution, complex documents. To start using the Chipper model in your API request, you can utilize the `hi_res_model_name=chipper` parameter. Please refer to the documentation [here](https://unstructured-io.github.io/unstructured/api.html#beta-version-hi-res-strategy-with-chipper-model).
As the Chipper model is in beta version, we welcome feedback and suggestions. For those interested in testing the Chipper model, we encourage you to connect with us on [Slack community](https://short.unstructured.io/pzw05l7).
## :eight_pointed_black_star: Quick Start
There are several ways to use the `unstructured` library:
* [Run the library in a container](https://github.com/Unstructured-IO/unstructured#using-the-library-in-a-container) or
* Install the library
1. [Install from PyPI](https://github.com/Unstructured-IO/unstructured#installing-the-library)
2. [Install for local development](https://github.com/Unstructured-IO/unstructured#installation-instructions-for-local-development)
* For installation with `conda` on Windows system, please refer to the [documentation](https://unstructured-io.github.io/unstructured/installing.html#installation-with-conda-on-windows)
### Run the library in a container
The following instructions are intended to help you get up and running using Docker to interact with `unstructured`.
See [here](https://docs.docker.com/get-docker/) if you don't already have docker installed on your machine.
NOTE: we build multi-platform images to support both x86_64 and Apple silicon hardware. `docker pull` should download the corresponding image for your architecture, but you can specify with `--platform` (e.g. `--platform linux/amd64`) if needed.
We build Docker images for all pushes to `main`. We tag each image with the corresponding short commit hash (e.g. `fbc7a69`) and the application version (e.g. `0.5.5-dev1`). We also tag the most recent image with `latest`. To leverage this, `docker pull` from our image repository.
```bash
docker pull downloads.unstructured.io/unstructured-io/unstructured:latest
```
Once pulled, you can create a container from this image and shell to it.
```bash
# create the container
docker run -dt --name unstructured downloads.unstructured.io/unstructured-io/unstructured:latest
# this will drop you into a bash shell where the Docker image is running
docker exec -it unstructured bash
```
You can also build your own Docker image.
If you only plan on parsing one type of data you can speed up building the image by commenting out some
of the packages/requirements necessary for other data types. See Dockerfile to know which lines are necessary
for your use case.
```bash
make docker-build
# this will drop you into a bash shell where the Docker image is running
make docker-start-bash
```
Once in the running container, you can try things directly in Python interpreter's interactive mode.
```bash
# this will drop you into a python console so you can run the below partition functions
python3
>>> from unstructured.partition.pdf import partition_pdf
>>> elements = partition_pdf(filename="example-docs/layout-parser-paper-fast.pdf")
>>> from unstructured.partition.text import partition_text
>>> elements = partition_text(filename="example-docs/fake-text.txt")
```
### Installing the library
Use the following instructions to get up and running with `unstructured` and test your
installation.
- Install the Python SDK to support all document types with `pip install "unstructured[all-docs]"`
- For plain text files, HTML, XML, JSON and Emails that do not require any extra dependencies, you can run `pip install unstructured`
- To process other doc types, you can install the extras required for those documents, such as `pip install "unstructured[docx,pptx]"`
- Install the following system dependencies if they are not already available on your system.
Depending on what document types you're parsing, you may not need all of these.
- `libmagic-dev` (filetype detection)
- `poppler-utils` (images and PDFs)
- `tesseract-ocr` (images and PDFs, install `tesseract-lang` for additional language support)
- `libreoffice` (MS Office docs)
- `pandoc` (EPUBs, RTFs and Open Office docs). Please note that to handle RTF files, you need version `2.14.2` or newer. Running either `make install-pandoc` or `./scripts/install-pandoc.sh` will install the correct version for you.
- For suggestions on how to install on the Windows and to learn about dependencies for other features, see the
installation documentation [here](https://unstructured-io.github.io/unstructured/installing.html).
At this point, you should be able to run the following code:
```python
from unstructured.partition.auto import partition
elements = partition(filename="example-docs/eml/fake-email.eml")
print("\n\n".join([str(el) for el in elements]))
```
### Installation Instructions for Local Development
The following instructions are intended to help you get up and running with `unstructured`
locally if you are planning to contribute to the project.
* Using `pyenv` to manage virtualenv's is recommended but not necessary
* Mac install instructions. See [here](https://github.com/Unstructured-IO/community#mac--homebrew) for more detailed instructions.
* `brew install pyenv-virtualenv`
* `pyenv install 3.10`
* Linux instructions are available [here](https://github.com/Unstructured-IO/community#linux).
* Create a virtualenv to work in and activate it, e.g. for one named `unstructured`:
`pyenv virtualenv 3.10 unstructured` <br />
`pyenv activate unstructured`
* Run `make install`
* Optional:
* To install models and dependencies for processing images and PDFs locally, run `make install-local-inference`.
* For processing image files, `tesseract` is required. See [here](https://tesseract-ocr.github.io/tessdoc/Installation.html) for installation instructions.
* For processing PDF files, `tesseract` and `poppler` are required. The [pdf2image docs](https://pdf2image.readthedocs.io/en/latest/installation.html) have instructions on installing `poppler` across various platforms.
Additionally, if you're planning to contribute to `unstructured`, we provide you an optional `pre-commit` configuration
file to ensure your code matches the formatting and linting standards used in `unstructured`.
If you'd prefer not to have code changes auto-tidied before every commit, you can use `make check` to see
whether any linting or formatting changes should be applied, and `make tidy` to apply them.
If using the optional `pre-commit`, you'll just need to install the hooks with `pre-commit install` since the
`pre-commit` package is installed as part of `make install` mentioned above. Finally, if you decided to use `pre-commit`
you can also uninstall the hooks with `pre-commit uninstall`.
In addition to develop in your local OS we also provide a helper to use docker providing a development environment:
```bash
make docker-start-dev
```
This starts a docker container with your local repo mounted to `/mnt/local_unstructured`. This docker image allows you to develop without worrying about your OS's compatibility with the repo and its dependencies.
## :clap: Quick Tour
### Documentation
This README overviews how to install, use and develop the library. For more comprehensive documentation, visit https://unstructured-io.github.io/unstructured/ .
### Concepts Guide
The `unstructured` library includes core functionality for partitioning, chunking, cleaning, and
staging raw documents for NLP tasks.
You can see a complete list of available functions and how to use them from the [Core Functionality documentation](https://unstructured-io.github.io/unstructured/core.html).
In general, these functions fall into several categories:
- *Partitioning* functions break raw documents into standard, structured elements.
- *Cleaning* functions remove unwanted text from documents, such as boilerplate and sentence fragments.
- *Staging* functions format data for downstream tasks, such as ML inference and data labeling.
- *Chunking* functions split documents into smaller sections for use in RAG apps and similarity
search.
- *Embedding* encoder classes provide an interfaces for easily converting preprocessed text to
vectors.
The **Connectors** 🔗 in `unstructured` serve as vital links between the pre-processing pipeline and various data storage platforms. They allow for the batch processing of documents across various sources, including cloud services, repositories, and local directories. Each connector is tailored to a specific platform, such as Azure, Google Drive, or Github, and comes with unique commands and dependencies. To see the list of Connectors available in `unstructured` library, please check out the [Connectors GitHub folder](https://github.com/Unstructured-IO/unstructured/tree/main/unstructured/ingest/connector) and [documentation](https://unstructured-io.github.io/unstructured/ingest/index.html)
### PDF Document Parsing Example
The following examples show how to get started with the `unstructured` library. You can parse over a dozen document types with one line of code! Use this [Colab notebook](https://colab.research.google.com/drive/1U8VCjY2-x8c6y5TYMbSFtQGlQVFHCVIW) to run the example below.
The easiest way to parse a document in unstructured is to use the `partition` function. If you use `partition` function, `unstructured` will detect the file type and route it to the appropriate file-specific partitioning function. If you are using the `partition` function, you may need to install additional parameters via `pip install unstructured[local-inference]`. Ensure you first install `libmagic` using the instructions outlined [here](https://unstructured-io.github.io/unstructured/installing.html#filetype-detection) `partition` will always apply the default arguments. If you need advanced features, use a document-specific partitioning function.
```python
from unstructured.partition.auto import partition
elements = partition("example-docs/layout-parser-paper.pdf")
```
Run `print("\n\n".join([str(el) for el in elements]))` to get a string representation of the
output, which looks like:
```
LayoutParser : A Unified Toolkit for Deep Learning Based Document Image Analysis
Zejiang Shen 1 ( (cid:0) ), Ruochen Zhang 2 , Melissa Dell 3 , Benjamin Charles Germain Lee 4 , Jacob Carlson 3 , and
Weining Li 5
Abstract. Recent advances in document image analysis (DIA) have been primarily driven by the application of neural
networks. Ideally, research outcomes could be easily deployed in production and extended for further investigation.
However, various factors like loosely organized codebases and sophisticated model configurations complicate the easy
reuse of important innovations by a wide audience. Though there have been ongoing efforts to improve reusability and
simplify deep learning (DL) model development in disciplines like natural language processing and computer vision, none
of them are optimized for challenges in the domain of DIA. This represents a major gap in the existing toolkit, as DIA
is central to academic research across a wide range of disciplines in the social sciences and humanities. This paper
introduces LayoutParser, an open-source library for streamlining the usage of DL in DIA research and applications.
The core LayoutParser library comes with a set of simple and intuitive interfaces for applying and customizing DL models
for layout detection, character recognition, and many other document processing tasks. To promote extensibility,
LayoutParser also incorporates a community platform for sharing both pre-trained models and full document digitization
pipelines. We demonstrate that LayoutParser is helpful for both lightweight and large-scale digitization pipelines in
real-word use cases. The library is publicly available at https://layout-parser.github.io
Keywords: Document Image Analysis · Deep Learning · Layout Analysis · Character Recognition · Open Source library ·
Toolkit.
Introduction
Deep Learning(DL)-based approaches are the state-of-the-art for a wide range of document image analysis (DIA) tasks
including document image classification [11,
```
See the [partitioning](https://unstructured-io.github.io/unstructured/core.html#partitioning)
section in our documentation for a full list of options and instructions on how to use
file-specific partitioning functions.
## :guardsman: Security Policy
See our [security policy](https://github.com/Unstructured-IO/unstructured/security/policy) for
information on how to report security vulnerabilities.
## :bug: Reporting Bugs
Encountered a bug? Please create a new [GitHub issue](https://github.com/Unstructured-IO/unstructured/issues/new/choose) and use our bug report template to describe the problem. To help us diagnose the issue, use the `python scripts/collect_env.py` command to gather your system's environment information and include it in your report. Your assistance helps us continuously improve our software - thank you!
## :books: Learn more
| Section | Description |
|-|-|
| [Company Website](https://unstructured.io) | Unstructured.io product and company info |
| [Documentation](https://unstructured-io.github.io/unstructured) | Full API documentation |
| [Batch Processing](unstructured/ingest/README.md) | Ingesting batches of documents through Unstructured |
## :chart_with_upwards_trend: Analytics
We’ve partnered with Scarf (https://scarf.sh) to collect anonymized user statistics to understand which features our community is using and how to prioritize product decision-making in the future. To learn more about how we collect and use this data, please read our [Privacy Policy](https://unstructured.io/privacy-policy).
To opt out of this data collection, you can set the environment variable `SCARF_NO_ANALYTICS=true` before running any `unstructured` commands.
Raw data
{
"_id": null,
"home_page": "https://github.com/Unstructured-IO/unstructured",
"name": "unstructured",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.12,>=3.9.0",
"maintainer_email": null,
"keywords": "NLP PDF HTML CV XML parsing preprocessing",
"author": "Unstructured Technologies",
"author_email": "devops@unstructuredai.io",
"download_url": "https://files.pythonhosted.org/packages/de/9d/10c00011dc6c28d9528a45dc3c4284d306c0b27207bd9d5cfa01b3c49929/unstructured-0.13.3.tar.gz",
"platform": null,
"description": "<h3 align=\"center\">\n <img\n src=\"https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/img/unstructured_logo.png\"\n height=\"200\"\n >\n</h3>\n\n<div align=\"center\">\n\n <a href=\"https://github.com/Unstructured-IO/unstructured/blob/main/LICENSE.md\"></a>\n <a href=\"https://pypi.python.org/pypi/unstructured/\"></a>\n <a href=\"https://GitHub.com/unstructured-io/unstructured/graphs/contributors\"></a>\n <a href=\"https://github.com/Unstructured-IO/unstructured/blob/main/CODE_OF_CONDUCT.md\"> </a>\n <a href=\"https://GitHub.com/unstructured-io/unstructured/releases\"></a>\n <a href=\"https://pypi.python.org/pypi/unstructured/\"></a>\n [](https://pepy.tech/project/unstructured)\n [](https://pepy.tech/project/unstructured)\n <a\n href=\"https://www.phorm.ai/query?projectId=34efc517-2201-4376-af43-40c4b9da3dc5\">\n\t<img src=\"https://img.shields.io/badge/Phorm-Ask_AI-%23F2777A.svg?&logo=data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNSIgaGVpZ2h0PSI0IiBmaWxsPSJub25lIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogIDxwYXRoIGQ9Ik00LjQzIDEuODgyYTEuNDQgMS40NCAwIDAgMS0uMDk4LjQyNmMtLjA1LjEyMy0uMTE1LjIzLS4xOTIuMzIyLS4wNzUuMDktLjE2LjE2NS0uMjU1LjIyNmExLjM1MyAxLjM1MyAwIDAgMS0uNTk1LjIxMmMtLjA5OS4wMTItLjE5Mi4wMTQtLjI3OS4wMDZsLTEuNTkzLS4xNHYtLjQwNmgxLjY1OGMuMDkuMDAxLjE3LS4xNjkuMjQ2LS4xOTFhLjYwMy42MDMgMCAwIDAgLjItLjEwNi41MjkuNTI5IDAgMCAwIC4xMzgtLjE3LjY1NC42NTQgMCAwIDAgLjA2NS0uMjRsLjAyOC0uMzJhLjkzLjkzIDAgMCAwLS4wMzYtLjI0OS41NjcuNTY3IDAgMCAwLS4xMDMtLjIuNTAyLjUwMiAwIDAgMC0uMTY4LS4xMzguNjA4LjYwOCAwIDAgMC0uMjQtLjA2N0wyLjQzNy43MjkgMS42MjUuNjcxYS4zMjIuMzIyIDAgMCAwLS4yMzIuMDU4LjM3NS4zNzUgMCAwIDAtLjExNi4yMzJsLS4xMTYgMS40NS0uMDU4LjY5Ny0uMDU4Ljc1NEwuNzA1IDRsLS4zNTctLjA3OUwuNjAyLjkwNkMuNjE3LjcyNi42NjMuNTc0LjczOS40NTRhLjk1OC45NTggMCAwIDEgLjI3NC0uMjg1Ljk3MS45NzEgMCAwIDEgLjMzNy0uMTRjLjExOS0uMDI2LjIyNy0uMDM0LjMyNS0uMDI2TDMuMjMyLjE2Yy4xNTkuMDE0LjMzNi4wMy40NTkuMDgyYTEuMTczIDEuMTczIDAgMCAxIC41NDUuNDQ3Yy4wNi4wOTQuMTA5LjE5Mi4xNDQuMjkzYTEuMzkyIDEuMzkyIDAgMCAxIC4wNzguNThsLS4wMjkuMzJaIiBmaWxsPSIjRjI3NzdBIi8+CiAgPHBhdGggZD0iTTQuMDgyIDIuMDA3YTEuNDU1IDEuNDU1IDAgMCAxLS4wOTguNDI3Yy0uMDUuMTI0LS4xMTQuMjMyLS4xOTIuMzI0YTEuMTMgMS4xMyAwIDAgMS0uMjU0LjIyNyAxLjM1MyAxLjM1MyAwIDAgMS0uNTk1LjIxNGMtLjEuMDEyLS4xOTMuMDE0LS4yOC4wMDZsLTEuNTYtLjEwOC4wMzQtLjQwNi4wMy0uMzQ4IDEuNTU5LjE1NGMuMDkgMCAuMTczLS4wMS4yNDgtLjAzM2EuNjAzLjYwMyAwIDAgMCAuMi0uMTA2LjUzMi41MzIgMCAwIDAgLjEzOS0uMTcyLjY2LjY2IDAgMCAwIC4wNjQtLjI0MWwuMDI5LS4zMjFhLjk0Ljk0IDAgMCAwLS4wMzYtLjI1LjU3LjU3IDAgMCAwLS4xMDMtLjIwMi41MDIuNTAyIDAgMCAwLS4xNjgtLjEzOC42MDUuNjA1IDAgMCAwLS4yNC0uMDY3TDEuMjczLjgyN2MtLjA5NC0uMDA4LS4xNjguMDEtLjIyMS4wNTUtLjA1My4wNDUtLjA4NC4xMTQtLjA5Mi4yMDZMLjcwNSA0IDAgMy45MzhsLjI1NS0yLjkxMUExLjAxIDEuMDEgMCAwIDEgLjM5My41NzIuOTYyLjk2MiAwIDAgMSAuNjY2LjI4NmEuOTcuOTcgMCAwIDEgLjMzOC0uMTRDMS4xMjIuMTIgMS4yMy4xMSAxLjMyOC4xMTlsMS41OTMuMTRjLjE2LjAxNC4zLjA0Ny40MjMuMWExLjE3IDEuMTcgMCAwIDEgLjU0NS40NDhjLjA2MS4wOTUuMTA5LjE5My4xNDQuMjk1YTEuNDA2IDEuNDA2IDAgMCAxIC4wNzcuNTgzbC0uMDI4LjMyMloiIGZpbGw9IndoaXRlIi8+CiAgPHBhdGggZD0iTTQuMDgyIDIuMDA3YTEuNDU1IDEuNDU1IDAgMCAxLS4wOTguNDI3Yy0uMDUuMTI0LS4xMTQuMjMyLS4xOTIuMzI0YTEuMTMgMS4xMyAwIDAgMS0uMjU0LjIyNyAxLjM1MyAxLjM1MyAwIDAgMS0uNTk1LjIxNGMtLjEuMDEyLS4xOTMuMDE0LS4yOC4wMDZsLTEuNTYtLjEwOC4wMzQtLjQwNi4wMy0uMzQ4IDEuNTU5LjE1NGMuMDkgMCAuMTczLS4wMS4yNDgtLjAzM2EuNjAzLjYwMyAwIDAgMCAuMi0uMTA2LjUzMi41MzIgMCAwIDAgLjEzOS0uMTcyLjY2LjY2IDAgMCAwIC4wNjQtLjI0MWwuMDI5LS4zMjFhLjk0Ljk0IDAgMCAwLS4wMzYtLjI1LjU3LjU3IDAgMCAwLS4xMDMtLjIwMi41MDIuNTAyIDAgMCAwLS4xNjgtLjEzOC42MDUuNjA1IDAgMCAwLS4yNC0uMDY3TDEuMjczLjgyN2MtLjA5NC0uMDA4LS4xNjguMDEtLjIyMS4wNTUtLjA1My4wNDUtLjA4NC4xMTQtLjA5Mi4yMDZMLjcwNSA0IDAgMy45MzhsLjI1NS0yLjkxMUExLjAxIDEuMDEgMCAwIDEgLjM5My41NzIuOTYyLjk2MiAwIDAgMSAuNjY2LjI4NmEuOTcuOTcgMCAwIDEgLjMzOC0uMTRDMS4xMjIuMTIgMS4yMy4xMSAxLjMyOC4xMTlsMS41OTMuMTRjLjE2LjAxNC4zLjA0Ny40MjMuMWExLjE3IDEuMTcgMCAwIDEgLjU0NS40NDhjLjA2MS4wOTUuMTA5LjE5My4xNDQuMjk1YTEuNDA2IDEuNDA2IDAgMCAxIC4wNzcuNTgzbC0uMDI4LjMyMloiIGZpbGw9IndoaXRlIi8+Cjwvc3ZnPgo=\" />\n </a>\n\n</div>\n\n<div>\n <p align=\"center\">\n <a\n href=\"https://short.unstructured.io/pzw05l7\">\n <img src=\"https://img.shields.io/badge/JOIN US ON SLACK-4A154B?style=for-the-badge&logo=slack&logoColor=white\" />\n </a>\n <a href=\"https://www.linkedin.com/company/unstructuredio/\">\n <img src=\"https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white\" />\n </a>\n</div>\n\n<h2 align=\"center\">\n <p>Open-Source Pre-Processing Tools for Unstructured Data</p>\n</h2>\n\nThe `unstructured` library provides open-source components for ingesting and pre-processing images and text documents, such as PDFs, HTML, Word docs, and [many more](https://unstructured-io.github.io/unstructured/core.html#partitioning). The use cases of `unstructured` revolve around streamlining and optimizing the data processing workflow for LLMs. `unstructured` modular functions and connectors form a cohesive system that simplifies data ingestion and pre-processing, making it adaptable to different platforms and efficient in transforming unstructured data into structured outputs.\n\n<h3 align=\"center\">\n <p>API Announcement!</p>\n</h3>\n\nWe are thrilled to announce our newly launched [Unstructured API](https://unstructured-io.github.io/unstructured/api.html), providing the Unstructured capabilities from `unstructured` as an API. Check out the [`unstructured-api` GitHub repository](https://github.com/Unstructured-IO/unstructured-api) to start making API calls. You\u2019ll also find instructions about how to host your own API version.\n\nWhile access to the hosted Unstructured API will remain free, API Keys are required to make requests. To prevent disruption, get yours [here](https://unstructured.io/api-key) and start using it today! Check out the [`unstructured-api` README](https://github.com/Unstructured-IO/unstructured-api#--) to start making API calls.</p>\n\n#### :rocket: Beta Feature: Chipper Model\n\nWe are releasing the beta version of our Chipper model to deliver superior performance when processing high-resolution, complex documents. To start using the Chipper model in your API request, you can utilize the `hi_res_model_name=chipper` parameter. Please refer to the documentation [here](https://unstructured-io.github.io/unstructured/api.html#beta-version-hi-res-strategy-with-chipper-model).\n\nAs the Chipper model is in beta version, we welcome feedback and suggestions. For those interested in testing the Chipper model, we encourage you to connect with us on [Slack community](https://short.unstructured.io/pzw05l7).\n\n## :eight_pointed_black_star: Quick Start\n\nThere are several ways to use the `unstructured` library:\n* [Run the library in a container](https://github.com/Unstructured-IO/unstructured#using-the-library-in-a-container) or\n* Install the library\n 1. [Install from PyPI](https://github.com/Unstructured-IO/unstructured#installing-the-library)\n 2. [Install for local development](https://github.com/Unstructured-IO/unstructured#installation-instructions-for-local-development)\n* For installation with `conda` on Windows system, please refer to the [documentation](https://unstructured-io.github.io/unstructured/installing.html#installation-with-conda-on-windows)\n\n### Run the library in a container\n\nThe following instructions are intended to help you get up and running using Docker to interact with `unstructured`.\nSee [here](https://docs.docker.com/get-docker/) if you don't already have docker installed on your machine.\n\nNOTE: we build multi-platform images to support both x86_64 and Apple silicon hardware. `docker pull` should download the corresponding image for your architecture, but you can specify with `--platform` (e.g. `--platform linux/amd64`) if needed.\n\nWe build Docker images for all pushes to `main`. We tag each image with the corresponding short commit hash (e.g. `fbc7a69`) and the application version (e.g. `0.5.5-dev1`). We also tag the most recent image with `latest`. To leverage this, `docker pull` from our image repository.\n\n```bash\ndocker pull downloads.unstructured.io/unstructured-io/unstructured:latest\n```\n\nOnce pulled, you can create a container from this image and shell to it.\n\n```bash\n# create the container\ndocker run -dt --name unstructured downloads.unstructured.io/unstructured-io/unstructured:latest\n\n# this will drop you into a bash shell where the Docker image is running\ndocker exec -it unstructured bash\n```\n\nYou can also build your own Docker image.\n\nIf you only plan on parsing one type of data you can speed up building the image by commenting out some\nof the packages/requirements necessary for other data types. See Dockerfile to know which lines are necessary\nfor your use case.\n\n```bash\nmake docker-build\n\n# this will drop you into a bash shell where the Docker image is running\nmake docker-start-bash\n```\n\nOnce in the running container, you can try things directly in Python interpreter's interactive mode.\n```bash\n# this will drop you into a python console so you can run the below partition functions\npython3\n\n>>> from unstructured.partition.pdf import partition_pdf\n>>> elements = partition_pdf(filename=\"example-docs/layout-parser-paper-fast.pdf\")\n\n>>> from unstructured.partition.text import partition_text\n>>> elements = partition_text(filename=\"example-docs/fake-text.txt\")\n```\n\n### Installing the library\nUse the following instructions to get up and running with `unstructured` and test your\ninstallation.\n\n- Install the Python SDK to support all document types with `pip install \"unstructured[all-docs]\"`\n - For plain text files, HTML, XML, JSON and Emails that do not require any extra dependencies, you can run `pip install unstructured`\n - To process other doc types, you can install the extras required for those documents, such as `pip install \"unstructured[docx,pptx]\"`\n- Install the following system dependencies if they are not already available on your system.\n Depending on what document types you're parsing, you may not need all of these.\n - `libmagic-dev` (filetype detection)\n - `poppler-utils` (images and PDFs)\n - `tesseract-ocr` (images and PDFs, install `tesseract-lang` for additional language support)\n - `libreoffice` (MS Office docs)\n - `pandoc` (EPUBs, RTFs and Open Office docs). Please note that to handle RTF files, you need version `2.14.2` or newer. Running either `make install-pandoc` or `./scripts/install-pandoc.sh` will install the correct version for you.\n\n- For suggestions on how to install on the Windows and to learn about dependencies for other features, see the\n installation documentation [here](https://unstructured-io.github.io/unstructured/installing.html).\n\nAt this point, you should be able to run the following code:\n\n```python\nfrom unstructured.partition.auto import partition\n\nelements = partition(filename=\"example-docs/eml/fake-email.eml\")\nprint(\"\\n\\n\".join([str(el) for el in elements]))\n```\n\n### Installation Instructions for Local Development\n\nThe following instructions are intended to help you get up and running with `unstructured`\nlocally if you are planning to contribute to the project.\n\n* Using `pyenv` to manage virtualenv's is recommended but not necessary\n\t* Mac install instructions. See [here](https://github.com/Unstructured-IO/community#mac--homebrew) for more detailed instructions.\n\t\t* `brew install pyenv-virtualenv`\n\t * `pyenv install 3.10`\n * Linux instructions are available [here](https://github.com/Unstructured-IO/community#linux).\n\n* Create a virtualenv to work in and activate it, e.g. for one named `unstructured`:\n\n\t`pyenv virtualenv 3.10 unstructured` <br />\n\t`pyenv activate unstructured`\n\n* Run `make install`\n\n* Optional:\n * To install models and dependencies for processing images and PDFs locally, run `make install-local-inference`.\n * For processing image files, `tesseract` is required. See [here](https://tesseract-ocr.github.io/tessdoc/Installation.html) for installation instructions.\n * For processing PDF files, `tesseract` and `poppler` are required. The [pdf2image docs](https://pdf2image.readthedocs.io/en/latest/installation.html) have instructions on installing `poppler` across various platforms.\n\nAdditionally, if you're planning to contribute to `unstructured`, we provide you an optional `pre-commit` configuration\nfile to ensure your code matches the formatting and linting standards used in `unstructured`.\nIf you'd prefer not to have code changes auto-tidied before every commit, you can use `make check` to see\nwhether any linting or formatting changes should be applied, and `make tidy` to apply them.\n\nIf using the optional `pre-commit`, you'll just need to install the hooks with `pre-commit install` since the\n`pre-commit` package is installed as part of `make install` mentioned above. Finally, if you decided to use `pre-commit`\nyou can also uninstall the hooks with `pre-commit uninstall`.\n\nIn addition to develop in your local OS we also provide a helper to use docker providing a development environment:\n\n```bash\nmake docker-start-dev\n```\n\nThis starts a docker container with your local repo mounted to `/mnt/local_unstructured`. This docker image allows you to develop without worrying about your OS's compatibility with the repo and its dependencies.\n\n## :clap: Quick Tour\n\n### Documentation\nThis README overviews how to install, use and develop the library. For more comprehensive documentation, visit https://unstructured-io.github.io/unstructured/ .\n\n### Concepts Guide\n\nThe `unstructured` library includes core functionality for partitioning, chunking, cleaning, and\nstaging raw documents for NLP tasks.\nYou can see a complete list of available functions and how to use them from the [Core Functionality documentation](https://unstructured-io.github.io/unstructured/core.html).\n\nIn general, these functions fall into several categories:\n- *Partitioning* functions break raw documents into standard, structured elements.\n- *Cleaning* functions remove unwanted text from documents, such as boilerplate and sentence fragments.\n- *Staging* functions format data for downstream tasks, such as ML inference and data labeling.\n- *Chunking* functions split documents into smaller sections for use in RAG apps and similarity\n search.\n- *Embedding* encoder classes provide an interfaces for easily converting preprocessed text to\n vectors.\n\nThe **Connectors** \ud83d\udd17 in `unstructured` serve as vital links between the pre-processing pipeline and various data storage platforms. They allow for the batch processing of documents across various sources, including cloud services, repositories, and local directories. Each connector is tailored to a specific platform, such as Azure, Google Drive, or Github, and comes with unique commands and dependencies. To see the list of Connectors available in `unstructured` library, please check out the [Connectors GitHub folder](https://github.com/Unstructured-IO/unstructured/tree/main/unstructured/ingest/connector) and [documentation](https://unstructured-io.github.io/unstructured/ingest/index.html)\n\n### PDF Document Parsing Example\nThe following examples show how to get started with the `unstructured` library. You can parse over a dozen document types with one line of code! Use this [Colab notebook](https://colab.research.google.com/drive/1U8VCjY2-x8c6y5TYMbSFtQGlQVFHCVIW) to run the example below.\n\nThe easiest way to parse a document in unstructured is to use the `partition` function. If you use `partition` function, `unstructured` will detect the file type and route it to the appropriate file-specific partitioning function. If you are using the `partition` function, you may need to install additional parameters via `pip install unstructured[local-inference]`. Ensure you first install `libmagic` using the instructions outlined [here](https://unstructured-io.github.io/unstructured/installing.html#filetype-detection) `partition` will always apply the default arguments. If you need advanced features, use a document-specific partitioning function.\n\n```python\nfrom unstructured.partition.auto import partition\n\nelements = partition(\"example-docs/layout-parser-paper.pdf\")\n```\n\nRun `print(\"\\n\\n\".join([str(el) for el in elements]))` to get a string representation of the\noutput, which looks like:\n\n```\n\nLayoutParser : A Uni\ufb01ed Toolkit for Deep Learning Based Document Image Analysis\n\nZejiang Shen 1 ( (cid:0) ), Ruochen Zhang 2 , Melissa Dell 3 , Benjamin Charles Germain Lee 4 , Jacob Carlson 3 , and\nWeining Li 5\n\nAbstract. Recent advances in document image analysis (DIA) have been primarily driven by the application of neural\nnetworks. Ideally, research outcomes could be easily deployed in production and extended for further investigation.\nHowever, various factors like loosely organized codebases and sophisticated model con\ufb01gurations complicate the easy\nreuse of important innovations by a wide audience. Though there have been ongoing e\ufb00orts to improve reusability and\nsimplify deep learning (DL) model development in disciplines like natural language processing and computer vision, none\nof them are optimized for challenges in the domain of DIA. This represents a major gap in the existing toolkit, as DIA\nis central to academic research across a wide range of disciplines in the social sciences and humanities. This paper\nintroduces LayoutParser, an open-source library for streamlining the usage of DL in DIA research and applications.\nThe core LayoutParser library comes with a set of simple and intuitive interfaces for applying and customizing DL models\nfor layout detection, character recognition, and many other document processing tasks. To promote extensibility,\nLayoutParser also incorporates a community platform for sharing both pre-trained models and full document digitization\npipelines. We demonstrate that LayoutParser is helpful for both lightweight and large-scale digitization pipelines in\nreal-word use cases. The library is publicly available at https://layout-parser.github.io\n\nKeywords: Document Image Analysis \u00b7 Deep Learning \u00b7 Layout Analysis \u00b7 Character Recognition \u00b7 Open Source library \u00b7\nToolkit.\n\nIntroduction\n\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of document image analysis (DIA) tasks\nincluding document image classi\ufb01cation [11,\n```\n\nSee the [partitioning](https://unstructured-io.github.io/unstructured/core.html#partitioning)\nsection in our documentation for a full list of options and instructions on how to use\nfile-specific partitioning functions.\n\n## :guardsman: Security Policy\n\nSee our [security policy](https://github.com/Unstructured-IO/unstructured/security/policy) for\ninformation on how to report security vulnerabilities.\n\n## :bug: Reporting Bugs\n\nEncountered a bug? Please create a new [GitHub issue](https://github.com/Unstructured-IO/unstructured/issues/new/choose) and use our bug report template to describe the problem. To help us diagnose the issue, use the `python scripts/collect_env.py` command to gather your system's environment information and include it in your report. Your assistance helps us continuously improve our software - thank you!\n\n## :books: Learn more\n\n| Section | Description |\n|-|-|\n| [Company Website](https://unstructured.io) | Unstructured.io product and company info |\n| [Documentation](https://unstructured-io.github.io/unstructured) | Full API documentation |\n| [Batch Processing](unstructured/ingest/README.md) | Ingesting batches of documents through Unstructured |\n\n## :chart_with_upwards_trend: Analytics\n\nWe\u2019ve partnered with Scarf (https://scarf.sh) to collect anonymized user statistics to understand which features our community is using and how to prioritize product decision-making in the future. To learn more about how we collect and use this data, please read our [Privacy Policy](https://unstructured.io/privacy-policy).\nTo opt out of this data collection, you can set the environment variable `SCARF_NO_ANALYTICS=true` before running any `unstructured` commands.\n\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A library that prepares raw documents for downstream ML tasks.",
"version": "0.13.3",
"project_urls": {
"Homepage": "https://github.com/Unstructured-IO/unstructured"
},
"split_keywords": [
"nlp",
"pdf",
"html",
"cv",
"xml",
"parsing",
"preprocessing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "87866061616867b8304fc3ef37851feabf1c6377cb88a16088175eed61f0d056",
"md5": "31e21fdf47d7acf03bc6862a242333bf",
"sha256": "852f8db24d82568fbd26c0a90ab85803d2dcb507f8b64a56932455a46b543422"
},
"downloads": -1,
"filename": "unstructured-0.13.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "31e21fdf47d7acf03bc6862a242333bf",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.12,>=3.9.0",
"size": 1904420,
"upload_time": "2024-04-21T04:12:26",
"upload_time_iso_8601": "2024-04-21T04:12:26.087235Z",
"url": "https://files.pythonhosted.org/packages/87/86/6061616867b8304fc3ef37851feabf1c6377cb88a16088175eed61f0d056/unstructured-0.13.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "de9d10c00011dc6c28d9528a45dc3c4284d306c0b27207bd9d5cfa01b3c49929",
"md5": "91349c7947990f1e7645754f6e1dba60",
"sha256": "190b51fbd282602e68b44708798dedd85479a718e4e50a300f12abaa5643df32"
},
"downloads": -1,
"filename": "unstructured-0.13.3.tar.gz",
"has_sig": false,
"md5_digest": "91349c7947990f1e7645754f6e1dba60",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.12,>=3.9.0",
"size": 1705599,
"upload_time": "2024-04-21T04:12:29",
"upload_time_iso_8601": "2024-04-21T04:12:29.809547Z",
"url": "https://files.pythonhosted.org/packages/de/9d/10c00011dc6c28d9528a45dc3c4284d306c0b27207bd9d5cfa01b3c49929/unstructured-0.13.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-21 04:12:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Unstructured-IO",
"github_project": "unstructured",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "unstructured"
}
JSON
