<h3 align="center">
<img
src="https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/img/unstructured_logo.png"
height="200"
>
</h3>
<div align="center">
<a href="https://github.com/Unstructured-IO/unstructured/blob/main/LICENSE.md"></a>
<a href="https://pypi.python.org/pypi/unstructured/"></a>
<a href="https://GitHub.com/unstructured-io/unstructured/graphs/contributors"></a>
<a href="https://github.com/Unstructured-IO/unstructured/blob/main/CODE_OF_CONDUCT.md"> </a>
<a href="https://GitHub.com/unstructured-io/unstructured/releases"></a>
<a href="https://pypi.python.org/pypi/unstructured/"></a>
</div>
<div>
<p align="center">
<a
href="https://join.slack.com/t/unstructuredw-kbe4326/shared_invite/zt-1nlh1ot5d-dfY7zCRlhFboZrIWLA4Qgw">
<img src="https://img.shields.io/badge/JOIN US ON SLACK-4A154B?style=for-the-badge&logo=slack&logoColor=white" />
</a>
<a href="https://www.linkedin.com/company/unstructuredio/">
<img src="https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white" />
</a>
</div>
<h3 align="center">
<p>Open-Source Pre-Processing Tools for Unstructured Data</p>
</h3>
The `unstructured` library provides open-source components for pre-processing text documents
such as **PDFs**, **HTML** and **Word** Documents. These components are packaged as *bricks* 🧱, which provide
users the building blocks they need to build pipelines targeted at the documents they care
about. Bricks in the library fall into three categories:
- :jigsaw: ***Partitioning bricks*** that break raw documents down into standard, structured
elements.
- :broom: ***Cleaning bricks*** that remove unwanted text from documents, such as boilerplate and
sentence
fragments.
- :performing_arts: ***Staging bricks*** that format data for downstream tasks, such as ML inference
and data labeling.
<br></br>
## :eight_pointed_black_star: Quick Start
Use the following instructions to get up and running with `unstructured` and test your
installation. NOTE: We do not currently support python 3.11, please use an older version.
- Install the Python SDK with `pip install "unstructured[local-inference]"`
- If you do not need to process PDFs or images, you can run `pip install unstructured`
- Install the following system dependencies if they are not already available on your system.
Depending on what document types you're parsing, you may not need all of these.
- `libmagic-dev` (filetype detection)
- `poppler-utils` (images and PDFs)
- `tesseract-ocr` (images and PDFs)
- `libreoffice` (MS Office docs)
- If you are parsing PDFs, run the following to install the `detectron2` model, which
`unstructured` uses for layout detection:
- `pip install "detectron2@git+https://github.com/facebookresearch/detectron2.git@v0.6#egg=detectron2"`
At this point, you should be able to run the following code:
```python
from unstructured.partition.auto import partition
elements = partition(filename="example-docs/fake-email.eml")
print("\n\n".join([str(el) for el in elements]))
```
And if you installed with `local-inference`, you should be able to run this as well:
```python
from unstructured.partition.auto import partition
elements = partition("example-docs/layout-parser-paper.pdf")
print("\n\n".join([str(el) for el in elements]))
```
## :dizzy: Instructions for using the docker image
The following instructions are intended to help you get up and running using Docker to interact with `unstructured`.
See [here](https://docs.docker.com/get-docker/) if you don't already have docker installed on your machine.
NOTE: the image is only supported for x86_64 hardware and known to have issues on Apple silicon.
We build Docker images for all pushes to `main`. We tag each image with the corresponding short commit hash (e.g. `fbc7a69`) and the application version (e.g. `0.5.5-dev1`). We also tag the most recent image with `latest`. To leverage this, `docker pull` from our image repository.
```bash
docker pull quay.io/unstructured-io/unstructured:latest
```
Once pulled, you can create a container from this image and shell to it.
```bash
# create the container
docker run --platform linux/amd64 -d -t --name unstructured quay.io/unstructured-io/unstructured:latest
# this will drop you into a bash shell where the Docker image is running
docker exec -it unstructured bash
```
You can also build your own Docker image.
If you only plan on parsing one type of data you can speed up building the image by commenting out some
of the packages/requirements necessary for other data types. See Dockerfile to know which lines are necessary
for your use case.
```bash
make docker-build
# this will drop you into a bash shell where the Docker image is running
make docker-start-bash
```
Once in the running container, you can try things out directly in Python interpreter's interactive mode.
```bash
# this will drop you into a python console so you can run the below partition functions
python3
>>> from unstructured.partition.pdf import partition_pdf
>>> elements = partition_pdf(filename="example-docs/layout-parser-paper-fast.pdf")
>>> from unstructured.partition.text import partition_text
>>> elements = partition_text(filename="example-docs/fake-text.txt")
```
## :coffee: Installation Instructions for Local Development
The following instructions are intended to help you get up and running with `unstructured`
locally if you are planning to contribute to the project.
* Using `pyenv` to manage virtualenv's is recommended but not necessary
* Mac install instructions. See [here](https://github.com/Unstructured-IO/community#mac--homebrew) for more detailed instructions.
* `brew install pyenv-virtualenv`
* `pyenv install 3.8.15`
* Linux instructions are available [here](https://github.com/Unstructured-IO/community#linux).
* Create a virtualenv to work in and activate it, e.g. for one named `unstructured`:
`pyenv virtualenv 3.8.15 unstructured` <br />
`pyenv activate unstructured`
* Run `make install`
* Optional:
* To install models and dependencies for processing images and PDFs locally, run `make install-local-inference`.
* For processing image files, `tesseract` is required. See [here](https://tesseract-ocr.github.io/tessdoc/Installation.html) for installation instructions.
* For processing PDF files, `tesseract` and `poppler` are required. The [pdf2image docs](https://pdf2image.readthedocs.io/en/latest/installation.html) have instructions on installing `poppler` across various platforms.
Additionally, if you're planning to contribute to `unstructured`, we provide you an optional `pre-commit` configuration
file to ensure your code matches the formatting and linting standards used in `unstructured`.
If you'd prefer not having code changes auto-tidied before every commit, you can use `make check` to see
whether any linting or formatting changes should be applied, and `make tidy` to apply them.
If using the optional `pre-commit`, you'll just need to install the hooks with `pre-commit install` since the
`pre-commit` package is installed as part of `make install` mentioned above. Finally, if you decided to use `pre-commit`
you can also uninstall the hooks with `pre-commit uninstall`.
## :clap: Quick Tour
You can run this [Colab notebook](https://colab.research.google.com/drive/1U8VCjY2-x8c6y5TYMbSFtQGlQVFHCVIW) to run the examples below.
The following examples show how to get started with the `unstructured` library.
You can parse **TXT**, **HTML**, **PDF**, **EML**, **EPUB**, **DOC**, **DOCX**, **PPT**, **PPTX**, **JPG**,
and **PNG** documents with one line of code!
<br></br>
See our [documentation page](https://unstructured-io.github.io/unstructured) for a full description
of the features in the library.
### Document Parsing
The easiest way to parse a document in unstructured is to use the `partition` brick. If you
use `partition` brick, `unstructured` will detect the file type and route it to the appropriate
file-specific partitioning brick.
If you are using the `partition` brick, you may need to install additional parameters via `pip install unstructured[local-inference]`. Ensure you first install `libmagic` using the
instructions outlined [here](https://unstructured-io.github.io/unstructured/installing.html#filetype-detection)
`partition` will always apply the default arguments. If you need
advanced features, use a document-specific brick. The `partition` brick currently works for
`.txt`, `.doc`, `.docx`, `.ppt`, `.pptx`, `.jpg`, `.png`, `.eml`, `.html`, and `.pdf` documents.
```python
from unstructured.partition.auto import partition
elements = partition("example-docs/layout-parser-paper.pdf")
```
Run `print("\n\n".join([str(el) for el in elements]))` to get a string representation of the
output, which looks like:
```
LayoutParser : A Unified Toolkit for Deep Learning Based Document Image Analysis
Zejiang Shen 1 ( (cid:0) ), Ruochen Zhang 2 , Melissa Dell 3 , Benjamin Charles Germain Lee 4 , Jacob Carlson 3 , and
Weining Li 5
Abstract. Recent advances in document image analysis (DIA) have been primarily driven by the application of neural
networks. Ideally, research outcomes could be easily deployed in production and extended for further investigation.
However, various factors like loosely organized codebases and sophisticated model configurations complicate the easy
reuse of im- portant innovations by a wide audience. Though there have been on-going efforts to improve reusability and
simplify deep learning (DL) model development in disciplines like natural language processing and computer vision, none
of them are optimized for challenges in the domain of DIA. This represents a major gap in the existing toolkit, as DIA
is central to academic research across a wide range of disciplines in the social sciences and humanities. This paper
introduces LayoutParser , an open-source library for streamlining the usage of DL in DIA research and applica- tions.
The core LayoutParser library comes with a set of simple and intuitive interfaces for applying and customizing DL models
for layout de- tection, character recognition, and many other document processing tasks. To promote extensibility,
LayoutParser also incorporates a community platform for sharing both pre-trained models and full document digiti- zation
pipelines. We demonstrate that LayoutParser is helpful for both lightweight and large-scale digitization pipelines in
real-word use cases. The library is publicly available at https://layout-parser.github.io
Keywords: Document Image Analysis · Deep Learning · Layout Analysis · Character Recognition · Open Source library ·
Toolkit.
Introduction
Deep Learning(DL)-based approaches are the state-of-the-art for a wide range of document image analysis (DIA) tasks
including document image classification [11,
```
### HTML Parsing
You can parse an HTML document using the following workflow:
```python
from unstructured.partition.html import partition_html
elements = partition_html("example-docs/example-10k.html")
print("\n\n".join([str(el) for el in elements[:5]]))
```
The print statement will show the following text:
```
UNITED STATES
SECURITIES AND EXCHANGE COMMISSION
Washington, D.C. 20549
FORM 10-K
ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934
```
And `elements` will be a list of elements in the HTML document, similar to the following:
```python
[<unstructured.documents.elements.Title at 0x169cbe820>,
<unstructured.documents.elements.NarrativeText at 0x169cbe8e0>,
<unstructured.documents.elements.NarrativeText at 0x169cbe3a0>]
```
### PDF Parsing
You can use the following workflow to parse PDF documents.
```python
from unstructured.partition.pdf import partition_pdf
elements = partition_pdf("example-docs/layout-parser-paper.pdf")
```
The output will look the same as the example from the document parsing section above.
### E-mail Parsing
The `partition_email` function within `unstructured` is helpful for parsing `.eml` files. Common
e-mail clients such as Microsoft Outlook and Gmail support exporting e-mails as `.eml` files.
`partition_email` accepts filenames, file-like object, and raw text as input. The following
three snippets for parsing `.eml` files are equivalent:
```python
from unstructured.partition.email import partition_email
elements = partition_email(filename="example-docs/fake-email.eml")
with open("example-docs/fake-email.eml", "r") as f:
elements = partition_email(file=f)
with open("example-docs/fake-email.eml", "r") as f:
text = f.read()
elements = partition_email(text=text)
```
The `elements` output will look like the following:
```python
[<unstructured.documents.html.HTMLNarrativeText at 0x13ab14370>,
<unstructured.documents.html.HTMLTitle at 0x106877970>,
<unstructured.documents.html.HTMLListItem at 0x1068776a0>,
<unstructured.documents.html.HTMLListItem at 0x13fe4b0a0>]
```
Run `print("\n\n".join([str(el) for el in elements]))` to get a string representation of the
output, which looks like:
```python
This is a test email to use for unit tests.
Important points:
Roses are red
Violets are blue
```
### Text Document Parsing
The `partition_text` function within `unstructured` can be used to parse simple
text files into elements.
`partition_text` accepts filenames, file-like object, and raw text as input. The following three snippets are for parsing text files:
```python
from unstructured.partition.text import partition_text
elements = partition_text(filename="example-docs/fake-text.txt")
with open("example-docs/fake-text.txt", "r") as f:
elements = partition_text(file=f)
with open("example-docs/fake-text.txt", "r") as f:
text = f.read()
elements = partition_text(text=text)
```
The `elements` output will look like the following:
```python
[<unstructured.documents.html.HTMLNarrativeText at 0x13ab14370>,
<unstructured.documents.html.HTMLTitle at 0x106877970>,
<unstructured.documents.html.HTMLListItem at 0x1068776a0>,
<unstructured.documents.html.HTMLListItem at 0x13fe4b0a0>]
```
Run `print("\n\n".join([str(el) for el in elements]))` to get a string representation of the
output, which looks like:
```python
This is a test document to use for unit tests.
Important points:
Hamburgers are delicious
Dogs are the best
I love fuzzy blankets
```
## :guardsman: Security Policy
See our [security policy](https://github.com/Unstructured-IO/unstructured/security/policy) for
information on how to report security vulnerabilities.
## :books: Learn more
| Section | Description |
|-|-|
| [Company Website](https://unstructured.io) | Unstructured.io product and company info |
| [Documentation](https://unstructured-io.github.io/unstructured) | Full API documentation |
| [Batch Processing](Ingest.md) | Ingesting batches of documents through Unstructured |
Raw data
{
"_id": null,
"home_page": "https://github.com/Unstructured-IO/unstructured",
"name": "unstructured",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7.0",
"maintainer_email": "",
"keywords": "NLP PDF HTML CV XML parsing preprocessing",
"author": "Unstructured Technologies",
"author_email": "devops@unstructuredai.io",
"download_url": "https://files.pythonhosted.org/packages/dd/07/9bcc47fdc9cf967c9effc5c36474e04024eb82e9427fc49d6dbabe9bfd1b/unstructured-0.5.6.tar.gz",
"platform": null,
"description": "<h3 align=\"center\">\n <img\n src=\"https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/img/unstructured_logo.png\"\n height=\"200\"\n >\n</h3>\n\n<div align=\"center\">\n\n <a href=\"https://github.com/Unstructured-IO/unstructured/blob/main/LICENSE.md\"></a>\n <a href=\"https://pypi.python.org/pypi/unstructured/\"></a>\n <a href=\"https://GitHub.com/unstructured-io/unstructured/graphs/contributors\"></a>\n <a href=\"https://github.com/Unstructured-IO/unstructured/blob/main/CODE_OF_CONDUCT.md\"> </a>\n <a href=\"https://GitHub.com/unstructured-io/unstructured/releases\"></a>\n <a href=\"https://pypi.python.org/pypi/unstructured/\"></a>\n\n</div>\n\n<div>\n <p align=\"center\">\n <a\n href=\"https://join.slack.com/t/unstructuredw-kbe4326/shared_invite/zt-1nlh1ot5d-dfY7zCRlhFboZrIWLA4Qgw\">\n <img src=\"https://img.shields.io/badge/JOIN US ON SLACK-4A154B?style=for-the-badge&logo=slack&logoColor=white\" />\n </a>\n <a href=\"https://www.linkedin.com/company/unstructuredio/\">\n <img src=\"https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white\" />\n </a>\n</div>\n\n<h3 align=\"center\">\n <p>Open-Source Pre-Processing Tools for Unstructured Data</p>\n</h3>\n\nThe `unstructured` library provides open-source components for pre-processing text documents\nsuch as **PDFs**, **HTML** and **Word** Documents. These components are packaged as *bricks* \ud83e\uddf1, which provide\nusers the building blocks they need to build pipelines targeted at the documents they care\nabout. Bricks in the library fall into three categories:\n\n- :jigsaw: ***Partitioning bricks*** that break raw documents down into standard, structured\n elements.\n- :broom: ***Cleaning bricks*** that remove unwanted text from documents, such as boilerplate and\n sentence\n fragments.\n- :performing_arts: ***Staging bricks*** that format data for downstream tasks, such as ML inference\n and data labeling.\n\n<br></br>\n\n## :eight_pointed_black_star: Quick Start\n\nUse the following instructions to get up and running with `unstructured` and test your\ninstallation. NOTE: We do not currently support python 3.11, please use an older version.\n\n- Install the Python SDK with `pip install \"unstructured[local-inference]\"`\n\t\t- If you do not need to process PDFs or images, you can run `pip install unstructured`\n- Install the following system dependencies if they are not already available on your system.\n Depending on what document types you're parsing, you may not need all of these.\n - `libmagic-dev` (filetype detection)\n - `poppler-utils` (images and PDFs)\n - `tesseract-ocr` (images and PDFs)\n - `libreoffice` (MS Office docs)\n- If you are parsing PDFs, run the following to install the `detectron2` model, which\n `unstructured` uses for layout detection:\n - `pip install \"detectron2@git+https://github.com/facebookresearch/detectron2.git@v0.6#egg=detectron2\"`\n\nAt this point, you should be able to run the following code:\n\n```python\nfrom unstructured.partition.auto import partition\n\nelements = partition(filename=\"example-docs/fake-email.eml\")\nprint(\"\\n\\n\".join([str(el) for el in elements]))\n```\n\nAnd if you installed with `local-inference`, you should be able to run this as well:\n\n```python\nfrom unstructured.partition.auto import partition\n\nelements = partition(\"example-docs/layout-parser-paper.pdf\")\nprint(\"\\n\\n\".join([str(el) for el in elements]))\n```\n\n## :dizzy: Instructions for using the docker image\n\nThe following instructions are intended to help you get up and running using Docker to interact with `unstructured`.\nSee [here](https://docs.docker.com/get-docker/) if you don't already have docker installed on your machine.\n\nNOTE: the image is only supported for x86_64 hardware and known to have issues on Apple silicon.\n\nWe build Docker images for all pushes to `main`. We tag each image with the corresponding short commit hash (e.g. `fbc7a69`) and the application version (e.g. `0.5.5-dev1`). We also tag the most recent image with `latest`. To leverage this, `docker pull` from our image repository.\n\n```bash\ndocker pull quay.io/unstructured-io/unstructured:latest\n```\n\nOnce pulled, you can create a container from this image and shell to it.\n\n```bash\n# create the container\ndocker run --platform linux/amd64 -d -t --name unstructured quay.io/unstructured-io/unstructured:latest\n\n# this will drop you into a bash shell where the Docker image is running\ndocker exec -it unstructured bash \n```\n\nYou can also build your own Docker image.\n\nIf you only plan on parsing one type of data you can speed up building the image by commenting out some\nof the packages/requirements necessary for other data types. See Dockerfile to know which lines are necessary\nfor your use case.\n\n```bash\nmake docker-build\n\n# this will drop you into a bash shell where the Docker image is running\nmake docker-start-bash\n```\n\nOnce in the running container, you can try things out directly in Python interpreter's interactive mode.\n```bash\n# this will drop you into a python console so you can run the below partition functions\npython3\n\n>>> from unstructured.partition.pdf import partition_pdf\n>>> elements = partition_pdf(filename=\"example-docs/layout-parser-paper-fast.pdf\")\n\n>>> from unstructured.partition.text import partition_text\n>>> elements = partition_text(filename=\"example-docs/fake-text.txt\")\n```\n\n\n## :coffee: Installation Instructions for Local Development\n\nThe following instructions are intended to help you get up and running with `unstructured`\nlocally if you are planning to contribute to the project.\n\n* Using `pyenv` to manage virtualenv's is recommended but not necessary\n\t* Mac install instructions. See [here](https://github.com/Unstructured-IO/community#mac--homebrew) for more detailed instructions.\n\t\t* `brew install pyenv-virtualenv`\n\t * `pyenv install 3.8.15`\n * Linux instructions are available [here](https://github.com/Unstructured-IO/community#linux).\n\n* Create a virtualenv to work in and activate it, e.g. for one named `unstructured`:\n\n\t`pyenv virtualenv 3.8.15 unstructured` <br />\n\t`pyenv activate unstructured`\n\n* Run `make install`\n\n* Optional:\n * To install models and dependencies for processing images and PDFs locally, run `make install-local-inference`.\n * For processing image files, `tesseract` is required. See [here](https://tesseract-ocr.github.io/tessdoc/Installation.html) for installation instructions.\n * For processing PDF files, `tesseract` and `poppler` are required. The [pdf2image docs](https://pdf2image.readthedocs.io/en/latest/installation.html) have instructions on installing `poppler` across various platforms.\n\nAdditionally, if you're planning to contribute to `unstructured`, we provide you an optional `pre-commit` configuration\nfile to ensure your code matches the formatting and linting standards used in `unstructured`.\nIf you'd prefer not having code changes auto-tidied before every commit, you can use `make check` to see\nwhether any linting or formatting changes should be applied, and `make tidy` to apply them.\n\nIf using the optional `pre-commit`, you'll just need to install the hooks with `pre-commit install` since the\n`pre-commit` package is installed as part of `make install` mentioned above. Finally, if you decided to use `pre-commit`\nyou can also uninstall the hooks with `pre-commit uninstall`.\n\n## :clap: Quick Tour\n\nYou can run this [Colab notebook](https://colab.research.google.com/drive/1U8VCjY2-x8c6y5TYMbSFtQGlQVFHCVIW) to run the examples below.\n\nThe following examples show how to get started with the `unstructured` library.\nYou can parse **TXT**, **HTML**, **PDF**, **EML**, **EPUB**, **DOC**, **DOCX**, **PPT**, **PPTX**, **JPG**,\nand **PNG** documents with one line of code!\n<br></br>\nSee our [documentation page](https://unstructured-io.github.io/unstructured) for a full description\nof the features in the library.\n\n### Document Parsing\n\nThe easiest way to parse a document in unstructured is to use the `partition` brick. If you\nuse `partition` brick, `unstructured` will detect the file type and route it to the appropriate\nfile-specific partitioning brick.\nIf you are using the `partition` brick, you may need to install additional parameters via `pip install unstructured[local-inference]`. Ensure you first install `libmagic` using the\ninstructions outlined [here](https://unstructured-io.github.io/unstructured/installing.html#filetype-detection)\n`partition` will always apply the default arguments. If you need\nadvanced features, use a document-specific brick. The `partition` brick currently works for\n`.txt`, `.doc`, `.docx`, `.ppt`, `.pptx`, `.jpg`, `.png`, `.eml`, `.html`, and `.pdf` documents.\n\n```python\nfrom unstructured.partition.auto import partition\n\nelements = partition(\"example-docs/layout-parser-paper.pdf\")\n```\n\nRun `print(\"\\n\\n\".join([str(el) for el in elements]))` to get a string representation of the\noutput, which looks like:\n\n```\n\nLayoutParser : A Uni\ufb01ed Toolkit for Deep Learning Based Document Image Analysis\n\nZejiang Shen 1 ( (cid:0) ), Ruochen Zhang 2 , Melissa Dell 3 , Benjamin Charles Germain Lee 4 , Jacob Carlson 3 , and\nWeining Li 5\n\nAbstract. Recent advances in document image analysis (DIA) have been primarily driven by the application of neural\nnetworks. Ideally, research outcomes could be easily deployed in production and extended for further investigation.\nHowever, various factors like loosely organized codebases and sophisticated model con\ufb01gurations complicate the easy\nreuse of im- portant innovations by a wide audience. Though there have been on-going e\ufb00orts to improve reusability and\nsimplify deep learning (DL) model development in disciplines like natural language processing and computer vision, none\nof them are optimized for challenges in the domain of DIA. This represents a major gap in the existing toolkit, as DIA\nis central to academic research across a wide range of disciplines in the social sciences and humanities. This paper\nintroduces LayoutParser , an open-source library for streamlining the usage of DL in DIA research and applica- tions.\nThe core LayoutParser library comes with a set of simple and intuitive interfaces for applying and customizing DL models\nfor layout de- tection, character recognition, and many other document processing tasks. To promote extensibility,\nLayoutParser also incorporates a community platform for sharing both pre-trained models and full document digiti- zation\npipelines. We demonstrate that LayoutParser is helpful for both lightweight and large-scale digitization pipelines in\nreal-word use cases. The library is publicly available at https://layout-parser.github.io\n\nKeywords: Document Image Analysis \u00b7 Deep Learning \u00b7 Layout Analysis \u00b7 Character Recognition \u00b7 Open Source library \u00b7\nToolkit.\n\nIntroduction\n\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of document image analysis (DIA) tasks\nincluding document image classi\ufb01cation [11,\n```\n\n### HTML Parsing\n\nYou can parse an HTML document using the following workflow:\n\n```python\nfrom unstructured.partition.html import partition_html\n\nelements = partition_html(\"example-docs/example-10k.html\")\nprint(\"\\n\\n\".join([str(el) for el in elements[:5]]))\n```\n\nThe print statement will show the following text:\n```\nUNITED STATES\n\nSECURITIES AND EXCHANGE COMMISSION\n\nWashington, D.C. 20549\n\nFORM 10-K\n\nANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934\n```\n\nAnd `elements` will be a list of elements in the HTML document, similar to the following:\n\n```python\n[<unstructured.documents.elements.Title at 0x169cbe820>,\n <unstructured.documents.elements.NarrativeText at 0x169cbe8e0>,\n <unstructured.documents.elements.NarrativeText at 0x169cbe3a0>]\n```\n\n### PDF Parsing\n\nYou can use the following workflow to parse PDF documents.\n\n```python\nfrom unstructured.partition.pdf import partition_pdf\n\nelements = partition_pdf(\"example-docs/layout-parser-paper.pdf\")\n```\n\nThe output will look the same as the example from the document parsing section above.\n\n\n### E-mail Parsing\n\nThe `partition_email` function within `unstructured` is helpful for parsing `.eml` files. Common\ne-mail clients such as Microsoft Outlook and Gmail support exporting e-mails as `.eml` files.\n`partition_email` accepts filenames, file-like object, and raw text as input. The following\nthree snippets for parsing `.eml` files are equivalent:\n\n```python\nfrom unstructured.partition.email import partition_email\n\nelements = partition_email(filename=\"example-docs/fake-email.eml\")\n\nwith open(\"example-docs/fake-email.eml\", \"r\") as f:\n elements = partition_email(file=f)\n\nwith open(\"example-docs/fake-email.eml\", \"r\") as f:\n text = f.read()\nelements = partition_email(text=text)\n```\n\nThe `elements` output will look like the following:\n\n```python\n[<unstructured.documents.html.HTMLNarrativeText at 0x13ab14370>,\n<unstructured.documents.html.HTMLTitle at 0x106877970>,\n<unstructured.documents.html.HTMLListItem at 0x1068776a0>,\n<unstructured.documents.html.HTMLListItem at 0x13fe4b0a0>]\n```\n\nRun `print(\"\\n\\n\".join([str(el) for el in elements]))` to get a string representation of the\noutput, which looks like:\n\n```python\nThis is a test email to use for unit tests.\n\nImportant points:\n\nRoses are red\n\nViolets are blue\n```\n\n### Text Document Parsing\n\nThe `partition_text` function within `unstructured` can be used to parse simple\ntext files into elements.\n\n`partition_text` accepts filenames, file-like object, and raw text as input. The following three snippets are for parsing text files:\n\n```python\nfrom unstructured.partition.text import partition_text\n\nelements = partition_text(filename=\"example-docs/fake-text.txt\")\n\nwith open(\"example-docs/fake-text.txt\", \"r\") as f:\n elements = partition_text(file=f)\n\nwith open(\"example-docs/fake-text.txt\", \"r\") as f:\n text = f.read()\nelements = partition_text(text=text)\n```\n\nThe `elements` output will look like the following:\n\n```python\n[<unstructured.documents.html.HTMLNarrativeText at 0x13ab14370>,\n<unstructured.documents.html.HTMLTitle at 0x106877970>,\n<unstructured.documents.html.HTMLListItem at 0x1068776a0>,\n<unstructured.documents.html.HTMLListItem at 0x13fe4b0a0>]\n```\n\nRun `print(\"\\n\\n\".join([str(el) for el in elements]))` to get a string representation of the\noutput, which looks like:\n\n```python\nThis is a test document to use for unit tests.\n\nImportant points:\n\nHamburgers are delicious\n\nDogs are the best\n\nI love fuzzy blankets\n```\n\n\n## :guardsman: Security Policy\n\nSee our [security policy](https://github.com/Unstructured-IO/unstructured/security/policy) for\ninformation on how to report security vulnerabilities.\n\n## :books: Learn more\n\n| Section | Description |\n|-|-|\n| [Company Website](https://unstructured.io) | Unstructured.io product and company info |\n| [Documentation](https://unstructured-io.github.io/unstructured) | Full API documentation |\n| [Batch Processing](Ingest.md) | Ingesting batches of documents through Unstructured |",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "A library that prepares raw documents for downstream ML tasks.",
"version": "0.5.6",
"split_keywords": [
"nlp",
"pdf",
"html",
"cv",
"xml",
"parsing",
"preprocessing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "dd079bcc47fdc9cf967c9effc5c36474e04024eb82e9427fc49d6dbabe9bfd1b",
"md5": "17530a02c5d4fcdcda1c940cdad4b89d",
"sha256": "f8bc7dff96b2d35773e2be5ca0a655ece975e433ddd719ebbb6761112d434dc7"
},
"downloads": -1,
"filename": "unstructured-0.5.6.tar.gz",
"has_sig": false,
"md5_digest": "17530a02c5d4fcdcda1c940cdad4b89d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7.0",
"size": 1295937,
"upload_time": "2023-03-21T20:44:57",
"upload_time_iso_8601": "2023-03-21T20:44:57.714282Z",
"url": "https://files.pythonhosted.org/packages/dd/07/9bcc47fdc9cf967c9effc5c36474e04024eb82e9427fc49d6dbabe9bfd1b/unstructured-0.5.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-21 20:44:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "Unstructured-IO",
"github_project": "unstructured",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "unstructured"
}
JSON
