x-ray is a Python library for finding bad redactions in PDF documents.
## Why?
At Free Law Project, we collect millions of PDFs. An ongoing problem
is that people fail to properly redact things. Instead of doing it the right
way, they just draw a black rectangle or a black highlight on top of black
text and call it a day. Well, when that happens you just select the text under
the rectangle, and you can read it again. Not great.
After witnessing this problem for years, we decided it would be good to figure
out how common it is, so, with some help, we built this simple tool. You give
the tool the path to a PDF. It tells you if it has worthless redactions in it.
## What next?
Right now, `x-ray` works pretty well and we are using it to analyze documents
in our collections. It could be better though. Bad redactions take many forms.
See the issues tab for other examples we don't yet support. We'd love your
help solving some of tougher cases.
## Installation
With poetry, do:
```text
poetry add x-ray
```
With pip, that'd be:
```text
pip install x-ray
```
## Usage
You can easily use this on the command line. Once installed, just:
```bash
% xray path/to/your/file.pdf
{
"1": [
{
"bbox": [
58.550079345703125,
72.19873046875,
75.65007781982422,
739.3987426757812
],
"text": "The Ring travels by way of Cirith Ungol"
}
]
}
```
Or if you have the file on a server somewhere, give it a URL. If it starts
with `https://`, it will be interpreted as a PDF to download:
```bash
% xray https://free.law/pdf/congressional-testimony-michael-lissner-free-law-project-hearing-on-ethics-and-transparency-2021-10-26.pdf
{}
```
A fun trick you can do is to make a file with one URL per line, call it `urls.txt`. Then you can run this to check each URL:
```bash
xargs -n 1 xray < urls.txt
```
However you run `xray` on the command line, you'll get JSON as output. When you have that, you can use it with tools like [`jq`][jq]. The format is as follows:
- It's a dict.
- The keys are page numbers.
- Each page number maps to a list of dicts.
- Each of those dicts maps to two keys.
- The first key is `bbox`. This is a four-tuple that indicates the x,y positions of the upper left corner and then lower right corners of the bad redaction.
- The second key is `text`. This is the text under the bad rectangle.
Simple enough.
You can also use it as a Python module, if you prefer the long-form:
```
% python -m xray some-file.pdf
```
But that's not as easy to remember.
If you want a bit more, you can, of course, use `xray` in Python:
```python
from pprint import pprint
import xray
bad_redactions = xray.inspect("some/path/to/your/file.pdf") # Pathlib works too
pprint(bad_redactions)
{1: [{'bbox': (58.550079345703125,
72.19873046875,
75.65007781982422,
739.3987426757812),
'text': 'Aragorn is the one true king.'}]}
```
The output is the same as above, except it's a Python object, not a JSON object.
If you already have the file contents as a `bytes` object, that'll work too:
```python
some_bytes = requests.get("https://lotr-secrets.com/some-doc.pdf").content
bad_redactions = xray.inspect(some_bytes)
```
Note that because the `inspect` method uses the same signature no matter what,
the type of the object you give it is essential:
Input | `xray`'s Assumption
-- | --
`str` or Pathlib `Path` | local file
`str` that starts with `https://` | URL to download
`bytes` | PDF in memory
This means that if you provide the filename on disk as a bytes object instead
of a `str`, it's not going to work. This will fail:
```python
xray.inspect(b"some-file-path.pdf")
```
That's pretty much it. There are no configuration files or other variables to
learn. You give it a file name. If there is a bad redaction in it, you'll soon
find out.
## How it works
Under the covers, `xray` uses the high-performant [PyMuPDF project][mu] to parse PDFs. It has been a wonderful project to work with.
You can read the source to see how it works, but the general idea is to:
1. Find rectangles in a PDF.
2. Find letters in the same location
3. Render the rectangle as an image
4. Inspect the rectangle to see if it's all one color. If it is, then that's a
bad redaction. If not, then we assume you can see a mix of text and
drawings, indicating a redaction that's OK.
The PDF format is a big and complicated one, so it's difficult to do all this perfectly. We do our best, but there's always more to do to make it better. [Donations][d] and sponsored work help.
[d]: https://free.law/donate/
## Contributions
Please see the issues list on Github for things we need, or start a conversation if you have questions. Before you do your first contribution, we'll need a signed contributor license agreement. See the template in the repo.
## Deployment
Releases happen automatically via Github Actions. To trigger an automated build:
0. Update CHANGES.md
1. Update the version in pyproject.toml
2. Tag the commit with something like "v0.0.0".
If you wish to create a new version manually, the process is:
1. Update version info in `pyproject.toml`
2. Configure your Pypi credentials [with Poetry][creds]
3. Build and publish the version:
```sh
poetry publish --build
```
## License
This repository is available under the permissive BSD license, making it easy and safe to incorporate in your own libraries.
Pull and feature requests welcome. Online editing in GitHub is possible (and easy!).
[jq]: https://stedolan.github.io/jq/
[mu]: https://pymupdf.readthedocs.io/
[asc]: https://en.wikipedia.org/wiki/Ascender_(typography)
[creds]: https://python-poetry.org/docs/repositories/#configuring-credentials
Raw data
{
"_id": null,
"home_page": "https://free.law/projects/x-ray",
"name": "x-ray",
"maintainer": "Free Law Project",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "info@free.law",
"keywords": "legal,courts,redactions",
"author": "Free Law Project",
"author_email": "info@free.law",
"download_url": "https://files.pythonhosted.org/packages/3f/2b/029fe5eebaffef4a768c3d73128e41a54a5b70b9b142e56ab93035934b9f/x_ray-0.3.4.tar.gz",
"platform": null,
"description": "\n\nx-ray is a Python library for finding bad redactions in PDF documents.\n\n## Why?\n\nAt Free Law Project, we collect millions of PDFs. An ongoing problem\nis that people fail to properly redact things. Instead of doing it the right\nway, they just draw a black rectangle or a black highlight on top of black\ntext and call it a day. Well, when that happens you just select the text under\nthe rectangle, and you can read it again. Not great.\n\nAfter witnessing this problem for years, we decided it would be good to figure\nout how common it is, so, with some help, we built this simple tool. You give\nthe tool the path to a PDF. It tells you if it has worthless redactions in it.\n\n\n## What next?\n\nRight now, `x-ray` works pretty well and we are using it to analyze documents\nin our collections. It could be better though. Bad redactions take many forms.\nSee the issues tab for other examples we don't yet support. We'd love your\nhelp solving some of tougher cases.\n\n\n## Installation\n\nWith poetry, do:\n\n```text\npoetry add x-ray\n```\n\nWith pip, that'd be:\n```text\npip install x-ray\n```\n\n## Usage\n\nYou can easily use this on the command line. Once installed, just:\n\n```bash\n% xray path/to/your/file.pdf\n{\n \"1\": [\n {\n \"bbox\": [\n 58.550079345703125,\n 72.19873046875,\n 75.65007781982422,\n 739.3987426757812\n ],\n \"text\": \"The Ring travels by way of Cirith Ungol\"\n }\n ]\n}\n```\n\nOr if you have the file on a server somewhere, give it a URL. If it starts\nwith `https://`, it will be interpreted as a PDF to download:\n\n```bash\n% xray https://free.law/pdf/congressional-testimony-michael-lissner-free-law-project-hearing-on-ethics-and-transparency-2021-10-26.pdf\n{}\n```\n\nA fun trick you can do is to make a file with one URL per line, call it `urls.txt`. Then you can run this to check each URL:\n\n```bash\nxargs -n 1 xray < urls.txt\n```\n\nHowever you run `xray` on the command line, you'll get JSON as output. When you have that, you can use it with tools like [`jq`][jq]. The format is as follows:\n\n - It's a dict.\n - The keys are page numbers.\n - Each page number maps to a list of dicts.\n - Each of those dicts maps to two keys.\n - The first key is `bbox`. This is a four-tuple that indicates the x,y positions of the upper left corner and then lower right corners of the bad redaction.\n - The second key is `text`. This is the text under the bad rectangle.\n\nSimple enough.\n\nYou can also use it as a Python module, if you prefer the long-form:\n\n```\n% python -m xray some-file.pdf\n```\n\nBut that's not as easy to remember.\n\nIf you want a bit more, you can, of course, use `xray` in Python:\n\n```python\nfrom pprint import pprint\nimport xray\nbad_redactions = xray.inspect(\"some/path/to/your/file.pdf\") # Pathlib works too\npprint(bad_redactions)\n{1: [{'bbox': (58.550079345703125,\n 72.19873046875,\n 75.65007781982422,\n 739.3987426757812),\n 'text': 'Aragorn is the one true king.'}]}\n```\n\nThe output is the same as above, except it's a Python object, not a JSON object.\n\nIf you already have the file contents as a `bytes` object, that'll work too:\n\n```python\nsome_bytes = requests.get(\"https://lotr-secrets.com/some-doc.pdf\").content\nbad_redactions = xray.inspect(some_bytes)\n```\n\nNote that because the `inspect` method uses the same signature no matter what,\nthe type of the object you give it is essential:\n\nInput | `xray`'s Assumption\n-- | --\n`str` or Pathlib `Path` | local file\n`str` that starts with `https://` | URL to download\n`bytes` | PDF in memory\n\nThis means that if you provide the filename on disk as a bytes object instead\nof a `str`, it's not going to work. This will fail:\n\n```python\nxray.inspect(b\"some-file-path.pdf\")\n```\n\nThat's pretty much it. There are no configuration files or other variables to\nlearn. You give it a file name. If there is a bad redaction in it, you'll soon\nfind out.\n\n\n## How it works\n\nUnder the covers, `xray` uses the high-performant [PyMuPDF project][mu] to parse PDFs. It has been a wonderful project to work with.\n\nYou can read the source to see how it works, but the general idea is to:\n\n1. Find rectangles in a PDF.\n\n2. Find letters in the same location\n\n3. Render the rectangle as an image\n\n4. Inspect the rectangle to see if it's all one color. If it is, then that's a\n bad redaction. If not, then we assume you can see a mix of text and\n drawings, indicating a redaction that's OK.\n\nThe PDF format is a big and complicated one, so it's difficult to do all this perfectly. We do our best, but there's always more to do to make it better. [Donations][d] and sponsored work help.\n\n[d]: https://free.law/donate/\n\n\n## Contributions\n\nPlease see the issues list on Github for things we need, or start a conversation if you have questions. Before you do your first contribution, we'll need a signed contributor license agreement. See the template in the repo.\n\n\n## Deployment\n\nReleases happen automatically via Github Actions. To trigger an automated build:\n\n0. Update CHANGES.md\n\n1. Update the version in pyproject.toml\n\n2. Tag the commit with something like \"v0.0.0\".\n\n\nIf you wish to create a new version manually, the process is:\n\n1. Update version info in `pyproject.toml`\n\n2. Configure your Pypi credentials [with Poetry][creds]\n\n3. Build and publish the version:\n\n```sh\npoetry publish --build\n```\n\n\n\n## License\n\nThis repository is available under the permissive BSD license, making it easy and safe to incorporate in your own libraries.\n\nPull and feature requests welcome. Online editing in GitHub is possible (and easy!).\n\n[jq]: https://stedolan.github.io/jq/\n[mu]: https://pymupdf.readthedocs.io/\n[asc]: https://en.wikipedia.org/wiki/Ascender_(typography)\n[creds]: https://python-poetry.org/docs/repositories/#configuring-credentials\n",
"bugtrack_url": null,
"license": "BSD-2-Clause",
"summary": "A library and microservice to find bad redactions in PDFs",
"version": "0.3.4",
"project_urls": {
"Change Log": "https://github.com/freelawproject/x-ray/blob/main/CHANGES.md",
"Documentation": "https://github.com/freelawproject/x-ray/blob/main/README.md",
"Funding": "https://www.courtlistener.com/donate/?referrer=pypi-xray",
"Homepage": "https://free.law/projects/x-ray",
"Issues": "https://github.com/freelawproject/x-ray/issues",
"Organisation Homepage": "https://free.law/",
"Repository": "https://github.com/freelawproject/x-ray"
},
"split_keywords": [
"legal",
"courts",
"redactions"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5ac4be4c73bc49617e85dc7d3128239af50348a65f6c87bcec007da47d1389ef",
"md5": "8845cc933760ee74a9bce67ab4913873",
"sha256": "205e7c37b423dbe139d78250f030e9d8cd78f9d317d12597e8fc8b31831e7ec3"
},
"downloads": -1,
"filename": "x_ray-0.3.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8845cc933760ee74a9bce67ab4913873",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 11761,
"upload_time": "2023-05-17T16:37:14",
"upload_time_iso_8601": "2023-05-17T16:37:14.791544Z",
"url": "https://files.pythonhosted.org/packages/5a/c4/be4c73bc49617e85dc7d3128239af50348a65f6c87bcec007da47d1389ef/x_ray-0.3.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3f2b029fe5eebaffef4a768c3d73128e41a54a5b70b9b142e56ab93035934b9f",
"md5": "b0d793ace737f820296c0ed1cc251a59",
"sha256": "d4202fd8e7c7b08edff8c2a65efba3a329a60b83710efc9f5a814903f53f8551"
},
"downloads": -1,
"filename": "x_ray-0.3.4.tar.gz",
"has_sig": false,
"md5_digest": "b0d793ace737f820296c0ed1cc251a59",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 12817,
"upload_time": "2023-05-17T16:37:17",
"upload_time_iso_8601": "2023-05-17T16:37:17.053134Z",
"url": "https://files.pythonhosted.org/packages/3f/2b/029fe5eebaffef4a768c3d73128e41a54a5b70b9b142e56ab93035934b9f/x_ray-0.3.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-17 16:37:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "freelawproject",
"github_project": "x-ray",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "x-ray"
}
JSON
