| Name | abi3audit JSON |
| Version |
0.0.20
JSON |
| download |
| home_page | None |
| Summary | Scans Python wheels for abi3 violations and inconsistencies |
| upload_time | 2025-01-06 22:07:17 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# abi3audit
<!--- @begin-badges@ --->
[](https://github.com/pypa/abi3audit/actions/workflows/tests.yml)
[](https://pypi.org/project/abi3audit)
[](https://repology.org/project/python:abi3audit/versions)
<!--- @end-badges@ --->
*[Read the Trail of Bits blog post about how we find bugs with `abi3audit`!](https://blog.trailofbits.com/2022/11/15/python-wheels-abi-abi3audit/)*
`abi3audit` scans Python extensions for `abi3` violations and inconsistencies.
It can scan individual (unpackaged) shared objects, packaged wheels, or entire
package version histories.
This project is maintained in part by [Trail of Bits](https://trailofbits.com).
This is not an official Trail of Bits product.
## Index
* [Motivation](#motivation)
* [Installation](#installation)
* [Usage](#usage)
* [Examples](#examples)
* [Limitations](#limitations)
* [Licensing](#licensing)
## Motivation
CPython (the reference implementation of Python) defines a stable API and corresponding
ABI ("`abi3`"). In principle, any CPython extension can be built against this
API/ABI and will remain forward compatible with future minor versions of CPython.
In other words: if you build against the stable ABI for Python 3.5, your
extension should work without modification on Python 3.9.
The stable ABI simplifies packaging of CPython extensions, since the packager
only needs to build one `abi3` wheel that targets the minimum supported Python
version.
To signal that a Python wheel contains `abi3`-compatible extensions,
the Python packaging ecosystem uses the `abi3` wheel tag, e.g.:
```text
pyrage-1.0.1-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl
```
Unfortunately, there is **no actual enforcement** of `abi3` compliance
in Python extensions at install or runtime: a wheel (or independent
shared object) that is tagged as `abi3` is assumed to be `abi3`, but
is not validated in any way.
To make matters worse, there is **no formal connection** between the flag
([`--py-limited-api`](https://setuptools.pypa.io/en/latest/userguide/ext_modules.html#setuptools.Extension))
that controls wheel tagging and the build macros
([`Py_LIMITED_API`](https://docs.python.org/3/c-api/stable.html#c.Py_LIMITED_API))
that actually lock a Python extension into a specific `abi3` version.
As a result: it is very easy to compile a Python extension for the wrong `abi3`
version, or to tag a Python wheel as `abi3` without actually compiling it
as `abi3`-compatible.
This has serious security and reliability implications: non-stable parts
of the CPython ABI can change between minor versions, resulting in crashes,
unpredictable behavior, or potentially exploitable memory corruption when
a Python extension incorrectly assumes the parameters of a function
or layout of a structure.
## Installation
`abi3audit` is available via `pip`:
```bash
pip install abi3audit
```
## Usage
You can run `abi3audit` as a standalone program, or via `python -m abi3audit`:
```bash
abi3audit --help
python -m abi3audit --help
```
Top-level:
<!-- @begin-abi3audit-help@ -->
```console
usage: abi3audit [-h] [-V] [--debug] [-v] [-R] [-o OUTPUT] [-s] [-S]
[--assume-minimum-abi3 ASSUME_MINIMUM_ABI3]
SPEC [SPEC ...]
Scans Python extensions for abi3 violations and inconsistencies
positional arguments:
SPEC the files or other dependency specs to scan
options:
-h, --help show this help message and exit
-V, --version show program's version number and exit
--debug emit debug statements; this setting also overrides
`ABI3AUDIT_LOGLEVEL` and is equivalent to setting it
to `debug`
-v, --verbose give more output, including pretty-printed results for
each audit step
-R, --report generate a JSON report; uses --output
-o OUTPUT, --output OUTPUT
the path to write the JSON report to (default: stdout)
-s, --summary always output a summary even if there are no
violations/ABI version mismatches
-S, --strict fail the entire audit if an individual audit step
fails
--assume-minimum-abi3 ASSUME_MINIMUM_ABI3
assumed abi3 version (3.x, with x>=2) if it cannot be
detected
```
<!-- @end-abi3audit-help@ -->
### Examples
Audit a single shared object, wheel, or PyPI package:
```bash
# audit a local copy of an abi3 extension
abi3audit procmaps.abi3.so
# audit a local copy of an abi3 wheel
abi3audit procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl
# audit every abi3 wheel for the package 'procmaps' on PyPI
abi3audit procmaps
```
Show additional detail (pretty tables and individual violations) while auditing:
```bash
abi3audit procmaps --verbose
```
yields:
```console
[17:59:46] 👎 procmaps:
procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl: procmaps.abi3.so
uses the Python 3.10 ABI, but is tagged for the Python 3.6 ABI
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ Symbol ┃ Version ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ PyUnicode_AsUTF8AndSize │ 3.10 │
└─────────────────────────┴─────────┘
[17:59:47] 💁 procmaps: 2 extensions scanned; 1 ABI version mismatches and 0
ABI violations found
```
Generate a JSON report for each input:
```bash
abi3audit procmaps --report | python -m json.tool
```
yields:
```json
{
"specs": {
"procmaps": {
"kind": "package",
"package": {
"procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl": [
{
"name": "procmaps.abi3.so",
"result": {
"is_abi3": true,
"is_abi3_baseline_compatible": false,
"baseline": "3.6",
"computed": "3.10",
"non_abi3_symbols": [],
"future_abi3_objects": {
"PyUnicode_AsUTF8AndSize": "3.10"
}
}
}
],
"procmaps-0.6.1-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl": [
{
"name": "procmaps.abi3.so",
"result": {
"is_abi3": true,
"is_abi3_baseline_compatible": true,
"baseline": "3.7",
"computed": "3.7",
"non_abi3_symbols": [],
"future_abi3_objects": {}
}
}
]
}
}
}
}
```
## Limitations
`abi3audit` is a *best-effort* tool, with some of the same limitations as
[`auditwheel`](https://github.com/pypa/auditwheel). In particular:
* `abi3audit` cannot check for *dynamic* abi3 violations, such as an extension
that calls [`dlsym(3)`](https://man7.org/linux/man-pages/man3/dlsym.3.html)
to invoke a non-abi3 function at runtime.
* `abi3audit` can confirm the presence of abi3-compatible symbols, but does
not have an exhaustive list of abi3-*incompatible* symbols. Instead, it looks
for violations by looking for symbols that start with `Py_` or `_Py_` that
are not in the abi3 compatibility list. This is *unlikely* to result in false
positives, but *could* if an extension incorrectly uses those reserved
prefixes.
* When auditing a "bare" shared object (e.g. `foo.abi3.so`), `abi3audit` cannot
assume anything about the minimum *intended* abi3 version. Instead, it
defaults to the lowest known abi3 version (`abi3-cp32`) and warns on any
version mismatches (e.g., a symbol that was only stabilized in 3.6).
This can result in false positives, so users are encouraged to audit entire
wheels or packages instead (since they contain the sufficient metadata).
* `abi3audit` considers the abi3 version when a symbol was *stabilized*,
not *introduced*. In other words: `abi3audit` will produce a warning
when an `abi3-cp36` extension contains a function stabilized in 3.7, even
if that function was introduced in 3.6. This is *not* a false positive
(it is an ABI version mismatch), but it's *generally* not a source of bugs.
* `abi3audit` checks both the "local" and "external" symbols for each extension,
for formats that support both. It does this to catch symbols that have been
inlined, such as `_Py_DECREF`. However, if the extension's symbol table
has been stripped, these may be missed.
## Licensing
`abi3audit` is licensed under the MIT license.
`abi3audit` includes ASN.1 and Mach-O parsers generated from
definitions provided by the [Kaitai Struct](https://kaitai.io/) project.
These vendored parsers are licensed by the Kaitai Struct authors under the MIT
license.
Raw data
{
"_id": null,
"home_page": null,
"name": "abi3audit",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": null,
"author_email": "William Woodruff <william@trailofbits.com>",
"download_url": "https://files.pythonhosted.org/packages/f7/b1/6f30cdfcf77d41b4a53e431acf458dc94e553697ea769a5397b3634afc43/abi3audit-0.0.20.tar.gz",
"platform": null,
"description": "# abi3audit\n\n<!--- @begin-badges@ --->\n[](https://github.com/pypa/abi3audit/actions/workflows/tests.yml)\n[](https://pypi.org/project/abi3audit)\n[](https://repology.org/project/python:abi3audit/versions)\n<!--- @end-badges@ --->\n\n*[Read the Trail of Bits blog post about how we find bugs with `abi3audit`!](https://blog.trailofbits.com/2022/11/15/python-wheels-abi-abi3audit/)*\n\n`abi3audit` scans Python extensions for `abi3` violations and inconsistencies.\n\nIt can scan individual (unpackaged) shared objects, packaged wheels, or entire\npackage version histories.\n\n\n\nThis project is maintained in part by [Trail of Bits](https://trailofbits.com).\nThis is not an official Trail of Bits product.\n\n## Index\n\n* [Motivation](#motivation)\n* [Installation](#installation)\n* [Usage](#usage)\n * [Examples](#examples)\n* [Limitations](#limitations)\n* [Licensing](#licensing)\n\n## Motivation\n\nCPython (the reference implementation of Python) defines a stable API and corresponding\nABI (\"`abi3`\"). In principle, any CPython extension can be built against this\nAPI/ABI and will remain forward compatible with future minor versions of CPython.\nIn other words: if you build against the stable ABI for Python 3.5, your\nextension should work without modification on Python 3.9.\n\nThe stable ABI simplifies packaging of CPython extensions, since the packager\nonly needs to build one `abi3` wheel that targets the minimum supported Python\nversion.\n\nTo signal that a Python wheel contains `abi3`-compatible extensions,\nthe Python packaging ecosystem uses the `abi3` wheel tag, e.g.:\n\n```text\npyrage-1.0.1-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl\n```\n\nUnfortunately, there is **no actual enforcement** of `abi3` compliance\nin Python extensions at install or runtime: a wheel (or independent\nshared object) that is tagged as `abi3` is assumed to be `abi3`, but\nis not validated in any way.\n\nTo make matters worse, there is **no formal connection** between the flag\n([`--py-limited-api`](https://setuptools.pypa.io/en/latest/userguide/ext_modules.html#setuptools.Extension))\nthat controls wheel tagging and the build macros\n([`Py_LIMITED_API`](https://docs.python.org/3/c-api/stable.html#c.Py_LIMITED_API))\nthat actually lock a Python extension into a specific `abi3` version.\n\nAs a result: it is very easy to compile a Python extension for the wrong `abi3`\nversion, or to tag a Python wheel as `abi3` without actually compiling it\nas `abi3`-compatible.\n\nThis has serious security and reliability implications: non-stable parts\nof the CPython ABI can change between minor versions, resulting in crashes,\nunpredictable behavior, or potentially exploitable memory corruption when\na Python extension incorrectly assumes the parameters of a function\nor layout of a structure.\n\n## Installation\n\n`abi3audit` is available via `pip`:\n\n```bash\npip install abi3audit\n```\n\n## Usage\n\nYou can run `abi3audit` as a standalone program, or via `python -m abi3audit`:\n\n```bash\nabi3audit --help\npython -m abi3audit --help\n```\n\nTop-level:\n\n<!-- @begin-abi3audit-help@ -->\n```console\nusage: abi3audit [-h] [-V] [--debug] [-v] [-R] [-o OUTPUT] [-s] [-S]\n [--assume-minimum-abi3 ASSUME_MINIMUM_ABI3]\n SPEC [SPEC ...]\n\nScans Python extensions for abi3 violations and inconsistencies\n\npositional arguments:\n SPEC the files or other dependency specs to scan\n\noptions:\n -h, --help show this help message and exit\n -V, --version show program's version number and exit\n --debug emit debug statements; this setting also overrides\n `ABI3AUDIT_LOGLEVEL` and is equivalent to setting it\n to `debug`\n -v, --verbose give more output, including pretty-printed results for\n each audit step\n -R, --report generate a JSON report; uses --output\n -o OUTPUT, --output OUTPUT\n the path to write the JSON report to (default: stdout)\n -s, --summary always output a summary even if there are no\n violations/ABI version mismatches\n -S, --strict fail the entire audit if an individual audit step\n fails\n --assume-minimum-abi3 ASSUME_MINIMUM_ABI3\n assumed abi3 version (3.x, with x>=2) if it cannot be\n detected\n```\n<!-- @end-abi3audit-help@ -->\n\n### Examples\n\nAudit a single shared object, wheel, or PyPI package:\n\n```bash\n# audit a local copy of an abi3 extension\nabi3audit procmaps.abi3.so\n\n# audit a local copy of an abi3 wheel\nabi3audit procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl\n\n# audit every abi3 wheel for the package 'procmaps' on PyPI\nabi3audit procmaps\n```\n\nShow additional detail (pretty tables and individual violations) while auditing:\n\n```bash\nabi3audit procmaps --verbose\n```\n\nyields:\n\n```console\n[17:59:46] \ud83d\udc4e procmaps:\n procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl: procmaps.abi3.so\n uses the Python 3.10 ABI, but is tagged for the Python 3.6 ABI\n \u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n \u2503 Symbol \u2503 Version \u2503\n \u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n \u2502 PyUnicode_AsUTF8AndSize \u2502 3.10 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n[17:59:47] \ud83d\udc81 procmaps: 2 extensions scanned; 1 ABI version mismatches and 0\n ABI violations found\n```\n\nGenerate a JSON report for each input:\n\n```bash\nabi3audit procmaps --report | python -m json.tool\n```\n\nyields:\n\n```json\n{\n \"specs\": {\n \"procmaps\": {\n \"kind\": \"package\",\n \"package\": {\n \"procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl\": [\n {\n \"name\": \"procmaps.abi3.so\",\n \"result\": {\n \"is_abi3\": true,\n \"is_abi3_baseline_compatible\": false,\n \"baseline\": \"3.6\",\n \"computed\": \"3.10\",\n \"non_abi3_symbols\": [],\n \"future_abi3_objects\": {\n \"PyUnicode_AsUTF8AndSize\": \"3.10\"\n }\n }\n }\n ],\n \"procmaps-0.6.1-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl\": [\n {\n \"name\": \"procmaps.abi3.so\",\n \"result\": {\n \"is_abi3\": true,\n \"is_abi3_baseline_compatible\": true,\n \"baseline\": \"3.7\",\n \"computed\": \"3.7\",\n \"non_abi3_symbols\": [],\n \"future_abi3_objects\": {}\n }\n }\n ]\n }\n }\n }\n}\n```\n\n## Limitations\n\n`abi3audit` is a *best-effort* tool, with some of the same limitations as\n[`auditwheel`](https://github.com/pypa/auditwheel). In particular:\n\n* `abi3audit` cannot check for *dynamic* abi3 violations, such as an extension\n that calls [`dlsym(3)`](https://man7.org/linux/man-pages/man3/dlsym.3.html)\n to invoke a non-abi3 function at runtime.\n\n* `abi3audit` can confirm the presence of abi3-compatible symbols, but does\n not have an exhaustive list of abi3-*incompatible* symbols. Instead, it looks\n for violations by looking for symbols that start with `Py_` or `_Py_` that\n are not in the abi3 compatibility list. This is *unlikely* to result in false\n positives, but *could* if an extension incorrectly uses those reserved\n prefixes.\n\n* When auditing a \"bare\" shared object (e.g. `foo.abi3.so`), `abi3audit` cannot\n assume anything about the minimum *intended* abi3 version. Instead, it\n defaults to the lowest known abi3 version (`abi3-cp32`) and warns on any\n version mismatches (e.g., a symbol that was only stabilized in 3.6).\n This can result in false positives, so users are encouraged to audit entire\n wheels or packages instead (since they contain the sufficient metadata).\n\n* `abi3audit` considers the abi3 version when a symbol was *stabilized*,\n not *introduced*. In other words: `abi3audit` will produce a warning\n when an `abi3-cp36` extension contains a function stabilized in 3.7, even\n if that function was introduced in 3.6. This is *not* a false positive\n (it is an ABI version mismatch), but it's *generally* not a source of bugs.\n\n* `abi3audit` checks both the \"local\" and \"external\" symbols for each extension,\n for formats that support both. It does this to catch symbols that have been\n inlined, such as `_Py_DECREF`. However, if the extension's symbol table\n has been stripped, these may be missed.\n\n## Licensing\n\n`abi3audit` is licensed under the MIT license.\n\n`abi3audit` includes ASN.1 and Mach-O parsers generated from\ndefinitions provided by the [Kaitai Struct](https://kaitai.io/) project.\nThese vendored parsers are licensed by the Kaitai Struct authors under the MIT\nlicense.\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Scans Python wheels for abi3 violations and inconsistencies",
"version": "0.0.20",
"project_urls": {
"Homepage": "https://pypi.org/project/abi3audit/",
"Issues": "https://github.com/pypa/abi3audit/issues",
"Source": "https://github.com/pypa/abi3audit"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "24687b79274efd98dbaef56acc928a864220b7e7806afac415ed63e7a2fd2e95",
"md5": "6dde47679d83474eccfac61fa914c93d",
"sha256": "1c5b36d1cff9ccd7f99336fa4f694f23eb6e15cb03418a385f110453c80d7fd0"
},
"downloads": -1,
"filename": "abi3audit-0.0.20-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6dde47679d83474eccfac61fa914c93d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 31097,
"upload_time": "2025-01-06T22:07:12",
"upload_time_iso_8601": "2025-01-06T22:07:12.474899Z",
"url": "https://files.pythonhosted.org/packages/24/68/7b79274efd98dbaef56acc928a864220b7e7806afac415ed63e7a2fd2e95/abi3audit-0.0.20-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f7b16f30cdfcf77d41b4a53e431acf458dc94e553697ea769a5397b3634afc43",
"md5": "abb7513cca8087ab15760d69c561e28c",
"sha256": "b31d9f0dde0cd46c13db13f71372235318dfb535a8df5f40839711b0968fd45f"
},
"downloads": -1,
"filename": "abi3audit-0.0.20.tar.gz",
"has_sig": false,
"md5_digest": "abb7513cca8087ab15760d69c561e28c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 30289,
"upload_time": "2025-01-06T22:07:17",
"upload_time_iso_8601": "2025-01-06T22:07:17.080079Z",
"url": "https://files.pythonhosted.org/packages/f7/b1/6f30cdfcf77d41b4a53e431acf458dc94e553697ea769a5397b3634afc43/abi3audit-0.0.20.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-06 22:07:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pypa",
"github_project": "abi3audit",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "abi3audit"
}
