Yorkshire
---------
🐶 Yorkshire is your friend; Yorkshire checks Python's requirements files for a
possible dependency confusion.
Note if `PEP-708: Extending the Repository API to Mitigate Dependency Confusion
Attacks
<https://discuss.python.org/t/pep-708-extending-the-repository-api-to-mitigate-dependency-confusion-attacks/24179>`__
gets accepted, you do not need to use Yorkshire anymore.
Yorkshire was developed to perform scans on all the possible files that can
manipulate with Python package index configuration. The scan will reveal
configuration of multiple Python package indexes to check for a possible
dependency confusion. By reviewing results, users can prevent from issues like
the one with `PyTorch's torchvision
<https://pytorch.org/blog/compromised-nightly-dependency/>`__. The tool does
not report whether there is an actual dependency confusion (that would require
more in-depth analysis), but whether there is a possibility for a dependency
confusion - whether packages could be consumed from multiple Python package
indexes.
The tool checks whether there are configured any extra index URLs in
corresponding files. Currently, there are supported the following installation
methods and their files:
* `PDM <https://pdm.fming.dev/>`__ - ``pyproject.toml`` and ``pdm.lock``
* `Pipenv <https://pipenv.pypa.io/en/latest/>`__ - ``Pipfile`` and ``Pipfile.lock``
* `Poetry <https://python-poetry.org/>`__ - ``pyproject.toml`` (poetry.lock is not sufficient for a dependency confusion detection)
* `pip <https://pypi.org/project/pip/>`__ - raw ``requirements.txt``
* `pip-tools <https://pypi.org/project/pip-tools/>`__ - ``requirements.txt`` and ``requirements.in``
* `setup.cfg <https://setuptools.pypa.io/en/latest/userguide/declarative_config.html>`__ - the tool parses setuptool's ``setup.cfg`` configuration
* `setup.py <https://setuptools.pypa.io/>`__ - the tool statically analyzes sources of the ``setup.py`` script
Installation
============
Yorkshire is available on PyPI:
.. code-block:: console
pip install yorkshire
yorkshire --help
To install the tool from this Git repository, issue the following command from
the root of the ``yorkshire`` directory:
.. code-block:: console
python3 -m venv venv
source venv/bin/activate
pip install -e .
yorkshire --help
Usage
=====
.. code-block:: console
yorkshire detect DIR|FILE|URL
* if the argument supplied is a directory, Yorkshire traverses the whole
directory tree and checks files present
* if the argument supplied is a file, Yorkshire performs analysis on the given
file
* if the argument supplied is URL, Yorksire downloads the referenced file and
perfoms analysis (the file is deleted as the analysis finishes)
See ``--help`` for more information:
.. code-block:: console
yorkshire --help
yorkshire detect --help
Example Run
===========
The tool can be run on a single requirements file and check Python package indexes configured:
.. code-block:: console
$ yorkshire detect tests/data/requirements_files/fail/pipfile/Pipfile
2023-03-10 14:07:01,640 [24252] INFO yorkshire._lib: Performing detection in Pipfile file located at 'tests/data/requirements_files/fail/pipfile'
2023-03-10 14:07:01,640 [24252] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/pipfile/Pipfile' states one or multiple Python package indexes: ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cpu']
Or, it can traverse a directory tree and report findings:
.. code-block:: console
$ yorkshire detect tests/data/requirements_files/fail
2023-03-10 14:08:39,811 [24502] INFO yorkshire._lib: Performing detection in setup.py file located at 'tests/data/requirements_files/fail/setup_py'
2023-03-10 14:08:39,811 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/setup_py/setup.py' uses dependency links
2023-03-10 14:08:39,811 [24502] INFO yorkshire._lib: Performing detection in pyproject.toml file located at 'tests/data/requirements_files/fail/pyproject_toml/poetry'
2023-03-10 14:08:39,811 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/pyproject_toml/poetry/pyproject.toml' uses an explicitly configured Poetry source: ['https://test.pypi.org/simple/']
2023-03-10 14:08:39,811 [24502] INFO yorkshire._lib: Performing detection in pyproject.toml file located at 'tests/data/requirements_files/fail/pyproject_toml/pdm'
2023-03-10 14:08:39,811 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/pyproject_toml/pdm/pyproject.toml' uses an explicitly configured PDM source: ['https://test.pypi.org/simple']
2023-03-10 14:08:39,811 [24502] INFO yorkshire._lib: Performing detection in setup.cfg file located at 'tests/data/requirements_files/fail/setup_cfg/01'
2023-03-10 14:08:39,811 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/setup_cfg/01/setup.cfg' uses dependency links: http://peak.telecommunity.com/snapshots/
2023-03-10 14:08:39,812 [24502] INFO yorkshire._lib: Performing detection in requirements.in file located at 'tests/data/requirements_files/fail/requirements/02'
2023-03-10 14:08:39,812 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/requirements/02/requirements.in' states one or multiple extra index URLs: ['https://download.pytorch.org/whl/cpu']
2023-03-10 14:08:39,812 [24502] INFO yorkshire._lib: Performing detection in requirements.in file located at 'tests/data/requirements_files/fail/requirements/01'
2023-03-10 14:08:39,812 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/requirements/01/requirements.in' states --find-links: ['https://github.com/NVIDIA/Torch-TensorRT/releases']
2023-03-10 14:08:39,813 [24502] INFO yorkshire._lib: Performing detection in pdm.lock file located at 'tests/data/requirements_files/fail/pdm_lock'
2023-03-10 14:08:39,813 [24502] WARNING yorkshire._lib: Package 'certifi 2021.10.8' is not consumed from PyPI: https://files.custom.org/packages/37/45/946c02767aabb873146011e665728b680884cd8fe70dde973c640e45b775/certifi-2021.10.8-py2.py3-none-any.whl
2023-03-10 14:08:39,813 [24502] INFO yorkshire._lib: Performing detection in Pipfile file located at 'tests/data/requirements_files/fail/pipfile'
2023-03-10 14:08:39,813 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/pipfile/Pipfile' states one or multiple Python package indexes: ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cpu']
2023-03-10 14:08:39,813 [24502] INFO yorkshire._lib: Performing detection in Pipfile.lock file located at 'tests/data/requirements_files/fail/pipfile_lock'
2023-03-10 14:08:39,813 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/pipfile_lock/Pipfile.lock' states one or multiple Python package indexes: ['https://pypi.org/simple', 'https://localhost:8080/simple']
The tool can also check a file referenced by URL (any query parameters are intentionally discarded):
.. code-block:: console
$ yorkshire detect https://raw.githubusercontent.com/pytorch/pytorch/master/requirements.txt
2023-03-10 14:11:45,774 [24832] INFO yorkshire._lib: Performing detection in requirements.txt file located at 'https://raw.githubusercontent.com/pytorch/pytorch/master'
$ echo $?
0
Using as Yorkshire as a library
===============================
Yorkshire can be used as a library in your application:
.. code-block:: python
>>> import yorkshire
>>> path = os.getcwd()
>>> yorkshire.detect(path)
>>> yorkshire.detect_file(path)
>>> help(yorkshire.detect)
>>> help(yorkshire.detect_file)
License
=======
See the LICENSE file.
Raw data
{
"_id": null,
"home_page": "https://github.com/DataDog/yorkshire",
"name": "yorkshire",
"maintainer": "Fridolin Pokorny",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "fridolin.pokorny@datadoghq.com",
"keywords": "packaging,pip,dependencies,dependency-management,utilities,dependency-confusion,guarddog",
"author": "Fridolin Pokorny",
"author_email": "fridolin.pokorny@datadoghq.com",
"download_url": "https://files.pythonhosted.org/packages/9d/64/b6d716faffc5fc988ddadafd1ffbec3d9900c4a1613b08bf2337d37f6bd9/yorkshire-0.0.0.tar.gz",
"platform": null,
"description": "Yorkshire\n---------\n\n\ud83d\udc36 Yorkshire is your friend; Yorkshire checks Python's requirements files for a\npossible dependency confusion.\n\nNote if `PEP-708: Extending the Repository API to Mitigate Dependency Confusion\nAttacks\n<https://discuss.python.org/t/pep-708-extending-the-repository-api-to-mitigate-dependency-confusion-attacks/24179>`__\ngets accepted, you do not need to use Yorkshire anymore.\n\nYorkshire was developed to perform scans on all the possible files that can\nmanipulate with Python package index configuration. The scan will reveal\nconfiguration of multiple Python package indexes to check for a possible\ndependency confusion. By reviewing results, users can prevent from issues like\nthe one with `PyTorch's torchvision\n<https://pytorch.org/blog/compromised-nightly-dependency/>`__. The tool does\nnot report whether there is an actual dependency confusion (that would require\nmore in-depth analysis), but whether there is a possibility for a dependency\nconfusion - whether packages could be consumed from multiple Python package\nindexes.\n\nThe tool checks whether there are configured any extra index URLs in\ncorresponding files. Currently, there are supported the following installation\nmethods and their files:\n\n* `PDM <https://pdm.fming.dev/>`__ - ``pyproject.toml`` and ``pdm.lock``\n* `Pipenv <https://pipenv.pypa.io/en/latest/>`__ - ``Pipfile`` and ``Pipfile.lock``\n* `Poetry <https://python-poetry.org/>`__ - ``pyproject.toml`` (poetry.lock is not sufficient for a dependency confusion detection)\n* `pip <https://pypi.org/project/pip/>`__ - raw ``requirements.txt``\n* `pip-tools <https://pypi.org/project/pip-tools/>`__ - ``requirements.txt`` and ``requirements.in``\n* `setup.cfg <https://setuptools.pypa.io/en/latest/userguide/declarative_config.html>`__ - the tool parses setuptool's ``setup.cfg`` configuration\n* `setup.py <https://setuptools.pypa.io/>`__ - the tool statically analyzes sources of the ``setup.py`` script\n\nInstallation\n============\n\nYorkshire is available on PyPI:\n\n.. code-block:: console\n\n pip install yorkshire\n yorkshire --help\n\nTo install the tool from this Git repository, issue the following command from\nthe root of the ``yorkshire`` directory:\n\n.. code-block:: console\n\n python3 -m venv venv\n source venv/bin/activate\n pip install -e .\n yorkshire --help\n\nUsage\n=====\n\n.. code-block:: console\n\n yorkshire detect DIR|FILE|URL\n\n* if the argument supplied is a directory, Yorkshire traverses the whole\n directory tree and checks files present\n* if the argument supplied is a file, Yorkshire performs analysis on the given\n file\n* if the argument supplied is URL, Yorksire downloads the referenced file and\n perfoms analysis (the file is deleted as the analysis finishes)\n\nSee ``--help`` for more information:\n\n.. code-block:: console\n\n yorkshire --help\n\n yorkshire detect --help\n\nExample Run\n===========\n\nThe tool can be run on a single requirements file and check Python package indexes configured:\n\n.. code-block:: console\n\n $ yorkshire detect tests/data/requirements_files/fail/pipfile/Pipfile\n 2023-03-10 14:07:01,640 [24252] INFO yorkshire._lib: Performing detection in Pipfile file located at 'tests/data/requirements_files/fail/pipfile'\n 2023-03-10 14:07:01,640 [24252] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/pipfile/Pipfile' states one or multiple Python package indexes: ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cpu']\n\nOr, it can traverse a directory tree and report findings:\n\n.. code-block:: console\n\n $ yorkshire detect tests/data/requirements_files/fail\n 2023-03-10 14:08:39,811 [24502] INFO yorkshire._lib: Performing detection in setup.py file located at 'tests/data/requirements_files/fail/setup_py'\n 2023-03-10 14:08:39,811 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/setup_py/setup.py' uses dependency links\n 2023-03-10 14:08:39,811 [24502] INFO yorkshire._lib: Performing detection in pyproject.toml file located at 'tests/data/requirements_files/fail/pyproject_toml/poetry'\n 2023-03-10 14:08:39,811 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/pyproject_toml/poetry/pyproject.toml' uses an explicitly configured Poetry source: ['https://test.pypi.org/simple/']\n 2023-03-10 14:08:39,811 [24502] INFO yorkshire._lib: Performing detection in pyproject.toml file located at 'tests/data/requirements_files/fail/pyproject_toml/pdm'\n 2023-03-10 14:08:39,811 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/pyproject_toml/pdm/pyproject.toml' uses an explicitly configured PDM source: ['https://test.pypi.org/simple']\n 2023-03-10 14:08:39,811 [24502] INFO yorkshire._lib: Performing detection in setup.cfg file located at 'tests/data/requirements_files/fail/setup_cfg/01'\n 2023-03-10 14:08:39,811 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/setup_cfg/01/setup.cfg' uses dependency links: http://peak.telecommunity.com/snapshots/\n 2023-03-10 14:08:39,812 [24502] INFO yorkshire._lib: Performing detection in requirements.in file located at 'tests/data/requirements_files/fail/requirements/02'\n 2023-03-10 14:08:39,812 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/requirements/02/requirements.in' states one or multiple extra index URLs: ['https://download.pytorch.org/whl/cpu']\n 2023-03-10 14:08:39,812 [24502] INFO yorkshire._lib: Performing detection in requirements.in file located at 'tests/data/requirements_files/fail/requirements/01'\n 2023-03-10 14:08:39,812 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/requirements/01/requirements.in' states --find-links: ['https://github.com/NVIDIA/Torch-TensorRT/releases']\n 2023-03-10 14:08:39,813 [24502] INFO yorkshire._lib: Performing detection in pdm.lock file located at 'tests/data/requirements_files/fail/pdm_lock'\n 2023-03-10 14:08:39,813 [24502] WARNING yorkshire._lib: Package 'certifi 2021.10.8' is not consumed from PyPI: https://files.custom.org/packages/37/45/946c02767aabb873146011e665728b680884cd8fe70dde973c640e45b775/certifi-2021.10.8-py2.py3-none-any.whl\n 2023-03-10 14:08:39,813 [24502] INFO yorkshire._lib: Performing detection in Pipfile file located at 'tests/data/requirements_files/fail/pipfile'\n 2023-03-10 14:08:39,813 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/pipfile/Pipfile' states one or multiple Python package indexes: ['https://pypi.org/simple', 'https://download.pytorch.org/whl/cpu']\n 2023-03-10 14:08:39,813 [24502] INFO yorkshire._lib: Performing detection in Pipfile.lock file located at 'tests/data/requirements_files/fail/pipfile_lock'\n 2023-03-10 14:08:39,813 [24502] WARNING yorkshire._lib: File 'tests/data/requirements_files/fail/pipfile_lock/Pipfile.lock' states one or multiple Python package indexes: ['https://pypi.org/simple', 'https://localhost:8080/simple']\n\nThe tool can also check a file referenced by URL (any query parameters are intentionally discarded):\n\n.. code-block:: console\n\n $ yorkshire detect https://raw.githubusercontent.com/pytorch/pytorch/master/requirements.txt\n 2023-03-10 14:11:45,774 [24832] INFO yorkshire._lib: Performing detection in requirements.txt file located at 'https://raw.githubusercontent.com/pytorch/pytorch/master'\n $ echo $?\n 0\n\nUsing as Yorkshire as a library\n===============================\n\nYorkshire can be used as a library in your application:\n\n.. code-block:: python\n\n >>> import yorkshire\n >>> path = os.getcwd()\n >>> yorkshire.detect(path)\n >>> yorkshire.detect_file(path)\n >>> help(yorkshire.detect)\n >>> help(yorkshire.detect_file)\n\nLicense\n=======\n\nSee the LICENSE file.\n\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Yorkshire is your friend who checks requirements files for a possible dependency confusion.",
"version": "0.0.0",
"split_keywords": [
"packaging",
"pip",
"dependencies",
"dependency-management",
"utilities",
"dependency-confusion",
"guarddog"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3a91c6d50fa2763a5e72ec38d66660e49a506c5ac67751b660a6dd06366e1646",
"md5": "33c31e0e377605a4de09f6e8f087e00e",
"sha256": "f0c6f630bd1b57274883bdac427f239eb10a29c262db9379dea11c4cbabba2c7"
},
"downloads": -1,
"filename": "yorkshire-0.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "33c31e0e377605a4de09f6e8f087e00e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 11985,
"upload_time": "2023-03-13T08:10:00",
"upload_time_iso_8601": "2023-03-13T08:10:00.367603Z",
"url": "https://files.pythonhosted.org/packages/3a/91/c6d50fa2763a5e72ec38d66660e49a506c5ac67751b660a6dd06366e1646/yorkshire-0.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9d64b6d716faffc5fc988ddadafd1ffbec3d9900c4a1613b08bf2337d37f6bd9",
"md5": "1fe3060a6f15e233e7d50d82af4c59d3",
"sha256": "e181e514acf15283037023df59b9a77b3d099d6495a9244a8d4d7e0ded0a1fc4"
},
"downloads": -1,
"filename": "yorkshire-0.0.0.tar.gz",
"has_sig": false,
"md5_digest": "1fe3060a6f15e233e7d50d82af4c59d3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 13970,
"upload_time": "2023-03-13T08:10:02",
"upload_time_iso_8601": "2023-03-13T08:10:02.889718Z",
"url": "https://files.pythonhosted.org/packages/9d/64/b6d716faffc5fc988ddadafd1ffbec3d9900c4a1613b08bf2337d37f6bd9/yorkshire-0.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-13 08:10:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "DataDog",
"github_project": "yorkshire",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "yorkshire"
}
JSON
