# poeblix
Poetry Plugin that adds various features that extend the `poetry` command such as building wheel files with locked dependencies, and validations of wheel/docker containers.
Supports poetry versions `1.2+`
# Overview
These contain custom poetry plugins that enable functionality not available in the official distribution of poetry. These include:
1. Using the Lock file to build a wheel file with pinned dependencies
2. Support for data_files (like with setup.py) such as jupyter extensions or font files
3. Validating a wheel file is consistent with dependencies specified in pyproject.toml/poetry.lock
4. Validating a docker container's `pip freeze` contains dependencies as specified in pyproject.toml/poetry.lock
These are not supported in Poetry due to debate in the community: https://github.com/python-poetry/poetry/issues/890, https://github.com/python-poetry/poetry/issues/4013, https://github.com/python-poetry/poetry/issues/2778
## Deterministic builds and environments
Poetry guarantees deterministic installations and environments thanks
to the `poetry.lock` file, which is where it stores the exact versions
of all the dependencies needed to install a package. However, this doesn't
occurs when wheel or package artifacts are build using `poetry build`
command.
To build a package, poetry uses the direct dependencies set in the
`pyproject.toml` and not all the other dependencies required to install
a package. For example, if `pyproject.toml` defines `pandas = "1.4.2"`
as dependency but `poetry.lock` also says that `pandas` requires of
`numpy-1.22.4`, poetry will build a package with `pandas` as dependency
but not with `numpy`.
Another problem that exists is that `pyproject.toml` can contain dependencies
with ranges of versions while `poetry.lock` has pinned versions. For instance,
if `pyproject.toml` has as dependency `pandas = ">=1.3"` but `poetry.lock`
sets `pandas-1.4.2`, poetry will build a package with the dependency
`Requires-Dist: pandas (>=0.1.3,<0.2.0)`. When the package is installed,
the resolver will install the newest package of `pandas` which its version
number is greater than or equal to `0.1.3` and lower than `0.2.0`.
Summing this up, the same python package created with `poetry build` and
installed several times won't install the same dependencies, making impossible
to have deterministic installations.
This plugin solves these problems building python packages that use the
dependencies defined in the `poetry.lock`.
# How to Use
### Prerequisite
Poetry Plugins are only supported in 1.2.0+ which, at the moment (5/29/22), can only be installed when using the [new poetry installer](https://python-poetry.org/docs/#installation)
```commandline
# You can update poetry using
poetry self update
```
## Installation
You can add the plugin via poetry's CLI:
```commandline
poetry self add poeblix
```
_For <= 1.2_:
```commandline
poetry plugin add poeblix
```
Or install directly from source/wheel, then add with the same above command using the absolute path to the built dist
To update the plugin:
```commandline
# Update to latest
poetry self add poeblix@latest
# Update to specific version
poetry self add poeblix==<version>
```
You should now see the blix* commands if you run `poetry list`
## Usage
1. To build a wheel from your package (pyproject.toml dependencies have precedence over poetry.lock ones, by default)
```commandline
poetry blixbuild
# Note: Options below are also available as part of the `blixvalidatewheel` and `blixvalidatedocker` commands
# To disable using lock file for building wheel and only use pyproject.toml
poetry blixbuild --no-lock
# Uses lock dependencies only which are pinned to exact versions, instead of pyproject.toml
poetry blixbuild --only-lock
# Specify additional dependency groups to include as Requires-Dist in the wheel
poetry blixbuild --with-groups=dev,integ,etc.
```
2. Validate a wheel file has consistent dependencies and data_files as specified in pyproject.toml/poetry.lock
```commandline
poetry blixvalidatewheel <path-to-wheel>
# Disable using lock file for validation
poetry blixvalidatewheel --no-lock <path-to-wheel>
```
_Note: this validates consistency in both directions_
3. Validate a docker container contains dependencies in a `pip freeze` as specified in pyproject.toml/poetry.lock
```commandline
poetry blixvalidatedocker <docker-container-ID>
# Disable using lock file for validation
poetry blixvalidatedocker --no-lock <docker-container-ID>
```
_Note: this only validates the docker container contains dependencies in the project, but not the other direction_
Here's an example series of commands to start up a temporary docker container using its tag, validate it, then stop the temporary container
```
# This will output the newly running container id
docker run --entrypoint=bash -it -d <docker-image-tag>
# Then validate the running docker container, and stop it when done
poetry blixvalidatedocker <container-id>
docker stop <container-id>
```
4. Adding data_files to pyproject.toml to mimic data_files in setup.py:
```yaml
...
[tool.blix.data]
data_files = [
{ destination = "share/data/", from = [ "data_files/test.txt", "data_files/anotherfile" ] },
{ destination = "share/data/threes", from = [ "data_files/athirdfile" ] }
]
...
```
data_files should be under the `[tool.blix.data]` category and is a list of objects, each containing the `destination` data folder, and a `from` list of files to add to the destination data folder.
_Note: the destination is a relative path that installs data to relative to the [installation prefix](https://docs.python.org/3/distutils/setupscript.html#installing-additional-files)_
Example: https://github.com/spoorn/poeblix/blob/main/test/positive_cases/happy_case_example/pyproject.toml
5. For more help on each command, use the --help argument
```commandline
poetry blixbuild --help
poetry blixvalidatewheel --help
poetry blixvalidatedocker --help
```
# Development
```bash
# Make a virtual environment on Python 3.9
# If using virtualenvwrapper, run `mkvirtualenv -p python3.9 venv`
virtualenv -p python3.9 venv
# Or activate existing virtualenv
# If using virtualenvwrapper, run `workon venv`
source venv/bin/activate
# installs the plugin in editable mode for easier testing via `poetry install`
./devtool bootstrap
# Lint checks
./devtool lint
# Tests
./devtool test
# Run all checks and tests
./devtool all
```
**plugins.py** : contains our plugin that adds the `poetry blixbuild` command for building our wheel file
**validatewheel.py**: adds a `poetry blixvalidatewheel` command that validates a wheel file contains the Required Dist as specified in pyproject.toml/poetry.lock
**validatedocker.py** : adds a command that validates a docker file contains dependencies as specified in pyproject.toml and poetry.lock. This does *NOT* validate that they are exactly matching, but rather that all dependencies in pyproject.toml/poetry.lock exist in the docker container on the correct versions. The docker image may contain more extra dependencies
Raw data
{
"_id": null,
"home_page": "https://github.com/spoorn/poeblix",
"name": "poeblix",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "",
"keywords": "poeblix,utility,poetry,plugin,poetry-plugin,wheel,distribution,data_files",
"author": "spoorn",
"author_email": "spookump@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/2d/46/6abbfd3fd4a4e5093d24069e5e1aca9f75f0fb44634679d464a6b0ed2c3f/poeblix-0.10.tar.gz",
"platform": null,
"description": "# poeblix\nPoetry Plugin that adds various features that extend the `poetry` command such as building wheel files with locked dependencies, and validations of wheel/docker containers.\n\nSupports poetry versions `1.2+`\n\n# Overview\nThese contain custom poetry plugins that enable functionality not available in the official distribution of poetry. These include:\n\n1. Using the Lock file to build a wheel file with pinned dependencies\n2. Support for data_files (like with setup.py) such as jupyter extensions or font files\n3. Validating a wheel file is consistent with dependencies specified in pyproject.toml/poetry.lock\n4. Validating a docker container's `pip freeze` contains dependencies as specified in pyproject.toml/poetry.lock\n\nThese are not supported in Poetry due to debate in the community: https://github.com/python-poetry/poetry/issues/890, https://github.com/python-poetry/poetry/issues/4013, https://github.com/python-poetry/poetry/issues/2778\n\n## Deterministic builds and environments\n\nPoetry guarantees deterministic installations and environments thanks\nto the `poetry.lock` file, which is where it stores the exact versions\nof all the dependencies needed to install a package. However, this doesn't\noccurs when wheel or package artifacts are build using `poetry build`\ncommand.\n\nTo build a package, poetry uses the direct dependencies set in the\n`pyproject.toml` and not all the other dependencies required to install\na package. For example, if `pyproject.toml` defines `pandas = \"1.4.2\"`\nas dependency but `poetry.lock` also says that `pandas` requires of\n`numpy-1.22.4`, poetry will build a package with `pandas` as dependency\nbut not with `numpy`.\n\nAnother problem that exists is that `pyproject.toml` can contain dependencies\nwith ranges of versions while `poetry.lock` has pinned versions. For instance,\nif `pyproject.toml` has as dependency `pandas = \">=1.3\"` but `poetry.lock`\nsets `pandas-1.4.2`, poetry will build a package with the dependency\n`Requires-Dist: pandas (>=0.1.3,<0.2.0)`. When the package is installed,\nthe resolver will install the newest package of `pandas` which its version\nnumber is greater than or equal to `0.1.3` and lower than `0.2.0`.\n\nSumming this up, the same python package created with `poetry build` and\ninstalled several times won't install the same dependencies, making impossible\nto have deterministic installations.\n\nThis plugin solves these problems building python packages that use the\ndependencies defined in the `poetry.lock`.\n\n\n# How to Use\n\n### Prerequisite\n\nPoetry Plugins are only supported in 1.2.0+ which, at the moment (5/29/22), can only be installed when using the [new poetry installer](https://python-poetry.org/docs/#installation)\n\n```commandline\n# You can update poetry using\npoetry self update\n```\n\n## Installation\n\nYou can add the plugin via poetry's CLI:\n\n```commandline\npoetry self add poeblix\n```\n\n_For <= 1.2_:\n\n```commandline\npoetry plugin add poeblix\n```\n\nOr install directly from source/wheel, then add with the same above command using the absolute path to the built dist\n\nTo update the plugin:\n\n```commandline\n# Update to latest\npoetry self add poeblix@latest\n\n# Update to specific version\npoetry self add poeblix==<version>\n```\n\nYou should now see the blix* commands if you run `poetry list`\n\n## Usage\n\n1. To build a wheel from your package (pyproject.toml dependencies have precedence over poetry.lock ones, by default)\n\n```commandline\npoetry blixbuild\n\n# Note: Options below are also available as part of the `blixvalidatewheel` and `blixvalidatedocker` commands\n\n# To disable using lock file for building wheel and only use pyproject.toml\npoetry blixbuild --no-lock\n\n# Uses lock dependencies only which are pinned to exact versions, instead of pyproject.toml\npoetry blixbuild --only-lock\n\n# Specify additional dependency groups to include as Requires-Dist in the wheel\npoetry blixbuild --with-groups=dev,integ,etc.\n```\n\n\n2. Validate a wheel file has consistent dependencies and data_files as specified in pyproject.toml/poetry.lock\n\n```commandline\npoetry blixvalidatewheel <path-to-wheel>\n\n# Disable using lock file for validation\npoetry blixvalidatewheel --no-lock <path-to-wheel>\n```\n\n_Note: this validates consistency in both directions_\n\n3. Validate a docker container contains dependencies in a `pip freeze` as specified in pyproject.toml/poetry.lock\n\n```commandline\npoetry blixvalidatedocker <docker-container-ID>\n\n# Disable using lock file for validation\npoetry blixvalidatedocker --no-lock <docker-container-ID>\n```\n\n_Note: this only validates the docker container contains dependencies in the project, but not the other direction_\n\nHere's an example series of commands to start up a temporary docker container using its tag, validate it, then stop the temporary container\n\n```\n# This will output the newly running container id\ndocker run --entrypoint=bash -it -d <docker-image-tag>\n\n# Then validate the running docker container, and stop it when done\npoetry blixvalidatedocker <container-id>\ndocker stop <container-id>\n```\n\n4. Adding data_files to pyproject.toml to mimic data_files in setup.py:\n\n```yaml\n...\n\n[tool.blix.data]\ndata_files = [\n { destination = \"share/data/\", from = [ \"data_files/test.txt\", \"data_files/anotherfile\" ] },\n { destination = \"share/data/threes\", from = [ \"data_files/athirdfile\" ] }\n]\n\n...\n```\n\ndata_files should be under the `[tool.blix.data]` category and is a list of objects, each containing the `destination` data folder, and a `from` list of files to add to the destination data folder.\n\n_Note: the destination is a relative path that installs data to relative to the [installation prefix](https://docs.python.org/3/distutils/setupscript.html#installing-additional-files)_\n\nExample: https://github.com/spoorn/poeblix/blob/main/test/positive_cases/happy_case_example/pyproject.toml\n\n5. For more help on each command, use the --help argument\n\n```commandline\npoetry blixbuild --help\npoetry blixvalidatewheel --help\npoetry blixvalidatedocker --help\n```\n\n# Development\n\n```bash\n# Make a virtual environment on Python 3.9\n# If using virtualenvwrapper, run `mkvirtualenv -p python3.9 venv`\nvirtualenv -p python3.9 venv\n\n# Or activate existing virtualenv\n# If using virtualenvwrapper, run `workon venv`\nsource venv/bin/activate\n\n# installs the plugin in editable mode for easier testing via `poetry install`\n./devtool bootstrap\n\n# Lint checks\n./devtool lint\n\n# Tests\n./devtool test\n\n# Run all checks and tests\n./devtool all\n```\n\n**plugins.py** : contains our plugin that adds the `poetry blixbuild` command for building our wheel file\n\n**validatewheel.py**: adds a `poetry blixvalidatewheel` command that validates a wheel file contains the Required Dist as specified in pyproject.toml/poetry.lock\n\n**validatedocker.py** : adds a command that validates a docker file contains dependencies as specified in pyproject.toml and poetry.lock. This does *NOT* validate that they are exactly matching, but rather that all dependencies in pyproject.toml/poetry.lock exist in the docker container on the correct versions. The docker image may contain more extra dependencies\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Poetry plugin that adds support for building wheel files using the poetry.lock file, and data_files just like in setup.py",
"version": "0.10",
"project_urls": {
"Homepage": "https://github.com/spoorn/poeblix",
"Repository": "https://github.com/spoorn/poeblix"
},
"split_keywords": [
"poeblix",
"utility",
"poetry",
"plugin",
"poetry-plugin",
"wheel",
"distribution",
"data_files"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "18c4f94859919e818c9f00cc542f58e3fc080b4ab8601ce8ecfd73a25515403a",
"md5": "4e476de59e9a00905d9911bc001832fb",
"sha256": "6c44f767b14e4a37c0c74cb45a119bfef98a42fba11889d8c72cc9dfd073c176"
},
"downloads": -1,
"filename": "poeblix-0.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4e476de59e9a00905d9911bc001832fb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 14895,
"upload_time": "2023-10-31T05:28:30",
"upload_time_iso_8601": "2023-10-31T05:28:30.331252Z",
"url": "https://files.pythonhosted.org/packages/18/c4/f94859919e818c9f00cc542f58e3fc080b4ab8601ce8ecfd73a25515403a/poeblix-0.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2d466abbfd3fd4a4e5093d24069e5e1aca9f75f0fb44634679d464a6b0ed2c3f",
"md5": "de87949523aaf9a4613aeb38c74070c0",
"sha256": "c7e9a4f6130eb0a5f49c6280271d87943b245a01e77a9ed25809b1a3741b3a8c"
},
"downloads": -1,
"filename": "poeblix-0.10.tar.gz",
"has_sig": false,
"md5_digest": "de87949523aaf9a4613aeb38c74070c0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 14255,
"upload_time": "2023-10-31T05:28:33",
"upload_time_iso_8601": "2023-10-31T05:28:33.628821Z",
"url": "https://files.pythonhosted.org/packages/2d/46/6abbfd3fd4a4e5093d24069e5e1aca9f75f0fb44634679d464a6b0ed2c3f/poeblix-0.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-31 05:28:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "spoorn",
"github_project": "poeblix",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "poeblix"
}
JSON
