<p align="center"></p>
<h2 align="center">Staging</h2>
<p align="center">
<a href="https://github.com/rgryta/Staging/actions/workflows/main.yml"><img alt="Build" src="https://github.com/rgryta/Staging/actions/workflows/main.yml/badge.svg?branch=main"></a>
<a href="https://pypi.org/project/staging/"><img alt="PyPI" src="https://img.shields.io/pypi/v/staging"></a>
<a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
<a href="https://github.com/PyCQA/pylint"><img alt="pylint" src="https://img.shields.io/badge/linting-pylint-yellowgreen"></a>
<a href="https://github.com/rgryta/NoPrint"><img alt="NoPrint" src="https://img.shields.io/badge/NoPrint-enabled-blueviolet"></a>
</p>
## About
Automate your commands - be it for environment setup or in CI/CD pipeline. Parallelize execution, and make it easier to handle dependent commands.
## Requirements
This package requires `tomlkit` and `dataclass-wizard` packages.
## Usage
### Full description
Staging is configurable in `pyproject.toml` file and is accessible through CLI interface.
In order to use staging, you have to define steps and then stages. Stages are built from steps.
Let's see example below (which uses all available options):
```toml
[tool.staging.steps]
# Test
get_test_packages = {execute = "command to get test packages", output="test_packages"}
coverage = { prepare = "coverage run -m pytest -xv {packages}", execute = "coverage report {flags}", cleanup = "coverage erase", output = "coverage_result", format = {flags = "flags", packages= "test_packages"} }
# Lint
clean = {execute = "command to execute", success_codes=[0,1,2,3,4,5]}
isort = { execute = "isort {flags} . tests", format = {flags = "flags"}}
black = { execute = "black {flags} . tests", format = {flags = "flags"}}
pylint = { execute = "pylint {package} tests", format = {package = "package"}}
noprint = { execute = "noprint -ve {package} tests", format = {package = "package"}, error_codes=[1,2]}
finish = { execute = "finishing command" }
[tool.staging.stages.format]
description = "Format code"
format = {package="staging"}
steps = [
{step = "isort"},
{step = "black"},
]
[tool.staging.stages.lint]
description = "Check linting"
format = {flags="--check", package="staging"}
steps = [
{step = "clean", continue_on_failure=true},
{parallel = {steps = ["isort", "black", "pylint", "noprint"]}, continue_on_failure=true},
{step = "finish"},
]
[tool.staging.stages.test]
description = "Test the package"
format = {flags="-m --fail-under=30"}
steps = [
{step = "get_test_packages"},
{step = "coverage"},
]
```
Here we have defined 7 different steps: `get_test_packages`, `coverage`, `clean`, and so on.
These 7 steps are later used in 3 stages: `test`, `lint`, and `format`.
Let's say we execute `staging format`. What happens?
1. Stage context is updated with formatter {key=`package`, value=`staging`}.
2. Step `isort` is executed. Command `isort {flags} . tests` is formatted using stage formatter. Since flags is empty,
the final command is simply `isort . tests`.
3. Step `black` is executed. This works the same as `isort` above.
Now let's say we execute `staging lint`.
1. Context has formatter for value `package`, but also for `flags`.
2. Step `clean` is executed. It might fail, but we don't care and instead the process is continued.
3. Steps `isort`, `black`, `pylint` and `noprint` are all executed in a thread pool due to being specified in a `parallel` block.
This time however, even though some of the steps are the same as in the `staging format`, we now have a `flags` formatter defined.
Thanks to this, we now have executed `isort --check . tests` command that would verify if the formatting was properly applied.
Due to how steps are configured - specified custom success and error codes - the parallel block would fail (one command will crash others).
However, the parallel block allows to specify `continue_on_error` as well, just like step block.
4. As such, `finish` block is also executed.
Last but not least, `staging test`.
1. Similarly to previous stages, we set up the stage formatter context.
2. Step `get_test_packages` outputs a package name to stdout. This value will be saved under `test_packages` key in the staging context.
3. Step `coverage` has a bit of a different structure, it has `prepare` and `cleanup` blocks on top of typical `execute`.
Prepare is executed before `execute`. However, it's not taken into account when writing to output. Moreover, it will always
fail when status code is different to 0. Block called `cleanup` is just like what it says it is. It's a cleanup process executed after `execute`.
It will be executed no matter what is the final status of the `execute` command.
You can also chain your stages in one command. Keep in mind however that formatting contexts are defined on **per stage** basis.
So if you execute `staging format lint` - formatting context from stage `format` won't be propagated to stage `lint`.
Running stages in one execution is also performed in serial manner - no parallelism.
### Example
For real-life examples, see `pyproject.toml` file for this package.
## Development
### Installation
Install virtual environment and check_bump package in editable mode with dev dependencies.
```bash
python -m venv venv
source venv/bin/activate
pip install -e .[dev]
```
### How to?
Automate as much as we can, see configuration in `pyproject.toml` file to see what are the flags used.
```bash
staging format # Reformat the code
staging lint # Check for linting issues
staging test # Run unit tests and coverage report
```
Raw data
{
"_id": null,
"home_page": null,
"name": "staging",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "CI, CD, stage, step, pipeline, workflow, automation, orchestration, execution, staging",
"author": null,
"author_email": "Radoslaw Gryta <radek.gryta@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/22/fb/1c4bd619dde709edd1647658d47dc159dc889287f2cc21b476cd3eda7eef/staging-0.0.2.tar.gz",
"platform": null,
"description": "<p align=\"center\"></p>\n<h2 align=\"center\">Staging</h2>\n<p align=\"center\">\n<a href=\"https://github.com/rgryta/Staging/actions/workflows/main.yml\"><img alt=\"Build\" src=\"https://github.com/rgryta/Staging/actions/workflows/main.yml/badge.svg?branch=main\"></a>\n<a href=\"https://pypi.org/project/staging/\"><img alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/staging\"></a>\n<a href=\"https://github.com/psf/black\"><img alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"></a>\n<a href=\"https://github.com/PyCQA/pylint\"><img alt=\"pylint\" src=\"https://img.shields.io/badge/linting-pylint-yellowgreen\"></a>\n<a href=\"https://github.com/rgryta/NoPrint\"><img alt=\"NoPrint\" src=\"https://img.shields.io/badge/NoPrint-enabled-blueviolet\"></a>\n</p>\n\n## About\n\nAutomate your commands - be it for environment setup or in CI/CD pipeline. Parallelize execution, and make it easier to handle dependent commands.\n\n## Requirements\n\nThis package requires `tomlkit` and `dataclass-wizard` packages.\n\n## Usage\n\n### Full description\nStaging is configurable in `pyproject.toml` file and is accessible through CLI interface.\n\nIn order to use staging, you have to define steps and then stages. Stages are built from steps.\n\nLet's see example below (which uses all available options):\n\n```toml\n[tool.staging.steps]\n# Test\nget_test_packages = {execute = \"command to get test packages\", output=\"test_packages\"}\ncoverage = { prepare = \"coverage run -m pytest -xv {packages}\", execute = \"coverage report {flags}\", cleanup = \"coverage erase\", output = \"coverage_result\", format = {flags = \"flags\", packages= \"test_packages\"} }\n# Lint\nclean = {execute = \"command to execute\", success_codes=[0,1,2,3,4,5]}\nisort = { execute = \"isort {flags} . tests\", format = {flags = \"flags\"}}\nblack = { execute = \"black {flags} . tests\", format = {flags = \"flags\"}}\npylint = { execute = \"pylint {package} tests\", format = {package = \"package\"}}\nnoprint = { execute = \"noprint -ve {package} tests\", format = {package = \"package\"}, error_codes=[1,2]}\nfinish = { execute = \"finishing command\" }\n\n[tool.staging.stages.format]\ndescription = \"Format code\"\nformat = {package=\"staging\"}\nsteps = [\n {step = \"isort\"},\n {step = \"black\"},\n]\n\n[tool.staging.stages.lint]\ndescription = \"Check linting\"\nformat = {flags=\"--check\", package=\"staging\"}\nsteps = [\n {step = \"clean\", continue_on_failure=true},\n {parallel = {steps = [\"isort\", \"black\", \"pylint\", \"noprint\"]}, continue_on_failure=true},\n {step = \"finish\"},\n]\n\n[tool.staging.stages.test]\ndescription = \"Test the package\"\nformat = {flags=\"-m --fail-under=30\"}\nsteps = [\n {step = \"get_test_packages\"},\n {step = \"coverage\"},\n]\n```\n\nHere we have defined 7 different steps: `get_test_packages`, `coverage`, `clean`, and so on.\nThese 7 steps are later used in 3 stages: `test`, `lint`, and `format`.\n\nLet's say we execute `staging format`. What happens?\n1. Stage context is updated with formatter {key=`package`, value=`staging`}.\n2. Step `isort` is executed. Command `isort {flags} . tests` is formatted using stage formatter. Since flags is empty, \nthe final command is simply `isort . tests`.\n3. Step `black` is executed. This works the same as `isort` above.\n\nNow let's say we execute `staging lint`.\n1. Context has formatter for value `package`, but also for `flags`.\n2. Step `clean` is executed. It might fail, but we don't care and instead the process is continued.\n3. Steps `isort`, `black`, `pylint` and `noprint` are all executed in a thread pool due to being specified in a `parallel` block. \nThis time however, even though some of the steps are the same as in the `staging format`, we now have a `flags` formatter defined.\nThanks to this, we now have executed `isort --check . tests` command that would verify if the formatting was properly applied. \nDue to how steps are configured - specified custom success and error codes - the parallel block would fail (one command will crash others). \nHowever, the parallel block allows to specify `continue_on_error` as well, just like step block.\n4. As such, `finish` block is also executed.\n\nLast but not least, `staging test`.\n1. Similarly to previous stages, we set up the stage formatter context.\n2. Step `get_test_packages` outputs a package name to stdout. This value will be saved under `test_packages` key in the staging context.\n3. Step `coverage` has a bit of a different structure, it has `prepare` and `cleanup` blocks on top of typical `execute`. \nPrepare is executed before `execute`. However, it's not taken into account when writing to output. Moreover, it will always\nfail when status code is different to 0. Block called `cleanup` is just like what it says it is. It's a cleanup process executed after `execute`.\nIt will be executed no matter what is the final status of the `execute` command.\n\nYou can also chain your stages in one command. Keep in mind however that formatting contexts are defined on **per stage** basis.\nSo if you execute `staging format lint` - formatting context from stage `format` won't be propagated to stage `lint`.\nRunning stages in one execution is also performed in serial manner - no parallelism.\n\n### Example\n\nFor real-life examples, see `pyproject.toml` file for this package.\n\n## Development\n\n### Installation\n\nInstall virtual environment and check_bump package in editable mode with dev dependencies.\n\n```bash\npython -m venv venv\nsource venv/bin/activate\npip install -e .[dev]\n```\n\n\n### How to?\n\nAutomate as much as we can, see configuration in `pyproject.toml` file to see what are the flags used.\n\n```bash\nstaging format # Reformat the code\nstaging lint # Check for linting issues\nstaging test # Run unit tests and coverage report\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "Execute pipeline stages and steps",
"version": "0.0.2",
"project_urls": {
"Bug Tracker": "https://github.com/rgryta/Staging/issues",
"Homepage": "https://github.com/rgryta/Staging"
},
"split_keywords": [
"ci",
" cd",
" stage",
" step",
" pipeline",
" workflow",
" automation",
" orchestration",
" execution",
" staging"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c5dbca8e01894f3b3bfa6d267d329324a1fa41137aede94e701b63708643c5a1",
"md5": "b25639d3cc4c23dee07e196ad365abd9",
"sha256": "7ed1ac96e566d2aeb6f336c9466d129a3b2f24a238c0fbee0ebacda96067c26c"
},
"downloads": -1,
"filename": "staging-0.0.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b25639d3cc4c23dee07e196ad365abd9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 9316,
"upload_time": "2024-04-15T23:26:49",
"upload_time_iso_8601": "2024-04-15T23:26:49.670285Z",
"url": "https://files.pythonhosted.org/packages/c5/db/ca8e01894f3b3bfa6d267d329324a1fa41137aede94e701b63708643c5a1/staging-0.0.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "22fb1c4bd619dde709edd1647658d47dc159dc889287f2cc21b476cd3eda7eef",
"md5": "1f96fa5bf106c1f4b3e8cf75d75564db",
"sha256": "f5e0c75c870a659e32d9d4fede3c6a23b4d547f82f30b2910be41bd902471c17"
},
"downloads": -1,
"filename": "staging-0.0.2.tar.gz",
"has_sig": false,
"md5_digest": "1f96fa5bf106c1f4b3e8cf75d75564db",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 10806,
"upload_time": "2024-04-15T23:26:51",
"upload_time_iso_8601": "2024-04-15T23:26:51.400938Z",
"url": "https://files.pythonhosted.org/packages/22/fb/1c4bd619dde709edd1647658d47dc159dc889287f2cc21b476cd3eda7eef/staging-0.0.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-15 23:26:51",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "rgryta",
"github_project": "Staging",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "staging"
}
JSON
