# step-pipeline
This library serves as a thin layer on top of Hail Batch (and eventually other execution engines)
to provide the following features:
- Skip steps that already ran and completed successfully (eg. their output files already exist and are newer than their input files). This is done prior to submitting the pipeline to the execution engine.
- Make it easier to switch between different ways of localizing files (copy, gcsfuse, etc.) with minimal changes to pipeline code
- Automatically define pipeline command-line args to force or skip execution of paticular steps
- Use a config file to store various pipeline parameters (eg. cloud project account, etc.)
- Add commands for sending pipeline completion notifications to Slack
- Add commands for profiling pipeline steps while they are running, and saving this info to external storage
- Generate a diagram of the pipeline execution graph (DAG)
Another longer-term goal is to allow the same pipeline definition to be submitted to different
backends with minimal changes - including Hail Batch, Terra/Cromwell, and others.
This can only work for pipelines that use the subset of workflow definition features that is shared
across the relevant execution engines.
---
### Installation
To install the `step-pipeline` library, run:
```
python3 -m pip install step-pipeline
```
---
### Docs
[API docs](https://bw2.github.io/step-pipeline/docs/_build/html/index.html)
---
Raw data
{
"_id": null,
"home_page": "https://github.com/broadinstitute/step_pipeline",
"name": "step-pipeline",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "pipelines,workflows,hail,Batch,cromwell,Terra,dsub,SGE",
"author": "",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/69/c8/e9e10c60d35b3171887456f70124fccf7ebc803ed74bb2920365421b119d/step_pipeline-0.2.7.tar.gz",
"platform": null,
"description": "# step-pipeline\n\n\nThis library serves as a thin layer on top of Hail Batch (and eventually other execution engines) \nto provide the following features:\n\n- Skip steps that already ran and completed successfully (eg. their output files already exist and are newer than their input files). This is done prior to submitting the pipeline to the execution engine.\n- Make it easier to switch between different ways of localizing files (copy, gcsfuse, etc.) with minimal changes to pipeline code\n- Automatically define pipeline command-line args to force or skip execution of paticular steps\n- Use a config file to store various pipeline parameters (eg. cloud project account, etc.)\n- Add commands for sending pipeline completion notifications to Slack\n- Add commands for profiling pipeline steps while they are running, and saving this info to external storage \n- Generate a diagram of the pipeline execution graph (DAG)\n\nAnother longer-term goal is to allow the same pipeline definition to be submitted to different \nbackends with minimal changes - including Hail Batch, Terra/Cromwell, and others. \nThis can only work for pipelines that use the subset of workflow definition features that is shared\nacross the relevant execution engines. \n\n---\n\n### Installation\n\nTo install the `step-pipeline` library, run:\n```\npython3 -m pip install step-pipeline\n```\n\n---\n\n### Docs\n\n[API docs](https://bw2.github.io/step-pipeline/docs/_build/html/index.html)\n\n---\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Pipeline library that simplifies creation of pipelines that run on top of hail Batch and other compute enviornments",
"version": "0.2.7",
"project_urls": {
"Homepage": "https://github.com/broadinstitute/step_pipeline"
},
"split_keywords": [
"pipelines",
"workflows",
"hail",
"batch",
"cromwell",
"terra",
"dsub",
"sge"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fa14acb4c857b006f90f34063861b2f4ce29f0832f244add38ca827eba606536",
"md5": "afceaac7020d9e6a08c611f3c8ac96e0",
"sha256": "c94ad376af880d783c2dbbb5c21a02e0c47a0ad29beed7ebb9b4f374c893e845"
},
"downloads": -1,
"filename": "step_pipeline-0.2.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "afceaac7020d9e6a08c611f3c8ac96e0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 44617,
"upload_time": "2023-11-27T17:32:51",
"upload_time_iso_8601": "2023-11-27T17:32:51.884929Z",
"url": "https://files.pythonhosted.org/packages/fa/14/acb4c857b006f90f34063861b2f4ce29f0832f244add38ca827eba606536/step_pipeline-0.2.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "69c8e9e10c60d35b3171887456f70124fccf7ebc803ed74bb2920365421b119d",
"md5": "65a376ae033e9ac65c1ae40ca62cb8f8",
"sha256": "8924891b74f434be9ef4a084c13f24f385bec7d01649b9431fd3a133fc93057f"
},
"downloads": -1,
"filename": "step_pipeline-0.2.7.tar.gz",
"has_sig": false,
"md5_digest": "65a376ae033e9ac65c1ae40ca62cb8f8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 39805,
"upload_time": "2023-11-27T17:32:53",
"upload_time_iso_8601": "2023-11-27T17:32:53.764595Z",
"url": "https://files.pythonhosted.org/packages/69/c8/e9e10c60d35b3171887456f70124fccf7ebc803ed74bb2920365421b119d/step_pipeline-0.2.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-27 17:32:53",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "broadinstitute",
"github_project": "step_pipeline",
"github_not_found": true,
"lcname": "step-pipeline"
}
JSON
