| Name | sbpack JSON |
| Version |
2022.3.16
JSON |
| download |
| home_page | |
| Summary | Command line tool to upload and download CWL to and from SB powered platforms. |
| upload_time | 2022-03-16 14:20:10 |
| maintainer | Seven Bridges |
| docs_url | None |
| author | Seven Bridges |
| requires_python | >=3.7 |
| license | |
| keywords |
seven
bridges
cwl
common
workflow
language
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# sbpack
[](https://pypi.org/project/sbpack/)
Upload (`sbpack`) and download (`sbpull`) CWL apps to/from any Seven Bridges powered platform.
Resolves linked processes, schemadefs and `$include`s and `$import`s.
## Installation
(It is good practice to install Python programs in a virtual environment.
[pipx] is a very effective tool for installing command line Python tools in isolated environments)
[pipx]: https://github.com/pipxproject/pipx
`sbpack` needs Python 3.7 or later
```
pip3 install pipx # in case you don't have pipx
pipx ensurepath # ensures CLI application directory is on your $PATH
```
### Install latest release on pypi
```bash
pipx install sbpack
# or pipx upgrade
```
### Install latest (unreleased) code
```
pipx install git+https://github.com/rabix/sbpack.git
# use pipx upgrade ... if upgrading an existing install
```
## Usage
```
$ sbpack -h
sbpack v2020.10.05
Upload CWL apps to any Seven Bridges powered platform
(c) Seven Bridges 2020
usage: sbpack [-h] [--filter-non-sbg-tags] profile appid cwl_path
positional arguments:
profile SB platform profile as set in the SB API credentials file.
appid Takes the form {user}/{project}/{app_id}.
cwl_path Path or URL to the main CWL file to be uploaded.
optional arguments:
-h, --help show this help message and exit
--filter-non-sbg-tags
Filter out custom tags that are not 'sbg:'
```
## Uploading workflows defined remotely
`sbpack` handles local paths and remote URLs in a principled manner. This means that
`sbpack` will handle packing and uploading a local workflow that links to a remote workflow
which itself has linked workflows. It will therefore also handle packing a fully
remote workflow.
For example, to pack and upload the workflow located at `https://github.com/Duke-GCB/GGR-cwl/blob/master/v1.0/ATAC-seq_pipeline/pipeline-se.cwl`
go to the `raw` button and use that URL, like:
```bash
sbpack sbg kghosesbg/sbpla-31744/ATAC-seq-pipeline-se https://raw.githubusercontent.com/Duke-GCB/GGR-cwl/master/v1.0/ATAC-seq_pipeline/pipeline-se.cwl
```
## Local packing
```
cwlpack <cwl> > packed.cwl
$ cwlpack -h
usage: cwlpack [-h] [--filter-non-sbg-tags] [--json] cwl_path
positional arguments:
cwl_path Path or URL to the main CWL file to be uploaded.
optional arguments:
-h, --help show this help message and exit
--filter-non-sbg-tags
Filter out custom tags that are not 'sbg:'
--json Output in JSON format, not YAML.
```
The `cwlpack` utility allows you to pack a workflow and print it out on `stdout` instead of
uploading it to a SB platform.
## Side-note
As an interesting side note, packing a workflow can get around at least two `cwltool` bugs
[[1]][cwltoolbug1], [[2]][cwltoolbug2].
[cwltoolbug1]: https://github.com/common-workflow-language/cwltool/issues/1304
[cwltoolbug2]: https://github.com/common-workflow-language/cwltool/issues/1306
## Pulling (and unpacking)
`sbpull` will retrieve CWL from any SB powered platform and save it to local disk.
```bash
sbpull sbg admin/sbg-public-data/salmon-workflow-1-2-0/ salmon.cwl
```
With the `--unpack` option set, it will also explode the workflow recursively, extracting out each
sub-process into its own file.
```bash
sbpull sbg admin/sbg-public-data/salmon-workflow-1-2-0/ salmon.cwl --unpack
```
> This is useful if you want to use SB platform CWL with your own workflows. You can pull the relevant
CWL into your code repository and use it with the rest of your code. If you use the `--unpack` option
you can access the individual components of the SB CWL workflow separately.
### Pulling a particular revision
While
```bash
sbpull sbg admin/sbg-public-data/bismark-0-21-0/ bismark.cwl
```
will pull the latest version of Bismark on the platform,
```bash
sbpull sbg admin/sbg-public-data/bismark-0-21-0/2 bismark.cwl
```
will pull revision 2 of this tool
## Note on reversibility
**`sbpack` and `sbpull --unpack` are not textually reversible. The packed and unpacked CWL
representations are functionally identical, however if you `sbpack` a workflow, and
then `sbpull --unpack` it, they will look different.**
## Credentials file and profiles
If you use the SBG API you already have an API configuration file. If
not, you should create one. It is located in
`~/.sevenbridges/credentials`. ([Documentation][cred-doc])
[cred-doc]: https://docs.sevenbridges.com/docs/store-credentials-to-access-seven-bridges-client-applications-and-libraries
Briefly, each section in the SBG configuration file (e.g. `[cgc]`) is a
profile name and has two entries. The end-point and an authentication
token, which you get from your developer tab on the platform.
```
[sbg-us]
api_endpoint = https://api.sbgenomics.com/v2
auth_token = <dev token here>
[sbg-eu]
api_endpoint = https://eu-api.sbgenomics.com/v2
auth_token = <dev token here>
[sbg-china]
api_endpoint = https://api.sevenbridges.cn/v2
auth_token = <dev token here>
[cgc]
api_endpoint = https://cgc-api.sbgenomics.com/v2
auth_token = <dev token here>
[cavatica]
api_endpoint = https://cavatica-api.sbgenomics.com/v2
auth_token = <dev token here>
[nhlbi]
api_endpoint = https://api.sb.biodatacatalyst.nhlbi.nih.gov/v2
auth_token = <dev token here>
```
You can have several profiles on the same platform if, for example, you
are an enterprise user and you belong to several divisions. Please refer
to the API documentation for more detail.
### Reading credentials from env variables
Instead of using the credentials file, you can specify environment variables
`SB_API_ENDPOINT` and `SB_AUTH_TOKEN`. To use the env variables in `sbpack` simply
specify profile `.` in the command, e.g.
```bash
sbpack . kghosesbg/sbpla-31744/ATAC-seq-pipeline-se https://raw.githubusercontent.com/Duke-GCB/GGR-cwl/master/v1.0/ATAC-seq_pipeline/pipeline-se.cwl
```
By specifying `.` profile, `sbpack` will use env variables. If these are not found, the default profile
from the credentials file is used.
### Running the test suite
The pulling test requires two environment variables to be set
```
SB_AUTH_TOKEN
SB_API_ENDPOINT
```
Raw data
{
"_id": null,
"home_page": "",
"name": "sbpack",
"maintainer": "Seven Bridges",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "kaushik.ghose@sbgenomics.com",
"keywords": "seven bridges cwl common workflow language",
"author": "Seven Bridges",
"author_email": "kaushik.ghose@sbgenomics.com",
"download_url": "https://files.pythonhosted.org/packages/8b/75/45a182c84ee1239a4ecf5236e50067d9ff7dfc8082cd70b0f33ff60b23fa/sbpack-2022.3.16.tar.gz",
"platform": "POSIX",
"description": "# sbpack\n\n\n[](https://pypi.org/project/sbpack/)\n\nUpload (`sbpack`) and download (`sbpull`) CWL apps to/from any Seven Bridges powered platform. \nResolves linked processes, schemadefs and `$include`s and `$import`s.\n\n## Installation\n\n(It is good practice to install Python programs in a virtual environment. \n[pipx] is a very effective tool for installing command line Python tools in isolated environments)\n\n[pipx]: https://github.com/pipxproject/pipx\n\n`sbpack` needs Python 3.7 or later\n\n```\npip3 install pipx # in case you don't have pipx\npipx ensurepath # ensures CLI application directory is on your $PATH\n```\n\n### Install latest release on pypi\n```bash\npipx install sbpack\n# or pipx upgrade\n```\n\n### Install latest (unreleased) code\n```\npipx install git+https://github.com/rabix/sbpack.git\n# use pipx upgrade ... if upgrading an existing install\n```\n\n## Usage\n```\n$ sbpack -h\n\nsbpack v2020.10.05\nUpload CWL apps to any Seven Bridges powered platform\n(c) Seven Bridges 2020\n\nusage: sbpack [-h] [--filter-non-sbg-tags] profile appid cwl_path\n\npositional arguments:\n profile SB platform profile as set in the SB API credentials file.\n appid Takes the form {user}/{project}/{app_id}.\n cwl_path Path or URL to the main CWL file to be uploaded.\n\noptional arguments:\n -h, --help show this help message and exit\n --filter-non-sbg-tags\n Filter out custom tags that are not 'sbg:'\n```\n\n## Uploading workflows defined remotely\n\n`sbpack` handles local paths and remote URLs in a principled manner. This means that\n`sbpack` will handle packing and uploading a local workflow that links to a remote workflow\nwhich itself has linked workflows. It will therefore also handle packing a fully \nremote workflow.\n\nFor example, to pack and upload the workflow located at `https://github.com/Duke-GCB/GGR-cwl/blob/master/v1.0/ATAC-seq_pipeline/pipeline-se.cwl`\ngo to the `raw` button and use that URL, like:\n\n```bash\nsbpack sbg kghosesbg/sbpla-31744/ATAC-seq-pipeline-se https://raw.githubusercontent.com/Duke-GCB/GGR-cwl/master/v1.0/ATAC-seq_pipeline/pipeline-se.cwl\n``` \n\n## Local packing\n```\ncwlpack <cwl> > packed.cwl\n\n$ cwlpack -h\nusage: cwlpack [-h] [--filter-non-sbg-tags] [--json] cwl_path\n\npositional arguments:\n cwl_path Path or URL to the main CWL file to be uploaded.\n\noptional arguments:\n -h, --help show this help message and exit\n --filter-non-sbg-tags\n Filter out custom tags that are not 'sbg:'\n --json Output in JSON format, not YAML.\n```\n\nThe `cwlpack` utility allows you to pack a workflow and print it out on `stdout` instead of \nuploading it to a SB platform.\n\n\n## Side-note\nAs an interesting side note, packing a workflow can get around at least two `cwltool` bugs \n[[1]][cwltoolbug1], [[2]][cwltoolbug2].\n\n[cwltoolbug1]: https://github.com/common-workflow-language/cwltool/issues/1304\n[cwltoolbug2]: https://github.com/common-workflow-language/cwltool/issues/1306\n\n\n## Pulling (and unpacking)\n`sbpull` will retrieve CWL from any SB powered platform and save it to local disk. \n\n```bash\nsbpull sbg admin/sbg-public-data/salmon-workflow-1-2-0/ salmon.cwl\n```\n\nWith the `--unpack` option set, it will also explode the workflow recursively, extracting out each\nsub-process into its own file. \n\n```bash\nsbpull sbg admin/sbg-public-data/salmon-workflow-1-2-0/ salmon.cwl --unpack\n```\n\n> This is useful if you want to use SB platform CWL with your own workflows. You can pull the relevant\nCWL into your code repository and use it with the rest of your code. If you use the `--unpack` option \nyou can access the individual components of the SB CWL workflow separately.\n\n### Pulling a particular revision\n\nWhile\n```bash\nsbpull sbg admin/sbg-public-data/bismark-0-21-0/ bismark.cwl\n```\nwill pull the latest version of Bismark on the platform,\n\n```bash\nsbpull sbg admin/sbg-public-data/bismark-0-21-0/2 bismark.cwl\n```\nwill pull revision 2 of this tool\n\n\n## Note on reversibility\n**`sbpack` and `sbpull --unpack` are not textually reversible. The packed and unpacked CWL \nrepresentations are functionally identical, however if you `sbpack` a workflow, and \nthen `sbpull --unpack` it, they will look different.**\n\n\n## Credentials file and profiles\n\nIf you use the SBG API you already have an API configuration file. If\nnot, you should create one. It is located in \n`~/.sevenbridges/credentials`. ([Documentation][cred-doc])\n\n[cred-doc]: https://docs.sevenbridges.com/docs/store-credentials-to-access-seven-bridges-client-applications-and-libraries\n\nBriefly, each section in the SBG configuration file (e.g. `[cgc]`) is a \nprofile name and has two entries. The end-point and an authentication\ntoken, which you get from your developer tab on the platform.\n\n```\n[sbg-us]\napi_endpoint = https://api.sbgenomics.com/v2\nauth_token = <dev token here>\n\n[sbg-eu]\napi_endpoint = https://eu-api.sbgenomics.com/v2\nauth_token = <dev token here>\n\n[sbg-china]\napi_endpoint = https://api.sevenbridges.cn/v2\nauth_token = <dev token here>\n\n[cgc]\napi_endpoint = https://cgc-api.sbgenomics.com/v2\nauth_token = <dev token here>\n\n[cavatica]\napi_endpoint = https://cavatica-api.sbgenomics.com/v2\nauth_token = <dev token here>\n\n[nhlbi]\napi_endpoint = https://api.sb.biodatacatalyst.nhlbi.nih.gov/v2\nauth_token = <dev token here>\n```\n\nYou can have several profiles on the same platform if, for example, you \nare an enterprise user and you belong to several divisions. Please refer\nto the API documentation for more detail.\n\n### Reading credentials from env variables\n\nInstead of using the credentials file, you can specify environment variables\n`SB_API_ENDPOINT` and `SB_AUTH_TOKEN`. To use the env variables in `sbpack` simply\nspecify profile `.` in the command, e.g.\n\n```bash\nsbpack . kghosesbg/sbpla-31744/ATAC-seq-pipeline-se https://raw.githubusercontent.com/Duke-GCB/GGR-cwl/master/v1.0/ATAC-seq_pipeline/pipeline-se.cwl\n```\n\nBy specifying `.` profile, `sbpack` will use env variables. If these are not found, the default profile\nfrom the credentials file is used.\n\n### Running the test suite\n\nThe pulling test requires two environment variables to be set\n\n```\nSB_AUTH_TOKEN\nSB_API_ENDPOINT\n```\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Command line tool to upload and download CWL to and from SB powered platforms.",
"version": "2022.3.16",
"project_urls": null,
"split_keywords": [
"seven",
"bridges",
"cwl",
"common",
"workflow",
"language"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ef8171b7bdcedb834659d5da3fe138a311e37f6cd0dd7cdd01f6b44a8a287773",
"md5": "d94be53a89b9451aa0f5430830ff4514",
"sha256": "203ef0d5a9ae808521d0a7276b9162abf24a34ad6b5bf274f36e680ebfcf4ff1"
},
"downloads": -1,
"filename": "sbpack-2022.3.16-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d94be53a89b9451aa0f5430830ff4514",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 26786,
"upload_time": "2022-03-16T14:20:08",
"upload_time_iso_8601": "2022-03-16T14:20:08.993769Z",
"url": "https://files.pythonhosted.org/packages/ef/81/71b7bdcedb834659d5da3fe138a311e37f6cd0dd7cdd01f6b44a8a287773/sbpack-2022.3.16-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "8b7545a182c84ee1239a4ecf5236e50067d9ff7dfc8082cd70b0f33ff60b23fa",
"md5": "2f2d971a2e526b36cd04dc50a7442925",
"sha256": "35ece178dd1b11a8ca1010bfa9ac891be608f759ea696eb14cd9dacabb5adbfa"
},
"downloads": -1,
"filename": "sbpack-2022.3.16.tar.gz",
"has_sig": false,
"md5_digest": "2f2d971a2e526b36cd04dc50a7442925",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 24128,
"upload_time": "2022-03-16T14:20:10",
"upload_time_iso_8601": "2022-03-16T14:20:10.780883Z",
"url": "https://files.pythonhosted.org/packages/8b/75/45a182c84ee1239a4ecf5236e50067d9ff7dfc8082cd70b0f33ff60b23fa/sbpack-2022.3.16.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-03-16 14:20:10",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "sbpack"
}
