# <img src="https://raw.githubusercontent.com/seqeralabs/twkit/main/assets/twkit.svg" width=50 alt="twkit logo"> twkit
`twkit` is a Python wrapper for the [Nextflow Tower CLI](https://github.com/seqeralabs/tower-cli). It can be leveraged to automate the creation of all of the entities in Nextflow Tower via a simple configuration file in YAML format.
The key features are:
- **Simple configuration**: All of the command-line options available when using the Nextflow Tower CLI can be defined in simple YAML format.
- **Infrastructure as Code**: Enable users to manage and provision their infrastructure specifications.
- **Automation**: End-to-end creation of entities within Nextflow Tower, all the way from adding an Organization to launching pipeline(s) within that Organization.
## Prerequisites
You will need to have an account on Nextflow Tower (see [Plans and pricing](https://cloud.tower.nf/pricing/)).
### 1. Dependencies
`twkit` requires the following dependencies:
1. [Nextflow Tower CLI](https://github.com/seqeralabs/tower-cli#1-installation)
2. [Python (`>=3.8`)](https://www.python.org/downloads/)
3. [PyYAML](https://pypi.org/project/PyYAML/)
Alternatively, you can install the dependencies via Conda by downloading and using the [Conda environment file](https://github.com/seqeralabs/twkit/blob/main/environment.yml) that has been supplied in this repository:
```console
conda env create -f environment.yml
conda activate twkit
```
### 2. Installation
The scripts in this repository are packaged and available on [PyPI](https://pypi.org/project/twkit/). They can be installed via `pip`:
```
pip install twkit
```
You can force overwrite the installation to use the latest changes with the command below:
```
pip install --upgrade --force-reinstall twkit
```
### 3. Configuration
Create a Tower access token using the [Nextflow Tower](https://tower.nf/) web interface via the **Your Tokens** page in your profile.
`twkit` reads this token from the environment variable `TOWER_ACCESS_TOKEN`. Please export it into your terminal as shown below:
```bash
export TOWER_ACCESS_TOKEN=<your access token>
```
## Usage
Use the -h or --help parameter to list the available commands and their associated options:
```
twkit -h
```
### Dryrun
To print the commands that would executed with `tw` when using a YAML file, you can run `twkit` with the `--dryrun` flag:
```
twkit file.yaml --dryrun
```
### Recursively delete
Instead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the `--delete` flag:
```
twkit file.yaml --delete
```
For example, if you have a YAML file that defines an Organization -> Workspace -> Team -> Credentials -> Compute Environment that have already been created, with the `--delete` flag, `twkit` will recursively delete the Compute Environment -> Credentials -> Team -> Workspace -> Organization.
### Using `tw` specific CLI options
`tw` specific CLI options can be specified with the `--cli=` flag:
```
twkit file.yaml --cli="--arg1 --arg2"
```
You can find the full list of options by running `tw -h`.
The Tower CLI expects to connect to a Tower instance that is secured by a TLS certificate. If your Tower instance does not present a certificate, you will need to qualify and run your `tw` commands with the `--insecure` flag.
To use `tw` specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options you would like to use enclosed in double quotes.
For example:
```
twkit file.yaml --cli="--insecure"
```
To use an SSL certificate that is not accepted by the default Java certificate authorities and specify a custom `cacerts` store as accepted by the `tw` CLI, you can specify the `-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts` option enclosed in double quotes to `twkit` as you would to `tw`, preceded by `--cli=`.
For example:
```
twkit hello-world-config.yml --cli="-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts"
```
<b>Note</b>: Use of `--verbose` option for the `tw` CLI is currently not supported by `twkit`. Supplying `--cli="--verbose"` will raise an error.
## Quick start
You must provide a YAML file that defines the options for each of the entities you would like to create in Nextflow Tower.
You will need to have an account on Nextflow Tower (see [Plans and pricing](https://cloud.tower.nf/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline.
### Launch via YAML
1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>` entries as required:
```
launch:
- name: 'hello-world' # Workflow name
workspace: '<YOUR_WORKSPACE>' # Workspace name
compute-env: '<YOUR_COMPUTE_ENVIRONMENT>' # Compute environment
revision: 'master' # Pipeline revision
pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL
```
2. Launch the pipeline with `twkit`:
```
twkit hello-world-config.yml
```
3. Login to your Tower instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!
### Launch via a Python script
You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Tower CLI by leveraging the flexibility and customisation options available in Python.
1. Download the [`launch_hello_world.py`](https://github.com/seqeralabs/twkit/blob/main/examples/python/launch_hello_world.py) Python script and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>` entries as required.
2. Launch the pipeline with `twkit`:
```
python launch_hello_world.py
```
3. Login to your Tower instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!
## Real world example
Please see [`twkit-e2e.yml`](https://github.com/seqeralabs/twkit/blob/main/examples/yaml/twkit-e2e.yml) for an end-to-end example that highlights how you can use `twkit` to create everything sequentially in Nextflow Tower all the way from creating a new Organization to launching a pipeline.
You can modify this YAML to similarly create Nextflow Tower resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it with `twkit examples/yaml/twkit-e2e.yml`, you will have to set the following environment variables:
```
$TOWER_GITHUB_PASSWORD
$DOCKERHUB_PASSWORD
$AWS_ACCESS_KEY_ID
$AWS_SECRET_ACCESS_KEY
$AWS_ASSUME_ROLE_ARN
$AZURE_BATCH_KEY
$AZURE_STORAGE_KEY
$GOOGLE_KEY
$SENTIEON_LICENSE_BASE64
```
## Templates
We have provided template YAML files for each of the entities that can be created on Tower. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/twkit/templates) directory and should form a good starting point for you to add your own customization:
- [organizations.yml](https://github.com/seqeralabs/twkit/blob/main/templates/organizations.yml)
- [teams.yml](https://github.com/seqeralabs/twkit/blob/main/templates/teams.yml)
- [workspaces.yml](https://github.com/seqeralabs/twkit/blob/main/templates/workspaces.yml)
- [participants.yml](https://github.com/seqeralabs/twkit/blob/main/templates/participants.yml)
- [credentials.yml](https://github.com/seqeralabs/twkit/blob/main/templates/credentials.yml)
- [secrets.yml](https://github.com/seqeralabs/twkit/blob/main/templates/secrets.yml)
- [compute-envs.yml](https://github.com/seqeralabs/twkit/blob/main/templates/compute-envs.yml)
- [actions.yml](https://github.com/seqeralabs/twkit/blob/main/templates/actions.yml)
- [datasets.yml](https://github.com/seqeralabs/twkit/blob/main/templates/datasets.yml)
- [pipelines.yml](https://github.com/seqeralabs/twkit/blob/main/templates/pipelines.yml)
- [launch.yml](https://github.com/seqeralabs/twkit/blob/main/templates/launch.yml)
## Contributions and Support
If you would like to contribute to `twkit`, please see the [contributing guidelines](https://github.com/seqeralabs/twkit/blob/main/.github/CONTRIBUTING.md).
For further information or help, please don't hesitate to create an [issue](https://github.com/seqeralabs/twkit/issues) in this repository.
## Credits
`twkit` was written by [Esha Joshi](https://github.com/ejseqera), [Adam Talbot](https://github.com/adamrtalbot) and [Harshil Patel](https://github.com/drpatelh) from the Scientific Development Team at [Seqera Labs](https://seqera.io/).
Raw data
{
"_id": null,
"home_page": "https://github.com/seqeralabs/twkit",
"name": "twkit",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8, <4",
"maintainer_email": "",
"keywords": "nextflow,bioinformatics,workflow,pipeline,nextflow-tower",
"author": "Esha Joshi, Adam Talbot, Harshil Patel",
"author_email": "esha.joshi@seqera.io, adam.talbot@seqera.io, harshil.patel@seqera.io",
"download_url": "https://files.pythonhosted.org/packages/b4/cb/7450626ac3411c3d5a45f7a277c61bbc26880ceea3d5fed78ca5e44d0cbe/twkit-0.3.0.tar.gz",
"platform": null,
"description": "# <img src=\"https://raw.githubusercontent.com/seqeralabs/twkit/main/assets/twkit.svg\" width=50 alt=\"twkit logo\"> twkit\n\n`twkit` is a Python wrapper for the [Nextflow Tower CLI](https://github.com/seqeralabs/tower-cli). It can be leveraged to automate the creation of all of the entities in Nextflow Tower via a simple configuration file in YAML format.\n\nThe key features are:\n\n- **Simple configuration**: All of the command-line options available when using the Nextflow Tower CLI can be defined in simple YAML format.\n- **Infrastructure as Code**: Enable users to manage and provision their infrastructure specifications.\n- **Automation**: End-to-end creation of entities within Nextflow Tower, all the way from adding an Organization to launching pipeline(s) within that Organization.\n\n## Prerequisites\n\nYou will need to have an account on Nextflow Tower (see [Plans and pricing](https://cloud.tower.nf/pricing/)).\n\n### 1. Dependencies\n\n`twkit` requires the following dependencies:\n\n1. [Nextflow Tower CLI](https://github.com/seqeralabs/tower-cli#1-installation)\n\n2. [Python (`>=3.8`)](https://www.python.org/downloads/)\n\n3. [PyYAML](https://pypi.org/project/PyYAML/)\n\nAlternatively, you can install the dependencies via Conda by downloading and using the [Conda environment file](https://github.com/seqeralabs/twkit/blob/main/environment.yml) that has been supplied in this repository:\n\n```console\nconda env create -f environment.yml\nconda activate twkit\n```\n\n### 2. Installation\n\nThe scripts in this repository are packaged and available on [PyPI](https://pypi.org/project/twkit/). They can be installed via `pip`:\n\n```\npip install twkit\n```\n\nYou can force overwrite the installation to use the latest changes with the command below:\n\n```\npip install --upgrade --force-reinstall twkit\n```\n\n### 3. Configuration\n\nCreate a Tower access token using the [Nextflow Tower](https://tower.nf/) web interface via the **Your Tokens** page in your profile.\n\n`twkit` reads this token from the environment variable `TOWER_ACCESS_TOKEN`. Please export it into your terminal as shown below:\n\n```bash\nexport TOWER_ACCESS_TOKEN=<your access token>\n```\n\n## Usage\n\nUse the -h or --help parameter to list the available commands and their associated options:\n\n```\ntwkit -h\n```\n\n### Dryrun\n\nTo print the commands that would executed with `tw` when using a YAML file, you can run `twkit` with the `--dryrun` flag:\n\n```\ntwkit file.yaml --dryrun\n```\n\n### Recursively delete\n\nInstead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the `--delete` flag:\n\n```\ntwkit file.yaml --delete\n```\n\nFor example, if you have a YAML file that defines an Organization -> Workspace -> Team -> Credentials -> Compute Environment that have already been created, with the `--delete` flag, `twkit` will recursively delete the Compute Environment -> Credentials -> Team -> Workspace -> Organization.\n\n### Using `tw` specific CLI options\n\n`tw` specific CLI options can be specified with the `--cli=` flag:\n\n```\ntwkit file.yaml --cli=\"--arg1 --arg2\"\n```\n\nYou can find the full list of options by running `tw -h`.\n\nThe Tower CLI expects to connect to a Tower instance that is secured by a TLS certificate. If your Tower instance does not present a certificate, you will need to qualify and run your `tw` commands with the `--insecure` flag.\n\nTo use `tw` specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options you would like to use enclosed in double quotes.\n\nFor example:\n\n```\ntwkit file.yaml --cli=\"--insecure\"\n```\n\nTo use an SSL certificate that is not accepted by the default Java certificate authorities and specify a custom `cacerts` store as accepted by the `tw` CLI, you can specify the `-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts` option enclosed in double quotes to `twkit` as you would to `tw`, preceded by `--cli=`.\n\nFor example:\n\n```\ntwkit hello-world-config.yml --cli=\"-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts\"\n```\n\n<b>Note</b>: Use of `--verbose` option for the `tw` CLI is currently not supported by `twkit`. Supplying `--cli=\"--verbose\"` will raise an error.\n\n## Quick start\n\nYou must provide a YAML file that defines the options for each of the entities you would like to create in Nextflow Tower.\n\nYou will need to have an account on Nextflow Tower (see [Plans and pricing](https://cloud.tower.nf/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline.\n\n### Launch via YAML\n\n1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>` entries as required:\n\n ```\n launch:\n - name: 'hello-world' # Workflow name\n workspace: '<YOUR_WORKSPACE>' # Workspace name\n compute-env: '<YOUR_COMPUTE_ENVIRONMENT>' # Compute environment\n revision: 'master' # Pipeline revision\n pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL\n ```\n\n2. Launch the pipeline with `twkit`:\n\n ```\n twkit hello-world-config.yml\n ```\n\n3. Login to your Tower instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!\n\n### Launch via a Python script\n\nYou can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Tower CLI by leveraging the flexibility and customisation options available in Python.\n\n1. Download the [`launch_hello_world.py`](https://github.com/seqeralabs/twkit/blob/main/examples/python/launch_hello_world.py) Python script and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>` entries as required.\n\n2. Launch the pipeline with `twkit`:\n\n ```\n python launch_hello_world.py\n ```\n\n3. Login to your Tower instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!\n\n## Real world example\n\nPlease see [`twkit-e2e.yml`](https://github.com/seqeralabs/twkit/blob/main/examples/yaml/twkit-e2e.yml) for an end-to-end example that highlights how you can use `twkit` to create everything sequentially in Nextflow Tower all the way from creating a new Organization to launching a pipeline.\n\nYou can modify this YAML to similarly create Nextflow Tower resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it with `twkit examples/yaml/twkit-e2e.yml`, you will have to set the following environment variables:\n\n```\n$TOWER_GITHUB_PASSWORD\n$DOCKERHUB_PASSWORD\n$AWS_ACCESS_KEY_ID\n$AWS_SECRET_ACCESS_KEY\n$AWS_ASSUME_ROLE_ARN\n$AZURE_BATCH_KEY\n$AZURE_STORAGE_KEY\n$GOOGLE_KEY\n$SENTIEON_LICENSE_BASE64\n```\n\n## Templates\n\nWe have provided template YAML files for each of the entities that can be created on Tower. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/twkit/templates) directory and should form a good starting point for you to add your own customization:\n\n- [organizations.yml](https://github.com/seqeralabs/twkit/blob/main/templates/organizations.yml)\n- [teams.yml](https://github.com/seqeralabs/twkit/blob/main/templates/teams.yml)\n- [workspaces.yml](https://github.com/seqeralabs/twkit/blob/main/templates/workspaces.yml)\n- [participants.yml](https://github.com/seqeralabs/twkit/blob/main/templates/participants.yml)\n- [credentials.yml](https://github.com/seqeralabs/twkit/blob/main/templates/credentials.yml)\n- [secrets.yml](https://github.com/seqeralabs/twkit/blob/main/templates/secrets.yml)\n- [compute-envs.yml](https://github.com/seqeralabs/twkit/blob/main/templates/compute-envs.yml)\n- [actions.yml](https://github.com/seqeralabs/twkit/blob/main/templates/actions.yml)\n- [datasets.yml](https://github.com/seqeralabs/twkit/blob/main/templates/datasets.yml)\n- [pipelines.yml](https://github.com/seqeralabs/twkit/blob/main/templates/pipelines.yml)\n- [launch.yml](https://github.com/seqeralabs/twkit/blob/main/templates/launch.yml)\n\n## Contributions and Support\n\nIf you would like to contribute to `twkit`, please see the [contributing guidelines](https://github.com/seqeralabs/twkit/blob/main/.github/CONTRIBUTING.md).\n\nFor further information or help, please don't hesitate to create an [issue](https://github.com/seqeralabs/twkit/issues) in this repository.\n\n## Credits\n\n`twkit` was written by [Esha Joshi](https://github.com/ejseqera), [Adam Talbot](https://github.com/adamrtalbot) and [Harshil Patel](https://github.com/drpatelh) from the Scientific Development Team at [Seqera Labs](https://seqera.io/).\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Automate creation of Nextflow Tower resources",
"version": "0.3.0",
"project_urls": {
"Homepage": "https://github.com/seqeralabs/twkit"
},
"split_keywords": [
"nextflow",
"bioinformatics",
"workflow",
"pipeline",
"nextflow-tower"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "bb975adbafe1da542d5eb6610d09f4cf2533dfd40622911a534ffd0d0089731c",
"md5": "18b00d9ae2710f98b49336a4a40fcc82",
"sha256": "a1e4274ea96ebbe47c495e72c58b2ca265a17102a76d66f29f06a80c25047432"
},
"downloads": -1,
"filename": "twkit-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "18b00d9ae2710f98b49336a4a40fcc82",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8, <4",
"size": 23964,
"upload_time": "2023-10-13T00:48:06",
"upload_time_iso_8601": "2023-10-13T00:48:06.901062Z",
"url": "https://files.pythonhosted.org/packages/bb/97/5adbafe1da542d5eb6610d09f4cf2533dfd40622911a534ffd0d0089731c/twkit-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b4cb7450626ac3411c3d5a45f7a277c61bbc26880ceea3d5fed78ca5e44d0cbe",
"md5": "e7d567e34e1b2f2916d38460ebc6cec4",
"sha256": "512174263599de840684aa33b61893bc36f1a0dc56da38975e004365cc3fecfb"
},
"downloads": -1,
"filename": "twkit-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "e7d567e34e1b2f2916d38460ebc6cec4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8, <4",
"size": 36058,
"upload_time": "2023-10-13T00:48:08",
"upload_time_iso_8601": "2023-10-13T00:48:08.399049Z",
"url": "https://files.pythonhosted.org/packages/b4/cb/7450626ac3411c3d5a45f7a277c61bbc26880ceea3d5fed78ca5e44d0cbe/twkit-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-13 00:48:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "seqeralabs",
"github_project": "twkit",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "twkit"
}
JSON
