# <img src="assets/seqerakit.svg" width=50 alt="seqerakit logo"> seqerakit
`seqerakit` is a Python wrapper for the [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli). It can be leveraged to automate the creation of all of the entities in Seqera Platform via a simple configuration file in YAML format.
The key features are:
- **Simple configuration**: All of the command-line options available when using the Seqera Platform CLI can be defined in simple YAML format.
- **Infrastructure as Code**: Enable users to manage and provision their infrastructure specifications.
- **Automation**: End-to-end creation of entities within Seqera Platform, all the way from adding an Organization to launching pipeline(s) within that Organization.
## Prerequisites
You will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)).
## Installation
`seqerakit` requires the following dependencies:
1. [Seqera Platform CLI (`>=0.9.2`)](https://github.com/seqeralabs/tower-cli#1-installation)
2. [Python (`>=3.8`)](https://www.python.org/downloads/)
3. [PyYAML](https://pypi.org/project/PyYAML/)
### Conda
You can install `seqerakit` and its dependencies via Conda. Ensure that you have the correct channels configured:
```console
conda config --add channels bioconda
conda config --add channels conda-forge
conda config --set channel_priority strict
```
You can then create a conda environment with `seqerakit` installed using the following:
```
conda env create -n seqerakit seqerakit
conda activate seqerakit
```
### Pip
If you already have [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli#1-installation) and Python installed on your system, you can install `seqerakit` directly from [PyPI](https://pypi.org/project/seqerakit/):
```console
pip install seqerakit
```
You can force overwrite the installation to use the latest changes with the command below:
```console
pip install --upgrade --force-reinstall seqerakit
```
### Local development installation
You can install the development branch of `seqerakit` on your local machine to test feature updates of the tool. Before proceeding, ensure that you have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system.
1. To install directly from pip:
```bash
pip install git+https://github.com/seqeralabs/seqera-kit.git@dev
```
2. Alternatively, you may clone the repository locally and install manually:
```bash
git clone https://github.com/seqeralabs/seqera-kit.git
cd seqera-kit
git checkout dev
pip install .
```
You can verify your installation with:
```bash
pip show seqerakit
```
## Configuration
Create a Seqera Platform access token using the [Seqera Platform](https://seqera.io/) web interface via the **Your Tokens** page in your profile.
`seqerakit` reads this token from the environment variable `TOWER_ACCESS_TOKEN`. Please export it into your terminal as shown below:
```bash
export TOWER_ACCESS_TOKEN=<Your access token>
```
For Enterprise installations of Seqera Platform, you will also need to configure the API endpoint that will be used to connect to the Platform. You can do so by exporting the following environment variable:
```bash
export TOWER_API_ENDPOINT=<Tower API URL>
```
By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud.
## Usage
To confirm the installation of `seqerakit`, configuration of the Seqera Platform CLI and connection to the Platform is working as expected. This will run the `tw info` command under the hood:
```bash
seqerakit --info
```
Use the `--help` or `-h` parameter to list the available commands and their associated options:
```bash
seqerakit --help
```
Use `--version` or `-v` to retrieve the current version of your seqerakit installation:
```bash
seqerakit --version
```
### Input
`seqerakit` supports input through either file paths to YAMLs or directly from standard input (stdin).
#### Using File Path
```bash
seqerakit /path/to/file.yaml
```
#### Using stdin
```console
$ cat file.yaml | seqerakit -
```
See the [Defining your YAML file using CLI options](#defining-your-yaml-file-using-cli-options) section for guidance on formatting your input YAML file(s).
### Dryrun
To print the commands that would executed with `tw` when using a YAML file, you can run `seqerakit` with the `--dryrun` flag:
```bash
seqerakit file.yaml --dryrun
```
### Recursively delete
Instead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the `--delete` flag:
```bash
seqerakit file.yaml --delete
```
For example, if you have a YAML file that defines an Organization -> Workspace -> Team -> Credentials -> Compute Environment that have already been created, with the `--delete` flag, `seqerakit` will recursively delete the Compute Environment -> Credentials -> Team -> Workspace -> Organization.
### Using `tw` specific CLI options
`tw` specific CLI options can be specified with the `--cli=` flag:
```bash
seqerakit file.yaml --cli="--arg1 --arg2"
```
You can find the full list of options by running `tw -h`.
The Seqera Platform CLI expects to connect to a Seqera Platform instance that is secured by a TLS certificate. If your Seqera Platform Enterprise instance does not present a certificate, you will need to qualify and run your `tw` commands with the `--insecure` flag.
To use `tw` specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options you would like to use enclosed in double quotes.
For example:
```bash
seqerakit file.yaml --cli="--insecure"
```
For Seqera Platform Enterprise, to use an SSL certificate that is not accepted by the default Java certificate authorities and specify a custom `cacerts` store as accepted by the `tw` CLI, you can specify the `-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts` option enclosed in double quotes to `seqerakit` as you would to `tw`, preceded by `--cli=`.
For example:
```bash
seqerakit hello-world-config.yml --cli="-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts"
```
<b>Note</b>: Use of `--verbose` option for the `tw` CLI is currently not supported by `seqerakit`. Supplying `--cli="--verbose"` will raise an error.
## Specify targets
When using a YAML file as input that defines multiple resources, you can use the `--targets` flag to specify which resources to create. This flag takes a comma-separated list of resource names.
For example, given a YAML file that defines the following resources:
```yaml
workspaces:
- name: 'showcase'
organization: 'seqerakit_automation'
...
compute-envs:
- name: 'compute-env'
type: 'aws-batch forge'
workspace: 'seqerakit/test'
...
pipelines:
- name: "hello-world-test-seqerakit"
url: "https://github.com/nextflow-io/hello"
workspace: 'seqerakit/test'
compute-env: "compute-env"
...
```
You can target the creation of `pipelines` only by running:
```bash
seqerakit test.yml --targets pipelines
```
This will process only the pipelines block from the YAML file and ignore other blocks such as `workspaces` and `compute-envs`.
### Multiple Targets
You can also specify multiple resources to create by separating them with commas. For example, to create both workspaces and pipelines, run:
```bash
seqerakit test.yml --targets workspaces,pipelines
```
## YAML Configuration Options
There are several options that can be provided in your YAML configuration file, that are handled specially by seqerakit and/or are not exposed as `tw` CLI options.
### 1. Pipeline parameters using `params` and `params-file`
To specify pipeline parameters, you may either use `params:` to specify a list of parameters, or use `params-file:` to point to a parameters file.
For example, to specify pipeline parameters within your YAML:
```yaml
params:
outdir: 's3://path/to/outdir'
fasta: 's3://path/to/reference.fasta'
```
Alternatively, to specify a file containing pipeline parameters:
```yaml
params-file: '/path/to/my/parameters.yaml'
```
Optionally, you may provide both:
```yaml
params-file: '/path/to/my/parameters.yaml'
params:
outdir: 's3://path/to/outdir'
fasta: 's3://path/to/reference.fasta'
```
**Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`.
### 2. `overwrite` Functionality
For every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name.
`seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw <subcommand> delete` command before attempting to create it based on the options defined in the YAML file.
```console
DEBUG:root: Overwrite is set to 'True' for organizations
DEBUG:root: Running command: tw -o json organizations list
DEBUG:root: The attempted organizations resource already exists. Overwriting.
DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME
DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization'
```
### 3. Specifying JSON configuration files with `file-path`
The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file.
An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template:
```yaml
compute-envs:
- name: 'my_aws_compute_environment' # required
workspace: 'my_organization/my_workspace' # required
credentials: 'my_aws_credentials' # required
wait: 'AVAILABLE' # optional
file-path: './compute-envs/my_aws_compute_environment.json' # required
overwrite: True
```
## Quick start
You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform.
You will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline.
### Launch via YAML
1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>` entries as required:
```yaml # noqa
launch:
- name: 'hello-world' # Workflow name
workspace: '<YOUR_WORKSPACE>' # Workspace name
compute-env: '<YOUR_COMPUTE_ENVIRONMENT>' # Compute environment
revision: 'master' # Pipeline revision
pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL
```
2. Launch the pipeline with `seqerakit`:
```bash
seqerakit hello-world-config.yml
```
3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!
### Launch via a Python script
You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Seqera Platform CLI by leveraging the flexibility and customisation options available in Python.
1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>` entries as required.
2. Launch the pipeline with `seqerakit`:
```bash
python launch_hello_world.py
```
3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!
## Defining your YAML file using CLI options
All available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for your desired entity.
1. Retrieve CLI Options
To obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows:
```console
$ tw pipelines add -h
Usage: tw pipelines add [OPTIONS] PIPELINE_URL
Add a workspace pipeline.
Parameters:
* PIPELINE_URL Nextflow pipeline URL.
Options:
* -n, --name=<name> Pipeline name.
-w, --workspace=<workspace> Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName
-d, --description=<description> Pipeline description.
--labels=<labels>[,<labels>...] List of labels seperated by coma.
-c, --compute-env=<computeEnv> Compute environment name.
--work-dir=<workDir> Path where the pipeline scratch data is stored.
-p, --profile=<profile>[,<profile>...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution.
--params-file=<paramsFile> Pipeline parameters in either JSON or YML format.
--revision=<revision> A valid repository commit Id, tag or branch name.
...
```
2. Define Key-Value Pairs in YAML
Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance:
```yaml
pipelines:
- name: 'my_first_pipeline'
url: 'https://github.com/username/my_pipeline'
workspace: 'my_organization/my_workspace'
description: 'My test pipeline'
labels: 'yeast,test_data'
compute-env: 'my_compute_environment'
work-dir: 's3://my_bucket'
profile: 'test'
params-file: '/path/to/params.yaml'
revision: '1.0'
```
In this example:
- `name`, `url`, `workspace`, etc., are the keys derived from the CLI options.
- The corresponding values are user-defined
### Best Practices:
- Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting.
- Use quotes around strings that contain special characters or spaces.
- When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above.
- For complex configurations, refer to the [Templates](./templates/) provided in this repository.
## Templates
We have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization:
- [organizations.yml](./templates/organizations.yml)
- [teams.yml](./templates/teams.yml)
- [workspaces.yml](./templates/workspaces.yml)
- [participants.yml](./templates/participants.yml)
- [credentials.yml](./templates/credentials.yml)
- [secrets.yml](./templates/secrets.yml)
- [compute-envs.yml](./templates/compute-envs.yml)
- [actions.yml](./templates/actions.yml)
- [datasets.yml](./templates/datasets.yml)
- [labels.yml](./templates/labels.yml)
- [pipelines.yml](./templates/pipelines.yml)
- [launch.yml](./templates/launch.yml)
## Real world example
Please see [`seqerakit-e2e.yml`](./examples/yaml/seqerakit-e2e.yml) for an end-to-end example that highlights how you can use `seqerakit` to create everything sequentially in Seqera Platform all the way from creating a new Organization to launching a pipeline.
You can modify this YAML to similarly create Seqera Platform resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it with `seqerakit examples/yaml/seqerakit-e2e.yml`, you will have to set the following environment variables:
```console
$TOWER_GITHUB_PASSWORD
$DOCKERHUB_PASSWORD
$AWS_ACCESS_KEY_ID
$AWS_SECRET_ACCESS_KEY
$AWS_ASSUME_ROLE_ARN
$AZURE_BATCH_KEY
$AZURE_STORAGE_KEY
$GOOGLE_KEY
$SENTIEON_LICENSE_BASE64
```
## Contributions and Support
If you would like to contribute to `seqerakit`, please see the [contributing guidelines](./.github/CONTRIBUTING.md).
For further information or help, please don't hesitate to create an [issue](https://github.com/seqeralabs/seqera-kit/issues) in this repository.
## Credits
`seqerakit` was written by [Esha Joshi](https://github.com/ejseqera), [Adam Talbot](https://github.com/adamrtalbot) and [Harshil Patel](https://github.com/drpatelh) from the Scientific Development Team at [Seqera Labs](https://seqera.io/).
Raw data
{
"_id": null,
"home_page": "https://github.com/seqeralabs/seqera-kit",
"name": "seqerakit",
"maintainer": null,
"docs_url": null,
"requires_python": "<4,>=3.8",
"maintainer_email": null,
"keywords": "nextflow, bioinformatics, workflow, pipeline, seqera-platform, seqera",
"author": "Esha Joshi, Adam Talbot, Harshil Patel",
"author_email": "esha.joshi@seqera.io, adam.talbot@seqera.io, harshil.patel@seqera.io",
"download_url": "https://files.pythonhosted.org/packages/6a/69/50c5b9ab566d81ab7fcca41eceb82f239dc97838f3f3ca77c15ebb9f494f/seqerakit-0.4.9.tar.gz",
"platform": null,
"description": "# <img src=\"assets/seqerakit.svg\" width=50 alt=\"seqerakit logo\"> seqerakit\n\n`seqerakit` is a Python wrapper for the [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli). It can be leveraged to automate the creation of all of the entities in Seqera Platform via a simple configuration file in YAML format.\n\nThe key features are:\n\n- **Simple configuration**: All of the command-line options available when using the Seqera Platform CLI can be defined in simple YAML format.\n- **Infrastructure as Code**: Enable users to manage and provision their infrastructure specifications.\n- **Automation**: End-to-end creation of entities within Seqera Platform, all the way from adding an Organization to launching pipeline(s) within that Organization.\n\n## Prerequisites\n\nYou will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)).\n\n## Installation\n\n`seqerakit` requires the following dependencies:\n\n1. [Seqera Platform CLI (`>=0.9.2`)](https://github.com/seqeralabs/tower-cli#1-installation)\n\n2. [Python (`>=3.8`)](https://www.python.org/downloads/)\n\n3. [PyYAML](https://pypi.org/project/PyYAML/)\n\n### Conda\n\nYou can install `seqerakit` and its dependencies via Conda. Ensure that you have the correct channels configured:\n\n```console\nconda config --add channels bioconda\nconda config --add channels conda-forge\nconda config --set channel_priority strict\n```\n\nYou can then create a conda environment with `seqerakit` installed using the following:\n\n```\nconda env create -n seqerakit seqerakit\nconda activate seqerakit\n```\n\n### Pip\n\nIf you already have [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli#1-installation) and Python installed on your system, you can install `seqerakit` directly from [PyPI](https://pypi.org/project/seqerakit/):\n\n```console\npip install seqerakit\n```\n\nYou can force overwrite the installation to use the latest changes with the command below:\n\n```console\npip install --upgrade --force-reinstall seqerakit\n```\n\n### Local development installation\nYou can install the development branch of `seqerakit` on your local machine to test feature updates of the tool. Before proceeding, ensure that you have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system.\n\n1. To install directly from pip:\n```bash\npip install git+https://github.com/seqeralabs/seqera-kit.git@dev\n```\n\n2. Alternatively, you may clone the repository locally and install manually:\n```bash\ngit clone https://github.com/seqeralabs/seqera-kit.git\ncd seqera-kit\ngit checkout dev\npip install .\n```\n\nYou can verify your installation with:\n```bash\npip show seqerakit\n```\n\n## Configuration\n\nCreate a Seqera Platform access token using the [Seqera Platform](https://seqera.io/) web interface via the **Your Tokens** page in your profile.\n\n`seqerakit` reads this token from the environment variable `TOWER_ACCESS_TOKEN`. Please export it into your terminal as shown below:\n\n```bash\nexport TOWER_ACCESS_TOKEN=<Your access token>\n```\n\nFor Enterprise installations of Seqera Platform, you will also need to configure the API endpoint that will be used to connect to the Platform. You can do so by exporting the following environment variable:\n```bash\nexport TOWER_API_ENDPOINT=<Tower API URL>\n```\nBy default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud.\n\n\n## Usage\n\nTo confirm the installation of `seqerakit`, configuration of the Seqera Platform CLI and connection to the Platform is working as expected. This will run the `tw info` command under the hood:\n\n```bash\nseqerakit --info\n```\n\nUse the `--help` or `-h` parameter to list the available commands and their associated options:\n```bash\nseqerakit --help\n```\n\nUse `--version` or `-v` to retrieve the current version of your seqerakit installation:\n```bash\nseqerakit --version\n```\n### Input\n`seqerakit` supports input through either file paths to YAMLs or directly from standard input (stdin).\n\n#### Using File Path\n```bash\nseqerakit /path/to/file.yaml\n```\n#### Using stdin\n```console\n$ cat file.yaml | seqerakit -\n```\nSee the [Defining your YAML file using CLI options](#defining-your-yaml-file-using-cli-options) section for guidance on formatting your input YAML file(s).\n\n### Dryrun\n\nTo print the commands that would executed with `tw` when using a YAML file, you can run `seqerakit` with the `--dryrun` flag:\n\n```bash\nseqerakit file.yaml --dryrun\n```\n\n### Recursively delete\n\nInstead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the `--delete` flag:\n\n```bash\nseqerakit file.yaml --delete\n```\n\nFor example, if you have a YAML file that defines an Organization -> Workspace -> Team -> Credentials -> Compute Environment that have already been created, with the `--delete` flag, `seqerakit` will recursively delete the Compute Environment -> Credentials -> Team -> Workspace -> Organization.\n\n### Using `tw` specific CLI options\n\n`tw` specific CLI options can be specified with the `--cli=` flag:\n\n```bash\nseqerakit file.yaml --cli=\"--arg1 --arg2\"\n```\n\nYou can find the full list of options by running `tw -h`.\n\nThe Seqera Platform CLI expects to connect to a Seqera Platform instance that is secured by a TLS certificate. If your Seqera Platform Enterprise instance does not present a certificate, you will need to qualify and run your `tw` commands with the `--insecure` flag.\n\nTo use `tw` specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options you would like to use enclosed in double quotes.\n\nFor example:\n\n```bash\nseqerakit file.yaml --cli=\"--insecure\"\n```\n\nFor Seqera Platform Enterprise, to use an SSL certificate that is not accepted by the default Java certificate authorities and specify a custom `cacerts` store as accepted by the `tw` CLI, you can specify the `-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts` option enclosed in double quotes to `seqerakit` as you would to `tw`, preceded by `--cli=`.\n\nFor example:\n\n```bash\nseqerakit hello-world-config.yml --cli=\"-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts\"\n```\n\n<b>Note</b>: Use of `--verbose` option for the `tw` CLI is currently not supported by `seqerakit`. Supplying `--cli=\"--verbose\"` will raise an error.\n\n## Specify targets\nWhen using a YAML file as input that defines multiple resources, you can use the `--targets` flag to specify which resources to create. This flag takes a comma-separated list of resource names.\n\nFor example, given a YAML file that defines the following resources:\n\n```yaml\nworkspaces:\n - name: 'showcase'\n organization: 'seqerakit_automation'\n...\ncompute-envs:\n - name: 'compute-env'\n type: 'aws-batch forge'\n workspace: 'seqerakit/test'\n...\npipelines:\n - name: \"hello-world-test-seqerakit\"\n url: \"https://github.com/nextflow-io/hello\"\n workspace: 'seqerakit/test'\n compute-env: \"compute-env\"\n...\n```\n\nYou can target the creation of `pipelines` only by running:\n\n```bash\nseqerakit test.yml --targets pipelines\n```\nThis will process only the pipelines block from the YAML file and ignore other blocks such as `workspaces` and `compute-envs`.\n\n### Multiple Targets\nYou can also specify multiple resources to create by separating them with commas. For example, to create both workspaces and pipelines, run:\n\n```bash\nseqerakit test.yml --targets workspaces,pipelines\n```\n\n## YAML Configuration Options\n\nThere are several options that can be provided in your YAML configuration file, that are handled specially by seqerakit and/or are not exposed as `tw` CLI options.\n\n### 1. Pipeline parameters using `params` and `params-file`\n\nTo specify pipeline parameters, you may either use `params:` to specify a list of parameters, or use `params-file:` to point to a parameters file.\n\nFor example, to specify pipeline parameters within your YAML:\n\n```yaml\nparams:\n outdir: 's3://path/to/outdir'\n fasta: 's3://path/to/reference.fasta'\n```\n\nAlternatively, to specify a file containing pipeline parameters:\n\n```yaml\nparams-file: '/path/to/my/parameters.yaml'\n```\n\nOptionally, you may provide both:\n\n```yaml\nparams-file: '/path/to/my/parameters.yaml'\nparams:\n outdir: 's3://path/to/outdir'\n fasta: 's3://path/to/reference.fasta'\n```\n\n**Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`.\n\n### 2. `overwrite` Functionality\nFor every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name.\n\n`seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw <subcommand> delete` command before attempting to create it based on the options defined in the YAML file.\n\n```console\nDEBUG:root: Overwrite is set to 'True' for organizations\n\nDEBUG:root: Running command: tw -o json organizations list\nDEBUG:root: The attempted organizations resource already exists. Overwriting.\n\nDEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME\nDEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization'\n```\n### 3. Specifying JSON configuration files with `file-path`\nThe Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file.\n\nAn example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template:\n\n```yaml\ncompute-envs:\n - name: 'my_aws_compute_environment' # required\n workspace: 'my_organization/my_workspace' # required\n credentials: 'my_aws_credentials' # required\n wait: 'AVAILABLE' # optional\n file-path: './compute-envs/my_aws_compute_environment.json' # required\n overwrite: True\n```\n\n\n## Quick start\n\nYou must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform.\n\nYou will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline.\n\n### Launch via YAML\n\n1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>` entries as required:\n\n ```yaml # noqa\n launch:\n - name: 'hello-world' # Workflow name\n workspace: '<YOUR_WORKSPACE>' # Workspace name\n compute-env: '<YOUR_COMPUTE_ENVIRONMENT>' # Compute environment\n revision: 'master' # Pipeline revision\n pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL\n ```\n\n2. Launch the pipeline with `seqerakit`:\n\n ```bash\n seqerakit hello-world-config.yml\n ```\n\n3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!\n\n### Launch via a Python script\n\nYou can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Seqera Platform CLI by leveraging the flexibility and customisation options available in Python.\n\n1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>` entries as required.\n\n2. Launch the pipeline with `seqerakit`:\n\n```bash\n python launch_hello_world.py\n```\n\n3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched!\n\n## Defining your YAML file using CLI options\n\nAll available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for your desired entity.\n\n1. Retrieve CLI Options\n\nTo obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows:\n\n```console\n$ tw pipelines add -h\n\nUsage: tw pipelines add [OPTIONS] PIPELINE_URL\n\nAdd a workspace pipeline.\n\nParameters:\n* PIPELINE_URL Nextflow pipeline URL.\n\nOptions:\n* -n, --name=<name> Pipeline name.\n -w, --workspace=<workspace> Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName\n -d, --description=<description> Pipeline description.\n --labels=<labels>[,<labels>...] List of labels seperated by coma.\n -c, --compute-env=<computeEnv> Compute environment name.\n --work-dir=<workDir> Path where the pipeline scratch data is stored.\n -p, --profile=<profile>[,<profile>...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution.\n --params-file=<paramsFile> Pipeline parameters in either JSON or YML format.\n --revision=<revision> A valid repository commit Id, tag or branch name.\n ...\n```\n2. Define Key-Value Pairs in YAML\n\nTranslate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance:\n\n```yaml\npipelines:\n - name: 'my_first_pipeline'\n url: 'https://github.com/username/my_pipeline'\n workspace: 'my_organization/my_workspace'\n description: 'My test pipeline'\n labels: 'yeast,test_data'\n compute-env: 'my_compute_environment'\n work-dir: 's3://my_bucket'\n profile: 'test'\n params-file: '/path/to/params.yaml'\n revision: '1.0'\n```\n\nIn this example:\n\n- `name`, `url`, `workspace`, etc., are the keys derived from the CLI options.\n- The corresponding values are user-defined\n\n### Best Practices:\n- Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting.\n- Use quotes around strings that contain special characters or spaces.\n- When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above.\n- For complex configurations, refer to the [Templates](./templates/) provided in this repository.\n\n\n## Templates\n\nWe have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization:\n\n- [organizations.yml](./templates/organizations.yml)\n- [teams.yml](./templates/teams.yml)\n- [workspaces.yml](./templates/workspaces.yml)\n- [participants.yml](./templates/participants.yml)\n- [credentials.yml](./templates/credentials.yml)\n- [secrets.yml](./templates/secrets.yml)\n- [compute-envs.yml](./templates/compute-envs.yml)\n- [actions.yml](./templates/actions.yml)\n- [datasets.yml](./templates/datasets.yml)\n- [labels.yml](./templates/labels.yml)\n- [pipelines.yml](./templates/pipelines.yml)\n- [launch.yml](./templates/launch.yml)\n\n## Real world example\n\nPlease see [`seqerakit-e2e.yml`](./examples/yaml/seqerakit-e2e.yml) for an end-to-end example that highlights how you can use `seqerakit` to create everything sequentially in Seqera Platform all the way from creating a new Organization to launching a pipeline.\n\nYou can modify this YAML to similarly create Seqera Platform resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it with `seqerakit examples/yaml/seqerakit-e2e.yml`, you will have to set the following environment variables:\n\n```console\n$TOWER_GITHUB_PASSWORD\n$DOCKERHUB_PASSWORD\n$AWS_ACCESS_KEY_ID\n$AWS_SECRET_ACCESS_KEY\n$AWS_ASSUME_ROLE_ARN\n$AZURE_BATCH_KEY\n$AZURE_STORAGE_KEY\n$GOOGLE_KEY\n$SENTIEON_LICENSE_BASE64\n```\n\n## Contributions and Support\n\nIf you would like to contribute to `seqerakit`, please see the [contributing guidelines](./.github/CONTRIBUTING.md).\n\nFor further information or help, please don't hesitate to create an [issue](https://github.com/seqeralabs/seqera-kit/issues) in this repository.\n\n## Credits\n\n`seqerakit` was written by [Esha Joshi](https://github.com/ejseqera), [Adam Talbot](https://github.com/adamrtalbot) and [Harshil Patel](https://github.com/drpatelh) from the Scientific Development Team at [Seqera Labs](https://seqera.io/).\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Automate creation of Seqera Platform resources",
"version": "0.4.9",
"project_urls": {
"Homepage": "https://github.com/seqeralabs/seqera-kit"
},
"split_keywords": [
"nextflow",
" bioinformatics",
" workflow",
" pipeline",
" seqera-platform",
" seqera"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9ef5028161c2358574d8206298ee522e6dace1d36bcabf96665d12a2428d61f2",
"md5": "27c2eadee1e31477906738ecb3aeed4e",
"sha256": "c8895c63f77250f165494de6c3e1d8b08945c2854df25d3b55b2025f899be6df"
},
"downloads": -1,
"filename": "seqerakit-0.4.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "27c2eadee1e31477906738ecb3aeed4e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.8",
"size": 27308,
"upload_time": "2024-08-07T01:59:18",
"upload_time_iso_8601": "2024-08-07T01:59:18.346662Z",
"url": "https://files.pythonhosted.org/packages/9e/f5/028161c2358574d8206298ee522e6dace1d36bcabf96665d12a2428d61f2/seqerakit-0.4.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6a6950c5b9ab566d81ab7fcca41eceb82f239dc97838f3f3ca77c15ebb9f494f",
"md5": "9925bfb24b0eeb6b4bb40ce851dd56d2",
"sha256": "ecca8afc49498226a8c9ad81723a609ae324993e4352fd9561dd2ab657eff378"
},
"downloads": -1,
"filename": "seqerakit-0.4.9.tar.gz",
"has_sig": false,
"md5_digest": "9925bfb24b0eeb6b4bb40ce851dd56d2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.8",
"size": 79123,
"upload_time": "2024-08-07T01:59:19",
"upload_time_iso_8601": "2024-08-07T01:59:19.614322Z",
"url": "https://files.pythonhosted.org/packages/6a/69/50c5b9ab566d81ab7fcca41eceb82f239dc97838f3f3ca77c15ebb9f494f/seqerakit-0.4.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-07 01:59:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "seqeralabs",
"github_project": "seqera-kit",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "seqerakit"
}
JSON
