| Name | grz-cli JSON |
| Version |
1.1.2
JSON |
| download |
| home_page | None |
| Summary | Tool for validation, encryption and upload of MV submissions to GDCs. |
| upload_time | 2025-08-05 11:50:04 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.12 |
| license | None |
| keywords |
gdc
grz
s3
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# GRZ CLI
A command-line tool for validating, encrypting, uploading and downloading submissions to/from a GDC/GRZ (Genomrechenzentrum).
## Table of Contents
- [Introduction](#introduction)
- [Features](#features)
- [Installation](#installation)
- [Requirements](#requirements)
- [End-user setup](#end-user-setup)
- [Installation via `conda` (recommended)](#installation-via-conda-recommended)
- [Installation via `pip` (not recommended)](#installation-via-pip-not-recommended)
- [Usage](#usage)
- [Configuration](#configuration)
- [Exemplary submission procedure](#exemplary-submission-procedure)
- [Command-Line Interface](#command-line-interface)
- [Testing](#testing)
- [Contributing](#contributing)
- [License](#license)
- [Acknowledgements](#acknowledgements)
## Introduction
This tool provides a way to validate files, encrypt/decrypt files using the [crypt4gh](https://crypt4gh.readthedocs.io/en/latest/) library and upload/download the encrypted files to an S3 bucket of a GDC/GRZ. It also logs the progress and outcomes of these operations in a metadata file.
It is recommended to have the following folder structure for a single submission:
```
EXAMPLE_SUBMISSION
├── files
│ ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read1.fastq.gz
│ ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read2.fastq.gz
│ ├── aaaaaaaa00000000aaaaaaaa00000000_blood_normal.vcf
│ ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read1.fastq.gz
│ ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read2.fastq.gz
│ ├── aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.vcf
│ ├── target_regions.bed
└── metadata
└── metadata.json
```
The current version of the tool requires the `working_dir` to have at least as much free disk space as the total size of the data being submitted.
## Features
- **Validation**: Validate file checksums, basic file metadata and BfArM requirements.
- **Encryption**: Encrypt files using `crypt4gh`.
- **Decryption**: Encrypt files using `crypt4gh`.
- **Upload**: Upload encrypted files directly to a GRZ either (via built-in `boto3`).
- **Download**: Download encrypted files from a GRZ (via built-in `boto3`).
- **Logging**: Log progress and results of operations
## Installation
### Requirements
Beside of the disk space requirements for the submission data, this tool also requires a linux environment, e.g.:
- Linux server
- Virtual machine running linux
- Docker container
- Windows subsystem for linux
- ...
### End-user setup
The recommended method to install this tool is using the conda package manager.
#### Installation via `conda` (recommended)
If `conda` is not yet available on your system, we recommend to install the [Miniforge conda distribution](https://github.com/conda-forge/miniforge) by running the following commands:
```bash
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh
```
There are also alternative ways to install conda:
- [Micromamba, a single executable that does not require a base environment](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)
- [Official installation instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)
Next, install the `grz-cli` tool:
```bash
# create conda environment and activate it
conda create -n grz-tools -c conda-forge -c bioconda "grz-cli"
conda activate grz-tools
```
##### Update instructions
Use the following command to update the tool:
```bash
conda update -n grz-tools -c conda-forge -c bioconda grz-cli
```
#### Installation via `pip` (not recommended)
While installation via `pip` is possible, it is not recommended because users must ensure
that the correct Python version is already installed and that they are using a virtual python environment.
```bash
pip install grz-cli
```
##### Update instructions:
Use the following command to update the tool:
```bash
pip upgrade grz-cli
```
#### Docker
Docker images are available via biocontainers at [https://biocontainers.pro/tools/grz-cli](https://biocontainers.pro/tools/grz-cli).
The build process can take at least a few days after the Bioconda release, so double-check that the latest version in Bioconda is also the latest Docker image version.
## Usage
### Configuration
**The configuration file will be provided by your associated GRZ, please place it into `~/.config/grz-cli/config.yaml`.**
The tool requires a configuration file in YAML format to specify the S3 bucket and other options.
For an exemplary configuration, see [resources/config.yaml](resources/config.yaml).
S3 access and secret key can be listed either in the config file or as environment variable (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`).
### Exemplary submission procedure
After preparing your submission as outlined above, you can use the following commands to validate, encrypt and upload the submission:
```sh
# Validate the submission
grz-cli validate --submission-dir EXAMPLE_SUBMISSION
# Encrypt the submission
grz-cli encrypt --submission-dir EXAMPLE_SUBMISSION
# Upload the submission
grz-cli upload --submission-dir EXAMPLE_SUBMISSION
```
### Troubleshooting
**In case of issues, please re-run your commands with `grz-cli --log-level DEBUG --log-file <your-log-file.log> [...]` and submit the log file to the GRZ data steward!**
## Command-Line Interface
`grz-cli` provides a command-line interface with the following subcommands:
### validate
It is recommended to run this command before continuing with encryption and upload.
Progress files are stored relative to the submission directory.
- `--submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]
Example usage:
```bash
grz_cli validate --submission-dir foo
```
### encrypt
If a working directory is not provided, then the current directory is used automatically. The log-files are going to be stored in the sub-folder of the working directory.
Files are stored in a folder named `encrypted_files` as a sub-folder of the working directory.
- `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]
- `-c, --config-file`: Path to config file [_optional_]
```bash
grz-cli encrypt --submission-dir foo
```
### upload
Upload the submission into a S3 structure of a GRZ.
- `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'encrypted_files/' directories [**Required**]
- `-c, --config-file`: Path to config file [_optional_]
Example usage:
```bash
grz-cli upload --submission-dir foo
```
## Testing
Please note that binary files used for testing are managed with [Git LFS](https://git-lfs.com), which will be needed to clone them locally with the git repository.
To run the tests, navigate to the root directory of your project and invoke `pytest`.
Alternatively, install `uv` and `tox` and run `uv run tox`.
## Contributing
### Running unreleased/development versions
First, install `uv`.
We recommend using Conda or [Pixi](https://pixi.sh/latest).
After cloning the desired branch of the `grz-tools` repo locally, you can run `grz-cli` directly from the repo using:
```
uv run --project path/to/cloned/grz-tools grz-cli --help
```
## License
This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
## Acknowledgements
Parts of `crypt4gh` code is used in modified form
Raw data
{
"_id": null,
"home_page": null,
"name": "grz-cli",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.12",
"maintainer_email": null,
"keywords": "GDC, GRZ, S3",
"author": null,
"author_email": "Koray Kirli <koraykirli@gmail.com>, Mathias Lesche <mathias.lesche@tu-dresden.de>, \"Florian R. H\u00f6lzlwimmer\" <git.ich@frhoelzlwimmer.de>, Till Hartmann <till.hartmann@bih-charite.de>, Thomas Sell <thomas.sell@bih-charite.de>, Travis Wrightsman <travis.wrightsman@uni-tuebingen.de>",
"download_url": "https://files.pythonhosted.org/packages/c3/08/112b49a582725e4593624717c4ac2fc3cb53c231d1e9a348de9e7d6e747b/grz_cli-1.1.2.tar.gz",
"platform": null,
"description": "# GRZ CLI\n\nA command-line tool for validating, encrypting, uploading and downloading submissions to/from a GDC/GRZ (Genomrechenzentrum).\n\n## Table of Contents\n\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n - [Requirements](#requirements)\n - [End-user setup](#end-user-setup)\n - [Installation via `conda` (recommended)](#installation-via-conda-recommended)\n - [Installation via `pip` (not recommended)](#installation-via-pip-not-recommended)\n- [Usage](#usage)\n - [Configuration](#configuration)\n - [Exemplary submission procedure](#exemplary-submission-procedure)\n- [Command-Line Interface](#command-line-interface)\n- [Testing](#testing)\n- [Contributing](#contributing)\n- [License](#license)\n- [Acknowledgements](#acknowledgements)\n\n## Introduction\n\nThis tool provides a way to validate files, encrypt/decrypt files using the [crypt4gh](https://crypt4gh.readthedocs.io/en/latest/) library and upload/download the encrypted files to an S3 bucket of a GDC/GRZ. It also logs the progress and outcomes of these operations in a metadata file.\n\nIt is recommended to have the following folder structure for a single submission:\n\n```\nEXAMPLE_SUBMISSION\n\u251c\u2500\u2500 files\n\u2502 \u251c\u2500\u2500 aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read1.fastq.gz\n\u2502 \u251c\u2500\u2500 aaaaaaaa00000000aaaaaaaa00000000_blood_normal.read2.fastq.gz\n\u2502 \u251c\u2500\u2500 aaaaaaaa00000000aaaaaaaa00000000_blood_normal.vcf\n\u2502 \u251c\u2500\u2500 aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read1.fastq.gz\n\u2502 \u251c\u2500\u2500 aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.read2.fastq.gz\n\u2502 \u251c\u2500\u2500 aaaaaaaa00000000aaaaaaaa00000000_blood_tumor.vcf\n\u2502 \u251c\u2500\u2500 target_regions.bed\n\u2514\u2500\u2500 metadata\n \u2514\u2500\u2500 metadata.json\n```\n\nThe current version of the tool requires the `working_dir` to have at least as much free disk space as the total size of the data being submitted.\n## Features\n\n- **Validation**: Validate file checksums, basic file metadata and BfArM requirements.\n- **Encryption**: Encrypt files using `crypt4gh`.\n- **Decryption**: Encrypt files using `crypt4gh`.\n- **Upload**: Upload encrypted files directly to a GRZ either (via built-in `boto3`).\n- **Download**: Download encrypted files from a GRZ (via built-in `boto3`).\n- **Logging**: Log progress and results of operations\n\n\n## Installation\n\n### Requirements\nBeside of the disk space requirements for the submission data, this tool also requires a linux environment, e.g.:\n- Linux server\n- Virtual machine running linux\n- Docker container\n- Windows subsystem for linux\n- ...\n\n\n### End-user setup\nThe recommended method to install this tool is using the conda package manager.\n\n\n#### Installation via `conda` (recommended)\n\nIf `conda` is not yet available on your system, we recommend to install the [Miniforge conda distribution](https://github.com/conda-forge/miniforge) by running the following commands:\n```bash\ncurl -L -O \"https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh\"\nbash Miniforge3-$(uname)-$(uname -m).sh\n```\nThere are also alternative ways to install conda:\n- [Micromamba, a single executable that does not require a base environment](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html)\n- [Official installation instructions](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)\n\nNext, install the `grz-cli` tool:\n```bash\n# create conda environment and activate it\nconda create -n grz-tools -c conda-forge -c bioconda \"grz-cli\"\nconda activate grz-tools\n```\n\n##### Update instructions\n\nUse the following command to update the tool:\n\n```bash\nconda update -n grz-tools -c conda-forge -c bioconda grz-cli\n```\n\n\n#### Installation via `pip` (not recommended)\nWhile installation via `pip` is possible, it is not recommended because users must ensure\nthat the correct Python version is already installed and that they are using a virtual python environment.\n\n```bash\npip install grz-cli\n```\n##### Update instructions:\nUse the following command to update the tool:\n```bash\npip upgrade grz-cli\n```\n\n\n#### Docker\nDocker images are available via biocontainers at [https://biocontainers.pro/tools/grz-cli](https://biocontainers.pro/tools/grz-cli).\n\nThe build process can take at least a few days after the Bioconda release, so double-check that the latest version in Bioconda is also the latest Docker image version.\n\n## Usage\n\n### Configuration\n**The configuration file will be provided by your associated GRZ, please place it into `~/.config/grz-cli/config.yaml`.**\n\nThe tool requires a configuration file in YAML format to specify the S3 bucket and other options.\nFor an exemplary configuration, see [resources/config.yaml](resources/config.yaml).\n\nS3 access and secret key can be listed either in the config file or as environment variable (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`).\n\n### Exemplary submission procedure\nAfter preparing your submission as outlined above, you can use the following commands to validate, encrypt and upload the submission: \n```sh\n# Validate the submission\ngrz-cli validate --submission-dir EXAMPLE_SUBMISSION\n\n# Encrypt the submission\ngrz-cli encrypt --submission-dir EXAMPLE_SUBMISSION\n\n# Upload the submission\ngrz-cli upload --submission-dir EXAMPLE_SUBMISSION\n```\n\n### Troubleshooting\n**In case of issues, please re-run your commands with `grz-cli --log-level DEBUG --log-file <your-log-file.log> [...]` and submit the log file to the GRZ data steward!**\n\n## Command-Line Interface\n\n`grz-cli` provides a command-line interface with the following subcommands:\n\n### validate\n\nIt is recommended to run this command before continuing with encryption and upload.\nProgress files are stored relative to the submission directory.\n\n- `--submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]\n\nExample usage:\n\n```bash\ngrz_cli validate --submission-dir foo\n```\n\n### encrypt\n\nIf a working directory is not provided, then the current directory is used automatically. The log-files are going to be stored in the sub-folder of the working directory.\nFiles are stored in a folder named `encrypted_files` as a sub-folder of the working directory.\n\n- `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'files/' directories [**Required**]\n- `-c, --config-file`: Path to config file [_optional_]\n\n```bash\ngrz-cli encrypt --submission-dir foo\n```\n\n### upload\n\nUpload the submission into a S3 structure of a GRZ.\n\n- `-s, --submission-dir`: Path to the submission directory containing both 'metadata/' and 'encrypted_files/' directories [**Required**]\n- `-c, --config-file`: Path to config file [_optional_]\n\nExample usage:\n\n```bash\ngrz-cli upload --submission-dir foo\n```\n\n## Testing\n\nPlease note that binary files used for testing are managed with [Git LFS](https://git-lfs.com), which will be needed to clone them locally with the git repository.\n\nTo run the tests, navigate to the root directory of your project and invoke `pytest`.\nAlternatively, install `uv` and `tox` and run `uv run tox`.\n\n## Contributing\n\n### Running unreleased/development versions\n\nFirst, install `uv`.\nWe recommend using Conda or [Pixi](https://pixi.sh/latest).\n\nAfter cloning the desired branch of the `grz-tools` repo locally, you can run `grz-cli` directly from the repo using:\n\n```\nuv run --project path/to/cloned/grz-tools grz-cli --help\n```\n\n## License\n\nThis project is licensed under the MIT License \u2014 see the [LICENSE](LICENSE) file for details.\n\n## Acknowledgements\n\nParts of `crypt4gh` code is used in modified form\n\n",
"bugtrack_url": null,
"license": null,
"summary": "Tool for validation, encryption and upload of MV submissions to GDCs.",
"version": "1.1.2",
"project_urls": {
"Documentation": "https://github.com/BfArM-MVH/grz-tools/tree/main/packages/grz-cli",
"Homepage": "https://github.com/BfArM-MVH/grz-tools",
"Issues": "https://github.com/BfArM-MVH/grz-tools/issues",
"Repository": "https://github.com/BfArM-MVH/grz-tools"
},
"split_keywords": [
"gdc",
" grz",
" s3"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "83661bf5203713f3c0c0a96e6524091d6ee3ca179e987362a6bd14ead542f328",
"md5": "e1530bd13a0d9235165fd7b307620fa7",
"sha256": "5bb75039c5dfc5027badf937c5b0cccd4c4845dd12cacc9d69892fabe9b9d8f3"
},
"downloads": -1,
"filename": "grz_cli-1.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e1530bd13a0d9235165fd7b307620fa7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.12",
"size": 10002,
"upload_time": "2025-08-05T11:50:03",
"upload_time_iso_8601": "2025-08-05T11:50:03.843477Z",
"url": "https://files.pythonhosted.org/packages/83/66/1bf5203713f3c0c0a96e6524091d6ee3ca179e987362a6bd14ead542f328/grz_cli-1.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "c308112b49a582725e4593624717c4ac2fc3cb53c231d1e9a348de9e7d6e747b",
"md5": "c5577a9248902bae7d7a9c90d2fae278",
"sha256": "0ab58cc4c0101eb0c6106f969785ebea986ebb217c084be218de9050936bf338"
},
"downloads": -1,
"filename": "grz_cli-1.1.2.tar.gz",
"has_sig": false,
"md5_digest": "c5577a9248902bae7d7a9c90d2fae278",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.12",
"size": 46254,
"upload_time": "2025-08-05T11:50:04",
"upload_time_iso_8601": "2025-08-05T11:50:04.978140Z",
"url": "https://files.pythonhosted.org/packages/c3/08/112b49a582725e4593624717c4ac2fc3cb53c231d1e9a348de9e7d6e747b/grz_cli-1.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-05 11:50:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "BfArM-MVH",
"github_project": "grz-tools",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "grz-cli"
}
