# Gen3 Utilities
Utilities to manage Gen3 schemas, projects and submissions.
## Installation
```
$ pip install gen3_util
$ gen3_util version
version: 0.0.12
```
## Use
```
$gen3_util --help
Usage: gen3_util [OPTIONS] COMMAND [ARGS]...
Gen3 Management Utilities
Options:
--config TEXT Path to config file. GEN3_UTIL_CONFIG
--format [yaml|json|text] Result format. GEN3_UTIL_FORMAT [default: yaml]
--profile TEXT Connection name. GEN3_UTIL_PROFILE See
https://bit.ly/3NbKGi4
--state_dir TEXT Directory for file transfer state
GEN3_UTIL_STATE_DIR [default: ~/.gen3/gen3_util]
--help Show this message and exit.
Commands:
projects Manage Gen3 projects.
buckets Project buckets.
meta Manage meta data.
files Manage file transfers.
access Manage access requests.
config Configure this utility.
jobs Manage Gen3 jobs.
users Manage project membership.
version Print version
ping Test connectivity to Gen3 endpoint.
```
## Connectivity
* Uses [gen3-client](https://gen3.org/resources/user/gen3-client/#2-configure-a-profile-with-credentials) for authentication
## Use cases
> I need to verify connectivity.
```
$ gen3_util --profile <connection-name> ping
msg: 'Configuration OK: Connected using profile:production'
endpoint: https://aced-idp.org
username: user@example.com
```
> I need to see what projects exist
```
$ gen3_util projects ls
endpoint: https://aced-training.compbio.ohsu.edu
msg: OK
projects:
- /programs
- /programs/aced
- /programs/aced/projects
- /programs/aced/projects/Alcoholism
- /programs/aced/projects/Alzheimers
- /programs/aced/projects/Breast_Cancer
- /programs/aced/projects/Colon_Cancer
- /programs/aced/projects/Diabetes
- /programs/aced/projects/Lung_Cancer
- /programs/aced/projects/Prostate_Cancer
```
> I need to see what buckets are associated with the commons
```
$ gen3_util buckets ls
endpoint: https://aced-idp.org
buckets:
GS_BUCKETS: {}
S3_BUCKETS:
<bucket-name>:
endpoint_url: https://<example.com>/<bucket-name>
programs:
- <program-name>
region: <region-name>
```
Note: the workflow to create new projects and adding users to those projects is a multi-step process. The following commands will create the requests, but they will need to be signed by a user with approval privileges. See requestor's ["Functionality and flow" documentation](https://github.com/uc-cdis/requestor/blob/master/docs/functionality_and_flow.md).
> I need to create a project
```text
$ gen3_util projects new --project_id=aced-my_new_project
requests:
- status: DRAFT
policy_id: programs.aced.projects.my_new_project_writer
- status: DRAFT
policy_id: programs.aced.projects.my_new_project_reader
```
> I need to add a user to that project
```text
$ gen3_util users add --project_id=aced-my_new_project --username linus.pauling@osu.edu --write
requests:
- status: DRAFT
username: linus.pauling@osu.edu
policy_id: programs.aced.projects.my_new_project_writer
- status: DRAFT
username: linus.pauling@osu.edu
policy_id: programs.aced.projects.my_new_project_reader
```
> Before proceeding, a user with approval privileges will need to sign the requests
```text
gen3_util access sign
```
Note: Adding files to a project requires a multi-step process. In addition to uploading a file, associated metadata must be created and uploaded to the commons. This will be done automatically for simple use cases, but may be overridden for more complex use cases such as bulk upload or prepared FHIR data.
> I want to create a simple project structure with a set of files
```text
Usage: gen3_util files manifest put [OPTIONS] LOCAL_PATH [REMOTE_PATH]
Add file meta information to the manifest.
local_path: path to file on local file system
remote_path: name of the file in bucket, defaults to local_path
Options:
--project_id TEXT Gen3 program-project
--specimen_id TEXT fhir specimen identifier
--patient_id TEXT fhir patient identifier
--task_id TEXT fhir task identifier
--observation_id TEXT fhir observation identifier
--md5 TEXT MD5 sum, if not provided, will be calculated before
upload
--help Show this message and exit.
```
<img width="673" alt="image" src="https://github.com/ACED-IDP/aced-idp.github.io/assets/47808/801cfb7a-bff3-4c4f-97be-acba79830787">
The following identifiers create a simple graph of the data from the DocumentReference(file) to its parents.
```text
--project_id TEXT Gen3 program-project [required]
--specimen_id TEXT fhir specimen identifier [optional]
--patient_id TEXT fhir patient identifier [optional]
--task_id TEXT fhir task identifier [optional]
--observation_id TEXT fhir observation identifier [optional]
```
Adding one or more commands via `gen3_util files manifest put` will create a manifest file that can be used to upload files and metadata to the commons.
> I need to upload those files to the instance, and automatically create meta data
```
$ gen3_util files manifest upload
```
## Development Setup
```
$ git clone git@github.com:ACED-IDP/gen3_util.git
$ cd gen3_util
$ python3 -m venv venv ; source venv/bin/activate
$ pip install -r requirements.txt
$ pip install -e .
```
## Test
* fixtures - data for testing environment
```
tests/fixtures/
└── custom_config
└── config.yaml # testing configuration
```
* test parameters
```
tests/
├── integration
│ └── conftest.py
└── unit
└── conftest.py
```
* running tests
```
$ pytest --cov=gen3_util
88%
```
* pre commit tests
A reasonable set of checks, including running unit tests prior to each commit. You can run these tests on demand by:
```
$ pre-commit install
$ pre-commit run --all-files
debug statements (python)................................................Passed
check python ast.........................................................Passed
fix utf-8 byte order marker..............................................Passed
check json...........................................(no files to check)Skipped
detect private key.......................................................Passed
check yaml...............................................................Passed
check for added large files..............................................Passed
check that scripts with shebangs are executable..........................Passed
check for case conflicts.................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
mixed line ending........................................................Passed
run our unit tests.......................................................Passed
```
## Distribution
- PyPi
```
# update pypi
# pypi credentials - see https://twine.readthedocs.io/en/stable/#environment-variables
export TWINE_USERNAME= # the username to use for authentication to the repository.
export TWINE_PASSWORD= # the password to use for authentication to the repository.
# this could be maintained as so: export $(cat .env | xargs)
rm -r dist/
python3 setup.py sdist bdist_wheel
twine upload dist/*
```
Raw data
{
"_id": null,
"home_page": "https://github.com/ACED-IDP/gen3_util",
"name": "gen3-util",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9, <4",
"maintainer_email": "",
"keywords": "gen3 bioinformatics",
"author": "Ellrott Lab",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/fc/f6/c6a5d4bdb88e89b874ba471cdae77cc195f52241b652ebe01cca50b29899/gen3_util-0.0.16.tar.gz",
"platform": null,
"description": "\n# Gen3 Utilities\n\nUtilities to manage Gen3 schemas, projects and submissions.\n\n## Installation\n```\n\n$ pip install gen3_util\n\n$ gen3_util version\nversion: 0.0.12\n\n\n```\n\n\n## Use\n\n```\n$gen3_util --help\nUsage: gen3_util [OPTIONS] COMMAND [ARGS]...\n\n Gen3 Management Utilities\n\nOptions:\n --config TEXT Path to config file. GEN3_UTIL_CONFIG\n --format [yaml|json|text] Result format. GEN3_UTIL_FORMAT [default: yaml]\n --profile TEXT Connection name. GEN3_UTIL_PROFILE See\n https://bit.ly/3NbKGi4\n\n --state_dir TEXT Directory for file transfer state\n GEN3_UTIL_STATE_DIR [default: ~/.gen3/gen3_util]\n\n --help Show this message and exit.\n\nCommands:\n projects Manage Gen3 projects.\n buckets Project buckets.\n meta Manage meta data.\n files Manage file transfers.\n access Manage access requests.\n config Configure this utility.\n jobs Manage Gen3 jobs.\n users Manage project membership.\n version Print version\n ping Test connectivity to Gen3 endpoint.\n\n\n```\n\n## Connectivity\n\n* Uses [gen3-client](https://gen3.org/resources/user/gen3-client/#2-configure-a-profile-with-credentials) for authentication\n\n\n## Use cases\n\n> I need to verify connectivity.\n\n```\n$ gen3_util --profile <connection-name> ping\nmsg: 'Configuration OK: Connected using profile:production'\nendpoint: https://aced-idp.org\nusername: user@example.com\n\n```\n\n> I need to see what projects exist\n\n```\n$ gen3_util projects ls\n\nendpoint: https://aced-training.compbio.ohsu.edu\nmsg: OK\nprojects:\n- /programs\n- /programs/aced\n- /programs/aced/projects\n- /programs/aced/projects/Alcoholism\n- /programs/aced/projects/Alzheimers\n- /programs/aced/projects/Breast_Cancer\n- /programs/aced/projects/Colon_Cancer\n- /programs/aced/projects/Diabetes\n- /programs/aced/projects/Lung_Cancer\n- /programs/aced/projects/Prostate_Cancer\n```\n\n> I need to see what buckets are associated with the commons\n\n```\n$ gen3_util buckets ls\nendpoint: https://aced-idp.org\nbuckets:\n GS_BUCKETS: {}\n S3_BUCKETS:\n <bucket-name>:\n endpoint_url: https://<example.com>/<bucket-name>\n programs:\n - <program-name>\n region: <region-name>\n\n```\n\nNote: the workflow to create new projects and adding users to those projects is a multi-step process. The following commands will create the requests, but they will need to be signed by a user with approval privileges. See requestor's [\"Functionality and flow\" documentation](https://github.com/uc-cdis/requestor/blob/master/docs/functionality_and_flow.md).\n\n> I need to create a project\n\n```text\n$ gen3_util projects new --project_id=aced-my_new_project\nrequests:\n- status: DRAFT\n policy_id: programs.aced.projects.my_new_project_writer\n- status: DRAFT\n policy_id: programs.aced.projects.my_new_project_reader\n\n```\n\n\n> I need to add a user to that project\n\n```text\n$ gen3_util users add --project_id=aced-my_new_project --username linus.pauling@osu.edu --write\nrequests:\n- status: DRAFT\n username: linus.pauling@osu.edu\n policy_id: programs.aced.projects.my_new_project_writer\n- status: DRAFT\n username: linus.pauling@osu.edu\n policy_id: programs.aced.projects.my_new_project_reader\n\n\n```\n\n> Before proceeding, a user with approval privileges will need to sign the requests\n\n```text\ngen3_util access sign\n```\n\nNote: Adding files to a project requires a multi-step process. In addition to uploading a file, associated metadata must be created and uploaded to the commons. This will be done automatically for simple use cases, but may be overridden for more complex use cases such as bulk upload or prepared FHIR data.\n\n\n\n> I want to create a simple project structure with a set of files\n\n```text\nUsage: gen3_util files manifest put [OPTIONS] LOCAL_PATH [REMOTE_PATH]\n\n Add file meta information to the manifest.\n\n local_path: path to file on local file system\n remote_path: name of the file in bucket, defaults to local_path\n\nOptions:\n --project_id TEXT Gen3 program-project\n --specimen_id TEXT fhir specimen identifier\n --patient_id TEXT fhir patient identifier\n --task_id TEXT fhir task identifier\n --observation_id TEXT fhir observation identifier\n --md5 TEXT MD5 sum, if not provided, will be calculated before\n upload\n\n --help Show this message and exit.\n\n```\n\n<img width=\"673\" alt=\"image\" src=\"https://github.com/ACED-IDP/aced-idp.github.io/assets/47808/801cfb7a-bff3-4c4f-97be-acba79830787\">\n\nThe following identifiers create a simple graph of the data from the DocumentReference(file) to its parents.\n\n```text\n --project_id TEXT Gen3 program-project [required]\n --specimen_id TEXT fhir specimen identifier [optional]\n --patient_id TEXT fhir patient identifier [optional]\n --task_id TEXT fhir task identifier [optional]\n --observation_id TEXT fhir observation identifier [optional]\n\n```\n\nAdding one or more commands via `gen3_util files manifest put` will create a manifest file that can be used to upload files and metadata to the commons.\n\n\n> I need to upload those files to the instance, and automatically create meta data\n\n```\n$ gen3_util files manifest upload\n```\n\n\n\n## Development Setup\n\n```\n$ git clone git@github.com:ACED-IDP/gen3_util.git\n$ cd gen3_util\n$ python3 -m venv venv ; source venv/bin/activate\n$ pip install -r requirements.txt\n$ pip install -e .\n\n```\n\n\n## Test\n\n* fixtures - data for testing environment\n\n```\ntests/fixtures/\n\u2514\u2500\u2500 custom_config\n \u2514\u2500\u2500 config.yaml # testing configuration\n\n```\n\n* test parameters\n\n```\ntests/\n\u251c\u2500\u2500 integration\n\u2502 \u2514\u2500\u2500 conftest.py\n\u2514\u2500\u2500 unit\n \u2514\u2500\u2500 conftest.py\n```\n\n* running tests\n\n```\n\n$ pytest --cov=gen3_util\n\n 88%\n\n\n```\n\n* pre commit tests\n\nA reasonable set of checks, including running unit tests prior to each commit. You can run these tests on demand by:\n\n```\n$ pre-commit install\n\n$ pre-commit run --all-files\ndebug statements (python)................................................Passed\ncheck python ast.........................................................Passed\nfix utf-8 byte order marker..............................................Passed\ncheck json...........................................(no files to check)Skipped\ndetect private key.......................................................Passed\ncheck yaml...............................................................Passed\ncheck for added large files..............................................Passed\ncheck that scripts with shebangs are executable..........................Passed\ncheck for case conflicts.................................................Passed\nfix end of files.........................................................Passed\ntrim trailing whitespace.................................................Passed\nmixed line ending........................................................Passed\nrun our unit tests.......................................................Passed\n\n```\n\n## Distribution\n\n- PyPi\n\n```\n# update pypi\n\n# pypi credentials - see https://twine.readthedocs.io/en/stable/#environment-variables\n\nexport TWINE_USERNAME= # the username to use for authentication to the repository.\nexport TWINE_PASSWORD= # the password to use for authentication to the repository.\n\n# this could be maintained as so: export $(cat .env | xargs)\n\nrm -r dist/\npython3 setup.py sdist bdist_wheel\ntwine upload dist/*\n```\n",
"bugtrack_url": null,
"license": "",
"summary": "Commons utilities",
"version": "0.0.16",
"project_urls": {
"Bug Reports": "https://github.com/ACED-IDP/gen3_util/issues",
"Homepage": "https://github.com/ACED-IDP/gen3_util",
"Source": "https://github.com/ACED-IDP/gen3_util"
},
"split_keywords": [
"gen3",
"bioinformatics"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fe91555fba8d62ed3be3ecd62008341ac8ebd91cf6c68ee69d9a7ebfc15f2a16",
"md5": "cf8662df72191c697ab886cb74c4ae0e",
"sha256": "56f87c94c5c5d6be0a02c327624cc61f4a91887d1cc7e1c4e5122b56da76cb4c"
},
"downloads": -1,
"filename": "gen3_util-0.0.16-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cf8662df72191c697ab886cb74c4ae0e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9, <4",
"size": 123123,
"upload_time": "2024-02-09T14:42:26",
"upload_time_iso_8601": "2024-02-09T14:42:26.064769Z",
"url": "https://files.pythonhosted.org/packages/fe/91/555fba8d62ed3be3ecd62008341ac8ebd91cf6c68ee69d9a7ebfc15f2a16/gen3_util-0.0.16-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fcf6c6a5d4bdb88e89b874ba471cdae77cc195f52241b652ebe01cca50b29899",
"md5": "92538f23f67610ba496851d696fe6fe7",
"sha256": "637bf0b54e330d036922555860374f05464d58c56caf18472d3a9a3a987cc876"
},
"downloads": -1,
"filename": "gen3_util-0.0.16.tar.gz",
"has_sig": false,
"md5_digest": "92538f23f67610ba496851d696fe6fe7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9, <4",
"size": 76966,
"upload_time": "2024-02-09T14:42:27",
"upload_time_iso_8601": "2024-02-09T14:42:27.771006Z",
"url": "https://files.pythonhosted.org/packages/fc/f6/c6a5d4bdb88e89b874ba471cdae77cc195f52241b652ebe01cca50b29899/gen3_util-0.0.16.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-09 14:42:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ACED-IDP",
"github_project": "gen3_util",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "gen3-util"
}
JSON
