# gen3utils
Utils for Gen3 commons management. Gen3utils includes a CLI and is intended to be used as an
external facing tool for validating configuration, generating release notes, posting deployment changes,
logging parser information, and more.
## Install with pip
pip install gen3utils
## manifest.json validation
Validate one or more `manifest.json` files:
```
gen3utils validate_manifest cdis-manifest/*/manifest.json
```
The validation settings can be updated by modifying [this file](gen3utils/manifest/validation_config.yaml).
## etlMapping.yaml validation
Validate an `etlMapping.yaml` file against the dictionary URL specified in a `manifest.json` file:
```
gen3utils validate_etl_mapping etlMapping.yaml manifest.json
```
## Portal Configuration (gitops.json) validation
Validate a `gitops.json` file against the dictionary URL specified in a `manifest.json` file and an etlMapping.yaml file. Adds a comment to a pull request listing all the errors encountered when validating against etlMapping.yaml
```
gen3utils validate_portal_config etlMapping.yaml manifest.json gitops.json <username>/<repository> <pull request number>
```
To run without making a pull request comment
```
gen3utils validate_portal_config etlMapping.yaml manifest.json gitops.json
```
## Comment on a PR with any deployment changes when updating manifest services
The command requires the name of the repository, the pull request number and **a `GITHUB_TOKEN` environment variable** containing a token with read and write access to the repository. It also comments a warning if a service is pinned on a branch.
```
pip install gen3utils
gen3utils post-deployment-changes <username>/<repository> <pull request number>
```
## Log parser for CTDS log pipeline
```
pip install gen3utils
gen3utils s3log --help
gen3utils s3log [OPTIONS] BUCKET PREFIX SCRIPT
```
Run `SCRIPT` in Gen3 logs under S3 `BUCKET:PREFIX`. The `SCRIPT` should be importable defining a method like this:
```
def handle_row(obj, line):
if 1 + 1 == 2:
return line
```
For example, to process logs in bucket `my-commons-logs` at prefix `my-logs` with a `gen3utils/script.py` file:
```
pip install gen3utils
gen3utils s3log my-commons-logs my-logs gen3utils.script
```
## Running tests locally
```
poetry install -vv
poetry run pytest -vv ./tests
```
Raw data
{
"_id": null,
"home_page": "https://github.com/uc-cdis/gen3utils",
"name": "gen3utils",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": null,
"author": "CTDS UChicago",
"author_email": "cdis@uchicago.edu",
"download_url": "https://files.pythonhosted.org/packages/18/73/f3fc6852f5f333297792e33161789294d6871c220f39595dccff1f056fbf/gen3utils-0.10.6.tar.gz",
"platform": null,
"description": "# gen3utils\n\nUtils for Gen3 commons management. Gen3utils includes a CLI and is intended to be used as an\nexternal facing tool for validating configuration, generating release notes, posting deployment changes,\nlogging parser information, and more.\n\n## Install with pip\npip install gen3utils\n\n## manifest.json validation\n\nValidate one or more `manifest.json` files:\n```\ngen3utils validate_manifest cdis-manifest/*/manifest.json\n```\n\nThe validation settings can be updated by modifying [this file](gen3utils/manifest/validation_config.yaml).\n\n## etlMapping.yaml validation\n\nValidate an `etlMapping.yaml` file against the dictionary URL specified in a `manifest.json` file:\n```\ngen3utils validate_etl_mapping etlMapping.yaml manifest.json\n```\n\n## Portal Configuration (gitops.json) validation\n\nValidate a `gitops.json` file against the dictionary URL specified in a `manifest.json` file and an etlMapping.yaml file. Adds a comment to a pull request listing all the errors encountered when validating against etlMapping.yaml\n```\ngen3utils validate_portal_config etlMapping.yaml manifest.json gitops.json <username>/<repository> <pull request number>\n```\nTo run without making a pull request comment\n```\ngen3utils validate_portal_config etlMapping.yaml manifest.json gitops.json\n```\n\n\n## Comment on a PR with any deployment changes when updating manifest services\n\nThe command requires the name of the repository, the pull request number and **a `GITHUB_TOKEN` environment variable** containing a token with read and write access to the repository. It also comments a warning if a service is pinned on a branch.\n```\npip install gen3utils\ngen3utils post-deployment-changes <username>/<repository> <pull request number>\n```\n\n## Log parser for CTDS log pipeline\n\n```\npip install gen3utils\ngen3utils s3log --help\ngen3utils s3log [OPTIONS] BUCKET PREFIX SCRIPT\n```\n\nRun `SCRIPT` in Gen3 logs under S3 `BUCKET:PREFIX`. The `SCRIPT` should be importable defining a method like this:\n```\ndef handle_row(obj, line):\n if 1 + 1 == 2:\n return line\n```\n\nFor example, to process logs in bucket `my-commons-logs` at prefix `my-logs` with a `gen3utils/script.py` file:\n```\npip install gen3utils\ngen3utils s3log my-commons-logs my-logs gen3utils.script\n```\n\n## Running tests locally\n\n```\npoetry install -vv\npoetry run pytest -vv ./tests\n```\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Utils for Gen3 Commons management",
"version": "0.10.6",
"project_urls": {
"Homepage": "https://github.com/uc-cdis/gen3utils",
"Repository": "https://github.com/uc-cdis/gen3utils"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "86e13f492251ab03a22e2f59ebbf35e7571aa6c1f6d5a2b03f883538c0d70b88",
"md5": "d509c5d2673b984ae42a72bcb232c024",
"sha256": "7494a5ce0624139e1555f3c2eac7b570baa457d0f533b7ce3b0fb3e3a0ff0fa7"
},
"downloads": -1,
"filename": "gen3utils-0.10.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d509c5d2673b984ae42a72bcb232c024",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 31551,
"upload_time": "2024-10-01T16:55:32",
"upload_time_iso_8601": "2024-10-01T16:55:32.132659Z",
"url": "https://files.pythonhosted.org/packages/86/e1/3f492251ab03a22e2f59ebbf35e7571aa6c1f6d5a2b03f883538c0d70b88/gen3utils-0.10.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1873f3fc6852f5f333297792e33161789294d6871c220f39595dccff1f056fbf",
"md5": "89b8af657d7b28f8b1721aad156c19a2",
"sha256": "407fd63ca4925fd5941d3e6c2b7de7be0a22a73a6d030608c807a6f8f597a32c"
},
"downloads": -1,
"filename": "gen3utils-0.10.6.tar.gz",
"has_sig": false,
"md5_digest": "89b8af657d7b28f8b1721aad156c19a2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 26207,
"upload_time": "2024-10-01T16:55:33",
"upload_time_iso_8601": "2024-10-01T16:55:33.626409Z",
"url": "https://files.pythonhosted.org/packages/18/73/f3fc6852f5f333297792e33161789294d6871c220f39595dccff1f056fbf/gen3utils-0.10.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-01 16:55:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "uc-cdis",
"github_project": "gen3utils",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "gen3utils"
}
JSON
