[](https://pypi.org/project/otf-addons-gcp/)
[](https://codecov.io/gh/adammcdonagh/otf-addons-gcp)
[](https://github.com/adammcdonagh/otf-addons-gcp/blob/master/LICENSE)
[](https://github.com/adammcdonagh/otf-addons-gcp/issues)
[](https://github.com/adammcdonagh/otf-addons-gcp/stargazers)
This repository contains addons to allow integration with GCP Cloud Storage via [Open Task Framework (OTF)](https://github.com/adammcdonagh/open-task-framework)
Open Task Framework (OTF) is a Python based framework to make it easy to run predefined file transfers and scripts/commands on remote machines.
This addons allows pushes and pulls of Files from (and to) GCP Cloud Storage Buckets.
# GCP SA Credentials
This package uses `google-auth` to get OAuth2.0 creds for the GCP Token API. This includes `access_token`.
Prior to using this framework, a GCP IAM Service Account should be created through gcloud CLI or Web portal. The JSON credentials for the Service account should be exported and stored locally.
(currently the .gitignore excludes any .json in the tests folder by default)
The Service account crednetials should be included in `tests/testFiles` and require only the following properties:
```
{
private_key= "-----BEGIN PRIVATE KEY-----\nMIIEv ....",
client_email= "file.upload@projectName.iam.gserviceaccount.com",
token_uri= "https://oauth2.googleapis.com/token"
}
```
The rest of the keys can be removed.
Running this OTF Addon, requires test files being placed in the `src/tmp` directory. Running the tests will perform an upload and download from/to GCP Cloud Storage.
Each request will generate a new Service Account Token, token refresh/state-keeping through AWS SSM or local cache has not been implemented yet.
# Transfers
Transfers require a few additional arguments to work. These are:
- bucket: The bucket name of the Cloud Storage instance
- credentials: JSON object containing the above 3 credential properties.
### Supported features
- File transfer: ingress/egress from/to Cloud Storage
- Renaming functionality
- PostCopy functionality
- fileWatch functionality
# Configuration
JSON configs for transfers can be defined as follows:
## Example file upload
```json
"destination": {
"bucket": "bucketname",
"directory": "directory/nested",
"protocol": {
"name": "opentaskpy.addons.gcp.remotehandlers.bucket.BucketTransfer",
"credentials": "{LOOKUP DEFINITION FOR SA CREDENTIALS}",
},
"rename":{
"pattern":"Hithere",
"sub": "Hi"
}
}
```
## Example file download
```json
"source": {
"bucket": "bucketname",
"directory": "directory/nested",
"protocol": {
"name": "opentaskpy.addons.gcp.remotehandlers.bucket.BucketTransfer",
"credentials": "{LOOKUP DEFINITION FOR SA CREDENTIALS}",
},
"postCopyAction":{
"action":"move",
"destination": "directory/nested/processed",
"pattern" : "(?<![^ ])(?=[^ ])(?!ab)", ## Regex for prefixing
"sub" : "Archived_"
},
"fileRegex": ".*//.txt$" ## accepts re module matching
}
```
Raw data
{
"_id": null,
"home_page": null,
"name": "otf-addons-gcp",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "automation, task, framework, gcp, cloudstorage, otf",
"author": null,
"author_email": "Aldo Troiano <a.troiano@reply.com>",
"download_url": "https://files.pythonhosted.org/packages/5f/03/f16879b62fe42cc03428bfe37373528c6914012b15474cca4637c4e07ccc/otf_addons_gcp-24.37.0.tar.gz",
"platform": null,
"description": "[](https://pypi.org/project/otf-addons-gcp/)\n\n[](https://codecov.io/gh/adammcdonagh/otf-addons-gcp)\n[](https://github.com/adammcdonagh/otf-addons-gcp/blob/master/LICENSE)\n[](https://github.com/adammcdonagh/otf-addons-gcp/issues)\n[](https://github.com/adammcdonagh/otf-addons-gcp/stargazers)\n\nThis repository contains addons to allow integration with GCP Cloud Storage via [Open Task Framework (OTF)](https://github.com/adammcdonagh/open-task-framework)\n\nOpen Task Framework (OTF) is a Python based framework to make it easy to run predefined file transfers and scripts/commands on remote machines.\n\nThis addons allows pushes and pulls of Files from (and to) GCP Cloud Storage Buckets.\n\n# GCP SA Credentials\n\nThis package uses `google-auth` to get OAuth2.0 creds for the GCP Token API. This includes `access_token`.\n\nPrior to using this framework, a GCP IAM Service Account should be created through gcloud CLI or Web portal. The JSON credentials for the Service account should be exported and stored locally.\n(currently the .gitignore excludes any .json in the tests folder by default)\n\nThe Service account crednetials should be included in `tests/testFiles` and require only the following properties:\n\n```\n{\n private_key= \"-----BEGIN PRIVATE KEY-----\\nMIIEv ....\",\n client_email= \"file.upload@projectName.iam.gserviceaccount.com\",\n token_uri= \"https://oauth2.googleapis.com/token\"\n}\n```\n\nThe rest of the keys can be removed.\n\nRunning this OTF Addon, requires test files being placed in the `src/tmp` directory. Running the tests will perform an upload and download from/to GCP Cloud Storage.\n\nEach request will generate a new Service Account Token, token refresh/state-keeping through AWS SSM or local cache has not been implemented yet.\n\n# Transfers\n\nTransfers require a few additional arguments to work. These are:\n\n- bucket: The bucket name of the Cloud Storage instance\n- credentials: JSON object containing the above 3 credential properties.\n\n### Supported features\n\n- File transfer: ingress/egress from/to Cloud Storage\n - Renaming functionality\n - PostCopy functionality\n - fileWatch functionality\n\n# Configuration\n\nJSON configs for transfers can be defined as follows:\n\n## Example file upload\n\n```json\n\"destination\": {\n \"bucket\": \"bucketname\",\n \"directory\": \"directory/nested\",\n \"protocol\": {\n \"name\": \"opentaskpy.addons.gcp.remotehandlers.bucket.BucketTransfer\",\n \"credentials\": \"{LOOKUP DEFINITION FOR SA CREDENTIALS}\",\n },\n \"rename\":{\n \"pattern\":\"Hithere\",\n \"sub\": \"Hi\"\n }\n}\n```\n\n## Example file download\n\n```json\n\"source\": {\n \"bucket\": \"bucketname\",\n \"directory\": \"directory/nested\",\n \"protocol\": {\n \"name\": \"opentaskpy.addons.gcp.remotehandlers.bucket.BucketTransfer\",\n \"credentials\": \"{LOOKUP DEFINITION FOR SA CREDENTIALS}\",\n },\n \"postCopyAction\":{\n \"action\":\"move\",\n \"destination\": \"directory/nested/processed\",\n \"pattern\" : \"(?<![^ ])(?=[^ ])(?!ab)\", ## Regex for prefixing\n \"sub\" : \"Archived_\"\n },\n \"fileRegex\": \".*//.txt$\" ## accepts re module matching\n}\n```\n",
"bugtrack_url": null,
"license": "GPLv3",
"summary": "Addons for opentaskpy, giving it the ability to push/pull files via GCP Cloud Storage.",
"version": "24.37.0",
"project_urls": {
"Bug Tracker": "https://github.com/adammcdonagh/otf-addons-gcp/issues",
"Changelog": "https://github.com/adammcdonagh/otf-addons-gcp/blob/main/CHANGELOG.md",
"Homepage": "https://github.com/adammcdonagh/otf-addons-gcp"
},
"split_keywords": [
"automation",
" task",
" framework",
" gcp",
" cloudstorage",
" otf"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5f03f16879b62fe42cc03428bfe37373528c6914012b15474cca4637c4e07ccc",
"md5": "002683569c71f4126f779db01451907e",
"sha256": "07a25bc42e2ad9dd3d4b52231f658a13768dc6bcb283d0eea72b29d63ac47910"
},
"downloads": -1,
"filename": "otf_addons_gcp-24.37.0.tar.gz",
"has_sig": false,
"md5_digest": "002683569c71f4126f779db01451907e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 28306,
"upload_time": "2024-09-09T17:39:21",
"upload_time_iso_8601": "2024-09-09T17:39:21.187452Z",
"url": "https://files.pythonhosted.org/packages/5f/03/f16879b62fe42cc03428bfe37373528c6914012b15474cca4637c4e07ccc/otf_addons_gcp-24.37.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-09 17:39:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "adammcdonagh",
"github_project": "otf-addons-gcp",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "otf-addons-gcp"
}
JSON
