| Name | pipen-gcs JSON |
| Version |
0.0.3
JSON |
| download |
| home_page | None |
| Summary | A plugin for pipen to handle file metadata in Google Cloud Storage |
| upload_time | 2024-12-20 03:40:04 |
| maintainer | None |
| docs_url | None |
| author | pwwang |
| requires_python | <4.0,>=3.9 |
| license | MIT |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# pipen-gcs
A plugin for [pipen][1] to handle files in Google Cloud Storage
## Installation
```bash
pip install -U pipen-gcs
# uninstall to disable
pip uninstall pipen-gcs
```
## Usage
```python
from pipen import Proc, Pipen
class MyProc(Proc):
input = "infile:file"
input_data = ["gs://bucket/path/to/file"]
output = "outfile:file:gs://bucket/path/to/output"
script = "cat {{in.infile}} > {{out.outfile}}"
class MyPipen(Pipen):
starts = MyProc
# input files/directories will be downloaded to /tmp
# output files/directories will be generated in /tmp and then uploaded
# to the cloud storage
plugin_opts = {"gcs_localize": "/tmp"}
if __name__ == "__main__":
MyPipen().run()
```
You can also disable localization, then you will have to handle the
cloud storage files yourself.
```python
from pipen import Proc, Pipen
class MyProc(Proc):
input = "infile:file"
input_data = ["gs://bucket/path/to/file"]
output = "outfile:file:gs://bucket/path/to/output"
script = "gsutil cp {{in.infile}} {{out.outfile}}"
class MyPipen(Pipen):
starts = MyProc
plugin_opts = {"gcs_localize": False}
if __name__ == "__main__":
MyPipen().run()
```
## Configuration
- `gcs_localize`: The directory to localize the cloud storage files. If
set to `False`, the files will not be localized. Default is `False`.
- `gcs_localize_force`: If set to `True`, the files will be localized
even if they exist locally. Default is `False`.
- `gcs_credentials`: The path to the Google Cloud Service Account
credentials file.
[1]: https://github.com/pwwang/pipen
Raw data
{
"_id": null,
"home_page": null,
"name": "pipen-gcs",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": null,
"author": "pwwang",
"author_email": "1188067+pwwang@users.noreply.github.com",
"download_url": "https://files.pythonhosted.org/packages/2b/96/c91877960e6dc89c9bd7833f75dbaa7ea93c6849c9ef2ce4110cf0e459a0/pipen_gcs-0.0.3.tar.gz",
"platform": null,
"description": "# pipen-gcs\n\nA plugin for [pipen][1] to handle files in Google Cloud Storage\n\n## Installation\n\n```bash\npip install -U pipen-gcs\n\n# uninstall to disable\npip uninstall pipen-gcs\n```\n\n## Usage\n\n```python\nfrom pipen import Proc, Pipen\n\nclass MyProc(Proc):\n input = \"infile:file\"\n input_data = [\"gs://bucket/path/to/file\"]\n output = \"outfile:file:gs://bucket/path/to/output\"\n script = \"cat {{in.infile}} > {{out.outfile}}\"\n\nclass MyPipen(Pipen):\n starts = MyProc\n # input files/directories will be downloaded to /tmp\n # output files/directories will be generated in /tmp and then uploaded\n # to the cloud storage\n plugin_opts = {\"gcs_localize\": \"/tmp\"}\n\nif __name__ == \"__main__\":\n MyPipen().run()\n```\n\nYou can also disable localization, then you will have to handle the\ncloud storage files yourself.\n\n```python\nfrom pipen import Proc, Pipen\n\nclass MyProc(Proc):\n input = \"infile:file\"\n input_data = [\"gs://bucket/path/to/file\"]\n output = \"outfile:file:gs://bucket/path/to/output\"\n script = \"gsutil cp {{in.infile}} {{out.outfile}}\"\n\nclass MyPipen(Pipen):\n starts = MyProc\n plugin_opts = {\"gcs_localize\": False}\n\nif __name__ == \"__main__\":\n MyPipen().run()\n```\n\n## Configuration\n\n- `gcs_localize`: The directory to localize the cloud storage files. If\n set to `False`, the files will not be localized. Default is `False`.\n- `gcs_localize_force`: If set to `True`, the files will be localized\n even if they exist locally. Default is `False`.\n- `gcs_credentials`: The path to the Google Cloud Service Account\n credentials file.\n\n[1]: https://github.com/pwwang/pipen\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A plugin for pipen to handle file metadata in Google Cloud Storage",
"version": "0.0.3",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8f053b2cbdb39211a32092c54db9a3cb0aad61261310720f6d27c1045ff4e9f2",
"md5": "a9675be16a5d36c6239df71f8bb3d80d",
"sha256": "cc72d667440f1ac489cd23f7613d35dfb161fbda110d80253db81fd7a122adc1"
},
"downloads": -1,
"filename": "pipen_gcs-0.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a9675be16a5d36c6239df71f8bb3d80d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 9963,
"upload_time": "2024-12-20T03:40:02",
"upload_time_iso_8601": "2024-12-20T03:40:02.338856Z",
"url": "https://files.pythonhosted.org/packages/8f/05/3b2cbdb39211a32092c54db9a3cb0aad61261310720f6d27c1045ff4e9f2/pipen_gcs-0.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2b96c91877960e6dc89c9bd7833f75dbaa7ea93c6849c9ef2ce4110cf0e459a0",
"md5": "a61469c49bfeaa842ab28ac3b98d1e91",
"sha256": "0ec9e7c053f2bbf6913b1677ba6194601dffe1d3e234012aad0f3caf66078ba4"
},
"downloads": -1,
"filename": "pipen_gcs-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "a61469c49bfeaa842ab28ac3b98d1e91",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 9363,
"upload_time": "2024-12-20T03:40:04",
"upload_time_iso_8601": "2024-12-20T03:40:04.452277Z",
"url": "https://files.pythonhosted.org/packages/2b/96/c91877960e6dc89c9bd7833f75dbaa7ea93c6849c9ef2ce4110cf0e459a0/pipen_gcs-0.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-20 03:40:04",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "pipen-gcs"
}
