gs-chunked-io

Name	gs-chunked-io JSON
Version	0.5.2 JSON
	download
home_page	https://github.com/xbrianh/gs-chunked-io.git
Summary	Streaming read/writes to Google Storage blobs with ascynchronous buffering.
upload_time	2020-10-21 18:24:29
maintainer
docs_url	None
author	Brian Hannafious
requires_python
license	MIT
keywords
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            # gs-chunked-io: Streams for Google Storage
_gs-chunked-io_ provides transparently chunked io streams for google storage objects.
Writable streams are managed as multipart objects, composed when the stream is closed.

IO opperations are concurrent by default. The number of concurrent threads can be adjusted using the `threads`
parameter, or disabled entirely with `threads=None`.
```
import gs_chunked_io as gscio
from google.cloud.storage import Client

client = Client()
bucket = client.bucket("my-bucket")
blob = bucket.get_blob("my-key")

# Readable stream:
with gscio.Reader(blob) as fh:
    fh.read(size)

# Writable stream:
with gscio.Writer("my_new_key", bucket) as fh:
    fh.write(data)

# Process blob in chunks:
for chunk in gscio.for_each_chunk(blob):
    my_chunk_processor(chunk)

# Multipart copy with processing:
dst_bucket = client.bucket("my_dest_bucket")
with gscio.Writer("my_dest_key", dst_bucket) as writer:
    for chunk in gscio.for_each_chunk(blob)
	    process_my_chunk(chunk)
	    writer(chunk)

# Extract .tar.gz on the fly:
import gzip
import tarfile
with gscio.Reader(blob) as fh:
    gzip_reader = gzip.GzipFile(fileobj=fh)
    tf = tarfile.TarFile(fileobj=gzip_reader)
    for tarinfo in tf:
        process_my_tarinfo(tarinfo)
```

## Installation
```
pip install gs-chunked-io
```

## Links
Project home page [GitHub](https://github.com/xbrianh/gs-chunked-io)  
Package distribution [PyPI](https://pypi.org/project/gs-chunked-io/)

### Bugs
Please report bugs, issues, feature requests, etc. on [GitHub](https://github.com/xbrianh/gs-chunked-io).

![](https://travis-ci.org/xbrianh/gs-chunked-io.svg?branch=master) ![](https://badge.fury.io/py/gs-chunked-io.svg)
            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/xbrianh/gs-chunked-io.git",
    "name": "gs-chunked-io",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "",
    "author": "Brian Hannafious",
    "author_email": "bhannafi@ucsc.edu",
    "download_url": "https://files.pythonhosted.org/packages/aa/76/5f86e2c4cc5d09c74e39f633a01bb75c3ad78d59e00b90105bb51e611288/gs-chunked-io-0.5.2.tar.gz",
    "platform": "MacOS X",
    "description": "# gs-chunked-io: Streams for Google Storage\n_gs-chunked-io_ provides transparently chunked io streams for google storage objects.\nWritable streams are managed as multipart objects, composed when the stream is closed.\n\nIO opperations are concurrent by default. The number of concurrent threads can be adjusted using the `threads`\nparameter, or disabled entirely with `threads=None`.\n```\nimport gs_chunked_io as gscio\nfrom google.cloud.storage import Client\n\nclient = Client()\nbucket = client.bucket(\"my-bucket\")\nblob = bucket.get_blob(\"my-key\")\n\n# Readable stream:\nwith gscio.Reader(blob) as fh:\n    fh.read(size)\n\n# Writable stream:\nwith gscio.Writer(\"my_new_key\", bucket) as fh:\n    fh.write(data)\n\n# Process blob in chunks:\nfor chunk in gscio.for_each_chunk(blob):\n    my_chunk_processor(chunk)\n\n# Multipart copy with processing:\ndst_bucket = client.bucket(\"my_dest_bucket\")\nwith gscio.Writer(\"my_dest_key\", dst_bucket) as writer:\n    for chunk in gscio.for_each_chunk(blob)\n\t    process_my_chunk(chunk)\n\t    writer(chunk)\n\n# Extract .tar.gz on the fly:\nimport gzip\nimport tarfile\nwith gscio.Reader(blob) as fh:\n    gzip_reader = gzip.GzipFile(fileobj=fh)\n    tf = tarfile.TarFile(fileobj=gzip_reader)\n    for tarinfo in tf:\n        process_my_tarinfo(tarinfo)\n```\n\n## Installation\n```\npip install gs-chunked-io\n```\n\n## Links\nProject home page [GitHub](https://github.com/xbrianh/gs-chunked-io)  \nPackage distribution [PyPI](https://pypi.org/project/gs-chunked-io/)\n\n### Bugs\nPlease report bugs, issues, feature requests, etc. on [GitHub](https://github.com/xbrianh/gs-chunked-io).\n\n![](https://travis-ci.org/xbrianh/gs-chunked-io.svg?branch=master) ![](https://badge.fury.io/py/gs-chunked-io.svg)",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Streaming read/writes to Google Storage blobs with ascynchronous buffering.",
    "version": "0.5.2",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "aa765f86e2c4cc5d09c74e39f633a01bb75c3ad78d59e00b90105bb51e611288",
                "md5": "910d44662bf4cbb5aae2c1329c92e1b2",
                "sha256": "722e5ff718d9c91f4ac5e4d285ace599642e7c7c31188c025f3d07216c8e61e6"
            },
            "downloads": -1,
            "filename": "gs-chunked-io-0.5.2.tar.gz",
            "has_sig": false,
            "md5_digest": "910d44662bf4cbb5aae2c1329c92e1b2",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 8063,
            "upload_time": "2020-10-21T18:24:29",
            "upload_time_iso_8601": "2020-10-21T18:24:29.654777Z",
            "url": "https://files.pythonhosted.org/packages/aa/76/5f86e2c4cc5d09c74e39f633a01bb75c3ad78d59e00b90105bb51e611288/gs-chunked-io-0.5.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2020-10-21 18:24:29",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "xbrianh",
    "github_project": "gs-chunked-io.git",
    "lcname": "gs-chunked-io"
}