[](https://travis-ci.org/cartoonist/pystream-protobuf)
[](https://pypi.python.org/pypi/pystream-protobuf)
[](https://pypi.python.org/pypi/pystream-protobuf)
[](https://www.python.org/download/releases/3.0/)
[](https://github.com/cartoonist/pystream-protobuf/blob/master/LICENSE)
# pyStream
Python implementation of [stream library](https://github.com/vgteam/stream).
## Introduction
This library enables _stream processing_ of protobuf messages (or any serializable
objects since v1.6.3); i.e. multiple protobuf messages can be written/read into/from a
single stream or file.
It was originally developed to parse/write [vg](https://github.com/vgteam/vg)
file formats (`.vg`, `.gam`, etc). However, it can be used for any arbitrary
protocol buffer messages.
Refer to the C++ [stream library](https://github.com/vgteam/stream) for more
details.
---
**NOTE**
**@vg users:** The new version of stream library, now as a part of
[libvgio](https://github.com/vgteam/libvgio), writes a header tag at the start of
the stream depending on the output format. For example, headers like `b'GAM'`
or `b'VG'` can be found before the actual protobuf messages in GAM and VG files
repectively. In this case, you should provide the expected value using `header`
keyword argument; e.g.
`stream.parse('file.gam', vg_pb2.Alignment, header=b'GAM', persistent_header=True)`
for GAM files (since version v1.6.2).
---
## Encoding
The encoding is simple. Messages are written in groups of different sizes. Each
group starts with its size; i.e. the number of messages in that group. Then, the
size of each message is followed by the encoded message itself. Quoted from
[Google Protobuf Developer Guide](https://developers.google.com/protocol-buffers/docs/techniques#streaming):
> The Protocol Buffer wire format is not self-delimiting, so protocol buffer
> parsers cannot determine where a message ends on their own. The easiest way to
> solve this problem is to write the size of each message before you write the
> message itself. When you read the messages back in, you read the size, then
> read the bytes into a separate buffer, then parse from that buffer.
By default, the stream is considered compressed by GZip. However, uncompressed
stream processing is possible by passing `gzip=False` to any API calls.
## Installation
You can install pyStream using `pip`:
pip install pystream-protobuf
## Usage
See [Wiki](https://github.com/cartoonist/pystream-protobuf/wiki) for usage documentation.
## Development
In case, you work with the source code and need to build the package:
python setup.py build
The proto files in the test module required to be compiled before running test
cases. To do so, it is required to have Google protobuf compiler (>=3.0.2)
installed. After installing protobuf compiler, run:
make init
to compile proto files required for test module and then:
make test
to run tests.
Raw data
{
"_id": null,
"home_page": "https://github.com/cartoonist/pystream-protobuf",
"name": "pystream-protobuf",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "stream protocol buffer protobuf",
"author": "Ali Ghaffaari",
"author_email": "ali.ghaffaari@mpi-inf.mpg.de",
"download_url": "https://files.pythonhosted.org/packages/89/4a/ab5bd8cca64786d2167743a68c60affd7148789fb497d984bc428bcae1be/pystream-protobuf-1.6.4.tar.gz",
"platform": null,
"description": "[](https://travis-ci.org/cartoonist/pystream-protobuf)\n[](https://pypi.python.org/pypi/pystream-protobuf)\n[](https://pypi.python.org/pypi/pystream-protobuf)\n[](https://www.python.org/download/releases/3.0/)\n[](https://github.com/cartoonist/pystream-protobuf/blob/master/LICENSE)\n\n# pyStream\nPython implementation of [stream library](https://github.com/vgteam/stream).\n\n## Introduction\nThis library enables _stream processing_ of protobuf messages (or any serializable\nobjects since v1.6.3); i.e. multiple protobuf messages can be written/read into/from a\nsingle stream or file.\n\nIt was originally developed to parse/write [vg](https://github.com/vgteam/vg)\nfile formats (`.vg`, `.gam`, etc). However, it can be used for any arbitrary\nprotocol buffer messages.\n\nRefer to the C++ [stream library](https://github.com/vgteam/stream) for more\ndetails.\n\n---\n\n**NOTE**\n\n**@vg users:** The new version of stream library, now as a part of\n[libvgio](https://github.com/vgteam/libvgio), writes a header tag at the start of\nthe stream depending on the output format. For example, headers like `b'GAM'`\nor `b'VG'` can be found before the actual protobuf messages in GAM and VG files\nrepectively. In this case, you should provide the expected value using `header`\nkeyword argument; e.g.\n`stream.parse('file.gam', vg_pb2.Alignment, header=b'GAM', persistent_header=True)`\nfor GAM files (since version v1.6.2).\n\n---\n\n## Encoding\nThe encoding is simple. Messages are written in groups of different sizes. Each\ngroup starts with its size; i.e. the number of messages in that group. Then, the\nsize of each message is followed by the encoded message itself. Quoted from\n[Google Protobuf Developer Guide](https://developers.google.com/protocol-buffers/docs/techniques#streaming):\n\n> The Protocol Buffer wire format is not self-delimiting, so protocol buffer\n> parsers cannot determine where a message ends on their own. The easiest way to\n> solve this problem is to write the size of each message before you write the\n> message itself. When you read the messages back in, you read the size, then\n> read the bytes into a separate buffer, then parse from that buffer.\n\nBy default, the stream is considered compressed by GZip. However, uncompressed\nstream processing is possible by passing `gzip=False` to any API calls.\n\n## Installation\nYou can install pyStream using `pip`:\n\n pip install pystream-protobuf\n\n## Usage\nSee [Wiki](https://github.com/cartoonist/pystream-protobuf/wiki) for usage documentation.\n\n## Development\nIn case, you work with the source code and need to build the package:\n\n python setup.py build\n\nThe proto files in the test module required to be compiled before running test\ncases. To do so, it is required to have Google protobuf compiler (>=3.0.2)\ninstalled. After installing protobuf compiler, run:\n\n make init\n\nto compile proto files required for test module and then:\n\n make test\n\nto run tests.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Python implementation of stream library",
"version": "1.6.4",
"split_keywords": [
"stream",
"protocol",
"buffer",
"protobuf"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "ba165de204e3dc28f8841283a37b197e5a535d3f62906106ca52d6d03b6fea2e",
"md5": "d04c9ae88dcbc23883a616fb0e7b95e5",
"sha256": "7129949a278276a7fc54497ca3fb0333ce905777b871eeccefcc0936bb30f631"
},
"downloads": -1,
"filename": "pystream_protobuf-1.6.4-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "d04c9ae88dcbc23883a616fb0e7b95e5",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 9669,
"upload_time": "2023-04-20T17:35:25",
"upload_time_iso_8601": "2023-04-20T17:35:25.492517Z",
"url": "https://files.pythonhosted.org/packages/ba/16/5de204e3dc28f8841283a37b197e5a535d3f62906106ca52d6d03b6fea2e/pystream_protobuf-1.6.4-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "894aab5bd8cca64786d2167743a68c60affd7148789fb497d984bc428bcae1be",
"md5": "869e0794d400eb120d92164c7b772491",
"sha256": "1fb2c71c54bb84b56f40fc07d5a6539bbb1fb226b294c62d46e65127952c2a7c"
},
"downloads": -1,
"filename": "pystream-protobuf-1.6.4.tar.gz",
"has_sig": false,
"md5_digest": "869e0794d400eb120d92164c7b772491",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 8614,
"upload_time": "2023-04-20T17:35:27",
"upload_time_iso_8601": "2023-04-20T17:35:27.237558Z",
"url": "https://files.pythonhosted.org/packages/89/4a/ab5bd8cca64786d2167743a68c60affd7148789fb497d984bc428bcae1be/pystream-protobuf-1.6.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-20 17:35:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "cartoonist",
"github_project": "pystream-protobuf",
"travis_ci": true,
"coveralls": false,
"github_actions": false,
"requirements": [],
"tox": true,
"lcname": "pystream-protobuf"
}
JSON
