| Name | snapstream JSON |
| Version |
1.0.0
JSON |
| download |
| home_page | https://github.com/Menziess/snapstream |
| Summary | Streamline your Kafka data processing, this tool aims to standardize streaming data from multiple Kafka clusters. With a pub-sub approach, multiple functions can easily subscribe to incoming messages, serialization can be specified per topic, and data is automatically processed by data sink functions. |
| upload_time | 2024-04-01 19:16:15 |
| maintainer | None |
| docs_url | None |
| author | Menziess |
| requires_python | <4.0.0,>=3.8.1 |
| license | MIT |
| keywords |
kafka
pubsub
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
[](https://github.com/Menziess/snapstream/actions/workflows/python-test.yml) [](https://snapstream.readthedocs.io/en/latest/?badge=latest) [](https://pepy.tech/project/snapstream)
# Snapstream
<img src="https://raw.githubusercontent.com/menziess/snapstream/master/res/logo.png" width="25%" height="25%" align="right" />
Snapstream provides a data-flow model to simplify development of stateful streaming applications.
## Installation
```sh
pip install snapstream
```
## Usage
We `snap` iterables to user functions, and process them in parallel when we call `stream`:
We pass the callable `print` to print out the return value. Multiple iterables and sinks can be passed.
```py
from snapstream import snap, stream
@snap(range(5), sink=[print])
def handler(msg):
yield f'Hello {msg}'
stream()
```
```sh
Hello 0
Hello 1
Hello 2
Hello 3
Hello 4
```
To try it out for yourself, spin up a local kafka broker with [docker-compose.yml](docker-compose.yml), using `localhost:29091` to connect:
```sh
docker compose up broker -d
```
Use the cli tool to inspect Topic/Cache:
```sh
snapstream topic emoji --offset -2
```
```
>>> timestamp: 2023-04-28T17:31:51.775000+00:00
>>> offset: 0
>>> key:
🏆
```
## Features
- [`snapstream.snap`](snapstream/__init__.py): bind streams (iterables) and sinks (callables) to user defined handler functions
- [`snapstream.stream`](snapstream/__init__.py): start streaming
- [`snapstream.Topic`](snapstream/core.py): consume from (iterable), and produce to (callable) kafka using [**confluent-kafka**](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html)
- [`snapstream.Cache`](snapstream/caching.py): store data to disk using [**rocksdict**](https://congyuwang.github.io/RocksDict/rocksdict.html)
- [`snapstream.Conf`](snapstream/core.py): set global kafka configuration (can be overridden per topic)
- [`snapstream.codecs.AvroCodec`](snapstream/codecs.py): serialize and deserialize avro messages
- [`snapstream.codecs.JsonCodec`](snapstream/codecs.py): serialize and deserialize json messages
Raw data
{
"_id": null,
"home_page": "https://github.com/Menziess/snapstream",
"name": "snapstream",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0.0,>=3.8.1",
"maintainer_email": null,
"keywords": "kafka, pubsub",
"author": "Menziess",
"author_email": "stefan_schenk@hotmail.com",
"download_url": "https://files.pythonhosted.org/packages/c9/67/acd181918a0bc94ca852e29d2d09966e5424f5bbb2b183147cd413947ea0/snapstream-1.0.0.tar.gz",
"platform": null,
"description": "[](https://github.com/Menziess/snapstream/actions/workflows/python-test.yml) [](https://snapstream.readthedocs.io/en/latest/?badge=latest) [](https://pepy.tech/project/snapstream)\n\n# Snapstream\n\n<img src=\"https://raw.githubusercontent.com/menziess/snapstream/master/res/logo.png\" width=\"25%\" height=\"25%\" align=\"right\" />\n\nSnapstream provides a data-flow model to simplify development of stateful streaming applications.\n\n## Installation\n\n```sh\npip install snapstream\n```\n\n## Usage\n\nWe `snap` iterables to user functions, and process them in parallel when we call `stream`:\n\n\n\nWe pass the callable `print` to print out the return value. Multiple iterables and sinks can be passed.\n\n```py\nfrom snapstream import snap, stream\n\n@snap(range(5), sink=[print])\ndef handler(msg):\n yield f'Hello {msg}'\n\nstream()\n```\n\n```sh\nHello 0\nHello 1\nHello 2\nHello 3\nHello 4\n```\n\nTo try it out for yourself, spin up a local kafka broker with [docker-compose.yml](docker-compose.yml), using `localhost:29091` to connect:\n\n```sh\ndocker compose up broker -d\n```\n\nUse the cli tool to inspect Topic/Cache:\n\n```sh\nsnapstream topic emoji --offset -2\n```\n\n```\n>>> timestamp: 2023-04-28T17:31:51.775000+00:00\n>>> offset: 0\n>>> key:\n\ud83c\udfc6\n```\n\n## Features\n\n- [`snapstream.snap`](snapstream/__init__.py): bind streams (iterables) and sinks (callables) to user defined handler functions\n- [`snapstream.stream`](snapstream/__init__.py): start streaming\n- [`snapstream.Topic`](snapstream/core.py): consume from (iterable), and produce to (callable) kafka using [**confluent-kafka**](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html)\n- [`snapstream.Cache`](snapstream/caching.py): store data to disk using [**rocksdict**](https://congyuwang.github.io/RocksDict/rocksdict.html)\n- [`snapstream.Conf`](snapstream/core.py): set global kafka configuration (can be overridden per topic)\n- [`snapstream.codecs.AvroCodec`](snapstream/codecs.py): serialize and deserialize avro messages\n- [`snapstream.codecs.JsonCodec`](snapstream/codecs.py): serialize and deserialize json messages\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Streamline your Kafka data processing, this tool aims to standardize streaming data from multiple Kafka clusters. With a pub-sub approach, multiple functions can easily subscribe to incoming messages, serialization can be specified per topic, and data is automatically processed by data sink functions.",
"version": "1.0.0",
"project_urls": {
"Documentation": "https://snapstream.readthedocs.io",
"Homepage": "https://github.com/Menziess/snapstream",
"Repository": "https://github.com/Menziess/snapstream"
},
"split_keywords": [
"kafka",
" pubsub"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "740f9bd2576285f13a92facdf08a8f03a9b6b2ff4fcf0abd40d319a14f3ccc45",
"md5": "f0169ee279c281b6028d1dbb25d05fdb",
"sha256": "f5323d0c7f5c94a459f7dfed460514b5e823784b65155f003b289b7c27603358"
},
"downloads": -1,
"filename": "snapstream-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f0169ee279c281b6028d1dbb25d05fdb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0.0,>=3.8.1",
"size": 16126,
"upload_time": "2024-04-01T19:16:13",
"upload_time_iso_8601": "2024-04-01T19:16:13.841719Z",
"url": "https://files.pythonhosted.org/packages/74/0f/9bd2576285f13a92facdf08a8f03a9b6b2ff4fcf0abd40d319a14f3ccc45/snapstream-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c967acd181918a0bc94ca852e29d2d09966e5424f5bbb2b183147cd413947ea0",
"md5": "bf77f134867289779d1ff5b4da03437d",
"sha256": "3d0ba0794a26ce33d402ea3470dc196ce4ced1ed72e3a18b6157b122b9d78dad"
},
"downloads": -1,
"filename": "snapstream-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "bf77f134867289779d1ff5b4da03437d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0.0,>=3.8.1",
"size": 14977,
"upload_time": "2024-04-01T19:16:15",
"upload_time_iso_8601": "2024-04-01T19:16:15.663811Z",
"url": "https://files.pythonhosted.org/packages/c9/67/acd181918a0bc94ca852e29d2d09966e5424f5bbb2b183147cd413947ea0/snapstream-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-01 19:16:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Menziess",
"github_project": "snapstream",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "snapstream"
}
