Name | snapstream JSON |
Version |
0.0.9
JSON |
| download |
home_page | https://github.com/Menziess/snapstream |
Summary | Streamline your Kafka data processing, this tool aims to standardize streaming data from multiple Kafka clusters. With a pub-sub approach, multiple functions can easily subscribe to incoming messages, serialization can be specified per topic, and data is automatically processed by data sink functions. |
upload_time | 2023-07-11 21:07:26 |
maintainer | |
docs_url | None |
author | Menziess |
requires_python | >=3.8.1,<4.0.0 |
license | MIT |
keywords |
kafka
pubsub
|
VCS |
|
bugtrack_url |
|
requirements |
No requirements were recorded.
|
Travis-CI |
No Travis.
|
coveralls test coverage |
No coveralls.
|
[![Test Python Package](https://github.com/Menziess/snapstream/actions/workflows/python-test.yml/badge.svg)](https://github.com/Menziess/snapstream/actions/workflows/python-test.yml) [![Documentation Status](https://readthedocs.org/projects/snapstream/badge/?version=latest)](https://snapstream.readthedocs.io/en/latest/?badge=latest) [![Downloads](https://static.pepy.tech/personalized-badge/snapstream?period=month&units=international_system&left_color=grey&right_color=brightgreen&left_text=downloads/month)](https://pepy.tech/project/snapstream)
# Snapstream
<img src="https://raw.githubusercontent.com/menziess/snapstream/master/res/logo.png" width="25%" height="25%" align="right" />
Snapstream provides a data-flow model to simplify development of stateful streaming applications.
## Installation
```sh
pip install snapstream
```
## Usage
We `snap` iterables to user functions, and process them in parallel when we call `stream`:
![demo](https://raw.githubusercontent.com/menziess/snapstream/master/res/demo.gif)
We pass the callable `print` to print out the return value. Multiple iterables and sinks can be passed.
```py
from snapstream import snap, stream
@snap(range(5), sink=[print])
def handler(msg):
yield f'Hello {msg}'
stream()
```
```sh
Hello 0
Hello 1
Hello 2
Hello 3
Hello 4
```
To try it out for yourself, spin up a local kafka broker with [docker-compose.yml](docker-compose.yml), using `localhost:29091` to connect:
```sh
docker compose up broker -d
```
Use the cli tool to inspect Topic/Cache:
```sh
snapstream topic emoji --offset -2
```
```
>>> timestamp: 2023-04-28T17:31:51.775000+00:00
>>> offset: 0
>>> key:
🏆
```
## Features
- [`snapstream.snap`](snapstream/__init__.py): bind streams (iterables) and sinks (callables) to user defined handler functions
- [`snapstream.stream`](snapstream/__init__.py): start streaming
- [`snapstream.Topic`](snapstream/core.py): consume from (iterable), and produce to (callable) kafka using [**confluent-kafka**](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html)
- [`snapstream.Cache`](snapstream/caching.py): store data to disk using [**rocksdict**](https://congyuwang.github.io/RocksDict/rocksdict.html)
- [`snapstream.Conf`](snapstream/core.py): set global kafka configuration (can be overridden per topic)
- [`snapstream.codecs.AvroCodec`](snapstream/codecs.py): serialize and deserialize avro messages
- [`snapstream.codecs.JsonCodec`](snapstream/codecs.py): serialize and deserialize json messages
Raw data
{
"_id": null,
"home_page": "https://github.com/Menziess/snapstream",
"name": "snapstream",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8.1,<4.0.0",
"maintainer_email": "",
"keywords": "kafka,pubsub",
"author": "Menziess",
"author_email": "stefan_schenk@hotmail.com",
"download_url": "https://files.pythonhosted.org/packages/5f/e9/96e2231ba17ddcb74ac57be4ba26c4d1a556e32741c225f9b7d96569ebf0/snapstream-0.0.9.tar.gz",
"platform": null,
"description": "[![Test Python Package](https://github.com/Menziess/snapstream/actions/workflows/python-test.yml/badge.svg)](https://github.com/Menziess/snapstream/actions/workflows/python-test.yml) [![Documentation Status](https://readthedocs.org/projects/snapstream/badge/?version=latest)](https://snapstream.readthedocs.io/en/latest/?badge=latest) [![Downloads](https://static.pepy.tech/personalized-badge/snapstream?period=month&units=international_system&left_color=grey&right_color=brightgreen&left_text=downloads/month)](https://pepy.tech/project/snapstream)\n\n# Snapstream\n\n<img src=\"https://raw.githubusercontent.com/menziess/snapstream/master/res/logo.png\" width=\"25%\" height=\"25%\" align=\"right\" />\n\nSnapstream provides a data-flow model to simplify development of stateful streaming applications.\n\n## Installation\n\n```sh\npip install snapstream\n```\n\n## Usage\n\nWe `snap` iterables to user functions, and process them in parallel when we call `stream`:\n\n![demo](https://raw.githubusercontent.com/menziess/snapstream/master/res/demo.gif)\n\nWe pass the callable `print` to print out the return value. Multiple iterables and sinks can be passed.\n\n```py\nfrom snapstream import snap, stream\n\n@snap(range(5), sink=[print])\ndef handler(msg):\n yield f'Hello {msg}'\n\nstream()\n```\n\n```sh\nHello 0\nHello 1\nHello 2\nHello 3\nHello 4\n```\n\nTo try it out for yourself, spin up a local kafka broker with [docker-compose.yml](docker-compose.yml), using `localhost:29091` to connect:\n\n```sh\ndocker compose up broker -d\n```\n\nUse the cli tool to inspect Topic/Cache:\n\n```sh\nsnapstream topic emoji --offset -2\n```\n\n```\n>>> timestamp: 2023-04-28T17:31:51.775000+00:00\n>>> offset: 0\n>>> key:\n\ud83c\udfc6\n```\n\n## Features\n\n- [`snapstream.snap`](snapstream/__init__.py): bind streams (iterables) and sinks (callables) to user defined handler functions\n- [`snapstream.stream`](snapstream/__init__.py): start streaming\n- [`snapstream.Topic`](snapstream/core.py): consume from (iterable), and produce to (callable) kafka using [**confluent-kafka**](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html)\n- [`snapstream.Cache`](snapstream/caching.py): store data to disk using [**rocksdict**](https://congyuwang.github.io/RocksDict/rocksdict.html)\n- [`snapstream.Conf`](snapstream/core.py): set global kafka configuration (can be overridden per topic)\n- [`snapstream.codecs.AvroCodec`](snapstream/codecs.py): serialize and deserialize avro messages\n- [`snapstream.codecs.JsonCodec`](snapstream/codecs.py): serialize and deserialize json messages\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Streamline your Kafka data processing, this tool aims to standardize streaming data from multiple Kafka clusters. With a pub-sub approach, multiple functions can easily subscribe to incoming messages, serialization can be specified per topic, and data is automatically processed by data sink functions.",
"version": "0.0.9",
"project_urls": {
"Documentation": "https://snapstream.readthedocs.io",
"Homepage": "https://github.com/Menziess/snapstream",
"Repository": "https://github.com/Menziess/snapstream"
},
"split_keywords": [
"kafka",
"pubsub"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4dfff360d5cc820da30b83fe918c70bb26daf9f909cbe040e81a38039c0ab3c1",
"md5": "4460926429dc6756d72a7c1f367d13bd",
"sha256": "4d43db40324bb7e9287513fad20335cdc0babcd07f3eba9329dd54fedf2d325f"
},
"downloads": -1,
"filename": "snapstream-0.0.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4460926429dc6756d72a7c1f367d13bd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8.1,<4.0.0",
"size": 15791,
"upload_time": "2023-07-11T21:07:24",
"upload_time_iso_8601": "2023-07-11T21:07:24.580113Z",
"url": "https://files.pythonhosted.org/packages/4d/ff/f360d5cc820da30b83fe918c70bb26daf9f909cbe040e81a38039c0ab3c1/snapstream-0.0.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5fe996e2231ba17ddcb74ac57be4ba26c4d1a556e32741c225f9b7d96569ebf0",
"md5": "2e528cac2ff552ae0199cbabc2a89b3d",
"sha256": "90de97adaa0be78cb12588f5667a109828abc875fa8eabef62ebd5296eb45fbf"
},
"downloads": -1,
"filename": "snapstream-0.0.9.tar.gz",
"has_sig": false,
"md5_digest": "2e528cac2ff552ae0199cbabc2a89b3d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8.1,<4.0.0",
"size": 14701,
"upload_time": "2023-07-11T21:07:26",
"upload_time_iso_8601": "2023-07-11T21:07:26.108921Z",
"url": "https://files.pythonhosted.org/packages/5f/e9/96e2231ba17ddcb74ac57be4ba26c4d1a556e32741c225f9b7d96569ebf0/snapstream-0.0.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-11 21:07:26",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Menziess",
"github_project": "snapstream",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "snapstream"
}