snapstream


Namesnapstream JSON
Version 0.0.9 PyPI version JSON
download
home_pagehttps://github.com/Menziess/snapstream
SummaryStreamline your Kafka data processing, this tool aims to standardize streaming data from multiple Kafka clusters. With a pub-sub approach, multiple functions can easily subscribe to incoming messages, serialization can be specified per topic, and data is automatically processed by data sink functions.
upload_time2023-07-11 21:07:26
maintainer
docs_urlNone
authorMenziess
requires_python>=3.8.1,<4.0.0
licenseMIT
keywords kafka pubsub
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            [![Test Python Package](https://github.com/Menziess/snapstream/actions/workflows/python-test.yml/badge.svg)](https://github.com/Menziess/snapstream/actions/workflows/python-test.yml) [![Documentation Status](https://readthedocs.org/projects/snapstream/badge/?version=latest)](https://snapstream.readthedocs.io/en/latest/?badge=latest) [![Downloads](https://static.pepy.tech/personalized-badge/snapstream?period=month&units=international_system&left_color=grey&right_color=brightgreen&left_text=downloads/month)](https://pepy.tech/project/snapstream)

# Snapstream

<img src="https://raw.githubusercontent.com/menziess/snapstream/master/res/logo.png" width="25%" height="25%" align="right" />

Snapstream provides a data-flow model to simplify development of stateful streaming applications.

## Installation

```sh
pip install snapstream
```

## Usage

We `snap` iterables to user functions, and process them in parallel when we call `stream`:

![demo](https://raw.githubusercontent.com/menziess/snapstream/master/res/demo.gif)

We pass the callable `print` to print out the return value. Multiple iterables and sinks can be passed.

```py
from snapstream import snap, stream

@snap(range(5), sink=[print])
def handler(msg):
    yield f'Hello {msg}'

stream()
```

```sh
Hello 0
Hello 1
Hello 2
Hello 3
Hello 4
```

To try it out for yourself, spin up a local kafka broker with [docker-compose.yml](docker-compose.yml), using `localhost:29091` to connect:

```sh
docker compose up broker -d
```

Use the cli tool to inspect Topic/Cache:

```sh
snapstream topic emoji --offset -2
```

```
>>> timestamp: 2023-04-28T17:31:51.775000+00:00
>>> offset: 0
>>> key:
🏆
```

## Features

- [`snapstream.snap`](snapstream/__init__.py): bind streams (iterables) and sinks (callables) to user defined handler functions
- [`snapstream.stream`](snapstream/__init__.py): start streaming
- [`snapstream.Topic`](snapstream/core.py): consume from (iterable), and produce to (callable) kafka using [**confluent-kafka**](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html)
- [`snapstream.Cache`](snapstream/caching.py): store data to disk using [**rocksdict**](https://congyuwang.github.io/RocksDict/rocksdict.html)
- [`snapstream.Conf`](snapstream/core.py): set global kafka configuration (can be overridden per topic)
- [`snapstream.codecs.AvroCodec`](snapstream/codecs.py): serialize and deserialize avro messages
- [`snapstream.codecs.JsonCodec`](snapstream/codecs.py): serialize and deserialize json messages

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/Menziess/snapstream",
    "name": "snapstream",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.8.1,<4.0.0",
    "maintainer_email": "",
    "keywords": "kafka,pubsub",
    "author": "Menziess",
    "author_email": "stefan_schenk@hotmail.com",
    "download_url": "https://files.pythonhosted.org/packages/5f/e9/96e2231ba17ddcb74ac57be4ba26c4d1a556e32741c225f9b7d96569ebf0/snapstream-0.0.9.tar.gz",
    "platform": null,
    "description": "[![Test Python Package](https://github.com/Menziess/snapstream/actions/workflows/python-test.yml/badge.svg)](https://github.com/Menziess/snapstream/actions/workflows/python-test.yml) [![Documentation Status](https://readthedocs.org/projects/snapstream/badge/?version=latest)](https://snapstream.readthedocs.io/en/latest/?badge=latest) [![Downloads](https://static.pepy.tech/personalized-badge/snapstream?period=month&units=international_system&left_color=grey&right_color=brightgreen&left_text=downloads/month)](https://pepy.tech/project/snapstream)\n\n# Snapstream\n\n<img src=\"https://raw.githubusercontent.com/menziess/snapstream/master/res/logo.png\" width=\"25%\" height=\"25%\" align=\"right\" />\n\nSnapstream provides a data-flow model to simplify development of stateful streaming applications.\n\n## Installation\n\n```sh\npip install snapstream\n```\n\n## Usage\n\nWe `snap` iterables to user functions, and process them in parallel when we call `stream`:\n\n![demo](https://raw.githubusercontent.com/menziess/snapstream/master/res/demo.gif)\n\nWe pass the callable `print` to print out the return value. Multiple iterables and sinks can be passed.\n\n```py\nfrom snapstream import snap, stream\n\n@snap(range(5), sink=[print])\ndef handler(msg):\n    yield f'Hello {msg}'\n\nstream()\n```\n\n```sh\nHello 0\nHello 1\nHello 2\nHello 3\nHello 4\n```\n\nTo try it out for yourself, spin up a local kafka broker with [docker-compose.yml](docker-compose.yml), using `localhost:29091` to connect:\n\n```sh\ndocker compose up broker -d\n```\n\nUse the cli tool to inspect Topic/Cache:\n\n```sh\nsnapstream topic emoji --offset -2\n```\n\n```\n>>> timestamp: 2023-04-28T17:31:51.775000+00:00\n>>> offset: 0\n>>> key:\n\ud83c\udfc6\n```\n\n## Features\n\n- [`snapstream.snap`](snapstream/__init__.py): bind streams (iterables) and sinks (callables) to user defined handler functions\n- [`snapstream.stream`](snapstream/__init__.py): start streaming\n- [`snapstream.Topic`](snapstream/core.py): consume from (iterable), and produce to (callable) kafka using [**confluent-kafka**](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html)\n- [`snapstream.Cache`](snapstream/caching.py): store data to disk using [**rocksdict**](https://congyuwang.github.io/RocksDict/rocksdict.html)\n- [`snapstream.Conf`](snapstream/core.py): set global kafka configuration (can be overridden per topic)\n- [`snapstream.codecs.AvroCodec`](snapstream/codecs.py): serialize and deserialize avro messages\n- [`snapstream.codecs.JsonCodec`](snapstream/codecs.py): serialize and deserialize json messages\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Streamline your Kafka data processing, this tool aims to standardize streaming data from multiple Kafka clusters. With a pub-sub approach, multiple functions can easily subscribe to incoming messages, serialization can be specified per topic, and data is automatically processed by data sink functions.",
    "version": "0.0.9",
    "project_urls": {
        "Documentation": "https://snapstream.readthedocs.io",
        "Homepage": "https://github.com/Menziess/snapstream",
        "Repository": "https://github.com/Menziess/snapstream"
    },
    "split_keywords": [
        "kafka",
        "pubsub"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "4dfff360d5cc820da30b83fe918c70bb26daf9f909cbe040e81a38039c0ab3c1",
                "md5": "4460926429dc6756d72a7c1f367d13bd",
                "sha256": "4d43db40324bb7e9287513fad20335cdc0babcd07f3eba9329dd54fedf2d325f"
            },
            "downloads": -1,
            "filename": "snapstream-0.0.9-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "4460926429dc6756d72a7c1f367d13bd",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8.1,<4.0.0",
            "size": 15791,
            "upload_time": "2023-07-11T21:07:24",
            "upload_time_iso_8601": "2023-07-11T21:07:24.580113Z",
            "url": "https://files.pythonhosted.org/packages/4d/ff/f360d5cc820da30b83fe918c70bb26daf9f909cbe040e81a38039c0ab3c1/snapstream-0.0.9-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "5fe996e2231ba17ddcb74ac57be4ba26c4d1a556e32741c225f9b7d96569ebf0",
                "md5": "2e528cac2ff552ae0199cbabc2a89b3d",
                "sha256": "90de97adaa0be78cb12588f5667a109828abc875fa8eabef62ebd5296eb45fbf"
            },
            "downloads": -1,
            "filename": "snapstream-0.0.9.tar.gz",
            "has_sig": false,
            "md5_digest": "2e528cac2ff552ae0199cbabc2a89b3d",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.8.1,<4.0.0",
            "size": 14701,
            "upload_time": "2023-07-11T21:07:26",
            "upload_time_iso_8601": "2023-07-11T21:07:26.108921Z",
            "url": "https://files.pythonhosted.org/packages/5f/e9/96e2231ba17ddcb74ac57be4ba26c4d1a556e32741c225f9b7d96569ebf0/snapstream-0.0.9.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-07-11 21:07:26",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "Menziess",
    "github_project": "snapstream",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "snapstream"
}
        
Elapsed time: 0.26809s