django-kafka-streamer


Namedjango-kafka-streamer JSON
Version 1.4.1 PyPI version JSON
download
home_pageNone
SummaryStream data to Apache Kafka.
upload_time2024-11-02 19:42:04
maintainerNone
docs_urlNone
authorNone
requires_python>=3.10
licenseNone
keywords django kafka microservices
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            django-kafka-streamer
=====================

.. image:: https://github.com/lostclus/django-kafka-streamer/actions/workflows/tests.yml/badge.svg
    :target: https://github.com/lostclus/django-kafka-streamer/actions

.. image:: https://readthedocs.org/projects/django-kafka-streamer/badge/?version=latest
    :target: http://django-kafka-streamer.readthedocs.io/
    :alt: Documentation

.. image:: https://img.shields.io/pypi/v/django-kafka-streamer.svg
    :target: https://pypi.org/project/django-kafka-streamer/
    :alt: Current version on PyPi

.. image:: https://img.shields.io/pypi/pyversions/django-kafka-streamer
    :alt: PyPI - Python Version

.. image:: https://img.shields.io/pypi/djversions/django-kafka-streamer
    :alt: PyPI - Django Version

.. image:: https://img.shields.io/badge/Published%20on-Django%20Packages-0c3c26
    :target: https://djangopackages.org/packages/p/django-kafka-streamer/
    :alt: Published on Django Packages

django-kafka-streamer is a Django application and library for streaming data to
Apache Kafka.

Features:

* Setup signal handlers to ORM models to transparently send create/update/delete
  events to Kafka
* Handle database object relations
* Celery task to stream large amount of data in background

Links:

* GitHub: https://github.com/lostclus/django-kafka-streamer/
* PyPI: https://pypi.org/project/django-kafka-streamer/
* Documentation: http://django-kafka-streamer.readthedocs.io/
* Consumer library: https://github.com/lostclus/aiosafeconsumer
* Example application: https://github.com/lostclus/WeatherApp

Usage:

`yourapp/models.py`::

    from django.db import models

    class MyModel(models.Model):
        field1 = models.IntegerField()
        field2 = models.CharField(max_length=10)

`yourapp/stramers.py`::

    from kafkastreamer import Streamer, register
    from .models import MyModel

    @register(MyModel)
    class MyModelStreamer(Streamer):
        topic = "model-a"

`yourproject/settings.py`::

    INSTALLED_APPS = [
        ...
        "kafkastreamer",
    ]

    KAFKA_STREAMER = {
        "BOOTSTRAP_SERVERS": ["localhost:9092"],
    },

Any changes in ``MyModel`` data will be automatically streamed to Kafka. To
force stream all data in all registered models type::

    python manage.py kafkastreamer_refresh

The data streamed to the ``model-a`` Kafka topic has following structure::

    {
        "_time": "2023-01-01T00:00:00Z",
        "_type": "create",
        "id": 1,
        "field1": 1,
        "field2": "abc"
    }

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "django-kafka-streamer",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.10",
    "maintainer_email": null,
    "keywords": "Django, Kafka, microservices",
    "author": null,
    "author_email": "Kostiantyn Korikov <lostclus@gmail.com>",
    "download_url": "https://files.pythonhosted.org/packages/6c/2b/4e2d275d5bb570e44c8b5af07bcdfea38824afc83d507e1c0955445d9421/django_kafka_streamer-1.4.1.tar.gz",
    "platform": null,
    "description": "django-kafka-streamer\n=====================\n\n.. image:: https://github.com/lostclus/django-kafka-streamer/actions/workflows/tests.yml/badge.svg\n    :target: https://github.com/lostclus/django-kafka-streamer/actions\n\n.. image:: https://readthedocs.org/projects/django-kafka-streamer/badge/?version=latest\n    :target: http://django-kafka-streamer.readthedocs.io/\n    :alt: Documentation\n\n.. image:: https://img.shields.io/pypi/v/django-kafka-streamer.svg\n    :target: https://pypi.org/project/django-kafka-streamer/\n    :alt: Current version on PyPi\n\n.. image:: https://img.shields.io/pypi/pyversions/django-kafka-streamer\n    :alt: PyPI - Python Version\n\n.. image:: https://img.shields.io/pypi/djversions/django-kafka-streamer\n    :alt: PyPI - Django Version\n\n.. image:: https://img.shields.io/badge/Published%20on-Django%20Packages-0c3c26\n    :target: https://djangopackages.org/packages/p/django-kafka-streamer/\n    :alt: Published on Django Packages\n\ndjango-kafka-streamer is a Django application and library for streaming data to\nApache Kafka.\n\nFeatures:\n\n* Setup signal handlers to ORM models to transparently send create/update/delete\n  events to Kafka\n* Handle database object relations\n* Celery task to stream large amount of data in background\n\nLinks:\n\n* GitHub: https://github.com/lostclus/django-kafka-streamer/\n* PyPI: https://pypi.org/project/django-kafka-streamer/\n* Documentation: http://django-kafka-streamer.readthedocs.io/\n* Consumer library: https://github.com/lostclus/aiosafeconsumer\n* Example application: https://github.com/lostclus/WeatherApp\n\nUsage:\n\n`yourapp/models.py`::\n\n    from django.db import models\n\n    class MyModel(models.Model):\n        field1 = models.IntegerField()\n        field2 = models.CharField(max_length=10)\n\n`yourapp/stramers.py`::\n\n    from kafkastreamer import Streamer, register\n    from .models import MyModel\n\n    @register(MyModel)\n    class MyModelStreamer(Streamer):\n        topic = \"model-a\"\n\n`yourproject/settings.py`::\n\n    INSTALLED_APPS = [\n        ...\n        \"kafkastreamer\",\n    ]\n\n    KAFKA_STREAMER = {\n        \"BOOTSTRAP_SERVERS\": [\"localhost:9092\"],\n    },\n\nAny changes in ``MyModel`` data will be automatically streamed to Kafka. To\nforce stream all data in all registered models type::\n\n    python manage.py kafkastreamer_refresh\n\nThe data streamed to the ``model-a`` Kafka topic has following structure::\n\n    {\n        \"_time\": \"2023-01-01T00:00:00Z\",\n        \"_type\": \"create\",\n        \"id\": 1,\n        \"field1\": 1,\n        \"field2\": \"abc\"\n    }\n",
    "bugtrack_url": null,
    "license": null,
    "summary": "Stream data to Apache Kafka.",
    "version": "1.4.1",
    "project_urls": {
        "Documentation": "http://django-kafka-streamer.readthedocs.io/",
        "Repository": "http://github.com/lostclus/django-kafka-streamer"
    },
    "split_keywords": [
        "django",
        " kafka",
        " microservices"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "c8f12e1305181f75ede74762d058dd79d3d5506fedabc61f5e872fadff5d0b60",
                "md5": "6005fa37325f2b95586bec93189f05ba",
                "sha256": "23e672a79d39af08834c57a3434f687d30ae02d8285805953fd39414619e82b2"
            },
            "downloads": -1,
            "filename": "django_kafka_streamer-1.4.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "6005fa37325f2b95586bec93189f05ba",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.10",
            "size": 20314,
            "upload_time": "2024-11-02T19:42:03",
            "upload_time_iso_8601": "2024-11-02T19:42:03.256686Z",
            "url": "https://files.pythonhosted.org/packages/c8/f1/2e1305181f75ede74762d058dd79d3d5506fedabc61f5e872fadff5d0b60/django_kafka_streamer-1.4.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "6c2b4e2d275d5bb570e44c8b5af07bcdfea38824afc83d507e1c0955445d9421",
                "md5": "3e1bd0c2eeaf66201e76eeecd021e38d",
                "sha256": "20c043e6f9d865ea32ef9ef9182e07e7be167e8cc28121fb8ca311cfb5e50167"
            },
            "downloads": -1,
            "filename": "django_kafka_streamer-1.4.1.tar.gz",
            "has_sig": false,
            "md5_digest": "3e1bd0c2eeaf66201e76eeecd021e38d",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.10",
            "size": 30299,
            "upload_time": "2024-11-02T19:42:04",
            "upload_time_iso_8601": "2024-11-02T19:42:04.886778Z",
            "url": "https://files.pythonhosted.org/packages/6c/2b/4e2d275d5bb570e44c8b5af07bcdfea38824afc83d507e1c0955445d9421/django_kafka_streamer-1.4.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-11-02 19:42:04",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "lostclus",
    "github_project": "django-kafka-streamer",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "tox": true,
    "lcname": "django-kafka-streamer"
}
        
Elapsed time: 0.35834s