django-kafka-streamer
=====================
.. image:: https://github.com/lostclus/django-kafka-streamer/actions/workflows/tests.yml/badge.svg
:target: https://github.com/lostclus/django-kafka-streamer/actions
.. image:: https://readthedocs.org/projects/django-kafka-streamer/badge/?version=latest
:target: http://django-kafka-streamer.readthedocs.io/
:alt: Documentation
.. image:: https://img.shields.io/pypi/v/django-kafka-streamer.svg
:target: https://pypi.org/project/django-kafka-streamer/
:alt: Current version on PyPi
.. image:: https://img.shields.io/pypi/pyversions/django-kafka-streamer
:alt: PyPI - Python Version
.. image:: https://img.shields.io/pypi/djversions/django-kafka-streamer
:alt: PyPI - Django Version
.. image:: https://img.shields.io/badge/Published%20on-Django%20Packages-0c3c26
:target: https://djangopackages.org/packages/p/django-kafka-streamer/
:alt: Published on Django Packages
django-kafka-streamer is a Django application and library for streaming data to
Apache Kafka.
Features:
* Setup signal handlers to ORM models to transparently send create/update/delete
events to Kafka
* Handle database object relations
* Celery task to stream large amount of data in background
Links:
* GitHub: https://github.com/lostclus/django-kafka-streamer/
* PyPI: https://pypi.org/project/django-kafka-streamer/
* Documentation: http://django-kafka-streamer.readthedocs.io/
* Consumer library: https://github.com/lostclus/aiosafeconsumer
* Example application: https://github.com/lostclus/WeatherApp
Usage:
`yourapp/models.py`::
from django.db import models
class MyModel(models.Model):
field1 = models.IntegerField()
field2 = models.CharField(max_length=10)
`yourapp/stramers.py`::
from kafkastreamer import Streamer, register
from .models import MyModel
@register(MyModel)
class MyModelStreamer(Streamer):
topic = "model-a"
`yourproject/settings.py`::
INSTALLED_APPS = [
...
"kafkastreamer",
]
KAFKA_STREAMER = {
"BOOTSTRAP_SERVERS": ["localhost:9092"],
},
Any changes in ``MyModel`` data will be automatically streamed to Kafka. To
force stream all data in all registered models type::
python manage.py kafkastreamer_refresh
The data streamed to the ``model-a`` Kafka topic has following structure::
{
"_time": "2023-01-01T00:00:00Z",
"_type": "create",
"id": 1,
"field1": 1,
"field2": "abc"
}
Raw data
{
"_id": null,
"home_page": null,
"name": "django-kafka-streamer",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "Django, Kafka, microservices",
"author": null,
"author_email": "Kostiantyn Korikov <lostclus@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/6c/2b/4e2d275d5bb570e44c8b5af07bcdfea38824afc83d507e1c0955445d9421/django_kafka_streamer-1.4.1.tar.gz",
"platform": null,
"description": "django-kafka-streamer\n=====================\n\n.. image:: https://github.com/lostclus/django-kafka-streamer/actions/workflows/tests.yml/badge.svg\n :target: https://github.com/lostclus/django-kafka-streamer/actions\n\n.. image:: https://readthedocs.org/projects/django-kafka-streamer/badge/?version=latest\n :target: http://django-kafka-streamer.readthedocs.io/\n :alt: Documentation\n\n.. image:: https://img.shields.io/pypi/v/django-kafka-streamer.svg\n :target: https://pypi.org/project/django-kafka-streamer/\n :alt: Current version on PyPi\n\n.. image:: https://img.shields.io/pypi/pyversions/django-kafka-streamer\n :alt: PyPI - Python Version\n\n.. image:: https://img.shields.io/pypi/djversions/django-kafka-streamer\n :alt: PyPI - Django Version\n\n.. image:: https://img.shields.io/badge/Published%20on-Django%20Packages-0c3c26\n :target: https://djangopackages.org/packages/p/django-kafka-streamer/\n :alt: Published on Django Packages\n\ndjango-kafka-streamer is a Django application and library for streaming data to\nApache Kafka.\n\nFeatures:\n\n* Setup signal handlers to ORM models to transparently send create/update/delete\n events to Kafka\n* Handle database object relations\n* Celery task to stream large amount of data in background\n\nLinks:\n\n* GitHub: https://github.com/lostclus/django-kafka-streamer/\n* PyPI: https://pypi.org/project/django-kafka-streamer/\n* Documentation: http://django-kafka-streamer.readthedocs.io/\n* Consumer library: https://github.com/lostclus/aiosafeconsumer\n* Example application: https://github.com/lostclus/WeatherApp\n\nUsage:\n\n`yourapp/models.py`::\n\n from django.db import models\n\n class MyModel(models.Model):\n field1 = models.IntegerField()\n field2 = models.CharField(max_length=10)\n\n`yourapp/stramers.py`::\n\n from kafkastreamer import Streamer, register\n from .models import MyModel\n\n @register(MyModel)\n class MyModelStreamer(Streamer):\n topic = \"model-a\"\n\n`yourproject/settings.py`::\n\n INSTALLED_APPS = [\n ...\n \"kafkastreamer\",\n ]\n\n KAFKA_STREAMER = {\n \"BOOTSTRAP_SERVERS\": [\"localhost:9092\"],\n },\n\nAny changes in ``MyModel`` data will be automatically streamed to Kafka. To\nforce stream all data in all registered models type::\n\n python manage.py kafkastreamer_refresh\n\nThe data streamed to the ``model-a`` Kafka topic has following structure::\n\n {\n \"_time\": \"2023-01-01T00:00:00Z\",\n \"_type\": \"create\",\n \"id\": 1,\n \"field1\": 1,\n \"field2\": \"abc\"\n }\n",
"bugtrack_url": null,
"license": null,
"summary": "Stream data to Apache Kafka.",
"version": "1.4.1",
"project_urls": {
"Documentation": "http://django-kafka-streamer.readthedocs.io/",
"Repository": "http://github.com/lostclus/django-kafka-streamer"
},
"split_keywords": [
"django",
" kafka",
" microservices"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c8f12e1305181f75ede74762d058dd79d3d5506fedabc61f5e872fadff5d0b60",
"md5": "6005fa37325f2b95586bec93189f05ba",
"sha256": "23e672a79d39af08834c57a3434f687d30ae02d8285805953fd39414619e82b2"
},
"downloads": -1,
"filename": "django_kafka_streamer-1.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "6005fa37325f2b95586bec93189f05ba",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 20314,
"upload_time": "2024-11-02T19:42:03",
"upload_time_iso_8601": "2024-11-02T19:42:03.256686Z",
"url": "https://files.pythonhosted.org/packages/c8/f1/2e1305181f75ede74762d058dd79d3d5506fedabc61f5e872fadff5d0b60/django_kafka_streamer-1.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6c2b4e2d275d5bb570e44c8b5af07bcdfea38824afc83d507e1c0955445d9421",
"md5": "3e1bd0c2eeaf66201e76eeecd021e38d",
"sha256": "20c043e6f9d865ea32ef9ef9182e07e7be167e8cc28121fb8ca311cfb5e50167"
},
"downloads": -1,
"filename": "django_kafka_streamer-1.4.1.tar.gz",
"has_sig": false,
"md5_digest": "3e1bd0c2eeaf66201e76eeecd021e38d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 30299,
"upload_time": "2024-11-02T19:42:04",
"upload_time_iso_8601": "2024-11-02T19:42:04.886778Z",
"url": "https://files.pythonhosted.org/packages/6c/2b/4e2d275d5bb570e44c8b5af07bcdfea38824afc83d507e1c0955445d9421/django_kafka_streamer-1.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-02 19:42:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lostclus",
"github_project": "django-kafka-streamer",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "django-kafka-streamer"
}