edX Event Bus - Kafka
#####################
Kafka implementation for Open edX event bus.
|pypi-badge| |ci-badge| |codecov-badge| |doc-badge| |pyversions-badge|
|license-badge|
Overview
********
This package implements an event bus for Open edX using Kafka.
The event bus acts as a broker between services publishing events and other services that consume these events.
Implementing the event bus will allow for asynchronous messaging across services which greatly improves efficiency as we don't have to wait for scheduled batch synchronizations.
Additionally, since the services all speak to the event bus, they are independent of one another and can still function if one service crashes.
This package contains both the publishing code, which processes events into messages to send to the broker, and the consumer code,
which polls the broker using a `while True` loop in order to turn messages back into event data to be emitted.
The actual Kafka host will be configurable.
The goal for event-bus-kafka is to eventually have a flexible event bus that can be easily brought into other apps and repositories to produce and consume arbitrary topics.
Ideally, the event bus itself will also be an abstraction behind which platform maintainers can use non-Kafka implementations (Redis, Pulsar, etc.).
The documentation/ADRs may also be moved to more appropriate places as the process matures.
The repository works together with the openedx/openedx-events repository to make the fully functional event bus.
Documentation
*************
To use this implementation of the Event Bus with openedx-events, you'll need to ensure that you include the dependency ``confluent_kafka[avro,schema-registry]`` (see `ADR 5 <https://github.com/openedx/event-bus-kafka/blob/main/docs/decisions/0005-optional-import-of-confluent-kafka.rst>`_ for an explanation) and set the following Django settings::
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS: ...
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL: ...
EVENT_BUS_TOPIC_PREFIX: ...
# On the producing side:
EVENT_BUS_PRODUCER: edx_event_bus_kafka.create_producer
# On the consuming side:
EVENT_BUS_CONSUMER: edx_event_bus_kafka.KafkaEventConsumer
Optional settings that are worth considering:
- ``EVENT_BUS_KAFKA_CONSUMER_CONSECUTIVE_ERRORS_LIMIT``
For manual testing, see `<docs/how_tos/manual_testing.rst>`__.
Django management commands:
- If you need to test event production, use the ``produce_event`` management command
- To actually run the consumer, use openedx-events' ``consume_events`` management command
OEP-52 documentation: https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0052-arch-event-bus-architecture.html
(TODO: `Set up documentation <https://openedx.atlassian.net/wiki/spaces/DOC/pages/21627535/Publish+Documentation+on+Read+the+Docs>`_)
Development Workflow
********************
One Time Setup
==============
.. code-block::
# Clone the repository
git clone git@github.com:openedx/event-bus-kafka.git
cd event-bus-kafka
# Set up a virtualenv using virtualenvwrapper with the same name as the repo and activate it
mkvirtualenv -p python3.8 event-bus-kafka
Every time you develop something in this repo
=============================================
.. code-block::
# Activate the virtualenv
workon event-bus-kafka
# Grab the latest code
git checkout main
git pull
# Install/update the dev requirements
make requirements
# Run the tests and quality checks (to verify the status before you make any changes)
make validate
# Make a new branch for your changes
git checkout -b <your_github_username>/<short_description>
# Using your favorite editor, edit the code to make your change.
vim …
# Run your new tests
pytest ./path/to/new/tests
# Run all the tests and quality checks
make validate
# Commit all your changes
git commit …
git push
# Open a PR and ask for review.
Installation
************
This library is not intended to be a direct dependency of any service. Instead, it will likely be installed by some private mechanism. Unless the platform gains new processes for installing private dependencies, upgrades will need to be manually managed via a pin.
License
*******
The code in this repository is licensed under the AGPL 3.0 unless
otherwise noted.
Please see `LICENSE.txt <LICENSE.txt>`_ for details.
How To Contribute
*****************
Contributions are very welcome.
Please read `How To Contribute <https://github.com/openedx/.github/blob/master/CONTRIBUTING.md>`_ for details.
The pull request description template should be automatically applied if you are creating a pull request from GitHub. Otherwise you
can find it at `PULL_REQUEST_TEMPLATE.md <.github/PULL_REQUEST_TEMPLATE.md>`_.
The issue report template should be automatically applied if you are creating an issue on GitHub as well. Otherwise you
can find it at `ISSUE_TEMPLATE.md <.github/ISSUE_TEMPLATE.md>`_.
Reporting Security Issues
*************************
Please do not report security issues in public. Please email security@openedx.org.
Getting Help
************
If you're having trouble, we have discussion forums at https://discuss.openedx.org where you can connect with others in the community.
Our real-time conversations are on Slack. You can request a `Slack invitation`_, then join our `community Slack workspace`_.
For more information about these options, see the `Getting Help`_ page.
.. _Slack invitation: https://openedx-slack-invite.herokuapp.com/
.. _community Slack workspace: https://openedx.slack.com/
.. _Getting Help: https://openedx.org/getting-help
.. |pypi-badge| image:: https://img.shields.io/pypi/v/edx-event-bus-kafka.svg
:target: https://pypi.python.org/pypi/edx-event-bus-kafka/
:alt: PyPI
.. |ci-badge| image:: https://github.com/openedx/event-bus-kafka/workflows/Python%20CI/badge.svg?branch=main
:target: https://github.com/openedx/event-bus-kafka/actions
:alt: CI
.. |codecov-badge| image:: https://codecov.io/github/openedx/event-bus-kafka/coverage.svg?branch=main
:target: https://codecov.io/github/openedx/event-bus-kafka?branch=main
:alt: Codecov
.. |doc-badge| image:: https://readthedocs.org/projects/edx-event-bus-kafka/badge/?version=latest
:target: https://edx-event-bus-kafka.readthedocs.io/en/latest/
:alt: Documentation
.. |pyversions-badge| image:: https://img.shields.io/pypi/pyversions/edx-event-bus-kafka.svg
:target: https://pypi.python.org/pypi/edx-event-bus-kafka/
:alt: Supported Python versions
.. |license-badge| image:: https://img.shields.io/github/license/openedx/event-bus-kafka.svg
:target: https://github.com/openedx/event-bus-kafka/blob/main/LICENSE.txt
:alt: License
Change Log
##########
..
All enhancements and patches to edx_event_bus_kafka will be documented
in this file. It adheres to the structure of https://keepachangelog.com/ ,
but in reStructuredText instead of Markdown (for ease of incorporation into
Sphinx documentation and the PyPI description).
This project adheres to Semantic Versioning (https://semver.org/).
.. There should always be an "Unreleased" section for changes pending release.
Unreleased
**********
[5.8.1] - 2024-08-02
********************
Changed
=======
* Monitoring: Add a custom attribute, ``kafka_received_message`` to track whether a message was processed or not.
[5.8.0] - 2024-08-01
********************
Changed
=======
* Monitoring: Ensure that we have a root span for each iteration of the consume loop; renamed the trace name to be ``consumer.consume``.
[5.7.0] - 2024-03-22
********************
Changed
=======
* Dropped Support for Django 3.2, Django 4.0 and Django 4.1
* Added Support for Python 3.12
[5.6.0] - 2024-01-25
********************
Changed
=======
* Added client.id to base configuration.
[5.5.0] - 2023-09-21
********************
Changed
=======
* Reset edx-django-utils RequestCache before handling each event
[5.4.0] - 2023-08-28
********************
Changed
=======
* Changed ordering of certain context assignments in producer code.
* Adds custom exceptions for producing and consuming errors.
[5.3.1] - 2023-08-10
********************
Fixed
=====
* Constrain openedx-events version to at least 8.5.0 (for EventsMetadata serialization support)
[5.3.0] - 2023-08-08
********************
Changed
=======
* Added event_metadata_as_dict to ProducingContext for easier replay from logs
[5.2.0] - 2023-08-03
********************
Changed
=======
* Added event_data_as_dict to ProducingContext for easier replay from logs
[5.1.0] - 2023-05-17
********************
Changed
=======
* Reconfigured serializers to use topic_record_name_strategy, allowing multiple event types per topic
[5.0.0] - 2023-05-17
********************
Changed
=======
* **BREAKING CHANGE**: Removed deprecated ``signal`` argument from consumer initialization
[4.0.1] - 2023-05-10
********************
Fixed
=====
* Added ``signal`` back as an argument when creating a consumer for compatibility with the openedx-events API
[4.0.0] - 2023-05-10
********************
Changed
=======
* Implement openedx-events Event Bus Consumer API.
* **BREAKING CHANGE**: Remove ``consume_events`` management command as this command will be provided by openedx_events. To replay events using the
openedx-events version of the management command, pass ``--extra '{"offset_time": "2023-01-08T06:46:22"}'`` instead of ``-o 2023-01-08T06:46:22``.
* **BREAKING Change**: ``offset_timestamp`` argument has been removed from ``consume_indefinitely`` and ``reset_offsets_and_sleep_indefinitely`` methods.
It is now added as an optional argument named ``offset_time`` to ``KafkaEventConsumer`` constructor.
[3.10.0] - 2023-05-05
*********************
Changed
=======
* Switch from ``edx-sphinx-theme`` to ``sphinx-book-theme`` since the former is
deprecated
* Refactored consumer to manually deserialize messages instead of using DeserializingConsumer
* Make signal argument optional in consumer command (take signal from message headers)
[3.9.6] - 2023-02-24
********************
Added
=====
* Add function tracing to the event consumption workflow for better monitoring.
[3.9.5] - 2023-02-24
********************
Fixed
=====
* ``consume_events`` command now loads all public signals so that the consumer can load signals by ``event_type`` even if their modules were not already imported
[3.9.4] - 2023-02-16
********************
Fixed
=====
* produce_event management command fixed to pass required events_metadata parameter
[3.9.3] - 2023-02-10
********************
Fixed
=====
* Include ``message_id`` in audit log when message is produced (was ``None``)
[3.9.2] - 2023-02-08
********************
Fixed
=====
* Added documentation to all Django settings used in consumer and producer
[3.9.1] - 2023-02-07
********************
There was no version 3.9.0, due to a release issue. (Ignore any ``v3.9.0`` tag.)
Added
=====
* Added logging of successful message production, consumption, and processing (not just errors)
[3.8.1] - 2023-02-03
********************
Changed
=======
* Require and use openedx-events 5.0.0, which has a breaking API change that affects event bus consumers.
[3.8.0] - 2023-01-31
********************
Added
=====
* Producer now passes all metadata fields as headers
* Consumer emits events with the original metadata information (from the producer)
[3.7.1] - 2023-01-31
********************
Fixed
=====
* Consumer management command exits with useful error message if confluent-kafka library not available.
[3.7.0] - 2023-01-30
********************
Changed
=======
* Added ``reset_offsets_and_sleep_indefinitely`` method to consumer; relying code should switch to calling this when an offset timestamp is set.
* Deprecated the ``offset_timestamp`` parameter on the consumer's ``consume_indefinitely`` method, since ``reset_offsets_and_sleep_indefinitely`` should be used instead.
[3.6.3] - 2023-01-27
********************
Fixed
=====
* In the consumer loop, open new database connection if the old one has closed before processing messages
[3.6.2] - 2023-01-26
********************
Fixed
=====
* Reset mode now commits the correct offsets to Kafka.
[3.6.1] - 2023-01-20
********************
Fixed
=======
* Added a poll call to force resets to be processed during replay/offset-reset mode.
[3.6.0] - 2023-01-06
********************
Changed
=======
* Consumers do not consume events after resetting offsets.
[3.5.1] - 2023-01-06
********************
Fixed
=====
* Convert timestamps to millisecond offsets as expected by the Kafka API when replaying by timestamp in the consumer
[3.5.0] - 2023-01-05
********************
Added
=====
* New setting ``EVENT_BUS_KAFKA_CONSUMER_CONSECUTIVE_ERRORS_LIMIT`` will terminate the consumer if too many errors occur in a row, supporting an automated version of "have you tried turning it off and on again" (as long as consumer will automatically be restarted e.g. by Kubernetes).
[3.4.1] - 2022-12-20
********************
Fixed
=====
* Fixed bugs in the event replay/offset handling code for consumers.
[3.4.0] - 2022-12-16
********************
Changed
=======
* Kill infinite consumer loop when we see a fatal KafkaError, as recommended in the documentation. See https://github.com/confluentinc/librdkafka/blob/e0b9e92a0b492b5b1a6f1bcf08744928d45bf396/INTRODUCTION.md#fatal-consumer-errors.
[3.3.0] - 2022-12-15
********************
Changed
=======
* Added/removed some custom attributes used for monitoring. Search for custom_attribute_name annotations for details.
[3.2.0] - 2022-12-14
********************
Changed
=======
* Add timestamp parameter for consumer, allowing the starting offset for consuming to be overridden from the default.
[3.1.0] - 2022-12-07
********************
Added
=====
* A variety of custom attributes are now set for monitoring purposes. Search for custom_attribute_name annotations for details.
Changed
=======
* Error recording to the log will now include message details when the message is available on the error.
[3.0.0] - 2022-12-02
********************
Changed
=======
* **BREAKING CHANGE**: Make event_metadata parameter required
[2.1.0] - 2022-12-01
********************
Changed
=======
* Set CloudEvent headers on events using an optional event_metadata parameter
[2.0.0] - 2022-11-28
********************
Changed
=======
* Implement openedx-events Event Bus Producer API
* **BREAKING CHANGE**: Remove caching from ``get_producer`` and rename to ``create_producer``, as we now rely on the wrapper in openedx-events to cache that call
Upgrading library from 1.x:
- Replace calls to ``edx_event_bus_kafka.get_producer`` with ``openedx_events.event_bus.get_producer``
- Add Django setting ``EVENT_BUS_PRODUCER = "edx_event_bus_kafka.create_producer"``
These breaking changes are only relevant for the producing side. (This should only include the CMS at the moment.)
[1.10.0] - 2022-11-21
*********************
Changed
=======
* Improve receiver error log message -- mention that stack traces are elsewhere in log
[1.9.0] - 2022-11-15
********************
Changed
=======
* Log and record receiver errors the same way as other errors (with offset, partition, etc.)
[1.8.1] - 2022-11-10
********************
Changed
=======
* Commit consumer offset asynchronously
[1.8.0] - 2022-11-09
********************
Added
=====
* Consumer logs a warning for receivers that fail with an exception
[1.7.0] - 2022-11-04
********************
Changed
=======
* Manually manage commits instead of using auto-commit on the consumer
* Catch Exception instead of BaseException on both producer and consumer
[1.6.0] - 2022-11-04
********************
Changed
=======
* Enhanced error logging in consumer, including telemetry for exceptions
* Consumer loop will no longer exit when an error is encountered
[1.5.0] - 2022-11-01
********************
Changed
=======
* Log full event data on all producer errors
[1.4.3] - 2022-10-31
********************
Fixed
=====
* Upgrade openedx-events and fastavro to bring in a fix for schema creation
[1.4.2] - 2022-10-31
********************
Fixed
=====
* Removed proof-of-concept code that logged user-login events
[1.4.1] - 2022-10-28
********************
Fixed
=====
* Correct and clarify management command help strings (some copy-paste errors)
* Update TODO comments
[1.4.0] - 2022-10-21
********************
Changed
=======
* Remove override of auto.offset.reset on consumer (which will default to "latest"). New consumer groups will consume only messages that are sent after the group was initialized.
* Remove redundant lookup of signal in consumer loop (should not have any effect)
* Explicitly encode message header values as UTF-8 (no change in behavior)
[1.3.0] - 2022-10-20
********************
Changed
=======
* Upgrade openedx-events. When AvroSignalSerializer gets event schemas, it will get whatever is currently defined in openedx-events, so this will update the COURSE_CATALOG_EVENT_CHANGED schema (dropping `effort` field)
[1.2.0] - 2022-10-13
********************
Changed
=======
* ``EVENT_BUS_KAFKA_CONSUMERS_ENABLED`` now defaults to True instead of False
* Removed manual monitoring since New Relic tracks these now.
[1.1.0] - 2022-10-06
********************
Changed
=======
* Added monitoring for consumption tasks.
[1.0.0] - 2022-10-03
********************
Changed
=======
* Fixed bug in schema registry that was sending schemas to the wrong topic
* Bump version to 1.x to acknowledge that this is in use in production
[0.7.0] - 2022-09-08
********************
Changed
=======
* **Breaking changes** ``EventProducerKafka`` is now ``KafkaEventProducer``
* KafkaEventConsumer is now part of the public API
[0.6.2] - 2022-09-08
********************
Added
=====
* Topic names can be autoprefixed by setting ``EVENT_BUS_TOPIC_PREFIX``
[0.6.1] - 2022-09-06
********************
Added
=====
* Producer now polls on an interval, improving callback reliability. Configurable with ``EVENT_BUS_KAFKA_POLL_INTERVAL_SEC``.
[0.6.0] - 2022-09-01
********************
Changed
=======
* **Breaking change**: Public API is now defined in ``edx_event_bus_kafka`` package and ``edx_event_bus_kafka.management.commands`` package; all other modules should be considered unstable and not for external use.
[0.5.1] - 2022-08-31
********************
Fixed
=====
* Various lint issues (and missing ``__init__.py`` files.)
[0.5.0] - 2022-08-31
********************
Changed
=======
* **Breaking changes** in the producer module, refactored to expose a better API:
* Rather than ``send_to_event_bus(...)``, relying code should now call ``get_producer().send(...)``.
* The ``sync`` kwarg is gone; to flush and sync messages before shutdown, call ``get_producer().prepare_for_shutdown()`` instead.
* Clarify that config module is for internal use only.
* Implementation changes: Only a single Producer is created, and is used for all signals.
[0.4.4] - 2022-08-26
********************
Fixed
=====
* Fixed bug in test module for when confluent-kafka isn't present
[0.4.3] - 2022-08-24
********************
Fixed
=====
* Never evict producers from cache. There wasn't a real risk of this, but now we can rely on them being long-lived. Addresses remainder of `<https://github.com/openedx/event-bus-kafka/issues/16>`__.
[0.4.2] - 2022-08-24
********************
Fixed
=====
* Properly load auth settings for producer/consumer. (Auth settings were ignored since 0.3.1.)
[0.4.1] - 2022-08-18
********************
Changed
=======
* Remove confluent-kafka as a formal dependency of the repository.
* Note: This library will not work without confluent-kafka.
* Add an ADR to explain why this work was done.
[0.4.0] - 2022-08-15
********************
Changed
=======
* Rename settings to have consistent prefix.
* ``KAFKA_CONSUMERS_ENABLED`` becomes ``EVENT_BUS_KAFKA_CONSUMERS_ENABLED``
* ``CONSUMER_POLL_TIMEOUT`` becomes ``EVENT_BUS_KAFKA_CONSUMER_POLL_TIMEOUT``
* Updates to documentation and tests for various settings previously renamed
[0.3.1] - 2022-08-11
********************
Changed
=======
* Refactored consumer to use common configuration.
[0.3.0] - 2022-08-10
********************
Changed
=======
* Moved configuration onto separate file.
* Updated configuration settings to have EVENT_BUS_KAFKA prefix.
[0.2.0] - 2022-08-09
********************
Fixed
=====
* Cache producers so that they don't lose data.
[0.1.0] - 2022-06-16
********************
Added
=====
* First release on PyPI.
Raw data
{
"_id": null,
"home_page": "https://github.com/openedx/event-bus-kafka",
"name": "edx-event-bus-kafka",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Python edx",
"author": "edX",
"author_email": "oscm@edx.org",
"download_url": "https://files.pythonhosted.org/packages/18/fb/be5cbff255dd1a7e4a90dffe5b266ab5743e83f7934624d87d132f11d252/edx_event_bus_kafka-5.8.1.tar.gz",
"platform": null,
"description": "edX Event Bus - Kafka\n#####################\n\nKafka implementation for Open edX event bus.\n\n|pypi-badge| |ci-badge| |codecov-badge| |doc-badge| |pyversions-badge|\n|license-badge|\n\nOverview\n********\nThis package implements an event bus for Open edX using Kafka.\n\nThe event bus acts as a broker between services publishing events and other services that consume these events.\nImplementing the event bus will allow for asynchronous messaging across services which greatly improves efficiency as we don't have to wait for scheduled batch synchronizations.\nAdditionally, since the services all speak to the event bus, they are independent of one another and can still function if one service crashes.\n\nThis package contains both the publishing code, which processes events into messages to send to the broker, and the consumer code,\nwhich polls the broker using a `while True` loop in order to turn messages back into event data to be emitted.\nThe actual Kafka host will be configurable.\n\nThe goal for event-bus-kafka is to eventually have a flexible event bus that can be easily brought into other apps and repositories to produce and consume arbitrary topics.\nIdeally, the event bus itself will also be an abstraction behind which platform maintainers can use non-Kafka implementations (Redis, Pulsar, etc.).\nThe documentation/ADRs may also be moved to more appropriate places as the process matures.\n\nThe repository works together with the openedx/openedx-events repository to make the fully functional event bus.\n\nDocumentation\n*************\n\nTo use this implementation of the Event Bus with openedx-events, you'll need to ensure that you include the dependency ``confluent_kafka[avro,schema-registry]`` (see `ADR 5 <https://github.com/openedx/event-bus-kafka/blob/main/docs/decisions/0005-optional-import-of-confluent-kafka.rst>`_ for an explanation) and set the following Django settings::\n\n EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS: ...\n EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL: ...\n EVENT_BUS_TOPIC_PREFIX: ...\n\n # On the producing side:\n EVENT_BUS_PRODUCER: edx_event_bus_kafka.create_producer\n\n # On the consuming side:\n EVENT_BUS_CONSUMER: edx_event_bus_kafka.KafkaEventConsumer\n\n\nOptional settings that are worth considering:\n\n- ``EVENT_BUS_KAFKA_CONSUMER_CONSECUTIVE_ERRORS_LIMIT``\n\nFor manual testing, see `<docs/how_tos/manual_testing.rst>`__.\n\nDjango management commands:\n\n- If you need to test event production, use the ``produce_event`` management command\n- To actually run the consumer, use openedx-events' ``consume_events`` management command\n\nOEP-52 documentation: https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0052-arch-event-bus-architecture.html\n(TODO: `Set up documentation <https://openedx.atlassian.net/wiki/spaces/DOC/pages/21627535/Publish+Documentation+on+Read+the+Docs>`_)\n\nDevelopment Workflow\n********************\n\nOne Time Setup\n==============\n.. code-block::\n\n # Clone the repository\n git clone git@github.com:openedx/event-bus-kafka.git\n cd event-bus-kafka\n\n # Set up a virtualenv using virtualenvwrapper with the same name as the repo and activate it\n mkvirtualenv -p python3.8 event-bus-kafka\n\n\nEvery time you develop something in this repo\n=============================================\n.. code-block::\n\n # Activate the virtualenv\n workon event-bus-kafka\n\n # Grab the latest code\n git checkout main\n git pull\n\n # Install/update the dev requirements\n make requirements\n\n # Run the tests and quality checks (to verify the status before you make any changes)\n make validate\n\n # Make a new branch for your changes\n git checkout -b <your_github_username>/<short_description>\n\n # Using your favorite editor, edit the code to make your change.\n vim \u2026\n\n # Run your new tests\n pytest ./path/to/new/tests\n\n # Run all the tests and quality checks\n make validate\n\n # Commit all your changes\n git commit \u2026\n git push\n\n # Open a PR and ask for review.\n\nInstallation\n************\n\nThis library is not intended to be a direct dependency of any service. Instead, it will likely be installed by some private mechanism. Unless the platform gains new processes for installing private dependencies, upgrades will need to be manually managed via a pin.\n\nLicense\n*******\n\nThe code in this repository is licensed under the AGPL 3.0 unless\notherwise noted.\n\nPlease see `LICENSE.txt <LICENSE.txt>`_ for details.\n\nHow To Contribute\n*****************\n\nContributions are very welcome.\nPlease read `How To Contribute <https://github.com/openedx/.github/blob/master/CONTRIBUTING.md>`_ for details.\n\nThe pull request description template should be automatically applied if you are creating a pull request from GitHub. Otherwise you\ncan find it at `PULL_REQUEST_TEMPLATE.md <.github/PULL_REQUEST_TEMPLATE.md>`_.\n\nThe issue report template should be automatically applied if you are creating an issue on GitHub as well. Otherwise you\ncan find it at `ISSUE_TEMPLATE.md <.github/ISSUE_TEMPLATE.md>`_.\n\nReporting Security Issues\n*************************\n\nPlease do not report security issues in public. Please email security@openedx.org.\n\nGetting Help\n************\n\nIf you're having trouble, we have discussion forums at https://discuss.openedx.org where you can connect with others in the community.\n\nOur real-time conversations are on Slack. You can request a `Slack invitation`_, then join our `community Slack workspace`_.\n\nFor more information about these options, see the `Getting Help`_ page.\n\n.. _Slack invitation: https://openedx-slack-invite.herokuapp.com/\n.. _community Slack workspace: https://openedx.slack.com/\n.. _Getting Help: https://openedx.org/getting-help\n\n.. |pypi-badge| image:: https://img.shields.io/pypi/v/edx-event-bus-kafka.svg\n :target: https://pypi.python.org/pypi/edx-event-bus-kafka/\n :alt: PyPI\n\n.. |ci-badge| image:: https://github.com/openedx/event-bus-kafka/workflows/Python%20CI/badge.svg?branch=main\n :target: https://github.com/openedx/event-bus-kafka/actions\n :alt: CI\n\n.. |codecov-badge| image:: https://codecov.io/github/openedx/event-bus-kafka/coverage.svg?branch=main\n :target: https://codecov.io/github/openedx/event-bus-kafka?branch=main\n :alt: Codecov\n\n.. |doc-badge| image:: https://readthedocs.org/projects/edx-event-bus-kafka/badge/?version=latest\n :target: https://edx-event-bus-kafka.readthedocs.io/en/latest/\n :alt: Documentation\n\n.. |pyversions-badge| image:: https://img.shields.io/pypi/pyversions/edx-event-bus-kafka.svg\n :target: https://pypi.python.org/pypi/edx-event-bus-kafka/\n :alt: Supported Python versions\n\n.. |license-badge| image:: https://img.shields.io/github/license/openedx/event-bus-kafka.svg\n :target: https://github.com/openedx/event-bus-kafka/blob/main/LICENSE.txt\n :alt: License\n\n\nChange Log\n##########\n\n..\n All enhancements and patches to edx_event_bus_kafka will be documented\n in this file. It adheres to the structure of https://keepachangelog.com/ ,\n but in reStructuredText instead of Markdown (for ease of incorporation into\n Sphinx documentation and the PyPI description).\n\n This project adheres to Semantic Versioning (https://semver.org/).\n\n.. There should always be an \"Unreleased\" section for changes pending release.\n\nUnreleased\n**********\n\n[5.8.1] - 2024-08-02\n********************\nChanged\n=======\n* Monitoring: Add a custom attribute, ``kafka_received_message`` to track whether a message was processed or not.\n\n[5.8.0] - 2024-08-01\n********************\nChanged\n=======\n* Monitoring: Ensure that we have a root span for each iteration of the consume loop; renamed the trace name to be ``consumer.consume``.\n\n[5.7.0] - 2024-03-22\n********************\nChanged\n=======\n* Dropped Support for Django 3.2, Django 4.0 and Django 4.1\n* Added Support for Python 3.12\n\n[5.6.0] - 2024-01-25\n********************\nChanged\n=======\n* Added client.id to base configuration.\n\n[5.5.0] - 2023-09-21\n********************\nChanged\n=======\n* Reset edx-django-utils RequestCache before handling each event\n\n[5.4.0] - 2023-08-28\n********************\nChanged\n=======\n* Changed ordering of certain context assignments in producer code.\n* Adds custom exceptions for producing and consuming errors.\n\n[5.3.1] - 2023-08-10\n********************\nFixed\n=====\n* Constrain openedx-events version to at least 8.5.0 (for EventsMetadata serialization support)\n\n[5.3.0] - 2023-08-08\n********************\nChanged\n=======\n* Added event_metadata_as_dict to ProducingContext for easier replay from logs\n\n[5.2.0] - 2023-08-03\n********************\nChanged\n=======\n* Added event_data_as_dict to ProducingContext for easier replay from logs\n\n[5.1.0] - 2023-05-17\n********************\nChanged\n=======\n* Reconfigured serializers to use topic_record_name_strategy, allowing multiple event types per topic\n\n[5.0.0] - 2023-05-17\n********************\nChanged\n=======\n* **BREAKING CHANGE**: Removed deprecated ``signal`` argument from consumer initialization\n\n[4.0.1] - 2023-05-10\n********************\nFixed\n=====\n* Added ``signal`` back as an argument when creating a consumer for compatibility with the openedx-events API\n\n[4.0.0] - 2023-05-10\n********************\nChanged\n=======\n* Implement openedx-events Event Bus Consumer API.\n* **BREAKING CHANGE**: Remove ``consume_events`` management command as this command will be provided by openedx_events. To replay events using the\n openedx-events version of the management command, pass ``--extra '{\"offset_time\": \"2023-01-08T06:46:22\"}'`` instead of ``-o 2023-01-08T06:46:22``.\n* **BREAKING Change**: ``offset_timestamp`` argument has been removed from ``consume_indefinitely`` and ``reset_offsets_and_sleep_indefinitely`` methods.\n It is now added as an optional argument named ``offset_time`` to ``KafkaEventConsumer`` constructor.\n\n[3.10.0] - 2023-05-05\n*********************\nChanged\n=======\n* Switch from ``edx-sphinx-theme`` to ``sphinx-book-theme`` since the former is\n deprecated\n* Refactored consumer to manually deserialize messages instead of using DeserializingConsumer\n* Make signal argument optional in consumer command (take signal from message headers)\n\n[3.9.6] - 2023-02-24\n********************\nAdded\n=====\n* Add function tracing to the event consumption workflow for better monitoring.\n\n[3.9.5] - 2023-02-24\n********************\n\nFixed\n=====\n* ``consume_events`` command now loads all public signals so that the consumer can load signals by ``event_type`` even if their modules were not already imported\n\n[3.9.4] - 2023-02-16\n********************\n\nFixed\n=====\n* produce_event management command fixed to pass required events_metadata parameter\n\n[3.9.3] - 2023-02-10\n********************\nFixed\n=====\n* Include ``message_id`` in audit log when message is produced (was ``None``)\n\n[3.9.2] - 2023-02-08\n********************\nFixed\n=====\n* Added documentation to all Django settings used in consumer and producer\n\n[3.9.1] - 2023-02-07\n********************\nThere was no version 3.9.0, due to a release issue. (Ignore any ``v3.9.0`` tag.)\n\nAdded\n=====\n* Added logging of successful message production, consumption, and processing (not just errors)\n\n[3.8.1] - 2023-02-03\n********************\nChanged\n=======\n* Require and use openedx-events 5.0.0, which has a breaking API change that affects event bus consumers.\n\n[3.8.0] - 2023-01-31\n********************\nAdded\n=====\n* Producer now passes all metadata fields as headers\n* Consumer emits events with the original metadata information (from the producer)\n\n[3.7.1] - 2023-01-31\n********************\nFixed\n=====\n* Consumer management command exits with useful error message if confluent-kafka library not available.\n\n[3.7.0] - 2023-01-30\n********************\nChanged\n=======\n* Added ``reset_offsets_and_sleep_indefinitely`` method to consumer; relying code should switch to calling this when an offset timestamp is set.\n* Deprecated the ``offset_timestamp`` parameter on the consumer's ``consume_indefinitely`` method, since ``reset_offsets_and_sleep_indefinitely`` should be used instead.\n\n[3.6.3] - 2023-01-27\n********************\nFixed\n=====\n* In the consumer loop, open new database connection if the old one has closed before processing messages\n\n[3.6.2] - 2023-01-26\n********************\nFixed\n=====\n* Reset mode now commits the correct offsets to Kafka.\n\n[3.6.1] - 2023-01-20\n********************\nFixed\n=======\n* Added a poll call to force resets to be processed during replay/offset-reset mode.\n\n[3.6.0] - 2023-01-06\n********************\nChanged\n=======\n* Consumers do not consume events after resetting offsets.\n\n[3.5.1] - 2023-01-06\n********************\nFixed\n=====\n* Convert timestamps to millisecond offsets as expected by the Kafka API when replaying by timestamp in the consumer\n\n[3.5.0] - 2023-01-05\n********************\nAdded\n=====\n* New setting ``EVENT_BUS_KAFKA_CONSUMER_CONSECUTIVE_ERRORS_LIMIT`` will terminate the consumer if too many errors occur in a row, supporting an automated version of \"have you tried turning it off and on again\" (as long as consumer will automatically be restarted e.g. by Kubernetes).\n\n[3.4.1] - 2022-12-20\n********************\nFixed\n=====\n* Fixed bugs in the event replay/offset handling code for consumers.\n\n[3.4.0] - 2022-12-16\n********************\nChanged\n=======\n* Kill infinite consumer loop when we see a fatal KafkaError, as recommended in the documentation. See https://github.com/confluentinc/librdkafka/blob/e0b9e92a0b492b5b1a6f1bcf08744928d45bf396/INTRODUCTION.md#fatal-consumer-errors.\n\n[3.3.0] - 2022-12-15\n********************\nChanged\n=======\n* Added/removed some custom attributes used for monitoring. Search for custom_attribute_name annotations for details.\n\n[3.2.0] - 2022-12-14\n********************\nChanged\n=======\n* Add timestamp parameter for consumer, allowing the starting offset for consuming to be overridden from the default.\n\n[3.1.0] - 2022-12-07\n********************\n\nAdded\n=====\n* A variety of custom attributes are now set for monitoring purposes. Search for custom_attribute_name annotations for details.\n\nChanged\n=======\n* Error recording to the log will now include message details when the message is available on the error.\n\n[3.0.0] - 2022-12-02\n********************\nChanged\n=======\n* **BREAKING CHANGE**: Make event_metadata parameter required\n\n[2.1.0] - 2022-12-01\n********************\nChanged\n=======\n* Set CloudEvent headers on events using an optional event_metadata parameter\n\n[2.0.0] - 2022-11-28\n********************\nChanged\n=======\n* Implement openedx-events Event Bus Producer API\n* **BREAKING CHANGE**: Remove caching from ``get_producer`` and rename to ``create_producer``, as we now rely on the wrapper in openedx-events to cache that call\n\nUpgrading library from 1.x:\n\n- Replace calls to ``edx_event_bus_kafka.get_producer`` with ``openedx_events.event_bus.get_producer``\n- Add Django setting ``EVENT_BUS_PRODUCER = \"edx_event_bus_kafka.create_producer\"``\n\nThese breaking changes are only relevant for the producing side. (This should only include the CMS at the moment.)\n\n[1.10.0] - 2022-11-21\n*********************\nChanged\n=======\n* Improve receiver error log message -- mention that stack traces are elsewhere in log\n\n[1.9.0] - 2022-11-15\n********************\nChanged\n=======\n* Log and record receiver errors the same way as other errors (with offset, partition, etc.)\n\n[1.8.1] - 2022-11-10\n********************\nChanged\n=======\n* Commit consumer offset asynchronously\n\n[1.8.0] - 2022-11-09\n********************\nAdded\n=====\n* Consumer logs a warning for receivers that fail with an exception\n\n[1.7.0] - 2022-11-04\n********************\n\nChanged\n=======\n* Manually manage commits instead of using auto-commit on the consumer\n* Catch Exception instead of BaseException on both producer and consumer\n\n[1.6.0] - 2022-11-04\n********************\n\nChanged\n=======\n* Enhanced error logging in consumer, including telemetry for exceptions\n* Consumer loop will no longer exit when an error is encountered\n\n[1.5.0] - 2022-11-01\n********************\n\nChanged\n=======\n* Log full event data on all producer errors\n\n[1.4.3] - 2022-10-31\n********************\n\nFixed\n=====\n* Upgrade openedx-events and fastavro to bring in a fix for schema creation\n\n[1.4.2] - 2022-10-31\n********************\n\nFixed\n=====\n* Removed proof-of-concept code that logged user-login events\n\n[1.4.1] - 2022-10-28\n********************\n\nFixed\n=====\n* Correct and clarify management command help strings (some copy-paste errors)\n* Update TODO comments\n\n[1.4.0] - 2022-10-21\n********************\n\nChanged\n=======\n* Remove override of auto.offset.reset on consumer (which will default to \"latest\"). New consumer groups will consume only messages that are sent after the group was initialized.\n* Remove redundant lookup of signal in consumer loop (should not have any effect)\n* Explicitly encode message header values as UTF-8 (no change in behavior)\n\n[1.3.0] - 2022-10-20\n********************\n\nChanged\n=======\n\n* Upgrade openedx-events. When AvroSignalSerializer gets event schemas, it will get whatever is currently defined in openedx-events, so this will update the COURSE_CATALOG_EVENT_CHANGED schema (dropping `effort` field)\n\n[1.2.0] - 2022-10-13\n********************\n\nChanged\n=======\n\n* ``EVENT_BUS_KAFKA_CONSUMERS_ENABLED`` now defaults to True instead of False\n* Removed manual monitoring since New Relic tracks these now.\n\n[1.1.0] - 2022-10-06\n********************\n\nChanged\n=======\n\n* Added monitoring for consumption tasks.\n\n[1.0.0] - 2022-10-03\n********************\n\nChanged\n=======\n\n* Fixed bug in schema registry that was sending schemas to the wrong topic\n* Bump version to 1.x to acknowledge that this is in use in production\n\n[0.7.0] - 2022-09-08\n********************\n\nChanged\n=======\n\n* **Breaking changes** ``EventProducerKafka`` is now ``KafkaEventProducer``\n* KafkaEventConsumer is now part of the public API\n\n[0.6.2] - 2022-09-08\n********************\n\nAdded\n=====\n\n* Topic names can be autoprefixed by setting ``EVENT_BUS_TOPIC_PREFIX``\n\n[0.6.1] - 2022-09-06\n********************\n\nAdded\n=====\n\n* Producer now polls on an interval, improving callback reliability. Configurable with ``EVENT_BUS_KAFKA_POLL_INTERVAL_SEC``.\n\n[0.6.0] - 2022-09-01\n********************\n\nChanged\n=======\n\n* **Breaking change**: Public API is now defined in ``edx_event_bus_kafka`` package and ``edx_event_bus_kafka.management.commands`` package; all other modules should be considered unstable and not for external use.\n\n[0.5.1] - 2022-08-31\n********************\n\nFixed\n=====\n\n* Various lint issues (and missing ``__init__.py`` files.)\n\n[0.5.0] - 2022-08-31\n********************\n\nChanged\n=======\n\n* **Breaking changes** in the producer module, refactored to expose a better API:\n\n * Rather than ``send_to_event_bus(...)``, relying code should now call ``get_producer().send(...)``.\n * The ``sync`` kwarg is gone; to flush and sync messages before shutdown, call ``get_producer().prepare_for_shutdown()`` instead.\n\n* Clarify that config module is for internal use only.\n* Implementation changes: Only a single Producer is created, and is used for all signals.\n\n[0.4.4] - 2022-08-26\n********************\n\nFixed\n=====\n\n* Fixed bug in test module for when confluent-kafka isn't present\n\n[0.4.3] - 2022-08-24\n********************\n\nFixed\n=====\n\n* Never evict producers from cache. There wasn't a real risk of this, but now we can rely on them being long-lived. Addresses remainder of `<https://github.com/openedx/event-bus-kafka/issues/16>`__.\n\n[0.4.2] - 2022-08-24\n********************\n\nFixed\n=====\n\n* Properly load auth settings for producer/consumer. (Auth settings were ignored since 0.3.1.)\n\n[0.4.1] - 2022-08-18\n********************\n\nChanged\n=======\n\n* Remove confluent-kafka as a formal dependency of the repository.\n\n * Note: This library will not work without confluent-kafka.\n\n* Add an ADR to explain why this work was done.\n\n[0.4.0] - 2022-08-15\n********************\n\nChanged\n=======\n\n* Rename settings to have consistent prefix.\n\n * ``KAFKA_CONSUMERS_ENABLED`` becomes ``EVENT_BUS_KAFKA_CONSUMERS_ENABLED``\n * ``CONSUMER_POLL_TIMEOUT`` becomes ``EVENT_BUS_KAFKA_CONSUMER_POLL_TIMEOUT``\n * Updates to documentation and tests for various settings previously renamed\n\n[0.3.1] - 2022-08-11\n********************\n\nChanged\n=======\n\n* Refactored consumer to use common configuration.\n\n[0.3.0] - 2022-08-10\n********************\n\nChanged\n=======\n\n* Moved configuration onto separate file.\n* Updated configuration settings to have EVENT_BUS_KAFKA prefix.\n\n[0.2.0] - 2022-08-09\n********************\n\nFixed\n=====\n\n* Cache producers so that they don't lose data.\n\n[0.1.0] - 2022-06-16\n********************\n\nAdded\n=====\n\n* First release on PyPI.\n",
"bugtrack_url": null,
"license": "AGPL 3.0",
"summary": "Kafka implementation for Open edX event bus.",
"version": "5.8.1",
"project_urls": {
"Homepage": "https://github.com/openedx/event-bus-kafka"
},
"split_keywords": [
"python",
"edx"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "078d272fd5234cf2f05632190648e18be4d31b81b2c1c15aba6fd278e2d3f053",
"md5": "0f2ac00e1d4a404e4e580b1d0c5f040f",
"sha256": "890b15c9dbd1a5dc996cd8d3be38c2eb1419bb148fcd0ea9ace691e616b0a31b"
},
"downloads": -1,
"filename": "edx_event_bus_kafka-5.8.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "0f2ac00e1d4a404e4e580b1d0c5f040f",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8",
"size": 58882,
"upload_time": "2024-08-05T13:25:27",
"upload_time_iso_8601": "2024-08-05T13:25:27.585155Z",
"url": "https://files.pythonhosted.org/packages/07/8d/272fd5234cf2f05632190648e18be4d31b81b2c1c15aba6fd278e2d3f053/edx_event_bus_kafka-5.8.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "18fbbe5cbff255dd1a7e4a90dffe5b266ab5743e83f7934624d87d132f11d252",
"md5": "fcefe86619fb6b6fdde2ff52cf13fce4",
"sha256": "c4206194cc2a7b77a23647b8a19409d1e41828164c13beef4b43c73789edc6e2"
},
"downloads": -1,
"filename": "edx_event_bus_kafka-5.8.1.tar.gz",
"has_sig": false,
"md5_digest": "fcefe86619fb6b6fdde2ff52cf13fce4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 64738,
"upload_time": "2024-08-05T13:25:30",
"upload_time_iso_8601": "2024-08-05T13:25:30.079392Z",
"url": "https://files.pythonhosted.org/packages/18/fb/be5cbff255dd1a7e4a90dffe5b266ab5743e83f7934624d87d132f11d252/edx_event_bus_kafka-5.8.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-05 13:25:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "openedx",
"github_project": "event-bus-kafka",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "edx-event-bus-kafka"
}
JSON
