# KafkaObserver
## What is it?
The KafkaObserver package provides a set of tools for observing and retrieving information about Kafka topics and consumer groups. It facilitates monitoring and managing Kafka infrastructure by offering insights into offset statuses, watermarks, and retention policies.
## Main Features
- **Watermark Information:** Retrieve information about the watermark (low and high offsets) for a specified topic.
- **Retention Policy Information:** Retrieve information about the retention policy (time and size) for a specified topic.
- **Topic Information:** Obtain details about Kafka topics, including basic information and extended details like watermark information and retention policies.
- **Consumer Groups Information:** Retrieve information about existing consumer groups.
- **Offset Status:** Get the offset status (current, committed, etc.) for a specific consumer and topic. Plot the offset status for visualization.
## Requirements
- `confluent-kafka`
- `openpyxl`
You can install these dependencies manually or use the provided `requirement.txt` file in the repository.
## Installation
1. Create a New Virtual Environment (named `.venv` in this case):
```bash
python3 -m venv venv
```
2. Activate the Virtual Environment:
```bash
source venv/bin/activate
```
3. Install the Package:
To install the `qurix-kafka-observer` package, use `pip`:
```bash
pip install qurix-kafka-observer
```
## Usage
### Kafka Observer
To use the KafkaObserver class:
```python
from qurix.kafka.observer import KafkaObserver
from qurix.kafka.entities.config import ConsumerConfig
# Create a ConsumerConfig object with your Kafka settings
BOOTSTRAP_SERVERS = "localhost:9092"
consumer_config = ConsumerConfig(
bootstrap_servers=BOOTSTRAP_SERVERS,
# ...
)
observer = KafkaObserver(consumer_config)
# Get consumer groups
observer.get_consumer_groups()
# Get consumer groups with offsets
observer.get_consumer_groups_offsets()
from confluent_kafka import Consumer
my_consumer = Consumer({"bootstrap.servers": BOOTSTRAP_SERVERS, "group.id": "your_consumer_group"})
# Get offset status
observer.get_offset_status(my_consumer, "some_topic")
```
The observer has also the possibility to plot important water mark and offset information for a certain topic:
```python
# Plot offset status
observer.plot_offset_status(my_consumer, "some_topic")
## Contact
For any inquiries or questions, feel free [reach out](https://qurix.tech/about_us.html).
Raw data
{
"_id": null,
"home_page": "https://github.com/qurixtechnology/qurix-kafka-observer.git",
"name": "qurix-kafka-observer",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10, <4",
"maintainer_email": "",
"keywords": "python",
"author": "qurix Technology",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/59/df/3e009ab9e5f4f11a0b2d1189dfe6927960682636d6e59309f2c564e969d3/qurix-kafka-observer-0.2.0.tar.gz",
"platform": null,
"description": "# KafkaObserver\n\n## What is it?\n\nThe KafkaObserver package provides a set of tools for observing and retrieving information about Kafka topics and consumer groups. It facilitates monitoring and managing Kafka infrastructure by offering insights into offset statuses, watermarks, and retention policies.\n\n## Main Features\n\n- **Watermark Information:** Retrieve information about the watermark (low and high offsets) for a specified topic.\n\n- **Retention Policy Information:** Retrieve information about the retention policy (time and size) for a specified topic.\n\n- **Topic Information:** Obtain details about Kafka topics, including basic information and extended details like watermark information and retention policies.\n\n- **Consumer Groups Information:** Retrieve information about existing consumer groups.\n\n- **Offset Status:** Get the offset status (current, committed, etc.) for a specific consumer and topic. Plot the offset status for visualization.\n\n\n## Requirements\n\n- `confluent-kafka`\n- `openpyxl`\n\nYou can install these dependencies manually or use the provided `requirement.txt` file in the repository.\n\n## Installation\n\n1. Create a New Virtual Environment (named `.venv` in this case):\n\n```bash\npython3 -m venv venv\n```\n\n2. Activate the Virtual Environment:\n\n```bash\nsource venv/bin/activate\n```\n\n3. Install the Package:\n\nTo install the `qurix-kafka-observer` package, use `pip`:\n\n```bash\npip install qurix-kafka-observer\n```\n\n## Usage\n\n### Kafka Observer\n\nTo use the KafkaObserver class:\n\n```python\nfrom qurix.kafka.observer import KafkaObserver\nfrom qurix.kafka.entities.config import ConsumerConfig\n\n# Create a ConsumerConfig object with your Kafka settings\nBOOTSTRAP_SERVERS = \"localhost:9092\"\nconsumer_config = ConsumerConfig(\n bootstrap_servers=BOOTSTRAP_SERVERS,\n # ...\n)\n\nobserver = KafkaObserver(consumer_config)\n\n# Get consumer groups\nobserver.get_consumer_groups()\n\n# Get consumer groups with offsets\nobserver.get_consumer_groups_offsets()\n\nfrom confluent_kafka import Consumer\n\nmy_consumer = Consumer({\"bootstrap.servers\": BOOTSTRAP_SERVERS, \"group.id\": \"your_consumer_group\"})\n\n# Get offset status\nobserver.get_offset_status(my_consumer, \"some_topic\")\n```\n\nThe observer has also the possibility to plot important water mark and offset information for a certain topic:\n\n```python\n# Plot offset status\nobserver.plot_offset_status(my_consumer, \"some_topic\")\n\n\n## Contact\n\nFor any inquiries or questions, feel free [reach out](https://qurix.tech/about_us.html).\n",
"bugtrack_url": null,
"license": "",
"summary": "qurix package for observing kafka messages",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/qurixtechnology/qurix-kafka-observer.git"
},
"split_keywords": [
"python"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f757a67a62cb61bfd8d755999a093b6f3ce3e26a72d50b7bee53c455caaee127",
"md5": "85a41dff29691572b35a16d9256ef6da",
"sha256": "89f024821113ca554f5547a3e55f5f6fc3949637d909ed4027f8bc0b6c98de71"
},
"downloads": -1,
"filename": "qurix_kafka_observer-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "85a41dff29691572b35a16d9256ef6da",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10, <4",
"size": 10957,
"upload_time": "2023-11-22T12:39:47",
"upload_time_iso_8601": "2023-11-22T12:39:47.277949Z",
"url": "https://files.pythonhosted.org/packages/f7/57/a67a62cb61bfd8d755999a093b6f3ce3e26a72d50b7bee53c455caaee127/qurix_kafka_observer-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "59df3e009ab9e5f4f11a0b2d1189dfe6927960682636d6e59309f2c564e969d3",
"md5": "c9ad58460bda63eb1706985a42f5c220",
"sha256": "03e04a2518bc12798b2f8d2e1df618fff069954756fe03e13ad59670c6471d98"
},
"downloads": -1,
"filename": "qurix-kafka-observer-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "c9ad58460bda63eb1706985a42f5c220",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10, <4",
"size": 9094,
"upload_time": "2023-11-22T12:39:48",
"upload_time_iso_8601": "2023-11-22T12:39:48.621169Z",
"url": "https://files.pythonhosted.org/packages/59/df/3e009ab9e5f4f11a0b2d1189dfe6927960682636d6e59309f2c564e969d3/qurix-kafka-observer-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-11-22 12:39:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "qurixtechnology",
"github_project": "qurix-kafka-observer",
"github_not_found": true,
"lcname": "qurix-kafka-observer"
}
JSON
