========================================
kafka-overwatch
========================================
What started as a simple CLI/Service to evaluate Kafka cluster topics which have no activity,
ended up being a somewhat comprehensive way to monitor Kafka cluster activities.
Takes a configuration file as input, where you can list one or multiple cluster(s) you wish to monitor.
After a set period of time, it can produce a report (to local disk or AWS S3) with the list of topics that haven't seen any activity.
It also exposes metrics via a prometheus endpoint.
Usage
========
.. code-block::
kafka-overwatch -c config.local.yaml
Features
==========
* Supports evaluating multiple Kafka clusters at once
* Generates a report on topic usage based on topic watermarks offsets (store local or to S3)
* Generates commands script to re-create all the topics in case of DR (store local or to S3)
* Exposes metrics via prometheus
* Topics count
* Partitions count
* Number of new messages (measured with topic offsets)
* AWS Secret integration for client config values
* Schema Registry integration
* Scan schema registries, map 1 to many kafka clusters
* Backup of the schemas, and CLI to restore schemas to existing/new registry.
Upcoming
----------
* Multi-nodes awareness (split the load with multiple nodes)
* `cfn-kafka-admin`_ output format
* topic messages meta-data analysis (i.e are messages compressed?)
* scripts to perform cleanup
* Recommendations generated from/based on models
* Conduktor Gateway vClusters auto-discovery
Configuration
===============
Whilst a much more comprehensive documentation is yet to be written, please look at ``kafka_overwatch/specs/config.json``
which is used with `jsonschema`_ to perform validation of the input.
Return codes
=============
0 - all successful.
1 - error during execution
2 - error importing configuration.
Misc
=====
Thanks
-------
Thanks to the Apache Kafka OpenSource community for their continuous efforts in making the eco-system great.
Thanks to the `NASA`_ for having a public cluster to run tests with
Note
-----
Inspired by `kafka-idle-topics`_, yet completely re-written to be a continuous monitoring of the topics,
similar to `cruise-control`_.
.. _EMF: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html
.. _kafka-idle-topics: https://github.com/abraham-leal/kafka-idle-topics
.. _cfn-kafka-admin: https://github.com/compose-x/cfn-kafka-admin
.. _cruise-control: https://github.com/linkedin/cruise-control
.. _jsonschema: https://pypi.org/project/jsonschema/
.. _NASA: https://www.nasa.gov/
Status
=======
Images build status
|BUILD|
Docs build status
|DOCS_BUILD|
.. |BUILD| image:: https://codebuild.eu-west-1.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiU3RHQnZ2eFpnQTlOSmU2MUM3NDB5NW9uMDY2TS9DZXBWZ2hmejdoK2xJRStHK2Fhd3FkS1FoQjJOSTcvYjVBNkFTTW5kVDNZK0NqZEthU3gveFpOVEljPSIsIml2UGFyYW1ldGVyU3BlYyI6IjlUbE0vNmpPQU92U1o0SmkiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=main
.. |DOCS_BUILD| image:: https://codebuild.eu-west-1.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiSVNBZkVSUkx1NHhtamlqSEJqempIdHd2aVNqV2RkTTFVYlphUzJ2ekprOVU4ODZ4cUNWcTNVSkRVM2ovcGFyak5NTTNJZ1Vra2ErSzVOdi84TkVLOUp3PSIsIml2UGFyYW1ldGVyU3BlYyI6IjAvK25MSmNPcjNScVpwdTQiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=main
Raw data
{
"_id": null,
"home_page": null,
"name": "kafka-overwatch",
"maintainer": "John \"Preston\" Mille",
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": "john@ews-network.net",
"keywords": "kafka, monitoring, reporting, usage-analysis, cost-savings",
"author": "John \"Preston\" Mille",
"author_email": "john@ews-network.net",
"download_url": "https://files.pythonhosted.org/packages/c4/05/f211b2bf0745b9ec9f48a96ba23c7e17021dcc570c5527882a2661377695/kafka_overwatch-0.5.0.post0.tar.gz",
"platform": null,
"description": "========================================\nkafka-overwatch\n========================================\n\nWhat started as a simple CLI/Service to evaluate Kafka cluster topics which have no activity,\nended up being a somewhat comprehensive way to monitor Kafka cluster activities.\n\nTakes a configuration file as input, where you can list one or multiple cluster(s) you wish to monitor.\n\nAfter a set period of time, it can produce a report (to local disk or AWS S3) with the list of topics that haven't seen any activity.\nIt also exposes metrics via a prometheus endpoint.\n\nUsage\n========\n\n.. code-block::\n\n kafka-overwatch -c config.local.yaml\n\nFeatures\n==========\n\n* Supports evaluating multiple Kafka clusters at once\n* Generates a report on topic usage based on topic watermarks offsets (store local or to S3)\n* Generates commands script to re-create all the topics in case of DR (store local or to S3)\n\n* Exposes metrics via prometheus\n\n * Topics count\n * Partitions count\n * Number of new messages (measured with topic offsets)\n\n* AWS Secret integration for client config values\n* Schema Registry integration\n\n * Scan schema registries, map 1 to many kafka clusters\n * Backup of the schemas, and CLI to restore schemas to existing/new registry.\n\nUpcoming\n----------\n\n* Multi-nodes awareness (split the load with multiple nodes)\n* `cfn-kafka-admin`_ output format\n* topic messages meta-data analysis (i.e are messages compressed?)\n* scripts to perform cleanup\n* Recommendations generated from/based on models\n* Conduktor Gateway vClusters auto-discovery\n\n\nConfiguration\n===============\n\nWhilst a much more comprehensive documentation is yet to be written, please look at ``kafka_overwatch/specs/config.json``\nwhich is used with `jsonschema`_ to perform validation of the input.\n\nReturn codes\n=============\n\n0 - all successful.\n1 - error during execution\n2 - error importing configuration.\n\nMisc\n=====\n\nThanks\n-------\n\nThanks to the Apache Kafka OpenSource community for their continuous efforts in making the eco-system great.\nThanks to the `NASA`_ for having a public cluster to run tests with\n\nNote\n-----\n\nInspired by `kafka-idle-topics`_, yet completely re-written to be a continuous monitoring of the topics,\nsimilar to `cruise-control`_.\n\n.. _EMF: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html\n.. _kafka-idle-topics: https://github.com/abraham-leal/kafka-idle-topics\n.. _cfn-kafka-admin: https://github.com/compose-x/cfn-kafka-admin\n.. _cruise-control: https://github.com/linkedin/cruise-control\n.. _jsonschema: https://pypi.org/project/jsonschema/\n.. _NASA: https://www.nasa.gov/\n\n\nStatus\n=======\n\nImages build status\n\n|BUILD|\n\nDocs build status\n\n|DOCS_BUILD|\n\n\n.. |BUILD| image:: https://codebuild.eu-west-1.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiU3RHQnZ2eFpnQTlOSmU2MUM3NDB5NW9uMDY2TS9DZXBWZ2hmejdoK2xJRStHK2Fhd3FkS1FoQjJOSTcvYjVBNkFTTW5kVDNZK0NqZEthU3gveFpOVEljPSIsIml2UGFyYW1ldGVyU3BlYyI6IjlUbE0vNmpPQU92U1o0SmkiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=main\n\n.. |DOCS_BUILD| image:: https://codebuild.eu-west-1.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiSVNBZkVSUkx1NHhtamlqSEJqempIdHd2aVNqV2RkTTFVYlphUzJ2ekprOVU4ODZ4cUNWcTNVSkRVM2ovcGFyak5NTTNJZ1Vra2ErSzVOdi84TkVLOUp3PSIsIml2UGFyYW1ldGVyU3BlYyI6IjAvK25MSmNPcjNScVpwdTQiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=main\n",
"bugtrack_url": null,
"license": "MPL-2.0",
"summary": "Continuously monitors Kafka cluster topics & consumer groups",
"version": "0.5.0.post0",
"project_urls": {
"Bug Tracker": "https://github.com/JohnPreston/kafka-overwatch/issues"
},
"split_keywords": [
"kafka",
" monitoring",
" reporting",
" usage-analysis",
" cost-savings"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e712ed40e1661d8e61e3256a45321c2d00c86c5ae327328c3ba8776b522c6503",
"md5": "bcc49c6f1203f6a0004c234861c2c435",
"sha256": "9df30ff754b541b1c700b34ea4a14cfecfdcfb2b4886bca7f9b78a3ee191268f"
},
"downloads": -1,
"filename": "kafka_overwatch-0.5.0.post0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "bcc49c6f1203f6a0004c234861c2c435",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 69517,
"upload_time": "2024-05-23T19:41:44",
"upload_time_iso_8601": "2024-05-23T19:41:44.848718Z",
"url": "https://files.pythonhosted.org/packages/e7/12/ed40e1661d8e61e3256a45321c2d00c86c5ae327328c3ba8776b522c6503/kafka_overwatch-0.5.0.post0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c405f211b2bf0745b9ec9f48a96ba23c7e17021dcc570c5527882a2661377695",
"md5": "9630d384e8d482822ce4ed05aa402fb4",
"sha256": "05be02ab6616a43c345ced8796f6ef12580b4a93d23ef8df490393a753ed7e89"
},
"downloads": -1,
"filename": "kafka_overwatch-0.5.0.post0.tar.gz",
"has_sig": false,
"md5_digest": "9630d384e8d482822ce4ed05aa402fb4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 44901,
"upload_time": "2024-05-23T19:41:46",
"upload_time_iso_8601": "2024-05-23T19:41:46.312652Z",
"url": "https://files.pythonhosted.org/packages/c4/05/f211b2bf0745b9ec9f48a96ba23c7e17021dcc570c5527882a2661377695/kafka_overwatch-0.5.0.post0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-23 19:41:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "JohnPreston",
"github_project": "kafka-overwatch",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "kafka-overwatch"
}
JSON
