===============================
bluesky-kafka
===============================
.. image:: https://github.com/bluesky/bluesky-kafka/actions/workflows/tests.yml/badge.svg
:target: https://github.com/bluesky/bluesky-kafka/actions
.. image:: https://img.shields.io/pypi/v/bluesky-kafka.svg
:target: https://pypi.python.org/pypi/bluesky-kafka
Kafka integration for bluesky.
* Free software: 3-clause BSD license
Features
--------
* BlueskyConsumer
* MongoConsumer
* Publisher
* RemoteDispatcher
Test
----
Install docker and docker-compose.
Start a Kafka server:
::
$ cd bluesky-kafka/scripts
$ sudo docker-compose -f bitnami-kafka-docker-compose.yml up
Run tests:
::
$ cd bluesky-kafka
$ pytest
Optionally increase logging output to the console by specifying a logging level:
::
$ pytest --log-cli-level=DEBUG
Run a Mongo Consumer Group
--------------------------
Create a conda environment:
::
$ conda create -n consumers python=3.8
$ conda activate consumers
Install packages:
::
$ pip install bluesky-kafka supervisor
Setup environment variables:
mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/
bootstrap_servers: comma-separated list of brokers.
::
$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3
$ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"
Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.
Start the consumer processes:
::
$ supervisord -c bluesky_kafka/supervisor/supervisord.conf
Monitor the consumer processes:
::
$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf
Raw data
{
"_id": null,
"home_page": "https://github.com/bluesky/bluesky-kafka",
"name": "bluesky-kafka",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "",
"author": "Brookhaven National Lab",
"author_email": "jlynch@bnl.gov",
"download_url": "https://files.pythonhosted.org/packages/3f/3e/fd7381a7e712238a0fa0bc8a0feb5c2febd7b9422e4b74951a541affb5e0/bluesky-kafka-0.10.0.tar.gz",
"platform": null,
"description": "===============================\nbluesky-kafka\n===============================\n\n.. image:: https://github.com/bluesky/bluesky-kafka/actions/workflows/tests.yml/badge.svg\n :target: https://github.com/bluesky/bluesky-kafka/actions\n\n.. image:: https://img.shields.io/pypi/v/bluesky-kafka.svg\n :target: https://pypi.python.org/pypi/bluesky-kafka\n\n\nKafka integration for bluesky.\n\n* Free software: 3-clause BSD license\n\nFeatures\n--------\n\n* BlueskyConsumer\n* MongoConsumer\n* Publisher\n* RemoteDispatcher\n\nTest\n----\n\nInstall docker and docker-compose.\n\nStart a Kafka server:\n\n::\n\n $ cd bluesky-kafka/scripts\n $ sudo docker-compose -f bitnami-kafka-docker-compose.yml up\n\nRun tests:\n\n::\n\n $ cd bluesky-kafka\n $ pytest\n\nOptionally increase logging output to the console by specifying a logging level:\n\n::\n\n $ pytest --log-cli-level=DEBUG\n\nRun a Mongo Consumer Group\n--------------------------\n\nCreate a conda environment:\n\n::\n\n $ conda create -n consumers python=3.8\n $ conda activate consumers\n\nInstall packages:\n\n::\n\n $ pip install bluesky-kafka supervisor\n\nSetup environment variables:\nmongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/\nbootstrap_servers: comma-separated list of brokers.\n\n::\n\n $ export BLUESKY_MONGO_URI=\"mongodb://username:password@machine1:port1,machine2:port2,machine3:port3\n $ export KAFKA_BOOTSTRAP_SERVERS=\"machine1:9092, machine2:9092, machine3:9092\"\n\nUpdate the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.\n\nStart the consumer processes:\n\n::\n\n $ supervisord -c bluesky_kafka/supervisor/supervisord.conf\n\nMonitor the consumer processes:\n\n::\n\n $ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf\n",
"bugtrack_url": null,
"license": "BSD (3-clause)",
"summary": "Kafka integration for bluesky.",
"version": "0.10.0",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b1d3a3e8393cdf6e40e94b3a91ab8afdb010769829b520b2fbfdc4697f31a1c3",
"md5": "d3ab76a43bf22600c3f1be131da8d240",
"sha256": "8b82605d61213d9484e68c20a63d03f24246a775b3729fdc3fcb7259ccac5178"
},
"downloads": -1,
"filename": "bluesky_kafka-0.10.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d3ab76a43bf22600c3f1be131da8d240",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.6",
"size": 37043,
"upload_time": "2023-02-17T19:06:47",
"upload_time_iso_8601": "2023-02-17T19:06:47.957686Z",
"url": "https://files.pythonhosted.org/packages/b1/d3/a3e8393cdf6e40e94b3a91ab8afdb010769829b520b2fbfdc4697f31a1c3/bluesky_kafka-0.10.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3f3efd7381a7e712238a0fa0bc8a0feb5c2febd7b9422e4b74951a541affb5e0",
"md5": "c3f56564dfa31becbe5a6e6bb4414276",
"sha256": "5aab55d4d47261327d0ab6c7f76d2b079f93f625917ee118dd9e638b94f1929f"
},
"downloads": -1,
"filename": "bluesky-kafka-0.10.0.tar.gz",
"has_sig": false,
"md5_digest": "c3f56564dfa31becbe5a6e6bb4414276",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 44960,
"upload_time": "2023-02-17T19:06:49",
"upload_time_iso_8601": "2023-02-17T19:06:49.700331Z",
"url": "https://files.pythonhosted.org/packages/3f/3e/fd7381a7e712238a0fa0bc8a0feb5c2febd7b9422e4b74951a541affb5e0/bluesky-kafka-0.10.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-02-17 19:06:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "bluesky",
"github_project": "bluesky-kafka",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "bluesky",
"specs": []
},
{
"name": "confluent-kafka",
"specs": []
},
{
"name": "msgpack",
"specs": [
[
">=",
"1.0.0"
]
]
},
{
"name": "msgpack-numpy",
"specs": []
},
{
"name": "suitcase-mongo",
"specs": []
}
],
"lcname": "bluesky-kafka"
}
JSON
