.. _AiiDA: https://www.aiida.net
.. _rmq tutorial: https://www.rabbitmq.com/getstarted.html
.. _documentation: https://kiwipy.readthedocs.io/en/latest/index.html
kiwiPy
======
.. image:: docs/source/_static/logo.svg
:height: 64px
:width: 64px
:alt: kiwiPy
.. image:: https://codecov.io/gh/aiidateam/kiwipy/branch/master/graph/badge.svg
:target: https://codecov.io/gh/aiidateam/kiwipy
:alt: Coveralls
.. image:: https://github.com/aiidateam/kiwipy/workflows/continuous-integration/badge.svg
:target: https://github.com/aiidateam/kiwipy/actions?query=workflow%3Acontinuous-integration
:alt: Github Actions
.. image:: https://img.shields.io/pypi/v/kiwipy.svg
:target: https://pypi.python.org/pypi/kiwipy/
:alt: Latest Version
.. image:: https://img.shields.io/pypi/pyversions/kiwipy.svg
:target: https://pypi.python.org/pypi/kiwipy/
.. image:: https://img.shields.io/pypi/l/kiwipy.svg
:target: https://pypi.python.org/pypi/kiwipy/
.. image:: https://joss.theoj.org/papers/10.21105/joss.02351/status.svg
:target: https://doi.org/10.21105/joss.02351
`kiwiPy`_ is a library that makes remote messaging using RabbitMQ (and possibly other message brokers) EASY. It was
designed to support high-throughput workflows in big-data and computational science settings and is currently used
by `AiiDA`_ for computational materials research around the world. That said, kiwiPy is entirely general and can
be used anywhere where high-throughput and robust messaging are needed.
Here's what you get:
* RPC
* Broadcast (with filters)
* Task queue messages
Let's dive in, with some examples taken from the `rmq tutorial`_. To see more detail head over to the `documentation`_.
RPC
---
The client:
.. code-block:: python
import kiwipy
with kiwipy.connect('amqp://localhost') as comm:
# Send an RPC message
print(" [x] Requesting fib(30)")
response = comm.rpc_send('fib', 30).result()
print((" [.] Got %r" % response))
`(rmq_rpc_client.py source) <https://raw.githubusercontent.com/aiidateam/kiwipy/master/examples/rmq_rpc_client.py>`_
The server:
.. code-block:: python
import threading
import kiwipy
def fib(comm, num):
if num == 0:
return 0
if num == 1:
return 1
return fib(comm, num - 1) + fib(comm, num - 2)
with kiwipy.connect('amqp://127.0.0.1') as comm:
# Register an RPC subscriber with the name 'fib'
comm.add_rpc_subscriber(fib, 'fib')
# Now wait indefinitely for fibonacci calls
threading.Event().wait()
`(rmq_rpc_server.py source) <https://raw.githubusercontent.com/aiidateam/kiwipy/master/examples/rmq_rpc_server.py>`_
Worker
------
Create a new task:
.. code-block:: python
import sys
import kiwipy
message = ' '.join(sys.argv[1:]) or "Hello World!"
with rmq.connect('amqp://localhost') as comm:
comm.task_send(message)
`(rmq_new_task.py source) <https://raw.githubusercontent.com/aiidateam/kiwipy/master/examples/rmq_new_task.py>`_
And the worker:
.. code-block:: python
import time
import threading
import kiwipy
print(' [*] Waiting for messages. To exit press CTRL+C')
def callback(_comm, task):
print((" [x] Received %r" % task))
time.sleep(task.count(b'.'))
print(" [x] Done")
try:
with kiwipy.connect('amqp://localhost') as comm:
comm.add_task_subscriber(callback)
threading.Event().wait()
except KeyboardInterrupt:
pass
`(rmq_worker.py source) <https://raw.githubusercontent.com/aiidateam/kiwipy/master/examples/rmq_worker.py>`_
Citing
======
If you use kiwiPy directly or indirectly (e.g. by using `AiiDA`_) then please cite:
Uhrin, M., & Huber, S. P. (2020). kiwiPy : Robust , high-volume , messaging for big-data and computational science workflows, 5, 4–6. http://doi.org/10.21105/joss.02351
This helps us to keep making community software.
Versioning
==========
This software follows `Semantic Versioning`_
Contributing
============
Want a new feature? Found a bug? Want to contribute more documentation or a translation perhaps?
Help is always welcome, get started with the `contributing guide <https://github.com/aiidateam/kiwipy/wiki/Contributing>`__.
.. _Semantic Versioning: http://semver.org/
Development
===========
This package utilises `tox <https://tox.readthedocs.io>`__ for unit test automation, and `pre-commit <https://pre-commit.com>`__ for code style formatting and test automation.
To install these development dependencies:
.. code-block:: bash
pip install tox pre-commit
To run the unit tests:
.. code-block:: bash
tox
For the ``rmq`` tests you will require a running instance of RabbitMQ.
One way to achieve this is using Docker and launching ``test/rmq/docker-compose.yml``.
To run the pre-commit tests:
.. code-block:: bash
pre-commit run --all
To build the documentation:
.. code-block:: bash
tox -e docs-clean
Changes should be submitted as Pull Requests (PRs) to the ``develop`` branch.
Publishing Releases
===================
1. Create a release PR/commit to the ``develop`` branch, updating ``kiwipy/version.py`` and ``CHANGELOG.md``.
2. Fast-forward merge `develop` into the `master` branch
3. Create a release on GitHub (https://github.com/aiidateam/kiwipy/releases/new), pointing to the release commit on `master`, named ``v.X.Y.Z`` (identical to version in ``kiwipy/version.py``)
4. This will trigger the ``continuous-deployment`` GitHub workflow which, if all tests pass, will publish the package to PyPi. Check this has successfully completed in the GitHub Actions tab (https://github.com/aiidateam/kiwipy/actions).
(if the release fails, delete the release and tag)
Raw data
{
"_id": null,
"home_page": null,
"name": "kiwipy",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "ommunication,messaging,rpc,broadcast",
"author": "Sebastiaan P. Huber, Jason Yu, Sonia Collaud",
"author_email": "Martin Uhrin <martin.uhrin@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/0e/ae/3831e1562f1273195e686796690cba656e8937f414d2efcf47466ebfd949/kiwipy-0.8.4.tar.gz",
"platform": null,
"description": ".. _AiiDA: https://www.aiida.net\n.. _rmq tutorial: https://www.rabbitmq.com/getstarted.html\n.. _documentation: https://kiwipy.readthedocs.io/en/latest/index.html\n\n\nkiwiPy\n======\n\n.. image:: docs/source/_static/logo.svg\n :height: 64px\n :width: 64px\n :alt: kiwiPy\n\n.. image:: https://codecov.io/gh/aiidateam/kiwipy/branch/master/graph/badge.svg\n :target: https://codecov.io/gh/aiidateam/kiwipy\n :alt: Coveralls\n\n.. image:: https://github.com/aiidateam/kiwipy/workflows/continuous-integration/badge.svg\n :target: https://github.com/aiidateam/kiwipy/actions?query=workflow%3Acontinuous-integration\n :alt: Github Actions\n\n.. image:: https://img.shields.io/pypi/v/kiwipy.svg\n :target: https://pypi.python.org/pypi/kiwipy/\n :alt: Latest Version\n\n.. image:: https://img.shields.io/pypi/pyversions/kiwipy.svg\n :target: https://pypi.python.org/pypi/kiwipy/\n\n.. image:: https://img.shields.io/pypi/l/kiwipy.svg\n :target: https://pypi.python.org/pypi/kiwipy/\n\n.. image:: https://joss.theoj.org/papers/10.21105/joss.02351/status.svg\n :target: https://doi.org/10.21105/joss.02351\n\n\n\n`kiwiPy`_ is a library that makes remote messaging using RabbitMQ (and possibly other message brokers) EASY. It was\ndesigned to support high-throughput workflows in big-data and computational science settings and is currently used\nby `AiiDA`_ for computational materials research around the world. That said, kiwiPy is entirely general and can\nbe used anywhere where high-throughput and robust messaging are needed.\n\nHere's what you get:\n\n* RPC\n* Broadcast (with filters)\n* Task queue messages\n\nLet's dive in, with some examples taken from the `rmq tutorial`_. To see more detail head over to the `documentation`_.\n\nRPC\n---\n\nThe client:\n\n.. code-block:: python\n\n import kiwipy\n\n with kiwipy.connect('amqp://localhost') as comm:\n # Send an RPC message\n print(\" [x] Requesting fib(30)\")\n response = comm.rpc_send('fib', 30).result()\n print((\" [.] Got %r\" % response))\n\n`(rmq_rpc_client.py source) <https://raw.githubusercontent.com/aiidateam/kiwipy/master/examples/rmq_rpc_client.py>`_\n\n\nThe server:\n\n.. code-block:: python\n\n import threading\n import kiwipy\n\n def fib(comm, num):\n if num == 0:\n return 0\n if num == 1:\n return 1\n\n return fib(comm, num - 1) + fib(comm, num - 2)\n\n with kiwipy.connect('amqp://127.0.0.1') as comm:\n # Register an RPC subscriber with the name 'fib'\n comm.add_rpc_subscriber(fib, 'fib')\n # Now wait indefinitely for fibonacci calls\n threading.Event().wait()\n\n`(rmq_rpc_server.py source) <https://raw.githubusercontent.com/aiidateam/kiwipy/master/examples/rmq_rpc_server.py>`_\n\n\nWorker\n------\n\nCreate a new task:\n\n.. code-block:: python\n\n import sys\n import kiwipy\n\n message = ' '.join(sys.argv[1:]) or \"Hello World!\"\n\n with rmq.connect('amqp://localhost') as comm:\n comm.task_send(message)\n\n`(rmq_new_task.py source) <https://raw.githubusercontent.com/aiidateam/kiwipy/master/examples/rmq_new_task.py>`_\n\n\nAnd the worker:\n\n.. code-block:: python\n\n import time\n import threading\n import kiwipy\n\n print(' [*] Waiting for messages. To exit press CTRL+C')\n\n\n def callback(_comm, task):\n print((\" [x] Received %r\" % task))\n time.sleep(task.count(b'.'))\n print(\" [x] Done\")\n\n\n try:\n with kiwipy.connect('amqp://localhost') as comm:\n comm.add_task_subscriber(callback)\n threading.Event().wait()\n except KeyboardInterrupt:\n pass\n\n`(rmq_worker.py source) <https://raw.githubusercontent.com/aiidateam/kiwipy/master/examples/rmq_worker.py>`_\n\nCiting\n======\n\nIf you use kiwiPy directly or indirectly (e.g. by using `AiiDA`_) then please cite:\n\nUhrin, M., & Huber, S. P. (2020). kiwiPy : Robust , high-volume , messaging for big-data and computational science workflows, 5, 4\u20136. http://doi.org/10.21105/joss.02351\n\nThis helps us to keep making community software.\n\nVersioning\n==========\n\nThis software follows `Semantic Versioning`_\n\nContributing\n============\n\nWant a new feature? Found a bug? Want to contribute more documentation or a translation perhaps?\n\nHelp is always welcome, get started with the `contributing guide <https://github.com/aiidateam/kiwipy/wiki/Contributing>`__.\n\n.. _Semantic Versioning: http://semver.org/\n\nDevelopment\n===========\n\nThis package utilises `tox <https://tox.readthedocs.io>`__ for unit test automation, and `pre-commit <https://pre-commit.com>`__ for code style formatting and test automation.\n\nTo install these development dependencies:\n\n.. code-block:: bash\n\n pip install tox pre-commit\n\nTo run the unit tests:\n\n.. code-block:: bash\n\n tox\n\nFor the ``rmq`` tests you will require a running instance of RabbitMQ.\nOne way to achieve this is using Docker and launching ``test/rmq/docker-compose.yml``.\n\nTo run the pre-commit tests:\n\n.. code-block:: bash\n\n pre-commit run --all\n\nTo build the documentation:\n\n.. code-block:: bash\n\n tox -e docs-clean\n\nChanges should be submitted as Pull Requests (PRs) to the ``develop`` branch.\n\nPublishing Releases\n===================\n\n1. Create a release PR/commit to the ``develop`` branch, updating ``kiwipy/version.py`` and ``CHANGELOG.md``.\n2. Fast-forward merge `develop` into the `master` branch\n3. Create a release on GitHub (https://github.com/aiidateam/kiwipy/releases/new), pointing to the release commit on `master`, named ``v.X.Y.Z`` (identical to version in ``kiwipy/version.py``)\n4. This will trigger the ``continuous-deployment`` GitHub workflow which, if all tests pass, will publish the package to PyPi. Check this has successfully completed in the GitHub Actions tab (https://github.com/aiidateam/kiwipy/actions).\n\n(if the release fails, delete the release and tag)\n",
"bugtrack_url": null,
"license": null,
"summary": "Robust, high-volume, message based communication made easy.",
"version": "0.8.4",
"project_urls": {
"Documentation": "https://kiwipy.readthedocs.io",
"Home": "https://github.com/aiidateam/kiwipy",
"Source": "https://github.com/aiidateam/kiwipy"
},
"split_keywords": [
"ommunication",
"messaging",
"rpc",
"broadcast"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "f7fead668377767dcc44d94258ac3ccbedbd55645cba893da842ed820c4ed937",
"md5": "61b68ac8c17b602dbc128d6fb0297021",
"sha256": "f2b260f1595689a0d835f1e2bf5373714001203d17c4d94e56ec299e793da495"
},
"downloads": -1,
"filename": "kiwipy-0.8.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "61b68ac8c17b602dbc128d6fb0297021",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 41753,
"upload_time": "2024-02-02T14:32:52",
"upload_time_iso_8601": "2024-02-02T14:32:52.794300Z",
"url": "https://files.pythonhosted.org/packages/f7/fe/ad668377767dcc44d94258ac3ccbedbd55645cba893da842ed820c4ed937/kiwipy-0.8.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "0eae3831e1562f1273195e686796690cba656e8937f414d2efcf47466ebfd949",
"md5": "f499a508a53f922461c09cdfd26a1e0a",
"sha256": "625830fa07faac2c2a307bd3aa1caedac6596f37f9b9aba31b7f84c3d9c9f57a"
},
"downloads": -1,
"filename": "kiwipy-0.8.4.tar.gz",
"has_sig": false,
"md5_digest": "f499a508a53f922461c09cdfd26a1e0a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 40782,
"upload_time": "2024-02-02T14:32:54",
"upload_time_iso_8601": "2024-02-02T14:32:54.589669Z",
"url": "https://files.pythonhosted.org/packages/0e/ae/3831e1562f1273195e686796690cba656e8937f414d2efcf47466ebfd949/kiwipy-0.8.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-02 14:32:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "aiidateam",
"github_project": "kiwipy",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "kiwipy"
}
JSON
