<h1 align="center">
<img src=".github/images/intility.png" width="124px"/><br/>
Metroid
</h1>
<p align="center">
<em>Subscribe, act, publish.</em>
</p>
<p align="center">
<a href="https://python.org">
<img src="https://img.shields.io/badge/python-v3.10+-blue.svg" alt="Python version">
</a>
<a href="https://djangoproject.com">
<img src="https://img.shields.io/badge/django-4.2+%20-blue.svg" alt="Django version">
</a>
<a href="https://docs.celeryproject.org/en/stable/">
<img src="https://img.shields.io/badge/celery-5.3.0+%20-blue.svg" alt="Celery version">
</a>
<a href="https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/servicebus/azure-servicebus">
<img src="https://img.shields.io/badge/azure--servicebus-7.10.0+%20-blue.svg" alt="ServiceBus version">
</a>
<a href="https://github.com/snok/django-guid/">
<img src="https://img.shields.io/badge/django--guid-3.2.0+-blue.svg" alt="Django GUID version">
</a>
</p>
<p align="center">
<a href="https://codecov.io/gh/intility/metroid">
<img src="https://codecov.io/gh/intility/metroid/branch/main/graph/badge.svg" alt="Codecov">
</a>
<a href="https://github.com/pre-commit/pre-commit">
<img src="https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white" alt="Pre-commit">
</a>
<a href="https://github.com/psf/black">
<img src="https://img.shields.io/badge/code%20style-black-000000.svg" alt="Black">
</a>
<a href="http://mypy-lang.org">
<img src="http://www.mypy-lang.org/static/mypy_badge.svg" alt="mypy">
</a>
<a href="https://pycqa.github.io/isort/">
<img src="https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336" alt="isort">
</a>
</p>
# Metroid - Metro for Django
This app is intended to streamline integration with Metro for all Django+Celery users by:
* Asynchronous handling of subscriptions and messages with one command
* Execute Celery tasks based on message topics, defined in `settings.py`
* Retry failed tasks through your admin dashboard when using the `MetroidTask` base
## Overview
* `python` >= 3.10
* `django` >= 4.2 - For `asgiref`, settings
* `django-guid` >= 3.2.0 - Storing correlation IDs for failed tasks in the database, making debugging easy
* Choose one:
* `celery` >= 5.3.0 - Execute tasks based on a subject
* `django-rq` >= 2.4.1 - Execute tasks based on a subject
### Implementation
The `python manage.py metroid` app is fully asynchronous, and has no blocking code. It utilizes `Celery` to execute tasks.
It works by:
1. Going through all your configured subscriptions and start a new async connection for each one of them
2. Metro sends messages on the subscriptions
3. This app filters out messages matching subjects you have defined, and queues a celery task to execute
the function as specified for that subject
3.1. If no task is found for that subject, the message is marked as complete
4. The message is marked as complete after the Celery task has successfully been queued
5. If the task is failed, an entry is automatically created in your database
6. All failed tasks can be retried manually through the admin dashboard
### Configure and install this package
> **_Note_**
> For a complete example, have a look in `demoproj/settings.py`.
1. Create a `METROID` key in `settings.py` with all your subscriptions and handlers.
Example settings:
```python
METROID = {
'subscriptions': [
{
'topic_name': 'metro-demo',
'subscription_name': 'sub-metrodemo-metrodemoerfett',
'connection_string': config('CONNECTION_STRING_METRO_DEMO', None),
'handlers': [
{
'subject': 'MetroDemo/Type/GeekJokes',
'regex': False,
'handler_function': 'demoproj.demoapp.services.my_func'
}
],
},
],
'worker_type': 'celery', # default
}
```
The `handler_function` is defined by providing the full dotted path as a string. For example,`from demoproj.demoapp.services import my_func` is provided as `'demoproj.demoapp.services.my_func'`.
The handlers subject can be a regular expression or a string. If a regular expression is provided, the variable regex must be set to True. Example:
```python
'handlers': [{'subject': r'^MetroDemo/Type/.*$','regex':True,'handler_function': my_func}],
```
2. Configure `Django-GUID` by adding the app to your installed apps, to your middlewares and configuring logging
as described [here](https://github.com/snok/django-guid#configuration).
Make sure you enable the [`CeleryIntegration`](https://django-guid.readthedocs.io/en/latest/integrations.html#celery):
```python
from django_guid.integrations import CeleryIntegration
DJANGO_GUID = {
'INTEGRATIONS': [
CeleryIntegration(
use_django_logging=True,
log_parent=True,
)
],
}
```
#### Creating your own handler functions
Your functions will be called with keyword arguments for
`message`, `topic_name`, `subscription_name` and `subject`. You function should in other words
look something like this:
##### Celery
```python
@app.task(base=MetroidTask)
def my_func(*, message: dict, topic_name: str, subscription_name: str, subject: str) -> None:
```
##### rq
```python
def my_func(*, message: dict, topic_name: str, subscription_name: str, subject: str) -> None:
```
### Running the project
1. Ensure you have redis running:
```bash
docker-compose up
```
2. Run migrations
```bash
python manage.py migrate
```
3. Create an admin account
```bash
python manage.py createsuperuser
```
4. Start a worker:
```python
celery -A demoproj worker -l info
```
5. Run the subscriber:
```python
python manage.py metroid
```
6. Send messages to Metro. Example code can be found in [`demoproj/demoapp/services.py`](demoproj/demoapp/services.py)
7. Run the webserver:
```python
python manage.py runserver 8000
```
8. See failed messages under `http://localhost:8080/admin`
To contribute, please see [`CONTRIBUTING.md`](CONTRIBUTING.md)
Raw data
{
"_id": null,
"home_page": "https://github.com/intility/metroid",
"name": "metroid",
"maintainer": "Ali Arfan",
"docs_url": null,
"requires_python": ">=3.10,<4.0",
"maintainer_email": "ali.arfan@intility.no",
"keywords": "async,django,servicebus,task,celery,worker,rq",
"author": "Jonas Kr\u00fcger Svensson",
"author_email": "jonas.svensson@intility.no",
"download_url": "https://files.pythonhosted.org/packages/fa/e9/246f0d800ad901d77342e03d8ed0641600038f7e3e85214283b4a49a6be8/metroid-1.4.0.tar.gz",
"platform": null,
"description": "<h1 align=\"center\">\n <img src=\".github/images/intility.png\" width=\"124px\"/><br/>\n Metroid\n</h1>\n\n<p align=\"center\">\n <em>Subscribe, act, publish.</em>\n</p>\n<p align=\"center\">\n <a href=\"https://python.org\">\n <img src=\"https://img.shields.io/badge/python-v3.10+-blue.svg\" alt=\"Python version\">\n </a>\n <a href=\"https://djangoproject.com\">\n <img src=\"https://img.shields.io/badge/django-4.2+%20-blue.svg\" alt=\"Django version\">\n </a>\n <a href=\"https://docs.celeryproject.org/en/stable/\">\n <img src=\"https://img.shields.io/badge/celery-5.3.0+%20-blue.svg\" alt=\"Celery version\">\n </a>\n <a href=\"https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/servicebus/azure-servicebus\">\n <img src=\"https://img.shields.io/badge/azure--servicebus-7.10.0+%20-blue.svg\" alt=\"ServiceBus version\">\n </a>\n <a href=\"https://github.com/snok/django-guid/\">\n <img src=\"https://img.shields.io/badge/django--guid-3.2.0+-blue.svg\" alt=\"Django GUID version\">\n </a>\n</p>\n<p align=\"center\">\n <a href=\"https://codecov.io/gh/intility/metroid\">\n <img src=\"https://codecov.io/gh/intility/metroid/branch/main/graph/badge.svg\" alt=\"Codecov\">\n </a>\n <a href=\"https://github.com/pre-commit/pre-commit\">\n <img src=\"https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white\" alt=\"Pre-commit\">\n </a>\n <a href=\"https://github.com/psf/black\">\n <img src=\"https://img.shields.io/badge/code%20style-black-000000.svg\" alt=\"Black\">\n </a>\n <a href=\"http://mypy-lang.org\">\n <img src=\"http://www.mypy-lang.org/static/mypy_badge.svg\" alt=\"mypy\">\n </a>\n <a href=\"https://pycqa.github.io/isort/\">\n <img src=\"https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336\" alt=\"isort\">\n </a>\n</p>\n\n\n# Metroid - Metro for Django\n\nThis app is intended to streamline integration with Metro for all Django+Celery users by:\n\n* Asynchronous handling of subscriptions and messages with one command\n* Execute Celery tasks based on message topics, defined in `settings.py`\n* Retry failed tasks through your admin dashboard when using the `MetroidTask` base\n\n## Overview\n* `python` >= 3.10\n* `django` >= 4.2 - For `asgiref`, settings\n* `django-guid` >= 3.2.0 - Storing correlation IDs for failed tasks in the database, making debugging easy\n* Choose one:\n * `celery` >= 5.3.0 - Execute tasks based on a subject\n * `django-rq` >= 2.4.1 - Execute tasks based on a subject\n\n### Implementation\n\nThe `python manage.py metroid` app is fully asynchronous, and has no blocking code. It utilizes `Celery` to execute tasks.\n\nIt works by:\n1. Going through all your configured subscriptions and start a new async connection for each one of them\n2. Metro sends messages on the subscriptions\n3. This app filters out messages matching subjects you have defined, and queues a celery task to execute\n the function as specified for that subject \n 3.1. If no task is found for that subject, the message is marked as complete\n4. The message is marked as complete after the Celery task has successfully been queued\n5. If the task is failed, an entry is automatically created in your database\n6. All failed tasks can be retried manually through the admin dashboard\n\n\n### Configure and install this package\n\n\n> **_Note_**\n> For a complete example, have a look in `demoproj/settings.py`.\n\n1. Create a `METROID` key in `settings.py` with all your subscriptions and handlers.\nExample settings:\n```python\nMETROID = {\n 'subscriptions': [\n {\n 'topic_name': 'metro-demo',\n 'subscription_name': 'sub-metrodemo-metrodemoerfett',\n 'connection_string': config('CONNECTION_STRING_METRO_DEMO', None),\n 'handlers': [\n {\n 'subject': 'MetroDemo/Type/GeekJokes',\n 'regex': False,\n 'handler_function': 'demoproj.demoapp.services.my_func'\n }\n ],\n },\n ],\n 'worker_type': 'celery', # default\n}\n```\n\nThe `handler_function` is defined by providing the full dotted path as a string. For example,`from demoproj.demoapp.services import my_func` is provided as `'demoproj.demoapp.services.my_func'`.\n\nThe handlers subject can be a regular expression or a string. If a regular expression is provided, the variable regex must be set to True. Example:\n ```python\n'handlers': [{'subject': r'^MetroDemo/Type/.*$','regex':True,'handler_function': my_func}],\n ```\n\n\n\n2. Configure `Django-GUID` by adding the app to your installed apps, to your middlewares and configuring logging\nas described [here](https://github.com/snok/django-guid#configuration).\nMake sure you enable the [`CeleryIntegration`](https://django-guid.readthedocs.io/en/latest/integrations.html#celery):\n```python\nfrom django_guid.integrations import CeleryIntegration\n\nDJANGO_GUID = {\n 'INTEGRATIONS': [\n CeleryIntegration(\n use_django_logging=True,\n log_parent=True,\n )\n ],\n}\n```\n\n\n#### Creating your own handler functions\n\nYour functions will be called with keyword arguments for\n\n\n`message`, `topic_name`, `subscription_name` and `subject`. You function should in other words\nlook something like this:\n\n##### Celery\n```python\n@app.task(base=MetroidTask)\ndef my_func(*, message: dict, topic_name: str, subscription_name: str, subject: str) -> None:\n```\n\n##### rq\n```python\ndef my_func(*, message: dict, topic_name: str, subscription_name: str, subject: str) -> None:\n```\n\n\n### Running the project\n1. Ensure you have redis running:\n```bash\ndocker-compose up\n```\n2. Run migrations\n```bash\npython manage.py migrate\n```\n3. Create an admin account\n```bash\npython manage.py createsuperuser\n```\n4. Start a worker:\n```python\ncelery -A demoproj worker -l info\n```\n5. Run the subscriber:\n```python\npython manage.py metroid\n```\n6. Send messages to Metro. Example code can be found in [`demoproj/demoapp/services.py`](demoproj/demoapp/services.py)\n7. Run the webserver:\n```python\npython manage.py runserver 8000\n```\n8. See failed messages under `http://localhost:8080/admin`\n\nTo contribute, please see [`CONTRIBUTING.md`](CONTRIBUTING.md)\n",
"bugtrack_url": null,
"license": "",
"summary": "Metroid - Metro for Django",
"version": "1.4.0",
"project_urls": {
"Documentation": "https://github.com/intility/metroid",
"Homepage": "https://github.com/intility/metroid",
"Repository": "https://github.com/intility/metroid"
},
"split_keywords": [
"async",
"django",
"servicebus",
"task",
"celery",
"worker",
"rq"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7e180b24a18a8738462152941150e7dbd6724dae64f10847cefa56f8ef1e2946",
"md5": "989816f3f4f52ae967b914ab4b1537af",
"sha256": "a108154d7e02116941b8ac10f6077a4224c5143d4e65f30427ecd33c75cd002e"
},
"downloads": -1,
"filename": "metroid-1.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "989816f3f4f52ae967b914ab4b1537af",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10,<4.0",
"size": 18065,
"upload_time": "2024-01-03T14:53:39",
"upload_time_iso_8601": "2024-01-03T14:53:39.061043Z",
"url": "https://files.pythonhosted.org/packages/7e/18/0b24a18a8738462152941150e7dbd6724dae64f10847cefa56f8ef1e2946/metroid-1.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fae9246f0d800ad901d77342e03d8ed0641600038f7e3e85214283b4a49a6be8",
"md5": "baba7fbc2f45f5c8fe95453cee21217a",
"sha256": "166b17b1f84bf2bcfbea17a0695b8ece0ac18f1f490c07d6b761ec9b100650b0"
},
"downloads": -1,
"filename": "metroid-1.4.0.tar.gz",
"has_sig": false,
"md5_digest": "baba7fbc2f45f5c8fe95453cee21217a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10,<4.0",
"size": 15031,
"upload_time": "2024-01-03T14:53:40",
"upload_time_iso_8601": "2024-01-03T14:53:40.733796Z",
"url": "https://files.pythonhosted.org/packages/fa/e9/246f0d800ad901d77342e03d8ed0641600038f7e3e85214283b4a49a6be8/metroid-1.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-03 14:53:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "intility",
"github_project": "metroid",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "metroid"
}
JSON
