# EVECelery
[](https://pypi.org/project/EVECelery)
[](https://evecelery.nullsec.space/en/latest/?badge=latest)
[](https://github.com/NullsecSpace/EVECelery/actions/workflows/github-actions.yml)
[](https://github.com/NullsecSpace/EVECelery/blob/main/LICENSE)
EVECelery is a distributed task queue framework for building tools that interact with
the [EVE Online ESI API](https://esi.evetech.net/ui) using Celery, RabbitMQ, and Redis.
With EVECelery you can easily distribute ESI calls and tasks across a fleet of worker nodes built using [Celery](https://docs.celeryq.dev/).
You can create your own tasks on top of EVECelery defining custom functions and scheduled jobs that run alongside the included tasks.
NOTE: This software is in development and may rapidly change or have breaking bugs until the v1.0 release is ready.
Ensure you use version pinning in your ```requirements.txt```.
- :books: Documentation: https://evecelery.nullsec.space
- :bulb: Examples: https://evecelery.nullsec.space/en/latest/examples/index.html
## Features
* Built with [Celery](https://docs.celeryq.dev/) to distribute ESI calls and tasks across a fleet of worker nodes
* Distributed locking system with Redis ensures stateless workers won't make duplicate API calls if multiple clients run tasks with matching parameters at the same time
* Cache integration with Redis that caches ESI responses
* Easily define your own Celery tasks to register with the EVECelery worker nodes
* Client support for obtaining results synchronously or asynchronously. See [Celery calling tasks](https://docs.celeryq.dev/) for docs on calling tasks.
* Automated task retry and distributed error rate control limiting across the worker fleet
* ESI task API designed to mirror the [ESI Swagger Spec](https://esi.evetech.net/ui/) with the same parameter names, responses, and documentation for easy development and code completion
* Support for periodic scheduled tasks making use of the [Celery beat scheduler](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html)
## Installation
```
pip install EVECelery
```
## Requirements
EVECelery requires RabbitMQ for the message broker service and Redis for distributed locks, cache, and Celery's result
backend (fetching the result of completed tasks).
Deploying these two servers through the official Docker images for [RabbitMQ](https://hub.docker.com/_/rabbitmq)
and [Redis](https://hub.docker.com/_/redis) is recommended.
## Quickstart and Usage
EVECelery has two components:
* Celery Worker - The Celery worker server that processes tasks from the message broker and makes requests to ESI on behalf of the client applications
* Task api - Collection of Celery tasks to make ESI requests from your client application
You can deploy multiple worker servers that process tasks in the message queues. These worker nodes share locks, error limiting info, and cache requests for clients.
From your application you make requests using the task api.
### Starting the Celery Worker
You can start the worker server from either the CLI or your own Python script.
It is recommended to use the CLI unless you plan on registering your own tasks to the celery worker.
#### From CLI
Ensure the following environmental variables are set and run the application via bash:
* EVECelery_RabbitMQ_User
* EVECelery_RabbitMQ_Password
* EVECelery_RabbitMQ_Host
* EVECelery_RabbitMQ_Port
* EVECelery_RabbitMQ_Vhost
* EVECelery_Redis_ResultBackend_User
* EVECelery_Redis_ResultBackend_Password
* EVECelery_Redis_ResultBackend_Host
* EVECelery_Redis_ResultBackend_Port
* EVECelery_Redis_ResultBackend_DB
* EVECelery_Email
```shell
$ eve-celery
```
#### From your code
You can also start the worker from a Python script if you don't want to set environmental variables.
```python
from EVECelery import EVECeleryWorker
c = EVECeleryWorker(broker_user="user", broker_password="pass", broker_host="host", broker_port=5672,
broker_vhost="esi",
result_user="user", result_password="pass", result_host="host", result_port=6379, result_db=0)
c.start()
```
### Using the task API from your code
From another Python script you can send tasks to the queues and receive results:
```python
from EVECelery import EVECeleryWorker, TaskDirectory
import json
# only need to make one CeleryWorker in our code to init the tasks and setup connections to RabbitMQ and Redis
# by not passing connection params to CeleryWorker() the connection info will be read from environmental variables
c = EVECeleryWorker()
# note we don't call c.start() here as this is not a worker node script.
# we are calling the task api to submit requests to the message queue which run on the Celery worker nodes
r = TaskDirectory.ESI.Universe.get_universe_regions_region_id.get_sync(region_id=10000053)
# r is the response containing data obtained from ESI
# subsequent calls with the same parameters return results from the cache regardless of requesting client
print(json.dumps(r.dict(), indent=2, default=str))
{
"pydantic_model": "Response200_get_universe_regions_region_id",
"cache": {
"hit": true,
"key": "Cache.ESI.Universe.get_universe_regions_region_id.cd252dea2970194b46260124270444f07f4bf449a5fe37ef31f163005fcb50e7",
"ttl": 8940
},
"headers": {
"Cache_Control": "public",
"Content_Language": "en",
"ETag": null,
"Expires": "Sun, 23 Apr 2023 11:05:00 GMT",
"Last_Modified": "Sat, 22 Apr 2023 11:01:05 GMT"
},
"body": {
"constellations": [
20000609,
20000610,
20000611,
20000612,
20000613,
20000614,
20000615,
20000616,
20000617,
20000618
],
"description": "A natural destination for explorers and adventurers of all kinds, Cobalt Edge...",
"name": "Cobalt Edge",
"region_id": 10000053
}
}
```
## Copyright Notice
See [CCP.md](https://github.com/NullsecSpace/EVECelery/blob/main/CCP.md)
Raw data
{
"_id": null,
"home_page": "https://github.com/NullsecSpace/EVECelery",
"name": "EVECelery",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "eve online rabbitmq esi redis message queue broker",
"author": "",
"author_email": "maintainers@nullsec.space",
"download_url": "https://files.pythonhosted.org/packages/0e/0e/38bd199952f0b2a16d3f50bfed2d69574b43c91271a18b0eb52f0d9f45c9/EVECelery-0.20.tar.gz",
"platform": null,
"description": "# EVECelery\n\n[](https://pypi.org/project/EVECelery)\n[](https://evecelery.nullsec.space/en/latest/?badge=latest)\n[](https://github.com/NullsecSpace/EVECelery/actions/workflows/github-actions.yml)\n[](https://github.com/NullsecSpace/EVECelery/blob/main/LICENSE)\n\nEVECelery is a distributed task queue framework for building tools that interact with\nthe [EVE Online ESI API](https://esi.evetech.net/ui) using Celery, RabbitMQ, and Redis.\n\nWith EVECelery you can easily distribute ESI calls and tasks across a fleet of worker nodes built using [Celery](https://docs.celeryq.dev/).\nYou can create your own tasks on top of EVECelery defining custom functions and scheduled jobs that run alongside the included tasks.\n\nNOTE: This software is in development and may rapidly change or have breaking bugs until the v1.0 release is ready.\nEnsure you use version pinning in your ```requirements.txt```.\n\n- :books: Documentation: https://evecelery.nullsec.space\n- :bulb: Examples: https://evecelery.nullsec.space/en/latest/examples/index.html\n\n## Features\n* Built with [Celery](https://docs.celeryq.dev/) to distribute ESI calls and tasks across a fleet of worker nodes\n* Distributed locking system with Redis ensures stateless workers won't make duplicate API calls if multiple clients run tasks with matching parameters at the same time\n* Cache integration with Redis that caches ESI responses\n* Easily define your own Celery tasks to register with the EVECelery worker nodes\n* Client support for obtaining results synchronously or asynchronously. See [Celery calling tasks](https://docs.celeryq.dev/) for docs on calling tasks.\n* Automated task retry and distributed error rate control limiting across the worker fleet\n* ESI task API designed to mirror the [ESI Swagger Spec](https://esi.evetech.net/ui/) with the same parameter names, responses, and documentation for easy development and code completion\n* Support for periodic scheduled tasks making use of the [Celery beat scheduler](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html)\n\n## Installation\n\n```\npip install EVECelery\n```\n\n## Requirements\n\nEVECelery requires RabbitMQ for the message broker service and Redis for distributed locks, cache, and Celery's result\nbackend (fetching the result of completed tasks).\n\nDeploying these two servers through the official Docker images for [RabbitMQ](https://hub.docker.com/_/rabbitmq)\nand [Redis](https://hub.docker.com/_/redis) is recommended.\n\n## Quickstart and Usage\nEVECelery has two components:\n* Celery Worker - The Celery worker server that processes tasks from the message broker and makes requests to ESI on behalf of the client applications \n* Task api - Collection of Celery tasks to make ESI requests from your client application\n\nYou can deploy multiple worker servers that process tasks in the message queues. These worker nodes share locks, error limiting info, and cache requests for clients.\nFrom your application you make requests using the task api.\n\n\n### Starting the Celery Worker\nYou can start the worker server from either the CLI or your own Python script.\nIt is recommended to use the CLI unless you plan on registering your own tasks to the celery worker.\n\n#### From CLI\n\nEnsure the following environmental variables are set and run the application via bash:\n* EVECelery_RabbitMQ_User\n* EVECelery_RabbitMQ_Password\n* EVECelery_RabbitMQ_Host\n* EVECelery_RabbitMQ_Port\n* EVECelery_RabbitMQ_Vhost\n* EVECelery_Redis_ResultBackend_User\n* EVECelery_Redis_ResultBackend_Password\n* EVECelery_Redis_ResultBackend_Host\n* EVECelery_Redis_ResultBackend_Port\n* EVECelery_Redis_ResultBackend_DB\n* EVECelery_Email\n\n```shell\n$ eve-celery\n```\n\n#### From your code\nYou can also start the worker from a Python script if you don't want to set environmental variables.\n\n```python\nfrom EVECelery import EVECeleryWorker\n\nc = EVECeleryWorker(broker_user=\"user\", broker_password=\"pass\", broker_host=\"host\", broker_port=5672,\n broker_vhost=\"esi\",\n result_user=\"user\", result_password=\"pass\", result_host=\"host\", result_port=6379, result_db=0)\n\nc.start()\n```\n\n### Using the task API from your code\nFrom another Python script you can send tasks to the queues and receive results:\n\n```python\nfrom EVECelery import EVECeleryWorker, TaskDirectory\nimport json\n\n# only need to make one CeleryWorker in our code to init the tasks and setup connections to RabbitMQ and Redis\n# by not passing connection params to CeleryWorker() the connection info will be read from environmental variables\nc = EVECeleryWorker()\n\n# note we don't call c.start() here as this is not a worker node script.\n# we are calling the task api to submit requests to the message queue which run on the Celery worker nodes\n\nr = TaskDirectory.ESI.Universe.get_universe_regions_region_id.get_sync(region_id=10000053)\n# r is the response containing data obtained from ESI\n# subsequent calls with the same parameters return results from the cache regardless of requesting client\nprint(json.dumps(r.dict(), indent=2, default=str))\n{\n \"pydantic_model\": \"Response200_get_universe_regions_region_id\",\n \"cache\": {\n \"hit\": true,\n \"key\": \"Cache.ESI.Universe.get_universe_regions_region_id.cd252dea2970194b46260124270444f07f4bf449a5fe37ef31f163005fcb50e7\",\n \"ttl\": 8940\n },\n \"headers\": {\n \"Cache_Control\": \"public\",\n \"Content_Language\": \"en\",\n \"ETag\": null,\n \"Expires\": \"Sun, 23 Apr 2023 11:05:00 GMT\",\n \"Last_Modified\": \"Sat, 22 Apr 2023 11:01:05 GMT\"\n },\n \"body\": {\n \"constellations\": [\n 20000609,\n 20000610,\n 20000611,\n 20000612,\n 20000613,\n 20000614,\n 20000615,\n 20000616,\n 20000617,\n 20000618\n ],\n \"description\": \"A natural destination for explorers and adventurers of all kinds, Cobalt Edge...\",\n \"name\": \"Cobalt Edge\",\n \"region_id\": 10000053\n }\n}\n```\n\n## Copyright Notice\n\nSee [CCP.md](https://github.com/NullsecSpace/EVECelery/blob/main/CCP.md)\n\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Task queue framework for building tools that interact with the EVE Online ESI API using Celery, RabbitMQ, and Redis.",
"version": "0.20",
"split_keywords": [
"eve",
"online",
"rabbitmq",
"esi",
"redis",
"message",
"queue",
"broker"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "67251a578a8df86147461f2898684c75a2e614d83a3b77850f4bdb22e66d6ece",
"md5": "5f1531e30f1247d709066c0473b6302a",
"sha256": "5931d4bbe3d4538bff6cb4329b69bb62216aafa2bcc03403bbbbb842d4a17c6b"
},
"downloads": -1,
"filename": "EVECelery-0.20-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5f1531e30f1247d709066c0473b6302a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 683893,
"upload_time": "2023-04-23T09:32:12",
"upload_time_iso_8601": "2023-04-23T09:32:12.294339Z",
"url": "https://files.pythonhosted.org/packages/67/25/1a578a8df86147461f2898684c75a2e614d83a3b77850f4bdb22e66d6ece/EVECelery-0.20-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0e0e38bd199952f0b2a16d3f50bfed2d69574b43c91271a18b0eb52f0d9f45c9",
"md5": "9a8920fb9187496eb05173ee1ec2efb7",
"sha256": "de9213c5f7077dfdb52f8a766e5a62396a0babe592bf0f6ee48682d66332beef"
},
"downloads": -1,
"filename": "EVECelery-0.20.tar.gz",
"has_sig": false,
"md5_digest": "9a8920fb9187496eb05173ee1ec2efb7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 192382,
"upload_time": "2023-04-23T09:32:14",
"upload_time_iso_8601": "2023-04-23T09:32:14.170795Z",
"url": "https://files.pythonhosted.org/packages/0e/0e/38bd199952f0b2a16d3f50bfed2d69574b43c91271a18b0eb52f0d9f45c9/EVECelery-0.20.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-04-23 09:32:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "NullsecSpace",
"github_project": "EVECelery",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "evecelery"
}
JSON
