# Celery AsyncIO Pool
<p align="center" width="100%">
<img width="55%" src="https://raw.githubusercontent.com/the-wondersmith/celery-aio-pool/main/icon.svg">
</p>
> Free software: GNU Affero General Public License v3+
## Getting Started
### Installation
#### Using Poetry _(preferred)_
```bash
poetry add celery-aio-pool
```
#### Using `pip` & [PyPI.org](https://pypi.org/project/celery-aio-pool/)
```bash
pip install celery-aio-pool
```
#### Using `pip` & [GitHub](https://github.com/the-wondersmith/celery-aio-pool.git)
```bash
pip install git+https://github.com/the-wondersmith/celery-aio-pool.git
```
#### Using `pip` & A Local Copy Of The Repo
```bash
git clone https://github.com/the-wondersmith/celery-aio-pool.git
cd celery-aio-pool
pip install -e "$(pwd)"
```
## Configure Celery
### Option 1: Using Celery's Out-Of-Tree Worker Pool _(preferred)_
#### Configuring Custom Worker Pool
Celery now supports the configuration of out-of-tree worker pool classes, allowing you to configure custom pools like `celery-aio-pool` directly:
- Set the environment variable `CELERY_CUSTOM_WORKER_POOL` to the name of your desired worker pool implementation.
- **NOTE:** _The value must be formatted as `package:class`_
- ```bash
export CELERY_CUSTOM_WORKER_POOL='celery_aio_pool.pool:AsyncIOPool'
```
- Start Celery with `--pool=custom` to use the configured pool:
- ```bash
celery worker --pool=custom --loglevel=INFO --logfile="$(pwd)/worker.log"
```
To verify the pool configuration, use `celery inspect stats`:
```bash
celery --app=your_celery_project inspect stats
-> celery@freenas: OK
{
...
"pool": {
...
"implementation": "celery_aio_pool.pool:AsyncIOPool",
...
```
### Option 2: Monkey-patching Celery _(non-preferred)_ for older versions of Celery (<5.3)
#### Using `celery-aio-pool`'s Provided Patcher
- Import `celery_aio_pool` in the same module where your Celery "app" is defined
- Ensure that the `patch_celery_tracer` utility is called **_before_** any other
Celery code is called
```python
"""My super awesome Celery app."""
# ...
from celery import Celery
# add the following import
import celery_aio_pool as aio_pool
# ensure the patcher is called *before*
# your Celery app is defined
assert aio_pool.patch_celery_tracer() is True
app = Celery(
"my-super-awesome-celery-app",
broker="amqp://guest@localhost//",
# add the following keyword argument
worker_pool=aio_pool.pool.AsyncIOPool,
)
```
## Developing / Testing / Contributing
> **NOTE:** _Our preferred packaging and dependency manager is [Poetry](https://python-poetry.org/)._
> _Installation instructions can be found [here](https://python-poetry.org/docs/#installing-with-the-official-installer)._
### Developing
Clone the repo and install the dependencies
```bash
$ git clone https://github.com/the-wondersmith/celery-aio-pool.git \
&& cd celery-aio-pool \
&& poetry install --sync
```
Alternatively, if you prefer _not_ to use Poetry, `celery-aio-pool` is fully PEP-517
compliant and can be installed directly by any PEP-517-compliant package manager.
```bash
$ cd celery-aio-pool \
&& pip install -e "$(pwd)"
```
> **TODO:** _Coming Soon™_
### Testing
To run the test suite:
```bash
$ poetry run pytest tests/
```
### Contributing
> **TODO:** _Coming Soon™_
```
Raw data
{
"_id": null,
"home_page": "https://github.com/the-wondersmith/celery-aio-pool",
"name": "celery-aio-pool",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "pool, celery, asyncio, concurrency, background-jobs",
"author": "Mark S.",
"author_email": "the@wondersmithd.dev",
"download_url": "https://files.pythonhosted.org/packages/24/50/2d0962983fe1b21a4969454f804c12b5d10ad5ae335342b8520eed1d8899/celery_aio_pool-0.1.0rc8.tar.gz",
"platform": null,
"description": "# Celery AsyncIO Pool\n\n\n\n\n\n\n<p align=\"center\" width=\"100%\">\n <img width=\"55%\" src=\"https://raw.githubusercontent.com/the-wondersmith/celery-aio-pool/main/icon.svg\">\n</p>\n\n> Free software: GNU Affero General Public License v3+\n\n## Getting Started\n\n### Installation\n\n#### Using Poetry _(preferred)_\n\n```bash\npoetry add celery-aio-pool\n```\n\n#### Using `pip` & [PyPI.org](https://pypi.org/project/celery-aio-pool/)\n\n```bash\npip install celery-aio-pool\n```\n\n#### Using `pip` & [GitHub](https://github.com/the-wondersmith/celery-aio-pool.git)\n\n```bash\npip install git+https://github.com/the-wondersmith/celery-aio-pool.git\n```\n\n#### Using `pip` & A Local Copy Of The Repo\n\n```bash\ngit clone https://github.com/the-wondersmith/celery-aio-pool.git\ncd celery-aio-pool\npip install -e \"$(pwd)\"\n```\n\n## Configure Celery\n\n### Option 1: Using Celery's Out-Of-Tree Worker Pool _(preferred)_\n#### Configuring Custom Worker Pool\n\nCelery now supports the configuration of out-of-tree worker pool classes, allowing you to configure custom pools like `celery-aio-pool` directly:\n\n- Set the environment variable `CELERY_CUSTOM_WORKER_POOL` to the name of your desired worker pool implementation.\n - **NOTE:** _The value must be formatted as `package:class`_\n - ```bash\n export CELERY_CUSTOM_WORKER_POOL='celery_aio_pool.pool:AsyncIOPool'\n ```\n\n- Start Celery with `--pool=custom` to use the configured pool:\n - ```bash\n celery worker --pool=custom --loglevel=INFO --logfile=\"$(pwd)/worker.log\"\n ```\n\nTo verify the pool configuration, use `celery inspect stats`:\n\n```bash\ncelery --app=your_celery_project inspect stats\n-> celery@freenas: OK\n {\n ...\n \"pool\": {\n ...\n \"implementation\": \"celery_aio_pool.pool:AsyncIOPool\",\n ...\n```\n\n### Option 2: Monkey-patching Celery _(non-preferred)_ for older versions of Celery (<5.3)\n#### Using `celery-aio-pool`'s Provided Patcher\n\n- Import `celery_aio_pool` in the same module where your Celery \"app\" is defined\n- Ensure that the `patch_celery_tracer` utility is called **_before_** any other\n Celery code is called\n\n```python\n\"\"\"My super awesome Celery app.\"\"\"\n\n# ...\nfrom celery import Celery\n\n# add the following import\nimport celery_aio_pool as aio_pool\n\n# ensure the patcher is called *before*\n# your Celery app is defined\n\nassert aio_pool.patch_celery_tracer() is True\n\napp = Celery(\n \"my-super-awesome-celery-app\",\n broker=\"amqp://guest@localhost//\",\n # add the following keyword argument\n worker_pool=aio_pool.pool.AsyncIOPool,\n)\n```\n\n## Developing / Testing / Contributing\n\n> **NOTE:** _Our preferred packaging and dependency manager is [Poetry](https://python-poetry.org/)._\n> _Installation instructions can be found [here](https://python-poetry.org/docs/#installing-with-the-official-installer)._\n\n### Developing\n\nClone the repo and install the dependencies\n```bash\n$ git clone https://github.com/the-wondersmith/celery-aio-pool.git \\\n && cd celery-aio-pool \\\n && poetry install --sync\n```\n\nAlternatively, if you prefer _not_ to use Poetry, `celery-aio-pool` is fully PEP-517\ncompliant and can be installed directly by any PEP-517-compliant package manager.\n\n```bash\n$ cd celery-aio-pool \\\n && pip install -e \"$(pwd)\"\n```\n\n> **TODO:** _Coming Soon\u2122_\n\n### Testing\n\nTo run the test suite:\n\n```bash\n$ poetry run pytest tests/\n```\n\n### Contributing\n\n> **TODO:** _Coming Soon\u2122_\n```\n",
"bugtrack_url": null,
"license": "AGPL-3.0-or-later",
"summary": "Celery worker pool with support for asyncio coroutines as tasks",
"version": "0.1.0rc8",
"project_urls": {
"Documentation": "https://github.com/the-wondersmith/celery-aio-pool",
"Homepage": "https://github.com/the-wondersmith/celery-aio-pool",
"Repository": "https://github.com/the-wondersmith/celery-aio-pool"
},
"split_keywords": [
"pool",
" celery",
" asyncio",
" concurrency",
" background-jobs"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3fed27407bd7aa280c9d6a3a33d541277bc35ad9a97d2eabac3c67e148da29fb",
"md5": "22c8a36c9c9e4dc6a6ddc72c343fd10a",
"sha256": "9a7631ea2759e6c8ae769fe668a3af34299a6ee730dabe57e36e66d66611b14a"
},
"downloads": -1,
"filename": "celery_aio_pool-0.1.0rc8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "22c8a36c9c9e4dc6a6ddc72c343fd10a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 23305,
"upload_time": "2024-12-09T15:31:13",
"upload_time_iso_8601": "2024-12-09T15:31:13.081920Z",
"url": "https://files.pythonhosted.org/packages/3f/ed/27407bd7aa280c9d6a3a33d541277bc35ad9a97d2eabac3c67e148da29fb/celery_aio_pool-0.1.0rc8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "24502d0962983fe1b21a4969454f804c12b5d10ad5ae335342b8520eed1d8899",
"md5": "12df129c8e3d12b749c77b0575771a40",
"sha256": "47ffe9307a8be94eb39cc1c6607f49b18db2c797623fb6ea0825835dd50652f2"
},
"downloads": -1,
"filename": "celery_aio_pool-0.1.0rc8.tar.gz",
"has_sig": false,
"md5_digest": "12df129c8e3d12b749c77b0575771a40",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 24467,
"upload_time": "2024-12-09T15:31:13",
"upload_time_iso_8601": "2024-12-09T15:31:13.997466Z",
"url": "https://files.pythonhosted.org/packages/24/50/2d0962983fe1b21a4969454f804c12b5d10ad5ae335342b8520eed1d8899/celery_aio_pool-0.1.0rc8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-09 15:31:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "the-wondersmith",
"github_project": "celery-aio-pool",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "amqp",
"specs": [
[
"==",
"5.3.1"
]
]
},
{
"name": "backports-zoneinfo",
"specs": [
[
"==",
"0.2.1"
]
]
},
{
"name": "backports-zoneinfo",
"specs": [
[
"==",
"0.2.1"
]
]
},
{
"name": "billiard",
"specs": [
[
"==",
"4.2.1"
]
]
},
{
"name": "celery",
"specs": [
[
"==",
"5.4.0"
]
]
},
{
"name": "click-didyoumean",
"specs": [
[
"==",
"0.3.1"
]
]
},
{
"name": "click-plugins",
"specs": [
[
"==",
"1.1.1"
]
]
},
{
"name": "click-repl",
"specs": [
[
"==",
"0.3.0"
]
]
},
{
"name": "click",
"specs": [
[
"==",
"8.1.7"
]
]
},
{
"name": "colorama",
"specs": [
[
"==",
"0.4.6"
]
]
},
{
"name": "kombu",
"specs": [
[
"==",
"5.4.2"
]
]
},
{
"name": "prompt-toolkit",
"specs": [
[
"==",
"3.0.48"
]
]
},
{
"name": "python-dateutil",
"specs": [
[
"==",
"2.9.0.post0"
]
]
},
{
"name": "six",
"specs": [
[
"==",
"1.17.0"
]
]
},
{
"name": "typing-extensions",
"specs": [
[
"==",
"4.12.2"
]
]
},
{
"name": "tzdata",
"specs": [
[
"==",
"2024.2"
]
]
},
{
"name": "vine",
"specs": [
[
"==",
"5.1.0"
]
]
},
{
"name": "wcwidth",
"specs": [
[
"==",
"0.2.13"
]
]
}
],
"lcname": "celery-aio-pool"
}
JSON
