# Celery Redis Poll Backend
A specialized Redis backend for Celery that replaces the default pub/sub mechanism for task result retrieval with a polling-based approach.
## Why Polling Instead of Pub/Sub?
The default Celery Redis backend uses Redis pub/sub for real-time task result notifications. While pub/sub provides immediate updates, it can face challenges in certain scenarios:
- Deadlocks in highly concurrent/multi-threaded workloads due to single-threaded nature of Redis and Celery clients.
- Higher overhead with `SUBSCRIBE` channels.
This backend provides a more robust alternative by using a polling mechanism instead.
## Features
- **Polling-Based Results**: Replaces pub/sub with a polling mechanism for task result retrieval
- **Compatible with Existing Code**: Drop-in replacement for the standard Redis backend
- **Configurable Polling**: Adjust polling intervals and timeouts to match your needs
- **Resource Efficient**: Reduces Redis memory usage by eliminating pub/sub channels
## Installation
```bash
pip install celery-redis-poll
```
## Usage
Configure your Celery application to use the polling backend:
```python
from celery import Celery
app = Celery('your_app',
broker='redis://localhost:6379/0',
backend='redispoll://localhost:6379/0')
```
For clustered Redis, use `redisclusterpoll` instead of `redispoll`.
## Requirements
- Python >= 3.7
- Celery >= 5.0.0
- Redis >= 4.5.0
- celery-redis-cluster >= 0.1.6
## Development
For development, install extra dependencies:
```bash
pip install celery-redis-poll[dev]
```
## License
This project is licensed under the MIT License.
## Contributing
Contributions are welcome! Please feel free to submit issues and pull requests.
Raw data
{
"_id": null,
"home_page": "https://github.com/lan17/celery-redis-poll",
"name": "celery-redis-poll",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Lev Neiman",
"author_email": "lev.neiman@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/fc/e8/5e4faea4e83e24bc2f46b3333584ab4ef6f52a22e49b22f313ccb11a41e1/celery_redis_poll-0.1.6.tar.gz",
"platform": null,
"description": "# Celery Redis Poll Backend\n\nA specialized Redis backend for Celery that replaces the default pub/sub mechanism for task result retrieval with a polling-based approach.\n\n## Why Polling Instead of Pub/Sub?\n\nThe default Celery Redis backend uses Redis pub/sub for real-time task result notifications. While pub/sub provides immediate updates, it can face challenges in certain scenarios:\n\n- Deadlocks in highly concurrent/multi-threaded workloads due to single-threaded nature of Redis and Celery clients.\n- Higher overhead with `SUBSCRIBE` channels.\n\nThis backend provides a more robust alternative by using a polling mechanism instead.\n\n## Features\n\n- **Polling-Based Results**: Replaces pub/sub with a polling mechanism for task result retrieval\n- **Compatible with Existing Code**: Drop-in replacement for the standard Redis backend\n- **Configurable Polling**: Adjust polling intervals and timeouts to match your needs\n- **Resource Efficient**: Reduces Redis memory usage by eliminating pub/sub channels\n\n## Installation\n\n```bash\npip install celery-redis-poll\n```\n\n## Usage\n\nConfigure your Celery application to use the polling backend:\n\n```python\nfrom celery import Celery\n\napp = Celery('your_app',\n broker='redis://localhost:6379/0',\n backend='redispoll://localhost:6379/0')\n```\n\nFor clustered Redis, use `redisclusterpoll` instead of `redispoll`.\n\n## Requirements\n\n- Python >= 3.7\n- Celery >= 5.0.0\n- Redis >= 4.5.0\n- celery-redis-cluster >= 0.1.6\n\n## Development\n\nFor development, install extra dependencies:\n\n```bash\npip install celery-redis-poll[dev]\n```\n\n## License\n\nThis project is licensed under the MIT License.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit issues and pull requests.\n",
"bugtrack_url": null,
"license": null,
"summary": "A Redis Backend for Celery with polling instead of pub/sub",
"version": "0.1.6",
"project_urls": {
"Homepage": "https://github.com/lan17/celery-redis-poll"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "de143ef229e6a870b6cdbbf95d58def3394006edf030ebbac34ed27117234daf",
"md5": "8f931d7e2e5fa25bc003399af61aa685",
"sha256": "47695839e17098fc859c97e5d02efffffcf3d0d3476c40d3ed39b42fd783a722"
},
"downloads": -1,
"filename": "celery_redis_poll-0.1.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8f931d7e2e5fa25bc003399af61aa685",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 4358,
"upload_time": "2025-01-22T16:28:02",
"upload_time_iso_8601": "2025-01-22T16:28:02.080003Z",
"url": "https://files.pythonhosted.org/packages/de/14/3ef229e6a870b6cdbbf95d58def3394006edf030ebbac34ed27117234daf/celery_redis_poll-0.1.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "fce85e4faea4e83e24bc2f46b3333584ab4ef6f52a22e49b22f313ccb11a41e1",
"md5": "da82df4ba366111919a6b5b1621179a5",
"sha256": "d660e227312d4611547c729c6c967ec9b25ec0021f1ddd6adc2e2837039eba17"
},
"downloads": -1,
"filename": "celery_redis_poll-0.1.6.tar.gz",
"has_sig": false,
"md5_digest": "da82df4ba366111919a6b5b1621179a5",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 4322,
"upload_time": "2025-01-22T16:28:14",
"upload_time_iso_8601": "2025-01-22T16:28:14.959744Z",
"url": "https://files.pythonhosted.org/packages/fc/e8/5e4faea4e83e24bc2f46b3333584ab4ef6f52a22e49b22f313ccb11a41e1/celery_redis_poll-0.1.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-22 16:28:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lan17",
"github_project": "celery-redis-poll",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "celery-redis-poll"
}
JSON
