# Celery Redis Poll Backend
A specialized Redis backend for Celery that replaces the default pub/sub mechanism for task result retrieval with a polling-based approach.
## Why Polling Instead of Pub/Sub?
The default Celery Redis backend uses Redis pub/sub for real-time task result notifications. While pub/sub provides immediate updates, it can face challenges in certain scenarios:
- Deadlocks in highly concurrent/multi-threaded workloads due to single-threaded nature of Redis and Celery clients.
- Higher overhead with `SUBSCRIBE` channels.
This backend provides a more robust alternative by using a polling mechanism instead.
## Features
- **Polling-Based Results**: Replaces pub/sub with a polling mechanism for task result retrieval
- **Compatible with Existing Code**: Drop-in replacement for the standard Redis backend
- **Configurable Polling**: Adjust polling intervals and timeouts to match your needs
- **Resource Efficient**: Reduces Redis memory usage by eliminating pub/sub channels
## Installation
```bash
pip install celery-redis-poll
```
After installation new backends are automatically registered with Celery:
- `redis+poll`
- `rediss+poll`
- `redis+cluster_poll`
- `rediss+cluster_poll`
## Usage
Configure your Celery application to use the polling backend:
```python
from celery import Celery
app = Celery('your_app',
broker='redis://localhost:6379/0',
backend='redis+poll://localhost:6379/0')
```
For clustered Redis, use `redis+cluster_poll`.
## Requirements
- Python >= 3.7
- Celery >= 5.0.0
- Redis >= 4.5.0
- celery-redis-cluster >= 0.1.6
## Development
For development, install extra dependencies:
```bash
pip install celery-redis-poll[dev]
```
## License
This project is licensed under the MIT License.
## Contributing
Contributions are welcome! Please feel free to submit issues and pull requests.
Raw data
{
"_id": null,
"home_page": "https://github.com/lan17/celery-redis-poll",
"name": "celery-redis-poll",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Lev Neiman",
"author_email": "lev.neiman@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/47/0b/1995a29dc85f0ed0857f42db74f76d13b3da9b9a15eaa0438363bcae099d/celery_redis_poll-0.1.11.tar.gz",
"platform": null,
"description": "# Celery Redis Poll Backend\n\nA specialized Redis backend for Celery that replaces the default pub/sub mechanism for task result retrieval with a polling-based approach.\n\n## Why Polling Instead of Pub/Sub?\n\nThe default Celery Redis backend uses Redis pub/sub for real-time task result notifications. While pub/sub provides immediate updates, it can face challenges in certain scenarios:\n\n- Deadlocks in highly concurrent/multi-threaded workloads due to single-threaded nature of Redis and Celery clients.\n- Higher overhead with `SUBSCRIBE` channels.\n\nThis backend provides a more robust alternative by using a polling mechanism instead.\n\n## Features\n\n- **Polling-Based Results**: Replaces pub/sub with a polling mechanism for task result retrieval\n- **Compatible with Existing Code**: Drop-in replacement for the standard Redis backend\n- **Configurable Polling**: Adjust polling intervals and timeouts to match your needs\n- **Resource Efficient**: Reduces Redis memory usage by eliminating pub/sub channels\n\n## Installation\n\n```bash\npip install celery-redis-poll\n```\n\nAfter installation new backends are automatically registered with Celery:\n- `redis+poll`\n- `rediss+poll`\n- `redis+cluster_poll`\n- `rediss+cluster_poll`\n\n## Usage\n\nConfigure your Celery application to use the polling backend:\n\n```python\nfrom celery import Celery\n\napp = Celery('your_app',\n broker='redis://localhost:6379/0',\n backend='redis+poll://localhost:6379/0')\n```\n\nFor clustered Redis, use `redis+cluster_poll`.\n\n## Requirements\n\n- Python >= 3.7\n- Celery >= 5.0.0\n- Redis >= 4.5.0\n- celery-redis-cluster >= 0.1.6\n\n## Development\n\nFor development, install extra dependencies:\n\n```bash\npip install celery-redis-poll[dev]\n```\n\n## License\n\nThis project is licensed under the MIT License.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit issues and pull requests.\n",
"bugtrack_url": null,
"license": null,
"summary": "A Redis Backend for Celery with polling instead of pub/sub",
"version": "0.1.11",
"project_urls": {
"Homepage": "https://github.com/lan17/celery-redis-poll"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "606708546b5b4c4c81cafa86c93e2ac525419892792fe69b681d28e0d9047c59",
"md5": "c38dfac15fc3b54a525c4990b198186b",
"sha256": "1b2a8f29af75e3691c8e39434e388d6ebb75a23eb7071c8b59bff12b78d2ac99"
},
"downloads": -1,
"filename": "celery_redis_poll-0.1.11-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c38dfac15fc3b54a525c4990b198186b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 4500,
"upload_time": "2025-02-14T05:53:51",
"upload_time_iso_8601": "2025-02-14T05:53:51.248519Z",
"url": "https://files.pythonhosted.org/packages/60/67/08546b5b4c4c81cafa86c93e2ac525419892792fe69b681d28e0d9047c59/celery_redis_poll-0.1.11-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "470b1995a29dc85f0ed0857f42db74f76d13b3da9b9a15eaa0438363bcae099d",
"md5": "f48b370ee6ebd4b8bb62609288238d0d",
"sha256": "1444e6986ab409f2be02bedabe0e8c2f92513f41650001b7a99d202b23fc1bf8"
},
"downloads": -1,
"filename": "celery_redis_poll-0.1.11.tar.gz",
"has_sig": false,
"md5_digest": "f48b370ee6ebd4b8bb62609288238d0d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 4638,
"upload_time": "2025-02-14T05:53:52",
"upload_time_iso_8601": "2025-02-14T05:53:52.339840Z",
"url": "https://files.pythonhosted.org/packages/47/0b/1995a29dc85f0ed0857f42db74f76d13b3da9b9a15eaa0438363bcae099d/celery_redis_poll-0.1.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-14 05:53:52",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "lan17",
"github_project": "celery-redis-poll",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "celery-redis-poll"
}
JSON
