| Name | squeal-queue JSON |
| Version |
0.0.8
JSON |
| download |
| home_page | |
| Summary | Message queue backed by a database |
| upload_time | 2023-08-21 05:26:54 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | <4,>=3.8 |
| license | MIT License Copyright (c) [year] [fullname] Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
message
queue
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# squeal: SQL-Backed Message Queue
A python library implementing a message queue using a relational database as the storage backend.
**Note**: This is an alpha version. The interface is unstable. Feel free to try it out, though, and let me know what you think.
## Why?
`squeal` offers a lightweight implementation of a message queue, using a backend that you probably already have as part of your infrastructure. The basic functionality is exposed by the `squeal.Queue` object:
* `create` and `destroy` the required database tables
* `put` and `get` messages from a queue
* a message payload is just a binary blob
* messages have a priority, where higher-priority messages are retrieved first
* consumers can `ack` or `nack` messages to indicate success or failure
* if a consumer acquires a message but doesn't `ack` it, it will eventually be redelivered to another consumer
* a message that is `nack`ed will be put back in the queue with an exponential backoff delay
* a `Queue` object represents multiple logical queues, indicated by a message `topic`
* topics are dynamic: they only exist as long as there's a message with that topic
* a `Queue` can query for existing topics or the number of messages waiting in any particular topic
`Queue` objects delegate to a `Backend` object that implements database-specific methods. The only backend is currently the `MySQLBackend`, which wraps a `Connection` from a mysql library, like `pymysql`.
## What database backends are supported?
Currently, the only backend that has been tested is:
* [`pymysql`](https://github.com/PyMySQL/PyMySQL) with `mysql 8.1.0`
But theoretically other database libraries can be used, as long as they implement [PEP 249 (Python Database API Specification)](https://peps.python.org/pep-0249/). Other database engines can probably be supported with minimal effort by changing the dialect of SQL that's generated. (That is, creating a new subclass of `Backend`)
# Examples
Check the `examples/` directory.
# API
(Coming soon)
# Contributing
## To-Do
* dead letter queue for messages that fail repeatedly
* raise some better exceptions if we get an expected error from the SQL library (table doesn't exist, etc)
* do some benchmarking and add indices
* refactor tests so all backends are compared against the same expectations
Please feel free to submit an issue to the github for bugs, comments, or feature requests. Also feel free to fork and make a PR.
## Formatting
Please use `black` to format your code.
## Running tests
Install the dev requirements in a virtual env:
```python3
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
```
The tests assume you have a mysql instance running locally. The connection can be adjusted with envvars, but the defaults are:
```python3
SQUEAL_TEST_HOSTNAME = os.environ.get("SQUEAL_TEST_HOSTNAME", "localhost")
SQUEAL_TEST_PORT = os.environ.get("SQUEAL_TEST_PORT", "3306")
SQUEAL_TEST_USERNAME = os.environ.get("SQUEAL_TEST_USERNAME", "root")
SQUEAL_TEST_PASSWORD = os.environ.get("SQUEAL_TEST_PASSWORD", "password")
SQUEAL_TEST_DATABASE = os.environ.get("SQUEAL_TEST_DATABASE", "test")
```
The easiest way to get this running is to just use docker:
```bash
docker run --name mysql -e MYSQL_ROOT_PASSWORD=password -d -p 3306:3306 mysql:8.1.0
```
Then the tests can be run with `pytest`:
```bash
pytest tests
```
Raw data
{
"_id": null,
"home_page": "",
"name": "squeal-queue",
"maintainer": "",
"docs_url": null,
"requires_python": "<4,>=3.8",
"maintainer_email": "",
"keywords": "message queue",
"author": "",
"author_email": "Alex Dodge <alexdodge@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/de/08/c2257d4cea4341c8227cdbe2049f6b55a9716acaffdf943c4c9e5837edc5/squeal-queue-0.0.8.tar.gz",
"platform": null,
"description": "# squeal: SQL-Backed Message Queue\n\nA python library implementing a message queue using a relational database as the storage backend.\n\n**Note**: This is an alpha version. The interface is unstable. Feel free to try it out, though, and let me know what you think.\n\n## Why?\n\n`squeal` offers a lightweight implementation of a message queue, using a backend that you probably already have as part of your infrastructure. The basic functionality is exposed by the `squeal.Queue` object:\n\n* `create` and `destroy` the required database tables\n* `put` and `get` messages from a queue\n* a message payload is just a binary blob\n* messages have a priority, where higher-priority messages are retrieved first\n* consumers can `ack` or `nack` messages to indicate success or failure\n* if a consumer acquires a message but doesn't `ack` it, it will eventually be redelivered to another consumer\n* a message that is `nack`ed will be put back in the queue with an exponential backoff delay\n* a `Queue` object represents multiple logical queues, indicated by a message `topic`\n* topics are dynamic: they only exist as long as there's a message with that topic\n* a `Queue` can query for existing topics or the number of messages waiting in any particular topic\n\n`Queue` objects delegate to a `Backend` object that implements database-specific methods. The only backend is currently the `MySQLBackend`, which wraps a `Connection` from a mysql library, like `pymysql`.\n\n## What database backends are supported?\n\nCurrently, the only backend that has been tested is:\n\n* [`pymysql`](https://github.com/PyMySQL/PyMySQL) with `mysql 8.1.0`\n\nBut theoretically other database libraries can be used, as long as they implement [PEP 249 (Python Database API Specification)](https://peps.python.org/pep-0249/). Other database engines can probably be supported with minimal effort by changing the dialect of SQL that's generated. (That is, creating a new subclass of `Backend`)\n\n# Examples\nCheck the `examples/` directory.\n\n# API\n(Coming soon)\n\n# Contributing\n\n## To-Do\n* dead letter queue for messages that fail repeatedly\n* raise some better exceptions if we get an expected error from the SQL library (table doesn't exist, etc)\n* do some benchmarking and add indices\n* refactor tests so all backends are compared against the same expectations\n\nPlease feel free to submit an issue to the github for bugs, comments, or feature requests. Also feel free to fork and make a PR.\n\n## Formatting\nPlease use `black` to format your code.\n\n## Running tests\nInstall the dev requirements in a virtual env:\n```python3\npython3 -m venv venv\n. venv/bin/activate\npip install -r requirements.txt\n```\n\nThe tests assume you have a mysql instance running locally. The connection can be adjusted with envvars, but the defaults are:\n```python3\nSQUEAL_TEST_HOSTNAME = os.environ.get(\"SQUEAL_TEST_HOSTNAME\", \"localhost\")\nSQUEAL_TEST_PORT = os.environ.get(\"SQUEAL_TEST_PORT\", \"3306\")\nSQUEAL_TEST_USERNAME = os.environ.get(\"SQUEAL_TEST_USERNAME\", \"root\")\nSQUEAL_TEST_PASSWORD = os.environ.get(\"SQUEAL_TEST_PASSWORD\", \"password\")\nSQUEAL_TEST_DATABASE = os.environ.get(\"SQUEAL_TEST_DATABASE\", \"test\")\n```\n\nThe easiest way to get this running is to just use docker:\n```bash\ndocker run --name mysql -e MYSQL_ROOT_PASSWORD=password -d -p 3306:3306 mysql:8.1.0\n```\n\nThen the tests can be run with `pytest`:\n```bash\npytest tests\n```\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) [year] [fullname] Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "Message queue backed by a database",
"version": "0.0.8",
"project_urls": {
"Library Bug Tracker": "https://github.com/adodge/squeal/issues",
"Library Homepage": "https://github.com/adodge/squeal"
},
"split_keywords": [
"message",
"queue"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "843e89bf57f19ee5b03702851021f1f6d43c41b8ad75241fe95a7fb6831b8d14",
"md5": "e5e1ffe25024bd53ebaa2621d316d329",
"sha256": "80826edb0d34250d31f32fcc2ac8eea65f9408e76f5e77ff1e129fe694611b1f"
},
"downloads": -1,
"filename": "squeal_queue-0.0.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e5e1ffe25024bd53ebaa2621d316d329",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4,>=3.8",
"size": 12742,
"upload_time": "2023-08-21T05:26:52",
"upload_time_iso_8601": "2023-08-21T05:26:52.964224Z",
"url": "https://files.pythonhosted.org/packages/84/3e/89bf57f19ee5b03702851021f1f6d43c41b8ad75241fe95a7fb6831b8d14/squeal_queue-0.0.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "de08c2257d4cea4341c8227cdbe2049f6b55a9716acaffdf943c4c9e5837edc5",
"md5": "909eae46253a9b07350ae6f0e21e2e44",
"sha256": "95c2a2179a4bb00d49ba2003c52d0c083c78ed653e2d0d3e1a699f6a6286af36"
},
"downloads": -1,
"filename": "squeal-queue-0.0.8.tar.gz",
"has_sig": false,
"md5_digest": "909eae46253a9b07350ae6f0e21e2e44",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4,>=3.8",
"size": 14555,
"upload_time": "2023-08-21T05:26:54",
"upload_time_iso_8601": "2023-08-21T05:26:54.523846Z",
"url": "https://files.pythonhosted.org/packages/de/08/c2257d4cea4341c8227cdbe2049f6b55a9716acaffdf943c4c9e5837edc5/squeal-queue-0.0.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-21 05:26:54",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "adodge",
"github_project": "squeal",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "squeal-queue"
}
