[](https://github.com/alex-petrenko/faster-fifo/actions/workflows/test-ci.yml)
[](https://pepy.tech/project/faster-fifo)
[<img src="https://img.shields.io/discord/987232982798598164?label=discord">](https://discord.gg/d9VZmMWejh)
<!-- [](https://codecov.io/gh/alex-petrenko/faster-fifo) -->
# faster-fifo
Faster alternative to Python's standard multiprocessing.Queue (IPC FIFO queue). Up to 30x faster in some configurations.
Implemented in C++ using POSIX mutexes with PTHREAD_PROCESS_SHARED attribute. Based on a circular buffer, low footprint, brokerless.
Completely mimics the interface of the standard multiprocessing.Queue, so can be used as a drop-in replacement.
Adds `get_many()` and `put_many()` methods to receive/send multiple messages at once for the price of a single lock.
## Requirements
- Linux or MacOS
- Python 3.6 or newer
- GCC 4.9.0 or newer
## Installation
```pip install faster-fifo```
(on a fresh Linux installation you might need some basic compiling tools `sudo apt install --reinstall build-essential`)
## Manual build instructions
```
pip install Cython
python setup.py build_ext --inplace
pip install -e .
```
## Usage example
```Python
from faster_fifo import Queue
from queue import Full, Empty
q = Queue(1000 * 1000) # specify the size of the circular buffer in the ctor
# any pickle-able Python object can be added to the queue
py_obj = dict(a=42, b=33, c=(1, 2, 3), d=[1, 2, 3], e='123', f=b'kkk')
q.put(py_obj)
assert q.qsize() == 1
retrieved = q.get()
assert q.empty()
assert py_obj == retrieved
for i in range(100):
try:
q.put(py_obj, timeout=0.1)
except Full:
log.debug('Queue is full!')
num_received = 0
while num_received < 100:
# get multiple messages at once, returns a list of messages for better performance in many-to-few scenarios
# get_many does not guarantee that all max_messages_to_get will be received on the first call, in fact
# no such guarantee can be made in multiprocessing systems.
# get_many() will retrieve as many messages as there are available AND can fit in the pre-allocated memory
# buffer. The size of the buffer is increased gradually to match demand.
messages = q.get_many(max_messages_to_get=100)
num_received += len(messages)
try:
q.get(timeout=0.1)
assert True, 'This won\'t be called'
except Empty:
log.debug('Queue is empty')
```
## Performance comparison (faster-fifo vs multiprocessing.Queue)
##### System #1 (Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz, 10 cores, Ubuntu 18.04)
*(measured execution times in seconds)*
| | multiprocessing.Queue | faster-fifo, get() | faster-fifo, get_many() |
|---------------------------------------------------|:---------------------:|:-----------------------:|:-------------------------:|
| 1 producer 1 consumer (200K msgs per producer) | 2.54 | 0.86 | 0.92 |
| 1 producer 10 consumers (200K msgs per producer) | 4.00 | 1.39 | 1.36 |
| 10 producers 1 consumer (100K msgs per producer) | 13.19 | 6.74 | 0.94 |
| 3 producers 20 consumers (100K msgs per producer) | 9.30 | 2.22 | 2.17 |
| 20 producers 3 consumers (50K msgs per producer) | 18.62 | 7.41 | 0.64 |
| 20 producers 20 consumers (50K msgs per producer) | 36.51 | 1.32 | 3.79 |
##### System #2 (Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz, 2 cores, Ubuntu 18.04)
*(measured execution times in seconds)*
| | multiprocessing.Queue | faster-fifo, get() | faster-fifo, get_many() |
|---------------------------------------------------|:---------------------:|:-----------------------:|:-------------------------:|
| 1 producer 1 consumer (200K msgs per producer) | 7.86 | 2.09 | 2.2 |
| 1 producer 10 consumers (200K msgs per producer) | 11.68 | 4.01 | 3.88 |
| 10 producers 1 consumer (100K msgs per producer) | 44.48 | 16.68 | 5.98 |
| 3 producers 20 consumers (100K msgs per producer) | 22.59 | 7.83 | 7.49 |
| 20 producers 3 consumers (50K msgs per producer) | 66.3 | 22.3 | 6.35 |
| 20 producers 20 consumers (50K msgs per producer) | 78.75 | 14.39 | 15.78 |
## Run tests
`python -m unittest`
(there are also C++ unit tests, should run them if C++ code was altered)
## Recent PyPI releases
##### v1.4.5
* Added method `data_size()` to query the total size of the messages in queue (in bytes). Thank you [@LucaNicosia](https://github.com/LucaNicosia)!
##### v1.4.4
* Fixed an obscure issue with the TLSBuffer ctor being called without arguments (guessing it's Cython's weirdness)
##### v1.4.3
* Simplified usage with "spawn" multiprocessing context. No need to use `faster_fifo_reduction` anymore. Thank you [@MosBas](https://github.com/MosBas)!
##### v1.4.2
* Fixed an issue with the custom Queue pickler
##### v1.4.1
* Fixed multithreading issues using threading.local for message recv buffer (huge thanks to [@brianmacy](https://github.com/brianmacy)!)
* Better error reporting in Cython and C++
* Added threading tests
##### v1.4.0
* Increase default receive buffer size from 10 bytes to 5000 bytes.
##### v1.3.1
* Minor change: better debugging messages + improved C++ tests
##### v1.3.0
* Now support custom serializers and deserializers instead of Pickle (thank you [@beasteers](https://github.com/beasteers)!):
```Python
q = Queue(max_size_bytes=100000, loads=custom_deserializer, dumps=custom_serializer)
```
## Footnote
Originally designed for SampleFactory, a high-throughput asynchronous RL codebase https://github.com/alex-petrenko/sample-factory.
Programmed by [Aleksei Petrenko](https://alex-petrenko.github.io/) and Tushar Kumar at [USC RESL](https://robotics.usc.edu/resl/people/).
Developed under MIT License, feel free to use for any purpose, commercial or not, at your own risk.
If you wish to cite this repository:
```
@misc{faster-fifo,
author={Petrenko, Aleksei and Kumar, Tushar},
title={A Faster Alternative to Python's multiprocessing.Queue},
publisher={GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/alex-petrenko/faster-fifo}},
year={2020},
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/alex-petrenko/faster-fifo",
"name": "faster-fifo",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "multiprocessing data structures",
"author": "Aleksei Petrenko & Tushar Kumar",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/59/ec/18ec355ffd6e3ede5b15523df54f0e34615216c0f7cfe20560e0571e1e4e/faster-fifo-1.4.5.tar.gz",
"platform": null,
"description": "[](https://github.com/alex-petrenko/faster-fifo/actions/workflows/test-ci.yml)\n[](https://pepy.tech/project/faster-fifo)\n[<img src=\"https://img.shields.io/discord/987232982798598164?label=discord\">](https://discord.gg/d9VZmMWejh)\n\n<!-- [](https://codecov.io/gh/alex-petrenko/faster-fifo) -->\n\n# faster-fifo\n\nFaster alternative to Python's standard multiprocessing.Queue (IPC FIFO queue). Up to 30x faster in some configurations.\n\nImplemented in C++ using POSIX mutexes with PTHREAD_PROCESS_SHARED attribute. Based on a circular buffer, low footprint, brokerless.\nCompletely mimics the interface of the standard multiprocessing.Queue, so can be used as a drop-in replacement.\n\nAdds `get_many()` and `put_many()` methods to receive/send multiple messages at once for the price of a single lock.\n\n## Requirements \n\n- Linux or MacOS\n- Python 3.6 or newer\n- GCC 4.9.0 or newer\n\n## Installation\n\n```pip install faster-fifo```\n\n(on a fresh Linux installation you might need some basic compiling tools `sudo apt install --reinstall build-essential`)\n\n## Manual build instructions\n\n```\npip install Cython\npython setup.py build_ext --inplace\npip install -e .\n```\n\n## Usage example\n\n```Python\nfrom faster_fifo import Queue\nfrom queue import Full, Empty\n\nq = Queue(1000 * 1000) # specify the size of the circular buffer in the ctor\n\n# any pickle-able Python object can be added to the queue\npy_obj = dict(a=42, b=33, c=(1, 2, 3), d=[1, 2, 3], e='123', f=b'kkk')\nq.put(py_obj)\nassert q.qsize() == 1\n\nretrieved = q.get()\nassert q.empty()\nassert py_obj == retrieved\n\nfor i in range(100):\n try:\n q.put(py_obj, timeout=0.1)\n except Full:\n log.debug('Queue is full!')\n\nnum_received = 0\nwhile num_received < 100:\n # get multiple messages at once, returns a list of messages for better performance in many-to-few scenarios\n # get_many does not guarantee that all max_messages_to_get will be received on the first call, in fact\n # no such guarantee can be made in multiprocessing systems.\n # get_many() will retrieve as many messages as there are available AND can fit in the pre-allocated memory\n # buffer. The size of the buffer is increased gradually to match demand.\n messages = q.get_many(max_messages_to_get=100)\n num_received += len(messages)\n\ntry:\n q.get(timeout=0.1)\n assert True, 'This won\\'t be called'\nexcept Empty:\n log.debug('Queue is empty')\n\n```\n\n## Performance comparison (faster-fifo vs multiprocessing.Queue)\n\n##### System #1 (Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz, 10 cores, Ubuntu 18.04)\n\n*(measured execution times in seconds)*\n\n| | multiprocessing.Queue | faster-fifo, get() | faster-fifo, get_many() |\n|---------------------------------------------------|:---------------------:|:-----------------------:|:-------------------------:|\n| 1 producer 1 consumer (200K msgs per producer) | 2.54 | 0.86 | 0.92 |\n| 1 producer 10 consumers (200K msgs per producer) | 4.00 | 1.39 | 1.36 | \n| 10 producers 1 consumer (100K msgs per producer) | 13.19 | 6.74 | 0.94 |\n| 3 producers 20 consumers (100K msgs per producer) | 9.30 | 2.22 | 2.17 |\n| 20 producers 3 consumers (50K msgs per producer) | 18.62 | 7.41 | 0.64 |\n| 20 producers 20 consumers (50K msgs per producer) | 36.51 | 1.32 | 3.79 |\n\n\n##### System #2 (Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz, 2 cores, Ubuntu 18.04)\n\n*(measured execution times in seconds)*\n\n| | multiprocessing.Queue | faster-fifo, get() | faster-fifo, get_many() |\n|---------------------------------------------------|:---------------------:|:-----------------------:|:-------------------------:|\n| 1 producer 1 consumer (200K msgs per producer) | 7.86 | 2.09 | 2.2 |\n| 1 producer 10 consumers (200K msgs per producer) | 11.68 | 4.01 | 3.88 | \n| 10 producers 1 consumer (100K msgs per producer) | 44.48 | 16.68 | 5.98 |\n| 3 producers 20 consumers (100K msgs per producer) | 22.59 | 7.83 | 7.49 |\n| 20 producers 3 consumers (50K msgs per producer) | 66.3 | 22.3 | 6.35 |\n| 20 producers 20 consumers (50K msgs per producer) | 78.75 | 14.39 | 15.78 |\n\n## Run tests\n\n`python -m unittest`\n\n(there are also C++ unit tests, should run them if C++ code was altered)\n\n## Recent PyPI releases\n\n##### v1.4.5\n\n* Added method `data_size()` to query the total size of the messages in queue (in bytes). Thank you [@LucaNicosia](https://github.com/LucaNicosia)!\n\n##### v1.4.4\n\n* Fixed an obscure issue with the TLSBuffer ctor being called without arguments (guessing it's Cython's weirdness)\n\n##### v1.4.3\n\n* Simplified usage with \"spawn\" multiprocessing context. No need to use `faster_fifo_reduction` anymore. Thank you [@MosBas](https://github.com/MosBas)!\n\n##### v1.4.2\n\n* Fixed an issue with the custom Queue pickler\n\n##### v1.4.1\n\n* Fixed multithreading issues using threading.local for message recv buffer (huge thanks to [@brianmacy](https://github.com/brianmacy)!)\n* Better error reporting in Cython and C++\n* Added threading tests\n\n##### v1.4.0\n\n* Increase default receive buffer size from 10 bytes to 5000 bytes.\n\n##### v1.3.1\n\n* Minor change: better debugging messages + improved C++ tests\n\n##### v1.3.0\n* Now support custom serializers and deserializers instead of Pickle (thank you [@beasteers](https://github.com/beasteers)!):\n```Python\nq = Queue(max_size_bytes=100000, loads=custom_deserializer, dumps=custom_serializer)\n```\n\n## Footnote\n\nOriginally designed for SampleFactory, a high-throughput asynchronous RL codebase https://github.com/alex-petrenko/sample-factory.\n\nProgrammed by [Aleksei Petrenko](https://alex-petrenko.github.io/) and Tushar Kumar at [USC RESL](https://robotics.usc.edu/resl/people/).\n\nDeveloped under MIT License, feel free to use for any purpose, commercial or not, at your own risk.\n\nIf you wish to cite this repository:\n\n```\n@misc{faster-fifo,\n author={Petrenko, Aleksei and Kumar, Tushar},\n title={A Faster Alternative to Python's multiprocessing.Queue},\n publisher={GitHub},\n journal = {GitHub repository},\n howpublished = {\\url{https://github.com/alex-petrenko/faster-fifo}},\n year={2020},\n}\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A faster alternative to Python's standard multiprocessing.Queue (IPC FIFO queue)",
"version": "1.4.5",
"split_keywords": [
"multiprocessing",
"data",
"structures"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "59ec18ec355ffd6e3ede5b15523df54f0e34615216c0f7cfe20560e0571e1e4e",
"md5": "2e1e14b596e9e831d0460d226dc895d0",
"sha256": "0c4b30592d33d9ccda307d8e88c2142aa533b5ce334da6ee4350e289108566c7"
},
"downloads": -1,
"filename": "faster-fifo-1.4.5.tar.gz",
"has_sig": false,
"md5_digest": "2e1e14b596e9e831d0460d226dc895d0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 87049,
"upload_time": "2023-03-25T02:16:25",
"upload_time_iso_8601": "2023-03-25T02:16:25.447543Z",
"url": "https://files.pythonhosted.org/packages/59/ec/18ec355ffd6e3ede5b15523df54f0e34615216c0f7cfe20560e0571e1e4e/faster-fifo-1.4.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-25 02:16:25",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "alex-petrenko",
"github_project": "faster-fifo",
"travis_ci": true,
"coveralls": false,
"github_actions": true,
"lcname": "faster-fifo"
}
JSON
