===============================================
GeventMP_ - Gevent_ Multiprocessing Extension
===============================================
.. image:: https://img.shields.io/gitter/room/karellen/Lobby?logo=gitter
:target: https://app.gitter.im/#/room/#karellen_Lobby:gitter.im
:alt: Gitter
.. image:: https://img.shields.io/github/actions/workflow/status/karellen/geventmp/build.yml?branch=master
:target: https://github.com/karellen/geventmp/actions/workflows/build.yml
:alt: Build Status
.. image:: https://img.shields.io/coveralls/github/karellen/geventmp/master?logo=coveralls
:target: https://coveralls.io/r/karellen/geventmp?branch=master
:alt: Coverage Status
|
.. image:: https://img.shields.io/pypi/v/geventmp?logo=pypi
:target: https://pypi.org/project/geventmp/
:alt: GeventMP Version
.. image:: https://img.shields.io/pypi/pyversions/geventmp?logo=pypi
:target: https://pypi.org/project/geventmp/
:alt: GeventMP Python Versions
.. image:: https://img.shields.io/pypi/dd/geventmp?logo=pypi
:target: https://pypi.org/project/geventmp/
:alt: GeventMP Downloads Per Day
.. image:: https://img.shields.io/pypi/dw/geventmp?logo=pypi
:target: https://pypi.org/project/geventmp/
:alt: GeventMP Downloads Per Week
.. image:: https://img.shields.io/pypi/dm/geventmp?logo=pypi
:target: https://pypi.org/project/geventmp/
:alt: GeventMP Downloads Per Month
|
.. warning::
HIC SUNT DRACONES!!!
This code is experimental (beta). There is some testing, but a lots of things are in flux, and
some platforms don't work at all.
You MAY try to use this in production with the understanding that this is a beta-quality software.
That said, this code may crash your server, bankrupt your company, burn your house down and be mean
to your puppy.
You've been warned.
Problem
=======
Due to internal implementation, `multiprocessing` (`MP`) is unsafe to use with Gevent_ even when `monkey-patched`__.
Namely, the use of OS semaphore primitives and inter-process IO in `MP` will cause the main
loop to stall/deadlock/block (specific issue depends on the version of CPython).
__ monkey_
Solution
========
GeventMP_ (`Gee-vent Em-Pee`) is a gevent_ multiprocessing extension plugin for the `monkey-patching`__ subsystem.
As with the rest of the monkey patch subsystem the process is fairly clear:
__ monkey_
1. Identify all places where blocking occurs and where it may stall the loop.
2. If blocking occurs on a file descriptor (`FD`), try to convert the file descriptor from blocking to non-blocking
(sockets/pipes/fifos, sometimes even files where, rarely, appropriate) and replace blocking IO functions with their
gevent_ non-blocking equivalents.
3. If blocking occurs in a Python/OS primitive that does not support non-blocking access and thus cannot be geventized,
wrap all blocking access to that primitive with native thread-pool-based wrappers and call it a day (while fully
understanding that primitive access latency will increase and raw performance may suffer as a result).
4. If you are really brave and have lots of free time on your hands, completely replace a standard blocking Python
non-`FD`-based primitive with implementation based on an `FD`-based OS primitive (e.g. POSIX semaphore =>
Linux `eventfd-based semaphore for kernels > 2.6.30`__).
5. Due to launching of separate processes in `MP`, figure out how, when, and whether to `monkey-patch`__ spawned/forked
children and grandchildren.
__ eventfd_
__ monkey_
Installation
============
The package is hosted on PyPi_.
For stable version:
.. code-block:: bash
pip install geventmp
For unstable version:
.. code-block:: bash
pip install --pre geventmp
Once installed, `GeventMP`_ will activate by default in the below stanza.
.. code-block:: python
from gevent.monkey import patch_all
patch_all()
If you would like `GeventMP`_ to not activate by default, either do not install it or explicitly disable it:
.. code-block:: python
from gevent.monkey import patch_all
patch_all(geventmp=False)
That's it - there are no other flags, settings, properties or config values so far.
Supported Platforms
===================
.. note::
All claims of support may not be real at all. You're welcome to experiment. See warnings on top.
* Linux and Darwin.
* CPython 3.7, 3.8, 3.9, 3.10, 3.11, 3.12
* PyPy 3.8, 3.9
Known Issues
============
* Multiprocessing `forkserver` works in GeventMP_, but the spawned child isn't green.
TODO
====
1. Monkey patch Windows to the extent possible.
2. Lots of applications use `Billiard <https://github.com/celery/billiard>`_ for multiprocessing instead of stock Python
package. Consider monkey patching Billiard if detected.
Contact Us
==========
Post feedback and issues on the `Bug Tracker`_, `Gitter`_,
and `Twitter (@karelleninc)`_.
.. _Gevent: https://github.com/gevent/gevent/
.. _geventmp: https://github.com/karellen/geventmp
.. _bug tracker: https://github.com/karellen/geventmp/issues
.. _gitter: https://gitter.im/karellen/Lobby
.. _twitter (@karelleninc): https://twitter.com/karelleninc
.. _monkey: https://en.wikipedia.org/wiki/Monkey_patch
.. _eventfd: https://linux.die.net/man/2/eventfd
.. _pypi: https://pypi.org/project/geventmp/
Raw data
{
"_id": null,
"home_page": "https://github.com/karellen/geventmp",
"name": "geventmp",
"maintainer": "Arcadiy Ivanov",
"docs_url": null,
"requires_python": ">3.6",
"maintainer_email": "arcadiy@ivanov.biz",
"keywords": "gevent multiprocessing mp monkey",
"author": "Karellen, Inc.",
"author_email": "supervisor@karellen.co",
"download_url": "https://files.pythonhosted.org/packages/cb/21/b14c9c1c89d0e3f57969e006ae2a35ebf69862b9e24aec192a85af7f6b3e/geventmp-0.0.8.tar.gz",
"platform": null,
"description": "===============================================\n GeventMP_ - Gevent_ Multiprocessing Extension\n===============================================\n\n.. image:: https://img.shields.io/gitter/room/karellen/Lobby?logo=gitter\n :target: https://app.gitter.im/#/room/#karellen_Lobby:gitter.im\n :alt: Gitter\n.. image:: https://img.shields.io/github/actions/workflow/status/karellen/geventmp/build.yml?branch=master\n :target: https://github.com/karellen/geventmp/actions/workflows/build.yml\n :alt: Build Status\n.. image:: https://img.shields.io/coveralls/github/karellen/geventmp/master?logo=coveralls\n :target: https://coveralls.io/r/karellen/geventmp?branch=master\n :alt: Coverage Status\n\n|\n\n.. image:: https://img.shields.io/pypi/v/geventmp?logo=pypi\n :target: https://pypi.org/project/geventmp/\n :alt: GeventMP Version\n.. image:: https://img.shields.io/pypi/pyversions/geventmp?logo=pypi\n :target: https://pypi.org/project/geventmp/\n :alt: GeventMP Python Versions\n.. image:: https://img.shields.io/pypi/dd/geventmp?logo=pypi\n :target: https://pypi.org/project/geventmp/\n :alt: GeventMP Downloads Per Day\n.. image:: https://img.shields.io/pypi/dw/geventmp?logo=pypi\n :target: https://pypi.org/project/geventmp/\n :alt: GeventMP Downloads Per Week\n.. image:: https://img.shields.io/pypi/dm/geventmp?logo=pypi\n :target: https://pypi.org/project/geventmp/\n :alt: GeventMP Downloads Per Month\n\n|\n\n.. warning::\n HIC SUNT DRACONES!!!\n\n This code is experimental (beta). There is some testing, but a lots of things are in flux, and\n some platforms don't work at all.\n\n You MAY try to use this in production with the understanding that this is a beta-quality software.\n\n That said, this code may crash your server, bankrupt your company, burn your house down and be mean\n to your puppy.\n\n You've been warned.\n\nProblem\n=======\n\nDue to internal implementation, `multiprocessing` (`MP`) is unsafe to use with Gevent_ even when `monkey-patched`__.\nNamely, the use of OS semaphore primitives and inter-process IO in `MP` will cause the main\nloop to stall/deadlock/block (specific issue depends on the version of CPython).\n\n__ monkey_\n\nSolution\n========\nGeventMP_ (`Gee-vent Em-Pee`) is a gevent_ multiprocessing extension plugin for the `monkey-patching`__ subsystem.\nAs with the rest of the monkey patch subsystem the process is fairly clear:\n\n__ monkey_\n\n1. Identify all places where blocking occurs and where it may stall the loop.\n2. If blocking occurs on a file descriptor (`FD`), try to convert the file descriptor from blocking to non-blocking\n (sockets/pipes/fifos, sometimes even files where, rarely, appropriate) and replace blocking IO functions with their\n gevent_ non-blocking equivalents.\n3. If blocking occurs in a Python/OS primitive that does not support non-blocking access and thus cannot be geventized,\n wrap all blocking access to that primitive with native thread-pool-based wrappers and call it a day (while fully\n understanding that primitive access latency will increase and raw performance may suffer as a result).\n4. If you are really brave and have lots of free time on your hands, completely replace a standard blocking Python\n non-`FD`-based primitive with implementation based on an `FD`-based OS primitive (e.g. POSIX semaphore =>\n Linux `eventfd-based semaphore for kernels > 2.6.30`__).\n5. Due to launching of separate processes in `MP`, figure out how, when, and whether to `monkey-patch`__ spawned/forked\n children and grandchildren.\n\n__ eventfd_\n\n__ monkey_\n\nInstallation\n============\nThe package is hosted on PyPi_.\n\nFor stable version:\n\n.. code-block:: bash\n\n pip install geventmp\n\nFor unstable version:\n\n.. code-block:: bash\n\n pip install --pre geventmp\n\n\nOnce installed, `GeventMP`_ will activate by default in the below stanza.\n\n.. code-block:: python\n\n from gevent.monkey import patch_all\n patch_all()\n\nIf you would like `GeventMP`_ to not activate by default, either do not install it or explicitly disable it:\n\n.. code-block:: python\n\n from gevent.monkey import patch_all\n patch_all(geventmp=False)\n\nThat's it - there are no other flags, settings, properties or config values so far.\n\nSupported Platforms\n===================\n\n.. note::\n All claims of support may not be real at all. You're welcome to experiment. See warnings on top.\n\n* Linux and Darwin.\n* CPython 3.7, 3.8, 3.9, 3.10, 3.11, 3.12\n* PyPy 3.8, 3.9\n\nKnown Issues\n============\n\n* Multiprocessing `forkserver` works in GeventMP_, but the spawned child isn't green.\n\nTODO\n====\n1. Monkey patch Windows to the extent possible.\n2. Lots of applications use `Billiard <https://github.com/celery/billiard>`_ for multiprocessing instead of stock Python\n package. Consider monkey patching Billiard if detected.\n\nContact Us\n==========\n\nPost feedback and issues on the `Bug Tracker`_, `Gitter`_,\nand `Twitter (@karelleninc)`_.\n\n.. _Gevent: https://github.com/gevent/gevent/\n.. _geventmp: https://github.com/karellen/geventmp\n.. _bug tracker: https://github.com/karellen/geventmp/issues\n.. _gitter: https://gitter.im/karellen/Lobby\n.. _twitter (@karelleninc): https://twitter.com/karelleninc\n.. _monkey: https://en.wikipedia.org/wiki/Monkey_patch\n.. _eventfd: https://linux.die.net/man/2/eventfd\n.. _pypi: https://pypi.org/project/geventmp/\n",
"bugtrack_url": null,
"license": "Apache License, Version 2.0",
"summary": "Multiprocessing Gevent Extension",
"version": "0.0.8",
"project_urls": {
"Bug Tracker": "https://github.com/karellen/geventmp/issues",
"Documentation": "https://github.com/karellen/geventmp/",
"Homepage": "https://github.com/karellen/geventmp",
"Source Code": "https://github.com/karellen/geventmp/"
},
"split_keywords": [
"gevent",
"multiprocessing",
"mp",
"monkey"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "72deebd2930bc798df374e7afe2d3152e9e2f2a6d27e393362f870c3776e82b0",
"md5": "a5e633fe3f6f198a8c3e5b78be3720cc",
"sha256": "7926954280f65bf5265a77b52709986f131267f189b6d2d1b58d6875be7eb44c"
},
"downloads": -1,
"filename": "geventmp-0.0.8-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a5e633fe3f6f198a8c3e5b78be3720cc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">3.6",
"size": 21962,
"upload_time": "2023-07-11T18:04:13",
"upload_time_iso_8601": "2023-07-11T18:04:13.619255Z",
"url": "https://files.pythonhosted.org/packages/72/de/ebd2930bc798df374e7afe2d3152e9e2f2a6d27e393362f870c3776e82b0/geventmp-0.0.8-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cb21b14c9c1c89d0e3f57969e006ae2a35ebf69862b9e24aec192a85af7f6b3e",
"md5": "249319032c8ee6222e9adade150e9d13",
"sha256": "6dffface520414fd669b1b9d77ec66dcf8590a01fb1ca1185f195e8b8747cb49"
},
"downloads": -1,
"filename": "geventmp-0.0.8.tar.gz",
"has_sig": false,
"md5_digest": "249319032c8ee6222e9adade150e9d13",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">3.6",
"size": 16793,
"upload_time": "2023-07-11T18:04:15",
"upload_time_iso_8601": "2023-07-11T18:04:15.132556Z",
"url": "https://files.pythonhosted.org/packages/cb/21/b14c9c1c89d0e3f57969e006ae2a35ebf69862b9e24aec192a85af7f6b3e/geventmp-0.0.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-11 18:04:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "karellen",
"github_project": "geventmp",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "geventmp"
}
JSON
