greenback: reenter an asyncio or Trio event loop from synchronous code
======================================================================
.. image:: https://img.shields.io/pypi/v/greenback.svg
:target: https://pypi.org/project/greenback
:alt: Latest PyPI version
.. image:: https://img.shields.io/badge/docs-read%20now-blue.svg
:target: https://greenback.readthedocs.io/en/latest/?badge=latest
:alt: Documentation status
.. image:: https://travis-ci.org/oremanj/greenback.svg?branch=master
:target: https://travis-ci.org/oremanj/greenback
:alt: Automated test status
.. image:: https://codecov.io/gh/oremanj/greenback/branch/master/graph/badge.svg
:target: https://codecov.io/gh/oremanj/greenback
:alt: Test coverage
.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
:target: https://github.com/ambv/black
:alt: Code style: black
.. image:: http://www.mypy-lang.org/static/mypy_badge.svg
:target: http://www.mypy-lang.org/
:alt: Checked with mypy
Python 3.5 introduced ``async``/``await`` syntax for defining
functions that can run concurrently in a cooperative multitasking
framework such as ``asyncio`` or `Trio
<https://trio.readthedocs.io/>`__. Such frameworks have a number of advantages
over previous approaches to concurrency: they scale better than threads and are
`clearer about control flow <https://glyph.twistedmatrix.com/2014/02/unyielding.html>`__
than the implicit cooperative multitasking provided by ``gevent``. They're also being
actively developed to explore some `exciting new ideas about concurrent programming
<https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/>`__.
Porting an existing codebase to ``async``/``await`` syntax can be
challenging, though, since it's somewhat "viral": only an async
function can call another async function. That means you don't just have
to modify the functions that actually perform I/O; you also need to
(trivially) modify every function that directly or indirectly calls a
function that performs I/O. While the results are generally an improvement
("explicit is better than implicit"), getting there in one big step is not
always feasible, especially if some of these layers are in libraries that
you don't control.
``greenback`` is a small library that attempts to bridge this gap. It
allows you to **call back into async code from a syntactically
synchronous function**, as long as the synchronous function was
originally called from an async task (running in an asyncio or Trio
event loop) that set up a ``greenback`` "portal" as explained
below. This is potentially useful in a number of different situations:
* You can interoperate with some existing libraries that are not
``async``/``await`` aware, without pushing their work off into
another thread.
* You can migrate an existing program to ``async``/``await``
syntax one layer at a time, instead of all at once.
* You can (cautiously) design async APIs that block in places where
you can't write ``await``, such as on attribute accesses.
``greenback`` requires Python 3.8 or later and an implementation that
supports the ``greenlet`` library. Either CPython or PyPy should work.
There are no known OS dependencies.
Quickstart
----------
* Call ``await greenback.ensure_portal()`` at least once in each task that will be
using ``greenback``. (Additional calls in the same task do nothing.) You can think
of this as creating a portal that will be used by future calls to
``greenback.await_()`` in the same task.
* Later, use ``greenback.await_(foo())`` as a replacement for
``await foo()`` in places where you can't write ``await``.
* If all of the places where you want to use
``greenback.await_()`` are indirectly within a single function, you can
eschew the ``await greenback.ensure_portal()`` and instead write a wrapper
around calls to that function: ``await greenback.with_portal_run(...)``
for an async function, or ``await greenback.with_portal_run_sync(...)``
for a synchronous function. These have the advantage of cleaning up the
portal (and its associated minor performance impact) as soon as the
function returns, rather than leaving it open until the task terminates.
* For more details and additional helper methods, see the
`documentation <https://greenback.readthedocs.io>`__.
Example
-------
Suppose you start with this async-unaware program::
import subprocess
def main():
print_fact(10)
def print_fact(n, mult=1):
"""Print the value of *n* factorial times *mult*."""
if n <= 1:
print_value(mult)
else:
print_fact(n - 1, mult * n)
def print_value(n):
"""Print the value *n* in an unreasonably convoluted way."""
assert isinstance(n, int)
subprocess.run(f"echo {n}", shell=True)
if __name__ == "__main__":
main()
Using ``greenback``, you can change it to run in a Trio event loop by
changing only the top and bottom layers, with no change to ``print_fact()``. ::
import trio
import greenback
async def main():
await greenback.ensure_portal()
print_fact(10)
def print_fact(n, mult=1):
"""Print the value of *n* factorial times *mult*."""
if n <= 1:
print_value(mult)
else:
print_fact(n - 1, mult * n)
def print_value(n):
"""Print the value *n* in an unreasonably convoluted way."""
assert isinstance(n, int)
greenback.await_(trio.run_process(f"echo {n}", shell=True))
if __name__ == "__main__":
trio.run(main)
FAQ
---
**Why is it called "greenback"?** It uses the `greenlet
<https://greenlet.readthedocs.io/en/latest/>`__ library to get you
*back* to an enclosing async context. Also, maybe it saves you `money
<https://www.dictionary.com/browse/greenback>`__ (engineering time) or
something.
**How does it work?** After you run ``await greenback.ensure_portal()``
in a certain task, that task will run inside a greenlet.
(This is achieved by interposing a "shim" coroutine in between the event
loop and the coroutine for your task; see the source code for details.)
Calls to ``greenback.await_()`` are then able to switch from that greenlet
back to the parent greenlet, which can easily perform the necessary
``await`` since it has direct access to the async environment. The
task greenlet is then resumed with the value or exception
produced by the ``await``.
**Should I trust this in production?** Maybe; try it and see. The
technique is rather low-level, and has some minor
`performance implications <https://greenback.readthedocs.io/en/latest/principle.html#performance>`__ (any task in which you call ``await
greenback.ensure_portal()`` will run a bit slower), but we're in
good company: SQLAlchemy's async ORM support is implemented in much
the same way. ``greenback`` itself is a fairly small amount of
pure-Python code on top of ``greenlet``. (There is one small usage of
``ctypes`` to work around a knob that's not exposed by the asyncio
acceleration extension module on CPython.)
``greenlet`` is a C module full of platform-specific arcana, but
it's been around for a very long time and popular production-quality
concurrency systems such as ``gevent`` rely heavily on it.
**What won't work?** A few things:
* Greenlet switching works by moving parts of the C stack to different
memory addresses, relying on the assumption that Python objects are
fully heap-allocated and don't contain any pointers into the C
stack. Poorly-behaved C extension modules might violate this
assumption and are likely to crash if used with ``greenback``.
Such extension modules are buggy and could be made to crash without
``greenback`` too, but perhaps only under an obscure or unlikely
series of operations.
* Calling ``greenback.await_()`` inside a finalizer (``__del__``
method), signal handler, or weakref callback is unsupported. It
might work most of the time, or even all the time, but the
environment in which such methods run is weird enough that the
author isn't prepared to make any guarantees. (Not that you have
any guarantees about the rest of it, just some better theoretical
grounding.)
License
-------
``greenback`` is licensed under your choice of the MIT or Apache 2.0 license.
See `LICENSE <https://github.com/oremanj/greenback/blob/master/LICENSE>`__
for details.
Raw data
{
"_id": null,
"home_page": "https://github.com/oremanj/greenback",
"name": "greenback",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "async,io,trio,asyncio",
"author": "Joshua Oreman",
"author_email": "oremanj@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/f4/c4/3c87c103dd82e0e55955a92aeedec64003757490933df9c934386b590f01/greenback-1.2.0.tar.gz",
"platform": null,
"description": "greenback: reenter an asyncio or Trio event loop from synchronous code\n======================================================================\n\n.. image:: https://img.shields.io/pypi/v/greenback.svg\n :target: https://pypi.org/project/greenback\n :alt: Latest PyPI version\n\n.. image:: https://img.shields.io/badge/docs-read%20now-blue.svg\n :target: https://greenback.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation status\n\n.. image:: https://travis-ci.org/oremanj/greenback.svg?branch=master\n :target: https://travis-ci.org/oremanj/greenback\n :alt: Automated test status\n\n.. image:: https://codecov.io/gh/oremanj/greenback/branch/master/graph/badge.svg\n :target: https://codecov.io/gh/oremanj/greenback\n :alt: Test coverage\n\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg\n :target: https://github.com/ambv/black\n :alt: Code style: black\n\n.. image:: http://www.mypy-lang.org/static/mypy_badge.svg\n :target: http://www.mypy-lang.org/\n :alt: Checked with mypy\n\n\nPython 3.5 introduced ``async``/``await`` syntax for defining\nfunctions that can run concurrently in a cooperative multitasking\nframework such as ``asyncio`` or `Trio\n<https://trio.readthedocs.io/>`__. Such frameworks have a number of advantages\nover previous approaches to concurrency: they scale better than threads and are\n`clearer about control flow <https://glyph.twistedmatrix.com/2014/02/unyielding.html>`__\nthan the implicit cooperative multitasking provided by ``gevent``. They're also being\nactively developed to explore some `exciting new ideas about concurrent programming\n<https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/>`__.\n\nPorting an existing codebase to ``async``/``await`` syntax can be\nchallenging, though, since it's somewhat \"viral\": only an async\nfunction can call another async function. That means you don't just have\nto modify the functions that actually perform I/O; you also need to\n(trivially) modify every function that directly or indirectly calls a\nfunction that performs I/O. While the results are generally an improvement\n(\"explicit is better than implicit\"), getting there in one big step is not\nalways feasible, especially if some of these layers are in libraries that\nyou don't control.\n\n``greenback`` is a small library that attempts to bridge this gap. It\nallows you to **call back into async code from a syntactically\nsynchronous function**, as long as the synchronous function was\noriginally called from an async task (running in an asyncio or Trio\nevent loop) that set up a ``greenback`` \"portal\" as explained\nbelow. This is potentially useful in a number of different situations:\n\n* You can interoperate with some existing libraries that are not\n ``async``/``await`` aware, without pushing their work off into\n another thread.\n\n* You can migrate an existing program to ``async``/``await``\n syntax one layer at a time, instead of all at once.\n\n* You can (cautiously) design async APIs that block in places where\n you can't write ``await``, such as on attribute accesses.\n\n``greenback`` requires Python 3.8 or later and an implementation that\nsupports the ``greenlet`` library. Either CPython or PyPy should work.\nThere are no known OS dependencies.\n\nQuickstart\n----------\n\n* Call ``await greenback.ensure_portal()`` at least once in each task that will be\n using ``greenback``. (Additional calls in the same task do nothing.) You can think\n of this as creating a portal that will be used by future calls to\n ``greenback.await_()`` in the same task.\n\n* Later, use ``greenback.await_(foo())`` as a replacement for\n ``await foo()`` in places where you can't write ``await``.\n\n* If all of the places where you want to use\n ``greenback.await_()`` are indirectly within a single function, you can\n eschew the ``await greenback.ensure_portal()`` and instead write a wrapper\n around calls to that function: ``await greenback.with_portal_run(...)``\n for an async function, or ``await greenback.with_portal_run_sync(...)``\n for a synchronous function. These have the advantage of cleaning up the\n portal (and its associated minor performance impact) as soon as the\n function returns, rather than leaving it open until the task terminates.\n\n* For more details and additional helper methods, see the\n `documentation <https://greenback.readthedocs.io>`__.\n\nExample\n-------\n\nSuppose you start with this async-unaware program::\n\n import subprocess\n\n def main():\n print_fact(10)\n\n def print_fact(n, mult=1):\n \"\"\"Print the value of *n* factorial times *mult*.\"\"\"\n if n <= 1:\n print_value(mult)\n else:\n print_fact(n - 1, mult * n)\n\n def print_value(n):\n \"\"\"Print the value *n* in an unreasonably convoluted way.\"\"\"\n assert isinstance(n, int)\n subprocess.run(f\"echo {n}\", shell=True)\n\n if __name__ == \"__main__\":\n main()\n\nUsing ``greenback``, you can change it to run in a Trio event loop by\nchanging only the top and bottom layers, with no change to ``print_fact()``. ::\n\n import trio\n import greenback\n\n async def main():\n await greenback.ensure_portal()\n print_fact(10)\n\n def print_fact(n, mult=1):\n \"\"\"Print the value of *n* factorial times *mult*.\"\"\"\n if n <= 1:\n print_value(mult)\n else:\n print_fact(n - 1, mult * n)\n\n def print_value(n):\n \"\"\"Print the value *n* in an unreasonably convoluted way.\"\"\"\n assert isinstance(n, int)\n greenback.await_(trio.run_process(f\"echo {n}\", shell=True))\n\n if __name__ == \"__main__\":\n trio.run(main)\n\nFAQ\n---\n\n**Why is it called \"greenback\"?** It uses the `greenlet\n<https://greenlet.readthedocs.io/en/latest/>`__ library to get you\n*back* to an enclosing async context. Also, maybe it saves you `money\n<https://www.dictionary.com/browse/greenback>`__ (engineering time) or\nsomething.\n\n**How does it work?** After you run ``await greenback.ensure_portal()``\nin a certain task, that task will run inside a greenlet.\n(This is achieved by interposing a \"shim\" coroutine in between the event\nloop and the coroutine for your task; see the source code for details.)\nCalls to ``greenback.await_()`` are then able to switch from that greenlet\nback to the parent greenlet, which can easily perform the necessary\n``await`` since it has direct access to the async environment. The\ntask greenlet is then resumed with the value or exception\nproduced by the ``await``.\n\n**Should I trust this in production?** Maybe; try it and see. The\ntechnique is rather low-level, and has some minor\n`performance implications <https://greenback.readthedocs.io/en/latest/principle.html#performance>`__ (any task in which you call ``await\ngreenback.ensure_portal()`` will run a bit slower), but we're in\ngood company: SQLAlchemy's async ORM support is implemented in much\nthe same way. ``greenback`` itself is a fairly small amount of\npure-Python code on top of ``greenlet``. (There is one small usage of\n``ctypes`` to work around a knob that's not exposed by the asyncio\nacceleration extension module on CPython.)\n``greenlet`` is a C module full of platform-specific arcana, but\nit's been around for a very long time and popular production-quality\nconcurrency systems such as ``gevent`` rely heavily on it.\n\n**What won't work?** A few things:\n\n* Greenlet switching works by moving parts of the C stack to different\n memory addresses, relying on the assumption that Python objects are\n fully heap-allocated and don't contain any pointers into the C\n stack. Poorly-behaved C extension modules might violate this\n assumption and are likely to crash if used with ``greenback``.\n Such extension modules are buggy and could be made to crash without\n ``greenback`` too, but perhaps only under an obscure or unlikely\n series of operations.\n\n* Calling ``greenback.await_()`` inside a finalizer (``__del__``\n method), signal handler, or weakref callback is unsupported. It\n might work most of the time, or even all the time, but the\n environment in which such methods run is weird enough that the\n author isn't prepared to make any guarantees. (Not that you have\n any guarantees about the rest of it, just some better theoretical\n grounding.)\n\n\nLicense\n-------\n\n``greenback`` is licensed under your choice of the MIT or Apache 2.0 license.\nSee `LICENSE <https://github.com/oremanj/greenback/blob/master/LICENSE>`__\nfor details.\n",
"bugtrack_url": null,
"license": "MIT -or- Apache License 2.0",
"summary": "Reenter an async event loop from synchronous code",
"version": "1.2.0",
"project_urls": {
"Homepage": "https://github.com/oremanj/greenback"
},
"split_keywords": [
"async",
"io",
"trio",
"asyncio"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "e2c66d34fef317ef45691bf803187913d206b520a6402d50a6a60c370338e6a5",
"md5": "62f62bb8886f267663c2c217255677af",
"sha256": "7b510dd3c22cb3d20043a628b1a5e02713fc98fea59f9ebd299902a5d078c9ce"
},
"downloads": -1,
"filename": "greenback-1.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "62f62bb8886f267663c2c217255677af",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 28070,
"upload_time": "2024-02-08T00:27:57",
"upload_time_iso_8601": "2024-02-08T00:27:57.793313Z",
"url": "https://files.pythonhosted.org/packages/e2/c6/6d34fef317ef45691bf803187913d206b520a6402d50a6a60c370338e6a5/greenback-1.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f4c43c87c103dd82e0e55955a92aeedec64003757490933df9c934386b590f01",
"md5": "234a61d1a1b2ea9f61c96c06d113caec",
"sha256": "db8b323617e1e1a7cbb3615fb9048c44b483aab3d3415f49e07c6957c5fce60d"
},
"downloads": -1,
"filename": "greenback-1.2.0.tar.gz",
"has_sig": false,
"md5_digest": "234a61d1a1b2ea9f61c96c06d113caec",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 42450,
"upload_time": "2024-02-08T00:28:00",
"upload_time_iso_8601": "2024-02-08T00:28:00.249015Z",
"url": "https://files.pythonhosted.org/packages/f4/c4/3c87c103dd82e0e55955a92aeedec64003757490933df9c934386b590f01/greenback-1.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-08 00:28:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "oremanj",
"github_project": "greenback",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"lcname": "greenback"
}
JSON
