async-reduce


Nameasync-reduce JSON
Version 1.4 PyPI version JSON
download
home_pagehttps://github.com/sirkonst/async-reduce
SummaryReducer for similar simultaneously coroutines
upload_time2025-01-18 18:27:25
maintainerKonstantin Enchant
docs_urlNone
authorKonstantin Enchant
requires_python>=3.7
licenseMIT
keywords asyncio reduce
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage
            [![Python versions](https://img.shields.io/badge/python-3.7%2C%203.8%2C%203.9%2C%203.10%2C%203.11%2C%203.12%2C%203.13-green.svg)]()
[![PyPI version](https://badge.fury.io/py/async-reduce.svg)](https://pypi.org/project/async-reduce/)


About Async-Reduce
==================

``async_reduce(coroutine)`` allows aggregate all *similar simultaneous*
ready to run `coroutine`s and reduce to running **only one** `coroutine`.
Other aggregated `coroutine`s will get result from single `coroutine`.

It can boost application performance in highly competitive execution of the
similar asynchronous operations and reduce load for inner systems.


Quick example
-------------

```python
from async_reduce import async_reduce


async def fetch_user_data(user_id: int) -> dict:
    """" Get user data from inner service """
    url = 'http://inner-service/user/{}'.format(user_id)

    return await http.get(url, timeout=10).json()


@web_server.router('/users/(\d+)')
async def handler_user_detail(request, user_id: int):
    """ Handler for get detail information about user """

    # all simultaneous requests of fetching user data for `user_id` will
    # reduced to single request
    user_data = await async_reduce(
        fetch_user_data(user_id)
    )

    # sometimes ``async_reduce`` cannot detect similar coroutines and
    # you should provide special argument `ident` for manually determination
    user_statistics = await async_reduce(
        DataBase.query('user_statistics').where(id=user_id).fetch_one(),
        ident='db_user_statistics:{}'.format(user_id)
    )

    return Response(...)
```

In that example without using ``async_reduce`` if client performs **N**
simultaneous requests like `GET http://web_server/users/42` *web_server*
performs **N** requests to *inner-service* and **N** queries to *database*.
In total: **N** simultaneous requests emits **2 * N** requests to inner systems.

With ``async_reduce`` if client performs **N** simultaneous requests *web_server*
performs **one** request to *inner-service* and **one** query to *database*.
In total: **N** simultaneous requests emit only **2** requests to inner systems.

See other real [examples](https://github.com/sirkonst/async-reduce/tree/master/examples).


Similar coroutines determination
--------------------------------

``async_reduce(coroutine)`` tries to detect similar coroutines by hashing
local variables bounded on call. It does not work correctly if:

* one of the arguments is not hashable
* coroutine function is a method of class with specific state (like ORM)
* coroutine function has closure to unhashable variable

You can disable auto-determination by setting custom key to argument ``ident``.


Use as decorator
----------------

Also library provide special decorator ``@async_reduceable()``, example:

```python
from async_reduce import async_reduceable


@async_reduceable()
async def fetch_user_data(user_id: int) -> dict:
    """" Get user data from inner service """
    url = 'http://inner-servicce/user/{}'.format(user_id)

    return await http.get(url, timeout=10).json()


@web_server.router('/users/(\d+)')
async def handler_user_detail(request, user_id: int):
    """ Handler for get detail information about user """
    return await fetch_user_data(user_id)
```


Hooks
-----

Library supports hooks. Add-on hooks:

* **DebugHooks** - print about all triggered hooks
* **StatisticsOverallHooks** - general statistics on the use of `async_reduce`
* **StatisticsDetailHooks** - like `StatisticsOverallHooks` but detail statistics
about all `coroutine` processed by `async_reduce`

Example:

```python
from async_reduce import AsyncReducer
from async_reduce.hooks import DebugHooks

# define custom async_reduce with hooks
async_reduce = AsyncReducer(hooks=DebugHooks())


async def handler_user_detail(request, user_id: int):
    user_data = await async_reduce(fetch_user_data(user_id))
```

See more detail example in [examples/example_hooks.py](https://github.com/sirkonst/async-reduce/blob/master/examples/example_hooks.py).

You can write custom hooks via inherit from [BaseHooks](https://github.com/sirkonst/async-reduce/blob/master/async_reduce/hooks/base.py).


Caveats
-------

* If single `coroutine` raises exceptions all aggregated `coroutine`s will get
same exception too

* If single `coroutine` is stuck all aggregated `coroutine`s will stuck too.
Limit execution time for `coroutine` and add retries (optional) to avoid it.

* Be careful when return mutable value from `coroutine` because single value
will shared. Prefer to use non-mutable value as coroutine return.


Development
-----------

See [DEVELOPMENT.md](https://github.com/sirkonst/async-reduce/blob/master/DEVELOPMENT.md).

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/sirkonst/async-reduce",
    "name": "async-reduce",
    "maintainer": "Konstantin Enchant",
    "docs_url": null,
    "requires_python": ">=3.7",
    "maintainer_email": "sirkonst@gmail.com",
    "keywords": "asyncio, reduce",
    "author": "Konstantin Enchant",
    "author_email": "sirkonst@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/a8/81/ec55c99bf95349fd8bc24a9cb36903c0a842c4662ff803720e65d7c37d87/async_reduce-1.4.tar.gz",
    "platform": null,
    "description": "[![Python versions](https://img.shields.io/badge/python-3.7%2C%203.8%2C%203.9%2C%203.10%2C%203.11%2C%203.12%2C%203.13-green.svg)]()\n[![PyPI version](https://badge.fury.io/py/async-reduce.svg)](https://pypi.org/project/async-reduce/)\n\n\nAbout Async-Reduce\n==================\n\n``async_reduce(coroutine)`` allows aggregate all *similar simultaneous*\nready to run `coroutine`s and reduce to running **only one** `coroutine`.\nOther aggregated `coroutine`s will get result from single `coroutine`.\n\nIt can boost application performance in highly competitive execution of the\nsimilar asynchronous operations and reduce load for inner systems.\n\n\nQuick example\n-------------\n\n```python\nfrom async_reduce import async_reduce\n\n\nasync def fetch_user_data(user_id: int) -> dict:\n    \"\"\"\" Get user data from inner service \"\"\"\n    url = 'http://inner-service/user/{}'.format(user_id)\n\n    return await http.get(url, timeout=10).json()\n\n\n@web_server.router('/users/(\\d+)')\nasync def handler_user_detail(request, user_id: int):\n    \"\"\" Handler for get detail information about user \"\"\"\n\n    # all simultaneous requests of fetching user data for `user_id` will\n    # reduced to single request\n    user_data = await async_reduce(\n        fetch_user_data(user_id)\n    )\n\n    # sometimes ``async_reduce`` cannot detect similar coroutines and\n    # you should provide special argument `ident` for manually determination\n    user_statistics = await async_reduce(\n        DataBase.query('user_statistics').where(id=user_id).fetch_one(),\n        ident='db_user_statistics:{}'.format(user_id)\n    )\n\n    return Response(...)\n```\n\nIn that example without using ``async_reduce`` if client performs **N**\nsimultaneous requests like `GET http://web_server/users/42` *web_server*\nperforms **N** requests to *inner-service* and **N** queries to *database*.\nIn total: **N** simultaneous requests emits **2 * N** requests to inner systems.\n\nWith ``async_reduce`` if client performs **N** simultaneous requests *web_server*\nperforms **one** request to *inner-service* and **one** query to *database*.\nIn total: **N** simultaneous requests emit only **2** requests to inner systems.\n\nSee other real [examples](https://github.com/sirkonst/async-reduce/tree/master/examples).\n\n\nSimilar coroutines determination\n--------------------------------\n\n``async_reduce(coroutine)`` tries to detect similar coroutines by hashing\nlocal variables bounded on call. It does not work correctly if:\n\n* one of the arguments is not hashable\n* coroutine function is a method of class with specific state (like ORM)\n* coroutine function has closure to unhashable variable\n\nYou can disable auto-determination by setting custom key to argument ``ident``.\n\n\nUse as decorator\n----------------\n\nAlso library provide special decorator ``@async_reduceable()``, example:\n\n```python\nfrom async_reduce import async_reduceable\n\n\n@async_reduceable()\nasync def fetch_user_data(user_id: int) -> dict:\n    \"\"\"\" Get user data from inner service \"\"\"\n    url = 'http://inner-servicce/user/{}'.format(user_id)\n\n    return await http.get(url, timeout=10).json()\n\n\n@web_server.router('/users/(\\d+)')\nasync def handler_user_detail(request, user_id: int):\n    \"\"\" Handler for get detail information about user \"\"\"\n    return await fetch_user_data(user_id)\n```\n\n\nHooks\n-----\n\nLibrary supports hooks. Add-on hooks:\n\n* **DebugHooks** - print about all triggered hooks\n* **StatisticsOverallHooks** - general statistics on the use of `async_reduce`\n* **StatisticsDetailHooks** - like `StatisticsOverallHooks` but detail statistics\nabout all `coroutine` processed by `async_reduce`\n\nExample:\n\n```python\nfrom async_reduce import AsyncReducer\nfrom async_reduce.hooks import DebugHooks\n\n# define custom async_reduce with hooks\nasync_reduce = AsyncReducer(hooks=DebugHooks())\n\n\nasync def handler_user_detail(request, user_id: int):\n    user_data = await async_reduce(fetch_user_data(user_id))\n```\n\nSee more detail example in [examples/example_hooks.py](https://github.com/sirkonst/async-reduce/blob/master/examples/example_hooks.py).\n\nYou can write custom hooks via inherit from [BaseHooks](https://github.com/sirkonst/async-reduce/blob/master/async_reduce/hooks/base.py).\n\n\nCaveats\n-------\n\n* If single `coroutine` raises exceptions all aggregated `coroutine`s will get\nsame exception too\n\n* If single `coroutine` is stuck all aggregated `coroutine`s will stuck too.\nLimit execution time for `coroutine` and add retries (optional) to avoid it.\n\n* Be careful when return mutable value from `coroutine` because single value\nwill shared. Prefer to use non-mutable value as coroutine return.\n\n\nDevelopment\n-----------\n\nSee [DEVELOPMENT.md](https://github.com/sirkonst/async-reduce/blob/master/DEVELOPMENT.md).\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Reducer for similar simultaneously coroutines",
    "version": "1.4",
    "project_urls": {
        "Homepage": "https://github.com/sirkonst/async-reduce"
    },
    "split_keywords": [
        "asyncio",
        " reduce"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "1e09fa6b990a1c2f047230414ffc4a99504ff6a1f3f672bf109936deb313a9dd",
                "md5": "579563eb84b47b54913dfdf1c8994bc4",
                "sha256": "a94a95af5b4068241440bda4ceea9042dbb1798499656d49fece1d08e6c6b236"
            },
            "downloads": -1,
            "filename": "async_reduce-1.4-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "579563eb84b47b54913dfdf1c8994bc4",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.7",
            "size": 12240,
            "upload_time": "2025-01-18T18:27:23",
            "upload_time_iso_8601": "2025-01-18T18:27:23.983345Z",
            "url": "https://files.pythonhosted.org/packages/1e/09/fa6b990a1c2f047230414ffc4a99504ff6a1f3f672bf109936deb313a9dd/async_reduce-1.4-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "a881ec55c99bf95349fd8bc24a9cb36903c0a842c4662ff803720e65d7c37d87",
                "md5": "9a7b11e8cea59e06e6ee2e0301ce97c3",
                "sha256": "ce15bef1bd2b294f813f471c43a0f0fb7f77e27504e13b892dadd809430364d1"
            },
            "downloads": -1,
            "filename": "async_reduce-1.4.tar.gz",
            "has_sig": false,
            "md5_digest": "9a7b11e8cea59e06e6ee2e0301ce97c3",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.7",
            "size": 13023,
            "upload_time": "2025-01-18T18:27:25",
            "upload_time_iso_8601": "2025-01-18T18:27:25.857138Z",
            "url": "https://files.pythonhosted.org/packages/a8/81/ec55c99bf95349fd8bc24a9cb36903c0a842c4662ff803720e65d7c37d87/async_reduce-1.4.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2025-01-18 18:27:25",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "sirkonst",
    "github_project": "async-reduce",
    "travis_ci": false,
    "coveralls": true,
    "github_actions": false,
    "tox": true,
    "lcname": "async-reduce"
}
        
Elapsed time: 0.43570s