# RedisCache
## Presentation
There are already quite a few Python decorators to cache functions in a Redis database:
- [redis-cache](https://pypi.org/project/redis-cache/)
- [redis_cache_decorator](https://pypi.org/project/redis_cache_decorator/)
- [redis-simple-cache](https://pypi.org/project/redis-simple-cache/)
- [python-redis-cache](https://pypi.org/project/python-redis-cache/)
- [redis-simple-cache-3k](https://pypi.org/project/redis-simple-cache-3k/)
- [redis-simple-cache-py3](https://pypi.org/project/redis-simple-cache-py3/)
- and more ...
But none I could find allows to set two expiration times as we do it here. The first given time is how long before we should update the value stored in the cache. The second given time, longer of course, is how long the data stored in the cache is still good enough to be sent back to the caller. The refreshing of the cache is only done when the function is called. And by default it is done asynchronously, so the caller doesn't have to wait. When the data in the cache becomes too old, it disappear automatically.
This is a great caching mechanism for functions that will give a consistent output according to their parameters and at a given time. A purely random function should not be cached. And a function that is independent of the time should be cached with a different mechanism like the LRU cache in the [functools](https://docs.python.org/3/library/functools.html) standard module.
## Installation
Simply install the PyPi package:
```bash
pip install rediscache
```
## Requirements
Of course you need a Redis server installed. By default, the decorator will connect to `localhost:6379` with no password, using the database number `0`. This can be changed with parameters given to the `RedisCache` object.
## Usage of the RedisCache class
### RedisCache class
To avoid having too many connections to the Redis server, it is best to create only one instance of this class.
```python
rediscache = RedisCache()
```
All the parameters for the `RedisCache` constructor are optional. Their default value are in `[]`.
- host: IP or host name of the Redis server. [`'localhost'`]
- port: Port number of the Redis server. [`6379`]
- db: Database number in the Redis server. [`0`]
- password: Password required to read and write on the Redis server. [`None`]
- decode: Decode the data stored in the cache as byte string. For example, it should not be done if you actually want to cache byte strings. [`True`]
- enabled: When False it allows to programmatically disable the cache. It can be useful for unit tests. [`True`]
### Environment variables
In the case of a cloud deployment, for example, it might be easier to use environment variables to set the Redis server details:
- REDIS_SERVICE_HOST: IP or host name of the Redis server.
- REDIS_SERVICE_PORT: Port number of the Redis server.
- REDIS_SERVICE_DB: Database number in the Redis server.
- REDIS_SERVICE_PASSWORD: Password required to read and write on the Redis server.
The order of priority is the natural _parameter_ > _environment variable_ > _default value_.
### `cache` decorator
This is the main decorator. All the parameters are available. The mandatory ones do not have a default value:
- refresh: The amount of seconds before it would be a good idea to refresh the cached value.
- expire: How many seconds that the value in the cache is still considered good enough to be sent back to the caller.
- retry: While a value is being refreshed, we want to avoid to refresh it in parallel. But if it is taking too long, after the number of seconds provided here, we may want to try our luck again. If not specified, we will take the `refresh` value.
- default: If we do not have the value in the cache and we do not want to wait, what shall we send back to the caller? It has to be serializable because it will also be stored in the cache. [`''`]
- wait: If the value is not in the cache, do we wait for the return of the function? [`False`]
- serializer: The only type of data that can be stored directly in the Redis database are `byte`, `str`, `int` and `float`. Any other will have to be serialized with the function provided here. [`None`]
- deserializer: If the value was serialized to be stored in the cache, it needs to deserialized when it is retrieved. [`None`]
- use_args: This is the list of positional parameters (a list of integers) to be taken into account to generate the key that will be used in Redis. If `None`, they will all be used. [`None`]
- use_kwargs: This is the list of named parameters (a list of names) to be taken into account to generate the key that will be used in Redis. If `None`, they will all be used. [`None`]
Example:
```python
from rediscache import RedisCache
REDISCACHE = RedisCache()
@REDISCACHE.cache(10, 60) # Keep the value up to 1mn but ready to be refreshed every 10s.
def my_function(...) {
...
}
```
See `test_rediscache.py` for more examples.
Note: when you choose to wait for the value, you do not have an absolute guarantee that you will not get the default value. For example if it takes more than the retry time to get an answer from the function, the decorator will give up.
### `get_stats(delete=False)`
This will get the stats stored when using the cache. The `delete` option is to reset the counters after read.
The output is a dictionary with the following keys and values:
- **Refresh**: Number of times the cached function was actually called.
- **Wait**: Number of times we waited for the result when executing the function.
- **Sleep**: Number of 1 seconds we waited for the results to be found in the cache.
- **Failed**: Number of times the cached function raised an exception when called.
- **Missed**: Number of times the functions result was not found in the cache.
- **Success**: Number of times the function's result was found in the cache.
- **Default**: Number of times the default value was used because nothing is in the cache or the function failed.
### The `function` property
The decorator and its aliases add a new property to the decorated function to be able to bypass the cache as it may be required
in some cases.
```python
from rediscache import RedisCache
REDISCACHE = RedisCache()
@REDISCACHE.cache(2, 10)
def myfunc():
return "Hello"
# Invoke the function without caching it.
print(myfunc.function())
```
## The `decorate` decorator
In the `tools` submodule, the `decorate` decorator is a little helper to transform a serializer or deserializer function into a decorator.
The transformation function is expected to take a single argument of a certain type and transform it into another type.
For example, a typical serializer would transform a dictionary into a string.
```python
from json import dumps
from rediscache.tools import decorate
@decorate(dumps)
def myfunc():
return {"toto": 42}
assert isinstance(myfunc(), str)
```
You may use partial functions if your transformation function requires extra parameters.
```python
from datetime import date
from functools import partial
from json import dumps
from rediscache.tools import decorate
@decorate(partial(dumps, skipkeys=True))
def func_with_id():
"""My func_with_id"""
return {"name": "Toto", "age": 25, date.today(): "today"}
assert func_with_id() == '{"name": "Toto", "age": 25}'
```
## Development
### Poetry
My development environment is handled by Poetry. I use `Python 3.11.7`.
### Testing
To make sure we use Redis properly, we do not mock it in the unit tess. So you will need a localhost default instance of Redis server without a password. This means that the unit tests are more like integration tests.
The execution of the tests including coverage result can be done with `test.sh`. You can also run just `pytest`:
```bash
./test.sh
```
## CI/CD
### Workflow
We use the GitHub workflow to check each new commit. See `.github/workflows/python-package.yaml`.
We get help from re-usable actions. Here is the [Marketplace](https://github.com/marketplace?type=actions).
- [Checkout](https://github.com/marketplace/actions/checkout)
- [Install Poetry Action](https://github.com/marketplace/actions/install-poetry-action)
- [Setup Python](https://github.com/marketplace/actions/setup-python)
### Publish to PyPI
For the moment the publish to PyPI is done manually with the `publish.sh` script. You will need a PyPI API token in `PYPI_API_TOKEN`, stored in a `secrets.sh`.
Raw data
{
"_id": null,
"home_page": "https://github.com/AmadeusITGroup/RedisCache",
"name": "rediscache",
"maintainer": "Amadeus IT Group",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "opensource@amadeus.com",
"keywords": "redis,performance,cache",
"author": "Pierre Cart-Grandjean",
"author_email": "pcart-grandjean@amadeus.com",
"download_url": "https://files.pythonhosted.org/packages/36/53/544624b913dbe58587a1e7e9291bf67cd8ac6a09e28dc75318b951e16eff/rediscache-1.0.0.tar.gz",
"platform": null,
"description": "# RedisCache\n\n## Presentation\n\nThere are already quite a few Python decorators to cache functions in a Redis database:\n\n- [redis-cache](https://pypi.org/project/redis-cache/)\n- [redis_cache_decorator](https://pypi.org/project/redis_cache_decorator/)\n- [redis-simple-cache](https://pypi.org/project/redis-simple-cache/)\n- [python-redis-cache](https://pypi.org/project/python-redis-cache/)\n- [redis-simple-cache-3k](https://pypi.org/project/redis-simple-cache-3k/)\n- [redis-simple-cache-py3](https://pypi.org/project/redis-simple-cache-py3/)\n- and more ...\n\nBut none I could find allows to set two expiration times as we do it here. The first given time is how long before we should update the value stored in the cache. The second given time, longer of course, is how long the data stored in the cache is still good enough to be sent back to the caller. The refreshing of the cache is only done when the function is called. And by default it is done asynchronously, so the caller doesn't have to wait. When the data in the cache becomes too old, it disappear automatically.\n\nThis is a great caching mechanism for functions that will give a consistent output according to their parameters and at a given time. A purely random function should not be cached. And a function that is independent of the time should be cached with a different mechanism like the LRU cache in the [functools](https://docs.python.org/3/library/functools.html) standard module.\n\n## Installation\n\nSimply install the PyPi package:\n\n```bash\npip install rediscache\n```\n\n## Requirements\n\nOf course you need a Redis server installed. By default, the decorator will connect to `localhost:6379` with no password, using the database number `0`. This can be changed with parameters given to the `RedisCache` object.\n\n## Usage of the RedisCache class\n\n### RedisCache class\n\nTo avoid having too many connections to the Redis server, it is best to create only one instance of this class.\n\n```python\nrediscache = RedisCache()\n```\n\nAll the parameters for the `RedisCache` constructor are optional. Their default value are in `[]`.\n\n- host: IP or host name of the Redis server. [`'localhost'`]\n- port: Port number of the Redis server. [`6379`]\n- db: Database number in the Redis server. [`0`]\n- password: Password required to read and write on the Redis server. [`None`]\n- decode: Decode the data stored in the cache as byte string. For example, it should not be done if you actually want to cache byte strings. [`True`]\n- enabled: When False it allows to programmatically disable the cache. It can be useful for unit tests. [`True`]\n\n### Environment variables\n\nIn the case of a cloud deployment, for example, it might be easier to use environment variables to set the Redis server details:\n\n- REDIS_SERVICE_HOST: IP or host name of the Redis server.\n- REDIS_SERVICE_PORT: Port number of the Redis server.\n- REDIS_SERVICE_DB: Database number in the Redis server.\n- REDIS_SERVICE_PASSWORD: Password required to read and write on the Redis server.\n\nThe order of priority is the natural _parameter_ > _environment variable_ > _default value_.\n\n### `cache` decorator\n\nThis is the main decorator. All the parameters are available. The mandatory ones do not have a default value:\n\n- refresh: The amount of seconds before it would be a good idea to refresh the cached value.\n- expire: How many seconds that the value in the cache is still considered good enough to be sent back to the caller.\n- retry: While a value is being refreshed, we want to avoid to refresh it in parallel. But if it is taking too long, after the number of seconds provided here, we may want to try our luck again. If not specified, we will take the `refresh` value.\n- default: If we do not have the value in the cache and we do not want to wait, what shall we send back to the caller? It has to be serializable because it will also be stored in the cache. [`''`]\n- wait: If the value is not in the cache, do we wait for the return of the function? [`False`]\n- serializer: The only type of data that can be stored directly in the Redis database are `byte`, `str`, `int` and `float`. Any other will have to be serialized with the function provided here. [`None`]\n- deserializer: If the value was serialized to be stored in the cache, it needs to deserialized when it is retrieved. [`None`]\n- use_args: This is the list of positional parameters (a list of integers) to be taken into account to generate the key that will be used in Redis. If `None`, they will all be used. [`None`]\n- use_kwargs: This is the list of named parameters (a list of names) to be taken into account to generate the key that will be used in Redis. If `None`, they will all be used. [`None`]\n\nExample:\n\n```python\nfrom rediscache import RedisCache\nREDISCACHE = RedisCache()\n@REDISCACHE.cache(10, 60) # Keep the value up to 1mn but ready to be refreshed every 10s.\ndef my_function(...) {\n ...\n}\n```\n\nSee `test_rediscache.py` for more examples.\n\nNote: when you choose to wait for the value, you do not have an absolute guarantee that you will not get the default value. For example if it takes more than the retry time to get an answer from the function, the decorator will give up.\n\n### `get_stats(delete=False)`\n\nThis will get the stats stored when using the cache. The `delete` option is to reset the counters after read.\nThe output is a dictionary with the following keys and values:\n\n- **Refresh**: Number of times the cached function was actually called.\n- **Wait**: Number of times we waited for the result when executing the function.\n- **Sleep**: Number of 1 seconds we waited for the results to be found in the cache.\n- **Failed**: Number of times the cached function raised an exception when called.\n- **Missed**: Number of times the functions result was not found in the cache.\n- **Success**: Number of times the function's result was found in the cache.\n- **Default**: Number of times the default value was used because nothing is in the cache or the function failed.\n\n### The `function` property\n\nThe decorator and its aliases add a new property to the decorated function to be able to bypass the cache as it may be required\nin some cases.\n\n```python\nfrom rediscache import RedisCache\nREDISCACHE = RedisCache()\n@REDISCACHE.cache(2, 10)\ndef myfunc():\n return \"Hello\"\n# Invoke the function without caching it.\nprint(myfunc.function())\n```\n\n## The `decorate` decorator\n\nIn the `tools` submodule, the `decorate` decorator is a little helper to transform a serializer or deserializer function into a decorator.\nThe transformation function is expected to take a single argument of a certain type and transform it into another type.\nFor example, a typical serializer would transform a dictionary into a string.\n\n```python\nfrom json import dumps\n\nfrom rediscache.tools import decorate\n\n@decorate(dumps)\ndef myfunc():\n return {\"toto\": 42}\n\nassert isinstance(myfunc(), str)\n```\n\nYou may use partial functions if your transformation function requires extra parameters.\n\n```python\nfrom datetime import date\nfrom functools import partial\nfrom json import dumps\n\nfrom rediscache.tools import decorate\n\n@decorate(partial(dumps, skipkeys=True))\ndef func_with_id():\n \"\"\"My func_with_id\"\"\"\n return {\"name\": \"Toto\", \"age\": 25, date.today(): \"today\"}\n\nassert func_with_id() == '{\"name\": \"Toto\", \"age\": 25}'\n```\n\n## Development\n\n### Poetry\n\nMy development environment is handled by Poetry. I use `Python 3.11.7`.\n\n### Testing\n\nTo make sure we use Redis properly, we do not mock it in the unit tess. So you will need a localhost default instance of Redis server without a password. This means that the unit tests are more like integration tests.\n\nThe execution of the tests including coverage result can be done with `test.sh`. You can also run just `pytest`:\n\n```bash\n./test.sh\n```\n\n## CI/CD\n\n### Workflow\n\nWe use the GitHub workflow to check each new commit. See `.github/workflows/python-package.yaml`.\n\nWe get help from re-usable actions. Here is the [Marketplace](https://github.com/marketplace?type=actions).\n\n- [Checkout](https://github.com/marketplace/actions/checkout)\n- [Install Poetry Action](https://github.com/marketplace/actions/install-poetry-action)\n- [Setup Python](https://github.com/marketplace/actions/setup-python)\n\n### Publish to PyPI\n\nFor the moment the publish to PyPI is done manually with the `publish.sh` script. You will need a PyPI API token in `PYPI_API_TOKEN`, stored in a `secrets.sh`.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Redis caching of functions evolving over time",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/AmadeusITGroup/RedisCache",
"Repository": "https://github.com/AmadeusITGroup/RedisCache"
},
"split_keywords": [
"redis",
"performance",
"cache"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3285b6a8601ec96b8206b49c669251fbb8416d90fe3544668e5fdc55a17f7c1b",
"md5": "e5d7714c9091bb3c4aa3c4e04e098fc9",
"sha256": "b58fa0e9d834c2a7681dc11ffc7e3967aca51b93461bbc9bbbc2cbdd4dc96c2f"
},
"downloads": -1,
"filename": "rediscache-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e5d7714c9091bb3c4aa3c4e04e098fc9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 9550,
"upload_time": "2024-02-25T21:54:45",
"upload_time_iso_8601": "2024-02-25T21:54:45.575473Z",
"url": "https://files.pythonhosted.org/packages/32/85/b6a8601ec96b8206b49c669251fbb8416d90fe3544668e5fdc55a17f7c1b/rediscache-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3653544624b913dbe58587a1e7e9291bf67cd8ac6a09e28dc75318b951e16eff",
"md5": "f47c8bdfd0ba301881d55da26f330956",
"sha256": "f18b57cbcaf703480a1ea808ab915d1593238a1da78bbc2cff53d01848c7cb9f"
},
"downloads": -1,
"filename": "rediscache-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "f47c8bdfd0ba301881d55da26f330956",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 8797,
"upload_time": "2024-02-25T21:54:47",
"upload_time_iso_8601": "2024-02-25T21:54:47.296821Z",
"url": "https://files.pythonhosted.org/packages/36/53/544624b913dbe58587a1e7e9291bf67cd8ac6a09e28dc75318b951e16eff/rediscache-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-25 21:54:47",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "AmadeusITGroup",
"github_project": "RedisCache",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "rediscache"
}
JSON
