A few caching data structures and other lossy things with capped sizes.
*Latest release 20250103*:
* Pull in @cachedmethod from cs.deco for dependency reasons.
* Move file_based and @file_property from cs.fileutils to cs.cache.
## <a name="cachedmethod"></a>`cachedmethod(*da, **dkw)`
Decorator to cache the result of an instance or class method
and keep a revision counter for changes.
The cached values are stored on the instance (`self`).
The revision counter supports the `@revised` decorator.
This decorator may be used in 2 modes.
Directly:
@cachedmethod
def method(self, ...)
or indirectly:
@cachedmethod(poll_delay=0.25)
def method(self, ...)
Optional keyword arguments:
* `attr_name`: the basis name for the supporting attributes.
Default: the name of the method.
* `poll_delay`: minimum time between polls; after the first
access, subsequent accesses before the `poll_delay` has elapsed
will return the cached value.
Default: `None`, meaning the value never becomes stale.
* `sig_func`: a signature function, which should be significantly
cheaper than the method. If the signature is unchanged, the
cached value will be returned. The signature function
expects the instance (`self`) as its first parameter.
Default: `None`, meaning no signature function;
the first computed value will be kept and never updated.
* `unset_value`: the value to return before the method has been
called successfully.
Default: `None`.
If the method raises an exception, this will be logged and
the method will return the previously cached value,
unless there is not yet a cached value
in which case the exception will be reraised.
If the signature function raises an exception
then a log message is issued and the signature is considered unchanged.
An example use of this decorator might be to keep a "live"
configuration data structure, parsed from a configuration
file which might be modified after the program starts. One
might provide a signature function which called `os.stat()` on
the file to check for changes before invoking a full read and
parse of the file.
*Note*: use of this decorator requires the `cs.pfx` module.
## <a name="CachingMapping"></a>Class `CachingMapping(cs.resources.MultiOpenMixin, collections.abc.MutableMapping)`
A caching front end for another mapping.
This is intended as a generic superclass for a proxy to a
slower mapping such as a database or remote key value store.
Note that this subclasses `MultiOpenMixin` to start/stop the worker `Thread`.
Users must enclose use of a `CachingMapping` in a `with` statement.
If subclasses also subclass `MultiOpenMixin` their `startup_shutdown`
method needs to also call our `startup_shutdown` method.
Example:
class Store:
""" A key value store with a slower backend.
"""
def __init__(self, mapping:Mapping):
self.mapping = CachingMapping(mapping)
.....
S = Store(slow_mapping)
with S:
... work with S ...
*`CachingMapping.__init__(self, mapping: Mapping, *, max_size=1024, queue_length=1024, delitem_bg: Optional[Callable[[Any], cs.result.Result]] = None, setitem_bg: Optional[Callable[[Any, Any], cs.result.Result]] = None, missing_fallthrough: bool = False)`*:
Initialise the cache.
Parameters:
* `mapping`: the backing store, a mapping
* `max_size`: optional maximum size for the cache, default 1024
* `queue_length`: option size for the queue to the worker, default 1024
* `delitem_bg`: optional callable to queue a delete of a
key in the backing store; if unset then deleted are
serialised in the worker thread
* `setitem_bg`: optional callable to queue setting the value
for a key in the backing store; if unset then deleted are
serialised in the worker thread
* `missing_fallthrough`: is true (default `False`) always
fall back to the backing mapping if a key is not in the cache
*`CachingMapping.flush(self)`*:
Wait for outstanding requests in the queue to complete.
Return the UNIX time of completion.
*`CachingMapping.items(self)`*:
Generator yielding `(k,v)` pairs.
*`CachingMapping.keys(self)`*:
Generator yielding the keys.
## <a name="ConvCache"></a>Class `ConvCache(cs.fs.HasFSPath)`
A cache for conversions of file contents such as thumbnails
or transcoded media, etc. This keeps cached results in a file
tree based on a content key, whose default function is
`cs.hashutils.file_checksum('blake3')`.
*`ConvCache.__init__(self, fspath: Optional[str] = None, content_key_func=None)`*:
Initialise a `ConvCache`.
Parameters:
* `fspath`: optional base path of the cache, default from
`ConvCache.DEFAULT_CACHE_BASEPATH`;
if this does not exist it will be created using `os.mkdir`
* `content_key_func`: optional function to compute a key
from the contents of a file, default `cs.hashindex.file_checksum`
(the blake3 hash of the contents)
*`ConvCache.content_key(self, srcpath)`*:
Return a content key for the filesystem path `srcpath`.
*`ConvCache.content_subpath(self, srcpath) -> str`*:
Return the content key based subpath component.
This default assumes the content key is a hash code and
breaks it hex representation into a 3 level hierarchy
such as `'d6/d9/c510785c468c9aa4b7bda343fb79'`.
*`ConvCache.convof(self, srcpath, conv_subpath, conv_func, *, ext=None, force=False) -> str`*:
Return the filesystem path of the cached conversion of
`srcpath` via `conv_func`.
Parameters:
* `srcpath`: the source filesystem path
* `conv_subpath`: a name for the conversion which encompasses
the salient aspaects such as `'png/64/64'` for a 64x64 pixel
thumbnail in PNG format
* `conv_func`: a callable of the form `conv_func(srcpath,dstpath)`
to convert the contents of `srcpath` and write the result
to the filesystem path `dstpath`
* `ext`: an optional filename extension, default from the
first component of `conv_subpath`
* `force`: option flag to require conversion even if the
cache has an entry
## <a name="convof"></a>`convof(srcpath, conv_subpath, conv_func, *, ext=None, force=False)`
`ConvCache.convof` using the default cache.
## <a name="file_based"></a>`file_based(*da, **dkw)`
A decorator which caches a value obtained from a file.
In addition to all the keyword arguments for `@cs.cache.cachedmethod`,
this decorator also accepts the following arguments:
* `attr_name`: the name for the associated attribute, used as
the basis for the internal cache value attribute
* `filename`: the filename to monitor.
Default from the `._{attr_name}__filename` attribute.
This value will be passed to the method as the `filename` keyword
parameter.
* `poll_delay`: delay between file polls, default `DEFAULT_POLL_INTERVAL`.
* `sig_func`: signature function used to encapsulate the relevant
information about the file; default
cs.filestate.FileState({filename}).
If the decorated function raises OSError with errno == ENOENT,
this returns None. Other exceptions are reraised.
## <a name="file_property"></a>`file_property(*da, **dkw)`
A property whose value reloads if a file changes.
## <a name="LRU_Cache"></a>Class `LRU_Cache`
A simple least recently used cache.
Unlike `functools.lru_cache`
this provides `on_add` and `on_remove` callbacks.
*`LRU_Cache.__init__(self, max_size, *, on_add=None, on_remove=None)`*:
Initialise the LRU_Cache with maximum size `max`,
additon callback `on_add` and removal callback `on_remove`.
*`LRU_Cache.__delitem__(self, key)`*:
Delete the specified `key`, calling the on_remove callback.
*`LRU_Cache.__setitem__(self, key, value)`*:
Store the item in the cache. Prune if necessary.
*`LRU_Cache.flush(self)`*:
Clear the cache.
*`LRU_Cache.get(self, key, default=None)`*:
Mapping method: get value for `key` or `default`.
*`LRU_Cache.items(self)`*:
Items from the cache.
*`LRU_Cache.keys(self)`*:
Keys from the cache.
## <a name="lru_cache"></a>`lru_cache(max_size=None, cache=None, on_add=None, on_remove=None)`
Enhanced workalike of @functools.lru_cache.
# Release Log
*Release 20250103*:
* Pull in @cachedmethod from cs.deco for dependency reasons.
* Move file_based and @file_property from cs.fileutils to cs.cache.
*Release 20241122*:
* convof, ConvCache.convof: new options force=False parameter to bypass existing cache entries.
* ConvCache.convof: check the conv_subpath using cs.fs.validate_rpath.
*Release 20240422.1*:
ConvCache docstring update.
*Release 20240422*:
New ConvCache and convof: a cache for conversions of file contents such as thumbnails or transcoded media.
*Release 20240412*:
* New CachingMapping, a caching front end for another mapping.
* LRU_Cache: add keys() and items().
*Release 20181228*:
Initial PyPI release.
Raw data
{
"_id": null,
"home_page": null,
"name": "cs-cache",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "python2, python3",
"author": null,
"author_email": "Cameron Simpson <cs@cskk.id.au>",
"download_url": "https://files.pythonhosted.org/packages/0c/9e/9b5c25867a480abb5c2ef1a3f4e93060af1e9b9148dcdc7e3f9b6d51cb81/cs_cache-20250103.tar.gz",
"platform": null,
"description": "A few caching data structures and other lossy things with capped sizes.\n\n*Latest release 20250103*:\n* Pull in @cachedmethod from cs.deco for dependency reasons.\n* Move file_based and @file_property from cs.fileutils to cs.cache.\n\n## <a name=\"cachedmethod\"></a>`cachedmethod(*da, **dkw)`\n\nDecorator to cache the result of an instance or class method\nand keep a revision counter for changes.\n\nThe cached values are stored on the instance (`self`).\nThe revision counter supports the `@revised` decorator.\n\nThis decorator may be used in 2 modes.\nDirectly:\n\n @cachedmethod\n def method(self, ...)\n\nor indirectly:\n\n @cachedmethod(poll_delay=0.25)\n def method(self, ...)\n\nOptional keyword arguments:\n* `attr_name`: the basis name for the supporting attributes.\n Default: the name of the method.\n* `poll_delay`: minimum time between polls; after the first\n access, subsequent accesses before the `poll_delay` has elapsed\n will return the cached value.\n Default: `None`, meaning the value never becomes stale.\n* `sig_func`: a signature function, which should be significantly\n cheaper than the method. If the signature is unchanged, the\n cached value will be returned. The signature function\n expects the instance (`self`) as its first parameter.\n Default: `None`, meaning no signature function;\n the first computed value will be kept and never updated.\n* `unset_value`: the value to return before the method has been\n called successfully.\n Default: `None`.\n\nIf the method raises an exception, this will be logged and\nthe method will return the previously cached value,\nunless there is not yet a cached value\nin which case the exception will be reraised.\n\nIf the signature function raises an exception\nthen a log message is issued and the signature is considered unchanged.\n\nAn example use of this decorator might be to keep a \"live\"\nconfiguration data structure, parsed from a configuration\nfile which might be modified after the program starts. One\nmight provide a signature function which called `os.stat()` on\nthe file to check for changes before invoking a full read and\nparse of the file.\n\n*Note*: use of this decorator requires the `cs.pfx` module.\n\n## <a name=\"CachingMapping\"></a>Class `CachingMapping(cs.resources.MultiOpenMixin, collections.abc.MutableMapping)`\n\nA caching front end for another mapping.\nThis is intended as a generic superclass for a proxy to a\nslower mapping such as a database or remote key value store.\n\nNote that this subclasses `MultiOpenMixin` to start/stop the worker `Thread`.\nUsers must enclose use of a `CachingMapping` in a `with` statement.\nIf subclasses also subclass `MultiOpenMixin` their `startup_shutdown`\nmethod needs to also call our `startup_shutdown` method.\n\nExample:\n\n class Store:\n \"\"\" A key value store with a slower backend.\n \"\"\"\n def __init__(self, mapping:Mapping):\n self.mapping = CachingMapping(mapping)\n\n .....\n S = Store(slow_mapping)\n with S:\n ... work with S ...\n\n*`CachingMapping.__init__(self, mapping: Mapping, *, max_size=1024, queue_length=1024, delitem_bg: Optional[Callable[[Any], cs.result.Result]] = None, setitem_bg: Optional[Callable[[Any, Any], cs.result.Result]] = None, missing_fallthrough: bool = False)`*:\nInitialise the cache.\n\nParameters:\n* `mapping`: the backing store, a mapping\n* `max_size`: optional maximum size for the cache, default 1024\n* `queue_length`: option size for the queue to the worker, default 1024\n* `delitem_bg`: optional callable to queue a delete of a\n key in the backing store; if unset then deleted are\n serialised in the worker thread\n* `setitem_bg`: optional callable to queue setting the value\n for a key in the backing store; if unset then deleted are\n serialised in the worker thread\n* `missing_fallthrough`: is true (default `False`) always\n fall back to the backing mapping if a key is not in the cache\n\n*`CachingMapping.flush(self)`*:\nWait for outstanding requests in the queue to complete.\nReturn the UNIX time of completion.\n\n*`CachingMapping.items(self)`*:\nGenerator yielding `(k,v)` pairs.\n\n*`CachingMapping.keys(self)`*:\nGenerator yielding the keys.\n\n## <a name=\"ConvCache\"></a>Class `ConvCache(cs.fs.HasFSPath)`\n\nA cache for conversions of file contents such as thumbnails\nor transcoded media, etc. This keeps cached results in a file\ntree based on a content key, whose default function is\n`cs.hashutils.file_checksum('blake3')`.\n\n*`ConvCache.__init__(self, fspath: Optional[str] = None, content_key_func=None)`*:\nInitialise a `ConvCache`.\n\nParameters:\n* `fspath`: optional base path of the cache, default from\n `ConvCache.DEFAULT_CACHE_BASEPATH`;\n if this does not exist it will be created using `os.mkdir`\n* `content_key_func`: optional function to compute a key\n from the contents of a file, default `cs.hashindex.file_checksum`\n (the blake3 hash of the contents)\n\n*`ConvCache.content_key(self, srcpath)`*:\nReturn a content key for the filesystem path `srcpath`.\n\n*`ConvCache.content_subpath(self, srcpath) -> str`*:\nReturn the content key based subpath component.\n\nThis default assumes the content key is a hash code and\nbreaks it hex representation into a 3 level hierarchy\nsuch as `'d6/d9/c510785c468c9aa4b7bda343fb79'`.\n\n*`ConvCache.convof(self, srcpath, conv_subpath, conv_func, *, ext=None, force=False) -> str`*:\nReturn the filesystem path of the cached conversion of\n`srcpath` via `conv_func`.\n\nParameters:\n* `srcpath`: the source filesystem path\n* `conv_subpath`: a name for the conversion which encompasses\n the salient aspaects such as `'png/64/64'` for a 64x64 pixel\n thumbnail in PNG format\n* `conv_func`: a callable of the form `conv_func(srcpath,dstpath)`\n to convert the contents of `srcpath` and write the result\n to the filesystem path `dstpath`\n* `ext`: an optional filename extension, default from the\n first component of `conv_subpath`\n* `force`: option flag to require conversion even if the\n cache has an entry\n\n## <a name=\"convof\"></a>`convof(srcpath, conv_subpath, conv_func, *, ext=None, force=False)`\n\n`ConvCache.convof` using the default cache.\n\n## <a name=\"file_based\"></a>`file_based(*da, **dkw)`\n\nA decorator which caches a value obtained from a file.\n\nIn addition to all the keyword arguments for `@cs.cache.cachedmethod`,\nthis decorator also accepts the following arguments:\n* `attr_name`: the name for the associated attribute, used as\n the basis for the internal cache value attribute\n* `filename`: the filename to monitor.\n Default from the `._{attr_name}__filename` attribute.\n This value will be passed to the method as the `filename` keyword\n parameter.\n* `poll_delay`: delay between file polls, default `DEFAULT_POLL_INTERVAL`.\n* `sig_func`: signature function used to encapsulate the relevant\n information about the file; default\n cs.filestate.FileState({filename}).\n\nIf the decorated function raises OSError with errno == ENOENT,\nthis returns None. Other exceptions are reraised.\n\n## <a name=\"file_property\"></a>`file_property(*da, **dkw)`\n\nA property whose value reloads if a file changes.\n\n## <a name=\"LRU_Cache\"></a>Class `LRU_Cache`\n\nA simple least recently used cache.\n\nUnlike `functools.lru_cache`\nthis provides `on_add` and `on_remove` callbacks.\n\n*`LRU_Cache.__init__(self, max_size, *, on_add=None, on_remove=None)`*:\nInitialise the LRU_Cache with maximum size `max`,\nadditon callback `on_add` and removal callback `on_remove`.\n\n*`LRU_Cache.__delitem__(self, key)`*:\nDelete the specified `key`, calling the on_remove callback.\n\n*`LRU_Cache.__setitem__(self, key, value)`*:\nStore the item in the cache. Prune if necessary.\n\n*`LRU_Cache.flush(self)`*:\nClear the cache.\n\n*`LRU_Cache.get(self, key, default=None)`*:\nMapping method: get value for `key` or `default`.\n\n*`LRU_Cache.items(self)`*:\nItems from the cache.\n\n*`LRU_Cache.keys(self)`*:\nKeys from the cache.\n\n## <a name=\"lru_cache\"></a>`lru_cache(max_size=None, cache=None, on_add=None, on_remove=None)`\n\nEnhanced workalike of @functools.lru_cache.\n\n# Release Log\n\n\n\n*Release 20250103*:\n* Pull in @cachedmethod from cs.deco for dependency reasons.\n* Move file_based and @file_property from cs.fileutils to cs.cache.\n\n*Release 20241122*:\n* convof, ConvCache.convof: new options force=False parameter to bypass existing cache entries.\n* ConvCache.convof: check the conv_subpath using cs.fs.validate_rpath.\n\n*Release 20240422.1*:\nConvCache docstring update.\n\n*Release 20240422*:\nNew ConvCache and convof: a cache for conversions of file contents such as thumbnails or transcoded media.\n\n*Release 20240412*:\n* New CachingMapping, a caching front end for another mapping.\n* LRU_Cache: add keys() and items().\n\n*Release 20181228*:\nInitial PyPI release.\n",
"bugtrack_url": null,
"license": "GNU General Public License v3 or later (GPLv3+)",
"summary": "A few caching data structures and other lossy things with capped sizes.",
"version": "20250103",
"project_urls": {
"MonoRepo Commits": "https://bitbucket.org/cameron_simpson/css/commits/branch/main",
"Monorepo Git Mirror": "https://github.com/cameron-simpson/css",
"Monorepo Hg/Mercurial Mirror": "https://hg.sr.ht/~cameron-simpson/css",
"Source": "https://github.com/cameron-simpson/css/blob/main/lib/python/cs/cache.py"
},
"split_keywords": [
"python2",
" python3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a9faa0d871016f374cfd4f35cb7abc76d053907301eba6a5aefab341ff043b92",
"md5": "70c8fd8a7fda35659a0525d48ba09f9e",
"sha256": "1896db6ab21fc9aaf7cc590a665c98d8950060d22adf18da705b99b2e7906657"
},
"downloads": -1,
"filename": "cs_cache-20250103-py3-none-any.whl",
"has_sig": false,
"md5_digest": "70c8fd8a7fda35659a0525d48ba09f9e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 12184,
"upload_time": "2025-01-03T04:17:36",
"upload_time_iso_8601": "2025-01-03T04:17:36.989808Z",
"url": "https://files.pythonhosted.org/packages/a9/fa/a0d871016f374cfd4f35cb7abc76d053907301eba6a5aefab341ff043b92/cs_cache-20250103-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0c9e9b5c25867a480abb5c2ef1a3f4e93060af1e9b9148dcdc7e3f9b6d51cb81",
"md5": "41bfaf6046c13c48edb61f39486893c4",
"sha256": "c2ca8142fd0b1642ce272557f9c5417e8dba8af4d3e64e5c388544f6837de289"
},
"downloads": -1,
"filename": "cs_cache-20250103.tar.gz",
"has_sig": false,
"md5_digest": "41bfaf6046c13c48edb61f39486893c4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 12364,
"upload_time": "2025-01-03T04:17:41",
"upload_time_iso_8601": "2025-01-03T04:17:41.551202Z",
"url": "https://files.pythonhosted.org/packages/0c/9e/9b5c25867a480abb5c2ef1a3f4e93060af1e9b9148dcdc7e3f9b6d51cb81/cs_cache-20250103.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-03 04:17:41",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "cameron-simpson",
"github_project": "css",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "cs-cache"
}
JSON
