[](https://github.com/SETI/rms-filecache/releases)
[](https://github.com/SETI/rms-filecache/releases)
[](https://github.com/SETI/rms-filecache/actions)
[](https://rms-filecache.readthedocs.io/en/latest/?badge=latest)
[](https://codecov.io/gh/SETI/rms-filecache)
<br />
[](https://pypi.org/project/rms-filecache)
[](https://pypi.org/project/rms-filecache)
[](https://pypi.org/project/rms-filecache)
[](https://pypi.org/project/rms-filecache)
<br />
[](https://github.com/SETI/rms-filecache/commits/main/)
[](https://github.com/SETI/rms-filecache/commits/main/)
[](https://github.com/SETI/rms-filecache/commits/main/)
<br />
[](https://github.com/SETI/rms-filecache/issues)
[](https://github.com/SETI/rms-filecache/issues)
[](https://github.com/SETI/rms-filecache/pulls)
[](https://github.com/SETI/rms-filecache/pulls)
<br />
[](https://github.com/SETI/rms-filecache/stargazers)
# Introduction
`filecache` is a Python module that abstracts away the location where files used or
generated by a program are stored. Files can be on the local file system, in Google Cloud
Storage, on Amazon Web Services S3, or on a webserver. When files to be read are on the
local file system, they are simply accessed in-place. Otherwise, they are downloaded from
the remote source to a local temporary directory. When files to be written are on the
local file system, they are simply written in-place. Otherwise, they are written to a
local temporary directory and then uploaded to the remote location (it is not possible to
upload to a webserver). When a cache is no longer needed, it is deleted from the local
disk.
`filecache` is a product of the [PDS Ring-Moon Systems Node](https://pds-rings.seti.org).
# Installation
The `filecache` module is available via the `rms-filecache` package on PyPI and can be
installed with:
```sh
pip install rms-filecache
```
# Getting Started
The top-level file organization is provided by the `FileCache` class. A `FileCache`
instance is used to specify a particular **sharing policy** and **lifetime**. For example,
a cache could be private to the current process and group a set of files that all have the
same basic purpose. Once these files have been (downloaded and) read, they are deleted as
a group. Another cache could be shared among all processes on the current machine and
group a set of files that are needed by multiple processes, thus allowing them to be
downloaded from a remote source only one time, saving time and bandwidth.
A `FileCache` can be instantiated either directly or as a context manager. When
instantiated directly, the programmer is responsible for calling `FileCache.delete_cache`
directly to delete the cache when finished (a non-shared cache will be automatically
deleted on program exit). When instantiated as a context manager, a non-shared cache is
deleted on exit from the context. See the class documentation for full details.
Usage examples:
```python
from filecache import FileCache
# Create a cache with a unique name that will be deleted on exit
with FileCache(None) as fc: # Use as context manager
# Also use open() as a context manager
with fc.open('gs://rms-filecache-tests/subdir1/subdir2a/binary1.bin', 'rb',
anonymous=True) as fp:
bin1 = fp.read()
with fc.open('s3://rms-filecache-tests/subdir1/subdir2a/binary1.bin', 'rb',
anonymous=True) as fp:
bin2 = fp.read()
assert bin1 == bin2
# Cache automatically deleted here
fc = FileCache(None) # Use without context manager
# Also retrieve file without using open context manager
path1 = fc.retrieve('gs://rms-filecache-tests/subdir1/subdir2a/binary1.bin',
anonymous=True)
with open(path1, 'rb') as fp:
bin1 = fp.read()
path2 = fc.retrieve('s3://rms-filecache-tests/subdir1/subdir2a/binary1.bin',
anonymous=True)
with open(path2, 'rb') as fp:
bin2 = fp.read()
fc.delete_cache() # Cache manually deleted here
assert bin1 == bin2
# Write a file to a bucket and read it back
with FileCache(None) as fc:
with fc.open('gs://my-writable-bucket/output.txt', 'w') as fp:
fp.write('A')
# The cache will be deleted here so the file will have to be downloaded
with FileCache(None) as fc:
with fc.open('gs://my-writable-bucket/output.txt', 'r') as fp:
print(fp.read())
```
The `FCPath` class is a reimplementation of the Python `Path` class to support remote
acess using an associated `FileCache`. Like `Path`, an `FCPath` instance can contain any
part of a URI, but only an absolute URI can be used when actually accessing the file
specified by the `FCPath`. In addition, an `FCPath` can encapsulate various arguments such
as `anonymous` and `time_out` so that they do not need to be specified to each access
method. Thus, using this class can simplify the use of a `FileCache` by allowing the user
to operate on paths using the simpler syntax provided by `Path`, and to not specify
various other parameters at each method call site. If an `FCPath` instance is created
without an explicitly-associated `FileCache`, then the default `FileCache()` is used,
which specifies a shared cache named `"global"` that will persist after the program exits.
Compare this example to the one above:
```python
from filecache import FileCache, FCPath
# Create a cache with a unique name that will be deleted on exit
with FileCache(None) as fc: # Use as context manager
# Use GS by specifying the bucket name and one directory level
p1 = fc.new_path('gs://rms-filecache-tests/subdir1', anonymous=True)
# Use S3 by specifying the bucket name and two directory levels
# Alternative creation method
p2 = FCPath('s3://rms-filecache-tests/subdir1/subdir2a', filecache=fc,
anonymous=True)
# Access GS using a directory + filename (since only one directory level
# was specified by the FCPath)
# The additional directory and filename are specified as an argument to open()
# Also use open() as a context manager
with p1.open('subdir2a/binary1.bin', 'rb') as fp:
bin1 = fp.read()
# Access S3 using a filename only (since two directory levels were already
# specified by the FCPath)
# The additional filename is specified by using the / operator to create a new
# FCPath instance; anonymous=True is inherited
with (p2 / 'binary1.bin').open(mode='rb') as fp:
bin2 = fp.read()
assert bin1 == bin2
# Cache automatically deleted here
```
A benefit of the abstraction is that different environments can access the same files in
different ways without needing to change the program code. For example, consider a program
that needs to access the file ``COISS_2xxx/COISS_2001/voldesc.cat`` from the NASA PDS
archives. This file might be stored on the local disk in the user's home directory in a
subdirectory called ``pds3-holdings``. Or if the user does not have a local copy, it is
accessible from a webserver at
``https://pds-rings.seti.org/holdings/volumes/COISS_2xxx/COISS_2001/voldesc.cat``.
Finally, it could be accessible from Google Cloud Storage from the ``rms-node-holdings``
bucket at
``gs://rms-node-holdings/pds3-holdings/volumes/COISS_2xxx/COISS_2001/voldesc.cat``. Before
running the program, an environment variable could be set to one of these values:
```sh
$ export PDS3_HOLDINGS_SRC="~/pds3-holdings"
$ export PDS3_HOLDINGS_SRC="https://pds-rings.seti.org/holdings"
$ export PDS3_HOLDINGS_SRC="gs://rms-node-holdings/pds3-holdings"
```
Then the program could be written as:
```python
from filecache import FileCache
import os
with FileCache(None) as fc:
p = fc.new_path(os.getenv('PDS3_HOLDINGS_SRC'))
voldesc_path = p / 'volumes/COISS_2xxx/COISS_2001/voldesc.cat'
contents = voldesc_path.read_text()
# Cache automatically deleted here
```
If the program was going to be run multiple times in a row, or multiple copies were going
to be run simultaneously, using a shared cache would allow all of the processes to share
the same copy, thus requiring only a single download no matter how many times the program
was run. A shared cache is indicated by giving the cache a name (or no argument, which
defaults to ``"global"``); also `FCPath` defaults to using the global cache if no
`FileCache` is specified. This results in the simplest form of the program:
```python
from filecache import FCPath
import os
p = FCPath(os.getenv('PDS3_HOLDINGS_DIR'))
voldesc_path = p / 'volumes/COISS_2xxx/COISS_2001/voldesc.cat'
contents = voldesc_path.read_text()
```
Finally, there are four classes that allow direct access to the four possible storage
locations without invoking any caching behavior: `FileCacheSourceLocal`,
`FileCacheSourceHTTP`, `FileCacheSourceGS`, and `FileSourceCacheS3`:
```python
from filecache import FileCacheSourceGS
src = FileCacheSourceGS('gs://rms-filecache-tests', anonymous=True)
src.retrieve('subdir1/subdir2a/binary1.bin', 'local_file.bin')
```
Details of each class are available in the [module documentation](https://rms-filecache.readthedocs.io/en/latest/module.html).
# Contributing
Information on contributing to this package can be found in the
[Contributing Guide](https://github.com/SETI/rms-filecache/blob/main/CONTRIBUTING.md).
# Links
- [Documentation](https://rms-filecache.readthedocs.io)
- [Repository](https://github.com/SETI/rms-filecache)
- [Issue tracker](https://github.com/SETI/rms-filecache/issues)
- [PyPi](https://pypi.org/project/rms-filecache)
# Licensing
This code is licensed under the [Apache License v2.0](https://github.com/SETI/rms-filecache/blob/main/LICENSE).
Raw data
{
"_id": null,
"home_page": null,
"name": "rms-filecache",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "\"Robert S. French\" <rfrench@seti.org>",
"keywords": "Google Cloud Storage, Google Storage, Amazon Web Services, S3, file cache",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/d2/fd/9a0e8bb092a50a8b2116232ba7926f02eb2a4b067ded715fd967afe7ebe2/rms_filecache-2.3.0.tar.gz",
"platform": null,
"description": "[](https://github.com/SETI/rms-filecache/releases)\n[](https://github.com/SETI/rms-filecache/releases)\n[](https://github.com/SETI/rms-filecache/actions)\n[](https://rms-filecache.readthedocs.io/en/latest/?badge=latest)\n[](https://codecov.io/gh/SETI/rms-filecache)\n<br />\n[](https://pypi.org/project/rms-filecache)\n[](https://pypi.org/project/rms-filecache)\n[](https://pypi.org/project/rms-filecache)\n[](https://pypi.org/project/rms-filecache)\n<br />\n[](https://github.com/SETI/rms-filecache/commits/main/)\n[](https://github.com/SETI/rms-filecache/commits/main/)\n[](https://github.com/SETI/rms-filecache/commits/main/)\n<br />\n[](https://github.com/SETI/rms-filecache/issues)\n[](https://github.com/SETI/rms-filecache/issues)\n[](https://github.com/SETI/rms-filecache/pulls)\n[](https://github.com/SETI/rms-filecache/pulls)\n<br />\n\n[](https://github.com/SETI/rms-filecache/stargazers)\n\n\n# Introduction\n\n`filecache` is a Python module that abstracts away the location where files used or\ngenerated by a program are stored. Files can be on the local file system, in Google Cloud\nStorage, on Amazon Web Services S3, or on a webserver. When files to be read are on the\nlocal file system, they are simply accessed in-place. Otherwise, they are downloaded from\nthe remote source to a local temporary directory. When files to be written are on the\nlocal file system, they are simply written in-place. Otherwise, they are written to a\nlocal temporary directory and then uploaded to the remote location (it is not possible to\nupload to a webserver). When a cache is no longer needed, it is deleted from the local\ndisk.\n\n`filecache` is a product of the [PDS Ring-Moon Systems Node](https://pds-rings.seti.org).\n\n# Installation\n\nThe `filecache` module is available via the `rms-filecache` package on PyPI and can be\ninstalled with:\n\n```sh\npip install rms-filecache\n```\n\n# Getting Started\n\nThe top-level file organization is provided by the `FileCache` class. A `FileCache`\ninstance is used to specify a particular **sharing policy** and **lifetime**. For example,\na cache could be private to the current process and group a set of files that all have the\nsame basic purpose. Once these files have been (downloaded and) read, they are deleted as\na group. Another cache could be shared among all processes on the current machine and\ngroup a set of files that are needed by multiple processes, thus allowing them to be\ndownloaded from a remote source only one time, saving time and bandwidth.\n\nA `FileCache` can be instantiated either directly or as a context manager. When\ninstantiated directly, the programmer is responsible for calling `FileCache.delete_cache`\ndirectly to delete the cache when finished (a non-shared cache will be automatically\ndeleted on program exit). When instantiated as a context manager, a non-shared cache is\ndeleted on exit from the context. See the class documentation for full details.\n\nUsage examples:\n\n```python\nfrom filecache import FileCache\n# Create a cache with a unique name that will be deleted on exit\nwith FileCache(None) as fc: # Use as context manager\n # Also use open() as a context manager\n with fc.open('gs://rms-filecache-tests/subdir1/subdir2a/binary1.bin', 'rb',\n anonymous=True) as fp:\n bin1 = fp.read()\n with fc.open('s3://rms-filecache-tests/subdir1/subdir2a/binary1.bin', 'rb',\n anonymous=True) as fp:\n bin2 = fp.read()\n assert bin1 == bin2\n# Cache automatically deleted here\n\nfc = FileCache(None) # Use without context manager\n# Also retrieve file without using open context manager\npath1 = fc.retrieve('gs://rms-filecache-tests/subdir1/subdir2a/binary1.bin',\n anonymous=True)\nwith open(path1, 'rb') as fp:\n bin1 = fp.read()\npath2 = fc.retrieve('s3://rms-filecache-tests/subdir1/subdir2a/binary1.bin',\n anonymous=True)\nwith open(path2, 'rb') as fp:\n bin2 = fp.read()\nfc.delete_cache() # Cache manually deleted here\nassert bin1 == bin2\n\n# Write a file to a bucket and read it back\nwith FileCache(None) as fc:\n with fc.open('gs://my-writable-bucket/output.txt', 'w') as fp:\n fp.write('A')\n# The cache will be deleted here so the file will have to be downloaded\nwith FileCache(None) as fc:\n with fc.open('gs://my-writable-bucket/output.txt', 'r') as fp:\n print(fp.read())\n```\n\nThe `FCPath` class is a reimplementation of the Python `Path` class to support remote\nacess using an associated `FileCache`. Like `Path`, an `FCPath` instance can contain any\npart of a URI, but only an absolute URI can be used when actually accessing the file\nspecified by the `FCPath`. In addition, an `FCPath` can encapsulate various arguments such\nas `anonymous` and `time_out` so that they do not need to be specified to each access\nmethod. Thus, using this class can simplify the use of a `FileCache` by allowing the user\nto operate on paths using the simpler syntax provided by `Path`, and to not specify\nvarious other parameters at each method call site. If an `FCPath` instance is created\nwithout an explicitly-associated `FileCache`, then the default `FileCache()` is used,\nwhich specifies a shared cache named `\"global\"` that will persist after the program exits.\n\nCompare this example to the one above:\n\n```python\nfrom filecache import FileCache, FCPath\n# Create a cache with a unique name that will be deleted on exit\nwith FileCache(None) as fc: # Use as context manager\n # Use GS by specifying the bucket name and one directory level\n p1 = fc.new_path('gs://rms-filecache-tests/subdir1', anonymous=True)\n # Use S3 by specifying the bucket name and two directory levels\n # Alternative creation method\n p2 = FCPath('s3://rms-filecache-tests/subdir1/subdir2a', filecache=fc,\n anonymous=True)\n # Access GS using a directory + filename (since only one directory level\n # was specified by the FCPath)\n # The additional directory and filename are specified as an argument to open()\n # Also use open() as a context manager\n with p1.open('subdir2a/binary1.bin', 'rb') as fp:\n bin1 = fp.read()\n # Access S3 using a filename only (since two directory levels were already\n # specified by the FCPath)\n # The additional filename is specified by using the / operator to create a new\n # FCPath instance; anonymous=True is inherited\n with (p2 / 'binary1.bin').open(mode='rb') as fp:\n bin2 = fp.read()\n assert bin1 == bin2\n# Cache automatically deleted here\n```\n\nA benefit of the abstraction is that different environments can access the same files in\ndifferent ways without needing to change the program code. For example, consider a program\nthat needs to access the file ``COISS_2xxx/COISS_2001/voldesc.cat`` from the NASA PDS\narchives. This file might be stored on the local disk in the user's home directory in a\nsubdirectory called ``pds3-holdings``. Or if the user does not have a local copy, it is\naccessible from a webserver at\n``https://pds-rings.seti.org/holdings/volumes/COISS_2xxx/COISS_2001/voldesc.cat``.\nFinally, it could be accessible from Google Cloud Storage from the ``rms-node-holdings``\nbucket at\n``gs://rms-node-holdings/pds3-holdings/volumes/COISS_2xxx/COISS_2001/voldesc.cat``. Before\nrunning the program, an environment variable could be set to one of these values:\n\n```sh\n$ export PDS3_HOLDINGS_SRC=\"~/pds3-holdings\"\n$ export PDS3_HOLDINGS_SRC=\"https://pds-rings.seti.org/holdings\"\n$ export PDS3_HOLDINGS_SRC=\"gs://rms-node-holdings/pds3-holdings\"\n```\n\nThen the program could be written as:\n\n```python\nfrom filecache import FileCache\nimport os\nwith FileCache(None) as fc:\n p = fc.new_path(os.getenv('PDS3_HOLDINGS_SRC'))\n voldesc_path = p / 'volumes/COISS_2xxx/COISS_2001/voldesc.cat'\n contents = voldesc_path.read_text()\n# Cache automatically deleted here\n```\n\nIf the program was going to be run multiple times in a row, or multiple copies were going\nto be run simultaneously, using a shared cache would allow all of the processes to share\nthe same copy, thus requiring only a single download no matter how many times the program\nwas run. A shared cache is indicated by giving the cache a name (or no argument, which\ndefaults to ``\"global\"``); also `FCPath` defaults to using the global cache if no\n`FileCache` is specified. This results in the simplest form of the program:\n\n```python\nfrom filecache import FCPath\nimport os\np = FCPath(os.getenv('PDS3_HOLDINGS_DIR'))\nvoldesc_path = p / 'volumes/COISS_2xxx/COISS_2001/voldesc.cat'\ncontents = voldesc_path.read_text()\n```\n\nFinally, there are four classes that allow direct access to the four possible storage\nlocations without invoking any caching behavior: `FileCacheSourceLocal`,\n`FileCacheSourceHTTP`, `FileCacheSourceGS`, and `FileSourceCacheS3`:\n\n```python\nfrom filecache import FileCacheSourceGS\nsrc = FileCacheSourceGS('gs://rms-filecache-tests', anonymous=True)\nsrc.retrieve('subdir1/subdir2a/binary1.bin', 'local_file.bin')\n```\n\nDetails of each class are available in the [module documentation](https://rms-filecache.readthedocs.io/en/latest/module.html).\n\n# Contributing\n\nInformation on contributing to this package can be found in the\n[Contributing Guide](https://github.com/SETI/rms-filecache/blob/main/CONTRIBUTING.md).\n\n# Links\n\n- [Documentation](https://rms-filecache.readthedocs.io)\n- [Repository](https://github.com/SETI/rms-filecache)\n- [Issue tracker](https://github.com/SETI/rms-filecache/issues)\n- [PyPi](https://pypi.org/project/rms-filecache)\n\n# Licensing\n\nThis code is licensed under the [Apache License v2.0](https://github.com/SETI/rms-filecache/blob/main/LICENSE).\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "File cache for files retrieved from the cloud",
"version": "2.3.0",
"project_urls": {
"Documentation": "https://rms-filecache.readthedocs.io/en/latest",
"Homepage": "https://github.com/SETI/rms-filecache",
"Issues": "https://github.com/SETI/rms-filecache/issues",
"Repository": "https://github.com/SETI/rms-filecache",
"Source": "https://github.com/SETI/rms-filecache"
},
"split_keywords": [
"google cloud storage",
" google storage",
" amazon web services",
" s3",
" file cache"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d5b290785a63ef6251d26d5bcd7460a7b3ff5089648d93b1e6f4f5b02091495f",
"md5": "de6f9d6e1305d3077c023afbc25e4aec",
"sha256": "9d4f1ae34349d4ad49df3277646707cf19509d3aea023bdcf2dc45d8f7234632"
},
"downloads": -1,
"filename": "rms_filecache-2.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "de6f9d6e1305d3077c023afbc25e4aec",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 46440,
"upload_time": "2025-01-07T02:58:42",
"upload_time_iso_8601": "2025-01-07T02:58:42.607521Z",
"url": "https://files.pythonhosted.org/packages/d5/b2/90785a63ef6251d26d5bcd7460a7b3ff5089648d93b1e6f4f5b02091495f/rms_filecache-2.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d2fd9a0e8bb092a50a8b2116232ba7926f02eb2a4b067ded715fd967afe7ebe2",
"md5": "864a364f6ec675ef5179ea7ae194c0de",
"sha256": "7a13882973a2f5e2525be5e6bf9321f603fc0d03b66f32c140b35be35b3fb59e"
},
"downloads": -1,
"filename": "rms_filecache-2.3.0.tar.gz",
"has_sig": false,
"md5_digest": "864a364f6ec675ef5179ea7ae194c0de",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 209440,
"upload_time": "2025-01-07T02:58:45",
"upload_time_iso_8601": "2025-01-07T02:58:45.680274Z",
"url": "https://files.pythonhosted.org/packages/d2/fd/9a0e8bb092a50a8b2116232ba7926f02eb2a4b067ded715fd967afe7ebe2/rms_filecache-2.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-07 02:58:45",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "SETI",
"github_project": "rms-filecache",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"requirements": [
{
"name": "boto3",
"specs": []
},
{
"name": "boto3-stubs",
"specs": []
},
{
"name": "coverage",
"specs": []
},
{
"name": "filelock",
"specs": []
},
{
"name": "flake8",
"specs": []
},
{
"name": "google-api-python-client-stubs",
"specs": []
},
{
"name": "google-cloud-storage",
"specs": []
},
{
"name": "mypy",
"specs": []
},
{
"name": "myst-parser",
"specs": []
},
{
"name": "pytest",
"specs": []
},
{
"name": "pytest-order",
"specs": []
},
{
"name": "requests",
"specs": []
},
{
"name": "sphinx",
"specs": []
},
{
"name": "sphinxcontrib-napoleon",
"specs": []
},
{
"name": "sphinx-rtd-theme",
"specs": []
},
{
"name": "types-requests",
"specs": []
},
{
"name": "typing_extensions",
"specs": []
}
],
"lcname": "rms-filecache"
}
JSON
