# fw-storage
Unified file storage interface tuned for simple filtering, memory-efficiency and
performance to support processing large datasets in Flywheel imports and exports.
Supported storage backends:
- `fs://` - Local file-system
- `s3://` - Amazon S3
- `gs://` - Google Cloud Storage
- `az://` - Azure Blob Storage
## Installation
Add as a `poetry` dependency to your project:
```bash
poetry add fw-storage
```
The local file-system storage backend has no extra package dependencies. To use
any other backend, though, `fw-storage` needs to be installed with extras. For
example, in order to access S3:
```bash
poetry add fw-storage --extras s3
```
Alternatively, install with `all` extras to get everything:
```bash
poetry add fw-storage --extras all
```
## Usage
```python
from fw_storage import create_storage_client
# instantiate storage with URL
fs = create_storage_client("fs:///tmp")
# set objects from bytes, filepaths or open files
fs.set("test/file1.dat", b"content")
fs.set("test/file2.dat", "/tmp/test/file1.dat")
fs.set("test/file3.dat", open("/tmp/test/file2.dat"))
# list objects, filtering with expressions
files = list(fs.ls("test", include=["size<1kB"], exclude=["path!~file3"]))
len(files) == 2
# get object info with path, size, created and modified
info = fs.stat("test/file1.dat")
info.size == 7
# read object contents
file = fs.get("test/file1.dat")
file.read() == b"content"
# remove one or more objects
fs.rm("test", recurse=True)
```
## Configuration
Credentials for cloud storage providers are loaded using the vendor SDKs to
support every standard config file location and environment variable recommended
by the provider:
| Storage | Config docs |
| ------- | ----------- |
| `s3://` | <https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html>|
| `gs://` | <https://google-auth.readthedocs.io/en/latest/reference/google.auth.html>|
| `az://` | <https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential>|
In addition, `az://` can be configured with the envvar `AZ_ACCESS_KEY`.
## Development
Install the project using `poetry` and enable `pre-commit`:
```bash
poetry install -E all
pre-commit install
```
## License
[](LICENSE)
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/flywheel-io/tools/lib/fw-storage",
"name": "fw-storage",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "Flywheel, file, object, storage",
"author": "Flywheel",
"author_email": "support@flywheel.io",
"download_url": null,
"platform": null,
"description": "# fw-storage\n\nUnified file storage interface tuned for simple filtering, memory-efficiency and\nperformance to support processing large datasets in Flywheel imports and exports.\n\nSupported storage backends:\n\n- `fs://` - Local file-system\n- `s3://` - Amazon S3\n- `gs://` - Google Cloud Storage\n- `az://` - Azure Blob Storage\n\n## Installation\n\nAdd as a `poetry` dependency to your project:\n\n```bash\npoetry add fw-storage\n```\n\nThe local file-system storage backend has no extra package dependencies. To use\nany other backend, though, `fw-storage` needs to be installed with extras. For\nexample, in order to access S3:\n\n```bash\npoetry add fw-storage --extras s3\n```\n\nAlternatively, install with `all` extras to get everything:\n\n```bash\npoetry add fw-storage --extras all\n```\n\n## Usage\n\n```python\nfrom fw_storage import create_storage_client\n\n# instantiate storage with URL\nfs = create_storage_client(\"fs:///tmp\")\n\n# set objects from bytes, filepaths or open files\nfs.set(\"test/file1.dat\", b\"content\")\nfs.set(\"test/file2.dat\", \"/tmp/test/file1.dat\")\nfs.set(\"test/file3.dat\", open(\"/tmp/test/file2.dat\"))\n\n# list objects, filtering with expressions\nfiles = list(fs.ls(\"test\", include=[\"size<1kB\"], exclude=[\"path!~file3\"]))\nlen(files) == 2\n\n# get object info with path, size, created and modified\ninfo = fs.stat(\"test/file1.dat\")\ninfo.size == 7\n\n# read object contents\nfile = fs.get(\"test/file1.dat\")\nfile.read() == b\"content\"\n\n# remove one or more objects\nfs.rm(\"test\", recurse=True)\n```\n\n## Configuration\n\nCredentials for cloud storage providers are loaded using the vendor SDKs to\nsupport every standard config file location and environment variable recommended\nby the provider:\n\n| Storage | Config docs |\n| ------- | ----------- |\n| `s3://` | <https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html>|\n| `gs://` | <https://google-auth.readthedocs.io/en/latest/reference/google.auth.html>|\n| `az://` | <https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential>|\n\nIn addition, `az://` can be configured with the envvar `AZ_ACCESS_KEY`.\n\n## Development\n\nInstall the project using `poetry` and enable `pre-commit`:\n\n```bash\npoetry install -E all\npre-commit install\n```\n\n## License\n\n[](LICENSE)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Unified storage interface.",
"version": "3.5.1",
"project_urls": {
"Documentation": "https://gitlab.com/flywheel-io/tools/lib/fw-storage",
"Homepage": "https://gitlab.com/flywheel-io/tools/lib/fw-storage",
"Repository": "https://gitlab.com/flywheel-io/tools/lib/fw-storage"
},
"split_keywords": [
"flywheel",
" file",
" object",
" storage"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fd2111203ad4ff23c03f46e82d6252b46b250acecd109ac02dc0f0d93021d9ad",
"md5": "994627970d56a8af149a1483f2214edc",
"sha256": "e989e377319adc62286255355d9faa4f864cac974aaf1d3e979dcd17cbfdd5e0"
},
"downloads": -1,
"filename": "fw_storage-3.5.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "994627970d56a8af149a1483f2214edc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 35424,
"upload_time": "2024-10-02T07:59:06",
"upload_time_iso_8601": "2024-10-02T07:59:06.236566Z",
"url": "https://files.pythonhosted.org/packages/fd/21/11203ad4ff23c03f46e82d6252b46b250acecd109ac02dc0f0d93021d9ad/fw_storage-3.5.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-02 07:59:06",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "flywheel-io",
"gitlab_project": "tools",
"lcname": "fw-storage"
}
JSON