fw-storage


Namefw-storage JSON
Version 3.4.2 PyPI version JSON
download
home_pagehttps://gitlab.com/flywheel-io/tools/lib/fw-storage
SummaryUnified storage interface.
upload_time2024-03-05 14:50:31
maintainer
docs_urlNone
authorFlywheel
requires_python>=3.8,<4.0
licenseMIT
keywords flywheel file object storage
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # fw-storage

Unified file storage interface tuned for simple filtering, memory-efficiency and
performance to support processing large datasets in Flywheel imports and exports.

Supported storage backends:

- `fs://` - Local file-system
- `s3://` - Amazon S3
- `gs://` - Google Cloud Storage
- `az://` - Azure Blob Storage

## Installation

Add as a `poetry` dependency to your project:

```bash
poetry add fw-storage
```

The local file-system storage backend has no extra package dependencies. To use
any other backend, though, `fw-storage` needs to be installed with extras. For
example, in order to access S3:

```bash
poetry add fw-storage --extras s3
```

Alternatively, install with `all` extras to get everything:

```bash
poetry add fw-storage --extras all
```

## Usage

```python
from fw_storage import create_storage_client

# instantiate storage with URL
fs = create_storage_client("fs:///tmp")

# set objects from bytes, filepaths or open files
fs.set("test/file1.dat", b"content")
fs.set("test/file2.dat", "/tmp/test/file1.dat")
fs.set("test/file3.dat", open("/tmp/test/file2.dat"))

# list objects, filtering with expressions
files = list(fs.ls("test", include=["size<1kB"], exclude=["path!~file3"]))
len(files) == 2

# get object info with path, size, created and modified
info = fs.stat("test/file1.dat")
info.size == 7

# read object contents
file = fs.get("test/file1.dat")
file.read() == b"content"

# remove one or more objects
fs.rm("test", recurse=True)
```

## Configuration

Credentials for cloud storage providers are loaded using the vendor SDKs to
support every standard config file location and environment variable recommended
by the provider:

| Storage | Config docs |
| ------- | ----------- |
| `s3://` | <https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html>|
| `gs://` | <https://google-auth.readthedocs.io/en/latest/reference/google.auth.html>|
| `az://` | <https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential>|

In addition, `az://` can be configured with the envvar `AZ_ACCESS_KEY`.

## Development

Install the project using `poetry` and enable `pre-commit`:

```bash
poetry install -E all
pre-commit install
```

## License

[![MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

            

Raw data

            {
    "_id": null,
    "home_page": "https://gitlab.com/flywheel-io/tools/lib/fw-storage",
    "name": "fw-storage",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.8,<4.0",
    "maintainer_email": "",
    "keywords": "Flywheel,file,object,storage",
    "author": "Flywheel",
    "author_email": "support@flywheel.io",
    "download_url": "",
    "platform": null,
    "description": "# fw-storage\n\nUnified file storage interface tuned for simple filtering, memory-efficiency and\nperformance to support processing large datasets in Flywheel imports and exports.\n\nSupported storage backends:\n\n- `fs://` - Local file-system\n- `s3://` - Amazon S3\n- `gs://` - Google Cloud Storage\n- `az://` - Azure Blob Storage\n\n## Installation\n\nAdd as a `poetry` dependency to your project:\n\n```bash\npoetry add fw-storage\n```\n\nThe local file-system storage backend has no extra package dependencies. To use\nany other backend, though, `fw-storage` needs to be installed with extras. For\nexample, in order to access S3:\n\n```bash\npoetry add fw-storage --extras s3\n```\n\nAlternatively, install with `all` extras to get everything:\n\n```bash\npoetry add fw-storage --extras all\n```\n\n## Usage\n\n```python\nfrom fw_storage import create_storage_client\n\n# instantiate storage with URL\nfs = create_storage_client(\"fs:///tmp\")\n\n# set objects from bytes, filepaths or open files\nfs.set(\"test/file1.dat\", b\"content\")\nfs.set(\"test/file2.dat\", \"/tmp/test/file1.dat\")\nfs.set(\"test/file3.dat\", open(\"/tmp/test/file2.dat\"))\n\n# list objects, filtering with expressions\nfiles = list(fs.ls(\"test\", include=[\"size<1kB\"], exclude=[\"path!~file3\"]))\nlen(files) == 2\n\n# get object info with path, size, created and modified\ninfo = fs.stat(\"test/file1.dat\")\ninfo.size == 7\n\n# read object contents\nfile = fs.get(\"test/file1.dat\")\nfile.read() == b\"content\"\n\n# remove one or more objects\nfs.rm(\"test\", recurse=True)\n```\n\n## Configuration\n\nCredentials for cloud storage providers are loaded using the vendor SDKs to\nsupport every standard config file location and environment variable recommended\nby the provider:\n\n| Storage | Config docs |\n| ------- | ----------- |\n| `s3://` | <https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html>|\n| `gs://` | <https://google-auth.readthedocs.io/en/latest/reference/google.auth.html>|\n| `az://` | <https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential>|\n\nIn addition, `az://` can be configured with the envvar `AZ_ACCESS_KEY`.\n\n## Development\n\nInstall the project using `poetry` and enable `pre-commit`:\n\n```bash\npoetry install -E all\npre-commit install\n```\n\n## License\n\n[![MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Unified storage interface.",
    "version": "3.4.2",
    "project_urls": {
        "Documentation": "https://gitlab.com/flywheel-io/tools/lib/fw-storage",
        "Homepage": "https://gitlab.com/flywheel-io/tools/lib/fw-storage",
        "Repository": "https://gitlab.com/flywheel-io/tools/lib/fw-storage"
    },
    "split_keywords": [
        "flywheel",
        "file",
        "object",
        "storage"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "76d72bc31eb8b34380b65fb4d0382d973f9b323edf072ae3aed906f2c7a85d4a",
                "md5": "5e71c2abeb693a2773947aa947d876b4",
                "sha256": "19a0af5a176efdb24174e89226f8b614eed7f0b651067f332a9757a35d0f306d"
            },
            "downloads": -1,
            "filename": "fw_storage-3.4.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "5e71c2abeb693a2773947aa947d876b4",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.8,<4.0",
            "size": 35009,
            "upload_time": "2024-03-05T14:50:31",
            "upload_time_iso_8601": "2024-03-05T14:50:31.116483Z",
            "url": "https://files.pythonhosted.org/packages/76/d7/2bc31eb8b34380b65fb4d0382d973f9b323edf072ae3aed906f2c7a85d4a/fw_storage-3.4.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-03-05 14:50:31",
    "github": false,
    "gitlab": true,
    "bitbucket": false,
    "codeberg": false,
    "gitlab_user": "flywheel-io",
    "gitlab_project": "tools",
    "lcname": "fw-storage"
}
        
Elapsed time: 0.21743s