ezfs


Nameezfs JSON
Version 1.0.2 PyPI version JSON
download
home_pageNone
SummaryMinimalistic virtual filesystem adapters for Python
upload_time2024-05-04 14:21:43
maintainerNone
docs_urlNone
authorDavid Fritz
requires_python>=3.10
licenseMIT
keywords filesystem s3 compression
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            
[![os: windows mac linux](https://img.shields.io/badge/os-linux_|_macos_|_windows-blue)](https://docs.python.org/3.10/)
[![python: 3.10+](https://img.shields.io/badge/python-3.10_|_3.11_|_3.12-blue)](https://devguide.python.org/versions)
[![python style: google](https://img.shields.io/badge/python%20style-google-blue)](https://google.github.io/styleguide/pyguide.html)
[![imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://github.com/PyCQA/isort)
[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![code style: pycodestyle](https://img.shields.io/badge/code%20style-pycodestyle-green)](https://github.com/PyCQA/pycodestyle)
[![doc style: pydocstyle](https://img.shields.io/badge/doc%20style-pydocstyle-green)](https://github.com/PyCQA/pydocstyle)
[![static typing: mypy](https://img.shields.io/badge/static_typing-mypy-green)](https://github.com/python/mypy)
[![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/PyCQA/pylint)
[![testing: pytest](https://img.shields.io/badge/testing-pytest-yellowgreen)](https://github.com/pytest-dev/pytest)
[![security: bandit](https://img.shields.io/badge/security-bandit-black)](https://github.com/PyCQA/bandit)
[![license: MIT](https://img.shields.io/badge/license-MIT-lightgrey)](LICENSE)


# EZFS

EZFS (Easy Filesystem, or Everythingz a Filesystem) is an optimized, minimal dependency (down to 0),
virtual filesystem adapter library for Python. EZFS provides access to the most common operations
for "files" and "filesystems", while abstracting storage away from the user and developer, to simplify
both UX and DX. All you need to know is how to access files in Python, and EZFS will take care of the rest,
regardless of your selected backend.

In order to provide a streamlined UX and DX, EZFS leverages existing native Python designs and patterns,
such as open/read/write operations, and applies them all backend storage types. The learning curve is minimal:
if you know how to read/write a local file in Python, you also know how to read/write to any location in EZFS.

Reading and writing is supported for both text and binary files across local, remote, and memory "filesystems".
Additional compression types and storage types can be supported by extending the primary `File`, `Filesystem`,
and `Compressor` adapters. EZFS can also be faster than "native" open/read/write operations in some scenarios,
due to having a specialized focus. Refer to the compatibility guides, and "Why EZFS", for more information.


## Table Of Contents

  * [Compatibility](#compatibility)
  * [Getting Started](#getting-started)
    * [Installation](#installation)
  * [How Tos](#how-tos)
    * [Write a file with compression](#write-a-file-with-compression)
    * [Read a file with compression](#read-a-file-with-compression)
    * [Swap between filesystem types](#swap-between-filesystem-types-local-file-to-local-db)
    * [Access a file in an S3 bucket](#access-a-file-object-in-an-s3-bucket-and-use-compression)
  * [Why EZFS?](#why-ezfs)
    * [What does EZFS provide? What does EZFS not provide?](#what-does-ezfs-provide-what-does-ezfs-not-provide)
    * [Dependency Simplicity Example](#dependency-simplicity-example)
    * [Optimized Remote Filesystem Example](#optimized-remote-filesystem-example)
  * [Contributing](#contributing)


## Compatibility

- Supports Python 3.10+
- Supports multiple compression types
  - `bz2`, `gzip`, `lzma` (when built into Python)
  - `blosc`, `brotli`, `lz4`, `snappy`, and `zstd` (when installed separately)
- Supports multiple storage types
  - `sqlite3` (when built into Python)
  - `S3` (when installed separately)
- Theoretically any compression type, or backend storage type, by extending `Compressor`, `File`, and `Filesystem` 


## Getting Started

### Installation

Install EZFS via pip:
```shell
pip install ezfs
```

Or via git clone:
```shell
git clone <path to fork>
cd ezfs
pip install .
```

Or build and install from wheel:
```shell
# Build locally.
git clone <path to fork>
cd ezfs
make wheel

# Push dist/ezfs*.tar.gz to environment where it will be installed.
pip install dist/ezfs*.tar.gz
```

Or via copy and paste (only a single file is required):
```shell
# Copy:
cp ezfs.py <target project directory>
```


## How Tos

EZFS filesystems and file objects are designed to work nearly identical to native `open()` file handles.
Basic read and write operations can be directly swapped out after creating a filesystem adapter, and calling `open()`
against the filesystem instead of Python built-ins, or 3rd party compression libraries. Here are a few examples
of how to use the more advanced features, such as compression and remote storage. Refer to the supported operations
table in [Why EZFS?](#why-ezfs) for information on additional features.

### Write a file with compression
```python
import ezfs

filesystem = ezfs.LocalFilesystem('/tmp')
with filesystem.open('test-file.txt.gz', 'w+', compression='gzip') as out_file:
    out_file.write('test message')

# Or automatically compress all files on the "filesystem" on write:
filesystem = ezfs.LocalFilesystem('/tmp', compression='gzip')
with filesystem.open('test-file.txt.gz', 'w+') as out_file:
    out_file.write('test message')
```

### Read a file with compression
```python
import ezfs

filesystem = ezfs.LocalFilesystem('/tmp')
with filesystem.open('test-file.txt.gz', compression='gzip') as in_file:
    print(in_file.read())

# Or automatically decompress all files on the "filesystem" on read:
filesystem = ezfs.LocalFilesystem('/tmp', compression='gzip')
with filesystem.open('test-file.txt.gz') as in_file:
    print(in_file.read())
```

### Swap between filesystem types (local file to local db)
```python
import ezfs

# Only a single change is needed, such as from a local folder:
filesystem = ezfs.LocalFilesystem('/tmp')
# To a local database file:
filesystem = ezfs.SQLiteFilesystem('/tmp/tmp.db')

# No change is needed to open/read/write operations:
with filesystem.open('test-file.txt.gz', 'w+', compression='gzip') as out_file:
    out_file.write('test message')
```

### Access a file (object) in an S3 bucket, and use compression
```python
import ezfs

# To use advanced compression types, they must be installed separately.
filesystem = ezfs.S3BotoFilesystem(
    'my-bucket-1234',
    access_key_id='ABC123',
    secret_access_key='abcdefg1234567',
    compression='zstd',
)
with filesystem.open('test-file.txt.zst', 'w+') as out_file:
    out_file.write('test message')
with filesystem.open('test-file.txt.zst') as in_file:
    print(in_file.read())
```


## Why EZFS?

To simplify simple use cases.

EZFS is a very lightweight library (one file!), used to optimize "simple" use cases, or provide a starting point
for more complex use cases. What make a use case "simple? Reliance on core file/filesystem functionality, such as
create, read, write, and delete operations. What makes a use case "complex"? Reliance on complex file/filesystem
features, such as permissions, streaming, and seeking. The former benefit from EZFS out-of-the-box, while the latter
requires developers to extend the functionality further if they need this type of support.

While there are other libraries that can help accomplish file/filesystem-like use cases depending on the backend,
such as `s3fs` for S3, they may be more than needed or wanted. For example, perhaps
you have predictable logic to store/read files, and don't need to browse the filesystem tree. Perhaps you want
to leverage a custom service to act as storage interchangeably with local files, without installing extra
dependencies from other solutions. EZFS adapters can help with that. If you need full metadata support like
filesystem tree browsing, or file permissions, EZFS cannot help with that (natively), and recommends using a
more feature rich solution, or extending the adapters to fit your needs.

### What does EZFS provide? What does EZFS not provide?

EZFS provides a shared, optimized, interface to read and write files to various backend locations,
with or without compression. The backend for the storage can often be changed with a single line,
without changing the rest of the code.

EZFS does not provide a complex feature set for advanced use cases, such as managing permissions or other metadata
on filesystems. EZFS also does not provide streaming interfaces for processing larger than memory files in "chunks".
The following is a list of common file/filesystem operations, whether they are supported out-of-the-box, whether
they are supported with advanced installs (extras), and whether they are optimized/simplified by EZFS.

| Operations            | OOB | Extras | Optimized | Simplified |
|-----------------------|-----|--------|-----------|------------|
| open()                | ✅   | -      | ✅ ²       | ✅          |
| read()                | ✅   | -      | ✅         | ✅          |
| write()               | ✅   | -      | ✅         | ✅          |
| close()               | ✅   | -      | ✅ ²       | ✅          |
| exists()              | ✅   | -      | ✅ ³       | ✅          |
| isfile()              | ✅   | -      | ✅ ³       | ✅          |
| remove()              | ✅   | -      | ✅ ³       | ✅          |
| rename()              | ✅   | -      | ✅ ³       | ✅          |
| Memory file storage   | ✅   | ✅      | ✅         | ✅          |
| S3 file storage       | ❌   | ✅      | ✅         | ✅          |
| SQLite file storage   | ✅ ¹ | -      | ✅         | ✅          |
| bz2 compression       | ✅ ¹ | -      | ✅         | ✅          |
| gzip compression      | ✅ ¹ | -      | ✅         | ✅          |
| lzma compression      | ✅ ¹ | -      | ✅         | ✅          |
| blosc compression     | ❌   | ✅      | ✅         | ✅          |
| brotli compression    | ❌   | ✅      | ✅         | ✅          |
| lz4 compression       | ❌   | ✅      | ✅         | ✅          |
| snappy compression    | ❌   | ✅      | ✅         | ✅          |
| zstd compression      | ❌   | ✅      | ✅         | ✅          |
| isdir()               | ❌   | ❌      | -         | -          |
| listdir()             | ❌   | ❌      | -         | -          |
| mkdir()               | ❌   | ❌      | -         | -          |
| rmdir()               | ❌   | ❌      | -         | -          |
| Other "os" calls      | ❌   | ❌      | -         | -          |
| Other "os.path" calls | ❌   | ❌      | -         | -          |
| File permissions      | ❌   | ❌      | -         | -          |
| File streaming        | ❌   | ❌      | -         | -          |
| File seeking          | ❌   | ❌      | -         | -          |

¹ Depends on how Python was built
² Depends on compression module used  
³ Depends on the backend used

### Dependency Simplicity Example

Here is an example of using a library such as `sf3s` vs `ezfs` for basic read and write to S3, and its effect
on required dependencies in a project. A basic `boto3` install (only requirement for `ezfs` support) will add
the following to the environment:
- boto3
- botocore
- jmespath
- python-dateutil
- s3transfer
- six
- urllib3

An `s3fs` install will add the following in addition to the core `boto3` requirements:
- aiobotocore
- aiohttp
- aioitertools
- aiosignal
- async-timeout
- attrs
- idna
- frozenlist
- fsspec
- multidict
- s3fs
- wrapt
- yarl

Perhaps you already have all these requirements. Great! Then S3FS may be a better fit. Perhaps you don't have these,
and want to reduce requirements that may add maintenance overhead to resolve security vulnerabilities. Great!
EZFS may be a better fit. Still not sure? Continue reading for a performance example.

### Optimized Remote Filesystem Example

Here is a basic performance example, using S3 + `pandas` to store DataFrames. EZFS can optimize the S3 client used
to reduce networking overhead, leading to improved performance. The optimization benefit is greater with small files,
but even larger files benefit, and the simplicity to use stays the same.
- Small file: 100K
- Large file: 1M
- 100 iterations per test

| Scenario               | Write    | Read     |
|------------------------|----------|----------|
| pandas s3fs small raw  | 25.0 sec | 15.6 sec |
| pandas ezfs small raw  | 16.9 sec |  8.3 sec |
| pandas s3fs large raw  | 47.3 sec | 17.9 sec |
| pandas ezfs large raw  | 28.7 sec | 11.2 sec |
| pandas s3fs small zstd | 20.2 sec | 12.0 sec |
| pandas ezfs small zstd | 12.6 sec |  6.8 sec |
| pandas s3fs large zstd | 37.4 sec | 18.1 sec |
| pandas ezfs large zstd | 21.5 sec | 11.1 sec |


### Contributing

EZFS is not currently accepting new features. Minor features may be added to improve the native use cases,
but outside minor changes it will only receive bug fixes and dependency updates. This decision is to ensure
EZFS remains focused on its primary goal: stay simple and efficient, by focusing on simple use cases.
Feel free to import, fork, copy, etc., to other projects to expand the scope of its ecosystem. Refer to the
[Contributing Guide](CONTRIBUTING.md) for information on how to contribute fixes to this project.

            

Raw data

            {
    "_id": null,
    "home_page": null,
    "name": "ezfs",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.10",
    "maintainer_email": null,
    "keywords": "filesystem, s3, compression",
    "author": "David Fritz",
    "author_email": null,
    "download_url": "https://files.pythonhosted.org/packages/48/57/597aa08a7c06eb45b3b2a7a5578b2d5f1a4cbb19333d039dce509bfe6d08/ezfs-1.0.2.tar.gz",
    "platform": "MacOS",
    "description": "\n[![os: windows mac linux](https://img.shields.io/badge/os-linux_|_macos_|_windows-blue)](https://docs.python.org/3.10/)\n[![python: 3.10+](https://img.shields.io/badge/python-3.10_|_3.11_|_3.12-blue)](https://devguide.python.org/versions)\n[![python style: google](https://img.shields.io/badge/python%20style-google-blue)](https://google.github.io/styleguide/pyguide.html)\n[![imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://github.com/PyCQA/isort)\n[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![code style: pycodestyle](https://img.shields.io/badge/code%20style-pycodestyle-green)](https://github.com/PyCQA/pycodestyle)\n[![doc style: pydocstyle](https://img.shields.io/badge/doc%20style-pydocstyle-green)](https://github.com/PyCQA/pydocstyle)\n[![static typing: mypy](https://img.shields.io/badge/static_typing-mypy-green)](https://github.com/python/mypy)\n[![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/PyCQA/pylint)\n[![testing: pytest](https://img.shields.io/badge/testing-pytest-yellowgreen)](https://github.com/pytest-dev/pytest)\n[![security: bandit](https://img.shields.io/badge/security-bandit-black)](https://github.com/PyCQA/bandit)\n[![license: MIT](https://img.shields.io/badge/license-MIT-lightgrey)](LICENSE)\n\n\n# EZFS\n\nEZFS (Easy Filesystem, or Everythingz a Filesystem) is an optimized, minimal dependency (down to 0),\nvirtual filesystem adapter library for Python. EZFS provides access to the most common operations\nfor \"files\" and \"filesystems\", while abstracting storage away from the user and developer, to simplify\nboth UX and DX. All you need to know is how to access files in Python, and EZFS will take care of the rest,\nregardless of your selected backend.\n\nIn order to provide a streamlined UX and DX, EZFS leverages existing native Python designs and patterns,\nsuch as open/read/write operations, and applies them all backend storage types. The learning curve is minimal:\nif you know how to read/write a local file in Python, you also know how to read/write to any location in EZFS.\n\nReading and writing is supported for both text and binary files across local, remote, and memory \"filesystems\".\nAdditional compression types and storage types can be supported by extending the primary `File`, `Filesystem`,\nand `Compressor` adapters. EZFS can also be faster than \"native\" open/read/write operations in some scenarios,\ndue to having a specialized focus. Refer to the compatibility guides, and \"Why EZFS\", for more information.\n\n\n## Table Of Contents\n\n  * [Compatibility](#compatibility)\n  * [Getting Started](#getting-started)\n    * [Installation](#installation)\n  * [How Tos](#how-tos)\n    * [Write a file with compression](#write-a-file-with-compression)\n    * [Read a file with compression](#read-a-file-with-compression)\n    * [Swap between filesystem types](#swap-between-filesystem-types-local-file-to-local-db)\n    * [Access a file in an S3 bucket](#access-a-file-object-in-an-s3-bucket-and-use-compression)\n  * [Why EZFS?](#why-ezfs)\n    * [What does EZFS provide? What does EZFS not provide?](#what-does-ezfs-provide-what-does-ezfs-not-provide)\n    * [Dependency Simplicity Example](#dependency-simplicity-example)\n    * [Optimized Remote Filesystem Example](#optimized-remote-filesystem-example)\n  * [Contributing](#contributing)\n\n\n## Compatibility\n\n- Supports Python 3.10+\n- Supports multiple compression types\n  - `bz2`, `gzip`, `lzma` (when built into Python)\n  - `blosc`, `brotli`, `lz4`, `snappy`, and `zstd` (when installed separately)\n- Supports multiple storage types\n  - `sqlite3` (when built into Python)\n  - `S3` (when installed separately)\n- Theoretically any compression type, or backend storage type, by extending `Compressor`, `File`, and `Filesystem` \n\n\n## Getting Started\n\n### Installation\n\nInstall EZFS via pip:\n```shell\npip install ezfs\n```\n\nOr via git clone:\n```shell\ngit clone <path to fork>\ncd ezfs\npip install .\n```\n\nOr build and install from wheel:\n```shell\n# Build locally.\ngit clone <path to fork>\ncd ezfs\nmake wheel\n\n# Push dist/ezfs*.tar.gz to environment where it will be installed.\npip install dist/ezfs*.tar.gz\n```\n\nOr via copy and paste (only a single file is required):\n```shell\n# Copy:\ncp ezfs.py <target project directory>\n```\n\n\n## How Tos\n\nEZFS filesystems and file objects are designed to work nearly identical to native `open()` file handles.\nBasic read and write operations can be directly swapped out after creating a filesystem adapter, and calling `open()`\nagainst the filesystem instead of Python built-ins, or 3rd party compression libraries. Here are a few examples\nof how to use the more advanced features, such as compression and remote storage. Refer to the supported operations\ntable in [Why EZFS?](#why-ezfs) for information on additional features.\n\n### Write a file with compression\n```python\nimport ezfs\n\nfilesystem = ezfs.LocalFilesystem('/tmp')\nwith filesystem.open('test-file.txt.gz', 'w+', compression='gzip') as out_file:\n    out_file.write('test message')\n\n# Or automatically compress all files on the \"filesystem\" on write:\nfilesystem = ezfs.LocalFilesystem('/tmp', compression='gzip')\nwith filesystem.open('test-file.txt.gz', 'w+') as out_file:\n    out_file.write('test message')\n```\n\n### Read a file with compression\n```python\nimport ezfs\n\nfilesystem = ezfs.LocalFilesystem('/tmp')\nwith filesystem.open('test-file.txt.gz', compression='gzip') as in_file:\n    print(in_file.read())\n\n# Or automatically decompress all files on the \"filesystem\" on read:\nfilesystem = ezfs.LocalFilesystem('/tmp', compression='gzip')\nwith filesystem.open('test-file.txt.gz') as in_file:\n    print(in_file.read())\n```\n\n### Swap between filesystem types (local file to local db)\n```python\nimport ezfs\n\n# Only a single change is needed, such as from a local folder:\nfilesystem = ezfs.LocalFilesystem('/tmp')\n# To a local database file:\nfilesystem = ezfs.SQLiteFilesystem('/tmp/tmp.db')\n\n# No change is needed to open/read/write operations:\nwith filesystem.open('test-file.txt.gz', 'w+', compression='gzip') as out_file:\n    out_file.write('test message')\n```\n\n### Access a file (object) in an S3 bucket, and use compression\n```python\nimport ezfs\n\n# To use advanced compression types, they must be installed separately.\nfilesystem = ezfs.S3BotoFilesystem(\n    'my-bucket-1234',\n    access_key_id='ABC123',\n    secret_access_key='abcdefg1234567',\n    compression='zstd',\n)\nwith filesystem.open('test-file.txt.zst', 'w+') as out_file:\n    out_file.write('test message')\nwith filesystem.open('test-file.txt.zst') as in_file:\n    print(in_file.read())\n```\n\n\n## Why EZFS?\n\nTo simplify simple use cases.\n\nEZFS is a very lightweight library (one file!), used to optimize \"simple\" use cases, or provide a starting point\nfor more complex use cases. What make a use case \"simple? Reliance on core file/filesystem functionality, such as\ncreate, read, write, and delete operations. What makes a use case \"complex\"? Reliance on complex file/filesystem\nfeatures, such as permissions, streaming, and seeking. The former benefit from EZFS out-of-the-box, while the latter\nrequires developers to extend the functionality further if they need this type of support.\n\nWhile there are other libraries that can help accomplish file/filesystem-like use cases depending on the backend,\nsuch as `s3fs` for S3, they may be more than needed or wanted. For example, perhaps\nyou have predictable logic to store/read files, and don't need to browse the filesystem tree. Perhaps you want\nto leverage a custom service to act as storage interchangeably with local files, without installing extra\ndependencies from other solutions. EZFS adapters can help with that. If you need full metadata support like\nfilesystem tree browsing, or file permissions, EZFS cannot help with that (natively), and recommends using a\nmore feature rich solution, or extending the adapters to fit your needs.\n\n### What does EZFS provide? What does EZFS not provide?\n\nEZFS provides a shared, optimized, interface to read and write files to various backend locations,\nwith or without compression. The backend for the storage can often be changed with a single line,\nwithout changing the rest of the code.\n\nEZFS does not provide a complex feature set for advanced use cases, such as managing permissions or other metadata\non filesystems. EZFS also does not provide streaming interfaces for processing larger than memory files in \"chunks\".\nThe following is a list of common file/filesystem operations, whether they are supported out-of-the-box, whether\nthey are supported with advanced installs (extras), and whether they are optimized/simplified by EZFS.\n\n| Operations            | OOB | Extras | Optimized | Simplified |\n|-----------------------|-----|--------|-----------|------------|\n| open()                | \u2705   | -      | \u2705 \u00b2       | \u2705          |\n| read()                | \u2705   | -      | \u2705         | \u2705          |\n| write()               | \u2705   | -      | \u2705         | \u2705          |\n| close()               | \u2705   | -      | \u2705 \u00b2       | \u2705          |\n| exists()              | \u2705   | -      | \u2705 \u00b3       | \u2705          |\n| isfile()              | \u2705   | -      | \u2705 \u00b3       | \u2705          |\n| remove()              | \u2705   | -      | \u2705 \u00b3       | \u2705          |\n| rename()              | \u2705   | -      | \u2705 \u00b3       | \u2705          |\n| Memory file storage   | \u2705   | \u2705      | \u2705         | \u2705          |\n| S3 file storage       | \u274c   | \u2705      | \u2705         | \u2705          |\n| SQLite file storage   | \u2705 \u00b9 | -      | \u2705         | \u2705          |\n| bz2 compression       | \u2705 \u00b9 | -      | \u2705         | \u2705          |\n| gzip compression      | \u2705 \u00b9 | -      | \u2705         | \u2705          |\n| lzma compression      | \u2705 \u00b9 | -      | \u2705         | \u2705          |\n| blosc compression     | \u274c   | \u2705      | \u2705         | \u2705          |\n| brotli compression    | \u274c   | \u2705      | \u2705         | \u2705          |\n| lz4 compression       | \u274c   | \u2705      | \u2705         | \u2705          |\n| snappy compression    | \u274c   | \u2705      | \u2705         | \u2705          |\n| zstd compression      | \u274c   | \u2705      | \u2705         | \u2705          |\n| isdir()               | \u274c   | \u274c      | -         | -          |\n| listdir()             | \u274c   | \u274c      | -         | -          |\n| mkdir()               | \u274c   | \u274c      | -         | -          |\n| rmdir()               | \u274c   | \u274c      | -         | -          |\n| Other \"os\" calls      | \u274c   | \u274c      | -         | -          |\n| Other \"os.path\" calls | \u274c   | \u274c      | -         | -          |\n| File permissions      | \u274c   | \u274c      | -         | -          |\n| File streaming        | \u274c   | \u274c      | -         | -          |\n| File seeking          | \u274c   | \u274c      | -         | -          |\n\n\u00b9 Depends on how Python was built\n\u00b2 Depends on compression module used  \n\u00b3 Depends on the backend used\n\n### Dependency Simplicity Example\n\nHere is an example of using a library such as `sf3s` vs `ezfs` for basic read and write to S3, and its effect\non required dependencies in a project. A basic `boto3` install (only requirement for `ezfs` support) will add\nthe following to the environment:\n- boto3\n- botocore\n- jmespath\n- python-dateutil\n- s3transfer\n- six\n- urllib3\n\nAn `s3fs` install will add the following in addition to the core `boto3` requirements:\n- aiobotocore\n- aiohttp\n- aioitertools\n- aiosignal\n- async-timeout\n- attrs\n- idna\n- frozenlist\n- fsspec\n- multidict\n- s3fs\n- wrapt\n- yarl\n\nPerhaps you already have all these requirements. Great! Then S3FS may be a better fit. Perhaps you don't have these,\nand want to reduce requirements that may add maintenance overhead to resolve security vulnerabilities. Great!\nEZFS may be a better fit. Still not sure? Continue reading for a performance example.\n\n### Optimized Remote Filesystem Example\n\nHere is a basic performance example, using S3 + `pandas` to store DataFrames. EZFS can optimize the S3 client used\nto reduce networking overhead, leading to improved performance. The optimization benefit is greater with small files,\nbut even larger files benefit, and the simplicity to use stays the same.\n- Small file: 100K\n- Large file: 1M\n- 100 iterations per test\n\n| Scenario               | Write    | Read     |\n|------------------------|----------|----------|\n| pandas s3fs small raw  | 25.0 sec | 15.6 sec |\n| pandas ezfs small raw  | 16.9 sec |  8.3 sec |\n| pandas s3fs large raw  | 47.3 sec | 17.9 sec |\n| pandas ezfs large raw  | 28.7 sec | 11.2 sec |\n| pandas s3fs small zstd | 20.2 sec | 12.0 sec |\n| pandas ezfs small zstd | 12.6 sec |  6.8 sec |\n| pandas s3fs large zstd | 37.4 sec | 18.1 sec |\n| pandas ezfs large zstd | 21.5 sec | 11.1 sec |\n\n\n### Contributing\n\nEZFS is not currently accepting new features. Minor features may be added to improve the native use cases,\nbut outside minor changes it will only receive bug fixes and dependency updates. This decision is to ensure\nEZFS remains focused on its primary goal: stay simple and efficient, by focusing on simple use cases.\nFeel free to import, fork, copy, etc., to other projects to expand the scope of its ecosystem. Refer to the\n[Contributing Guide](CONTRIBUTING.md) for information on how to contribute fixes to this project.\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Minimalistic virtual filesystem adapters for Python",
    "version": "1.0.2",
    "project_urls": {
        "Changelog": "https://github.com/pyranha-labs/ezfs/releases",
        "Home": "https://github.com/pyranha-labs/ezfs",
        "Issues": "https://github.com/pyranha-labs/ezfs/issues"
    },
    "split_keywords": [
        "filesystem",
        " s3",
        " compression"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "f70522dfe26fe8f5a9cd4db1e261bd046f76ba4455bf1883c4050352e9c15e58",
                "md5": "1a9f6f600a982001d8aa52587559c7a1",
                "sha256": "a79e53bfda2cca674f590e5d23110a0edf1d8c20598b60a31677f308546b432a"
            },
            "downloads": -1,
            "filename": "ezfs-1.0.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "1a9f6f600a982001d8aa52587559c7a1",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.10",
            "size": 13708,
            "upload_time": "2024-05-04T14:21:42",
            "upload_time_iso_8601": "2024-05-04T14:21:42.064999Z",
            "url": "https://files.pythonhosted.org/packages/f7/05/22dfe26fe8f5a9cd4db1e261bd046f76ba4455bf1883c4050352e9c15e58/ezfs-1.0.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "4857597aa08a7c06eb45b3b2a7a5578b2d5f1a4cbb19333d039dce509bfe6d08",
                "md5": "b846fe1acb7b65c3701ff2a767421d09",
                "sha256": "6c0f7cb98e223d34b975877b4b1e154c699ad203840c4d42092caa26fb939160"
            },
            "downloads": -1,
            "filename": "ezfs-1.0.2.tar.gz",
            "has_sig": false,
            "md5_digest": "b846fe1acb7b65c3701ff2a767421d09",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.10",
            "size": 16319,
            "upload_time": "2024-05-04T14:21:43",
            "upload_time_iso_8601": "2024-05-04T14:21:43.645856Z",
            "url": "https://files.pythonhosted.org/packages/48/57/597aa08a7c06eb45b3b2a7a5578b2d5f1a4cbb19333d039dce509bfe6d08/ezfs-1.0.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-05-04 14:21:43",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "pyranha-labs",
    "github_project": "ezfs",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "lcname": "ezfs"
}
        
Elapsed time: 0.25199s