| Name | genro-storage JSON |
| Version |
0.3.0
JSON |
| download |
| home_page | None |
| Summary | Unified storage abstraction for Genropy framework |
| upload_time | 2025-10-29 05:25:31 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | MIT |
| keywords |
storage
filesystem
s3
cloud
genropy
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# genro-storage
[](https://www.python.org/downloads/)
[](https://opensource.org/licenses/MIT)
[](https://genro-storage.readthedocs.io/en/latest/?badge=latest)
[](https://github.com/genropy/genro-storage/actions)
[](https://codecov.io/gh/genropy/genro-storage)
[](https://github.com/psf/black)
**Universal storage abstraction for Python with pluggable backends**
A modern, elegant Python library that provides a unified interface for accessing files across local filesystems, cloud storage (S3, GCS, Azure), and remote protocols (HTTP). Built on top of **fsspec**, genro-storage adds an intuitive mount-point system and user-friendly API inspired by Unix filesystems.
## Status: Beta - Ready for Production Testing
**Current Version:** 0.3.0-dev
**Last Updated:** October 2025
✅ Core implementation complete
✅ All backends working (local, S3, GCS, Azure, HTTP, Memory, Base64)
✅ **NEW in v0.3.0:** Async/await support for FastAPI, asyncio applications (asyncer-based)
✅ 274 tests passing on Python 3.9-3.12
✅ Full documentation on ReadTheDocs
✅ Battle-tested code from Genropy (19+ years in production, storage abstraction since 2018)
✅ Available on PyPI
## Key Features
- **Async/await support** - Use in FastAPI, asyncio apps with AsyncStorageManager (NEW in v0.3.0!)
- **Powered by fsspec** - Leverage 20+ battle-tested storage backends
- **Mount point system** - Organize storage with logical names like `home:`, `uploads:`, `s3:`
- **Intuitive API** - Pathlib-inspired interface that feels natural and Pythonic
- **Intelligent copy strategies** - Skip files by existence, size, or hash for efficient incremental backups
- **Progress tracking** - Built-in callbacks for progress bars and logging during copy operations
- **Content-based comparison** - Compare files by MD5 hash across different backends
- **Efficient hashing** - Uses cloud metadata (S3 ETag) when available, avoiding downloads
- **External tool integration** - `call()` method for seamless integration with ffmpeg, imagemagick, pandoc, etc.
- **WSGI file serving** - `serve()` method for web frameworks (Flask, Django, Pyramid) with ETag caching
- **MIME type detection** - Automatic content-type detection from file extensions
- **Flexible configuration** - Load mounts from YAML, JSON, or code
- **Dynamic paths** - Support for callable paths that resolve at runtime (perfect for user-specific directories)
- **Cloud metadata** - Get/set custom metadata on S3, GCS, Azure files
- **URL generation** - Generate presigned URLs for S3, public URLs for sharing
- **Base64 utilities** - Encode files to data URIs, download from URLs
- **S3 versioning** - Access historical file versions (when S3 versioning enabled)
- **Test-friendly** - In-memory backend for fast, isolated testing
- **Base64 data URIs** - Embed data inline with automatic encoding (writable with mutable paths)
- **Production-ready backends** - Built on 6+ years of Genropy production experience
- **Lightweight core** - Optional backends installed only when needed
- **Cross-storage operations** - Copy/move files between different storage types seamlessly
## Why genro-storage vs raw fsspec?
While **fsspec** is powerful, genro-storage provides:
- **Mount point abstraction** - Work with logical names instead of full URIs
- **Simpler API** - Less verbose, more intuitive for common operations
- **Configuration management** - Load storage configs from files
- **Enhanced utilities** - Cross-storage copy, unified error handling
Think of it as **"requests" is to "urllib"** - a friendlier interface to an excellent foundation.
## Perfect For
- **Multi-cloud applications** that need storage abstraction
- **Data pipelines** processing files from various sources
- **Web applications** managing uploads across environments
- **CLI tools** that work with local and remote files
- **Testing scenarios** requiring storage mocking
## Quick Example
### Synchronous Usage
```python
from genro_storage import StorageManager
# Configure storage backends
storage = StorageManager()
storage.configure([
{'name': 'home', 'type': 'local', 'path': '/home/user'},
{'name': 'uploads', 'type': 's3', 'bucket': 'my-app-uploads'},
{'name': 'backups', 'type': 'gcs', 'bucket': 'my-backups'},
{'name': 'data', 'type': 'base64'} # Inline base64 data
])
# Work with files using a unified API
node = storage.node('uploads:users/123/avatar.jpg')
if node.exists:
# Copy from S3 to local
node.copy_to(storage.node('home:cache/avatar.jpg'))
# Read and process
data = node.read_bytes()
# Backup to GCS
node.copy_to(storage.node('backups:avatars/user_123.jpg'))
# Base64 backend: embed data directly in URIs (data URI style)
# Read inline data
import base64
text = "Configuration data"
b64_data = base64.b64encode(text.encode()).decode()
node = storage.node(f'data:{b64_data}')
print(node.read_text()) # "Configuration data"
# Or write to create base64 (path updates automatically)
node = storage.node('data:')
node.write_text("New content")
print(node.path) # "TmV3IGNvbnRlbnQ=" (base64 of "New content")
# Copy from S3 to base64 for inline use
s3_image = storage.node('uploads:photo.jpg')
b64_image = storage.node('data:')
s3_image.copy_to(b64_image)
data_uri = f"data:image/jpeg;base64,{b64_image.path}"
# Advanced features
# 1. Intelligent incremental backups (NEW!)
docs = storage.node('home:documents')
s3_backup = storage.node('uploads:backup/documents')
# Skip files that already exist (fastest)
docs.copy_to(s3_backup, skip='exists')
# Skip files with same size (fast, good accuracy)
docs.copy_to(s3_backup, skip='size')
# Skip files with same content (accurate, uses S3 ETag - fast!)
docs.copy_to(s3_backup, skip='hash')
# With progress tracking
from tqdm import tqdm
pbar = tqdm(desc="Backing up", unit="file")
docs.copy_to(s3_backup, skip='hash',
progress=lambda cur, tot: pbar.update(1))
pbar.close()
# 2. Work with external tools using call() (ffmpeg, imagemagick, etc.)
video = storage.node('uploads:video.mp4')
thumbnail = storage.node('uploads:thumb.jpg')
# Automatically handles cloud download/upload
video.call('ffmpeg', '-i', video, '-vf', 'thumbnail', '-frames:v', '1', thumbnail)
# Or use local_path() for more control
with video.local_path(mode='r') as local_path:
import subprocess
subprocess.run(['ffmpeg', '-i', local_path, 'output.mp4'])
# 3. Serve files via WSGI (Flask, Django, Pyramid)
from flask import Flask, request
app = Flask(__name__)
@app.route('/files/<path:filepath>')
def serve_file(filepath):
node = storage.node(f'uploads:{filepath}')
# ETag caching, streaming, MIME types - all automatic!
return node.serve(request.environ, lambda s, h: None, cache_max_age=3600)
# 4. Check MIME types
doc = storage.node('uploads:report.pdf')
print(doc.mimetype) # 'application/pdf'
# 5. Dynamic paths for multi-user apps
def get_user_storage():
user_id = get_current_user()
return f'/data/users/{user_id}'
storage.configure([
{'name': 'user', 'type': 'local', 'path': get_user_storage}
])
# Path resolves differently per user!
# 6. Cloud metadata
file = storage.node('uploads:document.pdf')
file.set_metadata({
'Author': 'John Doe',
'Department': 'Engineering'
})
# 7. Generate shareable URLs
url = file.url(expires_in=3600) # S3 presigned URL
# 8. Encode to data URI
img = storage.node('home:logo.png')
data_uri = img.to_base64() # data:image/png;base64,...
# 9. Download from internet
remote = storage.node('uploads:downloaded.pdf')
remote.fill_from_url('https://example.com/file.pdf')
```
### Async Usage (NEW in v0.3.0!)
Built on [asyncer](https://github.com/tiangolo/asyncer) by Sebastián Ramírez (FastAPI author) for automatic sync→async conversion with no event loop blocking. Native async implementation planned for v0.4.0.
```python
from genro_storage import AsyncStorageManager
# Initialize async storage manager
storage = AsyncStorageManager()
# Configure (sync - call at startup)
storage.configure([
{'name': 'uploads', 'type': 's3', 'bucket': 'my-app-uploads'},
{'name': 'cache', 'type': 'local', 'path': '/tmp/cache'}
])
# Use in async context (FastAPI, asyncio, etc.)
async def process_file(file_path: str):
node = storage.node(f'uploads:{file_path}')
# All I/O operations are async
if await node.exists():
data = await node.read_bytes()
# Process and cache
processed = process_data(data)
cache_node = storage.node('cache:processed.dat')
await cache_node.write_bytes(processed)
return processed
raise FileNotFoundError(file_path)
# FastAPI example
from fastapi import FastAPI, HTTPException
app = FastAPI()
@app.get("/files/{filepath:path}")
async def get_file(filepath: str):
"""Serve file from S3 storage."""
node = storage.node(f'uploads:{filepath}')
if not await node.exists():
raise HTTPException(status_code=404, detail="File not found")
return {
"data": await node.read_bytes(),
"size": await node.size(),
"mime_type": node.mimetype # Sync property
}
# Concurrent operations
import asyncio
async def backup_files(file_list):
"""Backup multiple files concurrently."""
async def backup_one(filepath):
source = storage.node(f'uploads:{filepath}')
target = storage.node(f'backups:{filepath}')
data = await source.read_bytes()
await target.write_bytes(data)
# Process all files in parallel
await asyncio.gather(*[backup_one(f) for f in file_list])
```
## Learning with Interactive Tutorials
The best way to learn genro-storage is through our **hands-on Jupyter notebooks** in the [`notebooks/`](notebooks/) directory.
### Run Online (No Installation Required)
[](https://mybinder.org/v2/gh/genropy/genro-storage/main?filepath=notebooks)
Click the badge above to launch an interactive Jupyter environment in your browser. Ready in ~2 minutes!
### Run Locally
```bash
# 1. Install Jupyter
pip install jupyter notebook
# 2. Navigate to notebooks directory
cd notebooks
# 3. Launch Jupyter
jupyter notebook
# 4. Open 01_quickstart.ipynb and start learning!
```
**Note:** Jupyter will open in your browser automatically. Execute cells sequentially with `Shift+Enter`.
### Tutorial Contents
| Notebook | Topic | Duration | Level |
|----------|-------|----------|-------|
| 01 - Quickstart | Basic concepts and first steps | 15 min | Beginner |
| 02 - Backends | Storage backends and configuration | 20 min | Beginner |
| 03 - File Operations | Read, write, copy, directories | 25 min | Beginner |
| 04 - Virtual Nodes | iternode, diffnode, zip archives | 30 min | Intermediate |
| 05 - Copy Strategies | Smart copying and filtering | 25 min | Intermediate |
| 06 - Versioning | S3 version history and rollback | 30 min | Intermediate |
| 07 - Advanced Features | External tools, WSGI, metadata | 35 min | Advanced |
| 08 - Real World Examples | Complete use cases | 40 min | Advanced |
**Total time:** ~3.5 hours • **Start here:** [01_quickstart.ipynb](notebooks/01_quickstart.ipynb)
See [notebooks/README.md](notebooks/README.md) for the complete learning guide.
## Installation
### From GitHub (Recommended)
Install directly from GitHub without cloning:
```bash
# Base package
pip install git+https://github.com/genropy/genro-storage.git
# With S3 support
pip install "genro-storage[s3] @ git+https://github.com/genropy/genro-storage.git"
# With all backends
pip install "genro-storage[all] @ git+https://github.com/genropy/genro-storage.git"
```
### From Source (Development)
Clone and install in editable mode:
```bash
# Clone repository
git clone https://github.com/genropy/genro-storage.git
cd genro-storage
# Install base package
pip install -e .
# Install with S3 support
pip install -e ".[s3]"
# Install with all backends
pip install -e ".[all]"
# Install for development
pip install -e ".[all,dev]"
```
### Supported Backends
Install optional dependencies for specific backends:
```bash
pip install genro-storage[s3] # Amazon S3
pip install genro-storage[gcs] # Google Cloud Storage
pip install genro-storage[azure] # Azure Blob Storage
pip install genro-storage[http] # HTTP/HTTPS
pip install genro-storage[async] # Async support (NEW in v0.3.0!)
pip install genro-storage[all] # All backends + async
```
## Testing
```bash
# Unit tests (fast, no external dependencies)
pytest tests/test_local_storage.py -v
# Integration tests (requires Docker + MinIO)
docker-compose up -d
pytest tests/test_s3_integration.py -v
# All tests
pytest tests/ -v
# With coverage
pytest tests/ -v --cov=genro_storage
```
See [TESTING.md](TESTING.md) for detailed testing instructions with MinIO.
## Documentation
- **[Full Documentation](https://genro-storage.readthedocs.io/)** - Complete API reference and guides
- **[API Design](API_DESIGN.md)** - Detailed design specification
- **[Testing Guide](TESTING.md)** - How to run tests with MinIO
## Built With
- [fsspec](https://filesystem-spec.readthedocs.io/) - Pythonic filesystem abstraction
- [asyncer](https://github.com/tiangolo/asyncer) - Async wrapper (v0.3.0+, bridge to native async in v0.4.0)
- Modern Python (3.9+) with full type hints
- Optional backends: s3fs, gcsfs, adlfs, aiohttp
## Origins
genro-storage is extracted and modernized from [Genropy](https://github.com/genropy/genropy), a Python web framework in production since 2006 (19+ years). The storage abstraction layer was introduced in 2018 and has been battle-tested in production for 6+ years. We're making this powerful storage abstraction available as a standalone library for the wider Python community.
## Development Status
**Phase:** Beta - Production Testing
- ✅ API Design Complete and Stable
- ✅ Core Implementation Complete
- ✅ FsspecBackend (all 7 storage types working: local, S3, GCS, Azure, HTTP, Memory, Base64)
- ✅ Comprehensive Test Suite (274 tests, 81% coverage)
- ✅ CI/CD with Python 3.9, 3.10, 3.11, 3.12
- ✅ MD5 hashing and content-based equality
- ✅ Base64 backend with writable mutable paths
- ✅ Intelligent copy skip strategies (exists, size, hash, custom)
- ✅ call() method for external tool integration (ffmpeg, imagemagick, etc.)
- ✅ serve() method for WSGI file serving (Flask, Django, Pyramid)
- ✅ mimetype property for automatic content-type detection
- ✅ local_path() context manager for external tools
- ✅ Callable path support for dynamic directories
- ✅ Cloud metadata get/set (S3, GCS, Azure)
- ✅ URL generation (presigned URLs, data URIs)
- ✅ S3 versioning support
- ✅ Full Documentation on ReadTheDocs
- ✅ MinIO Integration Testing
- 🚧 Async/await support (AsyncStorageManager, AsyncStorageNode) - v0.3.0 in development
- 🎯 Ready for early adopters and production testing
- ⏳ Extended GCS/Azure integration testing
**Roadmap:**
- v0.2.0 (October 2025) - Virtual nodes, tutorials, enhanced testing ✅ **RELEASED**
- v0.3.0 (Q4 2025) - Async support via asyncer wrapper 🚧 **IN DEVELOPMENT**
- v0.4.0 (Q1 2026) - Native async API, runtime mount management
- v1.0.0 (2026) - Production-ready, stable API guarantee
## Contributing
Contributions welcome! The library is in beta with a stable API.
**How to contribute:**
1. Review the [API Design Document](API_DESIGN.md)
2. Check existing [tests](tests/) to understand behavior
3. Open an issue to discuss major changes
4. Submit PRs with tests
**Testing contributions:**
- Add tests for GCS and Azure backends
- Improve test coverage (target: 90%+)
- Add integration tests for edge cases
## License
MIT License - See [LICENSE](LICENSE) for details
---
**Made with ❤️ by the Genropy team**
Raw data
{
"_id": null,
"home_page": null,
"name": "genro-storage",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "storage, filesystem, s3, cloud, genropy",
"author": null,
"author_email": "Genropy Team <info@genropy.org>",
"download_url": "https://files.pythonhosted.org/packages/03/86/291145bd054168d3cd7a322e5eb7a6e1fb6f0d8fa619272abde9c342ecb0/genro_storage-0.3.0.tar.gz",
"platform": null,
"description": "# genro-storage\n\n[](https://www.python.org/downloads/)\n[](https://opensource.org/licenses/MIT)\n[](https://genro-storage.readthedocs.io/en/latest/?badge=latest)\n[](https://github.com/genropy/genro-storage/actions)\n[](https://codecov.io/gh/genropy/genro-storage)\n[](https://github.com/psf/black)\n\n**Universal storage abstraction for Python with pluggable backends**\n\nA modern, elegant Python library that provides a unified interface for accessing files across local filesystems, cloud storage (S3, GCS, Azure), and remote protocols (HTTP). Built on top of **fsspec**, genro-storage adds an intuitive mount-point system and user-friendly API inspired by Unix filesystems.\n\n## Status: Beta - Ready for Production Testing\n\n**Current Version:** 0.3.0-dev\n**Last Updated:** October 2025\n\n\u2705 Core implementation complete\n\u2705 All backends working (local, S3, GCS, Azure, HTTP, Memory, Base64)\n\u2705 **NEW in v0.3.0:** Async/await support for FastAPI, asyncio applications (asyncer-based)\n\u2705 274 tests passing on Python 3.9-3.12\n\u2705 Full documentation on ReadTheDocs\n\u2705 Battle-tested code from Genropy (19+ years in production, storage abstraction since 2018)\n\u2705 Available on PyPI\n\n## Key Features\n\n- **Async/await support** - Use in FastAPI, asyncio apps with AsyncStorageManager (NEW in v0.3.0!)\n- **Powered by fsspec** - Leverage 20+ battle-tested storage backends\n- **Mount point system** - Organize storage with logical names like `home:`, `uploads:`, `s3:`\n- **Intuitive API** - Pathlib-inspired interface that feels natural and Pythonic\n- **Intelligent copy strategies** - Skip files by existence, size, or hash for efficient incremental backups\n- **Progress tracking** - Built-in callbacks for progress bars and logging during copy operations\n- **Content-based comparison** - Compare files by MD5 hash across different backends\n- **Efficient hashing** - Uses cloud metadata (S3 ETag) when available, avoiding downloads\n- **External tool integration** - `call()` method for seamless integration with ffmpeg, imagemagick, pandoc, etc.\n- **WSGI file serving** - `serve()` method for web frameworks (Flask, Django, Pyramid) with ETag caching\n- **MIME type detection** - Automatic content-type detection from file extensions\n- **Flexible configuration** - Load mounts from YAML, JSON, or code\n- **Dynamic paths** - Support for callable paths that resolve at runtime (perfect for user-specific directories)\n- **Cloud metadata** - Get/set custom metadata on S3, GCS, Azure files\n- **URL generation** - Generate presigned URLs for S3, public URLs for sharing\n- **Base64 utilities** - Encode files to data URIs, download from URLs\n- **S3 versioning** - Access historical file versions (when S3 versioning enabled)\n- **Test-friendly** - In-memory backend for fast, isolated testing\n- **Base64 data URIs** - Embed data inline with automatic encoding (writable with mutable paths)\n- **Production-ready backends** - Built on 6+ years of Genropy production experience\n- **Lightweight core** - Optional backends installed only when needed\n- **Cross-storage operations** - Copy/move files between different storage types seamlessly\n\n## Why genro-storage vs raw fsspec?\n\nWhile **fsspec** is powerful, genro-storage provides:\n\n- **Mount point abstraction** - Work with logical names instead of full URIs\n- **Simpler API** - Less verbose, more intuitive for common operations\n- **Configuration management** - Load storage configs from files\n- **Enhanced utilities** - Cross-storage copy, unified error handling\n\nThink of it as **\"requests\" is to \"urllib\"** - a friendlier interface to an excellent foundation.\n\n## Perfect For\n\n- **Multi-cloud applications** that need storage abstraction\n- **Data pipelines** processing files from various sources\n- **Web applications** managing uploads across environments\n- **CLI tools** that work with local and remote files\n- **Testing scenarios** requiring storage mocking\n\n## Quick Example\n\n### Synchronous Usage\n\n```python\nfrom genro_storage import StorageManager\n\n# Configure storage backends\nstorage = StorageManager()\nstorage.configure([\n {'name': 'home', 'type': 'local', 'path': '/home/user'},\n {'name': 'uploads', 'type': 's3', 'bucket': 'my-app-uploads'},\n {'name': 'backups', 'type': 'gcs', 'bucket': 'my-backups'},\n {'name': 'data', 'type': 'base64'} # Inline base64 data\n])\n\n# Work with files using a unified API\nnode = storage.node('uploads:users/123/avatar.jpg')\nif node.exists:\n # Copy from S3 to local\n node.copy_to(storage.node('home:cache/avatar.jpg'))\n\n # Read and process\n data = node.read_bytes()\n\n # Backup to GCS\n node.copy_to(storage.node('backups:avatars/user_123.jpg'))\n\n# Base64 backend: embed data directly in URIs (data URI style)\n# Read inline data\nimport base64\ntext = \"Configuration data\"\nb64_data = base64.b64encode(text.encode()).decode()\nnode = storage.node(f'data:{b64_data}')\nprint(node.read_text()) # \"Configuration data\"\n\n# Or write to create base64 (path updates automatically)\nnode = storage.node('data:')\nnode.write_text(\"New content\")\nprint(node.path) # \"TmV3IGNvbnRlbnQ=\" (base64 of \"New content\")\n\n# Copy from S3 to base64 for inline use\ns3_image = storage.node('uploads:photo.jpg')\nb64_image = storage.node('data:')\ns3_image.copy_to(b64_image)\ndata_uri = f\"data:image/jpeg;base64,{b64_image.path}\"\n\n# Advanced features\n# 1. Intelligent incremental backups (NEW!)\ndocs = storage.node('home:documents')\ns3_backup = storage.node('uploads:backup/documents')\n\n# Skip files that already exist (fastest)\ndocs.copy_to(s3_backup, skip='exists')\n\n# Skip files with same size (fast, good accuracy)\ndocs.copy_to(s3_backup, skip='size')\n\n# Skip files with same content (accurate, uses S3 ETag - fast!)\ndocs.copy_to(s3_backup, skip='hash')\n\n# With progress tracking\nfrom tqdm import tqdm\npbar = tqdm(desc=\"Backing up\", unit=\"file\")\ndocs.copy_to(s3_backup, skip='hash',\n progress=lambda cur, tot: pbar.update(1))\npbar.close()\n\n# 2. Work with external tools using call() (ffmpeg, imagemagick, etc.)\nvideo = storage.node('uploads:video.mp4')\nthumbnail = storage.node('uploads:thumb.jpg')\n\n# Automatically handles cloud download/upload\nvideo.call('ffmpeg', '-i', video, '-vf', 'thumbnail', '-frames:v', '1', thumbnail)\n\n# Or use local_path() for more control\nwith video.local_path(mode='r') as local_path:\n import subprocess\n subprocess.run(['ffmpeg', '-i', local_path, 'output.mp4'])\n\n# 3. Serve files via WSGI (Flask, Django, Pyramid)\nfrom flask import Flask, request\napp = Flask(__name__)\n\n@app.route('/files/<path:filepath>')\ndef serve_file(filepath):\n node = storage.node(f'uploads:{filepath}')\n # ETag caching, streaming, MIME types - all automatic!\n return node.serve(request.environ, lambda s, h: None, cache_max_age=3600)\n\n# 4. Check MIME types\ndoc = storage.node('uploads:report.pdf')\nprint(doc.mimetype) # 'application/pdf'\n\n# 5. Dynamic paths for multi-user apps\ndef get_user_storage():\n user_id = get_current_user()\n return f'/data/users/{user_id}'\n\nstorage.configure([\n {'name': 'user', 'type': 'local', 'path': get_user_storage}\n])\n# Path resolves differently per user!\n\n# 6. Cloud metadata\nfile = storage.node('uploads:document.pdf')\nfile.set_metadata({\n 'Author': 'John Doe',\n 'Department': 'Engineering'\n})\n\n# 7. Generate shareable URLs\nurl = file.url(expires_in=3600) # S3 presigned URL\n\n# 8. Encode to data URI\nimg = storage.node('home:logo.png')\ndata_uri = img.to_base64() # data:image/png;base64,...\n\n# 9. Download from internet\nremote = storage.node('uploads:downloaded.pdf')\nremote.fill_from_url('https://example.com/file.pdf')\n```\n\n### Async Usage (NEW in v0.3.0!)\n\nBuilt on [asyncer](https://github.com/tiangolo/asyncer) by Sebasti\u00e1n Ram\u00edrez (FastAPI author) for automatic sync\u2192async conversion with no event loop blocking. Native async implementation planned for v0.4.0.\n\n```python\nfrom genro_storage import AsyncStorageManager\n\n# Initialize async storage manager\nstorage = AsyncStorageManager()\n\n# Configure (sync - call at startup)\nstorage.configure([\n {'name': 'uploads', 'type': 's3', 'bucket': 'my-app-uploads'},\n {'name': 'cache', 'type': 'local', 'path': '/tmp/cache'}\n])\n\n# Use in async context (FastAPI, asyncio, etc.)\nasync def process_file(file_path: str):\n node = storage.node(f'uploads:{file_path}')\n\n # All I/O operations are async\n if await node.exists():\n data = await node.read_bytes()\n\n # Process and cache\n processed = process_data(data)\n cache_node = storage.node('cache:processed.dat')\n await cache_node.write_bytes(processed)\n\n return processed\n\n raise FileNotFoundError(file_path)\n\n# FastAPI example\nfrom fastapi import FastAPI, HTTPException\n\napp = FastAPI()\n\n@app.get(\"/files/{filepath:path}\")\nasync def get_file(filepath: str):\n \"\"\"Serve file from S3 storage.\"\"\"\n node = storage.node(f'uploads:{filepath}')\n\n if not await node.exists():\n raise HTTPException(status_code=404, detail=\"File not found\")\n\n return {\n \"data\": await node.read_bytes(),\n \"size\": await node.size(),\n \"mime_type\": node.mimetype # Sync property\n }\n\n# Concurrent operations\nimport asyncio\n\nasync def backup_files(file_list):\n \"\"\"Backup multiple files concurrently.\"\"\"\n async def backup_one(filepath):\n source = storage.node(f'uploads:{filepath}')\n target = storage.node(f'backups:{filepath}')\n data = await source.read_bytes()\n await target.write_bytes(data)\n\n # Process all files in parallel\n await asyncio.gather(*[backup_one(f) for f in file_list])\n```\n\n## Learning with Interactive Tutorials\n\nThe best way to learn genro-storage is through our **hands-on Jupyter notebooks** in the [`notebooks/`](notebooks/) directory.\n\n### Run Online (No Installation Required)\n\n[](https://mybinder.org/v2/gh/genropy/genro-storage/main?filepath=notebooks)\n\nClick the badge above to launch an interactive Jupyter environment in your browser. Ready in ~2 minutes!\n\n### Run Locally\n\n```bash\n# 1. Install Jupyter\npip install jupyter notebook\n\n# 2. Navigate to notebooks directory\ncd notebooks\n\n# 3. Launch Jupyter\njupyter notebook\n\n# 4. Open 01_quickstart.ipynb and start learning!\n```\n\n**Note:** Jupyter will open in your browser automatically. Execute cells sequentially with `Shift+Enter`.\n\n### Tutorial Contents\n\n| Notebook | Topic | Duration | Level |\n|----------|-------|----------|-------|\n| 01 - Quickstart | Basic concepts and first steps | 15 min | Beginner |\n| 02 - Backends | Storage backends and configuration | 20 min | Beginner |\n| 03 - File Operations | Read, write, copy, directories | 25 min | Beginner |\n| 04 - Virtual Nodes | iternode, diffnode, zip archives | 30 min | Intermediate |\n| 05 - Copy Strategies | Smart copying and filtering | 25 min | Intermediate |\n| 06 - Versioning | S3 version history and rollback | 30 min | Intermediate |\n| 07 - Advanced Features | External tools, WSGI, metadata | 35 min | Advanced |\n| 08 - Real World Examples | Complete use cases | 40 min | Advanced |\n\n**Total time:** ~3.5 hours \u2022 **Start here:** [01_quickstart.ipynb](notebooks/01_quickstart.ipynb)\n\nSee [notebooks/README.md](notebooks/README.md) for the complete learning guide.\n\n## Installation\n\n### From GitHub (Recommended)\n\nInstall directly from GitHub without cloning:\n\n```bash\n# Base package\npip install git+https://github.com/genropy/genro-storage.git\n\n# With S3 support\npip install \"genro-storage[s3] @ git+https://github.com/genropy/genro-storage.git\"\n\n# With all backends\npip install \"genro-storage[all] @ git+https://github.com/genropy/genro-storage.git\"\n```\n\n### From Source (Development)\n\nClone and install in editable mode:\n\n```bash\n# Clone repository\ngit clone https://github.com/genropy/genro-storage.git\ncd genro-storage\n\n# Install base package\npip install -e .\n\n# Install with S3 support\npip install -e \".[s3]\"\n\n# Install with all backends\npip install -e \".[all]\"\n\n# Install for development\npip install -e \".[all,dev]\"\n```\n\n### Supported Backends\n\nInstall optional dependencies for specific backends:\n\n```bash\npip install genro-storage[s3] # Amazon S3\npip install genro-storage[gcs] # Google Cloud Storage\npip install genro-storage[azure] # Azure Blob Storage\npip install genro-storage[http] # HTTP/HTTPS\npip install genro-storage[async] # Async support (NEW in v0.3.0!)\npip install genro-storage[all] # All backends + async\n```\n\n## Testing\n\n```bash\n# Unit tests (fast, no external dependencies)\npytest tests/test_local_storage.py -v\n\n# Integration tests (requires Docker + MinIO)\ndocker-compose up -d\npytest tests/test_s3_integration.py -v\n\n# All tests\npytest tests/ -v\n\n# With coverage\npytest tests/ -v --cov=genro_storage\n```\n\nSee [TESTING.md](TESTING.md) for detailed testing instructions with MinIO.\n\n## Documentation\n\n- **[Full Documentation](https://genro-storage.readthedocs.io/)** - Complete API reference and guides\n- **[API Design](API_DESIGN.md)** - Detailed design specification\n- **[Testing Guide](TESTING.md)** - How to run tests with MinIO\n\n## Built With\n\n- [fsspec](https://filesystem-spec.readthedocs.io/) - Pythonic filesystem abstraction\n- [asyncer](https://github.com/tiangolo/asyncer) - Async wrapper (v0.3.0+, bridge to native async in v0.4.0)\n- Modern Python (3.9+) with full type hints\n- Optional backends: s3fs, gcsfs, adlfs, aiohttp\n\n## Origins\n\ngenro-storage is extracted and modernized from [Genropy](https://github.com/genropy/genropy), a Python web framework in production since 2006 (19+ years). The storage abstraction layer was introduced in 2018 and has been battle-tested in production for 6+ years. We're making this powerful storage abstraction available as a standalone library for the wider Python community.\n\n## Development Status\n\n**Phase:** Beta - Production Testing\n\n- \u2705 API Design Complete and Stable\n- \u2705 Core Implementation Complete\n- \u2705 FsspecBackend (all 7 storage types working: local, S3, GCS, Azure, HTTP, Memory, Base64)\n- \u2705 Comprehensive Test Suite (274 tests, 81% coverage)\n- \u2705 CI/CD with Python 3.9, 3.10, 3.11, 3.12\n- \u2705 MD5 hashing and content-based equality\n- \u2705 Base64 backend with writable mutable paths\n- \u2705 Intelligent copy skip strategies (exists, size, hash, custom)\n- \u2705 call() method for external tool integration (ffmpeg, imagemagick, etc.)\n- \u2705 serve() method for WSGI file serving (Flask, Django, Pyramid)\n- \u2705 mimetype property for automatic content-type detection\n- \u2705 local_path() context manager for external tools\n- \u2705 Callable path support for dynamic directories\n- \u2705 Cloud metadata get/set (S3, GCS, Azure)\n- \u2705 URL generation (presigned URLs, data URIs)\n- \u2705 S3 versioning support\n- \u2705 Full Documentation on ReadTheDocs\n- \u2705 MinIO Integration Testing\n- \ud83d\udea7 Async/await support (AsyncStorageManager, AsyncStorageNode) - v0.3.0 in development\n- \ud83c\udfaf Ready for early adopters and production testing\n- \u23f3 Extended GCS/Azure integration testing\n\n**Roadmap:**\n- v0.2.0 (October 2025) - Virtual nodes, tutorials, enhanced testing \u2705 **RELEASED**\n- v0.3.0 (Q4 2025) - Async support via asyncer wrapper \ud83d\udea7 **IN DEVELOPMENT**\n- v0.4.0 (Q1 2026) - Native async API, runtime mount management\n- v1.0.0 (2026) - Production-ready, stable API guarantee\n\n## Contributing\n\nContributions welcome! The library is in beta with a stable API.\n\n**How to contribute:**\n1. Review the [API Design Document](API_DESIGN.md)\n2. Check existing [tests](tests/) to understand behavior\n3. Open an issue to discuss major changes\n4. Submit PRs with tests\n\n**Testing contributions:**\n- Add tests for GCS and Azure backends\n- Improve test coverage (target: 90%+)\n- Add integration tests for edge cases\n\n## License\n\nMIT License - See [LICENSE](LICENSE) for details\n\n---\n\n**Made with \u2764\ufe0f by the Genropy team**\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Unified storage abstraction for Genropy framework",
"version": "0.3.0",
"project_urls": {
"Bug Tracker": "https://github.com/genropy/genro-storage/issues",
"Documentation": "https://genro-storage.readthedocs.io",
"Homepage": "https://github.com/genropy/genro-storage",
"Repository": "https://github.com/genropy/genro-storage"
},
"split_keywords": [
"storage",
" filesystem",
" s3",
" cloud",
" genropy"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "59b6bd64e3c827ab20f89f5177bc68114e1699053b0de9faa7530c9312e1adb9",
"md5": "77f03cf7f3a0f945194cfe72128146bd",
"sha256": "024fa16a319a12d018df6890ecdc6de94ec5eb9f281141dbbf88174356b9fea5"
},
"downloads": -1,
"filename": "genro_storage-0.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "77f03cf7f3a0f945194cfe72128146bd",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 57407,
"upload_time": "2025-10-29T05:25:29",
"upload_time_iso_8601": "2025-10-29T05:25:29.727429Z",
"url": "https://files.pythonhosted.org/packages/59/b6/bd64e3c827ab20f89f5177bc68114e1699053b0de9faa7530c9312e1adb9/genro_storage-0.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "0386291145bd054168d3cd7a322e5eb7a6e1fb6f0d8fa619272abde9c342ecb0",
"md5": "bf6b1687859f150c7659a6b6e5d5969e",
"sha256": "a5c57776bd0e9a285cb5ebbd6dea08851c843637c1a45f601233742b6b8bf5ff"
},
"downloads": -1,
"filename": "genro_storage-0.3.0.tar.gz",
"has_sig": false,
"md5_digest": "bf6b1687859f150c7659a6b6e5d5969e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 85129,
"upload_time": "2025-10-29T05:25:31",
"upload_time_iso_8601": "2025-10-29T05:25:31.189107Z",
"url": "https://files.pythonhosted.org/packages/03/86/291145bd054168d3cd7a322e5eb7a6e1fb6f0d8fa619272abde9c342ecb0/genro_storage-0.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-29 05:25:31",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "genropy",
"github_project": "genro-storage",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "genro-storage"
}
