# AutoStore - File Storage With Automatic Backend Detection
AutoStore provides a dictionary-like interface for reading and writing files from cloud storage and local filesystems.
AutoStore eliminates the cognitive overhead of managing different file formats and storage backends, letting you focus on your data and analysis rather than the mechanics of file I/O. It automatically detects storage backends from URI prefixes (s3://, gcs://, etc.), handles file format detection, type inference, and provides a clean, intuitive API for data persistence across local and cloud storage.
## Features
- Automatically detects storage type from URI prefixes
- Use multiple S3-compatible services (AWS, Conductor, MinIO, etc.) with different configurations
- Access any storage backend from a single store instance using URI syntax
- Automatically handles both individual files and multi-file datasets (parquet, CSV collections)
- Caching system with configurable expiration reduces redundant downloads
- Built-in support for Polars DataFrames, JSON, CSV, images, PyTorch models, NumPy arrays, and more
- Configuration with IDE support and validation for each service
## Getting Started
AutoStore requires Python 3.10+ and can be installed via pip.
```bash
pip install autostore
```
### Basic Usage - Zero Configuration
```python
from autostore import AutoStore
# Local storage - no configuration needed
store = AutoStore("./data")
# Write data - automatically saves with appropriate extensions
store["my_dataframe"] = df # ./data/my_dataframe.parquet
store["config"] = {"key": "value"} # ./data/config.json
store["logs"] = [{"event": "start"}] # ./data/logs.jsonl
# Read data
df = store["my_dataframe"] # Returns a Polars DataFrame
config = store["config"] # Returns a dict
logs = store["logs"] # Returns a list of dicts
```
### Cloud Storage - Automatic Detection
```python
from autostore import AutoStore
from autostore.s3 import S3Options
# S3 - automatically detected from s3:// prefix
store = AutoStore(
"s3://my-bucket/data/",
profile_name="my-profile",
cache_enabled=True
)
# Or with explicit options
options = S3Options(
profile_name="my-profile",
region_name="us-east-1",
cache_enabled=True,
cache_expiry_hours=12
)
store = AutoStore("s3://my-bucket/data/", options=options)
# Write data to S3
store["experiment/results"] = {"accuracy": 0.95, "epochs": 100}
# Read data from S3
results = store["experiment/results"] # Uses cache on subsequent loads
```
### Cross-Backend Access
```python
from autostore import AutoStore
# Create a local store as primary backend
store = AutoStore("./local-cache", cache_enabled=True)
# Access different backends using full URIs
store["local_file"] = {"type": "local"} # Primary backend
store["s3://bucket/remote.json"] = {"type": "s3"} # S3 backend
# Read from any backend
local_data = store["local_file"] # From local
s3_data = store["s3://bucket/remote.json"] # From S3
```
### Multiple S3-Compatible Services
AutoStore supports multiple S3-compatible services with different configurations:
```python
from autostore import AutoStore
from autostore.s3 import S3Options
# Register new schemes for different S3-compatible services
AutoStore.register_scheme("minio", "autostore.s3")
AutoStore.register_scheme("digitalocean", "autostore.s3")
# Create service-specific options with different configurations
aws_options = S3Options(
scheme="s3",
profile_name="aws-production",
region_name="us-east-1",
cache_enabled=True
)
minio_options = S3Options(
scheme="minio",
endpoint_url="https://minio.mycompany.com",
aws_access_key_id="minioadmin",
aws_secret_access_key="minioadmin",
region_name="us-east-1"
)
digitalocean_options = S3Options(
scheme="digitalocean",
endpoint_url="https://nyc3.digitaloceanspaces.com",
region_name="nyc3",
cache_enabled=True
)
# Create AutoStore with multiple backend options
store = AutoStore(
"./local-cache",
options=[aws_options, minio_options, digitalocean_options]
)
# Each scheme automatically uses its appropriate configuration
store["s3://aws-bucket/data.json"] = {"source": "aws"}
store["minio://my-bucket/data.json"] = {"source": "minio"}
store["digitalocean://my-space/data.json"] = {"source": "digitalocean"}
# Cross-backend data access with automatic option selection
aws_data = store["s3://aws-bucket/data.json"]
minio_data = store["minio://my-bucket/data.json"]
digitalocean_data = store["digitalocean://my-space/data.json"]
```
### Dataset Support
```python
from autostore import AutoStore
# Automatically detects and handles datasets
# For example, if you have multiple parquet files in an S3 bucket:
# ├── weather
# │ ├── 2024
# │ │ ├── january.parquet
# │ │ ├── february.parquet
# │ │ └── march.parquet
store = AutoStore("s3://my-bucket/datasets/")
# Access parquet dataset (multiple files)
weather_data = store["weather/2024/"] # Loads entire dataset as LazyFrame
# Access individual file
single_file = store["weather/2024/january.parquet"]
# List files in dataset
files = list(store.list_files("weather/2024/*", recursive=True))
```
## AutoPath - Path-like Interface
AutoPath provides a pathlib.Path-like interface for unified access to both local filesystem and cloud storage. It combines the familiar Path API with AutoStore's automatic backend detection and data handling capabilities.
### Basic AutoPath Usage
```python
from autostore import AutoStore, AutoPath
from autostore.s3 import S3Options
# Create a store with multiple backends
store = AutoStore(
"./local-data",
options=[
S3Options(
scheme="s3",
profile_name="aws-prod",
cache_enabled=True,
cache_expiry_hours=6
)
]
)
# Create AutoPath instances
local_path = AutoPath("./local-data/config.json", store=store)
s3_path = AutoPath("s3://my-bucket/data/results.parquet", store=store)
# Path operations work the same for local and cloud storage
config_exists = local_path.exists() # True/False
results_exists = s3_path.exists() # True/False
# Read files as text or bytes
config_text = local_path.read_text() # Read as string
results_bytes = s3_path.read_bytes() # Read as bytes
# Write files
local_path.write_text('{"key": "value"}') # Write string
s3_path.write_bytes(b"binary data") # Write bytes
```
### Path Manipulation and Navigation
```python
# Path joining works like pathlib.Path
data_dir = AutoPath("s3://my-bucket/datasets", store=store)
experiment_dir = data_dir / "experiment_1"
results_file = experiment_dir / "results.parquet"
print(results_file) # s3://my-bucket/datasets/experiment_1/results.parquet
# Path properties
print(results_file.name) # results.parquet
print(results_file.stem) # results
print(results_file.suffix) # .parquet
print(results_file.parent) # s3://my-bucket/datasets/experiment_1
# Navigate parent directories
parent = results_file.parent
grandparent = parent.parent
all_parents = results_file.parents # List of all parent directories
```
### File and Directory Operations
```python
# File operations
if results_file.exists():
print("File exists")
if results_file.is_file():
print("It's a file")
if data_dir.is_dir():
print("It's a directory")
# Directory listing
for item in data_dir.iterdir():
print(f"Found: {item}")
if item.is_file():
print(f" File size: {item.stat().size}")
# Glob patterns
for parquet_file in data_dir.glob("**/*.parquet"):
print(f"Parquet file: {parquet_file}")
for csv_file in experiment_dir.glob("*.csv"):
print(f"CSV file: {csv_file}")
```
### Directory Management
```python
# For local paths, this creates real directories
local_dir = AutoPath("./data/analysis", store=store)
local_dir.mkdir(parents=True, exist_ok=True)
# Remove directories
empty_dir = AutoPath("s3://my-bucket/empty-folder", store=store)
empty_dir.rmdir() # Remove empty directory
# Delete files or directories with contents
old_experiment = AutoPath("s3://my-bucket/old-experiment", store=store)
old_experiment.delete() # Recursively deletes all contents
```
### File Transfer Operations
```python
# Copy files between any backends
local_file = AutoPath("./data/model.pt", store=store)
s3_backup = AutoPath("s3://backup-bucket/models/model.pt", store=store)
# Copy local file to S3
local_file.copy_to(s3_backup)
# Move files
temp_file = AutoPath("./temp/processing.csv", store=store)
final_location = AutoPath("s3://data-bucket/processed/final.csv", store=store)
temp_file.move_to(final_location)
# Upload from local filesystem
local_source = "./analysis/results.xlsx"
s3_destination = AutoPath("s3://reports/analysis/results.xlsx", store=store)
s3_destination.upload_from(local_source)
# Download to local filesystem
s3_source = AutoPath("s3://data/large_dataset.parquet", store=store)
local_destination = "./downloads/dataset.parquet"
s3_source.download_to(local_destination)
```
### Data Loading with Automatic Format Detection
AutoPath integrates with AutoStore's handler system to load data in the appropriate format based on file extensions or content type. It supports Polars DataFrames, JSON, CSV, and more.
```python
# Load data with automatic format detection
parquet_path = AutoPath("s3://data/sales.parquet", store=store)
df = parquet_path.load() # Returns Polars DataFrame
json_path = AutoPath("s3://config/settings.json", store=store)
settings = json_path.load() # Returns dict
# Force specific format
csv_as_parquet = AutoPath("s3://data/data.csv", store=store)
df = csv_as_parquet.load(format="parquet") # Force parquet parsing
# Bypass cache
fresh_data = parquet_path.load(ignore_cache=True)
# Save data with automatic format detection
results = {"accuracy": 0.95, "model": "transformer"}
results_path = AutoPath("s3://experiments/run_001/results.json", store=store)
results_path.save(results) # Automatically saves as JSON
# Save DataFrame
import polars as pl
df = pl.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
data_path = AutoPath("s3://datasets/processed.parquet", store=store)
data_path.save(df) # Automatically saves as Parquet
```
### AutoPath without Explicit Store
AutoPath can automatically create appropriate stores:
```python
# For local paths
local_path = AutoPath("./data/file.json") # Creates local store automatically
# For S3 URIs
s3_path = AutoPath("s3://bucket/file.json") # Creates S3 store with default options
# Path operations work the same
data = s3_path.load()
s3_path.save({"new": "data"})
```
### Advanced Path Operations
```python
# Path pattern matching
log_path = AutoPath("s3://logs/app.2024-01-15.log", store=store)
if log_path.match("*.log"):
print("It's a log file")
# Relative paths
base_path = AutoPath("s3://data/experiments", store=store)
result_path = AutoPath("s3://data/experiments/run_1/results.json", store=store)
relative = result_path.relative_to(base_path) # "run_1/results.json"
# Path transformations
config_path = AutoPath("s3://app/config.yaml", store=store)
backup_path = config_path.with_suffix(".yaml.bak") # config.yaml.bak
renamed_path = config_path.with_name("new_config.yaml") # new_config.yaml
stemmed_path = config_path.with_stem("production") # production.yaml
# Absolute and URI representations
print(local_path.as_posix()) # Forward slashes
print(local_path.as_uri()) # file:// URI
print(s3_path.as_uri()) # s3:// URI
print(s3_path.is_absolute()) # True for URIs
```
### Integration Example
```python
from autostore import AutoStore, AutoPath
from autostore.s3 import S3Options
import polars as pl
# Setup store with caching
store = AutoStore(
"./cache",
options=[S3Options(
scheme="s3",
profile_name="production",
cache_enabled=True,
cache_expiry_hours=0 # Never expire
)]
)
# Define paths
raw_data = AutoPath("s3://raw-data/sales/2024/", store=store)
processed_data = AutoPath("s3://processed/sales_summary.parquet", store=store)
local_backup = AutoPath("./backups/sales_summary.parquet", store=store)
# Process data using path-like interface
if raw_data.is_dir():
# Load all files in directory as dataset
df = raw_data.load() # Loads entire directory as LazyFrame
# Process data
summary = df.group_by("region").agg([
pl.col("sales").sum().alias("total_sales"),
pl.col("sales").count().alias("transaction_count")
])
# Save processed data
processed_data.save(summary.collect())
# Create local backup
processed_data.copy_to(local_backup)
print(f"Processed {summary.height} regions")
print(f"Backup created at: {local_backup}")
```
## Supported Data Types
| Data Type | File Extension | Description | Library Required |
| -------------------------- | ---------------------- | --------------------------- | ---------------- |
| Polars DataFrame/LazyFrame | `.parquet`, `.csv` | High-performance DataFrames | polars |
| Python dict/list | `.json` | Standard JSON serialization | built-in |
| List of dicts | `.jsonl` | JSON Lines format | built-in |
| Pydantic models | `.pydantic.json` | Structured data models | pydantic |
| Python dataclasses | `.dataclass.json` | Dataclass serialization | built-in |
| String data | `.txt`, `.html`, `.md` | Plain text files | built-in |
| NumPy arrays | `.npy`, `.npz` | Numerical data | numpy |
| SciPy sparse matrices | `.sparse` | Sparse matrix data | scipy |
| PyTorch tensors/models | `.pt`, `.pth` | Deep learning models | torch |
| PIL/Pillow images | `.png`, `.jpg`, etc. | Image data | Pillow |
| YAML data | `.yaml`, `.yml` | Human-readable config files | PyYAML |
| Any Python object | `.pkl` | Pickle fallback | built-in |
## Supported Storage Backends
AutoStore automatically detects the storage backend from URI prefixes:
| Backend | URI Prefix | Options Class | Example |
| ------- | ------------------- | ------------- | ----------------------------- |
| Local | `./path` or `/path` | `Options` | `./data`, `/Users/name/files` |
| S3 | `s3://` | `S3Options` | `s3://bucket/prefix/` |
## Configuration Options
### Base Options (All Backends)
```python
from autostore import Options
base_options = Options(
cache_enabled=True, # Enable local caching
cache_dir="./cache", # Custom cache directory
cache_expiry_hours=12, # Cache expiration time (0 = never expire)
timeout=30, # Request timeout in seconds
max_retries=3, # Maximum retry attempts
retry_delay=1.0 # Delay between retries
)
```
### S3Options
```python
from autostore.s3 import S3Options
s3_options = S3Options(
# Scheme specification for multi-backend support
scheme="s3", # URI scheme this options applies to
# Authentication
aws_access_key_id="your-key",
aws_secret_access_key="your-secret",
profile_name="my-profile", # AWS profile name
# Configuration
region_name="us-east-1",
endpoint_url="custom-endpoint", # For S3-compatible services
# Performance
multipart_threshold=64 * 1024 * 1024, # Files > 64MB use multipart
multipart_chunksize=16 * 1024 * 1024, # Chunk size for uploads
max_concurrency=10, # Concurrent operations
# Inherited from Options
cache_enabled=True,
cache_expiry_hours=6 # 0 = never expire
)
```
### Usage Patterns
```python
# Method 1: Keyword arguments
store = AutoStore("s3://bucket/", profile_name="prod", cache_enabled=True)
# Method 2: Single options object
options = S3Options(scheme="s3", profile_name="prod", cache_enabled=True)
store = AutoStore("s3://bucket/", options=options)
# Method 3: Multiple options for different services
aws_options = S3Options(scheme="s3", profile_name="aws-prod")
minio_options = S3Options(scheme="minio", endpoint_url="https://minio.example.com")
store = AutoStore("./cache", options=[aws_options, minio_options])
# Method 4: Mixed (options object + additional kwargs)
base_options = S3Options(scheme="s3", profile_name="prod")
store = AutoStore("s3://bucket/", options=base_options, cache_enabled=True)
```
## Advanced Features
### Backend Management
```python
# Register new S3-compatible services
AutoStore.register_scheme("minio", "autostore.s3")
AutoStore.register_scheme("digitalocean", "autostore.s3")
# Check supported backends
backends = store.get_supported_backends()
print(f"Available: {backends}") # ['s3', 'minio', 'digitalocean', 'file', '']
# View active backends
active = store.list_active_backends()
print(f"Active: {active}") # ['primary: ./data', 's3: s3://bucket/', 'minio: minio://bucket/']
# Backend auto-loading with appropriate options
data = store["s3://bucket/file.json"] # Uses AWS S3 options
data = store["minio://bucket/file.json"] # Uses MinIO options
data = store["digitalocean://space/file.json"] # Uses DigitalOcean options
```
### Dataset Operations
```python
# Dataset detection
is_dataset = store.primary_backend.is_dataset("path/to/data/")
is_directory = store.primary_backend.is_directory("path/")
# List dataset files
files = list(store.list_files("dataset/*", recursive=True))
# Load entire dataset (for parquet/CSV collections)
lazy_frame = store["weather_data/"] # Loads all parquet files as one LazyFrame
```
### Caching System
AutoStore includes caching that:
- Stores frequently accessed files locally
- Uses ETags for cache validation
- Automatically expires old cache entries (or never expires if cache_expiry_hours=0)
- Works across all backends
```python
# Enable caching for any backend
store = AutoStore("s3://bucket/", cache_enabled=True, cache_expiry_hours=6)
# Never expire cache entries (useful for immutable data)
store = AutoStore("s3://bucket/", cache_enabled=True, cache_expiry_hours=0)
# Cache management
store.cleanup_cache() # Remove expired cache entries
# Check cache status
metadata = store.get_metadata("large_file")
print(f"File size: {metadata.size} bytes")
print(f"ETag: {metadata.etag}")
```
### Custom Data Handlers
Add support for new data types by creating custom handlers:
```python
from pathlib import Path
from autostore.autostore import DataHandler
class CustomLogHandler(DataHandler):
def can_handle_extension(self, extension: str) -> bool:
return extension.lower() == ".log"
def can_handle_data(self, data) -> bool:
return isinstance(data, list) and all(
isinstance(item, dict) and "timestamp" in item
for item in data
)
def read_from_file(self, file_path: Path, file_extension: str):
logs = []
with open(file_path, 'r') as f:
for line in f:
if line.strip():
logs.append(json.loads(line))
return logs
def write_to_file(self, data, file_path: Path, file_extension: str):
file_path.parent.mkdir(parents=True, exist_ok=True)
with open(file_path, 'w') as f:
for entry in data:
f.write(json.dumps(entry) + '\n')
@property
def extensions(self):
return [".log"]
@property
def priority(self):
return 15
# Register the handler
store.register_handler(CustomLogHandler())
```
### File Operations
```python
# Check existence
if "config" in store:
print("Config file exists")
# List all files
for key in store.keys():
print(f"File: {key}")
# Get file metadata
metadata = store.get_metadata("large_dataset")
print(f"Size: {metadata.size} bytes")
print(f"Modified: {metadata.modified_time}")
# Copy and move files
store.copy("original", "backup")
store.move("temp_file", "permanent_file")
# Delete files
del store["old_data"]
```
### Context Management
```python
# Automatic cleanup of temporary files and cache
with AutoStore("./data", config=config) as store:
store["data"] = large_dataset
results = store["data"]
# Temporary files are automatically cleaned up here
```
## Performance Considerations
### Large File Handling
AutoStore automatically optimizes for large files:
- Multipart uploads/downloads for files > 64MB
- Configurable chunk sizes and concurrency
- Streaming operations to minimize memory usage
## When to Use AutoStore
Choose AutoStore when you need:
- Multi-cloud data access with seamless backend switching
- Dataset processing with automatic detection of file collections
- Zero-configuration setup for rapid prototyping and development
- Cross-backend operations without managing multiple client libraries
- Data science projects with mixed file types across storage systems
- Type-safe configuration with IDE support and validation
- Intelligent caching to optimize cloud storage costs and performance
Don't choose AutoStore when:
- You need complex queries or relational operations (use databases)
- You only work with one data type and one storage backend consistently
- You need zero dependencies (use built-in libraries like shelve)
- You require advanced database features like transactions or indexing
- You need fine-grained control over every storage operation
## Changes
- 0.1.14 - AutoPath now has a load and save method that uses the built-in handlers
- 0.1.13 - Added AutoPath class for path-like operations with automatic backend detection
- AutoPath supports all storage operations like read, write, upload, download, delete, etc.
- AutoPath can be used in place of AutoStore for path-like interactions
- 0.1.8 - Auto scheme registration enhancement
- 0.1.7 - Cache expiry can be set to 0 to never expire cache entries.
- 0.1.6 - Scheme-based backend detection and Options system with automatic backend detection from URI schemes
- Unified Options dataclass system replacing separate config classes
- Cross-backend access from single store instance
- Dataset support with automatic multi-file detection
- Enhanced error handling with dependency management
- Breaking: Removed manual backend registration
- Breaking: Replaced `S3StorageConfig` with `S3Options`
- 0.1.5 - Added StorePath to use the Autostore instance in path-like operations
- 0.1.4 - parquet and csv are loaded as LazyFrames by default and sparse matrices are now saved as .sparse.npz
- 0.1.3
- Refactored to use different storage backends including local file system and S3.
- Implement S3 storage backend with basic operations
- Added S3StorageConfig for configuration management.
- Implemented S3Backend class for handling S3 interactions.
- Included methods for file operations: upload, download, delete, copy, move, and list files.
- Added support for directory-like structures in S3.
- Implemented metadata retrieval for files.
- Integrated error handling for common S3 exceptions.
- Added support for multipart uploads and downloads.
- Included utility functions for path parsing and glob pattern matching.
- Calling store.keys() now only returns keys without extensions.
- 0.1.2 - config, setup_logging, and load_dotenv are now imported at the module top level
- 0.1.1 - Added config, setup_logging, and load_dotenv
- 0.1.0 - Initial release
Raw data
{
"_id": null,
"home_page": null,
"name": "autostore",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "file storage, file management",
"author": null,
"author_email": "Asif Rahman <asiftr@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/10/e3/5615191656a05b822b7dfb2021af6753c8849949d2140f36921deeb3f84e/autostore-0.1.18.tar.gz",
"platform": null,
"description": "# AutoStore - File Storage With Automatic Backend Detection\n\nAutoStore provides a dictionary-like interface for reading and writing files from cloud storage and local filesystems.\n\nAutoStore eliminates the cognitive overhead of managing different file formats and storage backends, letting you focus on your data and analysis rather than the mechanics of file I/O. It automatically detects storage backends from URI prefixes (s3://, gcs://, etc.), handles file format detection, type inference, and provides a clean, intuitive API for data persistence across local and cloud storage.\n\n## Features\n\n- Automatically detects storage type from URI prefixes\n- Use multiple S3-compatible services (AWS, Conductor, MinIO, etc.) with different configurations\n- Access any storage backend from a single store instance using URI syntax\n- Automatically handles both individual files and multi-file datasets (parquet, CSV collections)\n- Caching system with configurable expiration reduces redundant downloads\n- Built-in support for Polars DataFrames, JSON, CSV, images, PyTorch models, NumPy arrays, and more\n- Configuration with IDE support and validation for each service\n\n## Getting Started\n\nAutoStore requires Python 3.10+ and can be installed via pip.\n\n```bash\npip install autostore\n```\n\n### Basic Usage - Zero Configuration\n\n```python\nfrom autostore import AutoStore\n\n# Local storage - no configuration needed\nstore = AutoStore(\"./data\")\n\n# Write data - automatically saves with appropriate extensions\nstore[\"my_dataframe\"] = df # ./data/my_dataframe.parquet\nstore[\"config\"] = {\"key\": \"value\"} # ./data/config.json\nstore[\"logs\"] = [{\"event\": \"start\"}] # ./data/logs.jsonl\n\n# Read data\ndf = store[\"my_dataframe\"] # Returns a Polars DataFrame\nconfig = store[\"config\"] # Returns a dict\nlogs = store[\"logs\"] # Returns a list of dicts\n```\n\n### Cloud Storage - Automatic Detection\n\n```python\nfrom autostore import AutoStore\nfrom autostore.s3 import S3Options\n\n# S3 - automatically detected from s3:// prefix\nstore = AutoStore(\n \"s3://my-bucket/data/\",\n profile_name=\"my-profile\",\n cache_enabled=True\n)\n\n# Or with explicit options\noptions = S3Options(\n profile_name=\"my-profile\",\n region_name=\"us-east-1\",\n cache_enabled=True,\n cache_expiry_hours=12\n)\nstore = AutoStore(\"s3://my-bucket/data/\", options=options)\n\n# Write data to S3\nstore[\"experiment/results\"] = {\"accuracy\": 0.95, \"epochs\": 100}\n\n# Read data from S3\nresults = store[\"experiment/results\"] # Uses cache on subsequent loads\n```\n\n### Cross-Backend Access\n\n```python\nfrom autostore import AutoStore\n\n# Create a local store as primary backend\nstore = AutoStore(\"./local-cache\", cache_enabled=True)\n\n# Access different backends using full URIs\nstore[\"local_file\"] = {\"type\": \"local\"} # Primary backend\nstore[\"s3://bucket/remote.json\"] = {\"type\": \"s3\"} # S3 backend\n\n# Read from any backend\nlocal_data = store[\"local_file\"] # From local\ns3_data = store[\"s3://bucket/remote.json\"] # From S3\n```\n\n### Multiple S3-Compatible Services\n\nAutoStore supports multiple S3-compatible services with different configurations:\n\n```python\nfrom autostore import AutoStore\nfrom autostore.s3 import S3Options\n\n# Register new schemes for different S3-compatible services\nAutoStore.register_scheme(\"minio\", \"autostore.s3\")\nAutoStore.register_scheme(\"digitalocean\", \"autostore.s3\")\n\n# Create service-specific options with different configurations\naws_options = S3Options(\n scheme=\"s3\",\n profile_name=\"aws-production\",\n region_name=\"us-east-1\",\n cache_enabled=True\n)\n\nminio_options = S3Options(\n scheme=\"minio\",\n endpoint_url=\"https://minio.mycompany.com\",\n aws_access_key_id=\"minioadmin\",\n aws_secret_access_key=\"minioadmin\",\n region_name=\"us-east-1\"\n)\n\ndigitalocean_options = S3Options(\n scheme=\"digitalocean\",\n endpoint_url=\"https://nyc3.digitaloceanspaces.com\",\n region_name=\"nyc3\",\n cache_enabled=True\n)\n\n# Create AutoStore with multiple backend options\nstore = AutoStore(\n \"./local-cache\",\n options=[aws_options, minio_options, digitalocean_options]\n)\n\n# Each scheme automatically uses its appropriate configuration\nstore[\"s3://aws-bucket/data.json\"] = {\"source\": \"aws\"}\nstore[\"minio://my-bucket/data.json\"] = {\"source\": \"minio\"}\nstore[\"digitalocean://my-space/data.json\"] = {\"source\": \"digitalocean\"}\n\n# Cross-backend data access with automatic option selection\naws_data = store[\"s3://aws-bucket/data.json\"]\nminio_data = store[\"minio://my-bucket/data.json\"]\ndigitalocean_data = store[\"digitalocean://my-space/data.json\"]\n```\n\n### Dataset Support\n\n```python\nfrom autostore import AutoStore\n\n# Automatically detects and handles datasets\n# For example, if you have multiple parquet files in an S3 bucket:\n# \u251c\u2500\u2500 weather\n# \u2502 \u251c\u2500\u2500 2024\n# \u2502 \u2502 \u251c\u2500\u2500 january.parquet\n# \u2502 \u2502 \u251c\u2500\u2500 february.parquet\n# \u2502 \u2502 \u2514\u2500\u2500 march.parquet\nstore = AutoStore(\"s3://my-bucket/datasets/\")\n\n# Access parquet dataset (multiple files)\nweather_data = store[\"weather/2024/\"] # Loads entire dataset as LazyFrame\n\n# Access individual file\nsingle_file = store[\"weather/2024/january.parquet\"]\n\n# List files in dataset\nfiles = list(store.list_files(\"weather/2024/*\", recursive=True))\n```\n\n## AutoPath - Path-like Interface\n\nAutoPath provides a pathlib.Path-like interface for unified access to both local filesystem and cloud storage. It combines the familiar Path API with AutoStore's automatic backend detection and data handling capabilities.\n\n### Basic AutoPath Usage\n\n```python\nfrom autostore import AutoStore, AutoPath\nfrom autostore.s3 import S3Options\n\n# Create a store with multiple backends\nstore = AutoStore(\n \"./local-data\",\n options=[\n S3Options(\n scheme=\"s3\",\n profile_name=\"aws-prod\",\n cache_enabled=True,\n cache_expiry_hours=6\n )\n ]\n)\n\n# Create AutoPath instances\nlocal_path = AutoPath(\"./local-data/config.json\", store=store)\ns3_path = AutoPath(\"s3://my-bucket/data/results.parquet\", store=store)\n\n# Path operations work the same for local and cloud storage\nconfig_exists = local_path.exists() # True/False\nresults_exists = s3_path.exists() # True/False\n\n# Read files as text or bytes\nconfig_text = local_path.read_text() # Read as string\nresults_bytes = s3_path.read_bytes() # Read as bytes\n\n# Write files\nlocal_path.write_text('{\"key\": \"value\"}') # Write string\ns3_path.write_bytes(b\"binary data\") # Write bytes\n```\n\n### Path Manipulation and Navigation\n\n```python\n# Path joining works like pathlib.Path\ndata_dir = AutoPath(\"s3://my-bucket/datasets\", store=store)\nexperiment_dir = data_dir / \"experiment_1\"\nresults_file = experiment_dir / \"results.parquet\"\n\nprint(results_file) # s3://my-bucket/datasets/experiment_1/results.parquet\n\n# Path properties\nprint(results_file.name) # results.parquet\nprint(results_file.stem) # results\nprint(results_file.suffix) # .parquet\nprint(results_file.parent) # s3://my-bucket/datasets/experiment_1\n\n# Navigate parent directories\nparent = results_file.parent\ngrandparent = parent.parent\nall_parents = results_file.parents # List of all parent directories\n```\n\n### File and Directory Operations\n\n```python\n# File operations\nif results_file.exists():\n print(\"File exists\")\n\nif results_file.is_file():\n print(\"It's a file\")\n\nif data_dir.is_dir():\n print(\"It's a directory\")\n\n# Directory listing\nfor item in data_dir.iterdir():\n print(f\"Found: {item}\")\n if item.is_file():\n print(f\" File size: {item.stat().size}\")\n\n# Glob patterns\nfor parquet_file in data_dir.glob(\"**/*.parquet\"):\n print(f\"Parquet file: {parquet_file}\")\n\nfor csv_file in experiment_dir.glob(\"*.csv\"):\n print(f\"CSV file: {csv_file}\")\n```\n\n### Directory Management\n\n```python\n# For local paths, this creates real directories\nlocal_dir = AutoPath(\"./data/analysis\", store=store)\nlocal_dir.mkdir(parents=True, exist_ok=True)\n\n# Remove directories\nempty_dir = AutoPath(\"s3://my-bucket/empty-folder\", store=store)\nempty_dir.rmdir() # Remove empty directory\n\n# Delete files or directories with contents\nold_experiment = AutoPath(\"s3://my-bucket/old-experiment\", store=store)\nold_experiment.delete() # Recursively deletes all contents\n```\n\n### File Transfer Operations\n\n```python\n# Copy files between any backends\nlocal_file = AutoPath(\"./data/model.pt\", store=store)\ns3_backup = AutoPath(\"s3://backup-bucket/models/model.pt\", store=store)\n\n# Copy local file to S3\nlocal_file.copy_to(s3_backup)\n\n# Move files\ntemp_file = AutoPath(\"./temp/processing.csv\", store=store)\nfinal_location = AutoPath(\"s3://data-bucket/processed/final.csv\", store=store)\ntemp_file.move_to(final_location)\n\n# Upload from local filesystem\nlocal_source = \"./analysis/results.xlsx\"\ns3_destination = AutoPath(\"s3://reports/analysis/results.xlsx\", store=store)\ns3_destination.upload_from(local_source)\n\n# Download to local filesystem\ns3_source = AutoPath(\"s3://data/large_dataset.parquet\", store=store)\nlocal_destination = \"./downloads/dataset.parquet\"\ns3_source.download_to(local_destination)\n```\n\n### Data Loading with Automatic Format Detection\n\nAutoPath integrates with AutoStore's handler system to load data in the appropriate format based on file extensions or content type. It supports Polars DataFrames, JSON, CSV, and more.\n\n```python\n# Load data with automatic format detection\nparquet_path = AutoPath(\"s3://data/sales.parquet\", store=store)\ndf = parquet_path.load() # Returns Polars DataFrame\n\njson_path = AutoPath(\"s3://config/settings.json\", store=store)\nsettings = json_path.load() # Returns dict\n\n# Force specific format\ncsv_as_parquet = AutoPath(\"s3://data/data.csv\", store=store)\ndf = csv_as_parquet.load(format=\"parquet\") # Force parquet parsing\n\n# Bypass cache\nfresh_data = parquet_path.load(ignore_cache=True)\n\n# Save data with automatic format detection\nresults = {\"accuracy\": 0.95, \"model\": \"transformer\"}\nresults_path = AutoPath(\"s3://experiments/run_001/results.json\", store=store)\nresults_path.save(results) # Automatically saves as JSON\n\n# Save DataFrame\nimport polars as pl\ndf = pl.DataFrame({\"a\": [1, 2, 3], \"b\": [4, 5, 6]})\ndata_path = AutoPath(\"s3://datasets/processed.parquet\", store=store)\ndata_path.save(df) # Automatically saves as Parquet\n```\n\n### AutoPath without Explicit Store\n\nAutoPath can automatically create appropriate stores:\n\n```python\n# For local paths\nlocal_path = AutoPath(\"./data/file.json\") # Creates local store automatically\n\n# For S3 URIs\ns3_path = AutoPath(\"s3://bucket/file.json\") # Creates S3 store with default options\n\n# Path operations work the same\ndata = s3_path.load()\ns3_path.save({\"new\": \"data\"})\n```\n\n### Advanced Path Operations\n\n```python\n# Path pattern matching\nlog_path = AutoPath(\"s3://logs/app.2024-01-15.log\", store=store)\nif log_path.match(\"*.log\"):\n print(\"It's a log file\")\n\n# Relative paths\nbase_path = AutoPath(\"s3://data/experiments\", store=store)\nresult_path = AutoPath(\"s3://data/experiments/run_1/results.json\", store=store)\nrelative = result_path.relative_to(base_path) # \"run_1/results.json\"\n\n# Path transformations\nconfig_path = AutoPath(\"s3://app/config.yaml\", store=store)\nbackup_path = config_path.with_suffix(\".yaml.bak\") # config.yaml.bak\nrenamed_path = config_path.with_name(\"new_config.yaml\") # new_config.yaml\nstemmed_path = config_path.with_stem(\"production\") # production.yaml\n\n# Absolute and URI representations\nprint(local_path.as_posix()) # Forward slashes\nprint(local_path.as_uri()) # file:// URI\nprint(s3_path.as_uri()) # s3:// URI\nprint(s3_path.is_absolute()) # True for URIs\n```\n\n### Integration Example\n\n```python\nfrom autostore import AutoStore, AutoPath\nfrom autostore.s3 import S3Options\nimport polars as pl\n\n# Setup store with caching\nstore = AutoStore(\n \"./cache\",\n options=[S3Options(\n scheme=\"s3\",\n profile_name=\"production\",\n cache_enabled=True,\n cache_expiry_hours=0 # Never expire\n )]\n)\n\n# Define paths\nraw_data = AutoPath(\"s3://raw-data/sales/2024/\", store=store)\nprocessed_data = AutoPath(\"s3://processed/sales_summary.parquet\", store=store)\nlocal_backup = AutoPath(\"./backups/sales_summary.parquet\", store=store)\n\n# Process data using path-like interface\nif raw_data.is_dir():\n # Load all files in directory as dataset\n df = raw_data.load() # Loads entire directory as LazyFrame\n\n # Process data\n summary = df.group_by(\"region\").agg([\n pl.col(\"sales\").sum().alias(\"total_sales\"),\n pl.col(\"sales\").count().alias(\"transaction_count\")\n ])\n\n # Save processed data\n processed_data.save(summary.collect())\n\n # Create local backup\n processed_data.copy_to(local_backup)\n\n print(f\"Processed {summary.height} regions\")\n print(f\"Backup created at: {local_backup}\")\n```\n\n## Supported Data Types\n\n| Data Type | File Extension | Description | Library Required |\n| -------------------------- | ---------------------- | --------------------------- | ---------------- |\n| Polars DataFrame/LazyFrame | `.parquet`, `.csv` | High-performance DataFrames | polars |\n| Python dict/list | `.json` | Standard JSON serialization | built-in |\n| List of dicts | `.jsonl` | JSON Lines format | built-in |\n| Pydantic models | `.pydantic.json` | Structured data models | pydantic |\n| Python dataclasses | `.dataclass.json` | Dataclass serialization | built-in |\n| String data | `.txt`, `.html`, `.md` | Plain text files | built-in |\n| NumPy arrays | `.npy`, `.npz` | Numerical data | numpy |\n| SciPy sparse matrices | `.sparse` | Sparse matrix data | scipy |\n| PyTorch tensors/models | `.pt`, `.pth` | Deep learning models | torch |\n| PIL/Pillow images | `.png`, `.jpg`, etc. | Image data | Pillow |\n| YAML data | `.yaml`, `.yml` | Human-readable config files | PyYAML |\n| Any Python object | `.pkl` | Pickle fallback | built-in |\n\n## Supported Storage Backends\n\nAutoStore automatically detects the storage backend from URI prefixes:\n\n| Backend | URI Prefix | Options Class | Example |\n| ------- | ------------------- | ------------- | ----------------------------- |\n| Local | `./path` or `/path` | `Options` | `./data`, `/Users/name/files` |\n| S3 | `s3://` | `S3Options` | `s3://bucket/prefix/` |\n\n## Configuration Options\n\n### Base Options (All Backends)\n\n```python\nfrom autostore import Options\n\nbase_options = Options(\n cache_enabled=True, # Enable local caching\n cache_dir=\"./cache\", # Custom cache directory\n cache_expiry_hours=12, # Cache expiration time (0 = never expire)\n timeout=30, # Request timeout in seconds\n max_retries=3, # Maximum retry attempts\n retry_delay=1.0 # Delay between retries\n)\n```\n\n### S3Options\n\n```python\nfrom autostore.s3 import S3Options\n\ns3_options = S3Options(\n # Scheme specification for multi-backend support\n scheme=\"s3\", # URI scheme this options applies to\n\n # Authentication\n aws_access_key_id=\"your-key\",\n aws_secret_access_key=\"your-secret\",\n profile_name=\"my-profile\", # AWS profile name\n\n # Configuration\n region_name=\"us-east-1\",\n endpoint_url=\"custom-endpoint\", # For S3-compatible services\n\n # Performance\n multipart_threshold=64 * 1024 * 1024, # Files > 64MB use multipart\n multipart_chunksize=16 * 1024 * 1024, # Chunk size for uploads\n max_concurrency=10, # Concurrent operations\n\n # Inherited from Options\n cache_enabled=True,\n cache_expiry_hours=6 # 0 = never expire\n)\n```\n\n### Usage Patterns\n\n```python\n# Method 1: Keyword arguments\nstore = AutoStore(\"s3://bucket/\", profile_name=\"prod\", cache_enabled=True)\n\n# Method 2: Single options object\noptions = S3Options(scheme=\"s3\", profile_name=\"prod\", cache_enabled=True)\nstore = AutoStore(\"s3://bucket/\", options=options)\n\n# Method 3: Multiple options for different services\naws_options = S3Options(scheme=\"s3\", profile_name=\"aws-prod\")\nminio_options = S3Options(scheme=\"minio\", endpoint_url=\"https://minio.example.com\")\nstore = AutoStore(\"./cache\", options=[aws_options, minio_options])\n\n# Method 4: Mixed (options object + additional kwargs)\nbase_options = S3Options(scheme=\"s3\", profile_name=\"prod\")\nstore = AutoStore(\"s3://bucket/\", options=base_options, cache_enabled=True)\n```\n\n## Advanced Features\n\n### Backend Management\n\n```python\n# Register new S3-compatible services\nAutoStore.register_scheme(\"minio\", \"autostore.s3\")\nAutoStore.register_scheme(\"digitalocean\", \"autostore.s3\")\n\n# Check supported backends\nbackends = store.get_supported_backends()\nprint(f\"Available: {backends}\") # ['s3', 'minio', 'digitalocean', 'file', '']\n\n# View active backends\nactive = store.list_active_backends()\nprint(f\"Active: {active}\") # ['primary: ./data', 's3: s3://bucket/', 'minio: minio://bucket/']\n\n# Backend auto-loading with appropriate options\ndata = store[\"s3://bucket/file.json\"] # Uses AWS S3 options\ndata = store[\"minio://bucket/file.json\"] # Uses MinIO options\ndata = store[\"digitalocean://space/file.json\"] # Uses DigitalOcean options\n```\n\n### Dataset Operations\n\n```python\n# Dataset detection\nis_dataset = store.primary_backend.is_dataset(\"path/to/data/\")\nis_directory = store.primary_backend.is_directory(\"path/\")\n\n# List dataset files\nfiles = list(store.list_files(\"dataset/*\", recursive=True))\n\n# Load entire dataset (for parquet/CSV collections)\nlazy_frame = store[\"weather_data/\"] # Loads all parquet files as one LazyFrame\n```\n\n### Caching System\n\nAutoStore includes caching that:\n\n- Stores frequently accessed files locally\n- Uses ETags for cache validation\n- Automatically expires old cache entries (or never expires if cache_expiry_hours=0)\n- Works across all backends\n\n```python\n# Enable caching for any backend\nstore = AutoStore(\"s3://bucket/\", cache_enabled=True, cache_expiry_hours=6)\n\n# Never expire cache entries (useful for immutable data)\nstore = AutoStore(\"s3://bucket/\", cache_enabled=True, cache_expiry_hours=0)\n\n# Cache management\nstore.cleanup_cache() # Remove expired cache entries\n\n# Check cache status\nmetadata = store.get_metadata(\"large_file\")\nprint(f\"File size: {metadata.size} bytes\")\nprint(f\"ETag: {metadata.etag}\")\n```\n\n### Custom Data Handlers\n\nAdd support for new data types by creating custom handlers:\n\n```python\nfrom pathlib import Path\nfrom autostore.autostore import DataHandler\n\nclass CustomLogHandler(DataHandler):\n def can_handle_extension(self, extension: str) -> bool:\n return extension.lower() == \".log\"\n\n def can_handle_data(self, data) -> bool:\n return isinstance(data, list) and all(\n isinstance(item, dict) and \"timestamp\" in item\n for item in data\n )\n\n def read_from_file(self, file_path: Path, file_extension: str):\n logs = []\n with open(file_path, 'r') as f:\n for line in f:\n if line.strip():\n logs.append(json.loads(line))\n return logs\n\n def write_to_file(self, data, file_path: Path, file_extension: str):\n file_path.parent.mkdir(parents=True, exist_ok=True)\n with open(file_path, 'w') as f:\n for entry in data:\n f.write(json.dumps(entry) + '\\n')\n\n @property\n def extensions(self):\n return [\".log\"]\n\n @property\n def priority(self):\n return 15\n\n# Register the handler\nstore.register_handler(CustomLogHandler())\n```\n\n### File Operations\n\n```python\n# Check existence\nif \"config\" in store:\n print(\"Config file exists\")\n\n# List all files\nfor key in store.keys():\n print(f\"File: {key}\")\n\n# Get file metadata\nmetadata = store.get_metadata(\"large_dataset\")\nprint(f\"Size: {metadata.size} bytes\")\nprint(f\"Modified: {metadata.modified_time}\")\n\n# Copy and move files\nstore.copy(\"original\", \"backup\")\nstore.move(\"temp_file\", \"permanent_file\")\n\n# Delete files\ndel store[\"old_data\"]\n```\n\n### Context Management\n\n```python\n# Automatic cleanup of temporary files and cache\nwith AutoStore(\"./data\", config=config) as store:\n store[\"data\"] = large_dataset\n results = store[\"data\"]\n# Temporary files are automatically cleaned up here\n```\n\n## Performance Considerations\n\n### Large File Handling\n\nAutoStore automatically optimizes for large files:\n\n- Multipart uploads/downloads for files > 64MB\n- Configurable chunk sizes and concurrency\n- Streaming operations to minimize memory usage\n\n## When to Use AutoStore\n\nChoose AutoStore when you need:\n\n- Multi-cloud data access with seamless backend switching\n- Dataset processing with automatic detection of file collections\n- Zero-configuration setup for rapid prototyping and development\n- Cross-backend operations without managing multiple client libraries\n- Data science projects with mixed file types across storage systems\n- Type-safe configuration with IDE support and validation\n- Intelligent caching to optimize cloud storage costs and performance\n\nDon't choose AutoStore when:\n\n- You need complex queries or relational operations (use databases)\n- You only work with one data type and one storage backend consistently\n- You need zero dependencies (use built-in libraries like shelve)\n- You require advanced database features like transactions or indexing\n- You need fine-grained control over every storage operation\n\n## Changes\n\n- 0.1.14 - AutoPath now has a load and save method that uses the built-in handlers\n- 0.1.13 - Added AutoPath class for path-like operations with automatic backend detection\n - AutoPath supports all storage operations like read, write, upload, download, delete, etc.\n - AutoPath can be used in place of AutoStore for path-like interactions\n- 0.1.8 - Auto scheme registration enhancement\n- 0.1.7 - Cache expiry can be set to 0 to never expire cache entries.\n- 0.1.6 - Scheme-based backend detection and Options system with automatic backend detection from URI schemes\n - Unified Options dataclass system replacing separate config classes\n - Cross-backend access from single store instance\n - Dataset support with automatic multi-file detection\n - Enhanced error handling with dependency management\n - Breaking: Removed manual backend registration\n - Breaking: Replaced `S3StorageConfig` with `S3Options`\n- 0.1.5 - Added StorePath to use the Autostore instance in path-like operations\n- 0.1.4 - parquet and csv are loaded as LazyFrames by default and sparse matrices are now saved as .sparse.npz\n- 0.1.3\n - Refactored to use different storage backends including local file system and S3.\n - Implement S3 storage backend with basic operations\n - Added S3StorageConfig for configuration management.\n - Implemented S3Backend class for handling S3 interactions.\n - Included methods for file operations: upload, download, delete, copy, move, and list files.\n - Added support for directory-like structures in S3.\n - Implemented metadata retrieval for files.\n - Integrated error handling for common S3 exceptions.\n - Added support for multipart uploads and downloads.\n - Included utility functions for path parsing and glob pattern matching.\n - Calling store.keys() now only returns keys without extensions.\n- 0.1.2 - config, setup_logging, and load_dotenv are now imported at the module top level\n- 0.1.1 - Added config, setup_logging, and load_dotenv\n- 0.1.0 - Initial release\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "File Storage Made Simple",
"version": "0.1.18",
"project_urls": null,
"split_keywords": [
"file storage",
" file management"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "3679a98321984815812f60d670d87edf55a5be672a672ac2f283d1946019ca6c",
"md5": "7e41dc91076c883bf498c2f1251626cc",
"sha256": "da7bea41ebf9ed982af1906f9379d05e210ddb4e5e64899fddb8c27b239e100e"
},
"downloads": -1,
"filename": "autostore-0.1.18-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7e41dc91076c883bf498c2f1251626cc",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 44159,
"upload_time": "2025-08-20T22:21:55",
"upload_time_iso_8601": "2025-08-20T22:21:55.809575Z",
"url": "https://files.pythonhosted.org/packages/36/79/a98321984815812f60d670d87edf55a5be672a672ac2f283d1946019ca6c/autostore-0.1.18-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "10e35615191656a05b822b7dfb2021af6753c8849949d2140f36921deeb3f84e",
"md5": "8f99e6770fd95e8d39963a8227d5785c",
"sha256": "aa108ab6552dbd9684c98592f397d6c6b4485b242be6aa11d66dbaa9fef43cae"
},
"downloads": -1,
"filename": "autostore-0.1.18.tar.gz",
"has_sig": false,
"md5_digest": "8f99e6770fd95e8d39963a8227d5785c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 68361,
"upload_time": "2025-08-20T22:21:57",
"upload_time_iso_8601": "2025-08-20T22:21:57.170684Z",
"url": "https://files.pythonhosted.org/packages/10/e3/5615191656a05b822b7dfb2021af6753c8849949d2140f36921deeb3f84e/autostore-0.1.18.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-20 22:21:57",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "autostore"
}
JSON