# Perceptra Storage
A unified storage adapter interface for multi-cloud storage backends in Python.
## Features
- **Unified API**: Single interface for all storage backends
- **Multi-cloud support**: S3, Azure Blob, MinIO, and Local filesystem
- **Type-safe**: Full type hints for better IDE support
- **Easy to extend**: Register custom adapters
- **Production-ready**: Comprehensive error handling and logging
- **Zero dependencies**: Core package has no dependencies; install only what you need
## Installation
```bash
# Install core package (local filesystem only)
pip install perceptra-storage
# Install with S3 support
pip install perceptra-storage[s3]
# Install with Azure support
pip install perceptra-storage[azure]
# Install with MinIO support
pip install perceptra-storage[minio]
# Install with all backends
pip install perceptra-storage[all]
```
## Quick Start
```python
from perceptra_storage import get_storage_adapter
# Create an S3 adapter
config = {
'bucket_name': 'my-bucket',
'region': 'us-west-2'
}
credentials = {
'access_key_id': 'YOUR_ACCESS_KEY',
'secret_access_key': 'YOUR_SECRET_KEY'
}
adapter = get_storage_adapter('s3', config, credentials)
# Test connection
adapter.test_connection()
# Upload a file
with open('data.csv', 'rb') as f:
adapter.upload_file(f, 'datasets/data.csv', content_type='text/csv')
# Download a file
data = adapter.download_file('datasets/data.csv')
# List files
files = adapter.list_files(prefix='datasets/')
for file in files:
print(f"{file.key}: {file.size} bytes")
# Generate presigned URL
url = adapter.generate_presigned_url('datasets/data.csv', expiration=3600)
print(f"Temporary URL: {url.url}")
# Delete a file
adapter.delete_file('datasets/data.csv')
```
## Supported Backends
### Amazon S3
```python
config = {
'bucket_name': 'my-bucket',
'region': 'us-west-2' # optional
}
credentials = {
'access_key_id': 'YOUR_ACCESS_KEY',
'secret_access_key': 'YOUR_SECRET_KEY',
'session_token': 'TOKEN' # optional, for temporary credentials
}
adapter = get_storage_adapter('s3', config, credentials)
```
### Azure Blob Storage
```python
config = {
'container_name': 'my-container',
'account_name': 'mystorageaccount'
}
credentials = {
'account_key': 'YOUR_ACCOUNT_KEY'
# OR 'connection_string': 'DefaultEndpointsProtocol=https;...'
# OR 'sas_token': 'sv=2021-06-08&...'
}
adapter = get_storage_adapter('azure', config, credentials)
```
### MinIO / S3-Compatible
```python
config = {
'bucket_name': 'my-bucket',
'endpoint_url': 'play.min.io:9000',
'secure': True, # Use HTTPS
'region': 'us-east-1' # optional
}
credentials = {
'access_key': 'YOUR_ACCESS_KEY',
'secret_key': 'YOUR_SECRET_KEY'
}
adapter = get_storage_adapter('minio', config, credentials)
```
### Local Filesystem
```python
config = {
'base_path': '/var/perceptra/storage',
'create_dirs': True # Auto-create directories
}
adapter = get_storage_adapter('local', config)
```
## API Reference
### BaseStorageAdapter
All adapters implement the following methods:
#### test_connection(timeout: int = 10) -> bool
Test connection to storage backend.
#### upload_file(file_obj, key, content_type=None, metadata=None) -> str
Upload a file to storage.
#### download_file(key, destination=None) -> bytes
Download a file from storage.
#### delete_file(key) -> bool
Delete a file from storage.
#### file_exists(key) -> bool
Check if a file exists.
#### get_file_metadata(key) -> StorageObject
Get metadata about a stored file.
#### list_files(prefix="", max_results=1000) -> list[StorageObject]
List files with optional prefix filter.
#### generate_presigned_url(key, expiration=3600, method="GET") -> PresignedUrl
Generate a presigned URL for temporary access.
#### get_public_url(key) -> Optional[str]
Get public URL for a file (if supported).
## Error Handling
The package defines a hierarchy of exceptions:
```python
from perceptra_storage import (
StorageError, # Base exception
StorageConnectionError, # Connection failures
StorageOperationError, # Operation failures
StorageNotFoundError, # File not found
StoragePermissionError, # Permission denied
)
try:
adapter.download_file('missing.txt')
except StorageNotFoundError:
print("File not found")
except StoragePermissionError:
print("Access denied")
except StorageError as e:
print(f"Storage error: {e}")
```
## Advanced Usage
### Custom Adapters
You can register custom storage adapters:
```python
from perceptra_storage import BaseStorageAdapter, register_adapter
class CustomStorageAdapter(BaseStorageAdapter):
def _validate_config(self):
# Validation logic
pass
def test_connection(self, timeout=10):
# Implementation
pass
# Implement other required methods...
# Register the adapter
register_adapter('custom', CustomStorageAdapter)
# Use it
adapter = get_storage_adapter('custom', config, credentials)
```
### Context Manager Support
For automatic resource cleanup:
```python
class ManagedAdapter:
def __init__(self, adapter):
self.adapter = adapter
def __enter__(self):
self.adapter.test_connection()
return self.adapter
def __exit__(self, exc_type, exc_val, exc_tb):
# Cleanup if needed
pass
with ManagedAdapter(adapter) as storage:
storage.upload_file(file_obj, 'data.csv')
```
### Batch Operations
```python
# Upload multiple files
files_to_upload = [
('local/file1.txt', 'remote/file1.txt'),
('local/file2.txt', 'remote/file2.txt'),
]
for local_path, remote_key in files_to_upload:
with open(local_path, 'rb') as f:
adapter.upload_file(f, remote_key)
print(f"Uploaded {local_path} -> {remote_key}")
# Download multiple files
files_to_download = ['data1.csv', 'data2.csv', 'data3.csv']
for key in files_to_download:
if adapter.file_exists(key):
data = adapter.download_file(key)
with open(f"local_{key}", 'wb') as f:
f.write(data)
```
## Integration with Django
Use with your Django storage profiles:
```python
from perceptra_storage import get_storage_adapter
from myapp.models import StorageProfile
def get_adapter_for_profile(profile: StorageProfile):
"""Create adapter from Django StorageProfile model."""
# Retrieve credentials from SecretRef
credentials = None
if profile.credential_ref:
credentials = retrieve_credentials(profile.credential_ref)
return get_storage_adapter(
backend=profile.backend,
config=profile.config,
credentials=credentials
)
# Usage
profile = StorageProfile.objects.get(tenant=tenant, is_default=True)
adapter = get_adapter_for_profile(profile)
adapter.test_connection()
```
## Testing
The package includes comprehensive tests:
```bash
# Install dev dependencies
pip install perceptra-storage[dev]
# Run tests
pytest
# Run with coverage
pytest --cov=perceptra_storage --cov-report=html
# Run specific backend tests
pytest tests/test_s3.py
pytest tests/test_azure.py
pytest tests/test_minio.py
pytest tests/test_local.py
```
## Security Best Practices
1. **Never hardcode credentials**: Use environment variables or secret management systems
2. **Use IAM roles**: When running on AWS, use IAM roles instead of access keys
3. **Managed identities**: Use Azure Managed Identities when possible
4. **Encrypt at rest**: Enable encryption on your storage backends
5. **Use HTTPS**: Always use secure connections (set `secure=True` for MinIO)
6. **Rotate credentials**: Regularly rotate access keys and tokens
7. **Principle of least privilege**: Grant only necessary permissions
## Performance Tips
1. **Batch operations**: Use list operations instead of checking files individually
2. **Streaming**: For large files, use streaming where possible
3. **Parallel uploads**: Use thread pools for multiple file uploads
4. **Connection pooling**: Reuse adapter instances instead of creating new ones
5. **Presigned URLs**: Use presigned URLs for client-side uploads/downloads
## Logging
The package uses Python's standard logging:
```python
import logging
# Enable debug logging
logging.basicConfig(level=logging.DEBUG)
# Or configure specific logger
logger = logging.getLogger('perceptra_storage')
logger.setLevel(logging.INFO)
```
## Contributing
Contributions are welcome! Please:
1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request
## License
MIT License - see LICENSE file for details
## Support
- Documentation: https://docs.perceptra.ai/storage
- Issues: https://github.com/tannousgeagea/perceptra-storage/issues
- Email: support@perceptra.ai
## Changelog
### 0.1.0 (2025-10-18)
- Initial release
- Support for S3, Azure, MinIO, and Local storage
- Full test coverage
- Type hints and documentation
Raw data
{
"_id": null,
"home_page": "https://github.com/perceptra/perceptra-storage",
"name": "perceptra-storage",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "Tannous Geagea <tannousgeagea@hotmail.com>",
"keywords": "storage, cloud, s3, azure, minio, blob, filesystem, adapter, multi-cloud, computer-vision",
"author": "Perceptra Team",
"author_email": "Tannous Geagea <tannousgeagea@hotmail.com>",
"download_url": "https://files.pythonhosted.org/packages/c9/23/8e0e61fae938418bc92b6782a3a871776d1eda5e118fc9efeabc6bd45374/perceptra_storage-0.1.2.tar.gz",
"platform": null,
"description": "# Perceptra Storage\n\nA unified storage adapter interface for multi-cloud storage backends in Python.\n\n## Features\n\n- **Unified API**: Single interface for all storage backends\n- **Multi-cloud support**: S3, Azure Blob, MinIO, and Local filesystem\n- **Type-safe**: Full type hints for better IDE support\n- **Easy to extend**: Register custom adapters\n- **Production-ready**: Comprehensive error handling and logging\n- **Zero dependencies**: Core package has no dependencies; install only what you need\n\n## Installation\n\n```bash\n# Install core package (local filesystem only)\npip install perceptra-storage\n\n# Install with S3 support\npip install perceptra-storage[s3]\n\n# Install with Azure support\npip install perceptra-storage[azure]\n\n# Install with MinIO support\npip install perceptra-storage[minio]\n\n# Install with all backends\npip install perceptra-storage[all]\n```\n\n## Quick Start\n\n```python\nfrom perceptra_storage import get_storage_adapter\n\n# Create an S3 adapter\nconfig = {\n 'bucket_name': 'my-bucket',\n 'region': 'us-west-2'\n}\ncredentials = {\n 'access_key_id': 'YOUR_ACCESS_KEY',\n 'secret_access_key': 'YOUR_SECRET_KEY'\n}\n\nadapter = get_storage_adapter('s3', config, credentials)\n\n# Test connection\nadapter.test_connection()\n\n# Upload a file\nwith open('data.csv', 'rb') as f:\n adapter.upload_file(f, 'datasets/data.csv', content_type='text/csv')\n\n# Download a file\ndata = adapter.download_file('datasets/data.csv')\n\n# List files\nfiles = adapter.list_files(prefix='datasets/')\nfor file in files:\n print(f\"{file.key}: {file.size} bytes\")\n\n# Generate presigned URL\nurl = adapter.generate_presigned_url('datasets/data.csv', expiration=3600)\nprint(f\"Temporary URL: {url.url}\")\n\n# Delete a file\nadapter.delete_file('datasets/data.csv')\n```\n\n## Supported Backends\n\n### Amazon S3\n\n```python\nconfig = {\n 'bucket_name': 'my-bucket',\n 'region': 'us-west-2' # optional\n}\ncredentials = {\n 'access_key_id': 'YOUR_ACCESS_KEY',\n 'secret_access_key': 'YOUR_SECRET_KEY',\n 'session_token': 'TOKEN' # optional, for temporary credentials\n}\nadapter = get_storage_adapter('s3', config, credentials)\n```\n\n### Azure Blob Storage\n\n```python\nconfig = {\n 'container_name': 'my-container',\n 'account_name': 'mystorageaccount'\n}\ncredentials = {\n 'account_key': 'YOUR_ACCOUNT_KEY'\n # OR 'connection_string': 'DefaultEndpointsProtocol=https;...'\n # OR 'sas_token': 'sv=2021-06-08&...'\n}\nadapter = get_storage_adapter('azure', config, credentials)\n```\n\n### MinIO / S3-Compatible\n\n```python\nconfig = {\n 'bucket_name': 'my-bucket',\n 'endpoint_url': 'play.min.io:9000',\n 'secure': True, # Use HTTPS\n 'region': 'us-east-1' # optional\n}\ncredentials = {\n 'access_key': 'YOUR_ACCESS_KEY',\n 'secret_key': 'YOUR_SECRET_KEY'\n}\nadapter = get_storage_adapter('minio', config, credentials)\n```\n\n### Local Filesystem\n\n```python\nconfig = {\n 'base_path': '/var/perceptra/storage',\n 'create_dirs': True # Auto-create directories\n}\nadapter = get_storage_adapter('local', config)\n```\n\n## API Reference\n\n### BaseStorageAdapter\n\nAll adapters implement the following methods:\n\n#### test_connection(timeout: int = 10) -> bool\nTest connection to storage backend.\n\n#### upload_file(file_obj, key, content_type=None, metadata=None) -> str\nUpload a file to storage.\n\n#### download_file(key, destination=None) -> bytes\nDownload a file from storage.\n\n#### delete_file(key) -> bool\nDelete a file from storage.\n\n#### file_exists(key) -> bool\nCheck if a file exists.\n\n#### get_file_metadata(key) -> StorageObject\nGet metadata about a stored file.\n\n#### list_files(prefix=\"\", max_results=1000) -> list[StorageObject]\nList files with optional prefix filter.\n\n#### generate_presigned_url(key, expiration=3600, method=\"GET\") -> PresignedUrl\nGenerate a presigned URL for temporary access.\n\n#### get_public_url(key) -> Optional[str]\nGet public URL for a file (if supported).\n\n## Error Handling\n\nThe package defines a hierarchy of exceptions:\n\n```python\nfrom perceptra_storage import (\n StorageError, # Base exception\n StorageConnectionError, # Connection failures\n StorageOperationError, # Operation failures\n StorageNotFoundError, # File not found\n StoragePermissionError, # Permission denied\n)\n\ntry:\n adapter.download_file('missing.txt')\nexcept StorageNotFoundError:\n print(\"File not found\")\nexcept StoragePermissionError:\n print(\"Access denied\")\nexcept StorageError as e:\n print(f\"Storage error: {e}\")\n```\n\n## Advanced Usage\n\n### Custom Adapters\n\nYou can register custom storage adapters:\n\n```python\nfrom perceptra_storage import BaseStorageAdapter, register_adapter\n\nclass CustomStorageAdapter(BaseStorageAdapter):\n def _validate_config(self):\n # Validation logic\n pass\n \n def test_connection(self, timeout=10):\n # Implementation\n pass\n \n # Implement other required methods...\n\n# Register the adapter\nregister_adapter('custom', CustomStorageAdapter)\n\n# Use it\nadapter = get_storage_adapter('custom', config, credentials)\n```\n\n### Context Manager Support\n\nFor automatic resource cleanup:\n\n```python\nclass ManagedAdapter:\n def __init__(self, adapter):\n self.adapter = adapter\n \n def __enter__(self):\n self.adapter.test_connection()\n return self.adapter\n \n def __exit__(self, exc_type, exc_val, exc_tb):\n # Cleanup if needed\n pass\n\nwith ManagedAdapter(adapter) as storage:\n storage.upload_file(file_obj, 'data.csv')\n```\n\n### Batch Operations\n\n```python\n# Upload multiple files\nfiles_to_upload = [\n ('local/file1.txt', 'remote/file1.txt'),\n ('local/file2.txt', 'remote/file2.txt'),\n]\n\nfor local_path, remote_key in files_to_upload:\n with open(local_path, 'rb') as f:\n adapter.upload_file(f, remote_key)\n print(f\"Uploaded {local_path} -> {remote_key}\")\n\n# Download multiple files\nfiles_to_download = ['data1.csv', 'data2.csv', 'data3.csv']\n\nfor key in files_to_download:\n if adapter.file_exists(key):\n data = adapter.download_file(key)\n with open(f\"local_{key}\", 'wb') as f:\n f.write(data)\n```\n\n## Integration with Django\n\nUse with your Django storage profiles:\n\n```python\nfrom perceptra_storage import get_storage_adapter\nfrom myapp.models import StorageProfile\n\ndef get_adapter_for_profile(profile: StorageProfile):\n \"\"\"Create adapter from Django StorageProfile model.\"\"\"\n # Retrieve credentials from SecretRef\n credentials = None\n if profile.credential_ref:\n credentials = retrieve_credentials(profile.credential_ref)\n \n return get_storage_adapter(\n backend=profile.backend,\n config=profile.config,\n credentials=credentials\n )\n\n# Usage\nprofile = StorageProfile.objects.get(tenant=tenant, is_default=True)\nadapter = get_adapter_for_profile(profile)\nadapter.test_connection()\n```\n\n## Testing\n\nThe package includes comprehensive tests:\n\n```bash\n# Install dev dependencies\npip install perceptra-storage[dev]\n\n# Run tests\npytest\n\n# Run with coverage\npytest --cov=perceptra_storage --cov-report=html\n\n# Run specific backend tests\npytest tests/test_s3.py\npytest tests/test_azure.py\npytest tests/test_minio.py\npytest tests/test_local.py\n```\n\n## Security Best Practices\n\n1. **Never hardcode credentials**: Use environment variables or secret management systems\n2. **Use IAM roles**: When running on AWS, use IAM roles instead of access keys\n3. **Managed identities**: Use Azure Managed Identities when possible\n4. **Encrypt at rest**: Enable encryption on your storage backends\n5. **Use HTTPS**: Always use secure connections (set `secure=True` for MinIO)\n6. **Rotate credentials**: Regularly rotate access keys and tokens\n7. **Principle of least privilege**: Grant only necessary permissions\n\n## Performance Tips\n\n1. **Batch operations**: Use list operations instead of checking files individually\n2. **Streaming**: For large files, use streaming where possible\n3. **Parallel uploads**: Use thread pools for multiple file uploads\n4. **Connection pooling**: Reuse adapter instances instead of creating new ones\n5. **Presigned URLs**: Use presigned URLs for client-side uploads/downloads\n\n## Logging\n\nThe package uses Python's standard logging:\n\n```python\nimport logging\n\n# Enable debug logging\nlogging.basicConfig(level=logging.DEBUG)\n\n# Or configure specific logger\nlogger = logging.getLogger('perceptra_storage')\nlogger.setLevel(logging.INFO)\n```\n\n## Contributing\n\nContributions are welcome! Please:\n\n1. Fork the repository\n2. Create a feature branch\n3. Add tests for new functionality\n4. Ensure all tests pass\n5. Submit a pull request\n\n## License\n\nMIT License - see LICENSE file for details\n\n## Support\n\n- Documentation: https://docs.perceptra.ai/storage\n- Issues: https://github.com/tannousgeagea/perceptra-storage/issues\n- Email: support@perceptra.ai\n\n## Changelog\n\n### 0.1.0 (2025-10-18)\n- Initial release\n- Support for S3, Azure, MinIO, and Local storage\n- Full test coverage\n- Type hints and documentation\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Unified storage adapter interface for multi-cloud storage backends",
"version": "0.1.2",
"project_urls": {
"Changelog": "https://github.com/tannousgeagea/perceptra-storage/blob/main/CHANGELOG.md",
"Documentation": "https://docs.perceptra.ai/storage",
"Homepage": "https://github.com/tannousgeagea/perceptra-storage",
"Issue Tracker": "https://github.com/tannousgeagea/perceptra-storage/issues",
"Repository": "https://github.com/tannousgeagea/perceptra-storage"
},
"split_keywords": [
"storage",
" cloud",
" s3",
" azure",
" minio",
" blob",
" filesystem",
" adapter",
" multi-cloud",
" computer-vision"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "ce07ee2adea97fec622abf36698a8f96f8dac33768d63226f3b692968cc9c3bd",
"md5": "05aaee547e96181746e2f83b041b6022",
"sha256": "e426cf7f349542e9b9e7b053f2db40eb3bd8d76fafc550a606babdc2af51d6b3"
},
"downloads": -1,
"filename": "perceptra_storage-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "05aaee547e96181746e2f83b041b6022",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 25704,
"upload_time": "2025-10-24T19:58:59",
"upload_time_iso_8601": "2025-10-24T19:58:59.171014Z",
"url": "https://files.pythonhosted.org/packages/ce/07/ee2adea97fec622abf36698a8f96f8dac33768d63226f3b692968cc9c3bd/perceptra_storage-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "c9238e0e61fae938418bc92b6782a3a871776d1eda5e118fc9efeabc6bd45374",
"md5": "a5b1f231fe9a45c44ef1540e6b5a0695",
"sha256": "c2359ec0039788374e100c39bedfc6680c3cfb381a917b3bb3b511908705b226"
},
"downloads": -1,
"filename": "perceptra_storage-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "a5b1f231fe9a45c44ef1540e6b5a0695",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 55227,
"upload_time": "2025-10-24T19:59:00",
"upload_time_iso_8601": "2025-10-24T19:59:00.416258Z",
"url": "https://files.pythonhosted.org/packages/c9/23/8e0e61fae938418bc92b6782a3a871776d1eda5e118fc9efeabc6bd45374/perceptra_storage-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-24 19:59:00",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "perceptra",
"github_project": "perceptra-storage",
"github_not_found": true,
"lcname": "perceptra-storage"
}
JSON
