# Cloud Shelve
`Cloud Shelve (cshelve)` is a Python package that provides a seamless way to store and manage data in the cloud using the familiar [Python Shelve interface](https://docs.python.org/3/library/shelve.html). It is designed for efficient and scalable storage solutions, allowing you to leverage cloud providers for persistent storage while keeping the simplicity of the `shelve` API.
## Features
- Supports large file storage in the cloud
- Secure data in-transit encryption when using cloud storage
- Fully compatible with Python's `shelve` API
- Cross-platform compatibility for local and remote storage
## Installation
Install `cshelve` via pip:
```bash
pip install cshelve
```
## Usage
The `cshelve` module strictly follows the official `shelve` API. Consequently, you can refer to the [Python official documentation](https://docs.python.org/3/library/shelve.html) for general usage examples. Simply replace the `shelve` import with `cshelve`, and you're good to go.
### Local Storage
Here is an example, adapted from the [official shelve documentation](https://docs.python.org/3/library/shelve.html#example), demonstrating local storage usage. Just replace `shelve` with `cshelve`:
```python
import cshelve
d = cshelve.open('local.db') # Open the local database file
key = 'key'
data = 'data'
d[key] = data # Store data at the key (overwrites existing data)
data = d[key] # Retrieve a copy of data (raises KeyError if not found)
del d[key] # Delete data at the key (raises KeyError if not found)
flag = key in d # Check if the key exists in the database
klist = list(d.keys()) # List all existing keys (could be slow for large datasets)
# Note: Since writeback=True is not used, handle data carefully:
d['xx'] = [0, 1, 2] # Store a list
d['xx'].append(3) # This won't persist since writeback=True is not used
# Correct approach:
temp = d['xx'] # Extract the stored list
temp.append(5) # Modify the list
d['xx'] = temp # Store it back to persist changes
d.close() # Close the database
```
### Remote Storage (e.g., Azure)
To configure remote cloud storage, you need to provide an INI file containing your cloud provider's configuration. The file should have a `.ini` extension.
#### Example Azure Configuration
```bash
$ cat azure.ini
[default]
provider = azure
account_url = https://myaccount.blob.core.windows.net
auth_type = passwordless
container_name = mycontainer
```
Once the INI file is ready, you can interact with remote storage the same way as with local storage. Here's an example using Azure:
```python
import cshelve
d = cshelve.open('azure.ini') # Open using the remote storage configuration
key = 'key'
data = 'data'
d[key] = data # Store data at the key
data = d[key] # Retrieve the data
del d[key] # Delete the data
flag = key in d # Check if the key exists in the cloud store
klist = list(d.keys()) # List all keys in the remote storage
d['xx'] = [0, 1, 2] # Store a list remotely
d['xx'].append(3) # Changes to the list won't persist
# Correct approach:
temp = d['xx'] # Extract the stored list
temp.append(5) # Modify the list
d['xx'] = temp # Store it back to persist changes
d.close() # Close the connection to the remote store
```
More configuration examples for other cloud providers can be found [here](./tests/configurations/).
### Cloud Providers configuration
#### Azure
The Azure provider uses [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) as remote storage.
The module considers the provided container as dedicated to the application. The impact might be significant. For example, if the flag `n` is provided to the `open` function, the entire container will be purged, aligning with the [official interface](https://docs.python.org/3/library/shelve.html#shelve.open).
| Option | Description | Required | Default Value |
|----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------|---------------|
| `account_url` | The URL of your Azure storage account. | :x: | |
| `auth_type` | The authentication method to use. Currently, only `passwordless` is supported. | :x: | |
| `container_name` | The name of the container in your Azure storage account. | :x: | |
Depending on the `open` flag, the permissions required by `cshelve` for blob storage vary.
| Flag | Description | Permissions Needed |
|------|-------------|--------------------|
| `r` | Open an existing blob storage container for reading only. | [Storage Blob Data Reader](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-reader) |
| `w` | Open an existing blob storage container for reading and writing. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |
| `c` | Open a blob storage container for reading and writing, creating it if it doesn't exist. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |
| `n` | Purge the blob storage container before using it. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |
Authentication type supported:
| Auth Type | Description | Advantage | Disadvantage | Example Configuration |
|-------------------|-------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------|---------------------------------------|-----------------------|
| Connection String | Uses a connection string for authentication. Credentials are provided directly in the string. | Fast startup as no additional credential retrieval is needed. | Credentials need to be securely managed and provided. | [Example](tests/configurations/azure-integration/connection-string.ini) |
| Passwordless | Uses passwordless authentication methods such as Managed Identity. | Recommended for better security and easier credential management. | May impact startup time due to the need to retrieve authentication credentials. | [Example](./tests/configurations/azure-integration/standard.ini) |
## Roadmap
- **AWS S3 Support**: Integration for AWS S3 storage is planned in upcoming versions.
- **Google Cloud Storage Support**: Support for Google Cloud Storage is also on the roadmap.
Stay tuned for updates!
## Contributing
We welcome contributions from the community! If you'd like to contribute, please read our [contributing guidelines](CONTRIBUTING.md) for more details on how to get started.
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.
## Contact
If you have any questions, issues, or feedback, feel free to [open an issue]https://github.com/Standard-Cloud/cshelve/issues).
Raw data
{
"_id": null,
"home_page": null,
"name": "cshelve",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "shelve, database, azure-storage-account, azure",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/8d/92/6dba9f97705912b8e47d6f2702e1b5cad48568811dc9a53eb4fd02f0501b/cshelve-0.1.0.dev5.tar.gz",
"platform": null,
"description": "# Cloud Shelve\n`Cloud Shelve (cshelve)` is a Python package that provides a seamless way to store and manage data in the cloud using the familiar [Python Shelve interface](https://docs.python.org/3/library/shelve.html). It is designed for efficient and scalable storage solutions, allowing you to leverage cloud providers for persistent storage while keeping the simplicity of the `shelve` API.\n\n## Features\n\n- Supports large file storage in the cloud\n- Secure data in-transit encryption when using cloud storage\n- Fully compatible with Python's `shelve` API\n- Cross-platform compatibility for local and remote storage\n\n## Installation\n\nInstall `cshelve` via pip:\n\n```bash\npip install cshelve\n```\n\n## Usage\n\nThe `cshelve` module strictly follows the official `shelve` API. Consequently, you can refer to the [Python official documentation](https://docs.python.org/3/library/shelve.html) for general usage examples. Simply replace the `shelve` import with `cshelve`, and you're good to go.\n\n### Local Storage\n\nHere is an example, adapted from the [official shelve documentation](https://docs.python.org/3/library/shelve.html#example), demonstrating local storage usage. Just replace `shelve` with `cshelve`:\n\n```python\nimport cshelve\n\nd = cshelve.open('local.db') # Open the local database file\n\nkey = 'key'\ndata = 'data'\n\nd[key] = data # Store data at the key (overwrites existing data)\ndata = d[key] # Retrieve a copy of data (raises KeyError if not found)\ndel d[key] # Delete data at the key (raises KeyError if not found)\n\nflag = key in d # Check if the key exists in the database\nklist = list(d.keys()) # List all existing keys (could be slow for large datasets)\n\n# Note: Since writeback=True is not used, handle data carefully:\nd['xx'] = [0, 1, 2] # Store a list\nd['xx'].append(3) # This won't persist since writeback=True is not used\n\n# Correct approach:\ntemp = d['xx'] # Extract the stored list\ntemp.append(5) # Modify the list\nd['xx'] = temp # Store it back to persist changes\n\nd.close() # Close the database\n```\n\n### Remote Storage (e.g., Azure)\n\nTo configure remote cloud storage, you need to provide an INI file containing your cloud provider's configuration. The file should have a `.ini` extension.\n\n#### Example Azure Configuration\n\n```bash\n$ cat azure.ini\n[default]\nprovider = azure\naccount_url = https://myaccount.blob.core.windows.net\nauth_type = passwordless\ncontainer_name = mycontainer\n```\n\nOnce the INI file is ready, you can interact with remote storage the same way as with local storage. Here's an example using Azure:\n\n```python\nimport cshelve\n\nd = cshelve.open('azure.ini') # Open using the remote storage configuration\n\nkey = 'key'\ndata = 'data'\n\nd[key] = data # Store data at the key\ndata = d[key] # Retrieve the data\ndel d[key] # Delete the data\n\nflag = key in d # Check if the key exists in the cloud store\nklist = list(d.keys()) # List all keys in the remote storage\n\nd['xx'] = [0, 1, 2] # Store a list remotely\nd['xx'].append(3) # Changes to the list won't persist\n\n# Correct approach:\ntemp = d['xx'] # Extract the stored list\ntemp.append(5) # Modify the list\nd['xx'] = temp # Store it back to persist changes\n\nd.close() # Close the connection to the remote store\n```\n\nMore configuration examples for other cloud providers can be found [here](./tests/configurations/).\n\n### Cloud Providers configuration\n\n#### Azure\n\nThe Azure provider uses [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) as remote storage.\nThe module considers the provided container as dedicated to the application. The impact might be significant. For example, if the flag `n` is provided to the `open` function, the entire container will be purged, aligning with the [official interface](https://docs.python.org/3/library/shelve.html#shelve.open).\n\n| Option | Description | Required | Default Value |\n|----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------|---------------|\n| `account_url` | The URL of your Azure storage account. | :x: | |\n| `auth_type` | The authentication method to use. Currently, only `passwordless` is supported. | :x: | |\n| `container_name` | The name of the container in your Azure storage account. | :x: | |\n\nDepending on the `open` flag, the permissions required by `cshelve` for blob storage vary.\n\n| Flag | Description | Permissions Needed |\n|------|-------------|--------------------|\n| `r` | Open an existing blob storage container for reading only. | [Storage Blob Data Reader](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-reader) |\n| `w` | Open an existing blob storage container for reading and writing. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |\n| `c` | Open a blob storage container for reading and writing, creating it if it doesn't exist. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |\n| `n` | Purge the blob storage container before using it. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |\n\n\nAuthentication type supported:\n\n| Auth Type | Description | Advantage | Disadvantage | Example Configuration |\n|-------------------|-------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------|---------------------------------------|-----------------------|\n| Connection String | Uses a connection string for authentication. Credentials are provided directly in the string. | Fast startup as no additional credential retrieval is needed. | Credentials need to be securely managed and provided. | [Example](tests/configurations/azure-integration/connection-string.ini) |\n| Passwordless | Uses passwordless authentication methods such as Managed Identity. | Recommended for better security and easier credential management. | May impact startup time due to the need to retrieve authentication credentials. | [Example](./tests/configurations/azure-integration/standard.ini) |\n\n\n## Roadmap\n\n- **AWS S3 Support**: Integration for AWS S3 storage is planned in upcoming versions.\n- **Google Cloud Storage Support**: Support for Google Cloud Storage is also on the roadmap.\n\nStay tuned for updates!\n\n## Contributing\n\nWe welcome contributions from the community! If you'd like to contribute, please read our [contributing guidelines](CONTRIBUTING.md) for more details on how to get started.\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.\n\n## Contact\n\nIf you have any questions, issues, or feedback, feel free to [open an issue]https://github.com/Standard-Cloud/cshelve/issues).\n",
"bugtrack_url": null,
"license": null,
"summary": "Propulsing the shelve module to the cloud",
"version": "0.1.0.dev5",
"project_urls": {
"documentation": "https://github.com/Standard-Cloud/cshelve",
"homepage": "https://github.com/Standard-Cloud/cshelve",
"repository": "https://github.com/Standard-Cloud/cshelve"
},
"split_keywords": [
"shelve",
" database",
" azure-storage-account",
" azure"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "f53c2e758a083fdddb0133fab35ad7c4e8a71042e7e3d2e1b994e7d25aaabf8e",
"md5": "bbb5148b54634492c021639513795554",
"sha256": "3a42bde4cf3155b40ac447f0540ae46a83414d4b8cf49748b9d213c86f32c835"
},
"downloads": -1,
"filename": "cshelve-0.1.0.dev5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "bbb5148b54634492c021639513795554",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 11268,
"upload_time": "2024-10-20T19:00:48",
"upload_time_iso_8601": "2024-10-20T19:00:48.668286Z",
"url": "https://files.pythonhosted.org/packages/f5/3c/2e758a083fdddb0133fab35ad7c4e8a71042e7e3d2e1b994e7d25aaabf8e/cshelve-0.1.0.dev5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "8d926dba9f97705912b8e47d6f2702e1b5cad48568811dc9a53eb4fd02f0501b",
"md5": "7179d999dd38b0116f0ec7bc53386008",
"sha256": "5998e4945d61166112c68f8b866f60401d7435c2b95a02de471eb7ef54842ecd"
},
"downloads": -1,
"filename": "cshelve-0.1.0.dev5.tar.gz",
"has_sig": false,
"md5_digest": "7179d999dd38b0116f0ec7bc53386008",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 11610,
"upload_time": "2024-10-20T19:00:50",
"upload_time_iso_8601": "2024-10-20T19:00:50.334354Z",
"url": "https://files.pythonhosted.org/packages/8d/92/6dba9f97705912b8e47d6f2702e1b5cad48568811dc9a53eb4fd02f0501b/cshelve-0.1.0.dev5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-20 19:00:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Standard-Cloud",
"github_project": "cshelve",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "cshelve"
}
JSON
