# Cloud Shelve
`Cloud Shelve (cshelve)` is a Python package that provides a seamless way to store and manage data in the cloud using the familiar [Python Shelve interface](https://docs.python.org/3/library/shelve.html). It is designed for efficient and scalable storage solutions, allowing you to leverage cloud providers for persistent storage while keeping the simplicity of the `shelve` API.
## Features
- Supports large file storage in the cloud
- Secure data in-transit encryption when using cloud storage
- Fully compatible with Python's `shelve` API
- Cross-platform compatibility for local and remote storage
## Installation
Install `cshelve` via pip:
```bash
pip install cshelve
```
## Usage
The `cshelve` module strictly follows the official `shelve` API. Consequently, you can refer to the [Python official documentation](https://docs.python.org/3/library/shelve.html) for general usage examples. Simply replace the `shelve` import with `cshelve`, and you're good to go.
### Local Storage
Here is an example, adapted from the [official shelve documentation](https://docs.python.org/3/library/shelve.html#example), demonstrating local storage usage. Just replace `shelve` with `cshelve`:
```python
import cshelve
d = cshelve.open('local.db') # Open the local database file
key = 'key'
data = 'data'
d[key] = data # Store data at the key (overwrites existing data)
data = d[key] # Retrieve a copy of data (raises KeyError if not found)
del d[key] # Delete data at the key (raises KeyError if not found)
flag = key in d # Check if the key exists in the database
klist = list(d.keys()) # List all existing keys (could be slow for large datasets)
# Note: Since writeback=True is not used, handle data carefully:
d['xx'] = [0, 1, 2] # Store a list
d['xx'].append(3) # This won't persist since writeback=True is not used
# Correct approach:
temp = d['xx'] # Extract the stored list
temp.append(5) # Modify the list
d['xx'] = temp # Store it back to persist changes
d.close() # Close the database
```
### Debug/test Storage
For testing purposes, it is possible to use an in-memory provider that can:
- Persist the data during all the program execution.
- Remove the data object is deleted.
Here is a configuration example:
```bash
$ cat in-memory.ini
[default]
provider = in-memory
# If set, open twice the same database during the program execution will lead to open twice the same database.
persist-key = standard
```
A common use case for this provider is to simplify mocking.
Example:
```bash
$ cat persist.ini
[default]
provider = in-memory
# If set, open twice the same database during the program execution will lead to open twice the same database.
persist-key = my-db
$ cat do-not-persist.ini
[default]
provider = in-memory
```
```python
import cshelve
with cshelve.open('persist.ini') as db:
db["Asterix"] = "Gaulois"
with cshelve.open('persist.ini') as db:
assert db["Asterix"] == "Gaulois"
with cshelve.open('do-not-persist.ini') as db:
db["Obelix"] = "Gaulois"
with cshelve.open('do-not-persist.ini') as db:
assert "Obelix" not in db
```
### Remote Storage (e.g., Azure)
To configure remote cloud storage, you need to provide an INI file containing your cloud provider's configuration. The file should have a `.ini` extension. Remote storage also requires the installation of optional dependencies for the cloud provider you want to use.
#### Example Azure Blob Configuration
First, install the Azure Blob Storage provider:
```bash
pip install cshelve[azure-blob]
```
Then, create an INI file with the following configuration:
```bash
$ cat azure-blob.ini
[default]
provider = azure-blob
account_url = https://myaccount.blob.core.windows.net
# Note: The auth_type can be access_key, passwordless, connection_string, or anonymous.
# The passwordless authentication method is recommended, but the Azure CLI must be installed (https://learn.microsoft.com/en-us/cli/azure/install-azure-cli).
auth_type = passwordless
container_name = mycontainer
```
Once the INI file is ready, you can interact with remote storage the same way as with local storage. Here's an example using Azure:
```python
import cshelve
d = cshelve.open('azure-blob.ini') # Open using the remote storage configuration
key = 'key'
data = 'data'
d[key] = data # Store data at the key on the remote storage
data = d[key] # Retrieve the data from the remote storage
del d[key] # Delete the data from the remote storage
flag = key in d # Check if the key exists in the cloud storage
klist = list(d.keys()) # List all keys present in the remote storage
# Note: Since writeback=True is not used, handle data carefully:
d['xx'] = [0, 1, 2] # Store a list on the remote storage
d['xx'].append(3) # This won't persist since writeback=True is not used
# Correct approach:
temp = d['xx'] # Extract the stored list from the remote storage
temp.append(5) # Modify the list locally
d['xx'] = temp # Store it back on the remote storage to persist changes
d.close() # Close the connection to the remote storage
```
More configuration examples for other cloud providers can be found [here](./tests/configurations/).
### Providers configuration
#### In Memory
Provider: `in-memory`
Installation: No additional installation required.
The In Memory provider uses an in-memory data structure to simulate storage. This is useful for testing and development purposes.
| Option | Description | Required | Default Value |
|----------------|------------------------------------------------------------------------------|----------|---------------|
| `persist-key` | If set, its value will be conserved and reused during the program execution. | :x: | None |
| `exists` | If True, the database exists; otherwise, it will be created. | :x: | False |
#### Azure Blob
Provider: `azure-blob`
Installation: `pip install cshelve[azure-blob]`
The Azure provider uses [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) as remote storage.
The module considers the provided container as dedicated to the application. The impact might be significant. For example, if the flag `n` is provided to the `open` function, the entire container will be purged, aligning with the [official interface](https://docs.python.org/3/library/shelve.html#shelve.open).
| Option | Description | Required | Default Value |
|----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------|---------------|
| `account_url` | The URL of your Azure storage account. | :x: | |
| `auth_type` | The authentication method to use: `access_key`, `passwordless`, `connection_string` or `anonymous`. | :white_check_mark: | |
| `container_name` | The name of the container in your Azure storage account. | :white_check_mark: | |
Depending on the `open` flag, the permissions required by `cshelve` for blob storage vary.
| Flag | Description | Permissions Needed |
|------|-------------|--------------------|
| `r` | Open an existing blob storage container for reading only. | [Storage Blob Data Reader](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-reader) |
| `w` | Open an existing blob storage container for reading and writing. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |
| `c` | Open a blob storage container for reading and writing, creating it if it doesn't exist. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |
| `n` | Purge the blob storage container before using it. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |
Authentication type supported:
| Auth Type | Description | Advantage | Disadvantage | Example Configuration |
|-------------------|-------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------|---------------------------------------|-----------------------|
| Access Key | Uses an Access Key or a Shared Access Signature for authentication. | Fast startup as no additional credential retrieval is needed. | Credentials need to be securely managed and provided. | [Example](./tests/configurations/azure-integration/access-key.ini) |
| Anonymous | No authentication for anonymous access on public blob storage. | No configuration or credentials needed. | Read-only access. | [Example](./tests/configurations/azure-integration/anonymous.ini) |
| Connection String | Uses a connection string for authentication. Credentials are provided directly in the string. | Fast startup as no additional credential retrieval is needed. | Credentials need to be securely managed and provided. | [Example](./tests/configurations/azure-integration/connection-string.ini) |
| Passwordless | Uses passwordless authentication methods such as Managed Identity. | Recommended for better security and easier credential management. | May impact startup time due to the need to retrieve authentication credentials. | [Example](./tests/configurations/azure-integration/standard.ini) |
## Contributing
We welcome contributions from the community! Have a look at our [issues](https://github.com/Standard-Cloud/cshelve/issues).
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.
## Contact
If you have any questions, issues, or feedback, feel free to [open an issue]https://github.com/Standard-Cloud/cshelve/issues).
Raw data
{
"_id": null,
"home_page": null,
"name": "cshelve",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "azure, azure-storage-account, cloud, database, shelve",
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/5e/a1/e867f5cfa149b4cd7e37b428a2f678747d64760d886fa6a85bb3861a1e07/cshelve-0.8.0.tar.gz",
"platform": null,
"description": "# Cloud Shelve\n`Cloud Shelve (cshelve)` is a Python package that provides a seamless way to store and manage data in the cloud using the familiar [Python Shelve interface](https://docs.python.org/3/library/shelve.html). It is designed for efficient and scalable storage solutions, allowing you to leverage cloud providers for persistent storage while keeping the simplicity of the `shelve` API.\n\n## Features\n\n- Supports large file storage in the cloud\n- Secure data in-transit encryption when using cloud storage\n- Fully compatible with Python's `shelve` API\n- Cross-platform compatibility for local and remote storage\n\n## Installation\n\nInstall `cshelve` via pip:\n\n```bash\npip install cshelve\n```\n\n## Usage\n\nThe `cshelve` module strictly follows the official `shelve` API. Consequently, you can refer to the [Python official documentation](https://docs.python.org/3/library/shelve.html) for general usage examples. Simply replace the `shelve` import with `cshelve`, and you're good to go.\n\n### Local Storage\n\nHere is an example, adapted from the [official shelve documentation](https://docs.python.org/3/library/shelve.html#example), demonstrating local storage usage. Just replace `shelve` with `cshelve`:\n\n```python\nimport cshelve\n\nd = cshelve.open('local.db') # Open the local database file\n\nkey = 'key'\ndata = 'data'\n\nd[key] = data # Store data at the key (overwrites existing data)\ndata = d[key] # Retrieve a copy of data (raises KeyError if not found)\ndel d[key] # Delete data at the key (raises KeyError if not found)\n\nflag = key in d # Check if the key exists in the database\nklist = list(d.keys()) # List all existing keys (could be slow for large datasets)\n\n# Note: Since writeback=True is not used, handle data carefully:\nd['xx'] = [0, 1, 2] # Store a list\nd['xx'].append(3) # This won't persist since writeback=True is not used\n\n# Correct approach:\ntemp = d['xx'] # Extract the stored list\ntemp.append(5) # Modify the list\nd['xx'] = temp # Store it back to persist changes\n\nd.close() # Close the database\n```\n\n### Debug/test Storage\n\nFor testing purposes, it is possible to use an in-memory provider that can:\n- Persist the data during all the program execution.\n- Remove the data object is deleted.\n\nHere is a configuration example:\n```bash\n$ cat in-memory.ini\n[default]\nprovider = in-memory\n# If set, open twice the same database during the program execution will lead to open twice the same database.\npersist-key = standard\n```\n\nA common use case for this provider is to simplify mocking.\n\nExample:\n```bash\n$ cat persist.ini\n[default]\nprovider = in-memory\n# If set, open twice the same database during the program execution will lead to open twice the same database.\npersist-key = my-db\n\n$ cat do-not-persist.ini\n[default]\nprovider = in-memory\n```\n\n```python\nimport cshelve\n\nwith cshelve.open('persist.ini') as db:\n db[\"Asterix\"] = \"Gaulois\"\n\nwith cshelve.open('persist.ini') as db:\n assert db[\"Asterix\"] == \"Gaulois\"\n\nwith cshelve.open('do-not-persist.ini') as db:\n db[\"Obelix\"] = \"Gaulois\"\n\nwith cshelve.open('do-not-persist.ini') as db:\n assert \"Obelix\" not in db\n```\n\n### Remote Storage (e.g., Azure)\n\nTo configure remote cloud storage, you need to provide an INI file containing your cloud provider's configuration. The file should have a `.ini` extension. Remote storage also requires the installation of optional dependencies for the cloud provider you want to use.\n\n#### Example Azure Blob Configuration\n\nFirst, install the Azure Blob Storage provider:\n```bash\npip install cshelve[azure-blob]\n```\n\nThen, create an INI file with the following configuration:\n```bash\n$ cat azure-blob.ini\n[default]\nprovider = azure-blob\naccount_url = https://myaccount.blob.core.windows.net\n# Note: The auth_type can be access_key, passwordless, connection_string, or anonymous.\n# The passwordless authentication method is recommended, but the Azure CLI must be installed (https://learn.microsoft.com/en-us/cli/azure/install-azure-cli).\nauth_type = passwordless\ncontainer_name = mycontainer\n```\n\nOnce the INI file is ready, you can interact with remote storage the same way as with local storage. Here's an example using Azure:\n\n```python\nimport cshelve\n\nd = cshelve.open('azure-blob.ini') # Open using the remote storage configuration\n\nkey = 'key'\ndata = 'data'\n\nd[key] = data # Store data at the key on the remote storage\ndata = d[key] # Retrieve the data from the remote storage\ndel d[key] # Delete the data from the remote storage\n\nflag = key in d # Check if the key exists in the cloud storage\nklist = list(d.keys()) # List all keys present in the remote storage\n\n# Note: Since writeback=True is not used, handle data carefully:\nd['xx'] = [0, 1, 2] # Store a list on the remote storage\nd['xx'].append(3) # This won't persist since writeback=True is not used\n\n# Correct approach:\ntemp = d['xx'] # Extract the stored list from the remote storage\ntemp.append(5) # Modify the list locally\nd['xx'] = temp # Store it back on the remote storage to persist changes\n\nd.close() # Close the connection to the remote storage\n```\n\nMore configuration examples for other cloud providers can be found [here](./tests/configurations/).\n\n### Providers configuration\n\n#### In Memory\n\nProvider: `in-memory`\nInstallation: No additional installation required.\n\nThe In Memory provider uses an in-memory data structure to simulate storage. This is useful for testing and development purposes.\n\n| Option | Description | Required | Default Value |\n|----------------|------------------------------------------------------------------------------|----------|---------------|\n| `persist-key` | If set, its value will be conserved and reused during the program execution. | :x: | None |\n| `exists` | If True, the database exists; otherwise, it will be created. | :x: | False |\n\n\n#### Azure Blob\n\nProvider: `azure-blob`\nInstallation: `pip install cshelve[azure-blob]`\n\nThe Azure provider uses [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) as remote storage.\nThe module considers the provided container as dedicated to the application. The impact might be significant. For example, if the flag `n` is provided to the `open` function, the entire container will be purged, aligning with the [official interface](https://docs.python.org/3/library/shelve.html#shelve.open).\n\n| Option | Description | Required | Default Value |\n|----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------|---------------|\n| `account_url` | The URL of your Azure storage account. | :x: | |\n| `auth_type` | The authentication method to use: `access_key`, `passwordless`, `connection_string` or `anonymous`. | :white_check_mark: | |\n| `container_name` | The name of the container in your Azure storage account. | :white_check_mark: | |\n\nDepending on the `open` flag, the permissions required by `cshelve` for blob storage vary.\n\n| Flag | Description | Permissions Needed |\n|------|-------------|--------------------|\n| `r` | Open an existing blob storage container for reading only. | [Storage Blob Data Reader](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-reader) |\n| `w` | Open an existing blob storage container for reading and writing. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |\n| `c` | Open a blob storage container for reading and writing, creating it if it doesn't exist. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |\n| `n` | Purge the blob storage container before using it. | [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) |\n\n\nAuthentication type supported:\n\n| Auth Type | Description | Advantage | Disadvantage | Example Configuration |\n|-------------------|-------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------|---------------------------------------|-----------------------|\n| Access Key | Uses an Access Key or a Shared Access Signature for authentication. | Fast startup as no additional credential retrieval is needed. | Credentials need to be securely managed and provided. | [Example](./tests/configurations/azure-integration/access-key.ini) |\n| Anonymous | No authentication for anonymous access on public blob storage. | No configuration or credentials needed. | Read-only access. | [Example](./tests/configurations/azure-integration/anonymous.ini) |\n| Connection String | Uses a connection string for authentication. Credentials are provided directly in the string. | Fast startup as no additional credential retrieval is needed. | Credentials need to be securely managed and provided. | [Example](./tests/configurations/azure-integration/connection-string.ini) |\n| Passwordless | Uses passwordless authentication methods such as Managed Identity. | Recommended for better security and easier credential management. | May impact startup time due to the need to retrieve authentication credentials. | [Example](./tests/configurations/azure-integration/standard.ini) |\n\n\n## Contributing\n\nWe welcome contributions from the community! Have a look at our [issues](https://github.com/Standard-Cloud/cshelve/issues).\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.\n\n## Contact\n\nIf you have any questions, issues, or feedback, feel free to [open an issue]https://github.com/Standard-Cloud/cshelve/issues).\n",
"bugtrack_url": null,
"license": null,
"summary": "Propulsing the shelve module to the cloud",
"version": "0.8.0",
"project_urls": {
"documentation": "https://github.com/Standard-Cloud/cshelve",
"homepage": "https://github.com/Standard-Cloud/cshelve",
"repository": "https://github.com/Standard-Cloud/cshelve"
},
"split_keywords": [
"azure",
" azure-storage-account",
" cloud",
" database",
" shelve"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "18197bd58749a346ed683d13e2189b696830407ff55c98d4322e299792948897",
"md5": "7776cacf50789807a9e9eb7d269ee24c",
"sha256": "475bbb6c520ee76e8abaa54845d442c381813d6816d80ff049eb3bcee7e4280a"
},
"downloads": -1,
"filename": "cshelve-0.8.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "7776cacf50789807a9e9eb7d269ee24c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 18583,
"upload_time": "2024-12-17T03:11:09",
"upload_time_iso_8601": "2024-12-17T03:11:09.591549Z",
"url": "https://files.pythonhosted.org/packages/18/19/7bd58749a346ed683d13e2189b696830407ff55c98d4322e299792948897/cshelve-0.8.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "5ea1e867f5cfa149b4cd7e37b428a2f678747d64760d886fa6a85bb3861a1e07",
"md5": "05853e4c44ab89f67227559f4701f7ec",
"sha256": "3b4171cc95d07c95fc6c06a6a79aa186106e1f89e29217b7fd65df46a77cf064"
},
"downloads": -1,
"filename": "cshelve-0.8.0.tar.gz",
"has_sig": false,
"md5_digest": "05853e4c44ab89f67227559f4701f7ec",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 120478,
"upload_time": "2024-12-17T03:11:10",
"upload_time_iso_8601": "2024-12-17T03:11:10.656585Z",
"url": "https://files.pythonhosted.org/packages/5e/a1/e867f5cfa149b4cd7e37b428a2f678747d64760d886fa6a85bb3861a1e07/cshelve-0.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-17 03:11:10",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Standard-Cloud",
"github_project": "cshelve",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "cshelve"
}
JSON
