| Name | aiida-s3 JSON |
| Version |
0.1.0
JSON |
| download |
| home_page | None |
| Summary | AiiDA plugin that provides a storage backend using AWS S3 for the file repository. |
| upload_time | 2022-12-13 08:58:25 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | None |
| keywords |
aiida
workflows
s3
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# `aiida-s3`
AiiDA plugin that provides various storage backends that allow using cloud data storage services, such as AWS S3 and Azure Blob Storage.
Currently, the following storage backends are available:
* `s3.psql_aws_s3`: Database provided by PostgreSQL and file repository provided by [AWS S3](https://aws.amazon.com/s3/).
* `s3.psql_azure_blob`: Database provided by PostgreSQL and file repository provided by [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/).
## Installation
The recommended method of installation is through the [`pip`](https://pip.pypa.io/en/stable/) package installer for Python:
pip install aiida-s3
## Setup
To use one of the storage backends provided by `aiida-s3` with AiiDA, you need to create a profile for it:
1. List the available storage backends:
```bash
aiida-s3 profile setup --help
```
1. Create a profile using one of the available storage backends by passing it as an argument to `aiida-s3 profile setup`, for example:
```bash
aiida-s3 profile setup psql-aws-s3
```
The command will prompt for the information required to setup the storage backend.
After all information is entered, the storage backend is initialized, such as creating the database schema and creating file containers.
If
1. Create a default user for the profile:
```bash
verdi -p profile-name user configure --set-default
```
1. The profile is now ready to be used with AiiDA.
Optionally, you can set it as the new default profile:
```bash
verdi profile setdefault profile-name
```
1. Optionally, to test that everything is working as intended, launch a test calculation:
```bash
verdi devel launch-add
```
## Testing
The unit tests are implemented and run with [`pytest`](https://docs.pytest.org/).
To run them, install the package with the `tests` extra dependencies:
pip install aiida-s3[tests]
The plugin provides interfaces to various services that require credentials, such as AWS S3 and Azure Blob Storage.
To run the test suite, one has to provide these credentials or the services have to be mocked.
Instructions for each service that is supported are provided below.
### AWS S3
The [AWS S3](https://aws.amazon.com/s3/) service is interfaced with through the [`boto3`](https://pypi.org/project/boto3/) Python SDK.
The [`moto`](https://pypi.org/project/moto/) library allows to mock this interface.
This makes it possible to run the test suite without any credentials.
To run the tests, simply execute `pytest`:
pytest
By default, the interactions with AWS S3 are mocked through `moto` and no actual credentials are required.
To run the tests against an actual AWS S3 container, the credentials need to be specified through environment variables:
export AIIDA_S3_MOCK_AWS_S3=False
export AWS_BUCKET_NAME='some-bucket'
export AWS_ACCESS_KEY_ID='access-key'
export AWS_SECRET_ACCESS_KEY='secret-access-key'
pytest
### Azure Blob Storage
The [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/) is communicated with through the [`azure-blob-storage`](https://pypi.org/project/azure-storage-blob/) Python SDK.
Currently, there is no good way to mock the clients of this library.
Therefore, when the tests are run without credentials, and so the Azure Blob Storage client needs to be mocked, the tests are skipped.
To run the tests against an actual AWS S3 container, the credentials need to be specified through environment variables:
export AIIDA_S3_MOCK_AZURE_BLOB=False
export AZURE_BLOB_CONTAINER_NAME='some-container'
export AZURE_BLOB_CONNECTION_STRING='DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net'
pytest
The specified container does not have to exist yet, it will be created automatically.
The connection string can be obtained through the Azure portal.
Raw data
{
"_id": null,
"home_page": null,
"name": "aiida-s3",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "aiida,workflows,s3",
"author": null,
"author_email": "\"Sebastiaan P. Huber\" <mail@sphuber.net>",
"download_url": "https://files.pythonhosted.org/packages/63/02/94d3da7bde22d0a7e888c9208720255a35102c4ab1fe3e349d7ab5faac44/aiida-s3-0.1.0.tar.gz",
"platform": null,
"description": "# `aiida-s3`\n\nAiiDA plugin that provides various storage backends that allow using cloud data storage services, such as AWS S3 and Azure Blob Storage.\n\nCurrently, the following storage backends are available:\n\n* `s3.psql_aws_s3`: Database provided by PostgreSQL and file repository provided by [AWS S3](https://aws.amazon.com/s3/).\n* `s3.psql_azure_blob`: Database provided by PostgreSQL and file repository provided by [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/).\n\n\n## Installation\n\nThe recommended method of installation is through the [`pip`](https://pip.pypa.io/en/stable/) package installer for Python:\n\n pip install aiida-s3\n\n## Setup\n\nTo use one of the storage backends provided by `aiida-s3` with AiiDA, you need to create a profile for it:\n\n1. List the available storage backends:\n ```bash\n aiida-s3 profile setup --help\n ```\n\n1. Create a profile using one of the available storage backends by passing it as an argument to `aiida-s3 profile setup`, for example:\n ```bash\n aiida-s3 profile setup psql-aws-s3\n ```\n The command will prompt for the information required to setup the storage backend.\n After all information is entered, the storage backend is initialized, such as creating the database schema and creating file containers.\n If\n\n1. Create a default user for the profile:\n ```bash\n verdi -p profile-name user configure --set-default\n ```\n\n1. The profile is now ready to be used with AiiDA.\n Optionally, you can set it as the new default profile:\n ```bash\n verdi profile setdefault profile-name\n ```\n\n1. Optionally, to test that everything is working as intended, launch a test calculation:\n ```bash\n verdi devel launch-add\n ```\n\n## Testing\n\nThe unit tests are implemented and run with [`pytest`](https://docs.pytest.org/).\nTo run them, install the package with the `tests` extra dependencies:\n\n pip install aiida-s3[tests]\n\nThe plugin provides interfaces to various services that require credentials, such as AWS S3 and Azure Blob Storage.\nTo run the test suite, one has to provide these credentials or the services have to be mocked.\nInstructions for each service that is supported are provided below.\n\n### AWS S3\n\nThe [AWS S3](https://aws.amazon.com/s3/) service is interfaced with through the [`boto3`](https://pypi.org/project/boto3/) Python SDK.\nThe [`moto`](https://pypi.org/project/moto/) library allows to mock this interface.\nThis makes it possible to run the test suite without any credentials.\nTo run the tests, simply execute `pytest`:\n\n pytest\n\nBy default, the interactions with AWS S3 are mocked through `moto` and no actual credentials are required.\nTo run the tests against an actual AWS S3 container, the credentials need to be specified through environment variables:\n\n export AIIDA_S3_MOCK_AWS_S3=False\n export AWS_BUCKET_NAME='some-bucket'\n export AWS_ACCESS_KEY_ID='access-key'\n export AWS_SECRET_ACCESS_KEY='secret-access-key'\n pytest\n\n\n### Azure Blob Storage\n\nThe [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/) is communicated with through the [`azure-blob-storage`](https://pypi.org/project/azure-storage-blob/) Python SDK.\nCurrently, there is no good way to mock the clients of this library.\nTherefore, when the tests are run without credentials, and so the Azure Blob Storage client needs to be mocked, the tests are skipped.\nTo run the tests against an actual AWS S3 container, the credentials need to be specified through environment variables:\n\n export AIIDA_S3_MOCK_AZURE_BLOB=False\n export AZURE_BLOB_CONTAINER_NAME='some-container'\n export AZURE_BLOB_CONNECTION_STRING='DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net'\n pytest\n\nThe specified container does not have to exist yet, it will be created automatically.\nThe connection string can be obtained through the Azure portal.\n",
"bugtrack_url": null,
"license": null,
"summary": "AiiDA plugin that provides a storage backend using AWS S3 for the file repository.",
"version": "0.1.0",
"split_keywords": [
"aiida",
"workflows",
"s3"
],
"urls": [
{
"comment_text": null,
"digests": {
"md5": "55540b4ae0750090a7e461adea402117",
"sha256": "df65f53bea5979718dac459b1dafc58ef4bf0d16f33c5a5917a7ee10b456bfe8"
},
"downloads": -1,
"filename": "aiida_s3-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "55540b4ae0750090a7e461adea402117",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 14262,
"upload_time": "2022-12-13T08:58:22",
"upload_time_iso_8601": "2022-12-13T08:58:22.928870Z",
"url": "https://files.pythonhosted.org/packages/04/66/03958471717eb11ec4718818fe2b93c71cd9fb5597eda6940f39b1d8839a/aiida_s3-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"md5": "5944fc8567763254d110968cf688edaf",
"sha256": "c46cd919518c5c42421d55340cd19828601441945ec7cab6c54336407ed3b37f"
},
"downloads": -1,
"filename": "aiida-s3-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "5944fc8567763254d110968cf688edaf",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 11239,
"upload_time": "2022-12-13T08:58:25",
"upload_time_iso_8601": "2022-12-13T08:58:25.653565Z",
"url": "https://files.pythonhosted.org/packages/63/02/94d3da7bde22d0a7e888c9208720255a35102c4ab1fe3e349d7ab5faac44/aiida-s3-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-13 08:58:25",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "aiida-s3"
}
