# HEA Server AWS S3 Bucket Folders Microservice
[Research Informatics Shared Resource](https://risr.hci.utah.edu), [Huntsman Cancer Institute](https://healthcare.utah.edu/huntsmancancerinstitute/),
Salt Lake City, UT
The HEA Server AWS S3 Bucket Folders Microservice manages folders in AWS S3 buckets.
## Version 1.8.1
* Fixed bug where copy fails if the target folder contains an object with a name that is a prefix of the object to be copied.
* Performance improvements.
## Version 1.8.0
* Added support for python 3.12.
* Improved performance of getting projects.
## Version 1.7.1
* Fixed regression causing move to fail.
## Version 1.7.0
* Removed integration tests that overlap with the unit tests.
* Accept the data query parameter for get requests for a speed boost.
## Version 1.6.1
* Dependency upgrades for compatibility with heaserver-keychain 1.5.0.
* Fixed wrong readOnly status for a trash item's size attribute in unit test.
## Version 1.6.0
* Make metadata follow objects during moves, copies, renames, and trash restores.
* Clear cache properly after deleting and restoring objects.
* Clear cache properly after moves, copies, and renames.
* Merged trash microservice.
## Version 1.5.6
* Fixed regression causing the service to crash when the user closes their browser in the middle of a download.
## Version 1.5.5
* Only resort to asynchronous get-items call when the call has taken longer than 30 seconds to come back.
* Use cache when retrieving file desktop objects.
* Fixed caching issue when moving a file from the root of a bucket.
* Reimplemented download to work with aiohttp 3.10.
## Version 1.5.4
* Fixed stale cache when converting a folder to a project.
* Permissions calculation speedup.
* Fixed stale cache when archiving a file.
## Version 1.5.3
* Fixed stale cache when copying a file to the root of a bucket.
## Version 1.5.2
* Fixed file renaming regression.
## Version 1.5.1
* Caching optimizations.
## Version 1.5.0
* Support mode and access_token query parameters when getting a file object.
## Version 1.4.1
* Improved permission denied messages.
## Version 1.4.0
* Present accurate file permissions.
## Version 1.3.0
* Merged the AWS S3 files microservice into this one.
* Fixed caching bugs affecting web client object explorer refresh.
* Avoid timeouts loading objects, which sporadically caused objects not to be returned.
## Version 1.2.2
* Prevent failed content downloads from hanging the microservice.
## Version 1.2.1
* Install setuptools first during installation.
* Correct issue where some users lost access to folder and folder items because the user lacked permissions in AWS to simulate permissions. Instead, such users will appear to receive full permission for everything, which was the behavior prior to version 1.2.0. As before, AWS will still reject requests that users lack permission for.
## Version 1.2.0
* Present accurate bucket permissions.
## Version 1.1.6
* Minor bug fixes.
## Version 1.1.5
* Made a project's unarchive restore duration required in the unarchive card.
## Version 1.1.4
* Made a folder's unarchive restore duration required in the unarchive card.
## Version 1.1.3
* Fixed potential issue preventing the service from updating temporary credentials.
## Version 1.1.2
* Fixed new folder form submission.
## Version 1.1.1
* Display type display name in properties card, and return the type display name from GET calls.
## Version 1.1.0
* Pass folder and project permissions back to clients.
## Version 1.0.13
* Changed presented bucket owner to system|aws.
* Omitted shares from the properties template.
## Version 1.0.12
* Improved upload desktop object action message.
## Version 1.0.11
* Improved performance.
## Version 1.0.10
* Support getting the content of a folder as a zip file when the folder has files > 2GiB in size.
## Version 1.0.9
* Prevent zip file corruption when getting the content of a folder.
## Version 1.0.8
* Addressed issue where downloads start failing for all users if one user interrupts their download.
## Version 1.0.7
* Addressed potential failures to connect to other CORE Browser microservices.
## Version 1.0.6
* Addressed potential exception while unarchiving objects.
* Addressed issue preventing copying and moving folders containing unarchived objects.
* Improved error message when attempting to copy or move a folder with archived objects.
## Version 1.0.5
* Improved validation for downloading objects and generating presigned URLs.
## Version 1.0.4
* Improved performance.
* Allow unarchived S3 objects to be downloaded.
## Version 1.0.3
* Fixed project downloading.
## Version 1.0.2
* Fixed project copy and move causing 404 error.
## Version 1.0.1
* Improved performance.
* Corrected issue copying, moving, and renaming folders and projects containing archived objects.
* Corrected error opening a project with an archived README.*.
* Skip archived objects when downloading a folder or project.
## Version 1
Initial release.
## Runtime requirements
* Python 3.10, 3.11, or 3.12.
## Development environment
### Build requirements
* Any development environment is fine.
* On Windows, you also will need:
* Build Tools for Visual Studio 2019, found at https://visualstudio.microsoft.com/downloads/. Select the C++ tools.
* git, found at https://git-scm.com/download/win.
* On Mac, Xcode or the command line developer tools is required, found in the Apple Store app.
* Python 3.10, 3.11, or 3.12: Download and install Python from https://www.python.org, and select the options to
install for all users and add Python to your environment variables. The install for all users option will help keep you
from accidentally installing packages into your Python installation's site-packages directory instead of to your
virtualenv environment, described below.
* Create a virtualenv environment using the `python -m venv <venv_directory>` command, substituting `<venv_directory>`
with the directory name of your virtual environment. Run `source <venv_directory>/bin/activate` (or `<venv_directory>/Scripts/activate` on Windows) to activate the virtual
environment. You will need to activate the virtualenv every time before starting work, or your IDE may be able to do
this for you automatically. **Note that PyCharm will do this for you, but you have to create a new Terminal panel
after you newly configure a project with your virtualenv.**
* From the project's root directory, and using the activated virtualenv, run `pip install wheel` followed by
`pip install -r requirements_dev.txt`. **Do NOT run `python setup.py develop`. It will break your environment.**
### Running tests
Run tests with the `pytest` command from the project root directory. To improve performance, run tests in multiple
processes with `pytest -n auto`.
### Running integration tests
* Install Docker
* On Windows, install pywin32 version >= 223 from https://github.com/mhammond/pywin32/releases. In your venv, make sure that
`include-system-site-packages` is set to `true`.
* A compatible heaserver-registry Docker image must be available.
* Run tests with the `pytest integrationtests` command from the project root directory.
### Trying out the APIs
This microservice has Swagger3/OpenAPI support so that you can quickly test the APIs in a web browser. Do the following:
* Install Docker, if it is not installed already.
* Have a heaserver-registry docker image in your Docker cache. You can generate one using the Dockerfile in the
heaserver-registry project.
* Run the `run-swaggerui.py` file in your terminal. This file contains some test objects that are loaded into a MongoDB
Docker container.
* Go to http://127.0.0.1:8080/docs in your web browser.
Once `run-swaggerui.py` is running, you can also access the APIs via `curl` or other tool. For example, in Windows
PowerShell, execute:
```
Invoke-RestMethod -Uri http://localhost:8080/awss3folders/root/items -Method GET -Headers @{'accept' = 'application/json'}`
```
In MacOS or Linux, the equivalent command is:
```
curl -X GET http://localhost:8080/awss3folders/root/items -H 'accept: application/json'
```
### Packaging and releasing this project
See the [RELEASING.md](RELEASING.md) file for details.
Raw data
{
"_id": null,
"home_page": "https://risr.hci.utah.edu",
"name": "heaserver-folders-aws-s3",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Research Informatics Shared Resource, Huntsman Cancer Institute, Salt Lake City, UT",
"author_email": "Andrew.Post@hci.utah.edu",
"download_url": "https://files.pythonhosted.org/packages/77/04/c03a96e4186511fcb651c3cc3ce1b933298022ec62a2e3b7e66cb02d90b4/heaserver_folders_aws_s3-1.8.1.tar.gz",
"platform": null,
"description": "# HEA Server AWS S3 Bucket Folders Microservice\r\n[Research Informatics Shared Resource](https://risr.hci.utah.edu), [Huntsman Cancer Institute](https://healthcare.utah.edu/huntsmancancerinstitute/),\r\nSalt Lake City, UT\r\n\r\nThe HEA Server AWS S3 Bucket Folders Microservice manages folders in AWS S3 buckets.\r\n\r\n## Version 1.8.1\r\n* Fixed bug where copy fails if the target folder contains an object with a name that is a prefix of the object to be copied.\r\n* Performance improvements.\r\n\r\n## Version 1.8.0\r\n* Added support for python 3.12.\r\n* Improved performance of getting projects.\r\n\r\n## Version 1.7.1\r\n* Fixed regression causing move to fail.\r\n\r\n## Version 1.7.0\r\n* Removed integration tests that overlap with the unit tests.\r\n* Accept the data query parameter for get requests for a speed boost.\r\n\r\n## Version 1.6.1\r\n* Dependency upgrades for compatibility with heaserver-keychain 1.5.0.\r\n* Fixed wrong readOnly status for a trash item's size attribute in unit test.\r\n\r\n## Version 1.6.0\r\n* Make metadata follow objects during moves, copies, renames, and trash restores.\r\n* Clear cache properly after deleting and restoring objects.\r\n* Clear cache properly after moves, copies, and renames.\r\n* Merged trash microservice.\r\n\r\n## Version 1.5.6\r\n* Fixed regression causing the service to crash when the user closes their browser in the middle of a download.\r\n\r\n## Version 1.5.5\r\n* Only resort to asynchronous get-items call when the call has taken longer than 30 seconds to come back.\r\n* Use cache when retrieving file desktop objects.\r\n* Fixed caching issue when moving a file from the root of a bucket.\r\n* Reimplemented download to work with aiohttp 3.10.\r\n\r\n## Version 1.5.4\r\n* Fixed stale cache when converting a folder to a project.\r\n* Permissions calculation speedup.\r\n* Fixed stale cache when archiving a file.\r\n\r\n## Version 1.5.3\r\n* Fixed stale cache when copying a file to the root of a bucket.\r\n\r\n## Version 1.5.2\r\n* Fixed file renaming regression.\r\n\r\n## Version 1.5.1\r\n* Caching optimizations.\r\n\r\n## Version 1.5.0\r\n* Support mode and access_token query parameters when getting a file object.\r\n\r\n## Version 1.4.1\r\n* Improved permission denied messages.\r\n\r\n## Version 1.4.0\r\n* Present accurate file permissions.\r\n\r\n## Version 1.3.0\r\n* Merged the AWS S3 files microservice into this one.\r\n* Fixed caching bugs affecting web client object explorer refresh.\r\n* Avoid timeouts loading objects, which sporadically caused objects not to be returned.\r\n\r\n## Version 1.2.2\r\n* Prevent failed content downloads from hanging the microservice.\r\n\r\n## Version 1.2.1\r\n* Install setuptools first during installation.\r\n* Correct issue where some users lost access to folder and folder items because the user lacked permissions in AWS to simulate permissions. Instead, such users will appear to receive full permission for everything, which was the behavior prior to version 1.2.0. As before, AWS will still reject requests that users lack permission for.\r\n\r\n## Version 1.2.0\r\n* Present accurate bucket permissions.\r\n\r\n## Version 1.1.6\r\n* Minor bug fixes.\r\n\r\n## Version 1.1.5\r\n* Made a project's unarchive restore duration required in the unarchive card.\r\n\r\n## Version 1.1.4\r\n* Made a folder's unarchive restore duration required in the unarchive card.\r\n\r\n## Version 1.1.3\r\n* Fixed potential issue preventing the service from updating temporary credentials.\r\n\r\n## Version 1.1.2\r\n* Fixed new folder form submission.\r\n\r\n## Version 1.1.1\r\n* Display type display name in properties card, and return the type display name from GET calls.\r\n\r\n## Version 1.1.0\r\n* Pass folder and project permissions back to clients.\r\n\r\n## Version 1.0.13\r\n* Changed presented bucket owner to system|aws.\r\n* Omitted shares from the properties template.\r\n\r\n## Version 1.0.12\r\n* Improved upload desktop object action message.\r\n\r\n## Version 1.0.11\r\n* Improved performance.\r\n\r\n## Version 1.0.10\r\n* Support getting the content of a folder as a zip file when the folder has files > 2GiB in size.\r\n\r\n## Version 1.0.9\r\n* Prevent zip file corruption when getting the content of a folder.\r\n\r\n## Version 1.0.8\r\n* Addressed issue where downloads start failing for all users if one user interrupts their download.\r\n\r\n## Version 1.0.7\r\n* Addressed potential failures to connect to other CORE Browser microservices.\r\n\r\n## Version 1.0.6\r\n* Addressed potential exception while unarchiving objects.\r\n* Addressed issue preventing copying and moving folders containing unarchived objects.\r\n* Improved error message when attempting to copy or move a folder with archived objects.\r\n\r\n## Version 1.0.5\r\n* Improved validation for downloading objects and generating presigned URLs.\r\n\r\n## Version 1.0.4\r\n* Improved performance.\r\n* Allow unarchived S3 objects to be downloaded.\r\n\r\n## Version 1.0.3\r\n* Fixed project downloading.\r\n\r\n## Version 1.0.2\r\n* Fixed project copy and move causing 404 error.\r\n\r\n## Version 1.0.1\r\n* Improved performance.\r\n* Corrected issue copying, moving, and renaming folders and projects containing archived objects.\r\n* Corrected error opening a project with an archived README.*.\r\n* Skip archived objects when downloading a folder or project.\r\n\r\n## Version 1\r\nInitial release.\r\n\r\n## Runtime requirements\r\n* Python 3.10, 3.11, or 3.12.\r\n\r\n## Development environment\r\n\r\n### Build requirements\r\n* Any development environment is fine.\r\n* On Windows, you also will need:\r\n * Build Tools for Visual Studio 2019, found at https://visualstudio.microsoft.com/downloads/. Select the C++ tools.\r\n * git, found at https://git-scm.com/download/win.\r\n* On Mac, Xcode or the command line developer tools is required, found in the Apple Store app.\r\n* Python 3.10, 3.11, or 3.12: Download and install Python from https://www.python.org, and select the options to\r\ninstall for all users and add Python to your environment variables. The install for all users option will help keep you\r\nfrom accidentally installing packages into your Python installation's site-packages directory instead of to your\r\nvirtualenv environment, described below.\r\n* Create a virtualenv environment using the `python -m venv <venv_directory>` command, substituting `<venv_directory>`\r\nwith the directory name of your virtual environment. Run `source <venv_directory>/bin/activate` (or `<venv_directory>/Scripts/activate` on Windows) to activate the virtual\r\nenvironment. You will need to activate the virtualenv every time before starting work, or your IDE may be able to do\r\nthis for you automatically. **Note that PyCharm will do this for you, but you have to create a new Terminal panel\r\nafter you newly configure a project with your virtualenv.**\r\n* From the project's root directory, and using the activated virtualenv, run `pip install wheel` followed by\r\n `pip install -r requirements_dev.txt`. **Do NOT run `python setup.py develop`. It will break your environment.**\r\n\r\n### Running tests\r\nRun tests with the `pytest` command from the project root directory. To improve performance, run tests in multiple\r\nprocesses with `pytest -n auto`.\r\n\r\n### Running integration tests\r\n* Install Docker\r\n* On Windows, install pywin32 version >= 223 from https://github.com/mhammond/pywin32/releases. In your venv, make sure that\r\n`include-system-site-packages` is set to `true`.\r\n* A compatible heaserver-registry Docker image must be available.\r\n* Run tests with the `pytest integrationtests` command from the project root directory.\r\n\r\n### Trying out the APIs\r\nThis microservice has Swagger3/OpenAPI support so that you can quickly test the APIs in a web browser. Do the following:\r\n* Install Docker, if it is not installed already.\r\n* Have a heaserver-registry docker image in your Docker cache. You can generate one using the Dockerfile in the\r\n heaserver-registry project.\r\n* Run the `run-swaggerui.py` file in your terminal. This file contains some test objects that are loaded into a MongoDB\r\n Docker container.\r\n* Go to http://127.0.0.1:8080/docs in your web browser.\r\n\r\nOnce `run-swaggerui.py` is running, you can also access the APIs via `curl` or other tool. For example, in Windows\r\nPowerShell, execute:\r\n```\r\nInvoke-RestMethod -Uri http://localhost:8080/awss3folders/root/items -Method GET -Headers @{'accept' = 'application/json'}`\r\n```\r\nIn MacOS or Linux, the equivalent command is:\r\n```\r\ncurl -X GET http://localhost:8080/awss3folders/root/items -H 'accept: application/json'\r\n```\r\n\r\n### Packaging and releasing this project\r\nSee the [RELEASING.md](RELEASING.md) file for details.\r\n",
"bugtrack_url": null,
"license": null,
"summary": "The HEA AWS S3 bucket folder service.",
"version": "1.8.1",
"project_urls": {
"Homepage": "https://risr.hci.utah.edu"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "602bd6a82581af8f676e6b5330ba2dde3131c2e95b0997140c366ab8e6e417fb",
"md5": "aa0ba0a405ed61d13c6249d41fa9e353",
"sha256": "15f29c18cb8cd09b1aed614bbadef79809b38c50e3ce04704a68a4932637f8b2"
},
"downloads": -1,
"filename": "heaserver_folders_aws_s3-1.8.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "aa0ba0a405ed61d13c6249d41fa9e353",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 72963,
"upload_time": "2024-12-04T01:00:10",
"upload_time_iso_8601": "2024-12-04T01:00:10.498320Z",
"url": "https://files.pythonhosted.org/packages/60/2b/d6a82581af8f676e6b5330ba2dde3131c2e95b0997140c366ab8e6e417fb/heaserver_folders_aws_s3-1.8.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7704c03a96e4186511fcb651c3cc3ce1b933298022ec62a2e3b7e66cb02d90b4",
"md5": "8c41658da7f8da1361794226aba031df",
"sha256": "268547246e1afb70d5d4344f13a8239f7e690ae6030c64b46a16a87a765a20f0"
},
"downloads": -1,
"filename": "heaserver_folders_aws_s3-1.8.1.tar.gz",
"has_sig": false,
"md5_digest": "8c41658da7f8da1361794226aba031df",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 18494941,
"upload_time": "2024-12-04T01:00:19",
"upload_time_iso_8601": "2024-12-04T01:00:19.886374Z",
"url": "https://files.pythonhosted.org/packages/77/04/c03a96e4186511fcb651c3cc3ce1b933298022ec62a2e3b7e66cb02d90b4/heaserver_folders_aws_s3-1.8.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-04 01:00:19",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "heaserver-folders-aws-s3"
}
JSON