<p align="center">
<a href="https://logyca.com/"><img src="https://logyca.com/sites/default/files/logyca.png" alt="Logyca"></a>
</p>
<p align="center">
<em>LOGYCA public libraries</em>
</p>
<p align="center">
<a href="https://pypi.org/project/logyca-azure-storage-blob" target="_blank">
<img src="https://img.shields.io/pypi/v/logyca-azure-storage-blob?color=orange&label=PyPI%20Package" alt="Package version">
</a>
<a href="(https://www.python.org" target="_blank">
<img src="https://img.shields.io/badge/Python-%5B%3E%3D3.8%2C%3C%3D3.11%5D-orange" alt="Python">
</a>
</p>
---
# About us
* <a href="http://logyca.com" target="_blank">LOGYCA Company</a>
* <a href="https://www.youtube.com/channel/UCzcJtxfScoAtwFbxaLNnEtA" target="_blank">LOGYCA Youtube Channel</a>
* <a href="https://www.linkedin.com/company/logyca" target="_blank"><img src="https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white" alt="Linkedin"></a>
* <a href="https://twitter.com/LOGYCA_Org" target="_blank"><img src="https://img.shields.io/badge/Twitter-1DA1F2?style=for-the-badge&logo=twitter&logoColor=white" alt="Twitter"></a>
* <a href="https://www.facebook.com/OrganizacionLOGYCA/" target="_blank"><img src="https://img.shields.io/badge/Facebook-1877F2?style=for-the-badge&logo=facebook&logoColor=white" alt="Facebook"></a>
---
# LOGYCA public libraries: To interact with the blob container files in your Azure Storage account.
[Source code](https://github.com/logyca/python-libraries/tree/main/logyca-azure-storage-blob)
| [Package (PyPI)](https://pypi.org/project/logyca-azure-storage-blob/)
| [Samples](https://github.com/logyca/python-libraries/tree/main/logyca-azure-storage-blob/samples)
---
To install the libraries
```Python
# Check SQLAlchemy dependency that is installed
pip install logyca_azure_storage_blob
```
# How azcopy performs file consistency validation
Like azcopy, Microsoft's client for uploading large files to Blob, the md5 hash must be computed locally and uploaded to the blob properties along with the file, the blob service will commit this at the end of the process, returning an error if presents inconsistency in value.
## Calculates the MD5 hash of the file content and saves it as the Content-MD5 property of the blob:
```console
azcopy --put-md5 #
```
References:
- https://learn.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-copy
This library uses the concepts of the azcopy client for its implementation.
---
# Test upload
bandwith movistar: upload/download: 300 mbps
## Test 1, function: container_blob_upload_staging_blocks_commit with console output
file: 20240126084440-backup.dump
size: 2.03 GB
parameters:
- self.__chunk_size_bytes=4*1024*1024 # Defaults 4 MiB
Elapsed time: 00:05:22
Output:
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[42/522], Percentage[8.05%]
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[43/522], Percentage[8.24%]
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[44/522], Percentage[8.43%]
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[45/522], Percentage[8.62%]
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[46/522], Percentage[8.81%]
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[47/522], Percentage[9.00%]
...
...
...
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[517/522], Percentage[99.04%]
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[518/522], Percentage[99.23%]
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[519/522], Percentage[99.43%]
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[520/522], Percentage[99.62%]
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[521/522], Percentage[99.81%]
Upload process: file_to_upload[20240126084440-backup.dump], Chunks[522/522], Percentage[100.00%]
## Test 2, function: container_blob_upload_data_transfer_options no console output
file: 20240126084440-backup.dump
size: 2.03 GB
parameters:
- self.__chunk_size_bytes=4*1024*1024 # 4 MiB
- self.__max_single_put_size=8*1024*1024 # 8 MiB.
- max_concurrency=1
Elapsed time: 00:11:19
Unsupported formats: ---
Output: Not available
## Test 3, function: container_blob_upload_data_transfer_options no console output
file: 20240126084440-backup.dump
size: 2.03 GB
parameters:
- self.__chunk_size_bytes=4*1024*1024 # 4 MiB
- self.__max_single_put_size=8*1024*1024 # 8 MiB.
- max_concurrency=2
Elapsed time: 00:03:50
Unsupported formats: ---
Output: Not available
## Test 4, function: container_blob_upload_data_transfer_options no console output
file: 20240126084440-backup.dump
size: 2.03 GB
parameters:
- self.__chunk_size_bytes=4*1024*1024 # 4 MiB
- self.__max_single_put_size=8*1024*1024 # 8 MiB.
- max_concurrency=3
Elapsed time: 00:03:38
Unsupported formats: ---
Output: Not available
---
# Semantic Versioning
logyca_azure_storage_blob < MAJOR >.< MINOR >.< PATCH >
* **MAJOR**: version when you make incompatible API changes
* **MINOR**: version when you add functionality in a backwards compatible manner
* **PATCH**: version when you make backwards compatible bug fixes
## Definitions for releasing versions
* https://peps.python.org/pep-0440/
- X.YaN (Alpha release): Identify and fix early-stage bugs. Not suitable for production use.
- X.YbN (Beta release): Stabilize and refine features. Address reported bugs. Prepare for official release.
- X.YrcN (Release candidate): Final version before official release. Assumes all major features are complete and stable. Recommended for testing in non-critical environments.
- X.Y (Final release/Stable/Production): Completed, stable version ready for use in production. Full release for public use.
---
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## Types of changes
- Added for new features.
- Changed for changes in existing functionality.
- Deprecated for soon-to-be removed features.
- Removed for now removed features.
- Fixed for any bug fixes.
- Security in case of vulnerabilities.
## [0.0.1rc1] - 2024-05-24
### Added
- First tests using pypi.org in develop environment.
## [0.1.0] - 2024-05-24
### Added
- Completion of testing and launch into production.
## [0.1.2] - 2024-07-18
### Changed
- Two filter sa container blob list are added. modified hours ago to return modified blobs after x hours and include_subfolders to return only blobs from the received folder list.
Raw data
{
"_id": null,
"home_page": "https://github.com/logyca/python-libraries/tree/main/logyca-azure-storage-blob",
"name": "logyca-azure-storage-blob",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "azure, blob, storage, upload, download",
"author": "Jaime Andres Cardona Carrillo",
"author_email": "jacardona@outlook.com",
"download_url": "https://files.pythonhosted.org/packages/58/03/337dfc4a990343197fdebd02b7c333d9a8fcce14068c6fd0ac451c2bf7d3/logyca_azure_storage_blob-0.1.2.tar.gz",
"platform": null,
"description": "<p align=\"center\">\r\n <a href=\"https://logyca.com/\"><img src=\"https://logyca.com/sites/default/files/logyca.png\" alt=\"Logyca\"></a>\r\n</p>\r\n<p align=\"center\">\r\n <em>LOGYCA public libraries</em>\r\n</p>\r\n\r\n<p align=\"center\">\r\n<a href=\"https://pypi.org/project/logyca-azure-storage-blob\" target=\"_blank\">\r\n <img src=\"https://img.shields.io/pypi/v/logyca-azure-storage-blob?color=orange&label=PyPI%20Package\" alt=\"Package version\">\r\n</a>\r\n<a href=\"(https://www.python.org\" target=\"_blank\">\r\n <img src=\"https://img.shields.io/badge/Python-%5B%3E%3D3.8%2C%3C%3D3.11%5D-orange\" alt=\"Python\">\r\n</a>\r\n</p>\r\n\r\n\r\n---\r\n\r\n# About us\r\n\r\n* <a href=\"http://logyca.com\" target=\"_blank\">LOGYCA Company</a>\r\n* <a href=\"https://www.youtube.com/channel/UCzcJtxfScoAtwFbxaLNnEtA\" target=\"_blank\">LOGYCA Youtube Channel</a>\r\n* <a href=\"https://www.linkedin.com/company/logyca\" target=\"_blank\"><img src=\"https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white\" alt=\"Linkedin\"></a>\r\n* <a href=\"https://twitter.com/LOGYCA_Org\" target=\"_blank\"><img src=\"https://img.shields.io/badge/Twitter-1DA1F2?style=for-the-badge&logo=twitter&logoColor=white\" alt=\"Twitter\"></a>\r\n* <a href=\"https://www.facebook.com/OrganizacionLOGYCA/\" target=\"_blank\"><img src=\"https://img.shields.io/badge/Facebook-1877F2?style=for-the-badge&logo=facebook&logoColor=white\" alt=\"Facebook\"></a>\r\n\r\n---\r\n\r\n# LOGYCA public libraries: To interact with the blob container files in your Azure Storage account.\r\n\r\n[Source code](https://github.com/logyca/python-libraries/tree/main/logyca-azure-storage-blob)\r\n| [Package (PyPI)](https://pypi.org/project/logyca-azure-storage-blob/)\r\n| [Samples](https://github.com/logyca/python-libraries/tree/main/logyca-azure-storage-blob/samples)\r\n\r\n---\r\n\r\nTo install the libraries\r\n\r\n```Python\r\n# Check SQLAlchemy dependency that is installed\r\npip install logyca_azure_storage_blob\r\n```\r\n\r\n\r\n# How azcopy performs file consistency validation\r\n\r\nLike azcopy, Microsoft's client for uploading large files to Blob, the md5 hash must be computed locally and uploaded to the blob properties along with the file, the blob service will commit this at the end of the process, returning an error if presents inconsistency in value.\r\n\r\n## Calculates the MD5 hash of the file content and saves it as the Content-MD5 property of the blob:\r\n\r\n\r\n```console\r\nazcopy --put-md5 #\r\n```\r\n\r\nReferences:\r\n\r\n - https://learn.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-copy\r\n\r\nThis library uses the concepts of the azcopy client for its implementation.\r\n\r\n---\r\n\r\n# Test upload\r\n\r\nbandwith movistar: upload/download: 300 mbps\r\n\r\n## Test 1, function: container_blob_upload_staging_blocks_commit with console output\r\n\r\n file: 20240126084440-backup.dump\r\n size: 2.03 GB\r\n parameters: \r\n - self.__chunk_size_bytes=4*1024*1024 # Defaults 4 MiB\r\n Elapsed time: 00:05:22\r\n Output:\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[42/522], Percentage[8.05%]\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[43/522], Percentage[8.24%]\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[44/522], Percentage[8.43%]\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[45/522], Percentage[8.62%]\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[46/522], Percentage[8.81%]\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[47/522], Percentage[9.00%]\r\n ...\r\n ...\r\n ...\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[517/522], Percentage[99.04%]\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[518/522], Percentage[99.23%]\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[519/522], Percentage[99.43%]\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[520/522], Percentage[99.62%]\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[521/522], Percentage[99.81%]\r\n Upload process: file_to_upload[20240126084440-backup.dump], Chunks[522/522], Percentage[100.00%]\r\n\r\n## Test 2, function: container_blob_upload_data_transfer_options no console output\r\n file: 20240126084440-backup.dump\r\n size: 2.03 GB \r\n parameters:\r\n - self.__chunk_size_bytes=4*1024*1024 # 4 MiB\r\n - self.__max_single_put_size=8*1024*1024 # 8 MiB.\r\n - max_concurrency=1\r\n Elapsed time: 00:11:19\r\n Unsupported formats: ---\r\n Output: Not available\r\n\r\n## Test 3, function: container_blob_upload_data_transfer_options no console output\r\n file: 20240126084440-backup.dump\r\n size: 2.03 GB \r\n parameters:\r\n - self.__chunk_size_bytes=4*1024*1024 # 4 MiB\r\n - self.__max_single_put_size=8*1024*1024 # 8 MiB.\r\n - max_concurrency=2\r\n Elapsed time: 00:03:50\r\n Unsupported formats: ---\r\n Output: Not available\r\n\r\n## Test 4, function: container_blob_upload_data_transfer_options no console output\r\n file: 20240126084440-backup.dump\r\n size: 2.03 GB \r\n parameters:\r\n - self.__chunk_size_bytes=4*1024*1024 # 4 MiB\r\n - self.__max_single_put_size=8*1024*1024 # 8 MiB.\r\n - max_concurrency=3\r\n Elapsed time: 00:03:38\r\n Unsupported formats: ---\r\n Output: Not available\r\n\r\n---\r\n\r\n# Semantic Versioning\r\n\r\nlogyca_azure_storage_blob < MAJOR >.< MINOR >.< PATCH >\r\n\r\n* **MAJOR**: version when you make incompatible API changes\r\n* **MINOR**: version when you add functionality in a backwards compatible manner\r\n* **PATCH**: version when you make backwards compatible bug fixes\r\n\r\n## Definitions for releasing versions\r\n* https://peps.python.org/pep-0440/\r\n\r\n - X.YaN (Alpha release): Identify and fix early-stage bugs. Not suitable for production use.\r\n - X.YbN (Beta release): Stabilize and refine features. Address reported bugs. Prepare for official release.\r\n - X.YrcN (Release candidate): Final version before official release. Assumes all major features are complete and stable. Recommended for testing in non-critical environments.\r\n - X.Y (Final release/Stable/Production): Completed, stable version ready for use in production. Full release for public use.\r\n\r\n---\r\n\r\n# Changelog\r\n\r\nAll notable changes to this project will be documented in this file.\r\n\r\nThe format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),\r\nand this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).\r\n\r\n## Types of changes\r\n\r\n- Added for new features.\r\n- Changed for changes in existing functionality.\r\n- Deprecated for soon-to-be removed features.\r\n- Removed for now removed features.\r\n- Fixed for any bug fixes.\r\n- Security in case of vulnerabilities.\r\n\r\n## [0.0.1rc1] - 2024-05-24\r\n### Added\r\n- First tests using pypi.org in develop environment.\r\n\r\n## [0.1.0] - 2024-05-24\r\n### Added\r\n- Completion of testing and launch into production.\r\n\r\n## [0.1.2] - 2024-07-18\r\n### Changed\r\n- Two filter sa container blob list are added. modified hours ago to return modified blobs after x hours and include_subfolders to return only blobs from the received folder list.\r\n\r\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "An integration package created by the company LOGYCA to interact with the blob container files in your Azure Storage account.",
"version": "0.1.2",
"project_urls": {
"Homepage": "https://github.com/logyca/python-libraries/tree/main/logyca-azure-storage-blob"
},
"split_keywords": [
"azure",
" blob",
" storage",
" upload",
" download"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0c4e7b63b5afa726e3b8e072331ab211fb8da3cb184786e91ea4ecc7c37b4952",
"md5": "2d7f065b09863740055d14728e8bd2ea",
"sha256": "b7a773648e6de267747f50ccac45d1f377be454f700838b8637da75c8d9f8580"
},
"downloads": -1,
"filename": "logyca_azure_storage_blob-0.1.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "2d7f065b09863740055d14728e8bd2ea",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 12830,
"upload_time": "2024-07-18T15:00:04",
"upload_time_iso_8601": "2024-07-18T15:00:04.610565Z",
"url": "https://files.pythonhosted.org/packages/0c/4e/7b63b5afa726e3b8e072331ab211fb8da3cb184786e91ea4ecc7c37b4952/logyca_azure_storage_blob-0.1.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5803337dfc4a990343197fdebd02b7c333d9a8fcce14068c6fd0ac451c2bf7d3",
"md5": "89857f11b6edf309ae1c4821b605599a",
"sha256": "9c6e7f4ff1d9163a8cb5ac1a8e811db088cec4f031e3092802ae30549e1b7b07"
},
"downloads": -1,
"filename": "logyca_azure_storage_blob-0.1.2.tar.gz",
"has_sig": false,
"md5_digest": "89857f11b6edf309ae1c4821b605599a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 14585,
"upload_time": "2024-07-18T15:00:06",
"upload_time_iso_8601": "2024-07-18T15:00:06.379290Z",
"url": "https://files.pythonhosted.org/packages/58/03/337dfc4a990343197fdebd02b7c333d9a8fcce14068c6fd0ac451c2bf7d3/logyca_azure_storage_blob-0.1.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-18 15:00:06",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "logyca",
"github_project": "python-libraries",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "logyca-azure-storage-blob"
}
JSON
