kiwixstorage
============
[](https://www.codefactor.io/repository/github/openzim/python-storagelib)
[](https://www.gnu.org/licenses/gpl-3.0)
[](https://pypi.org/project/kiwixstorage/)
helpers for S3 storage, autoconf from URL + Wasabi (wasabisys.com) extras
Goal is mainly to provide a configured s3.client and s3.resource from an URL
Users could limit usage to this and use [boto3](https://boto3.amazonaws.com/) directly from there.
A few additional wrappers are in place to simplify common actions.
Also, non-S3, wasabi-specific features are exposed directly.
# Usage
``` sh
pip install kiwixstorage
```
## Connection
``` py
from kiwixstorage import KiwixStorage
url = "https://s3.us-east-1.wasabisys.com/?keyId=x&secretAccessKey=y&bucketName=z"
s3 = KiwixStorage(url)
# test credentials and ensure we can list buckets
if not s3.check_credentials(list_buckets=True, failsafe=True):
return # bad auth
```
## Scraper use-case
``` py
online_url = "https://xxx"
fpath = "/local/path.ext"
# retrieve origin etag
etag = requests.head(online_url, allow_redirects=True).headers.get("Etag")
# check if we have that very same version in store
if s3.has_matching_object(key=url, etag=etag)
# lastest version in our store, download from there (using progress output)
s3.download_file(key=url, fpath=fpath, progress=True)
else:
# download the origin file using your regular tools
download_file(url, fpath)
# upload it our storage
s3.upload_file(fpath=fpath, key=url)
# now you have a local file of lastest version and the storage is up to date
```
# Other use cases
``` py
# create a bucket
bucket = s3.create_bucket("bucket_name")
# set auto-delete on bucket
s3.set_bucket_autodelete_after(nb_days=7)
# allow public downloads from bucket
s3.allow_public_downloads_on()
# upload a file
s3.upload_file(fpath, "some/path/file.img", meta={"ENCODER_VERSION": "v1"})
# set autodelete on specific file
s3.set_object_autodelete_on(key, datetime.datetime.now())
# download a file
s3.download_file(key, fpath)
# get URL for external download
s3.get_download_url(key)
```
# Resources:
* https://wasabi.com/wp-content/themes/wasabi/docs/API_Guide
* https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html
Raw data
{
"_id": null,
"home_page": "https://github.com/kiwix/python_storagelib",
"name": "kiwixstorage",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "kiwix zim offline aws s3",
"author": "kiwix",
"author_email": "reg@kiwix.org",
"download_url": "https://files.pythonhosted.org/packages/31/8c/ad36d332a6cfddbdbd793f5a86978aa74e399798aa1862a8ead52dbcdd07/kiwixstorage-0.9.0.tar.gz",
"platform": null,
"description": "kiwixstorage\n============\n\n[](https://www.codefactor.io/repository/github/openzim/python-storagelib)\n[](https://www.gnu.org/licenses/gpl-3.0)\n[](https://pypi.org/project/kiwixstorage/)\n\nhelpers for S3 storage, autoconf from URL + Wasabi (wasabisys.com) extras\n\nGoal is mainly to provide a configured s3.client and s3.resource from an URL\nUsers could limit usage to this and use [boto3](https://boto3.amazonaws.com/) directly from there.\n\nA few additional wrappers are in place to simplify common actions.\nAlso, non-S3, wasabi-specific features are exposed directly.\n\n# Usage\n\n``` sh\npip install kiwixstorage\n```\n\n## Connection\n\n``` py\nfrom kiwixstorage import KiwixStorage\nurl = \"https://s3.us-east-1.wasabisys.com/?keyId=x&secretAccessKey=y&bucketName=z\"\ns3 = KiwixStorage(url)\n# test credentials and ensure we can list buckets\nif not s3.check_credentials(list_buckets=True, failsafe=True):\n return # bad auth\n```\n\n## Scraper use-case\n\n``` py\nonline_url = \"https://xxx\"\nfpath = \"/local/path.ext\"\n# retrieve origin etag\netag = requests.head(online_url, allow_redirects=True).headers.get(\"Etag\")\n# check if we have that very same version in store\nif s3.has_matching_object(key=url, etag=etag)\n # lastest version in our store, download from there (using progress output)\n s3.download_file(key=url, fpath=fpath, progress=True)\nelse:\n # download the origin file using your regular tools\n download_file(url, fpath)\n # upload it our storage\n s3.upload_file(fpath=fpath, key=url)\n# now you have a local file of lastest version and the storage is up to date\n```\n\n# Other use cases\n\n``` py\n# create a bucket\nbucket = s3.create_bucket(\"bucket_name\")\n\n# set auto-delete on bucket\ns3.set_bucket_autodelete_after(nb_days=7)\n\n# allow public downloads from bucket\ns3.allow_public_downloads_on()\n\n# upload a file\ns3.upload_file(fpath, \"some/path/file.img\", meta={\"ENCODER_VERSION\": \"v1\"})\n\n# set autodelete on specific file\ns3.set_object_autodelete_on(key, datetime.datetime.now())\n\n# download a file\ns3.download_file(key, fpath)\n\n# get URL for external download\ns3.get_download_url(key)\n\n```\n\n# Resources:\n\n* https://wasabi.com/wp-content/themes/wasabi/docs/API_Guide\n* https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html\n",
"bugtrack_url": null,
"license": "GPLv3+",
"summary": "Kiwix S3 Cache wrapper to use within Kiwix/OpenZIM projects",
"version": "0.9.0",
"project_urls": {
"Homepage": "https://github.com/kiwix/python_storagelib"
},
"split_keywords": [
"kiwix",
"zim",
"offline",
"aws",
"s3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b14e95fc535d9fb2b40c8f53af15fde87fa7436e849b126603ef986f8651bb2f",
"md5": "e8b1990582d7d39fe155f91bac603162",
"sha256": "f0772ed9d57cc62197dfdfbe479aefed84860002e274743d6d1cd3a643687e1a"
},
"downloads": -1,
"filename": "kiwixstorage-0.9.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e8b1990582d7d39fe155f91bac603162",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 37173,
"upload_time": "2024-05-08T14:17:54",
"upload_time_iso_8601": "2024-05-08T14:17:54.888366Z",
"url": "https://files.pythonhosted.org/packages/b1/4e/95fc535d9fb2b40c8f53af15fde87fa7436e849b126603ef986f8651bb2f/kiwixstorage-0.9.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "318cad36d332a6cfddbdbd793f5a86978aa74e399798aa1862a8ead52dbcdd07",
"md5": "6d9a48fd7512af2ae2518119fe27e12b",
"sha256": "1ea147ba253236d0dabfd8ea9f173544c680ce3721000e60f361f6fd010270ba"
},
"downloads": -1,
"filename": "kiwixstorage-0.9.0.tar.gz",
"has_sig": false,
"md5_digest": "6d9a48fd7512af2ae2518119fe27e12b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 38592,
"upload_time": "2024-05-08T14:17:57",
"upload_time_iso_8601": "2024-05-08T14:17:57.406117Z",
"url": "https://files.pythonhosted.org/packages/31/8c/ad36d332a6cfddbdbd793f5a86978aa74e399798aa1862a8ead52dbcdd07/kiwixstorage-0.9.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-08 14:17:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "kiwix",
"github_project": "python_storagelib",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "boto3",
"specs": [
[
">=",
"1.12.39"
],
[
"<",
"2"
]
]
},
{
"name": "requests",
"specs": [
[
"<",
"3.0"
],
[
">=",
"2.23"
]
]
},
{
"name": "aws-requests-auth",
"specs": [
[
">=",
"0.4.2"
],
[
"<",
"0.5"
]
]
}
],
"tox": true,
"lcname": "kiwixstorage"
}
JSON
