# saritasa-s3-tools
[](https://pypi.org/project/saritasa-s3-tools/)
[](https://github.com/astral-sh/ruff)
Extension for boto3 to ease work with s3
## Table of contents
* [Installation](#installation)
* [Features](#features)
* [Django](#django)
* [Optional dependencies](#optional-dependencies)
* [Direct upload example](#direct-upload-example)
* [Pytest](#pytest-plugin)
## Installation
```bash
pip install saritasa-s3-tools
```
or if you are using [poetry](https://python-poetry.org/)
```bash
poetry add saritasa-s3-tools
```
## Features
* `S3Client` and `AsyncS3Client` for integrations with s3 buckets. This clients
are extension to boto3 clients with proper typing, support for async and
method to generate signed urls for file upload.
* `S3FileTypeConfig` for defining configuration parameters for direct upload to s3.
[Check out more](saritasa_s3_tools/configs.py#L24)
* `S3Key` for generating unique keys for s3 upload, used for `S3FileTypeConfig`
* `S3FileField` and `S3ImageFileField` - [factory-boy](https://github.com/FactoryBoy/factory_boy) fields for generating files and saving it in `s3`
* `pytest` plugin with fixtures for `boto3`, `S3Client` and `AsyncS3Client` and etc
* `Django` plugin for setting up models and api
## Django
`saritasa-s3-tools` comes with django plugin which helps with models
config and api(`Django Rest Framework` and `drf_spectacular`).
You can try it out in `example` folder.
### Setup model
First you need to add file/image fields to you model, like below
```python
import saritasa_s3_tools.django
from django.db import models
class ModelWithFiles(models.Model):
"""Test model with different files configs."""
file = saritasa_s3_tools.django.S3FileField(
blank=True,
null=True,
# S3FileTypeConfig is needed so that we could understand where to save
# file, who can save file, what files can be save and what are size
# constraints
s3_config=saritasa_s3_tools.S3FileTypeConfig(
name="django-files",
key=saritasa_s3_tools.keys.WithPrefixUUIDFolder("django-files"),
allowed=("text/plain",),
auth=lambda user: bool(user and user.is_authenticated),
content_length_range=(1000, 20000000),
),
)
image = saritasa_s3_tools.django.S3ImageField(
blank=True,
null=True,
s3_config=saritasa_s3_tools.S3FileTypeConfig(
name="django-images",
key=saritasa_s3_tools.keys.WithPrefixUUIDFolder("django-images"),
allowed=("image/png",),
content_length_range=(5000, 20000000),
),
)
```
### Setup serializers
Then add `S3FieldsConfigMixin` mixin to your serializer, like this
```python
from rest_framework import serializers
import saritasa_s3_tools.django
from .. import models
class ModelWithFilesSerializer(
saritasa_s3_tools.django.S3FieldsConfigMixin,
serializers.ModelSerializer,
):
"""Serializer to show info model with files."""
class Meta:
model = models.ModelWithFiles
fields = "__all__"
```
### Setup view
Then just add `S3GetParamsView` view to your project urls like that.
```python
from django.urls import path
path(
"s3/",
include("saritasa_s3_tools.django.urls"),
name="saritasa-s3-tools",
),
```
### Setup pytest
Just add this to core `conftest.py` file
```python
import pytest
@pytest.fixture(scope="session", autouse=True)
def _adjust_s3_bucket(django_adjust_s3_bucket: None) -> None:
"""Set bucket to a test one."""
```
### Note about signature version
By default we assume `s3v4` version for signature. We recommend that you would
set `AWS_S3_SIGNATURE_VERSION` to `s3v4`. If you need other versions, set it in
`AWS_S3_SIGNATURE_VERSION` and update `SARITASA_S3_TOOLS_UPLOAD_PARAMS`
(defaults are [here](saritasa_s3_tools/constants.py)) setting
to reflect expected fields that would return.
## Optional dependencies
* `[async]` - Add this to enable async support
* `[factory]` - Add this to enable factory-boy field `S3FileField` and `S3ImageFileField`
from `saritasa_s3_tools.factory`
* `[testing]` - Add this to enable testing helping functions from
`saritasa_s3_tools.testing.shortcuts`
* `[django]` - Add this to enable [django support](#django)
* `[django-openapi]` - Add this to enable [drf-spectacular support](#django)
To install all optional dependencies add `[all]`
## Direct upload example
```python
import saritasa_s3_tools
import pathlib
import xml.etree.ElementTree
s3_client = saritasa_s3_tools.S3Client(
boto3_client=boto3_client,
default_bucket=s3_bucket,
)
s3_params = s3_client.generate_params(
filename=pathlib.Path(__file__).name,
config=saritasa_s3_tools.S3FileTypeConfig.configs["files"],
content_type="application/x-python-code",
extra_metadata={
"test": "123",
},
)
with (
httpx.Client() as client,
pathlib.Path(__file__).open("rb") as upload_file,
):
upload_response = client.post(
url=s3_params.url,
data={
key: value
for key, value in s3_params.params.items()
if value is not None
},
files={"file": upload_file.read()},
)
parsed_response = xml.etree.ElementTree.fromstring( # noqa: S314
upload_response.content.decode(),
)
file_key = parsed_response[2].text
file_url = parsed_response[0].text
```
## pytest plugin
`saritasa-s3-tools` comes with pytest plugin which can setup boto3 instances
and create/clean up buckets for testing. Supports `pytest-xdist`.
### Fixtures
* `access_key_getter`, `s3_endpoint_url_getter`, `s3_region` - are used to
configure boto3 session and clients/resources that are used in tests.
You can override them or set values in ini file for pytest. Plugin will tell
what you are missing.
* `aws_session` - Returns `boto3.Session`
* `aws_config` - Returns `botocore.config.Config`, override if you need
customization, None by default.
* `boto3_resource` - Returns s3 resource or in typing `mypy_boto3_s3.S3ServiceResource`
* `boto3_client`- Returns s3 client or in typing `mypy_boto3_s3.S3Client`
* `s3_bucket_name` - Name of bucket for testing, default: `saritasa-s3-tools`
or `s3_bucket_name` from ini file.
* `s3_bucket_cleaner` - Returns function which cleans all files from bucket
* `s3_bucket_factory` - Returns manager which creates bucket, and when it's no
longer needed deletes it
* `s3_bucket` - Creates bucket via `s3_bucket_factory` and return it's name
* `s3_client` - Returns `saritasa_s3_tools.S3Client`
* `async_s3_client` - Returns `saritasa_s3_tools.AsyncS3Client`
Raw data
{
"_id": null,
"home_page": "https://pypi.org/project/saritasa-s3-tools/",
"name": "saritasa-s3-tools",
"maintainer": "Stanislav Khlud",
"docs_url": null,
"requires_python": "<4.0,>=3.11",
"maintainer_email": "stanislav.khlud@saritasa.com",
"keywords": "python, aws, s3, boto3, django, drf",
"author": "Saritasa",
"author_email": "pypi@saritasa.com",
"download_url": "https://files.pythonhosted.org/packages/ac/a3/e4f040530fb81180a4dfc64cc79dd0c6dc1be31c02f97ef25b0b9dff9b3c/saritasa_s3_tools-0.3.1.tar.gz",
"platform": null,
"description": "# saritasa-s3-tools\n\n\n[](https://pypi.org/project/saritasa-s3-tools/)\n\n\n\n\n\n[](https://github.com/astral-sh/ruff)\n\nExtension for boto3 to ease work with s3\n\n## Table of contents\n\n* [Installation](#installation)\n* [Features](#features)\n* [Django](#django)\n* [Optional dependencies](#optional-dependencies)\n* [Direct upload example](#direct-upload-example)\n* [Pytest](#pytest-plugin)\n\n## Installation\n\n```bash\npip install saritasa-s3-tools\n```\n\nor if you are using [poetry](https://python-poetry.org/)\n\n```bash\npoetry add saritasa-s3-tools\n```\n\n## Features\n\n* `S3Client` and `AsyncS3Client` for integrations with s3 buckets. This clients\nare extension to boto3 clients with proper typing, support for async and\nmethod to generate signed urls for file upload.\n* `S3FileTypeConfig` for defining configuration parameters for direct upload to s3.\n[Check out more](saritasa_s3_tools/configs.py#L24)\n* `S3Key` for generating unique keys for s3 upload, used for `S3FileTypeConfig`\n* `S3FileField` and `S3ImageFileField` - [factory-boy](https://github.com/FactoryBoy/factory_boy) fields for generating files and saving it in `s3`\n* `pytest` plugin with fixtures for `boto3`, `S3Client` and `AsyncS3Client` and etc\n* `Django` plugin for setting up models and api\n\n## Django\n\n`saritasa-s3-tools` comes with django plugin which helps with models\nconfig and api(`Django Rest Framework` and `drf_spectacular`).\nYou can try it out in `example` folder.\n\n### Setup model\n\nFirst you need to add file/image fields to you model, like below\n\n```python\nimport saritasa_s3_tools.django\n\nfrom django.db import models\n\nclass ModelWithFiles(models.Model):\n \"\"\"Test model with different files configs.\"\"\"\n\n file = saritasa_s3_tools.django.S3FileField(\n blank=True,\n null=True,\n # S3FileTypeConfig is needed so that we could understand where to save\n # file, who can save file, what files can be save and what are size\n # constraints\n s3_config=saritasa_s3_tools.S3FileTypeConfig(\n name=\"django-files\",\n key=saritasa_s3_tools.keys.WithPrefixUUIDFolder(\"django-files\"),\n allowed=(\"text/plain\",),\n auth=lambda user: bool(user and user.is_authenticated),\n content_length_range=(1000, 20000000),\n ),\n )\n\n image = saritasa_s3_tools.django.S3ImageField(\n blank=True,\n null=True,\n s3_config=saritasa_s3_tools.S3FileTypeConfig(\n name=\"django-images\",\n key=saritasa_s3_tools.keys.WithPrefixUUIDFolder(\"django-images\"),\n allowed=(\"image/png\",),\n content_length_range=(5000, 20000000),\n ),\n )\n```\n\n### Setup serializers\n\nThen add `S3FieldsConfigMixin` mixin to your serializer, like this\n\n```python\nfrom rest_framework import serializers\n\nimport saritasa_s3_tools.django\n\nfrom .. import models\n\n\nclass ModelWithFilesSerializer(\n saritasa_s3_tools.django.S3FieldsConfigMixin,\n serializers.ModelSerializer,\n):\n \"\"\"Serializer to show info model with files.\"\"\"\n\n class Meta:\n model = models.ModelWithFiles\n fields = \"__all__\"\n\n```\n\n### Setup view\n\nThen just add `S3GetParamsView` view to your project urls like that.\n\n```python\nfrom django.urls import path\n\npath(\n \"s3/\",\n include(\"saritasa_s3_tools.django.urls\"),\n name=\"saritasa-s3-tools\",\n),\n```\n\n### Setup pytest\n\nJust add this to core `conftest.py` file\n\n```python\nimport pytest\n\n\n@pytest.fixture(scope=\"session\", autouse=True)\ndef _adjust_s3_bucket(django_adjust_s3_bucket: None) -> None:\n \"\"\"Set bucket to a test one.\"\"\"\n```\n\n### Note about signature version\n\nBy default we assume `s3v4` version for signature. We recommend that you would\nset `AWS_S3_SIGNATURE_VERSION` to `s3v4`. If you need other versions, set it in\n`AWS_S3_SIGNATURE_VERSION` and update `SARITASA_S3_TOOLS_UPLOAD_PARAMS`\n(defaults are [here](saritasa_s3_tools/constants.py)) setting\nto reflect expected fields that would return.\n\n## Optional dependencies\n\n* `[async]` - Add this to enable async support\n* `[factory]` - Add this to enable factory-boy field `S3FileField` and `S3ImageFileField`\nfrom `saritasa_s3_tools.factory`\n* `[testing]` - Add this to enable testing helping functions from\n`saritasa_s3_tools.testing.shortcuts`\n* `[django]` - Add this to enable [django support](#django)\n* `[django-openapi]` - Add this to enable [drf-spectacular support](#django)\n\nTo install all optional dependencies add `[all]`\n\n## Direct upload example\n\n```python\nimport saritasa_s3_tools\nimport pathlib\nimport xml.etree.ElementTree\n\ns3_client = saritasa_s3_tools.S3Client(\n boto3_client=boto3_client,\n default_bucket=s3_bucket,\n)\ns3_params = s3_client.generate_params(\n filename=pathlib.Path(__file__).name,\n config=saritasa_s3_tools.S3FileTypeConfig.configs[\"files\"],\n content_type=\"application/x-python-code\",\n extra_metadata={\n \"test\": \"123\",\n },\n)\nwith (\n httpx.Client() as client,\n pathlib.Path(__file__).open(\"rb\") as upload_file,\n):\n upload_response = client.post(\n url=s3_params.url,\n data={\n key: value\n for key, value in s3_params.params.items()\n if value is not None\n },\n files={\"file\": upload_file.read()},\n )\nparsed_response = xml.etree.ElementTree.fromstring( # noqa: S314\n upload_response.content.decode(),\n)\nfile_key = parsed_response[2].text\nfile_url = parsed_response[0].text\n```\n\n## pytest plugin\n\n`saritasa-s3-tools` comes with pytest plugin which can setup boto3 instances\nand create/clean up buckets for testing. Supports `pytest-xdist`.\n\n### Fixtures\n\n* `access_key_getter`, `s3_endpoint_url_getter`, `s3_region` - are used to\nconfigure boto3 session and clients/resources that are used in tests.\nYou can override them or set values in ini file for pytest. Plugin will tell\nwhat you are missing.\n* `aws_session` - Returns `boto3.Session`\n* `aws_config` - Returns `botocore.config.Config`, override if you need\ncustomization, None by default.\n* `boto3_resource` - Returns s3 resource or in typing `mypy_boto3_s3.S3ServiceResource`\n* `boto3_client`- Returns s3 client or in typing `mypy_boto3_s3.S3Client`\n* `s3_bucket_name` - Name of bucket for testing, default: `saritasa-s3-tools`\nor `s3_bucket_name` from ini file.\n* `s3_bucket_cleaner` - Returns function which cleans all files from bucket\n* `s3_bucket_factory` - Returns manager which creates bucket, and when it's no\nlonger needed deletes it\n* `s3_bucket` - Creates bucket via `s3_bucket_factory` and return it's name\n* `s3_client` - Returns `saritasa_s3_tools.S3Client`\n* `async_s3_client` - Returns `saritasa_s3_tools.AsyncS3Client`\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Tools For S3 Used By Saritasa",
"version": "0.3.1",
"project_urls": {
"Homepage": "https://pypi.org/project/saritasa-s3-tools/",
"Repository": "https://github.com/saritasa-nest/saritasa-s3-tools/"
},
"split_keywords": [
"python",
" aws",
" s3",
" boto3",
" django",
" drf"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8bd6b3957a594a6f5e65ec3a0f40f52791bfe2e4b40edaea180b92d0ce433107",
"md5": "ae296ceab8c662fc206059817b244776",
"sha256": "3618c6379795c6ca0f520a814503fa9bbd5c34d155cec6c3ed0c1b72f5e4136a"
},
"downloads": -1,
"filename": "saritasa_s3_tools-0.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "ae296ceab8c662fc206059817b244776",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.11",
"size": 22823,
"upload_time": "2024-12-09T08:09:05",
"upload_time_iso_8601": "2024-12-09T08:09:05.766399Z",
"url": "https://files.pythonhosted.org/packages/8b/d6/b3957a594a6f5e65ec3a0f40f52791bfe2e4b40edaea180b92d0ce433107/saritasa_s3_tools-0.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "aca3e4f040530fb81180a4dfc64cc79dd0c6dc1be31c02f97ef25b0b9dff9b3c",
"md5": "c9c6c762454d140e6785d308a7a1ef91",
"sha256": "3b537409e38b254cbc1c9c1b5f20dbc69e81129357e57d59e45ff165786146e6"
},
"downloads": -1,
"filename": "saritasa_s3_tools-0.3.1.tar.gz",
"has_sig": false,
"md5_digest": "c9c6c762454d140e6785d308a7a1ef91",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.11",
"size": 21773,
"upload_time": "2024-12-09T08:09:07",
"upload_time_iso_8601": "2024-12-09T08:09:07.646609Z",
"url": "https://files.pythonhosted.org/packages/ac/a3/e4f040530fb81180a4dfc64cc79dd0c6dc1be31c02f97ef25b0b9dff9b3c/saritasa_s3_tools-0.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-09 08:09:07",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "saritasa-nest",
"github_project": "saritasa-s3-tools",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "saritasa-s3-tools"
}
JSON
