===========================================================
Python library to copy, sync and move files to and from s3
===========================================================
|Unittests| |License| |Downloads| |Language| |PyVersions|
.. |Unittests| image:: https://github.com/andyil/s3shutil/actions/workflows/unitests.yml/badge.svg
.. |Downloads| image:: https://img.shields.io/pypi/dw/s3shutil
.. |License| image:: https://img.shields.io/github/license/andyil/s3shutil
:target: https://github.com/andyil/s3shutil/blob/develop/LICENSE
:alt: License
.. |Language| image:: https://img.shields.io/github/languages/top/andyil/s3shutil
.. |PyVersions| image:: https://img.shields.io/pypi/pyversions/s3shutil.svg
**s3shutil is the easiest to use and fastest way of moving around directories and files in s3.**
.. note::
December 1st, 2023. Just released sync operation
Sync operation allows you to incrementally copy to destination files that
were added to source since the last copy
Supports all directions: s3 to s3, s3 to local drive, local drive to s3.
Installation
---------------
We recommend installing from the official PyPI repository.
.. code-block:: sh
$ pip install s3shutil
Design Principles
~~~~~~~~~~~~~~~~~
* A simple and intuitive string based API.
* Symmetric API: download and uploads work equally
* Exposes powerful and performant one-liners.
* Emulate the well known `shutil <https://docs.python.org/3/library/shutil.html>`_ standard module API.
* Use performance boosts behind the scenes (multithreading, batching, server to server operations)
* No dependencies except boto3
Using s3shutil
~~~~~~~~~~~~~~
s3shutil uses `boto3 <https://github.com/boto/boto3>`_ internally and we assume you have your credentials set up properly.
Using s3shutil is super easy:
**Import is mandatory, no suprises here**:
.. code-block:: python
import s3shutil
**Then you can do powerful things with simple one liners:**:
.. code-block:: python
# download a tree from s3
s3shutil.copytree('s3://bucket/my/path', '/home/myuser/files/')
# upload a tree to s3
s3shutil.copytree('/home/users/pics/', 's3://bucket/path/archive/')
# copy between two s3 locations
# same or different bucket
s3shutil.copytree('s3://bucket2/files/someth/', 's3://bucket1/backup/old/')
# delete (recursively) entire prefix
s3shutil.rmtree('s3://bucket/my-files/documents/')
**Just released! (December 2023), tree_sync operation:**
Only copies files that are missing in the destination.
Also deletes extra files.
.. code-block:: python
# sync download
s3shutil.tree_sync('s3://bucket/files/docs/', '/home/myuser/docs')
# sync upload
s3shutil.tree_sync('/home/myuser/files/', 's3://bucket/files/docs-v2/')
# sync two bucket locations
s3shutil.tree_sync('s3://bucket/files/docs/', 's3://bucket2/a/b/c')
Conclusions
~~~~~~~~~~~~~~
s3shutil will notice alone if the location is s3 (starts with s3://) or not
All operations have a similar string based API of powerfull one liners
Test Matrix
~~~~~~~~~~~~~~
s3shutil is thoroughly unit tested in all the combinations of:
Python Versions:
+ 3.12
+ 3.11
+ 3.10
+ 3.9
+ 3.8
+ 3.7
And boto3 Versions:
+ 1.33
+ 1.30
+ 1.28
+ 1.27
+ 1.26
+ 1.25
+ 1.24
+ 1.23
Contact
~~~~~~~~~~~~~~
Just use it! You can send an email as well `andyworms@gmail.com`.
All emails are (eventually) answered.
Also read the code, fork, open a PR, start a discussion.
Raw data
{
"_id": null,
"home_page": "https://github.com/andyil/s3shutil",
"name": "s3shutil",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "aws,s3,cloud,storage,shutil,network",
"author": "",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/f3/22/ae8930d197c6a5b96b4e0162e0a37d601c83c81ba4b757e020b0332aaa67/s3shutil-0.37.tar.gz",
"platform": null,
"description": "===========================================================\nPython library to copy, sync and move files to and from s3\n===========================================================\n|Unittests| |License| |Downloads| |Language| |PyVersions|\n\n.. |Unittests| image:: https://github.com/andyil/s3shutil/actions/workflows/unitests.yml/badge.svg\n \n.. |Downloads| image:: https://img.shields.io/pypi/dw/s3shutil\n \n.. |License| image:: https://img.shields.io/github/license/andyil/s3shutil\n :target: https://github.com/andyil/s3shutil/blob/develop/LICENSE\n :alt: License\n\n.. |Language| image:: https://img.shields.io/github/languages/top/andyil/s3shutil\n\n.. |PyVersions| image:: https://img.shields.io/pypi/pyversions/s3shutil.svg\n\n**s3shutil is the easiest to use and fastest way of moving around directories and files in s3.**\n\n\n.. note::\n December 1st, 2023. Just released sync operation\n\n Sync operation allows you to incrementally copy to destination files that\n were added to source since the last copy\n Supports all directions: s3 to s3, s3 to local drive, local drive to s3.\n\n\nInstallation\n---------------\nWe recommend installing from the official PyPI repository.\n\n.. code-block:: sh\n\n $ pip install s3shutil\n \n\n\n\n\nDesign Principles\n~~~~~~~~~~~~~~~~~\n* A simple and intuitive string based API.\n* Symmetric API: download and uploads work equally\n* Exposes powerful and performant one-liners.\n* Emulate the well known `shutil <https://docs.python.org/3/library/shutil.html>`_ standard module API.\n* Use performance boosts behind the scenes (multithreading, batching, server to server operations)\n* No dependencies except boto3\n\n\nUsing s3shutil\n~~~~~~~~~~~~~~\ns3shutil uses `boto3 <https://github.com/boto/boto3>`_ internally and we assume you have your credentials set up properly.\n\nUsing s3shutil is super easy:\n\n**Import is mandatory, no suprises here**:\n\n.. code-block:: python\n\n import s3shutil\n\n**Then you can do powerful things with simple one liners:**:\n\n.. code-block:: python\n\n # download a tree from s3\n s3shutil.copytree('s3://bucket/my/path', '/home/myuser/files/')\n\n # upload a tree to s3\n s3shutil.copytree('/home/users/pics/', 's3://bucket/path/archive/')\n\n # copy between two s3 locations\n # same or different bucket\n s3shutil.copytree('s3://bucket2/files/someth/', 's3://bucket1/backup/old/')\n\n # delete (recursively) entire prefix\n s3shutil.rmtree('s3://bucket/my-files/documents/')\n\n\n**Just released! (December 2023), tree_sync operation:**\n\nOnly copies files that are missing in the destination.\nAlso deletes extra files.\n\n\n.. code-block:: python\n\n # sync download\n s3shutil.tree_sync('s3://bucket/files/docs/', '/home/myuser/docs')\n\n # sync upload\n s3shutil.tree_sync('/home/myuser/files/', 's3://bucket/files/docs-v2/')\n\n # sync two bucket locations\n s3shutil.tree_sync('s3://bucket/files/docs/', 's3://bucket2/a/b/c')\n\n\nConclusions\n~~~~~~~~~~~~~~\ns3shutil will notice alone if the location is s3 (starts with s3://) or not\nAll operations have a similar string based API of powerfull one liners\n\n\nTest Matrix\n~~~~~~~~~~~~~~\ns3shutil is thoroughly unit tested in all the combinations of:\n\nPython Versions:\n\n+ 3.12\n+ 3.11 \n+ 3.10\n+ 3.9\n+ 3.8\n+ 3.7\n\nAnd boto3 Versions: \n\n+ 1.33\n+ 1.30\n+ 1.28\n+ 1.27\n+ 1.26\n+ 1.25\n+ 1.24\n+ 1.23\n\n\nContact\n~~~~~~~~~~~~~~\nJust use it! You can send an email as well `andyworms@gmail.com`.\nAll emails are (eventually) answered.\nAlso read the code, fork, open a PR, start a discussion.\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Easy pythonic API to copy and sync to and from s3",
"version": "0.37",
"project_urls": {
"Download": "https://github.com/andyil/s3shutil/archive/0.28.tar.gz",
"Homepage": "https://github.com/andyil/s3shutil"
},
"split_keywords": [
"aws",
"s3",
"cloud",
"storage",
"shutil",
"network"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f322ae8930d197c6a5b96b4e0162e0a37d601c83c81ba4b757e020b0332aaa67",
"md5": "76bc0c1dd452babe5f46da05299e447a",
"sha256": "02001654150beca717bb288694f195c6cdadbdedec64392e43668654a486f910"
},
"downloads": -1,
"filename": "s3shutil-0.37.tar.gz",
"has_sig": false,
"md5_digest": "76bc0c1dd452babe5f46da05299e447a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 17302,
"upload_time": "2023-12-01T19:48:15",
"upload_time_iso_8601": "2023-12-01T19:48:15.642270Z",
"url": "https://files.pythonhosted.org/packages/f3/22/ae8930d197c6a5b96b4e0162e0a37d601c83c81ba4b757e020b0332aaa67/s3shutil-0.37.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-01 19:48:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "andyil",
"github_project": "s3shutil",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "s3shutil"
}
JSON
