# s3local
[](https://badge.fury.io/py/s3local)
[](https://github.com/toyama0919/s3local/actions/workflows/tests.yml)
Cache the object in s3 to localhost.
Create a cache corresponding to s3 and automatically create a path for localhost and return it.
Once downloaded files remain in localhost as cache, the second migration download will be skipped
works on python3.6 or higher
## Settings
aws auth support following.
* environment variables
* profile(use --aws-profile option.)
* instance profile
## Examples
#### download object and list object
```bash
$ s3local download -u s3://mybucket/artifacts/ --debug
2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log
2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main2.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log
2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main3.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.log
# next download is skip
$ s3local download -u s3://mybucket/artifacts/ --debug
2021-05-14 14:08:02,970 DEBUG - skip already exists in local: s3://mybucket/artifacts/main.log
2021-05-14 14:08:02,970 DEBUG - skip already exists in local: s3://mybucket/artifacts/main2.log
2021-05-14 14:08:02,970 DEBUG - skip already exists in local: s3://mybucket/artifacts/main3.log
# overwrite download. (not skip)
$ s3local download -u s3://mybucket/artifacts/ --debug --no-skip-exist
2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log
2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main2.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log
2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main3.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.log
```
By default `$HOME/.s3local` is the root directory.
The format of path in local is as follows:
```
$HOME/.s3local/s3/${bucket}/${key}
```
You can change root by setting an environment variable S3LOCAL_ROOT.
```bash
$ s3local list-local -u s3://mybucket/artifacts/
/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log
/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log
/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.log
```
#### upload object
```bash
$ s3local upload -s tox.ini -u s3://mybucket/test/
2023-05-31 10:44:08,474 INFO - Copying to s3: tox.ini => s3://mybucket/test/tox.ini
```
## Python API
### download
```python
from s3local import Downloader
s3local = Downloader("s3://mybucket/artifacts/")
list = s3local.list_local_path(download=True)
print(list)
#=> [
# "/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log",
# "/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log",
# "/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.log",
# ]
```
### upload
```python
from s3local import Uploader
s3local = Uploader("s3://mybucket/artifacts/")
uploader.upload("output/hoge.txt")
#=> s3://mybucket/artifacts/hoge.txt
uploader.upload("output")
#=> s3://mybucket/artifacts/output/hoge.txt
```
## Installation
```sh
pip install s3local
```
## CI
### install test package
```
$ ./scripts/ci.sh install
```
### test
```
$ ./scripts/ci.sh run-test
```
flake8 and black and pytest.
### release pypi
```
$ ./scripts/ci.sh release
```
git tag and pypi release.
Raw data
{
"_id": null,
"home_page": "https://github.com/toyama0919/s3local",
"name": "s3local",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "s3local tool aws s3",
"author": "Hiroshi Toyama",
"author_email": "toyama0919@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/c6/f9/3aad74633ee19f7b1433e5fc63924729a447dbce133d0b68d169f8daba0b/s3local-0.4.2.tar.gz",
"platform": null,
"description": "# s3local\n\n[](https://badge.fury.io/py/s3local)\n[](https://github.com/toyama0919/s3local/actions/workflows/tests.yml)\n\nCache the object in s3 to localhost.\n\nCreate a cache corresponding to s3 and automatically create a path for localhost and return it.\n\nOnce downloaded files remain in localhost as cache, the second migration download will be skipped\n\nworks on python3.6 or higher\n\n## Settings\n\naws auth support following.\n\n* environment variables\n* profile(use --aws-profile option.)\n* instance profile\n\n## Examples\n\n#### download object and list object\n\n```bash\n$ s3local download -u s3://mybucket/artifacts/ --debug\n2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log\n2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main2.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log\n2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main3.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.log\n\n# next download is skip\n$ s3local download -u s3://mybucket/artifacts/ --debug\n2021-05-14 14:08:02,970 DEBUG - skip already exists in local: s3://mybucket/artifacts/main.log\n2021-05-14 14:08:02,970 DEBUG - skip already exists in local: s3://mybucket/artifacts/main2.log\n2021-05-14 14:08:02,970 DEBUG - skip already exists in local: s3://mybucket/artifacts/main3.log\n\n# overwrite download. (not skip)\n$ s3local download -u s3://mybucket/artifacts/ --debug --no-skip-exist\n2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log\n2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main2.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log\n2021-05-14 11:27:13,367 DEBUG - Copying: s3://mybucket/artifacts/main3.log > /Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.log\n```\n\nBy default `$HOME/.s3local` is the root directory.\n\nThe format of path in local is as follows:\n\n```\n$HOME/.s3local/s3/${bucket}/${key}\n```\n\nYou can change root by setting an environment variable S3LOCAL_ROOT.\n\n```bash\n$ s3local list-local -u s3://mybucket/artifacts/\n/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log\n/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log\n/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.log\n```\n\n#### upload object\n\n```bash\n$ s3local upload -s tox.ini -u s3://mybucket/test/\n2023-05-31 10:44:08,474 INFO - Copying to s3: tox.ini => s3://mybucket/test/tox.ini\n```\n\n## Python API\n\n### download\n\n```python\nfrom s3local import Downloader\n\ns3local = Downloader(\"s3://mybucket/artifacts/\")\nlist = s3local.list_local_path(download=True)\nprint(list)\n#=> [\n# \"/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log\",\n# \"/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log\",\n# \"/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.log\",\n# ]\n\n```\n\n### upload\n\n```python\nfrom s3local import Uploader\n\ns3local = Uploader(\"s3://mybucket/artifacts/\")\n\nuploader.upload(\"output/hoge.txt\")\n#=> s3://mybucket/artifacts/hoge.txt\n\nuploader.upload(\"output\")\n#=> s3://mybucket/artifacts/output/hoge.txt\n\n```\n\n## Installation\n\n```sh\npip install s3local\n```\n\n## CI\n\n### install test package\n\n```\n$ ./scripts/ci.sh install\n```\n\n### test\n\n```\n$ ./scripts/ci.sh run-test\n```\n\nflake8 and black and pytest.\n\n### release pypi\n\n```\n$ ./scripts/ci.sh release\n```\n\ngit tag and pypi release.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Command Line utility for s3 local caching.",
"version": "0.4.2",
"project_urls": {
"Homepage": "https://github.com/toyama0919/s3local"
},
"split_keywords": [
"s3local",
"tool",
"aws",
"s3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c6f93aad74633ee19f7b1433e5fc63924729a447dbce133d0b68d169f8daba0b",
"md5": "337d51dfa676a9db2de50e672a97e3ab",
"sha256": "162ccc329e575023aba33a8dadadbca73a154c6c872dfcd19c69ef42a0b4e6f7"
},
"downloads": -1,
"filename": "s3local-0.4.2.tar.gz",
"has_sig": false,
"md5_digest": "337d51dfa676a9db2de50e672a97e3ab",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 7424,
"upload_time": "2024-07-16T13:30:56",
"upload_time_iso_8601": "2024-07-16T13:30:56.064058Z",
"url": "https://files.pythonhosted.org/packages/c6/f9/3aad74633ee19f7b1433e5fc63924729a447dbce133d0b68d169f8daba0b/s3local-0.4.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-07-16 13:30:56",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "toyama0919",
"github_project": "s3local",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "s3local"
}
JSON
