# ⚓ Minato
[](https://github.com/altescy/minato/actions/workflows/ci.yml)
[](https://github.com/altescy/minato)
[](https://github.com/altescy/minato/blob/master/LICENSE)
[](https://pypi.org/project/minato/)
A Unified File I/O Library for Python
Minato is a Python library that provides a unified and simple interface to work with local and remote files, as well as compressed and archived files.
With Minato, you can seamlessly read and write files from various sources like local filesystem, HTTP(S), Amazon S3, Google Cloud Storage, and Hugging Fase Hub.
It also supports reading and writing compressed files such as gzip, bz2, and lzma, as well as directly accessing files inside archives like zip and tar.
One of Minato's key features is its built-in caching mechanism, which allows you to cache remote files locally, and manage the cache with a provided CLI.
The cache is automatically updated based on ETag headers, ensuring that you always work with the latest version of the files.
## Features
- Unified file I/O for local and remote files (HTTP(S), S3, GCP, Hugging Face Hub)
- Support for reading and writing compressed files (gzip, bz2, lzma)
- Direct access to files inside archives (zip, tar)
- Local caching of remote files with cache management CLI
- Automatic cache updates based on ETag headers
## Installation
Install Minato using pip:
```bash
pip install minato # minimal installation for only local/http(s) file I/O
pip install minato[s3] # for Amazon S3
pip install minato[gcs] # for Google Cloud Storage
pip install minato[huggingface-hub] # for Hugging Face Hub
pip install minato[all] # for all supported file I/O
```
## Usage
### Quick Start
Here's a simple example demonstrating how to read and write files on online storage:
```python
import minato
# Write a file to an S3 bucket
s3_path = "s3://your_bucket/path/to/file"
with minato.open(s3_path, "w") as f:
f.write("Create a new file on AWS S3!")
```
Access cached online resources in local storage:
```python
# Cache a remote file and get its local path
remote_path = "http://example.com/path/to/archive.zip!inner/path/to/file"
local_filename = minato.cached_path(remote_path)
```
Access files inside archives like zip by connecting the archive path and inner file path with an exclamation mark (`!`) like above.
Automatically decompress files with gzip / lzma / bz2 compression:
```python
with minato.open("data.txt.gz", "rt", decompress=True) as f:
content = f.read()
```
In the example above, Minato will automatically detect the file format based on the file's content or filename and decompress the file accordingly.
### Cache Management
```bash
❯ poetry run minato --help
usage: minato
positional arguments:
{cache,list,remove,update}
cache cache remote file and return cached local file path
list show list of cached files
remove remove cached files
update update cached files
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
```
Raw data
{
"_id": null,
"home_page": "https://github.com/altescy/minato",
"name": "minato",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "python,cache,s3,google-cloud-storage",
"author": "Yasuhiro Yamaguchi",
"author_email": "altescy@fastmail.com",
"download_url": "https://files.pythonhosted.org/packages/db/57/a3d814449596f16fb580d1f9842a1bba80dac3cb162afeeaa409aac09145/minato-0.12.2.tar.gz",
"platform": null,
"description": "# \u2693 Minato\n\n[](https://github.com/altescy/minato/actions/workflows/ci.yml)\n[](https://github.com/altescy/minato)\n[](https://github.com/altescy/minato/blob/master/LICENSE)\n[](https://pypi.org/project/minato/)\n\nA Unified File I/O Library for Python\n\n\nMinato is a Python library that provides a unified and simple interface to work with local and remote files, as well as compressed and archived files.\nWith Minato, you can seamlessly read and write files from various sources like local filesystem, HTTP(S), Amazon S3, Google Cloud Storage, and Hugging Fase Hub.\nIt also supports reading and writing compressed files such as gzip, bz2, and lzma, as well as directly accessing files inside archives like zip and tar.\n\nOne of Minato's key features is its built-in caching mechanism, which allows you to cache remote files locally, and manage the cache with a provided CLI.\nThe cache is automatically updated based on ETag headers, ensuring that you always work with the latest version of the files.\n\n## Features\n\n- Unified file I/O for local and remote files (HTTP(S), S3, GCP, Hugging Face Hub)\n- Support for reading and writing compressed files (gzip, bz2, lzma)\n- Direct access to files inside archives (zip, tar)\n- Local caching of remote files with cache management CLI\n- Automatic cache updates based on ETag headers\n\n## Installation\n\nInstall Minato using pip:\n\n```bash\npip install minato # minimal installation for only local/http(s) file I/O\npip install minato[s3] # for Amazon S3\npip install minato[gcs] # for Google Cloud Storage\npip install minato[huggingface-hub] # for Hugging Face Hub\npip install minato[all] # for all supported file I/O\n```\n\n## Usage\n\n### Quick Start\n\nHere's a simple example demonstrating how to read and write files on online storage:\n\n```python\nimport minato\n\n# Write a file to an S3 bucket\ns3_path = \"s3://your_bucket/path/to/file\"\nwith minato.open(s3_path, \"w\") as f:\n f.write(\"Create a new file on AWS S3!\")\n```\n\nAccess cached online resources in local storage:\n\n```python\n# Cache a remote file and get its local path\nremote_path = \"http://example.com/path/to/archive.zip!inner/path/to/file\"\nlocal_filename = minato.cached_path(remote_path)\n```\n\nAccess files inside archives like zip by connecting the archive path and inner file path with an exclamation mark (`!`) like above.\n\nAutomatically decompress files with gzip / lzma / bz2 compression:\n\n```python\nwith minato.open(\"data.txt.gz\", \"rt\", decompress=True) as f:\n content = f.read()\n```\n\nIn the example above, Minato will automatically detect the file format based on the file's content or filename and decompress the file accordingly.\n\n### Cache Management\n\n```bash\n\u276f poetry run minato --help\nusage: minato\n\npositional arguments:\n {cache,list,remove,update}\n cache cache remote file and return cached local file path\n list show list of cached files\n remove remove cached files\n update update cached files\n\noptional arguments:\n -h, --help show this help message and exit\n --version show program's version number and exit\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A Unified File I/O Library for Python",
"version": "0.12.2",
"project_urls": {
"Homepage": "https://github.com/altescy/minato"
},
"split_keywords": [
"python",
"cache",
"s3",
"google-cloud-storage"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8ab1cb8c92a99a7b1f92f3d08aa0b3f14b93710ae41743cd112b0532a9aae879",
"md5": "3d09e9d4ba107d3de080e1a1bca380e4",
"sha256": "8ba0b07c72ca0cb2a456d0e6696ee47c4046b8af720e85e29a46af6409861508"
},
"downloads": -1,
"filename": "minato-0.12.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3d09e9d4ba107d3de080e1a1bca380e4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 34820,
"upload_time": "2023-07-24T10:18:54",
"upload_time_iso_8601": "2023-07-24T10:18:54.707343Z",
"url": "https://files.pythonhosted.org/packages/8a/b1/cb8c92a99a7b1f92f3d08aa0b3f14b93710ae41743cd112b0532a9aae879/minato-0.12.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "db57a3d814449596f16fb580d1f9842a1bba80dac3cb162afeeaa409aac09145",
"md5": "750c81a17e8d98863acd2aff3eedd20a",
"sha256": "1616fcb0f272e9302cea43e8a2449cc38efd974b774bd9c57ffcef7f16b4ee61"
},
"downloads": -1,
"filename": "minato-0.12.2.tar.gz",
"has_sig": false,
"md5_digest": "750c81a17e8d98863acd2aff3eedd20a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 24058,
"upload_time": "2023-07-24T10:18:56",
"upload_time_iso_8601": "2023-07-24T10:18:56.386680Z",
"url": "https://files.pythonhosted.org/packages/db/57/a3d814449596f16fb580d1f9842a1bba80dac3cb162afeeaa409aac09145/minato-0.12.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-24 10:18:56",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "altescy",
"github_project": "minato",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "minato"
}
JSON
