# tosfs
[](https://pypi.python.org/pypi/tosfs/)
[](https://pypi.org/project/tosfs/)
[](https://pypi.org/project/tosfs/)
[](https://opensource.org/licenses/Apache-2.0)
[](https://github.com/pre-commit/pre-commit)
[](https://github.com/psf/black)
[](https://tosfs.readthedocs.io/en/latest/?badge=latest)
TOSFS builds on [Volcengine TOS Python SDK](https://github.com/volcengine/ve-tos-python-sdk) to provide a convenient Python filesystem interface for [TOS（Tinder Object Storage）](https://www.volcengine.com/docs/6349/74820).
## Features
* Excellent read/write performance (optimized by multi-threading and multi-disk staging).
* Solid stability (fine-grained judgment on response codes for TOS services).
* Outstanding compatibility (cross-validation completed on the version matrix of four Python versions and two fsspec versions).
* TOS HNS (Hierarchical NameSpace) Bucket support (in adaptation and verification).
* Native [append API](https://www.volcengine.com/docs/6349/74863) support.
* Extended features:
* batch list iterate;
* multiple thread walk;
* support special characters in local path;
## Installation
You can install tosfs via [pip](https://pip.pypa.io/) from [PyPI](https://pypi.org/):
```shell
$ pip install tosfs
```
## Quick Start
### Init FileSystem
* Init via `ak/sk`
```python
from tosfs.core import TosFileSystem
tosfs = TosFileSystem(
key='ak',
secret='sk',
endpoint_url='http://tos-cn-beijing.volces.com',
region='cn-beijing',
)
```
* Init via system env
make sure these envs take effect:
```shell
export TOS_ACCESS_KEY=' your ak here '
export TOS_SECRET_KEY=' your sk here '
export TOS_ENDPOINT='http://tos-cn-beijing.volces.com'
export TOS_REGION='cn-beijing'
```
then init `TosFileSystem` by setting `credentials_provider` to `EnvCredentialsProvider`
```python
import os
from tosfs.core import TosFileSystem
from tos import EnvCredentialsProvider
tosfs = TosFileSystem(
endpoint_url=os.environ.get("TOS_ENDPOINT"),
region=os.environ.get("TOS_REGION"),
credentials_provider=EnvCredentialsProvider, # must
)
```
### Access FS and file operation APIs
After getting the instance of `TosFileSystem` by following the above guidance,
now we can access fs and file operation apis, e.g.
```python
# list
tosfs.ls("")
tosfs.ls("mybucket", detail=False)
tosfs.ls("mybucket/rootpath/", detail=False)
# file read/write
with tosfs.open('bucket/root/text.txt', mode='wb') as f:
f.write('hello tosfs!')
with tosfs.open('bucket/root/text.txt', mode='rb') as f:
content = f.read()
print(content)
```
### integration usages
For more usage in ray, pyspark, pytorch and so on, please refer to the [examples](https://github.com/fsspec/tosfs/tree/main/examples) dir.
## Compatibility
The tosfs package is compatible with the following Python and fsspec versions:
* Python
| Version | Supported |
|---------|-----------|
| 3.9 | ✅ |
| 3.10 | ✅ |
| 3.11 | ✅ |
| 3.12 | ✅ |
* fsspec
| Version | Supported |
|-----------|------|
| 2023.5.0 | ✅ |
| 2024.9.0 | ✅ |
| 2024.10.0 | ✅ |
## Contributing
Contributions are very welcome. To learn more, see the [Contributor Guide](https://github.com/fsspec/tosfs/blob/main/CONTRIBUTING.md).
## License
Distributed under the terms of the [Apache 2.0 license](https://github.com/fsspec/tosfs/blob/main/LICENSE), Tosfs is free and open source software.
## Issues
If you encounter any problems, please [file an issue](https://github.com/fsspec/tosfs/issues/new/choose) along with a detailed description.
## Privacy Statement
Welcome to use our open-source project [tosfs](https://github.com/fsspec/tosfs). We highly value your privacy and are committed to protecting your information. This privacy statement aims to inform you about how we collect, use, store, and protect your information within the project (we believe you have the right to know).
Information Collection:
We may collect the names of the buckets you access in the TOS service
(and only the names of the buckets you access).
You can opt out of this collection by setting the environment variable
"TOS_BUCKET_TAG_ENABLE" to "false", and we will respect your decision.
Information Usage:
We will only use the collected information to tag the source of access
and commit that this information will solely be used for the purpose of
tagging the source of access.
Information Storage:
We will take reasonable security measures to protect the information
we collect from unauthorized access, disclosure, use, or destruction.
We commit that this information will not be stored on third-party service
providers' servers. Additionally, we will comply with applicable laws,
regulations, and security standards.
Information Sharing:
We will not sell, rent, or share the collected information with any third parties.
If you need any further adjustments or additional details, feel free to let us know!
Raw data
{
"_id": null,
"home_page": "https://github.com/fsspec/tosfs",
"name": "tosfs",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": "fsspec, tos, volcengine",
"author": "Vino Yang",
"author_email": "yanghua@bytedance.com",
"download_url": "https://files.pythonhosted.org/packages/f0/cf/dafbadc98077769aae56730cfa00fa977d5be900589d1d761e0a2cf522ee/tosfs-2024.11.1.tar.gz",
"platform": null,
"description": "# tosfs\n\n[](https://pypi.python.org/pypi/tosfs/)\n[](https://pypi.org/project/tosfs/)\n[](https://pypi.org/project/tosfs/)\n[](https://opensource.org/licenses/Apache-2.0)\n[](https://github.com/pre-commit/pre-commit)\n[](https://github.com/psf/black)\n[](https://tosfs.readthedocs.io/en/latest/?badge=latest)\n\n\n\nTOSFS builds on [Volcengine TOS Python SDK](https://github.com/volcengine/ve-tos-python-sdk) to provide a convenient Python filesystem interface for [TOS\uff08Tinder Object Storage\uff09](https://www.volcengine.com/docs/6349/74820).\n\n## Features\n\n* Excellent read/write performance (optimized by multi-threading and multi-disk staging).\n* Solid stability (fine-grained judgment on response codes for TOS services).\n* Outstanding compatibility (cross-validation completed on the version matrix of four Python versions and two fsspec versions).\n* TOS HNS (Hierarchical NameSpace) Bucket support (in adaptation and verification).\n* Native [append API](https://www.volcengine.com/docs/6349/74863) support.\n* Extended features:\n * batch list iterate;\n * multiple thread walk;\n * support special characters in local path;\n\n## Installation\n\nYou can install tosfs via [pip](https://pip.pypa.io/) from [PyPI](https://pypi.org/):\n\n```shell\n$ pip install tosfs\n```\n\n## Quick Start\n\n### Init FileSystem\n\n* Init via `ak/sk`\n\n```python\nfrom tosfs.core import TosFileSystem\n\ntosfs = TosFileSystem(\n key='ak',\n secret='sk',\n endpoint_url='http://tos-cn-beijing.volces.com',\n region='cn-beijing',\n)\n```\n\n* Init via system env\n\nmake sure these envs take effect:\n\n```shell\nexport TOS_ACCESS_KEY=' your ak here '\nexport TOS_SECRET_KEY=' your sk here '\nexport TOS_ENDPOINT='http://tos-cn-beijing.volces.com'\nexport TOS_REGION='cn-beijing'\n```\nthen init `TosFileSystem` by setting `credentials_provider` to `EnvCredentialsProvider`\n\n```python\nimport os\nfrom tosfs.core import TosFileSystem\nfrom tos import EnvCredentialsProvider\n\ntosfs = TosFileSystem(\n endpoint_url=os.environ.get(\"TOS_ENDPOINT\"),\n region=os.environ.get(\"TOS_REGION\"),\n credentials_provider=EnvCredentialsProvider, # must\n)\n```\n\n### Access FS and file operation APIs\n\nAfter getting the instance of `TosFileSystem` by following the above guidance,\nnow we can access fs and file operation apis, e.g.\n\n```python\n# list\ntosfs.ls(\"\")\ntosfs.ls(\"mybucket\", detail=False)\ntosfs.ls(\"mybucket/rootpath/\", detail=False)\n\n# file read/write\nwith tosfs.open('bucket/root/text.txt', mode='wb') as f:\n f.write('hello tosfs!')\n \nwith tosfs.open('bucket/root/text.txt', mode='rb') as f:\n content = f.read()\n print(content)\n```\n\n### integration usages\n\nFor more usage in ray, pyspark, pytorch and so on, please refer to the [examples](https://github.com/fsspec/tosfs/tree/main/examples) dir.\n\n## Compatibility\n\nThe tosfs package is compatible with the following Python and fsspec versions:\n\n* Python\n\n| Version | Supported |\n|---------|-----------|\n| 3.9 | \u2705 |\n| 3.10 | \u2705 |\n| 3.11 | \u2705 |\n| 3.12 | \u2705 |\n\n* fsspec\n\n| Version | Supported |\n|-----------|------|\n| 2023.5.0 | \u2705 |\n| 2024.9.0 | \u2705 |\n| 2024.10.0 | \u2705 |\n\n## Contributing\nContributions are very welcome. To learn more, see the [Contributor Guide](https://github.com/fsspec/tosfs/blob/main/CONTRIBUTING.md).\n\n## License\nDistributed under the terms of the [Apache 2.0 license](https://github.com/fsspec/tosfs/blob/main/LICENSE), Tosfs is free and open source software.\n\n## Issues\nIf you encounter any problems, please [file an issue](https://github.com/fsspec/tosfs/issues/new/choose) along with a detailed description.\n\n## Privacy Statement\nWelcome to use our open-source project [tosfs](https://github.com/fsspec/tosfs). We highly value your privacy and are committed to protecting your information. This privacy statement aims to inform you about how we collect, use, store, and protect your information within the project (we believe you have the right to know).\n\n Information Collection: \n We may collect the names of the buckets you access in the TOS service \n (and only the names of the buckets you access). \n You can opt out of this collection by setting the environment variable \n \"TOS_BUCKET_TAG_ENABLE\" to \"false\", and we will respect your decision.\n\n Information Usage: \n We will only use the collected information to tag the source of access \n and commit that this information will solely be used for the purpose of \n tagging the source of access.\n\n Information Storage: \n We will take reasonable security measures to protect the information \n we collect from unauthorized access, disclosure, use, or destruction. \n We commit that this information will not be stored on third-party service \n providers' servers. Additionally, we will comply with applicable laws, \n regulations, and security standards.\n\n Information Sharing: \n We will not sell, rent, or share the collected information with any third parties.\n\nIf you need any further adjustments or additional details, feel free to let us know!\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "tosfs is a Pythonic file-system interface for TOS\uff08Tinder Object Storage\uff09.",
"version": "2024.11.1",
"project_urls": {
"Documentation": "https://tosfs.readthedocs.io/en/latest/",
"Homepage": "https://github.com/fsspec/tosfs",
"Repository": "https://github.com/fsspec/tosfs"
},
"split_keywords": [
"fsspec",
" tos",
" volcengine"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f31cc5fd8f1af0fe8987d450d09dcd240483639f824e4b967492d235eaf5d568",
"md5": "245cdee6d77faaa07f6169f0430db787",
"sha256": "a9fe8df3bef893bfcb1effc26f92708bab2b23d4afa33c329b6307ba4d27dad1"
},
"downloads": -1,
"filename": "tosfs-2024.11.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "245cdee6d77faaa07f6169f0430db787",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 58924,
"upload_time": "2024-11-08T12:12:55",
"upload_time_iso_8601": "2024-11-08T12:12:55.293963Z",
"url": "https://files.pythonhosted.org/packages/f3/1c/c5fd8f1af0fe8987d450d09dcd240483639f824e4b967492d235eaf5d568/tosfs-2024.11.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f0cfdafbadc98077769aae56730cfa00fa977d5be900589d1d761e0a2cf522ee",
"md5": "3c5466ca0bc411965c44abc0f85ccfa3",
"sha256": "5b17bfb6c68dbff901cc7f44558e65cb9ea0873a52c3b61f16adc5db9c5db86c"
},
"downloads": -1,
"filename": "tosfs-2024.11.1.tar.gz",
"has_sig": false,
"md5_digest": "3c5466ca0bc411965c44abc0f85ccfa3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 50444,
"upload_time": "2024-11-08T12:12:57",
"upload_time_iso_8601": "2024-11-08T12:12:57.213870Z",
"url": "https://files.pythonhosted.org/packages/f0/cf/dafbadc98077769aae56730cfa00fa977d5be900589d1d761e0a2cf522ee/tosfs-2024.11.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-08 12:12:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fsspec",
"github_project": "tosfs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "tosfs"
}
JSON
