| Name | hf-xet JSON |
| Version |
1.1.10
JSON |
| download |
| home_page | None |
| Summary | Fast transfer of large files with the Hugging Face Hub. |
| upload_time | 2025-09-12 20:10:27 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<!---
Copyright 2024 The HuggingFace Team. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<p align="center">
<a href="https://github.com/huggingface/xet-core/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/github/license/huggingface/xet-core.svg?color=blue"></a>
<a href="https://github.com/huggingface/xet-core/releases"><img alt="GitHub release" src="https://img.shields.io/github/release/huggingface/xet-core.svg"></a>
<a href="https://github.com/huggingface/xet-core/blob/main/CODE_OF_CONDUCT.md"><img alt="Contributor Covenant" src="https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg"></a>
</p>
<h3 align="center">
<p>🤗 hf-xet - xet client tech, used in <a target="_blank" href="https://github.com/huggingface/huggingface_hub/">huggingface_hub</a></p>
</h3>
## Welcome
`hf-xet` enables `huggingface_hub` to utilize xet storage for uploading and downloading to HF Hub. Xet storage provides chunk-based deduplication, efficient storage/retrieval with local disk caching, and backwards compatibility with Git LFS. This library is not meant to be used directly, and is instead intended to be used from [huggingface_hub](https://pypi.org/project/huggingface-hub).
## Key features
♻ **chunk-based deduplication implementation**: avoid transferring and storing chunks that are shared across binary files (models, datasets, etc).
🤗 **Python bindings**: bindings for [huggingface_hub](https://github.com/huggingface/huggingface_hub/) package.
↔ **network communications**: concurrent communication to HF Hub Xet backend services (CAS).
🔖 **local disk caching**: chunk-based cache that sits alongside the existing [huggingface_hub disk cache](https://huggingface.co/docs/huggingface_hub/guides/manage-cache).
## Installation
Install the `hf_xet` package with [pip](https://pypi.org/project/hf-xet/):
```bash
pip install hf_xet
```
## Quick Start
`hf_xet` is not intended to be run independently as it is expected to be used from `huggingface_hub`, so to get started with `huggingface_hub` check out the documentation [here]("https://hf.co/docs/huggingface_hub").
## Contributions (feature requests, bugs, etc.) are encouraged & appreciated 💙💚💛💜🧡❤️
Please join us in making hf-xet better. We value everyone's contributions. Code is not the only way to help. Answering questions, helping each other, improving documentation, filing issues all help immensely. If you are interested in contributing (please do!), check out the [contribution guide](https://github.com/huggingface/xet-core/blob/main/CONTRIBUTING.md) for this repository.
Raw data
{
"_id": null,
"home_page": null,
"name": "hf-xet",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Rajat Arya <rajat@rajatarya.com>, Jared Sulzdorf <j.sulzdorf@gmail.com>, Di Xiao <di@huggingface.co>, Assaf Vayner <assaf@huggingface.co>, Hoyt Koepke <hoytak@gmail.com>",
"keywords": null,
"author": null,
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/74/31/feeddfce1748c4a233ec1aa5b7396161c07ae1aa9b7bdbc9a72c3c7dd768/hf_xet-1.1.10.tar.gz",
"platform": null,
"description": "<!---\nCopyright 2024 The HuggingFace Team. All rights reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n-->\n<p align=\"center\">\n <a href=\"https://github.com/huggingface/xet-core/blob/main/LICENSE\"><img alt=\"License\" src=\"https://img.shields.io/github/license/huggingface/xet-core.svg?color=blue\"></a>\n <a href=\"https://github.com/huggingface/xet-core/releases\"><img alt=\"GitHub release\" src=\"https://img.shields.io/github/release/huggingface/xet-core.svg\"></a>\n <a href=\"https://github.com/huggingface/xet-core/blob/main/CODE_OF_CONDUCT.md\"><img alt=\"Contributor Covenant\" src=\"https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg\"></a>\n</p>\n\n<h3 align=\"center\">\n <p>\ud83e\udd17 hf-xet - xet client tech, used in <a target=\"_blank\" href=\"https://github.com/huggingface/huggingface_hub/\">huggingface_hub</a></p>\n</h3>\n\n## Welcome\n\n`hf-xet` enables `huggingface_hub` to utilize xet storage for uploading and downloading to HF Hub. Xet storage provides chunk-based deduplication, efficient storage/retrieval with local disk caching, and backwards compatibility with Git LFS. This library is not meant to be used directly, and is instead intended to be used from [huggingface_hub](https://pypi.org/project/huggingface-hub).\n\n## Key features\n\n\u267b **chunk-based deduplication implementation**: avoid transferring and storing chunks that are shared across binary files (models, datasets, etc).\n\n\ud83e\udd17 **Python bindings**: bindings for [huggingface_hub](https://github.com/huggingface/huggingface_hub/) package.\n\n\u2194 **network communications**: concurrent communication to HF Hub Xet backend services (CAS).\n\n\ud83d\udd16 **local disk caching**: chunk-based cache that sits alongside the existing [huggingface_hub disk cache](https://huggingface.co/docs/huggingface_hub/guides/manage-cache).\n\n## Installation\n\nInstall the `hf_xet` package with [pip](https://pypi.org/project/hf-xet/):\n\n```bash\npip install hf_xet\n```\n\n## Quick Start\n\n`hf_xet` is not intended to be run independently as it is expected to be used from `huggingface_hub`, so to get started with `huggingface_hub` check out the documentation [here](\"https://hf.co/docs/huggingface_hub\").\n\n## Contributions (feature requests, bugs, etc.) are encouraged & appreciated \ud83d\udc99\ud83d\udc9a\ud83d\udc9b\ud83d\udc9c\ud83e\udde1\u2764\ufe0f\n\nPlease join us in making hf-xet better. We value everyone's contributions. Code is not the only way to help. Answering questions, helping each other, improving documentation, filing issues all help immensely. If you are interested in contributing (please do!), check out the [contribution guide](https://github.com/huggingface/xet-core/blob/main/CONTRIBUTING.md) for this repository.\n",
"bugtrack_url": null,
"license": null,
"summary": "Fast transfer of large files with the Hugging Face Hub.",
"version": "1.1.10",
"project_urls": {
"Documentation": "https://huggingface.co/docs/hub/en/storage-backends#using-xet-storage",
"Homepage": "https://github.com/huggingface/xet-core",
"Issues": "https://github.com/huggingface/xet-core/issues",
"Repository": "https://github.com/huggingface/xet-core.git"
},
"split_keywords": [],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "f7a2343e6d05de96908366bdc0081f2d8607d61200be2ac802769c4284cc65bd",
"md5": "f0617e64df21f99745663e090fd77b7f",
"sha256": "686083aca1a6669bc85c21c0563551cbcdaa5cf7876a91f3d074a030b577231d"
},
"downloads": -1,
"filename": "hf_xet-1.1.10-cp37-abi3-macosx_10_12_x86_64.whl",
"has_sig": false,
"md5_digest": "f0617e64df21f99745663e090fd77b7f",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.8",
"size": 2761466,
"upload_time": "2025-09-12T20:10:22",
"upload_time_iso_8601": "2025-09-12T20:10:22.836076Z",
"url": "https://files.pythonhosted.org/packages/f7/a2/343e6d05de96908366bdc0081f2d8607d61200be2ac802769c4284cc65bd/hf_xet-1.1.10-cp37-abi3-macosx_10_12_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "31f96215f948ac8f17566ee27af6430ea72045e0418ce757260248b483f4183b",
"md5": "4b90c7a8902d546bdaf70a01e9e538d6",
"sha256": "71081925383b66b24eedff3013f8e6bbd41215c3338be4b94ba75fd75b21513b"
},
"downloads": -1,
"filename": "hf_xet-1.1.10-cp37-abi3-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "4b90c7a8902d546bdaf70a01e9e538d6",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.8",
"size": 2623807,
"upload_time": "2025-09-12T20:10:21",
"upload_time_iso_8601": "2025-09-12T20:10:21.118638Z",
"url": "https://files.pythonhosted.org/packages/31/f9/6215f948ac8f17566ee27af6430ea72045e0418ce757260248b483f4183b/hf_xet-1.1.10-cp37-abi3-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "150786397573efefff941e100367bbda0b21496ffcdb34db7ab51912994c32a2",
"md5": "9c1fb66cb41a6b681b1e694b5da99e4f",
"sha256": "6b6bceb6361c80c1cc42b5a7b4e3efd90e64630bcf11224dcac50ef30a47e435"
},
"downloads": -1,
"filename": "hf_xet-1.1.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"has_sig": false,
"md5_digest": "9c1fb66cb41a6b681b1e694b5da99e4f",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.8",
"size": 3186960,
"upload_time": "2025-09-12T20:10:19",
"upload_time_iso_8601": "2025-09-12T20:10:19.336059Z",
"url": "https://files.pythonhosted.org/packages/15/07/86397573efefff941e100367bbda0b21496ffcdb34db7ab51912994c32a2/hf_xet-1.1.10-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "01a70b2e242b918cc30e1f91980f3c4b026ff2eedaf1e2ad96933bca164b2869",
"md5": "519452b8ca877a3113cd2d8ec60cfe41",
"sha256": "eae7c1fc8a664e54753ffc235e11427ca61f4b0477d757cc4eb9ae374b69f09c"
},
"downloads": -1,
"filename": "hf_xet-1.1.10-cp37-abi3-manylinux_2_28_aarch64.whl",
"has_sig": false,
"md5_digest": "519452b8ca877a3113cd2d8ec60cfe41",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.8",
"size": 3087167,
"upload_time": "2025-09-12T20:10:17",
"upload_time_iso_8601": "2025-09-12T20:10:17.255095Z",
"url": "https://files.pythonhosted.org/packages/01/a7/0b2e242b918cc30e1f91980f3c4b026ff2eedaf1e2ad96933bca164b2869/hf_xet-1.1.10-cp37-abi3-manylinux_2_28_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "4a253e32ab61cc7145b11eee9d745988e2f0f4fafda81b25980eebf97d8cff15",
"md5": "a63cad363bf3e59eab0713e5bfa30cc1",
"sha256": "0a0005fd08f002180f7a12d4e13b22be277725bc23ed0529f8add5c7a6309c06"
},
"downloads": -1,
"filename": "hf_xet-1.1.10-cp37-abi3-musllinux_1_2_aarch64.whl",
"has_sig": false,
"md5_digest": "a63cad363bf3e59eab0713e5bfa30cc1",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.8",
"size": 3248612,
"upload_time": "2025-09-12T20:10:24",
"upload_time_iso_8601": "2025-09-12T20:10:24.093932Z",
"url": "https://files.pythonhosted.org/packages/4a/25/3e32ab61cc7145b11eee9d745988e2f0f4fafda81b25980eebf97d8cff15/hf_xet-1.1.10-cp37-abi3-musllinux_1_2_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "2c3dab7109e607ed321afaa690f557a9ada6d6d164ec852fd6bf9979665dc3d6",
"md5": "f6dfd264cad607dbd942f72c52a81903",
"sha256": "f900481cf6e362a6c549c61ff77468bd59d6dd082f3170a36acfef2eb6a6793f"
},
"downloads": -1,
"filename": "hf_xet-1.1.10-cp37-abi3-musllinux_1_2_x86_64.whl",
"has_sig": false,
"md5_digest": "f6dfd264cad607dbd942f72c52a81903",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.8",
"size": 3353360,
"upload_time": "2025-09-12T20:10:25",
"upload_time_iso_8601": "2025-09-12T20:10:25.563787Z",
"url": "https://files.pythonhosted.org/packages/2c/3d/ab7109e607ed321afaa690f557a9ada6d6d164ec852fd6bf9979665dc3d6/hf_xet-1.1.10-cp37-abi3-musllinux_1_2_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "ee0e471f0a21db36e71a2f1752767ad77e92d8cde24e974e03d662931b1305ec",
"md5": "e7c1b03cd49d6b676d3ce92a75cc9af1",
"sha256": "5f54b19cc347c13235ae7ee98b330c26dd65ef1df47e5316ffb1e87713ca7045"
},
"downloads": -1,
"filename": "hf_xet-1.1.10-cp37-abi3-win_amd64.whl",
"has_sig": false,
"md5_digest": "e7c1b03cd49d6b676d3ce92a75cc9af1",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.8",
"size": 2804691,
"upload_time": "2025-09-12T20:10:28",
"upload_time_iso_8601": "2025-09-12T20:10:28.433001Z",
"url": "https://files.pythonhosted.org/packages/ee/0e/471f0a21db36e71a2f1752767ad77e92d8cde24e974e03d662931b1305ec/hf_xet-1.1.10-cp37-abi3-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "7431feeddfce1748c4a233ec1aa5b7396161c07ae1aa9b7bdbc9a72c3c7dd768",
"md5": "8e6e04dc986cd59862fbc28aca66fad8",
"sha256": "408aef343800a2102374a883f283ff29068055c111f003ff840733d3b715bb97"
},
"downloads": -1,
"filename": "hf_xet-1.1.10.tar.gz",
"has_sig": false,
"md5_digest": "8e6e04dc986cd59862fbc28aca66fad8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 487910,
"upload_time": "2025-09-12T20:10:27",
"upload_time_iso_8601": "2025-09-12T20:10:27.120044Z",
"url": "https://files.pythonhosted.org/packages/74/31/feeddfce1748c4a233ec1aa5b7396161c07ae1aa9b7bdbc9a72c3c7dd768/hf_xet-1.1.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-12 20:10:27",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "huggingface",
"github_project": "xet-core",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "hf-xet"
}
