# hfutils
[](https://pypi.org/project/hfutils/)
[](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Code+Test%22)
[](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Package+Release%22)
[](https://codecov.io/gh/deepghs/hfutils)
[](https://discord.gg/TwdHJ42N72)
[](https://github.com/deepghs/hfutils/stargazers)
[](https://github.com/deepghs/hfutils/network)
[](https://github.com/deepghs/hfutils/issues)
[](https://github.com/deepghs/hfutils/pulls)
[](https://github.com/deepghs/hfutils/graphs/contributors)
[](https://github.com/deepghs/hfutils/blob/master/LICENSE)
Useful utilities for huggingface
## Quick Start
To get started with `hfutils`, you can install it using PyPI:
```shell
pip install hfutils
```
Alternatively, you can install it from the source code:
```shell
git clone https://github.com/deepghs/hfutils.git
cd hfutils
pip install .
```
Verify the installation by checking the version:
```shell
hfutils -v
```
If Python is not available in your local environment, we recommend downloading the pre-compiled executable version from
the [releases](https://github.com/deepghs/hfutils/releases).
## Upload Data
Upload data to repositories using the following commands:
```shell
# Upload a single file to the repository
hfutils upload -r your/repository -i /your/local/file -f file/in/your/repo
# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip
# Upload files in a directory as a directory tree to the repository
hfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo
```
You can achieve the same using the Python API:
```python
from hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory
# Upload a single file to the repository
upload_file_to_file(
local_file='/your/local/file',
repo_id='your/repository',
file_in_repo='file/in/your/repo'
)
# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
upload_directory_as_archive(
local_directory='/your/local/directory',
repo_id='your/repository',
archive_in_repo='archive/file/in/your/repo.zip'
)
# Upload files in a directory as a directory tree to the repository
upload_directory_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
path_in_repo='dir/in/your/repo'
)
```
Explore additional options for uploading:
```shell
hfutils upload -h
```
## Download Data
Download data from repositories using the following commands:
```shell
# Download a single file from the repository
hfutils download -r your/repository -o /your/local/file -f file/in/your/repo
# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip
# Download files from the repository as a directory tree
hfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo
```
Use the Python API for the same functionality:
```python
from hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory
# Download a single file from the repository
download_file_to_file(
local_file='/your/local/file',
repo_id='your/repository',
file_in_repo='file/in/your/repo'
)
# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
download_archive_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
file_in_repo='archive/file/in/your/repo.zip',
)
# Download files from the repository as a directory tree
download_directory_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
dir_in_repo='dir/in/your/repo'
)
```
Explore additional options for downloading:
```shell
hfutils download -h
```
## List Files in Repository
List files in repositories
```shell
hfutils ls -r your/repository -o /your/local/file -d subdir/in/repo
```
## Supported Formats
By default, we support the `zip` and `tar` formats, including `.zip`, `.tar`, `.tar.gz`, `.tar.bz2`, and `.tar.xz`.
If you require support for `rar` and `7z` files, install the extra dependencies using the following command:
```shell
pip install hfutils[rar,7z]
```
**NOTE:** Creating RAR archive files is not supported. We use the [rarfile](https://github.com/markokr/rarfile) library,
which lacks the functionality for creating RAR files.
## How to Access Private Repositories
Simply configure the `HF_TOKEN` environment variable by using your HuggingFace access token.
Note that write permissions are required if you plan to upload any content.
## How to Use Hf-Transfer for Acceleration
If you are using the PyPI CLI, you need to install `hfutils` with the following command:
```shell
pip install hfutils[transfer]
```
If you are using a precompiled executable file, the transfer module is integrated inside; simply use it.
Enable Hf-Transfer acceleration by setting the environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`.
## How to Change the Temporary Directory
The temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk
space.
If your disk, especially the C drive on Windows, does not have sufficient space, simply set the `TMPDIR` to designate
another directory as the temporary directory.
Raw data
{
"_id": null,
"home_page": "https://github.com/deepghs/hfutils",
"name": "hfutils",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Utilities of images.",
"author": "narugo1992",
"author_email": "narugo992@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/a9/ae/6d1068420a1bad1c9150faf5f0ecb4faa2ba0fe8176851201bced74256f4/hfutils-0.7.0.tar.gz",
"platform": null,
"description": "# hfutils\n\n[](https://pypi.org/project/hfutils/)\n\n\n\n\n[](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Code+Test%22)\n[](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Package+Release%22)\n[](https://codecov.io/gh/deepghs/hfutils)\n\n[](https://discord.gg/TwdHJ42N72)\n\n[](https://github.com/deepghs/hfutils/stargazers)\n[](https://github.com/deepghs/hfutils/network)\n\n[](https://github.com/deepghs/hfutils/issues)\n[](https://github.com/deepghs/hfutils/pulls)\n[](https://github.com/deepghs/hfutils/graphs/contributors)\n[](https://github.com/deepghs/hfutils/blob/master/LICENSE)\n\nUseful utilities for huggingface\n\n## Quick Start\n\nTo get started with `hfutils`, you can install it using PyPI:\n\n```shell\npip install hfutils\n\n```\n\nAlternatively, you can install it from the source code:\n\n```shell\ngit clone https://github.com/deepghs/hfutils.git\ncd hfutils\npip install .\n\n```\n\nVerify the installation by checking the version:\n\n```shell\nhfutils -v\n\n```\n\nIf Python is not available in your local environment, we recommend downloading the pre-compiled executable version from\nthe [releases](https://github.com/deepghs/hfutils/releases).\n\n## Upload Data\n\nUpload data to repositories using the following commands:\n\n```shell\n# Upload a single file to the repository\nhfutils upload -r your/repository -i /your/local/file -f file/in/your/repo\n\n# Upload files in a directory as an archive file to the repository\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\nhfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip\n\n# Upload files in a directory as a directory tree to the repository\nhfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo\n\n```\n\nYou can achieve the same using the Python API:\n\n```python\nfrom hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory\n\n# Upload a single file to the repository\nupload_file_to_file(\n local_file='/your/local/file',\n repo_id='your/repository',\n file_in_repo='file/in/your/repo'\n)\n\n# Upload files in a directory as an archive file to the repository\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\nupload_directory_as_archive(\n local_directory='/your/local/directory',\n repo_id='your/repository',\n archive_in_repo='archive/file/in/your/repo.zip'\n)\n\n# Upload files in a directory as a directory tree to the repository\nupload_directory_as_directory(\n local_directory='/your/local/directory',\n repo_id='your/repository',\n path_in_repo='dir/in/your/repo'\n)\n```\n\nExplore additional options for uploading:\n\n```shell\nhfutils upload -h\n\n```\n\n## Download Data\n\nDownload data from repositories using the following commands:\n\n```shell\n# Download a single file from the repository\nhfutils download -r your/repository -o /your/local/file -f file/in/your/repo\n\n# Download an archive file from the repository and extract it to the given directory\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\nhfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip\n\n# Download files from the repository as a directory tree\nhfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo\n\n```\n\nUse the Python API for the same functionality:\n\n```python\nfrom hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory\n\n# Download a single file from the repository\ndownload_file_to_file(\n local_file='/your/local/file',\n repo_id='your/repository',\n file_in_repo='file/in/your/repo'\n)\n\n# Download an archive file from the repository and extract it to the given directory\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\ndownload_archive_as_directory(\n local_directory='/your/local/directory',\n repo_id='your/repository',\n file_in_repo='archive/file/in/your/repo.zip',\n)\n\n# Download files from the repository as a directory tree\ndownload_directory_as_directory(\n local_directory='/your/local/directory',\n repo_id='your/repository',\n dir_in_repo='dir/in/your/repo'\n)\n```\n\nExplore additional options for downloading:\n\n```shell\nhfutils download -h\n\n```\n\n## List Files in Repository\n\nList files in repositories\n\n```shell\nhfutils ls -r your/repository -o /your/local/file -d subdir/in/repo\n```\n\n## Supported Formats\n\nBy default, we support the `zip` and `tar` formats, including `.zip`, `.tar`, `.tar.gz`, `.tar.bz2`, and `.tar.xz`.\n\nIf you require support for `rar` and `7z` files, install the extra dependencies using the following command:\n\n```shell\npip install hfutils[rar,7z]\n```\n\n**NOTE:** Creating RAR archive files is not supported. We use the [rarfile](https://github.com/markokr/rarfile) library,\nwhich lacks the functionality for creating RAR files.\n\n## How to Access Private Repositories\n\nSimply configure the `HF_TOKEN` environment variable by using your HuggingFace access token.\nNote that write permissions are required if you plan to upload any content.\n\n## How to Use Hf-Transfer for Acceleration\n\nIf you are using the PyPI CLI, you need to install `hfutils` with the following command:\n\n```shell\npip install hfutils[transfer]\n\n```\n\nIf you are using a precompiled executable file, the transfer module is integrated inside; simply use it.\n\nEnable Hf-Transfer acceleration by setting the environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`.\n\n## How to Change the Temporary Directory\n\nThe temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk\nspace.\n\nIf your disk, especially the C drive on Windows, does not have sufficient space, simply set the `TMPDIR` to designate\nanother directory as the temporary directory.\n",
"bugtrack_url": null,
"license": "Apache License, Version 2.0",
"summary": "Useful utilities for huggingface",
"version": "0.7.0",
"project_urls": {
"Homepage": "https://github.com/deepghs/hfutils"
},
"split_keywords": [
"utilities",
"of",
"images."
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5174ede20ce61ab38b10d2921c1a1ac4f55bbc34f30b08a78d0cfa6ea0376157",
"md5": "5238751c0812fe4c6d8d5db0aa9e0e12",
"sha256": "80a30990e97b137d08d80930dcb0cd7a827b02978a904f23566ae69dd44ac81b"
},
"downloads": -1,
"filename": "hfutils-0.7.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5238751c0812fe4c6d8d5db0aa9e0e12",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 96912,
"upload_time": "2024-12-11T12:02:50",
"upload_time_iso_8601": "2024-12-11T12:02:50.386497Z",
"url": "https://files.pythonhosted.org/packages/51/74/ede20ce61ab38b10d2921c1a1ac4f55bbc34f30b08a78d0cfa6ea0376157/hfutils-0.7.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a9ae6d1068420a1bad1c9150faf5f0ecb4faa2ba0fe8176851201bced74256f4",
"md5": "a227ac9e1a3696655074b92e6970cff0",
"sha256": "b413b011710462e199dace6464462073007d39eae2bd0e6a384fc75ebc2d9bed"
},
"downloads": -1,
"filename": "hfutils-0.7.0.tar.gz",
"has_sig": false,
"md5_digest": "a227ac9e1a3696655074b92e6970cff0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 71250,
"upload_time": "2024-12-11T12:02:52",
"upload_time_iso_8601": "2024-12-11T12:02:52.958138Z",
"url": "https://files.pythonhosted.org/packages/a9/ae/6d1068420a1bad1c9150faf5f0ecb4faa2ba0fe8176851201bced74256f4/hfutils-0.7.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-11 12:02:52",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "deepghs",
"github_project": "hfutils",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "hbutils",
"specs": [
[
">=",
"0.10.0"
]
]
},
{
"name": "huggingface_hub",
"specs": [
[
"!=",
"0.24.1"
],
[
"!=",
"0.24.0"
],
[
">=",
"0.23"
]
]
},
{
"name": "tqdm",
"specs": []
},
{
"name": "requests",
"specs": []
},
{
"name": "click",
"specs": [
[
">=",
"7"
]
]
},
{
"name": "tzlocal",
"specs": []
},
{
"name": "natsort",
"specs": []
},
{
"name": "urlobject",
"specs": []
},
{
"name": "fsspec",
"specs": [
[
">=",
"2024"
]
]
},
{
"name": "random_user_agent",
"specs": []
}
],
"lcname": "hfutils"
}
JSON
