# hfutils
[](https://pypi.org/project/hfutils/)
[](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Code+Test%22)
[](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Package+Release%22)
[](https://codecov.io/gh/deepghs/hfutils)
[](https://discord.gg/TwdHJ42N72)
[](https://github.com/deepghs/hfutils/stargazers)
[](https://github.com/deepghs/hfutils/network)
[](https://github.com/deepghs/hfutils/issues)
[](https://github.com/deepghs/hfutils/pulls)
[](https://github.com/deepghs/hfutils/graphs/contributors)
[](https://github.com/deepghs/hfutils/blob/master/LICENSE)
Useful utilities for huggingface
## Quick Start
To get started with `hfutils`, you can install it using PyPI:
```shell
pip install hfutils
```
Alternatively, you can install it from the source code:
```shell
git clone https://github.com/deepghs/hfutils.git
cd hfutils
pip install .
```
Verify the installation by checking the version:
```shell
hfutils -v
```
If Python is not available in your local environment, we recommend downloading the pre-compiled executable version from
the [releases](https://github.com/deepghs/hfutils/releases).
## Upload Data
Upload data to repositories using the following commands:
```shell
# Upload a single file to the repository
hfutils upload -r your/repository -i /your/local/file -f file/in/your/repo
# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip
# Upload files in a directory as a directory tree to the repository
hfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo
```
You can achieve the same using the Python API:
```python
from hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory
# Upload a single file to the repository
upload_file_to_file(
local_file='/your/local/file',
repo_id='your/repository',
file_in_repo='file/in/your/repo'
)
# Upload files in a directory as an archive file to the repository
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
upload_directory_as_archive(
local_directory='/your/local/directory',
repo_id='your/repository',
archive_in_repo='archive/file/in/your/repo.zip'
)
# Upload files in a directory as a directory tree to the repository
upload_directory_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
path_in_repo='dir/in/your/repo'
)
```
Explore additional options for uploading:
```shell
hfutils upload -h
```
## Download Data
Download data from repositories using the following commands:
```shell
# Download a single file from the repository
hfutils download -r your/repository -o /your/local/file -f file/in/your/repo
# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
hfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip
# Download files from the repository as a directory tree
hfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo
```
Use the Python API for the same functionality:
```python
from hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory
# Download a single file from the repository
download_file_to_file(
local_file='/your/local/file',
repo_id='your/repository',
file_in_repo='file/in/your/repo'
)
# Download an archive file from the repository and extract it to the given directory
# More formats of archive files are supported
# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html
download_archive_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
file_in_repo='archive/file/in/your/repo.zip',
)
# Download files from the repository as a directory tree
download_directory_as_directory(
local_directory='/your/local/directory',
repo_id='your/repository',
dir_in_repo='dir/in/your/repo'
)
```
Explore additional options for downloading:
```shell
hfutils download -h
```
## List Files in Repository
List files in repositories
```shell
hfutils ls -r your/repository -o /your/local/file -d subdir/in/repo
```
## Supported Formats
By default, we support the `zip` and `tar` formats, including `.zip`, `.tar`, `.tar.gz`, `.tar.bz2`, and `.tar.xz`.
If you require support for `rar` and `7z` files, install the extra dependencies using the following command:
```shell
pip install hfutils[rar,7z]
```
**NOTE:** Creating RAR archive files is not supported. We use the [rarfile](https://github.com/markokr/rarfile) library,
which lacks the functionality for creating RAR files.
## How to Access Private Repositories
Simply configure the `HF_TOKEN` environment variable by using your HuggingFace access token.
Note that write permissions are required if you plan to upload any content.
## How to Use Hf-Transfer for Acceleration
If you are using the PyPI CLI, you need to install `hfutils` with the following command:
```shell
pip install hfutils[transfer]
```
If you are using a precompiled executable file, the transfer module is integrated inside; simply use it.
Enable Hf-Transfer acceleration by setting the environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`.
## How to Change the Temporary Directory
The temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk
space.
If your disk, especially the C drive on Windows, does not have sufficient space, simply set the `TMPDIR` to designate
another directory as the temporary directory.
Raw data
{
"_id": null,
"home_page": "https://github.com/deepghs/hfutils",
"name": "hfutils",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Utilities of images.",
"author": "narugo1992",
"author_email": "narugo992@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/14/38/852b1806297475f3b586e7a5675b54c18ca59c731aca25c3abaadce3b8a3/hfutils-0.5.0.tar.gz",
"platform": null,
"description": "# hfutils\n\n[](https://pypi.org/project/hfutils/)\n\n\n\n\n[](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Code+Test%22)\n[](https://github.com/deepghs/hfutils/actions?query=workflow%3A%22Package+Release%22)\n[](https://codecov.io/gh/deepghs/hfutils)\n\n[](https://discord.gg/TwdHJ42N72)\n\n[](https://github.com/deepghs/hfutils/stargazers)\n[](https://github.com/deepghs/hfutils/network)\n\n[](https://github.com/deepghs/hfutils/issues)\n[](https://github.com/deepghs/hfutils/pulls)\n[](https://github.com/deepghs/hfutils/graphs/contributors)\n[](https://github.com/deepghs/hfutils/blob/master/LICENSE)\n\nUseful utilities for huggingface\n\n## Quick Start\n\nTo get started with `hfutils`, you can install it using PyPI:\n\n```shell\npip install hfutils\n\n```\n\nAlternatively, you can install it from the source code:\n\n```shell\ngit clone https://github.com/deepghs/hfutils.git\ncd hfutils\npip install .\n\n```\n\nVerify the installation by checking the version:\n\n```shell\nhfutils -v\n\n```\n\nIf Python is not available in your local environment, we recommend downloading the pre-compiled executable version from\nthe [releases](https://github.com/deepghs/hfutils/releases).\n\n## Upload Data\n\nUpload data to repositories using the following commands:\n\n```shell\n# Upload a single file to the repository\nhfutils upload -r your/repository -i /your/local/file -f file/in/your/repo\n\n# Upload files in a directory as an archive file to the repository\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\nhfutils upload -r your/repository -i /your/local/directory -a archive/file/in/your/repo.zip\n\n# Upload files in a directory as a directory tree to the repository\nhfutils upload -r your/repository -i /your/local/directory -d dir/in/your/repo\n\n```\n\nYou can achieve the same using the Python API:\n\n```python\nfrom hfutils.operate import upload_file_to_file, upload_directory_as_archive, upload_directory_as_directory\n\n# Upload a single file to the repository\nupload_file_to_file(\n local_file='/your/local/file',\n repo_id='your/repository',\n file_in_repo='file/in/your/repo'\n)\n\n# Upload files in a directory as an archive file to the repository\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\nupload_directory_as_archive(\n local_directory='/your/local/directory',\n repo_id='your/repository',\n archive_in_repo='archive/file/in/your/repo.zip'\n)\n\n# Upload files in a directory as a directory tree to the repository\nupload_directory_as_directory(\n local_directory='/your/local/directory',\n repo_id='your/repository',\n path_in_repo='dir/in/your/repo'\n)\n```\n\nExplore additional options for uploading:\n\n```shell\nhfutils upload -h\n\n```\n\n## Download Data\n\nDownload data from repositories using the following commands:\n\n```shell\n# Download a single file from the repository\nhfutils download -r your/repository -o /your/local/file -f file/in/your/repo\n\n# Download an archive file from the repository and extract it to the given directory\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\nhfutils download -r your/repository -o /your/local/directory -a archive/file/in/your/repo.zip\n\n# Download files from the repository as a directory tree\nhfutils download -r your/repository -o /your/local/directory -d dir/in/your/repo\n\n```\n\nUse the Python API for the same functionality:\n\n```python\nfrom hfutils.operate import download_file_to_file, download_archive_as_directory, download_directory_as_directory\n\n# Download a single file from the repository\ndownload_file_to_file(\n local_file='/your/local/file',\n repo_id='your/repository',\n file_in_repo='file/in/your/repo'\n)\n\n# Download an archive file from the repository and extract it to the given directory\n# More formats of archive files are supported\n# See: https://deepghs.github.io/hfutils/main/api_doc/archive/index.html\ndownload_archive_as_directory(\n local_directory='/your/local/directory',\n repo_id='your/repository',\n file_in_repo='archive/file/in/your/repo.zip',\n)\n\n# Download files from the repository as a directory tree\ndownload_directory_as_directory(\n local_directory='/your/local/directory',\n repo_id='your/repository',\n dir_in_repo='dir/in/your/repo'\n)\n```\n\nExplore additional options for downloading:\n\n```shell\nhfutils download -h\n\n```\n\n## List Files in Repository\n\nList files in repositories\n\n```shell\nhfutils ls -r your/repository -o /your/local/file -d subdir/in/repo\n```\n\n## Supported Formats\n\nBy default, we support the `zip` and `tar` formats, including `.zip`, `.tar`, `.tar.gz`, `.tar.bz2`, and `.tar.xz`.\n\nIf you require support for `rar` and `7z` files, install the extra dependencies using the following command:\n\n```shell\npip install hfutils[rar,7z]\n```\n\n**NOTE:** Creating RAR archive files is not supported. We use the [rarfile](https://github.com/markokr/rarfile) library,\nwhich lacks the functionality for creating RAR files.\n\n## How to Access Private Repositories\n\nSimply configure the `HF_TOKEN` environment variable by using your HuggingFace access token.\nNote that write permissions are required if you plan to upload any content.\n\n## How to Use Hf-Transfer for Acceleration\n\nIf you are using the PyPI CLI, you need to install `hfutils` with the following command:\n\n```shell\npip install hfutils[transfer]\n\n```\n\nIf you are using a precompiled executable file, the transfer module is integrated inside; simply use it.\n\nEnable Hf-Transfer acceleration by setting the environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`.\n\n## How to Change the Temporary Directory\n\nThe temporary directory is utilized for storing partially downloaded files, consuming a considerable amount of disk\nspace.\n\nIf your disk, especially the C drive on Windows, does not have sufficient space, simply set the `TMPDIR` to designate\nanother directory as the temporary directory.\n",
"bugtrack_url": null,
"license": "Apache License, Version 2.0",
"summary": "Useful utilities for huggingface",
"version": "0.5.0",
"project_urls": {
"Homepage": "https://github.com/deepghs/hfutils"
},
"split_keywords": [
"utilities",
"of",
"images."
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7357dde7dcbca2a430906e3b45467cd6da5862727d85b21436014c9871fcb353",
"md5": "3eb30e9f553b48fd9cc81271d7c4140a",
"sha256": "abc1356435e3b398a5b16fe4314304b92129043a257d9cf1094c339d31ec1b9d"
},
"downloads": -1,
"filename": "hfutils-0.5.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3eb30e9f553b48fd9cc81271d7c4140a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 84842,
"upload_time": "2024-11-05T13:07:32",
"upload_time_iso_8601": "2024-11-05T13:07:32.707234Z",
"url": "https://files.pythonhosted.org/packages/73/57/dde7dcbca2a430906e3b45467cd6da5862727d85b21436014c9871fcb353/hfutils-0.5.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1438852b1806297475f3b586e7a5675b54c18ca59c731aca25c3abaadce3b8a3",
"md5": "dba5dd5509004fdc876de7f5657009cf",
"sha256": "d0e00e3168ece8f09fe1ccf232dce69b7a33df3384fa06b2492ba0054633bcd1"
},
"downloads": -1,
"filename": "hfutils-0.5.0.tar.gz",
"has_sig": false,
"md5_digest": "dba5dd5509004fdc876de7f5657009cf",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 61819,
"upload_time": "2024-11-05T13:07:34",
"upload_time_iso_8601": "2024-11-05T13:07:34.076534Z",
"url": "https://files.pythonhosted.org/packages/14/38/852b1806297475f3b586e7a5675b54c18ca59c731aca25c3abaadce3b8a3/hfutils-0.5.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-05 13:07:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "deepghs",
"github_project": "hfutils",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "hfutils"
}
JSON
