## HF-Torrent
This repository contains an experimental torrent-style download for model weights in Hugging Face.
The core implementation is `main_hub.py`, which downloads files from Hugging Face and converts them into seedable torrents.
## How to use
`pip install hf-torrent`
Before launching downloads, make sure you have [aria2c](https://aria2.github.io/) installed and
<details>
<summary> run the following command in a separate terminal </summary>
```bash
aria2c --enable-rpc --rpc-listen-all \
--max-upload-limit=4M \
--optimize-concurrent-downloads=true \
--bt-detach-seed-only=true \
--seed-ratio=10.0
```
</details>
Then you can start using HF-Torrent with the following command:
```bash
hf-torrent <your target repo>
```
[](https://asciinema.org/a/4ppEU7HDGzUqwReSIosXOkCs9)
Or if you have your preferred torrent clients, you can pull the torrent file only
```bash
hf-torrent --get-torrent <your target repo>
```
## How to contribute
### Installation
To install, run the following command:
`pip install -r requirements.txt`
### Seeding
It is easy to contribute to seeding! First, change `--seed-ratio=0` (from 10 to 0) when launching the `aria2c` RPC. Setting `0` means the daemon will seed infinitely.
Next, change the upload bandwidth limit to a value that suits you, for example: `--max-upload-limit=20M` (4M by default).
After making these changes, launching seeding is as simple as downloading models. You can start seeding by running the following script and help distribute your favorite models!
```bash
python download.py <your target repo>
```
### Create
To create a popular torrent, run the following command:
`python generate_popular_torrents.py -r efederici/retnet-120m`
This script will (1) download model files from Hugging Face, (2) iterate through the folder, (3) generate torrents for each file, and then put them under `hf-torrent-store/{repo}`.
We are looking for a wanted model list. Please open an issue if you have any favorite models to share.
Raw data
{
"_id": null,
"home_page": "https://github.com/Lyken17/hf-torrent/",
"name": "hf-torrent",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Ligeng Zhu",
"author_email": "lykensyu+github@gmail.com",
"download_url": "",
"platform": null,
"description": "## HF-Torrent\n\nThis repository contains an experimental torrent-style download for model weights in Hugging Face.\n\nThe core implementation is `main_hub.py`, which downloads files from Hugging Face and converts them into seedable torrents.\n\n## How to use\n\n`pip install hf-torrent`\n\nBefore launching downloads, make sure you have [aria2c](https://aria2.github.io/) installed and \n\n<details>\n<summary> run the following command in a separate terminal </summary>\n\n```bash\naria2c --enable-rpc --rpc-listen-all \\\n --max-upload-limit=4M \\\n --optimize-concurrent-downloads=true \\\n --bt-detach-seed-only=true \\\n --seed-ratio=10.0 \n```\n</details>\n\n\nThen you can start using HF-Torrent with the following command:\n\n```bash\nhf-torrent <your target repo>\n```\n\n[](https://asciinema.org/a/4ppEU7HDGzUqwReSIosXOkCs9)\n\n\nOr if you have your preferred torrent clients, you can pull the torrent file only\n\n```bash\nhf-torrent --get-torrent <your target repo>\n```\n\n\n## How to contribute\n\n### Installation\n\nTo install, run the following command:\n\n`pip install -r requirements.txt`\n\n\n### Seeding\n\nIt is easy to contribute to seeding! First, change `--seed-ratio=0` (from 10 to 0) when launching the `aria2c` RPC. Setting `0` means the daemon will seed infinitely.\nNext, change the upload bandwidth limit to a value that suits you, for example: `--max-upload-limit=20M` (4M by default).\n\nAfter making these changes, launching seeding is as simple as downloading models. You can start seeding by running the following script and help distribute your favorite models!\n\n```bash\npython download.py <your target repo>\n\n```\n\n### Create\n\nTo create a popular torrent, run the following command:\n\n`python generate_popular_torrents.py -r efederici/retnet-120m`\n\nThis script will (1) download model files from Hugging Face, (2) iterate through the folder, (3) generate torrents for each file, and then put them under `hf-torrent-store/{repo}`.\n\nWe are looking for a wanted model list. Please open an issue if you have any favorite models to share.\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A tool to download HuggingFace models via BitTorrent.",
"version": "0.0.1.post2401021918",
"project_urls": {
"Homepage": "https://github.com/Lyken17/hf-torrent/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "77e41ed9495a015b7a4474dc6651f570eb38745018bcb0f2be96b8ba00fab2f2",
"md5": "e4881009ca47873b2cf1d218bdf9be9b",
"sha256": "fd3d20d4f3d0bddeb3763cbff06f2f3f6c4e881b1ad45ece0162e3f587a77de3"
},
"downloads": -1,
"filename": "hf_torrent-0.0.1.post2401021918-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e4881009ca47873b2cf1d218bdf9be9b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 6552,
"upload_time": "2024-01-02T11:18:48",
"upload_time_iso_8601": "2024-01-02T11:18:48.714593Z",
"url": "https://files.pythonhosted.org/packages/77/e4/1ed9495a015b7a4474dc6651f570eb38745018bcb0f2be96b8ba00fab2f2/hf_torrent-0.0.1.post2401021918-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-02 11:18:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Lyken17",
"github_project": "hf-torrent",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "hf-torrent"
}
JSON
