# SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning)
<p align="center">
<br>
<img src="asset/banner.png"/>
<br>
<p>
<p align="center">
<a href="https://modelscope.cn/home">ModelScope Community Website</a>
<br>
<a href="README_CN.md">中文</a>   ｜   English  
</p>
<p align="center">
<img src="https://img.shields.io/badge/python-%E2%89%A53.8-5be.svg">
<img src="https://img.shields.io/badge/pytorch-%E2%89%A52.0-orange.svg">
<a href="https://github.com/modelscope/modelscope/"><img src="https://img.shields.io/badge/modelscope-%E2%89%A51.19-5D91D4.svg"></a>
<a href="https://pypi.org/project/ms-swift/"><img src="https://badge.fury.io/py/ms-swift.svg"></a>
<a href="https://github.com/modelscope/swift/blob/main/LICENSE"><img src="https://img.shields.io/github/license/modelscope/swift"></a>
<a href="https://pepy.tech/project/ms-swift"><img src="https://pepy.tech/badge/ms-swift"></a>
<a href="https://github.com/modelscope/swift/pulls"><img src="https://img.shields.io/badge/PR-welcome-55EB99.svg"></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/6427" target="_blank"><img src="https://trendshift.io/api/badge/repositories/6427" alt="modelscope%2Fswift | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<p align="center">
<a href="https://arxiv.org/abs/2408.05517">Paper</a>   ｜ <a href="https://swift.readthedocs.io/en/latest/">English Documentation</a>   ｜   <a href="https://swift.readthedocs.io/zh-cn/latest/">中文文档</a>  
</p>
<p align="center">
<a href="https://swift2x-en.readthedocs.io/en/latest/">Swift2.x En Doc</a>   ｜   <a href="https://swift2x.readthedocs.io/zh-cn/latest/">Swift2.x中文文档</a>  
</p>
## 📖 Table of Contents
- [Groups](#-Groups)
- [Introduction](#-introduction)
- [News](#-news)
- [Installation](#%EF%B8%8F-installation)
- [Quick Start](#-quick-Start)
- [Usage](#-Usage)
- [License](#-License)
- [Citation](#-citation)
## ☎ Groups
You can contact us and communicate with us by adding our group:
[Discord Group](https://discord.com/invite/D27yfEFVz5) | 微信群
:-------------------------:|:-------------------------:
<img src="asset/discord_qr.jpg" width="200" height="200"> | <img src="asset/wechat.png" width="200" height="200">
## 📝 Introduction
🍲 ms-swift is an official framework provided by the ModelScope community for fine-tuning and deploying large language models and multi-modal large models. It currently supports the training (pre-training, fine-tuning, human alignment), inference, evaluation, quantization, and deployment of over 400 large models and 100+ multi-modal large models. These large language models (LLMs) include models such as Qwen2.5, Llama3.2, GLM4, Internlm2.5, Yi1.5, Mistral, DeepSeek, Baichuan2, Gemma2, and TeleChat2. The multi-modal LLMs include models such as Qwen2-VL, Qwen2-Audio, Llama3.2-Vision, Llava, InternVL2.5, MiniCPM-V-2.6, GLM4v, Xcomposer2.5, Yi-VL, DeepSeek-VL2, Phi3.5-Vision, and GOT-OCR2.
🍔 In addition, ms-swift gathers the latest training technologies, including LoRA, QLoRA, Llama-Pro, LongLoRA, GaLore, Q-GaLore, LoRA+, LISA, DoRA, FourierFt, ReFT, UnSloth, and Liger. ms-swift supports accelerating the inference, evaluation, and deployment modules using vLLM and LMDeploy. To help researchers and developers fine-tune and apply large models more easily, ms-swift also provides a Gradio-based Web-UI interface and a wealth of best practices.
**Why choose ms-swift?**
- 🍎 **Model Types**: Supports 400+ large language models and **100+ multi-modal large models** and all-to-all models, **providing a comprehensive solution from training to deployment**.
- **Dataset Types**: Comes with 150+ pre-training, fine-tuning, human alignment, multi-modal datasets, and supports custom datasets.
- **Hardware Support**: Compatible with CPU, RTX series, T4/V100, A10/A100/H100, Ascend NPU, etc.
- 🍊 **Lightweight Training**: Supports lightweight fine-tuning methods like LoRA, QLoRA, DoRA, LoRA+, ReFT, RS-LoRA, LLaMAPro, Adapter, GaLore, Q-Galore, LISA, UnSloth, Liger-Kernel.
- **Distributed Training**: Supports distributed data parallel (DDP), device_map simple model parallelism, DeepSpeed ZeRO2/ZeRO3, FSDP, and other distributed training techniques.
- **Quantization Training**: Supports training quantized models like BNB, AWQ, GPTQ, AQLM, HQQ, EETQ.
- **RLHF Training**: Supports human alignment training methods such as DPO, CPO, SimPO, ORPO, KTO, RM for both pure text and multi-modal large models.
- 🍓 **Multi-Modal Training**: Supports training on different modalities like images, videos, and audio, for tasks like VQA, captioning, OCR, and grounding.
- **Interface Training**: Provides capabilities for training, inference, evaluation, quantization through an interface, completing the whole large model pipeline.
- **Plugin and Extension**: Supports custom model and dataset extensions, as well as customization of components like loss, metric, trainer, loss-scale, callback, optimizer.
- 🍉 **Toolbox Capabilities**: Offers not only training support for large models and multi-modal large models but also covers the entire process of inference, evaluation, quantization, and deployment.
- **Inference Acceleration**: Supports inference acceleration engines like PyTorch, vLLM, LmDeploy, and provides OpenAI API for accelerating inference, deployment, and evaluation modules.
- **Model Evaluation**: Uses EvalScope as the evaluation backend and supports evaluation on 100+ datasets for both pure text and multi-modal models.
- **Model Quantization**: Supports AWQ, GPTQ, and BNB quantized exports, with models that can use vLLM/LmDeploy for inference acceleration and continue training.
## 🎉 News
- 🎁 2024.12.04: **SWIFT3.0** major version update. Please check the [Release Notes and Changes](https://swift.readthedocs.io/en/latest/Instruction/ReleaseNote3.0.html).
- 🎉 2024.08.12: The SWIFT paper has been published on arXiv, and you can read it [here](https://arxiv.org/abs/2408.05517).
- 🔥 2024.08.05: Support for using [evalscope](https://github.com/modelscope/evalscope/) as a backend for evaluating large models and multimodal models.
- 🔥 2024.07.29: Support for using [vllm](https://github.com/vllm-project/vllm) and [lmdeploy](https://github.com/InternLM/lmdeploy) to accelerate inference for large models and multimodal models. When performing infer/deploy/eval, you can specify `--infer_backend vllm/lmdeploy`.
- 🔥 2024.07.24: Support for human preference alignment training for multimodal large models, including DPO/ORPO/SimPO/CPO/KTO/RM.
- 🔥 2024.02.01: Support for Agent training! The training algorithm is derived from [this paper](https://arxiv.org/pdf/2309.00986.pdf).
## 🛠️ Installation
To install using pip:
```shell
pip install ms-swift -U
```
To install from source:
```shell
# pip install git+https://github.com/modelscope/ms-swift.git
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e .
```
## 🚀 Quick Start
10 minutes of self-cognition fine-tuning of Qwen2.5-7B-Instruct on a single 3090 GPU:
### Command Line Interface
```shell
# 22GB
CUDA_VISIBLE_DEVICES=0 \
swift sft \
--model Qwen/Qwen2.5-7B-Instruct \
--train_type lora \
--dataset AI-ModelScope/alpaca-gpt4-data-zh#500 \
AI-ModelScope/alpaca-gpt4-data-en#500 \
swift/self-cognition#500 \
--torch_dtype bfloat16 \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--learning_rate 1e-4 \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--gradient_accumulation_steps 16 \
--eval_steps 50 \
--save_steps 50 \
--save_total_limit 2 \
--logging_steps 5 \
--max_length 2048 \
--output_dir output \
--system 'You are a helpful assistant.' \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--model_author swift \
--model_name swift-robot
```
After training is complete, use the following command to perform inference with the trained weights. The `--adapters` option should be replaced with the last checkpoint folder generated from the training. Since the adapters folder contains the parameter files from the training, there is no need to specify `--model` or `--system` separately.
```shell
# Using an interactive command line for inference.
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters output/vx-xxx/checkpoint-xxx \
--stream true
# merge-lora and use vLLM for inference acceleration
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters output/vx-xxx/checkpoint-xxx \
--stream true \
--merge_lora true \
--infer_backend vllm \
--max_model_len 8192
```
### Web-UI
The Web-UI is a **zero-threshold** training and deployment interface solution based on Gradio interface technology. For more details, you can check [here](https://swift.readthedocs.io/en/latest/GetStarted/Web-UI.html).
```shell
SWIFT_UI_LANG=en swift web-ui
```
### Using Python
ms-swift also supports training and inference using Python. Below is pseudocode for training and inference. For more details, you can refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/notebook).
Training:
```python
# Retrieve the model and template, and add a trainable LoRA module
model, tokenizer = get_model_tokenizer(model_id_or_path, ...)
template = get_template(model.model_meta.template, tokenizer, ...)
model = Swift.prepare_model(model, lora_config)
# Download and load the dataset, and encode the text into tokens
train_dataset, val_dataset = load_dataset(dataset_id_or_path, ...)
train_dataset = EncodePreprocessor(template=template)(train_dataset, num_proc=num_proc)
val_dataset = EncodePreprocessor(template=template)(val_dataset, num_proc=num_proc)
# Train the model
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
data_collator=template.data_collator,
train_dataset=train_dataset,
eval_dataset=val_dataset,
template=template,
)
trainer.train()
```
Inference:
```python
# Perform inference using the native PyTorch engine
engine = PtEngine(model_id_or_path, adapters=[lora_checkpoint])
infer_request = InferRequest(messages=[{'role': 'user', 'content': 'who are you?'}])
request_config = RequestConfig(max_tokens=max_new_tokens, temperature=temperature)
resp_list = engine.infer([infer_request], request_config)
print(f'response: {resp_list[0].choices[0].message.content}')
```
## ✨ Usage
Here is the simplest example of training to deployment using ms-swift. For more details, you can check the [examples](https://github.com/modelscope/ms-swift/tree/main/examples).
| Useful Links |
| ------ |
| [Command Line Parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html) |
| [Supported Models and Datasets](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) |
| [Custom Models](https://swift.readthedocs.io/en/latest/Customization/Custom-model.html), [Custom Datasets](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html) |
| [LLM Tutorial](https://github.com/modelscope/modelscope-classroom/tree/main/LLM-tutorial) |
### Training
Pre-training:
```shell
# 8*A100
NPROC_PER_NODE=8 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
swift pt \
--model Qwen/Qwen2.5-7B \
--dataset swift/chinese-c4 \
--streaming true \
--train_type full \
--deepspeed zero2 \
--output_dir output \
--max_steps 100000 \
...
```
Fine-tuning:
```shell
CUDA_VISIBLE_DEVICES=0 swift sft \
--model Qwen/Qwen2.5-7B-Instruct \
--dataset AI-ModelScope/alpaca-gpt4-data-en \
--train_type lora \
--output_dir output \
...
```
RLHF:
```shell
CUDA_VISIBLE_DEVICES=0 swift rlhf \
--rlhf_type dpo \
--model Qwen/Qwen2.5-7B-Instruct \
--dataset hjh0119/shareAI-Llama3-DPO-zh-en-emoji:en \
--train_type lora \
--output_dir output \
...
```
### Inference
```shell
CUDA_VISIBLE_DEVICES=0 swift infer \
--model Qwen/Qwen2.5-7B-Instruct \
--stream true \
--infer_backend pt
# LoRA
CUDA_VISIBLE_DEVICES=0 swift infer \
--model Qwen/Qwen2.5-7B-Instruct \
--adapters swift/test_lora \
--stream true \
--infer_backend pt \
--temperature 0
```
### Deployment
```shell
CUDA_VISIBLE_DEVICES=0 swift deploy \
--model Qwen/Qwen2.5-7B-Instruct \
--infer_backend vllm
```
### Evaluation
```shell
CUDA_VISIBLE_DEVICES=0 swift eval \
--model Qwen/Qwen2.5-7B-Instruct \
--infer_backend lmdeploy \
--eval_dataset ARC_c
```
### Quantization
```shell
CUDA_VISIBLE_DEVICES=0 swift export \
--model Qwen/Qwen2.5-7B-Instruct \
--quant_bits 4 --quant_method awq \
--dataset AI-ModelScope/alpaca-gpt4-data-zh \
--output_dir Qwen2.5-7B-Instruct-AWQ
```
## 🏛 License
This framework is licensed under the [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE). For models and datasets, please refer to the original resource page and follow the corresponding License.
## 📎 Citation
```bibtex
@misc{zhao2024swiftascalablelightweightinfrastructure,
title={SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning},
author={Yuze Zhao and Jintao Huang and Jinghan Hu and Xingjun Wang and Yunlin Mao and Daoze Zhang and Zeyinzi Jiang and Zhikai Wu and Baole Ai and Ang Wang and Wenmeng Zhou and Yingda Chen},
year={2024},
eprint={2408.05517},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2408.05517},
}
```
## Star History
[](https://star-history.com/#modelscope/ms-swift&Date)
Raw data
{
"_id": null,
"home_page": "https://github.com/modelscope/swift",
"name": "ms-swift",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "python, petl, efficient tuners",
"author": "DAMO ModelScope teams",
"author_email": "contact@modelscope.cn",
"download_url": "https://files.pythonhosted.org/packages/a5/96/31c5ce85c8548afd2b8ca716bdc635f2e54591a01af163baea71dff4edd0/ms-swift-3.0.0.tar.gz",
"platform": null,
"description": "# SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning)\n\n<p align=\"center\">\n <br>\n <img src=\"asset/banner.png\"/>\n <br>\n<p>\n<p align=\"center\">\n<a href=\"https://modelscope.cn/home\">ModelScope Community Website</a>\n<br>\n <a href=\"README_CN.md\">\u4e2d\u6587</a>   \uff5c   English  \n</p>\n\n<p align=\"center\">\n<img src=\"https://img.shields.io/badge/python-%E2%89%A53.8-5be.svg\">\n<img src=\"https://img.shields.io/badge/pytorch-%E2%89%A52.0-orange.svg\">\n<a href=\"https://github.com/modelscope/modelscope/\"><img src=\"https://img.shields.io/badge/modelscope-%E2%89%A51.19-5D91D4.svg\"></a>\n<a href=\"https://pypi.org/project/ms-swift/\"><img src=\"https://badge.fury.io/py/ms-swift.svg\"></a>\n<a href=\"https://github.com/modelscope/swift/blob/main/LICENSE\"><img src=\"https://img.shields.io/github/license/modelscope/swift\"></a>\n<a href=\"https://pepy.tech/project/ms-swift\"><img src=\"https://pepy.tech/badge/ms-swift\"></a>\n<a href=\"https://github.com/modelscope/swift/pulls\"><img src=\"https://img.shields.io/badge/PR-welcome-55EB99.svg\"></a>\n</p>\n\n<p align=\"center\">\n<a href=\"https://trendshift.io/repositories/6427\" target=\"_blank\"><img src=\"https://trendshift.io/api/badge/repositories/6427\" alt=\"modelscope%2Fswift | Trendshift\" style=\"width: 250px; height: 55px;\" width=\"250\" height=\"55\"/></a>\n</p>\n\n<p align=\"center\">\n <a href=\"https://arxiv.org/abs/2408.05517\">Paper</a>   \uff5c <a href=\"https://swift.readthedocs.io/en/latest/\">English Documentation</a>   \uff5c   <a href=\"https://swift.readthedocs.io/zh-cn/latest/\">\u4e2d\u6587\u6587\u6863</a>  \n</p>\n<p align=\"center\">\n <a href=\"https://swift2x-en.readthedocs.io/en/latest/\">Swift2.x En Doc</a>   \uff5c   <a href=\"https://swift2x.readthedocs.io/zh-cn/latest/\">Swift2.x\u4e2d\u6587\u6587\u6863</a>  \n</p>\n\n\n## \ud83d\udcd6 Table of Contents\n- [Groups](#-Groups)\n- [Introduction](#-introduction)\n- [News](#-news)\n- [Installation](#%EF%B8%8F-installation)\n- [Quick Start](#-quick-Start)\n- [Usage](#-Usage)\n- [License](#-License)\n- [Citation](#-citation)\n\n\n## \u260e Groups\n\nYou can contact us and communicate with us by adding our group:\n\n\n[Discord Group](https://discord.com/invite/D27yfEFVz5) | \u5fae\u4fe1\u7fa4\n:-------------------------:|:-------------------------:\n<img src=\"asset/discord_qr.jpg\" width=\"200\" height=\"200\"> | <img src=\"asset/wechat.png\" width=\"200\" height=\"200\">\n\n\n## \ud83d\udcdd Introduction\n\ud83c\udf72 ms-swift is an official framework provided by the ModelScope community for fine-tuning and deploying large language models and multi-modal large models. It currently supports the training (pre-training, fine-tuning, human alignment), inference, evaluation, quantization, and deployment of over 400 large models and 100+ multi-modal large models. These large language models (LLMs) include models such as Qwen2.5, Llama3.2, GLM4, Internlm2.5, Yi1.5, Mistral, DeepSeek, Baichuan2, Gemma2, and TeleChat2. The multi-modal LLMs include models such as Qwen2-VL, Qwen2-Audio, Llama3.2-Vision, Llava, InternVL2.5, MiniCPM-V-2.6, GLM4v, Xcomposer2.5, Yi-VL, DeepSeek-VL2, Phi3.5-Vision, and GOT-OCR2.\n\n\ud83c\udf54 In addition, ms-swift gathers the latest training technologies, including LoRA, QLoRA, Llama-Pro, LongLoRA, GaLore, Q-GaLore, LoRA+, LISA, DoRA, FourierFt, ReFT, UnSloth, and Liger. ms-swift supports accelerating the inference, evaluation, and deployment modules using vLLM and LMDeploy. To help researchers and developers fine-tune and apply large models more easily, ms-swift also provides a Gradio-based Web-UI interface and a wealth of best practices.\n\n**Why choose ms-swift?**\n\n- \ud83c\udf4e **Model Types**: Supports 400+ large language models and **100+ multi-modal large models** and all-to-all models, **providing a comprehensive solution from training to deployment**.\n- **Dataset Types**: Comes with 150+ pre-training, fine-tuning, human alignment, multi-modal datasets, and supports custom datasets.\n- **Hardware Support**: Compatible with CPU, RTX series, T4/V100, A10/A100/H100, Ascend NPU, etc.\n- \ud83c\udf4a **Lightweight Training**: Supports lightweight fine-tuning methods like LoRA, QLoRA, DoRA, LoRA+, ReFT, RS-LoRA, LLaMAPro, Adapter, GaLore, Q-Galore, LISA, UnSloth, Liger-Kernel.\n- **Distributed Training**: Supports distributed data parallel (DDP), device_map simple model parallelism, DeepSpeed ZeRO2/ZeRO3, FSDP, and other distributed training techniques.\n- **Quantization Training**: Supports training quantized models like BNB, AWQ, GPTQ, AQLM, HQQ, EETQ.\n- **RLHF Training**: Supports human alignment training methods such as DPO, CPO, SimPO, ORPO, KTO, RM for both pure text and multi-modal large models.\n- \ud83c\udf53 **Multi-Modal Training**: Supports training on different modalities like images, videos, and audio, for tasks like VQA, captioning, OCR, and grounding.\n- **Interface Training**: Provides capabilities for training, inference, evaluation, quantization through an interface, completing the whole large model pipeline.\n- **Plugin and Extension**: Supports custom model and dataset extensions, as well as customization of components like loss, metric, trainer, loss-scale, callback, optimizer.\n- \ud83c\udf49 **Toolbox Capabilities**: Offers not only training support for large models and multi-modal large models but also covers the entire process of inference, evaluation, quantization, and deployment.\n- **Inference Acceleration**: Supports inference acceleration engines like PyTorch, vLLM, LmDeploy, and provides OpenAI API for accelerating inference, deployment, and evaluation modules.\n- **Model Evaluation**: Uses EvalScope as the evaluation backend and supports evaluation on 100+ datasets for both pure text and multi-modal models.\n- **Model Quantization**: Supports AWQ, GPTQ, and BNB quantized exports, with models that can use vLLM/LmDeploy for inference acceleration and continue training.\n\n\n## \ud83c\udf89 News\n\n- \ud83c\udf81 2024.12.04: **SWIFT3.0** major version update. Please check the [Release Notes and Changes](https://swift.readthedocs.io/en/latest/Instruction/ReleaseNote3.0.html).\n- \ud83c\udf89 2024.08.12: The SWIFT paper has been published on arXiv, and you can read it [here](https://arxiv.org/abs/2408.05517).\n- \ud83d\udd25 2024.08.05: Support for using [evalscope](https://github.com/modelscope/evalscope/) as a backend for evaluating large models and multimodal models.\n- \ud83d\udd25 2024.07.29: Support for using [vllm](https://github.com/vllm-project/vllm) and [lmdeploy](https://github.com/InternLM/lmdeploy) to accelerate inference for large models and multimodal models. When performing infer/deploy/eval, you can specify `--infer_backend vllm/lmdeploy`.\n- \ud83d\udd25 2024.07.24: Support for human preference alignment training for multimodal large models, including DPO/ORPO/SimPO/CPO/KTO/RM.\n- \ud83d\udd25 2024.02.01: Support for Agent training! The training algorithm is derived from [this paper](https://arxiv.org/pdf/2309.00986.pdf).\n\n\n## \ud83d\udee0\ufe0f Installation\nTo install using pip:\n```shell\npip install ms-swift -U\n```\n\nTo install from source:\n```shell\n# pip install git+https://github.com/modelscope/ms-swift.git\n\ngit clone https://github.com/modelscope/ms-swift.git\ncd ms-swift\npip install -e .\n```\n\n## \ud83d\ude80 Quick Start\n\n10 minutes of self-cognition fine-tuning of Qwen2.5-7B-Instruct on a single 3090 GPU:\n\n### Command Line Interface\n\n```shell\n# 22GB\nCUDA_VISIBLE_DEVICES=0 \\\nswift sft \\\n --model Qwen/Qwen2.5-7B-Instruct \\\n --train_type lora \\\n --dataset AI-ModelScope/alpaca-gpt4-data-zh#500 \\\n AI-ModelScope/alpaca-gpt4-data-en#500 \\\n swift/self-cognition#500 \\\n --torch_dtype bfloat16 \\\n --num_train_epochs 1 \\\n --per_device_train_batch_size 1 \\\n --per_device_eval_batch_size 1 \\\n --learning_rate 1e-4 \\\n --lora_rank 8 \\\n --lora_alpha 32 \\\n --target_modules all-linear \\\n --gradient_accumulation_steps 16 \\\n --eval_steps 50 \\\n --save_steps 50 \\\n --save_total_limit 2 \\\n --logging_steps 5 \\\n --max_length 2048 \\\n --output_dir output \\\n --system 'You are a helpful assistant.' \\\n --warmup_ratio 0.05 \\\n --dataloader_num_workers 4 \\\n --model_author swift \\\n --model_name swift-robot\n```\n\nAfter training is complete, use the following command to perform inference with the trained weights. The `--adapters` option should be replaced with the last checkpoint folder generated from the training. Since the adapters folder contains the parameter files from the training, there is no need to specify `--model` or `--system` separately.\n\n```shell\n# Using an interactive command line for inference.\nCUDA_VISIBLE_DEVICES=0 \\\nswift infer \\\n --adapters output/vx-xxx/checkpoint-xxx \\\n --stream true\n\n# merge-lora and use vLLM for inference acceleration\nCUDA_VISIBLE_DEVICES=0 \\\nswift infer \\\n --adapters output/vx-xxx/checkpoint-xxx \\\n --stream true \\\n --merge_lora true \\\n --infer_backend vllm \\\n --max_model_len 8192\n```\n\n### Web-UI\nThe Web-UI is a **zero-threshold** training and deployment interface solution based on Gradio interface technology. For more details, you can check [here](https://swift.readthedocs.io/en/latest/GetStarted/Web-UI.html).\n\n```shell\nSWIFT_UI_LANG=en swift web-ui\n```\n\n\n\n### Using Python\n\nms-swift also supports training and inference using Python. Below is pseudocode for training and inference. For more details, you can refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/notebook).\n\nTraining:\n\n```python\n# Retrieve the model and template, and add a trainable LoRA module\nmodel, tokenizer = get_model_tokenizer(model_id_or_path, ...)\ntemplate = get_template(model.model_meta.template, tokenizer, ...)\nmodel = Swift.prepare_model(model, lora_config)\n\n# Download and load the dataset, and encode the text into tokens\ntrain_dataset, val_dataset = load_dataset(dataset_id_or_path, ...)\ntrain_dataset = EncodePreprocessor(template=template)(train_dataset, num_proc=num_proc)\nval_dataset = EncodePreprocessor(template=template)(val_dataset, num_proc=num_proc)\n\n# Train the model\ntrainer = Seq2SeqTrainer(\n model=model,\n args=training_args,\n data_collator=template.data_collator,\n train_dataset=train_dataset,\n eval_dataset=val_dataset,\n template=template,\n)\ntrainer.train()\n```\nInference:\n\n```python\n# Perform inference using the native PyTorch engine\nengine = PtEngine(model_id_or_path, adapters=[lora_checkpoint])\ninfer_request = InferRequest(messages=[{'role': 'user', 'content': 'who are you?'}])\nrequest_config = RequestConfig(max_tokens=max_new_tokens, temperature=temperature)\n\nresp_list = engine.infer([infer_request], request_config)\nprint(f'response: {resp_list[0].choices[0].message.content}')\n```\n\n## \u2728 Usage\nHere is the simplest example of training to deployment using ms-swift. For more details, you can check the [examples](https://github.com/modelscope/ms-swift/tree/main/examples).\n\n| Useful Links |\n| ------ |\n| [Command Line Parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html) |\n| [Supported Models and Datasets](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) |\n| [Custom Models](https://swift.readthedocs.io/en/latest/Customization/Custom-model.html), [Custom Datasets](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html) |\n| [LLM Tutorial](https://github.com/modelscope/modelscope-classroom/tree/main/LLM-tutorial) |\n\n### Training\n\nPre-training:\n```shell\n# 8*A100\nNPROC_PER_NODE=8 \\\nCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \\\nswift pt \\\n --model Qwen/Qwen2.5-7B \\\n --dataset swift/chinese-c4 \\\n --streaming true \\\n --train_type full \\\n --deepspeed zero2 \\\n --output_dir output \\\n --max_steps 100000 \\\n ...\n```\n\nFine-tuning:\n```shell\nCUDA_VISIBLE_DEVICES=0 swift sft \\\n --model Qwen/Qwen2.5-7B-Instruct \\\n --dataset AI-ModelScope/alpaca-gpt4-data-en \\\n --train_type lora \\\n --output_dir output \\\n ...\n```\n\nRLHF:\n```shell\nCUDA_VISIBLE_DEVICES=0 swift rlhf \\\n --rlhf_type dpo \\\n --model Qwen/Qwen2.5-7B-Instruct \\\n --dataset hjh0119/shareAI-Llama3-DPO-zh-en-emoji:en \\\n --train_type lora \\\n --output_dir output \\\n ...\n```\n\n\n### Inference\n```shell\nCUDA_VISIBLE_DEVICES=0 swift infer \\\n --model Qwen/Qwen2.5-7B-Instruct \\\n --stream true \\\n --infer_backend pt\n\n# LoRA\nCUDA_VISIBLE_DEVICES=0 swift infer \\\n --model Qwen/Qwen2.5-7B-Instruct \\\n --adapters swift/test_lora \\\n --stream true \\\n --infer_backend pt \\\n --temperature 0\n```\n\n### Deployment\n```shell\nCUDA_VISIBLE_DEVICES=0 swift deploy \\\n --model Qwen/Qwen2.5-7B-Instruct \\\n --infer_backend vllm\n```\n\n### Evaluation\n```shell\nCUDA_VISIBLE_DEVICES=0 swift eval \\\n --model Qwen/Qwen2.5-7B-Instruct \\\n --infer_backend lmdeploy \\\n --eval_dataset ARC_c\n```\n\n### Quantization\n```shell\nCUDA_VISIBLE_DEVICES=0 swift export \\\n --model Qwen/Qwen2.5-7B-Instruct \\\n --quant_bits 4 --quant_method awq \\\n --dataset AI-ModelScope/alpaca-gpt4-data-zh \\\n --output_dir Qwen2.5-7B-Instruct-AWQ\n```\n\n## \ud83c\udfdb License\n\nThis framework is licensed under the [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE). For models and datasets, please refer to the original resource page and follow the corresponding License.\n\n## \ud83d\udcce Citation\n\n```bibtex\n@misc{zhao2024swiftascalablelightweightinfrastructure,\n title={SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning},\n author={Yuze Zhao and Jintao Huang and Jinghan Hu and Xingjun Wang and Yunlin Mao and Daoze Zhang and Zeyinzi Jiang and Zhikai Wu and Baole Ai and Ang Wang and Wenmeng Zhou and Yingda Chen},\n year={2024},\n eprint={2408.05517},\n archivePrefix={arXiv},\n primaryClass={cs.CL},\n url={https://arxiv.org/abs/2408.05517},\n}\n```\n\n## Star History\n\n[](https://star-history.com/#modelscope/ms-swift&Date)\n\n\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Swift: Scalable lightWeight Infrastructure for Fine-Tuning",
"version": "3.0.0",
"project_urls": {
"Homepage": "https://github.com/modelscope/swift"
},
"split_keywords": [
"python",
" petl",
" efficient tuners"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "25b3240e23172c494bd6a2504cc1fcaf2e73efd12e1df51f7b30586f21eac982",
"md5": "37a57d0cf7c73e32a1a0b527349f3ad1",
"sha256": "d8ebd7d46bd9b423e186f84b366c87cabed541c23d52aecdc054504102a82cf1"
},
"downloads": -1,
"filename": "ms_swift-3.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "37a57d0cf7c73e32a1a0b527349f3ad1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 481782,
"upload_time": "2024-12-20T11:06:53",
"upload_time_iso_8601": "2024-12-20T11:06:53.749873Z",
"url": "https://files.pythonhosted.org/packages/25/b3/240e23172c494bd6a2504cc1fcaf2e73efd12e1df51f7b30586f21eac982/ms_swift-3.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "a59631c5ce85c8548afd2b8ca716bdc635f2e54591a01af163baea71dff4edd0",
"md5": "0e6dfe7a2df7bcb81692d95df636f531",
"sha256": "7503bf31ef8ed411ae9bff5e9c813f4365367b62e8deb912421252fe44c86a92"
},
"downloads": -1,
"filename": "ms-swift-3.0.0.tar.gz",
"has_sig": false,
"md5_digest": "0e6dfe7a2df7bcb81692d95df636f531",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 354717,
"upload_time": "2024-12-20T11:06:57",
"upload_time_iso_8601": "2024-12-20T11:06:57.626034Z",
"url": "https://files.pythonhosted.org/packages/a5/96/31c5ce85c8548afd2b8ca716bdc635f2e54591a01af163baea71dff4edd0/ms-swift-3.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-20 11:06:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "modelscope",
"github_project": "swift",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "ms-swift"
}
JSON
