This is a fork of vLLM to support xfastertransformer backend. This version is based on official vllm `v0.4.2`.
## Notice
🎉🎉🎉***Continuous batching and distributed is supported.*** 🎇🎇🎇
- BeamSearch is not support yet.(WIP)
- LORA is not support yet.(WIP)
## Install
### From PyPI
`pip install vllm-xft`
### From Source
`python3 setup.py bdist_wheel --verbose`
## Usage
### Python offline
```
python examples/offline_inference_xfastertransformer.py
```
### Serving(OpenAI Compatible Server)
```shell
python -m vllm.entrypoints.openai.api_server \
--model /data/llama-2-7b-chat-cpu \
--tokenizer /data/llama-2-7b-chat-hf \
--dtype fp16 \
--kv-cache-dtype fp16 \
--served-model-name xft \
--port 8000 \
--trust-remote-code \
```
- `--max-num-batched-tokens`: max batched token, default value is max(MAX_SEQ_LEN_OF_MODEL, 2048).
- `--max-num-seqs`: max seqs batch, default is 256.
More Arguments please refer to [vllm official docs](https://docs.vllm.ai/en/latest/models/engine_args.html)
### Query example
```shell
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "xft",
"prompt": "San Francisco is a",
"max_tokens": 512,
"temperature": 0
}'
```
## Distributed(Multi-rank)
Use oneCCL's `mpirun` to run the workload. The master (`rank 0`) is the same as the single-rank above, and the slaves (`rank > 0`) should use the following command:
```bash
python -m vllm.entrypoints.slave --dtype fp16 --model ${MODEL_PATH} --kv-cache-dtype fp16
```
Please keep params of slaves align with master.
### Serving(OpenAI Compatible Server)
Here is a example on 2Socket platform, 48 cores pre socket.
```bash
OMP_NUM_THREADS=48 mpirun \
-n 1 numactl --all -C 0-47 -m 0 \
python -m vllm.entrypoints.openai.api_server \
--model ${MODEL_PATH} \
--tokenizer ${TOKEN_PATH} \
--dtype bf16 \
--kv-cache-dtype fp16 \
--served-model-name xft \
--port 8000 \
--trust-remote-code \
: -n 1 numactl --all -C 48-95 -m 1 \
python -m vllm.entrypoints.slave \
--dtype bf16 \
--model ${MODEL_PATH} \
--kv-cache-dtype fp16
```
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-dark.png">
<img alt="vLLM" src="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-light.png" width=55%>
</picture>
</p>
<h3 align="center">
Easy, fast, and cheap LLM serving for everyone
</h3>
<p align="center">
| <a href="https://docs.vllm.ai"><b>Documentation</b></a> | <a href="https://vllm.ai"><b>Blog</b></a> | <a href="https://arxiv.org/abs/2309.06180"><b>Paper</b></a> | <a href="https://discord.gg/jz7wjKhh6g"><b>Discord</b></a> | <a href="https://x.com/vllm_project"><b>Twitter/X</b></a> |
</p>
---
**vLLM & NVIDIA Triton User Meetup (Monday, September 9, 5pm-9pm PT) at Fort Mason, San Francisco**
We are excited to announce our sixth vLLM Meetup, in collaboration with NVIDIA Triton Team.
Join us to hear the vLLM's recent update about performance.
Register now [here](https://lu.ma/87q3nvnh) and be part of the event!
---
*Latest News* 🔥
- [2024/07] We hosted [the fifth vLLM meetup](https://lu.ma/lp0gyjqr) with AWS! Please find the meetup slides [here](https://docs.google.com/presentation/d/1RgUD8aCfcHocghoP3zmXzck9vX3RCI9yfUAB2Bbcl4Y/edit?usp=sharing).
- [2024/07] In partnership with Meta, vLLM officially supports Llama 3.1 with FP8 quantization and pipeline parallelism! Please check out our blog post [here](https://blog.vllm.ai/2024/07/23/llama31.html).
- [2024/06] We hosted [the fourth vLLM meetup](https://lu.ma/agivllm) with Cloudflare and BentoML! Please find the meetup slides [here](https://docs.google.com/presentation/d/1iJ8o7V2bQEi0BFEljLTwc5G1S10_Rhv3beed5oB0NJ4/edit?usp=sharing).
- [2024/04] We hosted [the third vLLM meetup](https://robloxandvllmmeetup2024.splashthat.com/) with Roblox! Please find the meetup slides [here](https://docs.google.com/presentation/d/1A--47JAK4BJ39t954HyTkvtfwn0fkqtsL8NGFuslReM/edit?usp=sharing).
- [2024/01] We hosted [the second vLLM meetup](https://lu.ma/ygxbpzhl) with IBM! Please find the meetup slides [here](https://docs.google.com/presentation/d/12mI2sKABnUw5RBWXDYY-HtHth4iMSNcEoQ10jDQbxgA/edit?usp=sharing).
- [2023/10] We hosted [the first vLLM meetup](https://lu.ma/first-vllm-meetup) with a16z! Please find the meetup slides [here](https://docs.google.com/presentation/d/1QL-XPFXiFpDBh86DbEegFXBXFXjix4v032GhShbKf3s/edit?usp=sharing).
- [2023/08] We would like to express our sincere gratitude to [Andreessen Horowitz](https://a16z.com/2023/08/30/supporting-the-open-source-ai-community/) (a16z) for providing a generous grant to support the open-source development and research of vLLM.
- [2023/06] We officially released vLLM! FastChat-vLLM integration has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid-April. Check out our [blog post](https://vllm.ai).
---
## About
vLLM is a fast and easy-to-use library for LLM inference and serving.
vLLM is fast with:
- State-of-the-art serving throughput
- Efficient management of attention key and value memory with **PagedAttention**
- Continuous batching of incoming requests
- Fast model execution with CUDA/HIP graph
- Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), INT4, INT8, and FP8.
- Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
- Speculative decoding
- Chunked prefill
**Performance benchmark**: We include a [performance benchmark](https://buildkite.com/vllm/performance-benchmark/builds/4068) that compares the performance of vLLM against other LLM serving engines ([TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [text-generation-inference](https://github.com/huggingface/text-generation-inference) and [lmdeploy](https://github.com/InternLM/lmdeploy)).
vLLM is flexible and easy to use with:
- Seamless integration with popular Hugging Face models
- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
- Tensor parallelism and pipeline parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
- Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
- Prefix caching support
- Multi-lora support
vLLM seamlessly supports most popular open-source models on HuggingFace, including:
- Transformer-like LLMs (e.g., Llama)
- Mixture-of-Expert LLMs (e.g., Mixtral)
- Embedding Models (e.g. E5-Mistral)
- Multi-modal LLMs (e.g., LLaVA)
Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).
## Getting Started
Install vLLM with `pip` or [from source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source):
```bash
pip install vllm
```
Visit our [documentation](https://vllm.readthedocs.io/en/latest/) to learn more.
- [Installation](https://vllm.readthedocs.io/en/latest/getting_started/installation.html)
- [Quickstart](https://vllm.readthedocs.io/en/latest/getting_started/quickstart.html)
- [Supported Models](https://vllm.readthedocs.io/en/latest/models/supported_models.html)
## Contributing
We welcome and value any contributions and collaborations.
Please check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.
## Sponsors
vLLM is a community project. Our compute resources for development and testing are supported by the following organizations. Thank you for your support!
<!-- Note: Please sort them in alphabetical order. -->
<!-- Note: Please keep these consistent with docs/source/community/sponsors.md -->
- a16z
- AMD
- Anyscale
- AWS
- Crusoe Cloud
- Databricks
- DeepInfra
- Dropbox
- Google Cloud
- Lambda Lab
- NVIDIA
- Replicate
- Roblox
- RunPod
- Sequoia Capital
- Skywork AI
- Trainy
- UC Berkeley
- UC San Diego
- ZhenFund
We also have an official fundraising venue through [OpenCollective](https://opencollective.com/vllm). We plan to use the fund to support the development, maintenance, and adoption of vLLM.
## Citation
If you use vLLM for your research, please cite our [paper](https://arxiv.org/abs/2309.06180):
```bibtex
@inproceedings{kwon2023efficient,
title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
year={2023}
}
```
Raw data
{
"_id": null,
"home_page": "https://github.com/vllm-project/vllm",
"name": "vllm-xft",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "vLLM Team",
"author_email": null,
"download_url": null,
"platform": null,
"description": "This is a fork of vLLM to support xfastertransformer backend. This version is based on official vllm `v0.4.2`.\n## Notice\n\ud83c\udf89\ud83c\udf89\ud83c\udf89***Continuous batching and distributed is supported.*** \ud83c\udf87\ud83c\udf87\ud83c\udf87\n- BeamSearch is not support yet.(WIP)\n- LORA is not support yet.(WIP)\n\n## Install\n### From PyPI\n`pip install vllm-xft`\n\n### From Source\n`python3 setup.py bdist_wheel --verbose`\n\n## Usage\n### Python offline\n```\npython examples/offline_inference_xfastertransformer.py\n```\n### Serving(OpenAI Compatible Server)\n```shell\npython -m vllm.entrypoints.openai.api_server \\\n --model /data/llama-2-7b-chat-cpu \\\n --tokenizer /data/llama-2-7b-chat-hf \\\n --dtype fp16 \\\n --kv-cache-dtype fp16 \\\n --served-model-name xft \\\n --port 8000 \\\n --trust-remote-code \\\n```\n- `--max-num-batched-tokens`: max batched token, default value is max(MAX_SEQ_LEN_OF_MODEL, 2048).\n- `--max-num-seqs`: max seqs batch, default is 256. \n\nMore Arguments please refer to [vllm official docs](https://docs.vllm.ai/en/latest/models/engine_args.html) \n\n### Query example\n```shell\n curl http://localhost:8000/v1/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"xft\",\n \"prompt\": \"San Francisco is a\",\n \"max_tokens\": 512,\n \"temperature\": 0\n }'\n```\n\n## Distributed(Multi-rank)\nUse oneCCL's `mpirun` to run the workload. The master (`rank 0`) is the same as the single-rank above, and the slaves (`rank > 0`) should use the following command:\n```bash\npython -m vllm.entrypoints.slave --dtype fp16 --model ${MODEL_PATH} --kv-cache-dtype fp16\n```\nPlease keep params of slaves align with master.\n\n### Serving(OpenAI Compatible Server)\nHere is a example on 2Socket platform, 48 cores pre socket.\n```bash\nOMP_NUM_THREADS=48 mpirun \\\n -n 1 numactl --all -C 0-47 -m 0 \\\n python -m vllm.entrypoints.openai.api_server \\\n --model ${MODEL_PATH} \\\n --tokenizer ${TOKEN_PATH} \\\n --dtype bf16 \\\n --kv-cache-dtype fp16 \\\n --served-model-name xft \\\n --port 8000 \\\n --trust-remote-code \\\n : -n 1 numactl --all -C 48-95 -m 1 \\\n python -m vllm.entrypoints.slave \\\n --dtype bf16 \\\n --model ${MODEL_PATH} \\\n --kv-cache-dtype fp16\n```\n\n<p align=\"center\">\n <picture>\n <source media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-dark.png\">\n <img alt=\"vLLM\" src=\"https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-light.png\" width=55%>\n </picture>\n</p>\n\n<h3 align=\"center\">\nEasy, fast, and cheap LLM serving for everyone\n</h3>\n\n<p align=\"center\">\n| <a href=\"https://docs.vllm.ai\"><b>Documentation</b></a> | <a href=\"https://vllm.ai\"><b>Blog</b></a> | <a href=\"https://arxiv.org/abs/2309.06180\"><b>Paper</b></a> | <a href=\"https://discord.gg/jz7wjKhh6g\"><b>Discord</b></a> | <a href=\"https://x.com/vllm_project\"><b>Twitter/X</b></a> |\n\n</p>\n\n\n---\n\n**vLLM & NVIDIA Triton User Meetup (Monday, September 9, 5pm-9pm PT) at Fort Mason, San Francisco**\n\nWe are excited to announce our sixth vLLM Meetup, in collaboration with NVIDIA Triton Team.\nJoin us to hear the vLLM's recent update about performance.\nRegister now [here](https://lu.ma/87q3nvnh) and be part of the event!\n\n---\n\n*Latest News* \ud83d\udd25\n- [2024/07] We hosted [the fifth vLLM meetup](https://lu.ma/lp0gyjqr) with AWS! Please find the meetup slides [here](https://docs.google.com/presentation/d/1RgUD8aCfcHocghoP3zmXzck9vX3RCI9yfUAB2Bbcl4Y/edit?usp=sharing).\n- [2024/07] In partnership with Meta, vLLM officially supports Llama 3.1 with FP8 quantization and pipeline parallelism! Please check out our blog post [here](https://blog.vllm.ai/2024/07/23/llama31.html).\n- [2024/06] We hosted [the fourth vLLM meetup](https://lu.ma/agivllm) with Cloudflare and BentoML! Please find the meetup slides [here](https://docs.google.com/presentation/d/1iJ8o7V2bQEi0BFEljLTwc5G1S10_Rhv3beed5oB0NJ4/edit?usp=sharing).\n- [2024/04] We hosted [the third vLLM meetup](https://robloxandvllmmeetup2024.splashthat.com/) with Roblox! Please find the meetup slides [here](https://docs.google.com/presentation/d/1A--47JAK4BJ39t954HyTkvtfwn0fkqtsL8NGFuslReM/edit?usp=sharing).\n- [2024/01] We hosted [the second vLLM meetup](https://lu.ma/ygxbpzhl) with IBM! Please find the meetup slides [here](https://docs.google.com/presentation/d/12mI2sKABnUw5RBWXDYY-HtHth4iMSNcEoQ10jDQbxgA/edit?usp=sharing).\n- [2023/10] We hosted [the first vLLM meetup](https://lu.ma/first-vllm-meetup) with a16z! Please find the meetup slides [here](https://docs.google.com/presentation/d/1QL-XPFXiFpDBh86DbEegFXBXFXjix4v032GhShbKf3s/edit?usp=sharing).\n- [2023/08] We would like to express our sincere gratitude to [Andreessen Horowitz](https://a16z.com/2023/08/30/supporting-the-open-source-ai-community/) (a16z) for providing a generous grant to support the open-source development and research of vLLM.\n- [2023/06] We officially released vLLM! FastChat-vLLM integration has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid-April. Check out our [blog post](https://vllm.ai).\n\n---\n## About\nvLLM is a fast and easy-to-use library for LLM inference and serving.\n\nvLLM is fast with:\n\n- State-of-the-art serving throughput\n- Efficient management of attention key and value memory with **PagedAttention**\n- Continuous batching of incoming requests\n- Fast model execution with CUDA/HIP graph\n- Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), INT4, INT8, and FP8.\n- Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.\n- Speculative decoding\n- Chunked prefill\n\n**Performance benchmark**: We include a [performance benchmark](https://buildkite.com/vllm/performance-benchmark/builds/4068) that compares the performance of vLLM against other LLM serving engines ([TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [text-generation-inference](https://github.com/huggingface/text-generation-inference) and [lmdeploy](https://github.com/InternLM/lmdeploy)).\n\nvLLM is flexible and easy to use with:\n\n- Seamless integration with popular Hugging Face models\n- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more\n- Tensor parallelism and pipeline parallelism support for distributed inference\n- Streaming outputs\n- OpenAI-compatible API server\n- Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.\n- Prefix caching support\n- Multi-lora support\n\nvLLM seamlessly supports most popular open-source models on HuggingFace, including:\n- Transformer-like LLMs (e.g., Llama)\n- Mixture-of-Expert LLMs (e.g., Mixtral)\n- Embedding Models (e.g. E5-Mistral)\n- Multi-modal LLMs (e.g., LLaVA)\n\nFind the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).\n\n## Getting Started\n\nInstall vLLM with `pip` or [from source](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source):\n\n```bash\npip install vllm\n```\n\nVisit our [documentation](https://vllm.readthedocs.io/en/latest/) to learn more.\n- [Installation](https://vllm.readthedocs.io/en/latest/getting_started/installation.html)\n- [Quickstart](https://vllm.readthedocs.io/en/latest/getting_started/quickstart.html)\n- [Supported Models](https://vllm.readthedocs.io/en/latest/models/supported_models.html)\n\n## Contributing\n\nWe welcome and value any contributions and collaborations.\nPlease check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.\n\n## Sponsors\n\nvLLM is a community project. Our compute resources for development and testing are supported by the following organizations. Thank you for your support!\n\n<!-- Note: Please sort them in alphabetical order. -->\n<!-- Note: Please keep these consistent with docs/source/community/sponsors.md -->\n\n- a16z\n- AMD\n- Anyscale\n- AWS\n- Crusoe Cloud\n- Databricks\n- DeepInfra\n- Dropbox\n- Google Cloud\n- Lambda Lab\n- NVIDIA\n- Replicate\n- Roblox\n- RunPod\n- Sequoia Capital\n- Skywork AI\n- Trainy\n- UC Berkeley\n- UC San Diego\n- ZhenFund\n\nWe also have an official fundraising venue through [OpenCollective](https://opencollective.com/vllm). We plan to use the fund to support the development, maintenance, and adoption of vLLM.\n\n## Citation\n\nIf you use vLLM for your research, please cite our [paper](https://arxiv.org/abs/2309.06180):\n```bibtex\n@inproceedings{kwon2023efficient,\n title={Efficient Memory Management for Large Language Model Serving with PagedAttention},\n author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},\n booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},\n year={2023}\n}\n```\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "A high-throughput and memory-efficient inference and serving engine for LLMs",
"version": "0.5.5.0",
"project_urls": {
"Documentation": "https://vllm.readthedocs.io/en/latest/",
"Homepage": "https://github.com/vllm-project/vllm"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "65a7ba9d06d827320dbb07d5bff91d02420dfd4138dc89992c80b3af79fe6a09",
"md5": "d67854c57e0fbce6ca79d7ac730ebd97",
"sha256": "65a120bdfa8e581ed45218b8bf5cbd75f317e56bd5610fa3a938cdbe0bb386fb"
},
"downloads": -1,
"filename": "vllm_xft-0.5.5.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d67854c57e0fbce6ca79d7ac730ebd97",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 1108142,
"upload_time": "2024-08-29T08:38:30",
"upload_time_iso_8601": "2024-08-29T08:38:30.163127Z",
"url": "https://files.pythonhosted.org/packages/65/a7/ba9d06d827320dbb07d5bff91d02420dfd4138dc89992c80b3af79fe6a09/vllm_xft-0.5.5.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-29 08:38:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "vllm-project",
"github_project": "vllm",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "vllm-xft"
}
JSON
