# vLLM: Easy, Fast, and Cheap LLM Serving for Everyone
| [**Documentation**](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) | [**Blog**]() |
vLLM is a fast and easy-to-use library for LLM inference and serving.
## Latest News 🔥
- [2023/06] We officially released vLLM! vLLM has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid April. Check out our [blog post]().
## Getting Started
Visit our [documentation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) to get started.
- [Installation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/installation.html): `pip install vllm`
- [Quickstart](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/quickstart.html)
- [Supported Models](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/models/supported_models.html)
## Key Features
vLLM comes with many powerful features that include:
- State-of-the-art performance in serving throughput
- Efficient management of attention key and value memory with **PagedAttention**
- Seamless integration with popular HuggingFace models
- Dynamic batching of incoming requests
- Optimized CUDA kernels
- High-throughput serving with various decoding algorithms, including *parallel sampling* and *beam search*
- Tensor parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
## Performance
vLLM outperforms HuggingFace Transformers (HF) by up to 24x and Text Generation Inference (TGI) by up to 3.5x, in terms of throughput.
For details, check out our [blog post]().
<p align="center">
<img src="./assets/figures/perf_a10g_n1.png" width="45%">
<img src="./assets/figures/perf_a100_n1.png" width="45%">
<br>
<em> Serving throughput when each request asks for 1 output completion. </em>
</p>
<p align="center">
<img src="./assets/figures/perf_a10g_n3.png" width="45%">
<img src="./assets/figures/perf_a100_n3.png" width="45%">
<br>
<em> Serving throughput when each request asks for 3 output completions. </em>
</p>
## Contributing
We welcome and value any contributions and collaborations.
Please check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.
Raw data
{
"_id": null,
"home_page": "https://github.com/vllm-project/vllm",
"name": "vllm-py",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "vLLM Team",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/73/dc/45b2816ff92728780dde08fcb13a79458b07a3231f8f1d60fdb61df3d6e3/vllm-py-0.0.1.tar.gz",
"platform": null,
"description": "# vLLM: Easy, Fast, and Cheap LLM Serving for Everyone\n\n| [**Documentation**](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) | [**Blog**]() |\n\nvLLM is a fast and easy-to-use library for LLM inference and serving.\n\n## Latest News \ud83d\udd25\n\n- [2023/06] We officially released vLLM! vLLM has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid April. Check out our [blog post]().\n\n## Getting Started\n\nVisit our [documentation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) to get started.\n- [Installation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/installation.html): `pip install vllm`\n- [Quickstart](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/quickstart.html)\n- [Supported Models](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/models/supported_models.html)\n\n## Key Features\n\nvLLM comes with many powerful features that include:\n\n- State-of-the-art performance in serving throughput\n- Efficient management of attention key and value memory with **PagedAttention**\n- Seamless integration with popular HuggingFace models\n- Dynamic batching of incoming requests\n- Optimized CUDA kernels\n- High-throughput serving with various decoding algorithms, including *parallel sampling* and *beam search*\n- Tensor parallelism support for distributed inference\n- Streaming outputs\n- OpenAI-compatible API server\n\n## Performance\n\nvLLM outperforms HuggingFace Transformers (HF) by up to 24x and Text Generation Inference (TGI) by up to 3.5x, in terms of throughput.\nFor details, check out our [blog post]().\n\n<p align=\"center\">\n <img src=\"./assets/figures/perf_a10g_n1.png\" width=\"45%\">\n <img src=\"./assets/figures/perf_a100_n1.png\" width=\"45%\">\n <br>\n <em> Serving throughput when each request asks for 1 output completion. </em>\n</p>\n\n<p align=\"center\">\n <img src=\"./assets/figures/perf_a10g_n3.png\" width=\"45%\">\n <img src=\"./assets/figures/perf_a100_n3.png\" width=\"45%\">\n <br>\n <em> Serving throughput when each request asks for 3 output completions. </em>\n</p>\n\n## Contributing\n\nWe welcome and value any contributions and collaborations.\nPlease check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "A high-throughput and memory-efficient inference and serving engine for LLMs",
"version": "0.0.1",
"project_urls": {
"Documentation": "https://vllm.readthedocs.io/en/latest/",
"Homepage": "https://github.com/vllm-project/vllm"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "73dc45b2816ff92728780dde08fcb13a79458b07a3231f8f1d60fdb61df3d6e3",
"md5": "42e929a0d558d8f9944f48b87b2e3471",
"sha256": "b336809fc0e9b3fd8a12671efb03b2a5a3c85ff60253dd6923aed0e8aa37774f"
},
"downloads": -1,
"filename": "vllm-py-0.0.1.tar.gz",
"has_sig": false,
"md5_digest": "42e929a0d558d8f9944f48b87b2e3471",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 82636,
"upload_time": "2023-06-19T03:47:08",
"upload_time_iso_8601": "2023-06-19T03:47:08.949939Z",
"url": "https://files.pythonhosted.org/packages/73/dc/45b2816ff92728780dde08fcb13a79458b07a3231f8f1d60fdb61df3d6e3/vllm-py-0.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-19 03:47:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "vllm-project",
"github_project": "vllm",
"github_not_found": true,
"lcname": "vllm-py"
}
JSON
