# vLLM: Easy, Fast, and Cheap LLM Serving for Everyone
| [**Documentation**](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) | [**Blog**]() |
vLLM is a fast and easy-to-use library for LLM inference and serving.
## Latest News 🔥
- [2023/06] We officially released vLLM! vLLM has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid April. Check out our [blog post]().
## Getting Started
Visit our [documentation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) to get started.
- [Installation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/installation.html): `pip install vllm`
- [Quickstart](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/quickstart.html)
- [Supported Models](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/models/supported_models.html)
## Key Features
vLLM comes with many powerful features that include:
- State-of-the-art performance in serving throughput
- Efficient management of attention key and value memory with **PagedAttention**
- Seamless integration with popular HuggingFace models
- Dynamic batching of incoming requests
- Optimized CUDA kernels
- High-throughput serving with various decoding algorithms, including *parallel sampling* and *beam search*
- Tensor parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
## Performance
vLLM outperforms HuggingFace Transformers (HF) by up to 24x and Text Generation Inference (TGI) by up to 3.5x, in terms of throughput.
For details, check out our [blog post]().
<p align="center">
<img src="./assets/figures/perf_a10g_n1.png" width="45%">
<img src="./assets/figures/perf_a100_n1.png" width="45%">
<br>
<em> Serving throughput when each request asks for 1 output completion. </em>
</p>
<p align="center">
<img src="./assets/figures/perf_a10g_n3.png" width="45%">
<img src="./assets/figures/perf_a100_n3.png" width="45%">
<br>
<em> Serving throughput when each request asks for 3 output completions. </em>
</p>
## Contributing
We welcome and value any contributions and collaborations.
Please check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.
Raw data
{
"_id": null,
"home_page": "https://github.com/vllm-project/vllm",
"name": "woosuk-vllm-test",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "vLLM Team",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/5e/8d/9c42c70c3c274b6881d4b33f67f1df7b7afb915056d88ae7dbbe0350cd61/woosuk-vllm-test-0.1.1.tar.gz",
"platform": null,
"description": "# vLLM: Easy, Fast, and Cheap LLM Serving for Everyone\n\n| [**Documentation**](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) | [**Blog**]() |\n\nvLLM is a fast and easy-to-use library for LLM inference and serving.\n\n## Latest News \ud83d\udd25\n\n- [2023/06] We officially released vLLM! vLLM has powered [LMSYS Vicuna and Chatbot Arena](https://chat.lmsys.org) since mid April. Check out our [blog post]().\n\n## Getting Started\n\nVisit our [documentation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/) to get started.\n- [Installation](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/installation.html): `pip install vllm`\n- [Quickstart](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/getting_started/quickstart.html)\n- [Supported Models](https://llm-serving-cacheflow.readthedocs-hosted.com/_/sharing/Cyo52MQgyoAWRQ79XA4iA2k8euwzzmjY?next=/en/latest/models/supported_models.html)\n\n## Key Features\n\nvLLM comes with many powerful features that include:\n\n- State-of-the-art performance in serving throughput\n- Efficient management of attention key and value memory with **PagedAttention**\n- Seamless integration with popular HuggingFace models\n- Dynamic batching of incoming requests\n- Optimized CUDA kernels\n- High-throughput serving with various decoding algorithms, including *parallel sampling* and *beam search*\n- Tensor parallelism support for distributed inference\n- Streaming outputs\n- OpenAI-compatible API server\n\n## Performance\n\nvLLM outperforms HuggingFace Transformers (HF) by up to 24x and Text Generation Inference (TGI) by up to 3.5x, in terms of throughput.\nFor details, check out our [blog post]().\n\n<p align=\"center\">\n <img src=\"./assets/figures/perf_a10g_n1.png\" width=\"45%\">\n <img src=\"./assets/figures/perf_a100_n1.png\" width=\"45%\">\n <br>\n <em> Serving throughput when each request asks for 1 output completion. </em>\n</p>\n\n<p align=\"center\">\n <img src=\"./assets/figures/perf_a10g_n3.png\" width=\"45%\">\n <img src=\"./assets/figures/perf_a100_n3.png\" width=\"45%\">\n <br>\n <em> Serving throughput when each request asks for 3 output completions. </em>\n</p>\n\n## Contributing\n\nWe welcome and value any contributions and collaborations.\nPlease check out [CONTRIBUTING.md](./CONTRIBUTING.md) for how to get involved.\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "vLLM: Easy, Fast, and Cheap LLM Serving for Everyone",
"version": "0.1.1",
"project_urls": {
"Documentation": "https://vllm.readthedocs.io/en/latest/",
"Homepage": "https://github.com/vllm-project/vllm"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5e8d9c42c70c3c274b6881d4b33f67f1df7b7afb915056d88ae7dbbe0350cd61",
"md5": "8ba7338b75b4264ef015b2154025c0d3",
"sha256": "1ead39a787287ff6760b77b9c757c2fdfaaf81c3700534c3cdf2d9b95bdade17"
},
"downloads": -1,
"filename": "woosuk-vllm-test-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "8ba7338b75b4264ef015b2154025c0d3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 82638,
"upload_time": "2023-06-18T20:09:46",
"upload_time_iso_8601": "2023-06-18T20:09:46.562385Z",
"url": "https://files.pythonhosted.org/packages/5e/8d/9c42c70c3c274b6881d4b33f67f1df7b7afb915056d88ae7dbbe0350cd61/woosuk-vllm-test-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-06-18 20:09:46",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "vllm-project",
"github_project": "vllm",
"github_not_found": true,
"lcname": "woosuk-vllm-test"
}
JSON
