<div align="center">
Optimum-NVIDIA
===========================
<h4> Optimized inference with NVIDIA and Hugging Face </h4>
[](https://huggingface.co/docs/optimum/index)
[](https://www.python.org/downloads/release/python-31013/)
[](https://developer.nvidia.com/cuda-downloads)
[](https://github.com/nvidia/tensorrt-llm)
[](./LICENSE)
---
<div align="left">
Optimum-NVIDIA delivers the best inference performance on the NVIDIA platform through Hugging Face. Run LLaMA 2 at 1,200 tokens/second (up to 28x faster than the framework) by changing just a single line in your existing transformers code.
</div></div>
# Installation
## Pip
Pip installation flow has been validated on Ubuntu only at this stage.
```shell
apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev
python -m pip install --pre --extra-index-url https://pypi.nvidia.com optimum-nvidia
```
For developers who want to target the best performances, please look at the installation methods below.
## Docker container
You can use a Docker container to try Optimum-NVIDIA today. Images are available on the Hugging Face Docker Hub.
```bash
docker pull huggingface/optimum-nvidia
```
## Building from source
<!---
Currently, TRT LLM is built and run with Docker, so we should wait until pip installation is available;
Ideally the user doesn't need to use docker at all to build from source, they should be able to run something like
`git clone [...] && pip install -e optimum-nvidia`
-->
Instead of using the pre-built docker container, you can build Optimum-NVIDIA from source:
```bash
TARGET_SM="90-real;89-real"
git clone --recursive --depth=1 https://github.com/huggingface/optimum-nvidia.git
cd optimum-nvidia/third-party/tensorrt-llm
make -C docker release_build CUDA_ARCHS=$TARGET_SM
cd ../.. && docker build -t <organisation_name/image_name>:<version> -f docker/Dockerfile .
```
<!--
```bash
git clone git@github.com:huggingface/optimum-nvidia.git
cd optimum-nvidia
docker build Dockerfile
docker run optimum-nvidia
``` -->
# Quickstart Guide
## Pipelines
Hugging Face pipelines provide a simple yet powerful abstraction to quickly set up inference. If you already have a pipeline from transformers, you can unlock the performance benefits of Optimum-NVIDIA by just changing one line.
```diff
- from transformers.pipelines import pipeline
+ from optimum.nvidia.pipelines import pipeline
pipe = pipeline('text-generation', 'meta-llama/Llama-2-7b-chat-hf', use_fp8=True)
pipe("Describe a real-world application of AI in sustainable energy.")
```
## Generate
If you want control over advanced features like quantization and token selection strategies, we recommend using the `generate()` API. Just like with `pipelines`, switching from existing transformers code is super simple.
```diff
- from transformers import AutoModelForCausalLM
+ from optimum.nvidia import AutoModelForCausalLM
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf", padding_side="left")
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-chat-hf",
+ use_fp8=True,
)
model_inputs = tokenizer(["How is autonomous vehicle technology transforming the future of transportation and urban planning?"], return_tensors="pt").to("cuda")
generated_ids = model.generate(
**model_inputs,
top_k=40,
top_p=0.7,
repetition_penalty=10,
)
tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
```
To learn more about text generation with LLMs, check out [this guide](https://huggingface.co/docs/transformers/llm_tutorial)!
<!-- For more details, read our [documentation](https://huggingface.com/docs/optimum/nvidia/index). -->
# Support Matrix
We test Optimum-NVIDIA on 4090, L40S, and H100 Tensor Core GPUs, though it is expected to work on any GPU based on the following architectures:
* Turing (with experimental support for T4 / RTX Quadro x000)
* Ampere (A100/A30 are supported. Experimental support for A10, A40, RTX Ax000)
* Hopper
* Ada-Lovelace
Note that FP8 support is only available on GPUs based on Hopper and Ada-Lovelace architectures.
Optimum-NVIDIA works on Linux will support Windows soon.
Optimum-NVIDIA currently accelerates text-generation with LLaMAForCausalLM, and we are actively working to expand support to include more model architectures and tasks.
<!-- Optimum-NVIDIA supports the following model architectures and tasks:
| Model | Tasks |
| :---- | :---- |
| Gemma | TextGeneration |
| Llama | TextGeneration |
| Mistral | TextGeneration |
| Additional Models | Coming soon | -->
# Contributing
Check out our [Contributing Guide](./CONTRIBUTING.md)
Raw data
{
"_id": null,
"home_page": "https://huggingface.co/hardware/nvidia",
"name": "optimum-nvidia",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "transformers, neural-network, inference, nvidia, tensorrt, ampere, hopper",
"author": "HuggingFace Inc. Machine Learning Optimization Team",
"author_email": "\"HuggingFace Inc. Machine Learning Optimization Team\" <hardware@huggingface.co>",
"download_url": "https://files.pythonhosted.org/packages/6b/82/f25f93ddba8a18d5aff8dc2b15ebf7aaf98d1e33823305b36ab49af0ed45/optimum-nvidia-0.1.0b6.tar.gz",
"platform": null,
"description": "\n<div align=\"center\">\n\nOptimum-NVIDIA\n===========================\n<h4> Optimized inference with NVIDIA and Hugging Face </h4>\n\n[](https://huggingface.co/docs/optimum/index)\n[](https://www.python.org/downloads/release/python-31013/)\n[](https://developer.nvidia.com/cuda-downloads)\n[](https://github.com/nvidia/tensorrt-llm)\n[](./LICENSE)\n\n---\n<div align=\"left\">\n\nOptimum-NVIDIA delivers the best inference performance on the NVIDIA platform through Hugging Face. Run LLaMA 2 at 1,200 tokens/second (up to 28x faster than the framework) by changing just a single line in your existing transformers code.\n\n</div></div>\n\n# Installation\n\n## Pip\n\nPip installation flow has been validated on Ubuntu only at this stage.\n\n```shell\napt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev\npython -m pip install --pre --extra-index-url https://pypi.nvidia.com optimum-nvidia\n``` \n\nFor developers who want to target the best performances, please look at the installation methods below.\n\n## Docker container\nYou can use a Docker container to try Optimum-NVIDIA today. Images are available on the Hugging Face Docker Hub.\n\n```bash\ndocker pull huggingface/optimum-nvidia\n```\n\n## Building from source\n<!---\nCurrently, TRT LLM is built and run with Docker, so we should wait until pip installation is available;\nIdeally the user doesn't need to use docker at all to build from source, they should be able to run something like\n`git clone [...] && pip install -e optimum-nvidia`\n-->\n\nInstead of using the pre-built docker container, you can build Optimum-NVIDIA from source:\n```bash\nTARGET_SM=\"90-real;89-real\"\ngit clone --recursive --depth=1 https://github.com/huggingface/optimum-nvidia.git\ncd optimum-nvidia/third-party/tensorrt-llm\nmake -C docker release_build CUDA_ARCHS=$TARGET_SM\ncd ../.. && docker build -t <organisation_name/image_name>:<version> -f docker/Dockerfile .\n```\n\n<!-- \n```bash\ngit clone git@github.com:huggingface/optimum-nvidia.git\ncd optimum-nvidia\ndocker build Dockerfile\ndocker run optimum-nvidia\n``` -->\n\n# Quickstart Guide\n## Pipelines\n\nHugging Face pipelines provide a simple yet powerful abstraction to quickly set up inference. If you already have a pipeline from transformers, you can unlock the performance benefits of Optimum-NVIDIA by just changing one line.\n\n```diff\n- from transformers.pipelines import pipeline\n+ from optimum.nvidia.pipelines import pipeline\n\npipe = pipeline('text-generation', 'meta-llama/Llama-2-7b-chat-hf', use_fp8=True)\npipe(\"Describe a real-world application of AI in sustainable energy.\")\n```\n\n## Generate\n\nIf you want control over advanced features like quantization and token selection strategies, we recommend using the `generate()` API. Just like with `pipelines`, switching from existing transformers code is super simple.\n\n```diff\n- from transformers import AutoModelForCausalLM\n+ from optimum.nvidia import AutoModelForCausalLM\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Llama-2-7b-chat-hf\", padding_side=\"left\")\n\nmodel = AutoModelForCausalLM.from_pretrained(\n \"meta-llama/Llama-2-7b-chat-hf\",\n+ use_fp8=True, \n)\n\nmodel_inputs = tokenizer([\"How is autonomous vehicle technology transforming the future of transportation and urban planning?\"], return_tensors=\"pt\").to(\"cuda\")\n\ngenerated_ids = model.generate(\n **model_inputs, \n top_k=40, \n top_p=0.7, \n repetition_penalty=10,\n)\n\ntokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n```\n\nTo learn more about text generation with LLMs, check out [this guide](https://huggingface.co/docs/transformers/llm_tutorial)!\n\n<!-- For more details, read our [documentation](https://huggingface.com/docs/optimum/nvidia/index). -->\n\n# Support Matrix\nWe test Optimum-NVIDIA on 4090, L40S, and H100 Tensor Core GPUs, though it is expected to work on any GPU based on the following architectures: \n* Turing (with experimental support for T4 / RTX Quadro x000)\n* Ampere (A100/A30 are supported. Experimental support for A10, A40, RTX Ax000)\n* Hopper\n* Ada-Lovelace\n\nNote that FP8 support is only available on GPUs based on Hopper and Ada-Lovelace architectures.\n\nOptimum-NVIDIA works on Linux will support Windows soon.\n\nOptimum-NVIDIA currently accelerates text-generation with LLaMAForCausalLM, and we are actively working to expand support to include more model architectures and tasks.\n\n<!-- Optimum-NVIDIA supports the following model architectures and tasks:\n\n| Model | Tasks |\n| :---- | :---- |\n| Gemma | TextGeneration |\n| Llama | TextGeneration |\n| Mistral | TextGeneration |\n| Additional Models | Coming soon | -->\n\n# Contributing\n\nCheck out our [Contributing Guide](./CONTRIBUTING.md)\n",
"bugtrack_url": null,
"license": "Apache/2.0",
"summary": "Optimum Nvidia is the interface between the Hugging Face Transformers and NVIDIA GPUs. \"",
"version": "0.1.0b6",
"project_urls": {
"Homepage": "https://huggingface.co/hardware/nvidia",
"Issues": "https://github.com/huggingface/optimum-nvidia/issues",
"Repository": "https://github.com/huggingface/optimum-nvidia"
},
"split_keywords": [
"transformers",
" neural-network",
" inference",
" nvidia",
" tensorrt",
" ampere",
" hopper"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d615fc0918621f3fcc1f34ac856684312b2839b8fe4faa296a89992cd3887186",
"md5": "c2a80dd896777db0e48c581cb72aeaff",
"sha256": "6aa8fb61be830a84df3a9bf3d08da174df69dc435b8589480770d9895354376e"
},
"downloads": -1,
"filename": "optimum_nvidia-0.1.0b6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c2a80dd896777db0e48c581cb72aeaff",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 73735,
"upload_time": "2024-04-11T21:13:36",
"upload_time_iso_8601": "2024-04-11T21:13:36.642294Z",
"url": "https://files.pythonhosted.org/packages/d6/15/fc0918621f3fcc1f34ac856684312b2839b8fe4faa296a89992cd3887186/optimum_nvidia-0.1.0b6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6b82f25f93ddba8a18d5aff8dc2b15ebf7aaf98d1e33823305b36ab49af0ed45",
"md5": "22e7e14c9941459aa6b056492beb47a4",
"sha256": "564f8cfcfae1582bc7c2bf5ed55adea9ee9663e4cfbf21272243c63c891e5e8f"
},
"downloads": -1,
"filename": "optimum-nvidia-0.1.0b6.tar.gz",
"has_sig": false,
"md5_digest": "22e7e14c9941459aa6b056492beb47a4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 53727,
"upload_time": "2024-04-11T21:13:38",
"upload_time_iso_8601": "2024-04-11T21:13:38.539937Z",
"url": "https://files.pythonhosted.org/packages/6b/82/f25f93ddba8a18d5aff8dc2b15ebf7aaf98d1e33823305b36ab49af0ed45/optimum-nvidia-0.1.0b6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-11 21:13:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "huggingface",
"github_project": "optimum-nvidia",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "optimum-nvidia"
}
JSON
