# mistral.rs PyO3 Bindings: `mistralrs`
`mistralrs` is a Python package which provides an API for `mistral.rs`. We build `mistralrs` with the `maturin` build manager.
## Installation from PyPi
0) Install Rust: https://rustup.rs/
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
```
1) `mistralrs` depends on the `openssl` library.
To install it on Ubuntu:
```
sudo apt install libssl-dev
sudo apt install pkg-config
```
2) Install it!
- CUDA
`pip install mistralrs-cuda`
- Metal
`pip install mistralrs-metal`
- Apple Accelerate
`pip install mistralrs-accelerate`
- Intel MKL
`pip install mistralrs-mkl`
- Without accelerators
`pip install mistralrs`
All installations will install the `mistralrs` package. The suffix on the package installed by `pip` only controls the feature activation.
## Installation from source
1) Install required packages
- `openssl` (ex., `sudo apt install libssl-dev`)
- `pkg-config` (ex., `sudo apt install pkg-config`)
2) Install Rust: https://rustup.rs/
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
```
3) Set HF token correctly (skip if already set or your model is not gated, or if you want to use the `token_source` parameters in Python or the command line.)
```bash
mkdir ~/.cache/huggingface
touch ~/.cache/huggingface/token
echo <HF_TOKEN_HERE> > ~/.cache/huggingface/token
```
4) Download the code
```bash
git clone https://github.com/EricLBuehler/mistral.rs.git
cd mistral.rs
```
5) `cd` into the correct directory for building `mistralrs`:
`cd mistralrs-pyo3`
6) Install `maturin`, our Rust + Python build system:
Maturin requires a Python virtual environment such as `venv` or `conda` to be active. The `mistralrs` package will be installed into that
environment.
```
pip install maturin[patchelf]
```
7) Install `mistralrs`
Install `mistralrs` by executing the following in this directory where [features](../README.md#supported-accelerators) such as `cuda` or `flash-attn` may be specified with the `--features` argument just like they would be for `cargo run`.
The base build command is:
```bash
maturin develop -r
```
- To build for CUDA:
```bash
maturin develop -r --features cuda
```
- To build for CUDA with flash attention:
```bash
maturin develop -r --features "cuda flash-attn"
```
- To build for Metal:
```bash
maturin develop -r --features metal
```
- To build for Accelerate:
```bash
maturin develop -r --features accelerate
```
- To build for MKL:
```bash
maturin develop -r --features mkl
```
Please find [API docs here](API.md) and the type stubs [here](mistralrs.pyi), which are another great form of documentation.
We also provide [a cookbook here](../examples/python/cookbook.ipynb)!
## Example
```python
from mistralrs import ModelKind, MistralLoader, ChatCompletionRequest
kind = ModelKind.QuantizedGGUF
loader = MistralLoader(
model_id="mistralai/Mistral-7B-Instruct-v0.1",
kind=kind,
no_kv_cache=False,
repeat_last_n=64,
quantized_model_id="TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
quantized_filename="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
)
runner = loader.load()
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="mistral",
messages=[
{"role": "user", "content": "Tell me a story about the Rust type system."}
],
max_tokens=256,
frequency_penalty=1.0,
top_p=0.1,
temperature=0.1,
)
)
print(res)
```
Raw data
{
"_id": null,
"home_page": "https://github.com/EricLBuehler/mistral.rs",
"name": "mistralrs",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "machine-learning",
"author": "Eric Buehler",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/4b/aa/2362b3852bc98c119268e4306a1b28f9d70ec7c6286b0cb460098bd9f156/mistralrs-0.1.8.tar.gz",
"platform": null,
"description": "# mistral.rs PyO3 Bindings: `mistralrs`\n\n`mistralrs` is a Python package which provides an API for `mistral.rs`. We build `mistralrs` with the `maturin` build manager.\n\n## Installation from PyPi\n0) Install Rust: https://rustup.rs/\n ```bash\n curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n source $HOME/.cargo/env\n ```\n\n1) `mistralrs` depends on the `openssl` library.\n\nTo install it on Ubuntu:\n```\nsudo apt install libssl-dev\nsudo apt install pkg-config\n```\n\n2) Install it!\n\n- CUDA\n\n `pip install mistralrs-cuda`\n- Metal\n\n `pip install mistralrs-metal`\n- Apple Accelerate\n\n `pip install mistralrs-accelerate`\n- Intel MKL\n\n `pip install mistralrs-mkl`\n- Without accelerators\n\n `pip install mistralrs`\n\nAll installations will install the `mistralrs` package. The suffix on the package installed by `pip` only controls the feature activation.\n\n## Installation from source\n1) Install required packages\n - `openssl` (ex., `sudo apt install libssl-dev`)\n - `pkg-config` (ex., `sudo apt install pkg-config`)\n\n2) Install Rust: https://rustup.rs/\n ```bash\n curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n source $HOME/.cargo/env\n ```\n\n3) Set HF token correctly (skip if already set or your model is not gated, or if you want to use the `token_source` parameters in Python or the command line.)\n ```bash\n mkdir ~/.cache/huggingface\n touch ~/.cache/huggingface/token\n echo <HF_TOKEN_HERE> > ~/.cache/huggingface/token\n ```\n\n4) Download the code\n ```bash\n git clone https://github.com/EricLBuehler/mistral.rs.git\n cd mistral.rs\n ```\n\n5) `cd` into the correct directory for building `mistralrs`:\n `cd mistralrs-pyo3`\n\n6) Install `maturin`, our Rust + Python build system:\n Maturin requires a Python virtual environment such as `venv` or `conda` to be active. The `mistralrs` package will be installed into that\n environment.\n ```\n pip install maturin[patchelf]\n ```\n\n7) Install `mistralrs`\n Install `mistralrs` by executing the following in this directory where [features](../README.md#supported-accelerators) such as `cuda` or `flash-attn` may be specified with the `--features` argument just like they would be for `cargo run`.\n\n The base build command is:\n ```bash\n maturin develop -r\n ```\n\n - To build for CUDA:\n \n ```bash\n maturin develop -r --features cuda\n ```\n \n - To build for CUDA with flash attention:\n \n ```bash\n maturin develop -r --features \"cuda flash-attn\"\n ```\n\n - To build for Metal: \n\n ```bash\n maturin develop -r --features metal\n ```\n\n - To build for Accelerate: \n \n ```bash\n maturin develop -r --features accelerate\n ```\n\n - To build for MKL: \n \n ```bash\n maturin develop -r --features mkl\n ```\n\nPlease find [API docs here](API.md) and the type stubs [here](mistralrs.pyi), which are another great form of documentation.\n\nWe also provide [a cookbook here](../examples/python/cookbook.ipynb)!\n\n## Example\n```python\nfrom mistralrs import ModelKind, MistralLoader, ChatCompletionRequest\n\nkind = ModelKind.QuantizedGGUF\nloader = MistralLoader(\n model_id=\"mistralai/Mistral-7B-Instruct-v0.1\",\n kind=kind,\n no_kv_cache=False,\n repeat_last_n=64,\n quantized_model_id=\"TheBloke/Mistral-7B-Instruct-v0.1-GGUF\",\n quantized_filename=\"mistral-7b-instruct-v0.1.Q4_K_M.gguf\",\n)\nrunner = loader.load()\nres = runner.send_chat_completion_request(\n ChatCompletionRequest(\n model=\"mistral\",\n messages=[\n {\"role\": \"user\", \"content\": \"Tell me a story about the Rust type system.\"}\n ],\n max_tokens=256,\n frequency_penalty=1.0,\n top_p=0.1,\n temperature=0.1,\n )\n)\nprint(res)\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Fast and easy LLM serving.",
"version": "0.1.8",
"project_urls": {
"Homepage": "https://github.com/EricLBuehler/mistral.rs",
"Source Code": "https://github.com/EricLBuehler/mistral.rs"
},
"split_keywords": [
"machine-learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4baa2362b3852bc98c119268e4306a1b28f9d70ec7c6286b0cb460098bd9f156",
"md5": "7bda68055a93dba9cb0465e170e84d9a",
"sha256": "da370c5f57678fa87d41400bc66978cd771c38e2233086805ad678c6052bf7b0"
},
"downloads": -1,
"filename": "mistralrs-0.1.8.tar.gz",
"has_sig": false,
"md5_digest": "7bda68055a93dba9cb0465e170e84d9a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 163107,
"upload_time": "2024-05-16T16:33:36",
"upload_time_iso_8601": "2024-05-16T16:33:36.058666Z",
"url": "https://files.pythonhosted.org/packages/4b/aa/2362b3852bc98c119268e4306a1b28f9d70ec7c6286b0cb460098bd9f156/mistralrs-0.1.8.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-16 16:33:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "EricLBuehler",
"github_project": "mistral.rs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "mistralrs"
}
JSON
