# mistral.rs
`mistralrs` is a Python package which provides an easy to use API for `mistral.rs`.
## Example
More examples can be found [here](https://github.com/EricLBuehler/mistral.rs/tree/master/examples/python)!
```python
from mistralrs import Runner, Which, ChatCompletionRequest
runner = Runner(
which=Which.Plain(
model_id="microsoft/Phi-3.5-mini-instruct",
),
in_situ_quant="Q4K",
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="mistral",
messages=[
{"role": "user", "content": "Tell me a story about the Rust type system."}
],
max_tokens=256,
presence_penalty=1.0,
top_p=0.1,
temperature=0.1,
)
)
print(res.choices[0].message.content)
print(res.usage)
```
Please find [API docs here](API.md) and the type stubs [here](mistralrs.pyi), which are another great form of documentation.
We also provide [a cookbook here](../examples/python/cookbook.ipynb)!
Raw data
{
"_id": null,
"home_page": "https://github.com/EricLBuehler/mistral.rs",
"name": "mistralrs-cuda",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "machine-learning",
"author": "Eric Buehler",
"author_email": null,
"download_url": null,
"platform": null,
"description": "# mistral.rs\n\n`mistralrs` is a Python package which provides an easy to use API for `mistral.rs`. \n\n## Example\nMore examples can be found [here](https://github.com/EricLBuehler/mistral.rs/tree/master/examples/python)!\n\n```python\nfrom mistralrs import Runner, Which, ChatCompletionRequest\n\nrunner = Runner(\n which=Which.Plain(\n model_id=\"microsoft/Phi-3.5-mini-instruct\",\n ),\n in_situ_quant=\"Q4K\",\n)\n\nres = runner.send_chat_completion_request(\n ChatCompletionRequest(\n model=\"mistral\",\n messages=[\n {\"role\": \"user\", \"content\": \"Tell me a story about the Rust type system.\"}\n ],\n max_tokens=256,\n presence_penalty=1.0,\n top_p=0.1,\n temperature=0.1,\n )\n)\nprint(res.choices[0].message.content)\nprint(res.usage)\n```\n\nPlease find [API docs here](API.md) and the type stubs [here](mistralrs.pyi), which are another great form of documentation.\n\nWe also provide [a cookbook here](../examples/python/cookbook.ipynb)!\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Fast and easy LLM serving.",
"version": "0.3.4",
"project_urls": {
"Homepage": "https://github.com/EricLBuehler/mistral.rs",
"Source Code": "https://github.com/EricLBuehler/mistral.rs"
},
"split_keywords": [
"machine-learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5b43d45e2453716b726796a33dba946f740c2a61503080d8933f57862f2b737d",
"md5": "ebb9ca0fb101a01885a196cc3eb15adf",
"sha256": "f8e9ed25c98f3650dc4c36498a70eb4439dd2ac269f20508affe4e6afdbfbb34"
},
"downloads": -1,
"filename": "mistralrs_cuda-0.3.4-cp310-cp310-manylinux_2_39_x86_64.whl",
"has_sig": false,
"md5_digest": "ebb9ca0fb101a01885a196cc3eb15adf",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.10",
"size": 24150116,
"upload_time": "2024-11-28T19:17:57",
"upload_time_iso_8601": "2024-11-28T19:17:57.951201Z",
"url": "https://files.pythonhosted.org/packages/5b/43/d45e2453716b726796a33dba946f740c2a61503080d8933f57862f2b737d/mistralrs_cuda-0.3.4-cp310-cp310-manylinux_2_39_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "951efce60728cee1416dc27a7ed2013a1020ce6406e118ed5c8e5e41dbd6adf9",
"md5": "293d91770890e12e6b6c0052d114009c",
"sha256": "41139400d2065240d83655c133086641ea125e51ce4a00285b46772d926d78cc"
},
"downloads": -1,
"filename": "mistralrs_cuda-0.3.4-cp310-none-win_amd64.whl",
"has_sig": false,
"md5_digest": "293d91770890e12e6b6c0052d114009c",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.10",
"size": 17050374,
"upload_time": "2024-11-28T19:18:07",
"upload_time_iso_8601": "2024-11-28T19:18:07.258856Z",
"url": "https://files.pythonhosted.org/packages/95/1e/fce60728cee1416dc27a7ed2013a1020ce6406e118ed5c8e5e41dbd6adf9/mistralrs_cuda-0.3.4-cp310-none-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "5c45b77d213f6689395a2b9efc87eb4ab965ab7cd1634d4ebbf1c2e5885a7728",
"md5": "0fc37ad6c7d3bc28c310a0de17cb542c",
"sha256": "36330f276f5b2e57449ca03b8f8879e27ec3385d203c78074a58e7589b1055cf"
},
"downloads": -1,
"filename": "mistralrs_cuda-0.3.4-cp311-cp311-manylinux_2_39_x86_64.whl",
"has_sig": false,
"md5_digest": "0fc37ad6c7d3bc28c310a0de17cb542c",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.10",
"size": 24152755,
"upload_time": "2024-11-28T19:18:18",
"upload_time_iso_8601": "2024-11-28T19:18:18.698557Z",
"url": "https://files.pythonhosted.org/packages/5c/45/b77d213f6689395a2b9efc87eb4ab965ab7cd1634d4ebbf1c2e5885a7728/mistralrs_cuda-0.3.4-cp311-cp311-manylinux_2_39_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "546cf03f898a181a9b417da565877abf9e6b9cb1a92f52a26ef560315132f22c",
"md5": "a4bb74dc3d88a0645a627ceefefda2e6",
"sha256": "d284bb876add040193fe833c3169fa09d90f562ad5ca161b1f781ab1be00ad52"
},
"downloads": -1,
"filename": "mistralrs_cuda-0.3.4-cp311-none-win_amd64.whl",
"has_sig": false,
"md5_digest": "a4bb74dc3d88a0645a627ceefefda2e6",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.10",
"size": 17050967,
"upload_time": "2024-11-28T19:18:27",
"upload_time_iso_8601": "2024-11-28T19:18:27.676608Z",
"url": "https://files.pythonhosted.org/packages/54/6c/f03f898a181a9b417da565877abf9e6b9cb1a92f52a26ef560315132f22c/mistralrs_cuda-0.3.4-cp311-none-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "af4d7f3c45bbbcce12eb3ba8483ae1c9105fcb46b409f47aefb22ba910badd59",
"md5": "779cc84a277eb7259a5446204425d8b7",
"sha256": "cb1c7c3242083a152ab7d0bf7e866fe8ea0d9474ff83088ff4415ee748fbd50b"
},
"downloads": -1,
"filename": "mistralrs_cuda-0.3.4-cp312-cp312-manylinux_2_39_x86_64.whl",
"has_sig": false,
"md5_digest": "779cc84a277eb7259a5446204425d8b7",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.10",
"size": 24161352,
"upload_time": "2024-11-28T19:18:38",
"upload_time_iso_8601": "2024-11-28T19:18:38.705314Z",
"url": "https://files.pythonhosted.org/packages/af/4d/7f3c45bbbcce12eb3ba8483ae1c9105fcb46b409f47aefb22ba910badd59/mistralrs_cuda-0.3.4-cp312-cp312-manylinux_2_39_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ec3cc8fa08c3d385291e25d0c3759da14ca9099c9a191ea9aa41b8ffabe94eb0",
"md5": "0618458f867247c39894fba78e27a8ed",
"sha256": "b1d6fdc86ac5125f3c95f77c36918a331c98301386fe69bbc0392c7a501ea143"
},
"downloads": -1,
"filename": "mistralrs_cuda-0.3.4-cp312-none-win_amd64.whl",
"has_sig": false,
"md5_digest": "0618458f867247c39894fba78e27a8ed",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.10",
"size": 17040890,
"upload_time": "2024-11-28T19:18:48",
"upload_time_iso_8601": "2024-11-28T19:18:48.131551Z",
"url": "https://files.pythonhosted.org/packages/ec/3c/c8fa08c3d385291e25d0c3759da14ca9099c9a191ea9aa41b8ffabe94eb0/mistralrs_cuda-0.3.4-cp312-none-win_amd64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-28 19:17:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "EricLBuehler",
"github_project": "mistral.rs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "mistralrs-cuda"
}
JSON
