# mistral.rs
`mistralrs` is a Python package which provides an easy to use API for `mistral.rs`.
## Example
More examples can be found [here](https://github.com/EricLBuehler/mistral.rs/tree/master/examples/python)!
```python
from mistralrs import Runner, Which, ChatCompletionRequest
runner = Runner(
which=Which.Plain(
model_id="microsoft/Phi-3.5-mini-instruct",
),
in_situ_quant="Q4K",
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="mistral",
messages=[
{"role": "user", "content": "Tell me a story about the Rust type system."}
],
max_tokens=256,
presence_penalty=1.0,
top_p=0.1,
temperature=0.1,
)
)
print(res.choices[0].message.content)
print(res.usage)
```
Please find [API docs here](API.md) and the type stubs [here](mistralrs.pyi), which are another great form of documentation.
We also provide [a cookbook here](../examples/python/cookbook.ipynb)!
Raw data
{
"_id": null,
"home_page": "https://github.com/EricLBuehler/mistral.rs",
"name": "mistralrs",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "machine-learning",
"author": "Eric Buehler",
"author_email": null,
"download_url": null,
"platform": null,
"description": "# mistral.rs\n\n`mistralrs` is a Python package which provides an easy to use API for `mistral.rs`. \n\n## Example\nMore examples can be found [here](https://github.com/EricLBuehler/mistral.rs/tree/master/examples/python)!\n\n```python\nfrom mistralrs import Runner, Which, ChatCompletionRequest\n\nrunner = Runner(\n which=Which.Plain(\n model_id=\"microsoft/Phi-3.5-mini-instruct\",\n ),\n in_situ_quant=\"Q4K\",\n)\n\nres = runner.send_chat_completion_request(\n ChatCompletionRequest(\n model=\"mistral\",\n messages=[\n {\"role\": \"user\", \"content\": \"Tell me a story about the Rust type system.\"}\n ],\n max_tokens=256,\n presence_penalty=1.0,\n top_p=0.1,\n temperature=0.1,\n )\n)\nprint(res.choices[0].message.content)\nprint(res.usage)\n```\n\nPlease find [API docs here](API.md) and the type stubs [here](mistralrs.pyi), which are another great form of documentation.\n\nWe also provide [a cookbook here](../examples/python/cookbook.ipynb)!\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Fast and easy LLM serving.",
"version": "0.3.4",
"project_urls": {
"Homepage": "https://github.com/EricLBuehler/mistral.rs",
"Source Code": "https://github.com/EricLBuehler/mistral.rs"
},
"split_keywords": [
"machine-learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9153f37e5fcc54648cf2bcbbdb6d683e2e595975c841e9fee80b4dd65ac780dc",
"md5": "74bd44211156ccc58b70d906d8c0f282",
"sha256": "9504782ecec12faf75f1ebb097f6e586d22c67cf80c9b3376c89748f925ed8f7"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp310-cp310-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "74bd44211156ccc58b70d906d8c0f282",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.10",
"size": 10881111,
"upload_time": "2024-11-28T19:19:14",
"upload_time_iso_8601": "2024-11-28T19:19:14.738469Z",
"url": "https://files.pythonhosted.org/packages/91/53/f37e5fcc54648cf2bcbbdb6d683e2e595975c841e9fee80b4dd65ac780dc/mistralrs-0.3.4-cp310-cp310-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0922809bcd4e763ac21308776f7832c9f6fc323ed8c11fc6b46df22a032c9c8a",
"md5": "efacbe7e4441a330a9b06e28ab62ca07",
"sha256": "6360b20daf20f62c7698d94acef31f037598bdf4566fa0d684bb4a1de62b435d"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "efacbe7e4441a330a9b06e28ab62ca07",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.10",
"size": 14187638,
"upload_time": "2024-11-28T19:19:22",
"upload_time_iso_8601": "2024-11-28T19:19:22.403963Z",
"url": "https://files.pythonhosted.org/packages/09/22/809bcd4e763ac21308776f7832c9f6fc323ed8c11fc6b46df22a032c9c8a/mistralrs-0.3.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9573b41e49982730aa6c015daf8100bf42958d46369d739f1bcc978a046be24d",
"md5": "7fd0025a07bdb353719c0e1cc3b6462f",
"sha256": "488c9d3ab0ea3da4e88c94a81b917a64b60e28c417e595355a4323a5addd2bd8"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp310-cp310-manylinux_2_39_x86_64.whl",
"has_sig": false,
"md5_digest": "7fd0025a07bdb353719c0e1cc3b6462f",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.10",
"size": 15189124,
"upload_time": "2024-11-28T19:19:29",
"upload_time_iso_8601": "2024-11-28T19:19:29.919690Z",
"url": "https://files.pythonhosted.org/packages/95/73/b41e49982730aa6c015daf8100bf42958d46369d739f1bcc978a046be24d/mistralrs-0.3.4-cp310-cp310-manylinux_2_39_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6d8478a7b4812b0f6a0c8f8b9215109ade0eb3895eca7a4b6a59e817eb8be30c",
"md5": "ba8b8c9895f34791640efcc3667fa344",
"sha256": "847c8f0c3af2a8a47ab0084bbac2eb91e42f98ea44441444a7caa0315827f83a"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp310-none-win_amd64.whl",
"has_sig": false,
"md5_digest": "ba8b8c9895f34791640efcc3667fa344",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.10",
"size": 10599324,
"upload_time": "2024-11-28T19:19:36",
"upload_time_iso_8601": "2024-11-28T19:19:36.391902Z",
"url": "https://files.pythonhosted.org/packages/6d/84/78a7b4812b0f6a0c8f8b9215109ade0eb3895eca7a4b6a59e817eb8be30c/mistralrs-0.3.4-cp310-none-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ff9ff452cc9e91a471530c6c87ac2e20bd7fa744a830f4851acae58ea6b54459",
"md5": "e6dedf694f1d75977295f0a9a6fee2c5",
"sha256": "a51f855aafc4d721ef52219644f11c0c2c731e5bfe8c1f3a6eb6b6bb5dd2ec27"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp311-cp311-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "e6dedf694f1d75977295f0a9a6fee2c5",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.10",
"size": 10879634,
"upload_time": "2024-11-28T19:19:42",
"upload_time_iso_8601": "2024-11-28T19:19:42.654759Z",
"url": "https://files.pythonhosted.org/packages/ff/9f/f452cc9e91a471530c6c87ac2e20bd7fa744a830f4851acae58ea6b54459/mistralrs-0.3.4-cp311-cp311-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2ac18fc60bb01354611736d3bc33acb7466af986f156f94126be485485eec0d8",
"md5": "4d17fc9e499d268b5a22e1645c3ce22c",
"sha256": "68c9cc3530340954175f1b8a67d22ac9968a37bb91be67c177265a990c842ba3"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "4d17fc9e499d268b5a22e1645c3ce22c",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.10",
"size": 14183359,
"upload_time": "2024-11-28T19:19:50",
"upload_time_iso_8601": "2024-11-28T19:19:50.644135Z",
"url": "https://files.pythonhosted.org/packages/2a/c1/8fc60bb01354611736d3bc33acb7466af986f156f94126be485485eec0d8/mistralrs-0.3.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b8bd76be83905f224ac48b7309020194d9c65c410faaff1438794fcae5820bd0",
"md5": "bd07f79654f3864ce368869026209573",
"sha256": "4b1204c21585dc460f571ad3779d8323120d91cd21b4503ac9c8d3fa53d72eda"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp311-cp311-manylinux_2_39_x86_64.whl",
"has_sig": false,
"md5_digest": "bd07f79654f3864ce368869026209573",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.10",
"size": 15194006,
"upload_time": "2024-11-28T19:19:58",
"upload_time_iso_8601": "2024-11-28T19:19:58.321422Z",
"url": "https://files.pythonhosted.org/packages/b8/bd/76be83905f224ac48b7309020194d9c65c410faaff1438794fcae5820bd0/mistralrs-0.3.4-cp311-cp311-manylinux_2_39_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "cabde322565da6331d4c9c276454b312699320039a02a7c70110c012e72720ce",
"md5": "d92b9f8fc89d3b4399eb01f0a78e7217",
"sha256": "1fd9789a7432962e7675837fd31e0429977dfd802ff7ea491235bcd3f6c4435c"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp311-none-win_amd64.whl",
"has_sig": false,
"md5_digest": "d92b9f8fc89d3b4399eb01f0a78e7217",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.10",
"size": 10599222,
"upload_time": "2024-11-28T19:20:04",
"upload_time_iso_8601": "2024-11-28T19:20:04.650944Z",
"url": "https://files.pythonhosted.org/packages/ca/bd/e322565da6331d4c9c276454b312699320039a02a7c70110c012e72720ce/mistralrs-0.3.4-cp311-none-win_amd64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6e96c642ddfca1315126fb9c505f017f3d20ebd091a2c4fde402c6275560fa03",
"md5": "e549c1e8c0bb296a0b3e422b3220c83d",
"sha256": "52051d8cdd5061193de3ea270790ea41202c99aa6abe898f27efa2ac5a0d1c2d"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp312-cp312-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "e549c1e8c0bb296a0b3e422b3220c83d",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.10",
"size": 10871165,
"upload_time": "2024-11-28T19:20:11",
"upload_time_iso_8601": "2024-11-28T19:20:11.068307Z",
"url": "https://files.pythonhosted.org/packages/6e/96/c642ddfca1315126fb9c505f017f3d20ebd091a2c4fde402c6275560fa03/mistralrs-0.3.4-cp312-cp312-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "54a7d8b214e7e658a23feb9de9491da8cb69050224da55c238c3a1da1b205441",
"md5": "cc9d1be555319389bd4ff56b9d6dd1e2",
"sha256": "e6e7adb1488760a4c73a9f9842752ff3c0522f240f00336cfc3a98fa48ef171b"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"has_sig": false,
"md5_digest": "cc9d1be555319389bd4ff56b9d6dd1e2",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.10",
"size": 14179982,
"upload_time": "2024-11-28T19:20:18",
"upload_time_iso_8601": "2024-11-28T19:20:18.910329Z",
"url": "https://files.pythonhosted.org/packages/54/a7/d8b214e7e658a23feb9de9491da8cb69050224da55c238c3a1da1b205441/mistralrs-0.3.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "6cbde5045f13eb65431e157df285012da2c6b4b96db099fa61cd900e737118d4",
"md5": "3fdee53f3db4479ba981c6dcfa2a4354",
"sha256": "8e848e8160fd04b29aec4eea65857bf148e44ad95927a9264afbc0bfb5bcbf8d"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp312-cp312-manylinux_2_39_x86_64.whl",
"has_sig": false,
"md5_digest": "3fdee53f3db4479ba981c6dcfa2a4354",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.10",
"size": 15187881,
"upload_time": "2024-11-28T19:20:27",
"upload_time_iso_8601": "2024-11-28T19:20:27.105299Z",
"url": "https://files.pythonhosted.org/packages/6c/bd/e5045f13eb65431e157df285012da2c6b4b96db099fa61cd900e737118d4/mistralrs-0.3.4-cp312-cp312-manylinux_2_39_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fddc743a1f379d9711c09e42c5bcd882c0400ef705aae9e510f6b6e8265b6b2a",
"md5": "b90583c21e42f3c80735f943d685df5e",
"sha256": "1e2e2029695b0f89900307bda41daa5c40e3f476efb94f81a760d189d232fcd7"
},
"downloads": -1,
"filename": "mistralrs-0.3.4-cp312-none-win_amd64.whl",
"has_sig": false,
"md5_digest": "b90583c21e42f3c80735f943d685df5e",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.10",
"size": 10609021,
"upload_time": "2024-11-28T19:20:33",
"upload_time_iso_8601": "2024-11-28T19:20:33.916852Z",
"url": "https://files.pythonhosted.org/packages/fd/dc/743a1f379d9711c09e42c5bcd882c0400ef705aae9e510f6b6e8265b6b2a/mistralrs-0.3.4-cp312-none-win_amd64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-28 19:19:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "EricLBuehler",
"github_project": "mistral.rs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "mistralrs"
}
JSON
