# mistral.rs
`mistralrs` is a Python package which provides an easy to use API for `mistral.rs`.
## Example
More examples can be found [here](https://github.com/EricLBuehler/mistral.rs/tree/master/examples/python)!
```python
from mistralrs import Runner, Which, ChatCompletionRequest
runner = Runner(
which=Which.Plain(
model_id="microsoft/Phi-3.5-mini-instruct",
),
in_situ_quant="Q4K",
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="mistral",
messages=[
{"role": "user", "content": "Tell me a story about the Rust type system."}
],
max_tokens=256,
presence_penalty=1.0,
top_p=0.1,
temperature=0.1,
)
)
print(res.choices[0].message.content)
print(res.usage)
```
Please find [API docs here](API.md) and the type stubs [here](mistralrs.pyi), which are another great form of documentation.
We also provide [a cookbook here](../examples/python/cookbook.ipynb)!
Raw data
{
"_id": null,
"home_page": "https://github.com/EricLBuehler/mistral.rs",
"name": "mistralrs-accelerate",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": "machine-learning",
"author": "Eric Buehler",
"author_email": null,
"download_url": null,
"platform": null,
"description": "# mistral.rs\n\n`mistralrs` is a Python package which provides an easy to use API for `mistral.rs`. \n\n## Example\nMore examples can be found [here](https://github.com/EricLBuehler/mistral.rs/tree/master/examples/python)!\n\n```python\nfrom mistralrs import Runner, Which, ChatCompletionRequest\n\nrunner = Runner(\n which=Which.Plain(\n model_id=\"microsoft/Phi-3.5-mini-instruct\",\n ),\n in_situ_quant=\"Q4K\",\n)\n\nres = runner.send_chat_completion_request(\n ChatCompletionRequest(\n model=\"mistral\",\n messages=[\n {\"role\": \"user\", \"content\": \"Tell me a story about the Rust type system.\"}\n ],\n max_tokens=256,\n presence_penalty=1.0,\n top_p=0.1,\n temperature=0.1,\n )\n)\nprint(res.choices[0].message.content)\nprint(res.usage)\n```\n\nPlease find [API docs here](API.md) and the type stubs [here](mistralrs.pyi), which are another great form of documentation.\n\nWe also provide [a cookbook here](../examples/python/cookbook.ipynb)!\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Fast and easy LLM serving.",
"version": "0.3.4",
"project_urls": {
"Homepage": "https://github.com/EricLBuehler/mistral.rs",
"Source Code": "https://github.com/EricLBuehler/mistral.rs"
},
"split_keywords": [
"machine-learning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9b8749d813199d860ff8b498f9d4691f7b411e60677b5abc5f012176c674bdc8",
"md5": "4b9237c52ad3826b70ede56fc84b5dff",
"sha256": "24497f1ce35a0e63e1f9f20a1f9411335e287d4fed0c5b55bea1de23835dfa01"
},
"downloads": -1,
"filename": "mistralrs_accelerate-0.3.4-cp310-cp310-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "4b9237c52ad3826b70ede56fc84b5dff",
"packagetype": "bdist_wheel",
"python_version": "cp310",
"requires_python": ">=3.10",
"size": 10809788,
"upload_time": "2024-11-28T18:40:34",
"upload_time_iso_8601": "2024-11-28T18:40:34.451476Z",
"url": "https://files.pythonhosted.org/packages/9b/87/49d813199d860ff8b498f9d4691f7b411e60677b5abc5f012176c674bdc8/mistralrs_accelerate-0.3.4-cp310-cp310-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "40c42dadf61c9872a1472ae84627239e4bf8284f352f464939ad617d0930dd03",
"md5": "6dbad7c7a6dacaa469e19cc62f927c59",
"sha256": "69b8e4ea418c0d555e785d8f2b7193dbdf00ff6eb522ef6555cbcf154b84c86b"
},
"downloads": -1,
"filename": "mistralrs_accelerate-0.3.4-cp311-cp311-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "6dbad7c7a6dacaa469e19cc62f927c59",
"packagetype": "bdist_wheel",
"python_version": "cp311",
"requires_python": ">=3.10",
"size": 10810262,
"upload_time": "2024-11-28T18:40:40",
"upload_time_iso_8601": "2024-11-28T18:40:40.703443Z",
"url": "https://files.pythonhosted.org/packages/40/c4/2dadf61c9872a1472ae84627239e4bf8284f352f464939ad617d0930dd03/mistralrs_accelerate-0.3.4-cp311-cp311-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "11031f8938fe4939ae8a07626c82d5967ca36b0e7240d3850b70af976702dbf3",
"md5": "4775c2fbc95b197797a3719e4e8a7b90",
"sha256": "98bc9c376f9655e6daa8568173da3bc73d60dc86f2d06cbf2a65fd5697de48f8"
},
"downloads": -1,
"filename": "mistralrs_accelerate-0.3.4-cp312-cp312-macosx_11_0_arm64.whl",
"has_sig": false,
"md5_digest": "4775c2fbc95b197797a3719e4e8a7b90",
"packagetype": "bdist_wheel",
"python_version": "cp312",
"requires_python": ">=3.10",
"size": 10811503,
"upload_time": "2024-11-28T18:40:47",
"upload_time_iso_8601": "2024-11-28T18:40:47.289346Z",
"url": "https://files.pythonhosted.org/packages/11/03/1f8938fe4939ae8a07626c82d5967ca36b0e7240d3850b70af976702dbf3/mistralrs_accelerate-0.3.4-cp312-cp312-macosx_11_0_arm64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-28 18:40:34",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "EricLBuehler",
"github_project": "mistral.rs",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "mistralrs-accelerate"
}
JSON
