mistralrs-cuda


Namemistralrs-cuda JSON
Version 0.3.4 PyPI version JSON
download
home_pagehttps://github.com/EricLBuehler/mistral.rs
SummaryFast and easy LLM serving.
upload_time2024-11-28 19:17:57
maintainerNone
docs_urlNone
authorEric Buehler
requires_python>=3.10
licenseMIT
keywords machine-learning
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            # mistral.rs

`mistralrs` is a Python package which provides an easy to use API for `mistral.rs`. 

## Example
More examples can be found [here](https://github.com/EricLBuehler/mistral.rs/tree/master/examples/python)!

```python
from mistralrs import Runner, Which, ChatCompletionRequest

runner = Runner(
    which=Which.Plain(
        model_id="microsoft/Phi-3.5-mini-instruct",
    ),
    in_situ_quant="Q4K",
)

res = runner.send_chat_completion_request(
    ChatCompletionRequest(
        model="mistral",
        messages=[
            {"role": "user", "content": "Tell me a story about the Rust type system."}
        ],
        max_tokens=256,
        presence_penalty=1.0,
        top_p=0.1,
        temperature=0.1,
    )
)
print(res.choices[0].message.content)
print(res.usage)
```

Please find [API docs here](API.md) and the type stubs [here](mistralrs.pyi), which are another great form of documentation.

We also provide [a cookbook here](../examples/python/cookbook.ipynb)!


            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/EricLBuehler/mistral.rs",
    "name": "mistralrs-cuda",
    "maintainer": null,
    "docs_url": null,
    "requires_python": ">=3.10",
    "maintainer_email": null,
    "keywords": "machine-learning",
    "author": "Eric Buehler",
    "author_email": null,
    "download_url": null,
    "platform": null,
    "description": "# mistral.rs\n\n`mistralrs` is a Python package which provides an easy to use API for `mistral.rs`. \n\n## Example\nMore examples can be found [here](https://github.com/EricLBuehler/mistral.rs/tree/master/examples/python)!\n\n```python\nfrom mistralrs import Runner, Which, ChatCompletionRequest\n\nrunner = Runner(\n    which=Which.Plain(\n        model_id=\"microsoft/Phi-3.5-mini-instruct\",\n    ),\n    in_situ_quant=\"Q4K\",\n)\n\nres = runner.send_chat_completion_request(\n    ChatCompletionRequest(\n        model=\"mistral\",\n        messages=[\n            {\"role\": \"user\", \"content\": \"Tell me a story about the Rust type system.\"}\n        ],\n        max_tokens=256,\n        presence_penalty=1.0,\n        top_p=0.1,\n        temperature=0.1,\n    )\n)\nprint(res.choices[0].message.content)\nprint(res.usage)\n```\n\nPlease find [API docs here](API.md) and the type stubs [here](mistralrs.pyi), which are another great form of documentation.\n\nWe also provide [a cookbook here](../examples/python/cookbook.ipynb)!\n\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Fast and easy LLM serving.",
    "version": "0.3.4",
    "project_urls": {
        "Homepage": "https://github.com/EricLBuehler/mistral.rs",
        "Source Code": "https://github.com/EricLBuehler/mistral.rs"
    },
    "split_keywords": [
        "machine-learning"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "5b43d45e2453716b726796a33dba946f740c2a61503080d8933f57862f2b737d",
                "md5": "ebb9ca0fb101a01885a196cc3eb15adf",
                "sha256": "f8e9ed25c98f3650dc4c36498a70eb4439dd2ac269f20508affe4e6afdbfbb34"
            },
            "downloads": -1,
            "filename": "mistralrs_cuda-0.3.4-cp310-cp310-manylinux_2_39_x86_64.whl",
            "has_sig": false,
            "md5_digest": "ebb9ca0fb101a01885a196cc3eb15adf",
            "packagetype": "bdist_wheel",
            "python_version": "cp310",
            "requires_python": ">=3.10",
            "size": 24150116,
            "upload_time": "2024-11-28T19:17:57",
            "upload_time_iso_8601": "2024-11-28T19:17:57.951201Z",
            "url": "https://files.pythonhosted.org/packages/5b/43/d45e2453716b726796a33dba946f740c2a61503080d8933f57862f2b737d/mistralrs_cuda-0.3.4-cp310-cp310-manylinux_2_39_x86_64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "951efce60728cee1416dc27a7ed2013a1020ce6406e118ed5c8e5e41dbd6adf9",
                "md5": "293d91770890e12e6b6c0052d114009c",
                "sha256": "41139400d2065240d83655c133086641ea125e51ce4a00285b46772d926d78cc"
            },
            "downloads": -1,
            "filename": "mistralrs_cuda-0.3.4-cp310-none-win_amd64.whl",
            "has_sig": false,
            "md5_digest": "293d91770890e12e6b6c0052d114009c",
            "packagetype": "bdist_wheel",
            "python_version": "cp310",
            "requires_python": ">=3.10",
            "size": 17050374,
            "upload_time": "2024-11-28T19:18:07",
            "upload_time_iso_8601": "2024-11-28T19:18:07.258856Z",
            "url": "https://files.pythonhosted.org/packages/95/1e/fce60728cee1416dc27a7ed2013a1020ce6406e118ed5c8e5e41dbd6adf9/mistralrs_cuda-0.3.4-cp310-none-win_amd64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "5c45b77d213f6689395a2b9efc87eb4ab965ab7cd1634d4ebbf1c2e5885a7728",
                "md5": "0fc37ad6c7d3bc28c310a0de17cb542c",
                "sha256": "36330f276f5b2e57449ca03b8f8879e27ec3385d203c78074a58e7589b1055cf"
            },
            "downloads": -1,
            "filename": "mistralrs_cuda-0.3.4-cp311-cp311-manylinux_2_39_x86_64.whl",
            "has_sig": false,
            "md5_digest": "0fc37ad6c7d3bc28c310a0de17cb542c",
            "packagetype": "bdist_wheel",
            "python_version": "cp311",
            "requires_python": ">=3.10",
            "size": 24152755,
            "upload_time": "2024-11-28T19:18:18",
            "upload_time_iso_8601": "2024-11-28T19:18:18.698557Z",
            "url": "https://files.pythonhosted.org/packages/5c/45/b77d213f6689395a2b9efc87eb4ab965ab7cd1634d4ebbf1c2e5885a7728/mistralrs_cuda-0.3.4-cp311-cp311-manylinux_2_39_x86_64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "546cf03f898a181a9b417da565877abf9e6b9cb1a92f52a26ef560315132f22c",
                "md5": "a4bb74dc3d88a0645a627ceefefda2e6",
                "sha256": "d284bb876add040193fe833c3169fa09d90f562ad5ca161b1f781ab1be00ad52"
            },
            "downloads": -1,
            "filename": "mistralrs_cuda-0.3.4-cp311-none-win_amd64.whl",
            "has_sig": false,
            "md5_digest": "a4bb74dc3d88a0645a627ceefefda2e6",
            "packagetype": "bdist_wheel",
            "python_version": "cp311",
            "requires_python": ">=3.10",
            "size": 17050967,
            "upload_time": "2024-11-28T19:18:27",
            "upload_time_iso_8601": "2024-11-28T19:18:27.676608Z",
            "url": "https://files.pythonhosted.org/packages/54/6c/f03f898a181a9b417da565877abf9e6b9cb1a92f52a26ef560315132f22c/mistralrs_cuda-0.3.4-cp311-none-win_amd64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "af4d7f3c45bbbcce12eb3ba8483ae1c9105fcb46b409f47aefb22ba910badd59",
                "md5": "779cc84a277eb7259a5446204425d8b7",
                "sha256": "cb1c7c3242083a152ab7d0bf7e866fe8ea0d9474ff83088ff4415ee748fbd50b"
            },
            "downloads": -1,
            "filename": "mistralrs_cuda-0.3.4-cp312-cp312-manylinux_2_39_x86_64.whl",
            "has_sig": false,
            "md5_digest": "779cc84a277eb7259a5446204425d8b7",
            "packagetype": "bdist_wheel",
            "python_version": "cp312",
            "requires_python": ">=3.10",
            "size": 24161352,
            "upload_time": "2024-11-28T19:18:38",
            "upload_time_iso_8601": "2024-11-28T19:18:38.705314Z",
            "url": "https://files.pythonhosted.org/packages/af/4d/7f3c45bbbcce12eb3ba8483ae1c9105fcb46b409f47aefb22ba910badd59/mistralrs_cuda-0.3.4-cp312-cp312-manylinux_2_39_x86_64.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "ec3cc8fa08c3d385291e25d0c3759da14ca9099c9a191ea9aa41b8ffabe94eb0",
                "md5": "0618458f867247c39894fba78e27a8ed",
                "sha256": "b1d6fdc86ac5125f3c95f77c36918a331c98301386fe69bbc0392c7a501ea143"
            },
            "downloads": -1,
            "filename": "mistralrs_cuda-0.3.4-cp312-none-win_amd64.whl",
            "has_sig": false,
            "md5_digest": "0618458f867247c39894fba78e27a8ed",
            "packagetype": "bdist_wheel",
            "python_version": "cp312",
            "requires_python": ">=3.10",
            "size": 17040890,
            "upload_time": "2024-11-28T19:18:48",
            "upload_time_iso_8601": "2024-11-28T19:18:48.131551Z",
            "url": "https://files.pythonhosted.org/packages/ec/3c/c8fa08c3d385291e25d0c3759da14ca9099c9a191ea9aa41b8ffabe94eb0/mistralrs_cuda-0.3.4-cp312-none-win_amd64.whl",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-11-28 19:17:57",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "EricLBuehler",
    "github_project": "mistral.rs",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "mistralrs-cuda"
}
        
Elapsed time: 0.34576s