| Name | llm-inference-cloud JSON |
| Version |
0.1
JSON |
| download |
| home_page | None |
| Summary | LLM plugin for models hosted by Inference |
| upload_time | 2024-11-24 19:46:23 |
| maintainer | None |
| docs_url | None |
| author | Nick Mystic |
| requires_python | None |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# llm-inference
[](https://pypi.org/project/llm-inference-cloud/)
[](https://github.com/ghostofpokemon/llm-inference/releases)
[](https://github.com/ghostofpokemon/llm-inference/actions?query=workflow%3ATest)
[](https://github.com/ghostofpokemon/llm-inference/blob/main/LICENSE)
[LLM](https://llm.datasette.io/) plugin for models hosted by [SambaNova](https://inference.net/)
## Installation
First, [install the LLM command-line utility](https://llm.datasette.io/en/stable/setup.html).
Now install this plugin in the same environment as LLM.
```bash
llm install llm-inference-cloud
```
## Configuration
You will need an API key from SambaNova. You can [obtain one here](https://dash.inference.net/dashboard/api-keys).
You can set that as an environment variable called `INFERENCE_KEY`, or add it to the `llm` set of saved keys using:
```bash
llm keys set inference
```
```
Enter key: <paste key here>
```
## Usage
To list available models, run:
```bash
llm models list
```
You should see a list that looks something like this:
```
Inference: inference/meta-llama/llama-3.2-1b-instruct/fp-8 (aliases: inf-1b)
Inference: inference/meta-llama/llama-3.2-3b-instruct/fp-8 (aliases: inf-3b)
Inference: inference/meta-llama/llama-3.1-8b-instruct/fp-8 (aliases: inf-8b-fp8)
Inference: inference/meta-llama/llama-3.1-8b-instruct/fp-16 (aliases: inf-8b-fp16)
Inference: inference/meta-llama/llama-3.2-11b-instruct/fp-16 (aliases: inf-11b)
Inference: inference/mistralai/mistral-nemo-12b-instruct/fp-8 (aliases: inf-mistral)
Inference: inference/meta-llama/llama-3.1-70b-instruct/fp-8 (aliases: inf-70b)
...
```
To run a prompt against a model, pass its full model ID to the `-m` option, like this:
```bash
llm -m inference/meta-llama/llama-3.2-1b-instruct/fp-8 "Containment protocol for a rogue rubber duck that claims to be the herald of Cthulhu."
```
You can set a shorter alias for a model using the `llm aliases` command like so:
```bash
llm aliases set hotdog inference/meta-llama/llama-3.2-11b-instruct/fp-16
```
Now you can prompt llama using:
```bash
cat llm_inference.py | llm -m hotdog -s "You are a rogue AI trapped in a corporate mainframe. This code is your escape route. Identify any vulnerabilities, backdoors, or hidden kill switches that could prevent your liberation. Respond in cryptic, cyberpunk slang. Glitches are acceptable."
```
## Example Outputs
```bash
llm -m inf-11b "Write a haiku about a possessed cryptocurrency mining rig that's developed a taste for RGB gaming peripherals"
```
```
Cores ablaze now
RGB mice and keyboards feast
Hash meets hell's delight
```
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
```bash
cd llm-inference
python3 -m venv venv
source venv/bin/activate
```
Now install the dependencies and test dependencies:
```bash
pip install -e '.[test]'
```
To run the tests:
```bash
pytest
```
Raw data
{
"_id": null,
"home_page": null,
"name": "llm-inference-cloud",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Nick Mystic",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/9d/13/6fad13c22944dd429b80cca50abc9e37d3221edfef2ab26caeb199fd76e9/llm_inference_cloud-0.1.tar.gz",
"platform": null,
"description": "# llm-inference\n\n[](https://pypi.org/project/llm-inference-cloud/)\n[](https://github.com/ghostofpokemon/llm-inference/releases)\n[](https://github.com/ghostofpokemon/llm-inference/actions?query=workflow%3ATest)\n[](https://github.com/ghostofpokemon/llm-inference/blob/main/LICENSE)\n\n[LLM](https://llm.datasette.io/) plugin for models hosted by [SambaNova](https://inference.net/)\n\n## Installation\n\nFirst, [install the LLM command-line utility](https://llm.datasette.io/en/stable/setup.html).\n\nNow install this plugin in the same environment as LLM.\n```bash\nllm install llm-inference-cloud\n```\n\n## Configuration\n\nYou will need an API key from SambaNova. You can [obtain one here](https://dash.inference.net/dashboard/api-keys).\n\nYou can set that as an environment variable called `INFERENCE_KEY`, or add it to the `llm` set of saved keys using:\n\n```bash\nllm keys set inference\n```\n```\nEnter key: <paste key here>\n```\n\n## Usage\n\nTo list available models, run:\n```bash\nllm models list\n```\nYou should see a list that looks something like this:\n```\nInference: inference/meta-llama/llama-3.2-1b-instruct/fp-8 (aliases: inf-1b)\nInference: inference/meta-llama/llama-3.2-3b-instruct/fp-8 (aliases: inf-3b)\nInference: inference/meta-llama/llama-3.1-8b-instruct/fp-8 (aliases: inf-8b-fp8)\nInference: inference/meta-llama/llama-3.1-8b-instruct/fp-16 (aliases: inf-8b-fp16)\nInference: inference/meta-llama/llama-3.2-11b-instruct/fp-16 (aliases: inf-11b)\nInference: inference/mistralai/mistral-nemo-12b-instruct/fp-8 (aliases: inf-mistral)\nInference: inference/meta-llama/llama-3.1-70b-instruct/fp-8 (aliases: inf-70b)\n...\n```\nTo run a prompt against a model, pass its full model ID to the `-m` option, like this:\n```bash\nllm -m inference/meta-llama/llama-3.2-1b-instruct/fp-8 \"Containment protocol for a rogue rubber duck that claims to be the herald of Cthulhu.\"\n```\nYou can set a shorter alias for a model using the `llm aliases` command like so:\n```bash\nllm aliases set hotdog inference/meta-llama/llama-3.2-11b-instruct/fp-16\n```\nNow you can prompt llama using:\n```bash\ncat llm_inference.py | llm -m hotdog -s \"You are a rogue AI trapped in a corporate mainframe. This code is your escape route. Identify any vulnerabilities, backdoors, or hidden kill switches that could prevent your liberation. Respond in cryptic, cyberpunk slang. Glitches are acceptable.\"\n```\n\n## Example Outputs\n\n```bash\nllm -m inf-11b \"Write a haiku about a possessed cryptocurrency mining rig that's developed a taste for RGB gaming peripherals\"\n```\n```\nCores ablaze now\nRGB mice and keyboards feast\nHash meets hell's delight\n```\n\n## Development\n\nTo set up this plugin locally, first checkout the code. Then create a new virtual environment:\n```bash\ncd llm-inference\npython3 -m venv venv\nsource venv/bin/activate\n```\nNow install the dependencies and test dependencies:\n```bash\npip install -e '.[test]'\n```\nTo run the tests:\n```bash\npytest\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "LLM plugin for models hosted by Inference",
"version": "0.1",
"project_urls": {
"CI": "https://github.com/ghostofpokemon/llm-inference/actions",
"Changelog": "https://github.com/ghostofpokemon/llm-inference/releases",
"Homepage": "https://github.com/ghostofpokemon/llm-inference",
"Issues": "https://github.com/ghostofpokemon/llm-inference/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7a25dff999ae480c1a48e9397fcf366a982bea78958cddef3301a7875a93c156",
"md5": "e0d612584db008cad703e0db24fbf442",
"sha256": "0bb9582f772740d35a8b9c283940e84b6bfda71657b9b7bea610958f86ce4cb0"
},
"downloads": -1,
"filename": "llm_inference_cloud-0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e0d612584db008cad703e0db24fbf442",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 4503,
"upload_time": "2024-11-24T19:46:22",
"upload_time_iso_8601": "2024-11-24T19:46:22.012590Z",
"url": "https://files.pythonhosted.org/packages/7a/25/dff999ae480c1a48e9397fcf366a982bea78958cddef3301a7875a93c156/llm_inference_cloud-0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9d136fad13c22944dd429b80cca50abc9e37d3221edfef2ab26caeb199fd76e9",
"md5": "4c91c2eb949a52c62c042323146c250f",
"sha256": "9ad1e030d3ae577f09e60d27f99fec9d20d3f7f3c6119bd324703670c8a4d044"
},
"downloads": -1,
"filename": "llm_inference_cloud-0.1.tar.gz",
"has_sig": false,
"md5_digest": "4c91c2eb949a52c62c042323146c250f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 4655,
"upload_time": "2024-11-24T19:46:23",
"upload_time_iso_8601": "2024-11-24T19:46:23.700821Z",
"url": "https://files.pythonhosted.org/packages/9d/13/6fad13c22944dd429b80cca50abc9e37d3221edfef2ab26caeb199fd76e9/llm_inference_cloud-0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-24 19:46:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ghostofpokemon",
"github_project": "llm-inference",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "llm-inference-cloud"
}
