# picoLLM Inference Engine Python Binding
Made in Vancouver, Canada by [Picovoice](https://picovoice.ai)
## picoLLM Inference Engine
picoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language
models. picoLLM Inference Engine is:
- Accurate; picoLLM Compression improves GPTQ by [significant margins](https://picovoice.ai/blog/picollm-towards-optimal-llm-quantization/)
- Private; LLM inference runs 100% locally.
- Cross-Platform
- Runs on CPU and GPU
- Free for open-weight models
## Compatibility
- Python 3.8+
- Runs on Linux (x86_64), macOS (arm64, x86_64), Windows (x86_64), and Raspberry Pi (5 and 4).
## Installation
```console
pip3 install picollm
```
## Models
picoLLM Inference Engine supports the following open-weight models. The models are on
[Picovoice Console](https://console.picovoice.ai/).
- Gemma
- `gemma-2b`
- `gemma-2b-it`
- `gemma-7b`
- `gemma-7b-it`
- Llama-2
- `llama-2-7b`
- `llama-2-7b-chat`
- `llama-2-13b`
- `llama-2-13b-chat`
- `llama-2-70b`
- `llama-2-70b-chat`
- Llama-3
- `llama-3-8b`
- `llama-3-8b-instruct`
- `llama-3-70b`
- `llama-3-70b-instruct`
- Llama-3.2
- `llama3.2-1b-instruct`
- `llama3.2-3b-instruct`
- Mistral
- `mistral-7b-v0.1`
- `mistral-7b-instruct-v0.1`
- `mistral-7b-instruct-v0.2`
- Mixtral
- `mixtral-8x7b-v0.1`
- `mixtral-8x7b-instruct-v0.1`
- Phi-2
- `phi2`
- Phi-3
- `phi3`
- Phi-3.5
- `phi3.5`
## AccessKey
AccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is
using Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet
connectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100%
offline and completely free for open-weight models. Everyone who signs up for
[Picovoice Console](https://console.picovoice.ai/) receives a unique AccessKey.
## Usage
Create an instance of the engine and generate a prompt completion:
```python
import picollm
pllm = picollm.create(
access_key='${ACCESS_KEY}',
model_path='${MODEL_PATH}')
res = pllm.generate(prompt='${PROMPT}')
print(res.completion)
```
Replace `${ACCESS_KEY}` with yours obtained from Picovoice Console, `${MODEL_PATH}` with the path to a model file
downloaded from Picovoice Console, and `${PROMPT}` with a prompt string.
Instruction-tuned models (e.g., `llama-3-8b-instruct`, `llama-2-7b-chat`, and `gemma-2b-it`) have a specific chat
template. You can either directly format the prompt or use a dialog helper:
```python
dialog = pllm.get_dialog()
dialog.add_human_request(prompt)
res = pllm.generate(prompt=dialog.prompt())
dialog.add_llm_response(res.completion)
print(res.completion)
```
To interrupt completion generation before it has finished:
```python
pllm.interrupt()
```
Finally, when done, be sure to release the resources explicitly:
```python
pllm.release()
```
## Demos
[picollmdemo](https://pypi.org/project/picollmdemo/) provides command-line utilities for LLM completion and chat using
picoLLM.
Raw data
{
"_id": null,
"home_page": "https://github.com/Picovoice/picollm",
"name": "picollm",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Large Language Model, LLM, Generative AI, GenAI, Llama, Mistral, Mixtral, Gemma, Phi",
"author": "Picovoice",
"author_email": "hello@picovoice.ai",
"download_url": null,
"platform": null,
"description": "# picoLLM Inference Engine Python Binding\n\nMade in Vancouver, Canada by [Picovoice](https://picovoice.ai)\n\n## picoLLM Inference Engine\n\npicoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language\nmodels. picoLLM Inference Engine is:\n\n- Accurate; picoLLM Compression improves GPTQ by [significant margins](https://picovoice.ai/blog/picollm-towards-optimal-llm-quantization/)\n- Private; LLM inference runs 100% locally.\n- Cross-Platform\n- Runs on CPU and GPU\n- Free for open-weight models\n\n## Compatibility\n\n- Python 3.8+\n- Runs on Linux (x86_64), macOS (arm64, x86_64), Windows (x86_64), and Raspberry Pi (5 and 4).\n\n## Installation\n\n```console\npip3 install picollm\n```\n\n## Models\n\npicoLLM Inference Engine supports the following open-weight models. The models are on\n[Picovoice Console](https://console.picovoice.ai/).\n\n- Gemma\n - `gemma-2b`\n - `gemma-2b-it`\n - `gemma-7b`\n - `gemma-7b-it`\n- Llama-2\n - `llama-2-7b`\n - `llama-2-7b-chat`\n - `llama-2-13b`\n - `llama-2-13b-chat`\n - `llama-2-70b`\n - `llama-2-70b-chat`\n- Llama-3\n - `llama-3-8b`\n - `llama-3-8b-instruct`\n - `llama-3-70b`\n - `llama-3-70b-instruct`\n- Llama-3.2\n - `llama3.2-1b-instruct`\n - `llama3.2-3b-instruct`\n- Mistral\n - `mistral-7b-v0.1`\n - `mistral-7b-instruct-v0.1`\n - `mistral-7b-instruct-v0.2`\n- Mixtral\n - `mixtral-8x7b-v0.1`\n - `mixtral-8x7b-instruct-v0.1`\n- Phi-2\n - `phi2`\n- Phi-3\n - `phi3`\n- Phi-3.5\n - `phi3.5`\n\n## AccessKey\n\nAccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is\nusing Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet\nconnectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100%\noffline and completely free for open-weight models. Everyone who signs up for\n[Picovoice Console](https://console.picovoice.ai/) receives a unique AccessKey.\n\n## Usage\n\nCreate an instance of the engine and generate a prompt completion:\n\n```python\nimport picollm\n\npllm = picollm.create(\n access_key='${ACCESS_KEY}',\n model_path='${MODEL_PATH}')\n\nres = pllm.generate(prompt='${PROMPT}')\nprint(res.completion)\n```\n\nReplace `${ACCESS_KEY}` with yours obtained from Picovoice Console, `${MODEL_PATH}` with the path to a model file\ndownloaded from Picovoice Console, and `${PROMPT}` with a prompt string.\n\nInstruction-tuned models (e.g., `llama-3-8b-instruct`, `llama-2-7b-chat`, and `gemma-2b-it`) have a specific chat\ntemplate. You can either directly format the prompt or use a dialog helper:\n\n```python\ndialog = pllm.get_dialog()\ndialog.add_human_request(prompt)\n\nres = pllm.generate(prompt=dialog.prompt())\ndialog.add_llm_response(res.completion)\nprint(res.completion)\n```\n\nTo interrupt completion generation before it has finished:\n```python\npllm.interrupt()\n```\n\nFinally, when done, be sure to release the resources explicitly:\n\n```python\npllm.release()\n```\n\n## Demos\n\n[picollmdemo](https://pypi.org/project/picollmdemo/) provides command-line utilities for LLM completion and chat using\npicoLLM.\n",
"bugtrack_url": null,
"license": null,
"summary": "picoLLM Inference Engine",
"version": "1.2.3",
"project_urls": {
"Homepage": "https://github.com/Picovoice/picollm"
},
"split_keywords": [
"large language model",
" llm",
" generative ai",
" genai",
" llama",
" mistral",
" mixtral",
" gemma",
" phi"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f33f3c5e21ff657548a9aef244c2d75beae7e033bbf70060d78d40a34a7f6dab",
"md5": "538314d2f3dcd80c85c41d3d20d96b01",
"sha256": "82ee329a8bb6bce3f64a3bcbb1fb4078e41e2b04cdc66332d4dd86ed6ec1f472"
},
"downloads": -1,
"filename": "picollm-1.2.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "538314d2f3dcd80c85c41d3d20d96b01",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 11081331,
"upload_time": "2024-12-20T23:49:53",
"upload_time_iso_8601": "2024-12-20T23:49:53.411924Z",
"url": "https://files.pythonhosted.org/packages/f3/3f/3c5e21ff657548a9aef244c2d75beae7e033bbf70060d78d40a34a7f6dab/picollm-1.2.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-20 23:49:53",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Picovoice",
"github_project": "picollm",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "picollm"
}
JSON
