# picoLLM Inference Engine Python Binding
Made in Vancouver, Canada by [Picovoice](https://picovoice.ai)
## picoLLM Inference Engine
picoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language
models. picoLLM Inference Engine is:
- Accurate; picoLLM Compression improves GPTQ by up to 98%.
- Private; LLM inference runs 100% locally.
- Cross-Platform
- Runs on CPU and GPU
- Free for open-weight models
## Compatibility
- Python 3.8+
- Runs on Linux (x86_64), macOS (arm64, x86_64), Windows (x86_64), and Raspberry Pi (5, 4, and 3).
## Installation
```console
pip3 install picollm
```
## Models
picoLLM Inference Engine supports the following open-weight models. The models are on
[Picovoice Console](https://console.picovoice.ai/).
- Gemma
- `gemma-2b`
- `gemma-2b-it`
- `gemma-7b`
- `gemma-7b-it`
- Llama-2
- `llama-2-7b`
- `llama-2-7b-chat`
- `llama-2-13b`
- `llama-2-13b-chat`
- `llama-2-70b`
- `llama-2-70b-chat`
- Llama-3
- `llama-3-8b`
- `llama-3-8b-instruct`
- `llama-3-70b`
- `llama-3-70b-instruct`
- Mistral
- `mistral-7b-v0.1`
- `mistral-7b-instruct-v0.1`
- `mistral-7b-instruct-v0.2`
- Mixtral
- `mixtral-8x7b-v0.1`
- `mixtral-8x7b-instruct-v0.1`
- Phi-2
- `phi2`
## AccessKey
AccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is
using Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet
connectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100%
offline and completely free for open-weight models. Everyone who signs up for
[Picovoice Console](https://console.picovoice.ai/) receives a unique AccessKey.
## Usage
Create an instance of the engine and generate a prompt completion:
```python
import picollm
pllm = picollm.create(
access_key='${ACCESS_KEY}',
model_path='${MODEL_PATH}')
res = pllm.generate(prompt='${PROMPT}')
print(res.completion)
```
Replace `${ACCESS_KEY}` with yours obtained from Picovoice Console, `${MODEL_PATH}` with the path to a model file
downloaded from Picovoice Console, and `${PROMPT}` with a prompt string.
Instruction-tuned models (e.g., `llama-3-8b-instruct`, `llama-2-7b-chat`, and `gemma-2b-it`) have a specific chat
template. You can either directly format the prompt or use a dialog helper:
```python
dialog = pllm.get_dialog()
dialog.add_human_request(prompt)
res = pllm.generate(prompt=dialog.prompt())
dialog.add_llm_response(res.completion)
print(res.completion)
```
Finally, when done, be sure to release the resources explicitly:
```python
pllm.release()
```
## Demos
[//]: # ([AK])
<!-- markdown-link-check-disable -->
[picollmdemo](https://pypi.org/project/picollmdemo/) provides command-line utilities for LLM completion and chat using
picoLLM.
<!-- markdown-link-check-enable -->
Raw data
{
"_id": null,
"home_page": "https://github.com/Picovoice/picollm",
"name": "picollm",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Large Language Model, LLM, Generative AI, GenAI, Llama, Mistral, Mixtral, Gemma, Phi",
"author": "Picovoice",
"author_email": "hello@picovoice.ai",
"download_url": "https://files.pythonhosted.org/packages/d1/d9/00a2f83f022ad391e041957721b34ec20e1b12085ba15cdaed98af4be93e/picollm-1.0.0.tar.gz",
"platform": null,
"description": "# picoLLM Inference Engine Python Binding\n\nMade in Vancouver, Canada by [Picovoice](https://picovoice.ai)\n\n## picoLLM Inference Engine\n\npicoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language\nmodels. picoLLM Inference Engine is:\n\n- Accurate; picoLLM Compression improves GPTQ by up to 98%.\n- Private; LLM inference runs 100% locally.\n- Cross-Platform\n- Runs on CPU and GPU\n- Free for open-weight models\n\n## Compatibility\n\n- Python 3.8+\n- Runs on Linux (x86_64), macOS (arm64, x86_64), Windows (x86_64), and Raspberry Pi (5, 4, and 3).\n\n## Installation\n\n```console\npip3 install picollm\n```\n\n## Models\n\npicoLLM Inference Engine supports the following open-weight models. The models are on\n[Picovoice Console](https://console.picovoice.ai/).\n\n- Gemma\n - `gemma-2b`\n - `gemma-2b-it`\n - `gemma-7b`\n - `gemma-7b-it`\n- Llama-2\n - `llama-2-7b`\n - `llama-2-7b-chat`\n - `llama-2-13b`\n - `llama-2-13b-chat`\n - `llama-2-70b`\n - `llama-2-70b-chat`\n- Llama-3\n - `llama-3-8b`\n - `llama-3-8b-instruct`\n - `llama-3-70b`\n - `llama-3-70b-instruct`\n- Mistral\n - `mistral-7b-v0.1`\n - `mistral-7b-instruct-v0.1`\n - `mistral-7b-instruct-v0.2`\n- Mixtral\n - `mixtral-8x7b-v0.1`\n - `mixtral-8x7b-instruct-v0.1`\n- Phi-2\n - `phi2`\n\n## AccessKey\n\nAccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is\nusing Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet\nconnectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100%\noffline and completely free for open-weight models. Everyone who signs up for\n[Picovoice Console](https://console.picovoice.ai/) receives a unique AccessKey.\n\n## Usage\n\nCreate an instance of the engine and generate a prompt completion:\n\n```python\nimport picollm\n\npllm = picollm.create(\n access_key='${ACCESS_KEY}',\n model_path='${MODEL_PATH}')\n\nres = pllm.generate(prompt='${PROMPT}')\nprint(res.completion)\n```\n\nReplace `${ACCESS_KEY}` with yours obtained from Picovoice Console, `${MODEL_PATH}` with the path to a model file\ndownloaded from Picovoice Console, and `${PROMPT}` with a prompt string.\n\nInstruction-tuned models (e.g., `llama-3-8b-instruct`, `llama-2-7b-chat`, and `gemma-2b-it`) have a specific chat\ntemplate. You can either directly format the prompt or use a dialog helper:\n\n```python\ndialog = pllm.get_dialog()\ndialog.add_human_request(prompt)\n\nres = pllm.generate(prompt=dialog.prompt())\ndialog.add_llm_response(res.completion)\nprint(res.completion)\n```\n\nFinally, when done, be sure to release the resources explicitly:\n\n```python\npllm.release()\n```\n\n## Demos\n\n[//]: # ([AK])\n\n<!-- markdown-link-check-disable -->\n\n[picollmdemo](https://pypi.org/project/picollmdemo/) provides command-line utilities for LLM completion and chat using\npicoLLM.\n\n<!-- markdown-link-check-enable -->\n",
"bugtrack_url": null,
"license": null,
"summary": "picoLLM Inference Engine",
"version": "1.0.0",
"project_urls": {
"Homepage": "https://github.com/Picovoice/picollm"
},
"split_keywords": [
"large language model",
" llm",
" generative ai",
" genai",
" llama",
" mistral",
" mixtral",
" gemma",
" phi"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2b4267ee8530ca1f7fecfe7fa3a43eea3cc6b6f78426ed80c629efeb5ff730b3",
"md5": "cc0b3788e2cd5224a4d107176279219c",
"sha256": "a24ce487a20aca7af8c3292d6bfba604e68df91e0e9bef03caf0802d9f337b75"
},
"downloads": -1,
"filename": "picollm-1.0.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "cc0b3788e2cd5224a4d107176279219c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 8996594,
"upload_time": "2024-05-16T18:23:54",
"upload_time_iso_8601": "2024-05-16T18:23:54.436498Z",
"url": "https://files.pythonhosted.org/packages/2b/42/67ee8530ca1f7fecfe7fa3a43eea3cc6b6f78426ed80c629efeb5ff730b3/picollm-1.0.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d1d900a2f83f022ad391e041957721b34ec20e1b12085ba15cdaed98af4be93e",
"md5": "82d2045ced88ae0f40d8cd5f73279fdc",
"sha256": "d14789f8640a933d6aa0452879c3cb2ba5e3cd0bca1561b6ffa0cfef256e380c"
},
"downloads": -1,
"filename": "picollm-1.0.0.tar.gz",
"has_sig": false,
"md5_digest": "82d2045ced88ae0f40d8cd5f73279fdc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 8973919,
"upload_time": "2024-05-16T18:23:57",
"upload_time_iso_8601": "2024-05-16T18:23:57.634094Z",
"url": "https://files.pythonhosted.org/packages/d1/d9/00a2f83f022ad391e041957721b34ec20e1b12085ba15cdaed98af4be93e/picollm-1.0.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-16 18:23:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Picovoice",
"github_project": "picollm",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "picollm"
}
JSON
