# picoLLM Inference Engine Python Demos
Made in Vancouver, Canada by [Picovoice](https://picovoice.ai)
## picoLLM Inference Engine
picoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language
models. picoLLM Inference Engine is:
- Accurate; picoLLM Compression improves GPTQ by [significant margins](https://picovoice.ai/blog/picollm-towards-optimal-llm-quantization/)
- Private; LLM inference runs 100% locally.
- Cross-Platform
- Runs on CPU and GPU
- Free for open-weight models
## Compatibility
- Python 3.8+
- Runs on Linux (x86_64), macOS (arm64, x86_64), Windows (x86_64), and Raspberry Pi (5 and 4).
## Installation
```console
pip3 install picollmdemo
```
## Models
picoLLM Inference Engine supports the following open-weight models. The models are on
[Picovoice Console](https://console.picovoice.ai/).
- Gemma
- `gemma-2b`
- `gemma-2b-it`
- `gemma-7b`
- `gemma-7b-it`
- Llama-2
- `llama-2-7b`
- `llama-2-7b-chat`
- `llama-2-13b`
- `llama-2-13b-chat`
- `llama-2-70b`
- `llama-2-70b-chat`
- Llama-3
- `llama-3-8b`
- `llama-3-8b-instruct`
- `llama-3-70b`
- `llama-3-70b-instruct`
- Mistral
- `mistral-7b-v0.1`
- `mistral-7b-instruct-v0.1`
- `mistral-7b-instruct-v0.2`
- Mixtral
- `mixtral-8x7b-v0.1`
- `mixtral-8x7b-instruct-v0.1`
- Phi-2
- `phi2`
- Phi-3
- `phi3`
## AccessKey
AccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is
using Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet
connectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100%
offline and completely free for open-weight models. Everyone who signs up for
[Picovoice Console](https://console.picovoice.ai/) receives a unique AccessKey.
## Usage
There are two demos available: completion and chat. The completion demo accepts a prompt and a set of optional
parameters and generates a single completion. It can run all models, whether instruction-tuned or not. The chat demo can
run instruction-tuned (chat) models such as `llama-3-8b-instruct`, `phi2`, etc. The chat demo enables a back-and-forth
conversation with the LLM, similar to ChatGPT.
### Completion Demo
Run the demo by entering the following in the terminal:
```console
picollm_demo_completion --access_key ${ACCESS_KEY} --model_path ${MODEL_PATH} --prompt ${PROMPT}
```
Replace `${ACCESS_KEY}` with yours obtained from Picovoice Console, `${MODEL_PATH}` with the path to a model file
downloaded from Picovoice Console, and `${PROMPT}` with a prompt string.
To get information about all the available options in the demo, run the following:
```console
picollm_demo_completion --help
```
### Chat Demo
To run an instruction-tuned model for chat, run the following in the terminal:
```console
picollm_demo_chat --access_key ${ACCESS_KEY} --model_path ${MODEL_PATH}
```
Replace `${ACCESS_KEY}` with yours obtained from Picovoice Console and `${MODEL_PATH}` with the path to a model file
downloaded from Picovoice Console.
To get information about all the available options in the demo, run the following:
```console
picollm_demo_chat --help
```
Raw data
{
"_id": null,
"home_page": "https://github.com/Picovoice/picollm",
"name": "picollmdemo",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Large Language Model, LLM, Generative AI, GenAI, Llama, Mistral, Mixtral, Gemma, Phi",
"author": "Picovoice",
"author_email": "hello@picovoice.ai",
"download_url": "https://files.pythonhosted.org/packages/7b/b1/476cd94fce4fd61b08ab2e9df417aab1bf6e0fd090ee7cff3b6e0700f631/picollmdemo-1.2.0.tar.gz",
"platform": null,
"description": "# picoLLM Inference Engine Python Demos\n\nMade in Vancouver, Canada by [Picovoice](https://picovoice.ai)\n\n## picoLLM Inference Engine\n\npicoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language\nmodels. picoLLM Inference Engine is:\n\n- Accurate; picoLLM Compression improves GPTQ by [significant margins](https://picovoice.ai/blog/picollm-towards-optimal-llm-quantization/)\n- Private; LLM inference runs 100% locally.\n- Cross-Platform\n- Runs on CPU and GPU\n- Free for open-weight models\n\n## Compatibility\n\n- Python 3.8+\n- Runs on Linux (x86_64), macOS (arm64, x86_64), Windows (x86_64), and Raspberry Pi (5 and 4).\n\n## Installation\n\n```console\npip3 install picollmdemo\n```\n\n## Models\n\npicoLLM Inference Engine supports the following open-weight models. The models are on\n[Picovoice Console](https://console.picovoice.ai/).\n\n- Gemma\n - `gemma-2b`\n - `gemma-2b-it`\n - `gemma-7b`\n - `gemma-7b-it`\n- Llama-2\n - `llama-2-7b`\n - `llama-2-7b-chat`\n - `llama-2-13b`\n - `llama-2-13b-chat`\n - `llama-2-70b`\n - `llama-2-70b-chat`\n- Llama-3\n - `llama-3-8b`\n - `llama-3-8b-instruct`\n - `llama-3-70b`\n - `llama-3-70b-instruct`\n- Mistral\n - `mistral-7b-v0.1`\n - `mistral-7b-instruct-v0.1`\n - `mistral-7b-instruct-v0.2`\n- Mixtral\n - `mixtral-8x7b-v0.1`\n - `mixtral-8x7b-instruct-v0.1`\n- Phi-2\n - `phi2`\n- Phi-3\n - `phi3`\n\n## AccessKey\n\nAccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is\nusing Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet\nconnectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100%\noffline and completely free for open-weight models. Everyone who signs up for\n[Picovoice Console](https://console.picovoice.ai/) receives a unique AccessKey.\n\n## Usage\n\nThere are two demos available: completion and chat. The completion demo accepts a prompt and a set of optional\nparameters and generates a single completion. It can run all models, whether instruction-tuned or not. The chat demo can\nrun instruction-tuned (chat) models such as `llama-3-8b-instruct`, `phi2`, etc. The chat demo enables a back-and-forth\nconversation with the LLM, similar to ChatGPT.\n\n### Completion Demo\n\nRun the demo by entering the following in the terminal:\n\n```console\npicollm_demo_completion --access_key ${ACCESS_KEY} --model_path ${MODEL_PATH} --prompt ${PROMPT}\n```\n\nReplace `${ACCESS_KEY}` with yours obtained from Picovoice Console, `${MODEL_PATH}` with the path to a model file\ndownloaded from Picovoice Console, and `${PROMPT}` with a prompt string.\n\nTo get information about all the available options in the demo, run the following:\n\n```console\npicollm_demo_completion --help\n```\n\n### Chat Demo\n\nTo run an instruction-tuned model for chat, run the following in the terminal:\n\n```console\npicollm_demo_chat --access_key ${ACCESS_KEY} --model_path ${MODEL_PATH}\n```\n\nReplace `${ACCESS_KEY}` with yours obtained from Picovoice Console and `${MODEL_PATH}` with the path to a model file\ndownloaded from Picovoice Console.\n\nTo get information about all the available options in the demo, run the following:\n\n```console\npicollm_demo_chat --help\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "picoLLM Inference Engine demos",
"version": "1.2.0",
"project_urls": {
"Homepage": "https://github.com/Picovoice/picollm"
},
"split_keywords": [
"large language model",
" llm",
" generative ai",
" genai",
" llama",
" mistral",
" mixtral",
" gemma",
" phi"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4524d157cd302afb9c897fdc48d308cc42372305f23c5732b4df005317ef2117",
"md5": "f70c2ede42edc9b9818d6336447c0aed",
"sha256": "64a17e55c795d292034fb4baf9faf9a9fada0cd69b2e96548fafc0845dbfa2bb"
},
"downloads": -1,
"filename": "picollmdemo-1.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f70c2ede42edc9b9818d6336447c0aed",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 12228,
"upload_time": "2024-11-14T23:26:37",
"upload_time_iso_8601": "2024-11-14T23:26:37.254883Z",
"url": "https://files.pythonhosted.org/packages/45/24/d157cd302afb9c897fdc48d308cc42372305f23c5732b4df005317ef2117/picollmdemo-1.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "7bb1476cd94fce4fd61b08ab2e9df417aab1bf6e0fd090ee7cff3b6e0700f631",
"md5": "78815ba56fe6c54dbc333a7e81d76daa",
"sha256": "f4d697f4429c3a0bd87530c78dcb69b1b6c1d58bb89f8f70f8d51c1f3c0d365b"
},
"downloads": -1,
"filename": "picollmdemo-1.2.0.tar.gz",
"has_sig": false,
"md5_digest": "78815ba56fe6c54dbc333a7e81d76daa",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 11111,
"upload_time": "2024-11-14T23:26:38",
"upload_time_iso_8601": "2024-11-14T23:26:38.183370Z",
"url": "https://files.pythonhosted.org/packages/7b/b1/476cd94fce4fd61b08ab2e9df417aab1bf6e0fd090ee7cff3b6e0700f631/picollmdemo-1.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-14 23:26:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Picovoice",
"github_project": "picollm",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "picollmdemo"
}
JSON
