# llm-gpt4all
[](https://pypi.org/project/llm-gpt4all/)
[](https://github.com/simonw/llm-gpt4all/releases)
[](https://github.com/simonw/llm-gpt4all/actions?query=workflow%3ATest)
[](https://github.com/simonw/llm-gpt4all/blob/main/LICENSE)
Plugin for [LLM](https://llm.datasette.io/) adding support for the [GPT4All](https://gpt4all.io/) collection of models.
## Installation
Install this plugin in the same environment as LLM.
```bash
llm install llm-gpt4all
```
After installing the plugin you can see a new list of available models like this:
```bash
llm models list
```
The output will include something like this:
```
gpt4all: orca-mini-3b-gguf2-q4_0 - Mini Orca (Small), 1.84GB download, needs 4GB RAM (installed)
gpt4all: nous-hermes-llama2-13b - Hermes, 6.86GB download, needs 16GB RAM (installed)
gpt4all: all-MiniLM-L6-v2-f16 - SBert, 43.76MB download, needs 1GB RAM
gpt4all: replit-code-v1_5-3b-q4_0 - Replit, 1.74GB download, needs 4GB RAM
gpt4all: mpt-7b-chat-merges-q4_0 - MPT Chat, 3.54GB download, needs 8GB RAM
gpt4all: rift-coder-v0-7b-q4_0 - Rift coder, 3.56GB download, needs 8GB RAM
gpt4all: em_german_mistral_v01 - EM German Mistral, 3.83GB download, needs 8GB RAM
gpt4all: mistral-7b-instruct-v0 - Mistral Instruct, 3.83GB download, needs 8GB RAM
gpt4all: mistral-7b-openorca - Mistral OpenOrca, 3.83GB download, needs 8GB RAM
gpt4all: gpt4all-falcon-q4_0 - GPT4All Falcon, 3.92GB download, needs 8GB RAM
gpt4all: gpt4all-13b-snoozy-q4_0 - Snoozy, 6.86GB download, needs 16GB RAM
gpt4all: wizardlm-13b-v1 - Wizard v1.2, 6.86GB download, needs 16GB RAM
gpt4all: starcoder-q4_0 - Starcoder, 8.37GB download, needs 4GB RAM
```
Further details on these models can be found [in this Observable notebook](https://observablehq.com/@simonw/gpt4all-models).
## Usage
### Model Installation and Prompting
You can execute a model using the name displayed in the `llm models list` output. The model file will be downloaded the first time you attempt to run it.
```bash
llm -m orca-mini-3b-gguf2-q4_0 '3 names for a pet cow'
```
The first time you run this you will see a progress bar for the model file download:
```
31%|█████████▋ | 1.16G/3.79G [00:26<01:02, 42.0MiB/s]
```
On subsequent uses the model output will be displayed immediately.
Note that the models will be downloaded to `~/.cache/gpt4all`.
### Model options
Run `llm models --options` for a list of available model options, which should include:
```
gpt4all: mistral-7b-instruct-v0 - Mistral Instruct, 3.83GB download, needs 8GB RAM (installed)
max_tokens: int
The maximum number of tokens to generate.
temp: float
The model temperature. Larger values increase creativity but decrease
factuality.
top_k: int
Randomly sample from the top_k most likely tokens at each generation
step. Set this to 1 for greedy decoding.
top_p: float
Randomly sample at each generation step from the top most likely
tokens whose probabilities add up to top_p.
repeat_penalty: float
Penalize the model for repetition. Higher values result in less
repetition.
repeat_last_n: int
How far in the models generation history to apply the repeat penalty.
n_batch: int
Number of prompt tokens processed in parallel. Larger values decrease
latency but increase resource requirements.
```
Use them like this:
```bash
llm -m mistral-7b-instruct-v0 -o max_tokens 2 'hi'
```
```
Hello!
```
### Chatting
To chat with a model, avoiding the need to load it into memory for every message, use `llm chat`:
```bash
llm chat -m orca-mini-3b-gguf2-q4_0
```
```
Chatting with orca-mini-3b-gguf2-q4_0
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
> hi
Hello! How can I assist you today?
> three jokes about a pelican and a chicken who are friends
Sure, here are three jokes about a pelican and a chicken who are friends:
1. Why did the pelican cross the road? To get to the other side where the chicken was waiting for him!
2. What do you call a group of chickens playing basketball? A flock of feathers!
3. Why did the chicken invite the pelican over for dinner? Because it had nothing else to eat and needed some extra protein in its diet!
```
### Removing Models
To remove a downloaded model, delete the `.gguf` file from `~/.cache/gpt4all`.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
```bash
cd llm-gpt4all
python3 -m venv venv
source venv/bin/activate
```
Now install the dependencies and test dependencies:
```bash
pip install -e '.[test]'
```
To run the tests:
```bash
pytest
```
Raw data
{
"_id": null,
"home_page": "https://github.com/simonw/llm-gpt4all",
"name": "llm-gpt4all",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Simon Willison",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/81/a7/1e33cc26e6489678f130c5539e011788e1d20b2ceec3414d3ac6012355f0/llm_gpt4all-0.4.tar.gz",
"platform": null,
"description": "# llm-gpt4all\n\n[](https://pypi.org/project/llm-gpt4all/)\n[](https://github.com/simonw/llm-gpt4all/releases)\n[](https://github.com/simonw/llm-gpt4all/actions?query=workflow%3ATest)\n[](https://github.com/simonw/llm-gpt4all/blob/main/LICENSE)\n\nPlugin for [LLM](https://llm.datasette.io/) adding support for the [GPT4All](https://gpt4all.io/) collection of models.\n\n## Installation\n\nInstall this plugin in the same environment as LLM.\n```bash\nllm install llm-gpt4all\n```\nAfter installing the plugin you can see a new list of available models like this:\n\n```bash\nllm models list\n```\nThe output will include something like this:\n```\ngpt4all: orca-mini-3b-gguf2-q4_0 - Mini Orca (Small), 1.84GB download, needs 4GB RAM (installed)\ngpt4all: nous-hermes-llama2-13b - Hermes, 6.86GB download, needs 16GB RAM (installed)\ngpt4all: all-MiniLM-L6-v2-f16 - SBert, 43.76MB download, needs 1GB RAM\ngpt4all: replit-code-v1_5-3b-q4_0 - Replit, 1.74GB download, needs 4GB RAM\ngpt4all: mpt-7b-chat-merges-q4_0 - MPT Chat, 3.54GB download, needs 8GB RAM\ngpt4all: rift-coder-v0-7b-q4_0 - Rift coder, 3.56GB download, needs 8GB RAM\ngpt4all: em_german_mistral_v01 - EM German Mistral, 3.83GB download, needs 8GB RAM\ngpt4all: mistral-7b-instruct-v0 - Mistral Instruct, 3.83GB download, needs 8GB RAM\ngpt4all: mistral-7b-openorca - Mistral OpenOrca, 3.83GB download, needs 8GB RAM\ngpt4all: gpt4all-falcon-q4_0 - GPT4All Falcon, 3.92GB download, needs 8GB RAM\ngpt4all: gpt4all-13b-snoozy-q4_0 - Snoozy, 6.86GB download, needs 16GB RAM\ngpt4all: wizardlm-13b-v1 - Wizard v1.2, 6.86GB download, needs 16GB RAM\ngpt4all: starcoder-q4_0 - Starcoder, 8.37GB download, needs 4GB RAM\n```\nFurther details on these models can be found [in this Observable notebook](https://observablehq.com/@simonw/gpt4all-models).\n\n## Usage\n\n### Model Installation and Prompting\n\nYou can execute a model using the name displayed in the `llm models list` output. The model file will be downloaded the first time you attempt to run it.\n\n```bash\nllm -m orca-mini-3b-gguf2-q4_0 '3 names for a pet cow'\n```\nThe first time you run this you will see a progress bar for the model file download:\n```\n 31%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b | 1.16G/3.79G [00:26<01:02, 42.0MiB/s]\n```\nOn subsequent uses the model output will be displayed immediately.\n\nNote that the models will be downloaded to `~/.cache/gpt4all`.\n\n### Model options\n\nRun `llm models --options` for a list of available model options, which should include:\n\n```\ngpt4all: mistral-7b-instruct-v0 - Mistral Instruct, 3.83GB download, needs 8GB RAM (installed)\n max_tokens: int\n The maximum number of tokens to generate.\n temp: float\n The model temperature. Larger values increase creativity but decrease\n factuality.\n top_k: int\n Randomly sample from the top_k most likely tokens at each generation\n step. Set this to 1 for greedy decoding.\n top_p: float\n Randomly sample at each generation step from the top most likely\n tokens whose probabilities add up to top_p.\n repeat_penalty: float\n Penalize the model for repetition. Higher values result in less\n repetition.\n repeat_last_n: int\n How far in the models generation history to apply the repeat penalty.\n n_batch: int\n Number of prompt tokens processed in parallel. Larger values decrease\n latency but increase resource requirements.\n```\nUse them like this:\n\n```bash\nllm -m mistral-7b-instruct-v0 -o max_tokens 2 'hi'\n```\n```\nHello!\n```\n\n### Chatting\n\nTo chat with a model, avoiding the need to load it into memory for every message, use `llm chat`:\n\n```bash\nllm chat -m orca-mini-3b-gguf2-q4_0\n```\n```\nChatting with orca-mini-3b-gguf2-q4_0\nType 'exit' or 'quit' to exit\nType '!multi' to enter multiple lines, then '!end' to finish\n> hi\n Hello! How can I assist you today?\n> three jokes about a pelican and a chicken who are friends\n Sure, here are three jokes about a pelican and a chicken who are friends:\n\n1. Why did the pelican cross the road? To get to the other side where the chicken was waiting for him!\n2. What do you call a group of chickens playing basketball? A flock of feathers!\n3. Why did the chicken invite the pelican over for dinner? Because it had nothing else to eat and needed some extra protein in its diet!\n```\n\n### Removing Models\n\nTo remove a downloaded model, delete the `.gguf` file from `~/.cache/gpt4all`.\n\n## Development\n\nTo set up this plugin locally, first checkout the code. Then create a new virtual environment:\n```bash\ncd llm-gpt4all\npython3 -m venv venv\nsource venv/bin/activate\n```\nNow install the dependencies and test dependencies:\n```bash\npip install -e '.[test]'\n```\nTo run the tests:\n```bash\npytest\n```\n",
"bugtrack_url": null,
"license": "Apache License, Version 2.0",
"summary": "Plugin for LLM adding support for GPT4ALL models",
"version": "0.4",
"project_urls": {
"CI": "https://github.com/simonw/llm-gpt4all/actions",
"Changelog": "https://github.com/simonw/llm-gpt4all/releases",
"Homepage": "https://github.com/simonw/llm-gpt4all",
"Issues": "https://github.com/simonw/llm-gpt4all/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1aaa9b130e426bec78591378c78227983c9dcc635af9847b1eca3eee01afb5f1",
"md5": "21278afc0491b5fec5cdbfcb77959d0f",
"sha256": "4f7dd866025bd7b804aab81b85e9f5a5ac45171e6fd52b3c4fb75f30e762b384"
},
"downloads": -1,
"filename": "llm_gpt4all-0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "21278afc0491b5fec5cdbfcb77959d0f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 10914,
"upload_time": "2024-04-20T00:33:47",
"upload_time_iso_8601": "2024-04-20T00:33:47.226967Z",
"url": "https://files.pythonhosted.org/packages/1a/aa/9b130e426bec78591378c78227983c9dcc635af9847b1eca3eee01afb5f1/llm_gpt4all-0.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "81a71e33cc26e6489678f130c5539e011788e1d20b2ceec3414d3ac6012355f0",
"md5": "25a141a74221a7613d938a8a4f584ce0",
"sha256": "3e498ae6fa0d146d11cf1885e24cc218b77280102d5cfe666d6732c6fdc59927"
},
"downloads": -1,
"filename": "llm_gpt4all-0.4.tar.gz",
"has_sig": false,
"md5_digest": "25a141a74221a7613d938a8a4f584ce0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 11069,
"upload_time": "2024-04-20T00:33:49",
"upload_time_iso_8601": "2024-04-20T00:33:49.379351Z",
"url": "https://files.pythonhosted.org/packages/81/a7/1e33cc26e6489678f130c5539e011788e1d20b2ceec3414d3ac6012355f0/llm_gpt4all-0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-20 00:33:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "simonw",
"github_project": "llm-gpt4all",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "llm-gpt4all"
}
JSON
