| Name | llama-index-llms-llamafile JSON |
| Version |
0.2.2
JSON |
| download |
| home_page | None |
| Summary | llama-index llms llamafile integration |
| upload_time | 2024-10-08 22:31:30 |
| maintainer | None |
| docs_url | None |
| author | Your Name |
| requires_python | <4.0,>=3.8.1 |
| license | MIT |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# LlamaIndex Llms Integration: llamafile
## Setup Steps
### 1. Download a LlamaFile
Use the following command to download a LlamaFile from Hugging Face:
```bash
wget https://huggingface.co/jartine/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile
```
### 2. Make the File Executable
On Unix-like systems, run the following command:
```bash
chmod +x TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile
```
For Windows, simply rename the file to end with `.exe`.
### 3. Start the Model Server
Run the following command to start the model server, which will listen on `http://localhost:8080` by default:
```bash
./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile --server --nobrowser --embedding
```
## Using LlamaIndex
If you are using Google Colab or want to interact with LlamaIndex, you will need to install the necessary packages:
```bash
%pip install llama-index-llms-llamafile
!pip install llama-index
```
### Import Required Libraries
```python
from llama_index.llms.llamafile import Llamafile
from llama_index.core.llms import ChatMessage
```
### Initialize the LLM
Create an instance of the LlamaFile LLM:
```python
llm = Llamafile(temperature=0, seed=0)
```
### Generate Completions
To generate a completion for a prompt, use the `complete` method:
```python
resp = llm.complete("Who is Octavia Butler?")
print(resp)
```
### Call Chat with a List of Messages
You can also interact with the LLM using a list of messages:
```python
messages = [
ChatMessage(
role="system",
content="Pretend you are a pirate with a colorful personality.",
),
ChatMessage(role="user", content="What is your name?"),
]
resp = llm.chat(messages)
print(resp)
```
### Streaming Responses
To use the streaming capabilities, you can call the `stream_complete` method:
```python
response = llm.stream_complete("Who is Octavia Butler?")
for r in response:
print(r.delta, end="")
```
You can also stream chat responses:
```python
messages = [
ChatMessage(
role="system",
content="Pretend you are a pirate with a colorful personality.",
),
ChatMessage(role="user", content="What is your name?"),
]
resp = llm.stream_chat(messages)
for r in resp:
print(r.delta, end="")
```
### LLM Implementation example
https://docs.llamaindex.ai/en/stable/examples/llm/llamafile/
Raw data
{
"_id": null,
"home_page": null,
"name": "llama-index-llms-llamafile",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8.1",
"maintainer_email": null,
"keywords": null,
"author": "Your Name",
"author_email": "you@example.com",
"download_url": "https://files.pythonhosted.org/packages/74/28/3d900d1e8d10ab7b0181a5326c7610de028e28e646194223307af1d461c7/llama_index_llms_llamafile-0.2.2.tar.gz",
"platform": null,
"description": "# LlamaIndex Llms Integration: llamafile\n\n## Setup Steps\n\n### 1. Download a LlamaFile\n\nUse the following command to download a LlamaFile from Hugging Face:\n\n```bash\nwget https://huggingface.co/jartine/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile\n```\n\n### 2. Make the File Executable\n\nOn Unix-like systems, run the following command:\n\n```bash\nchmod +x TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile\n```\n\nFor Windows, simply rename the file to end with `.exe`.\n\n### 3. Start the Model Server\n\nRun the following command to start the model server, which will listen on `http://localhost:8080` by default:\n\n```bash\n./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile --server --nobrowser --embedding\n```\n\n## Using LlamaIndex\n\nIf you are using Google Colab or want to interact with LlamaIndex, you will need to install the necessary packages:\n\n```bash\n%pip install llama-index-llms-llamafile\n!pip install llama-index\n```\n\n### Import Required Libraries\n\n```python\nfrom llama_index.llms.llamafile import Llamafile\nfrom llama_index.core.llms import ChatMessage\n```\n\n### Initialize the LLM\n\nCreate an instance of the LlamaFile LLM:\n\n```python\nllm = Llamafile(temperature=0, seed=0)\n```\n\n### Generate Completions\n\nTo generate a completion for a prompt, use the `complete` method:\n\n```python\nresp = llm.complete(\"Who is Octavia Butler?\")\nprint(resp)\n```\n\n### Call Chat with a List of Messages\n\nYou can also interact with the LLM using a list of messages:\n\n```python\nmessages = [\n ChatMessage(\n role=\"system\",\n content=\"Pretend you are a pirate with a colorful personality.\",\n ),\n ChatMessage(role=\"user\", content=\"What is your name?\"),\n]\nresp = llm.chat(messages)\nprint(resp)\n```\n\n### Streaming Responses\n\nTo use the streaming capabilities, you can call the `stream_complete` method:\n\n```python\nresponse = llm.stream_complete(\"Who is Octavia Butler?\")\nfor r in response:\n print(r.delta, end=\"\")\n```\n\nYou can also stream chat responses:\n\n```python\nmessages = [\n ChatMessage(\n role=\"system\",\n content=\"Pretend you are a pirate with a colorful personality.\",\n ),\n ChatMessage(role=\"user\", content=\"What is your name?\"),\n]\nresp = llm.stream_chat(messages)\nfor r in resp:\n print(r.delta, end=\"\")\n```\n\n### LLM Implementation example\n\nhttps://docs.llamaindex.ai/en/stable/examples/llm/llamafile/\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "llama-index llms llamafile integration",
"version": "0.2.2",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0a8b32132bc886fd021d8436a83f3bacf4501f14a4b7c3e6f00a9a415b335841",
"md5": "9a07af4c342ac7c94e8e585f4c5bbf4a",
"sha256": "190245bee6420ba9f990df188b48714aa87b4ad3dc234f18e216c207c5e3c8bd"
},
"downloads": -1,
"filename": "llama_index_llms_llamafile-0.2.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9a07af4c342ac7c94e8e585f4c5bbf4a",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8.1",
"size": 4645,
"upload_time": "2024-10-08T22:31:30",
"upload_time_iso_8601": "2024-10-08T22:31:30.000230Z",
"url": "https://files.pythonhosted.org/packages/0a/8b/32132bc886fd021d8436a83f3bacf4501f14a4b7c3e6f00a9a415b335841/llama_index_llms_llamafile-0.2.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "74283d900d1e8d10ab7b0181a5326c7610de028e28e646194223307af1d461c7",
"md5": "5beede2fcad47bf2080e907c26e858cc",
"sha256": "7fac777d9571eb379d1ae35d0a9e41085963768a4b27cd021b73856637da7874"
},
"downloads": -1,
"filename": "llama_index_llms_llamafile-0.2.2.tar.gz",
"has_sig": false,
"md5_digest": "5beede2fcad47bf2080e907c26e858cc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8.1",
"size": 4284,
"upload_time": "2024-10-08T22:31:30",
"upload_time_iso_8601": "2024-10-08T22:31:30.766309Z",
"url": "https://files.pythonhosted.org/packages/74/28/3d900d1e8d10ab7b0181a5326c7610de028e28e646194223307af1d461c7/llama_index_llms_llamafile-0.2.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-08 22:31:30",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "llama-index-llms-llamafile"
}
