# Zrb Ollama
Zrb Ollama is a [Pypi](https://pypi.org) package that acts as LiteLLM's wrapper, allowing you to incorporate LLM into your workflow.
Zrb Ollama is a part of the [Zrb](https://pypi.org/project/zrb) ecosystem, but you can install it independently from Zrb.
# Installation
You can install Zrb Ollama by invoking any of the following commands:
```bash
# From pypi
pip install zrb-ollama[rag,aws]
# From github
pip install git+https://github.com/state-alchemists/zrb-ollama.git@main
# From directory
pip install --use-feature=in-tree-build path/to/this/directory
```
By default, Zrb Ollama uses Ollama-based LLM. You can install Ollama by visiting the official website: [`https://ollama.ai/`](https://ollama.ai/).
The default LLM is `ollama/mistral:7b-instruct`, while the default embedding LLM is `ollama/nomic-embed-text`.
You can change this by setting the `model` parameter on `LLMTask` or the `create_rag` function. See [LiteLLM provider](https://docs.litellm.ai/docs/providers/) to use custom LLM.
# Setup LLM
## Using Ollama
You can intall and use [Ollama](https://ollama.ai) to run models locally. To use Ollama in `zrb-ollama`, you need to set two variables:
- `ZRB_OLLAMA_LLM_MODEL` (set this to `ollama/gemma2`, `ollama/qwen2` or other ollama models)
- `ZRB_OLLAMA_EMBEDDING_MODEL` (set this to `ollama/nomic-embed-text` or other ollama models)
## Using OpenAI
To use OpenAI, you need to set three variables:
- `OPENAI_API_KEY`
- `ZRB_OLLAMA_LLM_MODEL` (set this to `gpt-4o`, `gpt-4o-mini` or other OpenAI models)
- `ZRB_OLLAMA_EMBEDDING_MODEL` (set this to `text-embedding-ada-001` or other OpenAI models)
# Interactive Mode
Zrb Ollama provides a simple CLI command so you can interact with the LLM immediately.
To interact with the LLM, you can invoke the following command.
```bash
zrb-ollama
```
To enhance `zrb-ollama` with tools, you can create a file named `zrb_ollama_init.py` and register the tools:
```python
import os
from zrb_ollama import interactive_tools
from zrb_ollama.tools import create_rag, get_rag_documents
_CURRENT_DIR = os.path.dirname(__file__)
# Create RAG function
retrieve_john_titor_info = create_rag(
tool_name='retrieve_john_titor_info',
tool_description="Look for anything related to John Titor",
documents=get_rag_documents(os.path.join(_CURRENT_DIR, "rag", "document")),
vector_db_path=os.path.join(_CURRENT_DIR, "rag", "vector"),
# reset_db=True,
)
# Register RAG function as zrb-ollama tool
interactive_tools.register(retrieve_john_titor_info)
# Create a simple function
def add(a: int, b: int) -> int:
"""Adding two numbers and return the result"""
return a + b
# Register the function as zrb-ollama tool
interactive_tools.register(add)
```
`zrb-ollama` automatically load `zrb_ollama_init.py` and make any registered tools available in the interface.
## Command
While in interactive mode, you can use the following commands:
```
/? Show help
/bye Quit
/clear Clear context
/multi Start multiline mode
/end Stop multiline mode
/model [model] Get/set current model (e.g., ollama/mistral:7b-instruct, gpt-4o)
/tool Get list of tools
/tool add <tool-name> Add tool
/tool rm <tool-name> Remove tool
```
All commands are started with a `/`.
# Using LLMTask (For Integration with Zrb)
Zrb Ollama provides a task named `LLMTask`, allowing you to create a Zrb Task with a custom model or tools.
```python
import os
from zrb import CmdTask, StrInput, runner
from zrb_ollama import LLMTask, ToolFactory
from zrb_ollama.tools import (
create_rag, get_rag_documents, query_internet
)
_CURRENT_DIR = os.path.dirname(__file__)
_RAG_DIR = os.path.join(_CURRENT_DIR, "rag")
rag = LLMTask(
name="rag",
inputs=[
StrInput(name="user-prompt", default="How John Titor introduce himself?"),
],
# model="gpt-4o",
user_message="{{input.user_prompt}}",
tools=[query_internet],
tool_factories=[
ToolFactory(
create_rag,
tool_name="retrieve_john_titor_info",
tool_description="Look for anything related to John Titor",
documents=get_rag_documents(os.path.join(_RAG_DIR, "document")),
# model="text-embedding-ada-002",
vector_db_path=os.path.join(_RAG_DIR, "vector"),
# reset_db=True,
)
],
)
runner.register(rag)
```
Assuming there is a file named `john-titor.md` inside `rag/documents` folder, you can invoke the Task by invoking the following command.
```bash
zrb rag
```
The LLM can browse the article or look for anything on the internet.
# Using Agent (For Integration with Anything Else)
Under the hood, LLMTask makes use of Agent. You can create and interact with the agent programmatically as follows.
```python
import asyncio
import os
from zrb import CmdTask, StrInput, runner
from zrb_ollama import agent
from zrb_ollama.tools import (
create_rag, get_rag_documents, query_internet
)
_CURRENT_DIR = os.path.dirname(__file__)
_RAG_DIR = os.path.join(_CURRENT_DIR, "rag")
from zrb_ollama.tools import create_rag, query_internet
agent = Agent(
model="gpt-4o",
tools=[
create_rag(
tool_name="retrieve",
tool_description="Look for anything related to John Titor"
documents=get_rag_documents(os.path.join(_RAG_DIR, "document")),
# model="text-embedding-ada-002",
vector_db_path=os.path.join(_RAG_DIR, "vector"),
# reset_db=True,
),
query_internet,
]
)
result = asyncio.run(agent.add_user_message("How John Titor introduce himself?"))
print(result)
```
# Configurations
You can set Zrb Ollama configurations using environment variables.
- `LLM_MODEL`
- Default: `ollama/mistral:7b-instruct`
- Description: Default LLM model for `LLMTask` and interactive mode. See [Lite LLM](https://docs.litellm.ai/docs/providers) for valid values.
- `INTERACTIVE_ENABLED_TOOL_NAMES`
- Default: `query_internet,open_web_page,run_shell_command`
- Description: Default tools enabled for interactive mode.
- `RAG_EMBEDDING_MODEL`
- Default: `ollama/nomic-embed-text`
- Description: Default RAG embedding model for `LLMTask` and interactive mode. See [Lite LLM](https://docs.litellm.ai/docs/providers) for valid values.
- `RAG_CHUNK_SIZE`
- Default: `1024`
- Description: Default chunk size for RAG.
- `RAG_OVERLAP`
- Default: `128`
- Description: Default chunk overlap size for RAG.
- `RAG_MAX_RESULT_COUNT`
- Default: `5`
- Description: Default result count for RAG.
- `DEFAULT_SYSTEM_PROMPT`
- Default: `You are a helpful assistant. You provide accurate and comprehensive answers.`
- Description: Default system prompt for LLM Agent.
- `DEFAULT_SYSTEM_MESSAGE_TEMPLATE`
- Default: See [config.py](https://github.com/state-alchemists/zrb-ollama/blob/main/src/zrb_ollama/config.py)
- Description: Default template for LLM AGENT's system message. May contains the following:
- `{system_prompt}`
- `{response_format}`
- `{function_signatures}`
- `DEFAULT_JSON_FIXER_SYSTEM_PROMPT`
- Default: `You are a message fixer. You turn any message into JSON format. Your user is a LLM assistant that need to provide the correctly formatted message to serve the end user (human). If you think the LLM should end the conversation, make sure all necessary information for the human is included in the final_answer.`
- Description: System prompt to fix main LLM response in case it produces invalid JSON
- `DEFAULT_JSON_FIXER_SYSTEM_MESSAGE_TEMPLATE`
- Default: See [config.py](https://github.com/state-alchemists/zrb-ollama/blob/main/src/zrb_ollama/config.py)
- Description: Default system message template to fix main LLM response. May contains the following:
- `{system_prompt}`
- `{response_format}`
- `{function_signatures}`
Raw data
{
"_id": null,
"home_page": "https://github.com/state-alchemists/zrb-ollama",
"name": "zrb-ollama",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0.0,>=3.10.0",
"maintainer_email": null,
"keywords": "Ollama, OpenAI, Bedrock, AI Agent, LLM",
"author": "Go Frendi",
"author_email": "gofrendiasgard@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e8/31/de167b28c4d11310bbbf21c9a244216cf28bf9d07761e63a755adb253e7d/zrb_ollama-0.2.14.tar.gz",
"platform": null,
"description": "\n\n# Zrb Ollama\n\nZrb Ollama is a [Pypi](https://pypi.org) package that acts as LiteLLM's wrapper, allowing you to incorporate LLM into your workflow.\n\nZrb Ollama is a part of the [Zrb](https://pypi.org/project/zrb) ecosystem, but you can install it independently from Zrb. \n\n# Installation\n\nYou can install Zrb Ollama by invoking any of the following commands:\n\n```bash\n# From pypi\npip install zrb-ollama[rag,aws]\n\n# From github\npip install git+https://github.com/state-alchemists/zrb-ollama.git@main\n\n# From directory\npip install --use-feature=in-tree-build path/to/this/directory\n```\n\nBy default, Zrb Ollama uses Ollama-based LLM. You can install Ollama by visiting the official website: [`https://ollama.ai/`](https://ollama.ai/).\n\nThe default LLM is `ollama/mistral:7b-instruct`, while the default embedding LLM is `ollama/nomic-embed-text`.\n\nYou can change this by setting the `model` parameter on `LLMTask` or the `create_rag` function. See [LiteLLM provider](https://docs.litellm.ai/docs/providers/) to use custom LLM.\n\n# Setup LLM\n\n## Using Ollama\n\nYou can intall and use [Ollama](https://ollama.ai) to run models locally. To use Ollama in `zrb-ollama`, you need to set two variables:\n\n- `ZRB_OLLAMA_LLM_MODEL` (set this to `ollama/gemma2`, `ollama/qwen2` or other ollama models)\n- `ZRB_OLLAMA_EMBEDDING_MODEL` (set this to `ollama/nomic-embed-text` or other ollama models)\n\n## Using OpenAI\n\nTo use OpenAI, you need to set three variables:\n\n- `OPENAI_API_KEY`\n- `ZRB_OLLAMA_LLM_MODEL` (set this to `gpt-4o`, `gpt-4o-mini` or other OpenAI models)\n- `ZRB_OLLAMA_EMBEDDING_MODEL` (set this to `text-embedding-ada-001` or other OpenAI models)\n\n# Interactive Mode \n\nZrb Ollama provides a simple CLI command so you can interact with the LLM immediately. \nTo interact with the LLM, you can invoke the following command.\n\n```bash\nzrb-ollama\n```\n\nTo enhance `zrb-ollama` with tools, you can create a file named `zrb_ollama_init.py` and register the tools:\n\n```python\nimport os\nfrom zrb_ollama import interactive_tools\nfrom zrb_ollama.tools import create_rag, get_rag_documents\n\n_CURRENT_DIR = os.path.dirname(__file__)\n\n# Create RAG function\nretrieve_john_titor_info = create_rag(\n tool_name='retrieve_john_titor_info',\n tool_description=\"Look for anything related to John Titor\",\n documents=get_rag_documents(os.path.join(_CURRENT_DIR, \"rag\", \"document\")),\n vector_db_path=os.path.join(_CURRENT_DIR, \"rag\", \"vector\"),\n # reset_db=True,\n)\n\n# Register RAG function as zrb-ollama tool\ninteractive_tools.register(retrieve_john_titor_info)\n\n\n# Create a simple function\ndef add(a: int, b: int) -> int:\n \"\"\"Adding two numbers and return the result\"\"\"\n return a + b\n\n\n# Register the function as zrb-ollama tool\ninteractive_tools.register(add)\n```\n\n`zrb-ollama` automatically load `zrb_ollama_init.py` and make any registered tools available in the interface.\n\n## Command\n\nWhile in interactive mode, you can use the following commands:\n\n```\n/? Show help\n/bye Quit\n/clear Clear context\n/multi Start multiline mode\n/end Stop multiline mode\n/model [model] Get/set current model (e.g., ollama/mistral:7b-instruct, gpt-4o)\n\n/tool Get list of tools\n/tool add <tool-name> Add tool\n/tool rm <tool-name> Remove tool\n```\n\nAll commands are started with a `/`.\n\n# Using LLMTask (For Integration with Zrb)\n\nZrb Ollama provides a task named `LLMTask`, allowing you to create a Zrb Task with a custom model or tools.\n\n```python\nimport os\n\nfrom zrb import CmdTask, StrInput, runner\nfrom zrb_ollama import LLMTask, ToolFactory\nfrom zrb_ollama.tools import (\n create_rag, get_rag_documents, query_internet\n)\n\n_CURRENT_DIR = os.path.dirname(__file__)\n_RAG_DIR = os.path.join(_CURRENT_DIR, \"rag\")\n\nrag = LLMTask(\n name=\"rag\",\n inputs=[\n StrInput(name=\"user-prompt\", default=\"How John Titor introduce himself?\"),\n ],\n # model=\"gpt-4o\",\n user_message=\"{{input.user_prompt}}\",\n tools=[query_internet],\n tool_factories=[\n ToolFactory(\n create_rag,\n tool_name=\"retrieve_john_titor_info\",\n tool_description=\"Look for anything related to John Titor\",\n documents=get_rag_documents(os.path.join(_RAG_DIR, \"document\")),\n # model=\"text-embedding-ada-002\",\n vector_db_path=os.path.join(_RAG_DIR, \"vector\"),\n # reset_db=True,\n )\n ],\n)\nrunner.register(rag)\n```\n\nAssuming there is a file named `john-titor.md` inside `rag/documents` folder, you can invoke the Task by invoking the following command.\n\n```bash\nzrb rag\n```\n\nThe LLM can browse the article or look for anything on the internet.\n\n# Using Agent (For Integration with Anything Else)\n\nUnder the hood, LLMTask makes use of Agent. You can create and interact with the agent programmatically as follows.\n\n```python\nimport asyncio\nimport os\n\nfrom zrb import CmdTask, StrInput, runner\nfrom zrb_ollama import agent\nfrom zrb_ollama.tools import (\n create_rag, get_rag_documents, query_internet\n)\n\n_CURRENT_DIR = os.path.dirname(__file__)\n_RAG_DIR = os.path.join(_CURRENT_DIR, \"rag\")\n\n\nfrom zrb_ollama.tools import create_rag, query_internet\n\n\nagent = Agent(\n model=\"gpt-4o\",\n tools=[\n create_rag(\n tool_name=\"retrieve\",\n tool_description=\"Look for anything related to John Titor\"\n documents=get_rag_documents(os.path.join(_RAG_DIR, \"document\")),\n # model=\"text-embedding-ada-002\",\n vector_db_path=os.path.join(_RAG_DIR, \"vector\"),\n # reset_db=True,\n ),\n query_internet,\n ]\n)\nresult = asyncio.run(agent.add_user_message(\"How John Titor introduce himself?\"))\nprint(result)\n```\n\n# Configurations\n\nYou can set Zrb Ollama configurations using environment variables.\n\n- `LLM_MODEL`\n - Default: `ollama/mistral:7b-instruct`\n - Description: Default LLM model for `LLMTask` and interactive mode. See [Lite LLM](https://docs.litellm.ai/docs/providers) for valid values.\n- `INTERACTIVE_ENABLED_TOOL_NAMES`\n - Default: `query_internet,open_web_page,run_shell_command`\n - Description: Default tools enabled for interactive mode.\n- `RAG_EMBEDDING_MODEL`\n - Default: `ollama/nomic-embed-text`\n - Description: Default RAG embedding model for `LLMTask` and interactive mode. See [Lite LLM](https://docs.litellm.ai/docs/providers) for valid values.\n- `RAG_CHUNK_SIZE`\n - Default: `1024`\n - Description: Default chunk size for RAG.\n- `RAG_OVERLAP`\n - Default: `128`\n - Description: Default chunk overlap size for RAG.\n- `RAG_MAX_RESULT_COUNT`\n - Default: `5`\n - Description: Default result count for RAG.\n- `DEFAULT_SYSTEM_PROMPT`\n - Default: `You are a helpful assistant. You provide accurate and comprehensive answers.`\n - Description: Default system prompt for LLM Agent.\n- `DEFAULT_SYSTEM_MESSAGE_TEMPLATE`\n - Default: See [config.py](https://github.com/state-alchemists/zrb-ollama/blob/main/src/zrb_ollama/config.py)\n - Description: Default template for LLM AGENT's system message. May contains the following:\n - `{system_prompt}`\n - `{response_format}`\n - `{function_signatures}`\n- `DEFAULT_JSON_FIXER_SYSTEM_PROMPT`\n - Default: `You are a message fixer. You turn any message into JSON format. Your user is a LLM assistant that need to provide the correctly formatted message to serve the end user (human). If you think the LLM should end the conversation, make sure all necessary information for the human is included in the final_answer.`\n - Description: System prompt to fix main LLM response in case it produces invalid JSON\n- `DEFAULT_JSON_FIXER_SYSTEM_MESSAGE_TEMPLATE`\n - Default: See [config.py](https://github.com/state-alchemists/zrb-ollama/blob/main/src/zrb_ollama/config.py)\n - Description: Default system message template to fix main LLM response. May contains the following:\n - `{system_prompt}`\n - `{response_format}`\n - `{function_signatures}`\n",
"bugtrack_url": null,
"license": "AGPL-3.0-or-later",
"summary": "Zrb LLM plugin",
"version": "0.2.14",
"project_urls": {
"Documentation": "https://github.com/state-alchemists/zrb-ollama",
"Homepage": "https://github.com/state-alchemists/zrb-ollama",
"Repository": "https://github.com/state-alchemists/zrb-ollama"
},
"split_keywords": [
"ollama",
" openai",
" bedrock",
" ai agent",
" llm"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "24cbc99505d513ddc987bcca2f428ee88effff6485624594df6d885b923235e5",
"md5": "e61134b6949588b8aad4ae00f6e49d82",
"sha256": "f95f16bd2cba08560833b4641c298c5686c06f355647c6cc583f3300f24ebc83"
},
"downloads": -1,
"filename": "zrb_ollama-0.2.14-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e61134b6949588b8aad4ae00f6e49d82",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0.0,>=3.10.0",
"size": 24397,
"upload_time": "2024-09-05T08:06:40",
"upload_time_iso_8601": "2024-09-05T08:06:40.852962Z",
"url": "https://files.pythonhosted.org/packages/24/cb/c99505d513ddc987bcca2f428ee88effff6485624594df6d885b923235e5/zrb_ollama-0.2.14-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e831de167b28c4d11310bbbf21c9a244216cf28bf9d07761e63a755adb253e7d",
"md5": "5a2bfca085279b1a45cdb24e7be391ef",
"sha256": "c1288e048cfdfcdb1fabe591b62efeb6a46db32c7e5e12425dc23ded5a50631a"
},
"downloads": -1,
"filename": "zrb_ollama-0.2.14.tar.gz",
"has_sig": false,
"md5_digest": "5a2bfca085279b1a45cdb24e7be391ef",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0.0,>=3.10.0",
"size": 20721,
"upload_time": "2024-09-05T08:06:43",
"upload_time_iso_8601": "2024-09-05T08:06:43.631359Z",
"url": "https://files.pythonhosted.org/packages/e8/31/de167b28c4d11310bbbf21c9a244216cf28bf9d07761e63a755adb253e7d/zrb_ollama-0.2.14.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-05 08:06:43",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "state-alchemists",
"github_project": "zrb-ollama",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "zrb-ollama"
}
JSON
