# dbt-llm-tools
LLM based tools for dbt projects. Answer data questions, generate documentation and more.
The library comes with a streamlit interface that allows you to interact with your dbt project via a UI.
In addition, you can also access the underlying classes that enable you to:
- Chatbot: ask questions about data and get answers based on your dbt model documentation
- Documentation Generator: generate documentation for dbt models based on model and upstream model definition.
**Here is a quick demo of how the Chatbot works:**
https://www.loom.com/share/abb0612c4e884d4cb8fabc22af964e7e?sid=f5f8c0e6-51f5-4afc-a7bf-51e9e182c2e7
## Get Started
You can install `dbt-llm-tools` with the UI and interact with your project via a streamlit interface.
Alternatively, you can also install without the UI to use the underlying classes only.
### Option 1 - With UI
To install with the UI:
1. Clone the repository on your computer: `gh repo clone pragunbhutani/dbt-llm-tools`
2. `cd` into the repository: `cd dbt-llm-tools`
3. The project uses poetry to install dependencies. If you don't have poetry installed already, run `make poetry`.
- Remember to add the poetry executable to your $PATH and refresh your terminal.
4. Run `make install` to download and set up all of the dependencies
- (optional) Download an example dbt project with `make fetch_example_project`.
5. Run the client with `make run_client`
You should then be able to see the client run in your browser at `http://localhost:8501/app`
**Note** - To use the tools, you'll need an OpenAI API Key.
### Option 2 - Without UI
dbt-llm-tools can be installed via pip.
```
pip install dbt-llm-tools
```
## Documentation
The following shows you examples of how to use the two main classes of `dbt-llm-tools`. You can also find full documentation for these and all other classes at https://dbt-llm-tools.readthedocs.io/en/latest/
### Class - Chatbot
How to load your dbt project into the Chatbot and ask questions about your data.
```Python
from dbt_llm_tools import Chatbot
# Instantiate a chatbot object
chatbot = Chatbot(
dbt_project_root='/path/to/dbt/project',
openai_api_key='YOUR_OPENAI_API_KEY',
)
# Step 1. Load models information from your dbt ymls into a local vector store
chatbot.load_models()
# Step 2. Ask the chatbot a question
response = chatbot.ask_question(
'How can I obtain the number of customers who upgraded to a paid plan in the last 3 months?'
)
print(response)
```
**Note**: dbt-llm-tools currently only supports OpenAI ChatGPT models for generating embeddings and responses to queries.
#### How it works
The Chatbot is based on the concept of Retrieval Augmented Generation and basically works as follows:
- When you call the `chatbot.load_models()` method, the bot scans all the folders in the locations specified by you for dbt YML files.
- It then converts all the models into a text description, which are stored as embeddings in a vector database. The bot currently only supports [ChromaDB](https://www.trychroma.com/) as a vector db, which is persisted in a file on your local machine.
- When you ask a query, it fetches 3 models whose description is found to be the most relevant for your query.
- These models are then fed into ChatGPT as a prompt, along with some basic instructions and your question.
- The response is returned to you as a string.
### Class - Documentation Generator
How to load your dbt project into the Documentation Generator and have it write documentation for your models.
```Python
from dbt_llm_tools import DocumentationGenerator
# Instantiate a Documentation Generator object
doc_gen = DocumentationGenerator(
dbt_project_root="YOUR_DBT_PROJECT_PATH",
openai_api_key="YOUR_OPENAI_API_KEY",
)
# Generate documentation for a model and all its upstream models
doc_gen.generate_documentation(
model_name='dbt_model_name',
write_documentation_to_yaml=False
)
```
Raw data
{
"_id": null,
"home_page": "https://github.com/pragunbhutani/dbt-llm-tools",
"name": "dbt-llm-tools",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": "dbt, openai, llm, data chatbot, dbt documentation",
"author": "Pragun Bhutani",
"author_email": "1109752+pragunbhutani@users.noreply.github.com",
"download_url": "https://files.pythonhosted.org/packages/e9/fc/cab846ada49e3c8c6ac0f37d0d5643900f47064c6b9825db51324c097efc/dbt_llm_tools-0.2.0.tar.gz",
"platform": null,
"description": "# dbt-llm-tools\n\nLLM based tools for dbt projects. Answer data questions, generate documentation and more.\nThe library comes with a streamlit interface that allows you to interact with your dbt project via a UI.\n\nIn addition, you can also access the underlying classes that enable you to:\n\n- Chatbot: ask questions about data and get answers based on your dbt model documentation\n- Documentation Generator: generate documentation for dbt models based on model and upstream model definition.\n\n**Here is a quick demo of how the Chatbot works:**\n\nhttps://www.loom.com/share/abb0612c4e884d4cb8fabc22af964e7e?sid=f5f8c0e6-51f5-4afc-a7bf-51e9e182c2e7\n\n## Get Started\n\nYou can install `dbt-llm-tools` with the UI and interact with your project via a streamlit interface.\nAlternatively, you can also install without the UI to use the underlying classes only.\n\n### Option 1 - With UI\n\nTo install with the UI:\n\n1. Clone the repository on your computer: `gh repo clone pragunbhutani/dbt-llm-tools`\n2. `cd` into the repository: `cd dbt-llm-tools`\n3. The project uses poetry to install dependencies. If you don't have poetry installed already, run `make poetry`.\n - Remember to add the poetry executable to your $PATH and refresh your terminal.\n4. Run `make install` to download and set up all of the dependencies\n - (optional) Download an example dbt project with `make fetch_example_project`.\n5. Run the client with `make run_client`\n\nYou should then be able to see the client run in your browser at `http://localhost:8501/app`\n\n**Note** - To use the tools, you'll need an OpenAI API Key.\n\n### Option 2 - Without UI\n\ndbt-llm-tools can be installed via pip.\n\n```\npip install dbt-llm-tools\n```\n\n## Documentation\n\nThe following shows you examples of how to use the two main classes of `dbt-llm-tools`. You can also find full documentation for these and all other classes at https://dbt-llm-tools.readthedocs.io/en/latest/\n\n### Class - Chatbot\n\nHow to load your dbt project into the Chatbot and ask questions about your data.\n\n```Python\nfrom dbt_llm_tools import Chatbot\n\n# Instantiate a chatbot object\nchatbot = Chatbot(\n\tdbt_project_root='/path/to/dbt/project',\n\topenai_api_key='YOUR_OPENAI_API_KEY',\n)\n\n# Step 1. Load models information from your dbt ymls into a local vector store\nchatbot.load_models()\n\n# Step 2. Ask the chatbot a question\nresponse = chatbot.ask_question(\n\t'How can I obtain the number of customers who upgraded to a paid plan in the last 3 months?'\n)\nprint(response)\n```\n\n**Note**: dbt-llm-tools currently only supports OpenAI ChatGPT models for generating embeddings and responses to queries.\n\n#### How it works\n\nThe Chatbot is based on the concept of Retrieval Augmented Generation and basically works as follows:\n\n- When you call the `chatbot.load_models()` method, the bot scans all the folders in the locations specified by you for dbt YML files.\n- It then converts all the models into a text description, which are stored as embeddings in a vector database. The bot currently only supports [ChromaDB](https://www.trychroma.com/) as a vector db, which is persisted in a file on your local machine.\n- When you ask a query, it fetches 3 models whose description is found to be the most relevant for your query.\n- These models are then fed into ChatGPT as a prompt, along with some basic instructions and your question.\n- The response is returned to you as a string.\n\n### Class - Documentation Generator\n\nHow to load your dbt project into the Documentation Generator and have it write documentation for your models.\n\n```Python\nfrom dbt_llm_tools import DocumentationGenerator\n\n# Instantiate a Documentation Generator object\ndoc_gen = DocumentationGenerator(\n\tdbt_project_root=\"YOUR_DBT_PROJECT_PATH\",\n\topenai_api_key=\"YOUR_OPENAI_API_KEY\",\n)\n\n# Generate documentation for a model and all its upstream models\ndoc_gen.generate_documentation(\n\tmodel_name='dbt_model_name',\n\twrite_documentation_to_yaml=False\n)\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "LLM based tools for dbt projects. Answer data questions, generate documentation and more.",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/pragunbhutani/dbt-llm-tools",
"Repository": "https://github.com/pragunbhutani/dbt-llm-tools"
},
"split_keywords": [
"dbt",
" openai",
" llm",
" data chatbot",
" dbt documentation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8fa1df07190f5eb6c83af04bf8132ce28647174f1b7b456f13b5aefa03fa683b",
"md5": "c82a805ddcc53fe1108eca2ab3031bfa",
"sha256": "720c36fc9b881df05ae3b15c7408c53e2046a93e490407ae8bee196e48f1dd00"
},
"downloads": -1,
"filename": "dbt_llm_tools-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c82a805ddcc53fe1108eca2ab3031bfa",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 16290,
"upload_time": "2024-04-20T08:23:14",
"upload_time_iso_8601": "2024-04-20T08:23:14.174410Z",
"url": "https://files.pythonhosted.org/packages/8f/a1/df07190f5eb6c83af04bf8132ce28647174f1b7b456f13b5aefa03fa683b/dbt_llm_tools-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e9fccab846ada49e3c8c6ac0f37d0d5643900f47064c6b9825db51324c097efc",
"md5": "0e77a52e95f89626847677d86d2f6e24",
"sha256": "e8507e676e7fe8c6f5f69cfdad5817127e75aeddc69dd98bcbf5aa422e067ab2"
},
"downloads": -1,
"filename": "dbt_llm_tools-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "0e77a52e95f89626847677d86d2f6e24",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 14407,
"upload_time": "2024-04-20T08:23:15",
"upload_time_iso_8601": "2024-04-20T08:23:15.215831Z",
"url": "https://files.pythonhosted.org/packages/e9/fc/cab846ada49e3c8c6ac0f37d0d5643900f47064c6b9825db51324c097efc/dbt_llm_tools-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-20 08:23:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pragunbhutani",
"github_project": "dbt-llm-tools",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "dbt-llm-tools"
}
JSON
