| Name | ragpipe JSON |
| Version |
0.0.2.9
JSON |
| download |
| home_page | None |
| Summary | ragpipe: iterate quickly on your RAG pipelines. |
| upload_time | 2024-11-27 15:56:17 |
| maintainer | None |
| docs_url | None |
| author | Nishant Sinha |
| requires_python | <3.13,>=3.8.1 |
| license | Apache 2.0 |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<h1 align="center" >ragpipe</h1>
<p align="center">
<img src="docs/src/assets/ragpipe.jpeg" width="30%" alt="Ragpipe Logo">
</p>
<h3 align="center">
Ragpipe: Iterate fast on your RAG pipelines.
<br><br>
<a href="https://ragpipe.github.io/">Docs</a> •
<a href="examples/">Examples</a> •
<a href="https://discord.com/invite/ATWd8A5cEh">Discord</a>
</h3>
## Introduction
Ragpipe helps you extract insights from large document repositories *quickly*.
Ragpipe is lean and nimble. Makes it easy to iterate fast, tweak components of your RAG pipeline until you get desired responses.
Yet another RAG framework? Although popular RAG frameworks make it easy to setup RAG pipelines, they lack primitives that enable you to iterate and get to desired responses quickly.
Watch a quick [video intro](https://www.youtube.com/playlist?list=PLLPfjV1xMkS1k9J7q2v3eQ2U-At6p3evM).
**Updates**
* 21.11.24. Added [SEC10k example](examples/sec10k/) - query any PDF using Colpali, Docling, VLMs.
* 25.10.24. Support for blazing fast [Model2Vec plugin](ext/libs/model2vec.py).
* 3.9.24. Quickstart *FounderMode* notebook [cookbook](examples/quickstart/pg.ipynb).
---
Instead of the usual `chunk-embed-match-rank` flow, Ragpipe adopts a holistic, end-to-end view of the pipeline:
- build a hierachical **document model**,
- **decompose** a complex query into sub-queries
- **resolve** sub-queries and obtain responses
- **aggregate** the query responses.
How do we resolve each sub-query?
- choose **representations** for document parts relevant to a sub-query,
- specify the **bridges** among those representations,
- **merge** the retrieved docs across bridges to setup a context,
- present the query and context to a language model to compute the final response
The `represent-bridge-merge` pattern is very powerful and allows us to build and iterate over all kinds of complex retrieval pipelines, including those based on the traditional `retrieve-rank-rerank` pattern and more recent advanced RAG patterns. Evals can be attached to `bridge` or `merge` nodes to verify intermediate results. See examples below.
*Note: Under active development. Expect breaking changes.*
## Installation
Using `pip`.
```bash
pip install ragpipe
```
Alternatively, clone the repository and use `pip` to install dependencies.
```bash
git clone https://github.com/ekshaks/ragpipe; cd ragpipe
#creating a new environment with python 3.10
conda create -n ragpipe python=3.10
#activating the environment
conda activate ragpipe
#install ragpipe dependencies
pip install -r requirements.txt
```
Note: For CUDA support on Windows/Linux you might need to install PyTorch with CUDA compiled.
For instructions follow https://pytorch.org/get-started/locally/
## Key Ideas
**Representations**. Choose the query/document fields as well as how to represent each chosen query / document field to aid similarity/relevance computation (*bridges*) over the entire document repository. Representations can be text strings, dense/sparse vector embeddings or arbitrary data objects, and help *bridge* the gap between the query and the documents.
**Bridges**. Choose a *pair* of query and document representation to *bridge*. A bridge serves as a relevance indicator: one of the several criteria for identifying the relevant documents for a query. In practice, several bridges together determine the degree to which a document is relevant to a query. A bridge is a ranker and top-k selector, rolled into one. Computing each bridge creates a unique ranked list of documents with respect to the relevance criteria.
**Merges**. Specify how to combine the bridges in sequential or parallel pipelines, e.g., combine ranked list of documents, retrieved via multiple bridges, into a single ranked list using rank fusion.
**Data Model**. A hierarchical data structure that consists of all the (nested) documents. The data model is created from the original document files and is retained over the entire pipeline. We compute representations for arbitrary nested fields of the data, without flattening the data tree.
## Querying with Ragpipe
To query over a data repository,
1. Build a hierachical data model over your data repositories, e.g., `{"documents" : [{"text": ...}, ...]}`.
2. In the `project.yml` config file:
- Specify which document fields will be represented and how.
- Specify which representations to compute for the query.
- Specify `bridges`: which pair of query and doc field representation should be matched to find relevant documents.
- Specify `merges`: how to combine multiple bridges, sequentially or in parallel, to yield the final ranked list of relevant documents.
3. Specify how to generate response to the query using the above ranked document list and a large language model.
4. Iterate by making quick changes to (1), (2) or (3).
## Quick Start
**Ragpipe CLI**. *(coming soon)*
**Configure and Run Pipeline**. Configure the query pipeline by building the data model, specifying representations/encoders for document fields and bridges for matching and ranking.
- Create two files `project.yml` (config file) and `project.py` (build data model) for your project.
- Run `python project.py`
- Quickstart templates are available here: [examples/quickstart/project.yml](project.yml), [examples/quickstart/project.py](project.py). Copy and modify as per your RAG pipeline and project structure.
The default LLM is [Groq](https://groq.com/). Please set GROQ_API_KEY in `.env`. Alternatively, openai LLMs (set `OPENAI_API_KEY`) and ollama based local LLMs (`ollama/..` or `local/..`) are also supported.
## Examples
A notebook explaining how to setup a simple **end-to-end RAG pipeline** with `ragpipe` is [here](examples/quickstart/pg.ipynb).
Several examples are in the [examples](examples) directory.
For instance, run [`examples/insurance`](examples/insurance).
```
examples/insurance/
|
|-- insurance.py
|-- insurance.yml
```
```bash
python -m examples.insurance.insurance
```
## API Usage
Embed ragpipe into your Agents by delegating fine-grained retrieval to ragpipe.
```python
def rag():
from ragpipe.config import load_config
config = load_config('examples/<project>/config.yml', show=True) #see examples/*/*.yml
query_text = config.queries[0] #user-provided query
D = build_data_model(config) # D.docs.<> contain documents
from ragpipe import Retriever
docs_retrieved = Retriever(config).eval(query_text, D)
for doc in docs_retrieved: doc.show()
from ragpipe.llms import respond_to_contextual_query as respond
result = respond(query_text, docs_retrieved, config.prompts['qa'], config.llm_models['default'])
print(f'\nQuery: {query_text}')
print('\nGenerated answer: ', result)
```
## Tests
```bash
pytest examples/test_all.py
```
## Key Dependencies
Ragpipe relies on
- `rank_bm25`: for BM25 based retrieval
- `fastembed`, `sentence-transformers`: dense and sparse embeddings
- `chromadb`, `qdrant-client`: vector databases (more coming..)
- `litellm`: interact with LLM APIs
- `jinja2`: prompt formatting
- `LlamaIndex`: for parsing documents
## Contribute
Ragpipe is open-source and under active development. We welcome contributions:
- Try out ragpipe on queries over your data. Open an issue or send a pull request.
- Join us as an early contributor to build a new, powerful and flexible RAG framework.
- Stuck on a RAG problem without progress? Share with us, iterate and overcome blockers.
Join discussion on our [Discord](https://discord.com/invite/ATWd8A5cEh) channel.
## Troubleshooting
- If you encounter errors related to protocol buffers, use the following fix: `export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python`
## Read More
- [Why your GPT + Vector Search RAG demo won't make it to production?](https://offnote.substack.com/p/llm-ir-1-why-your-gpt-vector-search)
- [RAG++: Bridging the Query - Doc Gap](https://offnote.substack.com/p/llm-ir-2-rag-from-scratch-bridging)
- [Lessons Building an Enterprise RAG Product](https://offnote.substack.com/p/lessons-building-an-enterprise-genai)
Raw data
{
"_id": null,
"home_page": null,
"name": "ragpipe",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.8.1",
"maintainer_email": null,
"keywords": null,
"author": "Nishant Sinha",
"author_email": "nishant@offnote.co",
"download_url": "https://files.pythonhosted.org/packages/c5/2c/a17078ed0ff22421802d364ba0d0a58661eb8a2ee2e30fbfcc9019897b76/ragpipe-0.0.2.9.tar.gz",
"platform": null,
"description": "\n<h1 align=\"center\" >ragpipe</h1>\n<p align=\"center\">\n\u00a0 \u00a0 <img src=\"docs/src/assets/ragpipe.jpeg\" width=\"30%\" alt=\"Ragpipe Logo\">\n</p>\n\n\n<h3 align=\"center\">\n Ragpipe: Iterate fast on your RAG pipelines.\n <br><br>\n\u00a0 <a href=\"https://ragpipe.github.io/\">Docs</a> \u2022\n\u00a0 <a href=\"examples/\">Examples</a> \u2022\n\u00a0<a href=\"https://discord.com/invite/ATWd8A5cEh\">Discord</a> \n</h3>\n\n\n## Introduction\n\nRagpipe helps you extract insights from large document repositories *quickly*. \n\nRagpipe is lean and nimble. Makes it easy to iterate fast, tweak components of your RAG pipeline until you get desired responses.\n\nYet another RAG framework? Although popular RAG frameworks make it easy to setup RAG pipelines, they lack primitives that enable you to iterate and get to desired responses quickly. \n\nWatch a quick [video intro](https://www.youtube.com/playlist?list=PLLPfjV1xMkS1k9J7q2v3eQ2U-At6p3evM).\n\n**Updates**\n\n* 21.11.24. Added [SEC10k example](examples/sec10k/) - query any PDF using Colpali, Docling, VLMs.\n* 25.10.24. Support for blazing fast [Model2Vec plugin](ext/libs/model2vec.py).\n* 3.9.24. Quickstart *FounderMode* notebook [cookbook](examples/quickstart/pg.ipynb).\n\n---\n\nInstead of the usual `chunk-embed-match-rank` flow, Ragpipe adopts a holistic, end-to-end view of the pipeline:\n\n- build a hierachical **document model**, \n- **decompose** a complex query into sub-queries \n- **resolve** sub-queries and obtain responses\n- **aggregate** the query responses.\n\nHow do we resolve each sub-query?\n- choose **representations** for document parts relevant to a sub-query, \n- specify the **bridges** among those representations, \n- **merge** the retrieved docs across bridges to setup a context,\n- present the query and context to a language model to compute the final response\n\nThe `represent-bridge-merge` pattern is very powerful and allows us to build and iterate over all kinds of complex retrieval pipelines, including those based on the traditional `retrieve-rank-rerank` pattern and more recent advanced RAG patterns. Evals can be attached to `bridge` or `merge` nodes to verify intermediate results. See examples below.\n\n*Note: Under active development. Expect breaking changes.*\n\n\n## Installation\n\nUsing `pip`.\n```bash\npip install ragpipe\n```\n\n\nAlternatively, clone the repository and use `pip` to install dependencies.\n```bash\ngit clone https://github.com/ekshaks/ragpipe; cd ragpipe\n#creating a new environment with python 3.10\nconda create -n ragpipe python=3.10\n#activating the environment\nconda activate ragpipe\n#install ragpipe dependencies\npip install -r requirements.txt\n```\n\nNote: For CUDA support on Windows/Linux you might need to install PyTorch with CUDA compiled.\nFor instructions follow https://pytorch.org/get-started/locally/\n\n## Key Ideas\n\n**Representations**. Choose the query/document fields as well as how to represent each chosen query / document field to aid similarity/relevance computation (*bridges*) over the entire document repository. Representations can be text strings, dense/sparse vector embeddings or arbitrary data objects, and help *bridge* the gap between the query and the documents.\n\n**Bridges**. Choose a *pair* of query and document representation to *bridge*. A bridge serves as a relevance indicator: one of the several criteria for identifying the relevant documents for a query. In practice, several bridges together determine the degree to which a document is relevant to a query. A bridge is a ranker and top-k selector, rolled into one. Computing each bridge creates a unique ranked list of documents with respect to the relevance criteria.\n\n**Merges**. Specify how to combine the bridges in sequential or parallel pipelines, e.g., combine ranked list of documents, retrieved via multiple bridges, into a single ranked list using rank fusion.\n\n**Data Model**. A hierarchical data structure that consists of all the (nested) documents. The data model is created from the original document files and is retained over the entire pipeline. We compute representations for arbitrary nested fields of the data, without flattening the data tree.\n\n\n## Querying with Ragpipe\n\nTo query over a data repository, \n\n1. Build a hierachical data model over your data repositories, e.g., `{\"documents\" : [{\"text\": ...}, ...]}`. \n\n2. In the `project.yml` config file:\n\n- Specify which document fields will be represented and how.\n- Specify which representations to compute for the query.\n- Specify `bridges`: which pair of query and doc field representation should be matched to find relevant documents.\n- Specify `merges`: how to combine multiple bridges, sequentially or in parallel, to yield the final ranked list of relevant documents.\n\n3. Specify how to generate response to the query using the above ranked document list and a large language model.\n4. Iterate by making quick changes to (1), (2) or (3).\n\n## Quick Start\n\n**Ragpipe CLI**. *(coming soon)*\n\n**Configure and Run Pipeline**. Configure the query pipeline by building the data model, specifying representations/encoders for document fields and bridges for matching and ranking.\n\n- Create two files `project.yml` (config file) and `project.py` (build data model) for your project.\n- Run `python project.py`\n- Quickstart templates are available here: [examples/quickstart/project.yml](project.yml), [examples/quickstart/project.py](project.py). Copy and modify as per your RAG pipeline and project structure.\n \n\nThe default LLM is [Groq](https://groq.com/). Please set GROQ_API_KEY in `.env`. Alternatively, openai LLMs (set `OPENAI_API_KEY`) and ollama based local LLMs (`ollama/..` or `local/..`) are also supported.\n\n## Examples\n\nA notebook explaining how to setup a simple **end-to-end RAG pipeline** with `ragpipe` is [here](examples/quickstart/pg.ipynb).\n\nSeveral examples are in the [examples](examples) directory.\n\nFor instance, run [`examples/insurance`](examples/insurance).\n```\nexamples/insurance/\n|\n|-- insurance.py\n|-- insurance.yml\n```\n\n```bash \npython -m examples.insurance.insurance\n```\n\n\n## API Usage\n\nEmbed ragpipe into your Agents by delegating fine-grained retrieval to ragpipe.\n\n```python\n\ndef rag():\n from ragpipe.config import load_config\n config = load_config('examples/<project>/config.yml', show=True) #see examples/*/*.yml\n\n query_text = config.queries[0] #user-provided query\n D = build_data_model(config) # D.docs.<> contain documents\n\n from ragpipe import Retriever\n docs_retrieved = Retriever(config).eval(query_text, D)\n for doc in docs_retrieved: doc.show()\n\n from ragpipe.llms import respond_to_contextual_query as respond\n result = respond(query_text, docs_retrieved, config.prompts['qa'], config.llm_models['default']) \n \n print(f'\\nQuery: {query_text}')\n print('\\nGenerated answer: ', result)\n```\n\n## Tests\n\n```bash\npytest examples/test_all.py\n```\n\n\n## Key Dependencies\n\nRagpipe relies on \n- `rank_bm25`: for BM25 based retrieval\n- `fastembed`, `sentence-transformers`: dense and sparse embeddings\n- `chromadb`, `qdrant-client`: vector databases (more coming..)\n- `litellm`: interact with LLM APIs\n- `jinja2`: prompt formatting\n- `LlamaIndex`: for parsing documents\n\n\n## Contribute\n\nRagpipe is open-source and under active development. We welcome contributions:\n- Try out ragpipe on queries over your data. Open an issue or send a pull request.\n- Join us as an early contributor to build a new, powerful and flexible RAG framework.\n- Stuck on a RAG problem without progress? Share with us, iterate and overcome blockers.\n\n\nJoin discussion on our [Discord](https://discord.com/invite/ATWd8A5cEh) channel.\n\n\n## Troubleshooting\n\n- If you encounter errors related to protocol buffers, use the following fix: `export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python`\n\n## Read More\n\n- [Why your GPT + Vector Search RAG demo won't make it to production?](https://offnote.substack.com/p/llm-ir-1-why-your-gpt-vector-search)\n- [RAG++: Bridging the Query - Doc Gap](https://offnote.substack.com/p/llm-ir-2-rag-from-scratch-bridging)\n- [Lessons Building an Enterprise RAG Product](https://offnote.substack.com/p/lessons-building-an-enterprise-genai)",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "ragpipe: iterate quickly on your RAG pipelines.",
"version": "0.0.2.9",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "af55485b0667fc8767ca74b2b97ad3e3d94ff5c0cba11d0fc7648f68df55ddb2",
"md5": "4b5167f6146da82ae35fd42a4d30b131",
"sha256": "d5cc69a7adce9b265b689a7c9afdc3fcf18fd71b4eaea36464ccf8fa58c16d59"
},
"downloads": -1,
"filename": "ragpipe-0.0.2.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4b5167f6146da82ae35fd42a4d30b131",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.8.1",
"size": 80520,
"upload_time": "2024-11-27T15:56:14",
"upload_time_iso_8601": "2024-11-27T15:56:14.802914Z",
"url": "https://files.pythonhosted.org/packages/af/55/485b0667fc8767ca74b2b97ad3e3d94ff5c0cba11d0fc7648f68df55ddb2/ragpipe-0.0.2.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c52ca17078ed0ff22421802d364ba0d0a58661eb8a2ee2e30fbfcc9019897b76",
"md5": "d1e75b427ba903382868df4dbd9e6e8e",
"sha256": "0ddd2dccd8da56f763721e5b3eac66f94e56fdf102bbd39b970c2055efe062a1"
},
"downloads": -1,
"filename": "ragpipe-0.0.2.9.tar.gz",
"has_sig": false,
"md5_digest": "d1e75b427ba903382868df4dbd9e6e8e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.8.1",
"size": 66510,
"upload_time": "2024-11-27T15:56:17",
"upload_time_iso_8601": "2024-11-27T15:56:17.286205Z",
"url": "https://files.pythonhosted.org/packages/c5/2c/a17078ed0ff22421802d364ba0d0a58661eb8a2ee2e30fbfcc9019897b76/ragpipe-0.0.2.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-27 15:56:17",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "ragpipe"
}
