Langchain-RAG
=============
[](https://codespaces.new/pprados/langchain-rag?quickstart=1)
> Note: A [pull-request](https://github.com/langchain-ai/langchain/pull/13910) with this code was proposed to langchain.
When splitting documents for retrieval, there are often conflicting desires:
1. You may want to keep documents small, ensuring that their embeddings accurately represent their meaning. If they become too long, the embeddings can lose their meaning.
2. You also want to maintain documents long enough to retain the context of each chunk.
When you have a lot of documents, and therefore a lot of pieces, it's likely that dozens of pieces have a distance close to the question. Taking only the top 4 is not a good idea. The answer may lie in the 6 or 7 tracks. How can we improve the match between the question and a fragment? By preparing several versions of the fragment, each with an embedding. In this way, one of the versions can be closer to the question than the original fragment. This version is stripped of context. But the context is still needed to answer the question correctly. One strategy consists of breaking down each fragment into different versions, but using the retriever to return to the original fragment.
The `RAGVectorStore` strikes a balance by splitting and storing small chunks and different variations of data. During retrieval, it initially retrieves the small chunks but then looks up the parent IDs for those chunks and returns the larger documents.
The challenge lies in correctly managing the lifecycle of the three levels of documents:
- Original documents
- Chunks extracted from the original documents
- Transformations of chunks to generate more vectors for improved retrieval
The `RAGVectorStore`, in combination with other components, is designed to address this challenge.
# Demo
Read [this notebook](docs/integrations/vectorstores/rag_vectorstore.ipynb)
Or :
- `poetry run python -m ipykernel install --user --name langchain-rag`
- `jupyter lab`
# Tips
`poetry run python -m ipykernel install --user --name langchain-parent`
Raw data
{
"_id": null,
"home_page": "https://www.github.com/pprados/langchain-rag",
"name": "langchain-rag",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.9",
"maintainer_email": null,
"keywords": null,
"author": "Philippe PRADOS",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/af/dd/44eb1e55af3993922cfb91f827008faab619a1f513c64f078ffb05e61edf/langchain_rag-0.2.0.tar.gz",
"platform": null,
"description": "Langchain-RAG\n=============\n\n[](https://codespaces.new/pprados/langchain-rag?quickstart=1)\n\n> Note: A [pull-request](https://github.com/langchain-ai/langchain/pull/13910) with this code was proposed to langchain.\n\nWhen splitting documents for retrieval, there are often conflicting desires:\n\n1. You may want to keep documents small, ensuring that their embeddings accurately represent their meaning. If they become too long, the embeddings can lose their meaning.\n2. You also want to maintain documents long enough to retain the context of each chunk.\n\nWhen you have a lot of documents, and therefore a lot of pieces, it's likely that dozens of pieces have a distance close to the question. Taking only the top 4 is not a good idea. The answer may lie in the 6 or 7 tracks. How can we improve the match between the question and a fragment? By preparing several versions of the fragment, each with an embedding. In this way, one of the versions can be closer to the question than the original fragment. This version is stripped of context. But the context is still needed to answer the question correctly. One strategy consists of breaking down each fragment into different versions, but using the retriever to return to the original fragment. \n\nThe `RAGVectorStore` strikes a balance by splitting and storing small chunks and different variations of data. During retrieval, it initially retrieves the small chunks but then looks up the parent IDs for those chunks and returns the larger documents.\n\nThe challenge lies in correctly managing the lifecycle of the three levels of documents:\n- Original documents\n- Chunks extracted from the original documents\n- Transformations of chunks to generate more vectors for improved retrieval\n\nThe `RAGVectorStore`, in combination with other components, is designed to address this challenge.\n\n# Demo\nRead [this notebook](docs/integrations/vectorstores/rag_vectorstore.ipynb)\nOr :\n- `poetry run python -m ipykernel install --user --name langchain-rag`\n- `jupyter lab`\n\n# Tips\n`poetry run python -m ipykernel install --user --name langchain-parent`\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "This is a temporary project while I wait for my langchain [pull-request](https://github.com/langchain-ai/langchain/pull/7278) to be validated.",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://www.github.com/pprados/langchain-rag",
"Repository": "https://www.github.com/pprados/langchain-rag"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a3166509e3ffdb00a8e0bdc942457b2535a4b313e70a21fdad52422e2241ad5d",
"md5": "5251b89d222d4b655a1bebf4fdbba459",
"sha256": "89bf9856944161c36f0a5cc498fa311562b9b701fcd40d284acfc1c81364a128"
},
"downloads": -1,
"filename": "langchain_rag-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5251b89d222d4b655a1bebf4fdbba459",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.9",
"size": 34121,
"upload_time": "2024-05-21T10:23:26",
"upload_time_iso_8601": "2024-05-21T10:23:26.825206Z",
"url": "https://files.pythonhosted.org/packages/a3/16/6509e3ffdb00a8e0bdc942457b2535a4b313e70a21fdad52422e2241ad5d/langchain_rag-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "afdd44eb1e55af3993922cfb91f827008faab619a1f513c64f078ffb05e61edf",
"md5": "495a67682b55fef1b4785da8f4fc6fbc",
"sha256": "754b6fa09ee716b90ffd93cd29cfc9526bb34272e1a5f9e2b0a23caa68fc8172"
},
"downloads": -1,
"filename": "langchain_rag-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "495a67682b55fef1b4785da8f4fc6fbc",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.9",
"size": 26922,
"upload_time": "2024-05-21T10:23:28",
"upload_time_iso_8601": "2024-05-21T10:23:28.549348Z",
"url": "https://files.pythonhosted.org/packages/af/dd/44eb1e55af3993922cfb91f827008faab619a1f513c64f078ffb05e61edf/langchain_rag-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-21 10:23:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "pprados",
"github_project": "langchain-rag",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "langchain-rag"
}
JSON
