| Name | pebblo JSON |
| Version |
0.1.20
JSON |
| download |
| home_page | None |
| Summary | Pebblo Gen-AI Data Analyzer |
| upload_time | 2024-10-10 06:34:53 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9 |
| license | None |
| keywords |
langchain
ai
rag
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<p align="center">
<img src="https://github.com/daxa-ai/pebblo/blob/main/docs/gh_pages/static/img/pebblo-logo-name.jpg?raw=true" />
</p>
---
[](https://github.com/daxa-ai/pebblo)
[](http://opensource.org/licenses/MIT)
[](https://daxa-ai.github.io/pebblo/)
[](https://pypi.org/project/pebblo/)
[](https://www.linkedin.com/company/daxa-ai)
[](https://discord.gg/wyAfaYXwwv)
[](https://twitter.com/daxa_ai)
---
**Pebblo** enables developers to safely load data and promote their Gen AI app to deployment without worrying about the organization’s compliance and security requirements. The project identifies semantic topics and entities found in the loaded data and summarizes them on the UI or a PDF report.
Pebblo has these components.
1. Pebblo Server - a REST api application with topic-classifier, entity-classifier and reporting features
1. Pebblo SafeLoader - a thin wrapper to Gen-AI framework's data loaders
1. Pebblo SafeRetriever - a retrieval QA chain that enforces identity and semantic rules on Vector database retrieval before LLM inference
## Pebblo Server
### Installation
#### Using `pip`
```bash
pip install pebblo --extra-index-url https://packages.daxa.ai/simple/
```
#### Download python package
Alternatively, download and install the latest Pebblo python `.whl` package from URL https://packages.daxa.ai/pebblo/0.1.13/pebblo-0.1.13-py3-none-any.whl
Example:
```bash
curl -LO "https://packages.daxa.ai/pebblo/0.1.13/pebblo-0.1.13-py3-none-any.whl"
pip install pebblo-0.1.13-py3-none-any.whl
```
### Run Pebblo Server
```bash
pebblo
```
Pebblo Server now listens to `localhost:8000` to accept Gen-AI application data snippets for inspection and reporting.
##### Pebblo Optional Flags
- `--config <file>`: specify a configuration file in yaml format.
See [configuration](docs/gh_pages/docs/config.md) guide for knobs to control Pebblo Server behavior like enabling snippet anonymization, selecting specific report renderer, etc.
### Using Docker
```bash
docker run -p 8000:8000 docker.daxa.ai/daxaai/pebblo
```
Local UI can be accessed by pointing the browser to `https://localhost:8000`.
See [installation](docs/gh_pages/docs/installation.md) guide for details on how to pass custom config.yaml and accessing PDF reports in the host machine.
### Troubleshooting
Refer to [troubleshooting](docs/gh_pages/docs/troubleshooting.md) guide.
## Pebblo SafeLoader
### Langchain
`Pebblo SafeLoader` is natively supported in Langchain framework. It is available in Langchain versions `>=0.1.7`
#### Enable Pebblo in Langchain Application
Add `PebbloSafeLoader` wrapper to the existing Langchain document loader(s) used in the RAG application. `PebbloSafeLoader` is interface compatible with Langchain `BaseLoader`. The application can continue to use `load()` and `lazy_load()` methods as it would on a Langchain document loader.
Here is the snippet of Langchain RAG application using `CSVLoader` before enabling `PebbloSafeLoader`.
```python
from langchain_community.document_loaders import CSVLoader
loader = CSVLoader(file_path)
documents = loader.load()
vectordb = Chroma.from_documents(documents, OpenAIEmbeddings())
```
The Pebblo SafeLoader can be enabled with few lines of code change to the above snippet.
```python
from langchain_community.document_loaders import CSVLoader
from langchain_community.document_loaders.pebblo import PebbloSafeLoader
loader = PebbloSafeLoader(
CSVLoader(file_path),
name="acme-corp-rag-1", # App name (Mandatory)
owner="Joe Smith", # Owner (Optional)
description="Support productivity RAG application", # Description (Optional)
)
documents = loader.load()
vectordb = Chroma.from_documents(documents, OpenAIEmbeddings())
```
See [here](https://github.com/srics/pebblo/tree/main/pebblo_safeloader) for samples with Pebblo SafeLoader enabled RAG applications and [this](https://daxa-ai.github.io/pebblo/rag) document for more details.
## Pebblo SafeRetriever
### Langchain
PebbloRetrievalQA chain uses a SafeRetrieval to enforce that the snippets used for in-context are retrieved
only from the documents authorized for the user and semantically allowed for the Gen-AI application.
Here is a sample code for the PebbloRetrievalQA with `authorized_identities` from the user accessing the RAG
application, passed in `auth_context`.
```python
from langchain_community.chains import PebbloRetrievalQA
from langchain_community.chains.pebblo_retrieval.models import AuthContext, ChainInput
safe_rag_chain = PebbloRetrievalQA.from_chain_type(
llm=llm,
app_name="pebblo-safe-retriever-demo",
owner="Joe Smith",
description="Safe RAG demo using Pebblo",
chain_type="stuff",
retriever=vectordb.as_retriever(),
verbose=True,
)
def ask(question: str, auth_context: dict):
auth_context_obj = AuthContext(**auth_context)
chain_input_obj = ChainInput(query=question, auth_context=auth_context_obj)
return safe_rag_chain.invoke(chain_input_obj.dict())
```
See [here](https://github.com/srics/pebblo/tree/main/pebblo_saferetriever) for samples with Pebblo SafeRetriever enabled RAG applications and [this](https://daxa-ai.github.io/pebblo/retrieval_chain) document for more details.
# Contribution
Pebblo is a open-source community project. If you want to contribute see [Contributor Guidelines](https://github.com/daxa-ai/pebblo/blob/main/CONTRIBUTING.md) for more details.
# License
Pebblo is released under the MIT License
Raw data
{
"_id": null,
"home_page": null,
"name": "pebblo",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "Pebblo Maintainer <pebblo@daxa.ai>",
"keywords": "langchain, ai, rag",
"author": null,
"author_email": "Pebblo Authors <pebblo@daxa.ai>",
"download_url": null,
"platform": null,
"description": "<p align=\"center\">\n <img src=\"https://github.com/daxa-ai/pebblo/blob/main/docs/gh_pages/static/img/pebblo-logo-name.jpg?raw=true\" />\n</p>\n\n---\n[](https://github.com/daxa-ai/pebblo)\n[](http://opensource.org/licenses/MIT)\n[](https://daxa-ai.github.io/pebblo/)\n\n[](https://pypi.org/project/pebblo/)\n\n\n\n[](https://www.linkedin.com/company/daxa-ai)\n[](https://discord.gg/wyAfaYXwwv)\n[](https://twitter.com/daxa_ai)\n---\n\n\n**Pebblo** enables developers to safely load data and promote their Gen AI app to deployment without worrying about the organization\u2019s compliance and security requirements. The project identifies semantic topics and entities found in the loaded data and summarizes them on the UI or a PDF report.\n\nPebblo has these components.\n\n1. Pebblo Server - a REST api application with topic-classifier, entity-classifier and reporting features\n1. Pebblo SafeLoader - a thin wrapper to Gen-AI framework's data loaders\n1. Pebblo SafeRetriever - a retrieval QA chain that enforces identity and semantic rules on Vector database retrieval before LLM inference\n\n## Pebblo Server\n\n### Installation\n \n#### Using `pip`\n\n```bash\npip install pebblo --extra-index-url https://packages.daxa.ai/simple/\n```\n\n#### Download python package\nAlternatively, download and install the latest Pebblo python `.whl` package from URL https://packages.daxa.ai/pebblo/0.1.13/pebblo-0.1.13-py3-none-any.whl\n\nExample:\n```bash\ncurl -LO \"https://packages.daxa.ai/pebblo/0.1.13/pebblo-0.1.13-py3-none-any.whl\" \npip install pebblo-0.1.13-py3-none-any.whl\n```\n### Run Pebblo Server\n\n```bash\npebblo\n```\n\nPebblo Server now listens to `localhost:8000` to accept Gen-AI application data snippets for inspection and reporting.\n\n##### Pebblo Optional Flags\n\n- `--config <file>`: specify a configuration file in yaml format.\n\n\nSee [configuration](docs/gh_pages/docs/config.md) guide for knobs to control Pebblo Server behavior like enabling snippet anonymization, selecting specific report renderer, etc.\n\n### Using Docker\n\n```bash\ndocker run -p 8000:8000 docker.daxa.ai/daxaai/pebblo\n```\n\nLocal UI can be accessed by pointing the browser to `https://localhost:8000`.\n\nSee [installation](docs/gh_pages/docs/installation.md) guide for details on how to pass custom config.yaml and accessing PDF reports in the host machine.\n\n### Troubleshooting\n\nRefer to [troubleshooting](docs/gh_pages/docs/troubleshooting.md) guide.\n\n## Pebblo SafeLoader\n\n### Langchain\n\n`Pebblo SafeLoader` is natively supported in Langchain framework. It is available in Langchain versions `>=0.1.7`\n\n#### Enable Pebblo in Langchain Application\n\nAdd `PebbloSafeLoader` wrapper to the existing Langchain document loader(s) used in the RAG application. `PebbloSafeLoader` is interface compatible with Langchain `BaseLoader`. The application can continue to use `load()` and `lazy_load()` methods as it would on a Langchain document loader.\n\nHere is the snippet of Langchain RAG application using `CSVLoader` before enabling `PebbloSafeLoader`.\n\n```python\n from langchain_community.document_loaders import CSVLoader\n\n loader = CSVLoader(file_path)\n documents = loader.load()\n vectordb = Chroma.from_documents(documents, OpenAIEmbeddings())\n```\n\nThe Pebblo SafeLoader can be enabled with few lines of code change to the above snippet.\n\n```python\n from langchain_community.document_loaders import CSVLoader\n from langchain_community.document_loaders.pebblo import PebbloSafeLoader\n\n loader = PebbloSafeLoader(\n CSVLoader(file_path),\n name=\"acme-corp-rag-1\", # App name (Mandatory)\n owner=\"Joe Smith\", # Owner (Optional)\n description=\"Support productivity RAG application\", # Description (Optional)\n )\n documents = loader.load()\n vectordb = Chroma.from_documents(documents, OpenAIEmbeddings())\n```\n\nSee [here](https://github.com/srics/pebblo/tree/main/pebblo_safeloader) for samples with Pebblo SafeLoader enabled RAG applications and [this](https://daxa-ai.github.io/pebblo/rag) document for more details.\n\n## Pebblo SafeRetriever\n\n### Langchain\n\n\nPebbloRetrievalQA chain uses a SafeRetrieval to enforce that the snippets used for in-context are retrieved\nonly from the documents authorized for the user and semantically allowed for the Gen-AI application.\n\nHere is a sample code for the PebbloRetrievalQA with `authorized_identities` from the user accessing the RAG\napplication, passed in `auth_context`.\n\n```python\nfrom langchain_community.chains import PebbloRetrievalQA\nfrom langchain_community.chains.pebblo_retrieval.models import AuthContext, ChainInput\n\nsafe_rag_chain = PebbloRetrievalQA.from_chain_type(\n llm=llm,\n app_name=\"pebblo-safe-retriever-demo\",\n owner=\"Joe Smith\",\n description=\"Safe RAG demo using Pebblo\",\n chain_type=\"stuff\",\n retriever=vectordb.as_retriever(),\n verbose=True,\n)\n\ndef ask(question: str, auth_context: dict):\n auth_context_obj = AuthContext(**auth_context)\n chain_input_obj = ChainInput(query=question, auth_context=auth_context_obj)\n return safe_rag_chain.invoke(chain_input_obj.dict())\n```\n\nSee [here](https://github.com/srics/pebblo/tree/main/pebblo_saferetriever) for samples with Pebblo SafeRetriever enabled RAG applications and [this](https://daxa-ai.github.io/pebblo/retrieval_chain) document for more details.\n\n# Contribution\n\nPebblo is a open-source community project. If you want to contribute see [Contributor Guidelines](https://github.com/daxa-ai/pebblo/blob/main/CONTRIBUTING.md) for more details.\n\n# License\n\nPebblo is released under the MIT License\n",
"bugtrack_url": null,
"license": null,
"summary": "Pebblo Gen-AI Data Analyzer",
"version": "0.1.20",
"project_urls": {
"Bug Reports": "https://github.com/daxa-ai/pebblo/issues",
"Funding": "https://donate.pypi.org",
"Homepage": "https://github.com/daxa-ai/pebblo",
"Source": "https://github.com/daxa-ai/pebblo/"
},
"split_keywords": [
"langchain",
" ai",
" rag"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a67c64e977eee8dc403b3bd553388af175bd2463f74d783a89c16c89f9b4a3ea",
"md5": "f0149eb70679b92b06b3777298e872bf",
"sha256": "c68007e92cab4969b34236c4d2f828f4efac8518b81bb0776217d8cb7eb4ea6c"
},
"downloads": -1,
"filename": "pebblo-0.1.20-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f0149eb70679b92b06b3777298e872bf",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 3509006,
"upload_time": "2024-10-10T06:34:53",
"upload_time_iso_8601": "2024-10-10T06:34:53.742218Z",
"url": "https://files.pythonhosted.org/packages/a6/7c/64e977eee8dc403b3bd553388af175bd2463f74d783a89c16c89f9b4a3ea/pebblo-0.1.20-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-10 06:34:53",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "daxa-ai",
"github_project": "pebblo",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pebblo"
}
