# RagXO
Export, version and reuse your E2E RAG pipeline everywhere 🚀
[](https://badge.fury.io/py/ragxo)
[](https://opensource.org/licenses/MIT)
[](https://www.python.org/downloads/release/python-380/)
## Table of Contents
- [Features](#features-)
- [Installation](#installation-️)
- [Quickstart](#quickstart-)
- [Build a RAG pipeline](#build-a-rag-pipeline)
- [Load a RAG pipeline](#load-a-rag-pipeline)
- [Usage Guide](#usage-guide-)
- [Import](#import)
- [Adding Preprocessing Steps](#adding-preprocessing-steps)
- [Custom Embedding Functions](#custom-embedding-functions)
- [Creating Documents](#creating-documents)
- [LLM Configuration](#llm-configuration)
- [Export and Load](#export-and-load)
- [Evaluation](#evaluation)
- [Best Practices](#best-practices-)
- [License](#license-)
- [Contributing](#contributing-)
RagXO extends the capabilities of traditional RAG (Retrieval-Augmented Generation) systems by providing a unified way to package, version, and deploy your entire RAG pipeline with LLM integration. Export your complete system—including embedding functions, preprocessing steps, vector store, and LLM configurations—into a single, portable artifact.
## Features ✨
- **Complete RAG Pipeline**: Package your entire RAG system into a versioned artifact
- **LLM Integration**: Built-in support for OpenAI models
- **Flexible Embedding**: Compatible with any embedding function (Sentence Transformers, OpenAI, etc.)
- **Custom Preprocessing**: Chain multiple preprocessing steps
- **Vector Store Integration**: Built-in Milvus support
- **System Prompts**: Include and version your system prompts
## Installation 🛠️
```bash
pip install ragxo
```
## Quickstart 🚀
### Build a RAG pipeline
```bash
export OPENAI_API_KEY=<openai_key>
```
```python
from ragxo import Ragxo, Document
ragxo_client = Ragxo(dimension=1536)
def preprocess_text_remove_special_chars(text: str) -> str:
return re.sub(r'[^a-zA-Z0-9\s]', '', text)
def preprocess_text_lower(text: str) -> str:
return text.lower()
def get_embeddings(text: str) -> list[float]:
return openai.embeddings.create(input=text, model="text-embedding-ada-002").data[0].embedding
ragxo_client.add_preprocess(preprocess_text_lower)
ragxo_client.add_preprocess(preprocess_text_remove_special_chars)
ragxo_client.add_embedding_fn(get_embeddings)
ragxo_client.add_system_prompt("You are a helpful assistant that can answer questions about the data provided.")
ragxo_client.add_model(
"gpt-4o-mini",
limit=10,
temperature=0.5,
max_tokens=1000,
top_p=1.0,
frequency_penalty=0.0,
presence_penalty=0.0
)
ragxo_client.index([
Document(text="Capital of France is Paris", metadata={"source": "example"}, id=1),
Document(text="Capital of Germany is Berlin", metadata={"source": "example"}, id=2),
Document(text="Capital of Italy is Rome", metadata={"source": "example"}, id=3),
])
ragxo_client.export("my_rag_v1.0.0")
# or export to s3
ragxo_client.export("my_rag_v1.0.0", s3_bucket="my_bucket")
```
### Load a RAG pipeline
```python
loaded_ragxo_client = Ragxo.load("my_rag_v1.0.0")
vector_search_results = loaded_ragxo_client.query("What is the capital of France?")
llm_response = loaded_ragxo_client.generate_llm_response(
"What is the capital of France?")
print(llm_response.choices[0].message.content)
```
## Usage Guide 📚
### Import
```python
from ragxo import Ragxo, Document
ragxo_client = Ragxo(dimension=768)
```
### Adding Preprocessing Steps
```python
import re
def remove_special_chars(text: str) -> str:
return re.sub(r'[^a-zA-Z0-9\s]', '', text)
def lowercase(text: str) -> str:
return text.lower()
ragxo_client.add_preprocess(remove_special_chars)
ragxo_client.add_preprocess(lowercase)
```
### Custom Embedding Functions
```python
# Using SentenceTransformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
def get_embeddings(text: str) -> list[float]:
return model.encode(text).tolist()
ragxo.add_embedding_fn(get_embeddings)
# Or using OpenAI
from openai import OpenAI
client = OpenAI()
def get_openai_embeddings(text: str) -> list[float]:
response = client.embeddings.create(
input=text,
model="text-embedding-ada-002"
)
return response.data[0].embedding
ragxo.add_embedding_fn(get_openai_embeddings)
```
### Creating Documents
```python
from ragxo import Document
doc = Document(
text="Your document content here",
metadata={"source": "wiki", "category": "science"},
id=1
)
ragxo_client.index([doc])
```
### LLM Configuration
```python
# Set system prompt
ragxo_client.add_system_prompt("""
You are a helpful assistant. Use the provided context to answer questions accurately.
If you're unsure about something, please say so.
""")
# Set LLM model
ragxo_client.add_model("gpt-4")
```
### Export and Load
```python
# Export your RAG pipeline
ragxo_client.export("rag_pipeline_v1")
# Load it elsewhere
loaded_ragxo_client = Ragxo.load("rag_pipeline_v1")
```
### Evaluation
```python
from ragxo import EvaluationExample
# Create test examples
test_data = [
EvaluationExample(
query="What is the capital of France?",
expected="The capital of France is Paris."
),
EvaluationExample(
query="What is the capital of Germany?",
expected="The capital of Germany is Berlin."
),
]
# Evaluate the RAG system
accuracy = ragxo_client.evaluate(
test_data=test_data,
batch_size=10, # Process 10 examples at a time
judge_model="gpt-4" # Optional: specify a different model for evaluation
)
print(f"Evaluation accuracy: {accuracy * 100:.2f}%")
```
The evaluation process:
1. Processes test examples in batches
2. Generates RAG responses for each query
3. Uses an LLM to compare generated answers with expected answers
4. Returns accuracy score (0.0 to 1.0)
Best practices for evaluation:
- Use diverse test examples
- Include edge cases
- Keep expected answers consistent in format
- Use a more capable model for evaluation (e.g., GPT-4)
- Adjust batch size based on your rate limits and needs
## Best Practices 💡
1. **Version Your Exports**: Use semantic versioning for your exports:
```python
ragxo.export("my_rag_v1.0.0")
```
2. **S3**: Use S3 to store your exports
```shell
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
```
```python
ragxo_client.export("my_rag_v1.0.0", s3_bucket="my_bucket")
```
## License 📝
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## Contributing 🤝
Contributions are welcome! Please feel free to submit a Pull Request.
Raw data
{
"_id": null,
"home_page": "https://github.com/yourusername/ragx",
"name": "ragxo",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.11",
"maintainer_email": null,
"keywords": "rag, milvus, nlp, embeddings, openai",
"author": "Mohamed Sadek",
"author_email": "mohamedfawzydes@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/37/50/694f16fb8135f67a4eb8d3c7f547a86373977f10cc832eb4958ba65802c3/ragxo-0.1.12.tar.gz",
"platform": null,
"description": "# RagXO\n\nExport, version and reuse your E2E RAG pipeline everywhere \ud83d\ude80\n\n[](https://badge.fury.io/py/ragxo)\n[](https://opensource.org/licenses/MIT)\n[](https://www.python.org/downloads/release/python-380/)\n\n## Table of Contents\n- [Features](#features-)\n- [Installation](#installation-\ufe0f)\n- [Quickstart](#quickstart-)\n - [Build a RAG pipeline](#build-a-rag-pipeline)\n - [Load a RAG pipeline](#load-a-rag-pipeline)\n- [Usage Guide](#usage-guide-)\n - [Import](#import)\n - [Adding Preprocessing Steps](#adding-preprocessing-steps)\n - [Custom Embedding Functions](#custom-embedding-functions)\n - [Creating Documents](#creating-documents)\n - [LLM Configuration](#llm-configuration)\n - [Export and Load](#export-and-load)\n - [Evaluation](#evaluation)\n- [Best Practices](#best-practices-)\n- [License](#license-)\n- [Contributing](#contributing-)\n\nRagXO extends the capabilities of traditional RAG (Retrieval-Augmented Generation) systems by providing a unified way to package, version, and deploy your entire RAG pipeline with LLM integration. Export your complete system\u2014including embedding functions, preprocessing steps, vector store, and LLM configurations\u2014into a single, portable artifact.\n\n## Features \u2728\n\n- **Complete RAG Pipeline**: Package your entire RAG system into a versioned artifact\n- **LLM Integration**: Built-in support for OpenAI models\n- **Flexible Embedding**: Compatible with any embedding function (Sentence Transformers, OpenAI, etc.)\n- **Custom Preprocessing**: Chain multiple preprocessing steps\n- **Vector Store Integration**: Built-in Milvus support\n- **System Prompts**: Include and version your system prompts\n\n## Installation \ud83d\udee0\ufe0f\n\n```bash\npip install ragxo\n```\n\n## Quickstart \ud83d\ude80\n\n### Build a RAG pipeline\n\n```bash\nexport OPENAI_API_KEY=<openai_key> \n```\n\n```python\nfrom ragxo import Ragxo, Document\n\n\n\nragxo_client = Ragxo(dimension=1536)\n\ndef preprocess_text_remove_special_chars(text: str) -> str:\n return re.sub(r'[^a-zA-Z0-9\\s]', '', text)\n\ndef preprocess_text_lower(text: str) -> str:\n return text.lower()\n\ndef get_embeddings(text: str) -> list[float]:\n return openai.embeddings.create(input=text, model=\"text-embedding-ada-002\").data[0].embedding\n\nragxo_client.add_preprocess(preprocess_text_lower)\nragxo_client.add_preprocess(preprocess_text_remove_special_chars)\nragxo_client.add_embedding_fn(get_embeddings)\n\nragxo_client.add_system_prompt(\"You are a helpful assistant that can answer questions about the data provided.\")\nragxo_client.add_model(\n \"gpt-4o-mini\",\n limit=10,\n temperature=0.5,\n max_tokens=1000,\n top_p=1.0,\n frequency_penalty=0.0,\n presence_penalty=0.0\n)\n\nragxo_client.index([\n Document(text=\"Capital of France is Paris\", metadata={\"source\": \"example\"}, id=1),\n Document(text=\"Capital of Germany is Berlin\", metadata={\"source\": \"example\"}, id=2),\n Document(text=\"Capital of Italy is Rome\", metadata={\"source\": \"example\"}, id=3),\n])\n\nragxo_client.export(\"my_rag_v1.0.0\")\n\n# or export to s3\nragxo_client.export(\"my_rag_v1.0.0\", s3_bucket=\"my_bucket\")\n\n```\n\n\n### Load a RAG pipeline\n\n```python\nloaded_ragxo_client = Ragxo.load(\"my_rag_v1.0.0\")\n\nvector_search_results = loaded_ragxo_client.query(\"What is the capital of France?\")\n\nllm_response = loaded_ragxo_client.generate_llm_response(\n \"What is the capital of France?\")\n\nprint(llm_response.choices[0].message.content)\n```\n\n\n## Usage Guide \ud83d\udcda\n\n### Import\n\n```python\nfrom ragxo import Ragxo, Document\n\nragxo_client = Ragxo(dimension=768)\n\n```\n\n### Adding Preprocessing Steps\n\n```python\nimport re\n\ndef remove_special_chars(text: str) -> str:\n return re.sub(r'[^a-zA-Z0-9\\s]', '', text)\n\ndef lowercase(text: str) -> str:\n return text.lower()\n\nragxo_client.add_preprocess(remove_special_chars)\nragxo_client.add_preprocess(lowercase)\n```\n\n### Custom Embedding Functions\n\n```python\n# Using SentenceTransformers\nfrom sentence_transformers import SentenceTransformer\nmodel = SentenceTransformer('all-MiniLM-L6-v2')\n\ndef get_embeddings(text: str) -> list[float]:\n return model.encode(text).tolist()\n\nragxo.add_embedding_fn(get_embeddings)\n\n# Or using OpenAI\nfrom openai import OpenAI\nclient = OpenAI()\n\ndef get_openai_embeddings(text: str) -> list[float]:\n response = client.embeddings.create(\n input=text,\n model=\"text-embedding-ada-002\"\n )\n return response.data[0].embedding\n\nragxo.add_embedding_fn(get_openai_embeddings)\n```\n\n\n### Creating Documents\n\n```python\nfrom ragxo import Document\n\ndoc = Document(\n text=\"Your document content here\",\n metadata={\"source\": \"wiki\", \"category\": \"science\"},\n id=1\n)\n\nragxo_client.index([doc])\n\n```\n\n### LLM Configuration\n\n```python\n# Set system prompt\nragxo_client.add_system_prompt(\"\"\"\nYou are a helpful assistant. Use the provided context to answer questions accurately.\nIf you're unsure about something, please say so.\n\"\"\")\n\n# Set LLM model\nragxo_client.add_model(\"gpt-4\")\n```\n\n### Export and Load\n\n```python\n# Export your RAG pipeline\nragxo_client.export(\"rag_pipeline_v1\")\n\n# Load it elsewhere\nloaded_ragxo_client = Ragxo.load(\"rag_pipeline_v1\")\n```\n\n### Evaluation\n\n```python\nfrom ragxo import EvaluationExample\n\n# Create test examples\ntest_data = [\n EvaluationExample(\n query=\"What is the capital of France?\",\n expected=\"The capital of France is Paris.\"\n ),\n EvaluationExample(\n query=\"What is the capital of Germany?\",\n expected=\"The capital of Germany is Berlin.\"\n ),\n]\n\n# Evaluate the RAG system\naccuracy = ragxo_client.evaluate(\n test_data=test_data,\n batch_size=10, # Process 10 examples at a time\n judge_model=\"gpt-4\" # Optional: specify a different model for evaluation\n)\n\nprint(f\"Evaluation accuracy: {accuracy * 100:.2f}%\")\n```\n\nThe evaluation process:\n1. Processes test examples in batches\n2. Generates RAG responses for each query\n3. Uses an LLM to compare generated answers with expected answers\n4. Returns accuracy score (0.0 to 1.0)\n\nBest practices for evaluation:\n- Use diverse test examples\n- Include edge cases\n- Keep expected answers consistent in format\n- Use a more capable model for evaluation (e.g., GPT-4)\n- Adjust batch size based on your rate limits and needs\n\n## Best Practices \ud83d\udca1\n\n1. **Version Your Exports**: Use semantic versioning for your exports:\n```python\nragxo.export(\"my_rag_v1.0.0\")\n```\n\n2. **S3**: Use S3 to store your exports\n\n```shell\nexport AWS_ACCESS_KEY_ID=your_access_key\nexport AWS_SECRET_ACCESS_KEY=your_secret_key\n```\n\n```python\nragxo_client.export(\"my_rag_v1.0.0\", s3_bucket=\"my_bucket\")\n```\n\n## License \ud83d\udcdd\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Contributing \ud83e\udd1d\n\nContributions are welcome! Please feel free to submit a Pull Request.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A RAG (Retrieval-Augmented Generation) toolkit with Milvus integration",
"version": "0.1.12",
"project_urls": {
"Homepage": "https://github.com/yourusername/ragx",
"Repository": "https://github.com/yourusername/ragx"
},
"split_keywords": [
"rag",
" milvus",
" nlp",
" embeddings",
" openai"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "baca4e17977b732452326470183395acca03c2c5b22e43b7fd8d438d1d9c2bb9",
"md5": "164b8662557e13fa077eb28eb6c318cb",
"sha256": "65e388dd87b86e8d566ba9e4460e50e2e196d6fb54934dae86f62f3486911862"
},
"downloads": -1,
"filename": "ragxo-0.1.12-py3-none-any.whl",
"has_sig": false,
"md5_digest": "164b8662557e13fa077eb28eb6c318cb",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.11",
"size": 8802,
"upload_time": "2025-02-09T20:02:17",
"upload_time_iso_8601": "2025-02-09T20:02:17.841944Z",
"url": "https://files.pythonhosted.org/packages/ba/ca/4e17977b732452326470183395acca03c2c5b22e43b7fd8d438d1d9c2bb9/ragxo-0.1.12-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3750694f16fb8135f67a4eb8d3c7f547a86373977f10cc832eb4958ba65802c3",
"md5": "bd5a4228af7d4859ddf5da4993a8ea27",
"sha256": "f01971fcebd70c10ec3981ea58923f6890e08bbbd94e7a2dd3c03ab2e8ec7e5d"
},
"downloads": -1,
"filename": "ragxo-0.1.12.tar.gz",
"has_sig": false,
"md5_digest": "bd5a4228af7d4859ddf5da4993a8ea27",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.11",
"size": 10407,
"upload_time": "2025-02-09T20:02:19",
"upload_time_iso_8601": "2025-02-09T20:02:19.633674Z",
"url": "https://files.pythonhosted.org/packages/37/50/694f16fb8135f67a4eb8d3c7f547a86373977f10cc832eb4958ba65802c3/ragxo-0.1.12.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-09 20:02:19",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yourusername",
"github_project": "ragx",
"github_not_found": true,
"lcname": "ragxo"
}
JSON
