| Name | LangRAGEval JSON |
| Version |
0.1.1
JSON |
| download |
| home_page | https://github.com/Vprashant/LangGPTEval |
| Summary | LangRAGEval is a library for evaluating responses based on faithfulness, context recall, answer relevancy, and context relevancy. |
| upload_time | 2024-05-31 08:38:45 |
| maintainer | None |
| docs_url | None |
| author | Prashant Verma |
| requires_python | None |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# LangGPTEval
[](https://www.python.org/downloads/release/python-370/)
[](https://pypi.org/project/fancybbox/)
[](https://opensource.org/licenses/MIT)
**LangGPTEval**, Evaluation library designed for Retrieval-Augmented Generation (RAG) responses.Evaluates the faithfulness, context recall, answer relevancy, and context relevancy of responses generated by various models, including OpenAI, Azure, and custom models. With a complex architecture and advanced Pydantic validation, LangGPTEval ensures reliable and accurate evaluation metrics.
## 🌟 Introduction
LangGPTEval is designed to evaluate the quality of responses generated by RAG models. It supports multiple metrics for evaluation:
- **Faithfulness**: How true the response is to the given context.
- **Context Recall**: How well the response recalls the given context.
- **Answer Relevancy**: How relevant the response is to the question.
- **Context Relevancy**: How relevant the response is to the context.
LangGPTEval is highly customizable, allowing users to plug in their models and tailor the evaluation to their specific needs.
## 🛠️ Installation
You can install LangGPTEval using pip:
```bash
pip install LangGPTEval
```
## ⚡ Quick Start
Here’s a quick start guide to get you up and running with LangGPTEval.
1. **Install the library**.
2. **Prepare your data**.
3. **Evaluate your model**.
## 📚 Usage
### Importing the Library
First, import the necessary components from the LangGPTEval library.
```python
from LangGPTEval.models import EvaluationInput, ContextData
from LangGPTEval.evaluation import evaluate_faithfulness, evaluate_context_recall, evaluate_answer_relevancy, evaluate_context_relevancy
from langchain.llms import OpenAI
```
### Setting Up Your Model
Create an instance of your model. Here, we demonstrate using LangChain’s OpenAI model.
```python
class LangChainOpenAIModel:
def __init__(self, api_key: str):
self.llm = OpenAI(api_key=api_key)
def invoke(self, prompt: Any) -> str:
response = self.llm(prompt)
score = response.strip()
return score
```
### Example Data
Prepare the input data for evaluation.
```python
context = [ContextData(page_content="Test context")]
response = "Test response"
input_data = EvaluationInput(context=context, response=response)
```
### Evaluating the Model
Use the evaluation functions to evaluate the model’s performance.
```python
# Replace 'your-openai-api-key' with your actual OpenAI API key
api_key = 'your-openai-api-key'
openai_model = LangChainOpenAIModel(api_key)
try:
# Evaluate with the LangChain OpenAI model
faithfulness_result = evaluate_faithfulness(input_data, openai_model)
context_recall_result = evaluate_context_recall(input_data, openai_model)
answer_relevancy_result = evaluate_answer_relevancy(input_data, openai_model)
context_relevancy_result = evaluate_context_relevancy(input_data, openai_model)
print(faithfulness_result.score)
print(context_recall_result.score)
print(answer_relevancy_result.score)
print(context_relevancy_result.score)
except ValueError as e:
print(f"An error occurred during evaluation: {str(e)}")
```
## 🔍 Examples
### Example with Custom Model
```python
class CustomModel:
def invoke(self, prompt):
# Custom model implementation
return "0.9" # Example score
# Create a custom model instance
custom_model = CustomModel()
try:
# Evaluate with the custom model
faithfulness_result = evaluate_faithfulness(input_data, custom_model)
context_recall_result = evaluate_context_recall(input_data, custom_model)
answer_relevancy_result = evaluate_answer_relevancy(input_data, custom_model)
context_relevancy_result = evaluate_context_relevancy(input_data, custom_model)
print(faithfulness_result.score)
print(context_recall_result.score)
print(answer_relevancy_result.score)
print(context_relevancy_result.score)
except ValueError as e:
print(f"An error occurred during evaluation: {str(e)}")
```
## 🤝 Contributing
Contributions are welcome! Please read the [contributing guidelines](CONTRIBUTING.md) before making a pull request.
### Steps to Contribute
1. Fork the repository.
2. Create a new branch (`git checkout -b feature-branch`).
3. Make your changes.
4. Commit your changes (`git commit -m 'Add new feature'`).
5. Push to the branch (`git push origin feature-branch`).
6. Open a pull request.
## 📜 License
LangGPTEval is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.
**Happy Evaluating!** 🎉
LangGPTEval is here to make your RAG model evaluations precise and easy. If you have any questions or need further assistance, feel free to reach out to me.
---
Raw data
{
"_id": null,
"home_page": "https://github.com/Vprashant/LangGPTEval",
"name": "LangRAGEval",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Prashant Verma",
"author_email": "prashant27050@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/f5/8f/a258a1de1702abc1871455cb8e3185419d1ae97e59035c1274ccc08eca0b/langrageval-0.1.1.tar.gz",
"platform": null,
"description": "# LangGPTEval\r\n[](https://www.python.org/downloads/release/python-370/)\r\n[](https://pypi.org/project/fancybbox/)\r\n[](https://opensource.org/licenses/MIT)\r\n\r\n**LangGPTEval**, Evaluation library designed for Retrieval-Augmented Generation (RAG) responses.Evaluates the faithfulness, context recall, answer relevancy, and context relevancy of responses generated by various models, including OpenAI, Azure, and custom models. With a complex architecture and advanced Pydantic validation, LangGPTEval ensures reliable and accurate evaluation metrics.\r\n\r\n## \ud83c\udf1f Introduction\r\n\r\nLangGPTEval is designed to evaluate the quality of responses generated by RAG models. It supports multiple metrics for evaluation:\r\n\r\n- **Faithfulness**: How true the response is to the given context.\r\n- **Context Recall**: How well the response recalls the given context.\r\n- **Answer Relevancy**: How relevant the response is to the question.\r\n- **Context Relevancy**: How relevant the response is to the context.\r\n\r\nLangGPTEval is highly customizable, allowing users to plug in their models and tailor the evaluation to their specific needs.\r\n\r\n## \ud83d\udee0\ufe0f Installation\r\n\r\nYou can install LangGPTEval using pip:\r\n\r\n```bash\r\npip install LangGPTEval\r\n```\r\n\r\n## \u26a1 Quick Start\r\n\r\nHere\u2019s a quick start guide to get you up and running with LangGPTEval.\r\n\r\n1. **Install the library**.\r\n2. **Prepare your data**.\r\n3. **Evaluate your model**.\r\n\r\n## \ud83d\udcda Usage\r\n\r\n### Importing the Library\r\n\r\nFirst, import the necessary components from the LangGPTEval library.\r\n\r\n```python\r\nfrom LangGPTEval.models import EvaluationInput, ContextData\r\nfrom LangGPTEval.evaluation import evaluate_faithfulness, evaluate_context_recall, evaluate_answer_relevancy, evaluate_context_relevancy\r\nfrom langchain.llms import OpenAI\r\n```\r\n\r\n### Setting Up Your Model\r\n\r\nCreate an instance of your model. Here, we demonstrate using LangChain\u2019s OpenAI model.\r\n\r\n```python\r\nclass LangChainOpenAIModel:\r\n def __init__(self, api_key: str):\r\n self.llm = OpenAI(api_key=api_key)\r\n\r\n def invoke(self, prompt: Any) -> str:\r\n response = self.llm(prompt)\r\n score = response.strip()\r\n return score\r\n```\r\n\r\n### Example Data\r\n\r\nPrepare the input data for evaluation.\r\n\r\n```python\r\ncontext = [ContextData(page_content=\"Test context\")]\r\nresponse = \"Test response\"\r\ninput_data = EvaluationInput(context=context, response=response)\r\n```\r\n\r\n### Evaluating the Model\r\n\r\nUse the evaluation functions to evaluate the model\u2019s performance.\r\n\r\n```python\r\n# Replace 'your-openai-api-key' with your actual OpenAI API key\r\napi_key = 'your-openai-api-key'\r\nopenai_model = LangChainOpenAIModel(api_key)\r\n\r\ntry:\r\n # Evaluate with the LangChain OpenAI model\r\n faithfulness_result = evaluate_faithfulness(input_data, openai_model)\r\n context_recall_result = evaluate_context_recall(input_data, openai_model)\r\n answer_relevancy_result = evaluate_answer_relevancy(input_data, openai_model)\r\n context_relevancy_result = evaluate_context_relevancy(input_data, openai_model)\r\n\r\n print(faithfulness_result.score)\r\n print(context_recall_result.score)\r\n print(answer_relevancy_result.score)\r\n print(context_relevancy_result.score)\r\nexcept ValueError as e:\r\n print(f\"An error occurred during evaluation: {str(e)}\")\r\n```\r\n\r\n## \ud83d\udd0d Examples\r\n\r\n### Example with Custom Model\r\n\r\n```python\r\nclass CustomModel:\r\n def invoke(self, prompt):\r\n # Custom model implementation\r\n return \"0.9\" # Example score\r\n\r\n# Create a custom model instance\r\ncustom_model = CustomModel()\r\n\r\ntry:\r\n # Evaluate with the custom model\r\n faithfulness_result = evaluate_faithfulness(input_data, custom_model)\r\n context_recall_result = evaluate_context_recall(input_data, custom_model)\r\n answer_relevancy_result = evaluate_answer_relevancy(input_data, custom_model)\r\n context_relevancy_result = evaluate_context_relevancy(input_data, custom_model)\r\n\r\n print(faithfulness_result.score)\r\n print(context_recall_result.score)\r\n print(answer_relevancy_result.score)\r\n print(context_relevancy_result.score)\r\nexcept ValueError as e:\r\n print(f\"An error occurred during evaluation: {str(e)}\")\r\n```\r\n\r\n## \ud83e\udd1d Contributing\r\n\r\nContributions are welcome! Please read the [contributing guidelines](CONTRIBUTING.md) before making a pull request.\r\n\r\n### Steps to Contribute\r\n\r\n1. Fork the repository.\r\n2. Create a new branch (`git checkout -b feature-branch`).\r\n3. Make your changes.\r\n4. Commit your changes (`git commit -m 'Add new feature'`).\r\n5. Push to the branch (`git push origin feature-branch`).\r\n6. Open a pull request.\r\n\r\n## \ud83d\udcdc License\r\n\r\nLangGPTEval is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.\r\n\r\n\r\n**Happy Evaluating!** \ud83c\udf89\r\n\r\nLangGPTEval is here to make your RAG model evaluations precise and easy. If you have any questions or need further assistance, feel free to reach out to me.\r\n\r\n---\r\n\r\n\r\n",
"bugtrack_url": null,
"license": null,
"summary": "LangRAGEval is a library for evaluating responses based on faithfulness, context recall, answer relevancy, and context relevancy.",
"version": "0.1.1",
"project_urls": {
"Homepage": "https://github.com/Vprashant/LangGPTEval"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fd0303c1bdbb34cd45ca97b6210fb2044cdae294c08ffe36f7592b26ee8acf65",
"md5": "466072abb59c1e5bb2bfc1037c6acf3c",
"sha256": "91f879feee995886d2f129f6326ff9bcf189a18ee1a7f48b0a0482c0b4dbfe73"
},
"downloads": -1,
"filename": "LangRAGEval-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "466072abb59c1e5bb2bfc1037c6acf3c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 6474,
"upload_time": "2024-05-31T08:38:43",
"upload_time_iso_8601": "2024-05-31T08:38:43.870044Z",
"url": "https://files.pythonhosted.org/packages/fd/03/03c1bdbb34cd45ca97b6210fb2044cdae294c08ffe36f7592b26ee8acf65/LangRAGEval-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f58fa258a1de1702abc1871455cb8e3185419d1ae97e59035c1274ccc08eca0b",
"md5": "8c3500066d7b461b7a3cefa5663abd51",
"sha256": "a358ec6dcf46097c1b309258922cbdef5eb92bba3419b387e27b5f74c9cdfc9c"
},
"downloads": -1,
"filename": "langrageval-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "8c3500066d7b461b7a3cefa5663abd51",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 5414,
"upload_time": "2024-05-31T08:38:45",
"upload_time_iso_8601": "2024-05-31T08:38:45.575810Z",
"url": "https://files.pythonhosted.org/packages/f5/8f/a258a1de1702abc1871455cb8e3185419d1ae97e59035c1274ccc08eca0b/langrageval-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-05-31 08:38:45",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Vprashant",
"github_project": "LangGPTEval",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "langrageval"
}
