# EvalLite 🚀
[](https://badge.fury.io/py/evallite)
[](https://opensource.org/licenses/MIT)
[](https://pypi.org/project/evallite/)
An efficient, zero-cost LLM evaluation framework combining the simplicity of [DeepEval](https://github.com/confident-ai/deepeval) with the power of free Hugging Face models through [AILite](https://github.com/yourusername/ailite).
[](LICENSE)
## 🌟 Key Features
- **Zero-Cost Evaluation**: Leverage free Hugging Face models for LLM evaluation
- **Simple Integration**: Drop-in replacement for DeepEval's evaluation capabilities
- **Extensive Model Support**: Access to leading open-source models including:
- Meta Llama 3.1 70B Instruct
- Qwen 2.5 72B Instruct
- Mistral Nemo Instruct
- Phi-3.5 Mini Instruct
- And more!
- **Comprehensive Metrics**: Full compatibility with DeepEval's evaluation metrics
- **Async Support**: Built-in asynchronous evaluation capabilities
## 📥 Installation
```bash
pip install evallite
```
## 🚀 Quick Start
Here's a simple example to get you started with EvalLite:
```python
from evallite import (
assert_test,
EvalLiteModel,
LLMTestCase,
evaluate,
AnswerRelevancyMetric
)
# Initialize metric with a specific model
answer_relevancy_metric = AnswerRelevancyMetric(
threshold=0.7,
model=EvalLiteModel(model="microsoft/Phi-3.5-mini-instruct")
)
# Create a test case
test_case = LLMTestCase(
input="What if these shoes don't fit?",
actual_output="We offer a 30-day full refund at no extra costs.",
retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
)
# Run evaluation
evaluate([test_case], [answer_relevancy_metric])
```
## 🔧 Available Models
EvalLite supports several powerful open-source models:
```python
from evallite import EvalLiteModel
# Available model options
models = [
'meta-llama/Meta-Llama-3.1-70B-Instruct',
'CohereForAI/c4ai-command-r-plus-08-2024',
'Qwen/Qwen2.5-72B-Instruct',
'nvidia/Llama-3.1-Nemotron-70B-Instruct-HF',
'meta-llama/Llama-3.2-11B-Vision-Instruct',
'NousResearch/Hermes-3-Llama-3.1-8B',
'mistralai/Mistral-Nemo-Instruct-2407',
'microsoft/Phi-3.5-mini-instruct'
]
# Initialize with specific model
evaluator = EvalLiteModel(model='microsoft/Phi-3.5-mini-instruct')
```
## 📊 Advanced Usage
### Custom Schema Support
EvalLite supports custom response schemas using Pydantic models:
```python
from pydantic import BaseModel
from typing import List
class Statements(BaseModel):
statements: List[str]
# Use with schema
result = evaluator.generate(
prompt="List three facts about climate change",
schema=Statements
)
```
### Async Evaluation
```python
async def evaluate_async():
response = await evaluator.a_generate(
prompt="What is the capital of France?",
schema=Statements
)
return response
```
### Batch Evaluation
```python
from evallite import EvaluationDataset
# Create multiple test cases
test_cases = [
LLMTestCase(
input="Question 1",
actual_output="Answer 1",
retrieval_context=["Context 1"]
),
LLMTestCase(
input="Question 2",
actual_output="Answer 2",
retrieval_context=["Context 2"]
)
]
# Create dataset
dataset = EvaluationDataset(test_cases=test_cases)
# Evaluate all at once
evaluate(dataset, [answer_relevancy_metric])
```
## 🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request
## 📄 License
This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.
## 🙏 Acknowledgments
- [DeepEval](https://github.com/confident-ai/deepeval) for the evaluation framework
- [AILite](https://github.com/yourusername/ailite) for providing free model access
- The open-source community for making powerful language models accessible
Raw data
{
"_id": null,
"home_page": "https://github.com/yourusername/pyopengenai",
"name": "evallite",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": null,
"keywords": null,
"author": "Kammari Santhosh",
"author_email": "Your Name <your.email@example.com>",
"download_url": "https://files.pythonhosted.org/packages/60/ae/d8550a42798c2874440a12c66296881e4a54d83d9af47691d57f965fe526/evallite-0.2.0.tar.gz",
"platform": null,
"description": "# EvalLite \ud83d\ude80\n\n[](https://badge.fury.io/py/evallite)\n[](https://opensource.org/licenses/MIT)\n[](https://pypi.org/project/evallite/)\n\nAn efficient, zero-cost LLM evaluation framework combining the simplicity of [DeepEval](https://github.com/confident-ai/deepeval) with the power of free Hugging Face models through [AILite](https://github.com/yourusername/ailite).\n\n[](LICENSE)\n\n## \ud83c\udf1f Key Features\n\n- **Zero-Cost Evaluation**: Leverage free Hugging Face models for LLM evaluation\n- **Simple Integration**: Drop-in replacement for DeepEval's evaluation capabilities\n- **Extensive Model Support**: Access to leading open-source models including:\n - Meta Llama 3.1 70B Instruct\n - Qwen 2.5 72B Instruct\n - Mistral Nemo Instruct\n - Phi-3.5 Mini Instruct\n - And more!\n- **Comprehensive Metrics**: Full compatibility with DeepEval's evaluation metrics\n- **Async Support**: Built-in asynchronous evaluation capabilities\n\n## \ud83d\udce5 Installation\n\n```bash\npip install evallite\n```\n\n## \ud83d\ude80 Quick Start\n\nHere's a simple example to get you started with EvalLite:\n\n```python\nfrom evallite import (\n assert_test,\n EvalLiteModel,\n LLMTestCase,\n evaluate,\n AnswerRelevancyMetric\n)\n\n# Initialize metric with a specific model\nanswer_relevancy_metric = AnswerRelevancyMetric(\n threshold=0.7,\n model=EvalLiteModel(model=\"microsoft/Phi-3.5-mini-instruct\")\n)\n\n# Create a test case\ntest_case = LLMTestCase(\n input=\"What if these shoes don't fit?\",\n actual_output=\"We offer a 30-day full refund at no extra costs.\",\n retrieval_context=[\"All customers are eligible for a 30 day full refund at no extra costs.\"]\n)\n\n# Run evaluation\nevaluate([test_case], [answer_relevancy_metric])\n```\n\n## \ud83d\udd27 Available Models\n\nEvalLite supports several powerful open-source models:\n\n```python\nfrom evallite import EvalLiteModel\n\n# Available model options\nmodels = [\n 'meta-llama/Meta-Llama-3.1-70B-Instruct',\n 'CohereForAI/c4ai-command-r-plus-08-2024',\n 'Qwen/Qwen2.5-72B-Instruct',\n 'nvidia/Llama-3.1-Nemotron-70B-Instruct-HF',\n 'meta-llama/Llama-3.2-11B-Vision-Instruct',\n 'NousResearch/Hermes-3-Llama-3.1-8B',\n 'mistralai/Mistral-Nemo-Instruct-2407',\n 'microsoft/Phi-3.5-mini-instruct'\n]\n\n# Initialize with specific model\nevaluator = EvalLiteModel(model='microsoft/Phi-3.5-mini-instruct')\n```\n\n## \ud83d\udcca Advanced Usage\n\n### Custom Schema Support\n\nEvalLite supports custom response schemas using Pydantic models:\n\n```python\nfrom pydantic import BaseModel\nfrom typing import List\n\nclass Statements(BaseModel):\n statements: List[str]\n\n# Use with schema\nresult = evaluator.generate(\n prompt=\"List three facts about climate change\",\n schema=Statements\n)\n```\n\n### Async Evaluation\n\n```python\nasync def evaluate_async():\n response = await evaluator.a_generate(\n prompt=\"What is the capital of France?\",\n schema=Statements\n )\n return response\n```\n\n### Batch Evaluation\n\n```python\nfrom evallite import EvaluationDataset\n\n# Create multiple test cases\ntest_cases = [\n LLMTestCase(\n input=\"Question 1\",\n actual_output=\"Answer 1\",\n retrieval_context=[\"Context 1\"]\n ),\n LLMTestCase(\n input=\"Question 2\",\n actual_output=\"Answer 2\",\n retrieval_context=[\"Context 2\"]\n )\n]\n\n# Create dataset\ndataset = EvaluationDataset(test_cases=test_cases)\n\n# Evaluate all at once\nevaluate(dataset, [answer_relevancy_metric])\n```\n\n## \ud83e\udd1d Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n## \ud83d\udcc4 License\n\nThis project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.\n\n## \ud83d\ude4f Acknowledgments\n\n- [DeepEval](https://github.com/confident-ai/deepeval) for the evaluation framework\n- [AILite](https://github.com/yourusername/ailite) for providing free model access\n- The open-source community for making powerful language models accessible\n",
"bugtrack_url": null,
"license": null,
"summary": "A powerful web content fetcher and processor",
"version": "0.2.0",
"project_urls": {
"Bug Tracker": "https://github.com/yourusername/pyopengenai/issues",
"Homepage": "https://github.com/yourusername/pyopengenai"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f9ec3a6cd2d7c5eda07042eb6c5b176f34eddc5cd6748aaf5dcc69ddb9cccb5c",
"md5": "57c112eb22cda7f42f3a5507624ac698",
"sha256": "802bbdefce814e7e71897644b2aaba50fff63f37ecd0a9e20dc8f774fd086c2d"
},
"downloads": -1,
"filename": "evallite-0.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "57c112eb22cda7f42f3a5507624ac698",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 4877,
"upload_time": "2024-11-09T07:13:19",
"upload_time_iso_8601": "2024-11-09T07:13:19.825350Z",
"url": "https://files.pythonhosted.org/packages/f9/ec/3a6cd2d7c5eda07042eb6c5b176f34eddc5cd6748aaf5dcc69ddb9cccb5c/evallite-0.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "60aed8550a42798c2874440a12c66296881e4a54d83d9af47691d57f965fe526",
"md5": "14252a88030e466f530a208f3f28445e",
"sha256": "fd50bc45b05c24c9d313568e2782efdddde5d8858616b99e05f5c2a9196f3556"
},
"downloads": -1,
"filename": "evallite-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "14252a88030e466f530a208f3f28445e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 4755,
"upload_time": "2024-11-09T07:13:21",
"upload_time_iso_8601": "2024-11-09T07:13:21.962001Z",
"url": "https://files.pythonhosted.org/packages/60/ae/d8550a42798c2874440a12c66296881e4a54d83d9af47691d57f965fe526/evallite-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-09 07:13:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yourusername",
"github_project": "pyopengenai",
"github_not_found": true,
"lcname": "evallite"
}
JSON
