<p align="center">
<br>
<img src="docs/en/_static/images/evalscope_logo.png"/>
<br>
<p>
<p align="center">
<a href="README_zh.md">δΈζ</a>   ο½   English  
</p>
<p align="center">
<img src="https://img.shields.io/badge/python-%E2%89%A53.8-5be.svg">
<a href="https://badge.fury.io/py/evalscope"><img src="https://badge.fury.io/py/evalscope.svg" alt="PyPI version" height="18"></a>
<a href="https://pypi.org/project/evalscope"><img alt="PyPI - Downloads" src="https://static.pepy.tech/badge/evalscope"></a>
<a href="https://github.com/modelscope/evalscope/pulls"><img src="https://img.shields.io/badge/PR-welcome-55EB99.svg"></a>
<a href='https://evalscope.readthedocs.io/en/latest/?badge=latest'><img src='https://readthedocs.org/projects/evalscope/badge/?version=latest' alt='Documentation Status' /></a>
<p>
<p align="center">
<a href="https://evalscope.readthedocs.io/zh-cn/latest/"> π δΈζζζ‘£</a>   ο½   <a href="https://evalscope.readthedocs.io/en/latest/"> π English Documents</a>
<p>
> β If you like this project, please click the "Star" button at the top right to support us. Your support is our motivation to keep going!
## π Contents
- [Introduction](#introduction)
- [News](#News)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Evaluation Backend](#evaluation-backend)
- [Custom Dataset Evaluation](#custom-dataset-evaluation)
- [Model Serving Performance Evaluation](#Model-Serving-Performance-Evaluation)
- [Arena Mode](#arena-mode)
## π Introduction
EvalScope is [ModelScope](https://modelscope.cn/)'s official framework for model evaluation and benchmarking, designed for diverse assessment needs. It supports various model types including large language models, multimodal, embedding, reranker, and CLIP models.
The framework accommodates multiple evaluation scenarios such as end-to-end RAG evaluation, arena mode, and inference performance testing. It features built-in benchmarks and metrics like MMLU, CMMLU, C-Eval, and GSM8K. Seamlessly integrated with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, EvalScope enables one-click evaluations, offering comprehensive support for model training and assessment π
<p align="center">
<img src="docs/en/_static/images/evalscope_framework.png" width="70%">
<br>EvalScope Framework.
</p>
The architecture includes the following modules:
1. **Model Adapter**: The model adapter is used to convert the outputs of specific models into the format required by the framework, supporting both API call models and locally run models.
2. **Data Adapter**: The data adapter is responsible for converting and processing input data to meet various evaluation needs and formats.
3. **Evaluation Backend**:
- **Native**: EvalScopeβs own **default evaluation framework**, supporting various evaluation modes, including single model evaluation, arena mode, baseline model comparison mode, etc.
- **OpenCompass**: Supports [OpenCompass](https://github.com/open-compass/opencompass) as the evaluation backend, providing advanced encapsulation and task simplification, allowing you to submit tasks for evaluation more easily.
- **VLMEvalKit**: Supports [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) as the evaluation backend, enabling easy initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets.
- **RAGEval**: Supports RAG evaluation, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
- **ThirdParty**: Other third-party evaluation tasks, such as ToolBench.
4. **Performance Evaluator**: Model performance evaluation, responsible for measuring model inference service performance, including performance testing, stress testing, performance report generation, and visualization.
5. **Evaluation Report**: The final generated evaluation report summarizes the model's performance, which can be used for decision-making and further model optimization.
6. **Visualization**: Visualization results help users intuitively understand evaluation results, facilitating analysis and comparison of different model performances.
## π News
- π₯ **[2024.12.13]** Model evaluation optimization: no need to pass the `--template-type` parameter anymore; supports starting evaluation with `evalscope eval --args`. Refer to the [π User Guide](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html) for more details.
- π₯ **[2024.11.26]** The model inference service performance evaluator has been completely refactored: it now supports local inference service startup and Speed Benchmark; asynchronous call error handling has been optimized. For more details, refer to the [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/index.html).
- π₯ **[2024.10.31]** The best practice for evaluating Multimodal-RAG has been updated, please check the [π Blog](https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html#multimodal-rag) for more details.
- π₯ **[2024.10.23]** Supports multimodal RAG evaluation, including the assessment of image-text retrieval using [CLIP_Benchmark](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/clip_benchmark.html), and extends [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html) to support end-to-end multimodal metrics evaluation.
- π₯ **[2024.10.8]** Support for RAG evaluation, including independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
- π₯ **[2024.09.18]** Our documentation has been updated to include a blog module, featuring some technical research and discussions related to evaluations. We invite you to [π read it](https://evalscope.readthedocs.io/en/refact_readme/blog/index.html).
- π₯ **[2024.09.12]** Support for LongWriter evaluation, which supports 10,000+ word generation. You can use the benchmark [LongBench-Write](evalscope/third_party/longbench_write/README.md) to measure the long output quality as well as the output length.
- π₯ **[2024.08.30]** Support for custom dataset evaluations, including text datasets and multimodal image-text datasets.
- π₯ **[2024.08.20]** Updated the official documentation, including getting started guides, best practices, and FAQs. Feel free to [πread it here](https://evalscope.readthedocs.io/en/latest/)!
- π₯ **[2024.08.09]** Simplified the installation process, allowing for pypi installation of vlmeval dependencies; optimized the multimodal model evaluation experience, achieving up to 10x acceleration based on the OpenAI API evaluation chain.
- π₯ **[2024.07.31]** Important change: The package name `llmuses` has been changed to `evalscope`. Please update your code accordingly.
- π₯ **[2024.07.26]** Support for **VLMEvalKit** as a third-party evaluation framework to initiate multimodal model evaluation tasks.
- π₯ **[2024.06.29]** Support for **OpenCompass** as a third-party evaluation framework, which we have encapsulated at a higher level, supporting pip installation and simplifying evaluation task configuration.
- π₯ **[2024.06.13]** EvalScope seamlessly integrates with the fine-tuning framework SWIFT, providing full-chain support from LLM training to evaluation.
- π₯ **[2024.06.13]** Integrated the Agent evaluation dataset ToolBench.
## π οΈ Installation
### Method 1: Install Using pip
We recommend using conda to manage your environment and installing dependencies with pip:
1. Create a conda environment (optional)
```shell
# It is recommended to use Python 3.10
conda create -n evalscope python=3.10
# Activate the conda environment
conda activate evalscope
```
2. Install dependencies using pip
```shell
pip install evalscope # Install Native backend (default)
# Additional options
pip install evalscope[opencompass] # Install OpenCompass backend
pip install evalscope[vlmeval] # Install VLMEvalKit backend
pip install evalscope[rag] # Install RAGEval backend
pip install evalscope[perf] # Install Perf dependencies
pip install evalscope[all] # Install all backends (Native, OpenCompass, VLMEvalKit, RAGEval)
```
> [!WARNING]
> As the project has been renamed to `evalscope`, for versions `v0.4.3` or earlier, you can install using the following command:
> ```shell
> pip install llmuses<=0.4.3
> ```
> To import relevant dependencies using `llmuses`:
> ``` python
> from llmuses import ...
> ```
### Method 2: Install from Source
1. Download the source code
```shell
git clone https://github.com/modelscope/evalscope.git
```
2. Install dependencies
```shell
cd evalscope/
pip install -e . # Install Native backend
# Additional options
pip install -e '.[opencompass]' # Install OpenCompass backend
pip install -e '.[vlmeval]' # Install VLMEvalKit backend
pip install -e '.[rag]' # Install RAGEval backend
pip install -e '.[perf]' # Install Perf dependencies
pip install -e '.[all]' # Install all backends (Native, OpenCompass, VLMEvalKit, RAGEval)
```
## π Quick Start
To evaluate a model on specified datasets using default configurations, this framework supports two ways to initiate evaluation tasks: using the command line or using Python code.
### Method 1. Using Command Line
Execute the `eval` command in any directory:
```bash
evalscope eval \
--model Qwen/Qwen2.5-0.5B-Instruct \
--datasets gsm8k arc \
--limit 5
```
### Method 2. Using Python Code
When using Python code for evaluation, you need to submit the evaluation task using the `run_task` function, passing a `TaskConfig` as a parameter. It can also be a Python dictionary, yaml file path, or json file path, for example:
**Using Python Dictionary**
```python
from evalscope.run import run_task
task_cfg = {
'model': 'Qwen/Qwen2.5-0.5B-Instruct',
'datasets': ['gsm8k', 'arc'],
'limit': 5
}
run_task(task_cfg=task_cfg)
```
<details><summary>More Startup Methods</summary>
**Using `TaskConfig`**
```python
from evalscope.run import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='Qwen/Qwen2.5-0.5B-Instruct',
datasets=['gsm8k', 'arc'],
limit=5
)
run_task(task_cfg=task_cfg)
```
**Using `yaml` file**
`config.yaml`:
```yaml
model: Qwen/Qwen2.5-0.5B-Instruct
datasets:
- gsm8k
- arc
limit: 5
```
```python
from evalscope.run import run_task
run_task(task_cfg="config.yaml")
```
**Using `json` file**
`config.json`:
```json
{
"model": "Qwen/Qwen2.5-0.5B-Instruct",
"datasets": ["gsm8k", "arc"],
"limit": 5
}
```
```python
from evalscope.run import run_task
run_task(task_cfg="config.json")
```
</details>
### Basic Parameter
- `--model`: Specifies the `model_id` of the model in [ModelScope](https://modelscope.cn/), which can be automatically downloaded, e.g., [Qwen/Qwen2.5-0.5B-Instruct](https://modelscope.cn/models/Qwen/Qwen2.5-0.5B-Instruct/summary); or use the local path of the model, e.g., `/path/to/model`
- `--datasets`: Dataset names, supports inputting multiple datasets separated by spaces. Datasets will be automatically downloaded from modelscope. For supported datasets, refer to the [Dataset List](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset.html)
- `--limit`: Maximum amount of evaluation data for each dataset. If not specified, it defaults to evaluating all data. Can be used for quick validation
### Output Results
```
+-----------------------+-------------------+-----------------+
| Model | ai2_arc | gsm8k |
+=======================+===================+=================+
| Qwen2.5-0.5B-Instruct | (ai2_arc/acc) 0.6 | (gsm8k/acc) 0.6 |
+-----------------------+-------------------+-----------------+
```
## βοΈ Complex Evaluation
For more customized evaluations, such as customizing model parameters or dataset parameters, you can use the following command. The evaluation startup method is the same as simple evaluation. Below shows how to start the evaluation using the `eval` command:
```shell
evalscope eval \
--model Qwen/Qwen2.5-0.5B-Instruct \
--model-args revision=master,precision=torch.float16,device_map=auto \
--generation-config do_sample=true,temperature=0.5 \
--dataset-args '{"gsm8k": {"few_shot_num": 0, "few_shot_random": false}}' \
--datasets gsm8k \
--limit 10
```
### Parameter
- `--model-args`: Model loading parameters, separated by commas in `key=value` format. Default parameters:
- `revision`: Model version, default is `master`
- `precision`: Model precision, default is `auto`
- `device_map`: Model device allocation, default is `auto`
- `--generation-config`: Generation parameters, separated by commas in `key=value` format. Default parameters:
- `do_sample`: Whether to use sampling, default is `false`
- `max_length`: Maximum length, default is 2048
- `max_new_tokens`: Maximum length of generation, default is 512
- `--dataset-args`: Configuration parameters for evaluation datasets, passed in `json` format. The key is the dataset name, and the value is the parameters. Note that it needs to correspond one-to-one with the values in the `--datasets` parameter:
- `few_shot_num`: Number of few-shot examples
- `few_shot_random`: Whether to randomly sample few-shot data, if not set, defaults to `true`
Reference: [Full Parameter Description](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)
## Evaluation Backend
EvalScope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes:
- **Native**: EvalScope's own **default evaluation framework**, supporting various evaluation modes including single model evaluation, arena mode, and baseline model comparison mode.
- [OpenCompass](https://github.com/open-compass/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/opencompass_backend.html)
- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/vlmevalkit_backend.html)
- **RAGEval**: Initiate RAG evaluation tasks through EvalScope, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html): [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/index.html)
- **ThirdParty**: Third-party evaluation tasks, such as [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) and [LongBench-Write](https://evalscope.readthedocs.io/en/latest/third_party/longwriter.html).
## Model Serving Performance Evaluation
A stress testing tool focused on large language models, which can be customized to support various dataset formats and different API protocol formats.
Reference: Performance Testing [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/index.html)
**Supports wandb for recording results**
**Supports Speed Benchmark**
It supports speed testing and provides speed benchmarks similar to those found in the [official Qwen](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html) reports:
```text
Speed Benchmark Results:
+---------------+-----------------+----------------+
| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |
+---------------+-----------------+----------------+
| 1 | 50.69 | 0.97 |
| 6144 | 51.36 | 1.23 |
| 14336 | 49.93 | 1.59 |
| 30720 | 49.56 | 2.34 |
+---------------+-----------------+----------------+
```
## Custom Dataset Evaluation
EvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation [πUser Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset/index.html)
## Arena Mode
The Arena mode allows multiple candidate models to be evaluated through pairwise battles, and can choose to use the AI Enhanced Auto-Reviewer (AAR) automatic evaluation process or manual evaluation to obtain the evaluation report.
Refer to: Arena Mode [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/arena.html)
## TO-DO List
- [x] RAG evaluation
- [x] VLM evaluation
- [x] Agents evaluation
- [x] vLLM
- [ ] Distributed evaluating
- [x] Multi-modal evaluation
- [ ] Benchmarks
- [ ] GAIA
- [ ] GPQA
- [x] MBPP
- [ ] Auto-reviewer
- [ ] Qwen-max
## Star History
[](https://star-history.com/#modelscope/evalscope&Date)
Raw data
{
"_id": null,
"home_page": "https://github.com/modelscope/evalscope",
"name": "evalscope",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "python, llm, evaluation",
"author": "ModelScope team",
"author_email": "contact@modelscope.cn",
"download_url": "https://files.pythonhosted.org/packages/63/5a/bbc230bb06a7bc40dd3985dde4615a8a71f111bf95761e40f6f0f8a7e1a6/evalscope-0.8.1.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <br>\n <img src=\"docs/en/_static/images/evalscope_logo.png\"/>\n <br>\n<p>\n\n\n<p align=\"center\">\n <a href=\"README_zh.md\">\u4e2d\u6587</a>   \uff5c   English  \n</p>\n\n<p align=\"center\">\n<img src=\"https://img.shields.io/badge/python-%E2%89%A53.8-5be.svg\">\n<a href=\"https://badge.fury.io/py/evalscope\"><img src=\"https://badge.fury.io/py/evalscope.svg\" alt=\"PyPI version\" height=\"18\"></a>\n<a href=\"https://pypi.org/project/evalscope\"><img alt=\"PyPI - Downloads\" src=\"https://static.pepy.tech/badge/evalscope\"></a>\n<a href=\"https://github.com/modelscope/evalscope/pulls\"><img src=\"https://img.shields.io/badge/PR-welcome-55EB99.svg\"></a>\n<a href='https://evalscope.readthedocs.io/en/latest/?badge=latest'><img src='https://readthedocs.org/projects/evalscope/badge/?version=latest' alt='Documentation Status' /></a>\n<p>\n\n<p align=\"center\">\n<a href=\"https://evalscope.readthedocs.io/zh-cn/latest/\"> \ud83d\udcd6 \u4e2d\u6587\u6587\u6863</a>   \uff5c   <a href=\"https://evalscope.readthedocs.io/en/latest/\"> \ud83d\udcd6 English Documents</a>\n<p>\n\n> \u2b50 If you like this project, please click the \"Star\" button at the top right to support us. Your support is our motivation to keep going!\n\n## \ud83d\udccb Contents\n- [Introduction](#introduction)\n- [News](#News)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Evaluation Backend](#evaluation-backend)\n- [Custom Dataset Evaluation](#custom-dataset-evaluation)\n- [Model Serving Performance Evaluation](#Model-Serving-Performance-Evaluation)\n- [Arena Mode](#arena-mode)\n\n\n## \ud83d\udcdd Introduction\n\nEvalScope is [ModelScope](https://modelscope.cn/)'s official framework for model evaluation and benchmarking, designed for diverse assessment needs. It supports various model types including large language models, multimodal, embedding, reranker, and CLIP models.\n\nThe framework accommodates multiple evaluation scenarios such as end-to-end RAG evaluation, arena mode, and inference performance testing. It features built-in benchmarks and metrics like MMLU, CMMLU, C-Eval, and GSM8K. Seamlessly integrated with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, EvalScope enables one-click evaluations, offering comprehensive support for model training and assessment \ud83d\ude80\n\n<p align=\"center\">\n <img src=\"docs/en/_static/images/evalscope_framework.png\" width=\"70%\">\n <br>EvalScope Framework.\n</p>\n\nThe architecture includes the following modules:\n1. **Model Adapter**: The model adapter is used to convert the outputs of specific models into the format required by the framework, supporting both API call models and locally run models.\n2. **Data Adapter**: The data adapter is responsible for converting and processing input data to meet various evaluation needs and formats.\n3. **Evaluation Backend**:\n - **Native**: EvalScope\u2019s own **default evaluation framework**, supporting various evaluation modes, including single model evaluation, arena mode, baseline model comparison mode, etc.\n - **OpenCompass**: Supports [OpenCompass](https://github.com/open-compass/opencompass) as the evaluation backend, providing advanced encapsulation and task simplification, allowing you to submit tasks for evaluation more easily.\n - **VLMEvalKit**: Supports [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) as the evaluation backend, enabling easy initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets.\n - **RAGEval**: Supports RAG evaluation, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).\n - **ThirdParty**: Other third-party evaluation tasks, such as ToolBench.\n4. **Performance Evaluator**: Model performance evaluation, responsible for measuring model inference service performance, including performance testing, stress testing, performance report generation, and visualization.\n5. **Evaluation Report**: The final generated evaluation report summarizes the model's performance, which can be used for decision-making and further model optimization.\n6. **Visualization**: Visualization results help users intuitively understand evaluation results, facilitating analysis and comparison of different model performances.\n\n\n## \ud83c\udf89 News\n- \ud83d\udd25 **[2024.12.13]** Model evaluation optimization: no need to pass the `--template-type` parameter anymore; supports starting evaluation with `evalscope eval --args`. Refer to the [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html) for more details.\n- \ud83d\udd25 **[2024.11.26]** The model inference service performance evaluator has been completely refactored: it now supports local inference service startup and Speed Benchmark; asynchronous call error handling has been optimized. For more details, refer to the [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/index.html).\n- \ud83d\udd25 **[2024.10.31]** The best practice for evaluating Multimodal-RAG has been updated, please check the [\ud83d\udcd6 Blog](https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html#multimodal-rag) for more details.\n- \ud83d\udd25 **[2024.10.23]** Supports multimodal RAG evaluation, including the assessment of image-text retrieval using [CLIP_Benchmark](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/clip_benchmark.html), and extends [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html) to support end-to-end multimodal metrics evaluation.\n- \ud83d\udd25 **[2024.10.8]** Support for RAG evaluation, including independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).\n- \ud83d\udd25 **[2024.09.18]** Our documentation has been updated to include a blog module, featuring some technical research and discussions related to evaluations. We invite you to [\ud83d\udcd6 read it](https://evalscope.readthedocs.io/en/refact_readme/blog/index.html).\n- \ud83d\udd25 **[2024.09.12]** Support for LongWriter evaluation, which supports 10,000+ word generation. You can use the benchmark [LongBench-Write](evalscope/third_party/longbench_write/README.md) to measure the long output quality as well as the output length.\n- \ud83d\udd25 **[2024.08.30]** Support for custom dataset evaluations, including text datasets and multimodal image-text datasets.\n- \ud83d\udd25 **[2024.08.20]** Updated the official documentation, including getting started guides, best practices, and FAQs. Feel free to [\ud83d\udcd6read it here](https://evalscope.readthedocs.io/en/latest/)!\n- \ud83d\udd25 **[2024.08.09]** Simplified the installation process, allowing for pypi installation of vlmeval dependencies; optimized the multimodal model evaluation experience, achieving up to 10x acceleration based on the OpenAI API evaluation chain.\n- \ud83d\udd25 **[2024.07.31]** Important change: The package name `llmuses` has been changed to `evalscope`. Please update your code accordingly.\n- \ud83d\udd25 **[2024.07.26]** Support for **VLMEvalKit** as a third-party evaluation framework to initiate multimodal model evaluation tasks.\n- \ud83d\udd25 **[2024.06.29]** Support for **OpenCompass** as a third-party evaluation framework, which we have encapsulated at a higher level, supporting pip installation and simplifying evaluation task configuration.\n- \ud83d\udd25 **[2024.06.13]** EvalScope seamlessly integrates with the fine-tuning framework SWIFT, providing full-chain support from LLM training to evaluation.\n- \ud83d\udd25 **[2024.06.13]** Integrated the Agent evaluation dataset ToolBench.\n\n\n\n## \ud83d\udee0\ufe0f Installation\n### Method 1: Install Using pip\nWe recommend using conda to manage your environment and installing dependencies with pip:\n\n1. Create a conda environment (optional)\n ```shell\n # It is recommended to use Python 3.10\n conda create -n evalscope python=3.10\n # Activate the conda environment\n conda activate evalscope\n ```\n\n2. Install dependencies using pip\n ```shell\n pip install evalscope # Install Native backend (default)\n # Additional options\n pip install evalscope[opencompass] # Install OpenCompass backend\n pip install evalscope[vlmeval] # Install VLMEvalKit backend\n pip install evalscope[rag] # Install RAGEval backend\n pip install evalscope[perf] # Install Perf dependencies\n pip install evalscope[all] # Install all backends (Native, OpenCompass, VLMEvalKit, RAGEval)\n ```\n\n> [!WARNING]\n> As the project has been renamed to `evalscope`, for versions `v0.4.3` or earlier, you can install using the following command:\n> ```shell\n> pip install llmuses<=0.4.3\n> ```\n> To import relevant dependencies using `llmuses`:\n> ``` python\n> from llmuses import ...\n> ```\n\n### Method 2: Install from Source\n1. Download the source code\n ```shell\n git clone https://github.com/modelscope/evalscope.git\n ```\n\n2. Install dependencies\n ```shell\n cd evalscope/\n pip install -e . # Install Native backend\n # Additional options\n pip install -e '.[opencompass]' # Install OpenCompass backend\n pip install -e '.[vlmeval]' # Install VLMEvalKit backend\n pip install -e '.[rag]' # Install RAGEval backend\n pip install -e '.[perf]' # Install Perf dependencies\n pip install -e '.[all]' # Install all backends (Native, OpenCompass, VLMEvalKit, RAGEval)\n ```\n\n\n## \ud83d\ude80 Quick Start\n\nTo evaluate a model on specified datasets using default configurations, this framework supports two ways to initiate evaluation tasks: using the command line or using Python code.\n\n### Method 1. Using Command Line\n\nExecute the `eval` command in any directory:\n```bash\nevalscope eval \\\n --model Qwen/Qwen2.5-0.5B-Instruct \\\n --datasets gsm8k arc \\\n --limit 5\n```\n\n### Method 2. Using Python Code\n\nWhen using Python code for evaluation, you need to submit the evaluation task using the `run_task` function, passing a `TaskConfig` as a parameter. It can also be a Python dictionary, yaml file path, or json file path, for example:\n\n**Using Python Dictionary**\n\n```python\nfrom evalscope.run import run_task\n\ntask_cfg = {\n 'model': 'Qwen/Qwen2.5-0.5B-Instruct',\n 'datasets': ['gsm8k', 'arc'],\n 'limit': 5\n}\n\nrun_task(task_cfg=task_cfg)\n```\n\n<details><summary>More Startup Methods</summary>\n\n**Using `TaskConfig`**\n\n```python\nfrom evalscope.run import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n model='Qwen/Qwen2.5-0.5B-Instruct',\n datasets=['gsm8k', 'arc'],\n limit=5\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n**Using `yaml` file**\n\n`config.yaml`:\n```yaml\nmodel: Qwen/Qwen2.5-0.5B-Instruct\ndatasets:\n - gsm8k\n - arc\nlimit: 5\n```\n\n```python\nfrom evalscope.run import run_task\n\nrun_task(task_cfg=\"config.yaml\")\n```\n\n**Using `json` file**\n\n`config.json`:\n```json\n{\n \"model\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"datasets\": [\"gsm8k\", \"arc\"],\n \"limit\": 5\n}\n```\n\n```python\nfrom evalscope.run import run_task\n\nrun_task(task_cfg=\"config.json\")\n```\n</details>\n\n### Basic Parameter\n- `--model`: Specifies the `model_id` of the model in [ModelScope](https://modelscope.cn/), which can be automatically downloaded, e.g., [Qwen/Qwen2.5-0.5B-Instruct](https://modelscope.cn/models/Qwen/Qwen2.5-0.5B-Instruct/summary); or use the local path of the model, e.g., `/path/to/model`\n- `--datasets`: Dataset names, supports inputting multiple datasets separated by spaces. Datasets will be automatically downloaded from modelscope. For supported datasets, refer to the [Dataset List](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset.html)\n- `--limit`: Maximum amount of evaluation data for each dataset. If not specified, it defaults to evaluating all data. Can be used for quick validation\n\n### Output Results\n```\n+-----------------------+-------------------+-----------------+\n| Model | ai2_arc | gsm8k |\n+=======================+===================+=================+\n| Qwen2.5-0.5B-Instruct | (ai2_arc/acc) 0.6 | (gsm8k/acc) 0.6 |\n+-----------------------+-------------------+-----------------+\n```\n\n## \u2699\ufe0f Complex Evaluation\nFor more customized evaluations, such as customizing model parameters or dataset parameters, you can use the following command. The evaluation startup method is the same as simple evaluation. Below shows how to start the evaluation using the `eval` command:\n\n```shell\nevalscope eval \\\n --model Qwen/Qwen2.5-0.5B-Instruct \\\n --model-args revision=master,precision=torch.float16,device_map=auto \\\n --generation-config do_sample=true,temperature=0.5 \\\n --dataset-args '{\"gsm8k\": {\"few_shot_num\": 0, \"few_shot_random\": false}}' \\\n --datasets gsm8k \\\n --limit 10\n```\n\n### Parameter\n- `--model-args`: Model loading parameters, separated by commas in `key=value` format. Default parameters:\n - `revision`: Model version, default is `master`\n - `precision`: Model precision, default is `auto`\n - `device_map`: Model device allocation, default is `auto`\n- `--generation-config`: Generation parameters, separated by commas in `key=value` format. Default parameters:\n - `do_sample`: Whether to use sampling, default is `false`\n - `max_length`: Maximum length, default is 2048\n - `max_new_tokens`: Maximum length of generation, default is 512\n- `--dataset-args`: Configuration parameters for evaluation datasets, passed in `json` format. The key is the dataset name, and the value is the parameters. Note that it needs to correspond one-to-one with the values in the `--datasets` parameter:\n - `few_shot_num`: Number of few-shot examples\n - `few_shot_random`: Whether to randomly sample few-shot data, if not set, defaults to `true`\n\nReference: [Full Parameter Description](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)\n\n\n## Evaluation Backend\nEvalScope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes:\n- **Native**: EvalScope's own **default evaluation framework**, supporting various evaluation modes including single model evaluation, arena mode, and baseline model comparison mode.\n- [OpenCompass](https://github.com/open-compass/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/opencompass_backend.html)\n- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/vlmevalkit_backend.html)\n- **RAGEval**: Initiate RAG evaluation tasks through EvalScope, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html): [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/index.html)\n- **ThirdParty**: Third-party evaluation tasks, such as [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) and [LongBench-Write](https://evalscope.readthedocs.io/en/latest/third_party/longwriter.html).\n\n\n## Model Serving Performance Evaluation\nA stress testing tool focused on large language models, which can be customized to support various dataset formats and different API protocol formats.\n\nReference: Performance Testing [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/index.html)\n\n**Supports wandb for recording results**\n\n\n\n**Supports Speed Benchmark**\n\nIt supports speed testing and provides speed benchmarks similar to those found in the [official Qwen](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html) reports:\n\n```text\nSpeed Benchmark Results:\n+---------------+-----------------+----------------+\n| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |\n+---------------+-----------------+----------------+\n| 1 | 50.69 | 0.97 |\n| 6144 | 51.36 | 1.23 |\n| 14336 | 49.93 | 1.59 |\n| 30720 | 49.56 | 2.34 |\n+---------------+-----------------+----------------+\n```\n\n## Custom Dataset Evaluation\nEvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation [\ud83d\udcd6User Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset/index.html)\n\n\n## Arena Mode\nThe Arena mode allows multiple candidate models to be evaluated through pairwise battles, and can choose to use the AI Enhanced Auto-Reviewer (AAR) automatic evaluation process or manual evaluation to obtain the evaluation report.\n\nRefer to: Arena Mode [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/arena.html)\n\n\n\n\n## TO-DO List\n- [x] RAG evaluation\n- [x] VLM evaluation\n- [x] Agents evaluation\n- [x] vLLM\n- [ ] Distributed evaluating\n- [x] Multi-modal evaluation\n- [ ] Benchmarks\n - [ ] GAIA\n - [ ] GPQA\n - [x] MBPP\n- [ ] Auto-reviewer\n - [ ] Qwen-max\n\n\n## Star History\n\n[](https://star-history.com/#modelscope/evalscope&Date)\n",
"bugtrack_url": null,
"license": null,
"summary": "EvalScope: Lightweight LLMs Evaluation Framework",
"version": "0.8.1",
"project_urls": {
"Homepage": "https://github.com/modelscope/evalscope"
},
"split_keywords": [
"python",
" llm",
" evaluation"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "37a9a9c6cda95a6b837c9303e9a1598999f2c4e605abb507365c6ff70b372a5a",
"md5": "a72c43255aa5fe54dae4d4a87aacc608",
"sha256": "ade94a8f9d86e78758fc3f71efd20439fe48c67b9e5b5c2929892537836559da"
},
"downloads": -1,
"filename": "evalscope-0.8.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a72c43255aa5fe54dae4d4a87aacc608",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 1409749,
"upload_time": "2024-12-17T12:09:18",
"upload_time_iso_8601": "2024-12-17T12:09:18.837910Z",
"url": "https://files.pythonhosted.org/packages/37/a9/a9c6cda95a6b837c9303e9a1598999f2c4e605abb507365c6ff70b372a5a/evalscope-0.8.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "635abbc230bb06a7bc40dd3985dde4615a8a71f111bf95761e40f6f0f8a7e1a6",
"md5": "66cc97d22345c630cb3e19909654610f",
"sha256": "afab90cdfc9812a3951c3837808a44cc0944502d7f42c4b2f0927b01df3c392e"
},
"downloads": -1,
"filename": "evalscope-0.8.1.tar.gz",
"has_sig": false,
"md5_digest": "66cc97d22345c630cb3e19909654610f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 1286612,
"upload_time": "2024-12-17T12:09:21",
"upload_time_iso_8601": "2024-12-17T12:09:21.944031Z",
"url": "https://files.pythonhosted.org/packages/63/5a/bbc230bb06a7bc40dd3985dde4615a8a71f111bf95761e40f6f0f8a7e1a6/evalscope-0.8.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-17 12:09:21",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "modelscope",
"github_project": "evalscope",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "evalscope"
}
JSON
