<p align="center">
<br>
<img src="docs/en/_static/images/evalscope_logo.png"/>
<br>
<p>
<p align="center">
<a href="README_zh.md">δΈζ</a>   ο½   English  
</p>
<p align="center">
<img src="https://img.shields.io/badge/python-%E2%89%A53.9-5be.svg">
<a href="https://badge.fury.io/py/evalscope"><img src="https://badge.fury.io/py/evalscope.svg" alt="PyPI version" height="18"></a>
<a href="https://pypi.org/project/evalscope"><img alt="PyPI - Downloads" src="https://static.pepy.tech/badge/evalscope"></a>
<a href="https://github.com/modelscope/evalscope/pulls"><img src="https://img.shields.io/badge/PR-welcome-55EB99.svg"></a>
<a href='https://evalscope.readthedocs.io/en/latest/?badge=latest'><img src='https://readthedocs.org/projects/evalscope/badge/?version=latest' alt='Documentation Status' /></a>
<p>
<p align="center">
<a href="https://evalscope.readthedocs.io/zh-cn/latest/"> π δΈζζζ‘£</a>   ο½   <a href="https://evalscope.readthedocs.io/en/latest/"> π English Documents</a>
<p>
> β If you like this project, please click the "Star" button at the top right to support us. Your support is our motivation to keep going!
## π Contents
- [π Contents](#-contents)
- [π Introduction](#-introduction)
- [β User Groups](#-user-groups)
- [π News](#-news)
- [π οΈ Installation](#οΈ-installation)
- [Method 1: Install Using pip](#method-1-install-using-pip)
- [Method 2: Install from Source](#method-2-install-from-source)
- [π Quick Start](#-quick-start)
- [Method 1. Using Command Line](#method-1-using-command-line)
- [Method 2. Using Python Code](#method-2-using-python-code)
- [Basic Parameter](#basic-parameter)
- [Output Results](#output-results)
- [π Visualization of Evaluation Results](#-visualization-of-evaluation-results)
- [π Evaluation of Model API](#-evaluation-of-model-api)
- [βοΈ Custom Parameter Evaluation](#οΈ-custom-parameter-evaluation)
- [Parameter Description](#parameter-description)
- [π§ͺ Other Evaluation Backends](#-other-evaluation-backends)
- [π Model Serving Performance Evaluation](#-model-serving-performance-evaluation)
- [ποΈ Custom Dataset Evaluation](#οΈ-custom-dataset-evaluation)
- [βοΈ Arena Mode](#οΈ-arena-mode)
- [π·ββοΈ Contribution](#οΈ-contribution)
- [π Citation](#-citation)
- [π Roadmap](#-roadmap)
- [β Star History](#-star-history)
## π Introduction
EvalScope is a comprehensive model evaluation and performance benchmarking framework meticulously crafted by the [ModelScope Community](https://modelscope.cn/), offering a one-stop solution for your model assessment needs. Regardless of the type of model you are developing, EvalScope is equipped to cater to your requirements:
- π§ Large Language Models
- π¨ Multimodal Models
- π Embedding Models
- π Reranker Models
- πΌοΈ CLIP Models
- π AIGC Models (Image-to-Text/Video)
- ...and more!
EvalScope is not merely an evaluation tool; it is a valuable ally in your model optimization journey:
- π
Equipped with multiple industry-recognized benchmarks and evaluation metrics: MMLU, CMMLU, C-Eval, GSM8K, etc.
- π Model inference performance stress testing: Ensuring your model excels in real-world applications.
- π Seamless integration with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, enabling one-click evaluations and providing full-chain support from training to assessment for your model development.
Below is the overall architecture diagram of EvalScope:
<p align="center">
<img src="https://sail-moe.oss-cn-hangzhou.aliyuncs.com/yunlin/images/evalscope/doc/EvalScope%E6%9E%B6%E6%9E%84%E5%9B%BE.png" width="70%">
<br>EvalScope Framework.
</p>
<details><summary>Framework Description</summary>
The architecture includes the following modules:
1. Input Layer
- **Model Sources**: API models (OpenAI API), local models (ModelScope)
- **Datasets**: Standard evaluation benchmarks (MMLU/GSM8k, etc.), custom data (MCQ/QA)
2. Core Functions
- **Multi-backend Evaluation**
- Native backends: Unified evaluation for LLM/VLM/Embedding/T2I models
- Integrated frameworks: OpenCompass/MTEB/VLMEvalKit/RAGAS
- **Performance Monitoring**
- Model plugins: Supports various model service APIs
- Data plugins: Supports multiple data formats
- Metric tracking: TTFT/TPOP/Stability and other metrics
- **Tool Extensions**
- Integration: Tool-Bench/Needle-in-a-Haystack/BFCL-v3
3. Output Layer
- **Structured Reports**: Supports JSON/Tables/Logs
- **Visualization Platforms**: Supports Gradio/Wandb/SwanLab
</details>
## β User Groups
Please scan the QR code below to join our community groups:
[Discord Group](https://discord.com/invite/D27yfEFVz5) | WeChat Group | DingTalk Group
:-------------------------:|:-------------------------:|:-------------------------:
<img src="docs/asset/discord_qr.jpg" width="160" height="160"> | <img src="docs/asset/wechat.png" width="160" height="160"> | <img src="docs/asset/dingding.png" width="160" height="160">
## π News
- π₯ **[2025.07.18]** The model stress testing now supports randomly generating image-text data for multimodal model evaluation. For usage instructions, refer to the [documentation](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/examples.html#id4).
- π₯ **[2025.07.16]** Support for [Ο-bench](https://github.com/sierra-research/tau-bench) has been added, enabling the evaluation of AI Agent performance and reliability in real-world scenarios involving dynamic user and tool interactions. For usage instructions, please refer to the [documentation](https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset/llm.html#bench).
- π₯ **[2025.07.14]** Support for "Humanity's Last Exam" ([Humanity's-Last-Exam](https://modelscope.cn/datasets/cais/hle)), a highly challenging evaluation benchmark. For usage instructions, refer to the [documentation](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset/llm.html#humanity-s-last-exam).
- π₯ **[2025.07.03]** Refactored Arena Mode: now supports custom model battles, outputs a model leaderboard, and provides battle result visualization. See [reference](https://evalscope.readthedocs.io/en/latest/user_guides/arena.html) for details.
- π₯ **[2025.06.28]** Optimized custom dataset evaluation: now supports evaluation without reference answers. Enhanced LLM judge usage, with built-in modes for "scoring directly without reference answers" and "checking answer consistency with reference answers". See [reference](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset/llm.html#qa) for details.
- π₯ **[2025.06.19]** Added support for the [BFCL-v3](https://modelscope.cn/datasets/AI-ModelScope/bfcl_v3) benchmark, designed to evaluate model function-calling capabilities across various scenarios. For more information, refer to the [documentation](https://evalscope.readthedocs.io/zh-cn/latest/third_party/bfcl_v3.html).
- π₯ **[2025.06.02]** Added support for the Needle-in-a-Haystack test. Simply specify `needle_haystack` to conduct the test, and a corresponding heatmap will be generated in the `outputs/reports` folder, providing a visual representation of the model's performance. Refer to the [documentation](https://evalscope.readthedocs.io/en/latest/third_party/needle_haystack.html) for more details.
- π₯ **[2025.05.29]** Added support for two long document evaluation benchmarks: [DocMath](https://modelscope.cn/datasets/yale-nlp/DocMath-Eval/summary) and [FRAMES](https://modelscope.cn/datasets/iic/frames/summary). For usage guidelines, please refer to the [documentation](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset.html).
- π₯ **[2025.05.16]** Model service performance stress testing now supports setting various levels of concurrency and outputs a performance test report. [Reference example](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#id3).
- π₯ **[2025.05.13]** Added support for the [ToolBench-Static](https://modelscope.cn/datasets/AI-ModelScope/ToolBench-Static) dataset to evaluate model's tool-calling capabilities. Refer to the [documentation](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) for usage instructions. Also added support for the [DROP](https://modelscope.cn/datasets/AI-ModelScope/DROP/dataPeview) and [Winogrande](https://modelscope.cn/datasets/AI-ModelScope/winogrande_val) benchmarks to assess the reasoning capabilities of models.
- π₯ **[2025.04.29]** Added Qwen3 Evaluation Best Practices, [welcome to read π](https://evalscope.readthedocs.io/en/latest/best_practice/qwen3.html)
- π₯ **[2025.04.27]** Support for text-to-image evaluation: Supports 8 metrics including MPS, HPSv2.1Score, etc., and evaluation benchmarks such as EvalMuse, GenAI-Bench. Refer to the [user documentation](https://evalscope.readthedocs.io/en/latest/user_guides/aigc/t2i.html) for more details.
- π₯ **[2025.04.10]** Model service stress testing tool now supports the `/v1/completions` endpoint (the default endpoint for vLLM benchmarking)
- π₯ **[2025.04.08]** Support for evaluating embedding model services compatible with the OpenAI API has been added. For more details, check the [user guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html#configure-evaluation-parameters).
<details><summary>More</summary>
- π₯ **[2025.03.27]** Added support for [AlpacaEval](https://www.modelscope.cn/datasets/AI-ModelScope/alpaca_eval/dataPeview) and [ArenaHard](https://modelscope.cn/datasets/AI-ModelScope/arena-hard-auto-v0.1/summary) evaluation benchmarks. For usage notes, please refer to the [documentation](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset.html)
- π₯ **[2025.03.20]** The model inference service stress testing now supports generating prompts of specified length using random values. Refer to the [user guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/examples.html#using-the-random-dataset) for more details.
- π₯ **[2025.03.13]** Added support for the [LiveCodeBench](https://www.modelscope.cn/datasets/AI-ModelScope/code_generation_lite/summary) code evaluation benchmark, which can be used by specifying `live_code_bench`. Supports evaluating QwQ-32B on LiveCodeBench, refer to the [best practices](https://evalscope.readthedocs.io/en/latest/best_practice/eval_qwq.html).
- π₯ **[2025.03.11]** Added support for the [SimpleQA](https://modelscope.cn/datasets/AI-ModelScope/SimpleQA/summary) and [Chinese SimpleQA](https://modelscope.cn/datasets/AI-ModelScope/Chinese-SimpleQA/summary) evaluation benchmarks. These are used to assess the factual accuracy of models, and you can specify `simple_qa` and `chinese_simpleqa` for use. Support for specifying a judge model is also available. For more details, refer to the [relevant parameter documentation](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html).
- π₯ **[2025.03.07]** Added support for the [QwQ-32B](https://modelscope.cn/models/Qwen/QwQ-32B/summary) model, evaluate the model's reasoning ability and reasoning efficiency, refer to [π Best Practices for QwQ-32B Evaluation](https://evalscope.readthedocs.io/en/latest/best_practice/eval_qwq.html) for more details.
- π₯ **[2025.03.04]** Added support for the [SuperGPQA](https://modelscope.cn/datasets/m-a-p/SuperGPQA/summary) dataset, which covers 13 categories, 72 first-level disciplines, and 285 second-level disciplines, totaling 26,529 questions. You can use it by specifying `super_gpqa`.
- π₯ **[2025.03.03]** Added support for evaluating the IQ and EQ of models. Refer to [π Best Practices for IQ and EQ Evaluation](https://evalscope.readthedocs.io/en/latest/best_practice/iquiz.html) to find out how smart your AI is!
- π₯ **[2025.02.27]** Added support for evaluating the reasoning efficiency of models. Refer to [π Best Practices for Evaluating Thinking Efficiency](https://evalscope.readthedocs.io/en/latest/best_practice/think_eval.html). This implementation is inspired by the works [Overthinking](https://doi.org/10.48550/arXiv.2412.21187) and [Underthinking](https://doi.org/10.48550/arXiv.2501.18585).
- π₯ **[2025.02.25]** Added support for two model inference-related evaluation benchmarks: [MuSR](https://modelscope.cn/datasets/AI-ModelScope/MuSR) and [ProcessBench](https://www.modelscope.cn/datasets/Qwen/ProcessBench/summary). To use them, simply specify `musr` and `process_bench` respectively in the datasets parameter.
- π₯ **[2025.02.18]** Supports the AIME25 dataset, which contains 15 questions (Grok3 scored 93 on this dataset).
- π₯ **[2025.02.13]** Added support for evaluating DeepSeek distilled models, including AIME24, MATH-500, and GPQA-Diamond datasetsοΌrefer to [best practice](https://evalscope.readthedocs.io/en/latest/best_practice/deepseek_r1_distill.html); Added support for specifying the `eval_batch_size` parameter to accelerate model evaluation.
- π₯ **[2025.01.20]** Support for visualizing evaluation results, including single model evaluation results and multi-model comparison, refer to the [π Visualizing Evaluation Results](https://evalscope.readthedocs.io/en/latest/get_started/visualization.html) for more details; Added [`iquiz`](https://modelscope.cn/datasets/AI-ModelScope/IQuiz/summary) evaluation example, evaluating the IQ and EQ of the model.
- π₯ **[2025.01.07]** Native backend: Support for model API evaluation is now available. Refer to the [π Model API Evaluation Guide](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#api) for more details. Additionally, support for the `ifeval` evaluation benchmark has been added.
- π₯π₯ **[2024.12.31]** Support for adding benchmark evaluations, refer to the [π Benchmark Evaluation Addition Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/add_benchmark.html); support for custom mixed dataset evaluations, allowing for more comprehensive model evaluations with less data, refer to the [π Mixed Dataset Evaluation Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/collection/index.html).
- π₯ **[2024.12.13]** Model evaluation optimization: no need to pass the `--template-type` parameter anymore; supports starting evaluation with `evalscope eval --args`. Refer to the [π User Guide](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html) for more details.
- π₯ **[2024.11.26]** The model inference service performance evaluator has been completely refactored: it now supports local inference service startup and Speed Benchmark; asynchronous call error handling has been optimized. For more details, refer to the [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/index.html).
- π₯ **[2024.10.31]** The best practice for evaluating Multimodal-RAG has been updated, please check the [π Blog](https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html#multimodal-rag) for more details.
- π₯ **[2024.10.23]** Supports multimodal RAG evaluation, including the assessment of image-text retrieval using [CLIP_Benchmark](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/clip_benchmark.html), and extends [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html) to support end-to-end multimodal metrics evaluation.
- π₯ **[2024.10.8]** Support for RAG evaluation, including independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).
- π₯ **[2024.09.18]** Our documentation has been updated to include a blog module, featuring some technical research and discussions related to evaluations. We invite you to [π read it](https://evalscope.readthedocs.io/en/refact_readme/blog/index.html).
- π₯ **[2024.09.12]** Support for LongWriter evaluation, which supports 10,000+ word generation. You can use the benchmark [LongBench-Write](evalscope/third_party/longbench_write/README.md) to measure the long output quality as well as the output length.
- π₯ **[2024.08.30]** Support for custom dataset evaluations, including text datasets and multimodal image-text datasets.
- π₯ **[2024.08.20]** Updated the official documentation, including getting started guides, best practices, and FAQs. Feel free to [πread it here](https://evalscope.readthedocs.io/en/latest/)!
- π₯ **[2024.08.09]** Simplified the installation process, allowing for pypi installation of vlmeval dependencies; optimized the multimodal model evaluation experience, achieving up to 10x acceleration based on the OpenAI API evaluation chain.
- π₯ **[2024.07.31]** Important change: The package name `llmuses` has been changed to `evalscope`. Please update your code accordingly.
- π₯ **[2024.07.26]** Support for **VLMEvalKit** as a third-party evaluation framework to initiate multimodal model evaluation tasks.
- π₯ **[2024.06.29]** Support for **OpenCompass** as a third-party evaluation framework, which we have encapsulated at a higher level, supporting pip installation and simplifying evaluation task configuration.
- π₯ **[2024.06.13]** EvalScope seamlessly integrates with the fine-tuning framework SWIFT, providing full-chain support from LLM training to evaluation.
- π₯ **[2024.06.13]** Integrated the Agent evaluation dataset ToolBench.
</details>
## π οΈ Installation
### Method 1: Install Using pip
We recommend using conda to manage your environment and installing dependencies with pip:
1. Create a conda environment (optional)
```shell
# It is recommended to use Python 3.10
conda create -n evalscope python=3.10
# Activate the conda environment
conda activate evalscope
```
2. Install dependencies using pip
```shell
pip install evalscope # Install Native backend (default)
# Additional options
pip install 'evalscope[opencompass]' # Install OpenCompass backend
pip install 'evalscope[vlmeval]' # Install VLMEvalKit backend
pip install 'evalscope[rag]' # Install RAGEval backend
pip install 'evalscope[perf]' # Install dependencies for the model performance testing module
pip install 'evalscope[app]' # Install dependencies for visualization
pip install 'evalscope[all]' # Install all backends (Native, OpenCompass, VLMEvalKit, RAGEval)
```
> [!WARNING]
> As the project has been renamed to `evalscope`, for versions `v0.4.3` or earlier, you can install using the following command:
> ```shell
> pip install llmuses<=0.4.3
> ```
> To import relevant dependencies using `llmuses`:
> ``` python
> from llmuses import ...
> ```
### Method 2: Install from Source
1. Download the source code
```shell
git clone https://github.com/modelscope/evalscope.git
```
2. Install dependencies
```shell
cd evalscope/
pip install -e . # Install Native backend
# Additional options
pip install -e '.[opencompass]' # Install OpenCompass backend
pip install -e '.[vlmeval]' # Install VLMEvalKit backend
pip install -e '.[rag]' # Install RAGEval backend
pip install -e '.[perf]' # Install Perf dependencies
pip install -e '.[app]' # Install visualization dependencies
pip install -e '.[all]' # Install all backends (Native, OpenCompass, VLMEvalKit, RAGEval)
```
## π Quick Start
To evaluate a model on specified datasets using default configurations, this framework supports two ways to initiate evaluation tasks: using the command line or using Python code.
### Method 1. Using Command Line
Execute the `eval` command in any directory:
```bash
evalscope eval \
--model Qwen/Qwen2.5-0.5B-Instruct \
--datasets gsm8k arc \
--limit 5
```
### Method 2. Using Python Code
When using Python code for evaluation, you need to submit the evaluation task using the `run_task` function, passing a `TaskConfig` as a parameter. It can also be a Python dictionary, yaml file path, or json file path, for example:
**Using `TaskConfig`**
```python
from evalscope import run_task, TaskConfig
task_cfg = TaskConfig(
model='Qwen/Qwen2.5-0.5B-Instruct',
datasets=['gsm8k', 'arc'],
limit=5
)
run_task(task_cfg=task_cfg)
```
<details><summary>More Startup Methods</summary>
**Using Python Dictionary**
```python
from evalscope.run import run_task
task_cfg = {
'model': 'Qwen/Qwen2.5-0.5B-Instruct',
'datasets': ['gsm8k', 'arc'],
'limit': 5
}
run_task(task_cfg=task_cfg)
```
**Using `yaml` file**
`config.yaml`:
```yaml
model: Qwen/Qwen2.5-0.5B-Instruct
datasets:
- gsm8k
- arc
limit: 5
```
```python
from evalscope.run import run_task
run_task(task_cfg="config.yaml")
```
**Using `json` file**
`config.json`:
```json
{
"model": "Qwen/Qwen2.5-0.5B-Instruct",
"datasets": ["gsm8k", "arc"],
"limit": 5
}
```
```python
from evalscope.run import run_task
run_task(task_cfg="config.json")
```
</details>
### Basic Parameter
- `--model`: Specifies the `model_id` of the model in [ModelScope](https://modelscope.cn/), which can be automatically downloaded, e.g., [Qwen/Qwen2.5-0.5B-Instruct](https://modelscope.cn/models/Qwen/Qwen2.5-0.5B-Instruct/summary); or use the local path of the model, e.g., `/path/to/model`
- `--datasets`: Dataset names, supports inputting multiple datasets separated by spaces. Datasets will be automatically downloaded from modelscope. For supported datasets, refer to the [Dataset List](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset.html)
- `--limit`: Maximum amount of evaluation data for each dataset. If not specified, it defaults to evaluating all data. Can be used for quick validation
### Output Results
```text
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
| Model Name | Dataset Name | Metric Name | Category Name | Subset Name | Num | Score |
+=======================+================+=================+=================+===============+=======+=========+
| Qwen2.5-0.5B-Instruct | gsm8k | AverageAccuracy | default | main | 5 | 0.4 |
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
| Qwen2.5-0.5B-Instruct | ai2_arc | AverageAccuracy | default | ARC-Easy | 5 | 0.8 |
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
| Qwen2.5-0.5B-Instruct | ai2_arc | AverageAccuracy | default | ARC-Challenge | 5 | 0.4 |
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
```
## π Visualization of Evaluation Results
1. Install the dependencies required for visualization, including gradio, plotly, etc.
```bash
pip install 'evalscope[app]'
```
2. Start the Visualization Service
Run the following command to start the visualization service.
```bash
evalscope app
```
You can access the visualization service in the browser if the following output appears.
```text
* Running on local URL: http://127.0.0.1:7861
To create a public link, set `share=True` in `launch()`.
```
<table>
<tr>
<td style="text-align: center;">
<img src="docs/en/get_started/images/setting.png" alt="Setting" style="width: 75%;" />
<p>Setting Interface</p>
</td>
<td style="text-align: center;">
<img src="docs/en/get_started/images/model_compare.png" alt="Model Compare" style="width: 100%;" />
<p>Model Comparison</p>
</td>
</tr>
<tr>
<td style="text-align: center;">
<img src="docs/en/get_started/images/report_overview.png" alt="Report Overview" style="width: 100%;" />
<p>Report Overview</p>
</td>
<td style="text-align: center;">
<img src="docs/en/get_started/images/report_details.png" alt="Report Details" style="width: 80%;" />
<p>Report Details</p>
</td>
</tr>
</table>
For more details, refer to: [π Visualization of Evaluation Results](https://evalscope.readthedocs.io/en/latest/get_started/visualization.html)
## π Evaluation of Model API
Specify the model API service address (api_url) and API Key (api_key) to evaluate the deployed model API service. In this case, the `eval-type` parameter must be specified as `service`, for example:
For example, to launch a model service using [vLLM](https://github.com/vllm-project/vllm):
```shell
export VLLM_USE_MODELSCOPE=True && python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-0.5B-Instruct --served-model-name qwen2.5 --trust_remote_code --port 8801
```
Then, you can use the following command to evaluate the model API service:
```shell
evalscope eval \
--model qwen2.5 \
--api-url http://127.0.0.1:8801/v1 \
--api-key EMPTY \
--eval-type service \
--datasets gsm8k \
--limit 10
```
## βοΈ Custom Parameter Evaluation
For more customized evaluations, such as customizing model parameters or dataset parameters, you can use the following command. The evaluation startup method is the same as simple evaluation. Below shows how to start the evaluation using the `eval` command:
```shell
evalscope eval \
--model Qwen/Qwen3-0.6B \
--model-args '{"revision": "master", "precision": "torch.float16", "device_map": "auto"}' \
--generation-config '{"do_sample":true,"temperature":0.6,"max_new_tokens":512,"chat_template_kwargs":{"enable_thinking": false}}' \
--dataset-args '{"gsm8k": {"few_shot_num": 0, "few_shot_random": false}}' \
--datasets gsm8k \
--limit 10
```
### Parameter Description
- `--model-args`: Model loading parameters, passed as a JSON string:
- `revision`: Model version
- `precision`: Model precision
- `device_map`: Device allocation for the model
- `--generation-config`: Generation parameters, passed as a JSON string and parsed as a dictionary:
- `do_sample`: Whether to use sampling
- `temperature`: Generation temperature
- `max_new_tokens`: Maximum length of generated tokens
- `chat_template_kwargs`: Model inference template parameters
- `--dataset-args`: Settings for the evaluation dataset, passed as a JSON string where the key is the dataset name and the value is the parameters. Note that these need to correspond one-to-one with the values in the `--datasets` parameter:
- `few_shot_num`: Number of few-shot examples
- `few_shot_random`: Whether to randomly sample few-shot data; if not set, defaults to `true`
Reference: [Full Parameter Description](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)
## π§ͺ Other Evaluation Backends
EvalScope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes:
- **Native**: EvalScope's own **default evaluation framework**, supporting various evaluation modes including single model evaluation, arena mode, and baseline model comparison mode.
- [OpenCompass](https://github.com/open-compass/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/opencompass_backend.html)
- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/vlmevalkit_backend.html)
- **RAGEval**: Initiate RAG evaluation tasks through EvalScope, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html): [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/index.html)
- **ThirdParty**: Third-party evaluation tasks, such as [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) and [LongBench-Write](https://evalscope.readthedocs.io/en/latest/third_party/longwriter.html).
## π Model Serving Performance Evaluation
A stress testing tool focused on large language models, which can be customized to support various dataset formats and different API protocol formats.
Reference: Performance Testing [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/index.html)
**Output example**
**Supports wandb for recording results**
**Supports swanlab for recording results**
**Supports Speed Benchmark**
It supports speed testing and provides speed benchmarks similar to those found in the [official Qwen](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html) reports:
```text
Speed Benchmark Results:
+---------------+-----------------+----------------+
| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |
+---------------+-----------------+----------------+
| 1 | 50.69 | 0.97 |
| 6144 | 51.36 | 1.23 |
| 14336 | 49.93 | 1.59 |
| 30720 | 49.56 | 2.34 |
+---------------+-----------------+----------------+
```
## ποΈ Custom Dataset Evaluation
EvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation [πUser Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset/index.html)
## βοΈ Arena Mode
Arena mode allows you to configure multiple candidate models and specify a baseline model. Evaluation is performed by pairwise battles between each candidate model and the baseline model, with the final output including each model's win rate and ranking. This method is suitable for comparative evaluation among multiple models, providing an intuitive reflection of each model's strengths and weaknesses. Refer to: Arena Mode [π User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/arena.html)
```text
Model WinRate (%) CI (%)
------------ ------------- ---------------
qwen2.5-72b 69.3 (-13.3 / +12.2)
qwen2.5-7b 50 (+0.0 / +0.0)
qwen2.5-0.5b 4.7 (-2.5 / +4.4)
```
## π·ββοΈ Contribution
EvalScope, as the official evaluation tool of [ModelScope](https://modelscope.cn), is continuously optimizing its benchmark evaluation features! We invite you to refer to the [Contribution Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/add_benchmark.html) to easily add your own evaluation benchmarks and share your contributions with the community. Letβs work together to support the growth of EvalScope and make our tools even better! Join us now!
<a href="https://github.com/modelscope/evalscope/graphs/contributors" target="_blank">
<table>
<tr>
<th colspan="2">
<br><img src="https://contrib.rocks/image?repo=modelscope/evalscope"><br><br>
</th>
</tr>
</table>
</a>
## π Citation
```bibtex
@misc{evalscope_2024,
title={{EvalScope}: Evaluation Framework for Large Models},
author={ModelScope Team},
year={2024},
url={https://github.com/modelscope/evalscope}
}
```
## π Roadmap
- [x] Support for better evaluation report visualization
- [x] Support for mixed evaluations across multiple datasets
- [x] RAG evaluation
- [x] VLM evaluation
- [x] Agents evaluation
- [x] vLLM
- [ ] Distributed evaluating
- [x] Multi-modal evaluation
- [ ] Benchmarks
- [x] BFCL-v3
- [x] GPQA
- [x] MBPP
## β Star History
[](https://star-history.com/#modelscope/evalscope&Date)
Raw data
{
"_id": null,
"home_page": "https://github.com/modelscope/evalscope",
"name": "evalscope",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "python, llm, evaluation",
"author": "ModelScope team",
"author_email": "contact@modelscope.cn",
"download_url": "https://files.pythonhosted.org/packages/38/88/326e48929bb9577a6a36e07afa65bbf6bd870c1b644f82e5713874ae3238/evalscope-0.17.1.tar.gz",
"platform": null,
"description": "<p align=\"center\">\n <br>\n <img src=\"docs/en/_static/images/evalscope_logo.png\"/>\n <br>\n<p>\n\n\n<p align=\"center\">\n <a href=\"README_zh.md\">\u4e2d\u6587</a>   \uff5c   English  \n</p>\n\n<p align=\"center\">\n<img src=\"https://img.shields.io/badge/python-%E2%89%A53.9-5be.svg\">\n<a href=\"https://badge.fury.io/py/evalscope\"><img src=\"https://badge.fury.io/py/evalscope.svg\" alt=\"PyPI version\" height=\"18\"></a>\n<a href=\"https://pypi.org/project/evalscope\"><img alt=\"PyPI - Downloads\" src=\"https://static.pepy.tech/badge/evalscope\"></a>\n<a href=\"https://github.com/modelscope/evalscope/pulls\"><img src=\"https://img.shields.io/badge/PR-welcome-55EB99.svg\"></a>\n<a href='https://evalscope.readthedocs.io/en/latest/?badge=latest'><img src='https://readthedocs.org/projects/evalscope/badge/?version=latest' alt='Documentation Status' /></a>\n<p>\n\n<p align=\"center\">\n<a href=\"https://evalscope.readthedocs.io/zh-cn/latest/\"> \ud83d\udcd6 \u4e2d\u6587\u6587\u6863</a>   \uff5c   <a href=\"https://evalscope.readthedocs.io/en/latest/\"> \ud83d\udcd6 English Documents</a>\n<p>\n\n> \u2b50 If you like this project, please click the \"Star\" button at the top right to support us. Your support is our motivation to keep going!\n\n## \ud83d\udccb Contents\n- [\ud83d\udccb Contents](#-contents)\n- [\ud83d\udcdd Introduction](#-introduction)\n- [\u260e User Groups](#-user-groups)\n- [\ud83c\udf89 News](#-news)\n- [\ud83d\udee0\ufe0f Installation](#\ufe0f-installation)\n - [Method 1: Install Using pip](#method-1-install-using-pip)\n - [Method 2: Install from Source](#method-2-install-from-source)\n- [\ud83d\ude80 Quick Start](#-quick-start)\n - [Method 1. Using Command Line](#method-1-using-command-line)\n - [Method 2. Using Python Code](#method-2-using-python-code)\n - [Basic Parameter](#basic-parameter)\n - [Output Results](#output-results)\n- [\ud83d\udcc8 Visualization of Evaluation Results](#-visualization-of-evaluation-results)\n- [\ud83c\udf10 Evaluation of Model API](#-evaluation-of-model-api)\n- [\u2699\ufe0f Custom Parameter Evaluation](#\ufe0f-custom-parameter-evaluation)\n - [Parameter Description](#parameter-description)\n- [\ud83e\uddea Other Evaluation Backends](#-other-evaluation-backends)\n- [\ud83d\udcc8 Model Serving Performance Evaluation](#-model-serving-performance-evaluation)\n- [\ud83d\udd8a\ufe0f Custom Dataset Evaluation](#\ufe0f-custom-dataset-evaluation)\n- [\u2694\ufe0f Arena Mode](#\ufe0f-arena-mode)\n- [\ud83d\udc77\u200d\u2642\ufe0f Contribution](#\ufe0f-contribution)\n- [\ud83d\udcda Citation](#-citation)\n- [\ud83d\udd1c Roadmap](#-roadmap)\n- [\u2b50 Star History](#-star-history)\n\n\n## \ud83d\udcdd Introduction\n\nEvalScope is a comprehensive model evaluation and performance benchmarking framework meticulously crafted by the [ModelScope Community](https://modelscope.cn/), offering a one-stop solution for your model assessment needs. Regardless of the type of model you are developing, EvalScope is equipped to cater to your requirements:\n\n- \ud83e\udde0 Large Language Models\n- \ud83c\udfa8 Multimodal Models\n- \ud83d\udd0d Embedding Models\n- \ud83c\udfc6 Reranker Models\n- \ud83d\uddbc\ufe0f CLIP Models\n- \ud83c\udfad AIGC Models (Image-to-Text/Video)\n- ...and more!\n\nEvalScope is not merely an evaluation tool; it is a valuable ally in your model optimization journey:\n\n- \ud83c\udfc5 Equipped with multiple industry-recognized benchmarks and evaluation metrics: MMLU, CMMLU, C-Eval, GSM8K, etc.\n- \ud83d\udcca Model inference performance stress testing: Ensuring your model excels in real-world applications.\n- \ud83d\ude80 Seamless integration with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, enabling one-click evaluations and providing full-chain support from training to assessment for your model development.\n\nBelow is the overall architecture diagram of EvalScope:\n\n<p align=\"center\">\n <img src=\"https://sail-moe.oss-cn-hangzhou.aliyuncs.com/yunlin/images/evalscope/doc/EvalScope%E6%9E%B6%E6%9E%84%E5%9B%BE.png\" width=\"70%\">\n <br>EvalScope Framework.\n</p>\n\n<details><summary>Framework Description</summary>\n\nThe architecture includes the following modules:\n1. Input Layer\n- **Model Sources**: API models (OpenAI API), local models (ModelScope)\n- **Datasets**: Standard evaluation benchmarks (MMLU/GSM8k, etc.), custom data (MCQ/QA)\n\n2. Core Functions\n- **Multi-backend Evaluation**\n - Native backends: Unified evaluation for LLM/VLM/Embedding/T2I models\n - Integrated frameworks: OpenCompass/MTEB/VLMEvalKit/RAGAS\n\n- **Performance Monitoring**\n - Model plugins: Supports various model service APIs\n - Data plugins: Supports multiple data formats\n - Metric tracking: TTFT/TPOP/Stability and other metrics\n\n- **Tool Extensions**\n - Integration: Tool-Bench/Needle-in-a-Haystack/BFCL-v3\n\n3. Output Layer\n- **Structured Reports**: Supports JSON/Tables/Logs\n- **Visualization Platforms**: Supports Gradio/Wandb/SwanLab\n\n</details>\n\n## \u260e User Groups\n\nPlease scan the QR code below to join our community groups:\n\n[Discord Group](https://discord.com/invite/D27yfEFVz5) | WeChat Group | DingTalk Group\n:-------------------------:|:-------------------------:|:-------------------------:\n<img src=\"docs/asset/discord_qr.jpg\" width=\"160\" height=\"160\"> | <img src=\"docs/asset/wechat.png\" width=\"160\" height=\"160\"> | <img src=\"docs/asset/dingding.png\" width=\"160\" height=\"160\">\n\n\n## \ud83c\udf89 News\n- \ud83d\udd25 **[2025.07.18]** The model stress testing now supports randomly generating image-text data for multimodal model evaluation. For usage instructions, refer to the [documentation](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/examples.html#id4).\n- \ud83d\udd25 **[2025.07.16]** Support for [\u03c4-bench](https://github.com/sierra-research/tau-bench) has been added, enabling the evaluation of AI Agent performance and reliability in real-world scenarios involving dynamic user and tool interactions. For usage instructions, please refer to the [documentation](https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset/llm.html#bench).\n- \ud83d\udd25 **[2025.07.14]** Support for \"Humanity's Last Exam\" ([Humanity's-Last-Exam](https://modelscope.cn/datasets/cais/hle)), a highly challenging evaluation benchmark. For usage instructions, refer to the [documentation](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset/llm.html#humanity-s-last-exam).\n- \ud83d\udd25 **[2025.07.03]** Refactored Arena Mode: now supports custom model battles, outputs a model leaderboard, and provides battle result visualization. See [reference](https://evalscope.readthedocs.io/en/latest/user_guides/arena.html) for details.\n- \ud83d\udd25 **[2025.06.28]** Optimized custom dataset evaluation: now supports evaluation without reference answers. Enhanced LLM judge usage, with built-in modes for \"scoring directly without reference answers\" and \"checking answer consistency with reference answers\". See [reference](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset/llm.html#qa) for details.\n- \ud83d\udd25 **[2025.06.19]** Added support for the [BFCL-v3](https://modelscope.cn/datasets/AI-ModelScope/bfcl_v3) benchmark, designed to evaluate model function-calling capabilities across various scenarios. For more information, refer to the [documentation](https://evalscope.readthedocs.io/zh-cn/latest/third_party/bfcl_v3.html).\n- \ud83d\udd25 **[2025.06.02]** Added support for the Needle-in-a-Haystack test. Simply specify `needle_haystack` to conduct the test, and a corresponding heatmap will be generated in the `outputs/reports` folder, providing a visual representation of the model's performance. Refer to the [documentation](https://evalscope.readthedocs.io/en/latest/third_party/needle_haystack.html) for more details.\n- \ud83d\udd25 **[2025.05.29]** Added support for two long document evaluation benchmarks: [DocMath](https://modelscope.cn/datasets/yale-nlp/DocMath-Eval/summary) and [FRAMES](https://modelscope.cn/datasets/iic/frames/summary). For usage guidelines, please refer to the [documentation](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset.html).\n- \ud83d\udd25 **[2025.05.16]** Model service performance stress testing now supports setting various levels of concurrency and outputs a performance test report. [Reference example](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#id3).\n- \ud83d\udd25 **[2025.05.13]** Added support for the [ToolBench-Static](https://modelscope.cn/datasets/AI-ModelScope/ToolBench-Static) dataset to evaluate model's tool-calling capabilities. Refer to the [documentation](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) for usage instructions. Also added support for the [DROP](https://modelscope.cn/datasets/AI-ModelScope/DROP/dataPeview) and [Winogrande](https://modelscope.cn/datasets/AI-ModelScope/winogrande_val) benchmarks to assess the reasoning capabilities of models.\n- \ud83d\udd25 **[2025.04.29]** Added Qwen3 Evaluation Best Practices, [welcome to read \ud83d\udcd6](https://evalscope.readthedocs.io/en/latest/best_practice/qwen3.html)\n- \ud83d\udd25 **[2025.04.27]** Support for text-to-image evaluation: Supports 8 metrics including MPS, HPSv2.1Score, etc., and evaluation benchmarks such as EvalMuse, GenAI-Bench. Refer to the [user documentation](https://evalscope.readthedocs.io/en/latest/user_guides/aigc/t2i.html) for more details.\n- \ud83d\udd25 **[2025.04.10]** Model service stress testing tool now supports the `/v1/completions` endpoint (the default endpoint for vLLM benchmarking)\n- \ud83d\udd25 **[2025.04.08]** Support for evaluating embedding model services compatible with the OpenAI API has been added. For more details, check the [user guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html#configure-evaluation-parameters).\n<details><summary>More</summary>\n\n- \ud83d\udd25 **[2025.03.27]** Added support for [AlpacaEval](https://www.modelscope.cn/datasets/AI-ModelScope/alpaca_eval/dataPeview) and [ArenaHard](https://modelscope.cn/datasets/AI-ModelScope/arena-hard-auto-v0.1/summary) evaluation benchmarks. For usage notes, please refer to the [documentation](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset.html)\n- \ud83d\udd25 **[2025.03.20]** The model inference service stress testing now supports generating prompts of specified length using random values. Refer to the [user guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/examples.html#using-the-random-dataset) for more details.\n- \ud83d\udd25 **[2025.03.13]** Added support for the [LiveCodeBench](https://www.modelscope.cn/datasets/AI-ModelScope/code_generation_lite/summary) code evaluation benchmark, which can be used by specifying `live_code_bench`. Supports evaluating QwQ-32B on LiveCodeBench, refer to the [best practices](https://evalscope.readthedocs.io/en/latest/best_practice/eval_qwq.html).\n- \ud83d\udd25 **[2025.03.11]** Added support for the [SimpleQA](https://modelscope.cn/datasets/AI-ModelScope/SimpleQA/summary) and [Chinese SimpleQA](https://modelscope.cn/datasets/AI-ModelScope/Chinese-SimpleQA/summary) evaluation benchmarks. These are used to assess the factual accuracy of models, and you can specify `simple_qa` and `chinese_simpleqa` for use. Support for specifying a judge model is also available. For more details, refer to the [relevant parameter documentation](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html).\n- \ud83d\udd25 **[2025.03.07]** Added support for the [QwQ-32B](https://modelscope.cn/models/Qwen/QwQ-32B/summary) model, evaluate the model's reasoning ability and reasoning efficiency, refer to [\ud83d\udcd6 Best Practices for QwQ-32B Evaluation](https://evalscope.readthedocs.io/en/latest/best_practice/eval_qwq.html) for more details.\n- \ud83d\udd25 **[2025.03.04]** Added support for the [SuperGPQA](https://modelscope.cn/datasets/m-a-p/SuperGPQA/summary) dataset, which covers 13 categories, 72 first-level disciplines, and 285 second-level disciplines, totaling 26,529 questions. You can use it by specifying `super_gpqa`.\n- \ud83d\udd25 **[2025.03.03]** Added support for evaluating the IQ and EQ of models. Refer to [\ud83d\udcd6 Best Practices for IQ and EQ Evaluation](https://evalscope.readthedocs.io/en/latest/best_practice/iquiz.html) to find out how smart your AI is!\n- \ud83d\udd25 **[2025.02.27]** Added support for evaluating the reasoning efficiency of models. Refer to [\ud83d\udcd6 Best Practices for Evaluating Thinking Efficiency](https://evalscope.readthedocs.io/en/latest/best_practice/think_eval.html). This implementation is inspired by the works [Overthinking](https://doi.org/10.48550/arXiv.2412.21187) and [Underthinking](https://doi.org/10.48550/arXiv.2501.18585).\n- \ud83d\udd25 **[2025.02.25]** Added support for two model inference-related evaluation benchmarks: [MuSR](https://modelscope.cn/datasets/AI-ModelScope/MuSR) and [ProcessBench](https://www.modelscope.cn/datasets/Qwen/ProcessBench/summary). To use them, simply specify `musr` and `process_bench` respectively in the datasets parameter.\n- \ud83d\udd25 **[2025.02.18]** Supports the AIME25 dataset, which contains 15 questions (Grok3 scored 93 on this dataset).\n- \ud83d\udd25 **[2025.02.13]** Added support for evaluating DeepSeek distilled models, including AIME24, MATH-500, and GPQA-Diamond datasets\uff0crefer to [best practice](https://evalscope.readthedocs.io/en/latest/best_practice/deepseek_r1_distill.html); Added support for specifying the `eval_batch_size` parameter to accelerate model evaluation.\n- \ud83d\udd25 **[2025.01.20]** Support for visualizing evaluation results, including single model evaluation results and multi-model comparison, refer to the [\ud83d\udcd6 Visualizing Evaluation Results](https://evalscope.readthedocs.io/en/latest/get_started/visualization.html) for more details; Added [`iquiz`](https://modelscope.cn/datasets/AI-ModelScope/IQuiz/summary) evaluation example, evaluating the IQ and EQ of the model.\n- \ud83d\udd25 **[2025.01.07]** Native backend: Support for model API evaluation is now available. Refer to the [\ud83d\udcd6 Model API Evaluation Guide](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#api) for more details. Additionally, support for the `ifeval` evaluation benchmark has been added.\n- \ud83d\udd25\ud83d\udd25 **[2024.12.31]** Support for adding benchmark evaluations, refer to the [\ud83d\udcd6 Benchmark Evaluation Addition Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/add_benchmark.html); support for custom mixed dataset evaluations, allowing for more comprehensive model evaluations with less data, refer to the [\ud83d\udcd6 Mixed Dataset Evaluation Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/collection/index.html).\n- \ud83d\udd25 **[2024.12.13]** Model evaluation optimization: no need to pass the `--template-type` parameter anymore; supports starting evaluation with `evalscope eval --args`. Refer to the [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html) for more details.\n- \ud83d\udd25 **[2024.11.26]** The model inference service performance evaluator has been completely refactored: it now supports local inference service startup and Speed Benchmark; asynchronous call error handling has been optimized. For more details, refer to the [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/index.html).\n- \ud83d\udd25 **[2024.10.31]** The best practice for evaluating Multimodal-RAG has been updated, please check the [\ud83d\udcd6 Blog](https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html#multimodal-rag) for more details.\n- \ud83d\udd25 **[2024.10.23]** Supports multimodal RAG evaluation, including the assessment of image-text retrieval using [CLIP_Benchmark](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/clip_benchmark.html), and extends [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html) to support end-to-end multimodal metrics evaluation.\n- \ud83d\udd25 **[2024.10.8]** Support for RAG evaluation, including independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html).\n- \ud83d\udd25 **[2024.09.18]** Our documentation has been updated to include a blog module, featuring some technical research and discussions related to evaluations. We invite you to [\ud83d\udcd6 read it](https://evalscope.readthedocs.io/en/refact_readme/blog/index.html).\n- \ud83d\udd25 **[2024.09.12]** Support for LongWriter evaluation, which supports 10,000+ word generation. You can use the benchmark [LongBench-Write](evalscope/third_party/longbench_write/README.md) to measure the long output quality as well as the output length.\n- \ud83d\udd25 **[2024.08.30]** Support for custom dataset evaluations, including text datasets and multimodal image-text datasets.\n- \ud83d\udd25 **[2024.08.20]** Updated the official documentation, including getting started guides, best practices, and FAQs. Feel free to [\ud83d\udcd6read it here](https://evalscope.readthedocs.io/en/latest/)!\n- \ud83d\udd25 **[2024.08.09]** Simplified the installation process, allowing for pypi installation of vlmeval dependencies; optimized the multimodal model evaluation experience, achieving up to 10x acceleration based on the OpenAI API evaluation chain.\n- \ud83d\udd25 **[2024.07.31]** Important change: The package name `llmuses` has been changed to `evalscope`. Please update your code accordingly.\n- \ud83d\udd25 **[2024.07.26]** Support for **VLMEvalKit** as a third-party evaluation framework to initiate multimodal model evaluation tasks.\n- \ud83d\udd25 **[2024.06.29]** Support for **OpenCompass** as a third-party evaluation framework, which we have encapsulated at a higher level, supporting pip installation and simplifying evaluation task configuration.\n- \ud83d\udd25 **[2024.06.13]** EvalScope seamlessly integrates with the fine-tuning framework SWIFT, providing full-chain support from LLM training to evaluation.\n- \ud83d\udd25 **[2024.06.13]** Integrated the Agent evaluation dataset ToolBench.\n\n</details>\n\n## \ud83d\udee0\ufe0f Installation\n### Method 1: Install Using pip\nWe recommend using conda to manage your environment and installing dependencies with pip:\n\n1. Create a conda environment (optional)\n ```shell\n # It is recommended to use Python 3.10\n conda create -n evalscope python=3.10\n # Activate the conda environment\n conda activate evalscope\n ```\n\n2. Install dependencies using pip\n ```shell\n pip install evalscope # Install Native backend (default)\n # Additional options\n pip install 'evalscope[opencompass]' # Install OpenCompass backend\n pip install 'evalscope[vlmeval]' # Install VLMEvalKit backend\n pip install 'evalscope[rag]' # Install RAGEval backend\n pip install 'evalscope[perf]' # Install dependencies for the model performance testing module\n pip install 'evalscope[app]' # Install dependencies for visualization\n pip install 'evalscope[all]' # Install all backends (Native, OpenCompass, VLMEvalKit, RAGEval)\n ```\n\n> [!WARNING]\n> As the project has been renamed to `evalscope`, for versions `v0.4.3` or earlier, you can install using the following command:\n> ```shell\n> pip install llmuses<=0.4.3\n> ```\n> To import relevant dependencies using `llmuses`:\n> ``` python\n> from llmuses import ...\n> ```\n\n### Method 2: Install from Source\n1. Download the source code\n ```shell\n git clone https://github.com/modelscope/evalscope.git\n ```\n\n2. Install dependencies\n ```shell\n cd evalscope/\n pip install -e . # Install Native backend\n # Additional options\n pip install -e '.[opencompass]' # Install OpenCompass backend\n pip install -e '.[vlmeval]' # Install VLMEvalKit backend\n pip install -e '.[rag]' # Install RAGEval backend\n pip install -e '.[perf]' # Install Perf dependencies\n pip install -e '.[app]' # Install visualization dependencies\n pip install -e '.[all]' # Install all backends (Native, OpenCompass, VLMEvalKit, RAGEval)\n ```\n\n\n## \ud83d\ude80 Quick Start\n\nTo evaluate a model on specified datasets using default configurations, this framework supports two ways to initiate evaluation tasks: using the command line or using Python code.\n\n### Method 1. Using Command Line\n\nExecute the `eval` command in any directory:\n```bash\nevalscope eval \\\n --model Qwen/Qwen2.5-0.5B-Instruct \\\n --datasets gsm8k arc \\\n --limit 5\n```\n\n### Method 2. Using Python Code\n\nWhen using Python code for evaluation, you need to submit the evaluation task using the `run_task` function, passing a `TaskConfig` as a parameter. It can also be a Python dictionary, yaml file path, or json file path, for example:\n\n**Using `TaskConfig`**\n\n```python\nfrom evalscope import run_task, TaskConfig\n\ntask_cfg = TaskConfig(\n model='Qwen/Qwen2.5-0.5B-Instruct',\n datasets=['gsm8k', 'arc'],\n limit=5\n)\n\nrun_task(task_cfg=task_cfg)\n```\n<details><summary>More Startup Methods</summary>\n\n**Using Python Dictionary**\n\n```python\nfrom evalscope.run import run_task\n\ntask_cfg = {\n 'model': 'Qwen/Qwen2.5-0.5B-Instruct',\n 'datasets': ['gsm8k', 'arc'],\n 'limit': 5\n}\n\nrun_task(task_cfg=task_cfg)\n```\n\n**Using `yaml` file**\n\n`config.yaml`:\n```yaml\nmodel: Qwen/Qwen2.5-0.5B-Instruct\ndatasets:\n - gsm8k\n - arc\nlimit: 5\n```\n\n```python\nfrom evalscope.run import run_task\n\nrun_task(task_cfg=\"config.yaml\")\n```\n\n**Using `json` file**\n\n`config.json`:\n```json\n{\n \"model\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"datasets\": [\"gsm8k\", \"arc\"],\n \"limit\": 5\n}\n```\n\n```python\nfrom evalscope.run import run_task\n\nrun_task(task_cfg=\"config.json\")\n```\n</details>\n\n### Basic Parameter\n- `--model`: Specifies the `model_id` of the model in [ModelScope](https://modelscope.cn/), which can be automatically downloaded, e.g., [Qwen/Qwen2.5-0.5B-Instruct](https://modelscope.cn/models/Qwen/Qwen2.5-0.5B-Instruct/summary); or use the local path of the model, e.g., `/path/to/model`\n- `--datasets`: Dataset names, supports inputting multiple datasets separated by spaces. Datasets will be automatically downloaded from modelscope. For supported datasets, refer to the [Dataset List](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset.html)\n- `--limit`: Maximum amount of evaluation data for each dataset. If not specified, it defaults to evaluating all data. Can be used for quick validation\n\n### Output Results\n```text\n+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+\n| Model Name | Dataset Name | Metric Name | Category Name | Subset Name | Num | Score |\n+=======================+================+=================+=================+===============+=======+=========+\n| Qwen2.5-0.5B-Instruct | gsm8k | AverageAccuracy | default | main | 5 | 0.4 |\n+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+\n| Qwen2.5-0.5B-Instruct | ai2_arc | AverageAccuracy | default | ARC-Easy | 5 | 0.8 |\n+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+\n| Qwen2.5-0.5B-Instruct | ai2_arc | AverageAccuracy | default | ARC-Challenge | 5 | 0.4 |\n+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+\n```\n\n## \ud83d\udcc8 Visualization of Evaluation Results\n\n1. Install the dependencies required for visualization, including gradio, plotly, etc.\n```bash\npip install 'evalscope[app]'\n```\n\n2. Start the Visualization Service\n\nRun the following command to start the visualization service.\n```bash\nevalscope app\n```\nYou can access the visualization service in the browser if the following output appears.\n```text\n* Running on local URL: http://127.0.0.1:7861\n\nTo create a public link, set `share=True` in `launch()`.\n```\n\n<table>\n <tr>\n <td style=\"text-align: center;\">\n <img src=\"docs/en/get_started/images/setting.png\" alt=\"Setting\" style=\"width: 75%;\" />\n <p>Setting Interface</p>\n </td>\n <td style=\"text-align: center;\">\n <img src=\"docs/en/get_started/images/model_compare.png\" alt=\"Model Compare\" style=\"width: 100%;\" />\n <p>Model Comparison</p>\n </td>\n </tr>\n <tr>\n <td style=\"text-align: center;\">\n <img src=\"docs/en/get_started/images/report_overview.png\" alt=\"Report Overview\" style=\"width: 100%;\" />\n <p>Report Overview</p>\n </td>\n <td style=\"text-align: center;\">\n <img src=\"docs/en/get_started/images/report_details.png\" alt=\"Report Details\" style=\"width: 80%;\" />\n <p>Report Details</p>\n </td>\n </tr>\n</table>\n\nFor more details, refer to: [\ud83d\udcd6 Visualization of Evaluation Results](https://evalscope.readthedocs.io/en/latest/get_started/visualization.html)\n\n## \ud83c\udf10 Evaluation of Model API\n\nSpecify the model API service address (api_url) and API Key (api_key) to evaluate the deployed model API service. In this case, the `eval-type` parameter must be specified as `service`, for example:\n\nFor example, to launch a model service using [vLLM](https://github.com/vllm-project/vllm):\n\n```shell\nexport VLLM_USE_MODELSCOPE=True && python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-0.5B-Instruct --served-model-name qwen2.5 --trust_remote_code --port 8801\n```\nThen, you can use the following command to evaluate the model API service:\n```shell\nevalscope eval \\\n --model qwen2.5 \\\n --api-url http://127.0.0.1:8801/v1 \\\n --api-key EMPTY \\\n --eval-type service \\\n --datasets gsm8k \\\n --limit 10\n```\n\n## \u2699\ufe0f Custom Parameter Evaluation\n\nFor more customized evaluations, such as customizing model parameters or dataset parameters, you can use the following command. The evaluation startup method is the same as simple evaluation. Below shows how to start the evaluation using the `eval` command:\n\n```shell\nevalscope eval \\\n --model Qwen/Qwen3-0.6B \\\n --model-args '{\"revision\": \"master\", \"precision\": \"torch.float16\", \"device_map\": \"auto\"}' \\\n --generation-config '{\"do_sample\":true,\"temperature\":0.6,\"max_new_tokens\":512,\"chat_template_kwargs\":{\"enable_thinking\": false}}' \\\n --dataset-args '{\"gsm8k\": {\"few_shot_num\": 0, \"few_shot_random\": false}}' \\\n --datasets gsm8k \\\n --limit 10\n```\n\n### Parameter Description\n- `--model-args`: Model loading parameters, passed as a JSON string:\n - `revision`: Model version\n - `precision`: Model precision\n - `device_map`: Device allocation for the model\n- `--generation-config`: Generation parameters, passed as a JSON string and parsed as a dictionary:\n - `do_sample`: Whether to use sampling\n - `temperature`: Generation temperature\n - `max_new_tokens`: Maximum length of generated tokens\n - `chat_template_kwargs`: Model inference template parameters\n- `--dataset-args`: Settings for the evaluation dataset, passed as a JSON string where the key is the dataset name and the value is the parameters. Note that these need to correspond one-to-one with the values in the `--datasets` parameter:\n - `few_shot_num`: Number of few-shot examples\n - `few_shot_random`: Whether to randomly sample few-shot data; if not set, defaults to `true`\n\nReference: [Full Parameter Description](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)\n\n\n## \ud83e\uddea Other Evaluation Backends\nEvalScope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes:\n- **Native**: EvalScope's own **default evaluation framework**, supporting various evaluation modes including single model evaluation, arena mode, and baseline model comparison mode.\n- [OpenCompass](https://github.com/open-compass/opencompass): Initiate OpenCompass evaluation tasks through EvalScope. Lightweight, easy to customize, supports seamless integration with the LLM fine-tuning framework ms-swift. [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/opencompass_backend.html)\n- [VLMEvalKit](https://github.com/open-compass/VLMEvalKit): Initiate VLMEvalKit multimodal evaluation tasks through EvalScope. Supports various multimodal models and datasets, and offers seamless integration with the LLM fine-tuning framework ms-swift. [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/vlmevalkit_backend.html)\n- **RAGEval**: Initiate RAG evaluation tasks through EvalScope, supporting independent evaluation of embedding models and rerankers using [MTEB/CMTEB](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/mteb.html), as well as end-to-end evaluation using [RAGAS](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html): [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/index.html)\n- **ThirdParty**: Third-party evaluation tasks, such as [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) and [LongBench-Write](https://evalscope.readthedocs.io/en/latest/third_party/longwriter.html).\n\n\n## \ud83d\udcc8 Model Serving Performance Evaluation\nA stress testing tool focused on large language models, which can be customized to support various dataset formats and different API protocol formats.\n\nReference: Performance Testing [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/index.html)\n\n**Output example**\n\n\n\n\n**Supports wandb for recording results**\n\n\n\n**Supports swanlab for recording results**\n\n\n\n**Supports Speed Benchmark**\n\nIt supports speed testing and provides speed benchmarks similar to those found in the [official Qwen](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html) reports:\n\n```text\nSpeed Benchmark Results:\n+---------------+-----------------+----------------+\n| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |\n+---------------+-----------------+----------------+\n| 1 | 50.69 | 0.97 |\n| 6144 | 51.36 | 1.23 |\n| 14336 | 49.93 | 1.59 |\n| 30720 | 49.56 | 2.34 |\n+---------------+-----------------+----------------+\n```\n\n## \ud83d\udd8a\ufe0f Custom Dataset Evaluation\nEvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation [\ud83d\udcd6User Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset/index.html)\n\n\n## \u2694\ufe0f Arena Mode\n\nArena mode allows you to configure multiple candidate models and specify a baseline model. Evaluation is performed by pairwise battles between each candidate model and the baseline model, with the final output including each model's win rate and ranking. This method is suitable for comparative evaluation among multiple models, providing an intuitive reflection of each model's strengths and weaknesses. Refer to: Arena Mode [\ud83d\udcd6 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/arena.html)\n\n```text\nModel WinRate (%) CI (%)\n------------ ------------- ---------------\nqwen2.5-72b 69.3 (-13.3 / +12.2)\nqwen2.5-7b 50 (+0.0 / +0.0)\nqwen2.5-0.5b 4.7 (-2.5 / +4.4)\n```\n\n## \ud83d\udc77\u200d\u2642\ufe0f Contribution\n\nEvalScope, as the official evaluation tool of [ModelScope](https://modelscope.cn), is continuously optimizing its benchmark evaluation features! We invite you to refer to the [Contribution Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/add_benchmark.html) to easily add your own evaluation benchmarks and share your contributions with the community. Let\u2019s work together to support the growth of EvalScope and make our tools even better! Join us now!\n\n<a href=\"https://github.com/modelscope/evalscope/graphs/contributors\" target=\"_blank\">\n <table>\n <tr>\n <th colspan=\"2\">\n <br><img src=\"https://contrib.rocks/image?repo=modelscope/evalscope\"><br><br>\n </th>\n </tr>\n </table>\n</a>\n\n## \ud83d\udcda Citation\n\n```bibtex\n@misc{evalscope_2024,\n title={{EvalScope}: Evaluation Framework for Large Models},\n author={ModelScope Team},\n year={2024},\n url={https://github.com/modelscope/evalscope}\n}\n```\n\n## \ud83d\udd1c Roadmap\n- [x] Support for better evaluation report visualization\n- [x] Support for mixed evaluations across multiple datasets\n- [x] RAG evaluation\n- [x] VLM evaluation\n- [x] Agents evaluation\n- [x] vLLM\n- [ ] Distributed evaluating\n- [x] Multi-modal evaluation\n- [ ] Benchmarks\n - [x] BFCL-v3\n - [x] GPQA\n - [x] MBPP\n\n\n## \u2b50 Star History\n\n[](https://star-history.com/#modelscope/evalscope&Date)\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "EvalScope: Lightweight LLMs Evaluation Framework",
"version": "0.17.1",
"project_urls": {
"Homepage": "https://github.com/modelscope/evalscope"
},
"split_keywords": [
"python",
" llm",
" evaluation"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "a9b5630d2c5dc5c32e9fbad5034e04d8aba6f6461dc08f255df77dd8d463857f",
"md5": "e3886ff4a1428cdfa970d025c3bdc017",
"sha256": "8cd7ce2bccf732e5e0c1c3e05604c3420e12b4c6e8820e1e4ab820fb0d9752cc"
},
"downloads": -1,
"filename": "evalscope-0.17.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e3886ff4a1428cdfa970d025c3bdc017",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 754236,
"upload_time": "2025-07-21T02:12:54",
"upload_time_iso_8601": "2025-07-21T02:12:54.633250Z",
"url": "https://files.pythonhosted.org/packages/a9/b5/630d2c5dc5c32e9fbad5034e04d8aba6f6461dc08f255df77dd8d463857f/evalscope-0.17.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "3888326e48929bb9577a6a36e07afa65bbf6bd870c1b644f82e5713874ae3238",
"md5": "18adc9ace9bf8173db6f60b244a30e66",
"sha256": "51977bc0f9022825b0ddf31605b2d172588a4b97dddbf0e60032dd68e591015e"
},
"downloads": -1,
"filename": "evalscope-0.17.1.tar.gz",
"has_sig": false,
"md5_digest": "18adc9ace9bf8173db6f60b244a30e66",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 558325,
"upload_time": "2025-07-21T02:12:56",
"upload_time_iso_8601": "2025-07-21T02:12:56.554169Z",
"url": "https://files.pythonhosted.org/packages/38/88/326e48929bb9577a6a36e07afa65bbf6bd870c1b644f82e5713874ae3238/evalscope-0.17.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-21 02:12:56",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "modelscope",
"github_project": "evalscope",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "evalscope"
}
JSON
