evalscope 0.15.1__tar.gz → 0.16.0__tar.gz

{evalscope-0.15.1/evalscope.egg-info → evalscope-0.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: evalscope
-Version: 0.15.1
+Version: 0.16.0
 Summary: EvalScope: Lightweight LLMs Evaluation Framework
 Home-page: https://github.com/modelscope/evalscope
 Author: ModelScope team
@@ -60,9 +60,10 @@ Provides-Extra: perf
 Requires-Dist: aiohttp; extra == "perf"
 Requires-Dist: fastapi; extra == "perf"
 Requires-Dist: numpy; extra == "perf"
+Requires-Dist: rich; extra == "perf"
 Requires-Dist: sse_starlette; extra == "perf"
 Requires-Dist: transformers; extra == "perf"
-Requires-Dist: unicorn; extra == "perf"
+Requires-Dist: uvicorn; extra == "perf"
 Provides-Extra: app
 Requires-Dist: gradio==5.4.0; extra == "app"
 Requires-Dist: plotly<6.0.0,>=5.23.0; extra == "app"
@@ -113,9 +114,10 @@ Requires-Dist: webdataset>0.2.0; extra == "all"
 Requires-Dist: aiohttp; extra == "all"
 Requires-Dist: fastapi; extra == "all"
 Requires-Dist: numpy; extra == "all"
+Requires-Dist: rich; extra == "all"
 Requires-Dist: sse_starlette; extra == "all"
 Requires-Dist: transformers; extra == "all"
-Requires-Dist: unicorn; extra == "all"
+Requires-Dist: uvicorn; extra == "all"
 Requires-Dist: gradio==5.4.0; extra == "all"
 Requires-Dist: plotly<6.0.0,>=5.23.0; extra == "all"
 Requires-Dist: diffusers; extra == "all"
@@ -177,9 +179,23 @@ Requires-Dist: opencv-python; extra == "all"
 
 ## 📝 Introduction
 
-EvalScope is [ModelScope](https://modelscope.cn/)'s official framework for model evaluation and benchmarking, designed for diverse assessment needs. It supports various model types including large language models, multimodal, embedding, reranker, and CLIP models.
+EvalScope is a comprehensive model evaluation and performance benchmarking framework meticulously crafted by the [ModelScope Community](https://modelscope.cn/), offering a one-stop solution for your model assessment needs. Regardless of the type of model you are developing, EvalScope is equipped to cater to your requirements:
 
-The framework accommodates multiple evaluation scenarios such as end-to-end RAG evaluation, arena mode, and inference performance testing. It features built-in benchmarks and metrics like MMLU, CMMLU, C-Eval, and GSM8K. Seamlessly integrated with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, EvalScope enables one-click evaluations, offering comprehensive support for model training and assessment 🚀
+- 🧠 Large Language Models
+- 🎨 Multimodal Models
+- 🔍 Embedding Models
+- 🏆 Reranker Models
+- 🖼️ CLIP Models
+- 🎭 AIGC Models (Image-to-Text/Video)
+- ...and more!
+
+EvalScope is not merely an evaluation tool; it is a valuable ally in your model optimization journey:
+
+- 🏅 Equipped with multiple industry-recognized benchmarks and evaluation metrics: MMLU, CMMLU, C-Eval, GSM8K, etc.
+- 📊 Model inference performance stress testing: Ensuring your model excels in real-world applications.
+- 🚀 Seamless integration with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, enabling one-click evaluations and providing full-chain support from training to assessment for your model development.
+
+Below is the overall architecture diagram of EvalScope:
 
 <p align="center">
   <img src="docs/en/_static/images/evalscope_framework.png" width="70%">
@@ -214,6 +230,8 @@ Please scan the QR code below to join our community groups:
 
 ## 🎉 News
 
+- 🔥 **[2025.05.16]** Model service performance stress testing now supports setting various levels of concurrency and outputs a performance test report. [Reference example](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#id3).
+- 🔥 **[2025.05.13]** Added support for the [ToolBench-Static](https://modelscope.cn/datasets/AI-ModelScope/ToolBench-Static) dataset to evaluate model's tool-calling capabilities. Refer to the [documentation](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) for usage instructions. Also added support for the [DROP](https://modelscope.cn/datasets/AI-ModelScope/DROP/dataPeview) and [Winogrande](https://modelscope.cn/datasets/AI-ModelScope/winogrande_val) benchmarks to assess the reasoning capabilities of models.
 - 🔥 **[2025.04.29]** Added Qwen3 Evaluation Best Practices, [welcome to read 📖](https://evalscope.readthedocs.io/en/latest/best_practice/qwen3.html)
 - 🔥 **[2025.04.27]** Support for text-to-image evaluation: Supports 8 metrics including MPS, HPSv2.1Score, etc., and evaluation benchmarks such as EvalMuse, GenAI-Bench. Refer to the [user documentation](https://evalscope.readthedocs.io/en/latest/user_guides/aigc/t2i.html) for more details.
 - 🔥 **[2025.04.10]** Model service stress testing tool now supports the `/v1/completions` endpoint (the default endpoint for vLLM benchmarking)
@@ -479,26 +497,27 @@ For more customized evaluations, such as customizing model parameters or dataset
 
 ```shell
 evalscope eval \
- --model Qwen/Qwen2.5-0.5B-Instruct \
- --model-args revision=master,precision=torch.float16,device_map=auto \
- --generation-config do_sample=true,temperature=0.5 \
+ --model Qwen/Qwen3-0.6B \
+ --model-args '{"revision": "master", "precision": "torch.float16", "device_map": "auto"}' \
+ --generation-config '{"do_sample":true,"temperature":0.6,"max_new_tokens":512,"chat_template_kwargs":{"enable_thinking": false}}' \
  --dataset-args '{"gsm8k": {"few_shot_num": 0, "few_shot_random": false}}' \
  --datasets gsm8k \
  --limit 10
 ```
 
-### Parameter
-- `--model-args`: Model loading parameters, separated by commas in `key=value` format. Default parameters:
-  - `revision`: Model version, default is `master`
-  - `precision`: Model precision, default is `auto`
-  - `device_map`: Model device allocation, default is `auto`
-- `--generation-config`: Generation parameters, separated by commas in `key=value` format. Default parameters:
-  - `do_sample`: Whether to use sampling, default is `false`
-  - `max_length`: Maximum length, default is 2048
-  - `max_new_tokens`: Maximum length of generation, default is 512
-- `--dataset-args`: Configuration parameters for evaluation datasets, passed in `json` format. The key is the dataset name, and the value is the parameters. Note that it needs to correspond one-to-one with the values in the `--datasets` parameter:
+### Parameter Description
+- `--model-args`: Model loading parameters, passed as a JSON string:
+  - `revision`: Model version
+  - `precision`: Model precision
+  - `device_map`: Device allocation for the model
+- `--generation-config`: Generation parameters, passed as a JSON string and parsed as a dictionary:
+  - `do_sample`: Whether to use sampling
+  - `temperature`: Generation temperature
+  - `max_new_tokens`: Maximum length of generated tokens
+  - `chat_template_kwargs`: Model inference template parameters
+- `--dataset-args`: Settings for the evaluation dataset, passed as a JSON string where the key is the dataset name and the value is the parameters. Note that these need to correspond one-to-one with the values in the `--datasets` parameter:
   - `few_shot_num`: Number of few-shot examples
-  - `few_shot_random`: Whether to randomly sample few-shot data, if not set, defaults to `true`
+  - `few_shot_random`: Whether to randomly sample few-shot data; if not set, defaults to `true`
 
 Reference: [Full Parameter Description](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)
 
@@ -517,6 +536,11 @@ A stress testing tool focused on large language models, which can be customized
 
 Reference: Performance Testing [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/index.html)
 
+**Output example**
+
+![multi_perf](docs/en/user_guides/stress_test/images/multi_perf.png)
+
+
 **Supports wandb for recording results**
 
 ![wandb sample](https://modelscope.oss-cn-beijing.aliyuncs.com/resource/wandb_sample.png)
@@ -565,7 +589,7 @@ EvalScope, as the official evaluation tool of [ModelScope](https://modelscope.cn
 </a>
 
 ## 🔜 Roadmap
-- [ ] Support for better evaluation report visualization
+- [x] Support for better evaluation report visualization
 - [x] Support for mixed evaluations across multiple datasets
 - [x] RAG evaluation
 - [x] VLM evaluation
@@ -575,7 +599,7 @@ EvalScope, as the official evaluation tool of [ModelScope](https://modelscope.cn
 - [x] Multi-modal evaluation
 - [ ] Benchmarks
   - [ ] GAIA
-  - [ ] GPQA
+  - [x] GPQA
   - [x] MBPP
 
 

{evalscope-0.15.1 → evalscope-0.16.0}/README.md

@@ -51,9 +51,23 @@
 
 ## 📝 Introduction
 
-EvalScope is [ModelScope](https://modelscope.cn/)'s official framework for model evaluation and benchmarking, designed for diverse assessment needs. It supports various model types including large language models, multimodal, embedding, reranker, and CLIP models.
+EvalScope is a comprehensive model evaluation and performance benchmarking framework meticulously crafted by the [ModelScope Community](https://modelscope.cn/), offering a one-stop solution for your model assessment needs. Regardless of the type of model you are developing, EvalScope is equipped to cater to your requirements:
 
-The framework accommodates multiple evaluation scenarios such as end-to-end RAG evaluation, arena mode, and inference performance testing. It features built-in benchmarks and metrics like MMLU, CMMLU, C-Eval, and GSM8K. Seamlessly integrated with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, EvalScope enables one-click evaluations, offering comprehensive support for model training and assessment 🚀
+- 🧠 Large Language Models
+- 🎨 Multimodal Models
+- 🔍 Embedding Models
+- 🏆 Reranker Models
+- 🖼️ CLIP Models
+- 🎭 AIGC Models (Image-to-Text/Video)
+- ...and more!
+
+EvalScope is not merely an evaluation tool; it is a valuable ally in your model optimization journey:
+
+- 🏅 Equipped with multiple industry-recognized benchmarks and evaluation metrics: MMLU, CMMLU, C-Eval, GSM8K, etc.
+- 📊 Model inference performance stress testing: Ensuring your model excels in real-world applications.
+- 🚀 Seamless integration with the [ms-swift](https://github.com/modelscope/ms-swift) training framework, enabling one-click evaluations and providing full-chain support from training to assessment for your model development.
+
+Below is the overall architecture diagram of EvalScope:
 
 <p align="center">
   <img src="docs/en/_static/images/evalscope_framework.png" width="70%">
@@ -88,6 +102,8 @@ Please scan the QR code below to join our community groups:
 
 ## 🎉 News
 
+- 🔥 **[2025.05.16]** Model service performance stress testing now supports setting various levels of concurrency and outputs a performance test report. [Reference example](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html#id3).
+- 🔥 **[2025.05.13]** Added support for the [ToolBench-Static](https://modelscope.cn/datasets/AI-ModelScope/ToolBench-Static) dataset to evaluate model's tool-calling capabilities. Refer to the [documentation](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html) for usage instructions. Also added support for the [DROP](https://modelscope.cn/datasets/AI-ModelScope/DROP/dataPeview) and [Winogrande](https://modelscope.cn/datasets/AI-ModelScope/winogrande_val) benchmarks to assess the reasoning capabilities of models.
 - 🔥 **[2025.04.29]** Added Qwen3 Evaluation Best Practices, [welcome to read 📖](https://evalscope.readthedocs.io/en/latest/best_practice/qwen3.html)
 - 🔥 **[2025.04.27]** Support for text-to-image evaluation: Supports 8 metrics including MPS, HPSv2.1Score, etc., and evaluation benchmarks such as EvalMuse, GenAI-Bench. Refer to the [user documentation](https://evalscope.readthedocs.io/en/latest/user_guides/aigc/t2i.html) for more details.
 - 🔥 **[2025.04.10]** Model service stress testing tool now supports the `/v1/completions` endpoint (the default endpoint for vLLM benchmarking)
@@ -353,26 +369,27 @@ For more customized evaluations, such as customizing model parameters or dataset
 
 ```shell
 evalscope eval \
- --model Qwen/Qwen2.5-0.5B-Instruct \
- --model-args revision=master,precision=torch.float16,device_map=auto \
- --generation-config do_sample=true,temperature=0.5 \
+ --model Qwen/Qwen3-0.6B \
+ --model-args '{"revision": "master", "precision": "torch.float16", "device_map": "auto"}' \
+ --generation-config '{"do_sample":true,"temperature":0.6,"max_new_tokens":512,"chat_template_kwargs":{"enable_thinking": false}}' \
  --dataset-args '{"gsm8k": {"few_shot_num": 0, "few_shot_random": false}}' \
  --datasets gsm8k \
  --limit 10
 ```
 
-### Parameter
-- `--model-args`: Model loading parameters, separated by commas in `key=value` format. Default parameters:
-  - `revision`: Model version, default is `master`
-  - `precision`: Model precision, default is `auto`
-  - `device_map`: Model device allocation, default is `auto`
-- `--generation-config`: Generation parameters, separated by commas in `key=value` format. Default parameters:
-  - `do_sample`: Whether to use sampling, default is `false`
-  - `max_length`: Maximum length, default is 2048
-  - `max_new_tokens`: Maximum length of generation, default is 512
-- `--dataset-args`: Configuration parameters for evaluation datasets, passed in `json` format. The key is the dataset name, and the value is the parameters. Note that it needs to correspond one-to-one with the values in the `--datasets` parameter:
+### Parameter Description
+- `--model-args`: Model loading parameters, passed as a JSON string:
+  - `revision`: Model version
+  - `precision`: Model precision
+  - `device_map`: Device allocation for the model
+- `--generation-config`: Generation parameters, passed as a JSON string and parsed as a dictionary:
+  - `do_sample`: Whether to use sampling
+  - `temperature`: Generation temperature
+  - `max_new_tokens`: Maximum length of generated tokens
+  - `chat_template_kwargs`: Model inference template parameters
+- `--dataset-args`: Settings for the evaluation dataset, passed as a JSON string where the key is the dataset name and the value is the parameters. Note that these need to correspond one-to-one with the values in the `--datasets` parameter:
   - `few_shot_num`: Number of few-shot examples
-  - `few_shot_random`: Whether to randomly sample few-shot data, if not set, defaults to `true`
+  - `few_shot_random`: Whether to randomly sample few-shot data; if not set, defaults to `true`
 
 Reference: [Full Parameter Description](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)
 
@@ -391,6 +408,11 @@ A stress testing tool focused on large language models, which can be customized
 
 Reference: Performance Testing [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/index.html)
 
+**Output example**
+
+![multi_perf](docs/en/user_guides/stress_test/images/multi_perf.png)
+
+
 **Supports wandb for recording results**
 
 ![wandb sample](https://modelscope.oss-cn-beijing.aliyuncs.com/resource/wandb_sample.png)
@@ -439,7 +461,7 @@ EvalScope, as the official evaluation tool of [ModelScope](https://modelscope.cn
 </a>
 
 ## 🔜 Roadmap
-- [ ] Support for better evaluation report visualization
+- [x] Support for better evaluation report visualization
 - [x] Support for mixed evaluations across multiple datasets
 - [x] RAG evaluation
 - [x] VLM evaluation
@@ -449,7 +471,7 @@ EvalScope, as the official evaluation tool of [ModelScope](https://modelscope.cn
 - [x] Multi-modal evaluation
 - [ ] Benchmarks
   - [ ] GAIA
-  - [ ] GPQA
+  - [x] GPQA
   - [x] MBPP
 
 

{evalscope-0.15.1 → evalscope-0.16.0}/evalscope/arguments.py

@@ -9,6 +9,15 @@ class ParseStrArgsAction(argparse.Action):
     def __call__(self, parser, namespace, values, option_string=None):
         assert isinstance(values, str), 'args should be a string.'
 
+        # try json load first
+        try:
+            arg_dict = json.loads(values)
+            setattr(namespace, self.dest, arg_dict)
+            return
+        except (json.JSONDecodeError, ValueError):
+            pass
+
+        # If JSON load fails, fall back to parsing as key=value pairs
         arg_dict = {}
         for arg in values.strip().split(','):
             key, value = map(str.strip, arg.split('=', 1))  # Use maxsplit=1 to handle multiple '='
@@ -67,6 +76,7 @@ def add_argument(parser: argparse.ArgumentParser):
     parser.add_argument('--work-dir', type=str, help='The root cache dir.')
 
     # Debug and runtime mode arguments
+    parser.add_argument('--ignore-errors', action='store_true', default=False, help='Ignore errors during evaluation.')
     parser.add_argument('--debug', action='store_true', default=False, help='Debug mode, will print information for debugging.')  # noqa: E501
     parser.add_argument('--dry-run', action='store_true', default=False, help='Dry run in single processing mode.')
     parser.add_argument('--seed', type=int, default=42, help='Random seed for reproducibility.')

{evalscope-0.15.1 → evalscope-0.16.0}/evalscope/backend/rag_eval/utils/llm.py

@@ -52,7 +52,7 @@ class LocalLLM(BaseLLM):
         """Run the LLM on the given input."""
         infer_cfg = {'stop': stop}
 
-        response, _ = self.model._model_generate([prompt], infer_cfg=infer_cfg)
+        response, _ = self.model.predict([{'data': [prompt]}], infer_cfg=infer_cfg)
         return response[0][0]
 
     @property

{evalscope-0.15.1 → evalscope-0.16.0}/evalscope/benchmarks/alpaca_eval/alpaca_eval_adapter.py

@@ -96,12 +96,6 @@ class AlpacaEvalAdapter(DataAdapter):
             return None
 
     def compute_metric(self, review_res_list: List[bool], **kwargs) -> List[dict]:
-        """
-        compute weighted mean of the bleu score of all samples
-
-        Args:
-            review_res_list: [{'is_correct': 1, 'is_incorrect': 0, 'is_not_attempted': 0}, ...]
-        """
         # zip dict answers
         res_list = [res for res in review_res_list if res is not None]
 

{evalscope-0.15.1 → evalscope-0.16.0}/evalscope/benchmarks/chinese_simple_qa/csimple_qa_adapter.py

@@ -148,6 +148,7 @@ class ChineseSimpleQAAdapter(DataAdapter):
             'is_correct': 1 if res == 'A' else 0,
             'is_incorrect': 1 if res == 'B' else 0,
             'is_not_attempted': 1 if res == 'C' else 0,
+            'judge_response': grading_response,
         }
 
     def compute_metric(self, review_res_list: List[dict], **kwargs) -> List[dict]:

{evalscope-0.15.1 → evalscope-0.16.0}/evalscope/benchmarks/data_adapter.py

@@ -258,7 +258,7 @@ class DataAdapter(ABC):
             avg_res: Dict[str, List[float]]
 
         """
-        if isinstance(review_res_list[0], list):
+        if len(review_res_list) > 0 and isinstance(review_res_list[0], list):
             review_res_list = [item for sublist in review_res_list for item in sublist]
 
         items = defaultdict(list)
@@ -322,6 +322,7 @@ class DataAdapter(ABC):
                         choices: Optional[List[str]] = None,
                         index: Optional[Union[int, str]] = None,
                         id: Optional[Union[int, str]] = None,
+                        messages: Optional[List[dict]] = None,
                         **kwargs) -> dict:
         data = [prompt] if not isinstance(prompt, list) else prompt
         prompt_data = PromptData(
@@ -329,7 +330,8 @@ class DataAdapter(ABC):
             multi_choices=choices or self.choices,
             system_prompt=system_prompt or self.system_prompt,
             index=index or 0,
-            id=id)
+            id=id,
+            messages=messages)
         return prompt_data.to_dict()
 
     def gen_prompt(self, input_d: dict, subset_name: str, few_shot_list: list, **kwargs) -> Any:

evalscope-0.16.0/evalscope/benchmarks/drop/drop_adapter.py

@@ -0,0 +1,133 @@
+import re
+from typing import List
+
+from evalscope.benchmarks import Benchmark, DataAdapter
+from evalscope.constants import EvalType
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+DROP_EXAMPLES = '''Some examples of passages and Q&A are provided below.
+
+# Examples
+---
+Passage: Trunajaya rebellion  or Trunajaya War was the ultimately unsuccessful rebellion waged by the Madurese prince Trunajaya and fighters from Makassar against the Mataram Sultanate and its Dutch East India Company  supporters in Java  during the 1670s. The rebellion was initially successful: the rebels defeated the royal army at Gegodog , captured most of the Javanese north coast, and took the Mataram capital Plered . King Amangkurat I died during the retreat of the royal court. His son and successor, Amangkurat II, requested help from the VOC in exchange for financial remuneration and geopolitical concessions. The VOC\'s subsequent involvement turned the tide of the war. VOC and Mataram forces recovered lost territories and overran Trunajaya\'s new capital at Kediri . However, the rebellion continued until the capture of Trunajaya at the end of 1679, and the defeat, death, or surrender of the other rebel leaders . Trunajaya was killed by Amangkurat II personally in 1680 while a prisoner of the VOC. After his father\'s death in 1677, Amangkurat II also faced rival claims to the throne. The most serious rival was his brother Pangeran Puger, who took the capital Plered in 1677 and did not surrender until 1681.
+Question: How many years was it between Trunajaya\'s capture and his death while prisoner of the VOC?
+Answer:  1
+
+---
+Passage: Led by former Giant Kurt Warner, the defending NFC champions took the field at Giants Stadium against a Giants team still reeling from their bad loss in New Orleans. The Giants scored first, sending Jacobs in for a 4-yard touchdown run following a Terrell Thomas interception. Later, Arizona running back Beanie Wells scored his first career touchdown on a 13-yard rush. Manning responded by throwing a 62-yard touchdown to Nicks for his longest reception of the year. In the second half, the Cardinals\' Tim Hightower and Jason Wright scored touchdowns. But it was turnovers that decided this game; Manning\'s 3 interceptions were as many as he had thrown all season. The Giants scored only 3 points in the second half, ending the game on an interception to Antrel Rolle. The Giants notable streak of 38 consecutive starts by the same offensive line unit was ended here, as offensive tackle Kareem McKenzie missed the game with a groin injury. McKenzie returned the following week.
+Question: Which player made the first score of the game?
+Answer:  Jacobs
+
+---
+Passage: Hoping to rebound from their road loss to the Bills, the Chargers flew to Wembley Stadium for the 2008 International Series game with the New Orleans Saints. In the first quarter, San Diego trailed early as kicker Taylor Mehlhaff got a 23-yard field goal.  The \'Bolts would respond with kicker Nate Kaeding getting a 33-yard field goal.  In the second quarter, New Orleans regained the lead as QB Drew Brees (a former Charger) completed a 12-yard TD pass to WR Devery Henderson (with a failed PAT) and RB Deuce McAllister getting a 1-yard TD run.  San Diego answered as QB Philip Rivers completed a 12-yard TD pass to RB LaDainian Tomlinson, but the Saints replied with Brees completing a 30-yard TD pass to WR Lance Moore.  The Chargers closed out the half with Rivers completing a 12-yard TD pass to TE Antonio Gates. In the third quarter, New Orleans increased its lead Brees completing a 1-yard TD pass to TE Mark Campbell, after a very controversial Pass interference call on cornerback Cletis Gordon put the Saints on the 1-yard line.  The \'Bolts would answer with Kaeding getting a 24-yard field goal.  In the fourth quarter, the Saints continued to build its lead as FB Mike Karney got a 1-yard TD run.  San Diego tried to rally as Kaeding nailed a 31-yard field goal, Rivers completed a 14-yard TD pass to WR Vincent Jackson, and Brees giving the \'Bolts a safety via an incomplete pass thrown into the back of his own endzone.  However, New Orleans\' defense stiffened for the win. With the loss, the Chargers went into their bye week at 3-5.
+Question: How many total yards of touchdown passes did Drew Brees make?
+Answer:  43
+
+'''  # noqa: E501
+
+
+@Benchmark.register(
+    name='drop',
+    pretty_name='DROP',
+    dataset_id='AI-ModelScope/DROP',
+    metric_list=['AverageAccuracy'],
+    few_shot_num=0,
+    train_split=None,
+    eval_split='validation',
+    prompt_template=
+    'You will be asked to read a passage and answer a question.{drop_examples}# Your Task\n\n---\n{query}\n\nThink step by step, then write a line of the form "Answer: $ANSWER" at the end of your response.',  # noqa: E501
+)
+class DROPAdapter(DataAdapter):
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+
+        few_shot_num = kwargs.get('few_shot_num', 0)
+        if few_shot_num != 0:
+            self.few_shot_num = 3
+            logger.info(f'Few shot num is set to {self.few_shot_num} for DROP dataset by system.')
+        else:
+            self.few_shot_num = 0
+
+    def gen_prompt(self, input_d: dict, subset_name: str, few_shot_list: list, **kwargs) -> dict:
+        """
+        Generate model prompt from input data.
+        """
+        drop_examples = '' if self.few_shot_num == 0 else DROP_EXAMPLES
+        query = f"Passage: {input_d['passage']}\nQuestion: {input_d['question']}"
+        prompt = self.prompt_template.format(
+            drop_examples=drop_examples,
+            query=query,
+        )
+        return self.gen_prompt_data(prompt)
+
+    def get_gold_answer(self, input_d: dict) -> List[str]:
+        """
+        Parse the raw input labels (gold).
+        """
+
+        def _flatten_validated_answers(validated_answers):
+            """Flattens a dict of lists of validated answers.
+            {"number": ['1', '8'], ...}
+            -> [{"number": ['1'], ...}, {"number": ['8'], ...}]
+            """
+            valid_answers = []
+            for i in range(len(validated_answers['number'])):
+                valid_answers.append({
+                    'number': validated_answers['number'][i],
+                    'date': validated_answers['date'][i],
+                    'spans': validated_answers['spans'][i],
+                })
+            return valid_answers
+
+        answers = []
+        answers_set = set()
+        candidates = [input_d['answer']] + _flatten_validated_answers(input_d['validated_answers'])
+        for candidate in candidates:
+            answer = DROPAdapter.parse_answer(candidate)
+            if answer in answers_set:
+                continue
+            answers_set.add(answer)
+            answers.append(answer)
+        return answers
+
+    def parse_pred_result(self, result: str, raw_input_d: dict = None, eval_type: str = EvalType.CHECKPOINT) -> str:
+        """
+        Parse the predicted result and extract proper answer.
+        """
+        match = re.search(r'(?i)Answer\s*:\s*([^\n]+)', result)
+        extracted_answer = match.group(1) if match else result
+        return extracted_answer
+
+    def match(self, gold: List[str], pred: str) -> float:
+        """
+        Match the gold answer and the predicted answer.
+        """
+        from .utils import _answer_to_bags
+
+        max_em = 0
+        for gold_answer in gold:
+            # Convert the answers to bags of answers
+            predicted_bags = _answer_to_bags(pred)
+            gold_bags = _answer_to_bags(gold_answer)
+
+            if set(predicted_bags[0]) == set(gold_bags[0]) and len(predicted_bags[0]) == len(gold_bags[0]):
+                exact_match = 1.0
+            else:
+                exact_match = 0.0
+            # Check if the answer is empty
+            if gold_answer[0].strip():
+                max_em = max(max_em, exact_match)
+
+        return max_em
+
+    @staticmethod
+    def parse_answer(answer):
+        # NOTE: Everything is returned as a tuple for uniformity and hashability.
+        if answer['number'] != '':
+            return (str(answer['number']), )
+        if answer['spans'] != []:
+            return tuple(answer['spans'])
+        return (' '.join([answer['date']['day'], answer['date']['month'], answer['date']['year']]).strip(), )

evalscope-0.16.0/evalscope/benchmarks/drop/utils.py

@@ -0,0 +1,59 @@
+import re
+import string
+
+_ARTICLES = re.compile(r'\b(a|an|the)\b', re.UNICODE)
+
+
+def _answer_to_bags(answer):
+    if isinstance(answer, (list, tuple)):
+        raw_spans = answer
+    else:
+        raw_spans = [answer]
+    normalized_spans = []
+    token_bags = []
+    for raw_span in raw_spans:
+        normalized_span = _normalize(raw_span)
+        normalized_spans.append(normalized_span)
+        token_bags.append(set(normalized_span.split()))
+    return normalized_spans, token_bags
+
+
+def _is_number(text):
+    try:
+        float(text)
+        return True
+    except ValueError:
+        return False
+
+
+def _remove_articles(text):
+    return _ARTICLES.sub(' ', text)
+
+
+def _white_space_fix(text):
+    return ' '.join(text.split())
+
+
+def _remove_punc(text):
+    exclude = set(string.punctuation)
+    if not _is_number(text):
+        return ''.join(ch for ch in text if ch not in exclude)
+    else:
+        return text
+
+
+def _fix_number(text):
+    return str(float(text)) if _is_number(text) else text
+
+
+def _tokenize(text):
+    return re.split(' |-', text)
+
+
+def _normalize(answer):
+    tokens = [
+        _white_space_fix(_remove_articles(_fix_number(_remove_punc(token.lower())))) for token in _tokenize(answer)
+    ]
+    tokens = [token for token in tokens if token.strip()]
+    normalized = ' '.join(tokens).strip()
+    return normalized

{evalscope-0.15.1 → evalscope-0.16.0}/evalscope/benchmarks/general_qa/general_qa_adapter.py

@@ -4,7 +4,7 @@ from collections import defaultdict
 from typing import List, Optional, Union
 
 from evalscope.benchmarks import Benchmark, DataAdapter
-from evalscope.metrics import bleu_ngram_one_sample, compute_rouge_score_one_sample_zh, mean
+from evalscope.metrics import mean
 from evalscope.utils.io_utils import jsonl_to_list
 from evalscope.utils.logger import get_logger
 
@@ -112,9 +112,13 @@ class GeneralQAAdapter(DataAdapter):
         """
         res = dict()
         if 'AverageRouge' in self.metric_list:
+            from evalscope.metrics.rouge_metric import compute_rouge_score_one_sample_zh
+
             rouge_dict = compute_rouge_score_one_sample_zh([pred], [gold])
             res.update(rouge_dict)
         if 'AverageBLEU' in self.metric_list:
+            from evalscope.metrics import bleu_ngram_one_sample
+
             bleu_dict = bleu_ngram_one_sample(pred, gold)
             res.update(bleu_dict)
         return res

{evalscope-0.15.1 → evalscope-0.16.0}/evalscope/benchmarks/simple_qa/simple_qa_adapter.py

@@ -148,6 +148,7 @@ class SimpleQAAdapter(DataAdapter):
             'is_correct': 1 if res == 'A' else 0,
             'is_incorrect': 1 if res == 'B' else 0,
             'is_not_attempted': 1 if res == 'C' else 0,
+            'judge_response': grading_response,
         }
 
     def compute_metric(self, review_res_list: List[dict], **kwargs) -> List[dict]:

evalscope-0.16.0/evalscope/benchmarks/tool_bench/tool_bench_adapter.py

@@ -0,0 +1,67 @@
+from typing import Dict, List
+
+from evalscope.benchmarks import Benchmark, DataAdapter
+from evalscope.constants import EvalType, OutputType
+from evalscope.metrics import Metric, mean, metric_registry
+
+
+@Benchmark.register(
+    name='tool_bench',
+    pretty_name='ToolBench-Static',
+    dataset_id='AI-ModelScope/ToolBench-Static',
+    subset_list=['in_domain', 'out_of_domain'],
+    metric_list=['Act.EM', 'Plan.EM', 'F1', 'HalluRate', 'Rouge-L'],
+    few_shot_num=0,
+    train_split=None,
+    eval_split='test',
+)
+class ToolBenchAdapter(DataAdapter):
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+
+        metric_registry.register(Metric(name='Rouge-L', object=mean))
+        metric_registry.register(Metric(name='Act.EM', object=mean))
+        metric_registry.register(Metric(name='Plan.EM', object=mean))
+        metric_registry.register(Metric(name='F1', object=mean))
+        metric_registry.register(Metric(name='HalluRate', object=mean))
+
+    def gen_prompt(self, input_d: dict, subset_name: str, few_shot_list: list, **kwargs) -> dict:
+        """
+        Generate model prompt from input data.
+        """
+        messages = input_d['messages']
+        # use prepared messages
+        return self.gen_prompt_data(prompt='', messages=messages)
+
+    def get_gold_answer(self, input_d: dict) -> str:
+        """
+        Parse the raw input labels (gold).
+        """
+        return input_d
+
+    def parse_pred_result(self, result: str, raw_input_d: dict = None, eval_type: str = EvalType.CHECKPOINT) -> str:
+        """
+        Parse the predicted result and extract proper answer.
+        """
+        return result
+
+    def match(self, gold: dict, pred: str) -> Dict:
+        """
+        Match the gold answer and the predicted answer.
+        """
+        from .utils import calculate_metrics
+
+        data = {
+            'target': gold['target'],
+            'predictions': pred,
+            'tools': gold['tools'],
+        }
+        metrics = calculate_metrics(data)
+        return metrics
+
+    def compute_metric(self, review_res_list: List[dict], **kwargs) -> Dict:
+        # aggregate review results
+        res_dict = super().compute_dict_metric(review_res_list, **kwargs)
+
+        return super().compute_metric(res_dict, **kwargs)