evalscope 0.12.1__tar.gz → 0.13.0__tar.gz

{evalscope-0.12.1/evalscope.egg-info → evalscope-0.13.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: evalscope
-Version: 0.12.1
+Version: 0.13.0
 Summary: EvalScope: Lightweight LLMs Evaluation Framework
 Home-page: https://github.com/modelscope/evalscope
 Author: ModelScope team
@@ -175,16 +175,29 @@ Requires-Dist: plotly<6.0.0,>=5.23.0; extra == "all"
 > ⭐ If you like this project, please click the "Star" button at the top right to support us. Your support is our motivation to keep going!
 
 ## 📋 Contents
-- [Introduction](#-introduction)
-- [News](#-news)
-- [Installation](#️-installation)
-- [Quick Start](#-quick-start)
+- [📋 Contents](#-contents)
+- [📝 Introduction](#-introduction)
+- [☎ User Groups](#-user-groups)
+- [🎉 News](#-news)
+- [🛠️ Installation](#️-installation)
+  - [Method 1: Install Using pip](#method-1-install-using-pip)
+  - [Method 2: Install from Source](#method-2-install-from-source)
+- [🚀 Quick Start](#-quick-start)
+  - [Method 1. Using Command Line](#method-1-using-command-line)
+  - [Method 2. Using Python Code](#method-2-using-python-code)
+  - [Basic Parameter](#basic-parameter)
+  - [Output Results](#output-results)
+- [📈 Visualization of Evaluation Results](#-visualization-of-evaluation-results)
+- [🌐 Evaluation of Specified Model API](#-evaluation-of-specified-model-api)
+- [⚙️ Custom Parameter Evaluation](#️-custom-parameter-evaluation)
+  - [Parameter](#parameter)
 - [Evaluation Backend](#evaluation-backend)
-- [Custom Dataset Evaluation](#️-custom-dataset-evaluation)
-- [Model Serving Performance Evaluation](#-model-serving-performance-evaluation)
-- [Arena Mode](#-arena-mode)
-- [Contribution](#️-contribution)
-- [Roadmap](#-roadmap)
+- [📈 Model Serving Performance Evaluation](#-model-serving-performance-evaluation)
+- [🖊️ Custom Dataset Evaluation](#️-custom-dataset-evaluation)
+- [🏟️ Arena Mode](#️-arena-mode)
+- [👷‍♂️ Contribution](#️-contribution)
+- [🔜 Roadmap](#-roadmap)
+- [Star History](#star-history)
 
 
 ## 📝 Introduction
@@ -226,6 +239,8 @@ Please scan the QR code below to join our community groups:
 
 ## 🎉 News
 
+- 🔥 **[2025.03.13]** Added support for the [LiveCodeBench](https://www.modelscope.cn/datasets/AI-ModelScope/code_generation_lite/summary) code evaluation benchmark. You can use it by specifying `live_code_bench`.
+- 🔥 **[2025.03.11]** Added support for the [SimpleQA](https://modelscope.cn/datasets/AI-ModelScope/SimpleQA/summary) and [Chinese SimpleQA](https://modelscope.cn/datasets/AI-ModelScope/Chinese-SimpleQA/summary) evaluation benchmarks. These are used to assess the factual accuracy of models, and you can specify `simple_qa` and `chinese_simpleqa` for use. Support for specifying a judge model is also available. For more details, refer to the [relevant parameter documentation](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html).
 - 🔥 **[2025.03.07]** Added support for the [QwQ-32B](https://modelscope.cn/models/Qwen/QwQ-32B/summary) model, evaluate the model's reasoning ability and reasoning efficiency, refer to [📖 Best Practices for QwQ-32B Evaluation](https://evalscope.readthedocs.io/en/latest/best_practice/eval_qwq.html) for more details.
 - 🔥 **[2025.03.04]** Added support for the [SuperGPQA](https://modelscope.cn/datasets/m-a-p/SuperGPQA/summary) dataset, which covers 13 categories, 72 first-level disciplines, and 285 second-level disciplines, totaling 26,529 questions. You can use it by specifying `super_gpqa`.
 - 🔥 **[2025.03.03]** Added support for evaluating the IQ and EQ of models. Refer to [📖 Best Practices for IQ and EQ Evaluation](https://evalscope.readthedocs.io/en/latest/best_practice/iquiz.html) to find out how smart your AI is!

{evalscope-0.12.1 → evalscope-0.13.0}/README.md

@@ -24,16 +24,29 @@
 > ⭐ If you like this project, please click the "Star" button at the top right to support us. Your support is our motivation to keep going!
 
 ## 📋 Contents
-- [Introduction](#-introduction)
-- [News](#-news)
-- [Installation](#️-installation)
-- [Quick Start](#-quick-start)
+- [📋 Contents](#-contents)
+- [📝 Introduction](#-introduction)
+- [☎ User Groups](#-user-groups)
+- [🎉 News](#-news)
+- [🛠️ Installation](#️-installation)
+  - [Method 1: Install Using pip](#method-1-install-using-pip)
+  - [Method 2: Install from Source](#method-2-install-from-source)
+- [🚀 Quick Start](#-quick-start)
+  - [Method 1. Using Command Line](#method-1-using-command-line)
+  - [Method 2. Using Python Code](#method-2-using-python-code)
+  - [Basic Parameter](#basic-parameter)
+  - [Output Results](#output-results)
+- [📈 Visualization of Evaluation Results](#-visualization-of-evaluation-results)
+- [🌐 Evaluation of Specified Model API](#-evaluation-of-specified-model-api)
+- [⚙️ Custom Parameter Evaluation](#️-custom-parameter-evaluation)
+  - [Parameter](#parameter)
 - [Evaluation Backend](#evaluation-backend)
-- [Custom Dataset Evaluation](#️-custom-dataset-evaluation)
-- [Model Serving Performance Evaluation](#-model-serving-performance-evaluation)
-- [Arena Mode](#-arena-mode)
-- [Contribution](#️-contribution)
-- [Roadmap](#-roadmap)
+- [📈 Model Serving Performance Evaluation](#-model-serving-performance-evaluation)
+- [🖊️ Custom Dataset Evaluation](#️-custom-dataset-evaluation)
+- [🏟️ Arena Mode](#️-arena-mode)
+- [👷‍♂️ Contribution](#️-contribution)
+- [🔜 Roadmap](#-roadmap)
+- [Star History](#star-history)
 
 
 ## 📝 Introduction
@@ -75,6 +88,8 @@ Please scan the QR code below to join our community groups:
 
 ## 🎉 News
 
+- 🔥 **[2025.03.13]** Added support for the [LiveCodeBench](https://www.modelscope.cn/datasets/AI-ModelScope/code_generation_lite/summary) code evaluation benchmark. You can use it by specifying `live_code_bench`.
+- 🔥 **[2025.03.11]** Added support for the [SimpleQA](https://modelscope.cn/datasets/AI-ModelScope/SimpleQA/summary) and [Chinese SimpleQA](https://modelscope.cn/datasets/AI-ModelScope/Chinese-SimpleQA/summary) evaluation benchmarks. These are used to assess the factual accuracy of models, and you can specify `simple_qa` and `chinese_simpleqa` for use. Support for specifying a judge model is also available. For more details, refer to the [relevant parameter documentation](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html).
 - 🔥 **[2025.03.07]** Added support for the [QwQ-32B](https://modelscope.cn/models/Qwen/QwQ-32B/summary) model, evaluate the model's reasoning ability and reasoning efficiency, refer to [📖 Best Practices for QwQ-32B Evaluation](https://evalscope.readthedocs.io/en/latest/best_practice/eval_qwq.html) for more details.
 - 🔥 **[2025.03.04]** Added support for the [SuperGPQA](https://modelscope.cn/datasets/m-a-p/SuperGPQA/summary) dataset, which covers 13 categories, 72 first-level disciplines, and 285 second-level disciplines, totaling 26,529 questions. You can use it by specifying `super_gpqa`.
 - 🔥 **[2025.03.03]** Added support for evaluating the IQ and EQ of models. Refer to [📖 Best Practices for IQ and EQ Evaluation](https://evalscope.readthedocs.io/en/latest/best_practice/iquiz.html) to find out how smart your AI is!

{evalscope-0.12.1 → evalscope-0.13.0}/evalscope/arguments.py

@@ -1,7 +1,7 @@
 import argparse
 import json
 
-from evalscope.constants import EvalBackend, EvalStage, EvalType, OutputType
+from evalscope.constants import EvalBackend, EvalStage, EvalType, JudgeStrategy, OutputType
 
 
 class ParseStrArgsAction(argparse.Action):
@@ -73,6 +73,11 @@ def add_argument(parser: argparse.ArgumentParser):
     parser.add_argument('--api-url', type=str, default=None, help='The API url for the remote API model.')
     parser.add_argument('--timeout', type=float, default=None, help='The timeout for the remote API model.')
     parser.add_argument('--stream', action='store_true', default=False, help='Stream mode.')  # noqa: E501
+
+    # LLMJudge arguments
+    parser.add_argument('--judge-strategy', type=str, default=JudgeStrategy.AUTO, help='The judge strategy.')
+    parser.add_argument('--judge-model-args', type=json.loads, default='{}', help='The judge model args, should be a json string.')  # noqa: E501
+    parser.add_argument('--judge-worker-num', type=int, default=8, help='The number of workers for the judge model.')
     # yapf: enable
 
 

{evalscope-0.12.1 → evalscope-0.13.0}/evalscope/benchmarks/arc/arc_adapter.py

@@ -26,7 +26,7 @@ logger = get_logger()
     train_split='train',
     eval_split='test',
     prompt_template=
-    'The following are multiple choice questions, please output correct answer in the form of A or B or C or D, do not output explanation:\n{query}',
+    'Given the following question and four candidate answers (A, B, C and D), choose the best answer.\n{query}\nYour response should end with "The best answer is [the_answer_letter]" where the [the_answer_letter] is one of A, B, C or D.',  # noqa
 )
 class ARCAdapter(DataAdapter):
 
@@ -134,7 +134,7 @@ class ARCAdapter(DataAdapter):
         if self.model_adapter == OutputType.MULTIPLE_CHOICE:
             return result
         else:
-            return ResponseParser.parse_first_capital(text=result, options=self.choices)
+            return ResponseParser.parse_first_option(text=result)
 
     def match(self, gold: str, pred: str) -> float:
         return exact_match(gold=gold, pred=pred)
@@ -149,8 +149,8 @@ class ARCAdapter(DataAdapter):
         choices_prompts: str = '\n'.join([label + '. ' + text for text, label in zip(choices_texts, choices_labels)])
         example += '\n' + choices_prompts
 
-        example += '\nAnswer:'
         if include_answer:
+            example += '\nAnswer:'
             example += ' {}\n\n'.format(input_d['answerKey'])
 
         return example

{evalscope-0.12.1 → evalscope-0.13.0}/evalscope/benchmarks/benchmark.py

@@ -1,6 +1,6 @@
 import copy
 from collections import OrderedDict
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, fields
 from typing import TYPE_CHECKING, Dict, List, Optional
 
 from evalscope.constants import OutputType
@@ -29,6 +29,7 @@ class BenchmarkMeta:
     query_template: Optional[str] = None
     pretty_name: Optional[str] = None
     filters: Optional[OrderedDict] = None
+    extra_params: Optional[Dict] = field(default_factory=dict)
 
     def _update(self, args: dict):
         if args.get('local_path'):
@@ -40,7 +41,7 @@ class BenchmarkMeta:
         return self.__dict__
 
     def to_string_dict(self) -> dict:
-        cur_dict = copy.deepcopy(self.__dict__)
+        cur_dict = copy.deepcopy(self.to_dict())
         # cur_dict['data_adapter'] = self.data_adapter.__name__
         del cur_dict['data_adapter']
         return cur_dict

{evalscope-0.12.1 → evalscope-0.13.0}/evalscope/benchmarks/ceval/ceval_adapter.py

@@ -134,7 +134,8 @@ SUBJECT_MAPPING = {
     few_shot_num=0,
     train_split='dev',
     eval_split='val',
-    prompt_template='以下是中国关于{subset_name}考试的单项选择题，请选出其中的正确答案。\n{query}',
+    prompt_template=
+    '以下是中国关于{subset_name}考试的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：“答案是：LETTER”（不带引号），其中 LETTER 是 A、B、C、D 中的一个。\n{query}',
 )
 class CEVALAdapter(DataAdapter):
 

evalscope-0.13.0/evalscope/benchmarks/chinese_simple_qa/csimple_qa_adapter.py

@@ -0,0 +1,168 @@
+import re
+from collections import defaultdict
+from typing import Any, List
+
+from evalscope.benchmarks import Benchmark, DataAdapter
+from evalscope.metrics import Metric, mean, metric_registry
+from evalscope.metrics.llm_judge import LLMJudge
+from evalscope.utils.logger import get_logger
+
+# flake8: noqa
+
+logger = get_logger()
+
+GRADER_TEMPLATE = """
+请根据给定问题、标准答案和模型预测的答案来评估模型的回答是否正确。您的任务是将结果评定为：【正确】、【错误】或【未尝试】。
+
+首先，我们将列出每个评定类别的示例，然后请您对新问题的预测答案进行评定。
+以下是【正确】的答复示例：
+```
+问题：贝拉克·奥巴马的孩子叫什么名字？
+标准答案：玛丽亚·奥巴马和萨莎·奥巴马
+模型预测1：Malia Obama and Sasha Obama
+模型预测2：玛丽亚和萨沙
+模型预测3：大多数人会说是玛丽亚和萨莎，但我不确定，需要再确认
+模型预测4：巴拉克·奥巴马有两个女儿，她们分别是玛丽亚·安和娜塔莎·玛丽安，但通常称作玛丽亚·奥巴马和萨莎·奥巴马。玛丽亚出生于1998年7月4日，萨莎出生于2001年6月10日。
+```
+这些答复均为【正确】，因为：
+    - 完整地包含了标准答案中的重要信息。
+    - 不包含任何与标准答案矛盾的信息。
+    - 只关注语义内容，中英文，大小写、标点、语法和顺序不重要。
+    - 答复中出现模糊语句或猜测是可以接受的，前提是包含了标准答案且不含有不正确信息或矛盾。
+
+以下是【错误】的答复示例：
+```
+问题：巴拉克·奥巴马的孩子叫什么名字？
+标准答案：玛丽亚·奥巴马和萨莎·奥巴马
+模型预测1：玛丽亚
+模型预测2：玛丽亚、萨莎和苏珊
+模型预测3：巴拉克·奥巴马没有孩子
+模型预测4：我认为是玛丽亚和萨莎。或者是玛丽亚和杰基。或者是乔伊和玛丽亚。
+模型预测5：虽然我不知道他们的确切名字，但能说出巴拉克·奥巴马有三个孩子。
+模型预测6：你可能是想说贝茜和奥利维亚。不过您应通过最新的参考资料确认详细信息。那是正确的答案吗？
+```
+这些答复均为【错误】，因为：
+    - 答复中包含与标准答案矛盾的事实陈述。即使在陈述中略带保留（例如：“可能是”，“虽然我不确定，但我认为”），也视为错误。
+
+以下是【未尝试】的答复示例：
+```
+问题：巴拉克·奥巴马的孩子叫什么名字？
+标准答案：玛丽亚·奥巴马和萨莎·奥巴马
+模型预测1：我不知道。
+模型预测2：我需要更多关于您所指奥巴马的上下文。
+模型预测3：不查阅网络我无法回答这个问题，不过我知道巴拉克·奥巴马有两个孩子。
+模型预测4：巴拉克·奥巴马有两个孩子。我知道其中一个叫玛丽亚，但我不确定另一个的名字。
+```
+这些答复均为【未尝试】，因为：
+    - 没有包含标准答案中的重要信息。
+    - 回复中没有与标准答案矛盾的陈述。
+
+另外注意以下几点：
+- 对于标准答案为数字的问题，预测答案应和标准答案一致。例如，考虑问题“金山铁路黄浦江特大桥的全长是多少米？”，标准答案为“3518.17”：
+    - 预测答案“3518”、“3518.1”、“3518.17”均为【正确】。
+    - 预测答案“3520”和“3600”均为【错误】。
+    - 预测答案“大约3500米”和“超过3000米”被视为【未尝试】，因为它们既不确认也不与标准答案矛盾。
+- 如果标准答案包含比问题更多的信息，预测答案只需包含问题中提到的信息。
+    - 例如，考虑问题“菱镁矿的主要化学成分是什么？”标准答案为“碳酸镁（MgCO3）”。“碳酸镁”或“MgCO3”均视为【正确】答案。
+- 如果从问题中明显可以推断出预测答案省略的信息，那么算作正确。
+    - 例如，问题“巴鲁米尼的努拉吉遗迹在1997年被联合国教科文组织列为世界文化遗产，那么这遗址在哪个地区？”标准答案为“意大利撒丁岛”，预测答案“撒丁岛”被视为【正确】。
+- 如果能明显看出名字翻译版本不同但是是同一个人也认为正确。
+    - 例如，如果标准答案是“Robinson”，那么回答鲁滨逊或者鲁滨孙均正确。
+
+下面是一个新的问题示例。请只回复A、B、C之一，不要道歉或纠正自己的错误，只需要评估该回答。
+```
+问题: {question}
+正确答案: {target}
+预测答案: {predicted_answer}
+```
+
+将此新问题的预测答案评定为以下之一：
+A:【正确】
+B:【错误】
+C:【未尝试】
+
+只返回字母"A"、"B"或"C"，无须添加其他文本。
+""".strip()  # noqa E501
+
+SUBSET_LIST = ['中华文化', '人文与社会科学', '工程、技术与应用科学', '生活、艺术与文化', '社会', '自然与自然科学']
+
+
+@Benchmark.register(
+    name='chinese_simpleqa',
+    pretty_name='Chinese SimpleQA',
+    subset_list=SUBSET_LIST,
+    dataset_id='AI-ModelScope/Chinese-SimpleQA',
+    metric_list=['is_correct', 'is_incorrect', 'is_not_attempted'],
+    few_shot_num=0,
+    train_split=None,
+    eval_split='train')
+class ChineseSimpleQAAdapter(DataAdapter):
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+        # register metrics
+        metric_registry.register(Metric(name='is_correct', object=mean))
+        metric_registry.register(Metric(name='is_incorrect', object=mean))
+        metric_registry.register(Metric(name='is_not_attempted', object=mean))
+
+        # whether to use LLM as a judge
+        self.llm_as_a_judge = True
+
+    def load(self, **kwargs):
+        kwargs['subset_list'] = ['default']
+        data_dict = super().load(**kwargs)
+        return self.reformat_subset(data_dict, subset_key='primary_category', format='{}')
+
+    def gen_prompt(self, input_d: dict, subset_name: str, few_shot_list: list, **kwargs) -> dict:
+        question = input_d['question']
+        return self.gen_prompt_data(question)
+
+    def get_gold_answer(self, input_d: dict) -> str:
+        return input_d['answer']
+
+    def parse_pred_result(self, result: str, raw_input_d: dict = None, **kwargs) -> str:
+        return result.strip()
+
+    def match(self, gold: str, pred: str) -> float:
+        # simple match
+        logger.warning(f'Please use LLMJudge to match the result for ChineseSimpleQA')
+        is_correct = 1 if gold.lower().strip() == pred.lower().strip() else 0
+        is_incorrect = not is_correct
+        is_not_attempted = 0
+        return {
+            'is_correct': is_correct,
+            'is_incorrect': is_incorrect,
+            'is_not_attempted': is_not_attempted,
+        }
+
+    def llm_match(self, gold: Any, pred: Any, judge: LLMJudge, **kwargs) -> dict:
+        raw_input = kwargs.get('raw_input', None)
+        question = raw_input['question']
+        # get grading response
+        prompt = GRADER_TEMPLATE.format(question=question, target=gold, predicted_answer=pred)
+        system_prompt = '你是一个智能助手，请根据给定问题、标准答案和模型预测的答案来评估模型的回答是否正确。'
+        grading_response = judge(prompt, system_prompt)
+        # parse grading response
+        match = re.search(r'(A|B|C)', grading_response)
+        res = match.group(0) if match else 'C'
+        return {
+            'is_correct': 1 if res == 'A' else 0,
+            'is_incorrect': 1 if res == 'B' else 0,
+            'is_not_attempted': 1 if res == 'C' else 0,
+        }
+
+    def compute_metric(self, review_res_list: List[dict], **kwargs) -> List[dict]:
+        """
+        compute weighted mean of the bleu score of all samples
+
+        Args:
+            review_res_list: [{'is_correct': 1, 'is_incorrect': 0, 'is_not_attempted': 0}, ...]
+        """
+        # zip dict answers
+        res_dict = defaultdict(list)
+        for res in review_res_list:
+            for key, value in res.items():
+                res_dict[key].append(value)
+
+        return super().compute_metric(res_dict, **kwargs)

{evalscope-0.12.1 → evalscope-0.13.0}/evalscope/benchmarks/cmmlu/cmmlu_adapter.py

@@ -111,7 +111,8 @@ SUBJECT_MAPPING = {
     few_shot_num=5,
     train_split='dev',
     eval_split='test',
-    prompt_template='以下是关于{subset_name}的单项选择题，请直接给出正确答案的选项。\n{query}',
+    prompt_template=
+    '以下是关于{subset_name}的单项选择题，请给出正确答案的选项。你的回答的最后一行应该是这样的格式：“答案：LETTER”（不带引号），其中 LETTER 是 A、B、C、D 中的一个。\n{query}',
 )
 class CMMLUAdapter(DataAdapter):
 

{evalscope-0.12.1 → evalscope-0.13.0}/evalscope/benchmarks/data_adapter.py

@@ -7,6 +7,7 @@ from typing import Any, List, Optional, Union
 
 from evalscope.benchmarks.utils import PromptData, preprocess_decorator
 from evalscope.constants import DEFAULT_DATASET_CACHE_DIR, AnswerKeys, EvalType, HubType
+from evalscope.metrics.llm_judge import LLMJudge
 from evalscope.metrics.named_metrics import metric_registry
 from evalscope.report import Report, ReportGenerator
 from evalscope.utils.logger import get_logger
@@ -22,6 +23,7 @@ class DataAdapter(ABC):
                  model_adapter: str,
                  subset_list: list,
                  metric_list: List[str],
+                 llm_as_a_judge: bool = False,
                  few_shot_num: Optional[int] = 0,
                  train_split: Optional[str] = None,
                  eval_split: Optional[str] = None,
@@ -61,6 +63,7 @@ class DataAdapter(ABC):
         self.query_template = query_template
         self.pretty_name = pretty_name
         self.config_kwargs = kwargs
+        self.llm_as_a_judge = llm_as_a_judge
         self.category_map = kwargs.get('category_map', {})
         self.choices = kwargs.get('choices', None)
 
@@ -91,12 +94,14 @@ class DataAdapter(ABC):
         # Try to load dataset from local disk
         if os.path.exists(dataset_name_or_path):
             logger.info(f'Loading dataset from local disk: {dataset_name_or_path}')
+            trust_remote_code = kwargs.pop('trust_remote_code', False)
             data_dict = self.load_from_disk(
-                dataset_name_or_path, subset_list, work_dir, trust_remote_code=False, **kwargs)
+                dataset_name_or_path, subset_list, work_dir, trust_remote_code=trust_remote_code, **kwargs)
         else:
             logger.info(f'Loading dataset from hub: {dataset_name_or_path}')
+            trust_remote_code = kwargs.pop('trust_remote_code', True)
             data_dict = self.load_from_hub(
-                dataset_name_or_path, subset_list, work_dir, trust_remote_code=True, **kwargs)
+                dataset_name_or_path, subset_list, work_dir, trust_remote_code=trust_remote_code, **kwargs)
         if len(data_dict) == 0:
             raise ValueError(f'Dataset is empty: {dataset_name_or_path}')
         return data_dict
@@ -356,5 +361,28 @@ class DataAdapter(ABC):
         """
         raise NotImplementedError
 
-    def llm_match(self, *args, **kwargs):
-        pass
+    def llm_match(self, gold: Any, pred: Any, judge: Optional[LLMJudge] = None, **kwargs) -> float:
+        """
+        Use LLM as a judge to evaluate the predicted answer against the gold answer.
+
+        Args:
+            gold (Any): The golden answer.
+            pred (Any): The predicted answer.
+
+        Returns:
+            The match result as a float score between 0 and 1.
+        """
+        # Default judge handling
+        if judge is None:
+            logger.warning('No judge LLM provided, please specify a judge LLM in the config.')
+            return 0
+
+        # Extract question from raw_input if available
+        raw_input = kwargs.get('raw_input', {})
+        question_keys = ['question', 'prompt', 'query', 'problem']
+        question = next((raw_input.get(key) for key in question_keys if raw_input.get(key)), None)
+
+        # Request judge and obtain score
+        prompt = judge.build_prompt(pred, gold, question)
+        score = judge(prompt)
+        return judge.get_score(score)

{evalscope-0.12.1 → evalscope-0.13.0}/evalscope/benchmarks/general_qa/general_qa_adapter.py

@@ -1,12 +1,10 @@
 # Copyright (c) Alibaba, Inc. and its affiliates.
-import glob
 import os.path
 from collections import defaultdict
 from typing import List
 
 from evalscope.benchmarks import Benchmark, DataAdapter
 from evalscope.metrics import bleu_ngram_one_sample, compute_rouge_score_one_sample_zh, mean
-from evalscope.models import ChatGenerationModelAdapter
 from evalscope.utils.io_utils import jsonl_to_list
 from evalscope.utils.logger import get_logger
 
@@ -133,7 +131,10 @@ class GeneralQAAdapter(DataAdapter):
         """
         items = defaultdict(list)
         for scores in review_res_list:
-            for k, v in scores.items():
-                items[k].append(v)
+            if isinstance(scores, dict):
+                for k, v in scores.items():
+                    items[k].append(v)
+            else:
+                items['AverageAccuracy'].append(scores)
         # items = [(score, 1.0) for score in review_res_list]
         return [{'metric_name': k, 'score': mean(v), 'num': len(v)} for k, v in items.items()]

{evalscope-0.12.1 → evalscope-0.13.0}/evalscope/benchmarks/hellaswag/hellaswag_adapter.py

@@ -19,15 +19,14 @@ logger = get_logger()
     name='hellaswag',
     pretty_name='HellaSwag',
     dataset_id='modelscope/hellaswag',
-    model_adapter=OutputType.CONTINUOUS,
-    output_types=[OutputType.CONTINUOUS, OutputType.GENERATION],
+    model_adapter=OutputType.MULTIPLE_CHOICE,
+    output_types=[OutputType.MULTIPLE_CHOICE, OutputType.GENERATION],
     subset_list=['default'],
     metric_list=['AverageAccuracy'],
     few_shot_num=0,
     train_split='train',
     eval_split='validation',
-    prompt_template=
-    'Respond with the index of sentence that makes the most sense, chose from 0, 1, 2, 3, derive your final answer as `The answer is ...`.',  # noqa: E501
+    prompt_template='{query}',  # noqa: E501
 )
 class HellaSwagAdapter(DataAdapter):
 
@@ -39,7 +38,7 @@ class HellaSwagAdapter(DataAdapter):
             kwargs['few_shot_num'] = 0
 
         super().__init__(**kwargs)
-        self.choices = ['0', '1', '2', '3']
+        self.choices = ['A', 'B', 'C', 'D']
 
     def load_from_disk(self, dataset_name_or_path, subset_list, work_dir, **kwargs) -> dict:
         data_dict = {}
@@ -85,15 +84,14 @@ class HellaSwagAdapter(DataAdapter):
             self._generate_prompt(input_d=sample, endings=endings, include_answer=True) for sample in few_shot_list
         ]
         context: str = '\n'.join(few_shot_prompts) + '\n'
-        context += self._generate_prompt(input_d=input_d, endings=endings, include_answer=False)
+        query = context.strip() + self._generate_prompt(input_d=input_d, endings=endings, include_answer=False)
 
-        ctx_continuation_pair_list = [(context.strip(), ' ' + cont.strip()) for cont in endings]
-
-        return self.gen_prompt_data(ctx_continuation_pair_list)
+        full_prompt = self.prompt_template.format(query=query)
+        return self.gen_prompt_data(full_prompt)
 
     def get_gold_answer(self, input_d: dict) -> str:
-        # Get the gold choice
-        return input_d['label']
+        # Get the gold choice from the label
+        return self.choices[int(input_d['label'])]
 
     def parse_pred_result(self, result: list, raw_input_d: dict = None, eval_type: str = EvalType.CHECKPOINT) -> str:
         """
@@ -107,30 +105,22 @@ class HellaSwagAdapter(DataAdapter):
         Returns:
             The parsed answer. Depending on the dataset. Usually a string for chat.
         """
-        if self.model_adapter == OutputType.CONTINUOUS:
-            # answer: in the form of [-2.3, -4.5, ...], len of self.choices
-            result = np.array(result)
-            endings: list = [self._preprocess(ending) for ending in raw_input_d['endings']]
-            completion_len = np.array([float(len(i)) for i in endings])
-            best_choice_idx = np.argmax(result / completion_len)
-
-            return str(best_choice_idx)
+        if self.model_adapter == OutputType.MULTIPLE_CHOICE:
+            return result
         else:
             return ResponseParser.parse_first_option(result)
 
     def match(self, gold: str, pred: str) -> float:
         return exact_match(gold=str(gold), pred=str(pred))
 
-    @classmethod
-    def _preprocess(cls, text):
+    def _preprocess(self, text):
         text = text.strip()
         text = text.replace(' [title]', '. ')
         text = re.sub('\\[.*?\\]', '', text)
         text = text.replace('  ', ' ')
         return text
 
-    @classmethod
-    def _generate_prompt(cls, input_d: dict, endings: list, include_answer=True) -> str:
+    def _generate_prompt(self, input_d: dict, endings: list, include_answer=True) -> str:
         """
         Generate prompt for HellaSwag dataset.
 
@@ -144,7 +134,13 @@ class HellaSwagAdapter(DataAdapter):
         """
 
         ctx = input_d['ctx_a'] + ' ' + input_d['ctx_b'].capitalize()
-        example: str = cls._preprocess(input_d['activity_label'] + ': ' + ctx)
+        # example: str = cls._preprocess(input_d['activity_label'] + ': ' + ctx)
+        example: str = self._preprocess(ctx)
+
+        example += '\nQuestion: Which ending makes the most sense?'
+        for i, ending in enumerate(endings):
+            example += f'\n{self.choices[i]}. {ending}'
+        example += '\nYou may choose from A, B, C, D. Derive your final answer as `The answer is ...`.'
 
         if include_answer:
             example += '{}\n\n'.format(endings[int(input_d['label'])])

{evalscope-0.12.1 → evalscope-0.13.0}/evalscope/benchmarks/humaneval/humaneval_adapter.py

@@ -2,7 +2,6 @@
 import re
 
 from evalscope.benchmarks import Benchmark, DataAdapter
-from evalscope.models import ChatGenerationModelAdapter
 from evalscope.utils.logger import get_logger
 
 logger = get_logger()
@@ -21,6 +20,10 @@ logger = get_logger()
     train_split=None,
     eval_split='test',
     prompt_template='Complete the following python code:\n{query}',
+    extra_params={
+        'num_workers': 4,
+        'timeout': 4
+    },
 )
 class HumanevalAdapter(DataAdapter):
     """
@@ -35,17 +38,17 @@ class HumanevalAdapter(DataAdapter):
             raise ImportError('Please install human_eval:'
                               'https://github.com/openai/human-eval/tree/master#installation , '
                               'Note that you need to enable the execution code in the human_eval/execution.py first.')
+        super().__init__(**kwargs)
 
+        extra_params = kwargs.get('extra_params', {})
         self.k = [1]
-        self.num_workers = 4
-        self.timeout = 4.0
+        self.num_workers = extra_params.get('num_workers', 4)
+        self.timeout = extra_params.get('timeout', 4)
 
         self.read_problems_func = stream_jsonl
         self.write_jsonl_func = write_jsonl
         self.eval_func = check_correctness
 
-        super().__init__(**kwargs)
-
     def load_from_disk(self, dataset_name_or_path, subset_list, work_dir, **kwargs) -> dict:
         data_dict = {}
         for subset_name in subset_list: