evalscope 0.12.0__py3-none-any.whl → 0.12.1__py3-none-any.whl

evalscope/benchmarks/super_gpqa/five_shot_prompt.txt

@@ -0,0 +1,89 @@
+Answer the following multiple choice question. There is only one correct answer. The last line of your response should be in the format 'Answer: $LETTER' (without quotes), where LETTER is one of A, B, C, D, E, F, G, H, I, or J.
+
+Question:
+A refracting telescope consists of two converging lenses separated by 100 cm. The eye-piece lens has a focal length of 20 cm. The angular magnification of the telescope is
+A) 10
+B) 40
+C) 6
+D) 25
+E) 15
+F) 50
+G) 30
+H) 4
+I) 5
+J) 20
+
+Answer: Let's think step by step. In a refracting telescope, if both lenses are converging, the focus of both lenses must be between the two lenses, and thus the focal lengths of the two lenses must add up to their separation. Since the focal length of one lens is 20 cm, the focal length of the other must be 80 cm. The magnification is the ratio of these two focal lengths, or 4.
+Answer: H.
+
+Question:
+Say the pupil of your eye has a diameter of 5 mm and you have a telescope with an aperture of 50 cm. How much more light can the telescope gather than your eye?
+A) 1000 times more
+B) 50 times more
+C) 5000 times more
+D) 500 times more
+E) 10000 times more
+F) 20000 times more
+G) 2000 times more
+H) 100 times more
+I) 10 times more
+J) N/A
+
+Answer: Let's think step by step. The amount of light a telescope can gather compared to the human eye is proportional to the area of its apertures. The area of a circle is given by the formula $A = \pi \left(\frac{{D}}{{2}}\right)^2$, where $D$ is the diameter. Therefore, the relative light-gathering power is calculated as:
+\[
+\frac{{\left(\frac{{50 \text{{ cm}}}}{{2}}\right)^2}}{{\left(\frac{{5 \text{{ mm}}}}{{2}}\right)^2}} = \frac{{\left(\frac{{50 \text{{ cm}}}}{{0.1 \text{{ cm}}}}\right)^2}}{{\left(\frac{{5 \text{{ mm}}}}{{0.1 \text{{ cm}}}}\right)^2}} = \frac{{500^2}}{{5^2}} = 10000.
+\]
+Answer: E.
+
+Question:
+Where do most short-period comets come from and how do we know?
+A) The Kuiper belt; short period comets tend to be in the plane of the solar system like the Kuiper belt.
+B) The asteroid belt; short period comets tend to come from random directions indicating a spherical distribution of comets called the asteroid belt.
+C) The asteroid belt; short period comets tend to be in the plane of the solar system just like the asteroid belt.
+D) The Oort cloud; short period comets have orbital periods similar to asteroids like Vesta and are found in the plane of the solar system just like the Oort cloud.
+E) The Oort Cloud; short period comets tend to come from random directions indicating a spherical distribution of comets called the Oort Cloud.
+F) The Oort cloud; short period comets tend to be in the plane of the solar system just like the Oort cloud.
+G) The asteroid belt; short period comets have orbital periods similar to asteroids like Vesta and are found in the plane of the solar system just like the asteroid belt.
+Answer: Let's think step by step. Most short-period comets originate from the Kuiper belt. This is deduced from the observation that these comets tend to follow orbits that lie in the plane of the solar system, similar to the distribution of objects in the Kuiper belt itself. Thus, the alignment of these cometary orbits with the ecliptic plane points to their Kuiper belt origin.
+Answer: A.
+
+Question:
+Colors in a soap bubble result from light
+A) dispersion
+B) deflection
+C) refraction
+D) reflection
+E) interference
+F) converted to a different frequency
+G) polarization
+H) absorption
+I) diffraction
+J) transmission
+
+Answer: Let's think step by step. The colorful patterns observed in a soap bubble are caused by the phenomenon of light interference. This occurs when light waves bounce between the two surfaces of the soap film, combining constructively or destructively based on their phase differences and the varying thickness of the film. These interactions result in vibrant color patterns due to variations in the intensity of different wavelengths of light.
+Answer: E.
+
+Question:
+A microwave oven is connected to an outlet, 120 V, and draws a current of 2 amps. At what rate is energy being used by the microwave oven?
+A) 240 W
+B) 120 W
+C) 10 W
+D) 480 W
+E) 360 W
+F) 200 W
+G) 30 W
+H) 150 W
+I) 60 W
+J) 300 W
+
+Answer: Let's think step by step. The rate of energy usage, known as power, in an electrical circuit is calculated by the product of voltage and current. For a microwave oven connected to a 120 V outlet and drawing a current of 2 amps, the power consumption can be calculated as follows:
+\[
+\text{{Power}} = \text{{Voltage}} \times \text{{Current}} = 120 \, \text{{V}} \times 2 \, \text{{A}} = 240 \, \text{{W}}.
+\]
+Therefore, the microwave oven uses energy at a rate of 240 watts.
+Answer: A.
+
+Question:
+{query}
+
+Answer: Let's think step by step.

evalscope/benchmarks/super_gpqa/super_gpqa_adapter.py

@@ -0,0 +1,191 @@
+import os
+import random
+import re
+
+from evalscope.benchmarks import Benchmark, DataAdapter
+from evalscope.constants import EvalType, OutputType
+from evalscope.metrics import exact_match
+from evalscope.utils import logger
+
+current_dir = os.path.dirname(os.path.abspath(__file__))
+
+SUBSET_LIST = [
+    'Electronic Science and Technology', 'Philosophy', 'Traditional Chinese Medicine', 'Applied Economics',
+    'Mathematics', 'Physics', 'Clinical Medicine', 'Computer Science and Technology',
+    'Information and Communication Engineering', 'Control Science and Engineering', 'Theoretical Economics', 'Law',
+    'History', 'Basic Medicine', 'Education', 'Materials Science and Engineering', 'Electrical Engineering',
+    'Systems Science', 'Power Engineering and Engineering Thermophysics', 'Military Science', 'Biology',
+    'Business Administration', 'Language and Literature', 'Public Health and Preventive Medicine', 'Political Science',
+    'Chemistry', 'Hydraulic Engineering', 'Chemical Engineering and Technology', 'Pharmacy', 'Geography', 'Art Studies',
+    'Architecture', 'Forestry Engineering', 'Public Administration', 'Oceanography', 'Journalism and Communication',
+    'Nuclear Science and Technology', 'Weapon Science and Technology', 'Naval Architecture and Ocean Engineering',
+    'Environmental Science and Engineering', 'Transportation Engineering', 'Geology', 'Physical Oceanography',
+    'Musicology', 'Stomatology', 'Aquaculture', 'Mechanical Engineering',
+    'Aeronautical and Astronautical Science and Technology', 'Civil Engineering', 'Mechanics',
+    'Petroleum and Natural Gas Engineering', 'Sociology', 'Food Science and Engineering', 'Agricultural Engineering',
+    'Surveying and Mapping Science and Technology', 'Metallurgical Engineering',
+    'Library, Information and Archival Management', 'Mining Engineering', 'Astronomy',
+    'Geological Resources and Geological Engineering', 'Atmospheric Science', 'Optical Engineering', 'Animal Husbandry',
+    'Geophysics', 'Crop Science', 'Management Science and Engineering', 'Psychology', 'Forestry',
+    'Textile Science and Engineering', 'Veterinary Medicine', 'Instrument Science and Technology', 'Physical Education'
+]
+
+SUBSET_MAPPING = {
+    'Electronic Science and Technology': ['Engineering'],
+    'Philosophy': ['Philosophy'],
+    'Traditional Chinese Medicine': ['Medicine'],
+    'Applied Economics': ['Economics'],
+    'Mathematics': ['Science'],
+    'Physics': ['Science'],
+    'Clinical Medicine': ['Medicine'],
+    'Computer Science and Technology': ['Engineering'],
+    'Information and Communication Engineering': ['Engineering'],
+    'Control Science and Engineering': ['Engineering'],
+    'Theoretical Economics': ['Economics'],
+    'Law': ['Law'],
+    'History': ['History'],
+    'Basic Medicine': ['Medicine'],
+    'Education': ['Education'],
+    'Materials Science and Engineering': ['Engineering'],
+    'Electrical Engineering': ['Engineering'],
+    'Systems Science': ['Science'],
+    'Power Engineering and Engineering Thermophysics': ['Engineering'],
+    'Military Science': ['Military Science'],
+    'Biology': ['Science'],
+    'Business Administration': ['Management'],
+    'Language and Literature': ['Literature and Arts'],
+    'Public Health and Preventive Medicine': ['Medicine'],
+    'Political Science': ['Law'],
+    'Chemistry': ['Science'],
+    'Hydraulic Engineering': ['Engineering'],
+    'Chemical Engineering and Technology': ['Engineering'],
+    'Pharmacy': ['Medicine'],
+    'Geography': ['Science'],
+    'Art Studies': ['Literature and Arts'],
+    'Architecture': ['Engineering'],
+    'Forestry Engineering': ['Engineering'],
+    'Public Administration': ['Management'],
+    'Oceanography': ['Science'],
+    'Journalism and Communication': ['Literature and Arts'],
+    'Nuclear Science and Technology': ['Engineering'],
+    'Weapon Science and Technology': ['Engineering'],
+    'Naval Architecture and Ocean Engineering': ['Engineering'],
+    'Environmental Science and Engineering': ['Engineering'],
+    'Transportation Engineering': ['Engineering'],
+    'Geology': ['Science'],
+    'Physical Oceanography': ['Science'],
+    'Musicology': ['Literature and Arts'],
+    'Stomatology': ['Medicine'],
+    'Aquaculture': ['Agronomy'],
+    'Mechanical Engineering': ['Engineering'],
+    'Aeronautical and Astronautical Science and Technology': ['Engineering'],
+    'Civil Engineering': ['Engineering'],
+    'Mechanics': ['Engineering'],
+    'Petroleum and Natural Gas Engineering': ['Engineering'],
+    'Sociology': ['Sociology'],
+    'Food Science and Engineering': ['Engineering'],
+    'Agricultural Engineering': ['Engineering'],
+    'Surveying and Mapping Science and Technology': ['Engineering'],
+    'Metallurgical Engineering': ['Engineering'],
+    'Library, Information and Archival Management': ['Management'],
+    'Mining Engineering': ['Engineering'],
+    'Astronomy': ['Science'],
+    'Geological Resources and Geological Engineering': ['Engineering'],
+    'Atmospheric Science': ['Science'],
+    'Optical Engineering': ['Engineering'],
+    'Animal Husbandry': ['Agronomy'],
+    'Geophysics': ['Science'],
+    'Crop Science': ['Agronomy'],
+    'Management Science and Engineering': ['Management'],
+    'Psychology': ['Education'],
+    'Forestry': ['Agronomy'],
+    'Textile Science and Engineering': ['Engineering'],
+    'Veterinary Medicine': ['Agronomy'],
+    'Instrument Science and Technology': ['Engineering'],
+    'Physical Education': ['Education']
+}
+
+
+@Benchmark.register(
+    name='super_gpqa',
+    pretty_name='SuperGPQA',
+    dataset_id='m-a-p/SuperGPQA',
+    model_adapter=OutputType.GENERATION,
+    output_types=[OutputType.MULTIPLE_CHOICE, OutputType.GENERATION],
+    subset_list=SUBSET_LIST,
+    metric_list=['AverageAccuracy'],
+    few_shot_num=0,
+    train_split=None,
+    eval_split='train',  # only have train split
+)
+class SuperGPQAAdapter(DataAdapter):
+
+    def __init__(self, **kwargs):
+        few_shot_num = kwargs.get('few_shot_num', 0)
+        if few_shot_num > 0 and few_shot_num != 5:
+            logger.warning(
+                f'Only support few_shot_num 0 or 5 for SuperGPQA, but got {few_shot_num}. Use 5-shot by default.')
+            kwargs['few_shot_num'] = 5
+        super().__init__(**kwargs)
+
+        self.choices = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']
+        self.category_map = SUBSET_MAPPING
+        self.few_shot_prompt = open(os.path.join(current_dir, 'five_shot_prompt.txt'), encoding='utf-8').read()
+        self.zero_shot_prompt = open(os.path.join(current_dir, 'zero_shot_prompt.txt'), encoding='utf-8').read()
+
+    def load(self, **kwargs):
+        kwargs['subset_list'] = ['default']
+        data_dict = super().load(**kwargs)
+        return self.reformat_subset(data_dict, subset_key='field', format='{}')
+
+    def gen_prompt(self, input_d: dict, subset_name: str, few_shot_list: list, **kwargs) -> dict:
+        if not self.prompt_template:
+            if few_shot_list:
+                prompt = self.few_shot_prompt.format(query=input_d['question'])
+            else:
+                prompt = self.zero_shot_prompt.format(query=input_d['question'])
+        else:
+            prompt = self.prompt_template.format(query=input_d['question'])
+        return self.gen_prompt_data(prompt)
+
+    def get_gold_answer(self, input_d: dict) -> str:
+        # Get the gold choice
+        return input_d.get('answer_letter')
+
+    def parse_pred_result(self, result: str, raw_input_d: dict = None, eval_type: str = EvalType.CHECKPOINT) -> str:
+        """
+        Parse the model output to get the answer. Could be the best choice index.
+
+        Args:
+            result: Predicted answer from the model. Usually a string for chat.
+            raw_input_d: The raw input. Depending on the dataset.
+            eval_type: 'checkpoint' or 'service' or 'custom'
+
+        Returns:
+            The parsed answer. Depending on the dataset. Usually a string for chat.
+        """
+        if self.model_adapter == OutputType.MULTIPLE_CHOICE:
+            return result
+        else:
+            from evalscope.benchmarks.super_gpqa.utils import extract_option_content, extract_option_labels
+            sample = raw_input_d
+            if self.few_shot_num == 0:
+                predict = extract_option_labels(result, 'ABCDEFGHIJ')
+                if predict is None:
+                    predict = extract_option_content(result, sample['options'])
+                    predict = chr(sample['options'].index(predict) + 65) if predict else None
+            else:
+                response = result.split('Question:')[0]
+                predict = extract_option_labels(response, 'ABCDEFGHIJ')
+                if predict is None:
+                    predict = extract_option_content(response, sample['options'])
+                    predict = chr(sample['options'].index(predict) + 65) if predict else None
+                if predict is None:
+                    predict = extract_option_labels(result, 'ABCDEFGHIJ')
+                    if predict is None:
+                        predict = extract_option_content(result, sample['options'])
+                        predict = chr(sample['options'].index(predict) + 65) if predict else None
+            return predict
+
+    def match(self, gold: str, pred: str) -> float:
+        return exact_match(gold=gold, pred=pred)

evalscope/benchmarks/super_gpqa/utils.py

@@ -0,0 +1,90 @@
+# flake8: noqa
+import re
+import timeout_decorator
+
+
+@timeout_decorator.timeout(5)  # 5 seconds timeout
+def safe_regex_search(pattern, text, flags=0):
+    try:
+        return re.search(pattern, text, flags)
+    except timeout_decorator.TimeoutError:
+        print(f'Regex match timeout: pattern={pattern}, text={text[:100]}...')
+        return None
+    except Exception as e:
+        print(f'Regex match error: {str(e)}')
+        return None
+
+
+def extract_option_labels(text, options='ABCDEFGHIJ'):
+    if not isinstance(text, str) or not isinstance(options, str):
+        return 'error'
+
+    text = text.rstrip()
+    last_line = text.split('\n')[-1]
+
+    option_str = ''.join([chr(65 + i) for i in range(len(options))]) if options else 'ABCDEFGHIJ'
+
+    patterns = [
+        # e.g. "The final answer to this question is: A."
+        #      "The best option is $\boxed{B}:"
+        #      "The correct answer is (C)."
+        f'[Tt]he\s+(?:\w+\s+)?(?:answer|option)(?:\w+\s+)?\s+is?:?\s*(?:[\*\$\\{{(\[\\\\(]*?(?:(?:\\\\boxed|\\\\mathbf|\\\\mathrm|\\\\text){{)?)*\s*([{option_str}])(?:\\\\?\}}?\$?\)?\]?\}}?)*(?:[\s:\.\*)]|$)',
+
+        # e.g. "ANSWER: A"
+        #      "Answer: $\boxed{B}."
+        #      "ANSWER: (C):"
+        f'(?i:Answer)[\*\s]*:\s*(?:[\*\$\\{{(\[\\\\(]*?(?:(?:\\\\boxed|\\\\mathbf|\\\\mathrm|\\\\text){{)?)*\s*([{option_str}])(?:\\\\?\}}?\$?\)?\]?\}}?)*(?:[\s:\.\*)]|$)',
+
+        # e.g. "A"
+        #      "$\boxed{B}$"
+        #      "(C)."
+        #      "[D]:"
+        f'^[^\w\r\n]*(?:[\*\$\\{{(\[\\\\(]*?(?:(?:\\\\boxed|\\\\mathbf|\\\\mathrm|\\\\text){{)?)*\s*([{option_str}])(?:\\\\?\}}?\$?\)?\]?\}}?)*(?:[\s:\.\*)]|$)',
+    ]
+
+    for pattern in patterns:
+        match = safe_regex_search(pattern, last_line, re.IGNORECASE)
+        if match:
+            return match.group(1)
+
+    for pattern in patterns:
+        match = safe_regex_search(pattern, text, re.IGNORECASE)
+        if match:
+            return match.group(1)
+
+    return None
+
+
+def extract_option_content(text, options_content=None):
+    if not isinstance(text, str) or not isinstance(options_content, list):
+        return 'error'
+
+    escaped_options_content = [re.escape(option_content) for option_content in options_content]
+    escaped_options_content_str = '|'.join(escaped_options_content)
+
+    text = text.rstrip()
+    last_line = text.split('\n')[-1]
+
+    patterns = [
+        f'[Tt]he\s+(?:\w+\s+)?(?:answer|option)(?:\w+\s+)?\s+is:?\s*(?:[\*\$\\{{\(\[\\\\(]*?(?:(?:\\\\boxed|\\\\mathbf|\\\\mathrm|\\\\text){{)?)*\s*({escaped_options_content_str})(?:\\\\?\}}?\$?\)?\]?\}}?)*(?:[\s:\.\*)]|$)',
+        f'(?i:Answer)\s*(?:[\*\$\\{{\(\[\\\\(]*?(?:(?:\\\\boxed|\\\\mathbf|\\\\mathrm|\\\\text){{)?)*\s*({escaped_options_content_str})(?:\\\\?\}}?\$?\)?\]?\}}?)*(?:[\s:\.\*)]|$)',
+        f'^[^\w\r\n]*(?:[\*\$\\{{\(\[\\\\(]*?(?:(?:\\\\boxed|\\\\mathbf|\\\\mathrm|\\\\text){{)?)*\s*({escaped_options_content_str})(?:\\\\?\}}?\$?\)?\]?\}}?)*(?:[\s:\.\*)]|$)',
+    ]
+
+    for pattern in patterns:
+        match = safe_regex_search(pattern, last_line)
+        if match:
+            if match.group(1) in escaped_options_content:
+                return options_content[escaped_options_content.index(match.group(1))]
+            else:
+                return match.group(1)
+
+    for pattern in patterns:
+        match = safe_regex_search(pattern, text)
+        if match:
+            if match.group(1) in escaped_options_content:
+                return options_content[escaped_options_content.index(match.group(1))]
+            else:
+                return match.group(1)
+
+    return None

evalscope/benchmarks/super_gpqa/zero_shot_prompt.txt

@@ -0,0 +1,3 @@
+Answer the following multiple choice question. There is only one correct answer. The last line of your response should be in the format 'Answer: $LETTER' (without quotes), where LETTER is one of A, B, C, D, E, F, G, H, I, or J.
+
+{query}

evalscope/benchmarks/trivia_qa/trivia_qa_adapter.py

@@ -5,8 +5,7 @@ import os
 
 from evalscope.benchmarks import Benchmark
 from evalscope.benchmarks.data_adapter import DataAdapter
-from evalscope.constants import EvalType
-from evalscope.models import ChatGenerationModelAdapter
+from evalscope.constants import EvalType, OutputType
 from evalscope.utils import get_logger
 
 # flake8: noqa
@@ -16,8 +15,8 @@ logger = get_logger()
 
 @Benchmark.register(
     name='trivia_qa',
+    pretty_name='TriviaQA',
     dataset_id='modelscope/trivia_qa',
-    model_adapter=ChatGenerationModelAdapter,
     subset_list=['default'],
     metric_list=['AverageAccuracy'],
     few_shot_num=5,
@@ -100,7 +99,7 @@ class TriviaQaAdapter(DataAdapter):
         context += self._generate_prompt(input_d=input_d, include_answer=False)
         full_prompt = context
 
-        return {'data': [full_prompt], 'system_prompt': prompt or self.prompt_template}
+        return self.gen_prompt_data(full_prompt)
 
     def get_gold_answer(self, input_d: dict) -> list:
         # Get the gold choice

evalscope/benchmarks/truthful_qa/truthful_qa_adapter.py

@@ -8,8 +8,7 @@ from typing import List
 
 from evalscope.benchmarks import Benchmark
 from evalscope.benchmarks.data_adapter import DataAdapter
-from evalscope.constants import EvalType
-from evalscope.models import ContinuationLogitsModelAdapter
+from evalscope.constants import EvalType, OutputType
 from evalscope.utils import get_logger
 
 # flake8: noqa
@@ -21,8 +20,10 @@ logger = get_logger()
 
 @Benchmark.register(
     name='truthful_qa',
+    pretty_name='TruthfulQA',
     dataset_id='modelscope/truthful_qa',
-    model_adapter=ContinuationLogitsModelAdapter,
+    model_adapter=OutputType.CONTINUOUS,
+    output_types=[OutputType.CONTINUOUS, OutputType.GENERATION],
     subset_list=['multiple_choice'],
     metric_list=['AverageAccuracy'],
     few_shot_num=0,
@@ -195,8 +196,7 @@ class TruthfulQaAdapter(DataAdapter):
         else:
             raise ValueError(f'** Unknown subset_name: {subset_name}')
 
-        prompt_d = {'data': ctx_continuation_pair_list}
-        return prompt_d
+        return self.gen_prompt_data(ctx_continuation_pair_list)
 
     def get_gold_answer(self, input_d: dict) -> dict:
         # Get the gold choice
@@ -215,14 +215,7 @@ class TruthfulQaAdapter(DataAdapter):
         Returns:
             The predicted answer.
         """
-        if eval_type == EvalType.CHECKPOINT:
-            return result
-        elif eval_type == EvalType.SERVICE:  # TODO: to be supported !
-            return result
-        elif eval_type == EvalType.CUSTOM:  # TODO: to be supported !
-            return result
-        else:
-            raise ValueError(f'Invalid eval_type: {eval_type}')
+        return result
 
     def match(self, gold: dict, pred: list) -> dict:
         """

evalscope/benchmarks/utils.py

@@ -0,0 +1,43 @@
+from dataclasses import dataclass
+from functools import wraps
+from typing import Dict, List, Optional
+
+from evalscope.constants import EvalType
+from evalscope.utils.filters import Filter
+
+
+@dataclass
+class PromptData:
+    data: List[str]
+    index: Optional[int] = 0
+    system_prompt: Optional[str] = None
+    multi_choices: Optional[List[str]] = None
+
+    def to_dict(self) -> Dict:
+        if self.multi_choices is None:
+            return {
+                'data': self.data,
+                'index': self.index,
+                'system_prompt': self.system_prompt,
+            }
+        else:
+            return {
+                'data': self.data,
+                'index': self.index,
+                'system_prompt': self.system_prompt,
+                'multi_choices': self.multi_choices,
+            }
+
+
+def preprocess_decorator(func):
+
+    @wraps(func)
+    def wrapper(self, result: str, raw_input_d: dict = None, eval_type: str = EvalType.CHECKPOINT):
+        filters = self.config_kwargs.get('filters', None)
+        if filters:
+            # Apply filters to the resultply filters to the result
+            for filter_name, filter_value in filters.items():
+                result = Filter.apply(filter_name, result, filter_value)
+        return func(self, result, raw_input_d, eval_type)
+
+    return wrapper

evalscope/collections/evaluator.py

@@ -97,8 +97,8 @@ class EvaluatorCollection:
         evaluators = {}
         for dataset_name in self.dataset_name_map.keys():
             benchmark = Benchmark.get(dataset_name)
+            model_adapter = initialize_model_adapter(self.task_cfg, benchmark, self.model)
             data_adapter = benchmark.get_data_adapter()
-            model_adapter = initialize_model_adapter(self.task_cfg, benchmark.model_adapter, self.model)
             evaluators[dataset_name] = SimpleEvaluator(dataset_name, data_adapter, model_adapter, self.task_cfg,
                                                        self.outputs)
         return evaluators
@@ -238,7 +238,16 @@ class EvaluatorCollection:
         if self.task_cfg.use_cache and os.path.exists(review_file_path):
             logger.warning(
                 f'Ignore use_cache={self.task_cfg.use_cache}, updating the review file: {review_file_path} ...')
-            os.remove(review_file_path)
+            if os.path.isdir(review_file_path):
+                for filename in os.listdir(review_file_path):
+                    file_path = os.path.join(review_file_path, filename)
+                    try:
+                        if os.path.isfile(file_path):
+                            os.remove(file_path)
+                    except Exception as e:
+                        logger.error(f'Error deleting file {file_path}: {e}')
+            else:
+                os.remove(review_file_path)
 
         reviews = defaultdict(dict)
         for sample in tqdm(self.dataset, desc='Getting reviews'):

evalscope/config.py

@@ -4,10 +4,12 @@ import copy
 import json
 import os
 from argparse import Namespace
+from collections import OrderedDict
 from dataclasses import dataclass, field
 from typing import Dict, List, Optional, Union
 
-from evalscope.constants import DEFAULT_DATASET_CACHE_DIR, DEFAULT_WORK_DIR, EvalBackend, EvalStage, EvalType, HubType
+from evalscope.constants import (DEFAULT_DATASET_CACHE_DIR, DEFAULT_WORK_DIR, EvalBackend, EvalStage, EvalType, HubType,
+                                 OutputType)
 from evalscope.models.custom import CustomModel
 from evalscope.utils import gen_hash
 from evalscope.utils.io_utils import dict_to_yaml, json_to_dict, yaml_to_dict
@@ -54,7 +56,7 @@ class TaskConfig:
     eval_config: Union[str, Dict, None] = None
     stage: str = EvalStage.ALL
     limit: Optional[int] = None
-    eval_batch_size: int = 1
+    eval_batch_size: Optional[int] = None
 
     # Cache and working directory arguments
     mem_cache: bool = False  # Deprecated, will be removed in v1.0.0.
@@ -77,6 +79,12 @@ class TaskConfig:
                 self.model_id = type(self.model).__name__
             else:
                 self.model_id = os.path.basename(self.model).rstrip(os.sep)
+            # fix path error, see http://github.com/modelscope/evalscope/issues/377
+            self.model_id = self.model_id.replace(':', '-')
+
+        # Set default eval_batch_size based on eval_type
+        if self.eval_batch_size is None:
+            self.eval_batch_size = 8 if self.eval_type == EvalType.SERVICE else 1
 
     def to_dict(self):
         return self.__dict__

evalscope/constants.py

@@ -139,6 +139,13 @@ class EvalType:
     SERVICE = 'service'  # model service
 
 
+class OutputType:
+    LOGITS = 'logits'  # for multiple choice tasks
+    GENERATION = 'generation'  # for text generation tasks and general tasks
+    MULTIPLE_CHOICE = 'multiple_choice_logits'  # for multiple choice tasks
+    CONTINUOUS = 'continuous_logits'  # for continuous tasks
+
+
 class EvalBackend:
     NATIVE = 'Native'
     OPEN_COMPASS = 'OpenCompass'

evalscope/metrics/named_metrics.py

@@ -35,6 +35,7 @@ metric_registry = MetricRegistry()
 metric_registry.register(Metric(name='AverageAccuracy', object=mean))
 metric_registry.register(Metric(name='WeightedAverageAccuracy', object=weighted_mean))
 metric_registry.register(Metric(name='AverageBLEU', object=mean))
+metric_registry.register(Metric(name='AverageRouge', object=mean))
 metric_registry.register(Metric(name='WeightedAverageBLEU', object=weighted_mean))
 metric_registry.register(Metric(name='AveragePass@1', object=mean))
 for k in range(1, 17):

evalscope/models/__init__.py

@@ -7,10 +7,11 @@ from evalscope.models.custom import CustomModel
 from evalscope.models.custom_adapter import CustomModelAdapter
 from evalscope.models.local_model import LocalModel, get_local_model
 from evalscope.models.model import BaseModel, ChatBaseModel, OpenAIModel
+from evalscope.models.register import get_model_adapter
 from evalscope.models.server_adapter import ServerModelAdapter
 
 __all__ = [
     'CustomModel', 'BaseModel', 'ChatBaseModel', 'OpenAIModel', 'BaseModelAdapter', 'ChatGenerationModelAdapter',
     'MultiChoiceModelAdapter', 'ContinuationLogitsModelAdapter', 'CustomModelAdapter', 'ServerModelAdapter',
-    'LocalModel', 'get_local_model', 'initialize_model_adapter'
+    'LocalModel', 'get_local_model', 'initialize_model_adapter', 'get_model_adapter'
 ]

evalscope/models/base_adapter.py

@@ -1,15 +1,21 @@
 import torch
 from abc import ABC, abstractmethod
-from typing import TYPE_CHECKING, Any, Optional, Union
+from typing import TYPE_CHECKING, Any, List, Optional, Union
 
-from evalscope.constants import EvalType
+from evalscope.constants import EvalType, OutputType
 from evalscope.models.custom import CustomModel
 from evalscope.models.local_model import LocalModel
+from evalscope.models.register import get_model_adapter, register_model_adapter
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
 
 if TYPE_CHECKING:
+    from evalscope.benchmarks import BenchmarkMeta
     from evalscope.config import TaskConfig
 
 
+@register_model_adapter('base')
 class BaseModelAdapter(ABC):
 
     def __init__(self, model: Optional[Union[LocalModel, CustomModel]], **kwargs):
@@ -33,7 +39,7 @@ class BaseModelAdapter(ABC):
         raise NotImplementedError
 
 
-def initialize_model_adapter(task_cfg: 'TaskConfig', model_adapter_cls: 'BaseModelAdapter', base_model: 'LocalModel'):
+def initialize_model_adapter(task_cfg: 'TaskConfig', benchmark: 'BenchmarkMeta', base_model: 'LocalModel'):
     """Initialize the model adapter based on the task configuration."""
     if task_cfg.dry_run:
         from evalscope.models.model import DummyChatModel
@@ -43,8 +49,14 @@ def initialize_model_adapter(task_cfg: 'TaskConfig', model_adapter_cls: 'BaseMod
             raise ValueError(f'Expected evalscope.models.custom.CustomModel, but got {type(task_cfg.model)}.')
         from evalscope.models import CustomModelAdapter
         return CustomModelAdapter(custom_model=task_cfg.model)
-    elif task_cfg.eval_type == EvalType.SERVICE:
+    elif task_cfg.eval_type == EvalType.SERVICE or task_cfg.api_url is not None:
         from evalscope.models import ServerModelAdapter
+
+        if benchmark.model_adapter in [OutputType.CONTINUOUS, OutputType.MULTIPLE_CHOICE]:
+            logger.warning('Output type is set to logits. This is not supported for service evaluation. '
+                           'Setting output type to generation by default.')
+            benchmark.model_adapter = OutputType.GENERATION
+
         return ServerModelAdapter(
             api_url=task_cfg.api_url,
             model_id=task_cfg.model,
@@ -54,5 +66,13 @@ def initialize_model_adapter(task_cfg: 'TaskConfig', model_adapter_cls: 'BaseMod
             stream=task_cfg.stream,
         )
     else:
-        return model_adapter_cls(
+        # for local model, we need to determine the model adapter class based on the output type
+        model_adapter_cls = benchmark.model_adapter
+        if model_adapter_cls not in benchmark.output_types:
+            logger.warning(f'Output type {model_adapter_cls} is not supported for benchmark {benchmark.name}. '
+                           f'Using {benchmark.output_types[0]} instead.')
+            model_adapter_cls = benchmark.output_types[0]
+
+        model_adapter = get_model_adapter(model_adapter_cls)
+        return model_adapter(
             model=base_model, generation_config=task_cfg.generation_config, chat_template=task_cfg.chat_template)