evalscope 0.17.1__py3-none-any.whl → 1.0.1__py3-none-any.whl

evalscope/benchmarks/super_gpqa/{five_shot_prompt.txt → prompt.py}

@@ -1,4 +1,5 @@
-Answer the following multiple choice question. There is only one correct answer. The last line of your response should be in the format 'Answer: $LETTER' (without quotes), where LETTER is one of A, B, C, D, E, F, G, H, I, or J.
+# flake8: noqa
+FEW_SHOT_SAMPLES = """Answer the following multiple choice question. There is only one correct answer. The last line of your response should be in the format 'ANSWER: $LETTER' (without quotes), where LETTER is one of A, B, C, D, E, F, G, H, I, or J.
 
 Question:
 A refracting telescope consists of two converging lenses separated by 100 cm. The eye-piece lens has a focal length of 20 cm. The angular magnification of the telescope is
@@ -13,8 +14,8 @@ H) 4
 I) 5
 J) 20
 
-Answer: Let's think step by step. In a refracting telescope, if both lenses are converging, the focus of both lenses must be between the two lenses, and thus the focal lengths of the two lenses must add up to their separation. Since the focal length of one lens is 20 cm, the focal length of the other must be 80 cm. The magnification is the ratio of these two focal lengths, or 4.
-Answer: H.
+ANSWER: Let's think step by step. In a refracting telescope, if both lenses are converging, the focus of both lenses must be between the two lenses, and thus the focal lengths of the two lenses must add up to their separation. Since the focal length of one lens is 20 cm, the focal length of the other must be 80 cm. The magnification is the ratio of these two focal lengths, or 4.
+ANSWER: H.
 
 Question:
 Say the pupil of your eye has a diameter of 5 mm and you have a telescope with an aperture of 50 cm. How much more light can the telescope gather than your eye?
@@ -29,11 +30,11 @@ H) 100 times more
 I) 10 times more
 J) N/A
 
-Answer: Let's think step by step. The amount of light a telescope can gather compared to the human eye is proportional to the area of its apertures. The area of a circle is given by the formula $A = \pi \left(\frac{{D}}{{2}}\right)^2$, where $D$ is the diameter. Therefore, the relative light-gathering power is calculated as:
+ANSWER: Let's think step by step. The amount of light a telescope can gather compared to the human eye is proportional to the area of its apertures. The area of a circle is given by the formula $A = \pi \left(\frac{{D}}{{2}}\right)^2$, where $D$ is the diameter. Therefore, the relative light-gathering power is calculated as:
 \[
 \frac{{\left(\frac{{50 \text{{ cm}}}}{{2}}\right)^2}}{{\left(\frac{{5 \text{{ mm}}}}{{2}}\right)^2}} = \frac{{\left(\frac{{50 \text{{ cm}}}}{{0.1 \text{{ cm}}}}\right)^2}}{{\left(\frac{{5 \text{{ mm}}}}{{0.1 \text{{ cm}}}}\right)^2}} = \frac{{500^2}}{{5^2}} = 10000.
 \]
-Answer: E.
+ANSWER: E.
 
 Question:
 Where do most short-period comets come from and how do we know?
@@ -44,8 +45,9 @@ D) The Oort cloud; short period comets have orbital periods similar to asteroids
 E) The Oort Cloud; short period comets tend to come from random directions indicating a spherical distribution of comets called the Oort Cloud.
 F) The Oort cloud; short period comets tend to be in the plane of the solar system just like the Oort cloud.
 G) The asteroid belt; short period comets have orbital periods similar to asteroids like Vesta and are found in the plane of the solar system just like the asteroid belt.
-Answer: Let's think step by step. Most short-period comets originate from the Kuiper belt. This is deduced from the observation that these comets tend to follow orbits that lie in the plane of the solar system, similar to the distribution of objects in the Kuiper belt itself. Thus, the alignment of these cometary orbits with the ecliptic plane points to their Kuiper belt origin.
-Answer: A.
+
+ANSWER: Let's think step by step. Most short-period comets originate from the Kuiper belt. This is deduced from the observation that these comets tend to follow orbits that lie in the plane of the solar system, similar to the distribution of objects in the Kuiper belt itself. Thus, the alignment of these cometary orbits with the ecliptic plane points to their Kuiper belt origin.
+ANSWER: A.
 
 Question:
 Colors in a soap bubble result from light
@@ -60,8 +62,8 @@ H) absorption
 I) diffraction
 J) transmission
 
-Answer: Let's think step by step. The colorful patterns observed in a soap bubble are caused by the phenomenon of light interference. This occurs when light waves bounce between the two surfaces of the soap film, combining constructively or destructively based on their phase differences and the varying thickness of the film. These interactions result in vibrant color patterns due to variations in the intensity of different wavelengths of light.
-Answer: E.
+ANSWER: Let's think step by step. The colorful patterns observed in a soap bubble are caused by the phenomenon of light interference. This occurs when light waves bounce between the two surfaces of the soap film, combining constructively or destructively based on their phase differences and the varying thickness of the film. These interactions result in vibrant color patterns due to variations in the intensity of different wavelengths of light.
+ANSWER: E.
 
 Question:
 A microwave oven is connected to an outlet, 120 V, and draws a current of 2 amps. At what rate is energy being used by the microwave oven?
@@ -76,15 +78,11 @@ H) 150 W
 I) 60 W
 J) 300 W
 
-Answer: Let's think step by step. The rate of energy usage, known as power, in an electrical circuit is calculated by the product of voltage and current. For a microwave oven connected to a 120 V outlet and drawing a current of 2 amps, the power consumption can be calculated as follows:
+ANSWER: Let's think step by step. The rate of energy usage, known as power, in an electrical circuit is calculated by the product of voltage and current. For a microwave oven connected to a 120 V outlet and drawing a current of 2 amps, the power consumption can be calculated as follows:
 \[
 \text{{Power}} = \text{{Voltage}} \times \text{{Current}} = 120 \, \text{{V}} \times 2 \, \text{{A}} = 240 \, \text{{W}}.
 \]
 Therefore, the microwave oven uses energy at a rate of 240 watts.
-Answer: A.
-
-Question:
-{query}
-{choices}
+ANSWER: A.
 
-Answer: Let's think step by step.
+"""

evalscope/benchmarks/super_gpqa/super_gpqa_adapter.py

@@ -1,34 +1,16 @@
-import os
-import random
-import re
+# Copyright (c) Alibaba, Inc. and its affiliates.
 
-from evalscope.benchmarks import Benchmark, DataAdapter
-from evalscope.constants import EvalType, OutputType
-from evalscope.metrics import exact_match
-from evalscope.utils import logger
+import os
+from typing import Any, Dict
 
-current_dir = os.path.dirname(os.path.abspath(__file__))
+from evalscope.api.benchmark import BenchmarkMeta, MultiChoiceAdapter
+from evalscope.api.dataset import Sample
+from evalscope.api.registry import register_benchmark
+from evalscope.constants import Tags
+from evalscope.utils.logger import get_logger
+from evalscope.utils.multi_choices import FEW_SHOT_TEMPLATE, MultipleChoiceTemplate
 
-SUBSET_LIST = [
-    'Electronic Science and Technology', 'Philosophy', 'Traditional Chinese Medicine', 'Applied Economics',
-    'Mathematics', 'Physics', 'Clinical Medicine', 'Computer Science and Technology',
-    'Information and Communication Engineering', 'Control Science and Engineering', 'Theoretical Economics', 'Law',
-    'History', 'Basic Medicine', 'Education', 'Materials Science and Engineering', 'Electrical Engineering',
-    'Systems Science', 'Power Engineering and Engineering Thermophysics', 'Military Science', 'Biology',
-    'Business Administration', 'Language and Literature', 'Public Health and Preventive Medicine', 'Political Science',
-    'Chemistry', 'Hydraulic Engineering', 'Chemical Engineering and Technology', 'Pharmacy', 'Geography', 'Art Studies',
-    'Architecture', 'Forestry Engineering', 'Public Administration', 'Oceanography', 'Journalism and Communication',
-    'Nuclear Science and Technology', 'Weapon Science and Technology', 'Naval Architecture and Ocean Engineering',
-    'Environmental Science and Engineering', 'Transportation Engineering', 'Geology', 'Physical Oceanography',
-    'Musicology', 'Stomatology', 'Aquaculture', 'Mechanical Engineering',
-    'Aeronautical and Astronautical Science and Technology', 'Civil Engineering', 'Mechanics',
-    'Petroleum and Natural Gas Engineering', 'Sociology', 'Food Science and Engineering', 'Agricultural Engineering',
-    'Surveying and Mapping Science and Technology', 'Metallurgical Engineering',
-    'Library, Information and Archival Management', 'Mining Engineering', 'Astronomy',
-    'Geological Resources and Geological Engineering', 'Atmospheric Science', 'Optical Engineering', 'Animal Husbandry',
-    'Geophysics', 'Crop Science', 'Management Science and Engineering', 'Psychology', 'Forestry',
-    'Textile Science and Engineering', 'Veterinary Medicine', 'Instrument Science and Technology', 'Physical Education'
-]
+logger = get_logger()
 
 SUBSET_MAPPING = {
     'Electronic Science and Technology': ['Engineering'],
@@ -106,104 +88,78 @@ SUBSET_MAPPING = {
 }
 
 
-@Benchmark.register(
-    name='super_gpqa',
-    pretty_name='SuperGPQA',
-    tags=['MCQ', 'Knowledge'],
-    description=
-    'SuperGPQA is a large-scale multiple-choice question answering dataset, designed to evaluate the generalization ability of models across different fields. It contains 100,000+ questions from 50+ fields, with each question having 10 options.',  # noqa: E501
-    dataset_id='m-a-p/SuperGPQA',
-    model_adapter=OutputType.GENERATION,
-    output_types=[OutputType.MULTIPLE_CHOICE, OutputType.GENERATION],
-    subset_list=SUBSET_LIST,
-    metric_list=['AverageAccuracy'],
-    few_shot_num=0,
-    train_split=None,
-    eval_split='train',  # only have train split
+@register_benchmark(
+    BenchmarkMeta(
+        name='super_gpqa',
+        pretty_name='SuperGPQA',
+        tags=[Tags.KNOWLEDGE, Tags.MULTIPLE_CHOICE],
+        description=
+        'SuperGPQA is a large-scale multiple-choice question answering dataset, designed to evaluate the generalization ability of models across different fields. It contains 100,000+ questions from 50+ fields, with each question having 10 options.',  # noqa: E501
+        dataset_id='m-a-p/SuperGPQA',
+        subset_list=list(SUBSET_MAPPING.keys()),
+        metric_list=['acc'],
+        few_shot_num=0,
+        train_split=None,
+        eval_split='train',  # only have train split
+        prompt_template=MultipleChoiceTemplate.SINGLE_ANSWER_COT,
+    )
 )
-class SuperGPQAAdapter(DataAdapter):
+class SuperGPQAAdapter(MultiChoiceAdapter):
 
     def __init__(self, **kwargs):
-        few_shot_num = kwargs.get('few_shot_num', 0)
-        if few_shot_num > 0 and few_shot_num != 5:
-            logger.warning(
-                f'Only support few_shot_num 0 or 5 for SuperGPQA, but got {few_shot_num}. Use 5-shot by default.')
-            kwargs['few_shot_num'] = 5
+
         super().__init__(**kwargs)
+        if self.few_shot_num > 0 and self.few_shot_num != 5:
+            logger.warning(
+                f'Only support few_shot_num 0 or 5 for SuperGPQA, but got {self.few_shot_num}. Use 5-shot by default.'
+            )
+            self.few_shot_num = 5
 
-        self.choices = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']
+        self.reformat_subset = True
         self.category_map = SUBSET_MAPPING
-        self.few_shot_prompt = open(os.path.join(current_dir, 'five_shot_prompt.txt'), encoding='utf-8').read()
-        self.zero_shot_prompt = open(os.path.join(current_dir, 'zero_shot_prompt.txt'), encoding='utf-8').read()
-
-    def load(self, **kwargs):
-        kwargs['subset_list'] = ['default']
-        data_dict = super().load(**kwargs)
-        return self.reformat_subset(data_dict, subset_key='field', format='{}')
-
-    def gen_prompt(self, input_d: dict, subset_name: str, few_shot_list: list, **kwargs) -> dict:
-        question = input_d['question']
-        choices = self._format_choices(input_d['options'])
-        if not self.prompt_template:
-            if few_shot_list:
-                prompt = self.few_shot_prompt.format(query=question, choices=choices)
-            else:
-                prompt = self.zero_shot_prompt.format(query=question, choices=choices)
-        else:
-            prompt = self.prompt_template.format(query=question, choices=choices)
-        return self.gen_prompt_data(prompt)
-
-    def get_gold_answer(self, input_d: dict) -> str:
-        # Get the gold choice
-        return input_d.get('answer_letter')
 
-    def parse_pred_result(self, result: str, raw_input_d: dict = None, eval_type: str = EvalType.CHECKPOINT) -> str:
+    def record_to_sample(self, record: Dict[str, Any]) -> Sample:
+        return Sample(
+            input=record['question'],
+            choices=record['options'],
+            target=record['answer_letter'],
+            subset_key=record['field'],
+            metadata={
+                'field': record['field'],
+                'discipline': record['discipline'],
+                'uuid': record.get('uuid', ''),
+                'explanation': record.get('answer', ''),
+            },
+        )
+
+    def format_fewshot_template(self, fewshot, sample):
+        from .prompt import FEW_SHOT_SAMPLES
+
+        return FEW_SHOT_TEMPLATE.format(fewshot=FEW_SHOT_SAMPLES, ) + self.format_prompt_template(sample)
+
+    def extract_answer(self, prediction: str, task_state) -> str:
         """
-        Parse the model output to get the answer. Could be the best choice index.
-
-        Args:
-            result: Predicted answer from the model. Usually a string for chat.
-            raw_input_d: The raw input. Depending on the dataset.
-            eval_type: 'checkpoint' or 'service' or 'custom'
-
-        Returns:
-            The parsed answer. Depending on the dataset. Usually a string for chat.
+        Extract the answer from the prediction.
         """
-        if self.model_adapter == OutputType.MULTIPLE_CHOICE:
-            return result
+        from .utils import extract_option_content, extract_option_labels
+
+        choices = [choice.value for choice in task_state.choices]
+        if self.few_shot_num == 0:
+            predict = extract_option_labels(prediction, 'ABCDEFGHIJ')
+            if predict is None:
+                # Try to extract by content matching
+                predict = extract_option_content(prediction, choices)
+                predict = chr(choices.index(predict) + 65) if predict else None
         else:
-            from evalscope.benchmarks.super_gpqa.utils import extract_option_content, extract_option_labels
-            sample = raw_input_d
-            if self.few_shot_num == 0:
-                predict = extract_option_labels(result, 'ABCDEFGHIJ')
-                if predict is None:
-                    predict = extract_option_content(result, sample['options'])
-                    predict = chr(sample['options'].index(predict) + 65) if predict else None
-            else:
-                response = result.split('Question:')[0]
-                predict = extract_option_labels(response, 'ABCDEFGHIJ')
+            response = prediction.split('Question:')[0]
+            predict = extract_option_labels(response, 'ABCDEFGHIJ')
+            if predict is None:
+                predict = extract_option_content(response, choices)
+                predict = chr(choices.index(predict) + 65) if predict else None
+            if predict is None:
+                predict = extract_option_labels(prediction, 'ABCDEFGHIJ')
                 if predict is None:
-                    predict = extract_option_content(response, sample['options'])
-                    predict = chr(sample['options'].index(predict) + 65) if predict else None
-                if predict is None:
-                    predict = extract_option_labels(result, 'ABCDEFGHIJ')
-                    if predict is None:
-                        predict = extract_option_content(result, sample['options'])
-                        predict = chr(sample['options'].index(predict) + 65) if predict else None
-            return predict
-
-    def match(self, gold: str, pred: str) -> float:
-        return exact_match(gold=gold, pred=pred)
+                    predict = extract_option_content(prediction, choices)
+                    predict = chr(choices.index(predict) + 65) if predict else None
 
-    def _format_choices(self, choices: list) -> str:
-        """
-        Format the choices into a string for display.
-
-        Args:
-            choices (list): List of choices.
-
-        Returns:
-            str: Formatted string of choices.
-        """
-        choice_list = [f'{option}) {content}' for option, content in zip(self.choices, choices)]
-        return '\n'.join(choice_list)
+        return predict or ''

evalscope/benchmarks/super_gpqa/utils.py

@@ -1,5 +1,6 @@
 # flake8: noqa
 import re
+from collections.abc import Sequence
 
 
 def safe_regex_search(pattern, text, flags=0):
@@ -51,7 +52,7 @@ def extract_option_labels(text, options='ABCDEFGHIJ'):
 
 
 def extract_option_content(text, options_content=None):
-    if not isinstance(text, str) or not isinstance(options_content, list):
+    if not isinstance(text, str) or not isinstance(options_content, Sequence):
         return 'error'
 
     escaped_options_content = [re.escape(option_content) for option_content in options_content]

evalscope/benchmarks/tau_bench/generation.py

@@ -0,0 +1,147 @@
+from typing import Any, Dict, List, Optional
+
+from evalscope.api.dataset import Sample
+from evalscope.api.messages import dict_to_chat_message
+from evalscope.api.model import Model, ModelOutput
+from evalscope.api.model.model_output import ChatCompletionChoice
+from evalscope.api.tool import ToolInfo
+from evalscope.models.utils.openai import openai_chat_choices
+from evalscope.utils.function_utils import run_once
+from evalscope.utils.logger import get_logger
+
+logger = get_logger()
+
+
+@run_once
+def _patch_agent_solve(model: Model):
+    """Patch ToolCallingAgent.solve method to use custom model configuration"""
+    from tau_bench.agents.tool_calling_agent import ToolCallingAgent, message_to_action
+    from tau_bench.envs.base import Env
+    from tau_bench.types import RESPOND_ACTION_NAME, Action, SolveResult
+
+    def patched_solve(
+        self,
+        env: Env,
+        task_index: Optional[int] = None,
+        max_num_steps: int = 30,
+    ) -> SolveResult:
+        env_reset_res = env.reset(task_index=task_index)
+        obs = env_reset_res.observation
+        info = env_reset_res.info.model_dump()
+        reward = 0.0
+        messages: List[Dict[str, Any]] = [
+            {
+                'role': 'system',
+                'content': self.wiki
+            },
+            {
+                'role': 'user',
+                'content': obs
+            },
+        ]
+
+        for step_index in range(max_num_steps):
+            res = model.generate(
+                input=[dict_to_chat_message(msg) for msg in messages],
+                tools=[ToolInfo.model_validate(tool['function']) for tool in self.tools_info]
+            )
+            oai_res = openai_chat_choices(res.choices, include_reasoning=False)
+
+            next_message = oai_res[0].message.model_dump(exclude_none=True)
+
+            action = message_to_action(next_message)
+
+            env_response = env.step(action)
+            reward = env_response.reward
+            info = {**info, **env_response.info.model_dump()}
+
+            if action.name != RESPOND_ACTION_NAME:
+                next_message['tool_calls'] = next_message['tool_calls'][:1]
+                messages.extend([
+                    next_message,
+                    {
+                        'role': 'tool',
+                        'tool_call_id': next_message['tool_calls'][0]['id'],
+                        'name': next_message['tool_calls'][0]['function']['name'],
+                        'content': env_response.observation,
+                    },
+                ])
+            else:
+                messages.extend([
+                    next_message,
+                    {
+                        'role': 'user',
+                        'content': env_response.observation
+                    },
+                ])
+            logger.debug(f'Task: {task_index} Step: {step_index} finished')
+
+            if env_response.done:
+                break
+
+        return SolveResult(
+            reward=reward,
+            info=info,
+            messages=messages,
+            total_cost=0,
+        )
+
+    ToolCallingAgent.solve = patched_solve
+
+    return 'ToolCallingAgent.solve patched successfully'
+
+
+def predict(model: Model, sample: Sample) -> ModelOutput:
+    """
+    Generate predictions for tau_bench tasks using the model.
+
+    Args:
+        model: The model to use for prediction
+        sample: The sample containing task metadata
+
+    Returns:
+        ModelOutput containing the prediction results
+    """
+    from tau_bench.agents.tool_calling_agent import ToolCallingAgent
+    from tau_bench.envs import get_env
+
+    _patch_agent_solve(model)
+    try:
+        # Extract task information from sample metadata
+        task_data = sample.metadata
+        env_name = task_data['env_name']
+        task_index = task_data['task_index']
+
+        # Direct call to tau_bench_server adapter's solve method
+        # This method can be implemented to solve specific tasks in the TauBench environment
+        isolated_env = get_env(
+            env_name=env_name,
+            user_strategy='llm',
+            user_model='dummy',  # Use dummy model to prevent errors
+            user_provider='openai',  # Use dummy provider to prevent errors
+            task_split='test',
+            task_index=task_index,
+        )
+        agent = ToolCallingAgent(
+            tools_info=isolated_env.tools_info,
+            wiki=isolated_env.wiki,
+            model='dummy',  # Use dummy model to prevent errors
+            provider='dummy',  # Use dummy provider to prevent errors
+            temperature=0,  # dummy temperature to prevent errors
+        )
+
+        res = agent.solve(env=isolated_env, task_index=task_index)
+
+        sample.metadata['task_result'] = res.model_dump(exclude_none=True)
+        return ModelOutput(
+            model=model.name,
+            choices=[ChatCompletionChoice.from_content(res.model_dump_json(indent=2))],
+        )
+
+    except Exception as e:
+        logger.error(f'Error in tau_bench prediction: {str(e)}')
+        sample.metadata['task_result'] = {'reward': 0, 'error': str(e)}
+        return ModelOutput(
+            model=model.name,
+            choices=[ChatCompletionChoice.from_content('')],
+        )