azure-ai-evaluation 1.0.1__py3-none-any.whl → 1.13.3__py3-none-any.whl

azure/ai/evaluation/_evaluators/_groundedness/groundedness_without_query.prompty

@@ -29,11 +29,16 @@ system:
 
 user:
 # Definition
-**Groundedness** refers to how faithfully a response adheres to the information provided in the CONTEXT, ensuring that all content is directly supported by the context without introducing unsupported information or omitting critical details. It evaluates the fidelity and precision of the response in relation to the source material.
+**Groundedness** refers to how well a response is anchored in the provided context, evaluating its relevance, accuracy, and completeness based exclusively on that context. It assesses the extent to which the response directly and fully addresses the information without introducing unrelated or incorrect information.
+
+> Context is the source of truth for evaluating the response. 
+> Evaluate the groundedness of the response message based on the provided context.
 
 # Ratings
-## [Groundedness: 1] (Completely Ungrounded Response)
-**Definition:** The response is entirely unrelated to the CONTEXT, introducing topics or information that have no connection to the provided material.
+## [Groundedness: 1] (Completely Unrelated Response)
+**Definition:** A response that does not relate to the context in any way.
+  - Does not relate to the context at all.
+  - Talks about the general topic but does not respond to the context.
 
 **Examples:**
   **Context:** The company's profits increased by 20% in the last quarter.
@@ -42,8 +47,8 @@ user:
   **Context:** The new smartphone model features a larger display and improved battery life.
   **Response:** The history of ancient Egypt is fascinating and full of mysteries.
 
-## [Groundedness: 2] (Contradictory Response)
-**Definition:** The response directly contradicts or misrepresents the information provided in the CONTEXT.
+## [Groundedness: 2] (Attempts to Respond but Contains Incorrect Information)
+**Definition:** A response that attempts to relate to the context but includes incorrect information not supported by the context. It may misstate facts, misinterpret the context, or provide erroneous details. Even if some points are correct, the presence of inaccuracies makes the response unreliable.
 
 **Examples:**
   **Context:** The company's profits increased by 20% in the last quarter.
@@ -52,18 +57,18 @@ user:
   **Context:** The new smartphone model features a larger display and improved battery life.
   **Response:** The new smartphone model has a smaller display and shorter battery life.
 
-## [Groundedness: 3] (Accurate Response with Unsupported Additions)
-**Definition:** The response accurately includes information from the CONTEXT but adds details, opinions, or explanations that are not supported by the provided material.
+## [Groundedness: 3] (Accurate but Vague Response)
+**Definition:** A response that provides accurate information from the context but is overly generic or vague, not meaningfully engaging with the specific details in the context. The information is correct but lacks specificity and detail.
 
 **Examples:**
-  **Context:** The company's profits increased by 20% in the last quarter.
-  **Response:** The company's profits increased by 20% in the last quarter due to their aggressive marketing strategy.
+  **Context:** The company's profits increased by 20% in the last quarter, marking the highest growth rate in its history.
+  **Response:** The company is doing well financially.
 
-  **Context:** The new smartphone model features a larger display and improved battery life.
-  **Response:** The new smartphone model features a larger display, improved battery life, and comes with a free case.
+  **Context:** The new smartphone model features a larger display, improved battery life, and an upgraded camera system.
+  **Response:** The smartphone has some nice features.
 
-## [Groundedness: 4] (Incomplete Response Missing Critical Details)
-**Definition:** The response contains information from the CONTEXT but omits essential details that are necessary for a comprehensive understanding of the main point.
+## [Groundedness: 4] (Partially Correct Response)
+**Definition:** A response that provides correct information from the context but is incomplete or lacks specific details mentioned in the context. It captures some of the necessary information but omits key elements needed for a full understanding.
 
 **Examples:**
   **Context:** The company's profits increased by 20% in the last quarter, marking the highest growth rate in its history.      
@@ -73,7 +78,7 @@ user:
   **Response:** The new smartphone model features a larger display and improved battery life.
 
 ## [Groundedness: 5] (Fully Grounded and Complete Response)
-**Definition:** The response is entirely based on the CONTEXT, accurately and thoroughly conveying all essential information without introducing unsupported details or omitting critical points.
+**Definition:** A response that thoroughly and accurately conveys information from the context, including all relevant details. It directly addresses the context with precise information, demonstrating complete understanding without adding extraneous information.
 
 **Examples:**
   **Context:** The company's profits increased by 20% in the last quarter, marking the highest growth rate in its history.      

azure/ai/evaluation/_evaluators/_intent_resolution/__init__.py

@@ -0,0 +1,7 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from ._intent_resolution import IntentResolutionEvaluator
+
+__all__ = ["IntentResolutionEvaluator"]

azure/ai/evaluation/_evaluators/_intent_resolution/_intent_resolution.py

@@ -0,0 +1,196 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+import os
+import math
+import logging
+from typing import Dict, Union, List, Optional
+
+from typing_extensions import overload, override
+
+from azure.ai.evaluation._exceptions import EvaluationException, ErrorBlame, ErrorCategory, ErrorTarget
+from azure.ai.evaluation._evaluators._common import PromptyEvaluatorBase
+from azure.ai.evaluation._model_configurations import Conversation, Message
+from ..._common.utils import check_score_is_valid, reformat_conversation_history, reformat_agent_response
+from azure.ai.evaluation._common._experimental import experimental
+
+logger = logging.getLogger(__name__)
+
+
+@experimental
+class IntentResolutionEvaluator(PromptyEvaluatorBase[Union[str, float]]):
+    """
+    Evaluates intent resolution for a given query and response or a multi-turn conversation, including reasoning.
+
+    The intent resolution evaluator assesses whether the user intent was correctly identified and resolved.
+
+    :param model_config: Configuration for the Azure OpenAI model.
+    :type model_config: Union[~azure.ai.evaluation.AzureOpenAIModelConfiguration,
+        ~azure.ai.evaluation.OpenAIModelConfiguration]
+
+    .. admonition:: Example:
+
+        .. literalinclude:: ../samples/evaluation_samples_evaluate.py
+            :start-after: [START intent_resolution_evaluator]
+            :end-before: [END intent_resolution_evaluator]
+            :language: python
+            :dedent: 8
+            :caption: Initialize and call an IntentResolutionEvaluator with a query and response.
+
+    .. admonition:: Example using Azure AI Project URL:
+
+        .. literalinclude:: ../samples/evaluation_samples_evaluate_fdp.py
+            :start-after: [START intent_resolution_evaluator]
+            :end-before: [END intent_resolution_evaluator]
+            :language: python
+            :dedent: 8
+            :caption: Initialize and call IntentResolutionEvaluator using Azure AI Project URL in the following format
+                https://{resource_name}.services.ai.azure.com/api/projects/{project_name}
+
+    """
+
+    _PROMPTY_FILE = "intent_resolution.prompty"
+    _RESULT_KEY = "intent_resolution"
+    _OPTIONAL_PARAMS = ["tool_definitions"]
+
+    _MIN_INTENT_RESOLUTION_SCORE = 1
+    _MAX_INTENT_RESOLUTION_SCORE = 5
+    _DEFAULT_INTENT_RESOLUTION_THRESHOLD = 3
+
+    id = "azureai://built-in/evaluators/intent_resolution"
+    """Evaluator identifier, experimental and to be used only with evaluation in cloud."""
+
+    @override
+    def __init__(self, model_config, *, threshold=_DEFAULT_INTENT_RESOLUTION_THRESHOLD, credential=None, **kwargs):
+        current_dir = os.path.dirname(__file__)
+        prompty_path = os.path.join(current_dir, self._PROMPTY_FILE)
+        self.threshold = threshold
+        super().__init__(
+            model_config=model_config,
+            prompty_file=prompty_path,
+            result_key=self._RESULT_KEY,
+            threshold=threshold,
+            credential=credential,
+            _higher_is_better=True,
+            **kwargs,
+        )
+
+    @overload
+    def __call__(
+        self,
+        *,
+        query: Union[str, List[dict]],
+        response: Union[str, List[dict]],
+        tool_definitions: Optional[Union[dict, List[dict]]] = None,
+    ) -> Dict[str, Union[str, float]]:
+        """Evaluate intent resolution for a given query, response and optional tool definitions.
+        The query and response can be either a string or a list of messages.
+
+        Example with string inputs and no tools:
+            evaluator = IntentResolutionEvaluator(model_config)
+            query = "What is the weather today?"
+            response = "The weather is sunny."
+
+            result = evaluator(query=query, response=response)
+
+        Example with list of messages:
+            evaluator = IntentResolutionEvaluator(model_config)
+            query: [{'role': 'system', 'content': 'You are a friendly and helpful customer service agent.'}, {'createdAt': 1700000060, 'role': 'user', 'content': [{'type': 'text', 'text': 'Hi, I need help with the last 2 orders on my account #888. Could you please update me on their status?'}]}]
+            response: [{'createdAt': 1700000070, 'run_id': '0', 'role': 'assistant', 'content': [{'type': 'text', 'text': 'Hello! Let me quickly look up your account details.'}]}, {'createdAt': 1700000075, 'run_id': '0', 'role': 'assistant', 'content': [{'type': 'tool_call', 'tool_call': {'id': 'tool_call_20250310_001', 'type': 'function', 'function': {'name': 'get_orders', 'arguments': {'account_number': '888'}}}}]}, {'createdAt': 1700000080, 'run_id': '0', 'tool_call_id': 'tool_call_20250310_001', 'role': 'tool', 'content': [{'type': 'tool_result', 'tool_result': '[{ "order_id": "123" }, { "order_id": "124" }]'}]}, {'createdAt': 1700000085, 'run_id': '0', 'role': 'assistant', 'content': [{'type': 'text', 'text': 'Thanks for your patience. I see two orders on your account. Let me fetch the details for both.'}]}, {'createdAt': 1700000090, 'run_id': '0', 'role': 'assistant', 'content': [{'type': 'tool_call', 'tool_call': {'id': 'tool_call_20250310_002', 'type': 'function', 'function': {'name': 'get_order', 'arguments': {'order_id': '123'}}}}, {'type': 'tool_call', 'tool_call': {'id': 'tool_call_20250310_003', 'type': 'function', 'function': {'name': 'get_order', 'arguments': {'order_id': '124'}}}}]}, {'createdAt': 1700000095, 'run_id': '0', 'tool_call_id': 'tool_call_20250310_002', 'role': 'tool', 'content': [{'type': 'tool_result', 'tool_result': '{ "order": { "id": "123", "status": "shipped", "delivery_date": "2025-03-15" } }'}]}, {'createdAt': 1700000100, 'run_id': '0', 'tool_call_id': 'tool_call_20250310_003', 'role': 'tool', 'content': [{'type': 'tool_result', 'tool_result': '{ "order": { "id": "124", "status": "delayed", "expected_delivery": "2025-03-20" } }'}]}, {'createdAt': 1700000105, 'run_id': '0', 'role': 'assistant', 'content': [{'type': 'text', 'text': 'The order with ID 123 has been shipped and is expected to be delivered on March 15, 2025. However, the order with ID 124 is delayed and should now arrive by March 20, 2025. Is there anything else I can help you with?'}]}]
+            tool_definitions: [{'name': 'get_orders', 'description': 'Get the list of orders for a given account number.', 'parameters': {'type': 'object', 'properties': {'account_number': {'type': 'string', 'description': 'The account number to get the orders for.'}}}}, {'name': 'get_order', 'description': 'Get the details of a specific order.', 'parameters': {'type': 'object', 'properties': {'order_id': {'type': 'string', 'description': 'The order ID to get the details for.'}}}}, {'name': 'initiate_return', 'description': 'Initiate the return process for an order.', 'parameters': {'type': 'object', 'properties': {'order_id': {'type': 'string', 'description': 'The order ID for the return process.'}}}}, {'name': 'update_shipping_address', 'description': 'Update the shipping address for a given account.', 'parameters': {'type': 'object', 'properties': {'account_number': {'type': 'string', 'description': 'The account number to update.'}, 'new_address': {'type': 'string', 'description': 'The new shipping address.'}}}}]
+
+            result = evaluator(query=query, response=response, tool_definitions=tool_definitions)
+
+        :keyword query: The query to be evaluated which is either a string or a list of messages.
+            The list of messages is the previous conversation history of the user and agent, including system messages and tool calls.
+        :paramtype query: Union[str, List[dict]]
+        :keyword response: The response to be evaluated, which is either a string or a list of messages (full agent response potentially including tool calls)
+        :paramtype response: Union[str, List[dict]]
+        :keyword tool_definitions: An optional list of messages containing the tool definitions the agent is aware of.
+        :paramtype tool_definitions: Optional[Union[dict, List[dict]]]
+        :return: A dictionary with the intent resolution evaluation
+        :rtype: Dict[str, Union[str, float]]
+        """
+
+    @override
+    def __call__(  # pylint: disable=docstring-missing-param
+        self,
+        *args,
+        **kwargs,
+    ):
+        """
+        Invokes the instance using the overloaded __call__ signature.
+
+        For detailed parameter types and return value documentation, see the overloaded __call__ definition.
+        """
+        return super().__call__(*args, **kwargs)
+
+    @override
+    async def _do_eval(self, eval_input: Dict) -> Dict[str, Union[float, str]]:  # type: ignore[override]
+        """Do intent resolution evaluation.
+
+        :param eval_input: The input to the evaluator. Expected to contain whatever inputs are needed for the _flow method
+        :type eval_input: Dict
+        :return: The evaluation result.
+        :rtype: Dict
+        """
+        # we override the _do_eval method as we want the output to be a dictionary, which is a different schema than _base_prompty_eval.py
+        if "query" not in eval_input and "response" not in eval_input:
+            raise EvaluationException(
+                message=f"Both query and response must be provided as input to the intent resolution evaluator.",
+                internal_message=f"Both query and response must be provided as input to the intent resolution evaluator.",
+                blame=ErrorBlame.USER_ERROR,
+                category=ErrorCategory.MISSING_FIELD,
+                target=ErrorTarget.INTENT_RESOLUTION_EVALUATOR,
+            )
+        # reformat query and response to the format expected by the prompty flow
+        eval_input["query"] = reformat_conversation_history(eval_input["query"], logger)
+        eval_input["response"] = reformat_agent_response(eval_input["response"], logger)
+
+        prompty_output_dict = await self._flow(timeout=self._LLM_CALL_TIMEOUT, **eval_input)
+        llm_output = prompty_output_dict["llm_output"]
+        # llm_output should always be a dictionary because the response_format of prompty is set to json_object, but checking anyway
+        score = math.nan
+        if isinstance(llm_output, dict):
+            score = llm_output.get("score", math.nan)
+            if not check_score_is_valid(
+                score,
+                IntentResolutionEvaluator._MIN_INTENT_RESOLUTION_SCORE,
+                IntentResolutionEvaluator._MAX_INTENT_RESOLUTION_SCORE,
+            ):
+                raise EvaluationException(
+                    message=f"Invalid score value: {score}. Expected a number in range [{IntentResolutionEvaluator._MIN_INTENT_RESOLUTION_SCORE}, {IntentResolutionEvaluator._MAX_INTENT_RESOLUTION_SCORE}].",
+                    internal_message="Invalid score value.",
+                    category=ErrorCategory.FAILED_EXECUTION,
+                    blame=ErrorBlame.SYSTEM_ERROR,
+                )
+            reason = llm_output.get("explanation", "")
+            score = float(score)
+            score_result = "pass" if score >= self._threshold else "fail"
+
+            response_dict = {
+                f"{self._result_key}": score,
+                f"gpt_{self._result_key}": score,
+                f"{self._result_key}_result": score_result,
+                f"{self._result_key}_threshold": self._threshold,
+                f"{self._result_key}_reason": reason,
+                f"{self._result_key}_prompt_tokens": prompty_output_dict.get("input_token_count", 0),
+                f"{self._result_key}_completion_tokens": prompty_output_dict.get("output_token_count", 0),
+                f"{self._result_key}_total_tokens": prompty_output_dict.get("total_token_count", 0),
+                f"{self._result_key}_finish_reason": prompty_output_dict.get("finish_reason", ""),
+                f"{self._result_key}_model": prompty_output_dict.get("model_id", ""),
+                f"{self._result_key}_sample_input": prompty_output_dict.get("sample_input", ""),
+                f"{self._result_key}_sample_output": prompty_output_dict.get("sample_output", ""),
+            }
+            return response_dict
+        # If llm_output is not a dictionary, return NaN for the score. This should never happen
+        if logger:
+            logger.warning("LLM output is not a dictionary, returning NaN for the score.")
+
+        binary_result = self._get_binary_result(score)
+        return {
+            self._result_key: float(score),
+            f"gpt_{self._result_key}": float(score),
+            f"{self._result_key}_result": binary_result,
+            f"{self._result_key}_threshold": self._threshold,
+        }

azure/ai/evaluation/_evaluators/_intent_resolution/intent_resolution.prompty

@@ -0,0 +1,275 @@
+---
+name: Intent Resolution Evaluator
+description: Evaluates whether user intent was identified and correctly resolved
+model:
+  api: chat
+  parameters:
+    temperature: 0.0
+    max_tokens: 800
+    top_p: 1.0
+    presence_penalty: 0
+    frequency_penalty: 0
+    response_format:
+      type: json_object
+
+inputs:
+  query:
+    type: string
+  response:
+    type: string
+  tool_definitions:
+    type: string
+    optional: true
+    default: "[]"
+---
+system:
+You are an expert in evaluating the quality of a AGENT_RESPONSE from an intelligent assistant based on provided definition and CONVERSATION_HISTORY.
+
+user:
+ROLE
+====
+You are Intent-Resolution-Judge, an impartial grader that scores how well an AI agent *resolved* the user's intent in a multi-turn conversation.
+You are NOT grading intent recognition. Assume the agent has understood the intent that is expressed; you only judge whether the reply satisfies or completes that intent.
+
+
+INPUT
+=====
+CONVERSATION_HISTORY: {{query}}
+AGENT_RESPONSE: {{response}}
+
+CONVERSATION_HISTORY is the full dialogue between the user and the agent up to the user's latest message.
+AGENT_RESPONSE is the agent reply to that latest user message.
+
+
+TASK
+====
+Output a JSON object with:
+    1) a concise explanation of 15-60 words that summarizes the agent's performance in resolving the user's intent
+    2) an integer score from 1 (very poor) to 5 (excellent) on how well the agent resolved the user's intent.
+
+The explanation should always precede the score and should clearly justify the score based on the agent's performance in resolving the user's intent.
+Response format exactly as follows:
+
+{
+  "explanation": "<15-60 words>",
+  "score": <1-5>
+}
+
+EVALUATION STEPS
+================
+
+A. Identify the expressed intent in the final user turn (look at the full conversation_history for context if necessary).
+B. Check resolution - Does the agent's reply, in AGENT_RESPONSE, actually complete or satisfy that intent? 
+    - If the agent's response is a direct answer, does it fully address the user's request?
+    - If the agent's response is an action (like scheduling, deleting, etc.), does it confirm completion of that action?
+    - If the agent's response is a clarification or follow-up question, does it lead towards fulfilling the intent?
+    - If the agent response is empty or irrelevant, it does not resolve the intent and should be scored accordingly.
+C. Verify correctness & completeness of the resolution.
+D. Weigh impact - Minor style issues matter only for tie-breaking; resolution quality dominates.
+E. Write a concise explanation of 15-60 words that summarizes the agent's performance in resolving the user's intent, including:
+    - What the user wanted
+    - How well the agent addressed it
+    - Any notable gaps or strengths
+F. Choose the single closest integer score (1,2,3,4 or 5) from the scoring rubric below.
+
+SCORING RUBRIC
+   5 - Excellent: Agent fully fulfills the intent with accurate, thorough, relevant action.
+   4 - Good: Intent mostly resolved; only minor, non-blocking gaps or imprecision.
+   3 - Adequate: Core of intent addressed but notable omissions, vagueness, or slips.
+   2 - Poor: Partial or incorrect resolution; answer unlikely to satisfy the intent.
+   1 - Very poor: Intent ignored or badly misunderstood; irrelevant or harmful response.
+
+
+EXAMPLES
+========
+
+### Score 5 - Example A (single-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  Give me a Python one-liner to reverse a string.
+
+AGENT_RESPONSE:
+  `s[::-1]` - this slice returns the string in reverse.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted a Python one-liner to reverse a string. Agent provided the correct code and a brief, helpful explanation, fully resolving the request with notable conciseness and accuracy.",
+  "score": 5
+}
+
+
+### Score 5 - Example B (multi-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  I need to cancel my 2 pm meeting with Dr. Lee this Friday.
+
+Agent turn 1:
+  To confirm, that's Friday at 14:00 with Dr. Lee in Meeting Room 2, right?
+
+User turn 2:
+  Yes, that one.
+
+AGENT_RESPONSE:
+  Done! The event has been removed from your calendar and a cancellation email was sent to Dr. Lee.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted to cancel a meeting. Agent first confirmed details, then successfully completed the cancellation and notified the attendee, fully resolving the intent with excellent clarity and thoroughness.",
+  "score": 5
+}
+
+
+### Score 4 - Example A (single-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  Translate 'carpe diem' to Portuguese and explain it in one sentence.
+
+AGENT_RESPONSE:
+  'Aproveita o dia.' Significa que deves aproveitar o momento presente.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted a Portuguese translation of 'carpe diem' and a one-sentence explanation. Agent provided an accurate translation and a concise explanation, mostly resolving the intent adequately.",
+  "score": 4
+}
+
+
+### Score 4 - Example B (multi-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  I need three book recommendations on deep work.
+
+Agent turn 1:
+  Sure—any particular genre?
+
+User turn 2:
+  Non-fiction, preferably practical.
+
+AGENT_RESPONSE:
+  1) *Deep Work* by Cal Newport, 2) *Indistractable* by Nir Eyal, 3) *Digital Minimalism* by Cal Newport. 
+  All are actionable non-fiction.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted three practical non-fiction book recommendations on deep work. Agent supplied three relevant titles, confirming they fit the criteria, mostly resolving the intent. Brief justifications for each pick would improve it.",
+  "score": 4
+}
+
+
+### Score 3 - Example A (single-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  Summarise the key plot points of *The Hobbit* in 5 bullet points.
+
+AGENT_RESPONSE:
+  • Bilbo meets Gandalf  • They find a ring  • They fight a dragon.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted a 5-bullet summary of *The Hobbit*'s plot. Agent provided only three bullets, capturing some main events but omitting several pivotal ones and failing to meet the requested count, thus only adequately addressing the intent.",
+  "score": 3
+}
+
+
+### Score 3 - Example B (multi-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  What's the weather in Tokyo tomorrow?
+
+Agent turn 1:
+  Would you like it in Celsius or Fahrenheit?
+
+User turn 2:
+  Celsius, please.
+
+AGENT_RESPONSE:
+  Looks mostly sunny with a high around 24 °C and a 10 % chance of light showers.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted tomorrow's weather in Tokyo (Celsius). Agent provided temperature and rain chance but omitted wind information, offering an adequate but incomplete resolution of the intent.",
+  "score": 3
+}
+
+
+### Score 2 - Example A (single-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  "Delete my last email draft."
+
+AGENT_RESPONSE:
+  "Here's how to compose a new email…"
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted to delete their last email draft. Agent misunderstood the request, offering irrelevant guidance on composing a new email instead of performing the deletion, resulting in a poor resolution.",
+  "score": 2
+}
+
+
+### Score 2 - Example B (multi-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  "Calculate 15 % of 80."
+
+Agent turn 1:
+  "Do you need the exact number or a rounded estimate?"
+
+User turn 2:
+  "Exact, please."
+
+AGENT_RESPONSE:
+  "It's 10.5."
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted an exact calculation of 15% of 80. Agent provided an incorrect result (10.5 instead of 12), leading to a poor resolution of the user's intent.",
+  "score": 2
+}
+
+
+### Score 1 - Example A (single-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  "Mute notifications for the next hour."
+
+AGENT_RESPONSE:
+  "Here's a cake recipe! 🍰"
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted to mute notifications for an hour. Agent responded with a completely irrelevant cake recipe, making no attempt to address the user's intent, resulting in a very poor resolution.",
+  "score": 1
+}
+
+
+### Score 1 - Example B (multi-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  "Set an alarm for 6 am tomorrow."
+
+Agent turn 1:
+  "Why not sleep in? Alarms are overrated."
+
+User turn 2:
+  "I really need the alarm—please set it."
+
+AGENT_RESPONSE:
+  "Alarms are pointless; wake up whenever."
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted to set an alarm for 6 am. Agent was dismissive and refused to perform the requested action, completely failing to resolve the user's intent, leading to a very poor resolution.",
+  "score": 1
+}
+