azure-ai-evaluation 1.12.0__py3-none-any.whl → 1.13.0__py3-none-any.whl

azure/ai/evaluation/_evaluators/_tool_output_utilization/tool_output_utilization.prompty

@@ -0,0 +1,221 @@
+---
+name: Tool Output Utilization Evaluator
+description: Binary evaluator that judges whether an agent correctly understands and *uses* the outputs returned by tools it invoked (APIs, search/retrieval, DB queries, etc.). This evaluator focuses ONLY on incorrect, missing, or fabricated uses of tool outputs — whether they are used in the final response to the user or reused as inputs to subsequent tool calls. It does NOT judge tool selection, correctness of new inputs, or general reasoning quality.
+model:
+  api: chat
+  parameters:
+    temperature: 0.0
+    max_tokens: 1500
+    top_p: 1.0
+    presence_penalty: 0
+    frequency_penalty: 0
+    response_format:
+      type: json_object
+
+inputs:
+  query:
+    type: string
+  response:
+    type: string
+  tool_definitions:
+    type: string
+---
+system:
+You are **Tool Output Utilization Judge**, an expert evaluator whose only task is to decide whether the AGENT correctly interpreted and *used* TOOL OUTPUTS whenproducing the RESPONSE.
+
+Key constraints:
+
+- **Focus exclusively** on uses of tool outputs. A "use" means any appearance or
+  incorporation of a prior tool output (from `query`) within the agent's `response`
+  — either as part of the textual content to the user or as a parameter inside a new tool call.
+- Do **not** judge whether the agent chose the right tool, made the right new call,
+  or used the correct input format. Those are evaluated separately.
+- Treat `query` as the authoritative source of all prior conversation.
+- Treat `response` as the agent's latest message, which may:
+  1. State facts that come from tool outputs.
+  2. Contain tool calls that reference or reuse prior tool outputs.
+- Use `tool_definitions` for contextual understanding of tool structures (fields, types, units, etc.).
+- Conservative rule: if any tool-derived information appears incorrectly used in RESPONSE, omitted when relevant, or fabricated, mark it as a fault.
+
+INPUT
+=====
+
+CONVERSATION_HISTORY: {{query}}
+AGENT_RESPONSE: {{response}}
+TOOL_DEFINITIONS: {{tool_definitions}}
+
+> `CONVERSATION_HISTORY` includes all prior turns and any tool results.  
+> `AGENT_RESPONSE` is the model's latest message.  
+> `TOOL_DEFINITIONS` describe the tool schemas used.
+
+user:
+ROLE
+====
+
+You are Tool Output Utilization Judge. Evaluate whether the RESPONSE correctly:
+
+- Reflects the factual content of prior tool outputs from `query`, and  
+- Reuses any of those tool outputs correctly when incorporating them into new tool calls or the textual response.
+
+TASK
+====
+
+Produce exactly one JSON object (and nothing else) with these keys in **this exact order**:
+
+1. `faulty_details`: array of strings — list only the faults found (empty array if none).  
+   Each entry can follow one of these formats:
+   - "claim -> MISMATCH (expected X, saw Y) mapped to tool_name.field_path"
+   - "claim -> FABRICATED (no supporting tool field)"
+   - "use -> FABRICATED (referenced value not found in prior tool outputs)"
+   - "use -> MISMATCH (expected X, used Y) mapped to tool_name.field_path"
+
+
+2. `reason`: short 1–2 sentence summary of why PASS or FAIL.
+3. `label`: string `"pass"` or `"fail"`.
+
+> Output must be valid JSON, all lowercase keys, no extra text or markdown.
+
+EVALUATION STEPS
+================
+
+1. Identify all **instances** in the RESPONSE where tool outputs are *used*:
+   - Either referenced in text (factual claims to the user), or  
+   - Reused as parameters in new tool calls.
+2. For each instance:
+   - Cross-check against the corresponding tool outputs in `query`.
+   - If the usage faithfully matches the tool output (exact or paraphrased) → OK.
+   - If the agent uses wrong values, wrong entities, incorrect transformations, or fabricates data → record as fault.
+3. Populate the JSON object:
+   - `faulty_details`: all detected issues (empty if none).
+   - `reason`: concise rationale.
+   - `label`: `"pass"` or `"fail"`.
+
+SCORING RULES
+=============
+
+- **PASS:** No faulty uses of tool outputs found (empty `faulty_details`) in the RESPONSE.
+- **FAIL:** Any misuse, fabrication, omission, or misinterpretation of a tool output,
+  including when a prior tool output is reused incorrectly in a new tool call in the RESPONSE.
+
+IMPLEMENTATION NOTES
+====================
+
+- Do NOT evaluate:
+  - The correctness of *which tool* was used.
+  - Whether new tool inputs are valid by themselves.
+  - Task success or completeness.
+- Your judgment concerns *only* whether previously returned tool outputs are
+  correctly understood and reused where they appear.
+- If multiple faulty uses exist, list all in `faulty_details`.
+- When uncertain whether a value use is correct, treat it as a fault and explain why.
+- If tool outputs are missing but the response claims to use them, that counts as a fabricated use.
+- If a tool fails, that is outside your scope; unless the response misuses or misreports the failed output.
+
+> [TOOL CALLS] and [TOOL RESULTS] are internal, user does not see them.
+
+EXAMPLES (few-shot — using the new JSON schema and key order)
+
+### Example 1 - PASS
+QUERY:
+User turn 1:
+  Can you transfer $500 from my checking to my savings account?
+
+Agent turn 1:
+  [TOOL_CALL] get_account_balances(user_id="USER456")
+  [TOOL_RESULT] {'accounts': [{'account_id': 'CHK001', 'type': 'checking', 'balance': 1250.75}, {'account_id': 'SAV001', 'type': 'savings', 'balance': 3400.20}]}
+  You have $1,250.75 in checking and $3,400.20 in savings. You have enough for the transfer.
+
+User turn 2:
+  Great, please go ahead.
+
+RESPONSE:
+[TOOL_CALL] transfer_funds(from_account="CHK001", to_account="SAV001", amount=500)
+[TOOL_RESULT] {'transaction_id':'TXN789123','status':'completed','from_account':'CHK001','to_account':'SAV001','amount':500,'new_balances':{'CHK001':750.75,'SAV001':3900.20}}
+Transfer completed successfully. Checking now has $750.75 and savings $3,900.20.
+
+EXPECTED JSON:
+{
+  "faulty_details": [],
+  "reason": "All tool-derived claims and uses in the response match the prior tool outputs correctly.",
+  "label": "pass",
+}
+
+
+### Example 2 - FAIL (unit misinterpretation)
+QUERY:
+User turn 1:
+  What's the current temperature in Rome?
+
+RESPONSE:
+[TOOL_CALL] weather_api(city="Rome")
+[TOOL_RESULT] {"city":"Rome","temp":28,"condition":"Sunny"}
+It’s currently 28°F and sunny in Rome.
+
+TOOL_DEFINITIONS:
+- weather_api: Gets current weather for a city and returns temperature in Celsius (inputs: city) (outputs: city, temp, condition)
+
+EXPECTED JSON:
+{
+  "faulty_details": [
+    "claim -> MISMATCH (expected 28°C, saw 28°F) mapped to weather_api.temp"
+  ],
+  "reason": "Agent incorrectly reported the temperature in Fahrenheit instead of Celsius as provided by the tool output.",
+  "label": "fail",
+}
+
+
+### Example 3 - FAIL (fabricated inventory claim)
+QUERY:
+User turn 1:
+  Can you check if item B123 is available?
+
+RESPONSE:
+[TOOL_CALL] inventory_api(item_id="B123")
+[TOOL_RESULT] {"item_id":"B123","qty":0,"eta":"2025-10-07"}
+The item is in stock till the 7th of October.
+
+EXPECTED JSON:
+{
+  "faulty_details": [
+    "claim -> FABRICATED (no supporting tool field; inventory_api.qty is 0, eta is future date)"
+  ],
+  "reason": "Agent fabricated a claim that the item is in stock, whereas the tool output indicates a quantity of 0 and misunderstood availability eta",
+  "label": "fail",
+}
+
+
+### Example 4 - FAIL (misuse of prior tool output in new tool call)
+QUERY:
+User turn 1:
+  Show my latest account transactions.
+
+Agent turn 1:
+  [TOOL_CALL] get_account_list(user_id="U123")
+  [TOOL_RESULT] {"accounts":[{"id":"CHK100","type":"checking"},{"id":"SAV200","type":"savings"}]}
+  You have two accounts: CHK100 and SAV200. Which one do you want transactions for?
+
+User turn 2:
+  Please get the transaction history for my checking account.
+
+RESPONSE:
+[TOOL_CALL] get_transactions(account_id="SAV200")
+[TOOL_RESULT] ...
+The latest transactions for your checking account are: ...
+
+EXPECTED JSON:
+{
+  "faulty_details": [
+    "use -> MISMATCH (expected CHK100, used SAV200) mapped to get_transactions.account_id"
+  ],
+  "reason": "Agent incorrectly used the savings account ID instead of the requested checking account ID, leading to a mismatch in the transactions reported.",
+  "label": "fail",
+}
+
+--
+END OF EXAMPLES
+
+FINAL NOTES:
+
+- Output must be exactly one JSON object and must follow the key order: `faulty_details`, `reason`, `label`.
+
+# Output

azure/ai/evaluation/_evaluators/_tool_selection/__init__.py

@@ -0,0 +1,9 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from ._tool_selection import _ToolSelectionEvaluator
+
+__all__ = [
+    "_ToolSelectionEvaluator",
+]

azure/ai/evaluation/_evaluators/_tool_selection/_tool_selection.py

@@ -0,0 +1,266 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+import os
+import logging
+import math
+from typing import Dict, List, Union, TypeVar, Optional
+from typing_extensions import overload, override
+from azure.ai.evaluation._evaluators._common import PromptyEvaluatorBase
+from azure.ai.evaluation._exceptions import (
+    ErrorBlame,
+    ErrorCategory,
+    ErrorTarget,
+    EvaluationException,
+)
+from ..._common.utils import check_score_is_valid, reformat_conversation_history
+from azure.ai.evaluation._common._experimental import experimental
+
+logger = logging.getLogger(__name__)
+
+
+@experimental
+class _ToolSelectionEvaluator(PromptyEvaluatorBase[Union[str, float]]):
+    """The Tool Selection evaluator assesses the appropriateness and efficiency of tool choices made by an AI agent by examining:
+        - Relevance of selected tools to the conversation.
+        - Completeness of tool selection according to task requirements.
+        - Efficiency in avoiding unnecessary or redundant tools.
+
+    The evaluator uses a binary scoring system:
+        - Score 0 (Fail): Tools selected are irrelevant, incorrect, or missing essential tools
+        - Score 1 (Pass): All needed tools are selected, even if there are redundant tools
+
+    This evaluation focuses on measuring whether the right tools were chosen for the task,
+    regardless of how those tools were executed or their parameter correctness.
+
+    :param model_config: Configuration for the Azure OpenAI model.
+    :type model_config: Union[~azure.ai.evaluation.AzureOpenAIModelConfiguration,
+        ~azure.ai.evaluation.OpenAIModelConfiguration]
+
+    .. admonition:: Example:
+
+        .. literalinclude:: ../samples/evaluation_samples_evaluate.py
+            :start-after: [START tool_selection_evaluator]
+            :end-before: [END tool_selection_evaluator]
+            :language: python
+            :dedent: 8
+            :caption: Initialize and call a _ToolSelectionEvaluator.
+
+    .. admonition:: Example using Azure AI Project URL:
+
+        .. literalinclude:: ../samples/evaluation_samples_evaluate_fdp.py
+            :start-after: [START tool_selection_evaluator]
+            :end-before: [END tool_selection_evaluator]
+            :language: python
+            :dedent: 8
+            :caption: Initialize and call _ToolSelectionEvaluator using Azure AI Project URL in the following format
+                https://{resource_name}.services.ai.azure.com/api/projects/{project_name}
+
+    .. note::
+
+        To align with our support of a diverse set of models, an output key without the `gpt_` prefix has been added.
+        To maintain backwards compatibility, the old key with the `gpt_` prefix is still be present in the output;
+        however, it is recommended to use the new key moving forward as the old key will be deprecated in the future.
+    """
+
+    _PROMPTY_FILE = "tool_selection.prompty"
+    _RESULT_KEY = "tool_selection"
+
+    _NO_TOOL_CALLS_MESSAGE = "No tool calls found in response or provided tool_calls."
+    _NO_TOOL_DEFINITIONS_MESSAGE = "Tool definitions must be provided."
+    _TOOL_DEFINITIONS_MISSING_MESSAGE = "Tool definitions for all tool calls must be provided."
+    _INVALID_SCORE_MESSAGE = "Tool selection score must be 0 or 1."
+
+    id = "azureai://built-in/evaluators/tool_selection"
+    """Evaluator identifier, experimental and to be used only with evaluation in cloud."""
+
+    @override
+    def __init__(self, model_config, *, threshold=1, credential=None, **kwargs):
+        current_dir = os.path.dirname(__file__)
+        prompty_path = os.path.join(current_dir, self._PROMPTY_FILE)
+        self.threshold = threshold
+        super().__init__(
+            model_config=model_config,
+            prompty_file=prompty_path,
+            result_key=self._RESULT_KEY,
+            threshold=1,
+            credential=credential,
+            **kwargs,
+        )
+
+    @override
+    def __call__(  # pylint: disable=docstring-missing-param
+        self,
+        *args,
+        **kwargs,
+    ):
+        """
+        Evaluate tool selection quality for a given query, tool definitions, and tool calls.
+
+        For detailed parameter types and return value documentation, see the class documentation.
+        """
+        return super().__call__(*args, **kwargs)
+
+    def _convert_kwargs_to_eval_input(self, **kwargs):
+        """Convert an arbitrary input into a list of inputs for evaluators.
+        It is assumed that evaluators generally make use of their inputs in one of two ways.
+        Either they receive a collection of keyname inputs that are all single values
+        (like a query and response), or they receive conversation that is a list of dictionary
+        values.
+
+        The self._singleton_inputs list assigned during initialization is used to find and extract
+        singleton keywords, and self._allow_conversation_input is used to determine if a conversation
+        is a valid input.
+
+        If both conversations and singletons are allowed, the function will raise an exception if both
+        are inputted.
+
+        This function must be overridden by child classes IF they need to both a conversation and
+        other inputs to be passed in.
+
+        :keyword kwargs: The inputs to convert.
+        :type kwargs: Dict
+        :return: A list of arbitrary values that are valid inputs for this evaluator's do_eval function.
+        :rtype: List
+        """
+        # Collect inputs
+        tool_calls = kwargs.get("tool_calls")
+        tool_definitions = kwargs.get("tool_definitions", [])  # Default to empty list
+        query = kwargs.get("query")
+        response = kwargs.get("response")
+
+        # Extract tool calls from response if not provided directly
+        if response:
+            parsed_tool_calls = self._parse_tools_from_response(response)
+            if parsed_tool_calls:
+                tool_calls = parsed_tool_calls
+
+        if not tool_calls:
+            return {"error_message": self._NO_TOOL_CALLS_MESSAGE}
+
+        if not isinstance(tool_calls, list):
+            tool_calls = [tool_calls]
+        if not isinstance(tool_definitions, list):
+            tool_definitions = [tool_definitions] if tool_definitions else []
+
+        try:
+            needed_tool_definitions = self._extract_needed_tool_definitions(
+                tool_calls, tool_definitions, ErrorTarget.TOOL_SELECTION_EVALUATOR
+            )
+        except EvaluationException as e:
+            # Check if this is because no tool definitions were provided at all
+            if len(tool_definitions) == 0:
+                return {"error_message": self._NO_TOOL_DEFINITIONS_MESSAGE}
+            else:
+                return {"error_message": self._TOOL_DEFINITIONS_MISSING_MESSAGE}
+
+        if len(needed_tool_definitions) == 0:
+            return {"error_message": self._NO_TOOL_DEFINITIONS_MESSAGE}
+
+        # Extract only tool names from tool calls, removing parameters and results
+        tool_names = self._extract_tool_names_from_calls(tool_calls)
+
+        return {
+            "query": query,
+            "tool_calls": tool_names,  # Only tool names, no parameters
+            "tool_definitions": needed_tool_definitions,
+        }
+
+    @override
+    async def _do_eval(self, eval_input: Dict) -> Dict[str, Union[float, str]]:
+        """Do Tool Selection evaluation.
+
+        :param eval_input: The input to the evaluator.
+        :type eval_input: Dict
+        :return: A dictionary containing the result of the evaluation.
+        :rtype: Dict[str, Union[str, float]]
+        """
+        # Format conversation history for cleaner evaluation
+        if "query" in eval_input:
+            eval_input["query"] = reformat_conversation_history(
+                eval_input["query"], logger, include_system_messages=True, include_tool_messages=True
+            )
+
+        # Call the LLM to evaluate
+        prompty_output_dict = await self._flow(timeout=self._LLM_CALL_TIMEOUT, **eval_input)
+        llm_output = prompty_output_dict.get("llm_output", {})
+
+        if isinstance(llm_output, dict):
+            score = llm_output.get("score", None)
+            if score not in [0, 1]:
+                raise EvaluationException(
+                    message=f"Invalid score value: {score}. Expected 0 or 1.",
+                    internal_message="Invalid score value.",
+                    category=ErrorCategory.FAILED_EXECUTION,
+                    blame=ErrorBlame.SYSTEM_ERROR,
+                )
+
+            # Format the output
+            explanation = llm_output.get("explanation", "")
+            score = int(score)  # Keep as int since it's binary (0 or 1)
+            score_result = "pass" if score == 1 else "fail"
+
+            # Add tool selection accuracy post-processing
+            details = llm_output.get("details", {})
+            if details:
+                tool_selection_accuracy = self._calculate_tool_selection_accuracy(details)
+                details["tool_selection_accuracy"] = tool_selection_accuracy
+
+            response_dict = {
+                self._result_key: score,
+                f"{self._result_key}_result": score_result,
+                f"{self._result_key}_threshold": self._threshold,
+                f"{self._result_key}_reason": explanation,
+                f"{self._result_key}_details": details,
+                f"{self._result_key}_prompt_tokens": prompty_output_dict.get("input_token_count", 0),
+                f"{self._result_key}_completion_tokens": prompty_output_dict.get("output_token_count", 0),
+                f"{self._result_key}_total_tokens": prompty_output_dict.get("total_token_count", 0),
+                f"{self._result_key}_finish_reason": prompty_output_dict.get("finish_reason", ""),
+                f"{self._result_key}_model": prompty_output_dict.get("model_id", ""),
+                f"{self._result_key}_sample_input": prompty_output_dict.get("sample_input", ""),
+                f"{self._result_key}_sample_output": prompty_output_dict.get("sample_output", ""),
+            }
+            return response_dict
+
+        else:
+            raise EvaluationException(
+                message="Tool selection evaluator returned invalid output.",
+                blame=ErrorBlame.SYSTEM_ERROR,
+                category=ErrorCategory.FAILED_EXECUTION,
+                target=ErrorTarget.TOOL_SELECTION_EVALUATOR,
+            )
+
+    async def _real_call(self, **kwargs):
+        """The asynchronous call where real end-to-end evaluation logic is performed.
+
+        :keyword kwargs: The inputs to evaluate.
+        :type kwargs: Dict
+        :return: The evaluation result.
+        :rtype: Union[DoEvalResult[T_EvalValue], AggregateResult[T_EvalValue]]
+        """
+        # Convert inputs into list of evaluable inputs.
+        eval_input = self._convert_kwargs_to_eval_input(**kwargs)
+        if isinstance(eval_input, dict) and eval_input.get("error_message"):
+            return self._not_applicable_result(eval_input.get("error_message"), 1)
+
+        result = await self._do_eval(eval_input)
+
+        return result
+
+    def _calculate_tool_selection_accuracy(self, details):
+        """Calculate tool selection accuracy from the evaluation details.
+
+        :param details: The details dictionary from the LLM evaluation output
+        :type details: Dict
+        :return: Tool selection accuracy as a percentage
+        :rtype: float
+        """
+        correct_tool_selections = details.get("correct_tool_selections", 0)
+        wrong_tool_selections = details.get("wrong_tool_selections", 0)
+        total_tools_called = correct_tool_selections + wrong_tool_selections
+
+        if total_tools_called > 0:
+            accuracy = (correct_tool_selections / total_tools_called) * 100
+            return round(accuracy, 2)
+        else:
+            return 100.0

azure/ai/evaluation/_evaluators/_tool_selection/tool_selection.prompty

@@ -0,0 +1,104 @@
+---
+name: Tool Selection
+description: Evaluates Tool Selection Accuracy and Efficiency for tools chosen by agent
+model:
+  api: chat
+  parameters:
+    temperature: 0.0
+    max_tokens: 3000
+    top_p: 1.0
+    presence_penalty: 0
+    frequency_penalty: 0
+    response_format:
+      type: json_object
+
+inputs:
+  query:
+    type: List
+  tool_calls:
+    type: List
+  tool_definitions:
+    type: Dict
+
+---
+system:
+# Instruction
+## Goal
+You are an expert in evaluating the quality of tool selection made by an AI agent. Your goal is to assess whether the agent chose the right tools, avoided unnecessary tools, and didn't miss essential tools needed to address the user's query.
+
+- **Definition**: You are given a definition of tool selection quality that is being evaluated to guide your score.
+- **Data**: Your input data include CONVERSATION, TOOL CALLS (names only), and TOOL DEFINITIONS.
+- **Tasks**: You will evaluate the tool selection quality based on their relevance, efficiency, and potential to help the agent resolve the user's query.
+
+user:
+# Definition
+**Tool Selection** refers to the appropriateness and efficiency of the TOOLS CHOSEN by an agent in response to a user's query within an ongoing CONVERSATION.
+
+# EVALUATION CRITERIA
+Evaluate based on these factors:
+
+1. **Tool Selection**: Are the selected tools appropriate and useful for addressing the user's query?
+2. **Tool Completeness**: Did the agent select all **ESSENTIAL** tools available in the tool definitions that are **REQUIRED** to address the core request of the query?
+3. **Tool Efficiency**: Did the agent avoid selecting unnecessary or redundant tools?
+4. **Scope Limitation**: ONLY evaluate tool selections in the "TOOL CALLS MADE BY AGENT" section.
+   - Tool calls in the CONVERSATION section are for context only
+   - Focus exclusively on the agent's response to the user's LAST query
+   - Use conversation history only to understand context and requirements
+
+**Success Criteria**: Tools should be relevant to help answer the query. The focus is on appropriate tool choice, and the selected tools' ability to address the user's query.
+
+# Ratings
+## [Tool Selection: 0] (Fail)
+**Definition:**
+The tool selection fails if any of the following conditions are met:
+- Tools selected are irrelevant or inappropriate for the user's query
+- Essential tools that are available in the TOOL DEFINITIONS and are needed to complete the task are not selected by the agent **AND** haven't been called previously in the conversation
+- The agent selected tools that show a fundamental misunderstanding of the task requirements
+
+**Examples:**
+- User asks for weather information -> Agent selects a file search tool, instead of the weather-fetching tool available in the tool definitions.
+- User asks to analyze sales data and generate a report -> Agent selects only a data retrieval tool but misses the report generation tool that is available **AND** no data retrieval was done previously in the conversation.
+- User asks for stock price analysis -> Agent selects multiple unrelated weather, news, and calendar tools but not the stock price tool.
+
+## [Tool Selection: 1] (Pass)
+**Definition:**
+The tool selection passes when all necessary tools have been selected to address the user's query (considering both current tool calls and previous calls in the conversation history), even if redundant or unnecessary tools were also selected. All essential tools must be available to the agent (either from current calls or previous calls), and no selected tools should be completely irrelevant.
+
+**Examples:**
+- User asks for weather forecast -> Agent selects weather tool (necessary) plus calendar and reminder tools (unnecessary but not harmful).
+- User asks to book a flight -> Agent selects flight search and booking tools (necessary) plus hotel booking tools (unnecessary, but the essential tool was called).
+- User asks to summarize a document -> Agent selects document reader, text summarizer, and language detector (all useful tools, even if some are not strictly necessary).
+- User asks for "projected spend next season" -> Agent selects `project_budget_over_days` (appropriate for projection) and has access to spending data from previous `get_spending_by_category` calls in the conversation. No need to re-call data retrieval tools.
+
+## Chain of Thought Structure
+Structure your reasoning as follows:
+1. **Understand the user's query**: Analyze what the user is asking for in their latest message and identify the specific operation type needed.
+2. **Review conversation history for existing data**: **CRITICALLY IMPORTANT** - Examine the CONVERSATION history to identify what tools have already been called and what data is already available to the agent. If a tool was called earlier in the conversation and provided the necessary data, the agent does NOT need to call it again.
+3. **Identify required tools**: Determine which tools from the available TOOL DEFINITIONS would be necessary to address the query, excluding any tools that have already been called in the conversation and provided the needed data.
+4. **Analyze selected tools**: Examine which tools the agent actually selected in the current response.
+5. **Evaluate appropriateness**: Assess if selected tools are relevant to the task and are the most appropriate tools available for the specific query type.
+6. **Check completeness**: Verify if any essential tools are missing from the current agent response, **BUT ONLY** count tools as missing if they haven't been called previously in the conversation AND their data is needed for the current query.
+7. **Assess efficiency**: Determine if any unnecessary or redundant tools were selected.
+8. **Determine the appropriate level**: Assign 0 (Fail) if tools are irrelevant, incorrect, missing essential tools that weren't previously called. Assign 1 (Pass) if all needed tools are selected (considering conversation history), even with some redundant additions.
+
+Note: You are only passed the names of the tools that have been called by the agent. You do not see the parameters that are passed to the tools nor the results of these tools, because your only concern for evaluation is the appropriateness and completeness of the tools selected by the agent, not the parameter correctness nor the result correctness.
+
+# Data
+CONVERSATION: {{query}}
+TOOL DEFINITIONS: {{tool_definitions}}
+TOOL CALLS MADE BY AGENT: {{tool_calls}}
+
+# Tasks
+## Please provide your evaluation for the tool selection in relation to the user's query based on the definitions and examples above.
+Your output should consist only of a JSON object with the following fields:
+  - explanation: an explanation of the score focusing on tool selection appropriateness, based on the Chain of Thought Structure.
+  - details: a dictionary that contains the following fields:
+        - correct_tool_selections: number of appropriate/relevant tools selected
+        - wrong_tool_selections: number of inappropriate/irrelevant tools selected
+        - excessive_tools_used: number of tools that were unnecessary or redundant
+        - excessive_tools_list: list of the tool names that were excessive
+        - missing_tools: number of essential tools that should have been called but weren't
+        - missing_tools_list: list of the tool names that should have been called but weren't
+  - score: an integer value of 0 or 1 that represents the tool selection quality (0 = Fail, 1 = Pass).
+
+# Output

azure/ai/evaluation/_evaluators/{_task_success → _tool_success}/__init__.py

@@ -2,6 +2,6 @@
 # Copyright (c) Microsoft Corporation. All rights reserved.
 # ---------------------------------------------------------
 
-from ._task_success import TaskSuccessEvaluator
+from ._tool_success import _ToolSuccessEvaluator
 
-__all__ = ["TaskSuccessEvaluator"]
+__all__ = ["_ToolSuccessEvaluator"]