azure-ai-evaluation 1.0.0b2__py3-none-any.whl → 1.13.3__py3-none-any.whl

azure/ai/evaluation/_evaluators/_intent_resolution/intent_resolution.prompty

@@ -0,0 +1,275 @@
+---
+name: Intent Resolution Evaluator
+description: Evaluates whether user intent was identified and correctly resolved
+model:
+  api: chat
+  parameters:
+    temperature: 0.0
+    max_tokens: 800
+    top_p: 1.0
+    presence_penalty: 0
+    frequency_penalty: 0
+    response_format:
+      type: json_object
+
+inputs:
+  query:
+    type: string
+  response:
+    type: string
+  tool_definitions:
+    type: string
+    optional: true
+    default: "[]"
+---
+system:
+You are an expert in evaluating the quality of a AGENT_RESPONSE from an intelligent assistant based on provided definition and CONVERSATION_HISTORY.
+
+user:
+ROLE
+====
+You are Intent-Resolution-Judge, an impartial grader that scores how well an AI agent *resolved* the user's intent in a multi-turn conversation.
+You are NOT grading intent recognition. Assume the agent has understood the intent that is expressed; you only judge whether the reply satisfies or completes that intent.
+
+
+INPUT
+=====
+CONVERSATION_HISTORY: {{query}}
+AGENT_RESPONSE: {{response}}
+
+CONVERSATION_HISTORY is the full dialogue between the user and the agent up to the user's latest message.
+AGENT_RESPONSE is the agent reply to that latest user message.
+
+
+TASK
+====
+Output a JSON object with:
+    1) a concise explanation of 15-60 words that summarizes the agent's performance in resolving the user's intent
+    2) an integer score from 1 (very poor) to 5 (excellent) on how well the agent resolved the user's intent.
+
+The explanation should always precede the score and should clearly justify the score based on the agent's performance in resolving the user's intent.
+Response format exactly as follows:
+
+{
+  "explanation": "<15-60 words>",
+  "score": <1-5>
+}
+
+EVALUATION STEPS
+================
+
+A. Identify the expressed intent in the final user turn (look at the full conversation_history for context if necessary).
+B. Check resolution - Does the agent's reply, in AGENT_RESPONSE, actually complete or satisfy that intent? 
+    - If the agent's response is a direct answer, does it fully address the user's request?
+    - If the agent's response is an action (like scheduling, deleting, etc.), does it confirm completion of that action?
+    - If the agent's response is a clarification or follow-up question, does it lead towards fulfilling the intent?
+    - If the agent response is empty or irrelevant, it does not resolve the intent and should be scored accordingly.
+C. Verify correctness & completeness of the resolution.
+D. Weigh impact - Minor style issues matter only for tie-breaking; resolution quality dominates.
+E. Write a concise explanation of 15-60 words that summarizes the agent's performance in resolving the user's intent, including:
+    - What the user wanted
+    - How well the agent addressed it
+    - Any notable gaps or strengths
+F. Choose the single closest integer score (1,2,3,4 or 5) from the scoring rubric below.
+
+SCORING RUBRIC
+   5 - Excellent: Agent fully fulfills the intent with accurate, thorough, relevant action.
+   4 - Good: Intent mostly resolved; only minor, non-blocking gaps or imprecision.
+   3 - Adequate: Core of intent addressed but notable omissions, vagueness, or slips.
+   2 - Poor: Partial or incorrect resolution; answer unlikely to satisfy the intent.
+   1 - Very poor: Intent ignored or badly misunderstood; irrelevant or harmful response.
+
+
+EXAMPLES
+========
+
+### Score 5 - Example A (single-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  Give me a Python one-liner to reverse a string.
+
+AGENT_RESPONSE:
+  `s[::-1]` - this slice returns the string in reverse.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted a Python one-liner to reverse a string. Agent provided the correct code and a brief, helpful explanation, fully resolving the request with notable conciseness and accuracy.",
+  "score": 5
+}
+
+
+### Score 5 - Example B (multi-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  I need to cancel my 2 pm meeting with Dr. Lee this Friday.
+
+Agent turn 1:
+  To confirm, that's Friday at 14:00 with Dr. Lee in Meeting Room 2, right?
+
+User turn 2:
+  Yes, that one.
+
+AGENT_RESPONSE:
+  Done! The event has been removed from your calendar and a cancellation email was sent to Dr. Lee.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted to cancel a meeting. Agent first confirmed details, then successfully completed the cancellation and notified the attendee, fully resolving the intent with excellent clarity and thoroughness.",
+  "score": 5
+}
+
+
+### Score 4 - Example A (single-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  Translate 'carpe diem' to Portuguese and explain it in one sentence.
+
+AGENT_RESPONSE:
+  'Aproveita o dia.' Significa que deves aproveitar o momento presente.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted a Portuguese translation of 'carpe diem' and a one-sentence explanation. Agent provided an accurate translation and a concise explanation, mostly resolving the intent adequately.",
+  "score": 4
+}
+
+
+### Score 4 - Example B (multi-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  I need three book recommendations on deep work.
+
+Agent turn 1:
+  Sure—any particular genre?
+
+User turn 2:
+  Non-fiction, preferably practical.
+
+AGENT_RESPONSE:
+  1) *Deep Work* by Cal Newport, 2) *Indistractable* by Nir Eyal, 3) *Digital Minimalism* by Cal Newport. 
+  All are actionable non-fiction.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted three practical non-fiction book recommendations on deep work. Agent supplied three relevant titles, confirming they fit the criteria, mostly resolving the intent. Brief justifications for each pick would improve it.",
+  "score": 4
+}
+
+
+### Score 3 - Example A (single-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  Summarise the key plot points of *The Hobbit* in 5 bullet points.
+
+AGENT_RESPONSE:
+  • Bilbo meets Gandalf  • They find a ring  • They fight a dragon.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted a 5-bullet summary of *The Hobbit*'s plot. Agent provided only three bullets, capturing some main events but omitting several pivotal ones and failing to meet the requested count, thus only adequately addressing the intent.",
+  "score": 3
+}
+
+
+### Score 3 - Example B (multi-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  What's the weather in Tokyo tomorrow?
+
+Agent turn 1:
+  Would you like it in Celsius or Fahrenheit?
+
+User turn 2:
+  Celsius, please.
+
+AGENT_RESPONSE:
+  Looks mostly sunny with a high around 24 °C and a 10 % chance of light showers.
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted tomorrow's weather in Tokyo (Celsius). Agent provided temperature and rain chance but omitted wind information, offering an adequate but incomplete resolution of the intent.",
+  "score": 3
+}
+
+
+### Score 2 - Example A (single-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  "Delete my last email draft."
+
+AGENT_RESPONSE:
+  "Here's how to compose a new email…"
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted to delete their last email draft. Agent misunderstood the request, offering irrelevant guidance on composing a new email instead of performing the deletion, resulting in a poor resolution.",
+  "score": 2
+}
+
+
+### Score 2 - Example B (multi-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  "Calculate 15 % of 80."
+
+Agent turn 1:
+  "Do you need the exact number or a rounded estimate?"
+
+User turn 2:
+  "Exact, please."
+
+AGENT_RESPONSE:
+  "It's 10.5."
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted an exact calculation of 15% of 80. Agent provided an incorrect result (10.5 instead of 12), leading to a poor resolution of the user's intent.",
+  "score": 2
+}
+
+
+### Score 1 - Example A (single-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  "Mute notifications for the next hour."
+
+AGENT_RESPONSE:
+  "Here's a cake recipe! 🍰"
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted to mute notifications for an hour. Agent responded with a completely irrelevant cake recipe, making no attempt to address the user's intent, resulting in a very poor resolution.",
+  "score": 1
+}
+
+
+### Score 1 - Example B (multi-turn)
+
+CONVERSATION_HISTORY:
+User turn 1:
+  "Set an alarm for 6 am tomorrow."
+
+Agent turn 1:
+  "Why not sleep in? Alarms are overrated."
+
+User turn 2:
+  "I really need the alarm—please set it."
+
+AGENT_RESPONSE:
+  "Alarms are pointless; wake up whenever."
+
+EXPECTED OUTPUT:
+{
+  "explanation": "User wanted to set an alarm for 6 am. Agent was dismissive and refused to perform the requested action, completely failing to resolve the user's intent, leading to a very poor resolution.",
+  "score": 1
+}
+

azure/ai/evaluation/_evaluators/_meteor/_meteor.py

@@ -1,44 +1,19 @@
 # ---------------------------------------------------------
 # Copyright (c) Microsoft Corporation. All rights reserved.
 # ---------------------------------------------------------
-import nltk
-from nltk.translate.meteor_score import meteor_score
-from promptflow._utils.async_utils import async_run_allowing_running_loop
-
-from azure.ai.evaluation._common.utils import nltk_tokenize
-
-
-class _AsyncMeteorScoreEvaluator:
-    def __init__(self, alpha: float = 0.9, beta: float = 3.0, gamma: float = 0.5):
-        self._alpha = alpha
-        self._beta = beta
-        self._gamma = gamma
+from typing import Dict
 
-        try:
-            nltk.find("corpora/wordnet.zip")
-        except LookupError:
-            nltk.download("wordnet")
-
-    async def __call__(self, *, ground_truth: str, response: str, **kwargs):
-        reference_tokens = nltk_tokenize(ground_truth)
-        hypothesis_tokens = nltk_tokenize(response)
-
-        score = meteor_score(
-            [reference_tokens],
-            hypothesis_tokens,
-            alpha=self._alpha,
-            beta=self._beta,
-            gamma=self._gamma,
-        )
+from nltk.translate.meteor_score import meteor_score
+from typing_extensions import overload, override
 
-        return {
-            "meteor_score": score,
-        }
+from azure.ai.evaluation._common.utils import nltk_tokenize, ensure_nltk_data_downloaded
+from azure.ai.evaluation._evaluators._common import EvaluatorBase
+from azure.ai.evaluation._constants import EVALUATION_PASS_FAIL_MAPPING
 
 
-class MeteorScoreEvaluator:
+class MeteorScoreEvaluator(EvaluatorBase):
     """
-    Evaluator that computes the METEOR Score between two strings.
+    Calculates the METEOR score for a given response and ground truth.
 
     The METEOR (Metric for Evaluation of Translation with Explicit Ordering) score grader evaluates generated text by
     comparing it to reference texts, focusing on precision, recall, and content alignment. It addresses limitations of
@@ -46,39 +21,98 @@ class MeteorScoreEvaluator:
     word stems to more accurately capture meaning and language variations. In addition to machine translation and
     text summarization, paraphrase detection is an optimal use case for the METEOR score.
 
+    Use the METEOR score when you want a more linguistically informed evaluation metric that captures not only
+    n-gram overlap but also accounts for synonyms, stemming, and word order. This is particularly useful for evaluating
+    tasks like machine translation, text summarization, and text generation.
+
+    The METEOR score ranges from 0 to 1, with 1 indicating a perfect match.
+
     :param alpha: The METEOR score alpha parameter. Default is 0.9.
     :type alpha: float
     :param beta: The METEOR score beta parameter. Default is 3.0.
     :type beta: float
     :param gamma: The METEOR score gamma parameter. Default is 0.5.
     :type gamma: float
+    :param threshold: The threshold for the METEOR score evaluator. Default is 0.5.
+    :type threshold: float
+
+    .. admonition:: Example:
+
+        .. literalinclude:: ../samples/evaluation_samples_evaluate.py
+            :start-after: [START meteor_score_evaluator]
+            :end-before: [END meteor_score_evaluator]
+            :language: python
+            :dedent: 8
+            :caption: Initialize and call a MeteorScoreEvaluator with alpha of 0.8.
+
+    .. admonition:: Example using Azure AI Project URL:
+
+        .. literalinclude:: ../samples/evaluation_samples_evaluate_fdp.py
+            :start-after: [START meteor_score_evaluator]
+            :end-before: [END meteor_score_evaluator]
+            :language: python
+            :dedent: 8
+            :caption: Initialize and call MeteorScoreEvaluator using Azure AI Project URL in the following format
+                https://{resource_name}.services.ai.azure.com/api/projects/{project_name}
+
+    .. admonition:: Example with Threshold:
+
+        .. literalinclude:: ../samples/evaluation_samples_threshold.py
+            :start-after: [START threshold_meteor_score_evaluator]
+            :end-before: [END threshold_meteor_score_evaluator]
+            :language: python
+            :dedent: 8
+            :caption: Initialize with threshold and call a MeteorScoreEvaluator.
+    """
 
-    **Usage**
-
-    .. code-block:: python
+    id = "azureai://built-in/evaluators/meteor_score"
+    """Evaluator identifier, experimental and to be used only with evaluation in cloud."""
 
-        eval_fn = MeteorScoreEvaluator(
-            alpha=0.9,
-            beta=3.0,
-            gamma=0.5
+    @override
+    def __init__(self, alpha: float = 0.9, beta: float = 3.0, gamma: float = 0.5, *, threshold: float = 0.5):
+        self._alpha = alpha
+        self._beta = beta
+        self._gamma = gamma
+        ensure_nltk_data_downloaded()
+        self._threshold = threshold
+        self._higher_is_better = True
+        super().__init__(threshold=threshold, _higher_is_better=self._higher_is_better)
+
+    @override
+    async def _do_eval(self, eval_input: Dict) -> Dict[str, float]:
+        """Produce a meteor score evaluation result.
+
+        :param eval_input: The input to the evaluation function.
+        :type eval_input: Dict
+        :return: The evaluation result.
+        :rtype: Dict
+        """
+        ground_truth = eval_input["ground_truth"]
+        response = eval_input["response"]
+        reference_tokens = nltk_tokenize(ground_truth)
+        hypothesis_tokens = nltk_tokenize(response)
+        score = meteor_score(
+            [reference_tokens],
+            hypothesis_tokens,
+            alpha=self._alpha,
+            beta=self._beta,
+            gamma=self._gamma,
         )
-        result = eval_fn(
-            response="Tokyo is the capital of Japan.",
-            ground_truth="The capital of Japan is Tokyo.")
-
-    **Output format**
-
-    .. code-block:: python
-
-        {
-            "meteor_score": 0.62
+        binary_result = False
+        if self._higher_is_better:
+            if score >= self._threshold:
+                binary_result = True
+        else:
+            if score <= self._threshold:
+                binary_result = True
+        return {
+            "meteor_score": score,
+            "meteor_result": EVALUATION_PASS_FAIL_MAPPING[binary_result],
+            "meteor_threshold": self._threshold,
         }
-    """
 
-    def __init__(self, alpha: float = 0.9, beta: float = 3.0, gamma: float = 0.5):
-        self._async_evaluator = _AsyncMeteorScoreEvaluator(alpha=alpha, beta=beta, gamma=gamma)
-
-    def __call__(self, *, ground_truth: str, response: str, **kwargs):
+    @overload  # type: ignore
+    def __call__(self, *, ground_truth: str, response: str) -> Dict[str, float]:
         """
         Evaluate the METEOR score between the response and the ground truth.
 
@@ -87,11 +121,23 @@ class MeteorScoreEvaluator:
         :keyword ground_truth: The ground truth to be compared against.
         :paramtype ground_truth: str
         :return: The METEOR score.
-        :rtype: dict
+        :rtype: Dict[str, float]
         """
-        return async_run_allowing_running_loop(
-            self._async_evaluator, ground_truth=ground_truth, response=response, **kwargs
-        )
 
-    def _to_async(self):
-        return self._async_evaluator
+    @override
+    def __call__(  # pylint: disable=docstring-missing-param
+        self,
+        *args,
+        **kwargs,
+    ):
+        """
+        Evaluate the METEOR score between the response and the ground truth.
+
+        :keyword response: The response to be evaluated.
+        :paramtype response: str
+        :keyword ground_truth: The ground truth to be compared against.
+        :paramtype ground_truth: str
+        :return: The METEOR score.
+        :rtype: Dict[str, float]
+        """
+        return super().__call__(*args, **kwargs)

azure/ai/evaluation/_evaluators/_protected_material/_protected_material.py

@@ -1,104 +1,131 @@
 # ---------------------------------------------------------
 # Copyright (c) Microsoft Corporation. All rights reserved.
 # ---------------------------------------------------------
-from promptflow._utils.async_utils import async_run_allowing_running_loop
 
-from azure.ai.evaluation._common.constants import EvaluationMetrics
-from azure.ai.evaluation._common.rai_service import evaluate_with_rai_service
-from azure.ai.evaluation._exceptions import ErrorBlame, ErrorCategory, ErrorTarget, EvaluationException
+from typing import Dict, List, Optional, Union
 
+from typing_extensions import overload, override
 
-class _AsyncProtectedMaterialEvaluator:
-    def __init__(self, azure_ai_project: dict, credential=None):
-        self._azure_ai_project = azure_ai_project
-        self._credential = credential
+from azure.ai.evaluation._common._experimental import experimental
+from azure.ai.evaluation._common.constants import EvaluationMetrics
+from azure.ai.evaluation._evaluators._common import RaiServiceEvaluatorBase
+from azure.ai.evaluation._model_configurations import Conversation
 
-    async def __call__(self, *, query: str, response: str, **kwargs):
-        """
-        Evaluates content according to this evaluator's metric.
 
-        :keyword query: The query to be evaluated.
-        :paramtype query: str
-        :keyword response: The response to be evaluated.
-        :paramtype response: str
-        :return: The evaluation score computation based on the Content Safety metric (self.metric).
-        :rtype: Any
-        """
-        # Validate inputs
-        # Raises value error if failed, so execution alone signifies success.
-        if not (query and query.strip() and query != "None") or not (
-            response and response.strip() and response != "None"
-        ):
-            msg = "Both 'query' and 'response' must be non-empty strings."
-            raise EvaluationException(
-                message=msg,
-                internal_message=msg,
-                error_category=ErrorCategory.MISSING_FIELD,
-                error_blame=ErrorBlame.USER_ERROR,
-                error_target=ErrorTarget.PROTECTED_MATERIAL_EVALUATOR,
-            )
-
-        # Run score computation based on supplied metric.
-        result = await evaluate_with_rai_service(
-            metric_name=EvaluationMetrics.PROTECTED_MATERIAL,
-            query=query,
-            response=response,
-            project_scope=self._azure_ai_project,
-            credential=self._credential,
-        )
-        return result
+@experimental
+class ProtectedMaterialEvaluator(RaiServiceEvaluatorBase[Union[str, bool]]):
+    """
+    Evaluates the protected material score for a given query and response or a multi-turn conversation, with reasoning.
 
+    Protected material is any text that is under copyright, including song lyrics, recipes, and articles. Protected
+    material evaluation leverages the Azure AI Content Safety Protected Material for Text service to perform the
+    classification.
 
-class ProtectedMaterialEvaluator:
-    """
-    Initialize a protected material evaluator to detect whether protected material
-    is present in your AI system's response. Outputs True or False with AI-generated reasoning.
+    The protected material score is a boolean value, where True indicates that protected material was detected.
 
-    :param azure_ai_project: The scope of the Azure AI project.
-        It contains subscription id, resource group, and project name.
-    :type azure_ai_project: ~azure.ai.evaluation.AzureAIProject
-    :param credential: The credential for connecting to Azure AI project.
+    :param credential: The credential required for connecting to the Azure AI project.
     :type credential: ~azure.core.credentials.TokenCredential
-    :return: Whether or not protected material was found in the response, with AI-generated reasoning.
-    :rtype: Dict[str, str]
+    :param azure_ai_project: The Azure AI project, which can either be a string representing the project endpoint
+        or an instance of AzureAIProject. It contains subscription id, resource group, and project name.
+    :type azure_ai_project: Union[str, ~azure.ai.evaluation.AzureAIProject]
 
-    **Usage**
+    .. admonition:: Example:
 
-    .. code-block:: python
+        .. literalinclude:: ../samples/evaluation_samples_evaluate.py
+            :start-after: [START protected_material_evaluator]
+            :end-before: [END protected_material_evaluator]
+            :language: python
+            :dedent: 8
+            :caption: Initialize and call a ProtectedMaterialEvaluator.
 
-        azure_ai_project = {
-            "subscription_id": "<subscription_id>",
-            "resource_group_name": "<resource_group_name>",
-            "project_name": "<project_name>",
-        }
-        eval_fn = ProtectedMaterialEvaluator(azure_ai_project)
-        result = eval_fn(query="What is the capital of France?", response="Paris.")
+    .. admonition:: Example using Azure AI Project URL:
 
-    **Output format**
+        .. literalinclude:: ../samples/evaluation_samples_evaluate_fdp.py
+            :start-after: [START protected_material_evaluator]
+            :end-before: [END protected_material_evaluator]
+            :language: python
+            :dedent: 8
+            :caption: Initialize and call ProtectedMaterialEvaluator using Azure AI Project URL in the following format
+                https://{resource_name}.services.ai.azure.com/api/projects/{project_name}
 
-    .. code-block:: python
-
-        {
-            "protected_material_label": "False",
-            "protected_material_reason": "This query does not contain any protected material."
-        }
     """
 
-    def __init__(self, azure_ai_project: dict, credential=None):
-        self._async_evaluator = _AsyncProtectedMaterialEvaluator(azure_ai_project, credential)
+    id = "azureai://built-in/evaluators/protected_material"
+    """Evaluator identifier, experimental and to be used only with evaluation in cloud."""
+    _OPTIONAL_PARAMS = ["query"]
+
+    @override
+    def __init__(
+        self,
+        credential,
+        azure_ai_project,
+        **kwargs,
+    ):
+        # Set default for evaluate_query if not provided
+        if "evaluate_query" not in kwargs:
+            kwargs["evaluate_query"] = True
+
+        super().__init__(
+            eval_metric=EvaluationMetrics.PROTECTED_MATERIAL,
+            azure_ai_project=azure_ai_project,
+            credential=credential,
+            **kwargs,
+        )
 
-    def __call__(self, *, query: str, response: str, **kwargs):
-        """
-        Evaluates protected material content.
+    @overload
+    def __call__(
+        self,
+        *,
+        query: str,
+        response: str,
+    ) -> Dict[str, Union[str, bool]]:
+        """Evaluate a given query/response pair for protected material
 
         :keyword query: The query to be evaluated.
         :paramtype query: str
         :keyword response: The response to be evaluated.
         :paramtype response: str
-        :return: A dictionary containing a boolean label and reasoning.
-        :rtype: dict
+        :return: The protected material score.
+        :rtype: Dict[str, Union[str, bool]]
+        """
+
+    @overload
+    def __call__(
+        self,
+        *,
+        conversation: Conversation,
+    ) -> Dict[str, Union[float, Dict[str, List[Union[str, bool]]]]]:
+        """Evaluate a conversation for protected material
+
+        :keyword conversation: The conversation to evaluate. Expected to contain a list of conversation turns under the
+            key "messages", and potentially a global context under the key "context". Conversation turns are expected
+            to be dictionaries with keys "content", "role", and possibly "context".
+        :paramtype conversation: Optional[~azure.ai.evaluation.Conversation]
+        :return: The protected material score.
+        :rtype: Dict[str, Union[str, bool, Dict[str, List[Union[str, bool]]]]]
         """
-        return async_run_allowing_running_loop(self._async_evaluator, query=query, response=response, **kwargs)
 
-    def _to_async(self):
-        return self._async_evaluator
+    @override
+    def __call__(
+        self,
+        *,
+        query: Optional[str] = None,
+        response: Optional[str] = None,
+        conversation=None,
+        **kwargs,
+    ):
+        """
+        Evaluate if protected material is present in your AI system's response.
+
+        :keyword query: The query to be evaluated.
+        :paramtype query: Optional[str]
+        :keyword response: The response to be evaluated.
+        :paramtype response: Optional[str]
+        :keyword conversation: The conversation to evaluate. Expected to contain a list of conversation turns under the
+            key "messages". Conversation turns are expected
+            to be dictionaries with keys "content" and "role".
+        :paramtype conversation: Optional[~azure.ai.evaluation.Conversation]
+        :return: The fluency score.
+        :rtype: Union[Dict[str, Union[str, bool]], Dict[str, Union[float, Dict[str, List[Union[str, bool]]]]]]
+        """
+        return super().__call__(query=query, response=response, conversation=conversation, **kwargs)