azure-ai-evaluation 0.0.0b0__py3-none-any.whl → 1.0.0__py3-none-any.whl

azure/ai/evaluation/_evaluators/_coherence/coherence.prompty

@@ -0,0 +1,99 @@
+---
+name: Coherence
+description: Evaluates coherence score for QA scenario
+model:
+  api: chat
+  parameters:
+    temperature: 0.0
+    max_tokens: 800
+    top_p: 1.0
+    presence_penalty: 0
+    frequency_penalty: 0
+    response_format:
+      type: text
+
+inputs:
+  query:
+    type: string
+  response:
+    type: string
+
+---
+system:
+# Instruction
+## Goal
+### You are an expert in evaluating the quality of a RESPONSE from an intelligent system based on provided definition and data. Your goal will involve answering the questions below using the information provided.
+- **Definition**: You are given a definition of the communication trait that is being evaluated to help guide your Score.
+- **Data**: Your input data include a QUERY and a RESPONSE.
+- **Tasks**: To complete your evaluation you will be asked to evaluate the Data in different ways.
+
+user:
+# Definition
+**Coherence** refers to the logical and orderly presentation of ideas in a response, allowing the reader to easily follow and understand the writer's train of thought. A coherent answer directly addresses the question with clear connections between sentences and paragraphs, using appropriate transitions and a logical sequence of ideas.
+
+# Ratings
+## [Coherence: 1] (Incoherent Response)
+**Definition:** The response lacks coherence entirely. It consists of disjointed words or phrases that do not form complete or meaningful sentences. There is no logical connection to the question, making the response incomprehensible.
+
+**Examples:**
+  **Query:** What are the benefits of renewable energy?
+  **Response:** Wind sun green jump apple silence over.
+
+  **Query:** Explain the process of photosynthesis.
+  **Response:** Plants light water flying blue music.
+
+## [Coherence: 2] (Poorly Coherent Response)
+**Definition:** The response shows minimal coherence with fragmented sentences and limited connection to the question. It contains some relevant keywords but lacks logical structure and clear relationships between ideas, making the overall message difficult to understand.
+
+**Examples:**
+  **Query:** How does vaccination work?
+  **Response:** Vaccines protect disease. Immune system fight. Health better.
+
+  **Query:** Describe how a bill becomes a law.
+  **Response:** Idea proposed. Congress discuss vote. President signs.
+
+## [Coherence: 3] (Partially Coherent Response)
+**Definition:** The response partially addresses the question with some relevant information but exhibits issues in the logical flow and organization of ideas. Connections between sentences may be unclear or abrupt, requiring the reader to infer the links. The response may lack smooth transitions and may present ideas out of order.
+
+**Examples:**
+  **Query:** What causes earthquakes?
+  **Response:** Earthquakes happen when tectonic plates move suddenly. Energy builds up then releases. Ground shakes and can cause damage.
+
+  **Query:** Explain the importance of the water cycle.
+  **Response:** The water cycle moves water around Earth. Evaporation, then precipitation occurs. It supports life by distributing water.
+
+## [Coherence: 4] (Coherent Response)
+**Definition:** The response is coherent and effectively addresses the question. Ideas are logically organized with clear connections between sentences and paragraphs. Appropriate transitions are used to guide the reader through the response, which flows smoothly and is easy to follow.
+
+**Examples:**
+  **Query:** What is the water cycle and how does it work?
+  **Response:** The water cycle is the continuous movement of water on Earth through processes like evaporation, condensation, and precipitation. Water evaporates from bodies of water, forms clouds through condensation, and returns to the surface as precipitation. This cycle is essential for distributing water resources globally.
+
+  **Query:** Describe the role of mitochondria in cellular function.
+  **Response:** Mitochondria are organelles that produce energy for the cell. They convert nutrients into ATP through cellular respiration. This energy powers various cellular activities, making mitochondria vital for cell survival.
+
+## [Coherence: 5] (Highly Coherent Response)
+**Definition:** The response is exceptionally coherent, demonstrating sophisticated organization and flow. Ideas are presented in a logical and seamless manner, with excellent use of transitional phrases and cohesive devices. The connections between concepts are clear and enhance the reader's understanding. The response thoroughly addresses the question with clarity and precision.
+
+**Examples:**
+  **Query:** Analyze the economic impacts of climate change on coastal cities.
+  **Response:** Climate change significantly affects the economies of coastal cities through rising sea levels, increased flooding, and more intense storms. These environmental changes can damage infrastructure, disrupt businesses, and lead to costly repairs. For instance, frequent flooding can hinder transportation and commerce, while the threat of severe weather may deter investment and tourism. Consequently, cities may face increased expenses for disaster preparedness and mitigation efforts, straining municipal budgets and impacting economic growth.
+
+  **Query:** Discuss the significance of the Monroe Doctrine in shaping U.S. foreign policy.
+  **Response:** The Monroe Doctrine was a pivotal policy declared in 1823 that asserted U.S. opposition to European colonization in the Americas. By stating that any intervention by external powers in the Western Hemisphere would be viewed as a hostile act, it established the U.S. as a protector of the region. This doctrine shaped U.S. foreign policy by promoting isolation from European conflicts while justifying American influence and expansion in the hemisphere. Its long-term significance lies in its enduring influence on international relations and its role in defining the U.S. position in global affairs.
+
+
+# Data
+QUERY: {{query}}
+RESPONSE: {{response}}
+
+
+# Tasks
+## Please provide your assessment Score for the previous RESPONSE in relation to the QUERY based on the Definitions above. Your output should include the following information:
+- **ThoughtChain**: To improve the reasoning process, think step by step and include a step-by-step explanation of your thought process as you analyze the data based on the definitions. Keep it brief and start your ThoughtChain with "Let's think step by step:".
+- **Explanation**: a very short explanation of why you think the input Data should get that Score.
+- **Score**: based on your previous analysis, provide your Score. The Score you give MUST be a integer score (i.e., "1", "2"...) based on the levels of the definitions.
+
+
+## Please provide your answers between the tags: <S0>your chain of thoughts</S0>, <S1>your explanation</S1>, <S2>your Score</S2>.
+# Output

azure/ai/evaluation/_evaluators/_common/__init__.py

@@ -0,0 +1,13 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from ._base_eval import EvaluatorBase
+from ._base_prompty_eval import PromptyEvaluatorBase
+from ._base_rai_svc_eval import RaiServiceEvaluatorBase
+
+__all__ = [
+    "EvaluatorBase",
+    "PromptyEvaluatorBase",
+    "RaiServiceEvaluatorBase",
+]

azure/ai/evaluation/_evaluators/_common/_base_eval.py

@@ -0,0 +1,344 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+import inspect
+from abc import ABC, abstractmethod
+from typing import Any, Callable, Dict, Generic, List, TypedDict, TypeVar, Union, cast, final
+
+from promptflow._utils.async_utils import async_run_allowing_running_loop
+from typing_extensions import ParamSpec, TypeAlias, get_overloads
+
+from azure.ai.evaluation._common.math import list_mean
+from azure.ai.evaluation._exceptions import ErrorBlame, ErrorCategory, ErrorTarget, EvaluationException
+from azure.ai.evaluation._common.utils import remove_optional_singletons
+
+P = ParamSpec("P")
+T = TypeVar("T")
+T_EvalValue = TypeVar("T_EvalValue")
+
+
+class DerivedEvalInput(TypedDict, total=False):
+    """The eval input generated by EvaluatorBase._derive_conversation_starter."""
+
+    query: Dict[str, Any]
+    response: Dict[str, Any]
+    context: str
+
+
+AggregateResult: TypeAlias = Dict[str, Union[float, Dict[str, List[T]]]]
+"""TypeAlias that models the return value of EvaluatorBase._aggregate_results
+
+    .. code-block:: python
+
+    foo: AggregateResult[float] = {
+        "evaluation_per_turn": {
+            "coherence": [1.0, 2.0, 3.0]
+        },
+        "coherence": 2.0
+    }
+"""
+
+DoEvalResult: TypeAlias = Dict[str, T]
+"""TypeAlias that models the return value of EvaluatorBase._do_eval
+
+    .. code-block:: python
+
+    foo: DoEvalResult[float] = {
+        "coherence": 2.0
+    }
+"""
+
+
+# TODO exception target pass down?
+class EvaluatorBase(ABC, Generic[T_EvalValue]):
+    """Base class for all evaluators that are capable of accepting either a group of single values,
+    or conversation as input. All such evaluators need to implement two functions of their own:
+        - _convert_conversation_to_eval_input
+        - _do_eval
+
+    Additionally, __call__ should be overridden to reshape the function header as needed to produce more informative
+    documentation, although ideally the actual child implementation of __call__ should just amount to
+    'super().__init__()'.
+
+
+    :param not_singleton_inputs: A list of strings that represent the names of
+        inputs to the child evaluator's __call__ function that are NOT singleton inputs. By default, this
+        is ["conversation", "kwargs"].
+    :type not_singleton_inputs: List[str]
+    :param eval_last_turn: If True, only the last turn of the conversation will be evaluated. Default is False.
+    :type eval_last_turn: bool
+    """
+
+    # ~~~ METHODS THAT ALMOST ALWAYS NEED TO BE OVERRIDDEN BY CHILDREN~~~
+
+    # Make sure to call super().__init__() in the child class's __init__ method.
+    # pylint: disable=dangerous-default-value
+    def __init__(
+        self,
+        *,
+        not_singleton_inputs: List[str] = ["conversation", "kwargs"],
+        eval_last_turn: bool = False,
+    ):
+        self._not_singleton_inputs = not_singleton_inputs
+        self._eval_last_turn = eval_last_turn
+        self._singleton_inputs = self._derive_singleton_inputs()
+        self._async_evaluator = AsyncEvaluatorBase(self._real_call)
+
+    # This needs to be overridden just to change the function header into something more informative,
+    # and to be able to add a more specific docstring. The actual function contents should just be
+    # super().__call__(<inputs>)
+    def __call__(  # pylint: disable=docstring-missing-param
+        self,
+        *args,
+        **kwargs,
+    ) -> Union[DoEvalResult[T_EvalValue], AggregateResult[T_EvalValue]]:
+        """Evaluate a given input. This method serves as a wrapper and is meant to be overridden by child classes for
+        one main reason - to overwrite the method headers and docstring to include additional inputs as needed.
+        The actual behavior of this function shouldn't change beyond adding more inputs to the
+        async_run_allowing_running_loop call.
+
+        :keyword kwargs: A dictionary that contains inputs needed to evaluate a conversation.
+        :type kwargs: Dict
+        :return: The evaluation result
+        :rtype: Union[DoEvalResult[T_EvalValue], AggregateResult[T_EvalValue]]
+        """
+        return async_run_allowing_running_loop(self._async_evaluator, **kwargs)
+
+    @abstractmethod
+    async def _do_eval(self, eval_input: Any) -> DoEvalResult[T_EvalValue]:
+        """Evaluate the input and produce a response. Must be overridden to produce a functional evaluator.
+        In the default case, all required inputs are assumed to be within eval_input, as user-friendly
+        typing is handled above this function in favor of polymorphic simplicity. This function must be
+        asynchronous.
+
+        :param eval_input: Whatever inputs are needed for this evaluator to perform a single evaluation.
+        :type eval_input: Any
+        :return: A single evaluation result
+        :rtype: DoEvalResult[T_EvalValue]
+        """
+
+    # ~~~ METHODS THAT MIGHT NEED TO BE OVERRIDDEN BY CHILDREN~~~
+
+    def _derive_singleton_inputs(self) -> List[str]:
+        """Inspect the evaluator's __call__ function to determine what singleton inputs are expected
+        when the evaluator is being used in a non-conversation context.
+        By default, it's assumed that any input that is NOT kwargs or a conversation are singleton inputs.
+        Thankfully this works the way you'd hope, with the call_signature being based on the child
+        function's signature, not the parent's.
+
+        :return: A list of strings representing the names of singleton inputs.
+        :rtype: List[str]
+        """
+
+        overloads = get_overloads(self.__call__)
+        if not overloads:
+            call_signatures = [inspect.signature(self.__call__)]
+        else:
+            call_signatures = [inspect.signature(overload) for overload in overloads]
+        call_signature = inspect.signature(self.__call__)
+        singletons = []
+        for call_signature in call_signatures:
+            params = call_signature.parameters
+            if any(not_singleton_input in params for not_singleton_input in self._not_singleton_inputs):
+                continue
+            # exclude self since it is not a singleton input
+            singletons.extend([p for p in params if p != "self"])
+        return singletons
+
+    def _derive_conversation_converter(self) -> Callable[[Dict], List[DerivedEvalInput]]:
+        """Produce the function that will be used to convert conversations to a list of evaluable inputs.
+        This uses the inputs derived from the _derive_singleton_inputs function to determine which
+        aspects of a conversation ought to be extracted.
+
+        :return: The function that will be used to convert conversations to evaluable inputs.
+        :rtype: Callable
+        """
+        include_context = "context" in self._singleton_inputs
+        include_query = "query" in self._singleton_inputs
+        include_response = "response" in self._singleton_inputs
+
+        def converter(conversation: Dict) -> List[DerivedEvalInput]:
+            messages = cast(List[Dict[str, Any]], conversation["messages"])
+            global_context = conversation.get("context", None)
+            # Extract queries, responses from conversation
+            queries: List[Dict[str, Any]] = []
+            responses: List[Dict[str, Any]] = []
+
+            # Convert conversation slice into queries and responses.
+            # Assume that 'user' role is asking queries and 'assistant' role is responding.
+            if self._eval_last_turn and len(messages) > 1:
+                messages = messages[-2:]
+
+            for each_turn in messages:
+                role = each_turn["role"]
+                if role == "user":
+                    queries.append(each_turn)
+                elif role == "assistant":
+                    responses.append(each_turn)
+            # TODO complain if len(queries) != len(responses)?
+            eval_inputs = []
+            for query, response in zip(queries, responses):
+                context = {}
+                if include_context:
+                    query_context = query.get("context", None)
+                    response_context = response.get("context", None)
+                    if global_context:
+                        context["global_context"] = global_context
+                    if query_context and include_query:
+                        context["query_context"] = query_context
+                    if response_context and include_response:
+                        context["response_context"] = response_context
+
+                eval_input: DerivedEvalInput = {}
+                if include_query:
+                    eval_input["query"] = query.get("content", "")
+                if include_response:
+                    eval_input["response"] = response.get("content", "")
+                if include_context:
+                    eval_input["context"] = str(context)
+                eval_inputs.append(eval_input)
+            return eval_inputs
+
+        return converter
+
+    def _convert_kwargs_to_eval_input(self, **kwargs) -> Union[List[Dict], List[DerivedEvalInput]]:
+        """Convert an arbitrary input into a list of inputs for evaluators.
+        It is assumed that evaluators generally make use of their inputs in one of two ways.
+        Either they receive a collection of keyname inputs that are all single values
+        (like a query and response), or they receive conversation that iss a list of dictionary
+        values.
+
+        The self._singleton_inputs list assigned during initialization is used to find and extract
+        singleton keywords, and self._allow_converssation_input is used to determine if a conversation
+        is a valid input.
+
+        If both conversations and singletons are allowed, the function will raise an exception if both
+        are inputted.
+
+        This function must be overridden by child classes IF they need to both a conversation and
+        other inputs to be passed in.
+
+        :keyword kwargs: The inputs to convert.
+        :type kwargs: Dict
+        :return: A list of arbitrary values that are valid inputs for this evaluator's do_eval function.
+        :rtype: List
+        """
+
+        # Collect inputs
+        conversation = kwargs.get("conversation", None)
+        singletons = {}
+        if len(self._singleton_inputs) > 0:
+            singletons = {key: kwargs.get(key, None) for key in self._singleton_inputs}
+        # Check that both conversation and other inputs aren't set
+        if conversation is not None and any(singletons.values()):
+            msg = f"{type(self).__name__}: Cannot provide both 'conversation' and individual inputs at the same time."
+            raise EvaluationException(
+                message=msg,
+                blame=ErrorBlame.USER_ERROR,
+                category=ErrorCategory.INVALID_VALUE,
+                target=ErrorTarget.CONVERSATION,
+            )
+        # Handle Conversation
+        if conversation is not None:
+            return self._derive_conversation_converter()(conversation)
+        # Handle Singletons
+        required_singletons = remove_optional_singletons(self, singletons)
+        if all(value is not None for value in required_singletons.values()):
+            return [singletons]
+        # Missing input
+        msg = f"{type(self).__name__}: Either 'conversation' or individual inputs must be provided."
+        raise EvaluationException(
+            message=msg,
+            blame=ErrorBlame.USER_ERROR,
+            category=ErrorCategory.INVALID_VALUE,
+            target=ErrorTarget.CONVERSATION,
+        )
+
+    def _aggregate_results(self, per_turn_results: List[DoEvalResult[T_EvalValue]]) -> AggregateResult[T_EvalValue]:
+        """Aggregate the evaluation results of each conversation turn into a single result.
+
+        Exact implementation might need to vary slightly depending on the results produced.
+        Default behavior is to average the all number-based outputs.
+
+        :param per_turn_results: List of evaluation results for each turn in the conversation.
+        :type per_turn_results: List[Dict]
+        :return: A dictionary containing aggregated results, with numeric metrics having their
+        means as top-level values in the dictionary, and all original
+        values (including non-numerics) located in under the "evaluation_per_turn" key,
+        which each sub-key being a metric and each sub-value being a the list of that metric's
+        per-turn values.
+        :rtype: AggregateResult[T_EvalValue]
+        """
+
+        aggregated: Dict[str, Union[float, Dict[str, List[T_EvalValue]]]] = {}
+        evaluation_per_turn: Dict[str, List[T_EvalValue]] = {}
+
+        # Go over each turn, and rotate the results into a
+        # metric: List[values] format for the evals_per_turn dictionary.
+        for turn in per_turn_results:
+            for metric, value in turn.items():
+                if metric not in evaluation_per_turn:
+                    evaluation_per_turn[metric] = []
+                evaluation_per_turn[metric].append(value)
+
+        # Find and average all numeric values
+        for metric, values in evaluation_per_turn.items():
+            if all(isinstance(value, (int, float)) for value in values):
+                aggregated[metric] = list_mean(cast(List[Union[int, float]], values))
+        # Slap the per-turn results back in.
+        aggregated["evaluation_per_turn"] = evaluation_per_turn
+        return aggregated
+
+    async def _real_call(self, **kwargs) -> Union[DoEvalResult[T_EvalValue], AggregateResult[T_EvalValue]]:
+        """The asynchronous call where real end-to-end evaluation logic is performed.
+
+        :keyword kwargs: The inputs to evaluate.
+        :type kwargs: Dict
+        :return: The evaluation result.
+        :rtype: Union[DoEvalResult[T_EvalValue], AggregateResult[T_EvalValue]]
+        """
+        # Convert inputs into list of evaluable inputs.
+        eval_input_list = self._convert_kwargs_to_eval_input(**kwargs)
+        per_turn_results = []
+        # Evaluate all inputs.
+        for eval_input in eval_input_list:
+            per_turn_results.append(await self._do_eval(eval_input))
+        # Return results as-is if only one result was produced.
+
+        if len(per_turn_results) == 1:
+            return per_turn_results[0]
+        if len(per_turn_results) == 0:
+            return {}  # TODO raise something?
+        # Otherwise, aggregate results.
+        return self._aggregate_results(per_turn_results=per_turn_results)
+
+    @final
+    def _to_async(self) -> "AsyncEvaluatorBase":
+        return self._async_evaluator
+
+
+class AsyncEvaluatorBase:
+    """The asynchronous evaluator hidden underneath all evaluators. This makes generous use passing functions
+    to ensure that no one ever needs to extend or otherwise modify this class directly.
+    """
+
+    def __init__(self, real_call):  # DO NOT ADD TYPEHINT PROMPT FLOW WILL SCREAM AT YOU ABOUT META GENERATION
+        self._real_call = real_call
+
+    # Don't look at my shame. Nothing to see here....
+    # Oh, you're still here? Ok, the reason this has such a gross call signature and behavior is due
+    # to our broken async code not properly handling inputs; keyword arguments that aren't in the signature
+    # are just not passed into this function instead of ending up in kwargs.
+    # Since we want this to be relatively call-agnostic, we just account for every input that any children
+    # are known to throw at this, mash them into kwargs, and then pass them into the real call.
+    async def __call__(self, *, query=None, response=None, context=None, conversation=None, **kwargs):
+        if conversation is not None:
+            kwargs["conversation"] = conversation
+        if query is not None:
+            kwargs["query"] = query
+        if response is not None:
+            kwargs["response"] = response
+        if context is not None:
+            kwargs["context"] = context
+        return await self._real_call(**kwargs)

azure/ai/evaluation/_evaluators/_common/_base_prompty_eval.py

@@ -0,0 +1,88 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+import math
+import re
+from typing import Dict, TypeVar, Union
+
+from promptflow.core import AsyncPrompty
+from typing_extensions import override
+
+from azure.ai.evaluation._common.constants import PROMPT_BASED_REASON_EVALUATORS
+from ..._common.utils import construct_prompty_model_config, validate_model_config, parse_quality_evaluator_reason_score
+from . import EvaluatorBase
+
+try:
+    from ..._user_agent import USER_AGENT
+except ImportError:
+    USER_AGENT = "None"
+
+T = TypeVar("T")
+
+
+class PromptyEvaluatorBase(EvaluatorBase[T]):
+    """Base class for all evaluators that make use of context as an input. It's also assumed that such evaluators
+    make use of a prompty file, and return their results as a dictionary, with a single key-value pair
+    linking the result name to a float value (unless multi-turn evaluation occurs, in which case the
+    per-turn results are stored in a list under the key "evaluation_per_turn").
+
+    :param result_key: The key to use for the result of the evaluation. Single turn evaluations will return
+        a dictionary in the format {result_key: float}.
+    :type result_key: str
+    :param prompty_file: The path to the prompty file to use for evaluation.
+    :type prompty_file: str
+    :param model_config: The model configuration to use for evaluation.
+    :type model_config: Union[AzureOpenAIModelConfiguration, OpenAIModelConfiguration]
+    :param ignore_queries: If True, queries will be ignored in conversation evaluations. Default is False.
+        Useful since some evaluators of this format are response-only.
+    :type ignore_queries: bool
+    """
+
+    _LLM_CALL_TIMEOUT = 600
+    _DEFAULT_OPEN_API_VERSION = "2024-02-15-preview"
+
+    def __init__(self, *, result_key: str, prompty_file: str, model_config: dict, eval_last_turn: bool = False):
+        self._result_key = result_key
+        self._prompty_file = prompty_file
+        super().__init__(eval_last_turn=eval_last_turn)
+
+        prompty_model_config = construct_prompty_model_config(
+            validate_model_config(model_config),
+            self._DEFAULT_OPEN_API_VERSION,
+            USER_AGENT,
+        )
+
+        self._flow = AsyncPrompty.load(source=prompty_file, model=prompty_model_config)
+
+    # __call__ not overridden here because child classes have such varied signatures that there's no point
+    # defining a default here.
+
+    @override
+    async def _do_eval(self, eval_input: Dict) -> Dict[str, Union[float, str]]:  # type: ignore[override]
+        """Do a relevance evaluation.
+
+        :param eval_input: The input to the evaluator. Expected to contain
+        whatever inputs are needed for the _flow method, including context
+        and other fields depending on the child class.
+        :type eval_input: Dict
+        :return: The evaluation result.
+        :rtype: Dict
+        """
+        llm_output = await self._flow(timeout=self._LLM_CALL_TIMEOUT, **eval_input)
+
+        score = math.nan
+        if llm_output:
+            # Parse out score and reason from evaluators known to possess them.
+            if self._result_key in PROMPT_BASED_REASON_EVALUATORS:
+                score, reason = parse_quality_evaluator_reason_score(llm_output)
+                return {
+                    self._result_key: float(score),
+                    f"gpt_{self._result_key}": float(score),
+                    f"{self._result_key}_reason": reason,
+                }
+            match = re.search(r"\d", llm_output)
+            if match:
+                score = float(match.group())
+            return {self._result_key: float(score), f"gpt_{self._result_key}": float(score)}
+        return {self._result_key: float(score), f"gpt_{self._result_key}": float(score)}

azure/ai/evaluation/_evaluators/_common/_base_rai_svc_eval.py

@@ -0,0 +1,133 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+from typing import Dict, TypeVar, Union
+
+from typing_extensions import override
+
+from azure.ai.evaluation._common.constants import (
+    EvaluationMetrics,
+    _InternalEvaluationMetrics,
+    Tasks,
+    _InternalAnnotationTasks,
+)
+from azure.ai.evaluation._common.rai_service import evaluate_with_rai_service
+from azure.ai.evaluation._common.utils import validate_azure_ai_project
+from azure.ai.evaluation._exceptions import EvaluationException
+from azure.core.credentials import TokenCredential
+
+from . import EvaluatorBase
+
+T = TypeVar("T")
+
+
+class RaiServiceEvaluatorBase(EvaluatorBase[T]):
+    """Base class for all evaluators that require the use of the Azure AI RAI service for evaluation.
+    This includes content safety evaluators, protected material evaluators, and others. These evaluators
+    are all assumed to be of the "query and response or conversation" input variety.
+
+    :param eval_metric: The evaluation metric to be used for evaluation. This is used by the API call logic
+        to specify which evaluation to perform.
+    :type eval_metric: ~azure.ai.evaluation._common.constants.EvaluationMetrics
+    :param eval_last_turn: If True, only the last turn of the conversation will be evaluated, and no
+        aggregation will be performed. If False, all turns will be evaluated and the numeric results will be,
+        aggregated. Per-turn results are still be available in the output via the "evaluation_per_turn" key
+        when this occurs. Default is False, resulting full conversation evaluation and aggregation.
+    :type eval_last_turn: bool
+    """
+
+    @override
+    def __init__(
+        self,
+        eval_metric: Union[EvaluationMetrics, _InternalEvaluationMetrics],
+        azure_ai_project: dict,
+        credential: TokenCredential,
+        eval_last_turn: bool = False,
+    ):
+        super().__init__(eval_last_turn=eval_last_turn)
+        self._eval_metric = eval_metric
+        self._azure_ai_project = validate_azure_ai_project(azure_ai_project)
+        self._credential = credential
+
+    @override
+    def __call__(  # pylint: disable=docstring-missing-param
+        self,
+        *args,
+        **kwargs,
+    ):
+        """Evaluate either a query and response or a conversation. Must supply either a query AND response,
+        or a conversation, but not both.
+
+        :keyword query: The query to evaluate.
+        :paramtype query: Optional[str]
+        :keyword response: The response to evaluate.
+        :paramtype response: Optional[str]
+        :keyword conversation: The conversation to evaluate. Expected to contain a list of conversation turns under the
+            key "messages", and potentially a global context under the key "context". Conversation turns are expected
+            to be dictionaries with keys "content", "role", and possibly "context".
+        :paramtype conversation: Optional[~azure.ai.evaluation.Conversation]
+        :rtype: Union[Dict[str, T], Dict[str, Union[float, Dict[str, List[T]]]]]
+        """
+        return super().__call__(*args, **kwargs)
+
+    @override
+    async def _do_eval(self, eval_input: Dict) -> Dict[str, T]:
+        """Perform the evaluation using the Azure AI RAI service.
+        The exact evaluation performed is determined by the evaluation metric supplied
+        by the child class initializer.
+
+        :param eval_input: The input to the evaluation function.
+        :type eval_input: Dict
+        :return: The evaluation result.
+        :rtype: Dict
+        """
+        query = eval_input.get("query", None)
+        response = eval_input.get("response", None)
+        if query is None or response is None:
+            raise EvaluationException(
+                message="Not implemented",
+                internal_message=(
+                    "Reached query/response evaluation without supplying query or response."
+                    + " This should have failed earlier."
+                ),
+            )
+        input_data = {"query": query, "response": response}
+
+        if "context" in self._singleton_inputs:
+            context = eval_input.get("context", None)
+            if context is None:
+                raise EvaluationException(
+                    message="Not implemented",
+                    internal_message=(
+                        "Attempted context-based evaluation without supplying context."
+                        + " This should have failed earlier."
+                    ),
+                )
+            input_data["context"] = context
+
+        return await evaluate_with_rai_service(  # type: ignore
+            metric_name=self._eval_metric,
+            data=input_data,
+            project_scope=self._azure_ai_project,
+            credential=self._credential,
+            annotation_task=self._get_task(),
+        )
+
+    def _get_task(self):
+        """Get the annotation task for the current evaluation metric.
+        The annotation task is used by the RAI service script to determine a the message format
+        of the API call, and how the output is processed, among other things.
+
+        :return: The annotation task for the evaluator's self._eval_metric value.
+        :rtype: ~azure.ai.evaluation._common.constants.Tasks
+
+        """
+        if self._eval_metric == EvaluationMetrics.GROUNDEDNESS:
+            return Tasks.GROUNDEDNESS
+        if self._eval_metric == EvaluationMetrics.XPIA:
+            return Tasks.XPIA
+        if self._eval_metric == _InternalEvaluationMetrics.ECI:
+            return _InternalAnnotationTasks.ECI
+        if self._eval_metric == EvaluationMetrics.PROTECTED_MATERIAL:
+            return Tasks.PROTECTED_MATERIAL
+        return Tasks.CONTENT_HARM