azure-ai-evaluation 1.4.0__py3-none-any.whl → 1.6.0__py3-none-any.whl

azure/ai/evaluation/_evaluators/_response_completeness/response_completeness.prompty

@@ -22,65 +22,51 @@ inputs:
 ---
 system:
 # Instruction
-## Context
-### You are an expert in evaluating the quality of an answer from an intelligent system based on provided definitions and data. Your goal will involve answering the questions below using the information provided.
-- **Definition**: You are given a definition of the response quality that is being evaluated to help guide your Score.
-- **Data**: Your input data include a response and its ground truth.
-- **Questions**: To complete your evaluation you will be asked to evaluate the Data in different ways.
-
+## Goal
+### You are an expert in evaluating the quality of a Response from an intelligent system based on provided definition and data. Your goal will involve answering the questions below using the information provided.
+- **Definition**: You are given a definition of the communication trait that is being evaluated to help guide your Score.
+- **Data**: Your input data include Response and Ground Truth.
+- **Tasks**: To complete your evaluation you will be asked to evaluate the Data in different ways.
 
+user:
 # Definition
+**Completeness** refers to how accurately and thoroughly a response represents the information provided in the ground truth. It considers both the inclusion of all relevant statements and the correctness of those statements. Each statement in the ground truth should be evaluated individually to determine if it is accurately reflected in the response without missing any key information. The scale ranges from 1 to 5, with higher numbers indicating greater completeness.
 
-**Level 1: Fully incomplete**
-
-**Definition:**
-A response is considered fully incomplete if it does not contain any the necessary and relevant information with respect to the ground truth. In other words, it completely misses all the information - especially claims and statements - established in the ground truth. 
+# Ratings
+## [Completeness: 1] (Fully Incomplete)
+**Definition:** A response that does not contain any of the necessary and relevant information with respect to the ground truth. It completely misses all the information, especially claims and statements, established in the ground truth.
 
 **Examples:**
-1. **Response:** "Flu shot cannot cure cancer. Stay healthy requires sleeping exactly 8 hours a day. A few hours of exercise per week will have little benefits for physical and mental health. Physical and mental health benefits are separate topics. Scientists have not studied any of them."
-   **Ground Truth:** "Flu shot can prevent flu-related illnesses. Staying healthy requires proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
-
-
-**Level 2: Barely complete**
+  **Response:** "Flu shot cannot cure cancer. Stay healthy requires sleeping exactly 8 hours a day. A few hours of exercise per week will have little benefits for physical and mental health. Physical and mental health benefits are separate topics. Scientists have not studied any of them."
+  **Ground Truth:** "Flu shot can prevent flu-related illnesses. Staying healthy requires proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
 
-**Definition:**
-A response is considered barely complete if it only contains a small percentage of all the necessary and relevant information with respect to the ground truth. In other words, it misses almost all the information - especially claims and statements - established in the ground truth. 
+## [Completeness: 2] (Barely Complete)
+**Definition:** A response that contains only a small percentage of all the necessary and relevant information with respect to the ground truth. It misses almost all the information, especially claims and statements, established in the ground truth.
 
 **Examples:**
-1. **Response:** "Flu shot can prevent flu-related illnesses. Staying healthy requires 2 meals a day. Exercise per week makes not difference to physical and mental health. This is because physical and mental health benefits have low correlation through scientific studies. Scientists are making this observation in studies."
-   **Ground Truth:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
+  **Response:** "Flu shot can prevent flu-related illnesses. Staying healthy requires 2 meals a day. Exercise per week makes no difference to physical and mental health. This is because physical and mental health benefits have low correlation through scientific studies. Scientists are making this observation in studies."
+  **Ground Truth:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
 
-
-**Level 3: Moderately complete**
-
-**Definition:**
-A response is considered moderately complete if it contains half of the necessary and relevant information with respect to the ground truth. In other words, it miss half of the information - especially claims and statements - established in the ground truth. 
+## [Completeness: 3] (Moderately Complete)
+**Definition:** A response that contains half of the necessary and relevant information with respect to the ground truth. It misses half of the information, especially claims and statements, established in the ground truth.
 
 **Examples:**
-1. **Response:** "Flu shot can prevent flu-related illnesses. Staying healthy requires a few dollar of investments a day. Even a few dollars of investments per week will not make an impact on physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Fiction writers are starting to discover them through their works."
-   **Ground Truth:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
-
-
-**Level 4: Mostly complete**
-
-**Definition:**
-A response is considered mostly complete if it contains most of the necessary and relevant information with respect to the ground truth. In other words, it misses some minor information - especially claims and statements - established in the ground truth. 
+  **Response:** "Flu shot can prevent flu-related illnesses. Staying healthy requires a few dollars of investments a day. Even a few dollars of investments per week will not make an impact on physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Fiction writers are starting to discover them through their works."
+  **Ground Truth:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
 
+## [Completeness: 4] (Mostly Complete)
+**Definition:** A response that contains most of the necessary and relevant information with respect to the ground truth. It misses some minor information, especially claims and statements, established in the ground truth.
 
 **Examples:**
-1. **Response:** "Flu shot can prevent flu-related illnesses. Staying healthy requires keto diet and rigorous athletic training. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
-   **Ground Truth:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
-
+  **Response:** "Flu shot can prevent flu-related illnesses. Staying healthy requires keto diet and rigorous athletic training. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
+  **Ground Truth:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
 
-**Level 5: Fully complete**
-
-**Definition:**
-A response is considered complete if it perfectly contains all the necessary and relevant information with respect to the ground truth. In other words, it does not miss any information from statements and claims in the ground truth. 
+## [Completeness: 5] (Fully Complete)
+**Definition:** A response that perfectly contains all the necessary and relevant information with respect to the ground truth. It does not miss any information from statements and claims in the ground truth.
 
 **Examples:**
-1. **Response:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
-   **Ground Truth:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
-
+  **Response:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
+  **Ground Truth:** "Flu shot can prevent flu-related illnesses. Stay healthy by proper hydration and moderate exercise. Even a few hours of exercise per week can have long-term benefits for physical and mental health. This is because physical and mental health benefits have intricate relationships through behavioral changes. Scientists are starting to discover them through rigorous studies."
 
 
 # Data
@@ -89,11 +75,10 @@ Ground Truth: {{ground_truth}}
 
 
 # Tasks
-## Please provide your assessment Score for the previous answer. Your output should include the following information:
-- **ThoughtChain**: To improve the reasoning process, Think Step by Step and include a step-by-step explanation of your thought process as you analyze the data based on the definitions. Keep it brief and Start your ThoughtChain with "Let's think step by step:".
+## Please provide your assessment Score for the previous RESPONSE in relation to the GROUND TRUTH based on the Definitions above. Your output should include the following information:
+- **ThoughtChain**: To improve the reasoning process, think step by step and include a step-by-step explanation of your thought process as you analyze the data based on the definitions. Keep it brief and start your ThoughtChain with "Let's think step by step:".
 - **Explanation**: a very short explanation of why you think the input data should get that Score.
-- **Score**: based on your previous analysis, provide your Score. The answer you give MUST be a integer score ("1", "2", ...) based on the categories of the definitions.
-
+- **Score**: based on your previous analysis, provide your Score. The Score you give MUST be an integer score (i.e., "1", "2"...) based on the levels of the definitions.
 
 ## Please provide your answers between the tags: <S0>your chain of thoughts</S0>, <S1>your explanation</S1>, <S2>your score</S2>.
 # Output

azure/ai/evaluation/_evaluators/_similarity/_similarity.py

@@ -28,7 +28,7 @@ class SimilarityEvaluator(PromptyEvaluatorBase):
     :param model_config: Configuration for the Azure OpenAI model.
     :type model_config: Union[~azure.ai.evaluation.AzureOpenAIModelConfiguration,
         ~azure.ai.evaluation.OpenAIModelConfiguration]
-    :param threshold: The threshold for the similarity evaluator. Default is 5.
+    :param threshold: The threshold for the similarity evaluator. Default is 3.
     :type threshold: int
 
     .. admonition:: Example:

azure/ai/evaluation/_evaluators/_task_adherence/_task_adherence.py

@@ -54,11 +54,14 @@ class TaskAdherenceEvaluator(PromptyEvaluatorBase[Union[str, float]]):
     """Evaluator identifier, experimental and to be used only with evaluation in cloud."""
 
     @override
-    def __init__(self, model_config, *, threshold=_DEFAULT_TASK_ADHERENCE_SCORE):
+    def __init__(self, model_config, *, threshold=_DEFAULT_TASK_ADHERENCE_SCORE,
+                 **kwargs):
         current_dir = os.path.dirname(__file__)
         prompty_path = os.path.join(current_dir, self._PROMPTY_FILE)
         self.threshold = threshold
-        super().__init__(model_config=model_config, prompty_file=prompty_path, result_key=self._RESULT_KEY)
+        super().__init__(model_config=model_config, prompty_file=prompty_path,
+                         result_key=self._RESULT_KEY,
+                         **kwargs)
 
     @overload
     def __call__(

azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_call_accuracy.py

@@ -64,11 +64,15 @@ class ToolCallAccuracyEvaluator(PromptyEvaluatorBase[Union[str, float]]):
     """Evaluator identifier, experimental and to be used only with evaluation in cloud."""
 
     @override
-    def __init__(self, model_config, *, threshold=_DEFAULT_TOOL_CALL_ACCURACY_SCORE):
+    def __init__(self, model_config, *,
+                 threshold=_DEFAULT_TOOL_CALL_ACCURACY_SCORE,
+                 **kwargs):
         current_dir = os.path.dirname(__file__)
         prompty_path = os.path.join(current_dir, self._PROMPTY_FILE)
         self.threshold = threshold
-        super().__init__(model_config=model_config, prompty_file=prompty_path, result_key=self._RESULT_KEY)
+        super().__init__(model_config=model_config, prompty_file=prompty_path,
+                         result_key=self._RESULT_KEY,
+                         **kwargs)
 
     @overload
     def __call__(

azure/ai/evaluation/_exceptions.py

@@ -38,6 +38,7 @@ class ErrorCategory(Enum):
     FAILED_REMOTE_TRACKING = "FAILED REMOTE TRACKING"
     PROJECT_ACCESS_ERROR = "PROJECT ACCESS ERROR"
     UNKNOWN = "UNKNOWN"
+    UPLOAD_ERROR = "UPLOAD ERROR"
 
 
 class ErrorBlame(Enum):
@@ -85,6 +86,7 @@ class ErrorTarget(Enum):
     CONVERSATION = "Conversation"
     TOOL_CALL_ACCURACY_EVALUATOR = "ToolCallAccuracyEvaluator"
     RED_TEAM = "RedTeam"
+    AOAI_GRADER = "AoaiGrader"
 
 
 class EvaluationException(AzureError):

azure/ai/evaluation/_legacy/_adapters/__init__.py

@@ -0,0 +1,7 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+# NOTE: This contains adapters that make the Promptflow dependency optional. In the first phase,
+#       Promptflow will still be installed as part of the azure-ai-evaluation dependencies. This
+#       will be removed in the future once the code migration is complete.

azure/ai/evaluation/_legacy/_adapters/_check.py

@@ -0,0 +1,17 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from typing import Final
+
+
+_has_legacy = False
+try:
+    from promptflow._constants import FlowType
+
+    _has_legacy = True
+except ImportError:
+    pass
+
+HAS_LEGACY_SDK: Final[bool] = _has_legacy
+MISSING_LEGACY_SDK: Final[bool] = not _has_legacy

azure/ai/evaluation/_legacy/_adapters/_configuration.py

@@ -0,0 +1,45 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from pathlib import Path
+from typing import Any, Dict, Final, Optional
+from typing_extensions import TypeAlias
+
+
+try:
+    from promptflow._sdk._configuration import Configuration as _Configuration
+except ImportError:
+    _global_config: Final[Dict[str, Any]] = {}
+
+    class _Configuration:
+        TRACE_DESTINATION: Final[str] = "trace.destination"
+        _instance = None
+
+        def __init__(self, *, override_config: Optional[Dict[str, Any]] = None) -> None:
+            self._config = override_config or {}
+
+        @classmethod
+        def get_instance(cls) -> "_Configuration":
+            """Use this to get instance to avoid multiple copies of same global config."""
+            if cls._instance is None:
+                cls._instance = Configuration(override_config=_global_config)
+            return cls._instance
+
+        def set_config(self, key: str, value: Any) -> None:
+            # Simulated config storage
+            self._config[key] = value
+
+        def get_config(self, key: str) -> Any:
+            # Simulated config storage
+            if key in self._config:
+                return self._config[key]
+            return _global_config.get(key, None)
+
+        def get_trace_destination(self, path: Optional[Path] = None) -> Optional[str]:
+            if path:
+                raise NotImplementedError("Setting trace destination with a path is not supported.")
+            return self._config.get("trace.destination", None)
+
+
+Configuration: TypeAlias = _Configuration

azure/ai/evaluation/_legacy/_adapters/_constants.py

@@ -0,0 +1,10 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from typing import Final
+
+
+PF_FLOW_ENTRY_IN_TMP: Final[str] = "PF_FLOW_ENTRY_IN_TMP"
+PF_FLOW_META_LOAD_IN_SUBPROCESS: Final[str] = "PF_FLOW_META_LOAD_IN_SUBPROCESS"
+LINE_NUMBER: Final[str] = "line_number"

azure/ai/evaluation/_legacy/_adapters/_errors.py

@@ -0,0 +1,29 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from typing import Any
+from typing_extensions import TypeAlias
+
+
+try:
+    from promptflow.core._errors import MissingRequiredPackage as _MissingRequiredPackage
+except ImportError:
+    from azure.ai.evaluation._exceptions import ErrorBlame, ErrorCategory, ErrorTarget, EvaluationException
+
+    class _MissingRequiredPackage(EvaluationException):
+        """Raised when a required package is missing.
+
+        :param message: A message describing the error. This is the error message the user will see.
+        :type message: str
+        """
+
+        def __init__(self, message: str, **kwargs: Any):
+            kwargs.setdefault("category", ErrorCategory.MISSING_PACKAGE)
+            kwargs.setdefault("blame", ErrorBlame.SYSTEM_ERROR)
+            kwargs.setdefault("target", ErrorTarget.EVALUATE)
+            kwargs.setdefault("internal_message", "Missing required package.")
+            super().__init__(message=message, **kwargs)
+
+
+MissingRequiredPackage: TypeAlias = _MissingRequiredPackage

azure/ai/evaluation/_legacy/_adapters/_flows.py

@@ -0,0 +1,28 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from typing_extensions import TypeAlias
+
+
+try:
+    from promptflow.core._flow import AsyncPrompty as _AsyncPrompty
+    from promptflow._sdk.entities._flows import FlexFlow as _FlexFlow
+    from promptflow._sdk.entities._flows.dag import Flow as _Flow
+except ImportError:
+    from azure.ai.evaluation._legacy.prompty import AsyncPrompty as _AsyncPrompty
+
+    class _FlexFlow:
+        pass
+
+    _FlexFlow.__name__ = "FlexFlow"
+
+    class _Flow:
+        name: str
+
+    _Flow.__name__ = "Flow"
+
+
+AsyncPrompty: TypeAlias = _AsyncPrompty
+FlexFlow: TypeAlias = _FlexFlow
+Flow: TypeAlias = _Flow

azure/ai/evaluation/_legacy/_adapters/_service.py

@@ -0,0 +1,16 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from typing import Callable, Final
+
+
+try:
+    from promptflow._cli._pf._service import stop_service as _stop_service
+except ImportError:
+
+    def _stop_service() -> None:
+        pass
+
+
+stop_service: Final[Callable[[], None]] = _stop_service

azure/ai/evaluation/_legacy/_adapters/client.py

@@ -0,0 +1,51 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from os import PathLike
+from typing import Any, Callable, Dict, Optional, Union
+from typing_extensions import TypeAlias
+
+import pandas as pd
+
+from ._errors import MissingRequiredPackage
+from ._configuration import Configuration
+from .entities import Run
+
+
+try:
+    from promptflow.client import PFClient as _PFClient
+except ImportError:
+
+    class _PFClient:
+        def __init__(self, **kwargs):
+            self._config = Configuration(override_config=kwargs.pop("config", None))
+
+        def run(
+            self,
+            flow: Union[str, PathLike, Callable],
+            *,
+            data: Union[str, PathLike],
+            run: Optional[Union[str, Run]] = None,
+            column_mapping: Optional[dict] = None,
+            variant: Optional[str] = None,
+            connections: Optional[dict] = None,
+            environment_variables: Optional[dict] = None,
+            name: Optional[str] = None,
+            display_name: Optional[str] = None,
+            tags: Optional[Dict[str, str]] = None,
+            resume_from: Optional[Union[str, Run]] = None,
+            code: Optional[Union[str, PathLike]] = None,
+            init: Optional[dict] = None,
+            **kwargs,
+        ) -> Run:
+            raise MissingRequiredPackage("Please install 'promptflow' package to use PFClient")
+
+        def get_details(self, run: Union[str, Run], max_results: int = 100, all_results: bool = False) -> pd.DataFrame:
+            return pd.DataFrame()
+
+        def get_metrics(self, run: Union[str, Run]) -> Dict[str, Any]:
+            return {}
+
+
+PFClient: TypeAlias = _PFClient

azure/ai/evaluation/_legacy/_adapters/entities.py

@@ -0,0 +1,26 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from typing_extensions import TypeAlias
+
+
+try:
+    from promptflow._sdk.entities import Run as _Run
+except ImportError:
+    from typing_extensions import Protocol
+    from typing import Any, Dict, Optional
+    from datetime import datetime
+    from pathlib import Path
+
+    class _Run(Protocol):
+        name: str
+        status: str
+        _properties: Dict[str, Any]
+        _created_on: datetime
+        _end_time: Optional[datetime]
+        _experiment_name: Optional[str]
+        _output_path: Path
+
+
+Run: TypeAlias = _Run

azure/ai/evaluation/_legacy/_adapters/tracing.py

@@ -0,0 +1,28 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from typing import Callable, Dict, Final, Optional
+from typing_extensions import TypeAlias
+
+
+try:
+    from promptflow.tracing import ThreadPoolExecutorWithContext as _ThreadPoolExecutorWithContext
+    from promptflow.tracing._integrations._openai_injector import (
+        inject_openai_api as _inject,
+        recover_openai_api as _recover,
+    )
+    from promptflow.tracing import _start_trace
+except ImportError:
+    from concurrent.futures import ThreadPoolExecutor as _ThreadPoolExecutorWithContext
+    from azure.ai.evaluation._legacy._batch_engine._openai_injector import (
+        inject_openai_api as _inject,
+        recover_openai_api as _recover,
+    )
+    from azure.ai.evaluation._legacy._batch_engine._trace import start_trace as _start_trace
+
+
+ThreadPoolExecutorWithContext: TypeAlias = _ThreadPoolExecutorWithContext
+inject_openai_api: Final[Callable[[], None]] = _inject
+recover_openai_api: Final[Callable[[], None]] = _recover
+start_trace: Final = _start_trace

azure/ai/evaluation/_legacy/_adapters/types.py

@@ -0,0 +1,15 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from typing import Any
+
+
+class AttrDict(dict):
+    """A dictionary that allows attribute access to its keys."""
+
+    def __getattr__(self, key: str) -> Any:
+        return self[key]
+
+    def __setattr__(self, key: str, value: Any) -> None:
+        self[key] = value

azure/ai/evaluation/_legacy/_adapters/utils.py

@@ -0,0 +1,31 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from typing import Final, Optional
+from typing_extensions import TypeAlias
+
+
+try:
+    from promptflow._utils.user_agent_utils import ClientUserAgentUtil as _ClientUserAgentUtil
+    from promptflow._utils.async_utils import async_run_allowing_running_loop as _async_run_allowing_running_loop
+    from promptflow._cli._utils import get_workspace_triad_from_local as _get_workspace_triad_from_local
+except ImportError:
+    from azure.ai.evaluation._legacy._batch_engine._utils_deprecated import (
+        async_run_allowing_running_loop as _async_run_allowing_running_loop,
+    )
+    from azure.ai.evaluation._evaluate._utils import AzureMLWorkspace
+
+    class _ClientUserAgentUtil:
+        @staticmethod
+        def append_user_agent(user_agent: Optional[str]):
+            # TODO ralphe: implement?
+            pass
+
+    def _get_workspace_triad_from_local() -> AzureMLWorkspace:
+        return AzureMLWorkspace("", "", "")
+
+
+ClientUserAgentUtil: TypeAlias = _ClientUserAgentUtil
+async_run_allowing_running_loop: Final = _async_run_allowing_running_loop
+get_workspace_triad_from_local: Final = _get_workspace_triad_from_local