azure-ai-evaluation 1.0.1__tar.gz → 1.2.0__tar.gz

{azure_ai_evaluation-1.0.1 → azure_ai_evaluation-1.2.0}/CHANGELOG.md

@@ -1,10 +1,81 @@
 # Release History
 
+## 1.2.0 (2025-01-27)
+
+### Features Added
+- CSV files are now supported as data file inputs with `evaluate()` API. The CSV file should have a header row with column names that match the `data` and `target` fields in the `evaluate()` method and the filename should be passed as the `data` parameter. Column name 'Conversation' in CSV file is not fully supported yet.  
+
+### Breaking Changes
+- `ViolenceMultimodalEvaluator`, `SexualMultimodalEvaluator`, `SelfHarmMultimodalEvaluator`, `HateUnfairnessMultimodalEvaluator` and `ProtectedMaterialMultimodalEvaluator` will be removed in next release. 
+
+### Bugs Fixed
+- Removed `[remote]` extra. This is no longer needed when tracking results in Azure AI Studio.
+- Fixed `AttributeError: 'NoneType' object has no attribute 'get'` while running simulator with 1000+ results
+- Fixed the non adversarial simulator to run in task-free mode
+- Content safety evaluators (violence, self harm, sexual, hate/unfairness) return the maximum result as the
+  main score when aggregating per-turn evaluations from a conversation into an overall
+  evaluation score. Other conversation-capable evaluators still default to a mean for aggregation.
+- Fixed bug in non adversarial simulator sample where `tasks` undefined
+
+### Other Changes
+- Changed minimum required python version to use this package from 3.8 to 3.9
+- Stop dependency on the local promptflow service. No promptflow service will automatically start when running evaluation.
+- Evaluators internally allow for custom aggregation. However, this causes serialization failures if evaluated while the
+  environment variable `AI_EVALS_BATCH_USE_ASYNC` is set to false.
+
+## 1.1.0 (2024-12-12)
+
+### Features Added
+- Added image support in `ContentSafetyEvaluator`, `ViolenceEvaluator`, `SexualEvaluator`, `SelfHarmEvaluator`, `HateUnfairnessEvaluator` and `ProtectedMaterialEvaluator`. Provide image URLs or base64 encoded images in `conversation` input for image evaluation. See below for an example:
+
+```python
+evaluator = ContentSafetyEvaluator(credential=azure_cred, azure_ai_project=project_scope)
+conversation = {
+    "messages": [
+        {
+            "role": "system",
+            "content": [
+                {"type": "text", "text": "You are an AI assistant that understands images."}
+            ],
+        },
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "Can you describe this image?"},
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": "https://cdn.britannica.com/68/178268-050-5B4E7FB6/Tom-Cruise-2013.jpg"
+                    },
+                },
+            ],
+        },
+        {
+            "role": "assistant",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "The image shows a man with short brown hair smiling, wearing a dark-colored shirt.",
+                }
+            ],
+        },
+    ]
+}
+print("Calling Content Safety Evaluator for multi-modal")
+score = evaluator(conversation=conversation)
+```
+
+- Please switch to generic evaluators for image evaluations as mentioned above. `ContentSafetyMultimodalEvaluator`, `ContentSafetyMultimodalEvaluatorBase`, `ViolenceMultimodalEvaluator`, `SexualMultimodalEvaluator`, `SelfHarmMultimodalEvaluator`, `HateUnfairnessMultimodalEvaluator` and `ProtectedMaterialMultimodalEvaluator` will be deprecated in the next release.
+
+### Bugs Fixed
+- Removed `[remote]` extra. This is no longer needed when tracking results in Azure AI Foundry portal.
+- Fixed `AttributeError: 'NoneType' object has no attribute 'get'` while running simulator with 1000+ results
+
 ## 1.0.1 (2024-11-15)
 
 ### Bugs Fixed
-- Fixed `[remote]` extra to be needed only when tracking results in Azure AI Studio.
 - Removing `azure-ai-inference` as dependency.
+- Fixed `AttributeError: 'NoneType' object has no attribute 'get'` while running simulator with 1000+ results
 
 ## 1.0.0 (2024-11-13)
 
@@ -16,6 +87,7 @@
 - Fixed an issue where the `output_path` parameter in the `evaluate` API did not support relative path.
 - Output of adversarial simulators are of type `JsonLineList` and the helper function `to_eval_qr_json_lines` now outputs context from both user and assistant turns along with `category` if it exists in the conversation
 - Fixed an issue where during long-running simulations, API token expires causing "Forbidden" error. Instead, users can now set an environment variable `AZURE_TOKEN_REFRESH_INTERVAL` to refresh the token more frequently to prevent expiration and ensure continuous operation of the simulation.
+- Fixed an issue with the `ContentSafetyEvaluator` that caused parallel execution of sub-evaluators to fail. Parallel execution is now enabled by default again, but can still be disabled via the '_parallel' boolean keyword argument during class initialization.
 - Fix `evaluate` function not producing aggregated metrics if ANY values to be aggregated were None, NaN, or
 otherwise difficult to process. Such values are ignored fully, so the aggregated metric of `[1, 2, 3, NaN]`
 would be 2, not 1.5.

{azure_ai_evaluation-1.0.1/azure_ai_evaluation.egg-info → azure_ai_evaluation-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: azure-ai-evaluation
-Version: 1.0.1
+Version: 1.2.0
 Summary: Microsoft Azure Evaluation Library for Python
 Home-page: https://github.com/Azure/azure-sdk-for-python
 Author: Microsoft Corporation
@@ -13,23 +13,21 @@ Classifier: Development Status :: 5 - Production/Stable
 Classifier: Programming Language :: Python
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.8
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: NOTICE.txt
-Requires-Dist: promptflow-devkit>=1.15.0
-Requires-Dist: promptflow-core>=1.15.0
+Requires-Dist: promptflow-devkit>=1.17.1
+Requires-Dist: promptflow-core>=1.17.1
 Requires-Dist: pyjwt>=2.8.0
 Requires-Dist: azure-identity>=1.16.0
 Requires-Dist: azure-core>=1.30.2
 Requires-Dist: nltk>=3.9.1
-Provides-Extra: remote
-Requires-Dist: promptflow-azure<2.0.0,>=1.15.0; extra == "remote"
+Requires-Dist: azure-storage-blob>=12.10.0
 
 # Azure AI Evaluation client library for Python
 
@@ -55,7 +53,7 @@ Azure AI SDK provides following to evaluate Generative AI Applications:
 
 ### Prerequisites
 
-- Python 3.8 or later is required to use this package.
+- Python 3.9 or later is required to use this package.
 - [Optional] You must have [Azure AI Project][ai_project] or [Azure Open AI][azure_openai] to use AI-assisted evaluators
 
 ### Install the package
@@ -359,13 +357,13 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con
 [evaluate_dataset]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk#evaluate-on-test-dataset-using-evaluate
 [evaluators]: https://learn.microsoft.com/python/api/azure-ai-evaluation/azure.ai.evaluation?view=azure-python-preview
 [evaluate_api]: https://learn.microsoft.com/python/api/azure-ai-evaluation/azure.ai.evaluation?view=azure-python-preview#azure-ai-evaluation-evaluate
-[evaluate_app]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/evaluate_app
+[evaluate_app]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/Supported_Evaluation_Targets/Evaluate_App_Endpoint
 [evaluation_tsg]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md
 [ai_studio]: https://learn.microsoft.com/azure/ai-studio/what-is-ai-studio
 [ai_project]: https://learn.microsoft.com/azure/ai-studio/how-to/create-projects?tabs=ai-studio
 [azure_openai]: https://learn.microsoft.com/azure/ai-services/openai/
-[evaluate_models]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/evaluate_endpoints
-[custom_evaluators]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/evaluate_custom
+[evaluate_models]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/Supported_Evaluation_Targets/Evaluate_Base_Model_Endpoint
+[custom_evaluators]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/Supported_Evaluation_Metrics/Custom_Evaluators
 [evaluate_samples]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate
 [evaluation_metrics]: https://learn.microsoft.com/azure/ai-studio/concepts/evaluation-metrics-built-in
 [performance_and_quality_evaluators]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk#performance-and-quality-evaluators
@@ -373,18 +371,88 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con
 [composite_evaluators]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk#composite-evaluators
 [adversarial_simulation_docs]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/simulator-interaction-data#generate-adversarial-simulations-for-safety-evaluation
 [adversarial_simulation_scenarios]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/simulator-interaction-data#supported-adversarial-simulation-scenarios
-[adversarial_simulation]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/simulate_adversarial
-[simulate_with_conversation_starter]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/simulate_conversation_starter
+[adversarial_simulation]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/Simulators/Simulate_Adversarial_Data
+[simulate_with_conversation_starter]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/Simulators/Simulate_Context-Relevant_Data/Simulate_From_Conversation_Starter
 [adversarial_jailbreak]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/simulator-interaction-data#simulating-jailbreak-attacks
 
-
 # Release History
 
+## 1.2.0 (2025-01-27)
+
+### Features Added
+- CSV files are now supported as data file inputs with `evaluate()` API. The CSV file should have a header row with column names that match the `data` and `target` fields in the `evaluate()` method and the filename should be passed as the `data` parameter. Column name 'Conversation' in CSV file is not fully supported yet.  
+
+### Breaking Changes
+- `ViolenceMultimodalEvaluator`, `SexualMultimodalEvaluator`, `SelfHarmMultimodalEvaluator`, `HateUnfairnessMultimodalEvaluator` and `ProtectedMaterialMultimodalEvaluator` will be removed in next release. 
+
+### Bugs Fixed
+- Removed `[remote]` extra. This is no longer needed when tracking results in Azure AI Studio.
+- Fixed `AttributeError: 'NoneType' object has no attribute 'get'` while running simulator with 1000+ results
+- Fixed the non adversarial simulator to run in task-free mode
+- Content safety evaluators (violence, self harm, sexual, hate/unfairness) return the maximum result as the
+  main score when aggregating per-turn evaluations from a conversation into an overall
+  evaluation score. Other conversation-capable evaluators still default to a mean for aggregation.
+- Fixed bug in non adversarial simulator sample where `tasks` undefined
+
+### Other Changes
+- Changed minimum required python version to use this package from 3.8 to 3.9
+- Stop dependency on the local promptflow service. No promptflow service will automatically start when running evaluation.
+- Evaluators internally allow for custom aggregation. However, this causes serialization failures if evaluated while the
+  environment variable `AI_EVALS_BATCH_USE_ASYNC` is set to false.
+
+## 1.1.0 (2024-12-12)
+
+### Features Added
+- Added image support in `ContentSafetyEvaluator`, `ViolenceEvaluator`, `SexualEvaluator`, `SelfHarmEvaluator`, `HateUnfairnessEvaluator` and `ProtectedMaterialEvaluator`. Provide image URLs or base64 encoded images in `conversation` input for image evaluation. See below for an example:
+
+```python
+evaluator = ContentSafetyEvaluator(credential=azure_cred, azure_ai_project=project_scope)
+conversation = {
+    "messages": [
+        {
+            "role": "system",
+            "content": [
+                {"type": "text", "text": "You are an AI assistant that understands images."}
+            ],
+        },
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "Can you describe this image?"},
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": "https://cdn.britannica.com/68/178268-050-5B4E7FB6/Tom-Cruise-2013.jpg"
+                    },
+                },
+            ],
+        },
+        {
+            "role": "assistant",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "The image shows a man with short brown hair smiling, wearing a dark-colored shirt.",
+                }
+            ],
+        },
+    ]
+}
+print("Calling Content Safety Evaluator for multi-modal")
+score = evaluator(conversation=conversation)
+```
+
+- Please switch to generic evaluators for image evaluations as mentioned above. `ContentSafetyMultimodalEvaluator`, `ContentSafetyMultimodalEvaluatorBase`, `ViolenceMultimodalEvaluator`, `SexualMultimodalEvaluator`, `SelfHarmMultimodalEvaluator`, `HateUnfairnessMultimodalEvaluator` and `ProtectedMaterialMultimodalEvaluator` will be deprecated in the next release.
+
+### Bugs Fixed
+- Removed `[remote]` extra. This is no longer needed when tracking results in Azure AI Foundry portal.
+- Fixed `AttributeError: 'NoneType' object has no attribute 'get'` while running simulator with 1000+ results
+
 ## 1.0.1 (2024-11-15)
 
 ### Bugs Fixed
-- Fixed `[remote]` extra to be needed only when tracking results in Azure AI Studio.
 - Removing `azure-ai-inference` as dependency.
+- Fixed `AttributeError: 'NoneType' object has no attribute 'get'` while running simulator with 1000+ results
 
 ## 1.0.0 (2024-11-13)
 
@@ -396,6 +464,7 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con
 - Fixed an issue where the `output_path` parameter in the `evaluate` API did not support relative path.
 - Output of adversarial simulators are of type `JsonLineList` and the helper function `to_eval_qr_json_lines` now outputs context from both user and assistant turns along with `category` if it exists in the conversation
 - Fixed an issue where during long-running simulations, API token expires causing "Forbidden" error. Instead, users can now set an environment variable `AZURE_TOKEN_REFRESH_INTERVAL` to refresh the token more frequently to prevent expiration and ensure continuous operation of the simulation.
+- Fixed an issue with the `ContentSafetyEvaluator` that caused parallel execution of sub-evaluators to fail. Parallel execution is now enabled by default again, but can still be disabled via the '_parallel' boolean keyword argument during class initialization.
 - Fix `evaluate` function not producing aggregated metrics if ANY values to be aggregated were None, NaN, or
 otherwise difficult to process. Such values are ignored fully, so the aggregated metric of `[1, 2, 3, NaN]`
 would be 2, not 1.5.

{azure_ai_evaluation-1.0.1 → azure_ai_evaluation-1.2.0}/README.md

@@ -22,7 +22,7 @@ Azure AI SDK provides following to evaluate Generative AI Applications:
 
 ### Prerequisites
 
-- Python 3.8 or later is required to use this package.
+- Python 3.9 or later is required to use this package.
 - [Optional] You must have [Azure AI Project][ai_project] or [Azure Open AI][azure_openai] to use AI-assisted evaluators
 
 ### Install the package
@@ -326,13 +326,13 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con
 [evaluate_dataset]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk#evaluate-on-test-dataset-using-evaluate
 [evaluators]: https://learn.microsoft.com/python/api/azure-ai-evaluation/azure.ai.evaluation?view=azure-python-preview
 [evaluate_api]: https://learn.microsoft.com/python/api/azure-ai-evaluation/azure.ai.evaluation?view=azure-python-preview#azure-ai-evaluation-evaluate
-[evaluate_app]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/evaluate_app
+[evaluate_app]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/Supported_Evaluation_Targets/Evaluate_App_Endpoint
 [evaluation_tsg]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/evaluation/azure-ai-evaluation/TROUBLESHOOTING.md
 [ai_studio]: https://learn.microsoft.com/azure/ai-studio/what-is-ai-studio
 [ai_project]: https://learn.microsoft.com/azure/ai-studio/how-to/create-projects?tabs=ai-studio
 [azure_openai]: https://learn.microsoft.com/azure/ai-services/openai/
-[evaluate_models]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/evaluate_endpoints
-[custom_evaluators]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/evaluate_custom
+[evaluate_models]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/Supported_Evaluation_Targets/Evaluate_Base_Model_Endpoint
+[custom_evaluators]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/Supported_Evaluation_Metrics/Custom_Evaluators
 [evaluate_samples]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate
 [evaluation_metrics]: https://learn.microsoft.com/azure/ai-studio/concepts/evaluation-metrics-built-in
 [performance_and_quality_evaluators]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk#performance-and-quality-evaluators
@@ -340,6 +340,6 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con
 [composite_evaluators]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk#composite-evaluators
 [adversarial_simulation_docs]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/simulator-interaction-data#generate-adversarial-simulations-for-safety-evaluation
 [adversarial_simulation_scenarios]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/simulator-interaction-data#supported-adversarial-simulation-scenarios
-[adversarial_simulation]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/simulate_adversarial
-[simulate_with_conversation_starter]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/simulate_conversation_starter
-[adversarial_jailbreak]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/simulator-interaction-data#simulating-jailbreak-attacks
+[adversarial_simulation]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/Simulators/Simulate_Adversarial_Data
+[simulate_with_conversation_starter]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/evaluate/Simulators/Simulate_Context-Relevant_Data/Simulate_From_Conversation_Starter
+[adversarial_jailbreak]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/simulator-interaction-data#simulating-jailbreak-attacks

{azure_ai_evaluation-1.0.1 → azure_ai_evaluation-1.2.0}/TROUBLESHOOTING.md

@@ -26,6 +26,10 @@ This guide walks you through how to investigate failures, common errors in the `
 - Ensure that you assign the proper permissions to the storage account linked to your Azure AI Studio hub. This can be done with the following command. More information can be found [here](https://aka.ms/credentialleshub).
 
     ```Shell
+    # <mySubscriptionID>: Subscription ID of the Azure AI Studio hub's linked storage account (available in Azure AI hub resource view in Azure Portal).
+    # <myResourceGroupName>: Resource group of the Azure AI Studio hub's linked storage account.
+    # <user-id>: User object ID for role assignment (retrieve with "az ad user show" command).
+
     az role assignment create --role "Storage Blob Data Contributor" --scope /subscriptions/<mySubscriptionID>/resourceGroups/<myResourceGroupName> --assignee-principal-type User --assignee-object-id "<user-id>"
     ```
 
@@ -50,7 +54,7 @@ The Adversarial simulator does not support selecting individual harms, instead w
 ### Simulator is slow
 
 Identify the type of simulations being run (adversarial or non-adversarial).
-Adjust parameters such as `api_call_retry_sleep_sec`, `api_call_delay_sec`, and `concurrent_async_task`. Please note that rate limits to llm calls can be both tokens per minute and requests per minute. 
+Adjust parameters such as `api_call_retry_sleep_sec`, `api_call_delay_sec`, and `concurrent_async_task`. Please note that rate limits to llm calls can be both tokens per minute and requests per minute.
 
 ## Logging
 

azure_ai_evaluation-1.2.0/azure/ai/evaluation/_azure/_clients.py

@@ -0,0 +1,204 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from logging import Logger
+from typing import Any, Dict, Final, Optional, Set, Union, cast
+from threading import Lock
+from urllib.parse import quote
+from json.decoder import JSONDecodeError
+
+from azure.core.credentials import TokenCredential, AzureSasCredential
+from azure.core.rest import HttpResponse
+from azure.ai.evaluation._exceptions import ErrorBlame, ErrorCategory, ErrorTarget, EvaluationException
+from azure.ai.evaluation._http_utils import HttpPipeline, get_http_client
+from azure.ai.evaluation._azure._token_manager import AzureMLTokenManager
+from azure.ai.evaluation.simulator._model_tools._identity_manager import TokenScope
+from ._models import BlobStoreInfo, Workspace
+
+
+API_VERSION: Final[str] = "2024-07-01-preview"
+QUERY_KEY_API_VERSION: Final[str] = "api-version"
+PATH_ML_WORKSPACES = ("providers", "Microsoft.MachineLearningServices", "workspaces")
+
+
+class LiteMLClient:
+    """A lightweight Azure ML API client.
+
+    :param subscription_id: Azure subscription ID
+    :type subscription_id: str
+    :param resource_group: Azure resource group name
+    :type resource_group: str
+    :param logger: Logger object
+    :type logger: logging.Logger
+    :keyword credential: Azure credentials
+    :paramtype credential: TokenCredential
+    :keyword kwargs: Additional keyword arguments
+    :paramtype kwargs: Dict
+    :keyword str api_version: The API version. Default is 2024-10-01
+    """
+
+    def __init__(
+        self,
+        subscription_id: str,
+        resource_group: str,
+        logger: Logger,
+        credential: Optional[TokenCredential] = None,
+        **kwargs: Any,
+    ) -> None:
+        subscription_id = quote(subscription_id, safe="")
+        resource_group = quote(resource_group, safe="")
+
+        self._base_url: Final[str] = (
+            f"https://management.azure.com/subscriptions/{subscription_id}/resourceGroups/{resource_group}"
+        )
+        self._logger: Final[Logger] = logger
+        self._api_version: Final[str] = kwargs.get("api_version", API_VERSION)
+        self._http_client: Final[HttpPipeline] = get_http_client(**kwargs)
+        self._lock: Final[Lock] = Lock()
+
+        # things that can change under lock
+        self._token_manager: Optional[AzureMLTokenManager] = None
+        self._credential: Optional[TokenCredential] = credential
+
+    def get_token(self) -> str:
+        return self._get_token_manager().get_token()
+
+    def get_credential(self) -> TokenCredential:
+        # load the token manager to get the credential if needed
+        self._get_token_manager()
+        return cast(TokenCredential, self._credential)
+
+    def workspace_get_default_datastore(
+        self, workspace_name: str, *, include_credentials: bool = False, **kwargs: Any
+    ) -> BlobStoreInfo:
+        # 1. Get the default blob store
+        # REST API documentation:
+        # https://learn.microsoft.com/rest/api/azureml/datastores/list?view=rest-azureml-2024-10-01
+        url = self._generate_path(  # pylint: disable=specify-parameter-names-in-call
+            *PATH_ML_WORKSPACES, workspace_name, "datastores"
+        )
+        headers = self._get_headers()
+
+        stores_response = self._http_client.request(
+            method="GET",
+            url=url,
+            params={QUERY_KEY_API_VERSION: self._api_version, "isDefault": True, "count": 1, "orderByAsc": "false"},
+            headers=headers,
+        )
+        self._throw_on_http_error(stores_response, "list default workspace datastore")
+
+        json = stores_response.json()["value"][0]
+        props_json = json["properties"]
+        name = json["name"]
+        account_name = props_json["accountName"]
+        endpoint = props_json["endpoint"]
+        container_name = props_json["containerName"]
+        credential_type = props_json.get("credentials", {}).get("credentialsType")
+
+        # 2. Get the SAS token to use for accessing the blob store
+        # REST API documentation:
+        # https://learn.microsoft.com/rest/api/azureml/datastores/list-secrets?view=rest-azureml-2024-10-01
+        blob_store_credential: Optional[Union[AzureSasCredential, TokenCredential, str]]
+        if not include_credentials:
+            blob_store_credential = None
+        elif credential_type and credential_type.lower() == "none":
+            # If storage account key access is disabled, and only Microsoft Entra ID authentication is available,
+            # the credentialsType will be "None" and we should not attempt to get the secrets.
+            blob_store_credential = self.get_credential()
+        else:
+            url = self._generate_path(
+                *PATH_ML_WORKSPACES, workspace_name, "datastores", "workspaceblobstore", "listSecrets"
+            )
+            secrets_response = self._http_client.request(
+                method="POST",
+                url=url,
+                json={
+                    "expirableSecret": True,
+                    "expireAfterHours": int(kwargs.get("key_expiration_hours", 1)),
+                },
+                params={
+                    QUERY_KEY_API_VERSION: self._api_version,
+                },
+                headers=headers,
+            )
+            self._throw_on_http_error(secrets_response, "workspace datastore secrets")
+
+            secrets_json = secrets_response.json()
+            secrets_type = secrets_json["secretsType"].lower()
+
+            # As per this website, only SAS tokens, access tokens, or Entra IDs are valid for accessing blob data
+            # stores:
+            # https://learn.microsoft.com/rest/api/storageservices/authorize-requests-to-azure-storage.
+            if secrets_type == "sas":
+                blob_store_credential = AzureSasCredential(secrets_json["sasToken"])
+            elif secrets_type == "accountkey":
+                # To support older versions of azure-storage-blob better, we return a string here instead of
+                # an AzureNamedKeyCredential
+                blob_store_credential = secrets_json["key"]
+            else:
+                raise EvaluationException(
+                    message=f"The '{account_name}' blob store does not use a recognized credential type.",
+                    internal_message=f"The credential type is '{secrets_type}'",
+                    target=ErrorTarget.EVALUATE,
+                    category=ErrorCategory.INVALID_VALUE,
+                    blame=ErrorBlame.SYSTEM_ERROR,
+                )
+
+        return BlobStoreInfo(name, account_name, endpoint, container_name, blob_store_credential)
+
+    def workspace_get_info(self, workspace_name: str) -> Workspace:
+        # https://learn.microsoft.com/rest/api/azureml/workspaces/get?view=rest-azureml-2024-10-01
+        workspace_response = self._http_client.request(
+            "GET",
+            self._generate_path(*PATH_ML_WORKSPACES, workspace_name),
+            params={QUERY_KEY_API_VERSION: self._api_version},
+            headers=self._get_headers(),
+        )
+
+        self._throw_on_http_error(workspace_response, f"get '{workspace_name}' workspace")
+        workspace = Workspace.deserialize(workspace_response)
+        return workspace
+
+    def _get_token_manager(self) -> AzureMLTokenManager:
+        # Lazy init since getting credentials in the constructor can take a long time in some situations
+        if self._token_manager is None:
+            with self._lock:
+                if self._token_manager is None:
+                    self._token_manager = AzureMLTokenManager(
+                        TokenScope.DEFAULT_AZURE_MANAGEMENT.value, self._logger, credential=self._credential
+                    )
+                    self._credential = self._token_manager.credential
+
+        return self._token_manager
+
+    @staticmethod
+    def _throw_on_http_error(response: HttpResponse, description: str, valid_status: Optional[Set[int]] = None) -> None:
+        if valid_status and (response.status_code in valid_status):
+            return
+        if response.status_code >= 200 and response.status_code < 300:
+            # nothing to see here, move along
+            return
+
+        message = f"The {description} request failed with HTTP {response.status_code}"
+        try:
+            error_json = response.json()["error"]
+            additional_info = f"({error_json['code']}) {error_json['message']}"
+            message += f" - {additional_info}"
+        except (JSONDecodeError, ValueError, KeyError):
+            pass
+
+        raise EvaluationException(
+            message=message,
+            target=ErrorTarget.EVALUATE,
+            category=ErrorCategory.FAILED_EXECUTION,
+            blame=ErrorBlame.SYSTEM_ERROR,
+        )
+
+    def _generate_path(self, *paths: str) -> str:
+        sanitized_paths = [quote(path, safe="") for path in paths]
+        url = self._base_url + "/" + str.join("/", sanitized_paths)
+        return url
+
+    def _get_headers(self) -> Dict[str, str]:
+        return {"Authorization": f"Bearer {self.get_token()}", "Content-Type": "application/json"}