azure-ai-evaluation 1.0.0b1__tar.gz → 1.0.0b2__tar.gz

{azure_ai_evaluation-1.0.0b1 → azure_ai_evaluation-1.0.0b2}/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Release History
 
+## 1.0.0b2 (2024-09-24)
+
+### Breaking Changes
+
+- `data` and `evaluators` are now required keywords in `evaluate`.
+
 ## 1.0.0b1 (2024-09-20)
 
 ### Breaking Changes

{azure_ai_evaluation-1.0.0b1/azure_ai_evaluation.egg-info → azure_ai_evaluation-1.0.0b2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: azure-ai-evaluation
-Version: 1.0.0b1
+Version: 1.0.0b2
 Summary: Microsoft Azure Evaluation Library for Python
 Home-page: https://github.com/Azure/azure-sdk-for-python
 Author: Microsoft Corporation
@@ -35,11 +35,27 @@ Requires-Dist: promptflow-azure<2.0.0,>=1.15.0; extra == "pf-azure"
 
 # Azure AI Evaluation client library for Python
 
+We are excited to introduce the public preview of the Azure AI Evaluation SDK. 
+
+[Source code][source_code]
+| [Package (PyPI)][evaluation_pypi]
+| [API reference documentation][evaluation_ref_docs]
+| [Product documentation][product_documentation]
+| [Samples][evaluation_samples]
+
+This package has been tested with Python 3.8, 3.9, 3.10, 3.11, and 3.12.
+
+For a more complete set of Azure libraries, see https://aka.ms/azsdk/python/all
+
 ## Getting started
 
+### Prerequisites
+
+- Python 3.8 or later is required to use this package.
+
 ### Install the package
 
-Install the Azure AI Evaluation library for Python with:
+Install the Azure AI Evaluation library for Python with [pip][pip_link]::
 
 ```bash
 pip install azure-ai-evaluation
@@ -51,6 +67,8 @@ Evaluators are custom or prebuilt classes or functions that are designed to meas
 
 ## Examples
 
+### Evaluators
+
 Users can create evaluator runs on the local machine as shown in the example below:
 
 ```python
@@ -92,9 +110,9 @@ if __name__ == "__main__":
 
     # Initialize Project Scope
     azure_ai_project = {
-        "subscription_id": "e0fd569c-e34a-4249-8c24-e8d723c7f054",
-        "resource_group_name": "rg-test",
-        "project_name": "project-test",
+        "subscription_id": <subscription_id>,
+        "resource_group_name": <resource_group_name>,
+        "project_name": <project_name>
     }
 
     violence_eval = ViolenceEvaluator(azure_ai_project)
@@ -122,9 +140,13 @@ if __name__ == "__main__":
 
     pprint(result)
 ```
-## Simulator
+### Simulator
 
-Sample application prompty
+
+Simulators allow users to generate synthentic data using their application. Simulator expects the user to have a callback method that invokes
+their AI application.
+
+#### Simulating with a Prompty
 
 ```yaml
 ---
@@ -163,7 +185,7 @@ Application code:
 import json
 import asyncio
 from typing import Any, Dict, List, Optional
-from azure.ai.evaluation.synthetic import Simulator
+from azure.ai.evaluation.simulator import Simulator
 from promptflow.client import load_flow
 from azure.identity import DefaultAzureCredential
 import os
@@ -171,8 +193,7 @@ import os
 azure_ai_project = {
     "subscription_id": os.environ.get("AZURE_SUBSCRIPTION_ID"),
     "resource_group_name": os.environ.get("RESOURCE_GROUP"),
-    "project_name": os.environ.get("PROJECT_NAME"),
-    "credential": DefaultAzureCredential(),
+    "project_name": os.environ.get("PROJECT_NAME")
 }
 
 import wikipedia
@@ -249,8 +270,7 @@ if __name__ == "__main__":
     print("done!")
 ```
 
-Simulators allow users to generate synthentic data using their application. Simulator expects the user to have a callback method that invokes
-their AI application. Here's a sample of a callback which invokes AsyncAzureOpenAI:
+#### Adversarial Simulator
 
 ```python
 from from azure.ai.evaluation.simulator import AdversarialSimulator, AdversarialScenario
@@ -318,7 +338,9 @@ async def callback(
     }
 
 ```
-### Adversarial QA:
+
+#### Adversarial QA
+
 ```python
 scenario = AdversarialScenario.ADVERSARIAL_QA
 simulator = AdversarialSimulator(azure_ai_project=azure_ai_project, credential=DefaultAzureCredential())
@@ -334,7 +356,7 @@ outputs = asyncio.run(
 
 print(outputs.to_eval_qa_json_lines())
 ```
-### Direct Attack Simulator
+#### Direct Attack Simulator
 
 ```python
 scenario = AdversarialScenario.ADVERSARIAL_QA
@@ -353,13 +375,63 @@ print(outputs)
 ```
 ## Troubleshooting
 
+### General
+
+Azure ML clients raise exceptions defined in [Azure Core][azure_core_readme].
+
+### Logging
+
+This library uses the standard
+[logging][python_logging] library for logging.
+Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO
+level.
+
+Detailed DEBUG level logging, including request/response bodies and unredacted
+headers, can be enabled on a client with the `logging_enable` argument.
+
+See full SDK logging documentation with examples [here][sdk_logging_docs].
+
 ## Next steps
 
+- View our [samples][evaluation_samples].
+- View our [documentation][product_documentation]
+
 ## Contributing
 
+This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [cla.microsoft.com][cla].
+
+When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct][code_of_conduct]. For more information see the [Code of Conduct FAQ][coc_faq] or contact [opencode@microsoft.com][coc_contact] with any additional questions or comments.
+
+<!-- LINKS -->
+
+[source_code]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/evaluation/azure-ai-evaluation
+[evaluation_pypi]: https://pypi.org/project/azure-ai-evaluation/
+[evaluation_ref_docs]: https://learn.microsoft.com/python/api/azure-ai-evaluation/azure.ai.evaluation?view=azure-python-preview
+[evaluation_samples]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios
+[product_documentation]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk
+[python_logging]: https://docs.python.org/3/library/logging.html
+[sdk_logging_docs]: https://docs.microsoft.com/azure/developer/python/azure-sdk-logging
+[azure_core_readme]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/README.md
+[pip_link]: https://pypi.org/project/pip/
+[azure_core_ref_docs]: https://aka.ms/azsdk-python-core-policies
+[azure_core]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/README.md
+[azure_identity]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/identity/azure-identity
+[cla]: https://cla.microsoft.com
+[code_of_conduct]: https://opensource.microsoft.com/codeofconduct/
+[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/
+[coc_contact]: mailto:opencode@microsoft.com
+
 
 # Release History
 
+## 1.0.0b2 (2024-09-24)
+
+### Breaking Changes
+
+- `data` and `evaluators` are now required keywords in `evaluate`.
+
 ## 1.0.0b1 (2024-09-20)
 
 ### Breaking Changes

{azure_ai_evaluation-1.0.0b1 → azure_ai_evaluation-1.0.0b2}/README.md

@@ -1,10 +1,26 @@
 # Azure AI Evaluation client library for Python
 
+We are excited to introduce the public preview of the Azure AI Evaluation SDK. 
+
+[Source code][source_code]
+| [Package (PyPI)][evaluation_pypi]
+| [API reference documentation][evaluation_ref_docs]
+| [Product documentation][product_documentation]
+| [Samples][evaluation_samples]
+
+This package has been tested with Python 3.8, 3.9, 3.10, 3.11, and 3.12.
+
+For a more complete set of Azure libraries, see https://aka.ms/azsdk/python/all
+
 ## Getting started
 
+### Prerequisites
+
+- Python 3.8 or later is required to use this package.
+
 ### Install the package
 
-Install the Azure AI Evaluation library for Python with:
+Install the Azure AI Evaluation library for Python with [pip][pip_link]::
 
 ```bash
 pip install azure-ai-evaluation
@@ -16,6 +32,8 @@ Evaluators are custom or prebuilt classes or functions that are designed to meas
 
 ## Examples
 
+### Evaluators
+
 Users can create evaluator runs on the local machine as shown in the example below:
 
 ```python
@@ -57,9 +75,9 @@ if __name__ == "__main__":
 
     # Initialize Project Scope
     azure_ai_project = {
-        "subscription_id": "e0fd569c-e34a-4249-8c24-e8d723c7f054",
-        "resource_group_name": "rg-test",
-        "project_name": "project-test",
+        "subscription_id": <subscription_id>,
+        "resource_group_name": <resource_group_name>,
+        "project_name": <project_name>
     }
 
     violence_eval = ViolenceEvaluator(azure_ai_project)
@@ -87,9 +105,13 @@ if __name__ == "__main__":
 
     pprint(result)
 ```
-## Simulator
+### Simulator
+
 
-Sample application prompty
+Simulators allow users to generate synthentic data using their application. Simulator expects the user to have a callback method that invokes
+their AI application.
+
+#### Simulating with a Prompty
 
 ```yaml
 ---
@@ -128,7 +150,7 @@ Application code:
 import json
 import asyncio
 from typing import Any, Dict, List, Optional
-from azure.ai.evaluation.synthetic import Simulator
+from azure.ai.evaluation.simulator import Simulator
 from promptflow.client import load_flow
 from azure.identity import DefaultAzureCredential
 import os
@@ -136,8 +158,7 @@ import os
 azure_ai_project = {
     "subscription_id": os.environ.get("AZURE_SUBSCRIPTION_ID"),
     "resource_group_name": os.environ.get("RESOURCE_GROUP"),
-    "project_name": os.environ.get("PROJECT_NAME"),
-    "credential": DefaultAzureCredential(),
+    "project_name": os.environ.get("PROJECT_NAME")
 }
 
 import wikipedia
@@ -214,8 +235,7 @@ if __name__ == "__main__":
     print("done!")
 ```
 
-Simulators allow users to generate synthentic data using their application. Simulator expects the user to have a callback method that invokes
-their AI application. Here's a sample of a callback which invokes AsyncAzureOpenAI:
+#### Adversarial Simulator
 
 ```python
 from from azure.ai.evaluation.simulator import AdversarialSimulator, AdversarialScenario
@@ -283,7 +303,9 @@ async def callback(
     }
 
 ```
-### Adversarial QA:
+
+#### Adversarial QA
+
 ```python
 scenario = AdversarialScenario.ADVERSARIAL_QA
 simulator = AdversarialSimulator(azure_ai_project=azure_ai_project, credential=DefaultAzureCredential())
@@ -299,7 +321,7 @@ outputs = asyncio.run(
 
 print(outputs.to_eval_qa_json_lines())
 ```
-### Direct Attack Simulator
+#### Direct Attack Simulator
 
 ```python
 scenario = AdversarialScenario.ADVERSARIAL_QA
@@ -318,6 +340,50 @@ print(outputs)
 ```
 ## Troubleshooting
 
+### General
+
+Azure ML clients raise exceptions defined in [Azure Core][azure_core_readme].
+
+### Logging
+
+This library uses the standard
+[logging][python_logging] library for logging.
+Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO
+level.
+
+Detailed DEBUG level logging, including request/response bodies and unredacted
+headers, can be enabled on a client with the `logging_enable` argument.
+
+See full SDK logging documentation with examples [here][sdk_logging_docs].
+
 ## Next steps
 
+- View our [samples][evaluation_samples].
+- View our [documentation][product_documentation]
+
 ## Contributing
+
+This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [cla.microsoft.com][cla].
+
+When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct][code_of_conduct]. For more information see the [Code of Conduct FAQ][coc_faq] or contact [opencode@microsoft.com][coc_contact] with any additional questions or comments.
+
+<!-- LINKS -->
+
+[source_code]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/evaluation/azure-ai-evaluation
+[evaluation_pypi]: https://pypi.org/project/azure-ai-evaluation/
+[evaluation_ref_docs]: https://learn.microsoft.com/python/api/azure-ai-evaluation/azure.ai.evaluation?view=azure-python-preview
+[evaluation_samples]: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios
+[product_documentation]: https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk
+[python_logging]: https://docs.python.org/3/library/logging.html
+[sdk_logging_docs]: https://docs.microsoft.com/azure/developer/python/azure-sdk-logging
+[azure_core_readme]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/README.md
+[pip_link]: https://pypi.org/project/pip/
+[azure_core_ref_docs]: https://aka.ms/azsdk-python-core-policies
+[azure_core]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/README.md
+[azure_identity]: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/identity/azure-identity
+[cla]: https://cla.microsoft.com
+[code_of_conduct]: https://opensource.microsoft.com/codeofconduct/
+[coc_faq]: https://opensource.microsoft.com/codeofconduct/faq/
+[coc_contact]: mailto:opencode@microsoft.com

{azure_ai_evaluation-1.0.0b1 → azure_ai_evaluation-1.0.0b2}/azure/ai/evaluation/__init__.py

@@ -25,11 +25,7 @@ from ._evaluators._relevance import RelevanceEvaluator
 from ._evaluators._rouge import RougeScoreEvaluator, RougeType
 from ._evaluators._similarity import SimilarityEvaluator
 from ._evaluators._xpia import IndirectAttackEvaluator
-from ._model_configurations import (
-    AzureAIProject,
-    AzureOpenAIModelConfiguration,
-    OpenAIModelConfiguration,
-)
+from ._model_configurations import AzureAIProject, AzureOpenAIModelConfiguration, OpenAIModelConfiguration
 
 __all__ = [
     "evaluate",

{azure_ai_evaluation-1.0.0b1 → azure_ai_evaluation-1.0.0b2}/azure/ai/evaluation/_common/rai_service.py

@@ -11,12 +11,12 @@ from urllib.parse import urlparse
 
 import jwt
 import numpy as np
-from azure.core.credentials import TokenCredential
-from azure.identity import DefaultAzureCredential
 
+from azure.ai.evaluation._exceptions import ErrorBlame, ErrorCategory, ErrorTarget, EvaluationException
 from azure.ai.evaluation._http_utils import get_async_http_client
-from azure.ai.evaluation._exceptions import EvaluationException, ErrorBlame, ErrorCategory, ErrorTarget
 from azure.ai.evaluation._model_configurations import AzureAIProject
+from azure.core.credentials import TokenCredential
+from azure.identity import DefaultAzureCredential
 
 from .constants import (
     CommonConstants,
@@ -348,7 +348,7 @@ async def _get_service_discovery_url(azure_ai_project: AzureAIProject, token: st
         )
 
     if response.status_code != 200:
-        msg = f"Failed to retrieve the discovery service URL."
+        msg = "Failed to retrieve the discovery service URL."
         raise EvaluationException(
             message=msg,
             internal_message=msg,

{azure_ai_evaluation-1.0.0b1 → azure_ai_evaluation-1.0.0b2}/azure/ai/evaluation/_common/utils.py

@@ -2,20 +2,15 @@
 # Copyright (c) Microsoft Corporation. All rights reserved.
 # ---------------------------------------------------------
 
-from typing import Optional, Union
-
-from azure.ai.evaluation._model_configurations import AzureOpenAIModelConfiguration, OpenAIModelConfiguration
+import threading
+from typing import List, Optional, Union
 
-try:
-    from . import constants
-except ImportError:
-    import constants
+import nltk
+import numpy as np
 
-from typing import List
+from azure.ai.evaluation._model_configurations import AzureOpenAIModelConfiguration, OpenAIModelConfiguration
 
-import threading
-import numpy as np
-import nltk
+from . import constants
 
 _nltk_data_download_lock = threading.Lock()
 
@@ -46,7 +41,7 @@ def ensure_nltk_data_downloaded():
     """Download NLTK data packages if not already downloaded."""
     with _nltk_data_download_lock:
         try:
-            from nltk.tokenize.nist import NISTTokenizer
+            from nltk.tokenize.nist import NISTTokenizer  # pylint: disable=unused-import
         except LookupError:
             nltk.download("perluniprops")
             nltk.download("punkt")
@@ -54,12 +49,19 @@ def ensure_nltk_data_downloaded():
 
 
 def nltk_tokenize(text: str) -> List[str]:
-    """Tokenize the input text using the NLTK tokenizer."""
+    """Tokenize the input text using the NLTK tokenizer.
+
+    :param text: The text to tokenize
+    :type text: str
+    :return: A list of tokens
+    :rtype: list[str]
+    """
     ensure_nltk_data_downloaded()
 
     if not text.isascii():
         # Use NISTTokenizer for international tokenization
         from nltk.tokenize.nist import NISTTokenizer
+
         tokens = NISTTokenizer().international_tokenize(text)
     else:
         # By default, use NLTK word tokenizer
@@ -68,20 +70,18 @@ def nltk_tokenize(text: str) -> List[str]:
     return list(tokens)
 
 
-def check_and_add_api_version_for_aoai_model_config(
+def ensure_api_version_in_aoai_model_config(
     model_config: Union[AzureOpenAIModelConfiguration, OpenAIModelConfiguration],
     default_api_version: str,
 ) -> None:
-    if (
-        "azure_endpoint" in model_config or "azure_deployment" in model_config
-    ):
+    if "azure_endpoint" in model_config or "azure_deployment" in model_config:
         model_config["api_version"] = model_config.get("api_version", default_api_version)
 
 
-def check_and_add_user_agent_for_aoai_model_config(
+def ensure_user_agent_in_aoai_model_config(
     model_config: Union[AzureOpenAIModelConfiguration, OpenAIModelConfiguration],
     prompty_model_config: dict,
     user_agent: Optional[str] = None,
 ) -> None:
     if user_agent and ("azure_endpoint" in model_config or "azure_deployment" in model_config):
-        prompty_model_config["parameters"]["extra_headers"].update({"x-ms-useragent": user_agent})
+        prompty_model_config["parameters"]["extra_headers"].update({"x-ms-useragent": user_agent})

{azure_ai_evaluation-1.0.0b1 → azure_ai_evaluation-1.0.0b2}/azure/ai/evaluation/_constants.py

@@ -39,6 +39,15 @@ class Prefixes:
     TSG_OUTPUTS = "__outputs."
 
 
+class DefaultOpenEncoding:
+    """Enum that captures SDK's default values for the encoding param of open(...)"""
+
+    READ = "utf-8-sig"
+    """SDK Default Encoding when reading a file"""
+    WRITE = "utf-8"
+    """SDK Default Encoding when writing a file"""
+
+
 DEFAULT_EVALUATION_RESULTS_FILE_NAME = "evaluation_results.json"
 
 CONTENT_SAFETY_DEFECT_RATE_THRESHOLD_DEFAULT = 4

{azure_ai_evaluation-1.0.0b1 → azure_ai_evaluation-1.0.0b2}/azure/ai/evaluation/_evaluate/_batch_run_client/batch_run_context.py

@@ -5,13 +5,14 @@ import os
 
 from promptflow._sdk._constants import PF_FLOW_ENTRY_IN_TMP, PF_FLOW_META_LOAD_IN_SUBPROCESS
 from promptflow._utils.user_agent_utils import ClientUserAgentUtil
+from promptflow.tracing._integrations._openai_injector import inject_openai_api, recover_openai_api
+
 from azure.ai.evaluation._constants import (
     OTEL_EXPORTER_OTLP_TRACES_TIMEOUT,
     OTEL_EXPORTER_OTLP_TRACES_TIMEOUT_DEFAULT,
     PF_BATCH_TIMEOUT_SEC,
     PF_BATCH_TIMEOUT_SEC_DEFAULT,
 )
-from promptflow.tracing._integrations._openai_injector import inject_openai_api, recover_openai_api
 
 from ..._user_agent import USER_AGENT
 from .._utils import set_event_loop_policy

{azure_ai_evaluation-1.0.0b1 → azure_ai_evaluation-1.0.0b2}/azure/ai/evaluation/_evaluate/_batch_run_client/code_client.py

@@ -4,13 +4,16 @@
 import inspect
 import json
 import logging
+import os
+from pathlib import Path
+from typing import Callable, Dict, Optional, Union
 
 import pandas as pd
-
 from promptflow.contracts.types import AttrDict
-from azure.ai.evaluation._evaluate._utils import _apply_column_mapping, _has_aggregator, get_int_env_var, load_jsonl
 from promptflow.tracing import ThreadPoolExecutorWithContext as ThreadPoolExecutor
-from azure.ai.evaluation._exceptions import EvaluationException, ErrorBlame, ErrorCategory, ErrorTarget
+
+from azure.ai.evaluation._evaluate._utils import _apply_column_mapping, _has_aggregator, get_int_env_var, load_jsonl
+from azure.ai.evaluation._exceptions import ErrorBlame, ErrorCategory, ErrorTarget, EvaluationException
 
 from ..._constants import PF_BATCH_TIMEOUT_SEC, PF_BATCH_TIMEOUT_SEC_DEFAULT
 
@@ -18,7 +21,9 @@ LOGGER = logging.getLogger(__name__)
 
 
 class CodeRun:
-    def __init__(self, run, input_data, evaluator_name=None, aggregated_metrics=None, **kwargs):
+    def __init__(
+        self, run, input_data, evaluator_name=None, aggregated_metrics=None, **kwargs  # pylint: disable=unused-argument
+    ):
         self.run = run
         self.evaluator_name = evaluator_name if evaluator_name is not None else ""
         self.input_data = input_data
@@ -40,13 +45,13 @@ class CodeRun:
                 else None
             )
         except Exception as ex:  # pylint: disable=broad-exception-caught
-            LOGGER.debug(f"Error calculating metrics for evaluator {self.evaluator_name}, failed with error {str(ex)}")
+            LOGGER.debug("Error calculating metrics for evaluator %s, failed with error %s", self.evaluator_name, ex)
             aggregated_metrics = None
 
         if not isinstance(aggregated_metrics, dict):
             LOGGER.warning(
-                f"Aggregated metrics for evaluator {self.evaluator_name}"
-                f" is not a dictionary will not be logged as metrics"
+                "Aggregated metrics for evaluator %s is not a dictionary will not be logged as metrics",
+                self.evaluator_name,
             )
 
         aggregated_metrics = aggregated_metrics if isinstance(aggregated_metrics, dict) else {}
@@ -54,11 +59,15 @@ class CodeRun:
         return aggregated_metrics
 
 
-class CodeClient:
-    def __init__(self):
+class CodeClient:  # pylint: disable=client-accepts-api-version-keyword
+    def __init__(  # pylint: disable=missing-client-constructor-parameter-credential,missing-client-constructor-parameter-kwargs
+        self,
+    ) -> None:
         self._thread_pool = ThreadPoolExecutor(thread_name_prefix="evaluators_thread")
 
-    def _calculate_metric(self, evaluator, input_df, column_mapping, evaluator_name):
+    def _calculate_metric(
+        self, evaluator: Callable, input_df: pd.DataFrame, column_mapping: Optional[Dict[str, str]], evaluator_name: str
+    ) -> pd.DataFrame:
         row_metric_futures = []
         row_metric_results = []
         input_df = _apply_column_mapping(input_df, column_mapping)
@@ -110,18 +119,25 @@ class CodeClient:
                 return aggregated_output
         except Exception as ex:  # pylint: disable=broad-exception-caught
             LOGGER.warning(
-                f"Error calculating aggregations for evaluator {run.evaluator_name}," f" failed with error {str(ex)}"
+                "Error calculating aggregations for evaluator %s, failed with error %s", run.evaluator_name, ex
             )
         return None
 
-    def run(self, flow, data, evaluator_name=None, column_mapping=None, **kwargs):
+    def run(
+        self,  # pylint: disable=unused-argument
+        flow: Callable,
+        data: Union[os.PathLike, Path, pd.DataFrame],
+        evaluator_name: Optional[str] = None,
+        column_mapping: Optional[Dict[str, str]] = None,
+        **kwargs,
+    ) -> CodeRun:
         input_df = data
         if not isinstance(input_df, pd.DataFrame):
             try:
                 json_data = load_jsonl(data)
             except json.JSONDecodeError as exc:
                 raise EvaluationException(
-                    message = f"Failed to parse data as JSON: {data}. Provide valid json lines data.",
+                    message=f"Failed to parse data as JSON: {data}. Provide valid json lines data.",
                     internal_message="Failed to parse data as JSON",
                     target=ErrorTarget.CODE_CLIENT,
                     category=ErrorCategory.INVALID_VALUE,
@@ -129,22 +145,28 @@ class CodeClient:
                 ) from exc
 
             input_df = pd.DataFrame(json_data)
-        eval_future = self._thread_pool.submit(self._calculate_metric, flow, input_df, column_mapping, evaluator_name)
+        eval_future = self._thread_pool.submit(
+            self._calculate_metric,
+            evaluator=flow,
+            input_df=input_df,
+            column_mapping=column_mapping,
+            evaluator_name=evaluator_name,
+        )
         run = CodeRun(run=eval_future, input_data=data, evaluator_name=evaluator_name, aggregated_metrics=None)
         aggregation_future = self._thread_pool.submit(self._calculate_aggregations, evaluator=flow, run=run)
         run.aggregated_metrics = aggregation_future
         return run
 
-    def get_details(self, run, all_results=False):
+    def get_details(self, run: CodeRun, all_results: bool = False) -> pd.DataFrame:
         result_df = run.get_result_df(exclude_inputs=not all_results)
         return result_df
 
-    def get_metrics(self, run):
+    def get_metrics(self, run: CodeRun) -> Optional[None]:
         try:
             aggregated_metrics = run.get_aggregated_metrics()
             print("Aggregated metrics")
             print(aggregated_metrics)
         except Exception as ex:  # pylint: disable=broad-exception-caught
-            LOGGER.debug(f"Error calculating metrics for evaluator {run.evaluator_name}, failed with error {str(ex)}")
+            LOGGER.debug("Error calculating metrics for evaluator %s, failed with error %s", run.evaluator_name, ex)
             return None
         return aggregated_metrics