agentx-python 0.6.3__tar.gz → 0.6.5__tar.gz

{agentx_python-0.6.3/agentx_python.egg-info → agentx_python-0.6.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentx-python
-Version: 0.6.3
+Version: 0.6.5
 Summary: Official Python SDK for AgentX (https://www.agentx.so/)
 Home-page: https://github.com/AgentX-ai/AgentX-python
 Author: Robin Wang and AgentX Team
@@ -49,8 +49,12 @@ Dynamic: summary
 ![Logo](https://agentx-resources.s3.us-west-1.amazonaws.com/AgentX-logo-387x60.png)
 
 [![PyPI version](https://img.shields.io/pypi/v/agentx-python)](https://pypi.org/project/agentx-python/)
+[![Python versions](https://img.shields.io/pypi/pyversions/agentx-python)](https://pypi.org/project/agentx-python/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
-The official Python SDK for **[AgentX](https://www.agentx.so/)** — build, chat with, and orchestrate AI agents in a few lines of code.
+The official Python SDK for **[AgentX](https://app.agentx.so/)** — build, chat with, orchestrate, and trace AI agents in a few lines of code.
+
+Also see [SDK Developer Docs](https://developers.agentx.so), [API Reference Docs](https://docs.agentx.so/reference)
 
 ---
 
@@ -64,8 +68,9 @@ The official Python SDK for **[AgentX](https://www.agentx.so/)** — build, chat
   - [List agents](#list-agents)
   - [Start a conversation](#start-a-conversation)
   - [Chat (streaming and non-streaming)](#chat-streaming-and-non-streaming)
-- [Workforce (multi-agent orchestration)](#workforce-multi-agent-orchestration)
-- [Agent Evaluations](#custom-agent-evaluations) — LLM-as-a-judge, cosine / Jaccard similarity
+- [Workforce (multi-agent orchestration)](#workforce-multi-agent-orchestration) — teams of agents with a designated manager
+- [Production tracing](#production-tracing) — record live agent runs from any framework
+- [Custom agent evaluations](#custom-agent-evaluations) — LLM-as-a-judge, cosine / Jaccard similarity
 - [Links](#links)
 
 ---
@@ -78,6 +83,7 @@ The official Python SDK for **[AgentX](https://www.agentx.so/)** — build, chat
 - **Batteries included** — voice (ASR/TTS), image generation, document/CSV/Excel/OCR, RAG with built-in re-ranking.
 - **MCP support** — connect any Model Context Protocol server.
 - **Multi-agent orchestration** — workforces of agents with a designated manager, across LLM vendors.
+- **Production tracing** — one decorator or context manager records every agent run (input, output, latency, tool calls, token usage) into your workspace, for any framework.
 - **Agent Evaluations** — score any agent (LangChain, CrewAI, OpenAI, Anthropic, HTTP, …) with LLM-as-a-judge ratings plus optional cosine and Jaccard similarity metrics.
 - **A2A** — Each agent can be published with agent-to-agent protocol compatible.
 
@@ -187,6 +193,58 @@ for chunk in workforce.chat_stream(conversation.id, "How can you help me with th
 
 ---
 
+## Production tracing
+
+Record live agent runs into your workspace with a single decorator or context manager — no changes to your agent's logic. Traces appear in the **Live Traces** tab and can be evaluated against your test datasets with [`tracer.evaluate_trace()`](TRACING.md#tracerevaluate_trace).
+
+```python
+from agentx import AgentX
+
+client = AgentX.from_env()
+tracer = client.tracer
+
+@tracer.trace("customer-support-agent", framework="langchain", model="gpt-4o")
+def handle_query(query: str) -> str:
+    return chain.invoke(query)
+
+# Every call is automatically traced: input, output, latency, tool calls, token usage
+handle_query("How do I reset my password?")
+tracer.flush(timeout=10)  # ensure delivery before the process exits
+```
+
+Prefer full control over what gets captured? Use the context manager instead:
+
+```python
+with tracer.trace("rag-agent", framework="langchain") as span:
+    span.input = {"query": query, "user_id": user_id}
+
+    kb_result = search_knowledge_base(query)
+    span.add_tool_call("search_knowledge_base", input=query, output=kb_result, latency_ms=190)
+
+    span.output = llm.invoke(f"Context: {kb_result}\n\nQuery: {query}")
+```
+
+### Framework integrations
+
+Each integration auto-captures LLM calls, tool calls, and token usage — install the matching extra:
+
+| Framework             | Install                                      | Integration              |
+| --------------------- | -------------------------------------------- | ------------------------ |
+| LangChain             | `pip install "agentx-python[langchain]"`     | `AgentXCallbackHandler`  |
+| CrewAI                | `pip install "agentx-python[crewai]"`        | `AgentXCrewObserver`     |
+| OpenAI Agents SDK     | `pip install "agentx-python[openai-agents]"` | `AgentXTracingProcessor` |
+| Anthropic             | `pip install "agentx-python[anthropic]"`     | `patch_anthropic_client` |
+| Google ADK            | `pip install "agentx-python[google-adk]"`    | `AgentXADKPlugin`        |
+| Google GenAI (Gemini) | `pip install "agentx-python[google-genai]"`  | `patch_genai_client`     |
+
+Or plain Python — wrap any function with `@tracer.trace(...)` and it just works, no framework required.
+
+Running specialist agents in parallel with a `ThreadPoolExecutor`? Wrap each worker body in `tracer.use_span(span)` so their steps land on the parent trace instead of becoming independent traces — see [TRACING.md](TRACING.md) for the full pattern.
+
+See **[TRACING.md](TRACING.md)** for the complete guide — session grouping, error handling, async support, and the full API reference.
+
+---
+
 ## Custom agent evaluations
 
 Evaluate **any** AI agent — LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP endpoints, or plain Python — using AgentX as the scoring and reporting backend. Includes optional **cosine** and **Jaccard** similarity metrics alongside LLM-graded ratings.
@@ -214,4 +272,5 @@ See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide — dataset builder,
 - **Dashboard** — [app.agentx.so](https://app.agentx.so)
 - **Website** — [agentx.so](https://www.agentx.so/)
 - **PyPI** — [agentx-python](https://pypi.org/project/agentx-python/)
+- **Tracing docs** — [TRACING.md](TRACING.md)
 - **Evaluations docs** — [EVALUATIONS.md](EVALUATIONS.md)

{agentx_python-0.6.3 → agentx_python-0.6.5}/README.md

@@ -1,8 +1,12 @@
 ![Logo](https://agentx-resources.s3.us-west-1.amazonaws.com/AgentX-logo-387x60.png)
 
 [![PyPI version](https://img.shields.io/pypi/v/agentx-python)](https://pypi.org/project/agentx-python/)
+[![Python versions](https://img.shields.io/pypi/pyversions/agentx-python)](https://pypi.org/project/agentx-python/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
-The official Python SDK for **[AgentX](https://www.agentx.so/)** — build, chat with, and orchestrate AI agents in a few lines of code.
+The official Python SDK for **[AgentX](https://app.agentx.so/)** — build, chat with, orchestrate, and trace AI agents in a few lines of code.
+
+Also see [SDK Developer Docs](https://developers.agentx.so), [API Reference Docs](https://docs.agentx.so/reference)
 
 ---
 
@@ -16,8 +20,9 @@ The official Python SDK for **[AgentX](https://www.agentx.so/)** — build, chat
   - [List agents](#list-agents)
   - [Start a conversation](#start-a-conversation)
   - [Chat (streaming and non-streaming)](#chat-streaming-and-non-streaming)
-- [Workforce (multi-agent orchestration)](#workforce-multi-agent-orchestration)
-- [Agent Evaluations](#custom-agent-evaluations) — LLM-as-a-judge, cosine / Jaccard similarity
+- [Workforce (multi-agent orchestration)](#workforce-multi-agent-orchestration) — teams of agents with a designated manager
+- [Production tracing](#production-tracing) — record live agent runs from any framework
+- [Custom agent evaluations](#custom-agent-evaluations) — LLM-as-a-judge, cosine / Jaccard similarity
 - [Links](#links)
 
 ---
@@ -30,6 +35,7 @@ The official Python SDK for **[AgentX](https://www.agentx.so/)** — build, chat
 - **Batteries included** — voice (ASR/TTS), image generation, document/CSV/Excel/OCR, RAG with built-in re-ranking.
 - **MCP support** — connect any Model Context Protocol server.
 - **Multi-agent orchestration** — workforces of agents with a designated manager, across LLM vendors.
+- **Production tracing** — one decorator or context manager records every agent run (input, output, latency, tool calls, token usage) into your workspace, for any framework.
 - **Agent Evaluations** — score any agent (LangChain, CrewAI, OpenAI, Anthropic, HTTP, …) with LLM-as-a-judge ratings plus optional cosine and Jaccard similarity metrics.
 - **A2A** — Each agent can be published with agent-to-agent protocol compatible.
 
@@ -139,6 +145,58 @@ for chunk in workforce.chat_stream(conversation.id, "How can you help me with th
 
 ---
 
+## Production tracing
+
+Record live agent runs into your workspace with a single decorator or context manager — no changes to your agent's logic. Traces appear in the **Live Traces** tab and can be evaluated against your test datasets with [`tracer.evaluate_trace()`](TRACING.md#tracerevaluate_trace).
+
+```python
+from agentx import AgentX
+
+client = AgentX.from_env()
+tracer = client.tracer
+
+@tracer.trace("customer-support-agent", framework="langchain", model="gpt-4o")
+def handle_query(query: str) -> str:
+    return chain.invoke(query)
+
+# Every call is automatically traced: input, output, latency, tool calls, token usage
+handle_query("How do I reset my password?")
+tracer.flush(timeout=10)  # ensure delivery before the process exits
+```
+
+Prefer full control over what gets captured? Use the context manager instead:
+
+```python
+with tracer.trace("rag-agent", framework="langchain") as span:
+    span.input = {"query": query, "user_id": user_id}
+
+    kb_result = search_knowledge_base(query)
+    span.add_tool_call("search_knowledge_base", input=query, output=kb_result, latency_ms=190)
+
+    span.output = llm.invoke(f"Context: {kb_result}\n\nQuery: {query}")
+```
+
+### Framework integrations
+
+Each integration auto-captures LLM calls, tool calls, and token usage — install the matching extra:
+
+| Framework             | Install                                      | Integration              |
+| --------------------- | -------------------------------------------- | ------------------------ |
+| LangChain             | `pip install "agentx-python[langchain]"`     | `AgentXCallbackHandler`  |
+| CrewAI                | `pip install "agentx-python[crewai]"`        | `AgentXCrewObserver`     |
+| OpenAI Agents SDK     | `pip install "agentx-python[openai-agents]"` | `AgentXTracingProcessor` |
+| Anthropic             | `pip install "agentx-python[anthropic]"`     | `patch_anthropic_client` |
+| Google ADK            | `pip install "agentx-python[google-adk]"`    | `AgentXADKPlugin`        |
+| Google GenAI (Gemini) | `pip install "agentx-python[google-genai]"`  | `patch_genai_client`     |
+
+Or plain Python — wrap any function with `@tracer.trace(...)` and it just works, no framework required.
+
+Running specialist agents in parallel with a `ThreadPoolExecutor`? Wrap each worker body in `tracer.use_span(span)` so their steps land on the parent trace instead of becoming independent traces — see [TRACING.md](TRACING.md) for the full pattern.
+
+See **[TRACING.md](TRACING.md)** for the complete guide — session grouping, error handling, async support, and the full API reference.
+
+---
+
 ## Custom agent evaluations
 
 Evaluate **any** AI agent — LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI, Anthropic, HTTP endpoints, or plain Python — using AgentX as the scoring and reporting backend. Includes optional **cosine** and **Jaccard** similarity metrics alongside LLM-graded ratings.
@@ -166,4 +224,5 @@ See **[EVALUATIONS.md](EVALUATIONS.md)** for the full guide — dataset builder,
 - **Dashboard** — [app.agentx.so](https://app.agentx.so)
 - **Website** — [agentx.so](https://www.agentx.so/)
 - **PyPI** — [agentx-python](https://pypi.org/project/agentx-python/)
+- **Tracing docs** — [TRACING.md](TRACING.md)
 - **Evaluations docs** — [EVALUATIONS.md](EVALUATIONS.md)

{agentx_python-0.6.3 → agentx_python-0.6.5}/agentx/agentx.py

@@ -1,4 +1,4 @@
-from typing import List
+from typing import List, Optional
 import requests
 import os
 import logging
@@ -10,7 +10,12 @@ from agentx.resources.workforce import Workforce
 
 class AgentX:
 
-    def __init__(self, api_key: str = None, base_url: str = None, workspace_id: str = None):
+    def __init__(
+        self,
+        api_key: Optional[str] = None,
+        base_url: Optional[str] = None,
+        workspace_id: Optional[str] = None,
+    ):
         self.api_key = api_key or os.getenv("AGENTX_API_KEY")
         if self.api_key and not os.getenv("AGENTX_API_KEY"):
             os.environ["AGENTX_API_KEY"] = self.api_key
@@ -32,6 +37,7 @@ class AgentX:
             api_key=self.api_key,
             sdk_version=VERSION,
             base_url=self.base_url,
+            workspace_id=self.workspace_id,
         )
         self.evaluations = EvaluationsRunner(_eval_client)
 

agentx_python-0.6.5/agentx/evaluations/__init__.py

@@ -0,0 +1,7 @@
+from agentx.evaluations.client import EvaluationsClient
+from agentx.evaluations.evaluation_settings import (
+    EvaluationSettingsBuilder,
+    EvaluationSettingsClient,
+)
+
+__all__ = ["EvaluationsClient", "EvaluationSettingsBuilder", "EvaluationSettingsClient"]

{agentx_python-0.6.3 → agentx_python-0.6.5}/agentx/evaluations/client.py

@@ -13,6 +13,7 @@ from agentx.evaluations.models import (
     Dataset,
     EvaluationResult,
     EvaluationRun,
+    EvaluationSettings,
     EvaluationSubject,
     ModelInfo,
     Report,
@@ -44,12 +45,20 @@ class AgentXValidationError(AgentXEvaluationsError):
 
 class EvaluationsClient:
     def __init__(
-        self, api_key: str, sdk_version: str = "unknown", base_url: str = None
+        self,
+        api_key: str,
+        sdk_version: str = "unknown",
+        base_url: Optional[str] = None,
+        workspace_id: Optional[str] = None,
     ):
         if not api_key:
             raise AgentXAuthError("AGENTX_API_KEY is required")
         self._api_key = api_key
         self._sdk_version = sdk_version
+        # Falls back to the caller's default workspace server-side when unset — see
+        # _with_workspace(). Without this, dataset/settings/run creation silently land in
+        # whatever workspace the API key's user defaults to, not the one the caller intended.
+        self._workspace_id = workspace_id
         # Priority: constructor arg > env var > SDK default
         # Always append /custom-agent-evaluations so users only need to provide /api/v1
         _api_base = (
@@ -67,15 +76,29 @@ class EvaluationsClient:
                 "accept": "*/*",
             }
         )
-        # Expose dataset builder factory
+        # Expose dataset / evaluation-settings builder factories
         from agentx.evaluations.datasets import DatasetClient
+        from agentx.evaluations.evaluation_settings import EvaluationSettingsClient
 
         self.datasets = DatasetClient(self)
+        self.settings = EvaluationSettingsClient(self)
 
     # ------------------------------------------------------------------
     # Low-level HTTP
     # ------------------------------------------------------------------
 
+    def _with_workspace(self, payload: dict) -> dict:
+        """Injects the client's workspace_id into a request payload, unless the caller already
+        set one explicitly. Without this, requests silently fall back to the API key user's
+        default workspace server-side, which may not be the workspace the caller intended."""
+        if self._workspace_id and not payload.get("workspaceId"):
+            return {**payload, "workspaceId": self._workspace_id}
+        return payload
+
+    def _workspace_params(self) -> Optional[dict]:
+        """Same as _with_workspace(), for GET requests that take workspaceId as a query param."""
+        return {"workspaceId": self._workspace_id} if self._workspace_id else None
+
     def _request(self, method: str, path: str, timeout: int = 30, **kwargs) -> Any:
         url = f"{self._base_url}{path}"
         last_exc: Optional[Exception] = None
@@ -125,20 +148,52 @@ class EvaluationsClient:
     # ------------------------------------------------------------------
 
     def create_dataset(self, payload: dict) -> Dataset:
-        data = self._request("POST", "/datasets", json=payload)
+        data = self._request("POST", "/datasets", json=self._with_workspace(payload))
         return Dataset(**data)
 
     def list_datasets(self) -> List[Dataset]:
-        data = self._request("GET", "/datasets")
+        data = self._request("GET", "/datasets", params=self._workspace_params())
         return [
             Dataset(**d)
             for d in (data if isinstance(data, list) else data.get("datasets", []))
         ]
 
     def get_dataset(self, dataset_id: str) -> Dataset:
-        data = self._request("GET", f"/datasets/{dataset_id}")
+        data = self._request(
+            "GET", f"/datasets/{dataset_id}", params=self._workspace_params()
+        )
         return Dataset(**data)
 
+    # ------------------------------------------------------------------
+    # Evaluation Settings endpoints — standalone grading config, reusable
+    # across datasets.
+    # ------------------------------------------------------------------
+
+    def create_evaluation_settings(self, payload: dict) -> EvaluationSettings:
+        data = self._request(
+            "POST", "/evaluation-settings", json=self._with_workspace(payload)
+        )
+        return EvaluationSettings(**data)
+
+    def list_evaluation_settings(self) -> List[EvaluationSettings]:
+        data = self._request(
+            "GET", "/evaluation-settings", params=self._workspace_params()
+        )
+        return [
+            EvaluationSettings(**e)
+            for e in (
+                data if isinstance(data, list) else data.get("evaluationSettings", [])
+            )
+        ]
+
+    def get_evaluation_settings(self, evaluation_settings_id: str) -> EvaluationSettings:
+        data = self._request(
+            "GET",
+            f"/evaluation-settings/{evaluation_settings_id}",
+            params=self._workspace_params(),
+        )
+        return EvaluationSettings(**data)
+
     # ------------------------------------------------------------------
     # Run endpoints
     # ------------------------------------------------------------------
@@ -148,6 +203,7 @@ class EvaluationsClient:
         dataset_id: str,
         subject: EvaluationSubject,
         python_version: Optional[str] = None,
+        evaluation_settings_id: Optional[str] = None,
     ) -> EvaluationRun:
         from agentx.version import VERSION
 
@@ -162,7 +218,9 @@ class EvaluationsClient:
                 "pythonVersion": python_version or _python_version(),
             },
         }
-        data = self._request("POST", "/runs", json=payload)
+        if evaluation_settings_id:
+            payload["evaluationSettingsId"] = evaluation_settings_id
+        data = self._request("POST", "/runs", json=self._with_workspace(payload))
         return EvaluationRun(**data)
 
     def append_results(
@@ -207,6 +265,7 @@ def _result_to_payload(r: EvaluationResult) -> dict:
     d["questionIndex"] = d.pop("question_index", d.get("questionIndex"))
     d["runNumber"] = d.pop("run_number", d.get("runNumber"))
     d["idempotencyKey"] = d.pop("idempotency_key", d.get("idempotencyKey"))
+    d["traceId"] = d.pop("trace_id", d.get("traceId"))
     return {k: v for k, v in d.items() if v is not None}
 
 

{agentx_python-0.6.3 → agentx_python-0.6.5}/agentx/evaluations/datasets.py

@@ -29,6 +29,8 @@ class DatasetBuilder:
         evaluation_criteria: Optional[str] = None,
         vector_similarity: bool = False,
         jaccard_similarity: bool = False,
+        bleu_score: bool = False,
+        rouge_score: bool = False,
         similarity_model: Optional[str] = None,
         sovereignty_models: Optional[List[str]] = None,
     ):
@@ -43,7 +45,8 @@ class DatasetBuilder:
             "questions": [],
         }
         # Opt-in similarity metrics, surfaced on the report as cosine_similarity /
-        # jaccard_similarity (computed against each case's expected_results).
+        # jaccard_similarity / bleu_score / rouge_score (computed against each
+        # case's expected_results).
         if vector_similarity:
             vs: Dict[str, Any] = {"enabled": True}
             if similarity_model:
@@ -51,6 +54,10 @@ class DatasetBuilder:
             self._payload["vectorSimilarity"] = vs
         if jaccard_similarity:
             self._payload["jaccardSimilarity"] = {"enabled": True}
+        if bleu_score:
+            self._payload["bleuScore"] = {"enabled": True}
+        if rouge_score:
+            self._payload["rougeScore"] = {"enabled": True}
         # Sovereignty & Portability — the models to compare on this dataset (use
         # client.evaluations.list_models() to discover valid ids).
         if sovereignty_models:
@@ -212,6 +219,8 @@ class DatasetClient:
         evaluation_criteria: Optional[str] = None,
         vector_similarity: bool = False,
         jaccard_similarity: bool = False,
+        bleu_score: bool = False,
+        rouge_score: bool = False,
         similarity_model: Optional[str] = None,
         sovereignty_models: Optional[List[str]] = None,
     ) -> DatasetBuilder:
@@ -225,6 +234,8 @@ class DatasetClient:
             evaluation_criteria=evaluation_criteria,
             vector_similarity=vector_similarity,
             jaccard_similarity=jaccard_similarity,
+            bleu_score=bleu_score,
+            rouge_score=rouge_score,
             similarity_model=similarity_model,
             sovereignty_models=sovereignty_models,
         )

agentx_python-0.6.5/agentx/evaluations/evaluation_settings.py

@@ -0,0 +1,109 @@
+from __future__ import annotations
+
+import logging
+from typing import Any, Dict, List, Optional, TYPE_CHECKING
+
+from agentx.evaluations.models import EvaluationSettings
+
+if TYPE_CHECKING:
+    from agentx.evaluations.client import EvaluationsClient
+
+logger = logging.getLogger(__name__)
+
+
+class EvaluationSettingsBuilder:
+    """Fluent builder for creating a standalone, reusable grading config (no
+    dataset/questions attached)."""
+
+    def __init__(
+        self,
+        client: "EvaluationsClient",
+        name: str,
+        description: Optional[str] = None,
+        number_of_requests: int = 1,
+        acceptance_criteria: Optional[str] = None,
+        rejection_criteria: Optional[str] = None,
+        evaluation_criteria: Optional[str] = None,
+        vector_similarity: bool = False,
+        jaccard_similarity: bool = False,
+        bleu_score: bool = False,
+        rouge_score: bool = False,
+        similarity_model: Optional[str] = None,
+        sovereignty_models: Optional[List[str]] = None,
+    ):
+        self._client = client
+        self._payload: Dict[str, Any] = {
+            "name": name,
+            "description": description,
+            "numberOfRequests": number_of_requests,
+            "acceptanceCriteria": acceptance_criteria,
+            "rejectionCriteria": rejection_criteria,
+            "evaluationCriteria": evaluation_criteria,
+        }
+        # Opt-in similarity metrics, mirrors DatasetBuilder's config kwargs.
+        if vector_similarity:
+            vs: Dict[str, Any] = {"enabled": True}
+            if similarity_model:
+                vs["model"] = similarity_model
+            self._payload["vectorSimilarity"] = vs
+        if jaccard_similarity:
+            self._payload["jaccardSimilarity"] = {"enabled": True}
+        if bleu_score:
+            self._payload["bleuScore"] = {"enabled": True}
+        if rouge_score:
+            self._payload["rougeScore"] = {"enabled": True}
+        # Sovereignty & Portability — the models to compare when this config runs
+        # (use client.evaluations.list_models() to discover valid ids).
+        if sovereignty_models:
+            self._payload["sovereigntyIndex"] = {
+                "enabled": True,
+                "models": list(sovereignty_models),
+            }
+
+    def publish(self) -> EvaluationSettings:
+        logger.info("Publishing evaluation settings '%s'", self._payload["name"])
+        return self._client.create_evaluation_settings(self._payload)
+
+
+class EvaluationSettingsClient:
+    """Thin wrapper surfaced as client.evaluations.settings."""
+
+    def __init__(self, client: "EvaluationsClient"):
+        self._client = client
+
+    def builder(
+        self,
+        name: str,
+        description: Optional[str] = None,
+        number_of_requests: int = 1,
+        acceptance_criteria: Optional[str] = None,
+        rejection_criteria: Optional[str] = None,
+        evaluation_criteria: Optional[str] = None,
+        vector_similarity: bool = False,
+        jaccard_similarity: bool = False,
+        bleu_score: bool = False,
+        rouge_score: bool = False,
+        similarity_model: Optional[str] = None,
+        sovereignty_models: Optional[List[str]] = None,
+    ) -> EvaluationSettingsBuilder:
+        return EvaluationSettingsBuilder(
+            self._client,
+            name=name,
+            description=description,
+            number_of_requests=number_of_requests,
+            acceptance_criteria=acceptance_criteria,
+            rejection_criteria=rejection_criteria,
+            evaluation_criteria=evaluation_criteria,
+            vector_similarity=vector_similarity,
+            jaccard_similarity=jaccard_similarity,
+            bleu_score=bleu_score,
+            rouge_score=rouge_score,
+            similarity_model=similarity_model,
+            sovereignty_models=sovereignty_models,
+        )
+
+    def get(self, evaluation_settings_id: str) -> EvaluationSettings:
+        return self._client.get_evaluation_settings(evaluation_settings_id)
+
+    def list(self) -> List[EvaluationSettings]:
+        return self._client.list_evaluation_settings()