evalgate-sdk 3.3.1__tar.gz → 3.5.0__tar.gz

{evalgate_sdk-3.3.1 → evalgate_sdk-3.5.0}/CHANGELOG.md

@@ -9,6 +9,28 @@ Version numbering is aligned with the TypeScript SDK (`@evalgate/sdk`) and the p
 
 **Version history note:** The Python SDK jumped from 1.0.0 → 1.9.x → 2.0.0 to stay in sync with the TypeScript SDK. The TypeScript SDK had many releases (1.1–1.9) before the Python SDK existed. We now align both SDKs on the same major.minor version.
 
+## [3.5.0] - 2026-04-09
+
+### Added
+- **Knowledge API parity** — new `client.knowledge.search()`, `upload_document()`, and `get_grounding_context()` methods matching the TypeScript SDK surface.
+- **Auto-session SDK methods** — `evaluations.create_auto_session()`, `list_auto_sessions()`, `get_auto_session()`, `get_quality_score()`, and `synthesize_test_cases()` for autonomous loop management.
+- **Batch trace ingestion** — `report_traces()` function for sending up to 100 traces per request.
+- **Offline trace persistence** — `WorkflowTracer` supports JSON file persistence when disconnected.
+
+### Changed
+- **Version alignment** — bumped Python SDK package metadata and spec constants to `3.5.0`.
+
+## [3.4.0] - 2026-03-29
+
+### Added
+- **Judge orchestration parity** — added Python client and CLI support for the normalized judge registry, presets, run/case judge inspection, and multi-judge evidence payloads.
+- **Workflow and controls parity** — expanded Python operator surfaces for workflow execution, system controls, and workflow-aware parity with the TypeScript SDK.
+- **Provider/PII safeguards** — outbound judge/assertion flows now enforce shared provider policy and pre-send PII protection semantics in the Python SDK path too.
+
+### Changed
+- **Canonical contract alignment** — updated Python types and CLI copy to match the workflow-native and judge-orchestration model used across the platform.
+- **Version alignment** — bumped Python SDK package metadata and spec constants to `3.4.0`.
+
 ## [3.3.1] - 2026-03-24
 
 ### Changed

evalgate_sdk-3.5.0/PKG-INFO

@@ -0,0 +1,220 @@
+Metadata-Version: 2.4
+Name: evalgate-sdk
+Version: 3.5.0
+Summary: EvalGate Python SDK — CI for AI behavior. Traces, evaluations, assertions, and regression gates for LLM apps.
+Project-URL: Homepage, https://evalgate.com
+Project-URL: Documentation, https://github.com/evalgate/ai-evaluation-platform#readme
+Project-URL: Repository, https://github.com/evalgate/ai-evaluation-platform
+Project-URL: Issues, https://github.com/evalgate/ai-evaluation-platform/issues
+Project-URL: Changelog, https://github.com/evalgate/ai-evaluation-platform/blob/main/src/packages/sdk-python/CHANGELOG.md
+Author-email: EvalGate <team@evalgate.com>
+License-Expression: MIT
+Keywords: ai,anthropic,assertions,ci,evaluation,llm,monitoring,observability,openai,regression,testing,tracing,workflow
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: httpx<1,>=0.27
+Requires-Dist: pydantic<3,>=2.0
+Provides-Extra: all
+Requires-Dist: anthropic>=0.20; extra == 'all'
+Requires-Dist: langchain-core>=0.2; extra == 'all'
+Requires-Dist: openai>=1.0; extra == 'all'
+Requires-Dist: rich>=13; extra == 'all'
+Requires-Dist: typer>=0.12; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.20; extra == 'anthropic'
+Provides-Extra: cli
+Requires-Dist: rich>=13; extra == 'cli'
+Requires-Dist: typer>=0.12; extra == 'cli'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest-cov>=5; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: respx>=0.21; extra == 'dev'
+Requires-Dist: rich>=13; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Requires-Dist: typer>=0.12; extra == 'dev'
+Provides-Extra: langchain
+Requires-Dist: langchain-core>=0.2; extra == 'langchain'
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# evalgate-sdk
+
+[![PyPI](https://img.shields.io/pypi/v/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
+[![Python](https://img.shields.io/pypi/pyversions/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+[![Typed](https://img.shields.io/badge/typing-typed-blue)](https://peps.python.org/pep-0561/)
+
+`evalgate-sdk` brings EvalGate's evaluation control plane to Python.
+
+Use it to run assertions, trace agent workflows, inspect platform runs, orchestrate judges, and participate in the same closed-loop evaluation workflow as the TypeScript SDK and the web app.
+
+## Install
+
+```bash
+pip install evalgate-sdk
+pip install "evalgate-sdk[cli]"
+```
+
+Optional extras:
+
+```bash
+pip install "evalgate-sdk[openai]"
+pip install "evalgate-sdk[anthropic]"
+pip install "evalgate-sdk[langchain]"
+pip install "evalgate-sdk[all]"
+```
+
+## Quick Start
+
+### Local Assertions
+
+```python
+from evalgate_sdk import expect
+
+result = expect("The capital of France is Paris.").to_contain("Paris")
+print(result.passed)
+```
+
+### Platform Client
+
+```python
+from evalgate_sdk import AIEvalClient
+from evalgate_sdk.types import CreateTraceParams
+
+client = AIEvalClient(api_key="sk-...")
+
+trace = await client.traces.create(
+    CreateTraceParams(
+        name="support-run",
+        input="Cancel my subscription",
+        output="I've canceled your plan effective today.",
+    )
+)
+
+print(trace.id)
+```
+
+## Closed-Loop Workflow
+
+The Python package participates in the same EvalGate loop:
+
+```text
+trace -> cluster -> synthesize -> gate -> review -> auto -> ship
+```
+
+Representative CLI commands:
+
+```bash
+evalgate gate
+evalgate cluster --run .evalgate/runs/latest.json
+evalgate synthesize --dataset .evalgate/golden/labeled.jsonl
+evalgate judge registry
+evalgate judge presets
+```
+
+## Judge Orchestration
+
+The Python SDK supports the same core judge model as the platform:
+
+- registry-backed judge selection
+- saved judge configs
+- multi-judge aggregation
+- per-run and per-case judge inspection
+- structured reasoning and signal breakdowns
+
+### CLI
+
+```bash
+evalgate judge registry
+evalgate judge presets
+
+evalgate judge test \
+  --provider openai \
+  --model gpt-5.2-chat-latest \
+  --judge openai:gpt-5.2-chat-latest \
+  --judge anthropic:claude-sonnet-4-20250514 \
+  --aggregation weighted \
+  --prompt-template "Return strict JSON with score, passed, reasoning, and signals." \
+  --input "Cancel my subscription" \
+  --output "I've canceled your plan effective today."
+```
+
+### Client
+
+```python
+from evalgate_sdk.types import TestLLMJudgeConfigParams
+
+registry = await client.llm_judge.list_registry()
+presets = await client.llm_judge.list_presets()
+
+result = await client.llm_judge.test_config(
+    TestLLMJudgeConfigParams(
+        provider="openai",
+        model="gpt-5.2-chat-latest",
+        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
+        judges=[
+            {
+                "id": "primary",
+                "type": "llm",
+                "provider": "openai",
+                "model": "gpt-5.2-chat-latest",
+                "weight": 0.6,
+            },
+            {
+                "id": "backup",
+                "type": "llm",
+                "provider": "anthropic",
+                "model": "claude-sonnet-4-20250514",
+                "weight": 0.4,
+            },
+        ],
+        aggregation="weighted",
+        input="Cancel my subscription",
+        output="I've canceled your plan effective today.",
+        behavior="tool_use",
+        task_type="support",
+    )
+)
+
+print(result.result.score, result.result.reasoning)
+```
+
+## Enterprise Controls
+
+The Python SDK respects the same platform guardrails:
+
+- provider allowlists
+- pre-send PII checks for outbound judge/provider calls
+- org-scoped policy
+- shared config precedence with the API and web app
+
+## Surface Parity
+
+The Python SDK now covers the primary EvalGate workflows used in the product:
+
+- traces and runs
+- clustering and synthesis
+- gate/check/review loops
+- judge registry, presets, configs, results, and comparisons
+- workflow and control-plane inspection commands
+
+TypeScript still has a few extra convenience wrappers, but the core control-plane model is shared.
+
+For broader product context, see:
+
+- [Root README](../../../README.md)
+- [Integration reference](../../app/docs/integration/page.tsx)
+- [SDK parity guide](../../app/docs/sdk-parity/page.tsx)

evalgate_sdk-3.5.0/README.md

@@ -0,0 +1,168 @@
+# evalgate-sdk
+
+[![PyPI](https://img.shields.io/pypi/v/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
+[![Python](https://img.shields.io/pypi/pyversions/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+[![Typed](https://img.shields.io/badge/typing-typed-blue)](https://peps.python.org/pep-0561/)
+
+`evalgate-sdk` brings EvalGate's evaluation control plane to Python.
+
+Use it to run assertions, trace agent workflows, inspect platform runs, orchestrate judges, and participate in the same closed-loop evaluation workflow as the TypeScript SDK and the web app.
+
+## Install
+
+```bash
+pip install evalgate-sdk
+pip install "evalgate-sdk[cli]"
+```
+
+Optional extras:
+
+```bash
+pip install "evalgate-sdk[openai]"
+pip install "evalgate-sdk[anthropic]"
+pip install "evalgate-sdk[langchain]"
+pip install "evalgate-sdk[all]"
+```
+
+## Quick Start
+
+### Local Assertions
+
+```python
+from evalgate_sdk import expect
+
+result = expect("The capital of France is Paris.").to_contain("Paris")
+print(result.passed)
+```
+
+### Platform Client
+
+```python
+from evalgate_sdk import AIEvalClient
+from evalgate_sdk.types import CreateTraceParams
+
+client = AIEvalClient(api_key="sk-...")
+
+trace = await client.traces.create(
+    CreateTraceParams(
+        name="support-run",
+        input="Cancel my subscription",
+        output="I've canceled your plan effective today.",
+    )
+)
+
+print(trace.id)
+```
+
+## Closed-Loop Workflow
+
+The Python package participates in the same EvalGate loop:
+
+```text
+trace -> cluster -> synthesize -> gate -> review -> auto -> ship
+```
+
+Representative CLI commands:
+
+```bash
+evalgate gate
+evalgate cluster --run .evalgate/runs/latest.json
+evalgate synthesize --dataset .evalgate/golden/labeled.jsonl
+evalgate judge registry
+evalgate judge presets
+```
+
+## Judge Orchestration
+
+The Python SDK supports the same core judge model as the platform:
+
+- registry-backed judge selection
+- saved judge configs
+- multi-judge aggregation
+- per-run and per-case judge inspection
+- structured reasoning and signal breakdowns
+
+### CLI
+
+```bash
+evalgate judge registry
+evalgate judge presets
+
+evalgate judge test \
+  --provider openai \
+  --model gpt-5.2-chat-latest \
+  --judge openai:gpt-5.2-chat-latest \
+  --judge anthropic:claude-sonnet-4-20250514 \
+  --aggregation weighted \
+  --prompt-template "Return strict JSON with score, passed, reasoning, and signals." \
+  --input "Cancel my subscription" \
+  --output "I've canceled your plan effective today."
+```
+
+### Client
+
+```python
+from evalgate_sdk.types import TestLLMJudgeConfigParams
+
+registry = await client.llm_judge.list_registry()
+presets = await client.llm_judge.list_presets()
+
+result = await client.llm_judge.test_config(
+    TestLLMJudgeConfigParams(
+        provider="openai",
+        model="gpt-5.2-chat-latest",
+        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
+        judges=[
+            {
+                "id": "primary",
+                "type": "llm",
+                "provider": "openai",
+                "model": "gpt-5.2-chat-latest",
+                "weight": 0.6,
+            },
+            {
+                "id": "backup",
+                "type": "llm",
+                "provider": "anthropic",
+                "model": "claude-sonnet-4-20250514",
+                "weight": 0.4,
+            },
+        ],
+        aggregation="weighted",
+        input="Cancel my subscription",
+        output="I've canceled your plan effective today.",
+        behavior="tool_use",
+        task_type="support",
+    )
+)
+
+print(result.result.score, result.result.reasoning)
+```
+
+## Enterprise Controls
+
+The Python SDK respects the same platform guardrails:
+
+- provider allowlists
+- pre-send PII checks for outbound judge/provider calls
+- org-scoped policy
+- shared config precedence with the API and web app
+
+## Surface Parity
+
+The Python SDK now covers the primary EvalGate workflows used in the product:
+
+- traces and runs
+- clustering and synthesis
+- gate/check/review loops
+- judge registry, presets, configs, results, and comparisons
+- workflow and control-plane inspection commands
+
+TypeScript still has a few extra convenience wrappers, but the core control-plane model is shared.
+
+For broader product context, see:
+
+- [Root README](../../../README.md)
+- [Integration reference](../../app/docs/integration/page.tsx)
+- [SDK parity guide](../../app/docs/sdk-parity/page.tsx)

{evalgate_sdk-3.3.1 → evalgate_sdk-3.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "evalgate-sdk"
-version = "3.3.1"
+version = "3.5.0"
 description = "EvalGate Python SDK — CI for AI behavior. Traces, evaluations, assertions, and regression gates for LLM apps."
 readme = "README.md"
 license = "MIT"
@@ -76,5 +76,5 @@ select = ["E", "F", "I", "UP", "B", "SIM"]
 
 [tool.mypy]
 python_version = "3.10"
-strict = true
-warn_return_any = true
+strict = false
+warn_return_any = false

{evalgate_sdk-3.3.1 → evalgate_sdk-3.5.0}/src/evalgate_sdk/__init__.py

@@ -59,25 +59,57 @@ from evalgate_sdk.auto import (
     run_auto_daemon,
     write_auto_report,
 )
+from evalgate_sdk.auto_experiment_runner import (
+    Experiment,
+    ExperimentChangeSet,
+    ExperimentDecision,
+    ExperimentMetrics,
+    ExperimentRunner,
+    KeepDiscardPolicy,
+    KeepDiscardResult,
+    evaluate_experiment,
+    select_best_experiment,
+)
+from evalgate_sdk.auto_failure_analyzer import (
+    AnalyzeInput,
+    AutoLLMConfig,
+    FailureAnalysis,
+    FailureAnalyzer,
+    FailureCase,
+    FailurePattern,
+    TasteContext,
+)
+from evalgate_sdk.auto_mutation_engine import (
+    GeneratedMutation,
+    GenerateInput,
+    KnowledgeHint,
+    ModelSettingsOverride,
+    MutationAttempt,
+    MutationEngine,
+    MutationStrategy,
+    TrajectorySummary,
+)
 from evalgate_sdk.batch import RequestBatcher, batch_process, can_batch
 from evalgate_sdk.cache import CacheTTL, RequestCache, get_ttl, should_cache
 from evalgate_sdk.ci_context import CIContext, detect_ci_context
-from evalgate_sdk.cluster import ClusterCase, ClusterSample, ClusterSummary, TraceCluster, cluster_run_result, format_cluster_human
 from evalgate_sdk.cli.api import (
     FetchOptions,
     PublishShareResult,
     QualityLatestData,
     RunDetailsData,
     fetch_api,
+    fetch_auto_plan_preview,
+    fetch_integrity,
+    fetch_knowledge,
     fetch_quality_latest,
     fetch_run_details,
     fetch_run_export,
     import_run_on_fail,
     publish_share,
+    refresh_integrity,
 )
 from evalgate_sdk.cli.cli_constants import EXIT
 from evalgate_sdk.cli.config import (
-    EvalAIConfig,
     EvalGateConfig,
     find_config_path,
     load_config,
@@ -130,6 +162,14 @@ from evalgate_sdk.cli.traces import (
 )
 from evalgate_sdk.cli.workspace import EvalWorkspace, resolve_eval_workspace
 from evalgate_sdk.client import AIEvalClient
+from evalgate_sdk.cluster import (
+    ClusterCase,
+    ClusterSample,
+    ClusterSummary,
+    TraceCluster,
+    cluster_run_result,
+    format_cluster_human,
+)
 from evalgate_sdk.collector import (
     CollectorFeedbackInput,
     CollectorSpanInput,
@@ -137,6 +177,7 @@ from evalgate_sdk.collector import (
     ReportTraceOptions,
     ReportTraceResult,
     report_trace,
+    report_traces,
 )
 from evalgate_sdk.constants import DEFAULT_BASE_URL
 from evalgate_sdk.context import (
@@ -202,6 +243,7 @@ from evalgate_sdk.integrations.openai_eval import (
     OpenAIChatEvalResult,
     openai_chat_eval,
 )
+from evalgate_sdk.knowledge import KnowledgeAPI
 from evalgate_sdk.local import LocalStorage, LocalStorageStats
 from evalgate_sdk.logger import Logger, RequestLogger, create_logger, get_logger, set_logger
 from evalgate_sdk.matchers import GateAssertionError, assert_passes_gate, to_pass_gate
@@ -223,7 +265,6 @@ from evalgate_sdk.pytest_plugin import (
     assert_score_between,
 )
 from evalgate_sdk.reason_codes import REASON_CODES, get_reason_info, is_blocking
-from evalgate_sdk.replay_decision import NormalizedBudgetConfig, ReplayDecision, determine_comparison_basis, evaluate_replay_outcome
 from evalgate_sdk.regression import (
     ARTIFACTS,
     GATE_CATEGORY,
@@ -237,6 +278,12 @@ from evalgate_sdk.regression import (
     evaluate_regression,
     verify_baseline_checksum,
 )
+from evalgate_sdk.replay_decision import (
+    NormalizedBudgetConfig,
+    ReplayDecision,
+    determine_comparison_basis,
+    evaluate_replay_outcome,
+)
 from evalgate_sdk.runtime import (
     EvalExecutionError,
     EvalRuntimeError,
@@ -270,7 +317,6 @@ from evalgate_sdk.runtime.eval import (
     define_eval_only,
     define_eval_skip,
     define_suite,
-    evalai,
     from_dataset,
     get_filtered_specs,
 )
@@ -334,7 +380,25 @@ from evalgate_sdk.synthesize import (
     synthesize_labeled_dataset,
 )
 from evalgate_sdk.testing import TestSuite, create_test_suite
-from evalgate_sdk.types import CamelModel, QualityBreakdown, QualityScore
+from evalgate_sdk.types import (
+    AgentApprovalMode,
+    AgentAutonomyLevel,
+    AgentMemoryMode,
+    AgentWorkflowClass,
+    AgentWorkflowExperimentRecipe,
+    AgentWorkflowHookPhase,
+    AgentWorkflowSpec,
+    AutoPlanPreviewParams,
+    AutoPlanPreviewResponse,
+    AutoSessionWorkflowProfile,
+    CamelModel,
+    EvaluationIntegritySnapshot,
+    EvaluationRunDetail,
+    KnowledgeHintSummary,
+    KnowledgeResponse,
+    QualityBreakdown,
+    QualityScore,
+)
 from evalgate_sdk.utils.input_hash import normalize_input, sha256_input
 from evalgate_sdk.workflows import WorkflowTracer, create_workflow_tracer, trace_workflow_step
 
@@ -474,13 +538,74 @@ __all__ = [
     "AutoDiffSnapshot",
     "AutoIterationResult",
     "AutoReport",
+    "Experiment",
+    "ExperimentChangeSet",
+    "ExperimentDecision",
+    "ExperimentMetrics",
+    "ExperimentRunner",
+    "KeepDiscardPolicy",
+    "KeepDiscardResult",
+    "AnalyzeInput",
+    "AutoLLMConfig",
+    "FailureAnalysis",
+    "FailureAnalyzer",
+    "FailureCase",
+    "FailurePattern",
+    "KnowledgeHint",
+    "GenerateInput",
+    "GeneratedMutation",
+    "ModelSettingsOverride",
+    "MutationAttempt",
+    "MutationEngine",
+    "MutationStrategy",
+    "TasteContext",
+    "TrajectorySummary",
     "build_auto_plan",
     "decide_auto_experiment",
     "build_auto_report",
+    "evaluate_experiment",
     "format_auto_human",
     "append_auto_history",
+    "select_best_experiment",
     "write_auto_report",
     "run_auto_daemon",
+    "Experiment",
+    "ExperimentChangeSet",
+    "ExperimentDecision",
+    "ExperimentMetrics",
+    "ExperimentRunner",
+    "KeepDiscardPolicy",
+    "KeepDiscardResult",
+    "evaluate_experiment",
+    "select_best_experiment",
+    "AnalyzeInput",
+    "AutoLLMConfig",
+    "FailureAnalysis",
+    "FailureAnalyzer",
+    "FailureCase",
+    "FailurePattern",
+    "TasteContext",
+    "GeneratedMutation",
+    "GenerateInput",
+    "KnowledgeHint",
+    "ModelSettingsOverride",
+    "MutationAttempt",
+    "MutationEngine",
+    "MutationStrategy",
+    "AgentApprovalMode",
+    "AgentAutonomyLevel",
+    "AgentMemoryMode",
+    "AgentWorkflowClass",
+    "AgentWorkflowExperimentRecipe",
+    "AgentWorkflowHookPhase",
+    "AgentWorkflowSpec",
+    "AutoPlanPreviewParams",
+    "AutoPlanPreviewResponse",
+    "AutoSessionWorkflowProfile",
+    "EvaluationIntegritySnapshot",
+    "EvaluationRunDetail",
+    "KnowledgeHintSummary",
+    "KnowledgeResponse",
     # Streaming
     "RateLimiter",
     "BatchProgress",
@@ -546,7 +671,6 @@ __all__ = [
     "define_eval_only",
     "define_suite",
     "create_result",
-    "evalai",
     "from_dataset",
     "get_filtered_specs",
     # Runtime management
@@ -576,8 +700,11 @@ __all__ = [
     "CamelModel",
     "QualityScore",
     "QualityBreakdown",
+    # Knowledge API
+    "KnowledgeAPI",
     # Collector (T2)
     "report_trace",
+    "report_traces",
     "ReportTraceInput",
     "ReportTraceOptions",
     "ReportTraceResult",
@@ -647,13 +774,16 @@ __all__ = [
     "is_git_ref",
     "get_github_step_summary_path",
     # CLI config (T14)
-    "EvalAIConfig",
     "EvalGateConfig",
     "find_config_path",
     "load_config",
     "merge_config_with_args",
     # CLI API (T14)
     "fetch_api",
+    "fetch_integrity",
+    "refresh_integrity",
+    "fetch_knowledge",
+    "fetch_auto_plan_preview",
     "fetch_quality_latest",
     "fetch_run_details",
     "fetch_run_export",

evalgate_sdk-3.5.0/src/evalgate_sdk/_version.py

@@ -0,0 +1,3 @@
+__version__ = "3.5.0"
+SDK_VERSION = __version__
+SPEC_VERSION = "3.5.0"