evalgate-sdk 3.3.1__tar.gz → 3.5.1__tar.gz

{evalgate_sdk-3.3.1 → evalgate_sdk-3.5.1}/CHANGELOG.md

@@ -9,6 +9,37 @@ Version numbering is aligned with the TypeScript SDK (`@evalgate/sdk`) and the p
 
 **Version history note:** The Python SDK jumped from 1.0.0 → 1.9.x → 2.0.0 to stay in sync with the TypeScript SDK. The TypeScript SDK had many releases (1.1–1.9) before the Python SDK existed. We now align both SDKs on the same major.minor version.
 
+## [3.5.1] - 2026-06-26
+
+### Fixed
+- **CLI JSON contracts** — `check --format json` now emits one parseable JSON document.
+- **Zero-config scaffolds** — `start --template` writes config that `run` and `doctor` discover consistently.
+- **Runtime scoring defaults** — boolean/dict executor fallbacks now use the SDK's 0-100 score scale.
+- **Sync safety** — missing labeled datasets surface an explicit error, and prompt pulls write UTF-8 content on Windows.
+- **Default CLI install** — `pip install evalgate-sdk` now includes the runtime dependencies needed for the `evalgate` command.
+
+## [3.5.0] - 2026-04-09
+
+### Added
+- **Knowledge API parity** — new `client.knowledge.search()`, `upload_document()`, and `get_grounding_context()` methods matching the TypeScript SDK surface.
+- **Auto-session SDK methods** — `evaluations.create_auto_session()`, `list_auto_sessions()`, `get_auto_session()`, `get_quality_score()`, and `synthesize_test_cases()` for autonomous loop management.
+- **Batch trace ingestion** — `report_traces()` function for sending up to 100 traces per request.
+- **Offline trace persistence** — `WorkflowTracer` supports JSON file persistence when disconnected.
+
+### Changed
+- **Version alignment** — bumped Python SDK package metadata and spec constants to `3.5.0`.
+
+## [3.4.0] - 2026-03-29
+
+### Added
+- **Judge orchestration parity** — added Python client and CLI support for the normalized judge registry, presets, run/case judge inspection, and multi-judge evidence payloads.
+- **Workflow and controls parity** — expanded Python operator surfaces for workflow execution, system controls, and workflow-aware parity with the TypeScript SDK.
+- **Provider/PII safeguards** — outbound judge/assertion flows now enforce shared provider policy and pre-send PII protection semantics in the Python SDK path too.
+
+### Changed
+- **Canonical contract alignment** — updated Python types and CLI copy to match the workflow-native and judge-orchestration model used across the platform.
+- **Version alignment** — bumped Python SDK package metadata and spec constants to `3.4.0`.
+
 ## [3.3.1] - 2026-03-24
 
 ### Changed

evalgate_sdk-3.5.1/PKG-INFO

@@ -0,0 +1,250 @@
+Metadata-Version: 2.4
+Name: evalgate-sdk
+Version: 3.5.1
+Summary: EvalGate Python SDK — CI for AI behavior. Traces, evaluations, assertions, and regression gates for LLM apps.
+Project-URL: Homepage, https://evalgate.com
+Project-URL: Documentation, https://github.com/evalgate/ai-evaluation-platform#readme
+Project-URL: Repository, https://github.com/evalgate/ai-evaluation-platform
+Project-URL: Issues, https://github.com/evalgate/ai-evaluation-platform/issues
+Project-URL: Changelog, https://github.com/evalgate/ai-evaluation-platform/blob/main/src/packages/sdk-python/CHANGELOG.md
+Author-email: EvalGate <team@evalgate.com>
+License-Expression: MIT
+Keywords: ai,anthropic,assertions,ci,evaluation,llm,monitoring,observability,openai,regression,testing,tracing,workflow
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: httpx<1,>=0.27
+Requires-Dist: pydantic<3,>=2.0
+Requires-Dist: rich>=13
+Requires-Dist: typer>=0.12
+Provides-Extra: all
+Requires-Dist: anthropic>=0.20; extra == 'all'
+Requires-Dist: langchain-core>=0.2; extra == 'all'
+Requires-Dist: openai>=1.0; extra == 'all'
+Requires-Dist: rich>=13; extra == 'all'
+Requires-Dist: typer>=0.12; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.20; extra == 'anthropic'
+Provides-Extra: cli
+Requires-Dist: rich>=13; extra == 'cli'
+Requires-Dist: typer>=0.12; extra == 'cli'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest-cov>=5; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: respx>=0.21; extra == 'dev'
+Requires-Dist: rich>=13; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Requires-Dist: typer>=0.12; extra == 'dev'
+Provides-Extra: langchain
+Requires-Dist: langchain-core>=0.2; extra == 'langchain'
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# evalgate-sdk
+
+[![PyPI](https://img.shields.io/pypi/v/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
+[![Python](https://img.shields.io/pypi/pyversions/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+[![Typed](https://img.shields.io/badge/typing-typed-blue)](https://peps.python.org/pep-0561/)
+
+`evalgate-sdk` brings EvalGate's evaluation control plane to Python.
+
+Use it to run assertions, trace agent workflows, inspect platform runs, orchestrate judges, and participate in the same closed-loop evaluation workflow as the TypeScript SDK and the web app.
+
+## Install
+
+```bash
+pip install evalgate-sdk
+```
+
+Optional extras:
+
+```bash
+pip install "evalgate-sdk[openai]"
+pip install "evalgate-sdk[anthropic]"
+pip install "evalgate-sdk[langchain]"
+pip install "evalgate-sdk[all]"
+```
+
+## Quick Start
+
+### Local Assertions
+
+```python
+from evalgate_sdk import expect
+
+result = expect("The capital of France is Paris.").to_contain("Paris")
+print(result.passed)
+```
+
+### Platform Client
+
+```python
+from evalgate_sdk import AIEvalClient
+from evalgate_sdk.types import CreateTraceParams
+
+client = AIEvalClient(api_key="sk-...")
+
+trace = await client.traces.create(
+    CreateTraceParams(
+        name="support-run",
+        input="Cancel my subscription",
+        output="I've canceled your plan effective today.",
+    )
+)
+
+print(trace.id)
+```
+
+## Closed-Loop Workflow
+
+The Python package participates in the same EvalGate loop:
+
+```text
+trace -> cluster -> synthesize -> gate -> review -> auto -> ship
+```
+
+Representative CLI commands:
+
+```bash
+evalgate gate
+evalgate cluster --run .evalgate/runs/latest.json
+evalgate synthesize --dataset .evalgate/golden/labeled.jsonl
+evalgate judge registry
+evalgate judge presets
+```
+
+## Web App Sync
+
+The Python CLI uses the same sync contracts as the TypeScript SDK. Local labeled
+JSONL can be pushed into the web app, web-labeled cases can be merged back into
+the same file, and prompt files can round-trip with stale active-version
+protection.
+
+```bash
+# Push local labels into the web app and pull web labels back into JSONL
+evalgate sync \
+  --project-key support-agent \
+  --labeled-dataset .evalgate/golden/labeled.jsonl \
+  --pull-labels
+
+# Pull the active web prompt into a local file
+evalgate sync \
+  --project-key support-agent \
+  --prompt-id prompt_support \
+  --target-path prompts/support.md \
+  --pull-prompt
+
+# Push an edited local prompt as a new web prompt version
+evalgate sync \
+  --project-key support-agent \
+  --prompt-id prompt_support \
+  --target-path prompts/support.md \
+  --push-prompt
+```
+
+## Judge Orchestration
+
+The Python SDK supports the same core judge model as the platform:
+
+- registry-backed judge selection
+- saved judge configs
+- multi-judge aggregation
+- per-run and per-case judge inspection
+- structured reasoning and signal breakdowns
+
+### CLI
+
+```bash
+evalgate judge registry
+evalgate judge presets
+
+evalgate judge test \
+  --provider openai \
+  --model gpt-5.2-chat-latest \
+  --judge openai:gpt-5.2-chat-latest \
+  --judge anthropic:claude-sonnet-4-20250514 \
+  --aggregation weighted \
+  --prompt-template "Return strict JSON with score, passed, reasoning, and signals." \
+  --input "Cancel my subscription" \
+  --output "I've canceled your plan effective today."
+```
+
+### Client
+
+```python
+from evalgate_sdk.types import TestLLMJudgeConfigParams
+
+registry = await client.llm_judge.list_registry()
+presets = await client.llm_judge.list_presets()
+
+result = await client.llm_judge.test_config(
+    TestLLMJudgeConfigParams(
+        provider="openai",
+        model="gpt-5.2-chat-latest",
+        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
+        judges=[
+            {
+                "id": "primary",
+                "type": "llm",
+                "provider": "openai",
+                "model": "gpt-5.2-chat-latest",
+                "weight": 0.6,
+            },
+            {
+                "id": "backup",
+                "type": "llm",
+                "provider": "anthropic",
+                "model": "claude-sonnet-4-20250514",
+                "weight": 0.4,
+            },
+        ],
+        aggregation="weighted",
+        input="Cancel my subscription",
+        output="I've canceled your plan effective today.",
+        behavior="tool_use",
+        task_type="support",
+    )
+)
+
+print(result.result.score, result.result.reasoning)
+```
+
+## Enterprise Controls
+
+The Python SDK respects the same platform guardrails:
+
+- provider allowlists
+- pre-send PII checks for outbound judge/provider calls
+- org-scoped policy
+- shared config precedence with the API and web app
+
+## Surface Parity
+
+The Python SDK now covers the primary EvalGate workflows used in the product:
+
+- traces and runs
+- clustering and synthesis
+- gate/check/review loops
+- judge registry, presets, configs, results, and comparisons
+- workflow and control-plane inspection commands
+
+TypeScript still has a few extra convenience wrappers, but the core control-plane model is shared.
+
+For broader product context, see:
+
+- [Root README](../../../README.md)
+- [Integration reference](../../app/docs/integration/page.tsx)
+- [SDK parity guide](../../app/docs/sdk-parity/page.tsx)

evalgate_sdk-3.5.1/README.md

@@ -0,0 +1,196 @@
+# evalgate-sdk
+
+[![PyPI](https://img.shields.io/pypi/v/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
+[![Python](https://img.shields.io/pypi/pyversions/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+[![Typed](https://img.shields.io/badge/typing-typed-blue)](https://peps.python.org/pep-0561/)
+
+`evalgate-sdk` brings EvalGate's evaluation control plane to Python.
+
+Use it to run assertions, trace agent workflows, inspect platform runs, orchestrate judges, and participate in the same closed-loop evaluation workflow as the TypeScript SDK and the web app.
+
+## Install
+
+```bash
+pip install evalgate-sdk
+```
+
+Optional extras:
+
+```bash
+pip install "evalgate-sdk[openai]"
+pip install "evalgate-sdk[anthropic]"
+pip install "evalgate-sdk[langchain]"
+pip install "evalgate-sdk[all]"
+```
+
+## Quick Start
+
+### Local Assertions
+
+```python
+from evalgate_sdk import expect
+
+result = expect("The capital of France is Paris.").to_contain("Paris")
+print(result.passed)
+```
+
+### Platform Client
+
+```python
+from evalgate_sdk import AIEvalClient
+from evalgate_sdk.types import CreateTraceParams
+
+client = AIEvalClient(api_key="sk-...")
+
+trace = await client.traces.create(
+    CreateTraceParams(
+        name="support-run",
+        input="Cancel my subscription",
+        output="I've canceled your plan effective today.",
+    )
+)
+
+print(trace.id)
+```
+
+## Closed-Loop Workflow
+
+The Python package participates in the same EvalGate loop:
+
+```text
+trace -> cluster -> synthesize -> gate -> review -> auto -> ship
+```
+
+Representative CLI commands:
+
+```bash
+evalgate gate
+evalgate cluster --run .evalgate/runs/latest.json
+evalgate synthesize --dataset .evalgate/golden/labeled.jsonl
+evalgate judge registry
+evalgate judge presets
+```
+
+## Web App Sync
+
+The Python CLI uses the same sync contracts as the TypeScript SDK. Local labeled
+JSONL can be pushed into the web app, web-labeled cases can be merged back into
+the same file, and prompt files can round-trip with stale active-version
+protection.
+
+```bash
+# Push local labels into the web app and pull web labels back into JSONL
+evalgate sync \
+  --project-key support-agent \
+  --labeled-dataset .evalgate/golden/labeled.jsonl \
+  --pull-labels
+
+# Pull the active web prompt into a local file
+evalgate sync \
+  --project-key support-agent \
+  --prompt-id prompt_support \
+  --target-path prompts/support.md \
+  --pull-prompt
+
+# Push an edited local prompt as a new web prompt version
+evalgate sync \
+  --project-key support-agent \
+  --prompt-id prompt_support \
+  --target-path prompts/support.md \
+  --push-prompt
+```
+
+## Judge Orchestration
+
+The Python SDK supports the same core judge model as the platform:
+
+- registry-backed judge selection
+- saved judge configs
+- multi-judge aggregation
+- per-run and per-case judge inspection
+- structured reasoning and signal breakdowns
+
+### CLI
+
+```bash
+evalgate judge registry
+evalgate judge presets
+
+evalgate judge test \
+  --provider openai \
+  --model gpt-5.2-chat-latest \
+  --judge openai:gpt-5.2-chat-latest \
+  --judge anthropic:claude-sonnet-4-20250514 \
+  --aggregation weighted \
+  --prompt-template "Return strict JSON with score, passed, reasoning, and signals." \
+  --input "Cancel my subscription" \
+  --output "I've canceled your plan effective today."
+```
+
+### Client
+
+```python
+from evalgate_sdk.types import TestLLMJudgeConfigParams
+
+registry = await client.llm_judge.list_registry()
+presets = await client.llm_judge.list_presets()
+
+result = await client.llm_judge.test_config(
+    TestLLMJudgeConfigParams(
+        provider="openai",
+        model="gpt-5.2-chat-latest",
+        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
+        judges=[
+            {
+                "id": "primary",
+                "type": "llm",
+                "provider": "openai",
+                "model": "gpt-5.2-chat-latest",
+                "weight": 0.6,
+            },
+            {
+                "id": "backup",
+                "type": "llm",
+                "provider": "anthropic",
+                "model": "claude-sonnet-4-20250514",
+                "weight": 0.4,
+            },
+        ],
+        aggregation="weighted",
+        input="Cancel my subscription",
+        output="I've canceled your plan effective today.",
+        behavior="tool_use",
+        task_type="support",
+    )
+)
+
+print(result.result.score, result.result.reasoning)
+```
+
+## Enterprise Controls
+
+The Python SDK respects the same platform guardrails:
+
+- provider allowlists
+- pre-send PII checks for outbound judge/provider calls
+- org-scoped policy
+- shared config precedence with the API and web app
+
+## Surface Parity
+
+The Python SDK now covers the primary EvalGate workflows used in the product:
+
+- traces and runs
+- clustering and synthesis
+- gate/check/review loops
+- judge registry, presets, configs, results, and comparisons
+- workflow and control-plane inspection commands
+
+TypeScript still has a few extra convenience wrappers, but the core control-plane model is shared.
+
+For broader product context, see:
+
+- [Root README](../../../README.md)
+- [Integration reference](../../app/docs/integration/page.tsx)
+- [SDK parity guide](../../app/docs/sdk-parity/page.tsx)

{evalgate_sdk-3.3.1 → evalgate_sdk-3.5.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "evalgate-sdk"
-version = "3.3.1"
+version = "3.5.1"
 description = "EvalGate Python SDK — CI for AI behavior. Traces, evaluations, assertions, and regression gates for LLM apps."
 readme = "README.md"
 license = "MIT"
@@ -31,6 +31,8 @@ classifiers = [
 dependencies = [
     "httpx>=0.27,<1",
     "pydantic>=2.0,<3",
+    "rich>=13",
+    "typer>=0.12",
 ]
 
 [project.optional-dependencies]
@@ -66,6 +68,8 @@ packages = ["src/evalgate_sdk"]
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"
+pythonpath = ["src"]
 
 [tool.ruff]
 target-version = "py310"
@@ -76,5 +80,5 @@ select = ["E", "F", "I", "UP", "B", "SIM"]
 
 [tool.mypy]
 python_version = "3.10"
-strict = true
-warn_return_any = true
+strict = false
+warn_return_any = false