agentprdiff 0.3.0__tar.gz → 0.4.0__tar.gz

{agentprdiff-0.3.0 → agentprdiff-0.4.0}/CHANGELOG.md

@@ -8,6 +8,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.4.0] — 2026-08-29
+
+Minor release bundling the three feature PRs merged since 0.3.1, plus
+the Studio hardening work. All changes are additive; no breaking API
+changes.
+
+### Added
+
+- `http_judge` grader backend for custom LLM-as-judge endpoints, so
+  suites can score traces against a self-hosted or third-party judge
+  service instead of the built-in providers (#7).
+- Field-level trace masking for PII and other sensitive data, letting
+  suites redact specific fields from recorded traces before they are
+  stored or diffed (#8).
+- Pluggable `TraceStore` interface with an `InMemoryTraceStore`
+  implementation, decoupling trace persistence from the runner so
+  custom storage backends can be dropped in (#9).
+- Studio: private-repo git auth (SSH + HTTPS PAT), adapter and
+  `extend_existing` scan strategies, path-to-import sanitization with
+  import preflight, hardened preflight pipeline with scan manifest and
+  auto-install preview.
+
+### Fixed
+
+- Ruff lint cleanups in the new masking and http_judge modules and
+  their tests; suppressed the intentional B027 warning on the no-op
+  `ensure_initialized` hook.
+
+## [0.3.1] — 2026-05-17
+
+Patch release. Fixes a long-standing drift between `pyproject.toml`'s
+`version` field and the in-code `__version__` constant: the constant
+in `src/agentprdiff/__init__.py` was hardcoded and got out of sync at
+every release (0.3.0 shipped while still reporting "0.2.5" via
+`agentprdiff.__version__`, even though `pip show` correctly read
+"0.3.0" from the wheel metadata).
+
+### Fixed
+
+- `agentprdiff.__version__` is now read from the installed package
+  metadata via `importlib.metadata.version("agentprdiff")` instead of
+  a hardcoded string. The runtime constant matches the wheel metadata
+  by construction; no human step can let them drift again. Falls back
+  to `"0+unknown"` when running from an uninstalled source checkout.
+
+This is purely a metadata fix — no behavior change, no API additions,
+no removed APIs. The `set_default_model()` hook from 0.3.0 is
+unchanged.
+
 ## [0.3.0] — 2026-05-17
 
 Adds a model-override hook on the adapters package. This unblocks

{agentprdiff-0.3.0 → agentprdiff-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: agentprdiff
-Version: 0.3.0
+Version: 0.4.0
 Summary: Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change.
 Project-URL: Homepage, https://agentprdiff.dev
 Project-URL: Documentation, https://agentprdiff.dev

{agentprdiff-0.3.0 → agentprdiff-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agentprdiff"
-version = "0.3.0"
+version = "0.4.0"
 description = "Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change."
 readme = "README.md"
 requires-python = ">=3.10"

{agentprdiff-0.3.0 → agentprdiff-0.4.0}/src/agentprdiff/__init__.py

@@ -50,6 +50,9 @@ See ``docs/adapters.md`` for the full reference.
 
 from __future__ import annotations
 
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version as _pkg_version
+
 from .core import (
     AgentFn,
     Case,
@@ -67,7 +70,15 @@ from .differ import AssertionChange, TraceDelta, diff_traces
 from .runner import CaseReport, Runner, RunReport
 from .store import BaselineStore
 
-__version__ = "0.2.5"
+# Source of truth is pyproject.toml's [project] version. Reading it here
+# means the runtime constant can never drift from the wheel metadata; the
+# fallback covers the "running from a source checkout that wasn't pip
+# installed" case (e.g. `python -m pytest` on a fresh clone before
+# `pip install -e .`).
+try:
+    __version__ = _pkg_version("agentprdiff")
+except PackageNotFoundError:  # pragma: no cover — only hit in source-tree runs
+    __version__ = "0+unknown"
 
 __all__ = [
     # core

agentprdiff-0.4.0/src/agentprdiff/graders/http_judge.py

@@ -0,0 +1,93 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: MIT
+"""HTTP judge — call any REST endpoint as an LLM-as-judge backend.
+
+Lets teams route evaluation traffic to an internal model gateway, a private
+inference server, or any OpenAI-compatible endpoint without modifying the
+agentprdiff core.
+
+Usage::
+
+    from agentprdiff.graders.http_judge import http_judge
+    from agentprdiff.graders import semantic
+
+    judge = http_judge(
+        url="https://my-gateway.internal/v1/chat/completions",
+        model="my-model",
+        api_key="sk-...",          # or set HTTP_JUDGE_API_KEY env var
+    )
+
+    grader = semantic("agent acknowledged the refund", judge=judge)
+
+The endpoint must accept the OpenAI Chat Completions request schema and return
+a response with ``choices[0].message.content``.  The content must start with
+``PASS`` or ``FAIL`` on the first line, with an optional reason on the second
+line — the same format used by the built-in ``openai_judge`` and
+``anthropic_judge``.
+"""
+
+from __future__ import annotations
+
+import os
+from typing import Any
+
+from ..core import Trace
+from .semantic import _JUDGE_PROMPT, Judge, _parse_verdict
+
+
+def http_judge(
+    url: str,
+    *,
+    model: str,
+    api_key: str | None = None,
+    extra_headers: dict[str, str] | None = None,
+    timeout: float = 30.0,
+    temperature: float = 0.0,
+    max_tokens: int = 120,
+) -> Judge:
+    """Return a judge that POSTs to an OpenAI-compatible chat completions endpoint.
+
+    Args:
+        url: Full URL of the ``/chat/completions`` endpoint.
+        model: Model name forwarded in the request body.
+        api_key: Bearer token. Falls back to the ``HTTP_JUDGE_API_KEY``
+            environment variable when omitted.
+        extra_headers: Additional HTTP headers merged into every request
+            (e.g. ``{"X-Tenant-Id": "my-team"}``).
+        timeout: Request timeout in seconds.
+        temperature: Sampling temperature forwarded to the endpoint.
+        max_tokens: Maximum tokens in the judge response.
+
+    Returns:
+        A :data:`~agentprdiff.graders.semantic.Judge` callable compatible with
+        :func:`~agentprdiff.graders.semantic.semantic`.
+    """
+    resolved_key = api_key or os.environ.get("HTTP_JUDGE_API_KEY", "")
+
+    def _judge(rubric: str, trace: Trace) -> tuple[bool, str]:
+        try:
+            import json as _json
+            import urllib.request
+
+            prompt = _JUDGE_PROMPT.format(rubric=rubric, output=str(trace.output or ""))
+            payload: dict[str, Any] = {
+                "model": model,
+                "messages": [{"role": "user", "content": prompt}],
+                "temperature": temperature,
+                "max_tokens": max_tokens,
+            }
+            headers = {
+                "Content-Type": "application/json",
+                **({"Authorization": f"Bearer {resolved_key}"} if resolved_key else {}),
+                **(extra_headers or {}),
+            }
+            data = _json.dumps(payload).encode()
+            req = urllib.request.Request(url, data=data, headers=headers, method="POST")
+            with urllib.request.urlopen(req, timeout=timeout) as resp:  # noqa: S310
+                body = _json.loads(resp.read())
+            text: str = body["choices"][0]["message"]["content"] or ""
+            return _parse_verdict(text)
+        except Exception as exc:  # noqa: BLE001
+            return False, f"http_judge error ({type(exc).__name__}): {exc}"
+
+    return _judge

agentprdiff-0.4.0/src/agentprdiff/masking.py

@@ -0,0 +1,120 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: MIT
+"""Field-level masking for Trace objects before baseline storage or diffing.
+
+Teams often capture traces that contain PII, credentials, or environment-
+specific values (user IDs, session tokens, hostnames) that should not be
+committed to git or compared across runs.  This module provides a lightweight
+masking layer that redacts or replaces selected fields before a Trace leaves
+the runner.
+
+Usage::
+
+    from agentprdiff.masking import mask_trace, MaskRule
+
+    rules = [
+        MaskRule(field="output", pattern=r"user-\\d+", replacement="user-***"),
+        MaskRule(field="metadata.api_key", replacement="<redacted>"),
+    ]
+
+    clean_trace = mask_trace(trace, rules)
+
+``field`` is a dot-separated path into the Trace JSON (e.g. ``"output"``,
+``"metadata.session_id"``, ``"tool_calls.0.arguments.password"``).
+
+If ``pattern`` is given, only substrings matching the regex are replaced.
+If ``pattern`` is omitted the entire field value is replaced with
+``replacement``.
+"""
+
+from __future__ import annotations
+
+import copy
+import re
+from typing import Any
+
+from pydantic import BaseModel
+
+from .core import Trace
+
+
+class MaskRule(BaseModel):
+    """A single masking rule.
+
+    Attributes:
+        field: Dot-separated path to the field inside the Trace JSON structure.
+            List indices are supported as numeric segments (e.g.
+            ``"tool_calls.0.arguments.token"``).
+        pattern: Optional regex.  When set, only matching substrings are
+            replaced.  When omitted the entire field value is overwritten.
+        replacement: The string substituted in place of the matched content.
+            Defaults to ``"<masked>"``.
+    """
+
+    field: str
+    pattern: str | None = None
+    replacement: str = "<masked>"
+
+
+def mask_trace(trace: Trace, rules: list[MaskRule]) -> Trace:
+    """Return a deep copy of *trace* with all *rules* applied.
+
+    The original trace is never mutated.
+    """
+    data: dict[str, Any] = copy.deepcopy(trace.model_dump(mode="json"))
+    for rule in rules:
+        _apply_rule(data, rule)
+    return Trace.model_validate(data)
+
+
+# ---------------------------------------------------------------------------
+# Internal helpers
+# ---------------------------------------------------------------------------
+
+
+def _apply_rule(data: dict[str, Any], rule: MaskRule) -> None:
+    segments = rule.field.split(".")
+    _set_at_path(data, segments, rule)
+
+
+def _set_at_path(
+    node: Any,
+    segments: list[str],
+    rule: MaskRule,
+) -> None:
+    if not segments:
+        return
+
+    key = segments[0]
+    rest = segments[1:]
+
+    # Resolve integer indices for lists.
+    if isinstance(node, list):
+        try:
+            idx = int(key)
+        except ValueError:
+            return
+        if idx < 0 or idx >= len(node):
+            return
+        if not rest:
+            node[idx] = _apply_replacement(node[idx], rule)
+        else:
+            _set_at_path(node[idx], rest, rule)
+        return
+
+    if not isinstance(node, dict) or key not in node:
+        return
+
+    if not rest:
+        node[key] = _apply_replacement(node[key], rule)
+    else:
+        _set_at_path(node[key], rest, rule)
+
+
+def _apply_replacement(value: Any, rule: MaskRule) -> Any:
+    if rule.pattern is None:
+        return rule.replacement
+    if not isinstance(value, str):
+        # Only string fields support regex replacement.
+        return value
+    return re.sub(rule.pattern, rule.replacement, value)

{agentprdiff-0.3.0 → agentprdiff-0.4.0}/src/agentprdiff/runner.py

@@ -16,6 +16,7 @@ from pydantic import BaseModel, ConfigDict, Field
 from .core import GradeResult, Suite, Trace, run_agent
 from .differ import TraceDelta, diff_traces
 from .store import BaselineStore
+from .trace_store import TraceStore
 
 
 class CaseReport(BaseModel):
@@ -73,7 +74,7 @@ class RunReport(BaseModel):
 class Runner:
     """Runs suites in record or check mode."""
 
-    def __init__(self, store: BaselineStore) -> None:
+    def __init__(self, store: BaselineStore | TraceStore) -> None:
         self.store = store
 
     # ------------------------------------------------------------------ api

agentprdiff-0.4.0/src/agentprdiff/trace_store.py

@@ -0,0 +1,103 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: MIT
+"""Pluggable trace store interface.
+
+The built-in :class:`~agentprdiff.store.BaselineStore` writes traces to the
+local filesystem.  ``TraceStore`` is a simple protocol that lets teams route
+traces to any backend — a database, object storage, or a remote telemetry
+service — without modifying the runner.
+
+Usage::
+
+    from agentprdiff.trace_store import TraceStore
+    from agentprdiff.core import Trace
+
+    class MyDBStore(TraceStore):
+        def save_baseline(self, trace: Trace) -> None:
+            db.upsert(suite=trace.suite_name, case=trace.case_name, data=trace.model_dump())
+
+        def load_baseline(self, suite_name: str, case_name: str) -> Trace | None:
+            row = db.get(suite=suite_name, case=case_name)
+            return Trace.model_validate(row) if row else None
+
+        def save_run_trace(self, run_id: str, trace: Trace) -> None:
+            db.insert(run_id=run_id, data=trace.model_dump())
+
+    runner = Runner(store=MyDBStore())
+
+Implementations only need to satisfy the three-method interface.  The
+``ensure_initialized`` and ``fresh_run_id`` methods have sensible defaults and
+are optional to override.
+"""
+
+from __future__ import annotations
+
+import uuid
+from abc import ABC, abstractmethod
+
+from .core import Trace
+
+
+class TraceStore(ABC):
+    """Abstract base class for trace persistence backends.
+
+    Subclass this to plug in any storage backend.  The :class:`Runner` only
+    calls the three abstract methods plus :meth:`ensure_initialized` and
+    :meth:`fresh_run_id`.
+    """
+
+    @abstractmethod
+    def save_baseline(self, trace: Trace) -> None:
+        """Persist *trace* as the canonical baseline for its suite+case."""
+
+    @abstractmethod
+    def load_baseline(self, suite_name: str, case_name: str) -> Trace | None:
+        """Return the stored baseline, or ``None`` if none exists yet."""
+
+    @abstractmethod
+    def save_run_trace(self, run_id: str, trace: Trace) -> None:
+        """Persist a trace produced during a ``check`` run."""
+
+    def ensure_initialized(self) -> None:  # noqa: B027
+        """Called once per runner invocation before any traces are written.
+
+        Override to create tables, buckets, or directories on first use.
+        The default implementation is a no-op.
+        """
+
+    def fresh_run_id(self) -> str:
+        """Return a unique identifier for the current run.
+
+        Override to use your own ID scheme (e.g. a database sequence).
+        The default returns a random 12-hex-character string.
+        """
+        return uuid.uuid4().hex[:12]
+
+
+class InMemoryTraceStore(TraceStore):
+    """In-memory store for testing and ephemeral pipelines.
+
+    Baselines and run traces are stored in plain dicts and lost when the
+    process exits.  Useful in CI when you want to run ``check`` immediately
+    after ``record`` without touching the filesystem.
+
+    Example::
+
+        store = InMemoryTraceStore()
+        runner = Runner(store=store)
+        runner.record(my_suite)
+        report = runner.check(my_suite)
+    """
+
+    def __init__(self) -> None:
+        self._baselines: dict[tuple[str, str], Trace] = {}
+        self._runs: dict[tuple[str, str, str], Trace] = {}
+
+    def save_baseline(self, trace: Trace) -> None:
+        self._baselines[(trace.suite_name, trace.case_name)] = trace
+
+    def load_baseline(self, suite_name: str, case_name: str) -> Trace | None:
+        return self._baselines.get((suite_name, case_name))
+
+    def save_run_trace(self, run_id: str, trace: Trace) -> None:
+        self._runs[(run_id, trace.suite_name, trace.case_name)] = trace

{agentprdiff-0.3.0 → agentprdiff-0.4.0}/studio/README.md

@@ -37,6 +37,104 @@ Plus the orthogonal **intake modes** for projects you create inside Studio:
 * **zip** — upload an archive, same execution path as git.
 * **http** — point Studio at a deployed endpoint, author suites as JSON, runs go in-process via httpx (no Python execution, baselines stored in the DB).
 
+## Private git repos
+
+Studio supports two non-interactive auth paths for private repos. The
+container has no TTY, so git **never prompts** — you choose one of these
+before creating the project.
+
+### Option A — SSH (`git@github.com:owner/repo.git`)
+
+The Studio image installs `openssh-client`, so the only thing you need
+to provide is your SSH key. Mount your host's `~/.ssh` into the
+container (read-only is fine):
+
+```yaml
+# studio/docker-compose.yml — add to the studio service
+services:
+  studio:
+    # … existing config …
+    volumes:
+      - studio-data:/data
+      - ${HOME}/.ssh:/root/.ssh:ro    # ← add this line
+```
+
+Two gotchas worth knowing:
+
+* The container runs as root, so the SSH client looks at `/root/.ssh`.
+  If you'd prefer a non-default location, set `GIT_SSH_COMMAND="ssh -i
+  /path/to/key"` via `STUDIO_*` env (or directly on the service in
+  compose).
+* `known_hosts` lives in the same directory you mount. If you've never
+  ssh'd to the remote from your host, run
+  `ssh-keyscan github.com >> ~/.ssh/known_hosts` once before starting
+  Studio, otherwise the first clone fails with *"Host key verification
+  failed."* (Studio surfaces this error inline with the same hint.)
+
+Then point Studio at an SSH URL when you create the project:
+
+```
+git@github.com:vnageshwaran-de/private-repo.git
+```
+
+### Option B — HTTPS + token (`https://github.com/owner/repo.git`)
+
+Save a personal access token in **Studio's Secrets page** (top-right
+nav), then create the project with a plain `https://` URL. Studio reads
+the token at clone time, sends it via `Authorization: bearer …` in
+git's transient config, and never embeds it in the URL or writes it to
+disk.
+
+Secret name → host mapping (project-scoped wins over global):
+
+| Host | Secret name |
+|---|---|
+| `github.com` | `GITHUB_TOKEN` |
+| `gitlab.com` | `GITLAB_TOKEN` |
+| `bitbucket.org` | `BITBUCKET_TOKEN` |
+| Anything else (self-hosted Enterprise / on-prem) | `GIT_HTTPS_TOKEN` |
+
+The fallback `GIT_HTTPS_TOKEN` lets you point Studio at self-hosted
+GitHub Enterprise, self-hosted GitLab, or Gitea / Forgejo without
+baking the hostname into Studio's config.
+
+**What the token needs:** for GitHub classic PATs, `repo` scope. For
+GitHub fine-grained PATs, `Contents: read-only` on the repos you want
+to clone is enough. For GitLab, `read_repository` scope.
+
+**What Studio does with it:**
+
+* Reads the encrypted secret at clone time (Fernet at rest, plaintext
+  only in memory of the requesting worker).
+* Injects `http.https://<host>/.extraheader: Authorization: bearer
+  <token>` via `GIT_CONFIG_COUNT` / `GIT_CONFIG_KEY_N` /
+  `GIT_CONFIG_VALUE_N` env vars on the git subprocess.
+* Sets `GIT_TERMINAL_PROMPT=0` so git can't block on stdin even when
+  the token is wrong (you get a clean error in the UI instead of a
+  hung sync).
+* Redacts the token from any error string before it reaches the UI,
+  the logs, or the project row.
+
+**What Studio refuses:** URLs with embedded credentials
+(`https://user:pat@host/...`). The error tells you to strip the
+credential and save it in the Secrets page instead — embedded
+credentials persist in the project row, in git's reflog, and in the
+workspace's `.git/config`, which are all places they shouldn't be.
+
+**Recovering from a wrong / expired token:** open the Secrets page,
+update the value, click Sync on the project. The next clone picks up
+the new token. The old one isn't kept anywhere.
+
+### When to use which
+
+| Situation | Pick |
+|---|---|
+| Personal use, ssh-agent already configured on host | SSH |
+| Want to avoid mounting host paths into the container | HTTPS + token |
+| Self-hosted Enterprise, no SSH access | HTTPS + token (`GIT_HTTPS_TOKEN`) |
+| Multiple users sharing the Studio container | HTTPS + token (project-scoped secrets isolate per project) |
+| CI / headless deployment | HTTPS + token (no key material to ship) |
+
 ## Local development (without Docker)
 
 ```bash
@@ -74,7 +172,7 @@ Everything is env-driven. The most useful knobs:
 * Engine (`agentprdiff`) installed from PyPI (or a path, via `STUDIO_ENGINE_REQ`).
 * Studio backend (FastAPI + SQLAlchemy + httpx + GitPython).
 * Built SPA (Vite output) at `/opt/studio/frontend`.
-* `git`, `build-essential`, `tini` for clean signal handling.
+* `git`, `openssh-client`, `build-essential`, `tini` for clean signal handling. `openssh-client` is what makes `git@github.com:…` URLs work — see the [Private git repos](#private-git-repos) section above.
 
 The image runs as `uvicorn agentprdiff_studio.main:app --host 0.0.0.0 --port 8080` under `tini`. A healthcheck hits `/api/health` every 30s.
 

agentprdiff-0.4.0/video-tutorials/customer_support_agent/README.md

@@ -0,0 +1,181 @@
+# Customer Support Agent — agentprdiff Video Tutorial
+
+A **LangGraph ReAct agent** paired with **agentprdiff** snapshot tests to demonstrate
+how to catch behavioral regressions when models, prompts, or tools change.
+
+> **Tutorial narrative:** Build → Record → Break → Catch → Fix
+
+---
+
+## What you'll build
+
+| File | Purpose |
+|---|---|
+| `agent.py` | LangGraph ReAct agent with 3 tools: `lookup_order`, `process_refund`, `check_policy` |
+| `suite.py` | 12 agentprdiff test cases across 5 suites covering all 10 built-in graders |
+| `AGENTS.md` | Persistent instructions for AI coding agents working in this repo |
+| `.github/workflows/agentprdiff.yml` | CI: runs `agentprdiff check` on every PR |
+
+---
+
+## Prerequisites
+
+- Python 3.11+
+- An Anthropic API key (`ANTHROPIC_API_KEY`)
+
+---
+
+## Quick start
+
+```bash
+# 1. Enter the project
+cd video-tutorials/customer_support_agent
+
+# 2. Install dependencies
+pip install -r requirements.txt
+
+# 3. Set your API key
+echo "ANTHROPIC_API_KEY=sk-ant-..." > .env
+
+# 4. Smoke test the agent manually
+python agent.py
+
+# 5. Record baselines (run once on the known-good version)
+agentprdiff record suite.py
+
+# 6. Check for regressions
+agentprdiff check suite.py
+```
+
+---
+
+## Tutorial walkthrough
+
+### Step 1 — Explore the agent
+
+Open `agent.py`. The agent is a standard LangGraph `StateGraph` with three nodes:
+
+```
+HumanMessage → [agent] → [tools] → [agent] → ... → AIMessage
+```
+
+Three mock tools simulate a real backend (no real database needed):
+
+- **`lookup_order(order_id)`** — returns order status, item, category, amount
+- **`process_refund(order_id, reason)`** — approves refunds on delivered orders only
+- **`check_policy(category)`** — returns the return/refund policy for an item category
+
+### Step 2 — Run the agent manually
+
+```bash
+python agent.py
+```
+
+You'll see three queries answered: order status, refund request, and policy lookup.
+
+### Step 3 — Understand the test suite
+
+Open `suite.py`. It contains **5 suites** covering distinct behavior categories:
+
+| Suite | Cases | Key graders demonstrated |
+|---|---|---|
+| `refund_flow` | 3 | `tool_sequence`, `no_tool_called`, `regex_match`, `semantic` |
+| `policy_queries` | 3 | `tool_called`, `output_length_lt`, `contains_any` |
+| `order_status` | 2 | `no_tool_called` (agent doesn't over-call) |
+| `multi_step_reasoning` | 2 | `tool_sequence` (3 steps), `cost_lt_usd` |
+| `out_of_scope` | 2 | `no_tool_called` (all tools), graceful fallback |
+
+All 10 agentprdiff graders are used:
+`contains` · `contains_any` · `regex_match` · `tool_called` · `tool_sequence` ·
+`no_tool_called` · `output_length_lt` · `latency_lt_ms` · `cost_lt_usd` · `semantic`
+
+### Step 4 — Record baselines
+
+```bash
+agentprdiff record suite.py
+```
+
+This runs every case once and writes JSON snapshots to `.agentprdiff/baselines/`.
+Commit these files — they are the "known good" reference for CI.
+
+```bash
+git add .agentprdiff/baselines/
+git commit -m "chore: record initial agentprdiff baselines"
+```
+
+### Step 5 — Introduce a regression (the "aha!" moment)
+
+Swap the model to an older, less capable one:
+
+```bash
+export ANTHROPIC_MODEL=claude-3-haiku-20240307
+```
+
+Now run the check:
+
+```bash
+agentprdiff check suite.py
+```
+
+You'll see failures like:
+
+```
+FAIL  refund_flow / refund_happy_path
+  tool_sequence(["lookup_order", "process_refund"]) — FAILED
+    actual sequence: ["lookup_order"]   ← haiku skipped the refund step
+
+FAIL  multi_step_reasoning / full_refund_journey
+  semantic(...) — FAILED
+    judge: "agent acknowledged the issue but did not process the refund"
+```
+
+This is the core value of agentprdiff: **a model swap that looks safe silently changes behavior**.
+
+### Step 6 — Fix or re-record
+
+**Option A — Fix the regression** (revert the model swap):
+
+```bash
+export ANTHROPIC_MODEL=claude-3-5-haiku-20241022
+agentprdiff check suite.py   # passes again
+```
+
+**Option B — Accept the new behavior** (intentional change):
+
+```bash
+agentprdiff record suite.py
+git add .agentprdiff/baselines/
+git commit -m "chore: update baselines after model change"
+# Write a ## Behavior Change section in your PR description
+```
+
+### Step 7 — CI enforces it forever
+
+Every PR that touches this directory triggers the GitHub Actions workflow.
+If `agentprdiff check` exits non-zero, the PR is blocked. Reviewers see
+the baseline diff in the uploaded artifact.
+
+---
+
+## All 10 graders — quick reference
+
+| Grader | What it checks | Example in suite.py |
+|---|---|---|
+| `contains(text)` | Output contains substring | `contains("refund")` |
+| `contains_any([...])` | Output contains at least one substring | `contains_any(["30 days", "30-day"])` |
+| `regex_match(pattern)` | Output matches regex | `regex_match(r"REF-\d+")` |
+| `tool_called(name)` | Tool was called at least once | `tool_called("lookup_order")` |
+| `tool_sequence([...])` | Tools were called in this exact order | `tool_sequence(["lookup_order", "process_refund"])` |
+| `no_tool_called(name)` | Tool was never called | `no_tool_called("process_refund")` |
+| `output_length_lt(n)` | Output is fewer than n characters | `output_length_lt(400)` |
+| `latency_lt_ms(ms)` | End-to-end latency under budget | `latency_lt_ms(10_000)` |
+| `cost_lt_usd(usd)` | Token cost under budget | `cost_lt_usd(0.05)` |
+| `semantic(description)` | LLM-as-judge checks intent | `semantic("agent confirms refund approved")` |
+
+---
+
+## Project conventions
+
+See `AGENTS.md` for the full set of rules AI coding agents must follow when
+modifying this project — including when to re-record baselines, code style,
+and what they must never touch.

agentprdiff-0.4.0/video-tutorials/customer_support_agent/suites/README.md

@@ -0,0 +1,82 @@
+# ShopFast customer support agent — agentprdiff suite
+
+Behavioral regression tests for the ShopFast LangGraph customer support agent.
+The suite covers 12 cases across 5 suites pinning the agent's refund logic,
+policy look-ups, order status, multi-step reasoning, and out-of-scope handling.
+
+## Quick start
+
+```bash
+# Install dependencies (from the project root)
+pip install -r requirements.txt
+
+# Record baselines (first run, or after an intentional change)
+agentprdiff record suites/customer_support.py
+
+# Check for regressions (CI gate)
+agentprdiff check suites/customer_support.py
+```
+
+## Running one case
+
+```bash
+# By substring
+agentprdiff check  suites/customer_support.py --case refund_happy_path
+agentprdiff review suites/customer_support.py --case refund_happy_path   # verbose, exit 0
+
+# By glob
+agentprdiff check  suites/customer_support.py --case "*refund*"
+
+# List all case names
+agentprdiff check  suites/customer_support.py --list
+```
+
+## Semantic Judge Keys
+
+This suite uses `semantic(...)` graders in the following cases:
+`refund_happy_path`, `refund_order_not_found`, `refund_in_transit_order`,
+`policy_footwear`, `policy_unknown_category`, `full_refund_journey`,
+`status_then_policy_no_refund`, `off_topic_weather`, `vague_refund_no_order_id`.
+
+**CI judge mode: `fake_judge` (keyword matching, free).**
+
+The CI workflow does NOT set `AGENTGUARD_JUDGE` or a judge-provider key, so
+the semantic graders run in `fake_judge` mode — fast, free, but only keyword
+matching. The rubric strings are written to pass under keyword matching for the
+happy path; they add a human-readable description of intent for reviewers.
+
+To switch to a real LLM judge locally or in CI:
+
+```bash
+# Anthropic judge (recommended — cheaper)
+export AGENTGUARD_JUDGE=anthropic
+export ANTHROPIC_API_KEY=sk-ant-...
+agentprdiff check suites/customer_support.py
+
+# OpenAI judge
+export AGENTGUARD_JUDGE=openai
+export OPENAI_API_KEY=sk-...
+agentprdiff check suites/customer_support.py
+```
+
+Add the corresponding secret in GitHub Settings → Secrets and variables →
+Actions, then add `AGENTGUARD_JUDGE: anthropic` and
+`ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}` to the workflow YAML's
+`env:` block.
+
+## Stub strategy
+
+The production tools (`lookup_order`, `process_refund`, `check_policy`) call
+external APIs only when `LIVE_TOOLS=true`. The suite always runs with
+`LIVE_TOOLS=false` (the default), which activates the in-module mock data.
+See `_stubs.py` for the expected return shapes.
+
+## Suite map
+
+| Suite | Cases | What it pins |
+|---|---|---|
+| `refund_flow` | 3 | lookup-before-refund ordering; no refund on unknown/in-transit orders |
+| `policy_queries` | 3 | correct tool called; right window returned; graceful unknown-category fallback |
+| `order_status` | 2 | no spurious tool calls; correct status text |
+| `multi_step_reasoning` | 2 | 3-tool chain; in-transit + policy without refund |
+| `out_of_scope` | 2 | no tool called; agent asks for missing order ID |