agentprdiff 0.4.0__tar.gz → 0.5.0__tar.gz

{agentprdiff-0.4.0 → agentprdiff-0.5.0}/CHANGELOG.md

@@ -8,6 +8,94 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.5.0] — 2026-08-30
+
+The "trustworthy green builds" release: strict judging, stable assertion
+identity, frozen baselines, a flakiness guard, parallel execution, async
+agents, and an official GitHub Action that posts the behavioral diff on
+pull requests.
+
+### Added
+
+- `async def` agents are supported transparently: `run_agent` resolves
+  the returned coroutine (including from inside an already-running event
+  loop, e.g. Jupyter — resolved on a dedicated thread there), so async
+  LangGraph/LlamaIndex-style pipelines no longer need a sync wrapper.
+- `--concurrency N` on `record` and `check`: executes up to N cases at
+  once on a thread pool. Suites are I/O-bound, so wall-clock time drops
+  near-linearly; works for sync and async agents, preserves suite order
+  in reports, and keeps storage writes and diffing on the calling
+  thread. Agents must be safe to call from multiple threads when
+  concurrency > 1. Pairs with `--runs`: `--runs 3 --concurrency 8`
+  costs 3× but takes roughly 1× wall-clock.
+
+- Official GitHub Action (`uses: vnageshwaran-de/agentprdiff@main`): a
+  composite action at the repo root that installs agentprdiff, runs
+  `check` over your suites, and posts the behavioral diff — assertion
+  flips, cost/latency deltas, output diffs, multi-run tallies — as a
+  single living comment on the pull request (updated in place per push,
+  stdlib-only comment script, graceful fallback to job-log output on
+  fork PRs). Inputs cover suites/version/install/root/runs/strict-judge/
+  comment/github-token/python-version; output `regressed`. Requires
+  agentprdiff ≥ 0.5.0.
+- `record` and `check` now accept multiple suite files
+  (`agentprdiff check suites/*.py` works as the docs always implied).
+- `--json-out` now writes every suite's report from the invocation as
+  `{"reports": [...]}`; previously each suite overwrote the same file so
+  only the last survived. Consumers of the old single-suite envelope
+  should read the `reports` list instead.
+
+- Multi-run flakiness handling: `agentprdiff check --runs N` executes
+  each case N times, and a case passes when at least its
+  `min_pass_rate` fraction of attempts fully pass —
+  `case(..., min_pass_rate=0.6)` with `--runs 3` tolerates one
+  stochastic wobble out of three. Defaults (`--runs 1`,
+  `min_pass_rate=1.0`) preserve the exact single-shot behavior. Diffs
+  and reports use a representative attempt (the last fully-passing one
+  when any exists, otherwise the last), the terminal report shows the
+  tally (`2/3 runs passed (required ≥ 60%)`), and `CaseReport` gains
+  `runs_total` / `runs_passed` / `min_pass_rate` and a `pass_rate`
+  property. `record` always runs once — a baseline is a single
+  known-good trace.
+
+- `agentprdiff check --strict-judge`: fails the run when any `semantic()`
+  grader was judged by `fake_judge` via **silent fallback** (no judge env
+  var and no provider API key). A green build whose semantic assertions
+  were graded by keyword matching is not a green build. Explicit opt-in
+  via `AGENTPRDIFF_JUDGE=fake` still passes. Strict mode will become the
+  default in v1.0. Silent-fallback grading is also flagged in
+  `GradeResult.metadata["silent_fallback"]` for programmatic use.
+- Stable assertion identity: every grader factory now accepts an optional
+  `id=` keyword (e.g. `contains("refund", id="mentions-refund")`). Diffs
+  match assertions by `id` when both sides have one, falling back to the
+  display name — so renaming an argument or rewording a semantic rubric
+  no longer registers as a removed + added assertion (a documented source
+  of false regressions). `GradeResult` and `AssertionChange` gain a
+  `grader_id` field.
+- Baselines now persist their grader verdicts
+  (`trace.metadata["grader_results"]`), recorded at `record` time. `check`
+  reads the stored verdicts instead of re-running graders against the
+  baseline — which previously meant a paid, nondeterministic LLM-judge
+  call against the baseline on every check. Legacy baselines without
+  stored verdicts fall back to the old re-run path; re-record to migrate.
+- `cost_lt_usd` flags suspicious passes: when a trace made LLM calls but
+  recorded `$0.0000` total cost (usually a model missing from the pricing
+  table), the pass is annotated in `reason` and
+  `metadata["zero_cost_with_llm_calls"]` instead of silently succeeding.
+
+### Changed
+
+- The judge-selection env var is now `AGENTPRDIFF_JUDGE`. The legacy
+  `AGENTGUARD_JUDGE` name (from a pre-release name of this project) still
+  works and emits a `DeprecationWarning`; it will be removed in v1.0.
+  When both are set, `AGENTPRDIFF_JUDGE` wins. The judge banner names
+  whichever variable was actually used.
+
+### Fixed
+
+- A mypy type error in the review reporter's assertion-pairing code;
+  the reporter now pairs assertions by stable `grader_id` when present.
+
 ## [0.4.0] — 2026-08-29
 
 Minor release bundling the three feature PRs merged since 0.3.1, plus

{agentprdiff-0.4.0 → agentprdiff-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentprdiff
-Version: 0.4.0
+Version: 0.5.0
 Summary: Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change.
 Project-URL: Homepage, https://agentprdiff.dev
 Project-URL: Documentation, https://agentprdiff.dev
@@ -117,7 +117,8 @@ That's the whole product. Five CLI commands (`init`, `record`, `check`, `review`
 - **10 batteries-included graders** — `contains`, `contains_any`, `regex_match`, `tool_called`, `tool_sequence`, `no_tool_called`, `output_length_lt`, `latency_lt_ms`, `cost_lt_usd`, `semantic` (LLM-as-judge with pluggable backend).
 - **Baseline store** — JSON files under `.agentprdiff/baselines/`, meant to be **committed**. Reviewers see trace changes in pull requests.
 - **Diff engine** — per-case `TraceDelta` with assertion pass/fail changes, cost delta, latency delta, tool-sequence changes, and a unified output diff.
-- **CI-ready CLI** — exit 1 on regression, `--json-out` for artifact archiving, Rich-formatted terminal output.
+- **CI-ready CLI** — exit 1 on regression, `--json-out` for artifact archiving, Rich-formatted terminal output, `--strict-judge` so a missing judge key can't green-light keyword matching, and `--runs N` + per-case `min_pass_rate` so one stochastic wobble doesn't fail the build.
+- **GitHub Action with PR comments** — `uses: vnageshwaran-de/agentprdiff@main` runs `check` and posts the behavioral diff (assertion flips, cost/latency deltas, output diffs) as a living comment on the pull request. [Docs](https://agentprdiff.dev/scenarios/ci-cd/).
 - **Zero SDK lock-in** — works with OpenAI, Anthropic, Gemini, Bedrock, LangChain, LangGraph, LlamaIndex, Vercel AI SDK, custom wrappers — if you can wrap your agent in a function, `agentprdiff` can test it.
 - **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (sync **or** async — `AsyncOpenAI` is supported by the same context manager) or any OpenAI-compatible provider (Groq / Gemini / OpenRouter / Ollama / vLLM / Together / Fireworks / DeepInfra) or the Anthropic SDK. No manual `Trace` wiring required.
 

{agentprdiff-0.4.0 → agentprdiff-0.5.0}/README.md

@@ -77,7 +77,8 @@ That's the whole product. Five CLI commands (`init`, `record`, `check`, `review`
 - **10 batteries-included graders** — `contains`, `contains_any`, `regex_match`, `tool_called`, `tool_sequence`, `no_tool_called`, `output_length_lt`, `latency_lt_ms`, `cost_lt_usd`, `semantic` (LLM-as-judge with pluggable backend).
 - **Baseline store** — JSON files under `.agentprdiff/baselines/`, meant to be **committed**. Reviewers see trace changes in pull requests.
 - **Diff engine** — per-case `TraceDelta` with assertion pass/fail changes, cost delta, latency delta, tool-sequence changes, and a unified output diff.
-- **CI-ready CLI** — exit 1 on regression, `--json-out` for artifact archiving, Rich-formatted terminal output.
+- **CI-ready CLI** — exit 1 on regression, `--json-out` for artifact archiving, Rich-formatted terminal output, `--strict-judge` so a missing judge key can't green-light keyword matching, and `--runs N` + per-case `min_pass_rate` so one stochastic wobble doesn't fail the build.
+- **GitHub Action with PR comments** — `uses: vnageshwaran-de/agentprdiff@main` runs `check` and posts the behavioral diff (assertion flips, cost/latency deltas, output diffs) as a living comment on the pull request. [Docs](https://agentprdiff.dev/scenarios/ci-cd/).
 - **Zero SDK lock-in** — works with OpenAI, Anthropic, Gemini, Bedrock, LangChain, LangGraph, LlamaIndex, Vercel AI SDK, custom wrappers — if you can wrap your agent in a function, `agentprdiff` can test it.
 - **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (sync **or** async — `AsyncOpenAI` is supported by the same context manager) or any OpenAI-compatible provider (Groq / Gemini / OpenRouter / Ollama / vLLM / Together / Fireworks / DeepInfra) or the Anthropic SDK. No manual `Trace` wiring required.
 

{agentprdiff-0.4.0 → agentprdiff-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agentprdiff"
-version = "0.4.0"
+version = "0.5.0"
 description = "Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change."
 readme = "README.md"
 requires-python = ">=3.10"

{agentprdiff-0.4.0 → agentprdiff-0.5.0}/src/agentprdiff/cli.py

@@ -59,6 +59,17 @@ _LIST_OPTION = click.option(
     is_flag=True,
     help="Print suite/case names without running anything, then exit.",
 )
+_CONCURRENCY_OPTION = click.option(
+    "--concurrency",
+    type=click.IntRange(min=1),
+    default=1,
+    show_default=True,
+    help=(
+        "Execute up to this many cases at once on a thread pool. Agent "
+        "suites are I/O-bound, so this cuts wall-clock time near-linearly. "
+        "Your agent must be safe to call from multiple threads."
+    ),
+)
 
 
 @click.group(help="Snapshot testing for LLM agents.")
@@ -87,26 +98,33 @@ def cmd_init(ctx: click.Context) -> None:
 
 
 @main.command("record")
-@click.argument("suite_file", type=click.Path(exists=True, dir_okay=False, path_type=Path))
+@click.argument(
+    "suite_files",
+    nargs=-1,
+    required=True,
+    type=click.Path(exists=True, dir_okay=False, path_type=Path),
+)
 @click.option("--json-out", type=click.Path(path_type=Path), help="Write JSON report to this path.")
 @_CASE_OPTION
 @_SKIP_OPTION
 @_LIST_OPTION
+@_CONCURRENCY_OPTION
 @click.pass_context
 def cmd_record(
     ctx: click.Context,
-    suite_file: Path,
+    suite_files: tuple[Path, ...],
     json_out: Path | None,
     case_patterns: tuple[str, ...],
     skip_patterns: tuple[str, ...],
     list_only: bool,
+    concurrency: int,
 ) -> None:
-    """Run every suite in SUITE_FILE and save each trace as the baseline."""
+    """Run every suite in SUITE_FILES and save each trace as the baseline."""
     store: BaselineStore = ctx.obj["store"]
-    runner = Runner(store)
+    runner = Runner(store, concurrency=concurrency)
     terminal = TerminalReporter()
 
-    suites_all = load_suites(suite_file)
+    suites_all = [s for f in suite_files for s in load_suites(f)]
     if list_only:
         _print_listing(suites_all)
         return
@@ -114,21 +132,28 @@ def cmd_record(
     suites = _select_or_exit(suites_all, case_patterns, skip_patterns)
 
     any_error = False
+    reports = []
     for s in suites:
         report = runner.record(s)
         terminal.render(report)
-        if json_out:
-            JsonReporter().render(report, json_out)
+        reports.append(report)
         # record mode doesn't fail on grader failures, but a literal exception
         # during execution still warrants a nonzero exit.
         if any(cr.trace.error for cr in report.case_reports):
             any_error = True
+    if json_out:
+        JsonReporter().render_many(reports, json_out)
 
     sys.exit(1 if any_error else 0)
 
 
 @main.command("check")
-@click.argument("suite_file", type=click.Path(exists=True, dir_okay=False, path_type=Path))
+@click.argument(
+    "suite_files",
+    nargs=-1,
+    required=True,
+    type=click.Path(exists=True, dir_okay=False, path_type=Path),
+)
 @click.option("--json-out", type=click.Path(path_type=Path), help="Write JSON report to this path.")
 @_CASE_OPTION
 @_SKIP_OPTION
@@ -140,22 +165,49 @@ def cmd_record(
     show_default=True,
     help="Exit non-zero when regressions are detected.",
 )
+@click.option(
+    "--strict-judge",
+    is_flag=True,
+    default=False,
+    help=(
+        "Fail when any semantic() grader silently fell back to fake_judge "
+        "(no AGENTPRDIFF_JUDGE and no provider API key set). Recommended in "
+        "CI; will become the default in v1.0. Explicit opt-in via "
+        "AGENTPRDIFF_JUDGE=fake still passes."
+    ),
+)
+@click.option(
+    "--runs",
+    type=click.IntRange(min=1),
+    default=1,
+    show_default=True,
+    help=(
+        "Execute each case this many times; a case passes when at least its "
+        "min_pass_rate fraction of attempts fully pass (default 1.0 — all "
+        "attempts). The flakiness guard for stochastic agents: e.g. --runs 3 "
+        "with case(..., min_pass_rate=0.6) tolerates one wobble out of three."
+    ),
+)
+@_CONCURRENCY_OPTION
 @click.pass_context
 def cmd_check(
     ctx: click.Context,
-    suite_file: Path,
+    suite_files: tuple[Path, ...],
     json_out: Path | None,
     case_patterns: tuple[str, ...],
     skip_patterns: tuple[str, ...],
     list_only: bool,
     fail_on_regression: bool,
+    strict_judge: bool,
+    runs: int,
+    concurrency: int,
 ) -> None:
-    """Run every suite in SUITE_FILE and diff against saved baselines."""
+    """Run every suite in SUITE_FILES and diff against saved baselines."""
     store: BaselineStore = ctx.obj["store"]
-    runner = Runner(store)
+    runner = Runner(store, runs=runs, concurrency=concurrency)
     terminal = TerminalReporter()
 
-    suites_all = load_suites(suite_file)
+    suites_all = [s for f in suite_files for s in load_suites(f)]
     if list_only:
         _print_listing(suites_all)
         return
@@ -163,12 +215,30 @@ def cmd_check(
     suites = _select_or_exit(suites_all, case_patterns, skip_patterns)
 
     any_regression = False
+    silent_judge_cases: list[str] = []
+    reports = []
     for s in suites:
         report = runner.check(s)
         terminal.render(report)
-        if json_out:
-            JsonReporter().render(report, json_out)
+        reports.append(report)
         any_regression = any_regression or report.has_regression
+        for cr in report.case_reports:
+            if any(r.metadata.get("silent_fallback") for r in cr.grader_results):
+                silent_judge_cases.append(f"{cr.suite_name}/{cr.case_name}")
+    if json_out:
+        JsonReporter().render_many(reports, json_out)
+
+    if strict_judge and silent_judge_cases:
+        click.echo(
+            "\n--strict-judge: semantic() graders were judged by fake_judge via "
+            "silent fallback (no AGENTPRDIFF_JUDGE, no OPENAI_API_KEY/"
+            "ANTHROPIC_API_KEY) in:\n  "
+            + "\n  ".join(silent_judge_cases)
+            + "\nSet a real judge (AGENTPRDIFF_JUDGE=openai|anthropic plus the "
+            "matching API key) or opt in explicitly with AGENTPRDIFF_JUDGE=fake.",
+            err=True,
+        )
+        sys.exit(1)
 
     sys.exit(1 if (any_regression and fail_on_regression) else 0)
 

{agentprdiff-0.4.0 → agentprdiff-0.5.0}/src/agentprdiff/core.py

@@ -11,9 +11,12 @@ and diffed across runs.
 
 from __future__ import annotations
 
+import asyncio
+import inspect
 import time
 import uuid
 from collections.abc import Callable
+from concurrent.futures import ThreadPoolExecutor
 from datetime import datetime, timezone
 from typing import Any
 
@@ -99,6 +102,11 @@ class GradeResult(BaseModel):
 
     passed: bool
     grader_name: str
+    # Optional stable identity for baseline matching. When set (via the
+    # `id=` argument on grader factories), diffs match assertions by this id
+    # instead of the display name, so renaming an argument or rewording a
+    # rubric doesn't register as a removed + added assertion.
+    grader_id: str | None = None
     reason: str = ""
     metadata: dict[str, Any] = Field(default_factory=dict)
 
@@ -122,14 +130,36 @@ class Case(BaseModel):
     input: Any
     expect: list[Grader] = Field(default_factory=list)
     tags: list[str] = Field(default_factory=list)
+    # Minimum fraction of attempts that must fully pass when the runner
+    # executes the case more than once (`check --runs N`). 1.0 = every
+    # attempt must pass (the single-run behavior). E.g. 0.6 with --runs 3
+    # tolerates one stochastic wobble out of three.
+    min_pass_rate: float = Field(default=1.0, gt=0.0, le=1.0)
 
 
 # An Agent is any callable `(input) -> (output, Trace)`. If the user's agent
 # returns only an output, the runner wraps it so latency is captured but the
-# returned `Trace` has empty llm_calls / tool_calls.
+# returned `Trace` has empty llm_calls / tool_calls. `async def` agents are
+# supported: the runner resolves the coroutine for you (see `run_agent`).
 AgentFn = Callable[[Any], Any]
 
 
+def _resolve_coroutine(coro: Any) -> Any:
+    """Run a coroutine to completion and return its result.
+
+    Works both from plain sync code (the normal CLI path) and from inside an
+    already-running event loop (Jupyter, async test runners), where
+    ``asyncio.run`` would raise — there we hand the coroutine to a fresh loop
+    on a dedicated thread instead.
+    """
+    try:
+        asyncio.get_running_loop()
+    except RuntimeError:
+        return asyncio.run(coro)
+    with ThreadPoolExecutor(max_workers=1) as pool:
+        return pool.submit(asyncio.run, coro).result()
+
+
 class Suite(BaseModel):
     """A named group of cases sharing one agent under test."""
 
@@ -154,12 +184,28 @@ def suite(name: str, agent: AgentFn, cases: list[Case], description: str = "") -
     return Suite(name=name, agent=agent, cases=cases, description=description)
 
 
-def case(name: str, input: Any, expect: list[Grader], tags: list[str] | None = None) -> Case:
+def case(
+    name: str,
+    input: Any,
+    expect: list[Grader],
+    tags: list[str] | None = None,
+    min_pass_rate: float = 1.0,
+) -> Case:
     """Create a Case.
 
+    `min_pass_rate` only matters with `check --runs N` (N > 1): the case
+    passes when at least this fraction of attempts fully pass. Defaults to
+    1.0 — every attempt must pass.
+
     >>> c = case("refund", input="I want a refund", expect=[contains("refund")])
     """
-    return Case(name=name, input=input, expect=expect or [], tags=tags or [])
+    return Case(
+        name=name,
+        input=input,
+        expect=expect or [],
+        tags=tags or [],
+        min_pass_rate=min_pass_rate,
+    )
 
 
 # ---------------------------------------------------------------------------
@@ -179,11 +225,16 @@ def run_agent(
     If the agent returns a `(output, Trace)` tuple, we use the returned trace
     and just fill in the metadata we can see from out here (suite/case names).
     Otherwise we build a minimal trace with latency only.
+
+    `async def` agents work transparently: the returned coroutine is resolved
+    here, so latency covers the full awaited execution.
     """
     start = time.perf_counter()
     trace: Trace
     try:
         result = agent(input_value)
+        if inspect.iscoroutine(result):
+            result = _resolve_coroutine(result)
     except Exception as exc:  # noqa: BLE001 — we want to capture any failure mode
         elapsed_ms = (time.perf_counter() - start) * 1000.0
         return Trace(

{agentprdiff-0.4.0 → agentprdiff-0.5.0}/src/agentprdiff/differ.py

@@ -23,6 +23,7 @@ from .core import GradeResult, Trace
 
 class AssertionChange(BaseModel):
     grader_name: str
+    grader_id: str | None = None
     baseline_passed: bool | None  # None = grader didn't exist in baseline
     current_passed: bool
     current_reason: str = ""
@@ -100,16 +101,22 @@ def diff_traces(
         baseline_error=baseline.error if baseline else None,
     )
 
-    baseline_by_name: dict[str, bool] = {}
+    # Assertions are matched baseline<->current by stable grader_id when one
+    # was provided (the `id=` argument on grader factories), falling back to
+    # the display name. Matching by name alone means renaming an argument or
+    # rewording a semantic rubric shows up as a removed + added assertion —
+    # a documented source of false regressions.
+    baseline_by_key: dict[str, bool] = {}
     if baseline_results:
         for r in baseline_results:
-            baseline_by_name[r.grader_name] = r.passed
+            baseline_by_key[r.grader_id or r.grader_name] = r.passed
 
     for r in current_results:
         delta.assertion_changes.append(
             AssertionChange(
                 grader_name=r.grader_name,
-                baseline_passed=baseline_by_name.get(r.grader_name),
+                grader_id=r.grader_id,
+                baseline_passed=baseline_by_key.get(r.grader_id or r.grader_name),
                 current_passed=r.passed,
                 current_reason=r.reason,
             )

{agentprdiff-0.4.0 → agentprdiff-0.5.0}/src/agentprdiff/graders/deterministic.py

@@ -3,6 +3,12 @@
 These never call an LLM. Prefer them whenever the assertion can be expressed
 mechanically; reserve the semantic grader for things you genuinely can't
 encode as a rule.
+
+Every factory accepts an optional ``id=`` keyword: a stable identity used to
+match the assertion against baselines in diffs. Without an id, matching falls
+back to the display name (e.g. ``contains('refund')``), which means changing
+an argument reads as a removed + added assertion. Give long-lived assertions
+an id and rename freely.
 """
 
 from __future__ import annotations
@@ -26,7 +32,7 @@ def _output_str(trace: Trace) -> str:
         return ""
 
 
-def contains(substring: str, *, case_sensitive: bool = False) -> Grader:
+def contains(substring: str, *, case_sensitive: bool = False, id: str | None = None) -> Grader:
     """Pass iff the agent's final output contains `substring`."""
 
     def _grader(trace: Trace) -> GradeResult:
@@ -37,6 +43,7 @@ def contains(substring: str, *, case_sensitive: bool = False) -> Grader:
         return GradeResult(
             passed=passed,
             grader_name=f"contains({substring!r})",
+            grader_id=id,
             reason=(
                 f"output {'contains' if passed else 'does not contain'} {substring!r}"
             ),
@@ -45,7 +52,9 @@ def contains(substring: str, *, case_sensitive: bool = False) -> Grader:
     return _grader
 
 
-def contains_any(substrings: Sequence[str], *, case_sensitive: bool = False) -> Grader:
+def contains_any(
+    substrings: Sequence[str], *, case_sensitive: bool = False, id: str | None = None
+) -> Grader:
     """Pass iff the output contains at least one of the listed substrings."""
 
     def _grader(trace: Trace) -> GradeResult:
@@ -56,6 +65,7 @@ def contains_any(substrings: Sequence[str], *, case_sensitive: bool = False) ->
         return GradeResult(
             passed=passed,
             grader_name=f"contains_any({list(substrings)!r})",
+            grader_id=id,
             reason=(
                 f"matched {matched!r}"
                 if passed
@@ -66,7 +76,7 @@ def contains_any(substrings: Sequence[str], *, case_sensitive: bool = False) ->
     return _grader
 
 
-def regex_match(pattern: str, *, flags: int = 0) -> Grader:
+def regex_match(pattern: str, *, flags: int = 0, id: str | None = None) -> Grader:
     """Pass iff `pattern` matches the agent's final output."""
     compiled = re.compile(pattern, flags=flags)
 
@@ -77,6 +87,7 @@ def regex_match(pattern: str, *, flags: int = 0) -> Grader:
         return GradeResult(
             passed=passed,
             grader_name=f"regex_match({pattern!r})",
+            grader_id=id,
             reason=(
                 f"matched {m.group(0)!r}" if m else f"no match for {pattern!r}"
             ),
@@ -85,7 +96,7 @@ def regex_match(pattern: str, *, flags: int = 0) -> Grader:
     return _grader
 
 
-def tool_called(name: str, *, min_times: int = 1) -> Grader:
+def tool_called(name: str, *, min_times: int = 1, id: str | None = None) -> Grader:
     """Pass iff the tool `name` was called at least `min_times` times."""
 
     def _grader(trace: Trace) -> GradeResult:
@@ -94,13 +105,14 @@ def tool_called(name: str, *, min_times: int = 1) -> Grader:
         return GradeResult(
             passed=passed,
             grader_name=f"tool_called({name!r}, min_times={min_times})",
+            grader_id=id,
             reason=f"tool {name!r} called {count} time(s), required >= {min_times}",
         )
 
     return _grader
 
 
-def no_tool_called(name: str) -> Grader:
+def no_tool_called(name: str, *, id: str | None = None) -> Grader:
     """Pass iff the tool `name` was NOT called."""
 
     def _grader(trace: Trace) -> GradeResult:
@@ -109,13 +121,16 @@ def no_tool_called(name: str) -> Grader:
         return GradeResult(
             passed=passed,
             grader_name=f"no_tool_called({name!r})",
+            grader_id=id,
             reason=f"tool {name!r} called {count} time(s); expected 0",
         )
 
     return _grader
 
 
-def tool_sequence(sequence: Sequence[str], *, strict: bool = False) -> Grader:
+def tool_sequence(
+    sequence: Sequence[str], *, strict: bool = False, id: str | None = None
+) -> Grader:
     """Pass iff the tool-call sequence matches `sequence`.
 
     If `strict=False` (default), `sequence` must appear as a subsequence of
@@ -137,13 +152,14 @@ def tool_sequence(sequence: Sequence[str], *, strict: bool = False) -> Grader:
         return GradeResult(
             passed=passed,
             grader_name=f"tool_sequence({list(sequence)!r}, strict={strict})",
+            grader_id=id,
             reason=f"actual tool sequence: {actual}",
         )
 
     return _grader
 
 
-def output_length_lt(max_chars: int) -> Grader:
+def output_length_lt(max_chars: int, *, id: str | None = None) -> Grader:
     """Pass iff the output has fewer than `max_chars` characters."""
 
     def _grader(trace: Trace) -> GradeResult:
@@ -152,13 +168,14 @@ def output_length_lt(max_chars: int) -> Grader:
         return GradeResult(
             passed=passed,
             grader_name=f"output_length_lt({max_chars})",
+            grader_id=id,
             reason=f"output length {n} chars, limit {max_chars}",
         )
 
     return _grader
 
 
-def latency_lt_ms(max_ms: float) -> Grader:
+def latency_lt_ms(max_ms: float, *, id: str | None = None) -> Grader:
     """Pass iff the trace's total latency is below `max_ms` milliseconds."""
 
     def _grader(trace: Trace) -> GradeResult:
@@ -166,21 +183,38 @@ def latency_lt_ms(max_ms: float) -> Grader:
         return GradeResult(
             passed=passed,
             grader_name=f"latency_lt_ms({max_ms})",
+            grader_id=id,
             reason=f"latency {trace.total_latency_ms:.1f} ms, limit {max_ms:.1f} ms",
         )
 
     return _grader
 
 
-def cost_lt_usd(max_usd: float) -> Grader:
-    """Pass iff the trace's total cost is below `max_usd` dollars."""
+def cost_lt_usd(max_usd: float, *, id: str | None = None) -> Grader:
+    """Pass iff the trace's total cost is below `max_usd` dollars.
+
+    When the trace made LLM calls but recorded $0.00 total cost, the pass is
+    flagged as suspicious in the reason and metadata — the usual cause is a
+    model missing from the pricing table, which would otherwise let this
+    grader trivially pass forever.
+    """
 
     def _grader(trace: Trace) -> GradeResult:
         passed = trace.total_cost_usd < max_usd
+        reason = f"cost ${trace.total_cost_usd:.4f}, limit ${max_usd:.4f}"
+        metadata: dict[str, object] = {}
+        if passed and trace.total_cost_usd == 0.0 and trace.llm_calls:
+            metadata["zero_cost_with_llm_calls"] = True
+            reason += (
+                f" (warning: {len(trace.llm_calls)} LLM call(s) recorded $0.0000 — "
+                "pricing may be missing for this model; see register_prices())"
+            )
         return GradeResult(
             passed=passed,
             grader_name=f"cost_lt_usd({max_usd})",
-            reason=f"cost ${trace.total_cost_usd:.4f}, limit ${max_usd:.4f}",
+            grader_id=id,
+            reason=reason,
+            metadata=metadata,
         )
 
     return _grader