agentprdiff 0.2.1__tar.gz → 0.2.2__tar.gz

agentprdiff-0.2.2/CHANGELOG.md

@@ -0,0 +1,243 @@
+# Changelog
+
+All notable changes to `agentprdiff` are documented in this file. Originally
+prototyped under the name `tracediff`; renamed before first public release.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.2] — 2026-04-28
+
+### Added
+
+- **Native `AsyncOpenAI` support in `agentprdiff.adapters.openai`.** The same
+  `instrument_client` and `instrument_tools` API now works with the async
+  OpenAI client — and any async OpenAI-compatible provider (Groq, Gemini,
+  OpenRouter, Ollama, vLLM, Together, Fireworks, DeepInfra). The adapter
+  inspects `client.chat.completions.create` at `with`-block entry; if it's
+  a coroutine function, an awaitable patched method is installed and the
+  user's `await client.chat.completions.create(...)` call sites work
+  unchanged. `instrument_tools` matches per-tool: `async def` tools come
+  back as `async def` wrappers (`await tools[name](**args)`), sync tools
+  stay sync — a single `TOOL_MAP` may freely mix the two. The `with` block
+  is a regular `with`, not `async with`, since the patch is bound to the
+  client instance rather than the running event loop. agentprdiff's runner
+  remains sync; async agents bridge with `asyncio.run` in their
+  `eval_agent` entry point. Removes the previous adoption recommendation
+  to use manual `Trace.record_llm_call` instrumentation for async agents.
+- **Updated `--recipe async-openai` scaffold** to use the new adapter
+  natively. The generated `_eval_agent.py` no longer carries TODO markers
+  for manual instrumentation; it imports `instrument_client` /
+  `instrument_tools` and wraps an async tool-calling loop with an
+  `asyncio.run` bridge to agentprdiff's sync runner.
+- **`agentprdiff review <suite_file>`** — new subcommand for local iteration
+  on a single failing case. Runs the same comparison `check` does, but
+  renders one verbose panel per case (input echo, full assertion table with
+  `was → now` baseline-vs-current marks, per-metric deltas for cost /
+  latency / prompt and completion tokens, tool-sequence diff, and a unified
+  output diff in its own panel when output changed) and **always exits 0**
+  so it can sit inside watcher / `entr` / `fzf` loops without flipping the
+  shell red on every regression. Accepts the same `--case` / `--skip` /
+  `--list` flags as `record` and `check`. The CI gate stays `agentprdiff
+  check`; `review` is the `pytest -k` of agentprdiff. New `ReviewReporter`
+  in `agentprdiff.reporters` powers the rendering and is reusable by
+  third-party tooling.
+- **`agentprdiff scaffold <name>`** — new subcommand that stamps out the
+  canonical adoption layout (`suites/__init__.py`, `_eval_agent.py`,
+  `_stubs.py`, `<name>.py`, `<name>_cases.md`, `suites/README.md`, and
+  `.github/workflows/agentprdiff.yml`). Three recipes via `--recipe`:
+  `sync-openai` (default; uses `instrument_client`), `async-openai` (manual
+  asyncio wrapper, until the async adapter ships in 0.3), and `stubbed`
+  (substitutes a single LLM helper — see the new "stubbed LLM-boundary
+  pattern" recipe in `docs/adapters.md`). The generated workflow includes
+  `permissions: contents: read` so GitHub Advanced Security stops flagging
+  it. Pre-existing files are never overwritten — they're reported as
+  `[skip]` and the rest are still written.
+- **Case dossier** (`suites/<name>_cases.md`) — new mandatory artifact
+  produced by `scaffold` and documented in AGENTS.md and
+  `docs/suite-layout.md`. Reviewer-facing markdown with one block per case
+  using a fixed five-field structure: *What it tests*, *Input*,
+  *Assertions* (plain English), *Code impacted* (file:line references back
+  to production code), and *Application impact* (one concrete sentence
+  about what breaks for end users on regression). Closes the gap between
+  case names that look meaningful in CI output ("article_summary_preserves_acquisition_entities")
+  and reviewers who need to know what each case actually pins.
+- New "stubbed LLM-boundary pattern" recipe in `docs/adapters.md` for
+  agents whose LLM call is wrapped in a single helper (summarization,
+  classification, embedding-prep). Stubbing the helper is cleaner than
+  stubbing the SDK client and works equally well for sync and async clients.
+- `agentprdiff record` and `agentprdiff check` now accept `--case PATTERN` and
+  `--skip PATTERN` for narrowing a run to a subset of cases. Patterns are
+  case-insensitive substrings by default and use `fnmatch` semantics when they
+  contain `*`, `?`, or `[`. Both flags are repeatable, accept comma-separated
+  lists (`--case refund,policy`), and support qualifier syntax
+  (`--case billing:refund*`). A leading `~` (or `!`) negates a pattern, so
+  `--case ~slow` is equivalent to `--skip slow`.
+- `agentprdiff record --list` / `check --list` prints suite and case names
+  without running anything, so you can discover what's filterable before
+  reaching for `--case`.
+- When a filter is active, the CLI now prints a per-suite header
+  (`running 2 of 4 cases in customer_support: ...`) so a partial selection is
+  visible at a glance. A filter that matches zero cases exits with code 2 and
+  prints the available case names — previously a typo'd filter would have
+  silently exited 0.
+
+### Changed
+
+- The CI workflow templates in `AGENTS.md`, `README.md`, and
+  `docs/ci-integration.md` now declare `permissions: contents: read`
+  explicitly. GitHub Advanced Security flags workflows without an explicit
+  permissions block, and least-privilege is the right default anyway.
+- Documented the recommended `.gitignore` entry (`artifacts/agentprdiff*.json`)
+  alongside every CI snippet that uses `--json-out artifacts/...`. The path
+  uploads cleanly as a CI build artifact, but the same file lands on every
+  local run; without an ignore line it eventually gets `git add`-ed by
+  accident.
+
+### Documentation
+
+- New "API keys" section in `AGENTS.md` covering both key surfaces (the
+  production agent's own keys vs. agentprdiff's semantic-judge keys —
+  `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `AGENTGUARD_JUDGE` with the
+  silent fake_judge fallback), local setup options (.env / shell export /
+  direnv), CI secret wiring, and a short list of "never do this." Adopting
+  AI agents are now instructed to explicitly prompt the user about which
+  env var their production agent reads, whether to use a real semantic
+  judge in CI, and to verify `.env` is gitignored.
+- The scaffolded `.github/workflows/agentprdiff.yml` now includes both
+  `OPENAI_API_KEY` and an optional `ANTHROPIC_API_KEY` (for the semantic
+  judge) with explanatory comments. The scaffolded `suites/README.md` has a
+  new "Setup" section walking through local key configuration.
+- New "Rerun semantics" section in `AGENTS.md` (referenced from README and
+  `docs/ci-integration.md`) covering what every subcommand does on the
+  second invocation: `record` overwrites baselines in place, `check`
+  accumulates a timestamped directory under `.agentprdiff/runs/` on every
+  call (gitignored; `rm -rf` to clean), `--json-out PATH` overwrites a
+  single file, and `scaffold`/`init` refuse-to-overwrite / are idempotent.
+  Adopting AI agents are now instructed to surface the `runs/` accumulation
+  behavior to the human user during handoff — it's the only one of the
+  four that creates new files on every invocation.
+
+## [0.2.1] — 2026-04-26
+
+### Changed
+
+- README links to `AGENTS.md`, `docs/adapters.md`, `docs/ai-driven-adoption.md`,
+  `docs/suite-layout.md`, `docs/ci-integration.md`, `LICENSE`, and `CHANGELOG.md`
+  are now absolute GitHub URLs instead of relative paths. PyPI's project page
+  and Libraries.io render the README but cannot resolve relative repo paths;
+  the previous links rendered as broken from those surfaces. Absolute URLs
+  fix the click-through from PyPI / Libraries.io directly to the docs on
+  GitHub.
+- Status section refreshed: 0.2.x is the current alpha line; OpenAI and
+  Anthropic SDK adapters are now shipped (previously listed as 0.2 roadmap).
+  LangChain/LangGraph adapters and the JS companion package moved to the
+  0.3 roadmap.
+
+### Fixed
+
+- Wheel metadata now emits a separate `Author:` header in addition to
+  `Author-email:`. Previously, the PEP 621 `authors = [{ name = ..., email = ... }]`
+  form generated only `Author-email: "Name" <email>`, which downstream
+  parsers like pypistats.org could not read (showing "Author: None").
+  Splitting `authors` into a name-only entry plus an email-only entry,
+  and adding a parallel `maintainers` field, makes the package author
+  display correctly across PyPI, Libraries.io, and pypistats.
+
+## [0.2.0] — 2026-04-26
+
+### Added
+
+- **SDK adapters** for the two dominant agent toolchains, eliminating the need
+  for manual `Trace` instrumentation:
+  - `agentprdiff.adapters.openai.instrument_client` — context manager that
+    monkey-patches `client.chat.completions.create` for the duration of one
+    agent call. Records each invocation as an `LLMCall` (provider, model,
+    input messages, output text, tool calls, tokens, cost, latency) and
+    restores the original on exit. Works with **OpenAI, Groq, Gemini's
+    OpenAI-compatible endpoint, OpenRouter, Ollama, vLLM, Together,
+    Fireworks, DeepInfra**, and any other SDK that follows the OpenAI client
+    shape.
+  - `agentprdiff.adapters.anthropic.instrument_client` — equivalent for the
+    Anthropic Messages API (`client.messages.create`). Handles the
+    content-block response shape (text + `tool_use` blocks) and the
+    Messages-API token field names.
+  - `instrument_tools(tool_map, trace)` — wraps a dict of callables so each
+    invocation records a `ToolCall` with name, arguments, result, latency,
+    and any raised exception. Shared between both adapters.
+  - `agentprdiff.adapters.pricing` — curated model→price table for cost
+    estimation, with `register_prices()` and per-call `prices=` overrides.
+    Unknown models record `cost_usd=0.0` and emit a single `RuntimeWarning`
+    per process so missing pricing is loud rather than silent.
+- Documentation: `docs/adapters.md` (full reference) and
+  `docs/adapters-vercel.md` (manual integration recipe for the Vercel AI
+  SDK, which is JS-only and lives in a future companion package).
+- `AGENTS.md` at the repo root — an instruction set written for AI
+  coding agents (Claude Code, Cursor, Aider, etc.) that have been asked
+  to add `agentprdiff` to a codebase. Covers codebase discovery,
+  contract identification, wrap-the-agent recipes (OpenAI / Anthropic /
+  custom), stub patterns, suite scaffolding, baseline recording, CI
+  wiring, common pitfalls, and a validation checklist. Optimized for
+  AI-agent-driven adoption with copy-paste templates.
+- `docs/ai-driven-adoption.md` — human-facing companion to AGENTS.md.
+  Three prompt templates (minimum viable / recommended / contract-driven)
+  for adopters using Claude Code / Cursor / Aider, plus a sample
+  first-session transcript and tips for working with the AI agent
+  through the adoption flow.
+- `docs/suite-layout.md` — canonical reference for the suite directory
+  structure. Lists each file (`suites/<project>.py`, `_eval_agent.py`,
+  `_stubs.py`, baselines, CI workflow, etc.), classifies them as
+  mandatory / recommended / optional, and specifies what each must
+  and must not contain. Cross-referenced from AGENTS.md and the
+  validation checklist.
+
+### Changed
+
+- The suite loader now inserts the current working directory onto
+  `sys.path` in addition to the suite file's parent directory. Adopters
+  who run `agentprdiff record suites/foo.py` from their project root no
+  longer have to manually patch `sys.path` to import their own modules
+  (e.g. `from agent.agent import ...`, `from config import ...`).
+  Both insertions are reverted after the suite loads, so no path leakage
+  between runs.
+
+### Notes
+
+- The base `pip install agentprdiff` does **not** require the `openai` or
+  `anthropic` packages. The adapters operate on a client object's shape,
+  not on imported SDK modules — so installing only the SDKs you actually
+  use keeps the dependency footprint small. Optional extras are still
+  declared (`agentprdiff[openai]`, `agentprdiff[anthropic]`) for adopters
+  who prefer to pin the SDK version alongside agentprdiff itself.
+
+## [0.1.0] — 2026-04-22
+
+Initial public release.
+
+### Added
+
+- Core `Suite` / `Case` / `Trace` model for defining agent regression tests.
+- Deterministic graders: `contains`, `contains_any`, `regex_match`, `tool_called`,
+  `tool_sequence`, `output_length_lt`, `latency_lt_ms`, `cost_lt_usd`,
+  `no_tool_called`.
+- Semantic grader (`semantic`) with a pluggable `judge` callable and built-in
+  fake judge for CI environments without API keys.
+- Baseline store (JSON files under `.agentprdiff/baselines/`) designed to be
+  committed to version control.
+- Trace diff engine producing a structured `TraceDelta` (assertion pass/fail
+  changes, cost delta, latency delta, tool-call sequence changes, output
+  change).
+- CLI: `agentprdiff init`, `agentprdiff record`, `agentprdiff check`, `agentprdiff diff`.
+- Rich-formatted terminal reporter and machine-readable JSON reporter for CI.
+- Quickstart example with a mock agent that runs without any API keys.
+- Pytest test suite covering graders, runner, differ, store, and CLI smoke.
+- GitHub Actions CI workflow.
+
+### Known limitations
+
+- Only a manual instrumentation API for provider SDKs is shipped in 0.1.0.
+  Drop-in wrappers for OpenAI / Anthropic / Vercel AI SDK are planned for 0.2.
+- The semantic grader's built-in judge supports OpenAI and Anthropic via user-
+  supplied API keys; hosted judge endpoints are not yet offered.

{agentprdiff-0.2.1 → agentprdiff-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentprdiff
-Version: 0.2.1
+Version: 0.2.2
 Summary: Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change.
 Project-URL: Homepage, https://github.com/vnageshwaran-de/agentprdiff
 Project-URL: Documentation, https://github.com/vnageshwaran-de/agentprdiff#readme
@@ -94,7 +94,7 @@ agentprdiff record suite.py     # save this run as the baseline
 agentprdiff check  suite.py     # in CI: diff vs baseline, exit 1 on regression
 ```
 
-That's the whole product. Four CLI commands. One Python file. Zero framework lock-in.
+That's the whole product. Five CLI commands (`init`, `record`, `check`, `review`, `scaffold`). One Python file. Zero framework lock-in.
 
 ## What's in the box
 
@@ -104,7 +104,7 @@ That's the whole product. Four CLI commands. One Python file. Zero framework loc
 - **Diff engine** — per-case `TraceDelta` with assertion pass/fail changes, cost delta, latency delta, tool-sequence changes, and a unified output diff.
 - **CI-ready CLI** — exit 1 on regression, `--json-out` for artifact archiving, Rich-formatted terminal output.
 - **Zero SDK lock-in** — works with OpenAI, Anthropic, Gemini, Bedrock, LangChain, LangGraph, LlamaIndex, Vercel AI SDK, custom wrappers — if you can wrap your agent in a function, `agentprdiff` can test it.
-- **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (or any OpenAI-compatible provider — Groq / Gemini / OpenRouter / Ollama / vLLM) or the Anthropic SDK. No manual `Trace` wiring required.
+- **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (sync **or** async — `AsyncOpenAI` is supported by the same context manager) or any OpenAI-compatible provider (Groq / Gemini / OpenRouter / Ollama / vLLM / Together / Fireworks / DeepInfra) or the Anthropic SDK. No manual `Trace` wiring required.
 
 ## How it compares
 
@@ -127,13 +127,49 @@ The value is in the combination: deterministic assertions for the 80% of behavio
 
 This is the same loop as Jest snapshot tests or VCR cassettes — applied to LLM agents.
 
+### API keys
+
+`agentprdiff` doesn't read your agent's API key — your agent does, through whatever env var it already uses. Set that locally (in `.env`, your shell, direnv, whatever) and as a GitHub Actions secret in CI. The scaffold's workflow YAML has the right shape; you fill in the env var name to match your agent.
+
+The `semantic()` grader is the one piece of agentprdiff that can use an API key directly — for the LLM judge. Without one, it silently falls back to keyword matching. Set `ANTHROPIC_API_KEY` (cheaper) or `OPENAI_API_KEY` if you want a real judge in CI; leave both unset to keep CI free with fake_judge.
+
+See [AGENTS.md → API keys](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md#api-keys--what-to-set-where-and-how-to-ask-the-user-about-them) for the full setup (local options, CI secrets, what never to do).
+
+### What each command does on rerun
+
+A common first-day question. Short version:
+
+- `record` — overwrites baselines in place. Re-recording an intentional change shows up as a regular git diff in your PR; that's the review surface.
+- `check` — creates a new timestamped directory under `.agentprdiff/runs/` on every invocation. It's gitignored by default, so it never reaches CI; clean local history any time with `rm -rf .agentprdiff/runs/`. `--json-out PATH` overwrites a single file at PATH.
+- `review` — same comparison as `check`, but renders one verbose panel per case and **always exits 0**. Designed for local iteration loops; not meant for CI. Writes to the same `.agentprdiff/runs/` directory.
+- `scaffold` — never overwrites. Skips files that already exist (`[skip]`) and writes the rest.
+- `init` — idempotent; running it twice does nothing the second time.
+
+See [AGENTS.md → Rerun semantics](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md#rerun-semantics--what-each-command-does-on-the-second-run) for examples.
+
+### Scaffolding a new suite
+
+Skip the copy-paste from [AGENTS.md](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md):
+
+```bash
+agentprdiff scaffold ai_content_summary --recipe sync-openai
+```
+
+Writes the canonical layout (`suites/__init__.py`, `_eval_agent.py`, `_stubs.py`, `<name>.py`, `<name>_cases.md`, `suites/README.md`, and `.github/workflows/agentprdiff.yml`) with TODO markers where you wire in your agent. The `<name>_cases.md` file is a *case dossier* — reviewer-facing prose with one block per case (what it tests, input, assertions in plain English, file:line references to production code, and the application impact if the case regresses). Three recipes:
+
+- `sync-openai` (default): uses `instrument_client` from the OpenAI adapter with a sync `OpenAI()` client.
+- `async-openai`: same `instrument_client`, paired with an `asyncio.run` bridge so an `AsyncOpenAI` agent works with agentprdiff's sync runner. The adapter detects the async client at entry — no separate API.
+- `stubbed`: substitutes a single LLM helper instead of the SDK client. Best for summarization / classification / embedding-prep agents — see [`docs/adapters.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/adapters.md#stubbed-llm-boundary-pattern).
+
+The generated workflow includes `permissions: contents: read` so GHAS doesn't flag it. Pre-existing files are never overwritten.
+
 ## Instrumenting your agent
 
 You have two paths. Most agents need the first.
 
 ### Option A — SDK adapters (zero manual work)
 
-If your agent uses the OpenAI Python SDK (or any OpenAI-compatible provider — Groq, Gemini, OpenRouter, Ollama, vLLM, Together, Fireworks, DeepInfra) or the Anthropic SDK, the SDK adapter captures every model and tool call automatically:
+If your agent uses the OpenAI Python SDK — sync `OpenAI` **or** async `AsyncOpenAI`, including any OpenAI-compatible provider (Groq, Gemini, OpenRouter, Ollama, vLLM, Together, Fireworks, DeepInfra) — or the Anthropic SDK, the SDK adapter captures every model and tool call automatically:
 
 ```python
 from openai import OpenAI
@@ -150,7 +186,26 @@ def my_agent(query: str):
         return final_text, trace
 ```
 
-The patch is scoped to the specific client instance and reversed when the `with` block exits — no global SDK state is touched. Anthropic adopters use `agentprdiff.adapters.anthropic` with the same shape.
+For `AsyncOpenAI`, the same `instrument_client` works — it inspects `client.chat.completions.create` at entry and installs an awaitable patched method when the underlying one is `async def`. `instrument_tools` mirrors per-tool: `async def` tools come back awaitable, sync tools stay sync. The `with` block is still a regular `with`:
+
+```python
+import asyncio
+from openai import AsyncOpenAI
+from agentprdiff.adapters.openai import instrument_client, instrument_tools
+
+async def my_agent_async(query: str):
+    client = AsyncOpenAI()
+    with instrument_client(client) as trace:
+        tools = instrument_tools(TOOL_MAP, trace)
+        response = await client.chat.completions.create(...)
+        # ... await tools[name](**args) for async tools, tools[name](**args) for sync ...
+        return final_text, trace
+
+def my_agent(query: str):
+    return asyncio.run(my_agent_async(query))
+```
+
+The patch is scoped to the specific client instance and reversed when the `with` block exits — no global SDK state is touched. Anthropic adopters use `agentprdiff.adapters.anthropic` with the same shape (sync clients today; async Anthropic is on the roadmap).
 
 See [`docs/adapters.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/adapters.md) for the full reference, including pricing overrides, custom provider tags, and recipes for nested agents.
 
@@ -186,6 +241,8 @@ Agents that return just an output still work — `agentprdiff` wraps them and ca
 # .github/workflows/agents.yml
 name: agent-regression
 on: [pull_request]
+permissions:
+  contents: read   # least-privilege; GHAS flags workflows without this.
 jobs:
   agentprdiff:
     runs-on: ubuntu-latest
@@ -200,6 +257,8 @@ jobs:
         with: { name: agentprdiff, path: artifacts/ }
 ```
 
+If you use `--json-out artifacts/...`, add `artifacts/agentprdiff*.json` (or the broader `artifacts/`) to your project's `.gitignore` — the CI artifact upload doesn't prevent a contributor from accidentally `git add`ing it locally.
+
 See [`docs/ci-integration.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/ci-integration.md) for GitLab, CircleCI, and Buildkite.
 
 ## Quickstart
@@ -221,9 +280,53 @@ sed -i "s/refund/noundr/g" agent.py
 agentprdiff check suite.py    # exit 1; see the diff
 ```
 
+### Running a subset of cases
+
+Iterating on a single failing case shouldn't require commenting out the rest. `record`, `check`, and `review` all accept `--case` and `--skip` for narrowing a run:
+
+```bash
+# Discover what's available.
+agentprdiff check suite.py --list
+
+# Single case (case-insensitive substring).
+agentprdiff check suite.py --case refund_happy_path
+
+# Glob across cases.
+agentprdiff check suite.py --case "*order*"
+
+# Multiple patterns (repeated flag or comma-separated).
+agentprdiff check suite.py --case refund --case policy
+agentprdiff check suite.py --case refund,policy
+
+# Everything except slow cases.
+agentprdiff check suite.py --skip slow
+agentprdiff check suite.py --case ~slow         # equivalent
+
+# Qualify by suite when names collide across suites.
+agentprdiff check suite.py --case "billing:refund*"
+```
+
+A filter that matches zero cases exits 2 and prints the available case names — `--list` is the discoverable counterpart. The selection summary (`running 2 of 4 cases in <suite>: ...`) is printed before each suite runs so a partial match is never silent.
+
+### Reviewing one case (the local-iteration loop)
+
+`agentprdiff check` is built for CI: a compact summary table and exit 1 on regression. While you're iterating on a single case, that's the wrong shape — you want to see *everything* about that one case, and you don't want your shell going red between every keystroke. That's `agentprdiff review`:
+
+```bash
+# Verbose per-case panel: input, every assertion's was→now verdict,
+# cost/latency/token deltas, tool-sequence diff, output diff.
+agentprdiff review suite.py --case refund_happy_path
+
+# Same filter syntax as check / record — globs, negation, multi-pattern.
+agentprdiff review suite.py --case "*refund*"
+agentprdiff review suite.py --skip slow
+```
+
+`review` runs the same comparison `check` does (and writes to the same `.agentprdiff/runs/` directory) but **always exits 0**, even on regression — so it sits cleanly inside watcher loops (`entr`, `watchexec`, `fzf` previews). Use `check` when you want CI's exit semantics locally; reach for `review` while you're working. Think `pytest -k`.
+
 ## Status
 
-`agentprdiff` is **alpha** (0.2.x). The core model, CLI, and OpenAI / Anthropic SDK adapters are stable. LangChain/LangGraph adapters and a JS companion package for the Vercel AI SDK are on the 0.3 roadmap. See [`CHANGELOG.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/CHANGELOG.md).
+`agentprdiff` is **alpha** (0.2.x). The core model, CLI, and OpenAI / Anthropic SDK adapters are stable. The OpenAI adapter covers both sync `OpenAI` and async `AsyncOpenAI` clients via the same `instrument_client` context manager. Async Anthropic, LangChain/LangGraph adapters, and a JS companion package for the Vercel AI SDK are on the 0.3 roadmap. See [`CHANGELOG.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/CHANGELOG.md).
 
 Feedback, bug reports, and PRs extremely welcome. Open an issue or @ me.
 

{agentprdiff-0.2.1 → agentprdiff-0.2.2}/README.md

@@ -55,7 +55,7 @@ agentprdiff record suite.py     # save this run as the baseline
 agentprdiff check  suite.py     # in CI: diff vs baseline, exit 1 on regression
 ```
 
-That's the whole product. Four CLI commands. One Python file. Zero framework lock-in.
+That's the whole product. Five CLI commands (`init`, `record`, `check`, `review`, `scaffold`). One Python file. Zero framework lock-in.
 
 ## What's in the box
 
@@ -65,7 +65,7 @@ That's the whole product. Four CLI commands. One Python file. Zero framework loc
 - **Diff engine** — per-case `TraceDelta` with assertion pass/fail changes, cost delta, latency delta, tool-sequence changes, and a unified output diff.
 - **CI-ready CLI** — exit 1 on regression, `--json-out` for artifact archiving, Rich-formatted terminal output.
 - **Zero SDK lock-in** — works with OpenAI, Anthropic, Gemini, Bedrock, LangChain, LangGraph, LlamaIndex, Vercel AI SDK, custom wrappers — if you can wrap your agent in a function, `agentprdiff` can test it.
-- **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (or any OpenAI-compatible provider — Groq / Gemini / OpenRouter / Ollama / vLLM) or the Anthropic SDK. No manual `Trace` wiring required.
+- **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (sync **or** async — `AsyncOpenAI` is supported by the same context manager) or any OpenAI-compatible provider (Groq / Gemini / OpenRouter / Ollama / vLLM / Together / Fireworks / DeepInfra) or the Anthropic SDK. No manual `Trace` wiring required.
 
 ## How it compares
 
@@ -88,13 +88,49 @@ The value is in the combination: deterministic assertions for the 80% of behavio
 
 This is the same loop as Jest snapshot tests or VCR cassettes — applied to LLM agents.
 
+### API keys
+
+`agentprdiff` doesn't read your agent's API key — your agent does, through whatever env var it already uses. Set that locally (in `.env`, your shell, direnv, whatever) and as a GitHub Actions secret in CI. The scaffold's workflow YAML has the right shape; you fill in the env var name to match your agent.
+
+The `semantic()` grader is the one piece of agentprdiff that can use an API key directly — for the LLM judge. Without one, it silently falls back to keyword matching. Set `ANTHROPIC_API_KEY` (cheaper) or `OPENAI_API_KEY` if you want a real judge in CI; leave both unset to keep CI free with fake_judge.
+
+See [AGENTS.md → API keys](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md#api-keys--what-to-set-where-and-how-to-ask-the-user-about-them) for the full setup (local options, CI secrets, what never to do).
+
+### What each command does on rerun
+
+A common first-day question. Short version:
+
+- `record` — overwrites baselines in place. Re-recording an intentional change shows up as a regular git diff in your PR; that's the review surface.
+- `check` — creates a new timestamped directory under `.agentprdiff/runs/` on every invocation. It's gitignored by default, so it never reaches CI; clean local history any time with `rm -rf .agentprdiff/runs/`. `--json-out PATH` overwrites a single file at PATH.
+- `review` — same comparison as `check`, but renders one verbose panel per case and **always exits 0**. Designed for local iteration loops; not meant for CI. Writes to the same `.agentprdiff/runs/` directory.
+- `scaffold` — never overwrites. Skips files that already exist (`[skip]`) and writes the rest.
+- `init` — idempotent; running it twice does nothing the second time.
+
+See [AGENTS.md → Rerun semantics](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md#rerun-semantics--what-each-command-does-on-the-second-run) for examples.
+
+### Scaffolding a new suite
+
+Skip the copy-paste from [AGENTS.md](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md):
+
+```bash
+agentprdiff scaffold ai_content_summary --recipe sync-openai
+```
+
+Writes the canonical layout (`suites/__init__.py`, `_eval_agent.py`, `_stubs.py`, `<name>.py`, `<name>_cases.md`, `suites/README.md`, and `.github/workflows/agentprdiff.yml`) with TODO markers where you wire in your agent. The `<name>_cases.md` file is a *case dossier* — reviewer-facing prose with one block per case (what it tests, input, assertions in plain English, file:line references to production code, and the application impact if the case regresses). Three recipes:
+
+- `sync-openai` (default): uses `instrument_client` from the OpenAI adapter with a sync `OpenAI()` client.
+- `async-openai`: same `instrument_client`, paired with an `asyncio.run` bridge so an `AsyncOpenAI` agent works with agentprdiff's sync runner. The adapter detects the async client at entry — no separate API.
+- `stubbed`: substitutes a single LLM helper instead of the SDK client. Best for summarization / classification / embedding-prep agents — see [`docs/adapters.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/adapters.md#stubbed-llm-boundary-pattern).
+
+The generated workflow includes `permissions: contents: read` so GHAS doesn't flag it. Pre-existing files are never overwritten.
+
 ## Instrumenting your agent
 
 You have two paths. Most agents need the first.
 
 ### Option A — SDK adapters (zero manual work)
 
-If your agent uses the OpenAI Python SDK (or any OpenAI-compatible provider — Groq, Gemini, OpenRouter, Ollama, vLLM, Together, Fireworks, DeepInfra) or the Anthropic SDK, the SDK adapter captures every model and tool call automatically:
+If your agent uses the OpenAI Python SDK — sync `OpenAI` **or** async `AsyncOpenAI`, including any OpenAI-compatible provider (Groq, Gemini, OpenRouter, Ollama, vLLM, Together, Fireworks, DeepInfra) — or the Anthropic SDK, the SDK adapter captures every model and tool call automatically:
 
 ```python
 from openai import OpenAI
@@ -111,7 +147,26 @@ def my_agent(query: str):
         return final_text, trace
 ```
 
-The patch is scoped to the specific client instance and reversed when the `with` block exits — no global SDK state is touched. Anthropic adopters use `agentprdiff.adapters.anthropic` with the same shape.
+For `AsyncOpenAI`, the same `instrument_client` works — it inspects `client.chat.completions.create` at entry and installs an awaitable patched method when the underlying one is `async def`. `instrument_tools` mirrors per-tool: `async def` tools come back awaitable, sync tools stay sync. The `with` block is still a regular `with`:
+
+```python
+import asyncio
+from openai import AsyncOpenAI
+from agentprdiff.adapters.openai import instrument_client, instrument_tools
+
+async def my_agent_async(query: str):
+    client = AsyncOpenAI()
+    with instrument_client(client) as trace:
+        tools = instrument_tools(TOOL_MAP, trace)
+        response = await client.chat.completions.create(...)
+        # ... await tools[name](**args) for async tools, tools[name](**args) for sync ...
+        return final_text, trace
+
+def my_agent(query: str):
+    return asyncio.run(my_agent_async(query))
+```
+
+The patch is scoped to the specific client instance and reversed when the `with` block exits — no global SDK state is touched. Anthropic adopters use `agentprdiff.adapters.anthropic` with the same shape (sync clients today; async Anthropic is on the roadmap).
 
 See [`docs/adapters.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/adapters.md) for the full reference, including pricing overrides, custom provider tags, and recipes for nested agents.
 
@@ -147,6 +202,8 @@ Agents that return just an output still work — `agentprdiff` wraps them and ca
 # .github/workflows/agents.yml
 name: agent-regression
 on: [pull_request]
+permissions:
+  contents: read   # least-privilege; GHAS flags workflows without this.
 jobs:
   agentprdiff:
     runs-on: ubuntu-latest
@@ -161,6 +218,8 @@ jobs:
         with: { name: agentprdiff, path: artifacts/ }
 ```
 
+If you use `--json-out artifacts/...`, add `artifacts/agentprdiff*.json` (or the broader `artifacts/`) to your project's `.gitignore` — the CI artifact upload doesn't prevent a contributor from accidentally `git add`ing it locally.
+
 See [`docs/ci-integration.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/ci-integration.md) for GitLab, CircleCI, and Buildkite.
 
 ## Quickstart
@@ -182,9 +241,53 @@ sed -i "s/refund/noundr/g" agent.py
 agentprdiff check suite.py    # exit 1; see the diff
 ```
 
+### Running a subset of cases
+
+Iterating on a single failing case shouldn't require commenting out the rest. `record`, `check`, and `review` all accept `--case` and `--skip` for narrowing a run:
+
+```bash
+# Discover what's available.
+agentprdiff check suite.py --list
+
+# Single case (case-insensitive substring).
+agentprdiff check suite.py --case refund_happy_path
+
+# Glob across cases.
+agentprdiff check suite.py --case "*order*"
+
+# Multiple patterns (repeated flag or comma-separated).
+agentprdiff check suite.py --case refund --case policy
+agentprdiff check suite.py --case refund,policy
+
+# Everything except slow cases.
+agentprdiff check suite.py --skip slow
+agentprdiff check suite.py --case ~slow         # equivalent
+
+# Qualify by suite when names collide across suites.
+agentprdiff check suite.py --case "billing:refund*"
+```
+
+A filter that matches zero cases exits 2 and prints the available case names — `--list` is the discoverable counterpart. The selection summary (`running 2 of 4 cases in <suite>: ...`) is printed before each suite runs so a partial match is never silent.
+
+### Reviewing one case (the local-iteration loop)
+
+`agentprdiff check` is built for CI: a compact summary table and exit 1 on regression. While you're iterating on a single case, that's the wrong shape — you want to see *everything* about that one case, and you don't want your shell going red between every keystroke. That's `agentprdiff review`:
+
+```bash
+# Verbose per-case panel: input, every assertion's was→now verdict,
+# cost/latency/token deltas, tool-sequence diff, output diff.
+agentprdiff review suite.py --case refund_happy_path
+
+# Same filter syntax as check / record — globs, negation, multi-pattern.
+agentprdiff review suite.py --case "*refund*"
+agentprdiff review suite.py --skip slow
+```
+
+`review` runs the same comparison `check` does (and writes to the same `.agentprdiff/runs/` directory) but **always exits 0**, even on regression — so it sits cleanly inside watcher loops (`entr`, `watchexec`, `fzf` previews). Use `check` when you want CI's exit semantics locally; reach for `review` while you're working. Think `pytest -k`.
+
 ## Status
 
-`agentprdiff` is **alpha** (0.2.x). The core model, CLI, and OpenAI / Anthropic SDK adapters are stable. LangChain/LangGraph adapters and a JS companion package for the Vercel AI SDK are on the 0.3 roadmap. See [`CHANGELOG.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/CHANGELOG.md).
+`agentprdiff` is **alpha** (0.2.x). The core model, CLI, and OpenAI / Anthropic SDK adapters are stable. The OpenAI adapter covers both sync `OpenAI` and async `AsyncOpenAI` clients via the same `instrument_client` context manager. Async Anthropic, LangChain/LangGraph adapters, and a JS companion package for the Vercel AI SDK are on the 0.3 roadmap. See [`CHANGELOG.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/CHANGELOG.md).
 
 Feedback, bug reports, and PRs extremely welcome. Open an issue or @ me.
 

{agentprdiff-0.2.1 → agentprdiff-0.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agentprdiff"
-version = "0.2.1"
+version = "0.2.2"
 description = "Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change."
 readme = "README.md"
 requires-python = ">=3.10"

{agentprdiff-0.2.1 → agentprdiff-0.2.2}/src/agentprdiff/__init__.py

@@ -67,7 +67,7 @@ from .differ import AssertionChange, TraceDelta, diff_traces
 from .runner import CaseReport, Runner, RunReport
 from .store import BaselineStore
 
-__version__ = "0.2.1"
+__version__ = "0.2.2"
 
 __all__ = [
     # core