agentprdiff 0.2.1__tar.gz → 0.2.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- agentprdiff-0.2.3/CHANGELOG.md +280 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/PKG-INFO +109 -6
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/README.md +108 -5
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/pyproject.toml +1 -1
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/__init__.py +1 -1
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/adapters/openai.py +273 -73
- agentprdiff-0.2.3/src/agentprdiff/cli.py +358 -0
- agentprdiff-0.2.3/src/agentprdiff/filtering.py +125 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/graders/semantic.py +53 -0
- agentprdiff-0.2.3/src/agentprdiff/reporters.py +461 -0
- agentprdiff-0.2.3/src/agentprdiff/scaffold.py +679 -0
- agentprdiff-0.2.1/CHANGELOG.md +0 -129
- agentprdiff-0.2.1/src/agentprdiff/cli.py +0 -124
- agentprdiff-0.2.1/src/agentprdiff/reporters.py +0 -127
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/.gitignore +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/LICENSE +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/examples/quickstart/README.md +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/examples/regression-tour/README.md +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/adapters/__init__.py +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/adapters/anthropic.py +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/adapters/pricing.py +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/core.py +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/differ.py +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/graders/__init__.py +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/graders/deterministic.py +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/loader.py +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/runner.py +0 -0
- {agentprdiff-0.2.1 → agentprdiff-0.2.3}/src/agentprdiff/store.py +0 -0
|
@@ -0,0 +1,280 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to `agentprdiff` are documented in this file. Originally
|
|
4
|
+
prototyped under the name `tracediff`; renamed before first public release.
|
|
5
|
+
|
|
6
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
7
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
8
|
+
|
|
9
|
+
## [Unreleased]
|
|
10
|
+
|
|
11
|
+
## [0.2.3] — 2026-04-28
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- **Semantic-judge banner in `check` and `review` output.** `TerminalReporter`
|
|
16
|
+
and `ReviewReporter` now print one line — `semantic judge: <description>`
|
|
17
|
+
— directly under the header whenever the suite contains at least one
|
|
18
|
+
`semantic(...)` grader. The description names the active backend
|
|
19
|
+
(`fake_judge`, `openai/<model>`, or `anthropic/<model>`) and the env-var
|
|
20
|
+
signal that selected it, with explicit `silent fallback` wording when
|
|
21
|
+
no judge is configured. Closes the most common adoption trap: shipping
|
|
22
|
+
suites whose semantic coverage is decorative because no key was set
|
|
23
|
+
and the runner stayed quiet about it. Suites without `semantic(...)`
|
|
24
|
+
are unaffected — no banner is printed. New helpers
|
|
25
|
+
`agentprdiff.graders.semantic.describe_default_judge()` and
|
|
26
|
+
`case_uses_semantic()` power the rendering and are reusable by
|
|
27
|
+
third-party tooling.
|
|
28
|
+
- **Scaffolded workflow YAML now flags judge-SDK installs explicitly.**
|
|
29
|
+
`_TPL_WORKFLOW` ships commented `pip install anthropic` /
|
|
30
|
+
`pip install openai` lines tied to the chosen `AGENTGUARD_JUDGE` mode,
|
|
31
|
+
with guidance that a missing SDK raises `ImportError` rather than
|
|
32
|
+
falling back silently. Pairs with the in-band judge banner for
|
|
33
|
+
end-to-end coverage of the silent-fake_judge trap.
|
|
34
|
+
|
|
35
|
+
### Fixed
|
|
36
|
+
|
|
37
|
+
- Adoption checklist in `AGENTS.md` now requires the adopter to verify
|
|
38
|
+
the installed CLI supports the documented commands (`agentprdiff
|
|
39
|
+
check --help`) before writing run commands into the case dossier —
|
|
40
|
+
prevents the "source docs reference `--case` but the pinned wheel
|
|
41
|
+
predates it" confusion reported during 0.2.x adoption.
|
|
42
|
+
- New `Step 5b — decide and document the semantic-judge mode` mandates
|
|
43
|
+
a `## Semantic Judge Keys` section in `suites/README.md` and an
|
|
44
|
+
explicit `AGENTGUARD_JUDGE=<mode>` line in the workflow YAML, replacing
|
|
45
|
+
the implicit "first available key wins" precedence with a deliberate
|
|
46
|
+
declaration.
|
|
47
|
+
|
|
48
|
+
## [0.2.2] — 2026-04-28
|
|
49
|
+
|
|
50
|
+
### Added
|
|
51
|
+
|
|
52
|
+
- **Native `AsyncOpenAI` support in `agentprdiff.adapters.openai`.** The same
|
|
53
|
+
`instrument_client` and `instrument_tools` API now works with the async
|
|
54
|
+
OpenAI client — and any async OpenAI-compatible provider (Groq, Gemini,
|
|
55
|
+
OpenRouter, Ollama, vLLM, Together, Fireworks, DeepInfra). The adapter
|
|
56
|
+
inspects `client.chat.completions.create` at `with`-block entry; if it's
|
|
57
|
+
a coroutine function, an awaitable patched method is installed and the
|
|
58
|
+
user's `await client.chat.completions.create(...)` call sites work
|
|
59
|
+
unchanged. `instrument_tools` matches per-tool: `async def` tools come
|
|
60
|
+
back as `async def` wrappers (`await tools[name](**args)`), sync tools
|
|
61
|
+
stay sync — a single `TOOL_MAP` may freely mix the two. The `with` block
|
|
62
|
+
is a regular `with`, not `async with`, since the patch is bound to the
|
|
63
|
+
client instance rather than the running event loop. agentprdiff's runner
|
|
64
|
+
remains sync; async agents bridge with `asyncio.run` in their
|
|
65
|
+
`eval_agent` entry point. Removes the previous adoption recommendation
|
|
66
|
+
to use manual `Trace.record_llm_call` instrumentation for async agents.
|
|
67
|
+
- **Updated `--recipe async-openai` scaffold** to use the new adapter
|
|
68
|
+
natively. The generated `_eval_agent.py` no longer carries TODO markers
|
|
69
|
+
for manual instrumentation; it imports `instrument_client` /
|
|
70
|
+
`instrument_tools` and wraps an async tool-calling loop with an
|
|
71
|
+
`asyncio.run` bridge to agentprdiff's sync runner.
|
|
72
|
+
- **`agentprdiff review <suite_file>`** — new subcommand for local iteration
|
|
73
|
+
on a single failing case. Runs the same comparison `check` does, but
|
|
74
|
+
renders one verbose panel per case (input echo, full assertion table with
|
|
75
|
+
`was → now` baseline-vs-current marks, per-metric deltas for cost /
|
|
76
|
+
latency / prompt and completion tokens, tool-sequence diff, and a unified
|
|
77
|
+
output diff in its own panel when output changed) and **always exits 0**
|
|
78
|
+
so it can sit inside watcher / `entr` / `fzf` loops without flipping the
|
|
79
|
+
shell red on every regression. Accepts the same `--case` / `--skip` /
|
|
80
|
+
`--list` flags as `record` and `check`. The CI gate stays `agentprdiff
|
|
81
|
+
check`; `review` is the `pytest -k` of agentprdiff. New `ReviewReporter`
|
|
82
|
+
in `agentprdiff.reporters` powers the rendering and is reusable by
|
|
83
|
+
third-party tooling.
|
|
84
|
+
- **`agentprdiff scaffold <name>`** — new subcommand that stamps out the
|
|
85
|
+
canonical adoption layout (`suites/__init__.py`, `_eval_agent.py`,
|
|
86
|
+
`_stubs.py`, `<name>.py`, `<name>_cases.md`, `suites/README.md`, and
|
|
87
|
+
`.github/workflows/agentprdiff.yml`). Three recipes via `--recipe`:
|
|
88
|
+
`sync-openai` (default; uses `instrument_client`), `async-openai` (manual
|
|
89
|
+
asyncio wrapper, until the async adapter ships in 0.3), and `stubbed`
|
|
90
|
+
(substitutes a single LLM helper — see the new "stubbed LLM-boundary
|
|
91
|
+
pattern" recipe in `docs/adapters.md`). The generated workflow includes
|
|
92
|
+
`permissions: contents: read` so GitHub Advanced Security stops flagging
|
|
93
|
+
it. Pre-existing files are never overwritten — they're reported as
|
|
94
|
+
`[skip]` and the rest are still written.
|
|
95
|
+
- **Case dossier** (`suites/<name>_cases.md`) — new mandatory artifact
|
|
96
|
+
produced by `scaffold` and documented in AGENTS.md and
|
|
97
|
+
`docs/suite-layout.md`. Reviewer-facing markdown with one block per case
|
|
98
|
+
using a fixed five-field structure: *What it tests*, *Input*,
|
|
99
|
+
*Assertions* (plain English), *Code impacted* (file:line references back
|
|
100
|
+
to production code), and *Application impact* (one concrete sentence
|
|
101
|
+
about what breaks for end users on regression). Closes the gap between
|
|
102
|
+
case names that look meaningful in CI output ("article_summary_preserves_acquisition_entities")
|
|
103
|
+
and reviewers who need to know what each case actually pins.
|
|
104
|
+
- New "stubbed LLM-boundary pattern" recipe in `docs/adapters.md` for
|
|
105
|
+
agents whose LLM call is wrapped in a single helper (summarization,
|
|
106
|
+
classification, embedding-prep). Stubbing the helper is cleaner than
|
|
107
|
+
stubbing the SDK client and works equally well for sync and async clients.
|
|
108
|
+
- `agentprdiff record` and `agentprdiff check` now accept `--case PATTERN` and
|
|
109
|
+
`--skip PATTERN` for narrowing a run to a subset of cases. Patterns are
|
|
110
|
+
case-insensitive substrings by default and use `fnmatch` semantics when they
|
|
111
|
+
contain `*`, `?`, or `[`. Both flags are repeatable, accept comma-separated
|
|
112
|
+
lists (`--case refund,policy`), and support qualifier syntax
|
|
113
|
+
(`--case billing:refund*`). A leading `~` (or `!`) negates a pattern, so
|
|
114
|
+
`--case ~slow` is equivalent to `--skip slow`.
|
|
115
|
+
- `agentprdiff record --list` / `check --list` prints suite and case names
|
|
116
|
+
without running anything, so you can discover what's filterable before
|
|
117
|
+
reaching for `--case`.
|
|
118
|
+
- When a filter is active, the CLI now prints a per-suite header
|
|
119
|
+
(`running 2 of 4 cases in customer_support: ...`) so a partial selection is
|
|
120
|
+
visible at a glance. A filter that matches zero cases exits with code 2 and
|
|
121
|
+
prints the available case names — previously a typo'd filter would have
|
|
122
|
+
silently exited 0.
|
|
123
|
+
|
|
124
|
+
### Changed
|
|
125
|
+
|
|
126
|
+
- The CI workflow templates in `AGENTS.md`, `README.md`, and
|
|
127
|
+
`docs/ci-integration.md` now declare `permissions: contents: read`
|
|
128
|
+
explicitly. GitHub Advanced Security flags workflows without an explicit
|
|
129
|
+
permissions block, and least-privilege is the right default anyway.
|
|
130
|
+
- Documented the recommended `.gitignore` entry (`artifacts/agentprdiff*.json`)
|
|
131
|
+
alongside every CI snippet that uses `--json-out artifacts/...`. The path
|
|
132
|
+
uploads cleanly as a CI build artifact, but the same file lands on every
|
|
133
|
+
local run; without an ignore line it eventually gets `git add`-ed by
|
|
134
|
+
accident.
|
|
135
|
+
|
|
136
|
+
### Documentation
|
|
137
|
+
|
|
138
|
+
- New "API keys" section in `AGENTS.md` covering both key surfaces (the
|
|
139
|
+
production agent's own keys vs. agentprdiff's semantic-judge keys —
|
|
140
|
+
`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `AGENTGUARD_JUDGE` with the
|
|
141
|
+
silent fake_judge fallback), local setup options (.env / shell export /
|
|
142
|
+
direnv), CI secret wiring, and a short list of "never do this." Adopting
|
|
143
|
+
AI agents are now instructed to explicitly prompt the user about which
|
|
144
|
+
env var their production agent reads, whether to use a real semantic
|
|
145
|
+
judge in CI, and to verify `.env` is gitignored.
|
|
146
|
+
- The scaffolded `.github/workflows/agentprdiff.yml` now includes both
|
|
147
|
+
`OPENAI_API_KEY` and an optional `ANTHROPIC_API_KEY` (for the semantic
|
|
148
|
+
judge) with explanatory comments. The scaffolded `suites/README.md` has a
|
|
149
|
+
new "Setup" section walking through local key configuration.
|
|
150
|
+
- New "Rerun semantics" section in `AGENTS.md` (referenced from README and
|
|
151
|
+
`docs/ci-integration.md`) covering what every subcommand does on the
|
|
152
|
+
second invocation: `record` overwrites baselines in place, `check`
|
|
153
|
+
accumulates a timestamped directory under `.agentprdiff/runs/` on every
|
|
154
|
+
call (gitignored; `rm -rf` to clean), `--json-out PATH` overwrites a
|
|
155
|
+
single file, and `scaffold`/`init` refuse-to-overwrite / are idempotent.
|
|
156
|
+
Adopting AI agents are now instructed to surface the `runs/` accumulation
|
|
157
|
+
behavior to the human user during handoff — it's the only one of the
|
|
158
|
+
four that creates new files on every invocation.
|
|
159
|
+
|
|
160
|
+
## [0.2.1] — 2026-04-26
|
|
161
|
+
|
|
162
|
+
### Changed
|
|
163
|
+
|
|
164
|
+
- README links to `AGENTS.md`, `docs/adapters.md`, `docs/ai-driven-adoption.md`,
|
|
165
|
+
`docs/suite-layout.md`, `docs/ci-integration.md`, `LICENSE`, and `CHANGELOG.md`
|
|
166
|
+
are now absolute GitHub URLs instead of relative paths. PyPI's project page
|
|
167
|
+
and Libraries.io render the README but cannot resolve relative repo paths;
|
|
168
|
+
the previous links rendered as broken from those surfaces. Absolute URLs
|
|
169
|
+
fix the click-through from PyPI / Libraries.io directly to the docs on
|
|
170
|
+
GitHub.
|
|
171
|
+
- Status section refreshed: 0.2.x is the current alpha line; OpenAI and
|
|
172
|
+
Anthropic SDK adapters are now shipped (previously listed as 0.2 roadmap).
|
|
173
|
+
LangChain/LangGraph adapters and the JS companion package moved to the
|
|
174
|
+
0.3 roadmap.
|
|
175
|
+
|
|
176
|
+
### Fixed
|
|
177
|
+
|
|
178
|
+
- Wheel metadata now emits a separate `Author:` header in addition to
|
|
179
|
+
`Author-email:`. Previously, the PEP 621 `authors = [{ name = ..., email = ... }]`
|
|
180
|
+
form generated only `Author-email: "Name" <email>`, which downstream
|
|
181
|
+
parsers like pypistats.org could not read (showing "Author: None").
|
|
182
|
+
Splitting `authors` into a name-only entry plus an email-only entry,
|
|
183
|
+
and adding a parallel `maintainers` field, makes the package author
|
|
184
|
+
display correctly across PyPI, Libraries.io, and pypistats.
|
|
185
|
+
|
|
186
|
+
## [0.2.0] — 2026-04-26
|
|
187
|
+
|
|
188
|
+
### Added
|
|
189
|
+
|
|
190
|
+
- **SDK adapters** for the two dominant agent toolchains, eliminating the need
|
|
191
|
+
for manual `Trace` instrumentation:
|
|
192
|
+
- `agentprdiff.adapters.openai.instrument_client` — context manager that
|
|
193
|
+
monkey-patches `client.chat.completions.create` for the duration of one
|
|
194
|
+
agent call. Records each invocation as an `LLMCall` (provider, model,
|
|
195
|
+
input messages, output text, tool calls, tokens, cost, latency) and
|
|
196
|
+
restores the original on exit. Works with **OpenAI, Groq, Gemini's
|
|
197
|
+
OpenAI-compatible endpoint, OpenRouter, Ollama, vLLM, Together,
|
|
198
|
+
Fireworks, DeepInfra**, and any other SDK that follows the OpenAI client
|
|
199
|
+
shape.
|
|
200
|
+
- `agentprdiff.adapters.anthropic.instrument_client` — equivalent for the
|
|
201
|
+
Anthropic Messages API (`client.messages.create`). Handles the
|
|
202
|
+
content-block response shape (text + `tool_use` blocks) and the
|
|
203
|
+
Messages-API token field names.
|
|
204
|
+
- `instrument_tools(tool_map, trace)` — wraps a dict of callables so each
|
|
205
|
+
invocation records a `ToolCall` with name, arguments, result, latency,
|
|
206
|
+
and any raised exception. Shared between both adapters.
|
|
207
|
+
- `agentprdiff.adapters.pricing` — curated model→price table for cost
|
|
208
|
+
estimation, with `register_prices()` and per-call `prices=` overrides.
|
|
209
|
+
Unknown models record `cost_usd=0.0` and emit a single `RuntimeWarning`
|
|
210
|
+
per process so missing pricing is loud rather than silent.
|
|
211
|
+
- Documentation: `docs/adapters.md` (full reference) and
|
|
212
|
+
`docs/adapters-vercel.md` (manual integration recipe for the Vercel AI
|
|
213
|
+
SDK, which is JS-only and lives in a future companion package).
|
|
214
|
+
- `AGENTS.md` at the repo root — an instruction set written for AI
|
|
215
|
+
coding agents (Claude Code, Cursor, Aider, etc.) that have been asked
|
|
216
|
+
to add `agentprdiff` to a codebase. Covers codebase discovery,
|
|
217
|
+
contract identification, wrap-the-agent recipes (OpenAI / Anthropic /
|
|
218
|
+
custom), stub patterns, suite scaffolding, baseline recording, CI
|
|
219
|
+
wiring, common pitfalls, and a validation checklist. Optimized for
|
|
220
|
+
AI-agent-driven adoption with copy-paste templates.
|
|
221
|
+
- `docs/ai-driven-adoption.md` — human-facing companion to AGENTS.md.
|
|
222
|
+
Three prompt templates (minimum viable / recommended / contract-driven)
|
|
223
|
+
for adopters using Claude Code / Cursor / Aider, plus a sample
|
|
224
|
+
first-session transcript and tips for working with the AI agent
|
|
225
|
+
through the adoption flow.
|
|
226
|
+
- `docs/suite-layout.md` — canonical reference for the suite directory
|
|
227
|
+
structure. Lists each file (`suites/<project>.py`, `_eval_agent.py`,
|
|
228
|
+
`_stubs.py`, baselines, CI workflow, etc.), classifies them as
|
|
229
|
+
mandatory / recommended / optional, and specifies what each must
|
|
230
|
+
and must not contain. Cross-referenced from AGENTS.md and the
|
|
231
|
+
validation checklist.
|
|
232
|
+
|
|
233
|
+
### Changed
|
|
234
|
+
|
|
235
|
+
- The suite loader now inserts the current working directory onto
|
|
236
|
+
`sys.path` in addition to the suite file's parent directory. Adopters
|
|
237
|
+
who run `agentprdiff record suites/foo.py` from their project root no
|
|
238
|
+
longer have to manually patch `sys.path` to import their own modules
|
|
239
|
+
(e.g. `from agent.agent import ...`, `from config import ...`).
|
|
240
|
+
Both insertions are reverted after the suite loads, so no path leakage
|
|
241
|
+
between runs.
|
|
242
|
+
|
|
243
|
+
### Notes
|
|
244
|
+
|
|
245
|
+
- The base `pip install agentprdiff` does **not** require the `openai` or
|
|
246
|
+
`anthropic` packages. The adapters operate on a client object's shape,
|
|
247
|
+
not on imported SDK modules — so installing only the SDKs you actually
|
|
248
|
+
use keeps the dependency footprint small. Optional extras are still
|
|
249
|
+
declared (`agentprdiff[openai]`, `agentprdiff[anthropic]`) for adopters
|
|
250
|
+
who prefer to pin the SDK version alongside agentprdiff itself.
|
|
251
|
+
|
|
252
|
+
## [0.1.0] — 2026-04-22
|
|
253
|
+
|
|
254
|
+
Initial public release.
|
|
255
|
+
|
|
256
|
+
### Added
|
|
257
|
+
|
|
258
|
+
- Core `Suite` / `Case` / `Trace` model for defining agent regression tests.
|
|
259
|
+
- Deterministic graders: `contains`, `contains_any`, `regex_match`, `tool_called`,
|
|
260
|
+
`tool_sequence`, `output_length_lt`, `latency_lt_ms`, `cost_lt_usd`,
|
|
261
|
+
`no_tool_called`.
|
|
262
|
+
- Semantic grader (`semantic`) with a pluggable `judge` callable and built-in
|
|
263
|
+
fake judge for CI environments without API keys.
|
|
264
|
+
- Baseline store (JSON files under `.agentprdiff/baselines/`) designed to be
|
|
265
|
+
committed to version control.
|
|
266
|
+
- Trace diff engine producing a structured `TraceDelta` (assertion pass/fail
|
|
267
|
+
changes, cost delta, latency delta, tool-call sequence changes, output
|
|
268
|
+
change).
|
|
269
|
+
- CLI: `agentprdiff init`, `agentprdiff record`, `agentprdiff check`, `agentprdiff diff`.
|
|
270
|
+
- Rich-formatted terminal reporter and machine-readable JSON reporter for CI.
|
|
271
|
+
- Quickstart example with a mock agent that runs without any API keys.
|
|
272
|
+
- Pytest test suite covering graders, runner, differ, store, and CLI smoke.
|
|
273
|
+
- GitHub Actions CI workflow.
|
|
274
|
+
|
|
275
|
+
### Known limitations
|
|
276
|
+
|
|
277
|
+
- Only a manual instrumentation API for provider SDKs is shipped in 0.1.0.
|
|
278
|
+
Drop-in wrappers for OpenAI / Anthropic / Vercel AI SDK are planned for 0.2.
|
|
279
|
+
- The semantic grader's built-in judge supports OpenAI and Anthropic via user-
|
|
280
|
+
supplied API keys; hosted judge endpoints are not yet offered.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: agentprdiff
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.3
|
|
4
4
|
Summary: Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change.
|
|
5
5
|
Project-URL: Homepage, https://github.com/vnageshwaran-de/agentprdiff
|
|
6
6
|
Project-URL: Documentation, https://github.com/vnageshwaran-de/agentprdiff#readme
|
|
@@ -94,7 +94,7 @@ agentprdiff record suite.py # save this run as the baseline
|
|
|
94
94
|
agentprdiff check suite.py # in CI: diff vs baseline, exit 1 on regression
|
|
95
95
|
```
|
|
96
96
|
|
|
97
|
-
That's the whole product.
|
|
97
|
+
That's the whole product. Five CLI commands (`init`, `record`, `check`, `review`, `scaffold`). One Python file. Zero framework lock-in.
|
|
98
98
|
|
|
99
99
|
## What's in the box
|
|
100
100
|
|
|
@@ -104,7 +104,7 @@ That's the whole product. Four CLI commands. One Python file. Zero framework loc
|
|
|
104
104
|
- **Diff engine** — per-case `TraceDelta` with assertion pass/fail changes, cost delta, latency delta, tool-sequence changes, and a unified output diff.
|
|
105
105
|
- **CI-ready CLI** — exit 1 on regression, `--json-out` for artifact archiving, Rich-formatted terminal output.
|
|
106
106
|
- **Zero SDK lock-in** — works with OpenAI, Anthropic, Gemini, Bedrock, LangChain, LangGraph, LlamaIndex, Vercel AI SDK, custom wrappers — if you can wrap your agent in a function, `agentprdiff` can test it.
|
|
107
|
-
- **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (or any OpenAI-compatible provider
|
|
107
|
+
- **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (sync **or** async — `AsyncOpenAI` is supported by the same context manager) or any OpenAI-compatible provider (Groq / Gemini / OpenRouter / Ollama / vLLM / Together / Fireworks / DeepInfra) or the Anthropic SDK. No manual `Trace` wiring required.
|
|
108
108
|
|
|
109
109
|
## How it compares
|
|
110
110
|
|
|
@@ -127,13 +127,49 @@ The value is in the combination: deterministic assertions for the 80% of behavio
|
|
|
127
127
|
|
|
128
128
|
This is the same loop as Jest snapshot tests or VCR cassettes — applied to LLM agents.
|
|
129
129
|
|
|
130
|
+
### API keys
|
|
131
|
+
|
|
132
|
+
`agentprdiff` doesn't read your agent's API key — your agent does, through whatever env var it already uses. Set that locally (in `.env`, your shell, direnv, whatever) and as a GitHub Actions secret in CI. The scaffold's workflow YAML has the right shape; you fill in the env var name to match your agent.
|
|
133
|
+
|
|
134
|
+
The `semantic()` grader is the one piece of agentprdiff that can use an API key directly — for the LLM judge. Without one, it silently falls back to keyword matching. Set `ANTHROPIC_API_KEY` (cheaper) or `OPENAI_API_KEY` if you want a real judge in CI; leave both unset to keep CI free with fake_judge.
|
|
135
|
+
|
|
136
|
+
See [AGENTS.md → API keys](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md#api-keys--what-to-set-where-and-how-to-ask-the-user-about-them) for the full setup (local options, CI secrets, what never to do).
|
|
137
|
+
|
|
138
|
+
### What each command does on rerun
|
|
139
|
+
|
|
140
|
+
A common first-day question. Short version:
|
|
141
|
+
|
|
142
|
+
- `record` — overwrites baselines in place. Re-recording an intentional change shows up as a regular git diff in your PR; that's the review surface.
|
|
143
|
+
- `check` — creates a new timestamped directory under `.agentprdiff/runs/` on every invocation. It's gitignored by default, so it never reaches CI; clean local history any time with `rm -rf .agentprdiff/runs/`. `--json-out PATH` overwrites a single file at PATH.
|
|
144
|
+
- `review` — same comparison as `check`, but renders one verbose panel per case and **always exits 0**. Designed for local iteration loops; not meant for CI. Writes to the same `.agentprdiff/runs/` directory.
|
|
145
|
+
- `scaffold` — never overwrites. Skips files that already exist (`[skip]`) and writes the rest.
|
|
146
|
+
- `init` — idempotent; running it twice does nothing the second time.
|
|
147
|
+
|
|
148
|
+
See [AGENTS.md → Rerun semantics](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md#rerun-semantics--what-each-command-does-on-the-second-run) for examples.
|
|
149
|
+
|
|
150
|
+
### Scaffolding a new suite
|
|
151
|
+
|
|
152
|
+
Skip the copy-paste from [AGENTS.md](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md):
|
|
153
|
+
|
|
154
|
+
```bash
|
|
155
|
+
agentprdiff scaffold ai_content_summary --recipe sync-openai
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
Writes the canonical layout (`suites/__init__.py`, `_eval_agent.py`, `_stubs.py`, `<name>.py`, `<name>_cases.md`, `suites/README.md`, and `.github/workflows/agentprdiff.yml`) with TODO markers where you wire in your agent. The `<name>_cases.md` file is a *case dossier* — reviewer-facing prose with one block per case (what it tests, input, assertions in plain English, file:line references to production code, and the application impact if the case regresses). Three recipes:
|
|
159
|
+
|
|
160
|
+
- `sync-openai` (default): uses `instrument_client` from the OpenAI adapter with a sync `OpenAI()` client.
|
|
161
|
+
- `async-openai`: same `instrument_client`, paired with an `asyncio.run` bridge so an `AsyncOpenAI` agent works with agentprdiff's sync runner. The adapter detects the async client at entry — no separate API.
|
|
162
|
+
- `stubbed`: substitutes a single LLM helper instead of the SDK client. Best for summarization / classification / embedding-prep agents — see [`docs/adapters.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/adapters.md#stubbed-llm-boundary-pattern).
|
|
163
|
+
|
|
164
|
+
The generated workflow includes `permissions: contents: read` so GHAS doesn't flag it. Pre-existing files are never overwritten.
|
|
165
|
+
|
|
130
166
|
## Instrumenting your agent
|
|
131
167
|
|
|
132
168
|
You have two paths. Most agents need the first.
|
|
133
169
|
|
|
134
170
|
### Option A — SDK adapters (zero manual work)
|
|
135
171
|
|
|
136
|
-
If your agent uses the OpenAI Python SDK
|
|
172
|
+
If your agent uses the OpenAI Python SDK — sync `OpenAI` **or** async `AsyncOpenAI`, including any OpenAI-compatible provider (Groq, Gemini, OpenRouter, Ollama, vLLM, Together, Fireworks, DeepInfra) — or the Anthropic SDK, the SDK adapter captures every model and tool call automatically:
|
|
137
173
|
|
|
138
174
|
```python
|
|
139
175
|
from openai import OpenAI
|
|
@@ -150,7 +186,26 @@ def my_agent(query: str):
|
|
|
150
186
|
return final_text, trace
|
|
151
187
|
```
|
|
152
188
|
|
|
153
|
-
|
|
189
|
+
For `AsyncOpenAI`, the same `instrument_client` works — it inspects `client.chat.completions.create` at entry and installs an awaitable patched method when the underlying one is `async def`. `instrument_tools` mirrors per-tool: `async def` tools come back awaitable, sync tools stay sync. The `with` block is still a regular `with`:
|
|
190
|
+
|
|
191
|
+
```python
|
|
192
|
+
import asyncio
|
|
193
|
+
from openai import AsyncOpenAI
|
|
194
|
+
from agentprdiff.adapters.openai import instrument_client, instrument_tools
|
|
195
|
+
|
|
196
|
+
async def my_agent_async(query: str):
|
|
197
|
+
client = AsyncOpenAI()
|
|
198
|
+
with instrument_client(client) as trace:
|
|
199
|
+
tools = instrument_tools(TOOL_MAP, trace)
|
|
200
|
+
response = await client.chat.completions.create(...)
|
|
201
|
+
# ... await tools[name](**args) for async tools, tools[name](**args) for sync ...
|
|
202
|
+
return final_text, trace
|
|
203
|
+
|
|
204
|
+
def my_agent(query: str):
|
|
205
|
+
return asyncio.run(my_agent_async(query))
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
The patch is scoped to the specific client instance and reversed when the `with` block exits — no global SDK state is touched. Anthropic adopters use `agentprdiff.adapters.anthropic` with the same shape (sync clients today; async Anthropic is on the roadmap).
|
|
154
209
|
|
|
155
210
|
See [`docs/adapters.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/adapters.md) for the full reference, including pricing overrides, custom provider tags, and recipes for nested agents.
|
|
156
211
|
|
|
@@ -186,6 +241,8 @@ Agents that return just an output still work — `agentprdiff` wraps them and ca
|
|
|
186
241
|
# .github/workflows/agents.yml
|
|
187
242
|
name: agent-regression
|
|
188
243
|
on: [pull_request]
|
|
244
|
+
permissions:
|
|
245
|
+
contents: read # least-privilege; GHAS flags workflows without this.
|
|
189
246
|
jobs:
|
|
190
247
|
agentprdiff:
|
|
191
248
|
runs-on: ubuntu-latest
|
|
@@ -200,6 +257,8 @@ jobs:
|
|
|
200
257
|
with: { name: agentprdiff, path: artifacts/ }
|
|
201
258
|
```
|
|
202
259
|
|
|
260
|
+
If you use `--json-out artifacts/...`, add `artifacts/agentprdiff*.json` (or the broader `artifacts/`) to your project's `.gitignore` — the CI artifact upload doesn't prevent a contributor from accidentally `git add`ing it locally.
|
|
261
|
+
|
|
203
262
|
See [`docs/ci-integration.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/ci-integration.md) for GitLab, CircleCI, and Buildkite.
|
|
204
263
|
|
|
205
264
|
## Quickstart
|
|
@@ -221,9 +280,53 @@ sed -i "s/refund/noundr/g" agent.py
|
|
|
221
280
|
agentprdiff check suite.py # exit 1; see the diff
|
|
222
281
|
```
|
|
223
282
|
|
|
283
|
+
### Running a subset of cases
|
|
284
|
+
|
|
285
|
+
Iterating on a single failing case shouldn't require commenting out the rest. `record`, `check`, and `review` all accept `--case` and `--skip` for narrowing a run:
|
|
286
|
+
|
|
287
|
+
```bash
|
|
288
|
+
# Discover what's available.
|
|
289
|
+
agentprdiff check suite.py --list
|
|
290
|
+
|
|
291
|
+
# Single case (case-insensitive substring).
|
|
292
|
+
agentprdiff check suite.py --case refund_happy_path
|
|
293
|
+
|
|
294
|
+
# Glob across cases.
|
|
295
|
+
agentprdiff check suite.py --case "*order*"
|
|
296
|
+
|
|
297
|
+
# Multiple patterns (repeated flag or comma-separated).
|
|
298
|
+
agentprdiff check suite.py --case refund --case policy
|
|
299
|
+
agentprdiff check suite.py --case refund,policy
|
|
300
|
+
|
|
301
|
+
# Everything except slow cases.
|
|
302
|
+
agentprdiff check suite.py --skip slow
|
|
303
|
+
agentprdiff check suite.py --case ~slow # equivalent
|
|
304
|
+
|
|
305
|
+
# Qualify by suite when names collide across suites.
|
|
306
|
+
agentprdiff check suite.py --case "billing:refund*"
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
A filter that matches zero cases exits 2 and prints the available case names — `--list` is the discoverable counterpart. The selection summary (`running 2 of 4 cases in <suite>: ...`) is printed before each suite runs so a partial match is never silent.
|
|
310
|
+
|
|
311
|
+
### Reviewing one case (the local-iteration loop)
|
|
312
|
+
|
|
313
|
+
`agentprdiff check` is built for CI: a compact summary table and exit 1 on regression. While you're iterating on a single case, that's the wrong shape — you want to see *everything* about that one case, and you don't want your shell going red between every keystroke. That's `agentprdiff review`:
|
|
314
|
+
|
|
315
|
+
```bash
|
|
316
|
+
# Verbose per-case panel: input, every assertion's was→now verdict,
|
|
317
|
+
# cost/latency/token deltas, tool-sequence diff, output diff.
|
|
318
|
+
agentprdiff review suite.py --case refund_happy_path
|
|
319
|
+
|
|
320
|
+
# Same filter syntax as check / record — globs, negation, multi-pattern.
|
|
321
|
+
agentprdiff review suite.py --case "*refund*"
|
|
322
|
+
agentprdiff review suite.py --skip slow
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
`review` runs the same comparison `check` does (and writes to the same `.agentprdiff/runs/` directory) but **always exits 0**, even on regression — so it sits cleanly inside watcher loops (`entr`, `watchexec`, `fzf` previews). Use `check` when you want CI's exit semantics locally; reach for `review` while you're working. Think `pytest -k`.
|
|
326
|
+
|
|
224
327
|
## Status
|
|
225
328
|
|
|
226
|
-
`agentprdiff` is **alpha** (0.2.x). The core model, CLI, and OpenAI / Anthropic SDK adapters are stable. LangChain/LangGraph adapters and a JS companion package for the Vercel AI SDK are on the 0.3 roadmap. See [`CHANGELOG.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/CHANGELOG.md).
|
|
329
|
+
`agentprdiff` is **alpha** (0.2.x). The core model, CLI, and OpenAI / Anthropic SDK adapters are stable. The OpenAI adapter covers both sync `OpenAI` and async `AsyncOpenAI` clients via the same `instrument_client` context manager. Async Anthropic, LangChain/LangGraph adapters, and a JS companion package for the Vercel AI SDK are on the 0.3 roadmap. See [`CHANGELOG.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/CHANGELOG.md).
|
|
227
330
|
|
|
228
331
|
Feedback, bug reports, and PRs extremely welcome. Open an issue or @ me.
|
|
229
332
|
|
|
@@ -55,7 +55,7 @@ agentprdiff record suite.py # save this run as the baseline
|
|
|
55
55
|
agentprdiff check suite.py # in CI: diff vs baseline, exit 1 on regression
|
|
56
56
|
```
|
|
57
57
|
|
|
58
|
-
That's the whole product.
|
|
58
|
+
That's the whole product. Five CLI commands (`init`, `record`, `check`, `review`, `scaffold`). One Python file. Zero framework lock-in.
|
|
59
59
|
|
|
60
60
|
## What's in the box
|
|
61
61
|
|
|
@@ -65,7 +65,7 @@ That's the whole product. Four CLI commands. One Python file. Zero framework loc
|
|
|
65
65
|
- **Diff engine** — per-case `TraceDelta` with assertion pass/fail changes, cost delta, latency delta, tool-sequence changes, and a unified output diff.
|
|
66
66
|
- **CI-ready CLI** — exit 1 on regression, `--json-out` for artifact archiving, Rich-formatted terminal output.
|
|
67
67
|
- **Zero SDK lock-in** — works with OpenAI, Anthropic, Gemini, Bedrock, LangChain, LangGraph, LlamaIndex, Vercel AI SDK, custom wrappers — if you can wrap your agent in a function, `agentprdiff` can test it.
|
|
68
|
-
- **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (or any OpenAI-compatible provider
|
|
68
|
+
- **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (sync **or** async — `AsyncOpenAI` is supported by the same context manager) or any OpenAI-compatible provider (Groq / Gemini / OpenRouter / Ollama / vLLM / Together / Fireworks / DeepInfra) or the Anthropic SDK. No manual `Trace` wiring required.
|
|
69
69
|
|
|
70
70
|
## How it compares
|
|
71
71
|
|
|
@@ -88,13 +88,49 @@ The value is in the combination: deterministic assertions for the 80% of behavio
|
|
|
88
88
|
|
|
89
89
|
This is the same loop as Jest snapshot tests or VCR cassettes — applied to LLM agents.
|
|
90
90
|
|
|
91
|
+
### API keys
|
|
92
|
+
|
|
93
|
+
`agentprdiff` doesn't read your agent's API key — your agent does, through whatever env var it already uses. Set that locally (in `.env`, your shell, direnv, whatever) and as a GitHub Actions secret in CI. The scaffold's workflow YAML has the right shape; you fill in the env var name to match your agent.
|
|
94
|
+
|
|
95
|
+
The `semantic()` grader is the one piece of agentprdiff that can use an API key directly — for the LLM judge. Without one, it silently falls back to keyword matching. Set `ANTHROPIC_API_KEY` (cheaper) or `OPENAI_API_KEY` if you want a real judge in CI; leave both unset to keep CI free with fake_judge.
|
|
96
|
+
|
|
97
|
+
See [AGENTS.md → API keys](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md#api-keys--what-to-set-where-and-how-to-ask-the-user-about-them) for the full setup (local options, CI secrets, what never to do).
|
|
98
|
+
|
|
99
|
+
### What each command does on rerun
|
|
100
|
+
|
|
101
|
+
A common first-day question. Short version:
|
|
102
|
+
|
|
103
|
+
- `record` — overwrites baselines in place. Re-recording an intentional change shows up as a regular git diff in your PR; that's the review surface.
|
|
104
|
+
- `check` — creates a new timestamped directory under `.agentprdiff/runs/` on every invocation. It's gitignored by default, so it never reaches CI; clean local history any time with `rm -rf .agentprdiff/runs/`. `--json-out PATH` overwrites a single file at PATH.
|
|
105
|
+
- `review` — same comparison as `check`, but renders one verbose panel per case and **always exits 0**. Designed for local iteration loops; not meant for CI. Writes to the same `.agentprdiff/runs/` directory.
|
|
106
|
+
- `scaffold` — never overwrites. Skips files that already exist (`[skip]`) and writes the rest.
|
|
107
|
+
- `init` — idempotent; running it twice does nothing the second time.
|
|
108
|
+
|
|
109
|
+
See [AGENTS.md → Rerun semantics](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md#rerun-semantics--what-each-command-does-on-the-second-run) for examples.
|
|
110
|
+
|
|
111
|
+
### Scaffolding a new suite
|
|
112
|
+
|
|
113
|
+
Skip the copy-paste from [AGENTS.md](https://github.com/vnageshwaran-de/agentprdiff/blob/main/AGENTS.md):
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
agentprdiff scaffold ai_content_summary --recipe sync-openai
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Writes the canonical layout (`suites/__init__.py`, `_eval_agent.py`, `_stubs.py`, `<name>.py`, `<name>_cases.md`, `suites/README.md`, and `.github/workflows/agentprdiff.yml`) with TODO markers where you wire in your agent. The `<name>_cases.md` file is a *case dossier* — reviewer-facing prose with one block per case (what it tests, input, assertions in plain English, file:line references to production code, and the application impact if the case regresses). Three recipes:
|
|
120
|
+
|
|
121
|
+
- `sync-openai` (default): uses `instrument_client` from the OpenAI adapter with a sync `OpenAI()` client.
|
|
122
|
+
- `async-openai`: same `instrument_client`, paired with an `asyncio.run` bridge so an `AsyncOpenAI` agent works with agentprdiff's sync runner. The adapter detects the async client at entry — no separate API.
|
|
123
|
+
- `stubbed`: substitutes a single LLM helper instead of the SDK client. Best for summarization / classification / embedding-prep agents — see [`docs/adapters.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/adapters.md#stubbed-llm-boundary-pattern).
|
|
124
|
+
|
|
125
|
+
The generated workflow includes `permissions: contents: read` so GHAS doesn't flag it. Pre-existing files are never overwritten.
|
|
126
|
+
|
|
91
127
|
## Instrumenting your agent
|
|
92
128
|
|
|
93
129
|
You have two paths. Most agents need the first.
|
|
94
130
|
|
|
95
131
|
### Option A — SDK adapters (zero manual work)
|
|
96
132
|
|
|
97
|
-
If your agent uses the OpenAI Python SDK
|
|
133
|
+
If your agent uses the OpenAI Python SDK — sync `OpenAI` **or** async `AsyncOpenAI`, including any OpenAI-compatible provider (Groq, Gemini, OpenRouter, Ollama, vLLM, Together, Fireworks, DeepInfra) — or the Anthropic SDK, the SDK adapter captures every model and tool call automatically:
|
|
98
134
|
|
|
99
135
|
```python
|
|
100
136
|
from openai import OpenAI
|
|
@@ -111,7 +147,26 @@ def my_agent(query: str):
|
|
|
111
147
|
return final_text, trace
|
|
112
148
|
```
|
|
113
149
|
|
|
114
|
-
|
|
150
|
+
For `AsyncOpenAI`, the same `instrument_client` works — it inspects `client.chat.completions.create` at entry and installs an awaitable patched method when the underlying one is `async def`. `instrument_tools` mirrors per-tool: `async def` tools come back awaitable, sync tools stay sync. The `with` block is still a regular `with`:
|
|
151
|
+
|
|
152
|
+
```python
|
|
153
|
+
import asyncio
|
|
154
|
+
from openai import AsyncOpenAI
|
|
155
|
+
from agentprdiff.adapters.openai import instrument_client, instrument_tools
|
|
156
|
+
|
|
157
|
+
async def my_agent_async(query: str):
|
|
158
|
+
client = AsyncOpenAI()
|
|
159
|
+
with instrument_client(client) as trace:
|
|
160
|
+
tools = instrument_tools(TOOL_MAP, trace)
|
|
161
|
+
response = await client.chat.completions.create(...)
|
|
162
|
+
# ... await tools[name](**args) for async tools, tools[name](**args) for sync ...
|
|
163
|
+
return final_text, trace
|
|
164
|
+
|
|
165
|
+
def my_agent(query: str):
|
|
166
|
+
return asyncio.run(my_agent_async(query))
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
The patch is scoped to the specific client instance and reversed when the `with` block exits — no global SDK state is touched. Anthropic adopters use `agentprdiff.adapters.anthropic` with the same shape (sync clients today; async Anthropic is on the roadmap).
|
|
115
170
|
|
|
116
171
|
See [`docs/adapters.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/adapters.md) for the full reference, including pricing overrides, custom provider tags, and recipes for nested agents.
|
|
117
172
|
|
|
@@ -147,6 +202,8 @@ Agents that return just an output still work — `agentprdiff` wraps them and ca
|
|
|
147
202
|
# .github/workflows/agents.yml
|
|
148
203
|
name: agent-regression
|
|
149
204
|
on: [pull_request]
|
|
205
|
+
permissions:
|
|
206
|
+
contents: read # least-privilege; GHAS flags workflows without this.
|
|
150
207
|
jobs:
|
|
151
208
|
agentprdiff:
|
|
152
209
|
runs-on: ubuntu-latest
|
|
@@ -161,6 +218,8 @@ jobs:
|
|
|
161
218
|
with: { name: agentprdiff, path: artifacts/ }
|
|
162
219
|
```
|
|
163
220
|
|
|
221
|
+
If you use `--json-out artifacts/...`, add `artifacts/agentprdiff*.json` (or the broader `artifacts/`) to your project's `.gitignore` — the CI artifact upload doesn't prevent a contributor from accidentally `git add`ing it locally.
|
|
222
|
+
|
|
164
223
|
See [`docs/ci-integration.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/docs/ci-integration.md) for GitLab, CircleCI, and Buildkite.
|
|
165
224
|
|
|
166
225
|
## Quickstart
|
|
@@ -182,9 +241,53 @@ sed -i "s/refund/noundr/g" agent.py
|
|
|
182
241
|
agentprdiff check suite.py # exit 1; see the diff
|
|
183
242
|
```
|
|
184
243
|
|
|
244
|
+
### Running a subset of cases
|
|
245
|
+
|
|
246
|
+
Iterating on a single failing case shouldn't require commenting out the rest. `record`, `check`, and `review` all accept `--case` and `--skip` for narrowing a run:
|
|
247
|
+
|
|
248
|
+
```bash
|
|
249
|
+
# Discover what's available.
|
|
250
|
+
agentprdiff check suite.py --list
|
|
251
|
+
|
|
252
|
+
# Single case (case-insensitive substring).
|
|
253
|
+
agentprdiff check suite.py --case refund_happy_path
|
|
254
|
+
|
|
255
|
+
# Glob across cases.
|
|
256
|
+
agentprdiff check suite.py --case "*order*"
|
|
257
|
+
|
|
258
|
+
# Multiple patterns (repeated flag or comma-separated).
|
|
259
|
+
agentprdiff check suite.py --case refund --case policy
|
|
260
|
+
agentprdiff check suite.py --case refund,policy
|
|
261
|
+
|
|
262
|
+
# Everything except slow cases.
|
|
263
|
+
agentprdiff check suite.py --skip slow
|
|
264
|
+
agentprdiff check suite.py --case ~slow # equivalent
|
|
265
|
+
|
|
266
|
+
# Qualify by suite when names collide across suites.
|
|
267
|
+
agentprdiff check suite.py --case "billing:refund*"
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
A filter that matches zero cases exits 2 and prints the available case names — `--list` is the discoverable counterpart. The selection summary (`running 2 of 4 cases in <suite>: ...`) is printed before each suite runs so a partial match is never silent.
|
|
271
|
+
|
|
272
|
+
### Reviewing one case (the local-iteration loop)
|
|
273
|
+
|
|
274
|
+
`agentprdiff check` is built for CI: a compact summary table and exit 1 on regression. While you're iterating on a single case, that's the wrong shape — you want to see *everything* about that one case, and you don't want your shell going red between every keystroke. That's `agentprdiff review`:
|
|
275
|
+
|
|
276
|
+
```bash
|
|
277
|
+
# Verbose per-case panel: input, every assertion's was→now verdict,
|
|
278
|
+
# cost/latency/token deltas, tool-sequence diff, output diff.
|
|
279
|
+
agentprdiff review suite.py --case refund_happy_path
|
|
280
|
+
|
|
281
|
+
# Same filter syntax as check / record — globs, negation, multi-pattern.
|
|
282
|
+
agentprdiff review suite.py --case "*refund*"
|
|
283
|
+
agentprdiff review suite.py --skip slow
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
`review` runs the same comparison `check` does (and writes to the same `.agentprdiff/runs/` directory) but **always exits 0**, even on regression — so it sits cleanly inside watcher loops (`entr`, `watchexec`, `fzf` previews). Use `check` when you want CI's exit semantics locally; reach for `review` while you're working. Think `pytest -k`.
|
|
287
|
+
|
|
185
288
|
## Status
|
|
186
289
|
|
|
187
|
-
`agentprdiff` is **alpha** (0.2.x). The core model, CLI, and OpenAI / Anthropic SDK adapters are stable. LangChain/LangGraph adapters and a JS companion package for the Vercel AI SDK are on the 0.3 roadmap. See [`CHANGELOG.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/CHANGELOG.md).
|
|
290
|
+
`agentprdiff` is **alpha** (0.2.x). The core model, CLI, and OpenAI / Anthropic SDK adapters are stable. The OpenAI adapter covers both sync `OpenAI` and async `AsyncOpenAI` clients via the same `instrument_client` context manager. Async Anthropic, LangChain/LangGraph adapters, and a JS companion package for the Vercel AI SDK are on the 0.3 roadmap. See [`CHANGELOG.md`](https://github.com/vnageshwaran-de/agentprdiff/blob/main/CHANGELOG.md).
|
|
188
291
|
|
|
189
292
|
Feedback, bug reports, and PRs extremely welcome. Open an issue or @ me.
|
|
190
293
|
|
|
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "agentprdiff"
|
|
7
|
-
version = "0.2.
|
|
7
|
+
version = "0.2.3"
|
|
8
8
|
description = "Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change."
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
requires-python = ">=3.10"
|