agentprdiff 0.1.0__tar.gz → 0.2.0__tar.gz

{agentprdiff-0.1.0 → agentprdiff-0.2.0}/.gitignore

@@ -26,7 +26,7 @@ env/
 .vscode/
 *.swp
 
-# agentguard runtime
-# Users SHOULD commit .agentguard/baselines; they should NOT commit run artifacts.
-.agentguard/runs/
-.agentguard/cache/
+# agentprdiff runtime
+# Users SHOULD commit .agentprdiff/baselines; they should NOT commit run artifacts.
+.agentprdiff/runs/
+.agentprdiff/cache/

agentprdiff-0.2.0/CHANGELOG.md

@@ -0,0 +1,103 @@
+# Changelog
+
+All notable changes to `agentprdiff` are documented in this file. Originally
+prototyped under the name `tracediff`; renamed before first public release.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.2.0] — 2026-04-26
+
+### Added
+
+- **SDK adapters** for the two dominant agent toolchains, eliminating the need
+  for manual `Trace` instrumentation:
+  - `agentprdiff.adapters.openai.instrument_client` — context manager that
+    monkey-patches `client.chat.completions.create` for the duration of one
+    agent call. Records each invocation as an `LLMCall` (provider, model,
+    input messages, output text, tool calls, tokens, cost, latency) and
+    restores the original on exit. Works with **OpenAI, Groq, Gemini's
+    OpenAI-compatible endpoint, OpenRouter, Ollama, vLLM, Together,
+    Fireworks, DeepInfra**, and any other SDK that follows the OpenAI client
+    shape.
+  - `agentprdiff.adapters.anthropic.instrument_client` — equivalent for the
+    Anthropic Messages API (`client.messages.create`). Handles the
+    content-block response shape (text + `tool_use` blocks) and the
+    Messages-API token field names.
+  - `instrument_tools(tool_map, trace)` — wraps a dict of callables so each
+    invocation records a `ToolCall` with name, arguments, result, latency,
+    and any raised exception. Shared between both adapters.
+  - `agentprdiff.adapters.pricing` — curated model→price table for cost
+    estimation, with `register_prices()` and per-call `prices=` overrides.
+    Unknown models record `cost_usd=0.0` and emit a single `RuntimeWarning`
+    per process so missing pricing is loud rather than silent.
+- Documentation: `docs/adapters.md` (full reference) and
+  `docs/adapters-vercel.md` (manual integration recipe for the Vercel AI
+  SDK, which is JS-only and lives in a future companion package).
+- `AGENTS.md` at the repo root — an instruction set written for AI
+  coding agents (Claude Code, Cursor, Aider, etc.) that have been asked
+  to add `agentprdiff` to a codebase. Covers codebase discovery,
+  contract identification, wrap-the-agent recipes (OpenAI / Anthropic /
+  custom), stub patterns, suite scaffolding, baseline recording, CI
+  wiring, common pitfalls, and a validation checklist. Optimized for
+  AI-agent-driven adoption with copy-paste templates.
+- `docs/ai-driven-adoption.md` — human-facing companion to AGENTS.md.
+  Three prompt templates (minimum viable / recommended / contract-driven)
+  for adopters using Claude Code / Cursor / Aider, plus a sample
+  first-session transcript and tips for working with the AI agent
+  through the adoption flow.
+- `docs/suite-layout.md` — canonical reference for the suite directory
+  structure. Lists each file (`suites/<project>.py`, `_eval_agent.py`,
+  `_stubs.py`, baselines, CI workflow, etc.), classifies them as
+  mandatory / recommended / optional, and specifies what each must
+  and must not contain. Cross-referenced from AGENTS.md and the
+  validation checklist.
+
+### Changed
+
+- The suite loader now inserts the current working directory onto
+  `sys.path` in addition to the suite file's parent directory. Adopters
+  who run `agentprdiff record suites/foo.py` from their project root no
+  longer have to manually patch `sys.path` to import their own modules
+  (e.g. `from agent.agent import ...`, `from config import ...`).
+  Both insertions are reverted after the suite loads, so no path leakage
+  between runs.
+
+### Notes
+
+- The base `pip install agentprdiff` does **not** require the `openai` or
+  `anthropic` packages. The adapters operate on a client object's shape,
+  not on imported SDK modules — so installing only the SDKs you actually
+  use keeps the dependency footprint small. Optional extras are still
+  declared (`agentprdiff[openai]`, `agentprdiff[anthropic]`) for adopters
+  who prefer to pin the SDK version alongside agentprdiff itself.
+
+## [0.1.0] — 2026-04-22
+
+Initial public release.
+
+### Added
+
+- Core `Suite` / `Case` / `Trace` model for defining agent regression tests.
+- Deterministic graders: `contains`, `contains_any`, `regex_match`, `tool_called`,
+  `tool_sequence`, `output_length_lt`, `latency_lt_ms`, `cost_lt_usd`,
+  `no_tool_called`.
+- Semantic grader (`semantic`) with a pluggable `judge` callable and built-in
+  fake judge for CI environments without API keys.
+- Baseline store (JSON files under `.agentprdiff/baselines/`) designed to be
+  committed to version control.
+- Trace diff engine producing a structured `TraceDelta` (assertion pass/fail
+  changes, cost delta, latency delta, tool-call sequence changes, output
+  change).
+- CLI: `agentprdiff init`, `agentprdiff record`, `agentprdiff check`, `agentprdiff diff`.
+- Rich-formatted terminal reporter and machine-readable JSON reporter for CI.
+- Quickstart example with a mock agent that runs without any API keys.
+- Pytest test suite covering graders, runner, differ, store, and CLI smoke.
+- GitHub Actions CI workflow.
+
+### Known limitations
+
+- Only a manual instrumentation API for provider SDKs is shipped in 0.1.0.
+  Drop-in wrappers for OpenAI / Anthropic / Vercel AI SDK are planned for 0.2.
+- The semantic grader's built-in judge supports OpenAI and Anthropic via user-
+  supplied API keys; hosted judge endpoints are not yet offered.

{agentprdiff-0.1.0 → agentprdiff-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentprdiff
-Version: 0.1.0
+Version: 0.2.0
 Summary: Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change.
 Project-URL: Homepage, https://github.com/vnageshwaran-de/agentprdiff
 Project-URL: Documentation, https://github.com/vnageshwaran-de/agentprdiff#readme
@@ -50,6 +50,8 @@ pip install agentprdiff
 [![Python](https://img.shields.io/pypi/pyversions/agentprdiff.svg)](https://pypi.org/project/agentprdiff/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](./LICENSE)
 
+> **Adopting with an AI coding agent?** Point Claude Code, Cursor, Aider, or any agentic IDE at [`AGENTS.md`](./AGENTS.md) — a step-by-step adoption playbook the agent reads directly. Humans driving the adoption: see [`docs/ai-driven-adoption.md`](./docs/ai-driven-adoption.md) for copy-paste prompt templates. The canonical file layout — what's mandatory, what's recommended, what's optional — is at [`docs/suite-layout.md`](./docs/suite-layout.md).
+
 ## Why
 
 Unit tests assume determinism. Agents aren't deterministic, but they do have *behaviors you rely on* — a specific tool gets called, a refund amount is quoted, a latency budget is respected, a safety guardrail fires. When a model or prompt changes, those behaviors drift. Today most teams find out in production.
@@ -100,6 +102,7 @@ That's the whole product. Four CLI commands. One Python file. Zero framework loc
 - **Diff engine** — per-case `TraceDelta` with assertion pass/fail changes, cost delta, latency delta, tool-sequence changes, and a unified output diff.
 - **CI-ready CLI** — exit 1 on regression, `--json-out` for artifact archiving, Rich-formatted terminal output.
 - **Zero SDK lock-in** — works with OpenAI, Anthropic, Gemini, Bedrock, LangChain, LangGraph, LlamaIndex, Vercel AI SDK, custom wrappers — if you can wrap your agent in a function, `agentprdiff` can test it.
+- **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (or any OpenAI-compatible provider — Groq / Gemini / OpenRouter / Ollama / vLLM) or the Anthropic SDK. No manual `Trace` wiring required.
 
 ## How it compares
 
@@ -124,7 +127,34 @@ This is the same loop as Jest snapshot tests or VCR cassettes — applied to LLM
 
 ## Instrumenting your agent
 
-`agentprdiff` doesn't monkey-patch anything. Your agent returns `(output, Trace)`:
+You have two paths. Most agents need the first.
+
+### Option A — SDK adapters (zero manual work)
+
+If your agent uses the OpenAI Python SDK (or any OpenAI-compatible provider — Groq, Gemini, OpenRouter, Ollama, vLLM, Together, Fireworks, DeepInfra) or the Anthropic SDK, the SDK adapter captures every model and tool call automatically:
+
+```python
+from openai import OpenAI
+from agentprdiff.adapters.openai import instrument_client, instrument_tools
+
+TOOL_MAP = {"lookup_order": lookup_order, "send_email": send_email}
+
+def my_agent(query: str):
+    client = OpenAI()
+    with instrument_client(client) as trace:
+        tools = instrument_tools(TOOL_MAP, trace)
+        # ... your existing tool-calling loop, untouched ...
+        # the only swap: TOOL_MAP[fn](**args) → tools[fn](**args)
+        return final_text, trace
+```
+
+The patch is scoped to the specific client instance and reversed when the `with` block exits — no global SDK state is touched. Anthropic adopters use `agentprdiff.adapters.anthropic` with the same shape.
+
+See [`docs/adapters.md`](./docs/adapters.md) for the full reference, including pricing overrides, custom provider tags, and recipes for nested agents.
+
+### Option B — Manual instrumentation
+
+If you're not on either SDK, or you want full control, build the `Trace` yourself — `agentprdiff` doesn't require any monkey-patching:
 
 ```python
 from agentprdiff import Trace, LLMCall, ToolCall

{agentprdiff-0.1.0 → agentprdiff-0.2.0}/README.md

@@ -13,6 +13,8 @@ pip install agentprdiff
 [![Python](https://img.shields.io/pypi/pyversions/agentprdiff.svg)](https://pypi.org/project/agentprdiff/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](./LICENSE)
 
+> **Adopting with an AI coding agent?** Point Claude Code, Cursor, Aider, or any agentic IDE at [`AGENTS.md`](./AGENTS.md) — a step-by-step adoption playbook the agent reads directly. Humans driving the adoption: see [`docs/ai-driven-adoption.md`](./docs/ai-driven-adoption.md) for copy-paste prompt templates. The canonical file layout — what's mandatory, what's recommended, what's optional — is at [`docs/suite-layout.md`](./docs/suite-layout.md).
+
 ## Why
 
 Unit tests assume determinism. Agents aren't deterministic, but they do have *behaviors you rely on* — a specific tool gets called, a refund amount is quoted, a latency budget is respected, a safety guardrail fires. When a model or prompt changes, those behaviors drift. Today most teams find out in production.
@@ -63,6 +65,7 @@ That's the whole product. Four CLI commands. One Python file. Zero framework loc
 - **Diff engine** — per-case `TraceDelta` with assertion pass/fail changes, cost delta, latency delta, tool-sequence changes, and a unified output diff.
 - **CI-ready CLI** — exit 1 on regression, `--json-out` for artifact archiving, Rich-formatted terminal output.
 - **Zero SDK lock-in** — works with OpenAI, Anthropic, Gemini, Bedrock, LangChain, LangGraph, LlamaIndex, Vercel AI SDK, custom wrappers — if you can wrap your agent in a function, `agentprdiff` can test it.
+- **One-line SDK adapters** — `with instrument_client(client) as trace:` automatically records every LLM and tool call when you're on the OpenAI Python SDK (or any OpenAI-compatible provider — Groq / Gemini / OpenRouter / Ollama / vLLM) or the Anthropic SDK. No manual `Trace` wiring required.
 
 ## How it compares
 
@@ -87,7 +90,34 @@ This is the same loop as Jest snapshot tests or VCR cassettes — applied to LLM
 
 ## Instrumenting your agent
 
-`agentprdiff` doesn't monkey-patch anything. Your agent returns `(output, Trace)`:
+You have two paths. Most agents need the first.
+
+### Option A — SDK adapters (zero manual work)
+
+If your agent uses the OpenAI Python SDK (or any OpenAI-compatible provider — Groq, Gemini, OpenRouter, Ollama, vLLM, Together, Fireworks, DeepInfra) or the Anthropic SDK, the SDK adapter captures every model and tool call automatically:
+
+```python
+from openai import OpenAI
+from agentprdiff.adapters.openai import instrument_client, instrument_tools
+
+TOOL_MAP = {"lookup_order": lookup_order, "send_email": send_email}
+
+def my_agent(query: str):
+    client = OpenAI()
+    with instrument_client(client) as trace:
+        tools = instrument_tools(TOOL_MAP, trace)
+        # ... your existing tool-calling loop, untouched ...
+        # the only swap: TOOL_MAP[fn](**args) → tools[fn](**args)
+        return final_text, trace
+```
+
+The patch is scoped to the specific client instance and reversed when the `with` block exits — no global SDK state is touched. Anthropic adopters use `agentprdiff.adapters.anthropic` with the same shape.
+
+See [`docs/adapters.md`](./docs/adapters.md) for the full reference, including pricing overrides, custom provider tags, and recipes for nested agents.
+
+### Option B — Manual instrumentation
+
+If you're not on either SDK, or you want full control, build the `Trace` yourself — `agentprdiff` doesn't require any monkey-patching:
 
 ```python
 from agentprdiff import Trace, LLMCall, ToolCall

agentprdiff-0.2.0/examples/regression-tour/README.md

@@ -0,0 +1,140 @@
+# Regression tour
+
+A complete walkthrough of every grader and every failure mode in `agentprdiff`. Runs without API keys (uses `fake_judge` for the semantic grader so no OpenAI/Anthropic key is required).
+
+## What this exercises
+
+- All 10 deterministic and semantic graders: `contains`, `contains_any`, `regex_match`, `tool_called`, `tool_sequence`, `no_tool_called`, `output_length_lt`, `latency_lt_ms`, `cost_lt_usd`, `semantic`.
+- All 6 regression scenarios the differ can detect: output drift, extra tool, missing tool, tool reordering, latency regression, cost regression.
+- The Rich terminal reporter and exit-code behavior used in CI.
+
+## A note on invocation
+
+The commands below use `agentprdiff` directly. If pip's user-script directory isn't on your PATH, substitute `python3 -m agentprdiff.cli` everywhere — both forms are equivalent.
+
+## Setup (one-time)
+
+```bash
+cd examples/regression-tour
+agentprdiff init                    # creates the .agentprdiff/ scaffolding
+agentprdiff record suite.py         # captures the baseline trace
+```
+
+You should see baselines written under `.agentprdiff/baselines/`. In a real project these get committed to git — that's the whole point of agentprdiff.
+
+## Happy path
+
+```bash
+agentprdiff check suite.py
+echo "exit: $?"                     # 0
+```
+
+All three cases pass with no diff against the recorded baseline.
+
+## Regression scenarios
+
+Each command below injects one specific regression by setting `MODE`. Every one should fail with a clear diff and a non-zero exit code.
+
+### 1. Output text drifted
+
+```bash
+MODE=output_changed agentprdiff check suite.py
+echo "exit: $?"                     # non-zero
+```
+
+The agent's refund response changes from the baseline phrasing to "Refund initiated. Please allow 7–10 business days for processing." That trips:
+
+- `contains("refund")` — still passes (the new text mentions refund)
+- `contains_any(["business days", "card", "processed"])` — still passes
+- `regex_match(r"\$\d+\.\d{2}")` — **fails** (no dollar amount in the new output)
+- `output_length_lt(500)` — still passes
+- `semantic(...)` — likely **fails** depending on judge backend
+
+The terminal reporter prints a unified output diff so the reviewer can see exactly what changed.
+
+### 2. Extra tool call
+
+```bash
+MODE=tool_added agentprdiff check suite.py
+echo "exit: $?"                     # non-zero
+```
+
+The agent calls `check_inventory` after `lookup_order`. That trips:
+
+- `tool_sequence(["lookup_order"])` — **fails** (sequence is now `["lookup_order", "check_inventory"]`)
+- `no_tool_called("check_inventory")` — **fails**
+
+### 3. Missing tool call
+
+```bash
+MODE=tool_removed agentprdiff check suite.py
+echo "exit: $?"                     # non-zero
+```
+
+The agent never calls `lookup_order` and produces a fallback "trouble looking up your order" response. That trips:
+
+- `tool_called("lookup_order")` — **fails**
+- `tool_sequence(["lookup_order"])` — **fails**
+- `contains("refund")` — **fails** (output text changed)
+- `regex_match(r"\$\d+\.\d{2}")` — **fails**
+
+### 4. Tool order swapped
+
+```bash
+MODE=tool_reordered agentprdiff check suite.py
+echo "exit: $?"                     # non-zero
+```
+
+The agent calls `check_inventory` *before* `lookup_order`. Same tools, wrong order:
+
+- `tool_sequence(["lookup_order"])` — **fails** (sequence is `["check_inventory", "lookup_order"]`)
+- `no_tool_called("check_inventory")` — **fails**
+
+### 5. Latency regression
+
+```bash
+MODE=latency_regressed agentprdiff check suite.py
+echo "exit: $?"                     # non-zero
+```
+
+Planner LLM call jumps from 180 ms to 8 s, blowing past the 5 s cap:
+
+- `latency_lt_ms(5_000)` — **fails**
+
+The reporter shows the latency delta against baseline.
+
+### 6. Cost regression
+
+```bash
+MODE=cost_regressed agentprdiff check suite.py
+echo "exit: $?"                     # non-zero
+```
+
+Responder cost jumps from $0.0008 to $0.10 per call:
+
+- `cost_lt_usd(0.01)` — **fails**
+
+The reporter shows the cost delta against baseline.
+
+## Run them all back to back
+
+```bash
+./tour.sh
+```
+
+Runs every scenario with banners, prints exit codes, and gives you a one-screen overview of what the tool detects.
+
+## Resetting the baseline
+
+If you want to make the new behavior the new baseline (e.g., you intentionally changed the agent), re-record:
+
+```bash
+MODE=output_changed agentprdiff record suite.py     # baseline now matches the changed output
+agentprdiff check suite.py                          # passes against the new baseline
+```
+
+This is the workflow you'll use in real projects when a model upgrade or prompt change is intentional.
+
+## Why this matters
+
+The point of `agentprdiff` is that every one of these scenarios should be caught **before** the change reaches production. Run `agentprdiff check` in CI on every PR and the merge is blocked when behavior changes — same as `pytest` blocks merges when tests break.

{agentprdiff-0.1.0 → agentprdiff-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agentprdiff"
-version = "0.1.0"
+version = "0.2.0"
 description = "Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change."
 readme = "README.md"
 requires-python = ">=3.10"

{agentprdiff-0.1.0 → agentprdiff-0.2.0}/src/agentprdiff/__init__.py

@@ -30,6 +30,22 @@ Run from the shell::
     agentprdiff init
     agentprdiff record path/to/my_suite.py     # save baselines
     agentprdiff check  path/to/my_suite.py     # diff against baselines; exit 1 on regression
+
+If your agent already uses the OpenAI Python SDK (or any OpenAI-compatible
+provider — Groq, Gemini, OpenRouter, Ollama, vLLM) or the Anthropic SDK, the
+SDK adapters capture every model and tool call automatically, no manual Trace
+wiring required::
+
+    from agentprdiff.adapters.openai import instrument_client, instrument_tools
+
+    def my_agent(query):
+        client = OpenAI(...)
+        with instrument_client(client) as trace:
+            tools = instrument_tools(TOOL_MAP, trace)
+            # ... your existing tool-calling loop, untouched ...
+            return final_text, trace
+
+See ``docs/adapters.md`` for the full reference.
 """
 
 from __future__ import annotations
@@ -51,7 +67,7 @@ from .differ import AssertionChange, TraceDelta, diff_traces
 from .runner import CaseReport, Runner, RunReport
 from .store import BaselineStore
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 
 __all__ = [
     # core

agentprdiff-0.2.0/src/agentprdiff/adapters/__init__.py

@@ -0,0 +1,43 @@
+"""SDK adapters for agentprdiff.
+
+The adapters take an agent that uses a real LLM SDK (OpenAI, Anthropic, or any
+OpenAI-compatible provider like Groq, Gemini's openai-compat endpoint,
+OpenRouter, Ollama, or vLLM) and capture every model call as an `LLMCall` on a
+`Trace` — without forcing the user to rewrite their agent loop.
+
+The pattern is::
+
+    from agentprdiff.adapters.openai import instrument_client, instrument_tools
+
+    def my_agent(query: str):
+        client = OpenAI(...)
+        with instrument_client(client) as trace:
+            tools = instrument_tools(TOOL_MAP, trace)
+            # ... user's existing tool-calling loop, untouched ...
+            return final_text, trace
+
+Submodules are imported lazily so the base `agentprdiff` install doesn't pull
+in `openai` / `anthropic` unless the user opts in via the extras::
+
+    pip install "agentprdiff[openai]"
+    pip install "agentprdiff[anthropic]"
+
+See `docs/adapters.md` for the full reference.
+"""
+
+from __future__ import annotations
+
+# Re-export pricing helpers — these are pure-Python and have no SDK dependency.
+from .pricing import (
+    DEFAULT_PRICES,
+    PriceTable,
+    estimate_cost_usd,
+    register_prices,
+)
+
+__all__ = [
+    "DEFAULT_PRICES",
+    "PriceTable",
+    "estimate_cost_usd",
+    "register_prices",
+]

agentprdiff-0.2.0/src/agentprdiff/adapters/anthropic.py

@@ -0,0 +1,191 @@
+"""Anthropic Messages API adapter.
+
+The Messages API has a different shape from OpenAI's Chat Completions:
+
+* Output is a list of content blocks (``text``, ``tool_use``, ``thinking``,
+  ...) on ``response.content``, not ``response.choices[0].message.content``.
+* Token usage is ``response.usage.input_tokens`` / ``output_tokens``.
+* Tool calls live as ``tool_use`` blocks inside ``response.content``, each with
+  an ``id``, ``name``, and ``input`` dict.
+
+The user's loop pattern is also different — they iterate the content blocks,
+execute matching tools, and feed back ``tool_result`` blocks. We don't try to
+hide that; we just record whatever Anthropic returns, on the same ``Trace``
+data model the rest of agentprdiff uses.
+
+Usage::
+
+    from anthropic import Anthropic
+    from agentprdiff.adapters.anthropic import instrument_client, instrument_tools
+
+    def my_agent(query: str):
+        client = Anthropic()
+        with instrument_client(client) as trace:
+            tools = instrument_tools(TOOL_MAP, trace)
+            # ...standard Anthropic Messages tool-use loop...
+            return final_text, trace
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Callable, Iterator, Mapping
+from contextlib import contextmanager, suppress
+from typing import Any
+
+from ..core import LLMCall, Trace
+from .openai import _jsonable, _make_tool_wrapper, _serialize_messages
+from .pricing import PriceTable, estimate_cost_usd
+
+
+def _extract_anthropic_blocks(content: Any) -> tuple[str, list[dict[str, Any]]]:
+    """Walk Anthropic content blocks and extract output text + tool_use calls.
+
+    Returns ``(output_text, tool_calls_summary)``. Any thinking / redacted /
+    unknown block types are quietly ignored — they're not asserted against by
+    any current grader.
+    """
+    output_parts: list[str] = []
+    tool_calls: list[dict[str, Any]] = []
+    if not content:
+        return "", []
+    for block in content:
+        btype = getattr(block, "type", None)
+        if btype is None and isinstance(block, dict):
+            btype = block.get("type")
+        if btype == "text":
+            text = getattr(block, "text", None)
+            if text is None and isinstance(block, dict):
+                text = block.get("text", "")
+            output_parts.append(text or "")
+        elif btype == "tool_use":
+            name = getattr(block, "name", None)
+            inputs = getattr(block, "input", None)
+            tu_id = getattr(block, "id", None)
+            if name is None and isinstance(block, dict):
+                name = block.get("name")
+                inputs = block.get("input")
+                tu_id = block.get("id")
+            if name:
+                tool_calls.append(
+                    {
+                        "id": tu_id,
+                        "name": name,
+                        "arguments": inputs or {},
+                    }
+                )
+    return "".join(output_parts), tool_calls
+
+
+@contextmanager
+def instrument_client(
+    client: Any,
+    *,
+    trace: Trace | None = None,
+    prices: PriceTable | None = None,
+    provider: str | None = None,
+) -> Iterator[Trace]:
+    """Patch ``client.messages.create`` to record onto a Trace.
+
+    See the OpenAI adapter docstring for parameter semantics — they match.
+    """
+    if trace is None:
+        trace = Trace(suite_name="", case_name="", input=None)
+    provider_str = provider or "anthropic"
+
+    messages_attr = getattr(client, "messages", None)
+    if messages_attr is None or not hasattr(messages_attr, "create"):
+        raise TypeError(
+            "instrument_client expected an Anthropic client with "
+            "client.messages.create; got "
+            f"{type(client).__name__}."
+        )
+
+    original_create: Callable[..., Any] = messages_attr.create
+    had_instance_attr = "create" in vars(messages_attr)
+    instance_attr_value = vars(messages_attr).get("create")
+
+    def patched_create(*args: Any, **kwargs: Any) -> Any:
+        start = time.perf_counter()
+        try:
+            response = original_create(*args, **kwargs)
+        except Exception as exc:  # noqa: BLE001
+            elapsed_ms = (time.perf_counter() - start) * 1000.0
+            trace.record_llm_call(
+                LLMCall(
+                    provider=provider_str,
+                    model=str(kwargs.get("model", "")),
+                    input_messages=_serialize_messages(kwargs.get("messages")),
+                    output_text=f"<exception: {type(exc).__name__}: {exc}>",
+                    latency_ms=elapsed_ms,
+                )
+            )
+            raise
+
+        elapsed_ms = (time.perf_counter() - start) * 1000.0
+
+        usage = getattr(response, "usage", None)
+        prompt_tokens = int(getattr(usage, "input_tokens", 0) or 0)
+        completion_tokens = int(getattr(usage, "output_tokens", 0) or 0)
+        model_id = str(getattr(response, "model", "") or kwargs.get("model", "") or "")
+
+        content = getattr(response, "content", None)
+        output_text, tool_calls_summary = _extract_anthropic_blocks(content)
+
+        cost = estimate_cost_usd(
+            model_id,
+            prompt_tokens=prompt_tokens,
+            completion_tokens=completion_tokens,
+            prices=prices,
+        )
+
+        trace.record_llm_call(
+            LLMCall(
+                provider=provider_str,
+                model=model_id,
+                input_messages=_serialize_messages(kwargs.get("messages")),
+                output_text=output_text,
+                tool_calls=tool_calls_summary,
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                cost_usd=cost,
+                latency_ms=elapsed_ms,
+            )
+        )
+        return response
+
+    messages_attr.create = patched_create  # type: ignore[method-assign]
+    try:
+        yield trace
+    finally:
+        if had_instance_attr:
+            messages_attr.create = instance_attr_value  # type: ignore[method-assign]
+        else:
+            with suppress(AttributeError):
+                del messages_attr.create  # type: ignore[attr-defined]
+
+
+def instrument_tools(
+    tool_map: Mapping[str, Callable[..., Any]],
+    trace: Trace,
+) -> dict[str, Callable[..., Any]]:
+    """Wrap each tool callable to record a ``ToolCall`` per invocation.
+
+    Identical semantics to the OpenAI adapter version — the data model is
+    SDK-agnostic. We re-export here so adopters can do the natural::
+
+        from agentprdiff.adapters.anthropic import instrument_client, instrument_tools
+    """
+    wrapped: dict[str, Callable[..., Any]] = {}
+    for name, fn in tool_map.items():
+        wrapped[name] = _make_tool_wrapper(name, fn, trace)
+    return wrapped
+
+
+# Re-export the helpers so tests / advanced users don't have to import from
+# the OpenAI module explicitly when they're already in the Anthropic adapter.
+__all__ = [
+    "instrument_client",
+    "instrument_tools",
+    "_jsonable",
+]

agentprdiff-0.2.0/src/agentprdiff/adapters/openai.py

@@ -0,0 +1,343 @@
+"""OpenAI / OpenAI-compatible adapter.
+
+The adapter monkey-patches ``client.chat.completions.create`` for the duration
+of a ``with`` block, recording one ``LLMCall`` per invocation onto a ``Trace``.
+The patch is reversed on exit, so the user's client object behaves identically
+outside the ``with`` block.
+
+This adapter works with any SDK that uses the OpenAI Python client shape:
+
+* OpenAI itself
+* Groq (``base_url="https://api.groq.com/openai/v1"``)
+* Google Gemini's OpenAI-compatible endpoint
+* OpenRouter
+* Ollama (``base_url="http://localhost:11434/v1"``)
+* vLLM, Together, Fireworks, DeepInfra, Anyscale, etc.
+
+Usage::
+
+    from agentprdiff.adapters.openai import instrument_client, instrument_tools
+
+    def my_agent(query: str):
+        client = OpenAI(api_key=...)
+        with instrument_client(client) as trace:
+            tools = instrument_tools(TOOL_MAP, trace)
+            # ...standard OpenAI tool-calling loop, unchanged...
+            return final_text, trace
+
+The Trace's ``suite_name`` / ``case_name`` / ``input`` are filled in by
+``run_agent`` after the agent returns; you can leave them blank inside the
+adapter.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Callable, Iterator, Mapping
+from contextlib import contextmanager, suppress
+from typing import Any
+
+from ..core import LLMCall, ToolCall, Trace
+from .pricing import PriceTable, estimate_cost_usd
+
+
+def _infer_provider_from_client(client: Any) -> str:
+    """Best-effort guess at the underlying provider from the client's base_url.
+
+    Falls back to ``"openai-compatible"`` if we can't tell. The provider string
+    only affects the recorded `LLMCall.provider` field — it doesn't change
+    behavior — so a fuzzy match is fine.
+    """
+    base_url = ""
+    try:
+        base_url = str(getattr(client, "base_url", "") or "")
+    except Exception:  # noqa: BLE001
+        return "openai-compatible"
+
+    url = base_url.lower()
+    if "groq" in url:
+        return "groq"
+    if "openrouter" in url:
+        return "openrouter"
+    if "googleapis" in url or "generativelanguage" in url:
+        return "gemini"
+    if "ollama" in url or "11434" in url:
+        return "ollama"
+    if "together" in url:
+        return "together"
+    if "fireworks" in url:
+        return "fireworks"
+    if "deepinfra" in url:
+        return "deepinfra"
+    if "anthropic" in url:
+        # Anthropic's OpenAI-compat shim. Use the native adapter instead for
+        # full fidelity, but still flag it.
+        return "anthropic-openai-compat"
+    if "openai" in url or url == "":
+        return "openai"
+    return "openai-compatible"
+
+
+def _extract_tool_calls(message: Any) -> list[dict[str, Any]]:
+    """Pull tool_calls off a ChatCompletionMessage in a defensive way."""
+    raw = getattr(message, "tool_calls", None) or []
+    out: list[dict[str, Any]] = []
+    for tc in raw:
+        # OpenAI SDK objects are pydantic-like; fall through to dict access too.
+        try:
+            fn = tc.function
+            name = getattr(fn, "name", None)
+            arguments = getattr(fn, "arguments", None)
+            tc_id = getattr(tc, "id", None)
+        except AttributeError:
+            try:
+                fn = tc.get("function", {})
+                name = fn.get("name")
+                arguments = fn.get("arguments")
+                tc_id = tc.get("id")
+            except Exception:  # noqa: BLE001
+                continue
+        if name is None:
+            continue
+        out.append({"id": tc_id, "name": name, "arguments": arguments})
+    return out
+
+
+def _serialize_messages(messages: Any) -> list[dict[str, Any]]:
+    """Best-effort JSON-friendly copy of the request messages.
+
+    Trace baselines must be JSON-serializable, and `messages` is sometimes a
+    list of pydantic objects, sometimes a list of plain dicts. Prefer dicts;
+    skip anything we can't represent cleanly.
+    """
+    if not messages:
+        return []
+    out: list[dict[str, Any]] = []
+    for m in messages:
+        if isinstance(m, dict):
+            out.append(m)
+            continue
+        # pydantic-ish?
+        for attr in ("model_dump", "dict"):
+            dump = getattr(m, attr, None)
+            if callable(dump):
+                try:
+                    out.append(dump())
+                    break
+                except Exception:  # noqa: BLE001
+                    pass
+        else:
+            # Last resort — string repr so the trace round-trips.
+            out.append({"_repr": repr(m)})
+    return out
+
+
+@contextmanager
+def instrument_client(
+    client: Any,
+    *,
+    trace: Trace | None = None,
+    prices: PriceTable | None = None,
+    provider: str | None = None,
+) -> Iterator[Trace]:
+    """Patch ``client.chat.completions.create`` to record onto a Trace.
+
+    Yields the ``Trace`` so the caller can return ``(output, trace)`` from
+    their agent function. The patch is restored on exit even if the agent
+    raises.
+
+    Parameters
+    ----------
+    client:
+        An ``openai.OpenAI`` (or compatible) client instance. We patch a bound
+        attribute on this specific instance — global SDK state is untouched.
+    trace:
+        Optional pre-existing ``Trace`` to record into. Useful when nesting
+        adapters or stitching together a multi-stage agent. If omitted, we
+        create a fresh one with empty suite/case/input fields (the runner
+        fills them in after the agent returns).
+    prices:
+        Optional override for the model→price table. See
+        :mod:`agentprdiff.adapters.pricing`.
+    provider:
+        Optional explicit provider tag; otherwise we infer from
+        ``client.base_url``.
+    """
+    if trace is None:
+        trace = Trace(suite_name="", case_name="", input=None)
+    provider_str = provider or _infer_provider_from_client(client)
+
+    # Locate the create function we need to patch. Newer SDKs:
+    #   client.chat.completions.create
+    chat = getattr(client, "chat", None)
+    completions = getattr(chat, "completions", None) if chat is not None else None
+    if completions is None or not hasattr(completions, "create"):
+        raise TypeError(
+            "instrument_client expected an OpenAI-style client with "
+            "client.chat.completions.create; got "
+            f"{type(client).__name__}. If you're using a non-OpenAI SDK, see "
+            "agentprdiff/adapters/anthropic.py or open an issue."
+        )
+
+    # Stash the original so we can call through, plus remember whether the
+    # attribute was carried as an instance attr or was descriptor-resolved
+    # from the class. We need that to cleanly restore on exit (del-on-exit if
+    # it wasn't originally an instance attr; assign-on-exit if it was).
+    original_create: Callable[..., Any] = completions.create
+    had_instance_attr = "create" in vars(completions)
+    instance_attr_value = vars(completions).get("create")
+
+    def patched_create(*args: Any, **kwargs: Any) -> Any:
+        start = time.perf_counter()
+        try:
+            response = original_create(*args, **kwargs)
+        except Exception as exc:  # noqa: BLE001
+            elapsed_ms = (time.perf_counter() - start) * 1000.0
+            # Record a failed LLMCall so the trace shows what happened.
+            trace.record_llm_call(
+                LLMCall(
+                    provider=provider_str,
+                    model=str(kwargs.get("model", "")),
+                    input_messages=_serialize_messages(kwargs.get("messages")),
+                    output_text=f"<exception: {type(exc).__name__}: {exc}>",
+                    latency_ms=elapsed_ms,
+                )
+            )
+            raise
+
+        elapsed_ms = (time.perf_counter() - start) * 1000.0
+
+        # Pull usage / model / output safely; some compatible servers omit
+        # fields we'd like to have.
+        usage = getattr(response, "usage", None)
+        prompt_tokens = int(getattr(usage, "prompt_tokens", 0) or 0)
+        completion_tokens = int(getattr(usage, "completion_tokens", 0) or 0)
+        model_id = str(getattr(response, "model", "") or kwargs.get("model", "") or "")
+
+        choices = getattr(response, "choices", None) or []
+        message = getattr(choices[0], "message", None) if choices else None
+        output_text = getattr(message, "content", None) or "" if message is not None else ""
+        tool_calls_summary = _extract_tool_calls(message) if message is not None else []
+
+        cost = estimate_cost_usd(
+            model_id,
+            prompt_tokens=prompt_tokens,
+            completion_tokens=completion_tokens,
+            prices=prices,
+        )
+
+        trace.record_llm_call(
+            LLMCall(
+                provider=provider_str,
+                model=model_id,
+                input_messages=_serialize_messages(kwargs.get("messages")),
+                output_text=output_text,
+                tool_calls=tool_calls_summary,
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                cost_usd=cost,
+                latency_ms=elapsed_ms,
+            )
+        )
+        return response
+
+    # Apply the patch on this specific instance only.
+    completions.create = patched_create  # type: ignore[method-assign]
+    try:
+        yield trace
+    finally:
+        if had_instance_attr:
+            completions.create = instance_attr_value  # type: ignore[method-assign]
+        else:
+            # Drop the instance attribute so the original class-level
+            # descriptor (the bound method) shines through again.
+            # Defensive: someone else already cleaned up → no-op.
+            with suppress(AttributeError):
+                del completions.create  # type: ignore[attr-defined]
+
+
+def instrument_tools(
+    tool_map: Mapping[str, Callable[..., Any]],
+    trace: Trace,
+) -> dict[str, Callable[..., Any]]:
+    """Wrap each callable in ``tool_map`` so invocations record ``ToolCall``s.
+
+    Returns a new dict — the original is untouched. Use it the same way you'd
+    use the original::
+
+        tools = instrument_tools(TOOL_MAP, trace)
+        result = tools[fn_name](**fn_args)
+
+    Each call records:
+
+    * ``name`` — the dict key
+    * ``arguments`` — the kwargs (and positional args under ``"_args"`` if any)
+    * ``result`` — the return value (or None if the call raised)
+    * ``latency_ms`` — wall-clock latency
+    * ``error`` — exception text on failure
+    """
+    wrapped: dict[str, Callable[..., Any]] = {}
+    for name, fn in tool_map.items():
+        wrapped[name] = _make_tool_wrapper(name, fn, trace)
+    return wrapped
+
+
+def _make_tool_wrapper(
+    name: str,
+    fn: Callable[..., Any],
+    trace: Trace,
+) -> Callable[..., Any]:
+    def _wrapped(*args: Any, **kwargs: Any) -> Any:
+        start = time.perf_counter()
+        arguments: dict[str, Any] = dict(kwargs)
+        if args:
+            arguments["_args"] = list(args)
+        try:
+            result = fn(*args, **kwargs)
+        except Exception as exc:  # noqa: BLE001
+            elapsed_ms = (time.perf_counter() - start) * 1000.0
+            trace.record_tool_call(
+                ToolCall(
+                    name=name,
+                    arguments=arguments,
+                    result=None,
+                    latency_ms=elapsed_ms,
+                    error=f"{type(exc).__name__}: {exc}",
+                )
+            )
+            raise
+        elapsed_ms = (time.perf_counter() - start) * 1000.0
+        trace.record_tool_call(
+            ToolCall(
+                name=name,
+                arguments=arguments,
+                result=_jsonable(result),
+                latency_ms=elapsed_ms,
+            )
+        )
+        return result
+
+    _wrapped.__name__ = f"instrumented_{name}"
+    _wrapped.__doc__ = getattr(fn, "__doc__", None)
+    return _wrapped
+
+
+def _jsonable(value: Any) -> Any:
+    """Best-effort coerce a tool's return value to something JSON-serializable.
+
+    Pydantic models get model_dump'd; primitive types, lists, and dicts pass
+    through; anything else falls back to repr.
+    """
+    if value is None or isinstance(value, (bool, int, float, str)):
+        return value
+    if isinstance(value, (list, tuple)):
+        return [_jsonable(v) for v in value]
+    if isinstance(value, dict):
+        return {str(k): _jsonable(v) for k, v in value.items()}
+    dump = getattr(value, "model_dump", None)
+    if callable(dump):
+        try:
+            return dump(mode="json")
+        except Exception:  # noqa: BLE001
+            pass
+    return repr(value)

agentprdiff-0.2.0/src/agentprdiff/adapters/pricing.py

@@ -0,0 +1,136 @@
+"""Per-model price table used by SDK adapters to fill in `LLMCall.cost_usd`.
+
+The shape of an entry is::
+
+    "model-id": (input_usd_per_1k_tokens, output_usd_per_1k_tokens)
+
+Prices change. We ship a curated default table that's accurate at release time
+and intentionally easy to override:
+
+* Per-call:   pass `prices=` to `instrument_client(...)`.
+* Per-model:  call `register_prices({"my-model": (0.001, 0.002)})` once at
+              import time.
+* Globally:   replace `DEFAULT_PRICES` with your own dict.
+
+If a model is not in the table, the adapter records `cost_usd=0.0` and emits
+exactly one `RuntimeWarning` per process per model name, so cost regressions
+based on missing pricing are loud rather than silent.
+
+Sources for the bundled defaults: each provider's published pricing page as of
+2026-04. Submit a PR if you spot drift.
+"""
+
+from __future__ import annotations
+
+import warnings
+from collections.abc import Mapping
+
+# (input_$/1k, output_$/1k)
+PriceTable = Mapping[str, tuple[float, float]]
+
+# Curated defaults. Conservative — if pricing tiers exist (e.g. cached input vs
+# fresh input on Anthropic), we use the headline number. Override per-call for
+# precision.
+DEFAULT_PRICES: dict[str, tuple[float, float]] = {
+    # ── OpenAI ─────────────────────────────────────────────────────────────
+    "gpt-4o":                       (0.0025,   0.0100),
+    "gpt-4o-2024-08-06":            (0.0025,   0.0100),
+    "gpt-4o-mini":                  (0.00015,  0.00060),
+    "gpt-4-turbo":                  (0.0100,   0.0300),
+    "gpt-4":                        (0.0300,   0.0600),
+    "gpt-3.5-turbo":                (0.0005,   0.0015),
+    "o1":                           (0.0150,   0.0600),
+    "o1-mini":                      (0.0030,   0.0120),
+    "o1-preview":                   (0.0150,   0.0600),
+    "o3-mini":                      (0.0011,   0.0044),
+
+    # ── Anthropic ──────────────────────────────────────────────────────────
+    "claude-opus-4-6":              (0.0150,   0.0750),
+    "claude-sonnet-4-6":            (0.0030,   0.0150),
+    "claude-haiku-4-5-20251001":    (0.0008,   0.0040),
+    "claude-3-5-sonnet-20241022":   (0.0030,   0.0150),
+    "claude-3-5-sonnet-latest":     (0.0030,   0.0150),
+    "claude-3-5-haiku-20241022":    (0.0008,   0.0040),
+    "claude-3-5-haiku-latest":      (0.0008,   0.0040),
+    "claude-3-opus-20240229":       (0.0150,   0.0750),
+    "claude-3-sonnet-20240229":     (0.0030,   0.0150),
+    "claude-3-haiku-20240307":      (0.00025,  0.00125),
+
+    # ── Groq (LPU inference of OSS models, OpenAI-compatible API) ─────────
+    "llama-3.3-70b-versatile":      (0.00059,  0.00079),
+    "llama-3.1-70b-versatile":      (0.00059,  0.00079),
+    "llama-3.1-8b-instant":         (0.00005,  0.00008),
+    "mixtral-8x7b-32768":           (0.00024,  0.00024),
+    "gemma2-9b-it":                 (0.00020,  0.00020),
+
+    # ── Google Gemini (via OpenAI-compatible endpoint or native) ──────────
+    "gemini-1.5-pro":               (0.00125,  0.00500),
+    "gemini-1.5-flash":             (0.000075, 0.000300),
+    "gemini-2.0-flash":             (0.00010,  0.00040),
+    "gemini-2.0-flash-exp":         (0.00010,  0.00040),
+
+    # ── OpenRouter (passthrough; varies by upstream model) ────────────────
+    # OpenRouter prefixes upstream IDs as "<provider>/<model>". Add the
+    # specific routes you use; we list a few common ones as starters.
+    "openai/gpt-4o":                (0.0025,   0.0100),
+    "openai/gpt-4o-mini":           (0.00015,  0.00060),
+    "anthropic/claude-3.5-sonnet":  (0.0030,   0.0150),
+    "google/gemini-2.0-flash-001":  (0.00010,  0.00040),
+    "meta-llama/llama-3.3-70b-instruct": (0.00012, 0.00030),
+
+    # ── Ollama (local; cost is electricity, not API spend) ────────────────
+    # Recorded as zero so cost_lt_usd graders pass naturally for local runs.
+    "llama3.1":                     (0.0,      0.0),
+    "llama3.2":                     (0.0,      0.0),
+    "qwen2.5":                      (0.0,      0.0),
+    "mistral":                      (0.0,      0.0),
+}
+
+
+# Track which models we've already warned about so we don't spam logs across a
+# large suite. Keyed per-process; reset by tests via `_reset_warnings()`.
+_warned_models: set[str] = set()
+
+
+def register_prices(prices: PriceTable) -> None:
+    """Merge `prices` into the global default table.
+
+    Useful at the top of a suite file::
+
+        from agentprdiff.adapters import register_prices
+        register_prices({"my-finetune-v3": (0.0009, 0.0018)})
+    """
+    DEFAULT_PRICES.update(prices)
+
+
+def estimate_cost_usd(
+    model: str,
+    *,
+    prompt_tokens: int,
+    completion_tokens: int,
+    prices: PriceTable | None = None,
+) -> float:
+    """Compute USD cost for a single LLM call from token counts.
+
+    Returns 0.0 and warns once per process if the model isn't in the table.
+    """
+    table: PriceTable = prices if prices is not None else DEFAULT_PRICES
+    entry = table.get(model)
+    if entry is None:
+        if model not in _warned_models:
+            _warned_models.add(model)
+            warnings.warn(
+                f"[agentprdiff] no pricing entry for model {model!r}; cost_usd will be "
+                "recorded as 0.0. Pass prices={...} to instrument_client(...) or call "
+                "agentprdiff.adapters.register_prices({...}) to fix.",
+                RuntimeWarning,
+                stacklevel=3,
+            )
+        return 0.0
+    in_price, out_price = entry
+    return (prompt_tokens * in_price + completion_tokens * out_price) / 1000.0
+
+
+def _reset_warnings() -> None:
+    """Test helper — clear the per-process warning memo."""
+    _warned_models.clear()

{agentprdiff-0.1.0 → agentprdiff-0.2.0}/src/agentprdiff/loader.py

@@ -3,12 +3,27 @@
 We deliberately keep this ultra-simple for v0.1: the user points at a python
 file path (or module) and we import it; every module-level `Suite` instance
 is a suite to run.
+
+Import-path setup: when we exec the suite file, we insert two directories
+onto ``sys.path``:
+
+1. The suite file's own parent directory — for sibling helpers, e.g. a
+   ``suite.py`` next to a ``stubs.py``.
+2. The current working directory — typically the project root from which
+   ``agentprdiff record`` was invoked. This lets the suite import the
+   adopter's own modules (``from agent.agent import ...``,
+   ``from config import ...``, ``from suites._eval_agent import ...``)
+   without forcing every adopter to manipulate ``sys.path`` themselves.
+
+Both insertions are reversed after exec so the runner's environment doesn't
+leak suite-specific paths into subsequent loads.
 """
 
 from __future__ import annotations
 
 import contextlib
 import importlib.util
+import os
 import sys
 from pathlib import Path
 
@@ -30,13 +45,23 @@ def load_suites(path: str | Path) -> list[Suite]:
     if spec is None or spec.loader is None:  # pragma: no cover
         raise ImportError(f"could not load suite file: {p}")
     module = importlib.util.module_from_spec(spec)
-    # Ensure the file's own directory is importable (for relative helpers).
-    sys.path.insert(0, str(p.parent))
+
+    # Make both the suite file's directory and the cwd importable. We track
+    # what we actually inserted so we only remove our own contributions.
+    parent_dir = str(p.parent)
+    cwd = os.getcwd()
+    inserted: list[str] = []
+    for entry in (parent_dir, cwd):
+        if entry and entry not in sys.path:
+            sys.path.insert(0, entry)
+            inserted.append(entry)
+
     try:
         spec.loader.exec_module(module)
     finally:
-        with contextlib.suppress(ValueError):
-            sys.path.remove(str(p.parent))
+        for entry in inserted:
+            with contextlib.suppress(ValueError):
+                sys.path.remove(entry)
 
     suites = [v for v in vars(module).values() if isinstance(v, Suite)]
     if not suites:

agentprdiff-0.1.0/CHANGELOG.md

@@ -1,37 +0,0 @@
-# Changelog
-
-All notable changes to `agentprdiff` are documented in this file. Originally
-prototyped under the name `tracediff`; renamed before first public release.
-
-The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
-
-## [0.1.0] — 2026-04-22
-
-Initial public release.
-
-### Added
-
-- Core `Suite` / `Case` / `Trace` model for defining agent regression tests.
-- Deterministic graders: `contains`, `contains_any`, `regex_match`, `tool_called`,
-  `tool_sequence`, `output_length_lt`, `latency_lt_ms`, `cost_lt_usd`,
-  `no_tool_called`.
-- Semantic grader (`semantic`) with a pluggable `judge` callable and built-in
-  fake judge for CI environments without API keys.
-- Baseline store (JSON files under `.agentprdiff/baselines/`) designed to be
-  committed to version control.
-- Trace diff engine producing a structured `TraceDelta` (assertion pass/fail
-  changes, cost delta, latency delta, tool-call sequence changes, output
-  change).
-- CLI: `agentprdiff init`, `agentprdiff record`, `agentprdiff check`, `agentprdiff diff`.
-- Rich-formatted terminal reporter and machine-readable JSON reporter for CI.
-- Quickstart example with a mock agent that runs without any API keys.
-- Pytest test suite covering graders, runner, differ, store, and CLI smoke.
-- GitHub Actions CI workflow.
-
-### Known limitations
-
-- Only a manual instrumentation API for provider SDKs is shipped in 0.1.0.
-  Drop-in wrappers for OpenAI / Anthropic / Vercel AI SDK are planned for 0.2.
-- The semantic grader's built-in judge supports OpenAI and Anthropic via user-
-  supplied API keys; hosted judge endpoints are not yet offered.