agentverify 0.3.0__tar.gz → 0.5.0__tar.gz

{agentverify-0.3.0 → agentverify-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: agentverify
-Version: 0.3.0
+Version: 0.5.0
 Summary: Deterministic testing for AI agents. Assert agent actions, not vibes.
 Author: simukappu
 License: MIT
@@ -19,6 +19,7 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
 Classifier: Topic :: Software Development :: Testing
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
@@ -56,25 +57,36 @@ Dynamic: license-file
 
 **pytest for AI agents.** Assert agent actions, not vibes.
 
-agentverify is a pytest plugin for deterministic testing of AI agent actions. Record real LLM calls once and replay them in CI with zero cost, or run against a live LLM with tolerance-aware assertions. Either way, assert exactly what your agent did: which tools it called, what data flowed between steps, how much it cost, and whether it stayed within your safety rules.
+agentverify is a pytest plugin for deterministic testing of AI agent actions. Record real LLM calls once and replay them in CI with zero cost, or run against a live LLM with flexible matchers that tolerate prompt variation. Either way, assert exactly what your agent did: which tools it called, what data flowed between steps, how much it cost, and whether it stayed within your safety rules.
+
+![agentverify failure output showing the exact mismatch between expected and actual tool calls](docs/screenshot-error-output.png)
 
 Specifically:
 
-- **Tool call sequences** — exact order, subsequence, or set membership, with regex / wildcard argument matching
-- **Step-level execution** — each LLM call is a step; assert what tool was called at step N, what the step's output was, and that step N's input references data produced by step M
-- **Budgets** — token counts, cost in USD, end-to-end latency
-- **Safety** — a list of tools that must never be called, no matter what the LLM decides
+- **Tool call sequences**: exact order, subsequence, or set membership, with regex / wildcard argument matching
+- **Step-level execution**: each LLM call is a step; assert what tool was called at step N, what the step's output was, and that step N's input references data produced by step M
+- **Tool outcomes**: whether each tool call actually returned a usable result (vs. errored, timed out, or came back empty), what the result contained, and how many times a flaky tool was retried
+- **Budgets**: token counts, cost in USD, end-to-end latency
+- **Safety**: a list of tools that must never be called, no matter what the LLM decides
+
+Built-in adapters cover LangChain, LangGraph, Strands Agents, and OpenAI Agents SDK. A small [custom converter](#custom-converters) plugs in anything else, including pure-Python ReAct loops, the Anthropic / OpenAI SDK directly, and frameworks not listed above. LLM providers supported: OpenAI, Amazon Bedrock, Google Gemini, Anthropic, and LiteLLM. Cassettes are human-readable YAML. Commit them to git, review in PRs, run in CI without an API key.
+
+## Why I built this
 
-Built-in adapters cover LangChain, LangGraph, Strands Agents, and OpenAI Agents SDK, and a small custom converter plugs in anything else — pure-Python ReAct loops, the Anthropic / OpenAI SDK directly, or frameworks not listed above. LLM providers supported: OpenAI, Amazon Bedrock, Google Gemini, Anthropic, and LiteLLM. Cassettes are human-readable YAML — commit them to git, review in PRs, run in CI without an API key.
+When I started building agents that call tools in a loop, I ran into a failure mode that is easy to introduce and easy to miss: the agent calls the wrong tool, ignores a tool result, or quietly retries a failing call until the budget runs over. I wanted those caught in CI the way any other regression is. But the usual options did not fit. Mocking everything proves nothing, and running against a real LLM is unreliable, costs money, and needs an API key in CI.
 
-## Who is this for?
+What I wanted was boring. Run the agent once, record what it did, and assert on it in CI like any other function: the tool sequence, the arguments, the cost, the steps where one tool's output feeds the next. Review the recording in the PR next to the prompt change that produced it. No judge model, no scoring, no cloud account. agentverify is deliberately narrow, and the cassette-replay path is what keeps those tests in CI without a key or a bill.
+
+This is probably for you if:
 
 - You've shipped an agent that calls tools in a loop and need CI-safe regression tests for it
-- Your agent calls multiple tools and you've hit bugs like "it ignored the tool result" or "it called the wrong tool"
+- You've hit bugs like "it ignored the tool result" or "it called the wrong tool"
 - You want prompt and tool changes reviewed in PRs via a concrete diff, not an informal "looks fine locally" signoff
 - You're building on MCP servers and need to pin down which tools get called with which arguments
 - You want token, cost, and latency budgets enforced in CI, not caught only on the billing dashboard
 
+This is the **regression-test layer** for agents: deterministic assertions on what the agent did. It's distinct from quality evaluation (LLM-as-judge scoring on open-ended output) and production trace observability, both of which are often used alongside agentverify rather than instead of it. See [How agentverify compares](#how-agentverify-compares) for when to reach for each.
+
 ## Install
 
 ```bash
@@ -83,7 +95,7 @@ pip install agentverify
 
 ## Quick Start
 
-Copy this into `test_agent.py` and run `pytest`. No LLM calls, no API keys, no cassettes needed — just pure assertions on an `ExecutionResult`.
+After [installing](#install), copy this into `test_agent.py` and run `pytest`. No LLM calls, no API keys, no cassettes needed. Just pure assertions on an `ExecutionResult`.
 
 ```python
 from agentverify import (
@@ -118,8 +130,6 @@ def test_output():
     assert_final_output(result, contains="Tokyo")
 ```
 
-> **Using a real agent framework?** Walk through the [Real-World Examples](#real-world-examples) below for end-to-end tests with cassettes, or jump straight to the [Examples](#examples) table for the full list of runnable samples.
-
 ```
 $ pytest test_agent.py -v
 test_agent.py::test_tool_sequence PASSED
@@ -128,13 +138,15 @@ test_agent.py::test_safety PASSED
 test_agent.py::test_output PASSED
 ```
 
+> **Using a real agent framework?** Walk through the [Real-World Examples](#real-world-examples) below for in-depth walkthroughs with cassettes, or jump to the [Examples](#examples) table for a quick index of all runnable samples.
+
 ## Real-World Examples
 
-End-to-end examples that showcase different agent patterns and testing angles. All run with pre-recorded cassettes — no API keys needed.
+End-to-end examples that showcase different agent patterns and testing angles. All run with pre-recorded cassettes, no API keys needed.
 
 ### Strands Weather Forecaster — Two-Step ReAct
 
-Test the [Strands Weather Forecaster](https://strandsagents.com/docs/examples/python/weather_forecaster/) — an official Strands Agents sample. The agent uses the `http_request` tool to call the National Weather Service API, which returns the forecast URL for the requested location; the agent then calls that URL to get the actual forecast. Two steps, one piece of data flowing between them — the simplest shape of a ReAct loop.
+Test the [Strands Weather Forecaster](https://strandsagents.com/docs/examples/python/weather_forecaster/), an official Strands Agents sample. The agent uses the `http_request` tool to call the National Weather Service API, which returns the forecast URL for the requested location; the agent then calls that URL to get the actual forecast. Two steps, one piece of data flowing between them: the simplest shape of a ReAct loop.
 
 ```python
 import pytest
@@ -158,7 +170,7 @@ def test_weather_agent(cassette):
     result = rec.to_execution_result()
     assert_all(
         result,
-        # Two http_request calls with distinct URL shapes — location lookup then forecast
+        # Two http_request calls with distinct URL shapes: location lookup then forecast
         lambda r: assert_tool_calls(r, expected=[
             ToolCall("http_request", {"method": "GET", "url": MATCHES(r"/points/")}),
             ToolCall("http_request", {"method": "GET", "url": MATCHES(r"/forecast")}),
@@ -193,11 +205,11 @@ def test_weather_agent_steps(cassette):
     assert_step_uses_result_from(result, step=1, depends_on=0)
 ```
 
-`MATCHES(pattern)` is a regex matcher — see [Assertion Modes](#assertion-modes) for details on `ANY`, `MATCHES`, `OrderMode`, and `partial_args`. Step assertions are covered in [Step-Level Assertions](#step-level-assertions).
+`MATCHES(pattern)` is a regex matcher. See [Assertion Modes](#assertion-modes) for details on `ANY`, `MATCHES`, `OrderMode`, and `partial_args`. Step assertions are covered in [Step-Level Assertions](#step-level-assertions).
 
 ### OpenAI Agents SDK LLM as a Judge — Freezing a Probabilistic Refinement Loop
 
-Test the [OpenAI Agents SDK `agent_patterns/llm_as_a_judge.py`](https://github.com/openai/openai-agents-python/blob/main/examples/agent_patterns/llm_as_a_judge.py) sample: a generator writes a story outline, an evaluator grades it with a structured `{feedback, score}` verdict, and the loop re-runs the generator with that feedback until the evaluator accepts the draft. Everything about this agent is probabilistic — how many rounds it takes, what the feedback says, whether the rewrite incorporates it. Recording one run freezes all of it into a deterministic test.
+Test the [OpenAI Agents SDK `agent_patterns/llm_as_a_judge.py`](https://github.com/openai/openai-agents-python/blob/main/examples/agent_patterns/llm_as_a_judge.py) sample: a generator writes a story outline, an evaluator grades it with a structured `{feedback, score}` verdict, and the loop re-runs the generator with that feedback until the evaluator accepts the draft. Everything about this agent is probabilistic: how many rounds it takes, what the feedback says, whether the rewrite incorporates it. Recording one run freezes all of it into a deterministic test.
 
 The headline assertion is the feedback chain: round 2's generator must actually consume round 1's evaluator feedback, not silently regenerate the same outline.
 
@@ -221,12 +233,14 @@ def test_generator_consumes_evaluator_feedback(cassette):
     assert_step_output(
         result, step=1, matches=r'"score"\s*:\s*"(needs_improvement|fail)"'
     )
-    # Step 2 (generator, round 2) must actually consume step 1's feedback —
-    # catches "agent regenerated but ignored the judge" bugs.
+    # Step 2 (generator, round 2) must actually consume step 1's feedback.
+    # This catches "agent regenerated but ignored the judge" bugs.
     assert_step_uses_result_from(result, step=2, depends_on=1)
 ```
 
-`assert_step_uses_result_from` finds the multi-line evaluator feedback inside the next step's user message even when JSON serialization would escape the newlines; the match succeeds across that boundary. Because each `Runner.run` call is recorded individually, the example builds the `ExecutionResult` straight off the cassette recorder — `from_openai_agents` is the right adapter for single-run agents, but the cassette layer is the natural single source of truth when the workflow spans several runs.
+`assert_step_uses_result_from` matches across JSON-serialization boundaries. The multi-line evaluator feedback is found inside the next step's user message even when newlines are escaped during serialization.
+
+Because each `Runner.run` call is recorded individually and the workflow spans several runs, the test builds the `ExecutionResult` straight off the cassette recorder rather than the single-run `from_openai_agents` adapter. The cassette layer is the natural single source of truth when a test covers multiple agent runs.
 
 See [`examples/openai-agents-llm-as-a-judge/`](examples/openai-agents-llm-as-a-judge/) for the full loop, the cassette, and the remaining flat + step-level assertions (budget, safety, pass-verdict reached).
 
@@ -234,27 +248,27 @@ See [`examples/openai-agents-llm-as-a-judge/`](examples/openai-agents-llm-as-a-j
 
 Full-depth examples for the LangChain ecosystem live under [`examples/`](examples/):
 
-- **[LangGraph multi-agent supervisor](examples/langgraph-multi-agent-supervisor/)** — a supervisor routes to a research expert and a math expert; each `add` in the math chain must consume the previous `add`'s result. `assert_step_uses_result_from` matches the numeric running total (`"231317.0"` produced as a string, consumed as `231317` int/float) across type boundaries, with digit-boundary checks that keep `"231"` from falsely appearing inside `1231`.
-- **[LangChain GitHub issue triage](examples/langchain-issue-triage/)** — a `create_react_agent` drives the GitHub MCP server to list issues, read one, propose labels. The step-level tests verify that the `issue_number` passed to `get_issue` actually came from the preceding `list_issues` response, and that destructive tools like `close_issue` are never called.
+- **[LangGraph multi-agent supervisor](examples/langgraph-multi-agent-supervisor/)**: a supervisor routes to a research expert and a math expert; each `add` in the math chain must consume the previous `add`'s result. `assert_step_uses_result_from` matches the numeric running total (`"231317.0"` produced as a string, consumed as `231317` int/float) across type boundaries, with digit-boundary checks that keep `"231"` from falsely appearing inside `1231`.
+- **[LangChain GitHub issue triage](examples/langchain-issue-triage/)**: a `create_react_agent` drives the GitHub MCP server to list issues, read one, propose labels. The step-level tests verify that the `issue_number` passed to `get_issue` actually came from the preceding `list_issues` response, that the GitHub reads returned usable results (`assert_no_tool_errors`, `assert_tool_result_matches`), and that destructive tools like `close_issue` are never called.
 
-Each example ships with its own pre-recorded cassette, a detailed `README.md`, and a full test file — pick whichever framework fits your stack.
+Each example ships with its own pre-recorded cassette, a detailed `README.md`, and a full test file. Pick whichever framework fits your stack.
 
 ## Build an ExecutionResult
 
 Every agentverify assertion takes an `ExecutionResult`. You can build one three ways:
 
-1. **From a dict** — convenient for quick tests and fixtures, as in the [Quick Start](#quick-start).
-2. **From a built-in adapter** — one-liner for LangChain, LangGraph, Strands Agents, OpenAI Agents SDK. See [Framework Integration](#framework-integration).
-3. **From a custom converter** — for frameworks or pure-Python agents not covered by a built-in adapter, map your output to the schema below. See [`examples/custom-converter-python-agent/`](examples/custom-converter-python-agent/) for a complete walkthrough with an Anthropic SDK agent, including a pre-recorded cassette and step-level tests.
+1. **From a dict**: convenient for quick tests and fixtures, as in the [Quick Start](#quick-start).
+2. **From a built-in adapter**: one-liner for LangChain, LangGraph, Strands Agents, OpenAI Agents SDK. See [Framework Integration](#framework-integration).
+3. **From a custom converter**: for frameworks or pure-Python agents not covered by a built-in adapter, map your output to the schema below. See [`examples/custom-converter-python-agent/`](examples/custom-converter-python-agent/) for a complete walkthrough with an Anthropic SDK agent, including a pre-recorded cassette and step-level tests.
 
 `ExecutionResult.from_dict()` accepts these keys:
 
 | Key | Type | Description |
 |---|---|---|
 | `steps` | `list[dict]` | Each step dict has `index` (int), `source` (`"llm"` / `"probe"` / `"tool"`), `tool_calls` (list), `tool_results` (list), `output` (str or None), `input_context` (dict or None), `name` (str or None). See [Step-Level Assertions](#step-level-assertions) |
-| `tool_calls` | `list[dict]` | **Shortcut for single-step cases** — when `steps` is absent, a flat `tool_calls` list is wrapped into a single synthetic step. Used throughout the [Quick Start](#quick-start). Each dict has `name` (str, required), `arguments` (dict, optional), `result` (any, optional) |
+| `tool_calls` | `list[dict]` | **Shortcut for single-step cases**: when `steps` is absent, a flat `tool_calls` list is wrapped into a single synthetic step. Used throughout the [Quick Start](#quick-start). Each dict has `name` (str, required), `arguments` (dict, optional), `result` (any, optional) |
 | `token_usage` | `dict` or `None` | `{"input_tokens": int, "output_tokens": int}` |
-| `total_cost_usd` | `float` or `None` | Total cost in USD (must be set manually — not auto-calculated from tokens or populated from cassettes) |
+| `total_cost_usd` | `float` or `None` | Total cost in USD (must be set manually, not auto-calculated from tokens or populated from cassettes) |
 | `duration_ms` | `float` or `None` | Wall-clock duration in milliseconds (auto-populated by the `cassette` fixture and `MockLLM`) |
 | `final_output` | `str` or `None` | The agent's final text response |
 
@@ -262,7 +276,7 @@ Every agentverify assertion takes an `ExecutionResult`. You can build one three
 
 ## Record & Replay with Cassettes
 
-Record real LLM API calls once. Replay them in CI forever — zero cost, deterministic.
+Record real LLM API calls once, then replay them in CI forever. Zero cost, fully deterministic.
 
 ```python
 import pytest
@@ -283,7 +297,7 @@ def test_weather_agent(cassette):
     ])
 ```
 
-The `cassette` fixture is provided by the agentverify pytest plugin. It creates an `LLMCassetteRecorder` that intercepts LLM SDK calls (not HTTP — it patches the SDK's chat completion method directly). Use `@pytest.mark.agentverify` to mark your test, and call your agent code inside the `with cassette(...)` block. After the block exits, call `rec.to_execution_result()` to build the result for assertions.
+The `cassette` fixture is provided by the agentverify pytest plugin. It creates an `LLMCassetteRecorder` that intercepts LLM SDK calls (not HTTP; it patches the SDK's chat completion method directly). Use `@pytest.mark.agentverify` to mark your test, and call your agent code inside the `with cassette(...)` block. After the block exits, call `rec.to_execution_result()` to build the result for assertions.
 
 Record the cassette once, then replay it forever:
 
@@ -297,7 +311,7 @@ pytest
 
 Cassettes are human-readable YAML (or JSON). Commit them to git, review in PRs.
 
-**Cassette modes:**
+agentverify supports three cassette modes:
 
 | Mode | Behavior |
 |---|---|
@@ -332,25 +346,40 @@ Cassettes drift when your prompts, tool schemas, or model versions change. agent
 
 - Re-record on major prompt rewrites or tool schema changes
 - Pin model versions (`gpt-4o-2024-08-06`, not `gpt-4o`) in your agent code so cassettes don't silently shift underneath you
-- Treat cassette diffs as part of the PR review — a YAML diff next to the prompt diff is a feature, not noise
+- Treat cassette diffs as part of the PR review. A YAML diff next to the prompt diff is a feature, not noise
 - Use `MATCHES` and `partial_args=True` to assert on the stable parts of tool arguments, so small wording changes in LLM outputs don't cascade into test failures
 
+### Calls Cassettes Cannot Record
+
+Cassettes record non-streaming chat calls: OpenAI `chat.completions.create`, Anthropic `messages.create`, Gemini `models.generate_content`, Bedrock `Converse`, and LiteLLM `completion`. Inside a cassette or `MockLLM`, agentverify raises `CassetteUnsupportedCallError` instead of letting other LLM calls reach the real API:
+- Streaming entry points (`stream=True`, `messages.stream()`, `generate_content_stream()`, Bedrock `ConverseStream`, OpenAI Responses `stream()`), in record and replay mode.
+- In replay mode, every other request through an OpenAI, Anthropic, or Gemini SDK client, including the Responses API, `parse()`, embeddings, and the async Anthropic and Gemini clients. Bedrock operations that generate (`InvokeModel`, `InvokeAgent`, `RetrieveAndGenerate`, ...) and LiteLLM `acompletion`, `text_completion`, and `responses` raise too.
+- In record mode, the entry points named above run with a warning. Other requests (beta resources, WebSocket APIs) run without one, and replay then fails at that call.
+- With no cassette file (`AUTO` passthrough), nothing is saved and no guard applies.
+
+Frameworks that stream by default need streaming turned off, e.g. `BedrockModel(streaming=False)` in Strands.
+
 ### Cassette Sanitization
 
-Cassette files are sanitized by default when recording. API keys, tokens, and other sensitive data are automatically redacted before saving to disk.
+Cassettes are sanitized by default when recording. Before saving, agentverify redacts:
+- values that match a built-in key pattern, anywhere in the cassette
+- call arguments and headers under a credential key, whatever their format: `api_key`, `authorization`, `x-api-key`, `cookie`, and names ending in `token`, `secret`, `password`, and similar (for example in `extra_headers`)
 
-Built-in patterns cover:
-- OpenAI API keys (`sk-...`)
-- Anthropic API keys (`sk-ant-...`)
-- AWS access keys (`AKIA...`)
+Built-in patterns (`DEFAULT_PATTERNS`) cover:
+- OpenAI and Anthropic API keys (`sk-...`, `sk-ant-...`)
+- AWS access keys (`AKIA...`, `ASIA...`)
+- Amazon Bedrock API keys (`ABSK...`, `bedrock-api-key-...`)
+- Google API keys (`AIza...`)
+- GitHub tokens (`ghp_...`, `github_pat_...`)
 - Bearer tokens (`Bearer ...`)
 
-Add custom patterns:
+A custom list replaces the built-in patterns, so extend `DEFAULT_PATTERNS` to keep them. Credential keys in call arguments are redacted either way:
 
 ```python
-from agentverify import SanitizePattern
+from agentverify import DEFAULT_PATTERNS, SanitizePattern
 
 custom_patterns = [
+    *DEFAULT_PATTERNS,
     SanitizePattern(name="internal_token", pattern=r"tok-[a-f0-9]{32}", replacement="tok-***REDACTED***"),
 ]
 
@@ -365,27 +394,27 @@ Pass `sanitize=False` to disable sanitization entirely (not recommended).
 ```python
 from agentverify import assert_tool_calls, OrderMode, ToolCall, ANY, MATCHES
 
-# Exact match — same tools, same order, same count (default)
+# Exact match: same tools, same order, same count (default)
 assert_tool_calls(result, expected=[...])
 
-# Subsequence — these tools appeared in this order (other calls in between are OK)
+# Subsequence: these tools appeared in this order (other calls in between are OK)
 assert_tool_calls(result, expected=[...], order=OrderMode.IN_ORDER)
 
-# Set membership — these tools were called (order doesn't matter)
+# Set membership: these tools were called (order doesn't matter)
 assert_tool_calls(result, expected=[...], order=OrderMode.ANY_ORDER)
 
-# Partial args — only check the keys you care about
+# Partial args: only check the keys you care about
 assert_tool_calls(result, expected=[
     ToolCall("search", {"query": "Tokyo"}),
 ], partial_args=True)
 
-# ANY wildcard — ignore a specific argument value
+# ANY wildcard: ignore a specific argument value
 assert_tool_calls(result, expected=[
     ToolCall("get_weather", {"lat": ANY, "lon": ANY}),
 ])
 
-# MATCHES regex — verify a string argument follows a pattern
-# (re.search semantics — use ^/$ anchors for full match)
+# MATCHES regex: verify a string argument follows a pattern
+# (re.search semantics; use ^/$ anchors for full match)
 assert_tool_calls(result, expected=[
     ToolCall("send_email", {"to": MATCHES(r"^[\w.+-]+@example\.com$")}),
     ToolCall("log_event", {"event_id": MATCHES(r"^evt_[a-f0-9]{16}$")}),
@@ -408,7 +437,7 @@ assert_all(
 
 ### Cost Assertions
 
-`assert_cost()` enforces token and dollar budgets on an agent run. Either bound is optional — pass `max_tokens` only, `max_cost_usd` only, or both together:
+`assert_cost()` enforces token and dollar budgets on an agent run. Either bound is optional. Pass `max_tokens` only, `max_cost_usd` only, or both together:
 
 ```python
 from agentverify import assert_cost
@@ -419,11 +448,11 @@ assert_cost(result, max_tokens=500)
 # Dollar budget only
 assert_cost(result, max_cost_usd=0.01)
 
-# Both — fail if either is exceeded
+# Both: fail if either is exceeded
 assert_cost(result, max_tokens=500, max_cost_usd=0.01)
 ```
 
-`result.token_usage` is populated automatically by the `cassette` fixture, `MockLLM`, and every built-in framework adapter. `result.total_cost_usd` must be set manually (see [Build an ExecutionResult](#build-an-executionresult)) — agentverify does not compute dollar costs from tokens, since the pricing table would need per-model maintenance and get stale quickly.
+`result.token_usage` is populated automatically by the `cassette` fixture, `MockLLM`, and every built-in framework adapter. `result.total_cost_usd` must be set manually (see [Build an ExecutionResult](#build-an-executionresult)). agentverify does not compute dollar costs from tokens, since the pricing table would need per-model maintenance and get stale quickly.
 
 ### Latency Assertions
 
@@ -461,9 +490,16 @@ assert_final_output(result, equals="The weather in Tokyo is sunny, 22°C.")
 assert_final_output(result, matches=r"\d+°C")
 ```
 
+`contains` also accepts a list; every substring must appear. `case_sensitive=False` ignores case in `contains` only; for a case-insensitive regex, put `(?i)` in the pattern. `assert_step_output` and `assert_tool_result_matches` accept the same options:
+
+```python
+assert_final_output(result, contains=["Tokyo", "22°C"])
+assert_final_output(result, contains="tokyo", case_sensitive=False)
+```
+
 ### Strict Mode
 
-`assert_cost()` and `assert_latency()` silently pass when the underlying data (`token_usage`, `total_cost_usd`, `duration_ms`) is `None` — useful during cassette replay where some data may be unavailable. Pass `strict=True` to require the data to be present:
+`assert_cost()` and `assert_latency()` silently pass when the underlying data (`token_usage`, `total_cost_usd`, `duration_ms`) is `None`. Useful during cassette replay where some data may be unavailable. Pass `strict=True` to require the data to be present:
 
 ```python
 # Fails if token_usage or total_cost_usd is None
@@ -475,7 +511,7 @@ assert_latency(result, max_ms=3000, strict=True)
 
 ## Step-Level Assertions
 
-For agents that make multiple LLM calls per execution (ReAct, Plan-and-Execute, multi-hop retrieval, workflow-style), agentverify exposes the full step structure — not just a flat list of tool calls.
+For agents that make multiple LLM calls per execution (ReAct, Plan-and-Execute, multi-hop retrieval, workflow-style), agentverify exposes the full step structure, not just a flat list of tool calls.
 
 `ExecutionResult.steps` is the single source of truth; `result.tool_calls` remains as a derived flat view for simpler cases.
 
@@ -502,7 +538,7 @@ def test_plan_and_execute(cassette):
 
 ### Assert Step-to-Step Data Flow
 
-`assert_step_uses_result_from()` verifies that one step's input references data produced by another — catches "agent ignored the tool result" bugs.
+`assert_step_uses_result_from()` verifies that one step's input references data produced by another. Catches "agent ignored the tool result" bugs.
 
 ```python
 from agentverify import assert_step_uses_result_from
@@ -516,13 +552,13 @@ assert_step_uses_result_from(result, step=2, depends_on=1, via="tool_result")
 
 The check searches for any produced value (step M's `tool_results`, `tool_calls[*].result`, or `output`) inside step N's `input_context` or `tool_calls[*].arguments`. Strings are substring-matched; primitives (numbers, booleans) are matched structurally inside containers; JSON-encoded tool results are descended automatically.
 
-To see this concretely with a different example: if step 0's `list_issues` tool returned `[{"number": 1}, {"number": 2}]` and step 1 called `get_issue(issue_number=2)`, the check finds that the number `2` produced by step 0 shows up in step 1's arguments — data flows correctly. This is exactly the pattern the [LangChain GitHub issue triage example](examples/langchain-issue-triage/) verifies.
+To see this concretely with a different example: if step 0's `list_issues` tool returned `[{"number": 1}, {"number": 2}]` and step 1 called `get_issue(issue_number=2)`, the check finds that the number `2` produced by step 0 shows up in step 1's arguments. Data flows correctly. This is exactly the pattern the [LangChain GitHub issue triage example](examples/langchain-issue-triage/) verifies.
 
-**Works on cassette replay.** Cassette recorders don't record tool execution results directly, but agentverify backfills them from the next step's `input_context` so you can assert data flow without any special adapter setup.
+This works on cassette replay. Cassette recorders don't record tool execution results directly, but agentverify backfills them from the next step's `input_context` so you can assert data flow without any special adapter setup.
 
 ### Workflow-Style Agents with `step_probe`
 
-Many production agents aren't pure ReAct loops — they mix LLM calls with caching, state management, validation, and conditional branches. `step_probe()` lets you mark logical step boundaries in your agent code so tests can assert on those non-LLM steps.
+Many production agents aren't pure ReAct loops. They mix LLM calls with caching, state management, validation, and conditional branches. `step_probe()` lets you mark logical step boundaries in your agent code so tests can assert on those non-LLM steps.
 
 ```python
 # agent.py
@@ -551,13 +587,49 @@ def test_cache_miss_path():
     assert_step_output(result, name="postprocess", equals="formatted")
 ```
 
-**`step_probe` is a zero-cost no-op outside of test recording contexts** — when no `LLMCassetteRecorder` or `MockLLM` is active, it's a pass-through context manager. Safe to leave in production code.
+**`step_probe` is a zero-cost no-op outside of test recording contexts.** When no `LLMCassetteRecorder` or `MockLLM` is active, it's a pass-through context manager. Safe to leave in production code.
+
+## Tool Result Assertions
+
+Knowing the agent *called* the right tool is not the same as knowing the call *worked*. A tool can return a 5xx, time out, or hand back an empty body, and the LLM will often smooth that over in its final reply. These assertions check the result a tool handed back, not just which tool was called and with what arguments.
+
+```python
+from agentverify import (
+    assert_tool_invocation_succeeded,
+    assert_no_tool_errors,
+    assert_tool_result_matches,
+    assert_retry_count,
+)
+
+# Affirmative: the tool call(s) at this step returned a usable result
+assert_tool_invocation_succeeded(result, step=1)
+
+# Blanket safety net: no tool anywhere in the run errored
+assert_no_tool_errors(result)
+
+# The tool returned a well-shaped / non-empty result
+assert_tool_result_matches(result, step=1, contains="temperature")
+assert_tool_result_matches(result, step=1, matches=r'"temp":\s*\d+')
+# Every substring must appear in the same tool result; case_sensitive=False ignores case
+assert_tool_result_matches(result, step=1, contains=["temp", "unit"], case_sensitive=False)
+
+# Budget on retries of a flaky tool (counted from consecutive repeat calls)
+assert_retry_count(result, tool_name="http_request", max=2)
+```
+
+agentverify reads a per-result error signal that the built-in adapters populate from each framework's native tool-error convention (LangGraph `ToolMessage.status == "error"`, Strands `toolResult` status, OpenAI Agents SDK default failure and timeout messages, LangChain error-shaped observations), and that cassette replay reconstructs from the recorded tool messages.
+
+The signal is three-valued: a result is a **known error**, a **known success**, or **unknown** (for example, a plain-string tool output on cassette replay where no structured status survived). `assert_tool_invocation_succeeded` and `assert_no_tool_errors` fail only on a *known* error; unknown and absent results pass, so the same test gives the same verdict whether it runs against a live agent or a replayed cassette. To require that a result is actually present and well-formed, use `assert_tool_result_matches`, which fails when the step has no tool result at all.
+
+`assert_no_tool_errors` is a blanket gate, so reach for it only when every tool in the run is expected to succeed. If your agent's correct behaviour includes recovering from a failure (a 429 then a retry, a primary tool then a fallback), that trajectory contains an expected error, and the blanket gate would flag it. In that case assert success on the specific steps that must succeed with `assert_tool_invocation_succeeded(step=N)` instead.
+
+If you build an `ExecutionResult` by hand or with a custom converter, set the signal yourself via the `tool_results_meta` field on each `Step`: a list positionally aligned with `tool_results`, where entry `i` is `{"is_error": True}`, `{"is_error": False}`, or `{}` (unknown). With `step_probe`, pass it through `handle.set_tool_result(result, is_error=True)`.
 
 ## Framework Integration
 
 ### Built-in Adapters
 
-agentverify ships with built-in adapters for popular agent frameworks. No converter function needed — just import and call:
+agentverify ships with built-in adapters for popular agent frameworks. No converter function needed. Just import and call:
 
 | Framework | Adapter | Input |
 |---|---|---|
@@ -593,22 +665,22 @@ execution_result = from_openai_agents(Runner.run_sync(agent, "your prompt"))
 
 ### Custom Converters
 
-For other frameworks, build an `ExecutionResult` from your agent's output using a small converter function. See [`examples/custom-converter-python-agent/`](examples/custom-converter-python-agent/) for a complete reference — an Anthropic SDK ReAct agent paired with a ~80-line `converter.py`, a recorded cassette, and step-level tests.
+For other frameworks, build an `ExecutionResult` from your agent's output using a small converter function. See [`examples/custom-converter-python-agent/`](examples/custom-converter-python-agent/) for a complete reference: an Anthropic SDK ReAct agent paired with a ~80-line `converter.py`, a recorded cassette, and step-level tests.
 
 ## Supported LLM Providers
 
 | Provider | Extra |
 |---|---|
-| OpenAI | `pip install agentverify[openai]` |
-| Amazon Bedrock | `pip install agentverify[bedrock]` |
-| Google Gemini | `pip install agentverify[gemini]` |
-| Anthropic | `pip install agentverify[anthropic]` |
-| LiteLLM | `pip install agentverify[litellm]` |
-| All providers | `pip install agentverify[all]` |
+| OpenAI | `pip install "agentverify[openai]"` |
+| Amazon Bedrock | `pip install "agentverify[bedrock]"` |
+| Google Gemini | `pip install "agentverify[gemini]"` |
+| Anthropic | `pip install "agentverify[anthropic]"` |
+| LiteLLM | `pip install "agentverify[litellm]"` |
+| All providers | `pip install "agentverify[all]"` |
 
-## Tool Mocking
+## LLM Mocking with MockLLM
 
-`MockLLM` replays a list of predefined LLM responses that you define in code. Test your agent's routing logic without a cassette or a real LLM call — useful when you haven't recorded yet, or when you want to drive the agent through hypothetical responses (errors, edge cases) that would be awkward to elicit from a real model.
+`MockLLM` replays a list of predefined LLM responses that you define in code. Test your agent's routing logic without a cassette or a real LLM call. Useful when you haven't recorded yet, or when you want to drive the agent through hypothetical responses (errors, edge cases) that would be awkward to elicit from a real model.
 
 ```python
 import pytest
@@ -631,11 +703,11 @@ def test_agent_routes_to_weather_tool():
 
 `mock_response()` accepts:
 
-- `content` — the final text message the LLM should return.
-- `tool_calls` — a list of `(name, arguments)` tuples or `{"name": ..., "arguments": ...}` dicts.
-- `input_tokens` / `output_tokens` — optional token usage to report.
+- `content`: the final text message the LLM should return.
+- `tool_calls`: a list of `(name, arguments)` tuples or `{"name": ..., "arguments": ...}` dicts.
+- `input_tokens` / `output_tokens`: optional token usage to report.
 
-Provide one `mock_response(...)` per LLM call your agent is expected to make. If your agent makes more calls than you've queued, `MockLLM` raises `CassetteMissingRequestError` so under-specified tests fail loudly.
+Provide one `mock_response(...)` per LLM call your agent is expected to make. If your agent makes more calls than you've queued, `MockLLM` raises [`CassetteMissingRequestError`](#error-messages) so under-specified tests fail loudly.
 
 ### When to Use MockLLM vs Cassettes
 
@@ -664,21 +736,21 @@ jobs:
       - uses: actions/setup-python@v6
         with:
           python-version: "3.14"
-      # Shown separately here for clarity — in practice, list agentverify
+      # Shown separately here for clarity. In practice, list agentverify
       # alongside the rest of your project's deps in requirements.txt.
       - run: pip install "agentverify[all]"
       - run: pip install -r requirements.txt  # your project's deps
       - run: pytest --tb=short -v
 ```
 
-Cassette files in `tests/cassettes/` are replayed automatically — no API keys or secrets needed in CI.
+Cassette files in `tests/cassettes/` are replayed automatically. No API keys or secrets needed in CI.
 
 ## Error Messages
 
 Clear, structured output when assertions fail:
 
 ```
-ToolCallSequenceError: Tool call sequence mismatch at index 1
+ToolCallSequenceError: Tool call sequence mismatch at index 1 (step 0)
 
 Expected:
   [0] get_location(city="Tokyo")
@@ -705,7 +777,7 @@ LatencyBudgetError: Latency budget exceeded
   Exceeded by: 450.0 ms (15.0%)
 ```
 
-Other error types follow the same pattern: `SafetyRuleViolationError`, `FinalOutputError`.
+Other error types follow the same pattern: `SafetyRuleViolationError`, `FinalOutputError`, `CassetteRequestMismatchError` (stale cassette detected), `CassetteMissingRequestError` (cassette or MockLLM ran out of queued responses), and the step-level `StepIndexError`, `StepNameNotFoundError`, `StepNameAmbiguousError`, `StepOutputError`, `StepDependencyError`. All inherit from `AgentVerifyError`, which itself extends `AssertionError` so pytest treats them as normal test failures.
 
 ## Requirements
 
@@ -714,28 +786,52 @@ Other error types follow the same pattern: `SafetyRuleViolationError`, `FinalOut
 
 ## Examples
 
-The [`examples/`](examples/) directory contains end-to-end examples with real agent frameworks and MCP servers. Each example ships with pre-recorded cassettes — run the tests without any API keys.
+The [`examples/`](examples/) directory contains end-to-end examples with real agent frameworks and MCP servers. Each example ships with pre-recorded cassettes, so tests run without any API keys.
 
 | Example | Framework | Description |
 |---|---|---|
-| [`langchain-issue-triage`](examples/langchain-issue-triage/) | LangChain + OpenAI | Triages GitHub issues via GitHub MCP, step-level data flow for the discovered issue number |
+| [`langchain-issue-triage`](examples/langchain-issue-triage/) | LangChain + OpenAI | Triages GitHub issues via GitHub MCP, step-level data flow for the discovered issue number, tool-result outcome checks |
 | [`langgraph-multi-agent-supervisor`](examples/langgraph-multi-agent-supervisor/) | LangGraph + OpenAI | Research + math multi-agent handoff with running-total data flow |
 | [`strands-weather-forecaster`](examples/strands-weather-forecaster/) | Strands Agents + Bedrock | Two-step ReAct that discovers a forecast URL, then calls it |
 | [`openai-agents-llm-as-a-judge`](examples/openai-agents-llm-as-a-judge/) | OpenAI Agents SDK | Probabilistic generator ↔ evaluator refinement loop, frozen into a deterministic test with feedback-chain data flow |
-| [`custom-converter-python-agent`](examples/custom-converter-python-agent/) | Anthropic SDK (no framework) | Reference for writing a custom converter — a hand-rolled ReAct loop calling the Anthropic Messages API directly |
-| [`mcp-server`](examples/mcp-server/) | — | Mock GitHub MCP server for token-free testing |
+| [`custom-converter-python-agent`](examples/custom-converter-python-agent/) | Anthropic SDK (no framework) | Reference for writing a custom converter: a hand-rolled ReAct loop calling the Anthropic Messages API directly |
+| [`mcp-server`](examples/mcp-server/) | (shared utility) | Mock GitHub MCP server for token-free testing |
 
 See each example's README for setup and recording mode details.
 
+## Benchmark
+
+The [`benchmarks/execution-model-trajectory/`](benchmarks/execution-model-trajectory/) directory contains a comparison study that writes the same execution-trajectory assertion in three different assertion-execution-model styles (inline pytest via agentverify, separate-runner via [DeepEval](https://www.deepeval.com/docs/evaluation-llm-tracing) [`ToolCorrectnessMetric`](https://www.deepeval.com/docs/metrics-tool-correctness) + `@observe`, managed remote via [Amazon Bedrock AgentCore Evaluations Custom code-based evaluator](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/code-based-evaluators.html) on Lambda) over two agents (Strands single-agent ReAct and LangGraph multi-agent supervisor). Per-cell numbers (LOC, wall time, API calls, dollar cost, CI secrets, cold-start) are recorded under [`results/`](benchmarks/execution-model-trajectory/results/). See the bench's [`README.md`](benchmarks/execution-model-trajectory/README.md) for the operating manual and [`DESIGN.md`](benchmarks/execution-model-trajectory/DESIGN.md) for the methodology and explicit non-goals. The benchmark is a one-time exercise that refreshes when tooling versions change materially, not a continuous CI gate.
+
+## How agentverify compares
+
+Agent evaluation covers a few different jobs, and agentverify does one of them. This table is meant to help you pick the right tool for a given job, not to rank them. In practice they are often used together.
+
+| Tool | Best at | Execution model |
+|---|---|---|
+| **agentverify** | Deterministic regression tests on what the agent did: tool calls, arguments, step-to-step data flow, cost, latency, safety | Inline pytest, in-process, no LLM call on replay, no cloud account |
+| [DeepEval](https://github.com/confident-ai/deepeval) (Confident AI) | A large catalog of quality metrics (relevancy, faithfulness, and 40+ others), run locally or on a cloud platform | Separate test runner / cloud |
+| [LangSmith](https://www.langchain.com/langsmith) (LangChain) | Agent tracing and dataset-based evaluation for the LangChain / LangGraph stack: trajectory matching and LLM-as-judge, with a pytest / vitest integration | Hosted platform (plus open-source evaluator packages) |
+| Managed cloud eval services: [Amazon Bedrock AgentCore Evaluations](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations), [Google Vertex AI Gen AI evaluation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview), [Azure AI Foundry evaluators](https://learn.microsoft.com/azure/ai-foundry/concepts/evaluation-evaluators/agent-evaluators) | Hosted quality scoring (LLM-as-judge) plus some deterministic trajectory and metric checks, with datasets, dashboards, and reuse across CI and production | Managed, tied to a cloud account |
+| [Scenario](https://github.com/langwatch/scenario) (LangWatch) | Multi-turn simulation of a user conversing with the agent | Simulation against a live LLM |
+
+Most of these tools focus on output quality, often via LLM-as-judge, and several now add deterministic trajectory checks too. The line agentverify draws is narrower and lower in the stack: it asserts the **trajectory** (what tools ran, with what arguments, in what order, feeding what into what) inline in pytest, with no LLM in the loop on replay and no cloud account. Quality judgements ("was this a good answer?") need a judge model and fit better on a separate stage of the pipeline, usually against a live agent in staging rather than on every PR. The two are complementary: one gates the agent's actions cheaply on every commit, the other scores answer quality less often, where the judge cost is justified.
+
+A rough guide:
+
+- Want a broad catalog of quality metrics you can run locally? DeepEval has the widest catalog.
+- Building entirely on the LangChain / LangGraph stack and want tracing plus evaluation in one place? LangSmith fits naturally.
+- Already invested in a cloud agent platform (AWS, Google, or Azure) and want hosted quality scoring with datasets and dashboards? The managed eval services fit, and several now include deterministic trajectory checks too.
+- Need to stress-test multi-turn conversations with a simulated user? Scenario is built for that.
+- Want trajectory assertions inline in pytest, on every PR, with no API key, no secrets, and no cloud account to stand up? That is what agentverify is for.
+
 ## Roadmap
 
-We want agentverify to become the way people regression-test agents in CI — tests that live next to the agent code, in git, reviewed in PRs alongside the prompt and tool changes that affect them.
+We want agentverify to become the way people regression-test agents in CI: tests that live next to the agent code, in git, reviewed in PRs alongside the prompt and tool changes that affect them.
 
-- Async support — first-class `asyncio` testing for async agents and tools
-- Responses API cassette adapter — record/replay for OpenAI Agents SDK (Responses API) with end-to-end example
-- Framework adapters for Google ADK and CrewAI — pending async support and stable tool-call APIs from these frameworks
-- Cost estimation from tokens — auto-calculate `total_cost_usd` from token usage and model pricing
-- Eval-framework bridges — compose ragas / deepeval quality scores with agentverify's action assertions in a single test report
+- Async support: first-class `asyncio` testing for async agents and tools
+- Responses API cassette adapter: record/replay for OpenAI Agents SDK (Responses API) with end-to-end example
+- Framework adapters for Google ADK and CrewAI: pending async support and stable tool-call APIs from these frameworks
 
 ## Changelog