agent-trajectory-diff 0.2.2__tar.gz → 0.3.0__tar.gz

{agent_trajectory_diff-0.2.2 → agent_trajectory_diff-0.3.0}/.gitignore

@@ -79,3 +79,6 @@ website/.env.local
 website/.env.development.local
 website/.env.test.local
 website/.env.production.local
+
+# pytest-benchmark local results
+.benchmarks/

{agent_trajectory_diff-0.2.2 → agent_trajectory_diff-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agent-trajectory-diff
-Version: 0.2.2
+Version: 0.3.0
 Summary: A developer-first package to evaluate and find regressions in agent trajectories
 Project-URL: Homepage, https://github.com/lostmartian/agentdiff
 Project-URL: Repository, https://github.com/lostmartian/agentdiff
@@ -31,6 +31,23 @@ Description-Content-Type: text/markdown
 * **A Local-First CI/CD Gate:** Runs locally in your terminal or inside `pytest` and GitHub Actions, raising errors or exit codes on regression violations.
 * **A Universal Comparator:** Ingests telemetry run files from **OpenInference/OTel**, **Langfuse**, **LangSmith**, **OpenAI Agents SDK**, or raw/custom JSON.
 
+### What AgentDiff Is Not
+
+* **Not an observability backend.** No hosted tracing, no APM, no log storage — AgentDiff works on trace files you already have, at test time.
+* **Not an LLM-as-a-judge scorer.** Semantic answer quality is DeepEval/Ragas territory; AgentDiff measures *how* your agent got there — structurally and deterministically.
+* **Not an agent framework.** It doesn't orchestrate or run agents; it evaluates the trajectories your existing agents (LangGraph, CrewAI, OpenAI Agents SDK, custom loops) already produce.
+
+### Local-First Privacy
+
+Agent trajectories contain your prompts, your tool outputs, and often your customers' data. AgentDiff is architected so that **nothing ever leaves your machine:**
+
+* **No network calls at diff time.** Parsing, DAG alignment, and scoring are pure local computation — run a diff on a plane, in a bank's air-gapped CI, or behind a strict egress firewall.
+* **No account, no telemetry.** AgentDiff doesn't phone home, has no API to sign up for, and collects nothing.
+* **Your baselines live in your repo.** Baseline traces are ordinary committed files (`--baseline` / `--update-baseline`), versioned with the code they gate — no external service holds them.
+* **CI stays inside your perimeter.** The GitHub Action reads traces from your checkout and posts reports with your own `GITHUB_TOKEN`; traces are never uploaded anywhere by us.
+
+Hosted eval platforms require shipping production traces to a third party before you can diff them. With AgentDiff, the diff is a file operation.
+
 ## Installation
 
 Install the PyPI package:
@@ -145,6 +162,10 @@ name: AgentDiff Gate
 on:
   pull_request:
 
+permissions:
+  contents: read
+  pull-requests: write   # lets the action post the PR comment
+
 jobs:
   agentdiff:
     runs-on: ubuntu-latest
@@ -160,11 +181,18 @@ jobs:
           update-baseline: "false"
           max-divergence: "0.3"
           max-cost-delta: "10.0"
+          # Optional: auto-post the report onto the triggering PR.
+          pr: ${{ github.event.pull_request.number }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
 ```
 
 The action installs the package (default `agent-trajectory-diff` from PyPI),
 runs `agentdiff --fail-on-regression`, and fails the job when divergence,
-loops, or cost spikes exceed the thresholds.
+loops, or cost spikes exceed the thresholds. When `pr` is set it also posts the
+PR-ready report (status, gate table, root-cause culprit, collapsed divergence
+tree, loops) as a comment on that PR — even when the gate blocks. See the
+[`agentdiff-demo`](https://github.com/lostmartian/agentdiff-demo) repository
+for a working, live example (real Gemini agent + auto PR comments).
 
 **Available inputs:**
 
@@ -178,6 +206,11 @@ loops, or cost spikes exceed the thresholds.
 | `max-loops` | `0` | Maximum loop count before regression. |
 | `max-cost-delta` | `10.0` | Maximum cost increase percentage before regression. |
 | `update-baseline` | `false` | Overwrite the stored baseline with the candidate when the run is clean. |
+| `pr` | *(empty)* | GitHub PR number to post the report comment to (e.g. `github.event.pull_request.number`). |
+| `github-token` | *(empty)* | GitHub token used to post the comment (e.g. `secrets.GITHUB_TOKEN`). Required when `pr` is set. |
+
+> **Permission:** to post the PR comment the workflow needs `pull-requests: write`
+> (the built-in `GITHUB_TOKEN` is otherwise read-only). No manual token required.
 
 ## Core Metrics
 
@@ -186,8 +219,35 @@ loops, or cost spikes exceed the thresholds.
 | **Trajectory Divergence Index (TDI)** | `0.0` (Identical) to `1.0` (Divergent) | $$1.0 - \frac{2 \times \vert{}\text{LCS}(\text{Steps}_A, \text{Steps}_B)\vert{}}{\vert{}\text{Steps}_A\vert{} + \vert{}\text{Steps}_B\vert{}}$$ |
 | **Wasted Effort Index (WEI)** | `0.0` (Optimal) to `1.0` (Total Waste) | $$\frac{\text{Count}(\text{Steps with status} \in \{\text{ERROR, RETRY, ABANDONED}\})}{\text{Total Execution Steps}}$$ |
 | **Loop Buster Index (LBI)** | Integer ($\ge 0$) | Detects consecutive repeating sequences of tools with stagnant state changes. |
+| **Recovery Step Ratio (RSR)** | `1.0` = parity; $> 1.0$ = slower recovery than baseline | Successful steps spent after ERROR/RETRY/ABANDONED clusters until re-aligning with the baseline path: $\text{RSR} = \frac{\text{Recovery}_{\text{candidate}}}{\text{Recovery}_{\text{baseline}}}$ (falls back to the raw candidate count when the baseline is clean). Gate via `--max-recovery-ratio` / `max_recovery_step_ratio`. |
 | **Resource Deltas ($\Delta\text{Res}$)** | Percentage ($\pm\%$) | Standard deltas for $\Delta\text{Tokens}$, $\Delta\text{Cost}$, and $\Delta\text{Latency}$. |
 
+## FAQ
+
+**How is AgentDiff different from DeepEval or Ragas?**
+They score *what* the agent said (semantic quality, via LLM judges). AgentDiff measures *how* the agent got there — step order, tool loops, wasted effort, cost/latency deltas — using deterministic graph algorithms. They complement each other; AgentDiff adds no LLM calls and is fully deterministic.
+
+**Do I need API keys to run a diff?**
+No. AgentDiff is pure math over trace files you already have. Keys are only needed by your own agent when it produces traces, or by the optional live cookbooks that generate them.
+
+**Where do trace files come from?**
+Export them from whatever already records your runs: Langfuse or LangSmith exports, OpenTelemetry/OpenInference span dumps, the OpenAI Agents SDK tracing processor, or hand-rolled JSON matching the generic schema. See [`cookbooks/`](cookbooks/) for working recipes per source.
+
+**Can I compare runs from different frameworks?**
+Yes. Traces are normalized to one canonical `AgentTrace` schema before comparison, so an OpenInference baseline can be diffed against a Langfuse candidate (or any other pairing).
+
+**What do TDI / WEI / LBI mean in one line each?**
+TDI: fraction of trajectory structure that changed (0 = identical). WEI: share of steps that were errors/retries/abandonments. LBI: count of repeating tool sequences with no state progress. Definitions above.
+
+**How does the pytest plugin know which baseline belongs to a test?**
+Mark tests with the `agentdiff` marker and use the `agentdiff_trace` fixture; a committed baseline file per test is compared automatically (`--agentdiff-update-baselines` advances baselines on clean runs). See the docs for setup.
+
+**Which Python versions are supported?**
+Python 3.10 through 3.13, tested in CI on every PR.
+
+**Is it production-safe to gate merges on this?**
+That's the point — exit codes 0/1 make it a drop-in CI gate, and the GitHub Action posts the culprit + divergence tree right onto the PR so reviewers see *why* a gate blocked.
+
 ## Development & Operations
 
 This project utilizes `uv` to manage environments and dependencies. Automation tasks are defined in the **[`Makefile`](file:///Users/lostmartian/Desktop/interview/agentdiff/Makefile)**:

{agent_trajectory_diff-0.2.2 → agent_trajectory_diff-0.3.0}/README.md

@@ -11,6 +11,23 @@
 * **A Local-First CI/CD Gate:** Runs locally in your terminal or inside `pytest` and GitHub Actions, raising errors or exit codes on regression violations.
 * **A Universal Comparator:** Ingests telemetry run files from **OpenInference/OTel**, **Langfuse**, **LangSmith**, **OpenAI Agents SDK**, or raw/custom JSON.
 
+### What AgentDiff Is Not
+
+* **Not an observability backend.** No hosted tracing, no APM, no log storage — AgentDiff works on trace files you already have, at test time.
+* **Not an LLM-as-a-judge scorer.** Semantic answer quality is DeepEval/Ragas territory; AgentDiff measures *how* your agent got there — structurally and deterministically.
+* **Not an agent framework.** It doesn't orchestrate or run agents; it evaluates the trajectories your existing agents (LangGraph, CrewAI, OpenAI Agents SDK, custom loops) already produce.
+
+### Local-First Privacy
+
+Agent trajectories contain your prompts, your tool outputs, and often your customers' data. AgentDiff is architected so that **nothing ever leaves your machine:**
+
+* **No network calls at diff time.** Parsing, DAG alignment, and scoring are pure local computation — run a diff on a plane, in a bank's air-gapped CI, or behind a strict egress firewall.
+* **No account, no telemetry.** AgentDiff doesn't phone home, has no API to sign up for, and collects nothing.
+* **Your baselines live in your repo.** Baseline traces are ordinary committed files (`--baseline` / `--update-baseline`), versioned with the code they gate — no external service holds them.
+* **CI stays inside your perimeter.** The GitHub Action reads traces from your checkout and posts reports with your own `GITHUB_TOKEN`; traces are never uploaded anywhere by us.
+
+Hosted eval platforms require shipping production traces to a third party before you can diff them. With AgentDiff, the diff is a file operation.
+
 ## Installation
 
 Install the PyPI package:
@@ -125,6 +142,10 @@ name: AgentDiff Gate
 on:
   pull_request:
 
+permissions:
+  contents: read
+  pull-requests: write   # lets the action post the PR comment
+
 jobs:
   agentdiff:
     runs-on: ubuntu-latest
@@ -140,11 +161,18 @@ jobs:
           update-baseline: "false"
           max-divergence: "0.3"
           max-cost-delta: "10.0"
+          # Optional: auto-post the report onto the triggering PR.
+          pr: ${{ github.event.pull_request.number }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
 ```
 
 The action installs the package (default `agent-trajectory-diff` from PyPI),
 runs `agentdiff --fail-on-regression`, and fails the job when divergence,
-loops, or cost spikes exceed the thresholds.
+loops, or cost spikes exceed the thresholds. When `pr` is set it also posts the
+PR-ready report (status, gate table, root-cause culprit, collapsed divergence
+tree, loops) as a comment on that PR — even when the gate blocks. See the
+[`agentdiff-demo`](https://github.com/lostmartian/agentdiff-demo) repository
+for a working, live example (real Gemini agent + auto PR comments).
 
 **Available inputs:**
 
@@ -158,6 +186,11 @@ loops, or cost spikes exceed the thresholds.
 | `max-loops` | `0` | Maximum loop count before regression. |
 | `max-cost-delta` | `10.0` | Maximum cost increase percentage before regression. |
 | `update-baseline` | `false` | Overwrite the stored baseline with the candidate when the run is clean. |
+| `pr` | *(empty)* | GitHub PR number to post the report comment to (e.g. `github.event.pull_request.number`). |
+| `github-token` | *(empty)* | GitHub token used to post the comment (e.g. `secrets.GITHUB_TOKEN`). Required when `pr` is set. |
+
+> **Permission:** to post the PR comment the workflow needs `pull-requests: write`
+> (the built-in `GITHUB_TOKEN` is otherwise read-only). No manual token required.
 
 ## Core Metrics
 
@@ -166,8 +199,35 @@ loops, or cost spikes exceed the thresholds.
 | **Trajectory Divergence Index (TDI)** | `0.0` (Identical) to `1.0` (Divergent) | $$1.0 - \frac{2 \times \vert{}\text{LCS}(\text{Steps}_A, \text{Steps}_B)\vert{}}{\vert{}\text{Steps}_A\vert{} + \vert{}\text{Steps}_B\vert{}}$$ |
 | **Wasted Effort Index (WEI)** | `0.0` (Optimal) to `1.0` (Total Waste) | $$\frac{\text{Count}(\text{Steps with status} \in \{\text{ERROR, RETRY, ABANDONED}\})}{\text{Total Execution Steps}}$$ |
 | **Loop Buster Index (LBI)** | Integer ($\ge 0$) | Detects consecutive repeating sequences of tools with stagnant state changes. |
+| **Recovery Step Ratio (RSR)** | `1.0` = parity; $> 1.0$ = slower recovery than baseline | Successful steps spent after ERROR/RETRY/ABANDONED clusters until re-aligning with the baseline path: $\text{RSR} = \frac{\text{Recovery}_{\text{candidate}}}{\text{Recovery}_{\text{baseline}}}$ (falls back to the raw candidate count when the baseline is clean). Gate via `--max-recovery-ratio` / `max_recovery_step_ratio`. |
 | **Resource Deltas ($\Delta\text{Res}$)** | Percentage ($\pm\%$) | Standard deltas for $\Delta\text{Tokens}$, $\Delta\text{Cost}$, and $\Delta\text{Latency}$. |
 
+## FAQ
+
+**How is AgentDiff different from DeepEval or Ragas?**
+They score *what* the agent said (semantic quality, via LLM judges). AgentDiff measures *how* the agent got there — step order, tool loops, wasted effort, cost/latency deltas — using deterministic graph algorithms. They complement each other; AgentDiff adds no LLM calls and is fully deterministic.
+
+**Do I need API keys to run a diff?**
+No. AgentDiff is pure math over trace files you already have. Keys are only needed by your own agent when it produces traces, or by the optional live cookbooks that generate them.
+
+**Where do trace files come from?**
+Export them from whatever already records your runs: Langfuse or LangSmith exports, OpenTelemetry/OpenInference span dumps, the OpenAI Agents SDK tracing processor, or hand-rolled JSON matching the generic schema. See [`cookbooks/`](cookbooks/) for working recipes per source.
+
+**Can I compare runs from different frameworks?**
+Yes. Traces are normalized to one canonical `AgentTrace` schema before comparison, so an OpenInference baseline can be diffed against a Langfuse candidate (or any other pairing).
+
+**What do TDI / WEI / LBI mean in one line each?**
+TDI: fraction of trajectory structure that changed (0 = identical). WEI: share of steps that were errors/retries/abandonments. LBI: count of repeating tool sequences with no state progress. Definitions above.
+
+**How does the pytest plugin know which baseline belongs to a test?**
+Mark tests with the `agentdiff` marker and use the `agentdiff_trace` fixture; a committed baseline file per test is compared automatically (`--agentdiff-update-baselines` advances baselines on clean runs). See the docs for setup.
+
+**Which Python versions are supported?**
+Python 3.10 through 3.13, tested in CI on every PR.
+
+**Is it production-safe to gate merges on this?**
+That's the point — exit codes 0/1 make it a drop-in CI gate, and the GitHub Action posts the culprit + divergence tree right onto the PR so reviewers see *why* a gate blocked.
+
 ## Development & Operations
 
 This project utilizes `uv` to manage environments and dependencies. Automation tasks are defined in the **[`Makefile`](file:///Users/lostmartian/Desktop/interview/agentdiff/Makefile)**:

{agent_trajectory_diff-0.2.2 → agent_trajectory_diff-0.3.0}/pyproject.toml

@@ -35,6 +35,7 @@ Repository = "https://github.com/lostmartian/agentdiff"
 dev = [
     "hypothesis>=6.165.10",
     "pytest>=8.0",
+    "pytest-benchmark>=4.0",
     "pytest-cov>=7.1.0",
     "ruff>=0.16.3",
 ]

{agent_trajectory_diff-0.2.2 → agent_trajectory_diff-0.3.0}/src/agentdiff/__init__.py

@@ -16,6 +16,9 @@ Public API
 - Adapters: ``GenericAdapter``, ``LangfuseAdapter``, ``LangSmithAdapter``,
     ``OpenInferenceAdapter``, ``OpenAIAgentsAdapter`` (all expose
     ``from_dict`` / ``from_file``).
+- ``register_adapter(name, cls)`` / ``available_adapters()``
+    Extend ingestion with custom or entry-point plugin adapters
+    (entry-point group ``agentdiff.adapters``).
 - ``load_config(path=None)`` -> ``AgentDiffConfig``
     Load defaults from ``agentdiff.toml`` (thresholds, adapter, baseline).
 - Models: ``AgentTrace`` (canonical, ``schema_version``-ed), ``DiffReport``,
@@ -29,7 +32,7 @@ Typical flow::
     assert_no_regressions(report)
 """
 
-__version__ = "0.2.2"
+__version__ = "0.3.0"
 
 from agentdiff.adapters import (
     BaseAdapter,
@@ -39,14 +42,27 @@ from agentdiff.adapters import (
     OpenAIAgentsAdapter,
     OpenInferenceAdapter,
 )
+from agentdiff.adapters.registry import (
+    available_adapters,
+    register_adapter,
+)
 from agentdiff.ci.baseline import decide_rotation
 from agentdiff.config import AgentDiffConfig, load_config
 from agentdiff.engine import compare
+from agentdiff.engine.benchmark import BenchmarkCase, BenchmarkReport, run_benchmark
 from agentdiff.engine.explanations import (
     format_explanations,
     generate_explanations,
     locate_culprit,
 )
+from agentdiff.engine.suite import (
+    GateThresholds,
+    Scenario,
+    ScenarioResult,
+    SuiteReport,
+    run_scenario,
+    run_scenarios,
+)
 from agentdiff.engine.tree import render_tree
 from agentdiff.loader import load_trace, parse_trace_data
 from agentdiff.models import (
@@ -66,20 +82,27 @@ __all__ = [
     "AgentDiffConfig",
     "AgentTrace",
     "BaseAdapter",
+    "BenchmarkCase",
+    "BenchmarkReport",
     "DiffReport",
+    "GateThresholds",
     "GenericAdapter",
     "LangSmithAdapter",
     "LangfuseAdapter",
     "OpenAIAgentsAdapter",
     "OpenInferenceAdapter",
+    "Scenario",
+    "ScenarioResult",
     "StepDiff",
     "StepDiffStatus",
     "StepStatus",
     "StepType",
+    "SuiteReport",
     "TokenUsage",
     "TraceStep",
     "__version__",
     "assert_no_regressions",
+    "available_adapters",
     "compare",
     "decide_rotation",
     "format_explanations",
@@ -89,5 +112,9 @@ __all__ = [
     "load_trace",
     "locate_culprit",
     "parse_trace_data",
+    "register_adapter",
     "render_tree",
+    "run_benchmark",
+    "run_scenario",
+    "run_scenarios",
 ]

{agent_trajectory_diff-0.2.2 → agent_trajectory_diff-0.3.0}/src/agentdiff/adapters/__init__.py

@@ -1,15 +1,31 @@
 from agentdiff.adapters.base import BaseAdapter
+from agentdiff.adapters.crewai import CrewAIAdapter
 from agentdiff.adapters.generic import GenericAdapter
 from agentdiff.adapters.langfuse import LangfuseAdapter
+from agentdiff.adapters.langgraph import LangGraphAdapter
 from agentdiff.adapters.langsmith import LangSmithAdapter
 from agentdiff.adapters.openai_agents import OpenAIAgentsAdapter
 from agentdiff.adapters.openinference import OpenInferenceAdapter
+from agentdiff.adapters.registry import (
+    available_adapters,
+    get_adapter,
+    register_adapter,
+    reset_registry,
+    unregister_adapter,
+)
 
 __all__ = [
     "BaseAdapter",
+    "CrewAIAdapter",
     "GenericAdapter",
+    "LangGraphAdapter",
     "LangSmithAdapter",
     "LangfuseAdapter",
     "OpenAIAgentsAdapter",
     "OpenInferenceAdapter",
+    "available_adapters",
+    "get_adapter",
+    "register_adapter",
+    "reset_registry",
+    "unregister_adapter",
 ]

agent_trajectory_diff-0.3.0/src/agentdiff/adapters/_messages.py

@@ -0,0 +1,255 @@
+"""Shared role-message parsing used by direct framework adapters (A5).
+
+LangGraph states and CrewAI task logs both reduce to OpenAI-style role
+messages - system/user/human, assistant/ai with ``tool_calls``, and tool
+results - serialized in one of several shapes. This module owns that
+interpretation once so every direct-ingestion adapter maps identically:
+
+- ``message_kind``    -> canonical role for any known message shape
+- ``tool_calls_of``   -> normalized [{name, args, id}] from an AI payload
+- ``usage_of``        -> best-effort TokenUsage across metadata layouts
+- ``RoleStepBuilder`` -> accumulates TraceSteps with collision-safe ids
+
+Mapping contract (shared by all consumers):
+
+- human/system messages -> task-input context / skipped, never steps
+- assistant tool calls  -> one ROUTING step per call, named after the tool
+- matching tool result  -> one TOOL_CALL step, status honored when present
+- final assistant text  -> one LLM_CALL step named ``response``
+"""
+
+from agentdiff.models.step import StepStatus, StepType, TokenUsage, TraceStep
+
+WASTED_STATUS = {
+    "error": StepStatus.ERROR,
+    "retry": StepStatus.RETRY,
+    "abandoned": StepStatus.ABANDONED,
+}
+
+
+def message_kind(msg):
+    """Returns ``(kind, payload)`` for any known message shape.
+
+    kind is one of ``human`` | ``ai`` | ``tool`` | ``system`` | ``unknown``;
+    payload is the dict carrying content/tool_calls/metadata fields.
+    """
+    if not isinstance(msg, dict):
+        return "unknown", {}
+
+    # Shape 1: message_to_dict dump - {"type": "ai", "data": {...}}
+    if isinstance(msg.get("data"), dict) and msg.get("type"):
+        return str(msg["type"]).lower(), msg["data"]
+
+    kwargs = msg.get("kwargs") if isinstance(msg.get("kwargs"), dict) else None
+
+    # Shape 2: LC constructor dump - {"lc": 1, "id": [.., "AIMessage"], ...}
+    if kwargs is not None:
+        ident = msg.get("id")
+        tail = ident[-1] if isinstance(ident, list) and ident else ""
+        tail = str(tail)
+        if "AIMessage" in tail:
+            return "ai", kwargs
+        if "HumanMessage" in tail:
+            return "human", kwargs
+        if "ToolMessage" in tail:
+            return "tool", kwargs
+        if "SystemMessage" in tail:
+            return "system", kwargs
+
+    # Shape 3: plain role dicts
+    role = msg.get("role")
+    if role in ("user", "human"):
+        return "human", msg
+    if role in ("assistant", "ai"):
+        return "ai", msg
+    if role == "tool":
+        return "tool", msg
+    if role == "system":
+        return "system", msg
+
+    # Fallback for partial shapes (e.g. type field without data wrapper)
+    t = msg.get("type")
+    if t in ("ai", "human", "tool", "system"):
+        return str(t), msg
+    return "unknown", {}
+
+
+def tool_calls_of(payload):
+    """Extracts normalized tool calls from an AI-message payload."""
+    calls = payload.get("tool_calls")
+    out = []
+    if isinstance(calls, list):
+        for call in calls:
+            if not isinstance(call, dict):
+                continue
+            name = call.get("name") or call.get("function", {}).get("name")
+            args = call.get("args")
+            if args is None and isinstance(call.get("function"), dict):
+                raw_args = call["function"].get("arguments")
+                try:
+                    import json
+
+                    args = json.loads(raw_args) if isinstance(raw_args, str) else None
+                except Exception:
+                    args = None
+            call_id = call.get("id") or call.get("tool_call_id")
+            if name:
+                out.append({"name": str(name), "args": args or {}, "id": call_id})
+    return out
+
+
+def usage_of(payload):
+    """Best-effort token extraction across known metadata layouts."""
+    usage = payload.get("usage_metadata")
+    if isinstance(usage, dict):
+        prompt = usage.get("input_tokens") or 0
+        completion = usage.get("output_tokens") or 0
+        total = usage.get("total_tokens") or (prompt + completion)
+        return TokenUsage(
+            prompt_tokens=int(prompt),
+            completion_tokens=int(completion),
+            total_tokens=int(total),
+        )
+    for holder in ("response_metadata", "additional_kwargs"):
+        meta = payload.get(holder)
+        if isinstance(meta, dict):
+            tu = meta.get("token_usage")
+            if isinstance(tu, dict):
+                prompt = int(tu.get("prompt_tokens") or 0)
+                completion = int(tu.get("completion_tokens") or 0)
+                return TokenUsage(
+                    prompt_tokens=prompt,
+                    completion_tokens=completion,
+                    total_tokens=int(tu.get("total_tokens") or (prompt + completion)),
+                )
+    return TokenUsage()
+
+
+class RoleStepBuilder:
+    """Accumulates canonical steps from a role-message stream.
+
+    Ids are collision-safe by construction: explicit call ids get role-scoped
+    suffixes (``<id>#decision`` / ``<id>#result``), anything else falls back to
+    sequential ids, and duplicates are disambiguated deterministically.
+    """
+
+    def __init__(self):
+        self.steps = []
+        self.task_input = {}
+        self.pending_calls = {}  # tool_call_id -> tool name
+        self._seen_ids = set()
+
+    def _unique(self, raw, idx):
+        candidate = raw or f"role_step_{idx}"
+        if candidate not in self._seen_ids:
+            self._seen_ids.add(candidate)
+            return candidate
+        n = 2
+        while f"{candidate}#{n}" in self._seen_ids:
+            n += 1
+        final = f"{candidate}#{n}"
+        self._seen_ids.add(final)
+        return final
+
+    def add_step(
+        self,
+        name,
+        step_type,
+        payload_in,
+        payload_out,
+        status=StepStatus.SUCCESS,
+        error_message=None,
+        tokens=None,
+        call_id=None,
+    ):
+        idx = len(self.steps)
+        self.steps.append(
+            TraceStep(
+                step_id=self._unique(call_id, idx),
+                parent_id=None,
+                step_index=idx,
+                step_type=step_type,
+                name=name,
+                input_payload=payload_in,
+                output_payload=payload_out,
+                status=status,
+                error_message=error_message,
+                latency_ms=0.0,
+                tokens=tokens or TokenUsage(),
+                metadata={},
+            )
+        )
+
+    def feed(self, message, *, task_prefix=""):
+        """Consumes one message dict into the builder.
+
+        Returns ``"step"`` when a step was appended, ``"input"`` when the
+        message was absorbed as task input, ``"skip"`` otherwise.
+        """
+        kind, payload = message_kind(message)
+
+        if kind == "human":
+            content = payload.get("content")
+            if not self.task_input and content:
+                self.task_input = {"input": content}
+            return "input"
+        if kind == "system":
+            return "skip"
+
+        prefix = f"{task_prefix}/" if task_prefix else ""
+
+        if kind == "ai":
+            tokens = usage_of(payload)
+            calls = tool_calls_of(payload)
+            content = payload.get("content")
+            if calls:
+                for call in calls:
+                    if call["id"]:
+                        self.pending_calls[call["id"]] = call["name"]
+                    self.add_step(
+                        name=f"{prefix}{call['name']}",
+                        step_type=StepType.ROUTING,
+                        payload_in={"arguments": call["args"]},
+                        payload_out={"decision": call["name"]},
+                        tokens=tokens,
+                        call_id=f"{call['id']}#decision" if call["id"] else None,
+                    )
+                return "step"
+            if content:
+                self.add_step(
+                    name=f"{prefix}response",
+                    step_type=StepType.LLM_CALL,
+                    payload_in={"prompt": ""},
+                    payload_out={"result": content},
+                    tokens=tokens,
+                )
+                return "step"
+            return "skip"
+
+        if kind == "tool":
+            call_id = payload.get("tool_call_id")
+            name = payload.get("name") or self.pending_calls.get(
+                call_id or "", "tool_response"
+            )
+            status = WASTED_STATUS.get(
+                str(payload.get("status")).lower(), StepStatus.SUCCESS
+            )
+            content = payload.get("content")
+            self.add_step(
+                name=f"{prefix}{name}",
+                step_type=StepType.TOOL_CALL,
+                payload_in={"tool_call_id": call_id} if call_id else {},
+                payload_out={"result": content},
+                status=status,
+                error_message=(
+                    content if status is StepStatus.ERROR and content else None
+                ),
+                call_id=f"{call_id}#result" if call_id else None,
+            )
+            return "step"
+
+        return "skip"
+
+    @property
+    def last_step_name(self):
+        return self.steps[-1].name if self.steps else None