tracefork 0.1.0__py3-none-any.whl

tracefork-0.1.0.dist-info/METADATA

@@ -0,0 +1,235 @@
+Metadata-Version: 2.4
+Name: tracefork
+Version: 0.1.0
+Summary: Time-travel debugger for AI agents: bit-exact record/replay, fork any step, causal blame with confidence intervals.
+Project-URL: Homepage, https://github.com/pratik916/tracefork
+Project-URL: Repository, https://github.com/pratik916/tracefork
+Project-URL: Issues, https://github.com/pratik916/tracefork/issues
+Project-URL: Changelog, https://github.com/pratik916/tracefork/blob/main/CHANGELOG.md
+Author-email: Pratik Soni <godofcode.pratik@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ai-agents,anthropic,causal-inference,debugging,determinism,llm,observability,record-replay,time-travel-debugger
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Debuggers
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: anthropic>=0.40
+Requires-Dist: fastapi>=0.115
+Requires-Dist: typer>=0.12
+Requires-Dist: uvicorn>=0.30
+Requires-Dist: zstandard>=0.22
+Provides-Extra: dev
+Requires-Dist: mypy>=1.11; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest-cov>=5; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# tracefork
+
+[![CI](https://github.com/pratik916/tracefork/actions/workflows/ci.yml/badge.svg)](https://github.com/pratik916/tracefork/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/)
+[![Code style: ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+
+**A time-travel debugger for AI agents that doesn't just replay a failed run — it
+proves the replay is bit-for-bit real, lets you fork any step, and measures *which*
+step caused the failure, with confidence intervals.**
+
+![tracefork report — timeline, exchange detail, and causal blame panel](docs/demo.png)
+
+*The three-panel report: a run's timeline (left) with causal-blame badges, the
+request/response for the selected exchange (center), and the blame ranking with 95%
+confidence intervals (right). Generated offline, for $0, by
+[`examples/demo_report.py`](examples/demo_report.py).*
+
+---
+
+## The idea
+
+Every agent-observability tool shows you a trace and asks you to eyeball it. tracefork
+treats an agent run like a recording you can rewind, branch, and reason about causally:
+
+- **Record** every model call into a content-addressed **tape** at the HTTP seam of the
+  Anthropic SDK, capturing the sources of nondeterminism (clock, ids) the agent reads.
+- **Replay** the tape **bit-exact for $0** — every replayed request's *body* is
+  sha256-checked against the tape, so it's *proven* identical, not asserted. (The matched
+  surface is the request body; request headers such as `anthropic-beta` are out of scope —
+  see [Determinism boundary](#determinism-boundary-v1-honest-scope).) No network, no key.
+- **Fork** any step: swap in a different model response and let the *same* agent run
+  forward from there. The unchanged prefix replays for free; only the new tail costs
+  anything.
+- **Blame**: resample those forks across every step and rank each by its **flip-rate** —
+  how often perturbing it changes the run's outcome — with **Wilson score** confidence
+  intervals so a small sample can't masquerade as certainty.
+- **Validate the instrument itself**: inject faults with *known* root causes and confirm
+  the blame engine fingers the right step. The engine is genuinely causal — it ranks
+  whichever step actually flips the outcome #1, not a fixed slot — and across five
+  injection mechanisms it hits **1.00 top-1 precision** offline against a flat negative
+  control (which is now *enforced*, not just printed). See
+  [Validation scope](#validation-scope) for exactly what that number does and doesn't claim.
+
+That last pillar is the point: a debugger that claims to find root causes has to be
+held to ground truth. `tracefork validate` is that proof, and it runs in under a second
+with no API key.
+
+## Quickstart (offline, $0, no API key)
+
+Python **3.12** via [uv](https://docs.astral.sh/uv/). Everything below is offline and
+makes no network calls.
+
+```bash
+uv sync --extra dev
+
+# 1. The full offline test suite (65 tests).
+uv run pytest -q
+
+# 2. The instrument validates itself against injected, known-root-cause faults.
+uv run tracefork validate
+
+# 3. Generate the demo report shown above, then open it in any browser.
+uv run python examples/demo_report.py
+open examples/demo_report.html      # macOS; or just open the file
+
+# 4. The original Spike 0 receipt: record → persist → replay → prove bit-exact.
+uv run python -m tracefork_spike
+```
+
+`tracefork validate` prints:
+
+```
+  [PASS] corrupted_tool_output               top-1: 1.00
+  [PASS] misleading_retrieval                top-1: 1.00
+  [PASS] wrong_system_prompt                 top-1: 1.00
+  [PASS] dropped_message                     top-1: 1.00
+  [PASS] poisoned_argument                   top-1: 1.00
+
+  overall top-1 precision: 1.00
+  negative control max flip: 0.00 (threshold 0.30)
+```
+
+## The CLI
+
+```bash
+uv run tracefork --help
+```
+
+| Command | What it does |
+|---|---|
+| `replay  <tape> --agent pkg.mod:fn` | Replay a tape and print the bit-exact verification receipt. |
+| `verify  <tape> --agent pkg.mod:fn` | Verify replay; exit non-zero on drift (CI gate). |
+| `fork    <run_id> --step N --response f --agent pkg.mod:fn` | Fork a run at step N with a mutated response; record the counterfactual branch. |
+| `blame   <run_id> --agent pkg.mod:fn [--k 10] [--budget 5.0]` | Rank every step by causal flip-rate with 95% CIs (re-runs the agent; budget-capped). |
+| `report  <run_id> \| --tape <tape> -o out.html` | Render the self-contained three-panel HTML report. |
+| `serve   [--store store.db] [--port 7777]` | Serve the live web UI (same-origin, 127.0.0.1). |
+| `validate [--k 3] [--n-runs 5] [--check]` | Run the fault-injection suite; `--check` gates against the committed report. |
+
+Replay, verify, fork, and the offline demos need no key. `blame` against a *real* run
+re-runs the agent's counterfactual tails against the live API, which is why it's
+budget-capped — the offline, $0 proof that blame works is `tracefork validate`.
+
+## How it works
+
+The spine is a **record/replay seam at the Anthropic SDK's httpx boundary** plus a
+**nondeterminism-virtualization seam** the agent reads time and ids through. Bit-exactness
+is the contract between them.
+
+- **`transport.py`** — `TraceforkTransport` (sync) / `AsyncTraceforkTransport` (async).
+  Record mode tees request+response bytes into the tape (buffering streaming SSE and
+  plain JSON identically via `.read()`/`.aread()`); replay mode serves recorded bytes and
+  sha256-asserts every request body matches the tape. A replay transport has **no inner
+  transport**, so an unrecorded request is a hard error, never a silent network call.
+- **`tape.py`** — content-addressed (sha256) blobs + an ordered event log, persistable to
+  SQLite, with a hash-chain `digest()` fingerprint.
+- **`nondet.py`** — `NondetSource` is the only way the agent gets time/ids;
+  `RecordingNondet` logs real draws, `ReplayNondet` serves them back, `DriftingNondet` is
+  the negative control. `find_divergence()` unwraps the `DivergenceError` the SDK buries
+  inside an `APIConnectionError` so a real divergence isn't mistaken for a network blip.
+- **`fork.py`** — `ForkTransport` runs three phases: **prefix-replay** (served from the
+  parent tape for $0, request asserted to match — the agent must be deterministic up to
+  the fork point), **mutation-injection** (same request, swapped response), and
+  **tail-record** (the counterfactual continuation recorded fresh). A `Branch` carries
+  `prefix_replayed`/`tail_recorded` counters that quantify the savings.
+- **`blame.py`** — forks each step `k` times, re-runs the agent, grades the outcome via an
+  `Oracle`, and counts flips vs. the parent outcome. `wilson_ci()` gives the interval;
+  `BudgetGovernor` estimates fork count and dollar cost before any spend.
+- **`faults.py` / `validate.py`** — five fault classes, each producing *valid* Anthropic
+  JSON with a marker embedded inside a content field. A synthetic agent echoes each
+  response into its next request, so an injected fault propagates through a fork to a
+  fault-aware tail and flips the outcome — letting the blame engine be scored against
+  ground truth entirely offline.
+- **`report.py` / `server.py` / `web/report.html`** — a single, dependency-free HTML file
+  (vanilla JS, no npm) rendered statically by `report` or served live by `serve`.
+
+## Determinism boundary (v1, honest scope)
+
+Bit-exact replay holds within a declared boundary: **single-process, clock + id
+nondeterminism, captured through `NondetSource`**. An agent that reads `datetime.now()` /
+`uuid` / `random` directly, or runs its loop across threads/subprocesses, steps outside
+that boundary — and the verifier will *detect* the resulting drift rather than paper over
+it. Forking and blame assume the agent rebuilds its prefix deterministically (the same
+property replay proves). See [`SPIKE0.md`](SPIKE0.md) for how the boundary was de-risked.
+
+## Validation scope
+
+What `tracefork validate` proves, stated precisely: the blame engine is **genuinely
+causal** — inject an outcome-flipping fault at *any* step and the engine ranks that step
+first (verified by also injecting at a non-root step), so the 1.00 is not a tautology or a
+fixed-slot artifact. The five "fault classes" carry two real injection mechanisms (a
+corrupted tool argument and a replaced text message) via a marker that survives the SDK's
+JSON round-trip, and the negative control — a no-op perturbation that must not flip the
+outcome — is enforced with a hard threshold (the run fails if it ever exceeds 0.30).
+
+What it does **not** yet claim: discrimination among *several competing* plausible causes
+on a long run. The fixture is a short tape where one step gets a flip-capable perturbation
+and the rest get an inert one — a clean positive-vs-control, but an easy one. A longer tape
+with a decoy step that changes the transcript without changing the outcome is the next
+iteration; until then, read 1.00 as "the instrument reliably finds the planted cause," not
+"it resolves ambiguous multi-cause blame."
+
+## Layout
+
+```
+src/tracefork/      transport, tape, nondet, recorder, fork, store,
+                    blame, faults, validate, report, server, wire, synthetic, cli
+src/tracefork_spike/  the original bit-exact record/replay spike
+web/report.html     the single-file three-panel UI
+examples/           runnable demo that produces the report above
+tests/              65 offline tests ($0, no key)
+experiments/        committed reference report for `validate --check`
+```
+
+## Testing
+
+```bash
+uv run pytest -q                                   # all 65 offline tests
+uv run pytest tests/test_faults.py -q              # the self-validation chain
+uv run tracefork validate --check                  # regression-gate vs committed report
+```
+
+## Contributing
+
+Contributions are welcome — see [`CONTRIBUTING.md`](CONTRIBUTING.md) for dev setup,
+the invariants a PR must respect, and commit/PR conventions. The whole dev loop
+(tests, `validate`, lint, type-check) is offline and $0, so you can run the full gate
+with no API key. Please also read the [Code of Conduct](CODE_OF_CONDUCT.md).
+
+## Security
+
+See [`SECURITY.md`](SECURITY.md) for how to report a vulnerability. In short: tapes
+are JSON + base64 (never pickle, so loading one can't execute code), and `tracefork
+serve` binds to 127.0.0.1 only.
+
+## License
+
+MIT — see [`LICENSE`](LICENSE).

tracefork-0.1.0.dist-info/RECORD

@@ -0,0 +1,32 @@
+tracefork/__init__.py,sha256=U-ttX1vJOy7j4PYvbTRwhu_4UXfhJmWwS6X2q0Qo158,175
+tracefork/blame.py,sha256=Rdnl4pfnm7CUftQtea1kfrpIlVTCU_Ge914ndJEhmH8,10802
+tracefork/cli.py,sha256=tO1w6tCocsUUYsx9MHFpx2zTLJgbOo06Gko2ibDGxoU,13430
+tracefork/constants.py,sha256=xBqcMokay6vOOU27v8iqUq8AQlTF-uB33LX9FM77F0g,936
+tracefork/faults.py,sha256=iM59in35CCEkFJ6_a35ev-g__51wwbBfA-iy5eI9cNk,5291
+tracefork/fork.py,sha256=CRR1LHHAPq7-fAPM38ts-71X0_frR3U5XuwtU3bbHN4,6334
+tracefork/nondet.py,sha256=odpYOSxTwV0srleC7xU0ErTBtE8BxnBD9f-P7N2JzzM,3083
+tracefork/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+tracefork/recorder.py,sha256=0NnA91Arq72u_dqUkohs0P0CCu6WHqEbGzV6KPEPbSg,5627
+tracefork/replay.py,sha256=6BgPBq6wgMYg0cPY2l98BQn-9SxZAzLT0-88j8W_LDc,3625
+tracefork/report.py,sha256=kzm1PDe7ajr7_M33oRaEZ0hZo4h6b91L5aZ5t_22Ivc,4628
+tracefork/server.py,sha256=jntUa-KX9qglsq1FS8kd21PS2946Woi2NZOQdtPk67o,2349
+tracefork/store.py,sha256=0qiwuoU2VeB9sa-aa_Dji7ksylrskvc434Nr12rRUbE,4427
+tracefork/synthetic.py,sha256=mikKoulr_pIpM7j6T4uBimPenyQonU-dHGXYrYwcXt0,3851
+tracefork/tape.py,sha256=eb-rm6zFGSOOn2ftjWh7dRuaW3w5zzljghEEkczgUOw,5082
+tracefork/transport.py,sha256=bM6AbjtQvYBLbQkCUNhlaadukfpWRTqWMY6n0CF0qPQ,4960
+tracefork/validate.py,sha256=upgFnRfxmgfysB2GmDf3GQRiLaUmb7Cjvl-pdhfu5MA,6821
+tracefork/wire.py,sha256=Bb0u_VSuYuWBZmDNOtHpCIblKU6XbF8dbbaQ8ZV9184,2199
+tracefork_spike/__init__.py,sha256=c_g12WwUaAn_mxKW2eQLG0WWx4zIwfrGIzKaN1mcNhA,246
+tracefork_spike/__main__.py,sha256=HnbKfCaV-iv-zjLiy1gciGBrWYt6muRXf0pTH2u_K9U,50
+tracefork_spike/agent.py,sha256=HsqBK39HtNjzYSrtUds6EpLyJMNUm2bbuym4oZB3ITU,3223
+tracefork_spike/fake_llm.py,sha256=V55y9zrPLLqsQWNSPfJz7hRbN33Zg3lb0oazfk38UAo,4094
+tracefork_spike/nondet.py,sha256=UPgMasUy8yRkdsFpqM3o-__JV_xc13LEZ8jEWqRwCJc,3298
+tracefork_spike/spike.py,sha256=rg20GHMKHJRBRe6HcEftY8NEau80ACUH7GRQ9WgdF5g,5214
+tracefork_spike/tape.py,sha256=AUlZvkP1hQXPUWa4chJaZLFdnOGGLIHtu45pKJeZYb8,3230
+tracefork_spike/transport.py,sha256=NCVURT18wUELqf9Z_z9iZxItH-D8YMk3FrjNPp9NfYM,2643
+tracefork/web/report.html,sha256=6NWNTrUy38zP_eZmqdvBhH7KrkU1etmv4QQHeMFaX3k,9857
+tracefork-0.1.0.dist-info/METADATA,sha256=mPQQKaDYHze_LLdt7g-Nvr_yKUN-Yd4dYaoxgVYP6AU,12277
+tracefork-0.1.0.dist-info/WHEEL,sha256=mffPy8wBnZQn2VnJUU5jE99KsxaSfiyMHV9Yt0aLVxs,87
+tracefork-0.1.0.dist-info/entry_points.txt,sha256=w8wDABmxwUHKYLeZ9q6VplfnMftA0Jws2hLZk0wNgCo,48
+tracefork-0.1.0.dist-info/licenses/LICENSE,sha256=Nze8S2cJmJD6Qe6HpaIXVJKAvh7bhrUfrksikSkS9V8,1068
+tracefork-0.1.0.dist-info/RECORD,,

tracefork-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.30.1
+Root-Is-Purelib: true
+Tag: py3-none-any

tracefork-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+tracefork = tracefork.cli:app

tracefork-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Pratik Soni
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tracefork_spike/__init__.py

@@ -0,0 +1,7 @@
+"""tracefork Spike 0 — bit-exact record/replay of an Anthropic-SDK agent run."""
+
+from .nondet import DivergenceError
+from .spike import record_replay_verify
+from .tape import Tape
+
+__all__ = ["record_replay_verify", "Tape", "DivergenceError"]

tracefork_spike/__main__.py

@@ -0,0 +1,3 @@
+from .spike import main
+
+raise SystemExit(main())

tracefork_spike/agent.py

@@ -0,0 +1,91 @@
+"""A tiny tool-using agent built on the real Anthropic SDK.
+
+This is the "agent under recording". It is deliberately nondeterministic: the
+`book_flight` tool stamps a wall-clock `booked_at` and a fresh `confirmation_id` on
+every run. Those values flow into the *next* request body, so an honest replay can
+only be byte-exact if that nondeterminism was captured and virtualized. The agent
+reads time/ids exclusively through the injected `NondetSource`.
+"""
+
+from __future__ import annotations
+
+import json
+
+import anthropic
+import httpx
+
+from .nondet import NondetSource, find_divergence
+
+MODEL = "claude-opus-4-8"
+
+TOOLS = [
+    {
+        "name": "book_flight",
+        "description": "Book a flight to a destination.",
+        "input_schema": {
+            "type": "object",
+            "properties": {
+                "destination": {"type": "string"},
+                "seats": {"type": "integer"},
+            },
+            "required": ["destination", "seats"],
+        },
+    }
+]
+
+
+def make_client(transport: httpx.BaseTransport) -> anthropic.Anthropic:
+    return anthropic.Anthropic(
+        api_key="sk-ant-offline-fake",
+        http_client=httpx.Client(transport=transport),
+        max_retries=0,
+    )
+
+
+def _execute_tool(name: str, tool_input: dict, nondet: NondetSource) -> dict:
+    if name == "book_flight":
+        return {
+            "confirmation_id": nondet.new_id("CONF"),  # virtualized nondeterminism
+            "booked_at": nondet.now_iso(),  # virtualized nondeterminism
+            "destination": tool_input["destination"],
+            "seats": tool_input["seats"],
+        }
+    raise ValueError(f"unknown tool {name!r}")
+
+
+def run_agent(client: anthropic.Anthropic, nondet: NondetSource) -> dict:
+    """Run the agent loop to completion; return its observable trajectory."""
+    messages: list[dict] = [{"role": "user", "content": "Book me a flight to Tokyo."}]
+    turns: list[dict] = []
+
+    while True:
+        try:
+            resp = client.messages.create(
+                model=MODEL, max_tokens=1024, tools=TOOLS, messages=messages
+            )
+        except anthropic.APIConnectionError as e:
+            # The SDK masks transport-layer exceptions as connection errors; recover
+            # a replay DivergenceError so callers see the real cause.
+            div = find_divergence(e)
+            if div is not None:
+                raise div from None
+            raise
+        turns.append({"id": resp.id, "stop_reason": resp.stop_reason})
+        messages.append({"role": "assistant", "content": resp.content})
+
+        if resp.stop_reason != "tool_use":
+            final_text = next((b.text for b in resp.content if b.type == "text"), "")
+            return {"final_text": final_text, "turns": turns}
+
+        results = []
+        for block in resp.content:
+            if block.type == "tool_use":
+                out = _execute_tool(block.name, dict(block.input), nondet)
+                results.append(
+                    {
+                        "type": "tool_result",
+                        "tool_use_id": block.id,
+                        "content": json.dumps(out),
+                    }
+                )
+        messages.append({"role": "user", "content": results})

tracefork_spike/fake_llm.py

@@ -0,0 +1,106 @@
+"""A fake Anthropic endpoint as an httpx transport.
+
+This stands in for the real /v1/messages API so the spike needs no key and no
+network ($0, offline, CI-safe). It emits real Anthropic *wire-format* JSON so the
+genuine `anthropic` SDK parses it into real `Message` objects — i.e. the spike
+exercises the actual SDK + transport seam, not a hand-rolled client.
+
+When a real key is available, this inner transport is simply swapped for the SDK's
+real network transport; the recording/replay machinery around it is unchanged. That
+is the whole point: the seam is provider-real.
+
+The fake is a two-turn agent script: first request -> a `tool_use` for `book_flight`;
+second request (which now carries a `tool_result`) -> a final `end_turn` answer that
+echoes the confirmation id the agent's tool produced.
+"""
+
+from __future__ import annotations
+
+import json
+
+import httpx
+
+from .tape import sha256_hex
+
+
+def _has_tool_result(payload: dict) -> bool:
+    for m in payload.get("messages", []):
+        content = m.get("content")
+        if isinstance(content, list):
+            for block in content:
+                if isinstance(block, dict) and block.get("type") == "tool_result":
+                    return True
+    return False
+
+
+class FakeAnthropicTransport(httpx.BaseTransport):
+    """Deterministic given the request bytes (ids derived from the request hash)."""
+
+    def handle_request(self, request: httpx.Request) -> httpx.Response:
+        payload = json.loads(request.content)
+        model = payload.get("model", "claude-opus-4-8")
+        rid = "msg_" + sha256_hex(request.content)[:20]
+
+        if not _has_tool_result(payload):
+            toolu = "toolu_" + sha256_hex(request.content)[:18]
+            message = {
+                "id": rid,
+                "type": "message",
+                "role": "assistant",
+                "model": model,
+                "content": [
+                    {"type": "text", "text": "Booking your flight now."},
+                    {
+                        "type": "tool_use",
+                        "id": toolu,
+                        "name": "book_flight",
+                        "input": {"destination": "Tokyo", "seats": 1},
+                    },
+                ],
+                "stop_reason": "tool_use",
+                "stop_sequence": None,
+                "usage": {"input_tokens": 48, "output_tokens": 22},
+            }
+        else:
+            confirmation = _last_tool_result_confirmation(payload)
+            message = {
+                "id": rid,
+                "type": "message",
+                "role": "assistant",
+                "model": model,
+                "content": [
+                    {
+                        "type": "text",
+                        "text": (
+                            f"Done — your flight to Tokyo is booked. Confirmation {confirmation}."
+                        ),
+                    }
+                ],
+                "stop_reason": "end_turn",
+                "stop_sequence": None,
+                "usage": {"input_tokens": 96, "output_tokens": 18},
+            }
+
+        return httpx.Response(
+            200,
+            headers={"content-type": "application/json"},
+            content=json.dumps(message).encode(),
+            request=request,
+        )
+
+
+def _last_tool_result_confirmation(payload: dict) -> str:
+    """Pull the confirmation id out of the most recent tool_result so the final
+    answer references it — makes the agent's nondeterminism observable end-to-end."""
+    for m in reversed(payload.get("messages", [])):
+        content = m.get("content")
+        if isinstance(content, list):
+            for block in content:
+                if isinstance(block, dict) and block.get("type") == "tool_result":
+                    raw = block.get("content")
+                    text = raw if isinstance(raw, str) else json.dumps(raw)
+                    try:
+                        return json.loads(text).get("confirmation_id", "UNKNOWN")
+                    except (ValueError, TypeError):
+                        return "UNKNOWN"
+    return "UNKNOWN"

tracefork_spike/nondet.py

@@ -0,0 +1,97 @@
+"""Virtualized nondeterminism sources.
+
+The headline tracefork claim is bit-exact replay. That is only possible if every
+source of nondeterminism the agent consumes is captured at record time and served
+back identically at replay time. This module is the seam: the toy agent reads the
+clock and generates IDs *only* through a `NondetSource`, never through `time` /
+`uuid` directly. Swap `RecordingNondet` for `ReplayNondet` and the same agent code
+produces a byte-identical trajectory.
+
+`DriftingNondet` is the negative control: it draws fresh real values during replay,
+which must make the replay diverge — proving the verifier actually detects drift
+rather than always passing.
+"""
+
+from __future__ import annotations
+
+import datetime
+import uuid
+from typing import Protocol
+
+
+class DivergenceError(RuntimeError):
+    """Raised when a replay diverges from the recorded tape."""
+
+
+def find_divergence(exc: BaseException | None) -> DivergenceError | None:
+    """Walk an exception's cause/context chain for a DivergenceError.
+
+    The Anthropic SDK wraps any exception raised inside its httpx transport in an
+    `APIConnectionError`, so a divergence we raise from the replay transport arrives
+    as `APIConnectionError.__cause__` (possibly nested). This recovers the original."""
+    seen: set[int] = set()
+    cur = exc
+    while cur is not None and id(cur) not in seen:
+        seen.add(id(cur))
+        if isinstance(cur, DivergenceError):
+            return cur
+        cur = cur.__cause__ or cur.__context__
+    return None
+
+
+class NondetSource(Protocol):
+    def now_iso(self) -> str: ...
+    def new_id(self, prefix: str) -> str: ...
+
+
+class RecordingNondet:
+    """Draws genuinely real values (wall clock, random UUIDs) and logs each draw."""
+
+    def __init__(self) -> None:
+        self.draws: list[tuple[str, str]] = []
+
+    def now_iso(self) -> str:
+        v = datetime.datetime.now(datetime.UTC).isoformat()
+        self.draws.append(("clock", v))
+        return v
+
+    def new_id(self, prefix: str) -> str:
+        v = f"{prefix}_{uuid.uuid4().hex[:16]}"
+        self.draws.append(("id", v))
+        return v
+
+
+class ReplayNondet:
+    """Serves recorded draws back in order; errors on order/kind/exhaustion mismatch."""
+
+    def __init__(self, draws: list[tuple[str, str]]) -> None:
+        self._draws = list(draws)
+        self._i = 0
+
+    def _next(self, kind: str) -> str:
+        if self._i >= len(self._draws):
+            raise DivergenceError(
+                f"replay asked for a {kind!r} draw but the tape is exhausted "
+                f"(consumed {self._i}/{len(self._draws)})"
+            )
+        rec_kind, value = self._draws[self._i]
+        if rec_kind != kind:
+            raise DivergenceError(
+                f"draw #{self._i}: replay asked for {kind!r}, tape has {rec_kind!r}"
+            )
+        self._i += 1
+        return value
+
+    def now_iso(self) -> str:
+        return self._next("clock")
+
+    def new_id(self, prefix: str) -> str:
+        return self._next("id")
+
+    def fully_consumed(self) -> bool:
+        return self._i == len(self._draws)
+
+
+class DriftingNondet(RecordingNondet):
+    """Negative control: behaves like RecordingNondet (fresh real values) during a
+    replay, which makes the rebuilt request bytes diverge from the tape."""