agenticreplay 0.1.0 → 0.1.2

package/server.json

@@ -1,28 +0,0 @@
-{
-  "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
-  "name": "io.github.Continuum-AI-Corp/agenticreplay",
-  "description": "Read, replay and fork recorded coding-agent runs.",
-  "repository": {
-    "url": "https://github.com/askdkc/AgenticReplay",
-    "source": "github"
-  },
-  "version": "0.2.2",
-  "packages": [
-    {
-      "registryType": "npm",
-      "identifier": "agenticreplay",
-      "version": "0.2.2",
-      "transport": {
-        "type": "stdio"
-      },
-      "runtimeArguments": [
-        {
-          "type": "positional",
-          "value": "mcp",
-          "isRequired": true,
-          "description": "Serves the six read-and-replay tools over stdio."
-        }
-      ]
-    }
-  ]
-}

package/skills/agenticreplay-replay/SKILL.md

@@ -1,211 +0,0 @@
----
-name: agenticreplay-replay
-description: Answers questions about a past agent run from its recording rather than from memory, and replays or forks that run. Use when asked why an earlier run did something, or to reproduce a failure.
-license: Apache-2.0
-compatibility: Requires the `agenticreplay` npm package (Node 24.0.0+) with its MCP server registered as `agenticreplay`, and at least one recorded run in the project's .agenticreplay/runs directory.
-metadata:
-  author: Continuum-AI-Corp
-  version: "0.1"
-  homepage: https://github.com/askdkc/AgenticReplay
----
-
-# Reading a recorded agent run
-
-A recording is evidence. Your memory of a session is not, and neither is a transcript you were
-handed — both are missing the tool results, the exit codes, and the files that changed without
-anyone mentioning it.
-
-**The rule: when a question is about something that already happened, read the trace before you
-answer.** Do not reconstruct it. If a recording exists, guessing is the wrong move even when the
-guess would have been right.
-
-## When to Use This Skill
-
-- "Why did you delete/overwrite/move X?"
-- "What changed this file?" / "Which step broke the build?"
-- "Can you reproduce yesterday's failure?"
-- "Does this still reproduce?" (see the limit on that in step 4 — replay cannot tell you
-  whether a *fresh* run would fail again)
-- "Would a different model have got this right?"
-
-## Workflow
-
-### 1. Find the run
-
-`agenticreplay_list_runs` — newest first, and it names the run each fork came from. Skip this only when the
-user clearly means the most recent one; every other tool defaults to `run: "last"`.
-
-### 2. Narrow to the chain that produced the thing being asked about
-
-`agenticreplay_show_run` gives the whole timeline: model turns with token counts and stop reasons, tool
-calls with arguments and results, shell commands with exit codes, and every file the run changed.
-Good for orientation, long for a specific question.
-
-`agenticreplay_graph` is usually the better tool. It returns causal edges — which event produced which. Pass
-`to: <event seq>` to get **only** the chain that produced one event. That is the shape of an answer
-to "why did this happen", where the full timeline is the shape of an answer to "what happened".
-
-### 3. Report `recorded` and `inferred` differently
-
-Every edge from `agenticreplay_graph` is labelled:
-
-- **`recorded`** — the recorder watched it happen and wrote it into the trace.
-- **`inferred`** — derived just now from a rule the edge names. The trace does not vouch for it.
-
-Carry that distinction into your answer. "The trace shows the `rm` at step 14 removed it" and "this
-looks like the `rm` at step 14, going by timing" are different claims, and flattening them into one
-confident sentence is the specific failure this tool exists to prevent. Name the rule when you lean
-on an inferred edge.
-
-### 4. Reproduce it before explaining it
-
-`agenticreplay_replay` re-runs the recording and reports what could not be reproduced — divergences, and
-requests the recording could not serve.
-
-**What "offline" covers, and what it does not.** Every model response comes from the trace and the
-proxy forwards nothing upstream, so no provider is contacted and no tokens are spent. An unmatched
-request halts the replay rather than falling through to the network, unless `--loose` was asked for.
-
-That covers the model traffic. It does not cover the agent's own subprocesses: unless the recording
-used `--tls-intercept` — in which case replay re-establishes interception for the hosts it recorded
-— a `curl`, `npm install`, `git push` or database call inside a recorded shell command goes
-straight out. Replay is not a sandbox; only a network-isolated container makes it one.
-
-**What a matching replay proves, and what it does not.** It shows the recorded decisions reproduce
-against today's environment. It cannot show the failure is deterministic, because the model is not
-being asked again — the same recorded responses are served back. If the user wants to know whether a
-fresh run would fail the same way, say that replay cannot answer it; that needs real runs.
-
-**Replay re-executes the agent, not just its model traffic.** The recorded model responses are
-served from the trace, but the agent process runs again for real — so every shell command it issued
-runs again too. `worktree: true` isolates repository files and nothing else. Anything the run
-touched outside the tree — `/tmp`, Docker, a local database, a package manager, another host — is
-mutated a second time.
-
-**So check before the first replay of a run, not after.** Read its shell commands with
-`agenticreplay_show_run` and tell the user what will re-execute. If any of it reached outside the working
-tree, get approval for that specifically or replay inside a container; do not treat the earlier
-`worktree` answer as covering it. A run that only read files and edited the repository is free and
-repeatable, and worth replaying before committing to any explanation.
-
-**Pass `worktree: true`.** It replays into a scratch copy and leaves the working tree alone.
-
-Without it, replay is destructive for as long as it runs: it restores the recorded filesystem over
-the working tree and puts the tree back when the replay ends. Uncommitted work is absent in the
-meantime, and stays absent if the replay is interrupted before it can restore. Run an in-place
-replay only when the user has been told that and has agreed to it. "They do not appear to be
-typing" is not consent.
-
-A replay reporting `reused=3/5` on an interactive recording is not a partial failure. Harnesses make
-calls for themselves — a quota probe, a session-naming request — and a replay does not repeat them.
-
-### 5. Only then consider comparing models
-
-`agenticreplay_compare` forks one run onto several models from the same checkpoint: same files, same
-conversation prefix, so the model is the only variable. Pick the fork point with `agenticreplay_checkpoints`
-and pass it as `from`.
-
-Grade with `verify` — a shell command whose exit code is the verdict. Use something the repository
-already declares (`"npm test"`, `"npm run typecheck"`) or an explicitly local binary
-(`"./node_modules/.bin/tsc --noEmit"`), not `npx <tool>`: with no local install, npx runs whatever
-the registry has under that name, and `npx tsc` resolves a package deprecated in 2016 that is not
-TypeScript.
-
-**`agenticreplay_compare` uploads the recording to other people's models, and spends real money doing it.**
-Each model named receives the same files and conversation prefix the original run had — so whatever
-that run touched (source, prompts, configuration, anything a credential was pasted into) is sent to
-every provider behind those model ids.
-
-**And each fork is a live agent, not a replay.** From the fork point onward the model is really
-being asked, and whatever it decides to do, it does — its shell commands execute for real, and so
-does the `verify` command you pass. Each fork gets its own worktree, so repository files are
-isolated per model; nothing outside the tree is. A fork can also take actions the original run never
-took, because it is a different model making fresh decisions.
-
-So the approval has three parts, and they are not the same question:
-
-1. **Disclosure** — what context is uploaded, and to which providers. Approving a bill is not
-   approving a disclosure, and the two need separate answers when the recording is from a private
-   codebase. `agenticreplay scrub` is for when the comparison is worth running but the trace is not safe to
-   send as-is.
-2. **Side effects** — what the recorded run did outside its worktree, since each fork may repeat it
-   and may go further. Same check as step 4, `agenticreplay_show_run`, and the same answer if it reached
-   Docker, a database, a deployment or another host: get approval for that specifically, or run the
-   comparison in an isolated environment.
-3. **Cost** — how many models times how many forks.
-
-Never run it to satisfy curiosity the user did not express.
-
-## If there is no recording yet
-
-Say so plainly rather than falling back to guessing, and offer to start one.
-
-If `agenticreplay` is already installed:
-
-```console
-agenticreplay record claude           # or codex, opencode, openclaw, grok
-```
-
-If it is not, ask before installing it — a global install changes the user's machine, and that is
-their call, not a detail of your task:
-
-```console
-npm i -g agenticreplay     # ask first
-```
-
-`agenticreplay record <agent>` runs the agent unmodified behind a local proxy. Nothing about the agent
-changes; two environment variables get set. Recording a session now is what makes the next "why did
-it do that" answerable.
-
-For a run started with a prompt in argv — `agenticreplay record claude -- -p "…"` — the replay is exact. A
-session someone typed into replays approximately, because the prompts were never on the wire and
-are recovered from the harness's own transcript; `agenticreplay replay` says which is which rather than
-papering over it.
-
-## Sharing a run with someone else
-
-`agenticreplay export last -o run.html` writes one self-contained file. A trace holds whatever the run held,
-so run `agenticreplay scrub` before sending one anywhere.
-
-Scrubbing is best-effort, not a guarantee. It matches known key shapes and high-entropy strings; it
-cannot know that a particular internal hostname, customer name, or unreleased feature is
-confidential to this user. So scrub, then have the user look at what is actually going out, and get
-their agreement — do not describe a scrubbed trace as safe on the strength of the scrubber alone.
-
-## Limitations
-
-- **It only sees what was recorded.** Runs started without `agenticreplay record` leave no trace, and
-  nothing here recovers them. The answer to "why did it do that" in an unrecorded session is
-  honestly "there is no recording", not a reconstruction.
-- **A typed session replays approximately, not exactly.** Prompts entered at a terminal were never
-  on the wire; agenticreplay recovers them from the harness's own transcript. Only a run started with the
-  prompt in argv (`agenticreplay record claude -- -p "…"`) replays byte-for-byte.
-- **Some turns are not repeated.** A harness makes calls for itself — a quota probe, a
-  session-naming request — and a replay steps over them. Tools that need a person
-  (`AskUserQuestion`, plan mode) are absent when the same agent runs without one, which can make a
-  replayed request differ from the recorded one by enough to halt.
-- **`inferred` edges are not evidence.** They are derived from a named rule at query time. Treat
-  them as a reading of the trace, never as something the recorder witnessed.
-- **Not every harness is recordable.** Agents that read no base-URL variable and pin their own
-  origin need `--tls-intercept`, and some cannot be reached at all. A recording that came back
-  empty means the harness was not captured, not that nothing happened.
-- **Replay is not a time machine, and not a sandbox.** It reproduces the agent's side of the run
-  against today's world. External state the run depended on — a database row, a remote branch, the
-  clock — is whatever it is now, and the run's own shell commands reach it for real.
-- **A matching replay is not a determinism result.** The model is not re-asked; its recorded
-  responses are served back. Whether a fresh run would fail the same way is a different question
-  that replay cannot answer.
-
-## Tools
-
-| tool | arguments | notes |
-|---|---|---|
-| `agenticreplay_list_runs` | — | newest first, names the parent of each fork |
-| `agenticreplay_show_run` | `run` | the full timeline |
-| `agenticreplay_checkpoints` | `run` | where a fork can start |
-| `agenticreplay_graph` | `run`, `to` | causal edges; `to` narrows to one chain |
-| `agenticreplay_replay` | `run`, `worktree` | offline, free, repeatable |
-| `agenticreplay_compare` | `run`, `models`*, `from`, `verify` | **spends real tokens** |
-
-`run` accepts a run id or `"last"`, and defaults to `"last"`. Replay traces are skipped when
-resolving `"last"`, so it means the newest run you actually recorded.

package/spec/agenticreplay-trace-v0.md

@@ -1,223 +0,0 @@
-# AgenticReplay Trace Format — v0
-
-Status: **Draft**. Schema version `0.4.0`. AgenticReplay readers reject older formats; existing recordings are not modified.
-
-This document is normative. The JSON Schema files in `packages/schema/schema/` are the
-machine-readable form of this document and take precedence where they disagree with the prose.
-TypeScript and Python types are *generated from or verified against* the schema, never the reverse.
-
-Licensed CC BY 4.0 so that other implementations may quote it freely.
-
-## 1. Layout
-
-```
-.agenticreplay/runs/<run_id>/
-├── manifest.json      # run metadata, integrity root, schema version
-├── events.jsonl       # append-only, totally ordered event log
-├── blobs/<ab>/<sha256>  # content-addressed payloads
-├── fs/                # shadow git object store (optional)
-└── redactions.json    # what was removed, by rule, with stable placeholders
-```
-
-`run_id` MUST match `^run_[0-9a-f]{6,32}$`.
-
-## 2. events.jsonl
-
-### Capture evidence in 0.4
-
-`net.request` with `phase: "start"` identifies a physical attempt before origin dispatch. Its
-`request_id`, timestamp, destination, method and assigned turn remain fixed through completion.
-The body record uses `phase: "body"`; its response names that record in `causes` and carries the
-same `request_id`. Retries receive different identities. Native hosts additionally supply explicit
-session/model-call context; the last active session is never an ownership oracle.
-
-Transport bodies retain serialized text, unknown fields and complete SSE content after redaction.
-They do not reveal provider-internal reasoning or SDK transformations before the stated boundary.
-`bytes` measures observed transport bytes; `body_bytes` and `body_sha256` describe persisted redacted
-UTF-8 body content. A blob digest describes the JSON-encoded storage object. Decode its JSON string
-wrapper to obtain the downloadable body.
-
-Every applied transformation has `kind`, `location`, `rule`, `count` and `reason` in the event's
-`transformations` array. This includes redaction, excluded headers, URL scrubbing, decompression,
-encoding and omissions. Removed secret values never appear in this ledger. Authentication is
-forwarded to the selected origin but excluded from persisted headers. Nonsecret repeated headers
-retain multiple values.
-
-The default body limit is 64 MiB and the concurrent buffering budget is 256 MiB. CLI callers can
-set `--max-body-bytes` and `--max-buffered-bytes`. Bodies are redacted as whole values, never as
-independent network chunks. No plaintext body is written to a temporary file. Overflow, decoding,
-redaction and persistence failures make capture incomplete. Blobs are written under temporary
-names only after redaction, then atomically renamed to their content hashes.
-
-`manifest.capture.status` is `complete`, `incomplete` or `unverified`, with machine-readable `issues`.
-It describes recorded evidence at the stated boundary. It is independent of upstream HTTP status
-and `manifest.exit_code`, which remains the agent's exit status. A fully recorded HTTP 500 is
-complete transport evidence; an interrupted HTTP 200 is not. A recording command returns nonzero
-when capture is incomplete, even if the agent exited successfully. No observed requests is an
-incomplete recording, not proof of successful interception.
-
-Replay admits only complete request/response transport pairs. Logical model events alone are
-insufficient and must never cause an invented successful response. The `canonical_exact` score
-measures equality after protocol normalization, not equality of transport bytes.
-
-One JSON object per line, UTF-8, `\n`-terminated. Readers MUST tolerate a truncated final line
-(a crash during write) and MUST skip unknown `type` values rather than failing. Writers MUST NOT
-rewrite or reorder previously written lines.
-
-### 2.1 Envelope
-
-| Field      | Type              | Required | Meaning |
-|------------|-------------------|----------|---------|
-| `seq`      | integer ≥ 0       | yes      | Dense total order. Strictly increasing, starts at 0. |
-| `ts`       | RFC3339 string    | yes      | Wall clock. Informational only — wall clocks lie. |
-| `mono_us`  | integer ≥ 0       | yes      | Microseconds since run start. Authoritative for duration. |
-| `turn`     | integer ≥ 0       | yes      | Groups events into model turns. |
-| `type`     | string            | yes      | See §2.3. |
-| `actor`    | string            | yes      | `agent`, `harness`, `model`, `agenticreplay`, `gateway`, `tool`, `user`. |
-| `causes`   | integer[]         | no       | Causal DAG over the total order. Each entry MUST be < `seq`. |
-| `attrs`    | object            | no       | Type-specific scalar data. Kept small and greppable. |
-| `payload`  | value \| BlobRef  | no       | Large or opaque data. See §2.2. |
-| `redacted` | string[]          | no       | Identifiers of removed material. Never the material itself. |
-
-### 2.2 Payloads and blobs
-
-A payload is either an inline JSON value or a blob reference:
-
-```json
-{ "$blob": "sha256:9f2c14…", "bytes": 20481, "media_type": "text/plain" }
-```
-
-Any object carrying a `$blob` key MUST be a well-formed blob reference; a malformed one is not
-merely an unrecognised payload, it is an invalid event. This keeps a corrupt digest from being
-silently accepted as opaque inline data.
-
-Writers MUST spill any payload whose serialized form exceeds **4096 bytes** to a blob, so that
-`events.jsonl` stays readable with `jq` and `rg`. Blobs are stored at
-`blobs/<first two hex chars>/<full hex digest>` and are immutable.
-
-Because every model turn resends the whole conversation, content addressing is what keeps a trace
-`O(n)` in new content rather than `O(n²)` in turns. Implementations MUST NOT store duplicate blobs.
-
-### 2.3 Event types
-
-| Type | Emitted when |
-|------|--------------|
-| `run.start` / `run.end` | Bracket the run. Exactly one of each. |
-| `model.request` / `model.response` | Logical projections for model navigation. Transport evidence is recorded separately. |
-| `tool.call` / `tool.result` | A tool invocation, usually reconstructed from the protocol. |
-| `mcp.request` / `mcp.response` | An MCP JSON-RPC exchange seen by the shim. |
-| `shell.exec` / `shell.result` | A shell command, from the PATH shim. |
-| `fs.snapshot` / `fs.change` | A workspace tree id, and a diff against the previous tree. |
-| `net.request` / `net.response` | Physical HTTP attempts, including model calls, retries and failures. A pair carrying `rule` and `replay_key` is a retrieval call. Digest-only responses cannot be replayed. |
-| `retrieval.context` | What a retriever put into a prompt, **derived** from the `model.request` that carried it — query and the retrieved passages. Same standing as `tool.call`: reconstructed from the wire, never separately captured. Scores and `top-k` are not here because they are not on the wire. |
-| `agent.start` | An agent began a turn: its name, the tools and handoffs it was given. From a harness's own tracing, not from the wire. |
-| `agent.handoff` | One agent handed control to another, naming both. A proxy sees the transfer as an ordinary tool call and cannot say which agent it came *from*. |
-| `agent.guardrail` | A guardrail ran, and whether it tripped. Guardrails need not make any request, so this can have no trace on the wire at all. |
-| `graph.node.start` / `graph.node.end` | A node of an agent graph ran: its name, the superstep it belonged to, the run ids that pair the two and place it under its parent, and on the end the class of the exception it raised, if any. From the framework's own callbacks. A node's name appears in no request, a node that calls no model makes no request, and two nodes of one parallel superstep are indistinguishable on the wire from two consecutive turns. |
-| `session.snapshot` | The harness's own transcript, captured at the point the run ended. Carries what the run was *asked*, which a hand-driven run leaves nowhere on the wire. |
-| `error` | A failure derived from another event or reported by the harness. |
-| `divergence` | Replay matched inexactly. See §4. |
-| `checkpoint` | A forkable point. Derived, not recorded live. See §3. |
-| `fork` | A run forked from this point into a child run. |
-| `route.decision` | A gateway chose a model. **Generic** — any gateway may emit it. |
-| `note` | Derived annotation from an analyzer (e.g. loop detection). |
-
-The `agent.*` and `graph.*` types are the ones that cannot come from the proxy at all. Every other
-type above is something agenticreplay observed itself; these are reported by the harness through its own
-tracing or callback interface, and a trace that has none of them is not missing anything — it is a
-run whose harness either has no such interface or was not asked to use it. Readers must treat them
-as optional, like any other type they do not find.
-
-They also differ from the rest in who wrote them. A `model.request` was assembled by agenticreplay out of
-bytes it saw; a `graph.node.start` was handed to agenticreplay by code running inside the agent's own
-interpreter. An implementation that writes these MUST keep them to structure the harness names for
-itself — a node's declared name, an agent's name, whether a guardrail tripped — and MUST NOT carry
-prompts, model output, tool arguments, tool results or exception messages in them. Those either
-already exist elsewhere in the trace, byte for byte, or are values the harness never intended to
-export.
-
-Adding a type is a MINOR version bump. Removing or changing the meaning of one is MAJOR.
-
-## 3. Checkpoints
-
-A checkpoint is **derived**, not recorded. `seq` *n* is a checkpoint when both hold:
-
-1. a `fs.snapshot` exists at or before *n* within the same `turn`; and
-2. the conversation prefix up to *n* is complete — every `model.request` before *n* has a
-   matching `model.response`.
-
-A fork target that is not a checkpoint MUST snap to the nearest preceding checkpoint, and the
-implementation MUST report that it did so rather than silently forking from different state.
-
-## 4. Divergence
-
-Replay of a nondeterministic client will not always see byte-identical requests. Implementations
-MUST match using this ladder and MUST record which rung matched:
-
-| Rung | Strategy | Result |
-|------|----------|--------|
-| 1 | Canonical hash of the normalized request | exact — no event |
-| 2 | Same turn index and message count, structural distance under threshold | `divergence` level `minor` |
-| 3 | Identical trailing message, different prefix (typical after compaction) | `divergence` level `major` |
-| 4 | No match | halt and report; `--loose` continues live instead |
-
-Redaction (§5) makes rung 1 unreachable for any request that contained a secret: the placeholder's
-digest is salted per run, so the same value becomes a different placeholder on replay. Implementations
-MUST therefore put both sides in the same representation before comparing — redacting the incoming
-request with the same policy — and MUST compare the *kind* of secret rather than its digest, which
-is the most a trace can know about a value it deliberately destroyed. A request that is equal only
-after that fold is a rung 2 `minor` divergence, never rung 1.
-
-A rendered image makes rung 1 unreachable for the same reason. Replay does not intercept the world,
-so an agent that sees through a browser or a camera renders its next frame afresh, and two renders
-of one scene are not the same bytes. Implementations MUST therefore compare image content parts
-below rung 1 on their presence and media type rather than on their payload, and MUST keep the
-payload in the comparison at rung 1, so an exact match still means exact. The text accompanying the
-image MUST remain fully compared: it is what identifies the request — a browser agent sends its
-element tree, the page URL and its own memory in the same message — and folding the payload is only
-sound because that text is untouched. A request that is equal only after this fold is a rung 2
-`minor` divergence, never rung 1, and the divergence MUST say how many images it covered.
-
-Distance MUST be measured per field rather than over the serialized body as a whole. A whole-body
-longest-common-prefix-and-suffix measure counts everything between two distant edits as changed, so
-two drifting identifiers in a large prompt score as a total rewrite and no request can reach rung 2.
-
-**Replay MUST NOT silently approximate.** Every inexact match is an event in the trace.
-
-## 5. Redaction
-
-Redaction happens in the write path, before bytes reach disk. Implementations MUST:
-
-- capture environment variables by **allowlist only**;
-- never write auth material (`authorization`, `x-api-key`, `cookie`, `proxy-authorization`),
-  substituting a stable placeholder `<secret:kind:hash8>` derived from a per-run salt, so that
-  replay still matches structurally and repeat occurrences remain equal;
-- record removals in `redactions.json` by rule and identifier, never by value.
-
-High-entropy detection MUST additionally require a **mixed alphabet** — the token must contain
-both a digit and a letter. Shannon entropy alone has false positives on ordinary source code
-(`getUserAuthenticationTokenFromRequestHeaders` scores 4.08 bits/char), and since agent traces are
-mostly source code, an unguarded rule corrupts exactly the payloads the trace exists to preserve.
-Recall is essentially unaffected: random base64url of 20+ characters contains a digit with
-probability ≈0.98.
-
-High-entropy detection MAY exempt a payload it can prove is a whole image, because a base64
-screenshot is a long random-looking run by construction and shredding one protects nothing while
-destroying the payload the trace exists to preserve. An implementation that does so MUST grant the
-exemption on the decoded content and MUST be able to account for every byte it covers: structural
-and pixel data whose size the header declares and whose stream is verified to produce it, plus
-chunks whose length the format fixes below the detector's own threshold. A container carrying
-caller-chosen bytes of unbounded length — PNG `tEXt`/`iTXt`/`zTXt`/`iCCP`/`PLTE`, JPEG `COM`/`APPn`,
-or a format with no way to validate its payload at all — MUST NOT be exempted, because those bytes
-reach the trace uncompressed and a credential is indistinguishable from any other content in them.
-
-The exemption's stated limitation: a secret written into the pixels themselves is not detectable,
-and no content rule could be. Implementations MUST bump `policy_version` when the exemption's
-extent changes, so a reader can tell a trace recorded under one policy from another.
-
-Redaction is best-effort mitigation, not a guarantee. A trace is sensitive material.
-
-## 6. Integrity
-
-`manifest.json.integrity.events_sha256` is the SHA-256 of `events.jsonl` at the moment the run
-ended. Readers SHOULD verify it and MUST report, not repair, a mismatch.

package/test/integrations/README.md

@@ -1,133 +0,0 @@
-# Integration checks
-
-`agenticreplay record generic-openai -- python your_agent.py` is documented as working for LangGraph,
-CrewAI, the OpenAI Agents SDK and anything else that reads a base-URL variable. Until this
-directory existed, that was a sentence.
-
-Each check here records a real framework against a stub origin, kills the origin, replays the
-recording offline, and asserts the numbers. It is the difference between "we support X" and a line
-in CI that goes red.
-
-## Why end to end rather than the adapter contract
-
-`checkAdapterContract` guards against adapter rot — a harness renames the variable it reads and the
-adapter keeps setting the old one. It cannot see whether anything was actually captured.
-
-`agenticreplay record opencode` once produced a trace with two events and no model exchange, twice over, for
-two independent reasons. The contract was green for both. What catches that shape is running the
-thing and counting what came back.
-
-## Why a stub origin
-
-Three properties, and every one of them is load-bearing:
-
-- **Free.** A matrix that costs tokens per run is a matrix somebody eventually turns off.
-- **Deterministic.** Replay is judged on matching the recorded bytes. An origin that varied its
-  answer would make an exact match indistinguishable from a lucky one.
-- **Offline.** The second half of every check kills the origin before replaying. If a replay reaches
-  the network the check fails by construction rather than by assertion.
-
-The stub answers the shapes real frameworks use, not the simplest one: streaming SSE, tool calls,
-and the Anthropic messages endpoint. Those are where a recording proxy is most likely to break, and
-a matrix that only exercised a plain completion would have said nothing about either.
-
-## What each check proves
-
-| check | the mechanism it covers | what else rides on it |
-|---|---|---|
-| `openai-sdk` | the official Python SDK reads `OPENAI_BASE_URL` | every framework that lets the SDK build its own client |
-| `openai-async` | the same for `AsyncOpenAI` | anything building its own async client |
-| `anthropic-sdk` | `ANTHROPIC_BASE_URL` | Claude-backed agents that are not Claude Code |
-| `litellm` | `OPENAI_API_BASE` through LiteLLM | **Aider** — and anything else riding that transport |
-| `openhands-sdk` | the OpenHands SDK's own `LLM`, which wraps LiteLLM | OpenHands |
-| `crewai` | CrewAI's own `Agent`, `Task` and `Crew` | CrewAI, which since 1.x does **not** use LiteLLM |
-| `openai-agents` | the Agents SDK on the **Responses API**, its default | the Agents SDK, and agenticreplay's `responses` dialect |
-| `openai-agents-handoff` | two agents, a handoff and a guardrail — reported by the SDK, not seen on the wire | the sixth capture layer |
-| `langgraph-stream` | LangChain's own `OPENAI_API_BASE`, over SSE | LangGraph, LangChain |
-| `langgraph-tools` | the same, with a bound tool | tool-calling graphs |
-| `langgraph-nodes` | which node ran, in which superstep — including one that calls no model | the sixth capture layer, for graphs |
-| `llama-index` | LlamaIndex's own OpenAI LLM, which reads the older base-URL variable and not the new one | LlamaIndex |
-| `haystack` | a Haystack `Pipeline` whose generator takes no `api_base_url` | Haystack 3.x, and any component that lets the OpenAI SDK build its own client |
-| `haystack-rag` | the same, embedding at query time: document embeddings, a query embedding, retrieval, then an answer | RAG pipelines whose embeddings must replay too |
-| `browser-use` | its own `ChatOpenAI` passes an unset `base_url` through | browser-use, and the pattern any wrapper using the official SDK follows |
-| `vision-repaint` | a screenshot recorded intact, and replayed against a different one | agents with eyes, and what a match means when the pixels move |
-| `ai-sdk-openai` | a real `@ai-sdk/openai` client reading `OPENAI_BASE_URL`, no fetch preload | the Vercel AI SDK's env-aware path, which the bare-`fetch` fixtures cannot pin |
-| `mastra` | a model provider that takes its origin in code rather than from the environment | Mastra, and any JS agent that never reads a variable |
-| `mcp-stdio` | an MCP server launched from a config, recorded and then taken away | the MCP shim, and replay with the server gone |
-| `fetch-hook` | `NODE_OPTIONS` preload on `globalThis.fetch` | the Vercel AI SDK, and any JS agent with its origin compiled in |
-| `rag-index` | a concurrent index build, an embedding batch, and an answer over retrieved context | IndexRAG, LlamaIndex, GraphRAG, LightRAG — every pipeline that indexes before it answers |
-| `rag-split-origin` | the same run with embeddings at a **second origin of the same wire dialect** | any stack whose chat and embeddings do not share a provider |
-
-Twenty-two checks. Covering the layer underneath still covers what stands on it — that is what the
-`litellm` row is for — but three of these exist because that stopped being enough, and each of the
-three was added after running the framework itself said something the layer could not.
-
-**CrewAI is the clearest case.** It sat under the `litellm` row for exactly the reason that row
-gives, and in CrewAI 1.x it is no longer true: LiteLLM became an optional extra and CrewAI grew
-native providers, so a default install never loads it. Capture survived — the native provider reads
-both variables `generic-openai` sets — but the stated reason had been wrong for a whole major
-version, and nothing here could have noticed. Running CrewAI also turned up two changes worth
-documenting: a bare model name always reaches the native provider while a prefixed one can need
-LiteLLM installed, and `LLM(base_url=…)`, long documented as silently ignored, is now honoured.
-
-**The two `rag-*` checks cover a shape none of the others could.** An indexing pipeline sends the
-same *message* many times over — the prompt template is fixed and the document that varies rides
-in the **system** prompt — so the matching ladder, built for a conversation where the trailing
-message is the question, served each document whichever answer the cursor happened to be on. Every
-check above would have stayed green through that: the counts were right, `unmatched=0`, `exit=0`,
-and only the contents were wrong. `rag-index` asserts each request gets its own answer and that
-the embedding calls replay at all, which before `RetrievalRule` was a 502 on the first one.
-`rag-split-origin` asserts *where* each half went, and that assertion is the whole reason it is
-not a copy of `rag-index` — without it, "two origins" and "one origin" look identical from the
-numbers, which is exactly how embedding traffic came to be forwarded to the chat origin.
-
-**`openai-agents` is not a duplicate of `openai-async` either.** The Agents SDK defaults to the
-Responses API; `AsyncOpenAI` in that check uses chat completions. So the older check covered the
-client underneath on a wire format the SDK does not use, and the format the README claimed support
-for had no end-to-end check at all until this one.
-
-## What the checks are worth, measured
-
-A green matrix proves nothing on its own. Each redirect was removed in turn to see which checks
-notice:
-
-| removed from `generic-openai` | what fails |
-|---|---|
-| `OPENAI_BASE_URL` | `openai-sdk`, `openai-async`, `openai-agents`, `browser-use` |
-| `ANTHROPIC_BASE_URL` | `anthropic-sdk` |
-| `OPENAI_API_BASE` | **nothing** |
-
-The last row is still the finding, and it survived the checks added since it was first measured.
-LangChain reads `OPENAI_API_BASE` itself but hands the rest to the official client, which reads
-`OPENAI_BASE_URL`; CrewAI's native provider reads **both**, so it survives losing either one on its
-own and appears in neither row. Both variables stay set, because pre-1.0 SDKs and several ports read
-only the second and this matrix does not cover those. It is worth knowing that today nothing here
-would notice if that line went away.
-
-Two notes on the re-measurement rather than the measurement:
-
-- `browser-use` is new to the first row. It was absent from the earlier table and it does depend on
-  `OPENAI_BASE_URL` — its `ChatOpenAI` passes an unset `base_url` to the official client, which is
-  precisely the dependency. Read as a correction to the table, not as a change in browser-use.
-- `litellm` is excluded from this run. A complete litellm install does not fit under the Windows
-  long-path limit on the machine this was re-measured on, so its result here would have been an
-  artefact of the environment rather than a fact about the redirect. It is unchanged from the
-  earlier measurement, where removing `OPENAI_API_BASE` did not break it either — because
-  `generic-openai` still sets the other two and LiteLLM is reached through the same client.
-
-## Running them
-
-```console
-node test/integrations/run.mjs                # all of them
-node test/integrations/run.mjs litellm        # one
-node test/integrations/run.mjs --require-all  # a skip is a failure — what CI runs
-```
-
-Python checks are skipped, not failed, when the package they need is not installed — a contributor
-without `langgraph` on their machine should still be able to run the suite.
-
-`--require-all` is what makes a skip a failure, and it exists because the sentence that used to be
-here said CI installing the packages was enough. It was not. Two checks were added without a line
-in the workflow's `pip install`, CI went green on the skips, and the README said "in CI" about both
-— a claim made false by an omission that nothing could fail on. Adding a check now means adding it
-to that line, or this run says so.

package/test/integrations/agents/ai_sdk_openai.mjs

@@ -1,24 +0,0 @@
-/**
- * A real `@ai-sdk/openai` client, configured only by the environment.
- *
- * This is the check issue #80 asked for. The existing `hardcoded_origin.mjs` and
- * `vercel-agent.mjs` fixtures are a bare `fetch` to a compiled-in origin — they prove the
- * preload, not the SDK. Set `OPENAI_BASE_URL`, install no fetch hook, and the request has
- * to land on the stub. A green result here cannot be "the preload caught it either way".
- *
- * `createOpenAI({ apiKey })` is the construction the docs recommend; an explicit `baseURL`
- * would win and make this check meaningless.
- */
-import { createOpenAI } from '@ai-sdk/openai';
-import { generateText } from 'ai';
-
-if (/\bagenticreplay-fetch-hook\b/.test(process.env.NODE_OPTIONS ?? '')) {
-  throw new Error('this check is the no-preload path; the fetch hook was injected');
-}
-
-const openai = createOpenAI({ apiKey: process.env.OPENAI_API_KEY ?? 'stub-key' });
-const { text } = await generateText({
-  model: openai('stub-1'),
-  prompt: 'hello',
-});
-console.log('GOT:', text);

package/test/integrations/agents/anthropic_sdk.py

@@ -1,10 +0,0 @@
-"""The Anthropic SDK with no base_url, so ANTHROPIC_BASE_URL has to reach it."""
-
-from anthropic import Anthropic
-
-client = Anthropic()
-print("base_url:", client.base_url)
-reply = client.messages.create(
-    model="claude-stub", max_tokens=64, messages=[{"role": "user", "content": "hello"}]
-)
-print("GOT:", reply.content[0].text)