@post-print/agent-test 0.3.3 → 0.3.4

package/README.md

@@ -1,210 +1,100 @@
 # @post-print/agent-test
 
-**Source of truth for** agent-test package.
+**Source of truth for** the direct-agent test package.
 
-<!-- doc-meta: owner=eng | last-reviewed=2026-07-29 -->
+<!-- doc-meta: owner=eng | last-reviewed=2026-09-02 -->
 
-Jest-shaped agent scenario runner built on `@post-print/agent-harness`.
+Scenario runner for real Cursor and Claude agents, built on `@post-print/agent-harness`.
 
-## In-repo smoke (this monorepo)
+> **Deprecated and removed:** stored-trace replay is not an agent test. `host: "replay"`, `replayTrace`, `--record-fixtures`, and the old `--live` mode flag are rejected with migration guidance. Every scenario execution now launches Cursor or Claude and can incur provider usage.
 
-After `bun run build` at the repo root:
+## Direct TypeScript API
 
-```bash
-node packages/test/dist/cli.js --suites-dir packages/test/fixtures --suite smoke
-node packages/test/dist/cli.js --doctor
+`runAgentTest` is the execution core. JSON suites load into this same function.
+
+```ts
+import { runAgentTest } from "@post-print/agent-test";
+
+const result = await runAgentTest({
+  cwd: process.cwd(),
+  scenario: {
+    name: "uses the project instructions",
+    prompt: "Review the current change.",
+    rubric: { mustReadPath: ["AGENTS.md"] },
+  },
+});
 ```
 
-There is no top-level `agent-suites/` here — consumer examples below assume a consuming repo. Live `--live` needs an **exported** `CURSOR_API_KEY` for Cursor (default). `--host claude` needs an explicit `CLAUDE_AUTH_MODE=api-key` (with `ANTHROPIC_API_KEY`) or `CLAUDE_AUTH_MODE=subscription` (the Claude Code CLI login); nothing is inferred (copy repo-root `.env.example`; the CLI does not auto-load `.env`). Judge classifiers still need `CURSOR_API_KEY` unless you pass `--no-judge`.
+The default host is Cursor. Set `scenario.host: "claude"`, `defaults.host`, or the top-level `host` option for Claude. The scenario host wins over the top-level override, which wins over suite defaults. The default suite name is `direct`; worktree isolation, judging, timeout enforcement, and announce-stop retry are enabled by default.
+
+## JSON suites and CLI
 
-## CLI (consumers, Node >= 22)
+JSON is an authoring adapter, not a stored answer. Consumer repositories can keep suites under `agent-suites/<suite>/scenarios.json`:
+
+```json
+{
+  "name": "routing",
+  "defaults": { "host": "cursor", "profile": "cursor" },
+  "scenarios": [
+    {
+      "name": "reads instructions",
+      "prompt": "Review the current change.",
+      "rubric": { "mustReadPath": ["AGENTS.md"] }
+    }
+  ]
+}
+```
 
-Works under **Node >= 22** (the published `agent-test` bin):
+Run Cursor by default or select Claude explicitly:
 
 ```bash
 npx agent-test --suites-dir agent-suites
-npx agent-test --suites-dir agent-suites --suite ambient-routing
-npx agent-test --suites-dir agent-suites --live --suite ambient-routing   # exported CURSOR_API_KEY required
-npx agent-test --suites-dir agent-suites --live --host claude --suite ambient-routing  # CLAUDE_AUTH_MODE + claude CLI
-npx agent-test --live --compare-pairs skeleton-clean:skeleton-messy --out-dir "$TMPDIR/compare"
-npx agent-test compare --a clean.suite-report.json --b messy.suite-report.json --out-dir "$TMPDIR/compare"
-npx agent-test --suites-dir agent-suites --report-out "$TMPDIR/agent-test-report"       # dir: html + suite JSON
-npx agent-test --suites-dir agent-suites --report-out "$TMPDIR/reports/run.html"        # file: html only
+npx agent-test --suites-dir agent-suites --suite routing
+npx agent-test --suites-dir agent-suites --host claude
 npx agent-test --doctor
 ```
 
-### Report output
+Direct runs require an exported `CURSOR_API_KEY` for Cursor or `ANTHROPIC_API_KEY` plus the Claude Code CLI for Claude. Judge criteria use `CURSOR_API_KEY` unless `--no-judge` is set. The CLI does not load `.env`.
 
-By default the HTML report goes to a fresh temp directory. `--report-out <path>` puts it where you want:
+The old `--live` flag is removed because direct execution is now the only execution mode. Direct runs capture transient traces automatically; use `--keep-recordings` or `--debug` to retain diagnostics.
 
-| `--report-out` value | Result |
-| --- | --- |
-| ends with `.html` | the HTML report is written to exactly that file (parent dirs created) |
-| any other path | treated as a directory: `report.html` plus all other report content — per-suite `<suite>.suite-report.json`, and the compare JSON / markdown / HTML when `--compare-pairs` is used |
+## Validation, rubrics, and comparison
 
-`--out-dir` still wins for compare output when both are given. `--no-html-report` skips the HTML report entirely.
-
-Bun is fine for local package development (`bun install` / `bun run build` in this monorepo), but consumers do not need Bun to run suites.
-
-Default suites root: `agent-suites/` (must exist, or pass `--suites-dir`). Absolute `--suites-dir` is supported. Optional `--rubrics-dir` loads harness-only answer keys from `<rubricsDir>/<suite>/rubrics.json` (preferred over a sibling file when present).
-
-### Harness-only rubrics
-
-`scenarios.json` may omit `rubric` (or use `{}`) and keep only prompts / `seedPatch` / `compareId`. Put `must` / `mustNot` / `judge` / tool matchers in:
-
-- sibling `rubrics.json` or `scenarios.rubric.json`: `{ "scenarios": { "<scenario name>": { "must": […] } } }`
-- or `--rubrics-dir <path>` → `<path>/<suiteName>/rubrics.json`
-
-External entries **replace** the inline rubric for that scenario name. Unknown names in the rubrics file are an error. Keep rubrics off agent-visible roots (absolute `--suites-dir` / `--rubrics-dir` under `$TMPDIR`, or outside the IDE-open workspace).
-
-Live output always uses ANSI color (including under Cursor agent shells that set `NO_COLOR`).
-
-## Environment variables
-
-See repo-root `.env.example`. Common knobs:
-
-| Variable                                    | Purpose                                                       |
-| ------------------------------------------- | ------------------------------------------------------------- |
-| `CURSOR_API_KEY`                            | Required for `--live` Cursor and judge classifiers            |
-| `CLAUDE_AUTH_MODE`                          | Required for `--live --host claude`: `api-key` or `subscription` (no default) |
-| `ANTHROPIC_API_KEY`                         | Required when `CLAUDE_AUTH_MODE=api-key`                      |
-| `CLAUDE_CODE_BIN`                           | Optional path to Claude Code CLI binary                       |
-| `CLAUDE_AGENT_MODEL`                        | Optional Claude model override                                |
-| `CLAUDE_CODE_ALLOWED_TOOLS`                 | Optional `--allowedTools` list for Claude live runs           |
-| `AGENT_TEST_DEBUG`                          | Same as `--debug` when `1`/`true`                             |
-| `AGENT_TEST_VERBOSE`                        | Extra tips (e.g. OOM isolation) when `1`                      |
-| `AGENT_TEST_VERBOSE_PATHS`                  | Print full paths when `1`                                     |
-| `AGENT_TEST_QUIET`                          | Suppress progress when `1`                                    |
-| `AGENT_TEST_TIMEOUT_MS`                     | Live hard timeout (default 600000; `0` disables)              |
-| `AGENT_TEST_LIVE_RETRIES`                   | Judge infra retry attempts (default 3)                        |
-| `AGENT_TEST_SCENARIO_RETRIES`               | Live announce-stop scenario retries (default 1; `0` disables) |
-| `AGENT_TEST_ALLOW_IN_PLACE`                 | Allow `--no-worktree` live runs when `1`                      |
-| `AGENT_TEST_NO_WORKTREE`                    | Disable worktree isolation when `1`/`true`                    |
-| `AGENT_TEST_NO_ISOLATE`                     | Disable isolated subprocesses when `1`                        |
-| `AGENT_TEST_SCENARIO_SETTLE_MS`             | Settle delay between live scenarios                           |
-| `CURSOR_AGENT_MODEL` / `CURSOR_JUDGE_MODEL` | Optional model overrides                                      |
-| `CURSOR_JUDGE_TEMPERATURE`                  | Optional judge temperature                                    |
-
-## Debug mode
+These commands inspect configuration or existing reports; they do not claim to execute an agent:
 
 ```bash
-npx agent-test --suites-dir agent-suites --suite smoke --debug
-npx agent-test --suites-dir agent-suites --live --debug
-npx agent-test --suites-dir agent-suites --live --debug --debug-dir "$TMPDIR/agent-test-debug"
+npx agent-test --validate-only --validate-paths --suites-dir agent-suites
+npx agent-test --validate-seeds --suites-dir agent-suites
+npx agent-test --compare-pairs clean:changed --out-dir "$TMPDIR/compare"
+npx agent-test compare --a clean.suite-report.json --b changed.suite-report.json --out-dir "$TMPDIR/compare"
 ```
 
-`--debug` (or `AGENT_TEST_DEBUG=1`) implies `--keep-recordings`, verbose failure detail, and full paths. Every non-skipped scenario writes a bundle under the session root:
-
-```
-sessions/<id>/<suite>/<scenario>.debug/
-  summary.md             # verdict + Why (category hint + evidence + trace stats)
-  transcript.md          # Why, prompt, rubric, interleaved messages/tools (incl. results), failures
-  scenario.json          # prompt + rubric + seed metadata
-  result.json            # pass/fail, duration, failures, usage, skillsInvoked, routing, counts
-  trace.json
-  failures.json          # includes category + evidence
-  judge-debug.json       # when judge criteria ran (SDK status/error, sizes, attempt)
-  environment.json       # versions/models/timeout/isolation; API keys only as booleans
-  rerun.sh               # shell-quoted exact re-run command (export API keys yourself)
-```
+`scenarios.json` may omit inline rubric keys when they are supplied by sibling `rubrics.json` / `scenarios.rubric.json`, or by `--rubrics-dir <path>` at `<path>/<suite>/rubrics.json`. External rubric entries replace the inline rubric for the same scenario.
 
-**Debug dir default:** omit `--debug-dir` to stage under `$TMPDIR/agent-spec/sessions/<id>/…` (outside the repo). Passing an in-repo `--debug-dir` (for example `./agent-test-debug`) is supported — harness staging paths under that dir are excluded from worktree leak checks — but prefer `$TMPDIR` so debug artifacts never appear in `git status`.
+Reports pair scenarios by `compareId` when present, otherwise by normalized scenario name. JSON, Markdown, and HTML reports include outcomes, token usage, duration, tools, and grounding signals.
 
-`--debug-dir <path>` replaces `$TMPDIR/agent-spec` as the sessions parent (`<path>/sessions/<id>/…`).
+## Isolation and diagnostics
 
-Failure categories printed on FAIL lines and in `failures.json`:
+Each scenario uses a detached git worktree by default. This isolates edits, but it does not prevent a local host from reading the IDE-open caller checkout. Keep answer keys outside agent-visible roots when that distinction matters.
 
-| Category          | Meaning                                                                                |
-| ----------------- | -------------------------------------------------------------------------------------- |
-| `rubric_miss`     | Assertion/judge criterion miss                                                         |
-| `judge_infra`     | Judge SDK/API failure (not a criterion miss)                                           |
-| `agent_runtime`   | Agent session error, timeout, AskQuestion, subprocess exit                             |
-| `worktree_leak`   | Live agent mutated the caller working tree (harness `--debug-dir` staging is excluded) |
-| `recording_error` | Failed to persist a staging/fixture trace                                              |
+`--debug` retains an evidence bundle under `$TMPDIR/agent-spec/sessions/<id>/` by default. It includes the transcript, scenario, result, trace, failures, environment metadata, judge details, and an exact direct-run rerun command. Use `--debug-dir` to override the parent directory.
 
-**Cancel:** `Ctrl+C` (SIGINT) kills in-flight isolated scenario subprocesses and best-effort cancels the active Cursor SDK run, then cleans scenario worktrees.
+`Ctrl+C` cancels active Cursor/Claude work and cleans worktrees. `--no-worktree` requires `AGENT_TEST_ALLOW_IN_PLACE=1` because agent edits will persist in the caller checkout.
 
-## Live dogfood
-
-Live runs need `CURSOR_API_KEY` and a suites directory that exists. Preflight fails when the resolved suites directory is missing (default `agent-suites/` if `--suites-dir` is omitted). `--suites-dir` may be relative to the repo cwd or an absolute path (for example a scrubbed suite tree under `$TMPDIR`).
-
-Passing live runs write staging traces under `$TMPDIR/agent-spec/sessions/<pid>-<timestamp>/` (removed on exit unless `--keep-recordings`). Use `--record-fixtures` to overwrite each scenario's committed `replayTrace` path. `--no-worktree` requires `AGENT_TEST_ALLOW_IN_PLACE=1`.
-
-### Isolation model
-
-Live runs use a **detached git worktree** for agent file edits. That is not full filesystem isolation:
-
-- **Worktree does:** keep seed/apply edits and agent Write/Edit tools off the caller's working tree (leak checks catch escapes).
-- **Worktree does not:** stop context (skills/rules) from loading from the caller checkout by design, or stop Cursor **local** agents from Shell/Read against the IDE-open workspace instead of only `local.cwd`.
-
-Therefore: do **not** put answer keys, golden replays, or judge-bearing scenario text where a null-arm agent can forage them on the caller/IDE root. Prefer:
-
-- opaque prompts + `compareId`
-- seeds that only mutate fixtures (not skill bodies)
-- harness-only rubrics (sibling `rubrics.json` / `scenarios.rubric.json`, or `--rubrics-dir` outside the open workspace)
-- consumer orchestrators that park answer keys off the open workspace (toolbox pattern)
-- cloud runtime when true FS isolation is required
-
-Live agent runs have a **hard timeout** (default **10 minutes**, override with `--timeout-ms` or `AGENT_TEST_TIMEOUT_MS`; disable with `--no-timeout` or `AGENT_TEST_TIMEOUT_MS=0`). If the agent invokes `AskQuestion` or similar user-input tools, the harness fails fast with a clear error — live mode is single-shot and cannot supply follow-up turns. Use `--allow-user-input` only for intentional multi-turn dogfood (the run may still hang waiting for stdin).
-
-Announce-stop flakes (agent exits after Routing with no tools) are retried once by default on live runs (`AGENT_TEST_SCENARIO_RETRIES=1` or `--scenario-retries 1`; set `0` to disable). This is separate from `AGENT_TEST_LIVE_RETRIES` (judge infra only).
-
-### Dialogue skills in live runs
-
-Skills that expect multi-turn Socratic dialogue (for example `crystallize`) will hang or fail in `--live` unless the scenario is written for one-shot completion:
-
-- **Replay-only** — commit a golden trace where the agent finishes without asking questions; use `host: "replay"` or `skip: true` with live skipped at suite level.
-- **One-shot live prompt** — instruct the agent to mirror intent and emit the final artifact in a single turn (`no AskQuestion; produce Crystallized idea now`).
-- **Ambient routing** — fuzzy-intent scenarios that only require mirroring + one assumption are naturally one-shot; full crystallize dialogue is not.
-
-Do not weaken dialogue-first product skills for CI; reshape the suite contract instead.
+Dialogue-first skills must be tested with a one-shot prompt or an intentional `--allow-user-input` run. Do not weaken the production skill to make a headless test pass.
 
 ## MCP servers
 
-Live Cursor runs can attach **inline** MCP servers from suite/scenario JSON. Ambient project/user MCP (`.cursor/mcp.json`) is not loaded — tests stay hermetic.
-
-```json
-{
-  "defaults": {
-    "mcpServers": {
-      "docs": {
-        "type": "http",
-        "url": "https://example.com/mcp",
-        "headers": { "Authorization": "Bearer ${DOCS_TOKEN}" }
-      }
-    }
-  },
-  "scenarios": [
-    {
-      "name": "use echo",
-      "prompt": "Call the echo tool with text hello.",
-      "mcpServers": {
-        "echo": {
-          "type": "stdio",
-          "command": "node",
-          "args": ["packages/test/fixtures/mcp-echo/server.mjs"]
-        }
-      },
-      "rubric": {
-        "mustCallTool": ["echo:hello"],
-        "mustNotCallTool": ["shell"]
-      }
-    }
-  ]
-}
-```
+Cursor and Claude scenarios can attach inline stdio or HTTP/SSE MCP servers through suite defaults or scenario overrides. Scenario server names replace matching defaults. `${ENV_VAR}` placeholders resolve at run time. Ambient project/user MCP configuration is not loaded.
 
-- Suite `defaults.mcpServers` merge with scenario `mcpServers` by server name (scenario wins).
-- `${ENV_VAR}` placeholders expand in `command`, `args`, `env`, `url`, `headers`, and OAuth fields at run time.
-- Rubric `mustCallTool` / `mustNotCallTool` match tool **name** substrings (works with MCP name prefixes). Use `name:argFragment` to also require a substring in JSON args.
-- Rubric `mustReadPath` / `mustNotReadPath` match substrings on **Read** tool JSON args (registry-first / avoid inventing paths). Keep hallucination scoring in live `judge` questions — no heavy factuality engine in v1.
-- Suite defaults may set `profile: "skeleton"` and/or `contextSources` (additive paths / `.skeleton/customize/` basenames). Shared/cursor/claude defaults stay backwards-compatible.
-- Live runs surface provider `usage` on traces/results; suite summary + HTML report include token sum / p50 / p95 when present.
-- `--compare-pairs A:B` (or `agent-test compare --a/--b`) pairs scenarios by **`compareId`** when present, else band-neutral scenario name (`outcome:` / `transfer:` stripped), and writes `compare-report.json` / `.md` / `.html` with pass/fail, tokens, toolCallCount, durationMs, and skill/registry hop proxies. Suite HTML also embeds an A/B table when two reports are present.
-- Replay hosts ignore `mcpServers` but still score recorded `toolCalls` against those matchers.
+## In-repo package checks
 
-## Library
+The repository does not run paid agents in its default CI package-integrity checks:
 
-```ts
-import { runAllSuites, expectTrace } from "@post-print/agent-test";
+```bash
+bun run build
+node packages/test/dist/cli.js --validate-only --suites-dir packages/test/fixtures --suite smoke
+node packages/test/dist/cli.js --doctor
 ```
+
+Credentialed acceptance requires one TypeScript `runAgentTest` call and one JSON-suite CLI run against a real host.

package/dist/cli.d.ts

@@ -10,9 +10,6 @@ export interface ParsedCliArgs {
     filter?: string;
     scenarioFilter?: string;
     stagingSessionId?: string;
-    record: boolean;
-    recordFixtures: boolean;
-    live: boolean;
     judge?: boolean;
     worktree?: boolean;
     keepRecordings: boolean;
@@ -35,16 +32,13 @@ export interface ParsedCliArgs {
     compareMode: boolean;
     compareA?: string;
     compareB?: string;
-    /** Live/replay A:B suite dirs or report JSON paths. */
+    /** Direct-run A:B suite dirs or report JSON paths. */
     comparePairs?: string;
     compareOutDir?: string;
 }
 /** Parse agent-test CLI argv (exported for unit tests). */
 export declare function parseCliArgs(argv: string[]): ParsedCliArgs;
-/**
- * Split `--report-out` into an HTML file path and, when a directory was given,
- * the directory every other report artifact is written to.
- */
+/** Resolve an explicit report target into the HTML path and optional artifact directory. */
 export declare function resolveReportOutput(reportOut?: string): {
     htmlPath?: string;
     outDir?: string;

package/dist/cli.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAKA,OAAO,EACN,KAAK,SAAS,EAKd,MAAM,2BAA2B,CAAC;AAuBnC,OAAO,EACN,KAAK,UAAU,EAIf,MAAM,oBAAoB,CAAC;AAU5B,MAAM,WAAW,aAAa;IAC7B,GAAG,EAAE,MAAM,CAAC;IACZ,SAAS,EAAE,MAAM,CAAC;IAClB,6EAA6E;IAC7E,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,IAAI,CAAC,EAAE,SAAS,CAAC;IACjB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,MAAM,EAAE,OAAO,CAAC;IAChB,cAAc,EAAE,OAAO,CAAC;IACxB,IAAI,EAAE,OAAO,CAAC;IACd,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,cAAc,EAAE,OAAO,CAAC;IACxB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,OAAO,CAAC;IACnB,cAAc,EAAE,OAAO,CAAC;IACxB,MAAM,EAAE,OAAO,CAAC;IAChB,UAAU,EAAE,OAAO,CAAC;IACpB,4FAA4F;IAC5F,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,KAAK,EAAE,OAAO,CAAC;IACf,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,OAAO,CAAC;IACtB,aAAa,EAAE,OAAO,CAAC;IACvB,aAAa,EAAE,OAAO,CAAC;IACvB,MAAM,EAAE,UAAU,CAAC;IACnB,0EAA0E;IAC1E,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,qEAAqE;IACrE,WAAW,EAAE,OAAO,CAAC;IACrB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,uDAAuD;IACvD,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,aAAa,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,2DAA2D;AAC3D,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,aAAa,CAiL1D;AAoED;;;GAGG;AACH,wBAAgB,mBAAmB,CAAC,SAAS,CAAC,EAAE,MAAM,GAAG;IAAE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAAC,MAAM,CAAC,EAAE,MAAM,CAAA;CAAE,CAQ9F"}
+{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAKA,OAAO,EACN,KAAK,SAAS,EAKd,MAAM,2BAA2B,CAAC;AAsBnC,OAAO,EACN,KAAK,UAAU,EAIf,MAAM,oBAAoB,CAAC;AAU5B,MAAM,WAAW,aAAa;IAC7B,GAAG,EAAE,MAAM,CAAC;IACZ,SAAS,EAAE,MAAM,CAAC;IAClB,6EAA6E;IAC7E,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,IAAI,CAAC,EAAE,SAAS,CAAC;IACjB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,cAAc,EAAE,OAAO,CAAC;IACxB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,OAAO,CAAC;IACnB,cAAc,EAAE,OAAO,CAAC;IACxB,MAAM,EAAE,OAAO,CAAC;IAChB,UAAU,EAAE,OAAO,CAAC;IACpB,4FAA4F;IAC5F,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,KAAK,EAAE,OAAO,CAAC;IACf,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,OAAO,CAAC;IACtB,aAAa,EAAE,OAAO,CAAC;IACvB,aAAa,EAAE,OAAO,CAAC;IACvB,MAAM,EAAE,UAAU,CAAC;IACnB,0EAA0E;IAC1E,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,qEAAqE;IACrE,WAAW,EAAE,OAAO,CAAC;IACrB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,sDAAsD;IACtD,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,aAAa,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,2DAA2D;AAC3D,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,aAAa,CAmL1D;AAED,4FAA4F;AAC5F,wBAAgB,mBAAmB,CAAC,SAAS,CAAC,EAAE,MAAM,GAAG;IAAE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAAC,MAAM,CAAC,EAAE,MAAM,CAAA;CAAE,CAQ9F"}

package/dist/cli.js

@@ -8,9 +8,9 @@ import { compareSuiteReports, labelForCompareSide, loadSuiteRunReport, parseComp
 import { discoverSuites } from "./discover-suites.js";
 import { runDoctor } from "./doctor.js";
 import { writeHtmlReport } from "./html-report.js";
-import { assertLiveDogfoodPreflight } from "./preflight.js";
+import { assertDirectAgentPreflight } from "./preflight.js";
 import { logProgress } from "./progress.js";
-import { cleanupLegacyRepoRecordings, cleanupStagingSession, createLiveStagingSessionId, getLiveStagingSessionRoot, setLiveStagingRootOverride, } from "./record-trace.js";
+import { cleanupStagingSession, createLiveStagingSessionId, getLiveStagingSessionRoot, setLiveStagingRootOverride, } from "./record-trace.js";
 import { registerLiveRunHandlers, runAllSuites, runSuite } from "./run-suite.js";
 import { formatRunSummary, shouldFailScenario, summarizeReports, } from "./suite-summary.js";
 import { configureCliColor, theme } from "./theme.js";
@@ -28,9 +28,6 @@ export function parseCliArgs(argv) {
     let filter;
     let scenarioFilter;
     let stagingSessionId;
-    let record = false;
-    let recordFixtures = false;
-    let live = false;
     let judge;
     let worktree;
     let keepRecordings = false;
@@ -59,7 +56,14 @@ export function parseCliArgs(argv) {
     for (let i = startIndex; i < argv.length; i++) {
         const token = argv[i];
         if (token === "--host" && argv[i + 1]) {
-            host = argv[++i];
+            const value = argv[++i];
+            if (value === "replay") {
+                throw new Error("Replay-based testing is deprecated and no longer supported; use --host cursor or --host claude.");
+            }
+            if (value !== "cursor" && value !== "claude") {
+                throw new Error("--host must be cursor|claude");
+            }
+            host = value;
         }
         else if (token === "--suites-dir" && argv[i + 1]) {
             suitesDir = argv[++i];
@@ -76,15 +80,14 @@ export function parseCliArgs(argv) {
         else if (token === "--staging-session-id" && argv[i + 1]) {
             stagingSessionId = argv[++i];
         }
+        else if (token === "--live") {
+            throw new Error("--live was removed because agent-test now always runs a real agent");
+        }
         else if (token === "--record") {
-            record = true;
+            throw new Error("--record was removed; direct runs capture transient traces automatically (use --keep-recordings to retain them)");
         }
         else if (token === "--record-fixtures") {
-            record = true;
-            recordFixtures = true;
-        }
-        else if (token === "--live") {
-            live = true;
+            throw new Error("--record-fixtures was removed because replay-based testing is deprecated and no longer supported");
         }
         else if (token === "--keep-recordings") {
             keepRecordings = true;
@@ -176,12 +179,8 @@ export function parseCliArgs(argv) {
             throw new Error("compare requires --a <report.json> and --b <report.json>");
         }
     }
-    if (live) {
-        host = host ?? "cursor";
-        record = true;
-        judge = judge ?? true;
-        worktree = worktree ?? true;
-    }
+    judge = judge ?? true;
+    worktree = worktree ?? true;
     if (debug) {
         keepRecordings = true;
         process.env.AGENT_TEST_DEBUG = "1";
@@ -196,9 +195,6 @@ export function parseCliArgs(argv) {
         filter,
         scenarioFilter,
         stagingSessionId,
-        record,
-        recordFixtures,
-        live,
         judge,
         worktree,
         keepRecordings,
@@ -222,6 +218,16 @@ export function parseCliArgs(argv) {
         compareOutDir: compareOutDir ? resolve(cwd, compareOutDir) : undefined,
     };
 }
+/** Resolve an explicit report target into the HTML path and optional artifact directory. */
+export function resolveReportOutput(reportOut) {
+    if (!reportOut) {
+        return {};
+    }
+    if (reportOut.toLowerCase().endsWith(".html")) {
+        return { htmlPath: reportOut };
+    }
+    return { htmlPath: join(reportOut, "report.html"), outDir: reportOut };
+}
 async function pathExists(path) {
     try {
         await access(path);
@@ -260,8 +266,6 @@ async function loadOrRunCompareSide(args, side, stagingSessionId) {
         suitePath,
         host: args.host,
         scenarioFilter: args.scenarioFilter,
-        record: args.record,
-        recordFixtures: args.recordFixtures,
         judge: args.judge,
         worktree: args.worktree,
         stagingSessionId,
@@ -275,26 +279,13 @@ async function loadOrRunCompareSide(args, side, stagingSessionId) {
         scenarioRetries: args.scenarioRetries,
     });
 }
-/**
- * Split `--report-out` into an HTML file path and, when a directory was given,
- * the directory every other report artifact is written to.
- */
-export function resolveReportOutput(reportOut) {
-    if (!reportOut) {
-        return {};
-    }
-    if (reportOut.toLowerCase().endsWith(".html")) {
-        return { htmlPath: reportOut };
-    }
-    return { htmlPath: join(reportOut, "report.html"), outDir: reportOut };
-}
 async function writeSuiteReportDump(outDir, label, report) {
     await mkdir(outDir, { recursive: true });
     const path = join(outDir, `${label}.suite-report.json`);
     await writeFile(path, `${JSON.stringify(report, null, 2)}\n`, "utf8");
     return path;
 }
-async function cleanupLiveRunArtifacts(cwd, stagingSessionRoot, keepRecordings) {
+async function cleanupRunArtifacts(stagingSessionRoot, keepRecordings) {
     if (keepRecordings) {
         return;
     }
@@ -306,10 +297,6 @@ async function cleanupLiveRunArtifacts(cwd, stagingSessionRoot, keepRecordings)
             // best-effort
         }
     }
-    const legacyRemoved = await cleanupLegacyRepoRecordings(cwd);
-    if (legacyRemoved.length > 0) {
-        console.log(`Removed legacy in-repo recording dir(s):\n  ${legacyRemoved.join("\n  ")}`);
-    }
 }
 async function main() {
     let args;
@@ -332,9 +319,7 @@ async function main() {
         const bPath = resolve(args.cwd, args.compareB);
         const aReport = await loadSuiteRunReport(aPath);
         const bReport = await loadSuiteRunReport(bPath);
-        const outDir = args.compareOutDir ??
-            resolveReportOutput(args.reportOut).outDir ??
-            resolve(args.cwd, "compare-out");
+        const outDir = args.compareOutDir ?? resolve(args.cwd, "compare-out");
         const compare = compareSuiteReports({
             aLabel: labelForCompareSide(aPath),
             bLabel: labelForCompareSide(bPath),
@@ -382,47 +367,41 @@ async function main() {
     }
     const isChild = process.env.AGENT_TEST_CHILD === "1";
     if (args.debugDir && isPathUnderRoot(args.debugDir, args.cwd) && !isChild) {
-        console.warn(theme.warn(`--debug-dir is inside the repo (${args.debugDir}). Default is $TMPDIR/agent-spec — prefer that for live runs so debug output stays out of git status.`));
+        console.warn(theme.warn(`--debug-dir is inside the repo (${args.debugDir}). Default is $TMPDIR/agent-spec — prefer that for direct runs so debug output stays out of git status.`));
     }
     const verbose = args.debug || process.env.AGENT_TEST_VERBOSE === "1" || process.env.AGENT_TEST_DEBUG === "1";
     const stagingSessionId = args.stagingSessionId?.trim() ||
         process.env.AGENT_TEST_STAGING_SESSION_ID?.trim() ||
-        (args.live || args.debug || (args.record && !args.recordFixtures)
-            ? createLiveStagingSessionId()
-            : undefined);
+        createLiveStagingSessionId();
     const stagingSessionRoot = stagingSessionId
         ? getLiveStagingSessionRoot(stagingSessionId)
         : undefined;
-    const reportOutput = resolveReportOutput(args.reportOut);
     try {
-        if (args.live) {
-            const host = args.host ?? "cursor";
-            if (host === "claude") {
-                try {
-                    const authMode = parseClaudeAuthMode(process.env[CLAUDE_AUTH_MODE_ENV]);
-                    if (authMode === "api-key" && !process.env.ANTHROPIC_API_KEY?.trim()) {
-                        console.error(`${CLAUDE_AUTH_MODE_ENV}=api-key requires ANTHROPIC_API_KEY for --live --host claude`);
-                        return 1;
-                    }
-                }
-                catch (error) {
-                    console.error(error instanceof Error ? error.message : error);
+        if (args.host === "claude") {
+            try {
+                const authMode = parseClaudeAuthMode(process.env[CLAUDE_AUTH_MODE_ENV]);
+                if (authMode === "api-key" && !process.env.ANTHROPIC_API_KEY?.trim()) {
+                    console.error(`${CLAUDE_AUTH_MODE_ENV}=api-key requires ANTHROPIC_API_KEY`);
                     return 1;
                 }
             }
-            if (host === "cursor" && !process.env.CURSOR_API_KEY?.trim()) {
-                console.error("CURSOR_API_KEY required for --live (Cursor SDK runs)");
-                return 1;
-            }
-            // Judge classifiers still use the Cursor SDK.
-            if (args.judge !== false && !process.env.CURSOR_API_KEY?.trim()) {
-                console.error("CURSOR_API_KEY required for live judge classifiers (use --no-judge to skip)");
+            catch (error) {
+                console.error(error instanceof Error ? error.message : error);
                 return 1;
             }
         }
-        if (args.live) {
+        if (args.host === "cursor" && !process.env.CURSOR_API_KEY?.trim()) {
+            console.error("CURSOR_API_KEY required for Cursor agent runs");
+            return 1;
+        }
+        // Judge classifiers still use the Cursor SDK.
+        if (args.judge !== false && !process.env.CURSOR_API_KEY?.trim()) {
+            console.error("CURSOR_API_KEY required for judge classifiers (use --no-judge to skip)");
+            return 1;
+        }
+        {
             try {
-                await assertLiveDogfoodPreflight(args.cwd, args.suitesDir);
+                await assertDirectAgentPreflight(args.cwd, args.suitesDir);
             }
             catch (error) {
                 console.error(error instanceof Error ? error.message : error);
@@ -433,7 +412,7 @@ async function main() {
                 process.env.AGENT_TEST_NO_WORKTREE === "1" ||
                 process.env.AGENT_TEST_NO_WORKTREE === "true";
             if (worktreeDisabled && !inPlaceAllowed) {
-                console.error("Live dogfood requires git worktree isolation. Set AGENT_TEST_ALLOW_IN_PLACE=1 to run in repo cwd (--no-worktree leaks agent edits into your working tree).");
+                console.error("Direct agent tests require git worktree isolation. Set AGENT_TEST_ALLOW_IN_PLACE=1 to run in repo cwd (--no-worktree leaks agent edits into your working tree).");
                 return 1;
             }
             registerLiveRunHandlers();
@@ -442,7 +421,7 @@ async function main() {
                 if (removed.length > 0) {
                     console.log(theme.warn(`Cleaned ${removed.length} stale agent-test worktree(s) from a prior crash`));
                 }
-                console.log(theme.banner(args.debug ? "live debug" : "live"));
+                console.log(theme.banner(args.debug ? "direct debug" : "direct"));
                 if (stagingSessionRoot) {
                     console.log(theme.bannerSession(stagingSessionRoot));
                 }
@@ -464,10 +443,6 @@ async function main() {
                 }
             }
         }
-        else if (args.debug && !isChild && stagingSessionRoot) {
-            console.log(theme.banner("debug"));
-            console.log(theme.bannerSession(stagingSessionRoot));
-        }
         let reports;
         let comparePassRegressions = 0;
         if (args.comparePairs) {
@@ -477,7 +452,6 @@ async function main() {
             reports = [aReport, bReport];
             if (!isChild) {
                 const outDir = args.compareOutDir ??
-                    reportOutput.outDir ??
                     (stagingSessionRoot
                         ? join(stagingSessionRoot, "compare")
                         : resolve(args.cwd, "compare-out"));
@@ -547,15 +521,8 @@ async function main() {
                         includeCompare: Boolean(args.comparePairs),
                         compareALabel: pair ? labelForCompareSide(pair.a) : undefined,
                         compareBLabel: pair ? labelForCompareSide(pair.b) : undefined,
-                    }, reportOutput.htmlPath);
+                    });
                     console.log(`\n${theme.fileTip("HTML report", reportPath)}`);
-                    // A directory collects everything, not just the HTML.
-                    if (reportOutput.outDir && !args.comparePairs) {
-                        for (const report of reports) {
-                            const dumpPath = await writeSuiteReportDump(reportOutput.outDir, report.suite, report);
-                            console.log(theme.tip(`suite JSON: ${dumpPath}`));
-                        }
-                    }
                 }
                 catch (error) {
                     console.warn(theme.warn(`HTML report failed: ${error instanceof Error ? error.message : String(error)}`));
@@ -581,7 +548,7 @@ async function main() {
     }
     finally {
         if (stagingSessionId && !isChild) {
-            await cleanupLiveRunArtifacts(args.cwd, stagingSessionRoot, args.keepRecordings);
+            await cleanupRunArtifacts(stagingSessionRoot, args.keepRecordings);
         }
     }
 }