@post-print/agent-test 0.3.4 → 0.3.5

package/README.md

@@ -1,12 +1,14 @@
 # @post-print/agent-test
 
-**Source of truth for** the direct-agent test package.
+<!-- source-of-truth: the direct-agent test package -->
 
-<!-- doc-meta: owner=eng | last-reviewed=2026-09-02 -->
+<!-- doc-meta: owner=eng | last-reviewed=2026-09-13 -->
 
-Scenario runner for real Cursor and Claude agents, built on `@post-print/agent-harness`.
+<!-- review-deps: paths=packages/test/src/*.ts,packages/test/src/**/*.ts,packages/test/package.json -->
 
-> **Deprecated and removed:** stored-trace replay is not an agent test. `host: "replay"`, `replayTrace`, `--record-fixtures`, and the old `--live` mode flag are rejected with migration guidance. Every scenario execution now launches Cursor or Claude and can incur provider usage.
+Scenario runner for real Cursor, Claude, and OpenAI Codex agents, built on `@post-print/agent-harness`.
+
+> **Deprecated and removed:** stored-trace replay is not an agent test. `host: "replay"`, `replayTrace`, `--record-fixtures`, and the old `--live` mode flag are rejected with migration guidance.
 
 ## Direct TypeScript API
 
@@ -25,76 +27,153 @@ const result = await runAgentTest({
 });
 ```
 
-The default host is Cursor. Set `scenario.host: "claude"`, `defaults.host`, or the top-level `host` option for Claude. The scenario host wins over the top-level override, which wins over suite defaults. The default suite name is `direct`; worktree isolation, judging, timeout enforcement, and announce-stop retry are enabled by default.
+The default host is Cursor. Set `scenario.host`, `defaults.host`, or the top-level `host` option to `claude` or `openai`. The scenario host wins over the top-level override, which wins over suite defaults. The default suite name is `direct`. Sealed-workspace isolation, judging, timeout enforcement, and announce-stop retry are enabled by default.
 
 ## JSON suites and CLI
 
-JSON is an authoring adapter, not a stored answer. Consumer repositories can keep suites under `agent-suites/<suite>/scenarios.json`:
+JSON is an authoring adapter, not a stored answer. This repository keeps host-agent suites under `agent-suites/<suite>/scenarios.json`. Consumer repositories can use the same layout.
 
-```json
-{
-  "name": "routing",
-  "defaults": { "host": "cursor", "profile": "cursor" },
-  "scenarios": [
-    {
-      "name": "reads instructions",
-      "prompt": "Review the current change.",
-      "rubric": { "mustReadPath": ["AGENTS.md"] }
-    }
-  ]
-}
+```bash
+npx agent-test --suites-dir agent-suites
+npx agent-test --suites-dir agent-suites --host cursor
+npx agent-test --suites-dir agent-suites --suite smoke
+npx agent-test --suites-dir agent-suites --suite tools
+npx agent-test --suites-dir agent-suites --suite mcp
+npx agent-test --suites-dir agent-suites --suite judge
+npx agent-test --suites-dir agent-suites --suite depth
+npx agent-test --check --suites-dir agent-suites
 ```
 
-Run Cursor by default or select Claude explicitly:
+In-repo suites list `hosts: ["cursor", "claude", "openai"]`. `npx agent-test --suites-dir agent-suites` and `bun run test` run every suite on each host. That is the consumer confidence gate. Pass `--host cursor` to pin one adapter.
 
-```bash
-npx agent-test --suites-dir agent-suites
-npx agent-test --suites-dir agent-suites --suite routing
-npx agent-test --suites-dir agent-suites --host claude
-npx agent-test --doctor
+`smoke` is the short host proof. `tools` checks Read and Write. `mcp` checks echo invoke and a lookup read. `judge` starts a second host call that scores the reply. One judge scenario runs two workspace arms and scores both transcripts together. `depth` checks workspace roots, seed patches, injected context, mustRun, and skill invoke. `bun run test:smoke` stays on Cursor.
+
+`scenario.host` pins that scenario to one host. A matrix run skips it on the other hosts.
+
+## Custom hosts
+
+A consumer repo can register its own adapter.
+
+```js
+// agent-test.config.mjs
+import { defineConfig } from "@post-print/agent-test";
+
+export default defineConfig({
+  adapters: [
+    {
+      host: "gemini",
+      missingAuth() {
+        return process.env.GEMINI_API_KEY ? undefined : "GEMINI_API_KEY not set";
+      },
+      classifierHost: "cursor",
+      async run(options) {
+        return {
+          host: "gemini",
+          status: "completed",
+          durationMs: 0,
+          trace: {
+            messages: [{ role: "assistant", content: options.prompt }],
+            toolCalls: [],
+            shellCommands: [],
+            artifacts: {},
+          },
+        };
+      },
+    },
+  ],
+});
 ```
 
-Direct runs require an exported `CURSOR_API_KEY` for Cursor or `ANTHROPIC_API_KEY` plus the Claude Code CLI for Claude. Judge criteria use `CURSOR_API_KEY` unless `--no-judge` is set. The CLI does not load `.env`.
+Then `--host gemini` and a suite `hosts` list can include `gemini`. You can also pass `--adapter ./hosts/gemini.mjs`. The module can export `{ adapters: [...] }` or call `registerHostAdapter` as a side effect.
+
+Do not reuse `cursor`, `claude`, `openai`, `replay`, or `all` as the slug.
+
+Direct runs need host auth. Cursor uses `CURSOR_API_KEY`, or `CURSOR_AUTH_MODE=subscription` after `Cursor.auth.login()`. The Cursor app login does not count. Claude uses `CLAUDE_AUTH_MODE` plus `ANTHROPIC_API_KEY` or a Claude Code login. OpenAI uses `OPENAI_API_KEY` or `CODEX_API_KEY`, or `OPENAI_AUTH_MODE=subscription` after `codex login`. The judge uses the same host family. It runs when a rubric has judge questions or `mustInvokeSkill`. A `compare` scenario runs two arms. `compare.faster` and `compare.cheaper` pick a metric winner. `rubric.judge` is optional and scores both transcripts. `--no-judge` turns the judge off. The CLI does not load `.env`.
 
-The old `--live` flag is removed because direct execution is now the only execution mode. Direct runs capture transient traces automatically; use `--keep-recordings` or `--debug` to retain diagnostics.
+`--allow-user-input` starts a second classifier as the user. That user agent answers AskQuestion-style tools. The test agent then continues with the original task plus the transcript.
 
-## Validation, rubrics, and comparison
+## Check, rubrics, and compare
 
-These commands inspect configuration or existing reports; they do not claim to execute an agent:
+`--check` inspects the suite, seeds, package, and host. It does not launch an agent. A live run runs the same check first, then starts the host.
 
 ```bash
-npx agent-test --validate-only --validate-paths --suites-dir agent-suites
-npx agent-test --validate-seeds --suites-dir agent-suites
-npx agent-test --compare-pairs clean:changed --out-dir "$TMPDIR/compare"
-npx agent-test compare --a clean.suite-report.json --b changed.suite-report.json --out-dir "$TMPDIR/compare"
+npx agent-test --check --suites-dir agent-suites
 ```
 
-`scenarios.json` may omit inline rubric keys when they are supplied by sibling `rubrics.json` / `scenarios.rubric.json`, or by `--rubrics-dir <path>` at `<path>/<suite>/rubrics.json`. External rubric entries replace the inline rubric for the same scenario.
+A compare scenario sets `compare.a` and `compare.b`. Each arm can override workspace, prompt, host, skills, or extra rubric checks. Arm rubric arrays append onto the scenario rubric. Do not put `judge` on an arm. `compare.faster` and `compare.cheaper` name the arm that must win on time or tokens. Add `rubric.judge` when you want a pairwise judge. Omit it for a metric-only compare. `--no-judge` skips the judge.
 
-Reports pair scenarios by `compareId` when present, otherwise by normalized scenario name. JSON, Markdown, and HTML reports include outcomes, token usage, duration, tools, and grounding signals.
+```json
+{
+  "name": "cheaper prompt",
+  "prompt": "Read README.txt. Reply with one sentence.",
+  "compare": {
+    "a": { "label": "verbose", "prompt": "Read README.txt. Write a long summary." },
+    "b": { "label": "short" },
+    "cheaper": "b"
+  },
+  "rubric": { "mustReadPath": ["README.txt"] }
+}
+```
+
+Use two workspaces when one arm has a skill and the other does not. Put `mustInvokeSkill` on the skill arm only.
+
+```json
+{
+  "name": "skill vs no skill",
+  "prompt": "Read every markdown file. Write a detailed shipping plan.",
+  "compare": {
+    "a": {
+      "label": "no skill",
+      "workspace": "workspaces/skill-off",
+      "rubric": { "mustNotInvokeSkill": ["brief-ship"] }
+    },
+    "b": {
+      "label": "with skill",
+      "workspace": "workspaces/skill-on",
+      "skills": [".agents/skills/brief-ship/SKILL.md"],
+      "rubric": { "mustInvokeSkill": ["brief-ship"] }
+    },
+    "cheaper": "b"
+  },
+  "rubric": {
+    "mustReadPath": ["note.md"],
+    "judge": ["Did the skill arm stay closer to the note?"]
+  }
+}
+```
+
+`--doctor`, `--validate-only`, `--validate-paths`, and `--validate-seeds` are aliases for `--check`.
+
+`scenarios.json` can omit inline rubric keys when a sibling `rubrics.json` / `scenarios.rubric.json` or `--rubrics-dir` supplies them.
 
 ## Isolation and diagnostics
 
-Each scenario uses a detached git worktree by default. This isolates edits, but it does not prevent a local host from reading the IDE-open caller checkout. Keep answer keys outside agent-visible roots when that distinction matters.
+In-repo suites set `workspace` on every scenario. The runner copies that folder into a temp repo. Omit `workspace`, or set `"."`, to copy HEAD plus caller context instead. The folder gets its own `.git`. The runner fails the scenario when tool paths leave that folder. A leftover caller-tree check still restores leaked caller edits.
+
+Host-global user skills stay out of the run by default. Those trees live under `~/.cursor/skills`, `~/.claude/skills`, `~/.codex/skills`, and `~/.agents/skills`. A custom `workspace` is a fixture. Keep `allowUserSkills` false for that case. Set `allowUserSkills` to true on the scenario or suite defaults when the test needs the machine skill set. The `skills` field only overlays repo-relative folders into the sealed workspace.
+
+`contextSources` and `skills` are relative to the workspace root when `workspace` is a subfolder. A bare `contextSources` name is a file in that root. `seedPatch` stays a caller-repo path. Its hunks are relative to that workspace.
+
+`--debug` retains an evidence bundle under `$TMPDIR/agent-spec/sessions/<id>/` by default. Use `--debug-dir` to override the parent directory.
 
-`--debug` retains an evidence bundle under `$TMPDIR/agent-spec/sessions/<id>/` by default. It includes the transcript, scenario, result, trace, failures, environment metadata, judge details, and an exact direct-run rerun command. Use `--debug-dir` to override the parent directory.
+Host SDK INFO lines stay hidden. Set `AGENT_TEST_HOST_LOGS=1` or `--debug` to print them.
 
-`Ctrl+C` cancels active Cursor/Claude work and cleans worktrees. `--no-worktree` requires `AGENT_TEST_ALLOW_IN_PLACE=1` because agent edits will persist in the caller checkout.
+A TTY run prints `agent started`, then updates an `agent` clock every 0.1s. Tool names and a short reply preview print as the host streams them. The clock line stays one row so the terminal can overwrite it. The HTML report line is a localhost link. A click opens the browser. The preview exits after 30 minutes idle. Set `AGENT_TEST_NO_REPORT_PREVIEW=1` to skip the preview.
 
-Dialogue-first skills must be tested with a one-shot prompt or an intentional `--allow-user-input` run. Do not weaken the production skill to make a headless test pass.
+`Ctrl+C` cancels active host work and deletes the temp folder. `--no-worktree` requires `AGENT_TEST_ALLOW_IN_PLACE=1` because agent edits will persist in the caller checkout.
 
 ## MCP servers
 
-Cursor and Claude scenarios can attach inline stdio or HTTP/SSE MCP servers through suite defaults or scenario overrides. Scenario server names replace matching defaults. `${ENV_VAR}` placeholders resolve at run time. Ambient project/user MCP configuration is not loaded.
+Scenarios can attach inline stdio or HTTP/SSE MCP servers through suite defaults or scenario overrides. Ambient project/user MCP configuration is not loaded. When `workspace` is a subfolder, stdio MCP cwd is the caller repo. Script args stay caller-relative.
 
 ## In-repo package checks
 
-The repository does not run paid agents in its default CI package-integrity checks:
+Default CI does not launch a paid agent:
 
 ```bash
 bun run build
-node packages/test/dist/cli.js --validate-only --suites-dir packages/test/fixtures --suite smoke
-node packages/test/dist/cli.js --doctor
+node packages/test/dist/cli.js --check --suites-dir packages/test/fixtures --suite smoke
+node packages/test/dist/cli.js --check --suites-dir agent-suites --suite smoke
 ```
 
-Credentialed acceptance requires one TypeScript `runAgentTest` call and one JSON-suite CLI run against a real host.
+Host-agent acceptance is `bun run test`. That command runs smoke, tools, mcp, judge, and depth on Cursor, Claude, and Codex. `bun run test:smoke`, `bun run test:tools`, `bun run test:mcp`, `bun run test:judge`, and `bun run test:depth` stay on Cursor. A key-gated GitHub Actions job runs the same suites on Cursor when `CURSOR_API_KEY` is present.

package/dist/check.d.ts

@@ -0,0 +1,38 @@
+import { type AgentHost } from "@post-print/agent-harness";
+export interface CheckOptions {
+    cwd: string;
+    suitesDir: string;
+    filter?: string;
+    rubricsDir?: string;
+    /** CLI host override. Wins over suite defaults when set. */
+    host?: AgentHost;
+    /** CLI host matrix. Wins over `host` when more than one name is listed. */
+    hosts?: readonly AgentHost[];
+    cliPath?: string;
+}
+export interface CheckReport {
+    /** Package, suite, and seed checks passed. Host readiness is separate. */
+    ok: boolean;
+    packageOk: boolean;
+    suiteOk: boolean;
+    seedsOk: boolean;
+    hostReady: boolean;
+    hosts: AgentHost[];
+    messages: string[];
+}
+/** Hosts a live run will launch after suite defaults and an optional CLI override. */
+export declare function collectSuiteHosts(options: {
+    cwd: string;
+    suitesDir: string;
+    filter?: string;
+    rubricsDir?: string;
+    host?: AgentHost;
+    hosts?: readonly AgentHost[];
+}): Promise<AgentHost[]>;
+/** Missing auth for every host this run will launch. Undefined when all hosts are ready. */
+export declare function missingHostsAuth(hosts: readonly AgentHost[]): string | undefined;
+/** Suite, seed, package, and host checks. Does not launch an agent. */
+export declare function runCheck(options: CheckOptions): Promise<CheckReport>;
+export declare function formatCheckReport(report: CheckReport): string;
+export declare function formatCheckSummary(report: CheckReport): string;
+//# sourceMappingURL=check.d.ts.map

package/dist/check.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"check.d.ts","sourceRoot":"","sources":["../src/check.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,KAAK,SAAS,EAAoB,MAAM,2BAA2B,CAAC;AAS7E,MAAM,WAAW,YAAY;IAC5B,GAAG,EAAE,MAAM,CAAC;IACZ,SAAS,EAAE,MAAM,CAAC;IAClB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,4DAA4D;IAC5D,IAAI,CAAC,EAAE,SAAS,CAAC;IACjB,2EAA2E;IAC3E,KAAK,CAAC,EAAE,SAAS,SAAS,EAAE,CAAC;IAC7B,OAAO,CAAC,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,WAAW;IAC3B,0EAA0E;IAC1E,EAAE,EAAE,OAAO,CAAC;IACZ,SAAS,EAAE,OAAO,CAAC;IACnB,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE,OAAO,CAAC;IACjB,SAAS,EAAE,OAAO,CAAC;IACnB,KAAK,EAAE,SAAS,EAAE,CAAC;IACnB,QAAQ,EAAE,MAAM,EAAE,CAAC;CACnB;AAoBD,sFAAsF;AACtF,wBAAsB,iBAAiB,CAAC,OAAO,EAAE;IAChD,GAAG,EAAE,MAAM,CAAC;IACZ,SAAS,EAAE,MAAM,CAAC;IAClB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,IAAI,CAAC,EAAE,SAAS,CAAC;IACjB,KAAK,CAAC,EAAE,SAAS,SAAS,EAAE,CAAC;CAC7B,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC,CAmDvB;AAED,4FAA4F;AAC5F,wBAAgB,gBAAgB,CAAC,KAAK,EAAE,SAAS,SAAS,EAAE,GAAG,MAAM,GAAG,SAAS,CAMhF;AAED,uEAAuE;AACvE,wBAAsB,QAAQ,CAAC,OAAO,EAAE,YAAY,GAAG,OAAO,CAAC,WAAW,CAAC,CAsE1E;AAED,wBAAgB,iBAAiB,CAAC,MAAM,EAAE,WAAW,GAAG,MAAM,CAE7D;AAED,wBAAgB,kBAAkB,CAAC,MAAM,EAAE,WAAW,GAAG,MAAM,CAO9D"}

package/dist/check.js

@@ -0,0 +1,169 @@
+import { access } from "node:fs/promises";
+import { resolve } from "node:path";
+import { isKnownAgentHost } from "@post-print/agent-harness";
+import { discoverSuites } from "./discover-suites.js";
+import { missingAgentAuth, runDoctor } from "./doctor.js";
+import { resolveSuiteHosts, uniqueHosts } from "./hosts.js";
+import { loadSuiteFile } from "./load-suite.js";
+import { formatSeedValidationReport, validateSeedPatches } from "./validate-seeds.js";
+import { formatValidationReport, validateSuitePaths } from "./validate-suite.js";
+function filterSuitePaths(suitePaths, filter) {
+    if (!filter) {
+        return suitePaths;
+    }
+    return suitePaths.filter((path) => path.includes(`/${filter}/`) || path.endsWith(`/${filter}/scenarios.json`));
+}
+async function suitesDirExists(cwd, suitesDir) {
+    try {
+        await access(resolve(cwd, suitesDir));
+        return true;
+    }
+    catch {
+        return false;
+    }
+}
+/** Hosts a live run will launch after suite defaults and an optional CLI override. */
+export async function collectSuiteHosts(options) {
+    const cliHosts = uniqueHosts(options.hosts ?? (options.host ? [options.host] : undefined));
+    if (cliHosts?.length === 1) {
+        return cliHosts;
+    }
+    const root = resolve(options.cwd, options.suitesDir);
+    let suitePaths = [];
+    try {
+        suitePaths = filterSuitePaths(await discoverSuites(root), options.filter);
+    }
+    catch {
+        return cliHosts ?? ["cursor"];
+    }
+    if (suitePaths.length === 0) {
+        return cliHosts ?? ["cursor"];
+    }
+    const hosts = new Set();
+    for (const suitePath of suitePaths) {
+        try {
+            const suite = await loadSuiteFile(suitePath, { rubricsDir: options.rubricsDir });
+            const resolved = resolveSuiteHosts({
+                cliHosts,
+                suiteHosts: suite.hosts,
+                defaultHost: suite.defaults?.host,
+            });
+            if (resolved.length > 1 || (suite.hosts !== undefined && suite.hosts.length > 0)) {
+                for (const host of resolved) {
+                    hosts.add(host);
+                }
+                continue;
+            }
+            const fallback = suite.defaults?.host ?? "cursor";
+            for (const scenario of suite.scenarios) {
+                if (scenario.skip) {
+                    continue;
+                }
+                const host = scenario.host ?? fallback;
+                if (isKnownAgentHost(host)) {
+                    hosts.add(host);
+                }
+            }
+            if (suite.scenarios.length === 0 && isKnownAgentHost(fallback)) {
+                hosts.add(fallback);
+            }
+        }
+        catch {
+            // Invalid files are reported by suite validation.
+        }
+    }
+    if (hosts.size === 0) {
+        return cliHosts ?? ["cursor"];
+    }
+    return [...hosts];
+}
+/** Missing auth for every host this run will launch. Undefined when all hosts are ready. */
+export function missingHostsAuth(hosts) {
+    const missing = hosts.map((host) => missingAgentAuth(host)).filter((message) => message);
+    if (missing.length === 0) {
+        return undefined;
+    }
+    return missing.join("\n");
+}
+/** Suite, seed, package, and host checks. Does not launch an agent. */
+export async function runCheck(options) {
+    const messages = [];
+    const doctor = runDoctor({ cliPath: options.cliPath });
+    messages.push("package");
+    for (const message of doctor.messages) {
+        messages.push(`  ${message}`);
+    }
+    const hosts = await collectSuiteHosts(options);
+    const hostError = missingHostsAuth(hosts);
+    const hostReady = hostError === undefined;
+    messages.push("host");
+    messages.push(`  ${hosts.join(", ")}: ${hostReady ? "ready" : (hostError ?? "not ready")}`);
+    let suiteOk = false;
+    const suitesRoot = resolve(options.cwd, options.suitesDir);
+    if (!(await suitesDirExists(options.cwd, options.suitesDir))) {
+        messages.push("suite");
+        messages.push(`  missing ${suitesRoot}`);
+    }
+    else {
+        const suitePaths = filterSuitePaths(await discoverSuites(suitesRoot), options.filter);
+        if (suitePaths.length === 0) {
+            messages.push("suite");
+            messages.push(`  No suites found under ${options.suitesDir}`);
+        }
+        else {
+            const suiteReport = await validateSuitePaths(suitePaths, {
+                validatePaths: true,
+                repoRoot: options.cwd,
+                rubricsDir: options.rubricsDir,
+            });
+            suiteOk = suiteReport.ok;
+            messages.push("suite");
+            for (const line of formatValidationReport(suiteReport).split("\n")) {
+                messages.push(`  ${line}`);
+            }
+        }
+    }
+    const seedReport = await validateSeedPatches({
+        cwd: options.cwd,
+        suitesDir: options.suitesDir,
+        rubricsDir: options.rubricsDir,
+        filter: options.filter,
+    }).catch((error) => ({
+        ok: false,
+        issues: [
+            {
+                suite: options.suitesDir,
+                scenario: "*",
+                seedPatch: "",
+                message: error instanceof Error ? error.message : String(error),
+            },
+        ],
+        checked: 0,
+    }));
+    const seedsOk = seedReport.ok;
+    messages.push("seeds");
+    for (const line of formatSeedValidationReport(seedReport).split("\n")) {
+        messages.push(`  ${line}`);
+    }
+    return {
+        ok: doctor.ok && suiteOk && seedsOk,
+        packageOk: doctor.ok,
+        suiteOk,
+        seedsOk,
+        hostReady,
+        hosts,
+        messages,
+    };
+}
+export function formatCheckReport(report) {
+    return ["agent-test check", ...report.messages].join("\n");
+}
+export function formatCheckSummary(report) {
+    const suite = report.suiteOk ? "suite ok" : "suite failed";
+    const pkg = report.packageOk ? "package ok" : "package failed";
+    const host = report.hostReady
+        ? `host ${report.hosts.join(",")} ready`
+        : `host ${report.hosts.join(",")} not ready`;
+    return `check  ${pkg} · ${suite} · ${host}`;
+}
+//# sourceMappingURL=check.js.map

package/dist/check.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"check.js","sourceRoot":"","sources":["../src/check.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,MAAM,EAAE,MAAM,kBAAkB,CAAC;AAC1C,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAEpC,OAAO,EAAkB,gBAAgB,EAAE,MAAM,2BAA2B,CAAC;AAE7E,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AACtD,OAAO,EAAE,gBAAgB,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AAC1D,OAAO,EAAE,iBAAiB,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AAC5D,OAAO,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAC;AAChD,OAAO,EAAE,0BAA0B,EAAE,mBAAmB,EAAE,MAAM,qBAAqB,CAAC;AACtF,OAAO,EAAE,sBAAsB,EAAE,kBAAkB,EAAE,MAAM,qBAAqB,CAAC;AAyBjF,SAAS,gBAAgB,CAAC,UAAoB,EAAE,MAAe;IAC9D,IAAI,CAAC,MAAM,EAAE,CAAC;QACb,OAAO,UAAU,CAAC;IACnB,CAAC;IACD,OAAO,UAAU,CAAC,MAAM,CACvB,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,QAAQ,CAAC,IAAI,MAAM,GAAG,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,IAAI,MAAM,iBAAiB,CAAC,CACpF,CAAC;AACH,CAAC;AAED,KAAK,UAAU,eAAe,CAAC,GAAW,EAAE,SAAiB;IAC5D,IAAI,CAAC;QACJ,MAAM,MAAM,CAAC,OAAO,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC,CAAC;QACtC,OAAO,IAAI,CAAC;IACb,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,KAAK,CAAC;IACd,CAAC;AACF,CAAC;AAED,sFAAsF;AACtF,MAAM,CAAC,KAAK,UAAU,iBAAiB,CAAC,OAOvC;IACA,MAAM,QAAQ,GAAG,WAAW,CAAC,OAAO,CAAC,KAAK,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC;IAC3F,IAAI,QAAQ,EAAE,MAAM,KAAK,CAAC,EAAE,CAAC;QAC5B,OAAO,QAAQ,CAAC;IACjB,CAAC;IACD,MAAM,IAAI,GAAG,OAAO,CAAC,OAAO,CAAC,GAAG,EAAE,OAAO,CAAC,SAAS,CAAC,CAAC;IACrD,IAAI,UAAU,GAAa,EAAE,CAAC;IAC9B,IAAI,CAAC;QACJ,UAAU,GAAG,gBAAgB,CAAC,MAAM,cAAc,CAAC,IAAI,CAAC,EAAE,OAAO,CAAC,MAAM,CAAC,CAAC;IAC3E,CAAC;IAAC,MAAM,CAAC;QACR,OAAO,QAAQ,IAAI,CAAC,QAAQ,CAAC,CAAC;IAC/B,CAAC;IACD,IAAI,UAAU,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC7B,OAAO,QAAQ,IAAI,CAAC,QAAQ,CAAC,CAAC;IAC/B,CAAC;IACD,MAAM,KAAK,GAAG,IAAI,GAAG,EAAa,CAAC;IACnC,KAAK,MAAM,SAAS,IAAI,UAAU,EAAE,CAAC;QACpC,IAAI,CAAC;YACJ,MAAM,KAAK,GAAG,MAAM,aAAa,CAAC,SAAS,EAAE,EAAE,UAAU,EAAE,OAAO,CAAC,UAAU,EAAE,CAAC,CAAC;YACjF,MAAM,QAAQ,GAAG,iBAAiB,CAAC;gBAClC,QAAQ;gBACR,UAAU,EAAE,KAAK,CAAC,KAAK;gBACvB,WAAW,EAAE,KAAK,CAAC,QAAQ,EAAE,IAAI;aACjC,CAAC,CAAC;YACH,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,KAAK,KAAK,SAAS,IAAI,KAAK,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,EAAE,CAAC;gBAClF,KAAK,MAAM,IAAI,IAAI,QAAQ,EAAE,CAAC;oBAC7B,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;gBACjB,CAAC;gBACD,SAAS;YACV,CAAC;YACD,MAAM,QAAQ,GAAG,KAAK,CAAC,QAAQ,EAAE,IAAI,IAAI,QAAQ,CAAC;YAClD,KAAK,MAAM,QAAQ,IAAI,KAAK,CAAC,SAAS,EAAE,CAAC;gBACxC,IAAI,QAAQ,CAAC,IAAI,EAAE,CAAC;oBACnB,SAAS;gBACV,CAAC;gBACD,MAAM,IAAI,GAAG,QAAQ,CAAC,IAAI,IAAI,QAAQ,CAAC;gBACvC,IAAI,gBAAgB,CAAC,IAAI,CAAC,EAAE,CAAC;oBAC5B,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;gBACjB,CAAC;YACF,CAAC;YACD,IAAI,KAAK,CAAC,SAAS,CAAC,MAAM,KAAK,CAAC,IAAI,gBAAgB,CAAC,QAAQ,CAAC,EAAE,CAAC;gBAChE,KAAK,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;YACrB,CAAC;QACF,CAAC;QAAC,MAAM,CAAC;YACR,kDAAkD;QACnD,CAAC;IACF,CAAC;IACD,IAAI,KAAK,CAAC,IAAI,KAAK,CAAC,EAAE,CAAC;QACtB,OAAO,QAAQ,IAAI,CAAC,QAAQ,CAAC,CAAC;IAC/B,CAAC;IACD,OAAO,CAAC,GAAG,KAAK,CAAC,CAAC;AACnB,CAAC;AAED,4FAA4F;AAC5F,MAAM,UAAU,gBAAgB,CAAC,KAA2B;IAC3D,MAAM,OAAO,GAAG,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,gBAAgB,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC;IACzF,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC1B,OAAO,SAAS,CAAC;IAClB,CAAC;IACD,OAAO,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC3B,CAAC;AAED,uEAAuE;AACvE,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,OAAqB;IACnD,MAAM,QAAQ,GAAa,EAAE,CAAC;IAC9B,MAAM,MAAM,GAAG,SAAS,CAAC,EAAE,OAAO,EAAE,OAAO,CAAC,OAAO,EAAE,CAAC,CAAC;IACvD,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACzB,KAAK,MAAM,OAAO,IAAI,MAAM,CAAC,QAAQ,EAAE,CAAC;QACvC,QAAQ,CAAC,IAAI,CAAC,KAAK,OAAO,EAAE,CAAC,CAAC;IAC/B,CAAC;IAED,MAAM,KAAK,GAAG,MAAM,iBAAiB,CAAC,OAAO,CAAC,CAAC;IAC/C,MAAM,SAAS,GAAG,gBAAgB,CAAC,KAAK,CAAC,CAAC;IAC1C,MAAM,SAAS,GAAG,SAAS,KAAK,SAAS,CAAC;IAC1C,QAAQ,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IACtB,QAAQ,CAAC,IAAI,CAAC,KAAK,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,SAAS,IAAI,WAAW,CAAC,EAAE,CAAC,CAAC;IAE5F,IAAI,OAAO,GAAG,KAAK,CAAC;IACpB,MAAM,UAAU,GAAG,OAAO,CAAC,OAAO,CAAC,GAAG,EAAE,OAAO,CAAC,SAAS,CAAC,CAAC;IAC3D,IAAI,CAAC,CAAC,MAAM,eAAe,CAAC,OAAO,CAAC,GAAG,EAAE,OAAO,CAAC,SAAS,CAAC,CAAC,EAAE,CAAC;QAC9D,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;QACvB,QAAQ,CAAC,IAAI,CAAC,aAAa,UAAU,EAAE,CAAC,CAAC;IAC1C,CAAC;SAAM,CAAC;QACP,MAAM,UAAU,GAAG,gBAAgB,CAAC,MAAM,cAAc,CAAC,UAAU,CAAC,EAAE,OAAO,CAAC,MAAM,CAAC,CAAC;QACtF,IAAI,UAAU,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YAC7B,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;YACvB,QAAQ,CAAC,IAAI,CAAC,2BAA2B,OAAO,CAAC,SAAS,EAAE,CAAC,CAAC;QAC/D,CAAC;aAAM,CAAC;YACP,MAAM,WAAW,GAAG,MAAM,kBAAkB,CAAC,UAAU,EAAE;gBACxD,aAAa,EAAE,IAAI;gBACnB,QAAQ,EAAE,OAAO,CAAC,GAAG;gBACrB,UAAU,EAAE,OAAO,CAAC,UAAU;aAC9B,CAAC,CAAC;YACH,OAAO,GAAG,WAAW,CAAC,EAAE,CAAC;YACzB,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;YACvB,KAAK,MAAM,IAAI,IAAI,sBAAsB,CAAC,WAAW,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;gBACpE,QAAQ,CAAC,IAAI,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC;YAC5B,CAAC;QACF,CAAC;IACF,CAAC;IAED,MAAM,UAAU,GAAG,MAAM,mBAAmB,CAAC;QAC5C,GAAG,EAAE,OAAO,CAAC,GAAG;QAChB,SAAS,EAAE,OAAO,CAAC,SAAS;QAC5B,UAAU,EAAE,OAAO,CAAC,UAAU;QAC9B,MAAM,EAAE,OAAO,CAAC,MAAM;KACtB,CAAC,CAAC,KAAK,CAAC,CAAC,KAAc,EAAE,EAAE,CAAC,CAAC;QAC7B,EAAE,EAAE,KAAK;QACT,MAAM,EAAE;YACP;gBACC,KAAK,EAAE,OAAO,CAAC,SAAS;gBACxB,QAAQ,EAAE,GAAG;gBACb,SAAS,EAAE,EAAE;gBACb,OAAO,EAAE,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC;aAC/D;SACD;QACD,OAAO,EAAE,CAAC;KACV,CAAC,CAAC,CAAC;IACJ,MAAM,OAAO,GAAG,UAAU,CAAC,EAAE,CAAC;IAC9B,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC;IACvB,KAAK,MAAM,IAAI,IAAI,0BAA0B,CAAC,UAAU,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;QACvE,QAAQ,CAAC,IAAI,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC;IAC5B,CAAC;IAED,OAAO;QACN,EAAE,EAAE,MAAM,CAAC,EAAE,IAAI,OAAO,IAAI,OAAO;QACnC,SAAS,EAAE,MAAM,CAAC,EAAE;QACpB,OAAO;QACP,OAAO;QACP,SAAS;QACT,KAAK;QACL,QAAQ;KACR,CAAC;AACH,CAAC;AAED,MAAM,UAAU,iBAAiB,CAAC,MAAmB;IACpD,OAAO,CAAC,kBAAkB,EAAE,GAAG,MAAM,CAAC,QAAQ,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC5D,CAAC;AAED,MAAM,UAAU,kBAAkB,CAAC,MAAmB;IACrD,MAAM,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,cAAc,CAAC;IAC3D,MAAM,GAAG,GAAG,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,gBAAgB,CAAC;IAC/D,MAAM,IAAI,GAAG,MAAM,CAAC,SAAS;QAC5B,CAAC,CAAC,QAAQ,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,QAAQ;QACxC,CAAC,CAAC,QAAQ,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,YAAY,CAAC;IAC9C,OAAO,UAAU,GAAG,MAAM,KAAK,MAAM,IAAI,EAAE,CAAC;AAC7C,CAAC"}

package/dist/cli.d.ts

@@ -7,6 +7,12 @@ export interface ParsedCliArgs {
     /** Prefer `<rubricsDir>/<suite>/rubrics.json` (harness-only answer keys). */
     rubricsDir?: string;
     host?: AgentHost;
+    /** Host matrix when `--host` lists more than one adapter. */
+    hosts?: AgentHost[];
+    /** True when `--host` included `all`. Re-expand after adapters load. */
+    hostAll?: boolean;
+    /** Consumer adapter modules (`--adapter`). */
+    adapterModules?: string[];
     filter?: string;
     scenarioFilter?: string;
     stagingSessionId?: string;
@@ -16,6 +22,9 @@ export interface ParsedCliArgs {
     timeoutMs?: number;
     noTimeout: boolean;
     allowUserInput: boolean;
+    /** Check the suite and host. Do not launch an agent. */
+    check: boolean;
+    help: boolean;
     doctor: boolean;
     htmlReport: boolean;
     /** Explicit HTML report path (ends in .html) or output directory for all report content. */
@@ -28,14 +37,9 @@ export interface ParsedCliArgs {
     failOn: FailOnMode;
     /** Live announce-stop retries (overrides AGENT_TEST_SCENARIO_RETRIES). */
     scenarioRetries?: number;
-    /** Offline compare subcommand (`agent-test compare --a … --b …`). */
-    compareMode: boolean;
-    compareA?: string;
-    compareB?: string;
-    /** Direct-run A:B suite dirs or report JSON paths. */
-    comparePairs?: string;
-    compareOutDir?: string;
 }
+/** Usage text for `--help`. */
+export declare function formatHelp(): string;
 /** Parse agent-test CLI argv (exported for unit tests). */
 export declare function parseCliArgs(argv: string[]): ParsedCliArgs;
 /** Resolve an explicit report target into the HTML path and optional artifact directory. */

package/dist/cli.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAKA,OAAO,EACN,KAAK,SAAS,EAKd,MAAM,2BAA2B,CAAC;AAsBnC,OAAO,EACN,KAAK,UAAU,EAIf,MAAM,oBAAoB,CAAC;AAU5B,MAAM,WAAW,aAAa;IAC7B,GAAG,EAAE,MAAM,CAAC;IACZ,SAAS,EAAE,MAAM,CAAC;IAClB,6EAA6E;IAC7E,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,IAAI,CAAC,EAAE,SAAS,CAAC;IACjB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,cAAc,EAAE,OAAO,CAAC;IACxB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,OAAO,CAAC;IACnB,cAAc,EAAE,OAAO,CAAC;IACxB,MAAM,EAAE,OAAO,CAAC;IAChB,UAAU,EAAE,OAAO,CAAC;IACpB,4FAA4F;IAC5F,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,KAAK,EAAE,OAAO,CAAC;IACf,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,OAAO,CAAC;IACtB,aAAa,EAAE,OAAO,CAAC;IACvB,aAAa,EAAE,OAAO,CAAC;IACvB,MAAM,EAAE,UAAU,CAAC;IACnB,0EAA0E;IAC1E,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,qEAAqE;IACrE,WAAW,EAAE,OAAO,CAAC;IACrB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,sDAAsD;IACtD,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,aAAa,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,2DAA2D;AAC3D,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,aAAa,CAmL1D;AAED,4FAA4F;AAC5F,wBAAgB,mBAAmB,CAAC,SAAS,CAAC,EAAE,MAAM,GAAG;IAAE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAAC,MAAM,CAAC,EAAE,MAAM,CAAA;CAAE,CAQ9F"}
+{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAIA,OAAO,EACN,KAAK,SAAS,EAKd,MAAM,2BAA2B,CAAC;AAmBnC,OAAO,EACN,KAAK,UAAU,EAIf,MAAM,oBAAoB,CAAC;AAO5B,MAAM,WAAW,aAAa;IAC7B,GAAG,EAAE,MAAM,CAAC;IACZ,SAAS,EAAE,MAAM,CAAC;IAClB,6EAA6E;IAC7E,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,IAAI,CAAC,EAAE,SAAS,CAAC;IACjB,6DAA6D;IAC7D,KAAK,CAAC,EAAE,SAAS,EAAE,CAAC;IACpB,wEAAwE;IACxE,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,8CAA8C;IAC9C,cAAc,CAAC,EAAE,MAAM,EAAE,CAAC;IAC1B,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,cAAc,EAAE,OAAO,CAAC;IACxB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,OAAO,CAAC;IACnB,cAAc,EAAE,OAAO,CAAC;IACxB,wDAAwD;IACxD,KAAK,EAAE,OAAO,CAAC;IACf,IAAI,EAAE,OAAO,CAAC;IACd,MAAM,EAAE,OAAO,CAAC;IAChB,UAAU,EAAE,OAAO,CAAC;IACpB,4FAA4F;IAC5F,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,KAAK,EAAE,OAAO,CAAC;IACf,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,OAAO,CAAC;IACtB,aAAa,EAAE,OAAO,CAAC;IACvB,aAAa,EAAE,OAAO,CAAC;IACvB,MAAM,EAAE,UAAU,CAAC;IACnB,0EAA0E;IAC1E,eAAe,CAAC,EAAE,MAAM,CAAC;CACzB;AA+CD,+BAA+B;AAC/B,wBAAgB,UAAU,IAAI,MAAM,CA8BnC;AAED,2DAA2D;AAC3D,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,aAAa,CAkP1D;AAED,4FAA4F;AAC5F,wBAAgB,mBAAmB,CAAC,SAAS,CAAC,EAAE,MAAM,GAAG;IAAE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAAC,MAAM,CAAC,EAAE,MAAM,CAAA;CAAE,CAQ9F"}