@post-print/agent-test 1.1.8 → 2.0.0

package/README.md

@@ -1,77 +1,29 @@
 # @post-print/agent-test
 
-<!-- source-of-truth: the direct-agent test package -->
-
+<!-- source-of-truth: named agent and judge factories for TypeScript tests -->
 <!-- doc-meta: owner=eng | last-reviewed=2026-09-16 -->
-
 <!-- review-deps: paths=packages/test/src/*.ts,packages/test/src/**/*.ts,packages/test/package.json -->
 
-Scenario runner for real Cursor, Claude, and OpenAI Codex agents, built on `@post-print/agent-harness`.
-
-> **Deprecated and removed:** stored-trace replay is not an agent test. `host: "replay"`, `replayTrace`, `--record-fixtures`, and the old `--live` mode flag are rejected with migration guidance.
-
-## Direct TypeScript API
-
-`runAgentTest` is the execution core. JSON suites load into this same function.
-
 ```ts
-import { runAgentTest } from "@post-print/agent-test";
-
-const result = await runAgentTest({
-  cwd: process.cwd(),
-  scenario: {
-    name: "uses the project instructions",
-    description: "Checks that the agent reads AGENTS.md.",
-    contextMode: "host-native",
-    prompt: "Review the current change.",
-    rubric: { reviewDepth: "standard" },
-  },
+import { describe, expect, z } from "@post-print/agent-test";
+
+const test = describe("project answers", ({ agent, judge }) => ({
+  coder: agent({ skills: ["./skills/research"] }),
+  accuracy: judge({
+    prompt: "Assess the answer against the reference.",
+    schema: z.object({ correct: z.boolean(), reason: z.string() }),
+  }),
+}));
+
+test("answers correctly", async ({ coder, accuracy }) => {
+  const run = await coder.run({ prompt: "Who owns the project?" });
+  const evaluation = await accuracy.run({ input: { answer: run.output, reference: "Mina" } });
+  expect(evaluation.output.correct).toBe(true);
 });
 ```
 
-The default host is Cursor. Set `contextMode: "host-native"` to send the scenario prompt unchanged and let the host discover its supported workspace context. The compatibility default, `harness-preamble`, composes configured sources and synthetic skill catalogs into the submitted input. Set `scenario.host`, `defaults.host`, or the top-level `host` option to `claude` or `openai`. The scenario host wins over the top-level override, which wins over suite defaults. The default suite name is `direct`. Sealed-workspace isolation, judging, timeout enforcement, and announce-stop retry are enabled by default.
-
-OpenAI scenarios can set `networkAccess: true` when the task must reach a package registry or another network service. It is scenario-only and defaults to `false`. The opt-in does not disable the sealed workspace, temporary user home, or leak checks, and it does not apply to the read-only judge.
-
-The viewer and HTML report put a **Starting context** summary beside each task. It names supplied files, servers, tools, and skills, while the recorded conversation shows the submitted task prompt. In host-native mode, the summary states that the host discovers workspace files and instructions; agent-test does not claim which host-owned context was loaded.
-
-## JSON suites and CLI
-
-JSON is an authoring adapter, not a stored answer. This repository keeps host-agent suites under `agent-suites/<suite>/scenarios.json`.
-
-```bash
-npx agent-test login
-npx agent-test --check --suites-dir agent-suites
-npx agent-test viewer --suites-dir agent-suites --workers 2
-npx agent-test --suites-dir agent-suites --host cursor --suite test-sdk-capabilities
-npx agent-test --suites-dir agent-suites --host cursor --auth-mode api-key
-```
-
-First run: [docs/getting-started.md](../../docs/getting-started.md). Flags: [docs/cli.md](../../docs/cli.md). Suite fields, rubric, compare, and MCP: [docs/suites.md](../../docs/suites.md). Auth and adapters: [docs/hosts.md](../../docs/hosts.md). Sealed workspace and debug: [docs/isolation.md](../../docs/isolation.md).
-
-`--doctor`, `--validate-only`, `--validate-paths`, and `--validate-seeds` are aliases for `--check`.
-
-## In-repo package checks
-
-Default CI does not launch a host agent:
-
-```bash
-bun run build
-node packages/test/dist/cli.js --check --suites-dir packages/test/fixtures --suite smoke
-node packages/test/dist/cli.js --check --suites-dir agent-suites
-```
-
-`bun run test` is offline and sandbox-safe. Provider-backed runs are manual: `bun run test:capabilities` exercises supported `runAgentTest` capabilities on Cursor, and the tour has its own command. The host matrix and 20-run qualification are also manual. CI does not launch a provider-backed host.
-
-`bun run test:sdk:consumer` packs both packages, installs them in a clean Node 22 fixture, compiles the public declarations, runs a fake adapter, inspects comparison results, and runs the installed CLI. It does not use host credentials.
+Set default `agent`, `judge`, and `workspace` directly in `defineConfig`. Harness definitions select providers, models, and auth. Factories only declare resources; each test receives its own executable handles. Every run starts fresh; use `run.continue` for the same task. Compare with ordinary JavaScript and `expect`.
 
-## Viewer e2e
-
-Playwright drives the suite viewer and the HTML report preview. The viewer shows completed tests and completed arms separately. Completed arms remain provisional until their comparison is finalized; each ready comparison finalizes within the worker limit while unrelated agents continue. Navigator, selected-test header, and host tabs share the same status, and reloading an active run restores its event stream. Cancellation suppresses unfinished verdicts. Each compare arm has a tab that swaps the visible chat.
-
-```bash
-bunx playwright install chromium
-bun run test:e2e
-```
+Judge input is explicitly selected JSON data. Output is schema-validated and typed. The runner records artifacts and usage, handles cancellation, and owns cleanup. Global skills default to false. No browser is required for CLI execution.
 
-The suite uses a fake runner. It does not launch a host agent.
+See [the SDK guide](../../docs/sdk-v2.md), [tour](../../agent-suites/tour/tour.spec.ts), and [getting started](../../docs/getting-started.md). The old `test.use`, `compare`, `defineJudge`, and run-owned judge API are removed.

package/dist/cli.d.ts

@@ -1,65 +1,3 @@
-#!/usr/bin/env -S node --disable-warning=ExperimentalWarning
-import { type AgentHost, type HostAuthMode } from "@post-print/agent-harness";
-import { type FailOnMode } from "./suite-summary.js";
-import type { CompareArmId } from "./types.js";
-export interface ParsedCliArgs {
-    cwd: string;
-    suitesDir: string;
-    /** Prefer `<rubricsDir>/<suite>/rubrics.json` (harness-only answer keys). */
-    rubricsDir?: string;
-    host?: AgentHost;
-    /** Host matrix when `--host` lists more than one adapter. */
-    hosts?: AgentHost[];
-    /** True when `--host` included `all`. Re-expand after adapters load. */
-    hostAll?: boolean;
-    /** Consumer adapter modules (`--adapter`). */
-    adapterModules?: string[];
-    filter?: string;
-    scenarioFilter?: string;
-    stagingSessionId?: string;
-    judge?: boolean;
-    worktree?: boolean;
-    keepRecordings: boolean;
-    timeoutMs?: number;
-    noTimeout: boolean;
-    allowUserInput: boolean;
-    /** Check the suite and host. Do not launch an agent. */
-    check: boolean;
-    help: boolean;
-    doctor: boolean;
-    htmlReport: boolean;
-    /** Explicit HTML report path (ends in .html) or output directory for all report content. */
-    reportOut?: string;
-    debug: boolean;
-    debugDir?: string;
-    validateOnly: boolean;
-    validateSeeds: boolean;
-    validatePaths: boolean;
-    failOn: FailOnMode;
-    /** Live announce-stop retries (overrides AGENT_TEST_SCENARIO_RETRIES). */
-    scenarioRetries?: number;
-    /** Host billing mode. Default is subscription when omitted. */
-    authMode?: HostAuthMode;
-    /** `agent-test login` — store host CLI or Cursor SDK credentials. */
-    login: boolean;
-    /** `agent-test viewer` — localhost catalog and live chat UI. */
-    viewer: boolean;
-    /** Isolated child: run one compare arm only. */
-    compareArm?: CompareArmId;
-    /** Viewer listen port. `0` picks a free port. */
-    viewerPort?: number;
-    /** How many live agents run at once. Viewer default is 4. CLI default is 1. */
-    workers?: number;
-}
-/** Usage text for `--help`. */
-export declare function formatHelp(): string;
-/** Parse agent-test CLI argv (exported for unit tests). */
-export declare function parseCliArgs(argv: string[]): ParsedCliArgs;
-/** Resolve an explicit report target into the HTML path and optional artifact directory. */
-export declare function resolveReportOutput(reportOut?: string): {
-    htmlPath?: string;
-    outDir?: string;
-};
-/** Path passed to `writeHtmlReport`. `undefined` keeps the temp default. */
-export declare function resolveHtmlReportWritePath(htmlReport: boolean, reportOut?: string): string | undefined;
+#!/usr/bin/env node
+export declare function runCli(args: string[]): Promise<number>;
 //# sourceMappingURL=cli.d.ts.map

package/dist/cli.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAIA,OAAO,EACN,KAAK,SAAS,EAEd,KAAK,YAAY,EAMjB,MAAM,2BAA2B,CAAC;AA+BnC,OAAO,EACN,KAAK,UAAU,EAIf,MAAM,oBAAoB,CAAC;AAE5B,OAAO,KAAK,EAAE,YAAY,EAAkB,MAAM,YAAY,CAAC;AAS/D,MAAM,WAAW,aAAa;IAC7B,GAAG,EAAE,MAAM,CAAC;IACZ,SAAS,EAAE,MAAM,CAAC;IAClB,6EAA6E;IAC7E,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,IAAI,CAAC,EAAE,SAAS,CAAC;IACjB,6DAA6D;IAC7D,KAAK,CAAC,EAAE,SAAS,EAAE,CAAC;IACpB,wEAAwE;IACxE,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,8CAA8C;IAC9C,cAAc,CAAC,EAAE,MAAM,EAAE,CAAC;IAC1B,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,cAAc,EAAE,OAAO,CAAC;IACxB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,OAAO,CAAC;IACnB,cAAc,EAAE,OAAO,CAAC;IACxB,wDAAwD;IACxD,KAAK,EAAE,OAAO,CAAC;IACf,IAAI,EAAE,OAAO,CAAC;IACd,MAAM,EAAE,OAAO,CAAC;IAChB,UAAU,EAAE,OAAO,CAAC;IACpB,4FAA4F;IAC5F,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,KAAK,EAAE,OAAO,CAAC;IACf,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,OAAO,CAAC;IACtB,aAAa,EAAE,OAAO,CAAC;IACvB,aAAa,EAAE,OAAO,CAAC;IACvB,MAAM,EAAE,UAAU,CAAC;IACnB,0EAA0E;IAC1E,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,+DAA+D;IAC/D,QAAQ,CAAC,EAAE,YAAY,CAAC;IACxB,qEAAqE;IACrE,KAAK,EAAE,OAAO,CAAC;IACf,gEAAgE;IAChE,MAAM,EAAE,OAAO,CAAC;IAChB,gDAAgD;IAChD,UAAU,CAAC,EAAE,YAAY,CAAC;IAC1B,iDAAiD;IACjD,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,+EAA+E;IAC/E,OAAO,CAAC,EAAE,MAAM,CAAC;CACjB;AA+CD,+BAA+B;AAC/B,wBAAgB,UAAU,IAAI,MAAM,CAwCnC;AAED,2DAA2D;AAC3D,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,aAAa,CA+R1D;AAED,4FAA4F;AAC5F,wBAAgB,mBAAmB,CAAC,SAAS,CAAC,EAAE,MAAM,GAAG;IAAE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAAC,MAAM,CAAC,EAAE,MAAM,CAAA;CAAE,CAQ9F;AAED,4EAA4E;AAC5E,wBAAgB,0BAA0B,CACzC,UAAU,EAAE,OAAO,EACnB,SAAS,CAAC,EAAE,MAAM,GAChB,MAAM,GAAG,SAAS,CAKpB"}
+{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAQA,wBAAsB,MAAM,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC,MAAM,CAAC,CAoB5D"}