explorbot 0.4.10 → 0.5.1

package/docs/superpowers/specs/2026-09-18-judge-decision-model-design.md

@@ -0,0 +1,79 @@
+# Judge — a decision model tier — Design
+
+An optional *decision model* answers one narrow question with a probability over the answers offered. Explorbot uses it where it would otherwise guess: a code call site asks, and a confident answer lets the site skip its own, more expensive decision. The model is TypeSafe's Jev (a "System One" model), reached through OpenRouter or TypeSafe's own API.
+
+With `ai.decisionModel` unset, nothing registers and nothing calls out.
+
+## The interface
+
+```ts
+judge.decide(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision>
+
+class Decision {
+  readonly value: string | null;   // the winning option; 'yes' for an approved yes/no
+  readonly confidence: number;     // P(yes) for a yes/no, the chosen option's probability for a list
+  get approved(): boolean;         // value !== null
+  get rejected(): boolean;         // value === null
+}
+```
+
+- **An array is a categorization**, sent as a Choice. `value` is the chosen option. Including `UNDECIDED` lets the model say none fits.
+- **A boolean or `null` is a yes/no**, sent as a Noul, whose single number is P(yes). Only a confident yes approves.
+- **Approved means the answer's probability is above 70%** and it isn't `UNDECIDED`.
+- **Everything else is rejected**: a confident no, a low probability, `UNDECIDED`, a timeout, a failed request, a list with fewer than two options, or the direct path being disabled. `decide` never throws and never returns `null`.
+
+The threshold and every failure mode live in one place. A call site only ever sees a confident decision or a rejection, so it cannot misread uncertainty.
+
+### The one rule for call sites
+
+**`rejected` means "not approved", never "confidently no".** It absorbs uncertainty, so a question must be phrased so that `approved` is the action the site would take, and `rejected` falls through to today's behaviour. Then `if (decision?.approved)` is correct by construction.
+
+`consult(...)` is the same call without the direct-path gate, used by the tool.
+
+## Configuration
+
+```js
+ai: { decisionModel: { provider: 'openrouter', model: 'typesafe/jev-1.13' } }             // via OpenRouter
+ai: { decisionModel: { provider: 'typesafe', model: 'jev-latest' } }                       // TypeSafe API directly
+ai: { decisionModel: { provider: 'openrouter', model: 'typesafe/jev-1.13', tool: false } } // direct sites only
+```
+
+`config.ts` holds only the field type. An unknown provider or a missing API key throws at startup, naming what to fix.
+
+Everything transport-specific lives in `src/ai/judge-provider.ts`: `endpointFor()` (a `switch` over `openrouter` and `typesafe`, each with its endpoint and API-key variable), the HTTP call, timeout, and the Noul/Choice wire format. That file is temporary: when the Vercel AI SDK supports decision models it is deleted and `decisionModel` becomes a regular provider-built model.
+
+`Judge.fromConfig()` builds the judge; it reaches agents through `AgentDeps` and tools through `ToolDeps`.
+
+## Where it is used
+
+| Site | Question | Approved means |
+|---|---|---|
+| `failedToolResult`, multi-element branch | Which listed element does the intent name? | suggest that element by `elementIndex` |
+| `Pilot.analyzeProgress` | The run is moving toward the goal and can continue without a supervisor now. | skip the review silently |
+| `Pilot.settleExpectations`, text-only path | What did this run establish about the expected outcome? | settle it; the rest go to the agentic model |
+| `Navigator.verifyState`, before the prompt | Which already verified claim means the same as this one? | treat it as verified, skip the HTML-bearing prompt |
+| `Navigator.verifyState`, inexpressible branch | The page shows that this claim is true. | report it as a judgement, not an assertion |
+| Prima `go()`, semantic target | Which listed control leads to the target? / The page now shows the target. | click the ref, confirm arrival, skip the navigator |
+| `judge` tool (Tester and Pilot) | whatever the model asks | the answer; otherwise "not confirmed" |
+
+Invariants the sites keep:
+
+- **A judge answer never enters `verifications`.** A dedup match skips the work but writes no cache entry for the new claim, because a cache entry stands in for a proof.
+- **Prima confirms arrival** before returning a success envelope. `cli.ts` exits on `envelope.ok`.
+- **The tool assembles its own state**: scenario, compact ARIA capped at `JUDGE_PAGE_CAP`, and recent steps. The model supplies only the question.
+
+## Deliberately not used
+
+- **Pilot's verdict.** A generic "the app never held the state the scenario assumes" would fire on boundary give-ups, like "previous page" on page 1 when `»` is listed, and turn an executable scenario into a clean-looking skip. That hides the execution gap. The fix for a boundary give-up belongs in planning, not the verdict.
+- **Experience filtering.** It removes no model call, and one question per stored block on every prompt build costs more than it saves.
+- **Generation, screenshots, tool-calling loops.** The model is text-only and does not generate.
+
+## Measurements behind the threshold
+
+On states reconstructed from recorded traces:
+
+- A literal, concrete phrasing beat an abstract one on identical state: 0.88 versus 0.58. Questions should name what they expect to see.
+- A composite "continue or call Pilot" Choice tied at confidence 0.08 on a run that was progressing. Under `rejected` semantics, a tie falls through to a review, which is the safe direction.
+- The first live probe picked `copy` over `root` at 0.48 versus 0.41 on a dialog whose Copy button was disabled. A 70% bar rejects that, correctly.
+
+Threshold and phrasing should be tuned from traces: every call records `question`, `value` and `confidence` on its `judge.decide` span, and failures record their reason.

package/docs/web-testing/decisions.md

@@ -0,0 +1,108 @@
+# Decisions
+
+A decision model answers one closed question about the current state: a yes/no statement, or a pick from a list of options. It returns the answer together with its probability. Explorbot uses it to settle narrow choices quickly and cheaply that would otherwise need a call to a larger model or a guess.
+
+The decision model is optional. Without it Explorbot behaves exactly as described elsewhere in these docs. Setup is covered in [AI providers](../basics/providers.md#decision-model).
+
+## Decision rules
+
+Every question has one of two shapes:
+
+| Shape | Asked as | Answer |
+|---|---|---|
+| **Statement** | "The page shows that this claim is true." | Probability that the statement holds |
+| **Choice** | "Which listed element does the intent name?" plus numbered options | The chosen option and its probability |
+
+A choice always includes an extra *undecided* option, so the model can say that none of the options fits.
+
+Explorbot acts on an answer only when it is **approved**:
+
+- a statement's probability is above **70%**, or
+- a choice picked a real option, not *undecided*, with a probability above **70%**.
+
+Anything else counts as **rejected**, including:
+
+- a confident "no", or any probability of 70% or lower
+- *undecided*
+- a choice offered with fewer than two options
+- a request timeout (15 seconds) or a failed request
+- `direct: false` in the config, for built-in decision points
+
+A rejected answer never means "confidently no". It means Explorbot has no usable answer, so it takes its usual path. Every question is worded so that approving it lets Explorbot take a shortcut and rejecting it keeps the usual path. A wrong or unsure answer can therefore cost a shortcut, but never a check.
+
+## Built-in decision points
+
+These are fixed places in the code where Explorbot asks the decision model before doing the usual, more expensive work. Setting `direct: false` turns them off.
+
+### Tester: locator matches several elements
+
+When a click or fill fails because the locator matched more than one element, Explorbot lists the matches and asks which one the step meant.
+
+- **Approved:** the Tester is told which element to use and how to target it with `elementIndex`.
+- **Fallback:** the Tester receives the numbered list of matches and picks one itself, narrows the locator, or uses a visual click.
+
+### Pilot: progress review
+
+Every few steps, Pilot reviews the Tester's recent actions with the `agenticModel`. First it asks whether the run is moving toward its goal and can continue unsupervised for now.
+
+- **Approved:** the review is skipped and the Tester continues. This saves one `agenticModel` call.
+- **Fallback:** Pilot runs its full review as usual.
+
+### Navigator: claim already verified
+
+Before Navigator writes assertions for a claim, it asks whether a claim it has already verified on this page means the same thing, even if it is worded differently.
+
+- **Approved:** the claim counts as verified and no assertions are generated or run.
+- **Fallback:** Navigator verifies the claim with assertions as usual.
+
+The new claim is not added to the page's verified claims. Only a claim backed by assertions that actually ran is recorded as verified.
+
+### Navigator: claim no assertion can express
+
+Some claims cannot be turned into an assertion. When that happens, Navigator asks whether the page shows the claim is true.
+
+- **Approved:** `verify` reports that the page appears to confirm the claim. The result is marked as a judgement, kept apart from assertions that ran, and the Tester is asked to restate the claim so it can be asserted.
+- **Fallback:** `verify` reports that no assertion could express the claim, and suggests restating it or checking it with `see`.
+
+In both cases `verify` returns a failure, because nothing ran in the browser.
+
+### Prima: `go` to a described page
+
+When `prima go` gets a description of a page instead of a URL, it asks which control on the current page leads there.
+
+- **Approved:** Prima clicks that control. It then asks a second question: whether the new page shows the target. Only when that is approved too does Prima report success.
+- **Fallback:** if the choice is rejected, or the click fails, or the page doesn't change, or arrival is not confirmed, Prima hands off to the Navigator as usual.
+
+### Prima: `check` expected outcomes
+
+When `prima check` settles the expected outcomes of a finished run without a screenshot, it asks what the run established about each outcome: that it happened, or that it did not.
+
+- **Approved:** that outcome is settled as passed or failed without calling the `agenticModel`.
+- **Fallback:** the remaining outcomes are settled by the `agenticModel` as usual. When a screenshot is available, all outcomes go to the vision model and the decision model is not asked.
+
+## The `judge` tool
+
+With `tool: true`, Tester and Pilot get a `judge` tool. They use it when a step depends on reading the page rather than running a command. The tool takes:
+
+| Argument | Description |
+|---|---|
+| `question` | A statement to confirm, or the question a list of options answers |
+| `options` | Possible answers. Leave it out to confirm a statement |
+| `context` | Anything the page itself doesn't show |
+
+The model supplies only the question. The tool gathers the rest of the state itself: the current scenario, the compact ARIA snapshot of the page (up to 12,000 characters), and the last eight test steps.
+
+- **Approved:** the tool returns the answer and its confidence.
+- **Rejected:** the tool returns "Not confirmed" and tells the agent to gather more context or try another route. "Not confirmed" means the page doesn't settle the question. It does not mean the statement is false.
+
+`tool` and `direct` are independent. For example, `tool: false` keeps the built-in decision points but stops the agents from asking questions of their own.
+
+## Where it is not used
+
+- **Pilot's verdict on a test.** Whether a scenario passed, failed or was impossible is always decided by the `agenticModel`.
+- **Filtering experience.** One question per stored experience block, on every prompt, would cost more than it saves.
+- **Generation, screenshots and tool calling.** The decision model reads text and answers questions. It does not write, look at images or act on the page.
+
+## Tracing
+
+Each question is recorded as a `judge.decide` span with the question, the answer and its confidence. A failed request also records the reason. The TUI shows `⚖️ Asking judge...` while a question is in flight. See [Observability](../contributing/observability.md) for setting up tracing.

package/docs/workflow/agentic-usage.md

@@ -52,6 +52,7 @@ No `init`, no config file, no project directory, no model IDs to look up. These
 | `EXPLORBOT_URL` | yes | Base URL to test; the API boat reads it as the base endpoint |
 | `EXPLORBOT_VISION_MODEL` | no | Screenshot analysis; overrides the provider recommendation |
 | `EXPLORBOT_AGENTIC_MODEL` | no | Captain and Pilot decisions; overrides the provider recommendation |
+| `EXPLORBOT_DECISION_MODEL` | no | Turns on the decision model, as provider/model-id (openrouter/typesafe/jev-1.13 or typesafe/jev-latest); overrides ai.decisionModel |
 | `EXPLORBOT_OUTPUT` | no | Output root for states, plans, research, and reports. Defaults to the site dir under ~/.explorbot/sites |
 | `EXPLORBOT_EPHEMERAL` | no | Keep no state between runs — output goes to a fresh temp directory instead of the site dir |
 | `EXPLORBOT_KNOWLEDGE` | no | Inline knowledge text, applied to every page |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "explorbot",
-  "version": "0.4.10",
+  "version": "0.5.1",
   "description": "CLI app built with React Ink, CodeceptJS, and Playwright",
   "license": "Elastic-2.0",
   "type": "module",
@@ -60,7 +60,8 @@
     "check": "biome check .",
     "check:fix": "biome check --write .",
     "langfuse:export": "bun run .claude/skills/explorbot-debug/langfuse-export.ts",
-    "build:prima": "bun run scripts/build-prima-npm.ts"
+    "build:prima": "bun run scripts/build-prima-npm.ts",
+    "build:mdq": "bun run scripts/build-mdq-npm.ts"
   },
   "keywords": [
     "cli",
@@ -101,7 +102,7 @@
     "bash-tool": "^1.3.15",
     "chalk": "^5.6.2",
     "cli-highlight": "^2.1.11",
-    "codeceptjs": "^4.2.0-beta.3",
+    "codeceptjs": "^4.2.0",
     "commander": "^14.0.1",
     "debug": "^4.4.3",
     "dedent": "^1.6.0",

package/src/ai/agent.ts

@@ -4,6 +4,7 @@ import type Explorer from '../explorer.ts';
 import type { KnowledgeTracker } from '../knowledge-tracker.ts';
 import type { PlaywrightRecorder } from '../playwright-recorder.ts';
 import type { StateManager } from '../state-manager.ts';
+import type { Judge } from './judge.ts';
 import type { AIProvider } from './provider.ts';
 
 export interface Agent {
@@ -18,6 +19,7 @@ export interface AgentDeps {
   knowledgeTracker: KnowledgeTracker;
   requestStore: RequestStore;
   playwrightRecorder: PlaywrightRecorder;
+  judge?: Judge;
 }
 
-export type ToolDeps = Pick<AgentDeps, 'explorer' | 'stateManager' | 'ai'>;
+export type ToolDeps = Pick<AgentDeps, 'explorer' | 'stateManager' | 'ai' | 'judge'>;

package/src/ai/judge-provider.ts

@@ -0,0 +1,62 @@
+const REQUEST_TIMEOUT_MS = 15000;
+
+export class JudgeProvider {
+  private fetchImpl: typeof fetch = fetch;
+  private requestTimeoutMs = REQUEST_TIMEOUT_MS;
+  private url: string;
+  private apiKey: string;
+
+  constructor(
+    provider: string,
+    readonly model: string
+  ) {
+    const { url, keyName } = JudgeProvider.endpointFor(provider);
+    const apiKey = process.env[keyName];
+    if (!apiKey) throw new Error(`Set ${keyName} to use the decision model`);
+    this.url = url;
+    this.apiKey = apiKey;
+  }
+
+  async decide(state: unknown, question: string, options?: string[]): Promise<ProviderDecision> {
+    let q: Record<string, unknown> = { type: 'noul', instructions: question };
+    if (options) q = { type: 'choice', instructions: question, criteria: Object.fromEntries(options.map((option, index) => [String(index), option])) };
+    const body = JSON.stringify({ model: this.model, state, questions: { q } });
+    const controller = new AbortController();
+    const timer = setTimeout(() => controller.abort(), this.requestTimeoutMs);
+    try {
+      const response = await this.fetchImpl(this.url, {
+        method: 'POST',
+        headers: { Authorization: `Bearer ${this.apiKey}`, 'Content-Type': 'application/json' },
+        body,
+        signal: controller.signal,
+      });
+      if (!response.ok) throw new Error(`http_${response.status}`);
+
+      const answer = (await response.json())?.answers?.q;
+      if (!options && typeof answer?.noul === 'number') return { value: 'yes', probability: answer.noul };
+
+      const value = options?.[Number(answer?.choice)];
+      const probability = answer?.probabilities?.[answer?.choice];
+      if (value === undefined || typeof probability !== 'number') throw new Error('malformed_body');
+      return { value, probability };
+    } finally {
+      clearTimeout(timer);
+    }
+  }
+
+  private static endpointFor(provider: string): { url: string; keyName: string } {
+    switch (provider) {
+      case 'openrouter':
+        return { url: 'https://openrouter.ai/api/alpha/decisions', keyName: 'OPENROUTER_API_KEY' };
+      case 'typesafe':
+        return { url: 'https://api.typesafe.ai/v1/systemone', keyName: 'TYPESAFE_API_KEY' };
+      default:
+        throw new Error(`Unknown decision model provider "${provider}" — use "openrouter" or "typesafe"`);
+    }
+  }
+}
+
+export interface ProviderDecision {
+  value: string;
+  probability: number;
+}

package/src/ai/judge-tool.ts

@@ -0,0 +1,35 @@
+import { tool } from 'ai';
+import dedent from 'dedent';
+import { z } from 'zod';
+import type { ToolDeps } from './agent.ts';
+import { failedToolResult, successToolResult } from './tools.ts';
+
+export function createJudgeTool(deps: ToolDeps, buildState: () => Promise<Record<string, unknown>>): Record<string, any> {
+  const judge = deps.judge;
+  if (!judge?.toolEnabled) return {};
+
+  return {
+    judge: tool({
+      description: dedent`
+        Settle one judgement about the current page instead of guessing. Phrase it literally and concretely.
+      `,
+      inputSchema: z.object({
+        question: z.string().describe('The statement to confirm, or the question the options answer'),
+        options: z.array(z.string()).optional().describe('Possible answers. Omit to confirm a statement'),
+        context: z.string().optional().describe('Anything the page observation does not already carry'),
+      }),
+      execute: async ({ question, options, context }) => {
+        const state = await buildState();
+        if (context) state.context = context;
+
+        const decision = await judge.consult(question, options ?? null, state);
+        if (decision.rejected) {
+          return failedToolResult('judge', `Not confirmed: ${question}`, {
+            suggestion: 'The page does not settle this. Gather more context or take another route; do not assume either answer.',
+          });
+        }
+        return successToolResult('judge', { question, answer: decision.value, confidence: decision.confidence });
+      },
+    }),
+  };
+}

package/src/ai/judge.ts

@@ -0,0 +1,75 @@
+import { clearActivity, setActivity } from '../activity.ts';
+import type { AIConfig } from '../config.ts';
+import { Observability } from '../observability.ts';
+import { createDebug } from '../utils/logger.ts';
+import { JudgeProvider } from './judge-provider.ts';
+
+const debugLog = createDebug('explorbot:judge');
+
+const APPROVAL_THRESHOLD = 0.7;
+
+export const UNDECIDED = 'undecided';
+export const JUDGE_PAGE_CAP = 12000;
+
+export class Decision {
+  constructor(
+    readonly value: string | null,
+    readonly confidence: number
+  ) {}
+
+  get approved(): boolean {
+    return this.value !== null;
+  }
+
+  get rejected(): boolean {
+    return this.value === null;
+  }
+}
+
+export class Judge {
+  constructor(
+    private provider: JudgeProvider,
+    private enabled: { tool: boolean; direct: boolean }
+  ) {}
+
+  static fromConfig(config: AIConfig['decisionModel']): Judge | null {
+    if (!config) return null;
+    return new Judge(new JudgeProvider(config.provider, config.model), { tool: config.tool !== false, direct: config.direct !== false });
+  }
+
+  get toolEnabled(): boolean {
+    return this.enabled.tool;
+  }
+
+  async decide(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision> {
+    if (!this.enabled.direct) return new Decision(null, 0);
+    return this.consult(question, options, state);
+  }
+
+  async consult(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision> {
+    if (Array.isArray(options) && options.length < 2) return new Decision(null, 0);
+    return Observability.run('judge.decide', { tags: ['judge'] }, async () => {
+      setActivity('⚖️ Asking judge...', 'ai');
+      const decision = await this.request(question, options, state).finally(() => clearActivity());
+      Observability.getSpan()?.setAttribute('ai.telemetry.metadata.judgeDecision', JSON.stringify({ question, value: decision.value, confidence: decision.confidence }));
+      return decision;
+    });
+  }
+
+  private async request(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision> {
+    let list: string[] | undefined;
+    if (Array.isArray(options)) list = options;
+
+    const answer = await this.provider.decide(state, question, list).catch((error: unknown) => this.recordFailure(error));
+    if (!answer) return new Decision(null, 0);
+    if (answer.probability <= APPROVAL_THRESHOLD) return new Decision(null, answer.probability);
+    if (answer.value === UNDECIDED) return new Decision(null, answer.probability);
+    return new Decision(answer.value, answer.probability);
+  }
+
+  private recordFailure(error: unknown): null {
+    debugLog('judge declined: %s', error);
+    Observability.getSpan()?.setAttribute('ai.telemetry.metadata.judgeError', String(error));
+    return null;
+  }
+}

package/src/ai/navigator.ts

@@ -21,6 +21,7 @@ import { normalizeInlineText } from '../utils/strings.ts';
 import { extractStatePath, isSameHostFamily, matchesNavigationUrl } from '../utils/url-matcher.js';
 import type { Agent, AgentDeps } from './agent.js';
 import type { Conversation } from './conversation.js';
+import { type Decision, JUDGE_PAGE_CAP, type Judge, UNDECIDED } from './judge.ts';
 import type { Provider } from './provider.js';
 import { Researcher } from './researcher.ts';
 import { actionRule, locatorRule, unexpectedPopupRule } from './rules.js';
@@ -80,6 +81,7 @@ class Navigator implements Agent {
   private explorer: Explorer;
   private config: ExplorbotConfig;
   private stateManager: StateManager;
+  private judge?: Judge;
 
   constructor(deps: AgentDeps) {
     this.provider = deps.ai;
@@ -89,6 +91,7 @@ class Navigator implements Agent {
     this.knowledgeTracker = deps.knowledgeTracker;
     this.experienceTracker = deps.stateManager.getExperienceTracker();
     this.hooksRunner = new HooksRunner(deps.explorer, deps.config);
+    this.judge = deps.judge;
   }
 
   private get verifyAttempts(): number {
@@ -570,7 +573,7 @@ class Navigator implements Agent {
     const countVisit = (value?: string | null) => {
       if (!value) return;
       const normalized = normalizeUrl(value);
-      if (normalized) visitCounts.set(normalized, (visitCounts.get(normalized) || 0) + 1);
+      visitCounts.set(normalized, (visitCounts.get(normalized) || 0) + 1);
     };
 
     for (const transition of history) {
@@ -581,7 +584,7 @@ class Navigator implements Agent {
     if (opts?.visitedUrls) {
       for (const url of opts.visitedUrls) {
         const normalized = normalizeUrl(url);
-        if (normalized && !visitCounts.has(normalized)) {
+        if (!visitCounts.has(normalized)) {
           visitCounts.set(normalized, 1);
         }
       }
@@ -682,7 +685,7 @@ class Navigator implements Agent {
     return suggestion;
   }
 
-  async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; inexpressible: boolean; results: AssertionResult[]; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number }> {
+  async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; inexpressible: boolean; results: AssertionResult[]; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number; judged?: Decision }> {
     tag('info').log('AI Navigator verifying state at', actionResult.url);
     debugLog('Verification message:', message);
 
@@ -692,6 +695,13 @@ class Navigator implements Agent {
       return { verified: cachedVerification, inexpressible: false, results: [], successfulCodes: [], assertionSteps: [], totalAttempted: 0 };
     }
 
+    const verifiedClaims = Object.keys(actionResult.verifications ?? {}).filter((claim) => actionResult.getVerification(claim) === true);
+    const same = await this.judge?.decide('Which already verified claim means the same as the claim under consideration?', [...verifiedClaims, UNDECIDED], { claim: message });
+    if (same?.approved) {
+      tag('operation').log(`Judge matched claim to an already verified one: "${same.value}"`);
+      return { verified: true, inexpressible: false, results: [], successfulCodes: [], assertionSteps: [], totalAttempted: 0 };
+    }
+
     const knowledge = this.knowledgeTracker.renderRelevantContext(actionResult);
     let experience = '';
 
@@ -832,7 +842,8 @@ class Navigator implements Agent {
     const inexpressible = !alreadyVerified && totalAttempted === 0;
     if (inexpressible) {
       tag('warning').log('No assertion could express this claim');
-      return { verified: false, inexpressible, results, successfulCodes, assertionSteps, totalAttempted };
+      const judged = await this.judge?.decide('The page shows that this claim is true.', null, { claim: message, page: actionResult.getCompactARIA().slice(0, JUDGE_PAGE_CAP) });
+      return { verified: false, inexpressible, results, successfulCodes, assertionSteps, totalAttempted, judged };
     }
 
     actionResult.addVerification(message, verified);

package/src/ai/pilot.ts

@@ -8,7 +8,7 @@ import type Explorer from '../explorer.ts';
 import type { PlaywrightRecorder } from '../playwright-recorder.ts';
 import type { StateManager } from '../state-manager.ts';
 import { Stats } from '../stats.ts';
-import { type Test, TestResult } from '../test-plan.ts';
+import { type Test, TestResult, TestStatus } from '../test-plan.ts';
 import { collectInteractiveNodes } from '../utils/aria.ts';
 import { ErrorPageError } from '../utils/error-page.ts';
 import { createDebug, tag } from '../utils/logger.ts';
@@ -19,6 +19,7 @@ import type { Agent, AgentDeps } from './agent.ts';
 import type { Conversation } from './conversation.ts';
 import type { Fisherman } from './fisherman.ts';
 import { createAskApiTool } from './fisherman/tools.ts';
+import { type Judge, UNDECIDED } from './judge.ts';
 import type { Navigator } from './navigator.ts';
 import type { Provider } from './provider.ts';
 import type { Researcher } from './researcher.ts';
@@ -33,6 +34,10 @@ const PILOT_REASONING_LIMIT = 500;
 const PILOT_MESSAGE_LIMIT = 2;
 const PILOT_MESSAGE_MAX_LENGTH = 160;
 const PILOT_REQUEST_LIMIT = 5;
+const OUTCOME_STATUS: Record<string, SettledStatus> = {
+  'The run shows this outcome happened.': 'passed',
+  'The run shows this outcome did not happen.': 'failed',
+};
 
 export class Pilot implements Agent {
   emoji = '🧭';
@@ -45,6 +50,7 @@ export class Pilot implements Agent {
   private requestStore: RequestStore;
   private playwrightRecorder: PlaywrightRecorder;
   private fisherman: Fisherman | null = null;
+  private judge?: Judge;
 
   constructor(deps: AgentDeps, agentTools: any, researcher: Researcher) {
     this.provider = deps.ai;
@@ -54,6 +60,7 @@ export class Pilot implements Agent {
     this.stateManager = deps.stateManager;
     this.requestStore = deps.requestStore;
     this.playwrightRecorder = deps.playwrightRecorder;
+    this.judge = deps.judge;
   }
 
   setFisherman(fisherman: Fisherman): void {
@@ -548,6 +555,9 @@ export class Pilot implements Agent {
     const actionsContext = this.formatActions(toolCalls);
     const stateContext = this.buildStateContext(currentState);
 
+    const healthy = await this.judge?.decide('The run is moving toward the goal and can continue without a supervisor reviewing it now.', null, { scenario: task.scenario, state: stateContext, recentActions: actionsContext });
+    if (healthy?.approved) return '';
+
     const hasFailures = toolCalls.length === 0 || toolCalls.some((t) => !t.wasSuccessful);
 
     const text = await this.sendToPilot(
@@ -592,7 +602,11 @@ export class Pilot implements Agent {
 
     let undecided = task.expected.filter((text) => !task.getCheckedExpectations().includes(text));
     if (image) undecided = task.expected;
-    if (!undecided.length) return task.expected.map((text) => ({ text, status: decided(text) }));
+
+    let settledByJudge = new Map<string, SettledStatus>();
+    if (!image) settledByJudge = await this.settleByJudge(task, undecided);
+    undecided = undecided.filter((text) => !settledByJudge.has(text));
+    if (!undecided.length) return task.expected.map((text) => ({ text, status: settledByJudge.get(text) || decided(text) }));
 
     const schema = z.object({
       outcomes: z.array(
@@ -665,6 +679,8 @@ export class Pilot implements Agent {
 
     const judged = new Map((response?.object?.outcomes || []).map((outcome: any) => [outcome.expectation, outcome]));
     return task.expected.map((text) => {
+      const byJudge = settledByJudge.get(text);
+      if (byJudge) return { text, status: byJudge };
       if (!undecided.includes(text)) return { text, status: decided(text) };
       const outcome = judged.get(text) as { status: SettledStatus; evidence?: string } | undefined;
       if (!outcome) return { text, status: 'unverified' as SettledStatus };
@@ -672,6 +688,22 @@ export class Pilot implements Agent {
     });
   }
 
+  private async settleByJudge(task: Test, expectations: string[]): Promise<Map<string, SettledStatus>> {
+    const settled = new Map<string, SettledStatus>();
+    const judge = this.judge;
+    if (!judge) return settled;
+
+    const state = { scenario: task.scenario, runLog: task.notesToString() || 'No steps recorded.' };
+    await Promise.all(
+      expectations.map(async (text) => {
+        const decision = await judge.decide(`What did this run establish about the expected outcome: ${text}`, [...Object.keys(OUTCOME_STATUS), UNDECIDED], state);
+        const status = OUTCOME_STATUS[decision.value ?? ''];
+        if (status) settled.set(text, status);
+      })
+    );
+    return settled;
+  }
+
   private formatExpectations(task: Test): string {
     const checked = task.getCheckedExpectations();
     const remaining = task.getRemainingExpectations();
@@ -690,6 +722,7 @@ export class Pilot implements Agent {
     this.conversation!.addUserText(finalUserText);
 
     const tools = { ...this.pickPlanningTools(), ...this.buildFishermanTools(opts.task) };
+    const preparedCount = opts.task.preparedData.length;
 
     const result = await this.provider.invokeConversation(this.conversation!, tools, {
       maxToolRoundtrips: opts.maxToolRoundtrips ?? 0,
@@ -698,7 +731,7 @@ export class Pilot implements Agent {
       stopWhen: () => opts.task.hasFinished,
       telemetry: { functionId },
     });
-    const text = result?.response?.text || '';
+    const text = this.announcePreparedData(result?.response?.text || '', opts.task, preparedCount);
     const learned = (result?.toolExecutions || []).filter((e: any) => e.toolName === 'learnExperience' && e.output?.content).map((e: any) => ({ url: e.output.url, content: e.output.content }));
     if (learned.length === 0) return text;
     opts.task.applyExperience(learned);
@@ -714,6 +747,24 @@ export class Pilot implements Agent {
     `;
   }
 
+  private announcePreparedData(text: string, task: Test, preparedCount: number): string {
+    const prepared = task.preparedData.slice(preparedCount);
+    if (prepared.length === 0) return text;
+
+    let refresh = '';
+    if (task.status === TestStatus.IN_PROGRESS) refresh = 'It was created after the page loaded, so the page does not show it yet. Run I.refreshPage() through form() before looking for it.';
+
+    return dedent`
+      ${text}
+
+      <prepared_data>
+      Pilot created this data through the API for this test. Use it instead of creating the same data through the UI:
+      ${prepared.map((item) => `- ${item}`).join('\n')}
+      ${refresh}
+      </prepared_data>
+    `;
+  }
+
   private getExperienceToc(): string {
     const state = this.stateManager.getCurrentState();
     if (!state) return '';
@@ -721,7 +772,7 @@ export class Pilot implements Agent {
   }
 
   private pickPlanningTools() {
-    const { see, context, verify, research, getVisitedStates, xpathCheck, learnExperience, askUser } = this.agentTools ?? {};
+    const { see, context, verify, research, getVisitedStates, xpathCheck, learnExperience, askUser, judge } = this.agentTools ?? {};
     const planning: Record<string, unknown> = {};
     if (see) planning.see = see;
     if (context) planning.context = context;
@@ -731,6 +782,7 @@ export class Pilot implements Agent {
     if (xpathCheck) planning.xpathCheck = xpathCheck;
     if (learnExperience) planning.learnExperience = learnExperience;
     if (askUser) planning.askUser = askUser;
+    if (judge) planning.judge = judge;
     withdrawVisionTools(planning);
     return planning;
   }
@@ -777,6 +829,7 @@ export class Pilot implements Agent {
           });
           const stepText = `Precondition: created ${items.join(', ')}`;
           task.addStep(stepText);
+          task.preparedData.push(...items);
           tag('success').log(stepText);
 
           return { noted: true, prepared: true, created: result.created };
@@ -1157,7 +1210,7 @@ export class Pilot implements Agent {
       Tester tools: click, pressKey, form, see, verify, interact, context, research, xpathCheck,
       visualClick, back, getVisitedStates, reset, stop, finish, record.
       Use tool names exactly as listed. Do not invent combined names or aliases.
-      Reloading is not a tool: to re-read a page from the server, instruct Tester to run I.reloadPage() through form.
+      Reloading is not a tool: to re-read a page from the server, instruct Tester to run I.refreshPage() through form.
 
       ${capabilityGroundingRule}