explorbot 0.4.9 → 0.5.0

package/docs/superpowers/specs/2026-09-18-judge-decision-model-design.md

@@ -0,0 +1,79 @@
+# Judge — a decision model tier — Design
+
+An optional *decision model* answers one narrow question with a probability over the answers offered. Explorbot uses it where it would otherwise guess: a code call site asks, and a confident answer lets the site skip its own, more expensive decision. The model is TypeSafe's Jev (a "System One" model), reached through OpenRouter or TypeSafe's own API.
+
+With `ai.decisionModel` unset, nothing registers and nothing calls out.
+
+## The interface
+
+```ts
+judge.decide(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision>
+
+class Decision {
+  readonly value: string | null;   // the winning option; 'yes' for an approved yes/no
+  readonly confidence: number;     // P(yes) for a yes/no, the chosen option's probability for a list
+  get approved(): boolean;         // value !== null
+  get rejected(): boolean;         // value === null
+}
+```
+
+- **An array is a categorization**, sent as a Choice. `value` is the chosen option. Including `UNDECIDED` lets the model say none fits.
+- **A boolean or `null` is a yes/no**, sent as a Noul, whose single number is P(yes). Only a confident yes approves.
+- **Approved means the answer's probability is above 70%** and it isn't `UNDECIDED`.
+- **Everything else is rejected**: a confident no, a low probability, `UNDECIDED`, a timeout, a failed request, a list with fewer than two options, or the direct path being disabled. `decide` never throws and never returns `null`.
+
+The threshold and every failure mode live in one place. A call site only ever sees a confident decision or a rejection, so it cannot misread uncertainty.
+
+### The one rule for call sites
+
+**`rejected` means "not approved", never "confidently no".** It absorbs uncertainty, so a question must be phrased so that `approved` is the action the site would take, and `rejected` falls through to today's behaviour. Then `if (decision?.approved)` is correct by construction.
+
+`consult(...)` is the same call without the direct-path gate, used by the tool.
+
+## Configuration
+
+```js
+ai: { decisionModel: { provider: 'openrouter', model: 'typesafe/jev-1.13' } }             // via OpenRouter
+ai: { decisionModel: { provider: 'typesafe', model: 'jev-latest' } }                       // TypeSafe API directly
+ai: { decisionModel: { provider: 'openrouter', model: 'typesafe/jev-1.13', tool: false } } // direct sites only
+```
+
+`config.ts` holds only the field type. An unknown provider or a missing API key throws at startup, naming what to fix.
+
+Everything transport-specific lives in `src/ai/judge-provider.ts`: `endpointFor()` (a `switch` over `openrouter` and `typesafe`, each with its endpoint and API-key variable), the HTTP call, timeout, and the Noul/Choice wire format. That file is temporary: when the Vercel AI SDK supports decision models it is deleted and `decisionModel` becomes a regular provider-built model.
+
+`Judge.fromConfig()` builds the judge; it reaches agents through `AgentDeps` and tools through `ToolDeps`.
+
+## Where it is used
+
+| Site | Question | Approved means |
+|---|---|---|
+| `failedToolResult`, multi-element branch | Which listed element does the intent name? | suggest that element by `elementIndex` |
+| `Pilot.analyzeProgress` | The run is moving toward the goal and can continue without a supervisor now. | skip the review silently |
+| `Pilot.settleExpectations`, text-only path | What did this run establish about the expected outcome? | settle it; the rest go to the agentic model |
+| `Navigator.verifyState`, before the prompt | Which already verified claim means the same as this one? | treat it as verified, skip the HTML-bearing prompt |
+| `Navigator.verifyState`, inexpressible branch | The page shows that this claim is true. | report it as a judgement, not an assertion |
+| Prima `go()`, semantic target | Which listed control leads to the target? / The page now shows the target. | click the ref, confirm arrival, skip the navigator |
+| `judge` tool (Tester and Pilot) | whatever the model asks | the answer; otherwise "not confirmed" |
+
+Invariants the sites keep:
+
+- **A judge answer never enters `verifications`.** A dedup match skips the work but writes no cache entry for the new claim, because a cache entry stands in for a proof.
+- **Prima confirms arrival** before returning a success envelope. `cli.ts` exits on `envelope.ok`.
+- **The tool assembles its own state**: scenario, compact ARIA capped at `JUDGE_PAGE_CAP`, and recent steps. The model supplies only the question.
+
+## Deliberately not used
+
+- **Pilot's verdict.** A generic "the app never held the state the scenario assumes" would fire on boundary give-ups, like "previous page" on page 1 when `»` is listed, and turn an executable scenario into a clean-looking skip. That hides the execution gap. The fix for a boundary give-up belongs in planning, not the verdict.
+- **Experience filtering.** It removes no model call, and one question per stored block on every prompt build costs more than it saves.
+- **Generation, screenshots, tool-calling loops.** The model is text-only and does not generate.
+
+## Measurements behind the threshold
+
+On states reconstructed from recorded traces:
+
+- A literal, concrete phrasing beat an abstract one on identical state: 0.88 versus 0.58. Questions should name what they expect to see.
+- A composite "continue or call Pilot" Choice tied at confidence 0.08 on a run that was progressing. Under `rejected` semantics, a tie falls through to a review, which is the safe direction.
+- The first live probe picked `copy` over `root` at 0.48 versus 0.41 on a dialog whose Copy button was disabled. A 70% bar rejects that, correctly.
+
+Threshold and phrasing should be tuned from traces: every call records `question`, `value` and `confidence` on its `judge.decide` span, and failures record their reason.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "explorbot",
-  "version": "0.4.9",
+  "version": "0.5.0",
   "description": "CLI app built with React Ink, CodeceptJS, and Playwright",
   "license": "Elastic-2.0",
   "type": "module",
@@ -60,7 +60,8 @@
     "check": "biome check .",
     "check:fix": "biome check --write .",
     "langfuse:export": "bun run .claude/skills/explorbot-debug/langfuse-export.ts",
-    "build:prima": "bun run scripts/build-prima-npm.ts"
+    "build:prima": "bun run scripts/build-prima-npm.ts",
+    "build:mdq": "bun run scripts/build-mdq-npm.ts"
   },
   "keywords": [
     "cli",

package/src/action.ts

@@ -16,7 +16,9 @@ import { createDebug, setStepSpanParent, tag } from './utils/logger.js';
 import { Overlay, OverlayPage } from './utils/overlay.js';
 import { sleep, waitForPageReadiness } from './utils/page-readiness.ts';
 import type { Region } from './utils/region.js';
+import { isInternalStep } from './utils/step-analyzer.ts';
 import { safeFilename } from './utils/strings.ts';
+import { isSameHostFamily } from './utils/url-matcher.js';
 import { codeceptJSSandbox, hasPlaywrightCommands, playwrightSandbox, sanitizeCodeBlock } from './utils/web-sandbox.ts';
 
 const debugLog = createDebug('explorbot:action');
@@ -317,7 +319,7 @@ class Action {
 
     const url = URL.parse(request.url());
     if (!url) return;
-    if (url.origin !== this.baseOrigin) return;
+    if (!isSameHostFamily(url.href, this.baseOrigin)) return;
 
     const call: NetworkCall = { method: request.method(), path: url.pathname, status };
     if (this.networkRequests.some((r) => r.method === call.method && r.path === call.path && r.status === call.status)) return;
@@ -563,7 +565,7 @@ export const attachStepLogger = (target: ExecutedStep[], assertionsTarget?: Arra
   let batchFailed = false;
   const listener: StepListener = (step, error) => {
     if (!step?.toCode) return;
-    if (step.name?.startsWith('grab')) return;
+    if (isInternalStep(step)) return;
 
     const existing = recorded.get(step);
     if (existing) {

package/src/ai/agent.ts

@@ -4,6 +4,7 @@ import type Explorer from '../explorer.ts';
 import type { KnowledgeTracker } from '../knowledge-tracker.ts';
 import type { PlaywrightRecorder } from '../playwright-recorder.ts';
 import type { StateManager } from '../state-manager.ts';
+import type { Judge } from './judge.ts';
 import type { AIProvider } from './provider.ts';
 
 export interface Agent {
@@ -18,6 +19,7 @@ export interface AgentDeps {
   knowledgeTracker: KnowledgeTracker;
   requestStore: RequestStore;
   playwrightRecorder: PlaywrightRecorder;
+  judge?: Judge;
 }
 
-export type ToolDeps = Pick<AgentDeps, 'explorer' | 'stateManager' | 'ai'>;
+export type ToolDeps = Pick<AgentDeps, 'explorer' | 'stateManager' | 'ai' | 'judge'>;

package/src/ai/judge-provider.ts

@@ -0,0 +1,62 @@
+const REQUEST_TIMEOUT_MS = 15000;
+
+export class JudgeProvider {
+  private fetchImpl: typeof fetch = fetch;
+  private requestTimeoutMs = REQUEST_TIMEOUT_MS;
+  private url: string;
+  private apiKey: string;
+
+  constructor(
+    provider: string,
+    readonly model: string
+  ) {
+    const { url, keyName } = JudgeProvider.endpointFor(provider);
+    const apiKey = process.env[keyName];
+    if (!apiKey) throw new Error(`Set ${keyName} to use the decision model`);
+    this.url = url;
+    this.apiKey = apiKey;
+  }
+
+  async decide(state: unknown, question: string, options?: string[]): Promise<ProviderDecision> {
+    let q: Record<string, unknown> = { type: 'noul', instructions: question };
+    if (options) q = { type: 'choice', instructions: question, criteria: Object.fromEntries(options.map((option, index) => [String(index), option])) };
+    const body = JSON.stringify({ model: this.model, state, questions: { q } });
+    const controller = new AbortController();
+    const timer = setTimeout(() => controller.abort(), this.requestTimeoutMs);
+    try {
+      const response = await this.fetchImpl(this.url, {
+        method: 'POST',
+        headers: { Authorization: `Bearer ${this.apiKey}`, 'Content-Type': 'application/json' },
+        body,
+        signal: controller.signal,
+      });
+      if (!response.ok) throw new Error(`http_${response.status}`);
+
+      const answer = (await response.json())?.answers?.q;
+      if (!options && typeof answer?.noul === 'number') return { value: 'yes', probability: answer.noul };
+
+      const value = options?.[Number(answer?.choice)];
+      const probability = answer?.probabilities?.[answer?.choice];
+      if (value === undefined || typeof probability !== 'number') throw new Error('malformed_body');
+      return { value, probability };
+    } finally {
+      clearTimeout(timer);
+    }
+  }
+
+  private static endpointFor(provider: string): { url: string; keyName: string } {
+    switch (provider) {
+      case 'openrouter':
+        return { url: 'https://openrouter.ai/api/alpha/decisions', keyName: 'OPENROUTER_API_KEY' };
+      case 'typesafe':
+        return { url: 'https://api.typesafe.ai/v1/systemone', keyName: 'TYPESAFE_API_KEY' };
+      default:
+        throw new Error(`Unknown decision model provider "${provider}" — use "openrouter" or "typesafe"`);
+    }
+  }
+}
+
+export interface ProviderDecision {
+  value: string;
+  probability: number;
+}

package/src/ai/judge-tool.ts

@@ -0,0 +1,35 @@
+import { tool } from 'ai';
+import dedent from 'dedent';
+import { z } from 'zod';
+import type { ToolDeps } from './agent.ts';
+import { failedToolResult, successToolResult } from './tools.ts';
+
+export function createJudgeTool(deps: ToolDeps, buildState: () => Promise<Record<string, unknown>>): Record<string, any> {
+  const judge = deps.judge;
+  if (!judge?.toolEnabled) return {};
+
+  return {
+    judge: tool({
+      description: dedent`
+        Settle one judgement about the current page instead of guessing. Phrase it literally and concretely.
+      `,
+      inputSchema: z.object({
+        question: z.string().describe('The statement to confirm, or the question the options answer'),
+        options: z.array(z.string()).optional().describe('Possible answers. Omit to confirm a statement'),
+        context: z.string().optional().describe('Anything the page observation does not already carry'),
+      }),
+      execute: async ({ question, options, context }) => {
+        const state = await buildState();
+        if (context) state.context = context;
+
+        const decision = await judge.consult(question, options ?? null, state);
+        if (decision.rejected) {
+          return failedToolResult('judge', `Not confirmed: ${question}`, {
+            suggestion: 'The page does not settle this. Gather more context or take another route; do not assume either answer.',
+          });
+        }
+        return successToolResult('judge', { question, answer: decision.value, confidence: decision.confidence });
+      },
+    }),
+  };
+}

package/src/ai/judge.ts

@@ -0,0 +1,75 @@
+import { clearActivity, setActivity } from '../activity.ts';
+import type { AIConfig } from '../config.ts';
+import { Observability } from '../observability.ts';
+import { createDebug } from '../utils/logger.ts';
+import { JudgeProvider } from './judge-provider.ts';
+
+const debugLog = createDebug('explorbot:judge');
+
+const APPROVAL_THRESHOLD = 0.7;
+
+export const UNDECIDED = 'undecided';
+export const JUDGE_PAGE_CAP = 12000;
+
+export class Decision {
+  constructor(
+    readonly value: string | null,
+    readonly confidence: number
+  ) {}
+
+  get approved(): boolean {
+    return this.value !== null;
+  }
+
+  get rejected(): boolean {
+    return this.value === null;
+  }
+}
+
+export class Judge {
+  constructor(
+    private provider: JudgeProvider,
+    private enabled: { tool: boolean; direct: boolean }
+  ) {}
+
+  static fromConfig(config: AIConfig['decisionModel']): Judge | null {
+    if (!config) return null;
+    return new Judge(new JudgeProvider(config.provider, config.model), { tool: config.tool !== false, direct: config.direct !== false });
+  }
+
+  get toolEnabled(): boolean {
+    return this.enabled.tool;
+  }
+
+  async decide(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision> {
+    if (!this.enabled.direct) return new Decision(null, 0);
+    return this.consult(question, options, state);
+  }
+
+  async consult(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision> {
+    if (Array.isArray(options) && options.length < 2) return new Decision(null, 0);
+    return Observability.run('judge.decide', { tags: ['judge'] }, async () => {
+      setActivity('⚖️ Asking judge...', 'ai');
+      const decision = await this.request(question, options, state).finally(() => clearActivity());
+      Observability.getSpan()?.setAttribute('ai.telemetry.metadata.judgeDecision', JSON.stringify({ question, value: decision.value, confidence: decision.confidence }));
+      return decision;
+    });
+  }
+
+  private async request(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision> {
+    let list: string[] | undefined;
+    if (Array.isArray(options)) list = options;
+
+    const answer = await this.provider.decide(state, question, list).catch((error: unknown) => this.recordFailure(error));
+    if (!answer) return new Decision(null, 0);
+    if (answer.probability <= APPROVAL_THRESHOLD) return new Decision(null, answer.probability);
+    if (answer.value === UNDECIDED) return new Decision(null, answer.probability);
+    return new Decision(answer.value, answer.probability);
+  }
+
+  private recordFailure(error: unknown): null {
+    debugLog('judge declined: %s', error);
+    Observability.getSpan()?.setAttribute('ai.telemetry.metadata.judgeError', String(error));
+    return null;
+  }
+}

package/src/ai/navigator.ts

@@ -18,9 +18,10 @@ import { createDebug, pluralize, tag } from '../utils/logger.js';
 import { loop, pause } from '../utils/loop.js';
 import { RulesLoader } from '../utils/rules-loader.ts';
 import { normalizeInlineText } from '../utils/strings.ts';
-import { extractStatePath, matchesNavigationUrl } from '../utils/url-matcher.js';
+import { extractStatePath, isSameHostFamily, matchesNavigationUrl } from '../utils/url-matcher.js';
 import type { Agent, AgentDeps } from './agent.js';
 import type { Conversation } from './conversation.js';
+import { type Decision, JUDGE_PAGE_CAP, type Judge, UNDECIDED } from './judge.ts';
 import type { Provider } from './provider.js';
 import { Researcher } from './researcher.ts';
 import { actionRule, locatorRule, unexpectedPopupRule } from './rules.js';
@@ -80,6 +81,7 @@ class Navigator implements Agent {
   private explorer: Explorer;
   private config: ExplorbotConfig;
   private stateManager: StateManager;
+  private judge?: Judge;
 
   constructor(deps: AgentDeps) {
     this.provider = deps.ai;
@@ -89,6 +91,7 @@ class Navigator implements Agent {
     this.knowledgeTracker = deps.knowledgeTracker;
     this.experienceTracker = deps.stateManager.getExperienceTracker();
     this.hooksRunner = new HooksRunner(deps.explorer, deps.config);
+    this.judge = deps.judge;
   }
 
   private get verifyAttempts(): number {
@@ -99,15 +102,6 @@ class Navigator implements Agent {
     return this.config.ai?.agents?.navigator?.verifyTimeout ?? 1500;
   }
 
-  private getBaseOrigin(): string | null {
-    const baseUrl = this.config.playwright.url;
-    try {
-      return new URL(baseUrl).origin;
-    } catch {
-      return null;
-    }
-  }
-
   private getComparableCurrentUrl(stateManager: any, expectedUrl: string): string {
     const currentState = stateManager.getCurrentState();
     if (!currentState) return '';
@@ -126,18 +120,12 @@ class Navigator implements Agent {
     const currentFullUrl = currentState.fullUrl || currentState.url || '';
     if (!currentFullUrl) return false;
 
-    try {
-      const currentOrigin = new URL(currentFullUrl).origin;
-      if (/^https?:\/\//i.test(expectedUrl)) {
-        return currentOrigin === new URL(expectedUrl).origin;
-      }
+    if (!/^https?:\/\//i.test(currentFullUrl)) return !/^https?:\/\//i.test(expectedUrl);
+    if (/^https?:\/\//i.test(expectedUrl)) return isSameHostFamily(currentFullUrl, expectedUrl);
 
-      const baseOrigin = this.getBaseOrigin();
-      if (!baseOrigin) return true;
-      return currentOrigin === baseOrigin;
-    } catch {
-      return !/^https?:\/\//i.test(expectedUrl);
-    }
+    const baseUrl = this.config.playwright.url;
+    if (!baseUrl) return true;
+    return isSameHostFamily(currentFullUrl, baseUrl);
   }
 
   private isOnExpectedPage(expectedUrl: string, stateManager: any): boolean {
@@ -325,8 +313,9 @@ class Navigator implements Agent {
               lastFailure = `Reached ${check.freshState.url} but the page state did not change`;
               tag('warning').log(`Page state did not change at ${check.freshState.url}`);
             } else {
-              lastFailure = `Reached ${check.freshState.url}, expected ${expectedUrl}`;
-              tag('warning').log(`URL verification failed: expected ${expectedUrl}, got ${check.freshState.url}`);
+              const reachedUrl = check.freshState.fullUrl || check.freshState.url;
+              lastFailure = `Reached ${reachedUrl}, expected ${expectedUrl}`;
+              tag('warning').log(`URL verification failed: expected ${expectedUrl}, got ${reachedUrl}`);
             }
             batchFailures.push({
               code: codeBlock,
@@ -584,7 +573,7 @@ class Navigator implements Agent {
     const countVisit = (value?: string | null) => {
       if (!value) return;
       const normalized = normalizeUrl(value);
-      if (normalized) visitCounts.set(normalized, (visitCounts.get(normalized) || 0) + 1);
+      visitCounts.set(normalized, (visitCounts.get(normalized) || 0) + 1);
     };
 
     for (const transition of history) {
@@ -595,7 +584,7 @@ class Navigator implements Agent {
     if (opts?.visitedUrls) {
       for (const url of opts.visitedUrls) {
         const normalized = normalizeUrl(url);
-        if (normalized && !visitCounts.has(normalized)) {
+        if (!visitCounts.has(normalized)) {
           visitCounts.set(normalized, 1);
         }
       }
@@ -696,7 +685,7 @@ class Navigator implements Agent {
     return suggestion;
   }
 
-  async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; inexpressible: boolean; results: AssertionResult[]; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number }> {
+  async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; inexpressible: boolean; results: AssertionResult[]; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number; judged?: Decision }> {
     tag('info').log('AI Navigator verifying state at', actionResult.url);
     debugLog('Verification message:', message);
 
@@ -706,6 +695,13 @@ class Navigator implements Agent {
       return { verified: cachedVerification, inexpressible: false, results: [], successfulCodes: [], assertionSteps: [], totalAttempted: 0 };
     }
 
+    const verifiedClaims = Object.keys(actionResult.verifications ?? {}).filter((claim) => actionResult.getVerification(claim) === true);
+    const same = await this.judge?.decide('Which already verified claim means the same as the claim under consideration?', [...verifiedClaims, UNDECIDED], { claim: message });
+    if (same?.approved) {
+      tag('operation').log(`Judge matched claim to an already verified one: "${same.value}"`);
+      return { verified: true, inexpressible: false, results: [], successfulCodes: [], assertionSteps: [], totalAttempted: 0 };
+    }
+
     const knowledge = this.knowledgeTracker.renderRelevantContext(actionResult);
     let experience = '';
 
@@ -846,7 +842,8 @@ class Navigator implements Agent {
     const inexpressible = !alreadyVerified && totalAttempted === 0;
     if (inexpressible) {
       tag('warning').log('No assertion could express this claim');
-      return { verified: false, inexpressible, results, successfulCodes, assertionSteps, totalAttempted };
+      const judged = await this.judge?.decide('The page shows that this claim is true.', null, { claim: message, page: actionResult.getCompactARIA().slice(0, JUDGE_PAGE_CAP) });
+      return { verified: false, inexpressible, results, successfulCodes, assertionSteps, totalAttempted, judged };
     }
 
     actionResult.addVerification(message, verified);

package/src/ai/pilot.ts

@@ -8,7 +8,7 @@ import type Explorer from '../explorer.ts';
 import type { PlaywrightRecorder } from '../playwright-recorder.ts';
 import type { StateManager } from '../state-manager.ts';
 import { Stats } from '../stats.ts';
-import { type Test, TestResult } from '../test-plan.ts';
+import { type Test, TestResult, TestStatus } from '../test-plan.ts';
 import { collectInteractiveNodes } from '../utils/aria.ts';
 import { ErrorPageError } from '../utils/error-page.ts';
 import { createDebug, tag } from '../utils/logger.ts';
@@ -19,6 +19,7 @@ import type { Agent, AgentDeps } from './agent.ts';
 import type { Conversation } from './conversation.ts';
 import type { Fisherman } from './fisherman.ts';
 import { createAskApiTool } from './fisherman/tools.ts';
+import { type Judge, UNDECIDED } from './judge.ts';
 import type { Navigator } from './navigator.ts';
 import type { Provider } from './provider.ts';
 import type { Researcher } from './researcher.ts';
@@ -33,6 +34,10 @@ const PILOT_REASONING_LIMIT = 500;
 const PILOT_MESSAGE_LIMIT = 2;
 const PILOT_MESSAGE_MAX_LENGTH = 160;
 const PILOT_REQUEST_LIMIT = 5;
+const OUTCOME_STATUS: Record<string, SettledStatus> = {
+  'The run shows this outcome happened.': 'passed',
+  'The run shows this outcome did not happen.': 'failed',
+};
 
 export class Pilot implements Agent {
   emoji = '🧭';
@@ -45,6 +50,7 @@ export class Pilot implements Agent {
   private requestStore: RequestStore;
   private playwrightRecorder: PlaywrightRecorder;
   private fisherman: Fisherman | null = null;
+  private judge?: Judge;
 
   constructor(deps: AgentDeps, agentTools: any, researcher: Researcher) {
     this.provider = deps.ai;
@@ -54,6 +60,7 @@ export class Pilot implements Agent {
     this.stateManager = deps.stateManager;
     this.requestStore = deps.requestStore;
     this.playwrightRecorder = deps.playwrightRecorder;
+    this.judge = deps.judge;
   }
 
   setFisherman(fisherman: Fisherman): void {
@@ -118,7 +125,9 @@ export class Pilot implements Agent {
     }
 
     const schema = z.object({
-      decision: z.enum(['pass', 'fail', 'continue', 'skipped']).describe('pass = test succeeded, fail = test failed, continue = tester should keep going, skipped = scenario is irrelevant OR systematic execution failures prevented testing'),
+      decision: z
+        .enum(['pass', 'fail', 'continue', 'skipped'])
+        .describe('pass = scenario goal accomplished, fail = the app misbehaved, continue = tester should keep going, skipped = the scenario cannot be judged against this app (its premise does not hold, it is irrelevant, or systematic execution failures prevented testing)'),
       reason: z.string().describe('Concise user-facing reason, maximum 1 short sentence and 120 characters. Do NOT repeat the decision status; explain only the evidence. For continue: explain why rejected and suggest alternatives.'),
       guidance: z.string().nullable().describe('Required for "continue": specific actionable instruction for the tester — what exactly to verify, retry differently, or complete next. Be concrete.'),
       requestVerification: z
@@ -407,9 +416,13 @@ export class Pilot implements Agent {
         DOM assertion can't be made.
         Do not pass when Tester achieved only a related navigation/filter/tab/status outcome instead of the
         requested action, workflow, or entity detail goal.
-      - "fail": goal not achieved and no further step toward it is available on the current page.
-      - "skipped": scenario is irrelevant to the app, OR systematic infrastructure failures (LLM errors,
-        crashes) prevented testing. NOT for "test failed to interact" — that's "fail" or "continue".
+      - "fail": the app misbehaved — the scenario's action ran against the right target and the app
+        produced a wrong, broken, or missing outcome. Not reaching the goal is not by itself a fail.
+      - "skipped": the scenario cannot be judged against this app — the page shows its premise does not
+        hold (the assumed constraint, field, or behaviour is designed differently), the target entity or
+        feature is not the one here, the scenario is irrelevant, OR systematic infrastructure failures
+        (LLM errors, crashes) prevented testing. NOT for "test failed to interact" — that's "fail" or
+        "continue".
       - "continue": goal incomplete but the control for the NEXT step is present on the current page, or a
         concrete missing check would change your verdict. Guidance must name that step.
         If a verify() asserted a state that was ALREADY TRUE before the test, it proves nothing — reject.
@@ -542,6 +555,9 @@ export class Pilot implements Agent {
     const actionsContext = this.formatActions(toolCalls);
     const stateContext = this.buildStateContext(currentState);
 
+    const healthy = await this.judge?.decide('The run is moving toward the goal and can continue without a supervisor reviewing it now.', null, { scenario: task.scenario, state: stateContext, recentActions: actionsContext });
+    if (healthy?.approved) return '';
+
     const hasFailures = toolCalls.length === 0 || toolCalls.some((t) => !t.wasSuccessful);
 
     const text = await this.sendToPilot(
@@ -586,7 +602,11 @@ export class Pilot implements Agent {
 
     let undecided = task.expected.filter((text) => !task.getCheckedExpectations().includes(text));
     if (image) undecided = task.expected;
-    if (!undecided.length) return task.expected.map((text) => ({ text, status: decided(text) }));
+
+    let settledByJudge = new Map<string, SettledStatus>();
+    if (!image) settledByJudge = await this.settleByJudge(task, undecided);
+    undecided = undecided.filter((text) => !settledByJudge.has(text));
+    if (!undecided.length) return task.expected.map((text) => ({ text, status: settledByJudge.get(text) || decided(text) }));
 
     const schema = z.object({
       outcomes: z.array(
@@ -659,6 +679,8 @@ export class Pilot implements Agent {
 
     const judged = new Map((response?.object?.outcomes || []).map((outcome: any) => [outcome.expectation, outcome]));
     return task.expected.map((text) => {
+      const byJudge = settledByJudge.get(text);
+      if (byJudge) return { text, status: byJudge };
       if (!undecided.includes(text)) return { text, status: decided(text) };
       const outcome = judged.get(text) as { status: SettledStatus; evidence?: string } | undefined;
       if (!outcome) return { text, status: 'unverified' as SettledStatus };
@@ -666,6 +688,22 @@ export class Pilot implements Agent {
     });
   }
 
+  private async settleByJudge(task: Test, expectations: string[]): Promise<Map<string, SettledStatus>> {
+    const settled = new Map<string, SettledStatus>();
+    const judge = this.judge;
+    if (!judge) return settled;
+
+    const state = { scenario: task.scenario, runLog: task.notesToString() || 'No steps recorded.' };
+    await Promise.all(
+      expectations.map(async (text) => {
+        const decision = await judge.decide(`What did this run establish about the expected outcome: ${text}`, [...Object.keys(OUTCOME_STATUS), UNDECIDED], state);
+        const status = OUTCOME_STATUS[decision.value ?? ''];
+        if (status) settled.set(text, status);
+      })
+    );
+    return settled;
+  }
+
   private formatExpectations(task: Test): string {
     const checked = task.getCheckedExpectations();
     const remaining = task.getRemainingExpectations();
@@ -684,6 +722,7 @@ export class Pilot implements Agent {
     this.conversation!.addUserText(finalUserText);
 
     const tools = { ...this.pickPlanningTools(), ...this.buildFishermanTools(opts.task) };
+    const preparedCount = opts.task.preparedData.length;
 
     const result = await this.provider.invokeConversation(this.conversation!, tools, {
       maxToolRoundtrips: opts.maxToolRoundtrips ?? 0,
@@ -692,7 +731,7 @@ export class Pilot implements Agent {
       stopWhen: () => opts.task.hasFinished,
       telemetry: { functionId },
     });
-    const text = result?.response?.text || '';
+    const text = this.announcePreparedData(result?.response?.text || '', opts.task, preparedCount);
     const learned = (result?.toolExecutions || []).filter((e: any) => e.toolName === 'learnExperience' && e.output?.content).map((e: any) => ({ url: e.output.url, content: e.output.content }));
     if (learned.length === 0) return text;
     opts.task.applyExperience(learned);
@@ -708,6 +747,24 @@ export class Pilot implements Agent {
     `;
   }
 
+  private announcePreparedData(text: string, task: Test, preparedCount: number): string {
+    const prepared = task.preparedData.slice(preparedCount);
+    if (prepared.length === 0) return text;
+
+    let refresh = '';
+    if (task.status === TestStatus.IN_PROGRESS) refresh = 'It was created after the page loaded, so the page does not show it yet. Run I.refreshPage() through form() before looking for it.';
+
+    return dedent`
+      ${text}
+
+      <prepared_data>
+      Pilot created this data through the API for this test. Use it instead of creating the same data through the UI:
+      ${prepared.map((item) => `- ${item}`).join('\n')}
+      ${refresh}
+      </prepared_data>
+    `;
+  }
+
   private getExperienceToc(): string {
     const state = this.stateManager.getCurrentState();
     if (!state) return '';
@@ -715,7 +772,7 @@ export class Pilot implements Agent {
   }
 
   private pickPlanningTools() {
-    const { see, context, verify, research, getVisitedStates, xpathCheck, learnExperience, askUser } = this.agentTools ?? {};
+    const { see, context, verify, research, getVisitedStates, xpathCheck, learnExperience, askUser, judge } = this.agentTools ?? {};
     const planning: Record<string, unknown> = {};
     if (see) planning.see = see;
     if (context) planning.context = context;
@@ -725,6 +782,7 @@ export class Pilot implements Agent {
     if (xpathCheck) planning.xpathCheck = xpathCheck;
     if (learnExperience) planning.learnExperience = learnExperience;
     if (askUser) planning.askUser = askUser;
+    if (judge) planning.judge = judge;
     withdrawVisionTools(planning);
     return planning;
   }
@@ -771,6 +829,7 @@ export class Pilot implements Agent {
           });
           const stepText = `Precondition: created ${items.join(', ')}`;
           task.addStep(stepText);
+          task.preparedData.push(...items);
           tag('success').log(stepText);
 
           return { noted: true, prepared: true, created: result.created };
@@ -1151,7 +1210,7 @@ export class Pilot implements Agent {
       Tester tools: click, pressKey, form, see, verify, interact, context, research, xpathCheck,
       visualClick, back, getVisitedStates, reset, stop, finish, record.
       Use tool names exactly as listed. Do not invent combined names or aliases.
-      Reloading is not a tool: to re-read a page from the server, instruct Tester to run I.reloadPage() through form.
+      Reloading is not a tool: to re-read a page from the server, instruct Tester to run I.refreshPage() through form.
 
       ${capabilityGroundingRule}