explorbot 0.4.10 → 0.5.0

package/docs/superpowers/specs/2026-09-18-judge-decision-model-design.md

@@ -0,0 +1,79 @@
+# Judge — a decision model tier — Design
+
+An optional *decision model* answers one narrow question with a probability over the answers offered. Explorbot uses it where it would otherwise guess: a code call site asks, and a confident answer lets the site skip its own, more expensive decision. The model is TypeSafe's Jev (a "System One" model), reached through OpenRouter or TypeSafe's own API.
+
+With `ai.decisionModel` unset, nothing registers and nothing calls out.
+
+## The interface
+
+```ts
+judge.decide(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision>
+
+class Decision {
+  readonly value: string | null;   // the winning option; 'yes' for an approved yes/no
+  readonly confidence: number;     // P(yes) for a yes/no, the chosen option's probability for a list
+  get approved(): boolean;         // value !== null
+  get rejected(): boolean;         // value === null
+}
+```
+
+- **An array is a categorization**, sent as a Choice. `value` is the chosen option. Including `UNDECIDED` lets the model say none fits.
+- **A boolean or `null` is a yes/no**, sent as a Noul, whose single number is P(yes). Only a confident yes approves.
+- **Approved means the answer's probability is above 70%** and it isn't `UNDECIDED`.
+- **Everything else is rejected**: a confident no, a low probability, `UNDECIDED`, a timeout, a failed request, a list with fewer than two options, or the direct path being disabled. `decide` never throws and never returns `null`.
+
+The threshold and every failure mode live in one place. A call site only ever sees a confident decision or a rejection, so it cannot misread uncertainty.
+
+### The one rule for call sites
+
+**`rejected` means "not approved", never "confidently no".** It absorbs uncertainty, so a question must be phrased so that `approved` is the action the site would take, and `rejected` falls through to today's behaviour. Then `if (decision?.approved)` is correct by construction.
+
+`consult(...)` is the same call without the direct-path gate, used by the tool.
+
+## Configuration
+
+```js
+ai: { decisionModel: { provider: 'openrouter', model: 'typesafe/jev-1.13' } }             // via OpenRouter
+ai: { decisionModel: { provider: 'typesafe', model: 'jev-latest' } }                       // TypeSafe API directly
+ai: { decisionModel: { provider: 'openrouter', model: 'typesafe/jev-1.13', tool: false } } // direct sites only
+```
+
+`config.ts` holds only the field type. An unknown provider or a missing API key throws at startup, naming what to fix.
+
+Everything transport-specific lives in `src/ai/judge-provider.ts`: `endpointFor()` (a `switch` over `openrouter` and `typesafe`, each with its endpoint and API-key variable), the HTTP call, timeout, and the Noul/Choice wire format. That file is temporary: when the Vercel AI SDK supports decision models it is deleted and `decisionModel` becomes a regular provider-built model.
+
+`Judge.fromConfig()` builds the judge; it reaches agents through `AgentDeps` and tools through `ToolDeps`.
+
+## Where it is used
+
+| Site | Question | Approved means |
+|---|---|---|
+| `failedToolResult`, multi-element branch | Which listed element does the intent name? | suggest that element by `elementIndex` |
+| `Pilot.analyzeProgress` | The run is moving toward the goal and can continue without a supervisor now. | skip the review silently |
+| `Pilot.settleExpectations`, text-only path | What did this run establish about the expected outcome? | settle it; the rest go to the agentic model |
+| `Navigator.verifyState`, before the prompt | Which already verified claim means the same as this one? | treat it as verified, skip the HTML-bearing prompt |
+| `Navigator.verifyState`, inexpressible branch | The page shows that this claim is true. | report it as a judgement, not an assertion |
+| Prima `go()`, semantic target | Which listed control leads to the target? / The page now shows the target. | click the ref, confirm arrival, skip the navigator |
+| `judge` tool (Tester and Pilot) | whatever the model asks | the answer; otherwise "not confirmed" |
+
+Invariants the sites keep:
+
+- **A judge answer never enters `verifications`.** A dedup match skips the work but writes no cache entry for the new claim, because a cache entry stands in for a proof.
+- **Prima confirms arrival** before returning a success envelope. `cli.ts` exits on `envelope.ok`.
+- **The tool assembles its own state**: scenario, compact ARIA capped at `JUDGE_PAGE_CAP`, and recent steps. The model supplies only the question.
+
+## Deliberately not used
+
+- **Pilot's verdict.** A generic "the app never held the state the scenario assumes" would fire on boundary give-ups, like "previous page" on page 1 when `»` is listed, and turn an executable scenario into a clean-looking skip. That hides the execution gap. The fix for a boundary give-up belongs in planning, not the verdict.
+- **Experience filtering.** It removes no model call, and one question per stored block on every prompt build costs more than it saves.
+- **Generation, screenshots, tool-calling loops.** The model is text-only and does not generate.
+
+## Measurements behind the threshold
+
+On states reconstructed from recorded traces:
+
+- A literal, concrete phrasing beat an abstract one on identical state: 0.88 versus 0.58. Questions should name what they expect to see.
+- A composite "continue or call Pilot" Choice tied at confidence 0.08 on a run that was progressing. Under `rejected` semantics, a tie falls through to a review, which is the safe direction.
+- The first live probe picked `copy` over `root` at 0.48 versus 0.41 on a dialog whose Copy button was disabled. A 70% bar rejects that, correctly.
+
+Threshold and phrasing should be tuned from traces: every call records `question`, `value` and `confidence` on its `judge.decide` span, and failures record their reason.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "explorbot",
-  "version": "0.4.10",
+  "version": "0.5.0",
   "description": "CLI app built with React Ink, CodeceptJS, and Playwright",
   "license": "Elastic-2.0",
   "type": "module",
@@ -60,7 +60,8 @@
     "check": "biome check .",
     "check:fix": "biome check --write .",
     "langfuse:export": "bun run .claude/skills/explorbot-debug/langfuse-export.ts",
-    "build:prima": "bun run scripts/build-prima-npm.ts"
+    "build:prima": "bun run scripts/build-prima-npm.ts",
+    "build:mdq": "bun run scripts/build-mdq-npm.ts"
   },
   "keywords": [
     "cli",

package/src/ai/agent.ts

@@ -4,6 +4,7 @@ import type Explorer from '../explorer.ts';
 import type { KnowledgeTracker } from '../knowledge-tracker.ts';
 import type { PlaywrightRecorder } from '../playwright-recorder.ts';
 import type { StateManager } from '../state-manager.ts';
+import type { Judge } from './judge.ts';
 import type { AIProvider } from './provider.ts';
 
 export interface Agent {
@@ -18,6 +19,7 @@ export interface AgentDeps {
   knowledgeTracker: KnowledgeTracker;
   requestStore: RequestStore;
   playwrightRecorder: PlaywrightRecorder;
+  judge?: Judge;
 }
 
-export type ToolDeps = Pick<AgentDeps, 'explorer' | 'stateManager' | 'ai'>;
+export type ToolDeps = Pick<AgentDeps, 'explorer' | 'stateManager' | 'ai' | 'judge'>;

package/src/ai/judge-provider.ts

@@ -0,0 +1,62 @@
+const REQUEST_TIMEOUT_MS = 15000;
+
+export class JudgeProvider {
+  private fetchImpl: typeof fetch = fetch;
+  private requestTimeoutMs = REQUEST_TIMEOUT_MS;
+  private url: string;
+  private apiKey: string;
+
+  constructor(
+    provider: string,
+    readonly model: string
+  ) {
+    const { url, keyName } = JudgeProvider.endpointFor(provider);
+    const apiKey = process.env[keyName];
+    if (!apiKey) throw new Error(`Set ${keyName} to use the decision model`);
+    this.url = url;
+    this.apiKey = apiKey;
+  }
+
+  async decide(state: unknown, question: string, options?: string[]): Promise<ProviderDecision> {
+    let q: Record<string, unknown> = { type: 'noul', instructions: question };
+    if (options) q = { type: 'choice', instructions: question, criteria: Object.fromEntries(options.map((option, index) => [String(index), option])) };
+    const body = JSON.stringify({ model: this.model, state, questions: { q } });
+    const controller = new AbortController();
+    const timer = setTimeout(() => controller.abort(), this.requestTimeoutMs);
+    try {
+      const response = await this.fetchImpl(this.url, {
+        method: 'POST',
+        headers: { Authorization: `Bearer ${this.apiKey}`, 'Content-Type': 'application/json' },
+        body,
+        signal: controller.signal,
+      });
+      if (!response.ok) throw new Error(`http_${response.status}`);
+
+      const answer = (await response.json())?.answers?.q;
+      if (!options && typeof answer?.noul === 'number') return { value: 'yes', probability: answer.noul };
+
+      const value = options?.[Number(answer?.choice)];
+      const probability = answer?.probabilities?.[answer?.choice];
+      if (value === undefined || typeof probability !== 'number') throw new Error('malformed_body');
+      return { value, probability };
+    } finally {
+      clearTimeout(timer);
+    }
+  }
+
+  private static endpointFor(provider: string): { url: string; keyName: string } {
+    switch (provider) {
+      case 'openrouter':
+        return { url: 'https://openrouter.ai/api/alpha/decisions', keyName: 'OPENROUTER_API_KEY' };
+      case 'typesafe':
+        return { url: 'https://api.typesafe.ai/v1/systemone', keyName: 'TYPESAFE_API_KEY' };
+      default:
+        throw new Error(`Unknown decision model provider "${provider}" — use "openrouter" or "typesafe"`);
+    }
+  }
+}
+
+export interface ProviderDecision {
+  value: string;
+  probability: number;
+}

package/src/ai/judge-tool.ts

@@ -0,0 +1,35 @@
+import { tool } from 'ai';
+import dedent from 'dedent';
+import { z } from 'zod';
+import type { ToolDeps } from './agent.ts';
+import { failedToolResult, successToolResult } from './tools.ts';
+
+export function createJudgeTool(deps: ToolDeps, buildState: () => Promise<Record<string, unknown>>): Record<string, any> {
+  const judge = deps.judge;
+  if (!judge?.toolEnabled) return {};
+
+  return {
+    judge: tool({
+      description: dedent`
+        Settle one judgement about the current page instead of guessing. Phrase it literally and concretely.
+      `,
+      inputSchema: z.object({
+        question: z.string().describe('The statement to confirm, or the question the options answer'),
+        options: z.array(z.string()).optional().describe('Possible answers. Omit to confirm a statement'),
+        context: z.string().optional().describe('Anything the page observation does not already carry'),
+      }),
+      execute: async ({ question, options, context }) => {
+        const state = await buildState();
+        if (context) state.context = context;
+
+        const decision = await judge.consult(question, options ?? null, state);
+        if (decision.rejected) {
+          return failedToolResult('judge', `Not confirmed: ${question}`, {
+            suggestion: 'The page does not settle this. Gather more context or take another route; do not assume either answer.',
+          });
+        }
+        return successToolResult('judge', { question, answer: decision.value, confidence: decision.confidence });
+      },
+    }),
+  };
+}

package/src/ai/judge.ts

@@ -0,0 +1,75 @@
+import { clearActivity, setActivity } from '../activity.ts';
+import type { AIConfig } from '../config.ts';
+import { Observability } from '../observability.ts';
+import { createDebug } from '../utils/logger.ts';
+import { JudgeProvider } from './judge-provider.ts';
+
+const debugLog = createDebug('explorbot:judge');
+
+const APPROVAL_THRESHOLD = 0.7;
+
+export const UNDECIDED = 'undecided';
+export const JUDGE_PAGE_CAP = 12000;
+
+export class Decision {
+  constructor(
+    readonly value: string | null,
+    readonly confidence: number
+  ) {}
+
+  get approved(): boolean {
+    return this.value !== null;
+  }
+
+  get rejected(): boolean {
+    return this.value === null;
+  }
+}
+
+export class Judge {
+  constructor(
+    private provider: JudgeProvider,
+    private enabled: { tool: boolean; direct: boolean }
+  ) {}
+
+  static fromConfig(config: AIConfig['decisionModel']): Judge | null {
+    if (!config) return null;
+    return new Judge(new JudgeProvider(config.provider, config.model), { tool: config.tool !== false, direct: config.direct !== false });
+  }
+
+  get toolEnabled(): boolean {
+    return this.enabled.tool;
+  }
+
+  async decide(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision> {
+    if (!this.enabled.direct) return new Decision(null, 0);
+    return this.consult(question, options, state);
+  }
+
+  async consult(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision> {
+    if (Array.isArray(options) && options.length < 2) return new Decision(null, 0);
+    return Observability.run('judge.decide', { tags: ['judge'] }, async () => {
+      setActivity('⚖️ Asking judge...', 'ai');
+      const decision = await this.request(question, options, state).finally(() => clearActivity());
+      Observability.getSpan()?.setAttribute('ai.telemetry.metadata.judgeDecision', JSON.stringify({ question, value: decision.value, confidence: decision.confidence }));
+      return decision;
+    });
+  }
+
+  private async request(question: string, options: string[] | boolean | null, state: unknown): Promise<Decision> {
+    let list: string[] | undefined;
+    if (Array.isArray(options)) list = options;
+
+    const answer = await this.provider.decide(state, question, list).catch((error: unknown) => this.recordFailure(error));
+    if (!answer) return new Decision(null, 0);
+    if (answer.probability <= APPROVAL_THRESHOLD) return new Decision(null, answer.probability);
+    if (answer.value === UNDECIDED) return new Decision(null, answer.probability);
+    return new Decision(answer.value, answer.probability);
+  }
+
+  private recordFailure(error: unknown): null {
+    debugLog('judge declined: %s', error);
+    Observability.getSpan()?.setAttribute('ai.telemetry.metadata.judgeError', String(error));
+    return null;
+  }
+}

package/src/ai/navigator.ts

@@ -21,6 +21,7 @@ import { normalizeInlineText } from '../utils/strings.ts';
 import { extractStatePath, isSameHostFamily, matchesNavigationUrl } from '../utils/url-matcher.js';
 import type { Agent, AgentDeps } from './agent.js';
 import type { Conversation } from './conversation.js';
+import { type Decision, JUDGE_PAGE_CAP, type Judge, UNDECIDED } from './judge.ts';
 import type { Provider } from './provider.js';
 import { Researcher } from './researcher.ts';
 import { actionRule, locatorRule, unexpectedPopupRule } from './rules.js';
@@ -80,6 +81,7 @@ class Navigator implements Agent {
   private explorer: Explorer;
   private config: ExplorbotConfig;
   private stateManager: StateManager;
+  private judge?: Judge;
 
   constructor(deps: AgentDeps) {
     this.provider = deps.ai;
@@ -89,6 +91,7 @@ class Navigator implements Agent {
     this.knowledgeTracker = deps.knowledgeTracker;
     this.experienceTracker = deps.stateManager.getExperienceTracker();
     this.hooksRunner = new HooksRunner(deps.explorer, deps.config);
+    this.judge = deps.judge;
   }
 
   private get verifyAttempts(): number {
@@ -570,7 +573,7 @@ class Navigator implements Agent {
     const countVisit = (value?: string | null) => {
       if (!value) return;
       const normalized = normalizeUrl(value);
-      if (normalized) visitCounts.set(normalized, (visitCounts.get(normalized) || 0) + 1);
+      visitCounts.set(normalized, (visitCounts.get(normalized) || 0) + 1);
     };
 
     for (const transition of history) {
@@ -581,7 +584,7 @@ class Navigator implements Agent {
     if (opts?.visitedUrls) {
       for (const url of opts.visitedUrls) {
         const normalized = normalizeUrl(url);
-        if (normalized && !visitCounts.has(normalized)) {
+        if (!visitCounts.has(normalized)) {
           visitCounts.set(normalized, 1);
         }
       }
@@ -682,7 +685,7 @@ class Navigator implements Agent {
     return suggestion;
   }
 
-  async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; inexpressible: boolean; results: AssertionResult[]; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number }> {
+  async verifyState(message: string, actionResult: ActionResult): Promise<{ verified: boolean; inexpressible: boolean; results: AssertionResult[]; successfulCodes: string[]; assertionSteps: Array<{ name: string; args: any[] }>; totalAttempted: number; judged?: Decision }> {
     tag('info').log('AI Navigator verifying state at', actionResult.url);
     debugLog('Verification message:', message);
 
@@ -692,6 +695,13 @@ class Navigator implements Agent {
       return { verified: cachedVerification, inexpressible: false, results: [], successfulCodes: [], assertionSteps: [], totalAttempted: 0 };
     }
 
+    const verifiedClaims = Object.keys(actionResult.verifications ?? {}).filter((claim) => actionResult.getVerification(claim) === true);
+    const same = await this.judge?.decide('Which already verified claim means the same as the claim under consideration?', [...verifiedClaims, UNDECIDED], { claim: message });
+    if (same?.approved) {
+      tag('operation').log(`Judge matched claim to an already verified one: "${same.value}"`);
+      return { verified: true, inexpressible: false, results: [], successfulCodes: [], assertionSteps: [], totalAttempted: 0 };
+    }
+
     const knowledge = this.knowledgeTracker.renderRelevantContext(actionResult);
     let experience = '';
 
@@ -832,7 +842,8 @@ class Navigator implements Agent {
     const inexpressible = !alreadyVerified && totalAttempted === 0;
     if (inexpressible) {
       tag('warning').log('No assertion could express this claim');
-      return { verified: false, inexpressible, results, successfulCodes, assertionSteps, totalAttempted };
+      const judged = await this.judge?.decide('The page shows that this claim is true.', null, { claim: message, page: actionResult.getCompactARIA().slice(0, JUDGE_PAGE_CAP) });
+      return { verified: false, inexpressible, results, successfulCodes, assertionSteps, totalAttempted, judged };
     }
 
     actionResult.addVerification(message, verified);

package/src/ai/pilot.ts

@@ -8,7 +8,7 @@ import type Explorer from '../explorer.ts';
 import type { PlaywrightRecorder } from '../playwright-recorder.ts';
 import type { StateManager } from '../state-manager.ts';
 import { Stats } from '../stats.ts';
-import { type Test, TestResult } from '../test-plan.ts';
+import { type Test, TestResult, TestStatus } from '../test-plan.ts';
 import { collectInteractiveNodes } from '../utils/aria.ts';
 import { ErrorPageError } from '../utils/error-page.ts';
 import { createDebug, tag } from '../utils/logger.ts';
@@ -19,6 +19,7 @@ import type { Agent, AgentDeps } from './agent.ts';
 import type { Conversation } from './conversation.ts';
 import type { Fisherman } from './fisherman.ts';
 import { createAskApiTool } from './fisherman/tools.ts';
+import { type Judge, UNDECIDED } from './judge.ts';
 import type { Navigator } from './navigator.ts';
 import type { Provider } from './provider.ts';
 import type { Researcher } from './researcher.ts';
@@ -33,6 +34,10 @@ const PILOT_REASONING_LIMIT = 500;
 const PILOT_MESSAGE_LIMIT = 2;
 const PILOT_MESSAGE_MAX_LENGTH = 160;
 const PILOT_REQUEST_LIMIT = 5;
+const OUTCOME_STATUS: Record<string, SettledStatus> = {
+  'The run shows this outcome happened.': 'passed',
+  'The run shows this outcome did not happen.': 'failed',
+};
 
 export class Pilot implements Agent {
   emoji = '🧭';
@@ -45,6 +50,7 @@ export class Pilot implements Agent {
   private requestStore: RequestStore;
   private playwrightRecorder: PlaywrightRecorder;
   private fisherman: Fisherman | null = null;
+  private judge?: Judge;
 
   constructor(deps: AgentDeps, agentTools: any, researcher: Researcher) {
     this.provider = deps.ai;
@@ -54,6 +60,7 @@ export class Pilot implements Agent {
     this.stateManager = deps.stateManager;
     this.requestStore = deps.requestStore;
     this.playwrightRecorder = deps.playwrightRecorder;
+    this.judge = deps.judge;
   }
 
   setFisherman(fisherman: Fisherman): void {
@@ -548,6 +555,9 @@ export class Pilot implements Agent {
     const actionsContext = this.formatActions(toolCalls);
     const stateContext = this.buildStateContext(currentState);
 
+    const healthy = await this.judge?.decide('The run is moving toward the goal and can continue without a supervisor reviewing it now.', null, { scenario: task.scenario, state: stateContext, recentActions: actionsContext });
+    if (healthy?.approved) return '';
+
     const hasFailures = toolCalls.length === 0 || toolCalls.some((t) => !t.wasSuccessful);
 
     const text = await this.sendToPilot(
@@ -592,7 +602,11 @@ export class Pilot implements Agent {
 
     let undecided = task.expected.filter((text) => !task.getCheckedExpectations().includes(text));
     if (image) undecided = task.expected;
-    if (!undecided.length) return task.expected.map((text) => ({ text, status: decided(text) }));
+
+    let settledByJudge = new Map<string, SettledStatus>();
+    if (!image) settledByJudge = await this.settleByJudge(task, undecided);
+    undecided = undecided.filter((text) => !settledByJudge.has(text));
+    if (!undecided.length) return task.expected.map((text) => ({ text, status: settledByJudge.get(text) || decided(text) }));
 
     const schema = z.object({
       outcomes: z.array(
@@ -665,6 +679,8 @@ export class Pilot implements Agent {
 
     const judged = new Map((response?.object?.outcomes || []).map((outcome: any) => [outcome.expectation, outcome]));
     return task.expected.map((text) => {
+      const byJudge = settledByJudge.get(text);
+      if (byJudge) return { text, status: byJudge };
       if (!undecided.includes(text)) return { text, status: decided(text) };
       const outcome = judged.get(text) as { status: SettledStatus; evidence?: string } | undefined;
       if (!outcome) return { text, status: 'unverified' as SettledStatus };
@@ -672,6 +688,22 @@ export class Pilot implements Agent {
     });
   }
 
+  private async settleByJudge(task: Test, expectations: string[]): Promise<Map<string, SettledStatus>> {
+    const settled = new Map<string, SettledStatus>();
+    const judge = this.judge;
+    if (!judge) return settled;
+
+    const state = { scenario: task.scenario, runLog: task.notesToString() || 'No steps recorded.' };
+    await Promise.all(
+      expectations.map(async (text) => {
+        const decision = await judge.decide(`What did this run establish about the expected outcome: ${text}`, [...Object.keys(OUTCOME_STATUS), UNDECIDED], state);
+        const status = OUTCOME_STATUS[decision.value ?? ''];
+        if (status) settled.set(text, status);
+      })
+    );
+    return settled;
+  }
+
   private formatExpectations(task: Test): string {
     const checked = task.getCheckedExpectations();
     const remaining = task.getRemainingExpectations();
@@ -690,6 +722,7 @@ export class Pilot implements Agent {
     this.conversation!.addUserText(finalUserText);
 
     const tools = { ...this.pickPlanningTools(), ...this.buildFishermanTools(opts.task) };
+    const preparedCount = opts.task.preparedData.length;
 
     const result = await this.provider.invokeConversation(this.conversation!, tools, {
       maxToolRoundtrips: opts.maxToolRoundtrips ?? 0,
@@ -698,7 +731,7 @@ export class Pilot implements Agent {
       stopWhen: () => opts.task.hasFinished,
       telemetry: { functionId },
     });
-    const text = result?.response?.text || '';
+    const text = this.announcePreparedData(result?.response?.text || '', opts.task, preparedCount);
     const learned = (result?.toolExecutions || []).filter((e: any) => e.toolName === 'learnExperience' && e.output?.content).map((e: any) => ({ url: e.output.url, content: e.output.content }));
     if (learned.length === 0) return text;
     opts.task.applyExperience(learned);
@@ -714,6 +747,24 @@ export class Pilot implements Agent {
     `;
   }
 
+  private announcePreparedData(text: string, task: Test, preparedCount: number): string {
+    const prepared = task.preparedData.slice(preparedCount);
+    if (prepared.length === 0) return text;
+
+    let refresh = '';
+    if (task.status === TestStatus.IN_PROGRESS) refresh = 'It was created after the page loaded, so the page does not show it yet. Run I.refreshPage() through form() before looking for it.';
+
+    return dedent`
+      ${text}
+
+      <prepared_data>
+      Pilot created this data through the API for this test. Use it instead of creating the same data through the UI:
+      ${prepared.map((item) => `- ${item}`).join('\n')}
+      ${refresh}
+      </prepared_data>
+    `;
+  }
+
   private getExperienceToc(): string {
     const state = this.stateManager.getCurrentState();
     if (!state) return '';
@@ -721,7 +772,7 @@ export class Pilot implements Agent {
   }
 
   private pickPlanningTools() {
-    const { see, context, verify, research, getVisitedStates, xpathCheck, learnExperience, askUser } = this.agentTools ?? {};
+    const { see, context, verify, research, getVisitedStates, xpathCheck, learnExperience, askUser, judge } = this.agentTools ?? {};
     const planning: Record<string, unknown> = {};
     if (see) planning.see = see;
     if (context) planning.context = context;
@@ -731,6 +782,7 @@ export class Pilot implements Agent {
     if (xpathCheck) planning.xpathCheck = xpathCheck;
     if (learnExperience) planning.learnExperience = learnExperience;
     if (askUser) planning.askUser = askUser;
+    if (judge) planning.judge = judge;
     withdrawVisionTools(planning);
     return planning;
   }
@@ -777,6 +829,7 @@ export class Pilot implements Agent {
           });
           const stepText = `Precondition: created ${items.join(', ')}`;
           task.addStep(stepText);
+          task.preparedData.push(...items);
           tag('success').log(stepText);
 
           return { noted: true, prepared: true, created: result.created };
@@ -1157,7 +1210,7 @@ export class Pilot implements Agent {
       Tester tools: click, pressKey, form, see, verify, interact, context, research, xpathCheck,
       visualClick, back, getVisitedStates, reset, stop, finish, record.
       Use tool names exactly as listed. Do not invent combined names or aliases.
-      Reloading is not a tool: to re-read a page from the server, instruct Tester to run I.reloadPage() through form.
+      Reloading is not a tool: to re-read a page from the server, instruct Tester to run I.refreshPage() through form.
 
       ${capabilityGroundingRule}
 

package/src/ai/planner.ts

@@ -301,13 +301,14 @@ export class Planner extends PlannerBase implements Agent {
         .replaceEach((section) => {
           const heading = section.query('heading').text().trim();
           const withoutHeadings = mdq(section.text()).query('heading').replace('');
-          const body = mdq(withoutHeadings).query('hr').replace('').trim();
+          const body = mdq(withoutHeadings).query('hr').replace('').toString().trim();
           if (body && !seenTitles.has(heading)) {
             seenTitles.add(heading);
             return section.text();
           }
           return '';
-        });
+        })
+        .toString();
     }
 
     const trimmedTitles = new Set<string>();
@@ -319,10 +320,11 @@ export class Planner extends PlannerBase implements Agent {
           if (trimmedTitles.has(heading)) return section.text();
           const count = section.query('blockquote').count();
           if (count <= 10) return section.text();
-          const kept = mdq(section.text()).query('blockquote[10:]').replace('');
+          const kept = mdq(section.text()).query('blockquote[10:]').replace('').toString();
           trimmedTitles.add(heading);
           return `${kept.trimEnd()}\n> ... and ${count - 10} more discoveries\n`;
-        });
+        })
+        .toString();
     }
 
     return result.trim() || null;
@@ -404,7 +406,7 @@ export class Planner extends PlannerBase implements Agent {
     if (this.scout && this.docsWeight > 0) {
       docsPromise = this.scout.collectDocs({ url: state.url, title: state.title, feature, excludeUrls: this.knowledgeTracker.applicationSpecUrls(state) });
     }
-    let plannerResearch = mdq(research).query('code').replace('');
+    let plannerResearch = mdq(research).query('code').replace('').toString();
     plannerResearch = mdq(plannerResearch)
       .query('table')
       .replaceEach((table) => {
@@ -415,7 +417,8 @@ export class Planner extends PlannerBase implements Agent {
           Type: r.Type || '',
         }));
         return jsonToTable(elementWithType, ['Element', 'Type']);
-      });
+      })
+      .toString();
 
     const hasFocusedOverlay = hasFocusedSection(plannerResearch);
     const focusNote = hasFocusedOverlay ? "IMPORTANT: One section is marked as **Focused** — this is the user's current focus area. Concentrate testing on the Focused section FIRST — test all interactions inside it before planning tests for the rest of the page." : '';