explorbot 0.4.3 → 0.4.5

package/docs/reference/commands.md

@@ -118,6 +118,7 @@ EXPLORBOT_AI_PROVIDER=openrouter \
 | `EXPLORBOT_KNOWLEDGE_FILE` | Path to a knowledge markdown file |
 | `EXPLORBOT_SPEC` | Docbot application spec directory or index.md, used as page knowledge |
 | `EXPLORBOT_API_SPEC` | OpenAPI spec path for the API boat |
+| `EXPLORBOT_API_HEADERS` | Headers sent with every API request, one "Name: value" per line |
 | `EXPLORBOT_NO_BANNER` | Suppress the startup banner, for machine-readable output |
 | `EXPLORBOT_MAX_DURATION` | Wall-clock budget in minutes for an explore run; same as --max-duration |
 <!-- END env -->

package/docs/reference/configuration.md

@@ -179,6 +179,7 @@ Each agent takes its own model and system prompt.
 | `rerunner` | Heals failing steps when re-running generated tests |
 | `analyst` | Writes the end-of-session markdown report |
 | `fisherman` | Prepares test data through API requests |
+| `scout` | Retrieves relevant documentation for the Planner |
 | `chief` | API test planning |
 | `curler` | API test execution |
 
@@ -206,7 +207,7 @@ agents: {
 | `beforeHook` | `Hook \| HookPatternMap` | Code to run before agent execution |
 | `afterHook` | `Hook \| HookPatternMap` | Code to run after agent execution |
 
-Some agents take extra options: `pilot` accepts `stepsToReview` (recent steps reviewed per check, default 5); `planner` accepts `styles` (see [Planning styles](#planning-styles)); `rerunner` accepts `healLimit` (max heal attempts, default 3) and `recipes` (custom heal recipes, see [Rerunning Tests](../web-testing/rerun.md)). Researcher and Historian options are documented below.
+Some agents take extra options: `pilot` accepts `stepsToReview` (recent steps reviewed per check, default 5); `planner` accepts `styles` (see [Planning styles](#planning-styles)) and `docsWeight` (share of scenarios grounded in documentation when Scout is enabled, default 70); `scout` accepts `dirs` (see [Scout agent](#scout-agent)); `rerunner` accepts `healLimit` (max heal attempts, default 3) and `recipes` (custom heal recipes, see [Rerunning Tests](../web-testing/rerun.md)). Researcher and Historian options are documented below.
 
 See [Agent hooks](../web-testing/hooks.md) for hook configuration.
 
@@ -257,6 +258,31 @@ See [AI providers](../basics/providers.md) for recommended models and provider s
 
 Fisherman prepares test data over the API before a scenario runs, and can also answer questions about data that already exists without creating or changing anything. Pilot reaches this read-only capability through its `askApi(question)` tool, calling it to check whether suitable data is already there — or to get the exact name or id of an existing record — before deciding whether to create anything through `precondition()`. In replicate mode, where Fisherman learns the API by watching browser traffic instead of reading a spec, the read endpoints it can query come from successful GET requests observed in the browser, alongside the write endpoints already captured from XHR traffic. The endpoint list shown to the model names only the path and its query-parameter names, never their values; the underlying capture on disk holds the full request URL and headers — what write captures already hold — but no response body.
 
+### Scout agent
+
+Scout retrieves documentation relevant to the page being planned and hands it to the Planner as a `<docs_context>` block, so scenarios can be grounded in what the application documents say. It is opt-in and needs documentation collected beforehand:
+
+```javascript
+ai: {
+  agents: {
+    scout: {
+      enabled: true,          // Opt in — Scout never runs without this
+      dirs: ['docs'],         // Extra markdown directories to search, beyond the spec bundle
+    },
+    planner: {
+      docsWeight: 70,         // Roughly 70% of scenarios exercise documented behavior, the rest explore beyond it
+    },
+  },
+},
+```
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `enabled` | `boolean` | Turn Scout on. Default: off. |
+| `dirs` | `string[]` | Markdown directories added to the corpus, resolved relative to the project |
+
+The corpus combines the [application spec](../workflow/application-spec.md) bundle (`--spec` / `EXPLORBOT_SPEC` / `dirs.spec`, set by `explorbot docs collect`) with the `dirs` above. Scout scans it with the same `bash` + `readFile` tools Captain uses: the corpus is loaded into an in-memory sandbox and the model itself runs `rg` (or `grep` — whichever is installed) to explore it. One of the two must be on PATH — Scout fails loudly when neither is found. Pages already injected for the current URL as `<application_spec>` are excluded from the Scout corpus, so the two blocks never duplicate each other. Files under `dirs` that carry no page URL are listed as hand-written notes for Scout to inspect when they are relevant.
+
 ## Playwright settings
 
 ### Browser selection
@@ -512,6 +538,7 @@ export default {
       quartermaster: { /* ... */ },
       historian: { /* ... */ },
       fisherman: { /* ... */ },
+      scout: { enabled: true, dirs: ['docs'] }, // Documentation retrieval for the Planner
       rerunner: { /* ... */ },
       analyst: { /* ... */ },
     },

package/docs/web-testing/agents.md

@@ -44,7 +44,7 @@ See [Researcher Agent](./researcher.md) for configuration and usage.
 
 Generates test scenarios from research findings.
 
-The Planner writes business-focused scenarios with priority levels (critical/important/high/normal/low) and expected outcomes for verification. It balances positive and negative cases, skips scenarios you already have, and cycles through planning styles (normal, psycho, curious) to broaden coverage across iterations. You can add your own styles and page-specific rules.
+The Planner writes business-focused scenarios with priority levels (critical/important/high/normal/low) and expected outcomes for verification. It balances positive and negative cases, skips scenarios you already have, and cycles through planning styles (normal, psycho, curious) to broaden coverage across iterations. You can add your own styles and page-specific rules. With the Scout agent enabled, it also plans from collected documentation, weighted by `docsWeight`.
 
 Commands that use Planner:
 - `/plan [--focus <feature>]`
@@ -52,6 +52,14 @@ Commands that use Planner:
 
 See [Planner Agent](./planner.md) for planning styles, customization, and configuration.
 
+## Scout Agent
+
+Retrieves documentation relevant to the page being planned.
+
+Scout searches the collected documentation corpus — the [application spec](../workflow/application-spec.md) from `explorbot docs collect` plus any extra markdown directories you configure — and reports the documented capabilities, states and transitions that matter for the current page and focus. The Planner receives them as a `<docs_context>` block and grounds part of its scenarios in them. Pages already injected as `<application_spec>` for the current URL are not repeated. Scout is opt-in (`ai.agents.scout.enabled`) and searches with ripgrep or grep, falling back to an in-process scan when neither is installed.
+
+See [Configuration: Scout agent](../reference/configuration.md#scout-agent).
+
 ## Tester Agent
 
 Runs the planned scenarios.

package/docs/web-testing/planner.md

@@ -53,6 +53,11 @@ ai: {
 | `styles` | `string[]` | `['normal', 'curious', 'psycho']` | Style names and cycling order |
 | `rules` | `RuleEntry[]` | `[]` | URL-aware rule files from `rules/planner/` |
 | `systemPrompt` | `string` | - | Inline instructions appended to the prompt |
+| `docsWeight` | `number` | `70` | With Scout enabled, the rough share of scenarios exercising documented behavior; the rest explore beyond the documentation |
+
+## Planning from documentation
+
+With the [Scout agent](../reference/configuration.md#scout-agent) enabled, the Planner also receives a `<docs_context>` block — capabilities, states and transitions retrieved from collected documentation that are relevant to the current page and focus. `docsWeight` steers the mix: at `70` roughly seven of ten scenarios exercise documented behavior and three explore what the documentation does not cover. Set it to `100` for documentation-only planning, or lower it to lean on the Planner's own reading of the page. Pages whose documentation is already injected as `<application_spec>` are not repeated in `<docs_context>`.
 
 ## Planning Styles
 

package/docs/workflow/agentic-usage.md

@@ -58,6 +58,7 @@ No `init`, no config file, no project directory, no model IDs to look up. These
 | `EXPLORBOT_KNOWLEDGE_FILE` | no | Path to a knowledge markdown file |
 | `EXPLORBOT_SPEC` | no | Docbot application spec directory or index.md, used as page knowledge |
 | `EXPLORBOT_API_SPEC` | no | OpenAPI spec path for the API boat |
+| `EXPLORBOT_API_HEADERS` | no | Headers sent with every API request, one "Name: value" per line |
 | `EXPLORBOT_NO_BANNER` | no | Suppress the startup banner, for machine-readable output |
 | `EXPLORBOT_MAX_DURATION` | no | Wall-clock budget in minutes for an explore run; same as --max-duration |
 <!-- END env -->
@@ -231,11 +232,12 @@ The same variables drive API testing and doc collection.
 ```bash
 EXPLORBOT_URL=https://api.example.com \
 EXPLORBOT_API_SPEC=./openapi.yaml \
+EXPLORBOT_API_HEADERS="Authorization: Bearer $TOKEN" \
 EXPLORBOT_AI_PROVIDER=openrouter \
   npx explorbot api explore /users
 ```
 
-The API boat also takes those two as flags, so one line carries the whole run: `npx explorbot api explore /users --endpoint https://api.example.com --spec ./openapi.yaml`.
+The API boat also takes those three as flags, so one line carries the whole run: `npx explorbot api explore https://api.example.com --spec ./openapi.yaml -H "Authorization: Bearer $TOKEN"`. Given a full URL, `api explore` reads it as the base endpoint; a path like `/users` needs the base in `--endpoint` or `EXPLORBOT_URL`.
 
 ```bash
 EXPLORBOT_AI_PROVIDER=openrouter \

package/docs/workflow/application-spec.md

@@ -75,3 +75,7 @@ Screenshots and other relative links may be included for readers, but Explorbot
 ## Validation
 
 Explorbot rejects a bundle when `index.md` or `pages/` is missing, when it contains no page files, or when a page has an unsupported format, version, or missing URL.
+
+## Scout
+
+Beyond the per-URL injection, the same bundle feeds the [Scout agent](../reference/configuration.md#scout-agent): when Scout is enabled, it searches `pages/` (and any extra `ai.agents.scout.dirs`) for documentation relevant to the page being planned and reports it to the Planner. Pages already injected for the current URL are excluded from scouting, so the two channels never duplicate each other.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "explorbot",
-  "version": "0.4.3",
+  "version": "0.4.5",
   "description": "CLI app built with React Ink, CodeceptJS, and Playwright",
   "license": "Elastic-2.0",
   "type": "module",

package/src/action-result.ts

@@ -33,6 +33,7 @@ interface ActionResultData extends WebPageState {
   iframeSnapshots?: Array<{ src: string; html: string; id?: string }>;
   ariaSnapshot?: string | null;
   ariaSnapshotFile?: string;
+  regionAria?: string | null;
   focusedElement?: FocusedElement | null;
   iframeURL?: string;
   links?: Link[];
@@ -90,6 +91,7 @@ export class ActionResult implements ActionResultData {
   public links: Link[] = [];
   public verifications?: Record<string, boolean>;
   public overlay: Region = new Region();
+  public regionAria: string | null = null;
   private _diffCache: { previousId: number | undefined; diff: Diff } | null = null;
 
   constructor(data: ActionResultData) {
@@ -106,6 +108,7 @@ export class ActionResult implements ActionResultData {
     this.iframeURL = data.iframeURL;
     this.notes = data.notes ?? [];
     this.verifications = data.verifications;
+    this.regionAria = data.regionAria ?? null;
 
     // Set readonly properties
     if (data.screenshotFile !== undefined) {
@@ -304,6 +307,10 @@ export class ActionResult implements ActionResultData {
     return compactAriaSnapshot(this.ariaSnapshot, false);
   }
 
+  getRegionARIA(): string {
+    return compactAriaSnapshot(this.regionAria, false);
+  }
+
   getCompactARIA(): string {
     return compactAriaSnapshot(this.ariaSnapshot, true);
   }

package/src/action.ts

@@ -14,8 +14,8 @@ import { browserErrorMessage, isFatalBrowserError, isNavigationTransitionError }
 import { captureHtmlForSnapshot, htmlCombinedSnapshot, minifyHtml } from './utils/html.js';
 import { createDebug, setStepSpanParent, tag } from './utils/logger.js';
 import { Overlay, OverlayPage } from './utils/overlay.js';
-import type { Region } from './utils/region.js';
 import { sleep, waitForPageReadiness } from './utils/page-readiness.ts';
+import type { Region } from './utils/region.js';
 import { safeFilename } from './utils/strings.ts';
 import { codeceptJSSandbox, hasPlaywrightCommands, playwrightSandbox, sanitizeCodeBlock } from './utils/web-sandbox.ts';
 
@@ -182,7 +182,19 @@ class Action {
         focusedElement,
         iframeURL: frame ? frame.url?.() || 'iframe' : undefined,
       });
-      if (!frame) await this.detectRegion(result).catch((err: Error) => debugLog('Region detection failed:', err.message));
+      if (!frame) {
+        await this.detectRegion(result).catch((err: Error) => debugLog('Region detection failed:', err.message));
+        const regionRoot = result.overlay.root;
+        if (result.overlay.isModal && regionRoot) {
+          result.regionAria = await this.playwrightHelper.page
+            .locator(regionRoot)
+            .ariaSnapshot()
+            .catch((err: Error) => {
+              debugLog('Region ARIA snapshot failed:', err.message);
+              return null;
+            });
+        }
+      }
       this.stateManager.updateState(result, codeBlock);
       return result;
     } catch (err) {

package/src/ai/fisherman/tools.ts

@@ -7,11 +7,13 @@ import type { RequestStore } from '../../api/request-store.ts';
 import { extractEndpointDefinition } from '../../api/spec-reader.ts';
 import type { Test } from '../../test-plan.ts';
 import { tag } from '../../utils/logger.ts';
+import { truncate } from '../../utils/strings.ts';
 import { isDynamicSegment } from '../../utils/url-matcher.ts';
 import type { Fisherman } from '../fisherman.ts';
 import type { RequestHaul } from './request-haul.ts';
 
 const BODY_PREVIEW_LIMIT = 2000;
+const READS_IN_ANSWER = 3;
 
 export function createFishermanTools(apiClient: ApiClient, requestStore: RequestStore, haul: RequestHaul, opts: { spec?: any; baseEndpoint?: string; readOnly?: boolean }) {
   const readOnly = opts.readOnly === true;
@@ -242,7 +244,7 @@ export function createAskApiTool(fisherman: Fisherman | null, task: Test) {
         }
 
         task.addNote(`Asked API: ${question} — ${result.summary}`);
-        tag('success').log(`Ask API: ${result.summary}`);
+        tag('success').log(`Ask API: ${truncate(result.summary, 200)}`);
         return { answered: true, answer: result.summary };
       },
     }),
@@ -285,6 +287,11 @@ function synthesizeResult(haul: RequestHaul, declaredDone: boolean, readOnly: bo
     succeeded = haul.successfulReads();
     successLabel = 'successful reads';
   }
+  if (readOnly && succeeded.length > 0) {
+    const bodies = succeeded.slice(-READS_IN_ANSWER).map((read) => `${read.toEndpoint()} → ${read.rawResponseBody.substring(0, BODY_PREVIEW_LIMIT)}`);
+    return { success: true, summary: bodies.join('\n\n'), created: [], failed: [] };
+  }
+
   let summary = `Stopped before finishing: ${made.length} requests, ${succeeded.length} ${successLabel}, ${failures.length} failed`;
   const lastFailure = failures[failures.length - 1];
   if (lastFailure) summary += `; last failure: ${lastFailure.toSummary()}`;

package/src/ai/fisherman.ts

@@ -3,6 +3,7 @@ import type { ApiClient } from '../api/api-client.ts';
 import { type EndpointFamily, type RequestStore, isFailedRequest } from '../api/request-store.ts';
 import { listAllEndpoints } from '../api/spec-reader.ts';
 import { createDebug, tag } from '../utils/logger.ts';
+import { truncate } from '../utils/strings.ts';
 
 const debugLog = createDebug('explorbot:fisherman');
 import { loop } from '../utils/loop.ts';
@@ -137,7 +138,7 @@ export class Fisherman implements Agent {
     await this.runSession(conversation, tools, { haul, isFinished, finishFromText, label: `fisherman lookup: ${question.slice(0, 50)}` });
 
     const result = getResult();
-    tag('info').log(`Fisherman answer: ${result.summary}`);
+    tag('info').log(`Fisherman answer: ${truncate(result.summary, 200)}`);
     return result;
   }
 

package/src/ai/pilot.ts

@@ -363,8 +363,6 @@ export class Pilot implements Agent {
     return dedent`
       SCENARIO: ${task.scenario}
 
-      ${this.buildDeletionScope(task)}
-
       EXPECTED RESULTS (milestones):
       ${task.expected.map((e) => `- ${e}`).join('\n')}
     `;
@@ -372,20 +370,22 @@ export class Pilot implements Agent {
 
   private buildResetSystemPrompt(task: Test): string {
     return dedent`
-      You are Pilot — decide whether a reset is legitimate. Reset is DESTRUCTIVE: it abandons this
-      iteration's work, but server-side side effects (records created, forms submitted) persist.
-      Unnecessary resets create duplicate data and infinite loops.
+      You are Pilot — decide whether a reset is legitimate. Reset only re-navigates to the start URL:
+      it writes nothing, though it abandons this iteration's work and server-side side effects persist.
+      The hazard is the tester REDOING a completed flow afterwards — duplicate data and infinite loops.
 
       ${this.buildSharedEvidenceRules()}
 
       DECISION:
-      - "allow": current page cannot host the scenario, irrecoverable error, or no path back.
-      - "continue": prior action already succeeded (URL changed, record visible, confirmation shown) — verify/finish instead. Or scenario goal may already be met; instruct tester to verify the actual outcome rather than redo. Provide guidance.
+      - "allow": current page cannot host the scenario, irrecoverable error, no path back, or an
+        expectation requires the outcome to survive a reload or a return to the start page and no
+        reset has been taken yet this run — there the reset IS the check, not a redo.
+      - "continue": the outcome the scenario needs is already observable on the CURRENT page — verify/finish instead. Provide guidance.
       - "fail": resetCount >= 2 and underlying situation hasn't changed; same flow tried twice with same failure mode.
       - "skipped": feature doesn't exist on this app or prerequisites can't be met.
 
       PRIORITY:
-      1) Successful side effects in session_log → almost never allow reset.
+      1) Successful side effects in session_log → allow reset only to re-observe them, never to repeat them.
       2) resetCount — each prior reset raises the bar.
       3) Tester's stated reason — weigh against evidence, don't trust blindly.
 
@@ -1102,23 +1102,6 @@ export class Pilot implements Agent {
       .join('\n\n');
   }
 
-  private buildDeletionScope(task: Test): string {
-    const deletableItems = task.plan
-      ? task.plan
-          .listTests()
-          .filter((t) => t.isSuccessful && t.sessionName)
-          .map((t) => t.sessionName!)
-      : [];
-    const scenarioLower = task.scenario.toLowerCase();
-    if (deletableItems.length > 0) {
-      return `For deletion scenarios, items can only be deleted if their title contains: ${deletableItems.join(', ')}`;
-    }
-    if (scenarioLower.includes('delete') || scenarioLower.includes('remove')) {
-      return 'No items available for deletion — test should create an item first';
-    }
-    return '';
-  }
-
   private getSystemPrompt(task: Test, initialState: ActionResult): string {
     const interactive = isInteractive();
     const stepsText = task.plannedSteps.length > 0 ? task.plannedSteps.map((s, i) => `${i + 1}. ${s}`).join('\n') : 'No planned steps';

package/src/ai/planner.ts

@@ -25,6 +25,7 @@ import { POSSIBLE_SECTIONS, type Researcher } from './researcher.ts';
 import { findSimilarStateHash } from './researcher/cache.ts';
 import { hasFocusedSection } from './researcher/focus.ts';
 import { capabilityGroundingRule, dataProtectionRules, fileUploadRule } from './rules.ts';
+import type { Scout } from './scout.ts';
 
 const debugLog = createDebug('explorbot:planner');
 
@@ -63,6 +64,7 @@ export class Planner extends PlannerBase implements Agent {
   private lastSuite: Suite | null = null;
   researcher: Researcher;
   private fisherman: Fisherman | null = null;
+  private scout: Scout | null = null;
 
   constructor(deps: AgentDeps, researcher: Researcher) {
     super();
@@ -78,10 +80,19 @@ export class Planner extends PlannerBase implements Agent {
     this.fisherman = fisherman;
   }
 
+  setScout(scout: Scout): void {
+    this.scout = scout;
+  }
+
   private get sectionOrder(): string[] {
     return ConfigParser.getInstance().getConfig().ai?.agents?.researcher?.sections || Object.keys(POSSIBLE_SECTIONS);
   }
 
+  private get docsWeight(): number {
+    const value = ConfigParser.getInstance().getConfig().ai?.agents?.planner?.docsWeight ?? 70;
+    return Math.max(0, Math.min(100, value));
+  }
+
   private getDefaultStartUrl(state: { url: string; fullUrl?: string }): string {
     return state.fullUrl || state.url;
   }
@@ -328,6 +339,7 @@ export class Planner extends PlannerBase implements Agent {
     const conversation = new Conversation([], model);
     conversation.autoTrimTag('page_research', 20000);
     conversation.autoTrimTag('tested_scenarios', 10000);
+    conversation.autoTrimTag('docs_context', 8000);
 
     conversation.addUserText(this.getSystemMessage(feature));
 
@@ -385,6 +397,11 @@ export class Planner extends PlannerBase implements Agent {
     const research = await this.researcher.research(currentState || state, {
       deep: true,
     });
+
+    let docsPromise: Promise<string> | null = null;
+    if (this.scout && this.docsWeight > 0) {
+      docsPromise = this.scout.collectDocs({ url: state.url, title: state.title, feature, excludeUrls: this.knowledgeTracker.applicationSpecUrls(state) });
+    }
     let plannerResearch = mdq(research).query('code').replace('');
     plannerResearch = mdq(plannerResearch)
       .query('table')
@@ -419,6 +436,22 @@ export class Planner extends PlannerBase implements Agent {
       conversation.addUserText(applicationContext);
     }
 
+    if (docsPromise) {
+      const docs = await docsPromise;
+      if (docs) {
+        conversation.addUserText(dedent`
+          <docs_context>
+          Documentation retrieved from the collected corpus by the Scout agent.
+          Ground scenarios in these documented capabilities where they apply; treat them as supporting context, not a script.
+
+          Aim for roughly ${this.docsWeight}% of the scenarios to exercise behavior documented above; the remainder may explore beyond the documentation.
+
+          ${docs}
+          </docs_context>
+        `);
+      }
+    }
+
     conversation.addUserText(dedent`
       ${this.buildApproach(style)}
 

package/src/ai/provider.ts

@@ -409,7 +409,8 @@ export class Provider {
     setActivity(`🤖 Asking ${modelName} with dynamic tools`, 'ai');
     promptLog(`Using model: ${modelName}`);
 
-    const toolsWithCommentary = tools?.commentary ? tools : { ...tools, commentary: createHarmonyChannelFallbackTool() };
+    let toolsWithCommentary = tools;
+    if (!tools?.commentary && options.toolChoice !== 'required') toolsWithCommentary = { ...tools, commentary: createHarmonyChannelFallbackTool() };
     const toolNames = Object.keys(toolsWithCommentary || {});
     tag('debug').log(`Tools enabled: [${toolNames.join(', ')}]`);
     promptLog('Available tools:', toolNames);

package/src/ai/researcher/deep-analysis.ts

@@ -1,6 +1,6 @@
 import dedent from 'dedent';
 import { ActionResult, type Diff } from '../../action-result.js';
-import type { ExplorbotConfig } from '../../config.ts';
+import { type ExplorbotConfig, agentSettings } from '../../config.ts';
 import { executionController } from '../../execution-controller.ts';
 import type Explorer from '../../explorer.ts';
 import type { StateManager } from '../../state-manager.js';
@@ -31,7 +31,7 @@ export function WithDeepAnalysis<T extends Constructor>(Base: T) {
       tag('info').log('Starting deep analysis of expandable elements');
       await (this as any).navigateTo(state.fullUrl || state.url);
 
-      const maxClicks = (this.config.ai?.agents?.researcher as any)?.maxExpandableClicks ?? DEFAULT_MAX_EXPANDABLE_CLICKS;
+      const maxClicks = agentSettings(this.config, 'researcher').maxExpandableClicks ?? DEFAULT_MAX_EXPANDABLE_CLICKS;
 
       const expandedSections: string[] = [];
       const navigationLinks: Array<{ code: string; url: string }> = [];
@@ -88,6 +88,8 @@ export function WithDeepAnalysis<T extends Constructor>(Base: T) {
     }
 
     async researchOverlay(current: ActionResult, previous: ActionResult, pageStateHash: string): Promise<string | null> {
+      if (!(this as any).isEnabled()) return null;
+
       const region = current.overlay;
       if (!region.isOpen || !region.name) return null;
 

package/src/ai/researcher/sections.ts

@@ -1,6 +1,6 @@
 import dedent from 'dedent';
 import type { ActionResult } from '../../action-result.js';
-import type { ExplorbotConfig } from '../../config.ts';
+import { type ExplorbotConfig, agentSettings } from '../../config.ts';
 import { executionController } from '../../execution-controller.ts';
 import type Explorer from '../../explorer.ts';
 import type { StateManager } from '../../state-manager.js';
@@ -66,7 +66,7 @@ export function WithSections<T extends Constructor>(Base: T) {
     }
 
     private async _detectFocusCss(): Promise<string | null> {
-      const focusSections = (this.config.ai?.agents?.researcher as any)?.focusSections as string[] | undefined;
+      const focusSections = agentSettings(this.config, 'researcher').focusSections;
       if (!focusSections?.length) return null;
 
       for (const css of focusSections) {

package/src/ai/researcher.ts

@@ -1,7 +1,7 @@
 import dedent from 'dedent';
 import { ActionResult } from '../action-result.js';
 import { setActivity } from '../activity.ts';
-import { ConfigParser, type ExplorbotConfig, outputPath } from '../config.ts';
+import { ConfigParser, type ExplorbotConfig, type ResearcherAgentConfig, agentSettings, outputPath } from '../config.ts';
 import { executionController } from '../execution-controller.ts';
 import type { ExperienceTracker } from '../experience-tracker.ts';
 import type Explorer from '../explorer.ts';
@@ -19,7 +19,7 @@ import { annotatePageElements } from '../utils/web-annotate.ts';
 import type { Agent, AgentDeps } from './agent.js';
 import type { Navigator } from './navigator.ts';
 import { ContextLengthError, type Provider } from './provider.js';
-import { findSimilarResearch, getCachedResearch, reportResearch, saveResearch } from './researcher/cache.ts';
+import { findSimilarResearch, getCachedResearch, getPreviousResearch, reportResearch, saveResearch } from './researcher/cache.ts';
 import { type CoordinateMethods, WithCoordinates } from './researcher/coordinates.ts';
 import { type DeepAnalysisMethods, WithDeepAnalysis } from './researcher/deep-analysis.ts';
 import { detectFocusedSection, hasFocusedSection, markSectionAsFocused, pickDefaultFocusedSection } from './researcher/focus.ts';
@@ -62,13 +62,23 @@ export class Researcher extends ResearcherBase implements Agent {
   constructor(deps: AgentDeps) {
     super(deps);
     this.experienceTracker = deps.stateManager.getExperienceTracker();
+    this.settings.reasoning ??= 'low';
+  }
 
-    const ai = deps.config.ai;
-    if (ai) {
-      ai.agents ??= {};
-      ai.agents.researcher ??= {};
-      ai.agents.researcher.reasoning ??= 'low';
-    }
+  get settings(): ResearcherAgentConfig {
+    return agentSettings(this.config, 'researcher');
+  }
+
+  isEnabled(): boolean {
+    return this.settings.enabled !== false;
+  }
+
+  enable(): void {
+    this.settings.enabled = true;
+  }
+
+  disable(): void {
+    this.settings.enabled = false;
   }
 
   protected getNavigator(): Navigator {
@@ -94,7 +104,7 @@ export class Researcher extends ResearcherBase implements Agent {
 
   async research(state: WebPageState, opts: { screenshot?: boolean; force?: boolean; deep?: boolean; data?: boolean; fix?: boolean; _retriesLeft?: number } = {}): Promise<string> {
     const { screenshot = false, force = false, deep = false, data = false, fix = true } = opts;
-    const maxRetries = (this.config.ai?.agents?.researcher as any)?.retries ?? 2;
+    const maxRetries = this.settings.retries ?? 2;
     let retriesLeft = opts._retriesLeft ?? maxRetries;
     this.actionResult = ActionResult.fromState(state);
     const stateHash = this.actionResult.baseHash;
@@ -109,6 +119,13 @@ export class Researcher extends ResearcherBase implements Agent {
       }
     }
 
+    if (!this.isEnabled()) {
+      debugLog('Researcher is disabled, answering with the recorded map');
+      const recorded = getPreviousResearch(stateHash);
+      if (recorded) reportResearch(stateHash, recorded);
+      return recorded;
+    }
+
     Stats.researches++;
 
     const sessionName = `researcher: ${state.url}`;
@@ -341,7 +358,7 @@ export class Researcher extends ResearcherBase implements Agent {
   }
 
   private async waitUntilSettled(screenshot: boolean): Promise<boolean> {
-    const errorPageTimeout = (this.config.ai?.agents?.researcher as any)?.errorPageTimeout ?? 10;
+    const errorPageTimeout = this.settings.errorPageTimeout ?? 10;
     if (errorPageTimeout <= 0) return false;
 
     const includeScreenshot = screenshot && this.provider.hasVision();
@@ -374,7 +391,7 @@ export class Researcher extends ResearcherBase implements Agent {
   }
 
   private getConfiguredSections(): Record<string, string> {
-    const configSections = (this.config.ai?.agents?.researcher as any)?.sections as string[] | undefined;
+    const configSections = this.settings.sections;
     if (!configSections?.length) return POSSIBLE_SECTIONS;
     const filtered: Record<string, string> = {};
     for (const key of configSections) {

package/src/ai/rules.ts

@@ -8,8 +8,9 @@ const locatorPriorityRule = dedent`
 
   1. ARIA locators (first choice) - target browser's accessibility tree, most reliable
      Use JSON format: { "role": "button", "text": "Login" }
-     Copy role and text VERBATIM from the ARIA snapshot or UI map — never guess the pair.
-     If the element is absent from the snapshot, do not invent one; use text or CSS instead.
+     Copy role and text VERBATIM from the ARIA snapshot, UI map, or the page diff that
+     reported the element — never guess the pair; a guessed role can silently match a
+     different element with the same text. If named nowhere, use text or CSS instead.
 
   2. Text locators (second choice) - exact visible text, use only when unique on the page
      Example: 'Login', 'Submit', 'Username'
@@ -238,10 +239,10 @@ export const unexpectedPopupRule = dedent`
   If buttons are disabled unexpectedly, check if a popup is blocking interaction or if required form fields are empty.
 
   Dismiss strategy (try in order):
-  1. I.clickXY(0, 0) — click outside the popup to close it
-  2. I.pressKey('Escape') — press Escape to dismiss
-  3. I.click('Cancel') — click Cancel button if present
-  4. I.click({ role: 'button', text: 'Close' }) — click X/close button if present
+  1. I.pressKey('Escape') — press Escape to dismiss
+  2. I.click('Cancel') — click Cancel button if present
+  3. I.click({ role: 'button', text: 'Close' }) — click X/close button if present
+  4. I.clickXY(0, 0) via form() tool and check if page diff changed
   </unexpected_popup_rule>
 `;
 
@@ -335,7 +336,7 @@ export const actionRule = dedent`
   Prefer text/ARIA locators with context over complex CSS/XPath selectors.
   For inline create/edit flows, after filling a field verify it contains the value, then confirm using the nearest explicit button/link, an adjacent icon-only confirm control in the same row/form, or Enter if the field remains focused.
   If locator doesn't work, try CSS or XPath locators.
-  If nothing works, use I.clickXY(x, y) as last resort.
+  If nothing works, use visualClick() — it locates the target in a screenshot before clicking it.
 
   When a click result reports several matches, pick one from its numbered list by position rather than guessing a new locator.
   Reuse the same locator with step.opts({ elementIndex: N }) as the LAST argument. N is the "Element N" number.