explorbot 0.3.5 → 0.4.1

package/docs/superpowers/specs/2026-08-29-region-states-design.md

@@ -0,0 +1,262 @@
+# Region-of-Interest States — Diff-Detected Modals, Drawers and Soft Navigation
+
+**Date:** 2026-08-29
+**Status:** Planned
+**Plan:** `docs/superpowers/plans/2026-08-29-region-states.md`
+
+## Problem
+
+A state is `url + h1 + h2` (`ActionResult.getStateHash`, `src/action-result.ts:478`). The only
+other state signal is `StateManager.hasDialogAppeared` (`src/state-manager.ts:209`), which fires
+when the ARIA snapshot suddenly contains a dialog/modal node (`Overlay.fromAria` →
+`detectFocusArea`). Everything that opens without an ARIA dialog role and without a URL change is
+invisible as a state:
+
+- drawers and side panels rendered as plain positioned `<div>`s;
+- soft navigation — a SPA swapping a large content region in place (wizard step, inline editor,
+  detail subview) with no full re-render and no URL change.
+
+The diff pipeline actually *sees* these. `htmlDiff` (`src/utils/html-diff.ts`) returns
+`HtmlDiffPart[]` — each with a stable `container` selector and the appeared `subtree` — but
+`collapseHtmlParts` (`src/action-result.ts:591`) treats any diff over 8K chars as a full page
+re-render and collapses it to `...collapsed (12000 chars)...`. The one signal that says "a modal
+just opened" is thrown away as noise.
+
+On top of that, overlay detection today is **scattered across three approaches in four files**:
+
+1. ARIA role detection — `detectFocusArea` in `aria.ts`, via `Overlay.fromAria`;
+2. a selector-heuristic browser extractor — `extractVisibleOverlayHtml` in `html.ts`, driven by
+   `OVERLAY_SELECTORS` class-name patterns (`[class*="modal"]`, `[class*="drawer"]`…) and a
+   z-index geometry fallback, invoked from `Action.captureOverlayHtml` and independently from
+   `Driller.getVisibleOverlayHtml`;
+3. the `overlayHtml` → `Overlay.resolve` priority chain in `ActionResult`.
+
+The class-name selector heuristic is exactly the kind of memorized surface form the project's
+Regex-vs-AI doctrine rejects: it works only on sites that happen to name their CSS that way.
+
+Consequences:
+
+- Tester gets either the strict `<focus_scope>` block (ARIA dialogs only) or nothing. For a
+  drawer without a role it keeps targeting elements behind the drawer.
+- Pilot's `<state>` says `modal: none` while half the screen is a drawer.
+- Modal open/close cycling is invisible to `isInDeadLoop` — every hash in the window is the base
+  page.
+- Experience recorded while a modal is open lands in the base page's experience file, with no
+  record that it only applies inside that modal.
+
+## Design
+
+### Unification: `overlay.ts` is the single detection module
+
+All area-of-interest semantics live in `src/utils/overlay.ts`. After this change there are
+exactly **two** detection signals, both general:
+
+1. **ARIA** — `Overlay.fromAria` (role-based dialogs, free at capture time, needs no previous
+   state). `aria.ts` keeps only the ARIA-tree *primitives* (`detectFocusArea`,
+   `focusAreaControls`); their sole overlay-semantics consumer is `overlay.ts`.
+2. **Diff + geometry** — the new pipeline below, for everything the ARIA tree does not label.
+
+The selector-heuristic path is **deleted entirely** (see "Removed code"). No third approach, no
+class-name patterns, no priority chain.
+
+`overlay.ts` exposes exactly two classes: `Overlay`, the immutable value describing what is open,
+and `OverlayPage`, which wraps the live page and owns detection. The single public entry point is
+`new OverlayPage(page).detectRegion(diffParts)`; every lower-level step (subroot picking, the
+browser probe, coverage classification) is a private member. The pipeline runs after every action
+inside `Action.capturePageState`, before the (sync) `stateManager.updateState` call — `Action`
+only hands the page over (it is the only browser mover):
+
+```
+capture html/aria
+  └─ same URL, not iframe, html changed, no ARIA overlay already detected
+       └─ diff vs previous state (parse5, memoized — shared with toToolResult)
+            └─ OverlayPage.detectRegion(parts)
+                 ├─ appeared subtree ≥ 5K chars       (private)
+                 ├─ coverage probe via page.evaluate   (private)
+                 └─ coverage classification            (private)
+                      ├─ overlays the page → Overlay 'modal' | 'drawer'
+                      └─ inline            → Overlay 'region'
+```
+
+Detection is 100% structural — size threshold, diff paths, geometry. No AI in the path. AI enters
+only downstream: `researchOverlay` describes the region, Tester/Pilot decide what to do in it.
+
+### 1. Appeared-subroot detection (`OverlayPage`, over `html-diff.ts` parts)
+
+The first private step of `detectRegion` picks the largest part that contains an appeared
+element (`ELEMENT:` line in `part.added`) and whose minified `subtree` is ≥ `SUBROOT_MIN_HTML`
+(10 000 chars, unexported const — no config knob). `html-diff.ts` stays a generic diff engine; it
+newly exports `pathToXPath` so overlay.ts can convert appeared-element paths.
+
+The part's `container` is by design an ancestor that exists in **both** snapshots
+(`findStableContainer`) — it is never the appeared element, and for portal roots
+(`#modal-root`-style, zero-height with fixed children) its geometry lies. So the result carries
+both:
+
+- `container` — the stable scoping selector, handed to Tester and stored as the experience `root`;
+- `elementXPath` — the appeared element itself (from the `ELEMENT:html[1]/body[1]/div[3]` path
+  via `pathToXPath`) — this is what the coverage probe measures.
+
+When `container` degrades to `body` (top-level appended node — the common portal case), the
+`elementXPath` doubles as the root selector.
+
+### 2. Openness verification (`OverlayPage`)
+
+One hit-test decides whether the appeared region is **actually open**: take the center of the
+region's visible (viewport-intersected) rect, ask `document.elementFromPoint` what lives there,
+and check the hit belongs to the region (`.modal` is the region; the input at its center belongs
+to it → it is on top). A region whose own center resolves to a foreign element is hidden or
+covered — verified not open, and **discarded** rather than classified.
+
+For an open region, two values collected in the same probe decide the kind:
+
+- computed position — floating (`fixed`/`absolute`/positive z-index) → overlaying; in-flow →
+  inline `region` (so soft navigation never triggers the strict focus scope);
+- visible-rect coverage of the viewport — overlaying with coverage ≥ 0.8 → `modal`, else
+  `drawer`.
+
+The probe is a module-private plain function shipped as a source string into `page.evaluate`
+(it must stay self-contained so `toString()` reconstruction works in the browser); tests drive
+`detectRegion` with fake pages returning canned probe results.
+
+When the probe cannot run at all (no page, evaluate throws, element already gone) the region
+degrades to inline `region` with a debug log — never to an overlay: a false "overlaying" verdict
+would make Tester refuse legitimate navigation. An off-screen in-flow region (below the fold)
+also stays inline instead of being discarded.
+
+### 3. Overlay carries the region (`overlay.ts`)
+
+`Overlay` is extended rather than a parallel concept added:
+
+- `type`: `'dialog' | 'modal' | 'drawer' | 'region' | null` — `region` means inline subview;
+- `name`: heading-derived (h1–h4 join over the region HTML, private `nameFromHtml`);
+- `root`: the scoping selector (container CSS, or element XPath when the container degraded to
+  `body`);
+- `get detected()` — **keeps meaning "verified overlaying"** (`type` is dialog/modal/drawer).
+  Every existing consumer of `detected` (Tester `<focus_scope>`, Pilot `modal:` line,
+  `hasDialogAppeared`) keeps its semantics.
+- `get present()` — any region, inline included. New consumers that want "an area of interest
+  exists" use this.
+- `html`: the region's minified subtree, carried on the overlay itself — `toToolResult` renders
+  it as the single diff part instead of a collapsed dump.
+- `describe()` — the one-line human/model-facing summary
+  (`drawer "Edit User" opened, scope: aside.panel`), used for `pageDiff.areaOfInterest`.
+- `OverlayPage.detectRegion` builds the Overlay: verdict `overlays: true` with coverage ≥ 0.8 →
+  `modal`; overlaying with partial coverage → `drawer`; otherwise `region`.
+- `Overlay.resolve` simplifies to two sources: stored `overlay` data, else `fromAria`.
+
+### 4. State identity (`src/action-result.ts`)
+
+- `getStateHash()` gains a `region_<name>` part when `overlay.present && overlay.name`.
+  **Named regions only**: names come from headings (stable across runs); selectors with dynamic
+  classes never enter a hash. An unnamed region does not fork the state — which is why
+  `hasDialogAppeared` survives (generalized to `hasRegionAppeared` over `present`) as the
+  transition trigger for unnamed overlays.
+- `baseHash` getter — the hash without the region part. The research cache and Tester's
+  `pageStateHash` key off `baseHash`, otherwise a modal open at capture time forks
+  `getCachedResearch` and poisons `researchOverlay`'s append-to-page-research flow.
+- `diff(previous)` is memoized on `previous.id` so capture-time detection and `toToolResult`
+  share one parse5 pass.
+- Side effect, intended: with the region in the hash, modal **close** also changes the hash — a
+  test cycling open/close now produces alternating hashes that `isInDeadLoop` can see.
+
+### 5. StateManager records region states (`src/state-manager.ts`)
+
+Named regions change the hash, so `updateState` records the transition through the existing
+hash-changed path — region states land in `stateHistory`, `getRecentTransitions`, visit counts,
+and the `tag('data').log('state', …)` remote frame (which gains a `region` field). Unnamed
+regions go through `hasRegionAppeared` (the renamed, `present`-based `hasDialogAppeared`).
+
+### 6. Experience envelope: `root:` (`src/experience-tracker.ts`)
+
+Experience files for region states get a new frontmatter key:
+
+```markdown
+---
+url: /users
+title: Users — Admin
+root: 'aside.detail-panel'
+---
+```
+
+Envelope checklist (per CLAUDE.md "Data Envelope Formats"):
+
+1. **Read deterministically by code** — retrieval gating below; never interpreted by the model.
+2. **Scoped to URL/state** — per `<stateHash>.md` file; region states have their own hash, so
+   their file is created while the region is open and `root` comes from `state.overlay.root`.
+3. **Optional with a default** — absent means "whole page"; every existing file on disk behaves
+   exactly as today.
+4. **Single writer** — `ExperienceTracker.ensureExperienceFile` only.
+
+**Retrieval rule** (in `ActionResult.isRelevantExperienceRecord`, where matching already lives):
+a record carrying `root` is loaded only when the current state has a region open —
+`overlay.present` — and, when the current region's own `root` is known, the selectors match
+exactly. Found by this state + root selector exists → the experience file is loaded; no region
+open → the file is skipped, so drawer recipes stop polluting base-page context. Matching stays
+structural (string equality), never semantic.
+
+### 7. Surfacing to the agents
+
+- **Tool results** (`toToolResult`): when the region appeared in this transition, `pageDiff`
+  gains `areaOfInterest` — e.g. `drawer "Edit User" opened, scope: aside.detail-panel` — and
+  `htmlParts` is replaced by a single part containing the region's cleaned snapshot within the
+  existing per-part budget, instead of the `...collapsed (12000 chars)...` marker. This is the
+  payoff: the diff signal that was discarded becomes the headline of the acting tool's result.
+- **Tester** (`reinjectContextIfNeeded`): verified overlays keep the strict `<focus_scope>`
+  block, now with the concrete root selector. Inline regions get a new, softer
+  `<area_of_interest>` block — injected once per state change (via the previously write-only
+  `previousStateHash`) — that names the region and its root but leaves page navigation
+  actionable. The strict "elements outside are not actionable" wording stays gated on the probe
+  verdict. The `researchOverlay` trigger widens from `detected` to `present`.
+- **Pilot** (`buildStateContext`): the `modal:` line stays for verified overlays (its diagnostic
+  prompt patterns keep working) and gains the root; inline regions get a new
+  `region: <name> (inline, root: <selector>)` line plus one general system-prompt bullet.
+- **Researcher** (`deep-analysis.ts` `researchOverlay`): the guard widens from
+  `type === 'dialog' | 'modal'` to any named present region, so drawers and subviews get the same
+  incremental Extended Research treatment, still appended under the base page's research (keyed
+  by `baseHash`).
+- **Driller** (`detectNestedOverlayContext`): stops re-querying the live DOM through the selector
+  extractor. The nested-overlay context is built from what the tool result already carries — the
+  appeared `pageDiff.htmlParts` subtrees (plus the region part when `areaOfInterest` is set).
+  What changed after the click *is* the nested UI; no second detection approach needed.
+
+## Removed code
+
+Unification means the selector-heuristic path is deleted, not deprecated:
+
+| Removed | Was |
+|---|---|
+| `Action.captureOverlayHtml` + `overlayHtml` capture in `capturePageState` | Selector-extractor invocation per capture |
+| `ActionResultData.overlayHtml` + `Overlay.resolve`'s overlayHtml branch + `Overlay.fromHtml` (public) | Priority chain feeding heading-named modals |
+| `OVERLAY_SELECTORS`, `Overlay.captureConfig` (`overlay.ts`) | Class-name patterns (`[class*="modal"]`…) |
+| `extractVisibleOverlayHtml`, `getVisibleOverlayHtmlExtractorSource`, `VisibleOverlayExtractionConfig` (`html.ts`), plus limit fields used only by them | The selector/z-index browser extractor |
+| `Driller.getVisibleOverlayHtml` | Driller's private extractor invocation |
+| `extractVisibleOverlayHtml` describe-block and `overlayHtml` resolve tests | Tests of the removed path |
+
+**Accepted trade-off:** an overlay that is *already open at the very first capture* and carries
+no ARIA dialog role is no longer detected (there is no previous state to diff). The moment any
+action happens, the diff path sees it. This trades a narrow first-paint case for removing a
+site-shape heuristic that violates the core "no memorized surface forms" principle.
+
+## Decisions
+
+| Decision | Choice | Why |
+|---|---|---|
+| Single detection home | `overlay.ts` owns every decision function; `aria.ts` keeps ARIA parsing primitives; `Action` only orchestrates | One place to reason about overlays; browser access stays in the Action tier |
+| Old selector path | Deleted, including Driller's use (rebuilt on `pageDiff`) | User decision: unify, old code gone; class-name selectors are memorized surface forms |
+| What the probe measures | The appeared element (`elementXPath`), never the diff `container` | Container is a both-sides ancestor; portal roots have lying geometry |
+| `detected` semantics | Unchanged: verified overlaying only; new `present` for any region | A false overlay claim makes Tester refuse legitimate navigation — worse than no detection |
+| Hash contribution | Named regions only; `baseHash` escape hatch for research keys | Heading names are stable; selectors are not; research must stay keyed to the page |
+| Threshold | `SUBROOT_MIN_HTML = 5_000` on the minified subtree, unexported const | Single named constant; no config knob until someone needs one |
+| `root` retrieval gating | Sync string equality against `overlay.root`, require `overlay.present` | Deterministic, no DOM query in the sync retrieval path |
+
+## Non-goals / follow-ups
+
+- **DOM-presence gating for experience `root`** (querySelector against stored HTML when the
+  current region is detected by ARIA and has no `root`). Needs an async retrieval path; revisit
+  if the equality rule proves too strict.
+- **Region-scoped ARIA slices** for the Tester context. v1 hands the root selector and the
+  region snapshot via the tool result; slicing the ARIA tree to the region is a later refinement.
+- **First-paint overlay detection without ARIA roles.** If the accepted trade-off above bites in
+  practice, the general fix is a geometry-only probe at first capture (top-most covering element),
+  not the return of class-name selectors.

package/docs/superpowers/specs/2026-08-29-region-states-fixes-design.md

@@ -0,0 +1,269 @@
+# Region States Fixes — Persist the Overlay, Widen the Gates, Reject the Shell
+
+**Date:** 2026-08-29
+**Status:** Planned
+**Follows:** `2026-08-29-region-states-design.md`
+**Evidence:** 7 traced test sessions against beta.testomat.io (Langfuse, 2026-08-29 10:19–13:01), analyzed
+trace-by-trace: `SharedConstitutionalChocolate674`, `ManualConsciousPeach543`, `MagicDepressedRose1`,
+`MedicalUnfairGold673`, `ProudHungryChocolate14`, `RegularQuarrelsomeTurquoise190`, `InevitablePoisedBlack301`.
+
+## Problem
+
+The region-of-interest feature detects correctly and then loses its own result. Across every traced run
+where the diff path fired, `pageDiff.areaOfInterest` carried the right announcement — and the state hash
+never forked (`region_` appears in **zero** traces), Pilot's `region:` line appeared **never**, and the
+Tester context blocks arrived once-late or not at all. The concrete bill for one run
+(`MagicDepressedRose1`): the correct scope `//body/div[16]` sat unused in the tool result while
+`interact()` burned 9 attempts on a guessed `.modal` selector that does not exist — roughly half of a
+4-minute run spent compensating for guidance the feature had already computed.
+
+Where detection did classify, quality was poor at the edges: every diff-detected root was either a
+positional XPath (`//body/div[9..16]`) or the whole app shell (`div.main-app`); a hydration burst was
+classified as a `drawer` covering the page; a modal-*closing* click registered as a drawer *opening*; a
+genuine overlay (its subtree literally contains `modal-footer`, Pilot's screenshot read "obscured by a
+modal") classified as inline `region`; and nested panels inherited the outer panel's name.
+
+Meanwhile the app pattern these runs actually exercise — a drawer that opens **with a URL change**
+(`/suites/suite/new-test`, `/plans/new/manual`) — is invisible to the diff path by design, and the one
+overlay the old ARIA path caught got no root because ARIA detection pre-empts the probe entirely.
+
+## Mechanisms found
+
+1. **Edge-triggered overlay, wiped one capture later.** `Action.detectRegionOfInterest` sets
+   `result.overlay` only on the action whose diff crossed the threshold. The next capture finds no new
+   appeared subtree (the region is no longer *new*), falls through to `Overlay.fromAria` — empty, these
+   drawers carry no `role=dialog` — and `updateState` replaces the current state with an overlay-less
+   one. Everything downstream of the state (hash fork, `isNewState`, `<focus_scope>`,
+   `<area_of_interest>`, Pilot's `region:` line, experience `root:` frontmatter) starves. Confirmed
+   empirically: `reinjectContextIfNeeded` demonstrably ran every iteration (658 `current_focus` tags in
+   one trace) while its region branches fired 0 times against 2 real detections.
+2. **URL-change blind spot.** `Diff.calculate` short-circuits to `liveRegionMessages` whenever the URL
+   changed, and `detectRegionOfInterest` guards on `isSameUrl`. A route-synced SPA drawer — the standard
+   pattern in the app under test — is therefore never evaluated, in any run.
+3. **Shell containers pass the body guard.** `toOverlay` falls back to the element XPath only when the
+   container is literally `body`. `div.main-app` — body's sole meaningful child — sailed through as a
+   "scope", producing `<focus_scope>` text that was factually false ("elements outside are not
+   actionable" about a container that wraps the entire page, `MedicalUnfairGold673` 12:49:08).
+4. **Hydration and modal-close read as regions.** A ~2s same-URL hydration window between a failed click
+   and a `context()` call crossed the flat 10K threshold and, with a stray fixed-position widget, was
+   classified `drawer (root: div.main-app)`. Separately, dismissing a picker re-rendered base content
+   and fired `drawer "New Plan" opened, scope: div.main-app` on a click that *closed* an overlay
+   (`InevitablePoisedBlack301` 13:00:53). Both false positives share a shell root.
+5. **Floating check ignores ancestors.** `inspectRegion` reads `position`/`z-index` off the appeared
+   node only. A large subtree appearing *inside* an already-floating drawer classifies as inline
+   `region` even when the subtree itself contains `modal-footer` and a screenshot shows an overlay
+   (`MagicDepressedRose1` 12:45:23).
+6. **Naming picks the largest part, not the newest heading.** `appearedSubRoot` maximizes subtree size,
+   then `nameFrom` joins h1–h4 of that subtree — so a nested picker is named after the outer panel
+   ("New Plan", "New Test") instead of its own heading ("Select tests for plan", "Select suite for
+   test"), which sat in the same state's h3 the whole time.
+7. **ARIA pre-empts the probe; failed batches capture nothing.** `if (result.overlay.detected) return`
+   means an ARIA-detected modal never gets `root`/`html` enrichment — every `<focus_scope>` in
+   `ManualConsciousPeach543` lacked the scope sentence. And `executeOnce` only captures state on
+   success: the click that opened that modal succeeded *inside* a `form()` batch whose third line
+   failed, so no capture, no detection, and a tool result that labeled all four sub-commands FAILED —
+   including the two that succeeded. Cost: ~26s clicking a button behind a modal the system had been
+   told twice (by its own `see()`) was open.
+
+## Changes
+
+### 1. Persist the detected overlay until it verifiably closes (the P0)
+
+The overlay stops being a per-action edge and becomes state that is carried forward. In
+`Action`'s detection step, per capture, in order:
+
+1. **Same URL, HTML unchanged** → carry the previous state's overlay verbatim. Nothing moved.
+2. **Close check before open check.** If the previous overlay was diff-detected (has an element XPath)
+   and its element is gone from the new HTML or the openness probe says its center no longer belongs to
+   it → drop it (debug log "region closed"). The hash reverts, `updateState` records the close
+   transition — open/close cycles become visible to `isInDeadLoop`, as the original spec intended.
+3. **New detection** from the diff, as today.
+4. **No new detection, previous overlay still confirmed open** → carry it forward onto the new
+   `ActionResult`.
+
+To make the close/confirm check possible, `Overlay` additionally records the appeared element's XPath
+(`xpath`, internal — never rendered into prompts; `root` remains the scoping selector shown to agents).
+The confirm probe is the existing center hit-test (`OverlayPage`), run against the stored XPath — one
+cheap probe per capture, and only while an overlay is being carried.
+
+Two persistence refinements keep common flows from losing state:
+
+- **One-deep parent restore.** When a new detection replaces a carried overlay (a picker opening
+  inside an open drawer), the replaced overlay's identity (`type`/`name`/`root`/`xpath`, no `html`)
+  travels on the new one as `parent`. When the nested region closes, the parent's XPath is probed —
+  still open → it is restored as the current overlay. One level, deterministic; the traced
+  suite-picker-inside-drawer flow keeps its drawer.
+- **ARIA continuity.** A fresh ARIA detection whose type and name match the previous overlay keeps the
+  previous *instance* — otherwise a root enriched by change 6 would survive exactly one capture before
+  a bare `fromAria` result replaced it.
+
+This single change is what unlocks the already-built downstream behavior: the hash forks
+(`region_<name>`), `isNewState` fires once, `<focus_scope>`/`<area_of_interest>` inject,
+Pilot's `<state>` shows the region, and experience files for region states get written with their
+`root:` frontmatter.
+
+### 2. Detect route-synced drawers across URL changes
+
+`Diff` always computes the HTML diff, URL change or not. Tool-result behavior for navigations does not
+change — `PageDiff` keeps surfacing only messages for a changed URL, never navigation-noise
+`htmlParts` — the diff is computed for detection's sake and stays memoized (one parse5 pass, shared).
+
+Detection drops the hard `isSameUrl` gate and replaces it with a structural rule: across a URL change, a
+region is considered only when `similarity >= SOFT_NAVIGATION_SIMILARITY` (unexported const, ~50) — the
+old page must still substantially exist under the new content. A real navigation (low similarity)
+produces no region; a drawer rendered over the still-present page does. The accepted-trade-off section
+of the original spec ("first paint with no ARIA role") stands; this closes the much larger gap the field
+runs actually hit.
+
+### 3. Bound the region: bigger than a widget, smaller than the page
+
+A region is a **band**, not just a floor. The appeared subtree must satisfy both:
+
+- `size >= SUBROOT_MIN_HTML` (5K minified; the second field run showed a real split-pane form panel
+  at 6.6K minified / 9K raw sitting under the original 10K floor, while dropdowns stay under 1K) —
+  below it, a widget, ignored;
+- `size <= REGION_MAX_RATIO * pageSize` (~0.6, raw serialized subtree against raw serialized body —
+  like against like, using the strings the diff already has in hand, no extra minify pass) — above it,
+  **this is not a region change anymore, it is a new state**. No overlay
+  is set; the ordinary state-change machinery (url + headings hash, research on change) owns a page
+  that mostly replaced itself. A full-page takeover that swaps more than 60% of the HTML *is* a new
+  state semantically, and takeovers bring their own headings, so state identity still forks.
+
+The cap is the primary defense against the observed false positives: a hydration burst that finishes
+rendering the page blows it, and a modal-close re-render of the base content blows it too. It is also
+the same principle as change 2's cross-URL similarity floor, seen from the other side — a region
+requires that most of the page **survived**.
+
+The band alone is not enough — the second field run produced a list re-render (search filter cleared,
+rows repopulated) that passed floor, cap and dominance yet had no heading and no semantic root
+anywhere in its dominant chain. Such a detection can neither fork the hash nor scope anything, so it
+is pure noise. Hence an **identity gate** after classification: an overlay with neither a `name` nor
+a `root` is dropped. Either one alone keeps it — a named rootless overlay still forks state identity,
+a rooted nameless one still scopes.
+
+A third field run (a suite-selector dialog) showed the gates must be **per-candidate filters, not
+whole-detection aborts**. Opening that dialog produced three appeared parts at once: a 622K flood of
+detached list nodes rendered directly under `body` (96% of the page), the app shell re-keyed by the
+flood's positional renumbering (196K of old content the differ saw as "appeared"), and the real 13K
+modal. "Largest candidate wins, then cap" picked the flood and aborted, losing the modal every time.
+Detection now walks all in-band candidates instead: cap-tripping parts are skipped (they are page
+redraw, not regions), dominance is measured across the surviving candidates, and among survivors the
+one that brings a **fresh heading** is preferred over larger heading-stale reflows — the same
+newest-heading principle naming already uses, applied to selection. The probe then walks the ordered
+candidates and skips any whose center is covered by another element, so a background pane never wins
+over the dialog stacked on top of it.
+
+Root selection still needs its own care, and its rule is now stricter: **roots are semantic only**. A
+positional XPath is never a `root` — it neither reaches a prompt nor an experience file. It survives
+strictly as the overlay's internal `xpath`, the handle the openness/persistence probe needs to find
+the element, never rendered to agents. (This supersedes the original design's body → element-XPath
+fallback.)
+
+The real root of the appeared subtree is often an anonymous wrapper with no identity of its own —
+`body > div > div > div.bg-overlay > div > div.modal` — so the search may **skip the real root in
+either direction** when a better candidate exists:
+
+- **Up:** stable ancestors, as today — except a container whose subtree spans most of the document is
+  a shell, never a scope (same treatment as literal `body`).
+- **The appeared element itself**, when it carries a stable id or meaningful classes (the same
+  filters `findStableContainer` already applies: dynamic ids out, digit/framework/Tailwind utility
+  classes out, document-unique required).
+- **Down:** unwrap the appeared subtree — repeatedly descend into the child that holds the bulk of
+  the subtree's content (`ROOT_CONTENT_RATIO`, ~0.8; a wrapper has one dominant child), testing each
+  node for a semantic selector. In the example above, the anonymous wrappers and the `bg-overlay`
+  utility class are all rejected by the existing filters, and the descent lands on `div.modal`.
+- **Nothing semantic anywhere → the overlay has no root.** The context blocks already render their
+  scope sentence only when a root exists; agents then scope by the region's name and ARIA, and the
+  experience file simply omits its optional `root:` key.
+
+Preference order: semantic non-shell ancestor → appeared element → deepest dominant descendant →
+none. Classification is unaffected: the openness probe keeps measuring the topmost appeared element
+(the whole overlay, backdrop included), while `root` answers the different question — where locators
+should aim.
+
+This kills both observed root failures: `div.main-app` (shell) and `//body/div[16]` (positional)
+can never be produced again.
+
+### 4. Classify with ancestors and isolation
+
+- **Floating check walks up.** `inspectRegion` reports floating when the element *or any ancestor up to
+  body* is fixed/absolute/z-indexed. A re-render inside an open drawer now classifies as the drawer it
+  is.
+- **Isolation guard.** A region requires a dominant single addition: the winning part must account for
+  the bulk of the diff's total changed subtree length (`REGION_DOMINANCE`, ~0.7). This catches what the
+  change-3 cap cannot: many *small* scattered changes with no dominant subtree.
+- **Close is not open.** Because change 1 runs the close check first, a click that dismisses the current
+  overlay is consumed as a close transition; re-rendered base content underneath cannot double as a
+  fresh region in the same capture.
+
+### 5. Name from the newest heading
+
+`nameFrom` filters the subtree's headings to those **absent from the previous HTML** (plain containment
+check against the prior snapshot — structural, no semantics) and takes the first survivor; only when
+none survives does it fall back to the current first-heading join. A nested picker opening inside "New
+Plan" is now named "Select tests for plan" — which also keys its own hash fork and its own experience
+file, instead of colliding with the outer panel's.
+
+### 6. Merge ARIA identity with probe geometry
+
+ARIA detection no longer pre-empts the probe — the two paths answer different questions and merge:
+
+- ARIA supplies **identity** (type `dialog`/`modal`, accessible name) — it is authoritative when
+  present.
+- The diff+probe supplies **geometry** (`root`, `xpath`, `html`) when a qualifying appeared subtree
+  exists for the same moment.
+
+An ARIA-detected modal whose opening also produced a large diff gets a root and a region snapshot; the
+`<focus_scope>` scope sentence and Pilot's `(root: …)` suffix stop being diff-path-only.
+
+### 7. Capture on the failure path, report batches truthfully
+
+- `executeOnce` captures page state (detection included) on non-fatal failures too, not only on
+  success. A batch that opened a drawer on line 2 and died on line 3 must still produce a state with
+  the drawer in it.
+- The `form()` failure report attributes per-line status from the steps that actually ran (the
+  `executedSteps` machinery from #150), instead of stamping every sub-command FAILED. The model must
+  learn "your click already opened something" from the tool result, not from 26 seconds of timeouts.
+
+### 8. Region transitions wake the Pilot
+
+`shouldAnalyzeProgress` treats a region open/close transition since the last analysis like a new-page
+event: analysis triggers at the next iteration regardless of the interval modulo. In
+`RegularQuarrelsomeTurquoise190`, Pilot went dark for 27 tool calls — the entire lifetime of the region,
+three server 400s included; the feature's Pilot surface is worthless if Pilot never runs while a region
+is open.
+
+## Out of scope — separate follow-ups
+
+Real issues from the same traces that are not this feature's subsystem, recorded here so they are not
+lost:
+
+- **Verdict integrity** (`MedicalUnfairGold673` false pass): pre-existing entities accepted as proof
+  despite the provenance rule; Pilot's own screenshot doubt discarded between two calls 3 seconds
+  apart; verbatim expected-result settlement unenforced at `finish()`; the post-run recipe compactor
+  correctly concluded "no step actually creates a test" and that signal reaches nothing.
+- **Reload primitive**: `pressKey('F5')` is a no-op that reports success; persistence scenarios need a
+  real reload action.
+- **Pilot server-error triage**: a raw backend error (`WRONGTYPE`) was first misdiagnosed as a form
+  problem; 4xx/5xx with non-validation bodies should bias to stop-and-report on first occurrence.
+- **Experience vocabulary**: a stored recipe describing the picker as a "modal" steered `interact()`
+  into guessing `.modal`; supersede stale wording when a fresh run's classification disagrees, instead
+  of suppressing the new write as a duplicate.
+- **App defects found** (report to the product, they are findings, not bugs here): plans `POST` → 400
+  `WRONGTYPE` with partial persistence; suites search → 500; stale Ember modal backdrop blocking Save;
+  Monaco editor duplicating filled content.
+
+## Validation
+
+- Unit: carry-forward and close transitions (StateManager history shows open → carried → closed);
+  cross-URL detection above/below the similarity floor; the size band — below 5K no region, inside
+  the band a region, above `REGION_MAX_RATIO` no overlay and a plain state change; root selection —
+  semantic ancestor preferred, wrapper-chain descent landing on a nested semantic container, shell
+  rejection, and the no-semantic case yielding a rootless overlay with no positional XPath anywhere
+  in prompts or frontmatter; ancestor-floating classification; dominance guard against scattered
+  diffs; newest-heading naming with a nested-panel fixture; ARIA+probe merge producing rooted
+  `dialog`; failure-path capture.
+- The seven Langfuse sessions above are the acceptance fixture: re-run the same two focus commands
+  (`create test`, `create plan of different kinds`) and require — hash forks containing `region_`,
+  Pilot `<state>` showing the region while open, zero `.modal`-guess `interact()` scopes, and no
+  `div.main-app` root anywhere.

package/docs/superpowers/specs/2026-08-30-fisherman-live-session-auth-design.md

@@ -0,0 +1,37 @@
+# Fisherman Live-Session Auth — Design
+
+## Problem
+
+Langfuse trace `de95bd1cffce09169599d99d1bee56cd` (2026-08-30, beta.testomat.io, project `zyntra-don-t-touch-cloned`): every Fisherman write returned `403 {"error":"Unauthorized"}` in ~8ms while the same browser session performed successful writes to the same project minutes earlier. The requests were rejected at the auth layer because Fisherman assembled stale credentials:
+
+1. `cookieProvider` (`src/explorbot.ts`) serialized `page.context().cookies()` — the **entire jar, unfiltered** — producing a doubled `Cookie` header with a dead `localhost` session pair ahead of the valid one. A real browser never sends localhost cookies to another host.
+2. `extractAuthHeaders` (`src/api/request-store.ts`) scraped `x-csrf-token` from a **2026-07-07 capture of another project** (`imr_manual12`). The store iterates `capturedRequests` from the end, but `loadFromDisk` fills it in alphabetical filename order, so "last" means alphabetically-late, not newest. `output/requests/` is a graveyard spanning months and many projects, so any stale credential can win.
+3. `refreshAuth` (`src/ai/fisherman.ts`) applies live cookies **before** capture-scraped headers, so a captured `cookie` header (in `AUTH_HEADERS`) could even clobber the fresh jar.
+
+## Principle
+
+**Captures are a source of API shape — endpoints, body examples — which is durable across sessions. They are never a source of credentials, which are ephemeral.** Credentials come from the live browser session or from explicit `api.headers` config. This rules out any future re-accretion of header scraping from old captures.
+
+## Decisions
+
+- **D1 — Cookies from the live jar, filtered by the API origin.** `page.context().cookies(baseEndpoint)`: Playwright applies the same domain/path matching a browser applies when sending to that URL. No manual dedup — same-name cookies on parent/child domains are legitimate browser behavior and Playwright's filter already yields exactly what the browser would send.
+- **D2 — Never scrape cookies from captures.** `'cookie'` leaves `AUTH_HEADERS`. The jar is the single, always-current source of cookies.
+- **D3 — Auth headers only from current-session captures, newest first.** `RequestStore` records `sessionStartedAt` at construction; `extractAuthHeaders` considers only captures with `timestamp >= sessionStartedAt`, sorted newest-first. Live captures (added by `XhrCapture` during this run) pass; the disk graveyard never does. This also resolves same-id collisions (an old `xhr_001_…` disk file vs a live capture reusing that counter id): the gate keeps only the live one.
+- **D4 — Precedence: captured < live browser < config.** `refreshAuth` applies session-capture headers first, live browser headers second (current session replaces old), explicit `configHeaders` last (user intent stays authoritative). Still refreshed once per episode — the browser is idle while Fisherman runs.
+- **D4a — Replicate mode only.** Browser-derived credentials (jar cookies, page CSRF token, session-capture headers) apply only in replicate mode, where Fisherman replays what the browser does. In achieve mode the API contract is explicit and authentication comes solely from `api.headers` config — injecting browser cookies there would be surprising and can leak a UI session into a separately-authenticated API.
+- **D5 — Live CSRF token from the page (severable).** The provider also reads `meta[name="csrf-token"]` from the current page and sends it as `x-csrf-token`. This is a cross-framework web convention (Rails, Laravel), the same class of structural knowledge as ARIA attributes or URL anatomy — not a site-specific locator. `cookieProvider` is renamed `browserHeaderProvider` since it now supplies all live-browser-derived headers. Cutting this decision cuts only Task 4 of the plan; Tasks 1–3 stand alone.
+- **D6 — A capture without a timestamp is stale.** `RequestResult.load` currently stamps load-time for a file missing `timestamp`, which would slip past the session gate; absent timestamp now parses as epoch.
+
+## Declared behavior change
+
+`Authorization` / `x-api-key` values scraped from **previous-session** captures are no longer sent. Cookie-authenticated apps are unaffected (the jar is live). An app that was only ever authenticated through a stale captured token now fails honestly instead of sending dead credentials — the remedy is `api.headers` in config.
+
+## Out of scope
+
+- Prompt changes to make the model stop faster on `authorization` failures (the 4-failure guard already bounds it).
+- Pilot's final verdict misattributing the failure to later tester errors instead of the failed precondition.
+- Mirroring the `XSRF-TOKEN` cookie into an `X-XSRF-TOKEN` header (Angular/Laravel convention) — add only if a real trace shows it's needed.
+
+## Acceptance
+
+A run against a cookie-authenticated app where Fisherman's writes carry only the cookies the current browser session would send to the API origin, plus a current CSRF token — verified by the request ledger in `output/requests/` showing a single-valued `Cookie` header matching the live session and no header value originating from a previous session's captures.

package/docs/workflow/agentic-usage.md

@@ -56,8 +56,10 @@ No `init`, no config file, no project directory, no model IDs to look up. These
 | `EXPLORBOT_EPHEMERAL` | no | Keep no state between runs — output goes to a fresh temp directory instead of the site dir |
 | `EXPLORBOT_KNOWLEDGE` | no | Inline knowledge text, applied to every page |
 | `EXPLORBOT_KNOWLEDGE_FILE` | no | Path to a knowledge markdown file |
+| `EXPLORBOT_SPEC` | no | Docbot application spec directory or index.md, used as page knowledge |
 | `EXPLORBOT_API_SPEC` | no | OpenAPI spec path for the API boat |
 | `EXPLORBOT_NO_BANNER` | no | Suppress the startup banner, for machine-readable output |
+| `EXPLORBOT_MAX_DURATION` | no | Wall-clock budget in minutes for an explore run; same as --max-duration |
 <!-- END env -->
 
 `EXPLORBOT_URL` is optional when the command itself carries an absolute URL, as `docs collect https://…` does. The [API boat](../api-testing/basics.md) reads it as the base endpoint.
@@ -123,6 +125,12 @@ EXPLORBOT_AI_PROVIDER=openrouter \
 EXPLORBOT_KNOWLEDGE_FILE=./checkout-knowledge.md npx explorbot explore /checkout
 ```
 
+Both variables work in config-free runs and in runs on the global configuration, where what they carry is written into the site's knowledge directory for that run — rewritten on the next run, and removed by a run that sets neither variable. Facts worth keeping belong in `learn` or `know`. The `--knowledge` flag does the same thing as an argument, works with a project config as well, and writes nothing, so prefer it when one command needs one fact:
+
+```bash
+npx explorbot explore /checkout --knowledge 'Use the sandbox card 4111 1111 1111 1111'
+```
+
 ### What this mode changes
 
 Config-free runs leave no trace in the working directory:
@@ -212,9 +220,11 @@ The same variables drive API testing and doc collection.
 EXPLORBOT_URL=https://api.example.com \
 EXPLORBOT_API_SPEC=./openapi.yaml \
 EXPLORBOT_AI_PROVIDER=openrouter \
-  npx explorbot api explore
+  npx explorbot api explore /users
 ```
 
+The API boat also takes those two as flags, so one line carries the whole run: `npx explorbot api explore /users --endpoint https://api.example.com --spec ./openapi.yaml`.
+
 ```bash
 EXPLORBOT_AI_PROVIDER=openrouter \
   npx explorbot docs collect https://app.example.com/dashboard --max-pages 20
@@ -222,7 +232,7 @@ EXPLORBOT_AI_PROVIDER=openrouter \
 
 `docs collect` takes its base URL from the absolute path argument, so `EXPLORBOT_URL` is optional there.
 
-Knowledge written by `EXPLORBOT_KNOWLEDGE` carries `endpoint: '*'` frontmatter alongside `url: '*'`, matching the convention `api init` and `api know` use. The API boat does not read knowledge at runtime yet; the frontmatter is there for when it does, and the web side ignores it.
+Knowledge written by `EXPLORBOT_KNOWLEDGE` carries `endpoint: '*'` frontmatter alongside `url: '*'`, matching the convention `api init` and `api know` use, so one variable reaches both boats.
 
 ## See Also
 

package/docs/workflow/ci.md

@@ -9,6 +9,7 @@ Once your Explorbot setup works locally — it logs in, explores, and produces s
 - **A headless browser.** That's the default — don't pass `--show` and there is nothing to configure.
 - **A start path and a test budget.** `npx explorbot explore / --max-tests 10` keeps the run bounded and predictable.
 - **A hard timeout as backstop.** AI runs can stall; a job-level timeout guarantees the pipeline never hangs.
+- **A time budget below that timeout.** Pass `--max-duration` (or set `EXPLORBOT_MAX_DURATION`) a few minutes under the job timeout: Explorbot stops starting new tests, wraps up the in-flight one, writes the report, and finalizes the Testomatio run on its own — instead of being killed mid-test, which loses the report. The budget minus a 3-minute teardown reserve is the hard cutoff; 5 minutes before that, no new work starts.
 
 For login, commit your `knowledge/` directory to the repo — it's the input you control, and Explorbot reads it on every run. Keep credentials out of the files with `${env.NAME}` interpolation, as described in [Knowledge](./knowledge.md), and set those variables as pipeline secrets too. Add `--session` to the run command so Explorbot logs in once and reuses the saved session.