a11y-loop 0.2.4 → 0.2.6

package/.claude-plugin/marketplace.json

@@ -6,13 +6,13 @@
   },
   "metadata": {
     "description": "Makes AI coding agents decide accessibility while the work is still being planned and write accessible UI by default, then verify it in a real browser across the states they built — and say exactly what could not be checked. Ships the a11y-loop Agent Skill, a /a11y-plan command, and a plan-mode gate that declines a UI plan with no accessibility content in it.",
-    "version": "0.2.4"
+    "version": "0.2.6"
   },
   "plugins": [
     {
       "name": "a11y-loop",
       "source": "./",
-      "version": "0.2.4",
+      "version": "0.2.6",
       "description": "Accessibility in the plan, not after the pull request. Adds §0 plan rules (conformance target, per-component criteria, the product decisions that foreclose accessibility, color tokens before components), standing WCAG 2.2 AA generation rules, and a PreToolUse hook on ExitPlanMode that declines a UI-touching plan with no accessibility content — once, handing back the section to fill in. Verification is the a11y-loop CLI: axe-core in Chromium across default, dark, forced-colors, reduced-motion and 320px passes.",
       "author": {
         "name": "Chan Meng",

package/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "a11y-loop",
   "displayName": "a11y-loop",
-  "version": "0.2.4",
+  "version": "0.2.6",
   "description": "Hard-gates plan mode: a plan that changes UI has to say what it decided about accessibility before it can be approved. Ships the a11y-loop skill and a /a11y-plan command alongside it.",
   "author": {
     "name": "Chan Meng",

package/README.md

@@ -49,6 +49,7 @@ check.
 - [🎥 Promo Video](#-promo-video)
 - [✨ Key Features](#-key-features)
 - [📊 Honest Coverage](#-honest-coverage)
+- [🔬 Field Notes](#-field-notes)
 - [📈 Benchmark](#-benchmark)
 - [🛠️ Tech Stack](#-tech-stack)
 - [🏗️ Architecture](#-architecture)
@@ -80,6 +81,13 @@ a11y-loop is two things working together:
 2. **A Node CLI** (`a11y-loop audit` / `contrast --fix` / `diff`) that verifies the result in a
    real browser, feeds failures back to the agent to fix, and re-audits until the loop converges.
 
+**About the name.** `a11y` is the standard numeronym for *accessibility*: the first letter, the last
+letter, and the 11 letters between them replaced by their count — the same construction as `i18n`
+(internationalization) and `k8s` (Kubernetes). Spelled out it reads as "ally", which is the pun the
+accessibility community has always liked, but `11` is the conventional form. `loop` is the other half
+of the method: the CLI is not a one-shot audit but a write → audit → fix → re-audit cycle that runs
+until the report converges.
+
 The skill follows the open [Agent Skills](https://agentskills.io/specification) standard, so the
 same `SKILL.md` is portable across 40+ clients — Claude Code, Cursor, GitHub Copilot, Codex,
 Gemini CLI, and more — not just one vendor's agent. The CLI runs the same checks standalone in CI
@@ -152,7 +160,9 @@ graph LR
 `2` **Checks axe-core can't run** — tab order, focus visibility (including focus-ring contrast),
 dialog focus trap / Escape / focus-return, target size (24×24 CSS px, SC 2.5.8), reduced-motion
 effectiveness, ambiguous link text, div-as-button, and positive `tabindex`. These sit alongside
-axe-core, not instead of it.
+axe-core, not instead of it. The dialog and tab-order checks understand **portalled** dialogs —
+a `<div role="dialog">` in a portal, the way Base UI, Radix, Headless UI and anything on floating-ui
+ships one — not only a native `<dialog>` opened with `showModal()`.
 
 `3` **State coverage, not just page-load** — `--interact` drives the states an agent just built
 (a modal opened, a menu expanded, a form in its error state) through the same five passes, because
@@ -225,6 +235,10 @@ of the other ~87%.
 - axe-core's own `incomplete` results are surfaced as `needsReview`, not suppressed. See
   `evals/benchmark-results.md` for a live example of two such findings being investigated and
   resolved rather than dismissed.
+- **The engine is wrong sometimes, and the honest thing is to say where.** A handful of rows have
+  a known non-defect explanation — contrast in the forced-colors pass being the loudest. They are
+  written up in [Field Notes](#-field-notes), with what to check before dismissing one. Verifying a
+  row is not the same as suppressing it, and a11y-loop suppresses nothing.
 - **SARIF caveat, stated honestly:** GitHub Code Scanning only displays SARIF results that carry a
   file-path location. a11y-loop's findings are located by rendered URL + CSS selector, which
   Code Scanning drops on ingestion — a naive upload produces an empty Code Scanning view. SARIF
@@ -235,6 +249,96 @@ of the other ~87%.
   Section 508 to the 2.0/2.1 AA subset, EU EN 301 549 to 2.1 AA (moving to 2.2 AA around October
   2026), NZ and UK to the full 2.2 AA set.
 
+## 🔬 Field Notes
+
+Things a report says that are not what they look like. These came out of running a11y-loop against
+a production app across a full audit-fix cycle; each one cost an hour to work out from the report
+alone, so it is written down here. Every number below is reproducible from a checkout at axe-core
+4.12.1.
+
+### `color-contrast` in the forced-colors pass is usually not a defect
+
+axe-core reads an element's foreground from `-webkit-text-fill-color`, falling back to `color` only
+when that is unset. Chromium's forced-colors emulation forces `color` and the background, and
+leaves `-webkit-text-fill-color` at the author's value — on every element, including ones that
+never declared it. So in the forced-colors pass axe compares the author's foreground against the
+forced background.
+
+Dark-on-light text survives that comparison. Light-on-dark text collapses:
+
+| Element | Default pass | forced-colors pass |
+|---|---|---|
+| `#6d6975` text on `#ffffff` | 4.9:1, silent | silent |
+| `#e8e6ef` text on a `#141218` card | 14:1, silent | `color-contrast`, 1.23:1 — *"foreground color: #e8e6ef, background color: #ffffff"* |
+
+The tell is inside the finding: the reported foreground is the author's color while the background
+is the forced `#ffffff` or `#000000`. On a page with any dark surface this arrives as a cluster
+covering every line of text on it. On one production page it accounted for the large majority of a
+153-finding run.
+
+**Do not apply the suggestion.** It is computed from the same mismatched pair: for the row above it
+proposes darkening `#e8e6ef` to `#77757d`, which then fails in the default pass. **Do not suppress
+the rule either.** Confirm the pair in the default and dark passes, check forced colors by eye or
+with the OS setting, and say in the hand-off that forced colors was reviewed rather than measured.
+
+### `div-button` on `<body>` or a portal root is an outside-press handler
+
+Popup libraries dismiss on a press outside the popup by attaching a click handler to the document
+body or to the portal container. `div-button` looks for exactly that shape — a click handler on an
+element with no role, no `tabindex` and no keyboard path — so while a popup is open it reports
+`html > body` and the portal root.
+
+The keyboard equivalent of "press outside to dismiss" is Escape, which those libraries wire
+separately. So the question the row should send you to is `dialog-escape-does-not-close`: if
+Escape dismisses the popup, the `div-button` row on the body is a false positive of this check.
+It is a real failure only when that handler is the only way to activate something.
+
+### Contrast sampled mid-animation measures a frame, not the design
+
+An element fading in is composited at partial opacity, and axe flattens that against what is behind
+it. Measured on a production page: `#6d6975` text at opacity 0.917 was read as `#797580`. The
+difference is small, but it is enough to move a value that sits near 4.5:1 across the line, and
+enough for two runs of the same page to disagree.
+
+If a contrast finding names a color you cannot find anywhere in the source, this is usually why.
+`openPage()` waits 120 ms after load for webfonts and short entrance animations; a longer reveal
+outlives it. Re-run, have the `--interact` state wait for the animation to settle before it
+returns, or read the number off the reduced-motion pass — which is the stable one, provided the
+app honors the preference rather than ignoring it.
+
+### `keyboard-unreachable` can mean the Tab budget ran out
+
+The keyboard survey presses Tab up to `MAX_TAB_STEPS` (60) times, then reports everything
+`tabbable()` expected but the walk never reached. Past 60 tab stops in one state, the tail of the
+document is reported unreachable because the budget ended, not because focus could not get there.
+The rule's own message names the budget.
+
+A cluster of unreachable findings that is contiguous and sits at the end of document order is the
+signature. Confirm by hand with the five-minute keyboard test in
+[`references/manual-testing.md`](skill/a11y-loop/references/manual-testing.md), and audit narrower
+states so each walk fits.
+
+### Invoking the CLI through a Windows junction silently does nothing
+
+`src/cli.js` runs `main()` only when `import.meta.url === pathToFileURL(process.argv[1]).href`.
+Node resolves `import.meta.url` through a junction to the real path while `process.argv[1]` keeps
+the literal path you typed, so the guard is false, nothing runs, and the process exits **0 with no
+output** — which reads like a pass:
+
+```bash
+node D:\link-to-repo\src\cli.js --version   # prints nothing, exit 0
+node D:\github_repository\a11y-loop\src\cli.js --version   # 0.2.6
+```
+
+This bites git worktrees and any checkout reached through a junction or symlink. Use the real path,
+the installed `a11y-loop` binary, or import the entry point and call it directly — `main` is
+exported for this:
+
+```js
+import { main } from './src/cli.js';
+process.exitCode = await main(['audit', 'http://localhost:3000', '--json']);
+```
+
 ## 📈 Benchmark
 
 A small, illustrative comparison in [`evals/benchmark-results.md`](evals/benchmark-results.md):
@@ -263,9 +367,9 @@ illustration of the effect's shape, not a controlled study or a precise effect s
 - **SARIF conversion:** [axe-sarif-converter](https://github.com/microsoft/axe-sarif-converter)
 - **Agent integration:** the open [Agent Skills](https://agentskills.io/specification) standard —
   `SKILL.md` + `references/`, no client-proprietary format
-- **Tests:** the built-in `node --test` runner, 337+ tests across unit and integration suites,
-  including an 18-fixture seeded-violation matrix, a demo end-to-end run, and a forced-colors
-  gradient regression test
+- **Tests:** the built-in `node --test` runner, 378 tests across unit and integration suites,
+  including a 19-fixture matrix of seeded violations and correct-behaviour regression guards, a
+  demo end-to-end run, and a forced-colors gradient regression test
 
 ## 🏗️ Architecture
 
@@ -284,7 +388,7 @@ graph TD
     C --> F["diff<br/>FIXED / NEW / REMAINING"]
     D --> G["Playwright + Chromium"]
     G --> H["axe-core"]
-    G --> I["a11y-loop's own checks<br/>focus, dialog trap, target size,<br/>reduced motion, link text, div-button"]
+    G --> I["a11y-loop's own checks<br/>focus, dialog trap (native and portalled),<br/>target size, reduced motion,<br/>link text, div-button"]
     D -->|--interact| J["Drive built states:<br/>modal open, menu expanded, form error"]
     H --> K["JSON report<br/>WCAG SC + ACT IDs + provenance"]
     I --> K

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "a11y-loop",
-  "version": "0.2.4",
+  "version": "0.2.6",
   "description": "Makes AI coding agents decide accessibility while the work is still being planned and write accessible UI by default, then proves what it can prove with a real browser audit across the states they built — and tells you exactly what it could not check.",
   "type": "module",
   "bin": {

package/skill/a11y-loop/SKILL.md

@@ -28,7 +28,7 @@ compatibility: >-
   The generation and honesty rules apply with or without the CLI; every step
   that says "audit" requires it.
 metadata:
-  "a11y-loop/version": "0.2.4"
+  "a11y-loop/version": "0.2.6"
 allowed-tools: 'Bash(a11y-loop *) Bash(npx a11y-loop *) Bash(node ${CLAUDE_SKILL_DIR}/../../src/cli.js *)'
 ---
 
@@ -274,6 +274,13 @@ ignored reduced motion, SPA focus loss, unannounced updates, positive
    Opened a modal? Added an error state? Added a route, a tab panel, an
    expanded menu, an async list? Each one is a state. Audit it.
 
+   While a modal dialog is open the keyboard survey is scoped to that dialog —
+   a native `<dialog>` opened with `showModal()`, or a portalled
+   `<div role="dialog">` that has hidden the page around it, which is what
+   every popup library ships. That is deliberate: inside an open modal, the
+   page behind it is not supposed to be reachable. Give the page behind it its
+   own state if you need it audited.
+
    ```
    A11Y audit http://localhost:5173 --interact .a11y/states.mjs --json --out .a11y/run-1.json
    ```
@@ -282,6 +289,17 @@ ignored reduced motion, SPA focus loss, unannounced updates, positive
    needsReview item is a place the engine knows it could not decide — usually
    contrast over a gradient, image, or translucent layer. Resolve it by
    reasoning about the source, not by ignoring it.
+
+   A few rows have a known non-defect explanation: a `color-contrast` cluster
+   in the forced-colors pass, a `div-button` on `<body>` while a popup is open,
+   a contrast value sampled while something was animating, a run of
+   `keyboard-unreachable` after a long Tab walk.
+   [references/manual-testing.md](references/manual-testing.md) lists each with
+   its tell. Check a row against it before changing code — fixing a false
+   positive breaks working UI, and the suggested color on a forced-colors
+   contrast row fails in the default pass. Verifying a row is not suppressing
+   it: never disable a rule, and say in your report which rows you verified and
+   how.
 4. **Fix in the source files**, never in the report or by suppressing a rule.
    For contrast findings, use the report's `suggestions[]` or
    `A11Y contrast <fg> <bg> --fix`, which returns both a lighter and a darker

package/skill/a11y-loop/references/manual-testing.md

@@ -112,6 +112,44 @@ content is any good.
   notwithstanding"), which means it stays silent wherever it is unsure. Silence
   is not a pass.
 
+### Rows with a known non-defect explanation
+
+Four of these have been seen on a production app. Check a row against them
+before changing code for it — fixing a false positive breaks working UI — and
+then say in the report which rows you verified and how. Verifying a row is not
+the same as suppressing it, and nothing here is a reason to disable a rule.
+
+- **`color-contrast` in the forced-colors pass is usually not a defect.** axe
+  reads an element's foreground from `-webkit-text-fill-color` before `color`.
+  Chromium's forced-colors emulation forces `color` and the background but
+  leaves `-webkit-text-fill-color` at the author's value, on every element,
+  so the pass compares the author's foreground against the forced background.
+  Dark-on-light text survives that; light-on-dark text reads near 1:1 and every
+  line on a dark surface is reported. The tell is a finding whose foreground is
+  a color you can find in your source and whose background is `#ffffff` or
+  `#000000`. Confirm the pair in the default and dark passes, check forced
+  colors by eye or with the OS setting, and report it as reviewed rather than
+  measured. **Never apply the suggested color** — it is derived from the same
+  mismatched pair and fails in the default pass.
+- **A click handler on `<body>` or on a portal root is how popup libraries
+  dismiss on an outside press**, and a keyboard-path check sees a clickable
+  element with no role, no `tabindex` and no keyboard handling. The keyboard
+  equivalent of "press outside" is Escape, wired separately. Check that Escape
+  closes the popup; if it does, the row on the body is explained.
+- **Contrast measured while something is animating is a measurement of one
+  frame.** An element fading in is composited at partial opacity and flattened
+  against what is behind it: at opacity 0.917, `#6d6975` was read as `#797580`.
+  That is enough to move a value near 4.5:1 across the line and enough for two
+  runs of the same page to disagree. If a finding names a color that is nowhere
+  in your source, this is usually why. Let entrance animations finish before
+  the state returns, or read the number from the reduced-motion pass if the
+  page honors the preference.
+- **A Tab walk has a budget** — 60 presses. Anything the walk did not reach
+  inside it is reported unreachable because the budget ended, not because focus
+  could not get there. A contiguous cluster at the end of document order is the
+  signature; confirm it by hand with the keyboard test below, and audit
+  narrower states so each walk fits.
+
 ---
 
 ## A five-minute keyboard test anyone can run
@@ -168,6 +206,39 @@ from the specification, which is why the APG says outright that testing with
 real assistive technology is essential and that its own examples target spec
 compliance rather than AT bug workarounds.
 
+### Automating a screen-reader pass
+
+A screen-reader pass can be scripted — [Guidepup](https://www.guidepup.dev/)
+drives NVDA on Windows and VoiceOver on macOS — and it is worth doing, because
+it turns "focus moves somewhere and something is announced" into a transcript
+you can quote and re-run. Four conditions, each of which decides whether the
+output is evidence or decoration:
+
+- **Drive a portable copy of the screen reader, not the installed one**, so a
+  run cannot inherit a person's own settings. Pin its language explicitly
+  (NVDA: `general.language`, e.g. `en`) — it otherwise follows the OS locale,
+  and a transcript in an unexpected language is unreadable as evidence.
+- **Set the synthesizer to `silence`.** The spoken text is what you are
+  capturing; the audio is not, and rendering it makes runs slow and
+  machine-dependent.
+- **A driver's relay log is focus speech, not everything spoken.** Guidepup's
+  `spokenPhraseLog()` reports what it was handed as focus moved. Live-region
+  announcements — `role="status"`, `role="alert"`, `aria-live` — do not
+  reliably appear there, so their absence proves nothing. A claim that a live
+  region spoke has to come from the screen reader's own debug log (NVDA:
+  `nvda.log` at DEBUG level), where the announcement is recorded by the screen
+  reader itself. Anything less is an assumption about the one mechanism most
+  likely to silently not fire.
+- **Copy the log out before stopping the screen reader.** `nvda.stop()` removes
+  the temporary config directory the log lives in; read it afterwards and you
+  will find nothing.
+
+What this buys you is a regression you can detect and a transcript you can
+quote. What it does not buy you is a judgment about whether the interface is
+usable — that is still answered only by people who use a screen reader every
+day. Report an automated pass as what it is: one screen reader, one version,
+one set of states.
+
 ---
 
 ## Involving disabled users

package/src/lib/browser-utils.js

@@ -133,6 +133,10 @@ export const HELPERS_SOURCE = `
     // this, every check that surveys "visible" elements would flood a report
     // with the rest of the page once a modal is open — not a real defect,
     // just this helper not knowing the platform already handled it.
+    //
+    // A PORTALLED dialog needs no equivalent clause here: it makes the rest of
+    // the page unreachable by marking it aria-hidden, which the check above
+    // already catches. modalDialogRoot() is what reads that shape.
     var modal = document.querySelector(':modal');
     if (modal && !modal.contains(el)) return false;
     return true;
@@ -190,15 +194,62 @@ export const HELPERS_SOURCE = `
     return 'rgb(255, 255, 255)';
   }
 
+  /**
+   * Has the page been taken out of reach around this element? That is how a
+   * portalled dialog declares modality when it does not set aria-modal: the
+   * library marks the dialog's body-level SIBLINGS aria-hidden (or inert) and
+   * leaves its own portal alone. Base UI sets no aria-modal attribute at all
+   * and relies on this entirely, which is the currently recommended pattern.
+   */
+  function outsideIsHidden(el) {
+    var top = el;
+    while (top.parentElement && top.parentElement !== document.body) top = top.parentElement;
+    return Array.prototype.some.call(document.body.children, function (sibling) {
+      if (sibling === top || sibling.contains(el)) return false;
+      if (!isRendered(sibling)) return false;
+      return sibling.getAttribute('aria-hidden') === 'true' || sibling.hasAttribute('inert');
+    });
+  }
+
+  /**
+   * The open modal dialog, or null. The :modal pseudo-class is the cheap
+   * answer and the right one for a native <dialog> opened with showModal() —
+   * but it is ONLY that. Every React popup library (Base UI, Radix, Headless
+   * UI, floating-ui) ships a portalled <div role="dialog"> instead, which
+   * :modal never matches, so anything relying on it alone silently treats an
+   * open sheet as though no dialog were there at all.
+   *
+   * The portalled shape is recognised by what it actually does: a visible
+   * dialog-role element that is itself exposed, and that has either declared
+   * aria-modal="true" or hidden the rest of the page around itself. The LAST
+   * such element in document order wins, because a portal appends to <body> —
+   * so the most recently opened dialog is the topmost one.
+   */
+  function modalDialogRoot() {
+    var native = document.querySelector(':modal');
+    if (native) return native;
+
+    var open = Array.prototype.filter.call(
+      document.querySelectorAll('[role="dialog"], [role="alertdialog"], [aria-modal="true"]'),
+      function (el) {
+        if (el.getAttribute('aria-modal') === 'false') return false;
+        if (!isVisible(el)) return false;
+        return el.getAttribute('aria-modal') === 'true' || outsideIsHidden(el);
+      },
+    );
+    return open.length ? open[open.length - 1] : null;
+  }
+
   /**
    * Where a forward Tab walk should be scoped: inside the open modal dialog
-   * if there is one (everything outside it is inert and cannot be reached),
-   * otherwise the whole document. tabbable() itself has no idea a native
-   * dialog element made the rest of the page inert, so callers must pass
-   * this as the container rather than always walking document.body.
+   * if there is one (everything outside it is unreachable while it is open),
+   * otherwise the whole document. tabbable() has no idea a dialog is open —
+   * it reads display and visibility, not aria-hidden, and knows nothing about
+   * :modal — so callers must pass this as the container rather than always
+   * walking document.body.
    */
   function tabbableRoot() {
-    return document.querySelector(':modal') || document.body;
+    return modalDialogRoot() || document.body;
   }
 
   /** The CSS path of whatever was last clicked, or null if nothing was. */
@@ -212,6 +263,7 @@ export const HELPERS_SOURCE = `
     accessibleName: accessibleName,
     isRendered: isRendered,
     isVisible: isVisible,
+    modalDialogRoot: modalDialogRoot,
     tabbableRoot: tabbableRoot,
     lastClickSelector: lastClickSelector,
     interactiveElements: interactiveElements,

package/src/lib/checks/dialog.js

@@ -16,6 +16,28 @@ import { makeFinding, SEVERITY } from '../finding.js';
 /** Tab presses used to probe a focus trap. */
 export const TRAP_PROBE_STEPS = 12;
 
+/**
+ * How many consecutive presses may land outside the dialog on something no user
+ * can act on before it stops counting as a wrap and starts counting as an escape.
+ *
+ * Not a fudge factor — it is the length of a real wrap, measured. A portalled
+ * dialog wraps through a CHAIN: floating-ui (Base UI, Radix) renders a focus
+ * guard beside the floating element and another pair at the edges of <body>, and
+ * hands focus back an animation frame later, so a forward Tab off the end of the
+ * sheet goes guard → body → back inside. Observed on a live Base UI sheet, three
+ * runs: two of them spent two presses out (guard, then body) and one spent a
+ * single press. Allowing exactly one press, as this check used to, therefore
+ * reported a perfectly trapped dialog as broken about two runs in three — which
+ * is what made the row look flaky rather than wrong.
+ *
+ * It stays small on purpose, and it only applies AFTER focus has been inside the
+ * dialog at least once: presses spent outside before that are the probe finding
+ * its way in from wherever the earlier surveys left focus, not a wrap. Landing on
+ * a genuine, exposed element outside the dialog is an escape on the first press
+ * either way, whatever this bound says.
+ */
+export const MAX_WRAP_TICKS = 3;
+
 /**
  * @param {object} observation
  * @param {string} observation.selector
@@ -196,6 +218,11 @@ export async function captureDialogInitialState(page, opts = {}) {
         selector: helpers.cssPath(dialog),
         html: helpers.shortHtml(dialog),
         trigger,
+        // Was this dialog MODAL at the moment it opened? Recorded now because
+        // it is the only way to tell, later, between a dialog that never
+        // trapped anything and one whose modal treatment was torn down while
+        // the probe was running. See surveyDialog.
+        modal: helpers.modalDialogRoot() === dialog,
         focusMovedIntoDialog: Boolean(active && dialog.contains(active)),
         initialFocus: active ? helpers.cssPath(active) : null,
       };
@@ -204,6 +231,26 @@ export async function captureDialogInitialState(page, opts = {}) {
   );
 }
 
+/**
+ * Let the page finish reacting to a key before reading where focus went.
+ *
+ * A focus trap built on requestAnimationFrame — floating-ui's enqueueFocus, and
+ * so Base UI's and Radix's — hands focus back one frame after a guard receives
+ * it, and cancels a pending hand-back when another arrives. Synthetic Tab
+ * presses can be dispatched faster than a frame, which cancels the redirect over
+ * and over and leaves focus parked outside the dialog: the probe outruns the
+ * page and then reports the page for not keeping up. No keyboard user can press
+ * Tab twice inside one frame, so waiting two frames does not weaken the check —
+ * it stops it measuring itself.
+ *
+ * @param {import('playwright').Page} page
+ */
+async function settleFrames(page) {
+  await page.evaluate(
+    () => new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve))),
+  );
+}
+
 /**
  * Probe focus trap, Escape, and focus return, given a dialog snapshot already
  * captured by `captureDialogInitialState`. This drives the keyboard and
@@ -217,22 +264,66 @@ export async function captureDialogInitialState(page, opts = {}) {
 export async function surveyDialog(page, found) {
   if (!found) return null;
 
+  // The probe asks exactly one question: with focus INSIDE this dialog, can Tab
+  // take it out? So it has to start inside. By the time it runs, three other
+  // surveys have each walked the page with real Tab presses and left focus
+  // wherever they happened to finish — and on a portalled dialog, focus sitting
+  // outside is itself enough for the library's focus-out handling to begin
+  // dismantling the modal treatment it put up. The probe would then walk a page
+  // that is no longer behind a modal and report a working trap as broken.
+  //
+  // Measured on a live Base UI sheet, ten runs: without this, four began outside
+  // the dialog and every one of those four reported the trap broken. With it,
+  // ten of ten began inside and none did.
+  const enteredDialog = await page.evaluate(
+    ({ selector }) => {
+      const helpers = window.__a11yLoop;
+      const dialog = Array.from(document.querySelectorAll(selector)).find((el) =>
+        helpers.isVisible(el),
+      );
+      if (!dialog) return false;
+      if (dialog.contains(document.activeElement)) return true;
+      const stops = window.tabbable ? window.tabbable.tabbable(dialog) : [];
+      if (!stops.length) return false;
+      stops[0].focus();
+      return dialog.contains(document.activeElement);
+    },
+    { selector: DIALOG_SELECTOR },
+  );
+
   // Focus trap: Tab repeatedly and watch for focus leaving the dialog.
   //
-  // Real Chromium, for a genuinely native <dialog> opened with showModal(),
-  // transiently moves document.activeElement to document.body for exactly
-  // one Tab press when wrapping past the dialog's last (or before its first)
-  // focusable descendant, then redirects back inside on the very next press —
-  // body itself stays inert throughout, so nothing is actually reachable
-  // there. Treating that single step as an escape would flag a correctly
-  // trapped native dialog as broken. It is only a real failure if landing
-  // outside the dialog is on a genuine element, or if it does not recover on
-  // the immediately following press.
-  let focusTrapped = true;
+  // Every modal dialog spends one tick outside itself when it wraps, and the
+  // shape of that tick depends on who implements the trap:
+  //
+  //  - Real Chromium, for a genuinely native <dialog> opened with showModal(),
+  //    transiently moves document.activeElement to document.body for exactly
+  //    one Tab press when wrapping past the dialog's last (or before its first)
+  //    focusable descendant, then redirects back inside on the very next press.
+  //    Body itself stays inert throughout, so nothing is reachable there.
+  //  - A portalled <div role="dialog"> — Base UI, Radix, Headless UI, anything
+  //    on floating-ui — wraps through a focus guard instead: a focusable span
+  //    outside the dialog, hidden from assistive technology, whose whole job is
+  //    to catch the wrap and hand focus back. Nothing is reachable there either.
+  //
+  // Treating either as an escape flags a correctly trapped dialog as broken,
+  // and that is precisely what happened to portalled dialogs while this gate
+  // asked `:modal` — which matches only the native case. It is a real failure
+  // if focus lands outside the dialog on a genuine, exposed element, or if it
+  // does not come back within MAX_WRAP_TICKS presses: a portalled dialog wraps
+  // through a chain of those ticks, not a single one.
+
+  // A dialog focus cannot be placed inside is a dialog whose trap this check
+  // cannot judge, so it says so with null rather than guessing — dialogFindings
+  // reports only on an explicit false.
+  let focusTrapped = enteredDialog ? true : null;
   let escapedTo = null;
-  let sawUnconfirmedTransient = false;
-  for (let i = 0; i < TRAP_PROBE_STEPS; i++) {
+  let insideAtLeastOnce = false;
+  let consecutiveWrapTicks = 0;
+  let tornDownMidProbe = false;
+  for (let i = 0; enteredDialog && i < TRAP_PROBE_STEPS; i++) {
     await page.keyboard.press('Tab');
+    await settleFrames(page);
     const check = await page.evaluate(
       ({ selector }) => {
         const helpers = window.__a11yLoop;
@@ -240,13 +331,31 @@ export async function surveyDialog(page, found) {
           helpers.isVisible(el),
         );
         const active = document.activeElement;
-        if (!dialog || !active) return { inside: false, transient: false, label: null };
-        if (dialog.contains(active)) return { inside: true, transient: false, label: null };
+        if (!dialog || !active) {
+          return { inside: false, transient: false, label: null, stillModalRoot: false };
+        }
+        if (dialog.contains(active)) {
+          return { inside: true, transient: false, label: null, stillModalRoot: true };
+        }
         const isBodyOrRoot = active === document.body || active === document.documentElement;
-        const stillModal = typeof dialog.matches === 'function' && dialog.matches(':modal');
+        // A focus guard is an EMPTY element that is not exposed to assistive
+        // technology — every library builds one the same way, as a bare
+        // aria-hidden span with nothing in it. Emptiness is what separates it
+        // from the page behind the dialog, which is also aria-hidden but is
+        // full of real content: focus landing THERE is a trap that failed, and
+        // must still be reported.
+        const isFocusGuard =
+          !isBodyOrRoot &&
+          !helpers.isVisible(active) &&
+          active.children.length === 0 &&
+          !(active.textContent || '').trim();
+        const stillModal =
+          (typeof dialog.matches === 'function' && dialog.matches(':modal')) ||
+          helpers.modalDialogRoot() === dialog;
         return {
           inside: false,
-          transient: isBodyOrRoot && stillModal,
+          transient: (isBodyOrRoot || isFocusGuard) && stillModal,
+          stillModalRoot: helpers.modalDialogRoot() === dialog,
           label: isBodyOrRoot ? 'the browser UI / document root' : helpers.cssPath(active),
         };
       },
@@ -254,18 +363,40 @@ export async function surveyDialog(page, found) {
     );
 
     if (check.inside) {
-      sawUnconfirmedTransient = false;
+      insideAtLeastOnce = true;
+      consecutiveWrapTicks = 0;
       continue;
     }
-    if (check.transient && !sawUnconfirmedTransient) {
-      sawUnconfirmedTransient = true;
+    if (check.transient && (!insideAtLeastOnce || consecutiveWrapTicks < MAX_WRAP_TICKS)) {
+      if (insideAtLeastOnce) consecutiveWrapTicks += 1;
       continue;
     }
+    // The dialog was modal when it opened and is not any more: the page took
+    // the modal treatment down — the backdrop, the aria-hidden on everything
+    // behind it — while this probe was running, usually because the sheet is
+    // closing. Focus is free to walk the page because the page let it, not
+    // because a trap failed. There is nothing to judge here, so judge nothing:
+    // reporting a trap failure off a dialog that stopped being modal mid-probe
+    // is how a correctly trapped sheet ends up accused about one run in seven.
+    if (found.modal && !check.stillModalRoot) {
+      tornDownMidProbe = true;
+      break;
+    }
     focusTrapped = false;
     escapedTo = check.label;
     break;
   }
 
+  if (tornDownMidProbe) {
+    return {
+      ...found,
+      focusTrapped: null,
+      escapedTo: null,
+      escapeClosed: null,
+      focusReturnedToTrigger: null,
+    };
+  }
+
   await page.keyboard.press('Escape');
   await page.waitForTimeout(150);
 

package/src/lib/checks/keyboard.js

@@ -155,10 +155,12 @@ export function keyboardFindings(survey, ctx) {
  * matters whenever a state setup function has clicked something, which is most of
  * the time. A focused sentinel at the start of the body fixes the origin.
  *
- * "Top of the document" is not always `document.body`: if a native `<dialog>`
- * is open, everything outside it is inert, so a sentinel inserted into `body`
- * could never be focused at all. The sentinel goes into whichever element
- * `tabbableRoot()` says is actually reachable right now.
+ * "Top of the document" is not always `document.body`: if a modal dialog is
+ * open, everything outside it is out of reach, so a sentinel inserted into
+ * `body` is either unfocusable (a native `<dialog>`) or sits in the very region
+ * the dialog's focus trap exists to keep focus out of (a portalled one — as far
+ * from the sheet as the document allows). The sentinel goes into whichever
+ * element `tabbableRoot()` says is actually reachable right now.
  *
  * @param {import('playwright').Page} page
  */