agent-sanitizer 2.60.3 → 2.61.1

package/THREAT-MODEL.md

@@ -30,11 +30,22 @@ table](./README.md#entry-points) maps each to its import.
 
 - A **single leading BOM** (a legitimate marker); interior BOMs are stripped.
 - **ZWNJ (U+200C) / ZWJ (U+200D)** in genuine linguistic context: between two
-  letters of a script whose orthography requires them (Arabic, Devanagari,
-  Bengali, Gurmukhi, Gujarati, Oriya, Tamil, Telugu, Kannada, Malayalam,
-  Sinhala) or inside an emoji ZWJ sequence. The carve-out fires only when **both**
+  letters of a script whose orthography requires them (the Arabic-family
+  cursive scripts by Joining_Type, and every script that owns a Unicode
+  virama — Devanagari through Sinhala, plus Brahmi, Grantha, Javanese,
+  Balinese, Newa and the rest — after a virama on a consonant), a ZWJ between
+  a consonant and the virama after it (the Bengali RA with ya-phala, the
+  Marathi eyelash RA), or inside an emoji ZWJ sequence.
+  The carve-out fires only when **both**
   neighbors clearly belong to the context, and it is disabled once the total
   invisible count crosses a scatter floor—over-stripping beats under-stripping.
+- **An emoji glyph's own spelling** draws on no preserve budget: the VS16 on a
+  text-default emoji (❤, ⚠), a ZWJ that UAX #29's GB11 keeps inside one
+  grapheme (🏳️‍🌈, 👨‍👩‍👧‍👦), and a registered subdivision flag's tags. A VS15, a
+  VS16 on an emoji-default base, and a ZWJ GB11 does not join stay metered.
+  Residuals: one visible bit per text-default emoji (VS16 or not), and about one
+  hidden bit per adjacent pair of pictographs (an unregistered ZWJ like 😀‍😀
+  renders as two glyphs either way). Past the scatter floor all are stripped.
 - **Blank fillers doing real work in their own script**: the Braille blank
   (U+2800) beside a real cell, a Hangul filler beside a real jamo/syllable. A
   _run_ of fillers has only fillers for neighbors, so it fails the anchor and is
@@ -64,8 +75,11 @@ does not run a fixed sequence of passes; it iterates the whole
 deletes at least one character), and only once that is stable does it sweep any
 residual raw control introducer—7-bit ESC (U+001B) or any 8-bit C1 control
 (U+0080–U+009F)—outright. Sweeping earlier would strand a hidden control as
-visible text; a final unconditional sweep after the loop keeps the
-no-raw-introducer guarantee independent of the iteration bound. The result
+visible text. Input that outruns the iteration bound—escapes hidden behind
+selectors the carve-out keeps until each removal shrinks its budget—falls back
+to a carve-out-free strip of every invisible, every sequence and every
+introducer, so neither guarantee below depends on the bound; the carve-out is
+given up only on input built to outrun it. The result
 carries no raw ANSI introducer for _any_ input, and re-cleaning it reproduces
 it exactly—the idempotence the Edit-repair rehydrator's soundness gate assumes.
 One tokenizer answers every ANSI question (what to splice, and whether what was
@@ -309,15 +323,27 @@ one in a command spoofs what the user is shown as it runs.
 Display-only SGR colour is the exception, and is preserved byte-for-byte. The
 SGR grammar is closed — parameters and a final `m` — so it restyles visible text
 and can do nothing else; stripping it costs a model the colourized fixture, TUI
-golden file or prompt string it deliberately wrote, and buys no safety. Two
-shapes end the exception, both of them the model-sees/human-sees divergence
+golden file or prompt string it deliberately wrote, and buys no safety. Three
+shapes end the exception, all of them the model-sees/human-sees divergence
 rather than a display choice: a CONCEAL parameter (`ESC[8m`), which blanks the
-text that follows for a human while its bytes stay readable to a model, and a
-run of sequences with nothing that RENDERS between them, which puts no glyph on
-the screen at all and is therefore carrying data rather than colour. Either one
-strips the whole field.
-
-Both shapes are read as terminal semantics rather than token shapes, because a
+text that follows for a human while its bytes stay readable to a model; a
+foreground painted in its own background (`ESC[30;40m`) while a glyph goes by,
+which hides it the same way; and a run of sequences with nothing that RENDERS
+between them, which puts no glyph on the screen at all and is therefore
+carrying data rather than colour. Conceal or a matching foreground still in
+force when the text ENDS counts too: rendition outlives its bytes. Any one
+strips the whole field, and the prompt gate blocks on the same answer. The
+invisible verdict reads the de-ANSI'd view on both surfaces, since SGR bytes
+render nothing and must not buy preserved joiners and selectors.
+
+The colour match is judged only where unambiguous: a default colour matches
+nothing, bold over a basic colour may render brighter, and whitespace hides
+nothing. Block Elements (U+2580–U+259F) are judged across the whole text: in
+matching colours each paints a solid cell, so one glyph repeated (a half-block
+image's flat region) is styling, while a second distinct one is a hidden
+channel.
+
+All three are read as terminal semantics rather than token shapes, because a
 reader that matches shapes is one encoding away from the wrong answer. Conceal
 is threaded as STATE, so a reveal or reset later in the same sequence cancels it
 (`ESC[8;28;31m` renders red and visible) while an erase between two sequences
@@ -338,7 +364,10 @@ its colour.
 Both protections run to a fixed point over each field rather than once each,
 because they feed one another: removing an invisible character completes a
 sequence that was incomplete when the terminal stage ran (`ESC[`⟨ZWSP⟩`2J`), and
-removing a sequence makes invisibles adjacent that were not.
+removing a sequence makes invisibles adjacent that were not. A field still
+changing at the round bound gets the same carve-out-free strip as Layer 1's
+fallback rather than being written unfinished, and a field the strip leaves
+byte for byte is reported as untouched, never as a rewrite.
 
 ## Confusable folding (tool input)
 

package/claude-hooks/lib/authored-content.mjs

@@ -7,9 +7,10 @@
  *      TAG block used for ASCII smuggling and zero-width joiners) and variation
  *      selectors can encode a hidden message that another AI reading the
  *      committed file / PR / commit later decodes, while staying invisible to a
- *      human reviewer. Stripped when payload-capable (see isPayloadCapable):
- *      gated on volume because incidental joiners / emoji selectors are benign
- *      and authored content is *persisted*, so over-stripping is costly.
+ *      human reviewer. Stripped when the engine's payloadVerdict calls the field
+ *      payload-capable: gated on volume because incidental joiners / emoji
+ *      selectors are benign and authored content is *persisted*, so
+ *      over-stripping is costly.
  *
  *   2. Terminal-display rewriting. ANSI/terminal control sequences (CSI/OSC)
  *      authored into a command — echoed and executed live — or into file
@@ -22,8 +23,9 @@
  *      carry an OSC payload, so stripping it only costs a model the colourized
  *      fixture, TUI golden file or prompt string it deliberately wrote. The
  *      carve-out ends where SGR stops being styling (see the engine's
- *      sgrCarriesPayload): a CONCEAL parameter blanks text for a human while a
- *      model still reads it, and a long run of sequences with nothing that
+ *      sgrCarriesPayload): a CONCEAL parameter or a foreground painted in its
+ *      own background blanks text for a human while a model still reads it,
+ *      and a long run of sequences with nothing that
  *      renders between them puts no glyph on the screen at all, which is a
  *      covert channel rather than colour.
  *
@@ -57,11 +59,11 @@ import { readKnob } from "./knobs.mjs";
 // static import of this module, before its fail-closed catch runs). A failed
 // load leaves these bindings undefined, so sanitizeField's calls throw into
 // the fail-closed catch (ask) instead.
-const { stripAnsiFully, isBenignAnsiKinds, sgrCarriesPayload } =
+const { stripAnsiFully, stripTerminalFully, escapesPreservable } =
   /** @type {typeof import("agent-sanitizer")} */ (
     await lazyImport("agent-sanitizer")
   );
-const { STRIP, SCATTERED_THRESHOLD, hasLongRun, stripInvisible } =
+const { payloadVerdict, stripInvisibleWithReport } =
   /** @type {typeof import("agent-sanitizer/invisible")} */ (
     await lazyImport("agent-sanitizer/invisible")
   );
@@ -143,32 +145,14 @@ export function authoredScopeDecision(tool) {
   return { kind: "undeclared" };
 }
 
-// Payload-capable: a long contiguous run, or enough scattered invisibles to
-// carry a message. Mirrors sanitize-user-prompt so the model→world and
-// user→model surfaces share one definition of "stego payload".
-/** @param {string} text */
-function isPayloadCapable(text) {
-  // hasLongRun, not a pattern built here: the engine's scan is bounded per
-  // `exec`, which is what keeps an 8 MB run of zero-widths in a Write body from
-  // throwing `RangeError: Maximum call stack size exceeded` out of this hook.
-  if (hasLongRun(text)) return true;
-  return (text.match(STRIP)?.length ?? 0) >= SCATTERED_THRESHOLD;
-}
-
 /**
  * The de-ANSI'd `text`, or `text` itself when its escapes are display-only
- * colour a reader authored on purpose.
- *
- * The two questions are answered by two different views, deliberately.
- * `isBenignAnsiKinds` reads the kinds the strip actually REMOVED, so a sequence
- * that only reconstitutes mid-strip is judged as the sequence it becomes rather
- * than as the incomplete one it started as. `sgrCarriesPayload` reads the RAW
- * bytes, because those are the bytes a terminal renders on the preserve path —
- * it makes no second pass, so there is nothing to reconstitute there.
+ * colour a reader authored on purpose — the engine's `escapesPreservable`, the
+ * same answer the prompt gate gives, judged on the kinds THIS strip removed.
  *
  * Only COMPLETE tokens reach `kinds`: an introducer that opens no sequence is
  * left in place by `stripAnsiFully` (it rewrites no display), so the orphan arms
- * of `isBenignAnsiKinds` never decide anything here.
+ * of the inert-kind test never decide anything here.
  * @param {string} text
  * @returns {string}
  */
@@ -177,7 +161,7 @@ function stripTerminalControls(text) {
   const kinds = new Set();
   const deAnsi = stripAnsiFully(text, kinds);
   if (deAnsi === text) return text;
-  if (isBenignAnsiKinds(kinds) && !sgrCarriesPayload(text)) return text;
+  if (escapesPreservable(text, kinds)) return text;
   return deAnsi;
 }
 
@@ -187,11 +171,36 @@ function stripTerminalControls(text) {
 // ZWSP goes), and removing a sequence makes invisibles adjacent that were not.
 // One round of each therefore leaves the composition unfinished. Each round
 // deletes at least one character, so the loop converges on its own; the bound is
-// a DoS guard. No residual sweep stands behind it — unlike the engine's
-// MAX_LAYER1_PASSES — so an unconverged value would persist a live sequence into
-// the file, and reaching the bound while still changing throws instead.
+// a DoS guard. Reaching it still changing falls back to settleField, never to
+// the unconverged value, which could persist a live sequence into the file.
 const MAX_FIELD_PASSES = 3;
 
+/**
+ * The carve-out-free strip a field that did not converge falls back to: every
+ * invisible, then every escape and raw introducer, each where its protection
+ * is on — a fixed point of sanitizeField, on adversarial input only.
+ * @param {string} value  the field as the caller sent it (for the leading-BOM rule)
+ * @param {string} cleaned  the loop's last state
+ * @param {Set<string>} actions  collects the protections that changed the field
+ * @returns {string}
+ */
+function settleField(value, cleaned, actions) {
+  let settled = cleaned;
+  if (!readKnob("AGENT_SANITIZER_INVISIBLE_DISABLED")) {
+    const bulk = stripInvisibleWithReport(settled, value, {
+      carveOut: false,
+    }).cleaned;
+    if (bulk !== settled) actions.add("invisible characters");
+    settled = bulk;
+  }
+  if (!readKnob("AGENT_SANITIZER_TERMINAL_DISABLED")) {
+    const swept = stripTerminalFully(settled);
+    if (swept !== settled) actions.add("terminal-control sequences");
+    settled = swept;
+  }
+  return settled;
+}
+
 // Returns the cleaned value plus the human-readable actions applied, or null if
 // the field is already clean. Each protection has its own opt-out (see the
 // header) so a deployment can keep one while dropping the other.
@@ -216,23 +225,31 @@ function sanitizeField(value) {
       }
     }
 
-    if (
-      !readKnob("AGENT_SANITIZER_INVISIBLE_DISABLED") &&
-      isPayloadCapable(cleaned)
-    ) {
-      cleaned = stripInvisible(cleaned);
-      actions.add("invisible characters");
+    // The engine's payloadVerdict on the de-ANSI'd view, as the prompt gate
+    // judges it: SGR the colour carve-out kept renders nothing, so it must not
+    // count as cover that buys preserved joiners and selectors.
+    if (!readKnob("AGENT_SANITIZER_INVISIBLE_DISABLED")) {
+      const view = stripAnsiFully(cleaned);
+      if (payloadVerdict(view).payloadCapable) {
+        // A payload forfeits the colour carve-out, so the strip reads the view
+        // the verdict read — unless the terminal opt-out keeps the escapes.
+        const source = readKnob("AGENT_SANITIZER_TERMINAL_DISABLED")
+          ? cleaned
+          : view;
+        const stripped = stripInvisibleWithReport(source, cleaned).cleaned;
+        if (source !== cleaned) actions.add("terminal-control sequences");
+        if (stripped !== source) actions.add("invisible characters");
+        cleaned = stripped;
+      }
     }
     if (cleaned === before) break;
-    // This refusal is what blocks a still-changing value — one whose next round
+    // This fallback is what blocks a still-changing value — one whose next round
     // would remove another live sequence — from being written to the file.
     if (pass === MAX_FIELD_PASSES - 1)
-      throw new Error(
-        "authored content did not reach a fixed point within MAX_FIELD_PASSES",
-      );
+      cleaned = settleField(value, cleaned, actions);
   }
 
-  return actions.size > 0 ? { cleaned, actions: [...actions] } : null;
+  return cleaned === value ? null : { cleaned, actions: [...actions] };
 }
 
 /** @param {string[]} changed */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-sanitizer",
-  "version": "2.60.3",
+  "version": "2.61.1",
   "description": "Defend an agent against hidden-content injection: strip payload-capable invisible Unicode and ANSI, splice out human-invisible HTML, and flag data-exfil URLs in untrusted text before any model sees it.",
   "type": "module",
   "repository": {
@@ -52,6 +52,7 @@
     "@stryker-mutator/core": "^9.6.1",
     "@stryker-mutator/tap-runner": "^9.6.1",
     "@types/node": "25.9.1",
+    "@unicode/unicode-17.0.0": "2.0.7",
     "acorn": "^8.18.0",
     "acorn-walk": "^8.3.5",
     "agent-sanitizer": "link:.",
@@ -247,6 +248,8 @@
     "typecheck": "tsc --noEmit && tsc -p tsconfig.hooks.json --noEmit",
     "build:types": "tsc -p tsconfig.build.json && tsc -p tsconfig.build-hooks.json",
     "gen:joining-type": "node scripts/gen-joining-type.mjs",
+    "gen:unicode": "node scripts/gen-unicode.mjs",
+    "gen:unicode-properties": "node scripts/gen-unicode-properties.mjs",
     "gen:fail-open-lib": "node scripts/gen-fail-open-lib.mjs",
     "gen:node-floor-lib": "node scripts/gen-node-floor-lib.mjs",
     "gen:hook-binaries": "node plugin/scripts/build-hook-binaries.mjs",

package/python/agent_sanitizer/data/invisible-charset.json

@@ -1,6 +1,6 @@
 {
-  "_comment": "SSOT for the payload-capable invisible code points, generated by scripts/gen-invisible-charset.mjs. `extra_codepoints` are the non-Cf extras (variation selectors, blank-rendering fillers, zero-width combining marks) from src/invisible.mjs (VS + BLANK_NON_CF). `cf_codepoints` is the general-category Cf set PINNED from Node's Unicode data at generation time (see `unicode_version`) — NOT resolved live per consumer, because Node and CPython ship different Unicode versions and a live-Cf split let a code point in the version delta escape one layer. The deletion set is the UNION of the two lists. `control_introducers` is the raw ANSI control-introducer set (ESC + the C1 block) from src/ansi.mjs, which Layer 1 sweeps and the Python textstrip port must sweep identically. `escape_sequence_pattern` is the escape GRAMMAR from that same module, as a regex source valid in JS and in Python `re` with no flags: src/ansi.mjs's scanner is the authoritative implementation, and a stdlib-only consumer compiles this rather than hand-writing a second spelling of it. Consumers in other languages read this file instead of forking the lists — a fork is a silent security regression.",
-  "unicode_version": "17.0",
+  "_comment": "SSOT for the payload-capable invisible code points, generated by scripts/gen-invisible-charset.mjs. `extra_codepoints` are the non-Cf extras (variation selectors, blank-rendering fillers, zero-width combining marks) from src/invisible.mjs (VS + BLANK_NON_CF). `cf_codepoints` is the general-category Cf set PINNED from the UCD package at generation time (see `unicode_version`) — NOT resolved live per consumer, because Node and CPython ship different Unicode versions and a live-Cf split let a code point in the version delta escape one layer. The deletion set is the UNION of the two lists. `control_introducers` is the raw ANSI control-introducer set (ESC + the C1 block) from src/ansi.mjs, which Layer 1 sweeps and the Python textstrip port must sweep identically. `escape_sequence_pattern` is the escape GRAMMAR from that same module, as a regex source valid in JS and in Python `re` with no flags: src/ansi.mjs's scanner is the authoritative implementation, and a stdlib-only consumer compiles this rather than hand-writing a second spelling of it. Consumers in other languages read this file instead of forking the lists — a fork is a silent security regression.",
+  "unicode_version": "17.0.0",
   "extra_codepoints": [
     847,
     4447,

package/src/ansi.mjs

@@ -515,15 +515,67 @@ export function scanAnsi(text) {
 }
 
 /**
- * The CONCEAL state after the SGR token `token` is applied to a terminal already
- * in state `concealed` — the state in which a terminal renders what follows as
- * blank while its bytes stay readable to anything reading the file.
+ * What a terminal's graphic rendition looks like after an SGR token: whether
+ * CONCEAL is on, whether BOLD is (which many terminals render as a brighter
+ * foreground), and the foreground/background colour as a comparable key — `i<n>`
+ * for a palette index (`31` and `38;5;1` are both `i1`), `rgb:<r>,<g>,<b>` for
+ * direct colour, `null` for the default or a colour this reader cannot size.
+ * @typedef {{ concealed: boolean, bold: boolean, fg: string | null, bg: string | null }} SgrRendition
+ */
+
+/** The rendition of a terminal no SGR has touched (and of `ESC[0m`).
+ * @type {Readonly<SgrRendition>} */
+export const DEFAULT_RENDITION = Object.freeze({
+  concealed: false,
+  bold: false,
+  fg: null,
+  bg: null,
+});
+
+const SGR_BOLD = 1;
+const SGR_NORMAL_INTENSITY = 22;
+const SGR_FOREGROUND = 38;
+const SGR_BACKGROUND = 48;
+const SGR_INDEXED = 5;
+const SGR_DIRECT = 2;
+
+/**
+ * The palette key a basic colour parameter sets on the layer at `base` (30
+ * foreground, 40 background), `null` for its default, undefined for none.
+ * @param {number} value @param {number} base
+ * @returns {string | null | undefined}
+ */
+function basicColour(value, base) {
+  if (value >= base && value <= base + 7) return `i${value - base}`;
+  if (value >= base + 60 && value <= base + 67) return `i${value - base - 52}`;
+  if (value === base + 9) return null;
+  return undefined;
+}
+
+/**
+ * The colour key extended-colour arguments name (`5;n` palette, `2;r;g;b`
+ * direct, either separator), or `null` — never equal to another colour.
+ * @param {string[]} args  the parameters after the selector, colour space first
+ * @returns {string | null}
+ */
+function extendedColour(args) {
+  const space = Number(args[0]);
+  if (space === SGR_INDEXED && args.length >= 2) return `i${Number(args[1])}`;
+  if (space === SGR_DIRECT && args.length >= 4)
+    return `rgb:${args
+      .slice(-3)
+      .map((component) => Number(component))
+      .join(",")}`;
+  return null;
+}
+
+/**
+ * The rendition after the SGR token `token` is applied to a terminal already in
+ * `state`.
  *
- * A TRANSITION, not a property of the token: conceal is terminal state that
+ * A TRANSITION, not a property of the token: rendition is terminal state that
  * outlives the sequence that set it, so `ESC[8m` followed by `ESC[31m` is still
- * concealed — a per-token predicate would read the second token as "no conceal
- * here" and lose the state. Only `8` (set), `28` (reveal) and `0` (reset all)
- * move it, and the token's parameters are applied in order, so a later reveal or
+ * concealed. The token's parameters are applied in order, so a later reveal or
  * reset in the SAME token cancels an earlier `8`: `ESC[8;28;31m` renders red and
  * visible.
  *
@@ -531,33 +583,60 @@ export function scanAnsi(text) {
  * the digit. `38`/`48`/`58` take their colour arguments from the parameters that
  * FOLLOW them in the semicolon form, so `ESC[38;5;8m` is bright-black foreground
  * and its `8` is a palette index; a parameter carrying its arguments as ITU T.416
- * sub-parameters (`38:5:8`) is self-contained, so only its head counts and
- * nothing after it is consumed. A colour-space selector T.416 gives no argument
- * count for is malformed, and a terminal that ignores the colour form still
- * applies what follows it, so the scan CONTINUES from the next parameter rather
- * than consuming arguments it cannot size — reading one parameter too many costs
- * a token shape no emitter produces, while stopping there would hand
- * `ESC[38;9;8m` a pass.
+ * sub-parameters (`38:5:8`) is self-contained, so nothing after it is consumed.
+ * A colour-space selector T.416 gives no argument count for is malformed, and a
+ * terminal that ignores the colour form still applies what follows it, so the
+ * scan CONTINUES from the next parameter rather than consuming arguments it
+ * cannot size — stopping there would hand `ESC[38;9;8m` a pass.
  * @param {string} token  a token {@link scanAnsi} classified {@link TOKEN_KIND.SGR}
- * @param {boolean} concealed  the state before this token
- * @returns {boolean}
+ * @param {Readonly<SgrRendition>} state  the rendition before this token
+ * @returns {SgrRendition}
  */
-export function sgrConcealState(token, concealed) {
+export function sgrRendition(token, state) {
   // The parameter bytes: everything between the introducer (two characters in
   // the 7-bit `ESC [` form, one in the 8-bit C1 form) and the final `m`.
   const params = token
     .slice(token.charCodeAt(0) === ESC ? 2 : 1, -1)
     .split(";");
-  let state = concealed;
+  let { concealed, bold, fg, bg } = state;
   for (let i = 0; i < params.length; i++) {
-    const colonForm = params[i].includes(":");
+    const subParams = params[i].split(":");
     // An omitted parameter is 0 (`ESC[m` is `ESC[0m`), which `Number("")` gives.
-    const value = Number(params[i].split(":")[0]);
-    if (value === SGR_CONCEAL) state = true;
-    else if (value === SGR_REVEAL || value === SGR_RESET) state = false;
-    if (colonForm || !SGR_EXTENDED_COLOUR.has(value)) continue;
-    const args = SGR_COLOUR_ARGS.get(Number(params[i + 1]));
-    if (args !== undefined) i += 1 + args;
+    const value = Number(subParams[0]);
+    if (value === SGR_RESET) ({ concealed, bold, fg, bg } = DEFAULT_RENDITION);
+    else if (value === SGR_CONCEAL) concealed = true;
+    else if (value === SGR_REVEAL) concealed = false;
+    else if (value === SGR_BOLD) bold = true;
+    else if (value === SGR_NORMAL_INTENSITY) bold = false;
+    const foreground = basicColour(value, 30);
+    if (foreground !== undefined) fg = foreground;
+    const background = basicColour(value, 40);
+    if (background !== undefined) bg = background;
+    if (!SGR_EXTENDED_COLOUR.has(value)) continue;
+    const colonForm = subParams.length > 1;
+    const count = SGR_COLOUR_ARGS.get(Number(params[i + 1]));
+    if (!colonForm && count === undefined) continue; // unsized: consume nothing
+    const args = colonForm
+      ? subParams.slice(1)
+      : params.slice(i + 1, i + 2 + /** @type {number} */ (count));
+    if (!colonForm) i += args.length;
+    const colour = extendedColour(args);
+    if (value === SGR_FOREGROUND) fg = colour;
+    else if (value === SGR_BACKGROUND) bg = colour;
   }
-  return state;
+  return { concealed, bold, fg, bg };
+}
+
+/**
+ * True when text drawn in `rendition` is invisible to a human because its
+ * foreground IS its background (`ESC[30;40m`, `ESC[38;5;7;48;5;7m`). Only an
+ * unambiguous match counts: a default colour is unknown, so it matches nothing,
+ * and BOLD over a basic colour is drawn brighter by many terminals, so
+ * `ESC[1;30;40m` is not called hidden — the precision direction.
+ * @param {Readonly<SgrRendition>} rendition
+ * @returns {boolean}
+ */
+export function foregroundMatchesBackground({ bold, fg, bg }) {
+  if (fg === null || fg !== bg) return false;
+  return !(bold && /^i[0-7]$/u.test(fg));
 }

package/src/cf-charset.mjs

@@ -1,19 +1,19 @@
 /**
- * GENERATED by scripts/gen-invisible-charset.mjs from Node's Unicode data
- * (\p{Cf}, Unicode 17.0) — DO NOT EDIT.
+ * GENERATED by scripts/gen-invisible-charset.mjs from the pinned UCD
+ * (General_Category=Cf, Unicode 17.0.0) — DO NOT EDIT.
  *
  * The general-category Cf code points, PINNED at generation time. invisible.mjs
- * strips exactly this set instead of testing \p{Cf} live, and the Python port
+ * strips exactly this set instead of asking the runtime for Cf, and the Python port
  * reads the SAME set from data/invisible-charset.json's `cf_codepoints`, so both
  * layers strip an identical Cf set regardless of each runtime's own Unicode
  * version. Regenerate with `node scripts/gen-invisible-charset.mjs`;
- * test/invisible-charset.test.mjs fails if this drifts from Node's \p{Cf}.
+ * test/invisible-charset.test.mjs fails if this drifts from the pinned UCD.
  */
 
-export const UNICODE_VERSION = "17.0";
+export const UNICODE_VERSION = "17.0.0";
 
 // GENERATED Unicode data, not hand-written logic. Its correctness is pinned by
-// the SSOT round-trip in test/invisible-charset.test.mjs (re-enumerate \p{Cf},
+// the SSOT round-trip in test/invisible-charset.test.mjs (re-derive the Cf set,
 // then assert every code point matches) — Stryker skips the numeric literals.
 // Stryker disable all
 /** @type {readonly number[]} Sorted ascending. */

package/src/gates.mjs

@@ -88,7 +88,7 @@ export function needsUrlScan(text) {
 // can't be retried at every offset; the atlasv1 arm in SECRET_HINT does the same.
 /** @type {RegExp} */
 export const SECRET_HINT =
-  /secret|token|password|passwd|pwd|bearer|credential|authorization|contrase[nñ]a|-----BEGIN|(?:api|auth|service|account|db|database|priv|private|client|access)[_-]?key|(?:db|database|key)[_-]?pass|(?:A3T|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}|gh[pousr]_[A-Za-z0-9]|github_pat_|gl[a-z]{2,12}-[0-9A-Za-z_-]{20}|sk-ant-|AIza[0-9A-Za-z_-]{35}|sk_live_|sk_test_|rk_live_|rk_test_|xox[bpasr]-|eyJ[A-Za-z0-9]|do[opr]_v1_[a-f0-9]{16}|v1\.0-[a-f0-9]{24}-|hv[sb]\.[A-Za-z0-9_-]{20}|(?<![a-z0-9])[a-z0-9]{14}\.atlasv1\.|sk-or-v1-[0-9a-f]{16}|gsk_[A-Za-z0-9]{16}|xai-[A-Za-z0-9]{16}|r8_[A-Za-z0-9]{16}/i;
+  /secret|token|password|passwd|pwd|bearer|credential|authorization|contrase[nñ]a|-----BEGIN|(?:api|auth|service|account|db|database|priv|private|client|access)[_-]?key|(?:db|database|key)[_-]?pass|(?:A3T|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}|gh[pousr]_[A-Za-z0-9]|github_pat_|gl[a-z]{2,12}-[0-9A-Za-z_-]{20}|sk-ant-|AIza[0-9A-Za-z_-]{35}|sk_live_|sk_test_|rk_live_|rk_test_|xox[bpasr]-|eyJ[A-Za-z0-9]|do[opr]_v1_[a-f0-9]{16}|v1\.0-[a-f0-9]{24}-|hv[sb]\.[A-Za-z0-9_-]{20}|(?<![a-z0-9])[a-z0-9]{14}\.atlasv1\.|sk-or-|gsk_[A-Za-z0-9]{16}|xai-[A-Za-z0-9]{16}|r8_[A-Za-z0-9]{16}/i;
 
 // Second alternation (see SECRET_HINT): kept a separate literal so a redos
 // analyzer vets each alternation in isolation.

package/src/index.mjs

@@ -26,10 +26,12 @@ import { sanitizeText } from "./output.mjs";
 export {
   applyLayer1,
   applyLayer1WellFormed,
+  escapesPreservable,
   isBenignAnsi,
   isBenignAnsiKinds,
   normalizeLoneSurrogates,
   stripAnsiFully,
+  stripTerminalFully,
   sgrCarriesPayload,
   SGR_RUN_THRESHOLD,
   LONE_SURROGATE_RE,

package/src/instructions.mjs

@@ -40,7 +40,8 @@ import { join, relative, resolve, isAbsolute, dirname, sep } from "node:path";
 import {
   findLongRuns,
   SCATTERED_THRESHOLD,
-  countPayloadInvisible,
+  payloadInvisibleView,
+  payloadVerdict,
   stripInvisible,
 } from "./invisible.mjs";
 import { excludeNodeModules } from "./claude-context.mjs";
@@ -209,15 +210,21 @@ export function decodeRun(run) {
 }
 
 /**
- * Scan a file's text for hidden-Unicode injection. Reports each long invisible
+ * Scan a file's text for hidden-Unicode injection. Reports each payload long
  * run (with its decoded payload) plus a single scattered-chars finding when the
- * non-run invisible count crosses the threshold-evasion floor.
+ * effective invisible count outside those runs crosses the threshold-evasion
+ * floor.
+ *
+ * Whether there is anything to report is {@link payloadVerdict}'s call, the
+ * one every other surface makes.
  * @param {string} content
  * @returns {Array<{ line: number | null, charCount: number, method: string, decoded: string }>}
  *   `line` is the 1-based line of a long-run finding, or `null` for the
  *   whole-file scattered-chars finding (not tied to a single line).
  */
 export function scanText(content) {
+  const verdict = payloadVerdict(content);
+  if (!verdict.payloadCapable) return [];
   const findings = [];
   let runChars = 0;
   // The line number is carried forward across runs. Deriving it per run from
@@ -228,7 +235,13 @@ export function scanText(content) {
   // order, so this scan only ever moves forward.
   let line = 1;
   let counted = 0;
-  for (const run of findLongRuns(content)) {
+  // The verdict keeps only the first run; the rest come off the payload view,
+  // which is index-aligned with `content` and masks preserved invisibles.
+  const runs =
+    verdict.firstRun === null
+      ? []
+      : findLongRuns(payloadInvisibleView(content));
+  for (const run of runs) {
     for (; counted < run.index; counted++)
       if (content.charCodeAt(counted) === NEWLINE) line++;
     runChars += run.charCount;
@@ -240,27 +253,9 @@ export function scanText(content) {
   }
 
   // Threshold-evasion: scattered invisible chars not in a long run can still be
-  // a payload. Always evaluated; chars already in a run are excluded so they
-  // aren't double-counted. countPayloadInvisible is invisible.mjs's carve-out
-  // counter (the SSOT the stripper itself gates on): it discounts every
-  // invisible that does real rendering work — emoji presentation selectors
-  // (VS16 *and* VS15) on a real pictograph, emoji-sequence ZWJ, and linguistic
-  // ZWNJ/ZWJ between cursive letters or after a virama. A hand-rolled emoji-only
-  // mirror lived here before and over-counted linguistic joiners and VS15, so a
-  // ZWNJ-dense Persian doc or a doc of text-presentation hearts (❤︎) tripped a
-  // scattered false positive on content stripInvisible would preserve.
-  //
-  // Asymmetry (deliberate, benign): the minuend discounts preserved
-  // selectors/joiners EVERYWHERE, while `runChars` is each run's RAW length. A
-  // long run is ≥LONG_RUN_THRESHOLD *consecutive* invisibles, and a preserved
-  // selector/joiner is always flanked by a visible neighbor — so it cannot sit
-  // inside such a run, and the two counts describe disjoint chars in practice.
-  // In the pathological case that they don't (e.g. a run of stacked VS16), the
-  // raw `runChars` subtracts at most a few more than the minuend added, biasing
-  // `scattered` slightly LOW — a false negative, the precision-favoring
-  // direction, never a spurious finding. `scattered` may even go negative; the
-  // `>=` gate treats that as "no scatter", which is correct.
-  const scattered = countPayloadInvisible(content) - runChars;
+  // a payload. Every run above is made of invisibles the verdict's effective
+  // count includes, so subtracting them leaves exactly the unreported ones.
+  const scattered = verdict.invisibleCount - runChars;
   if (scattered >= SCATTERED_THRESHOLD) {
     findings.push({
       line: null, // whole-file finding: scattered chars aren't tied to one line