agent-sanitizer 2.45.9 → 2.47.0

package/README.md

@@ -60,17 +60,18 @@ Split into subpaths so the heavy HTML dependency stays opt-in. **Seam** names
 the callback you inject for the agent-specific concern; `—` is a pure transform,
 `fs (direct)` does its own file I/O instead of taking one.
 
-| #   | Import          | Purpose                                                                                                                                                                                  | Seam                        |
-| --- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
-| 1   | `/invisible`    | Strip zero-width, bidi, variation-selector and tag chars + ANSI/SGR escapes. Preserves ZWNJ/ZWJ for Arabic/Indic/emoji. Zero deps.                                                       | —                           |
-| 2   | `/html`         | Splice out HTML comments and elements hidden via `display:none`, off-screen, white-on-white, `hidden`. Each splice leaves a keyed, round-trippable placeholder.                          | —                           |
-| 3   | `/html`         | Detect exfil-shaped URLs (payloads in query/path, embedded creds, `data:`/`javascript:`, off-origin redirects) and confusable HOSTS (`аpple.com`). Reports only — never rewrites.        | —                           |
-| 4   | `/confusables`  | Fold look-alike glyphs in tool-call input (paths, commands) to ASCII, closing a cross-script deny-rule bypass. Gated per token, so non-Latin prose passes through unfolded.              | `scan` (optional)           |
-| 5   | `/instructions` | Scan/auto-clean `CLAUDE.md`, `AGENTS.md`, `SKILL.md`, etc., decoding Unicode-tag + zero-width-binary payloads.                                                                           | `fs` (direct)               |
-| 6   | `/prompt`       | Classify a prompt pass / note / block on payload-capable invisible/ANSI content (inert escapes get the note).                                                                            | —                           |
-| 7   | `/output`       | Run Layers 1–4 over structured tool output, preserving shape. The Layer-5 slot takes a delete-only filter.                                                                               | `redact`, `filterInjection` |
-| 8   | `/rehydrate`    | Re-anchor a model Edit or whole-file Write composed from the _sanitized_ view back onto real bytes; gate MultiEdit on a verified view==disk; deny anything ambiguous or secret-exposing. | `io`                        |
-| —   | `/view-map`     | Pure offset/text machinery mapping a file's on-disk bytes ↔ the sanitized view (Layer-1 deletions, Layer-4 redactions). No I/O — consumed by `/rehydrate`.                               | —                           |
+| #   | Import          | Purpose                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Seam                        |
+| --- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
+| 1   | `/invisible`    | Strip zero-width, bidi, variation-selector and tag chars + ANSI/SGR escapes. Preserves ZWNJ/ZWJ for Arabic/Indic/emoji. Zero deps.                                                                                                                                                                                                                                                                                                                                                                                                     | —                           |
+| 1   | `/layer1`       | Layer 1 in one call: the invisible-char and ANSI strips run to a FIXED POINT over each other (either removal can reconstitute what the other matches), then a residual control-introducer sweep closes it — not a two-step pipeline a caller can reproduce in a fixed order. `applyLayer1` leaves an unpaired UTF-16 surrogate in place; `applyLayer1WellFormed` also maps it to U+FFFD, which is the string `/output` shows a model — take that one when the result feeds a redactor, an offset calculation or a view map. Zero deps. | —                           |
+| 2   | `/html`         | Splice out HTML comments and elements hidden via `display:none`, off-screen, white-on-white, `hidden`. Each splice leaves a keyed, round-trippable placeholder.                                                                                                                                                                                                                                                                                                                                                                        | —                           |
+| 3   | `/html`         | Detect exfil-shaped URLs (payloads in query/path, embedded creds, `data:`/`javascript:`, off-origin redirects) and confusable HOSTS (`аpple.com`). Reports only — never rewrites.                                                                                                                                                                                                                                                                                                                                                      | —                           |
+| 4   | `/confusables`  | Fold look-alike glyphs in tool-call input (paths, commands) to ASCII, closing a cross-script deny-rule bypass. Gated per token, so non-Latin prose passes through unfolded.                                                                                                                                                                                                                                                                                                                                                            | `scan` (optional)           |
+| 5   | `/instructions` | Scan/auto-clean `CLAUDE.md`, `AGENTS.md`, `SKILL.md`, etc., decoding Unicode-tag + zero-width-binary payloads.                                                                                                                                                                                                                                                                                                                                                                                                                         | `fs` (direct)               |
+| 6   | `/prompt`       | Classify a prompt pass / note / block on payload-capable invisible/ANSI content (inert escapes get the note).                                                                                                                                                                                                                                                                                                                                                                                                                          | —                           |
+| 7   | `/output`       | Run Layers 1–4 over structured tool output, preserving shape. The Layer-5 slot takes a delete-only filter.                                                                                                                                                                                                                                                                                                                                                                                                                             | `redact`, `filterInjection` |
+| 8   | `/rehydrate`    | Re-anchor a model Edit or whole-file Write composed from the _sanitized_ view back onto real bytes; gate MultiEdit on a verified view==disk; deny anything ambiguous or secret-exposing.                                                                                                                                                                                                                                                                                                                                               | `io`                        |
+| —   | `/view-map`     | Pure offset/text machinery mapping a file's on-disk bytes ↔ the sanitized view (Layer-1 deletions, Layer-4 redactions). No I/O — consumed by `/rehydrate`.                                                                                                                                                                                                                                                                                                                                                                             | —                           |
 
 See [`THREAT-MODEL.md`](./THREAT-MODEL.md) for per-vector detail.
 

package/claude-hooks/lib/hook-io.mjs

@@ -329,13 +329,6 @@ export function failOpenContext(
   );
 }
 
-// Unpaired UTF-16 surrogates: a high half with no low follower, or a low half
-// with no high lead. Hook text spliced into the model's context must be
-// well-formed UTF-16 there, so the sanitizers normalize these out before
-// serializing.
-const LONE_SURROGATE_RE =
-  /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/g;
-
 /**
  * Hard cap on hook stdin. A well-formed Claude Code hook payload is at most a
  * few MB (tool input plus the harness-truncated tool output); 64 MiB leaves
@@ -622,20 +615,24 @@ const UNTRUSTED_TEXT_CAP = 500;
 /**
  * Scrub untrusted text before it is spliced into the model's context via a
  * warning/reason field: strip ANSI and payload-capable invisibles to a fixed
- * point (via the injected `layer1`, the package's composite Layer-1 view),
- * replace lone surrogates so the model's UTF-16 context stays well-formed, then
- * cap by whole code points (never mid-pair, which the surrogate pass above
- * already swept). `layer1` is injected rather than imported so this
- * dependency-light module never eagerly loads the sanitizer package — each
- * caller passes its own caught-import binding.
+ * point (via the injected `layer1`), then cap by whole code points. `layer1` is
+ * injected rather than imported so this dependency-light module never eagerly
+ * loads the sanitizer package — each caller passes its own caught-import
+ * binding.
+ *
+ * Pass a layer1 that also normalizes lone surrogates — `applyLayer1WellFormed`,
+ * never the bare `applyLayer1`. The model's UTF-16 context must be well-formed,
+ * and the code-point cap below must not slice a half it mistook for a whole
+ * character. This module used to re-spell that substitution over a private
+ * regex; the package now owns the one definition.
  * @param {unknown} raw
- * @param {(text: string) => { cleaned: string }} layer1
+ * @param {(text: string) => { cleaned: string }} layer1  MUST normalize lone surrogates
  * @param {number} [cap]
  * @returns {string}
  */
 export function scrubUntrustedText(raw, layer1, cap = UNTRUSTED_TEXT_CAP) {
   if (typeof raw !== "string" || raw === "") return "";
-  const cleaned = layer1(raw).cleaned.replace(LONE_SURROGATE_RE, "�");
+  const cleaned = layer1(raw).cleaned;
   const points = [...cleaned];
   return points.length > cap
     ? points.slice(0, cap).join("") + "…[truncated]"

package/claude-hooks/lib/invisible-alert.mjs

@@ -30,13 +30,16 @@ import {
 } from "./hook-io.mjs";
 
 // Layer-1 scrubber for the untrusted alert-store contents the gate splices into a
-// permissionDecisionReason. Bound via lazyImport (see its doc for the fail-OPEN
-// hazard of a bare static npm import): a load failure leaves applyLayer1 undefined,
-// so scrubUntrustedText throws into the caller's fail-closed catch (→ ask) rather
-// than emitting an unscrubbed reason.
-const { applyLayer1 } = /** @type {typeof import("agent-sanitizer")} */ (
-  await lazyImport("agent-sanitizer")
-);
+// permissionDecisionReason. The WELL-FORMED composition, not the bare applyLayer1:
+// the reason is spliced into the model's UTF-16 context, and scrubUntrustedText's
+// code-point cap must not slice a lone surrogate half. Bound via lazyImport (see
+// its doc for the fail-OPEN hazard of a bare static npm import): a load failure
+// leaves it undefined, so scrubUntrustedText throws into the caller's fail-closed
+// catch (→ ask) rather than emitting an unscrubbed reason.
+const { applyLayer1WellFormed } =
+  /** @type {typeof import("agent-sanitizer")} */ (
+    await lazyImport("agent-sanitizer")
+  );
 
 /**
  * The path prefix every alert artifact of this PROJECT shares. Never a file
@@ -392,7 +395,7 @@ export function invisibleCharAlert(sessionId) {
     }
   }
   if (parts.length === 0) return null;
-  return scrubUntrustedText(parts.join("\n"), applyLayer1);
+  return scrubUntrustedText(parts.join("\n"), applyLayer1WellFormed);
 }
 
 /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-sanitizer",
-  "version": "2.45.9",
+  "version": "2.47.0",
   "description": "Defend an agent against hidden-content injection: strip payload-capable invisible Unicode and ANSI, splice out human-invisible HTML, and flag data-exfil URLs in untrusted text before any model sees it.",
   "type": "module",
   "repository": {
@@ -98,6 +98,10 @@
       "types": "./types/invisible.d.mts",
       "default": "./src/invisible.mjs"
     },
+    "./layer1": {
+      "types": "./types/layer1.d.mts",
+      "default": "./src/layer1.mjs"
+    },
     "./html": {
       "types": "./types/html.d.mts",
       "default": "./src/html.mjs"

package/src/index.mjs

@@ -25,8 +25,10 @@ import { sanitizeText } from "./output.mjs";
 // (`./rehydrate`) so every consumer derives the identical model-facing view.
 export {
   applyLayer1,
+  applyLayer1WellFormed,
   isBenignAnsi,
   isBenignAnsiKinds,
+  normalizeLoneSurrogates,
   stripAnsiFully,
   LONE_SURROGATE_RE,
 } from "./layer1.mjs";

package/src/layer1.mjs

@@ -6,11 +6,20 @@
  * re-implementation would drift, and rehydration's soundness gate depends on
  * re-cleaning reproducing the view).
  *
- * Lone-surrogate normalization is NOT applied by {@link applyLayer1} itself: it
- * is exported as {@link LONE_SURROGATE_RE} for consumers to apply at the boundary
- * that needs a well-formed string (the redactor input in output.mjs's
- * processLayer1 / re-redact, and before the HTML tokenizer), because that is the
- * point where a lone surrogate would otherwise corrupt a match or a parse.
+ * Lone-surrogate normalization is NOT applied by {@link applyLayer1} itself. It
+ * belongs at the boundary that needs a well-formed string — the redactor input
+ * in output.mjs's processLayer1 / re-redact, and before the HTML tokenizer —
+ * because that is the point where a lone surrogate would otherwise corrupt a
+ * match or a parse. Prefer {@link normalizeLoneSurrogates} at each such
+ * boundary, the one definition of that substitution, which output.mjs's
+ * processLayer1 and rehydrate.mjs both call — not a fresh `.replace` over
+ * {@link LONE_SURROGATE_RE}.
+ *
+ * So {@link applyLayer1} alone is NOT the string the tool-output pipeline shows a
+ * model: on an input carrying an unpaired surrogate the two differ at exactly
+ * that code unit. {@link applyLayer1WellFormed} is the composition the pipeline
+ * runs, and a consumer deriving offsets for redaction or view mapping wants that
+ * one — see its own doc for why the choice matters.
  */
 import { stripInvisibleWithReport, CATEGORY } from "./invisible.mjs";
 import {
@@ -266,3 +275,48 @@ export function applyLayer1(text) {
   if (ansiKinds.size > 0) found.add(CATEGORY.ANSI);
   return { cleaned, deAnsi, found: [...found], ansiKinds: [...ansiKinds] };
 }
+
+/**
+ * Map every lone UTF-16 surrogate to U+FFFD. Load-bearing on ANY path that feeds
+ * text to an injected redactor: a secret split by an interposed lone surrogate
+ * reads as adjacent to a model rendering its own UTF-16 but as broken to a
+ * redactor (Node maps the lone surrogate to U+FFFD en route), so a secret
+ * reconstituted across the surrogate survives redaction unless the text is
+ * normalized first. It also keeps an HTML tokenizer from throwing on a stray
+ * code unit. The substitution is same-LENGTH — one UTF-16 unit for one — so
+ * offsets computed against the un-normalized string stay valid against this one.
+ * @param {string} text
+ * @returns {string}
+ */
+export function normalizeLoneSurrogates(text) {
+  return text.replace(LONE_SURROGATE_RE, "\uFFFD");
+}
+
+/**
+ * {@link applyLayer1} followed by {@link normalizeLoneSurrogates} — the exact
+ * composition `output.mjs`'s processLayer1 runs before Layers 2+ see the text,
+ * so this is the string a model is actually shown.
+ *
+ * Take this one, not `applyLayer1`, whenever the result feeds a redactor, an
+ * offset calculation or a view map: those consumers compare their own string to
+ * the model-facing view, and an unpaired surrogate is exactly where the two
+ * spellings diverge. Take `applyLayer1` when you want Layer 1's removals alone
+ * and intend to hand a possibly ill-formed string onward unchanged.
+ *
+ * `found` gains {@link CATEGORY.LONE_SURROGATES} exactly when the normalization
+ * changed the text, matching what the pipeline reports for the same input.
+ * `deAnsi` is untouched: it is the ANSI strip of the ORIGINAL text, the scope
+ * the long-run payload check needs.
+ * @param {string} text
+ * @returns {{ cleaned: string, deAnsi: string, found: string[], ansiKinds: string[] }}
+ */
+export function applyLayer1WellFormed(text) {
+  const result = applyLayer1(text);
+  const cleaned = normalizeLoneSurrogates(result.cleaned);
+  if (cleaned === result.cleaned) return result;
+  return {
+    ...result,
+    cleaned,
+    found: [...result.found, CATEGORY.LONE_SURROGATES],
+  };
+}

package/src/output.mjs

@@ -34,7 +34,7 @@ import {
   applyLayer1,
   INERT_ANSI_NOTE,
   isBenignAnsiKinds,
-  LONE_SURROGATE_RE,
+  normalizeLoneSurrogates,
 } from "./layer1.mjs";
 import {
   describeConfusableHosts,
@@ -144,21 +144,6 @@ function errMessage(err) {
  *   message). Null means the filter made no finding.
  */
 
-/**
- * Map every lone UTF-16 surrogate to U+FFFD. Load-bearing on ANY path that feeds
- * text to the injected redactor: a secret split by an interposed lone surrogate
- * reads as adjacent to a model rendering its own UTF-16 but as broken to a
- * redactor (Node maps the lone surrogate to U+FFFD en route), so a secret
- * reconstituted across the surrogate survives redaction unless the text is
- * normalized first. Shared by {@link processLayer1} and the Layer-5 re-redact so
- * the two redact-input paths cannot drift.
- * @param {string} text
- * @returns {string}
- */
-function normalizeLoneSurrogates(text) {
-  return text.replace(LONE_SURROGATE_RE, "�");
-}
-
 /**
  * @typedef {{ text: string, found: string[], findings: import("./severity.mjs").Finding[], modified: boolean, unreportedChange: boolean }} PipelineState
  *   The running state of one {@link sanitizeText} call. Layers read `text` and

package/src/rehydrate.mjs

@@ -54,7 +54,7 @@
  * denied with use-single-Edit guidance (see the dispatch in
  * {@link rehydrateRedacted}).
  */
-import { applyLayer1, LONE_SURROGATE_RE } from "./layer1.mjs";
+import { applyLayer1, normalizeLoneSurrogates } from "./layer1.mjs";
 import {
   occurrences,
   overlapAwareCount,
@@ -114,7 +114,7 @@ function layer1View(text) {
   const { cleaned: layer1Cleaned } = applyLayer1(text);
   return {
     layer1Cleaned,
-    cleaned: layer1Cleaned.replace(LONE_SURROGATE_RE, "\uFFFD"),
+    cleaned: normalizeLoneSurrogates(layer1Cleaned),
   };
 }
 

package/types/claude-hooks/lib/hook-io.d.mts

@@ -245,14 +245,18 @@ export function makeDeadline(budgetMs: number, now?: () => number): {
 /**
  * Scrub untrusted text before it is spliced into the model's context via a
  * warning/reason field: strip ANSI and payload-capable invisibles to a fixed
- * point (via the injected `layer1`, the package's composite Layer-1 view),
- * replace lone surrogates so the model's UTF-16 context stays well-formed, then
- * cap by whole code points (never mid-pair, which the surrogate pass above
- * already swept). `layer1` is injected rather than imported so this
- * dependency-light module never eagerly loads the sanitizer package — each
- * caller passes its own caught-import binding.
+ * point (via the injected `layer1`), then cap by whole code points. `layer1` is
+ * injected rather than imported so this dependency-light module never eagerly
+ * loads the sanitizer package — each caller passes its own caught-import
+ * binding.
+ *
+ * Pass a layer1 that also normalizes lone surrogates — `applyLayer1WellFormed`,
+ * never the bare `applyLayer1`. The model's UTF-16 context must be well-formed,
+ * and the code-point cap below must not slice a half it mistook for a whole
+ * character. This module used to re-spell that substitution over a private
+ * regex; the package now owns the one definition.
  * @param {unknown} raw
- * @param {(text: string) => { cleaned: string }} layer1
+ * @param {(text: string) => { cleaned: string }} layer1  MUST normalize lone surrogates
  * @param {number} [cap]
  * @returns {string}
  */

package/types/index.d.mts

@@ -52,6 +52,6 @@ export function sanitize(text: string, options?: {
         original: string;
     }>;
 }>;
-export { applyLayer1, isBenignAnsi, isBenignAnsiKinds, stripAnsiFully, LONE_SURROGATE_RE } from "./layer1.mjs";
+export { applyLayer1, applyLayer1WellFormed, isBenignAnsi, isBenignAnsiKinds, normalizeLoneSurrogates, stripAnsiFully, LONE_SURROGATE_RE } from "./layer1.mjs";
 export { stripInvisible, stripInvisibleWithReport, isSgrOnly, STRIP, SGR_RE, CHECKS, CATEGORY, CATEGORY_LABELS, LINGUISTIC_SCRIPTS, VS, BLANK_NON_CF, LONG_RUN_RE, LONG_RUN_THRESHOLD, SCATTERED_THRESHOLD, findLongRuns, hasLongRun } from "./invisible.mjs";
 export { HTML_TAG_PRESENT, MD_LINK_HINT, SECRET_HINT, SECRET_HINT_EXT, matchesSecretHint } from "./gates.mjs";

package/types/layer1.d.mts

@@ -97,6 +97,43 @@ export function applyLayer1(text: string): {
     found: string[];
     ansiKinds: string[];
 };
+/**
+ * Map every lone UTF-16 surrogate to U+FFFD. Load-bearing on ANY path that feeds
+ * text to an injected redactor: a secret split by an interposed lone surrogate
+ * reads as adjacent to a model rendering its own UTF-16 but as broken to a
+ * redactor (Node maps the lone surrogate to U+FFFD en route), so a secret
+ * reconstituted across the surrogate survives redaction unless the text is
+ * normalized first. It also keeps an HTML tokenizer from throwing on a stray
+ * code unit. The substitution is same-LENGTH — one UTF-16 unit for one — so
+ * offsets computed against the un-normalized string stay valid against this one.
+ * @param {string} text
+ * @returns {string}
+ */
+export function normalizeLoneSurrogates(text: string): string;
+/**
+ * {@link applyLayer1} followed by {@link normalizeLoneSurrogates} — the exact
+ * composition `output.mjs`'s processLayer1 runs before Layers 2+ see the text,
+ * so this is the string a model is actually shown.
+ *
+ * Take this one, not `applyLayer1`, whenever the result feeds a redactor, an
+ * offset calculation or a view map: those consumers compare their own string to
+ * the model-facing view, and an unpaired surrogate is exactly where the two
+ * spellings diverge. Take `applyLayer1` when you want Layer 1's removals alone
+ * and intend to hand a possibly ill-formed string onward unchanged.
+ *
+ * `found` gains {@link CATEGORY.LONE_SURROGATES} exactly when the normalization
+ * changed the text, matching what the pipeline reports for the same input.
+ * `deAnsi` is untouched: it is the ANSI strip of the ORIGINAL text, the scope
+ * the long-run payload check needs.
+ * @param {string} text
+ * @returns {{ cleaned: string, deAnsi: string, found: string[], ansiKinds: string[] }}
+ */
+export function applyLayer1WellFormed(text: string): {
+    cleaned: string;
+    deAnsi: string;
+    found: string[];
+    ansiKinds: string[];
+};
 export const LONE_SURROGATE_RE: RegExp;
 /**
  * What a reader is told when the ONLY thing a strip removed was inert ANSI (see