agent-sanitizer 2.14.7 → 2.14.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -47,17 +47,17 @@ Split into subpaths so the heavy HTML dependency stays opt-in. **Seam** names
47
47
  the callback you inject for the agent-specific concern; `—` is a pure transform,
48
48
  `fs (direct)` does its own file I/O instead of taking one.
49
49
 
50
- | # | Import | Purpose | Seam |
51
- | --- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
52
- | 1 | `/invisible` | Strip zero-width, bidi, variation-selector and tag chars + ANSI/SGR escapes. Preserves ZWNJ/ZWJ for Arabic/Indic/emoji. Zero deps. | — |
53
- | 2 | `/html` | Splice out instructions hidden in comments, `display:none`, off-screen, white-on-white, `hidden`. Leaves a placeholder. | — |
54
- | 3 | `/html` | Detect exfil-shaped URLs (payloads in query/path, embedded creds, `data:`/`javascript:`, off-origin redirects). Reports only. | — |
55
- | 4 | `/confusables` | Fold look-alike glyphs in tool-call input (paths, commands) to ASCII, closing a cross-script deny-rule bypass. | `scan` |
56
- | 5 | `/instructions` | Scan/auto-clean `CLAUDE.md`, `AGENTS.md`, `SKILL.md`, etc., decoding Unicode-tag + zero-width-binary payloads. | `fs` (direct) |
57
- | 6 | `/prompt` | Classify a prompt pass / SGR-note / block on payload-capable invisible/ANSI content. | — |
58
- | 7 | `/output` | Run Layers 1–4 over structured tool output, preserving shape. The Layer-5 slot takes a delete-only filter. | `redact`, `filterInjection` |
59
- | 8 | `/rehydrate` | Re-anchor a model Edit composed from the _sanitized_ view back onto real bytes; deny anything ambiguous or secret-exposing. | `io` |
60
- | — | `/view-map` | Pure offset/text machinery mapping a file's on-disk bytes ↔ the sanitized view (Layer-1 deletions, Layer-4 redactions). No I/O — consumed by `/rehydrate`. | — |
50
+ | # | Import | Purpose | Seam |
51
+ | --- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
52
+ | 1 | `/invisible` | Strip zero-width, bidi, variation-selector and tag chars + ANSI/SGR escapes. Preserves ZWNJ/ZWJ for Arabic/Indic/emoji. Zero deps. | — |
53
+ | 2 | `/html` | Splice out instructions hidden in comments, `display:none`, off-screen, white-on-white, `hidden`. Leaves a placeholder. | — |
54
+ | 3 | `/html` | Detect exfil-shaped URLs (payloads in query/path, embedded creds, `data:`/`javascript:`, off-origin redirects). Reports only. | — |
55
+ | 4 | `/confusables` | Fold look-alike glyphs in tool-call input (paths, commands) to ASCII, closing a cross-script deny-rule bypass. Gated per token, so non-Latin prose passes through unfolded. | `scan` |
56
+ | 5 | `/instructions` | Scan/auto-clean `CLAUDE.md`, `AGENTS.md`, `SKILL.md`, etc., decoding Unicode-tag + zero-width-binary payloads. | `fs` (direct) |
57
+ | 6 | `/prompt` | Classify a prompt pass / SGR-note / block on payload-capable invisible/ANSI content. | — |
58
+ | 7 | `/output` | Run Layers 1–4 over structured tool output, preserving shape. The Layer-5 slot takes a delete-only filter. | `redact`, `filterInjection` |
59
+ | 8 | `/rehydrate` | Re-anchor a model Edit composed from the _sanitized_ view back onto real bytes; deny anything ambiguous or secret-exposing. | `io` |
60
+ | — | `/view-map` | Pure offset/text machinery mapping a file's on-disk bytes ↔ the sanitized view (Layer-1 deletions, Layer-4 redactions). No I/O — consumed by `/rehydrate`. | — |
61
61
 
62
62
  See [`THREAT-MODEL.md`](./THREAT-MODEL.md) for per-vector detail.
63
63
 
package/THREAT-MODEL.md CHANGED
@@ -92,12 +92,50 @@ payload-bearing query/fragment), suitable for a warning shown to the operator.
92
92
  `./confusables` folds look-alike glyphs in tool-call **input** fields (paths,
93
93
  commands) to their ASCII canon. A denied path/command spelled in homoglyphs (a
94
94
  Cyrillic `а` for ASCII `a`) would not match an ASCII deny rule; folding closes
95
- that cross-script bypass (CVE-2025-54794 class). Folding is per-character and
96
- context-free, so it also catches an **isolated** confusable with no ASCII anchor
97
- that a context-sensitive canonicaliser would leave alone. The homoglyph engine
98
- is **injected** (`{ scan }`)—the package owns no glyph map. An all-ASCII field
99
- never invokes the scanner. This narrows a steganographic channel; it is not an
100
- enforcement boundary (distinct code points would not match a deny rule anyway).
95
+ that cross-script bypass (CVE-2025-54794 class).
96
+
97
+ Folding is gated per **token** a maximal run of ASCII alphanumerics and
98
+ non-ASCII glyphs, with every other ASCII character a boundary. A token folds only
99
+ when both hold:
100
+
101
+ 1. **Folding leaves it pure ASCII.** The bypass requires the folded token to come
102
+ out byte-equal to an ASCII deny-rule target, so a token that still holds an
103
+ unmapped glyph after folding could not match one either way — skipping it
104
+ forfeits no enforcement.
105
+ 2. **It is more than a lone non-ASCII glyph.** A single-code-point token is a
106
+ one-letter foreign word — Russian `с` (with), `о` (about), `у` (at), `а`
107
+ (and), all mapped confusables and all among the most frequent words in the
108
+ language — as readily as it is a disguised argument, and no deny rule targets
109
+ a single character.
110
+
111
+ So `/etc/pаsswd` and an all-Cyrillic `раѕѕwd` still fold, while `Привет`,
112
+ `пароль.txt`, and `работа с файлом` pass through untouched. Without this gate the
113
+ layer transliterated ordinary Cyrillic and Greek text — a commit message, an
114
+ issue body, a filename — into garbage, which this repo weighs as the worse
115
+ failure.
116
+
117
+ The gate is per-token and never per-field: a field-level "any prose here → skip
118
+ the field" rule would let an attacker switch folding off for a whole command by
119
+ appending one foreign word.
120
+
121
+ **Known false positive:** a multi-letter foreign word composed _entirely_ of
122
+ mapped confusables — the Russian `сор` → `cop`, `со` → `co` — is by construction
123
+ indistinguishable from a disguised ASCII token, and still folds.
124
+
125
+ **Known false negative:** splicing one unmapped non-ASCII glyph into an otherwise
126
+ all-confusable token switches its fold off. It buys an attacker nothing on its
127
+ own — the unmapped glyph is still in the field, so the token cannot match an
128
+ ASCII deny rule either. A filter reading the raw field is what closes that, not
129
+ folding.
130
+
131
+ The soundness argument assumes no later layer erases code points from the same
132
+ field, which could remove an unmapped glyph the gate relied on after the decision
133
+ was made. Layer 4 runs before `sanitizeAuthoredContent` on `Bash.command`.
134
+
135
+ The homoglyph engine is **injected** (`{ scan }`) — the package owns no glyph
136
+ map. An all-ASCII field never invokes the scanner. This narrows a steganographic
137
+ channel; it is not an enforcement boundary (distinct code points would not match
138
+ a deny rule anyway).
101
139
 
102
140
  ## Instruction-file scanning
103
141
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "agent-sanitizer",
3
- "version": "2.14.7",
3
+ "version": "2.14.9",
4
4
  "description": "Defend an agent against hidden-content injection: strip payload-capable invisible Unicode and ANSI, splice out human-invisible HTML, and flag data-exfil URLs in untrusted text before any model sees it.",
5
5
  "type": "module",
6
6
  "repository": {
@@ -7,13 +7,42 @@
7
7
  * an ASCII deny rule, so an attacker could slip a denied path/command past a
8
8
  * filter by spelling it in look-alike code points.
9
9
  *
10
- * Folding is per-character and context-free: every glyph the injected scanner
11
- * flags is replaced with its ASCII (latin) equivalent regardless of its
12
- * neighbours. This deliberately catches an ISOLATED confusable with no ASCII
13
- * anchor (a lone Cyrillic "а" in "/а") that a context-SENSITIVE canonicaliser
14
- * would leave untouched exactly the bypass to close while leaving genuine
15
- * non-confusable non-ASCII (accented Latin, CJK, emoji) alone, since a faithful
16
- * scanner does not flag those.
10
+ * Folding is gated per TOKEN (a maximal run of ASCII alphanumerics and
11
+ * non-ASCII glyphs see isTokenBoundary). A token folds only when both hold:
12
+ *
13
+ * 1. Folding makes it pure ASCII every non-ASCII code point in it is
14
+ * flagged. The bypass requires the folded token to come out byte-equal to
15
+ * an ASCII deny-rule target, so a token left holding an unmapped glyph
16
+ * could never match one anyway; skipping it costs no enforcement.
17
+ * 2. It is more than a lone non-ASCII glyph standing between two boundaries.
18
+ * A one-code-point token is a one-letter foreign word — Russian "с", "о",
19
+ * "у", "а" are among the most frequent words in the language — as readily
20
+ * as it is a disguised argument, and no deny rule targets a single
21
+ * character, so the evidence does not support rewriting it.
22
+ *
23
+ * That still catches the all-confusable disguise ("раѕѕwd" → "passwd") and the
24
+ * anchored one ("pаsswd" → "passwd"), which is exactly the bypass to
25
+ * close, while leaving genuine non-Latin prose intact: "Привет" keeps unmapped
26
+ * П/и/в/т, so its mapped р/е are not folded and the word survives byte-for-byte.
27
+ *
28
+ * WHY THE GATE: without it, folding mangled any Cyrillic/Greek text passing
29
+ * through a command or path — a Russian commit message, issue body, or filename
30
+ * came out transliterated into garbage. A false positive here rewrites content
31
+ * the operator wrote, which this repo weighs as the worse failure.
32
+ *
33
+ * RESIDUAL: a multi-letter foreign word composed ENTIRELY of mapped confusables
34
+ * (the Russian "сор" → "cop") is indistinguishable from a disguised ASCII token
35
+ * by construction, and still folds. Documented in THREAT-MODEL.md, not worked
36
+ * around — the alternatives (a field-level gate, a "looks like a path" shape
37
+ * heuristic) either open a bypass or trade one guess for another.
38
+ *
39
+ * ORDERING: the soundness argument assumes no later layer erases code points
40
+ * from the same field, which would let an unmapped glyph the gate relied on
41
+ * disappear after the decision. Layer 4 runs before sanitizeAuthoredContent on
42
+ * Bash.command; keep it there.
43
+ *
44
+ * Genuine non-confusable non-ASCII (accented Latin, CJK, emoji) is untouched
45
+ * regardless, since a faithful scanner does not flag it.
17
46
  *
18
47
  * The confusable scanner is INJECTED, never imported: the canonical engine
19
48
  * (namespace-guard's vision-weighted map) is a heavy, separately-owned peer.
@@ -87,6 +116,157 @@ function describeFolds(findings) {
87
116
  return folds.length > MAX_REPORTED_FOLDS ? `${shown}, …` : shown;
88
117
  }
89
118
 
119
+ /**
120
+ * Reject a finding that does not describe a real, foldable glyph at its reported
121
+ * offset in `text`. Every consumer of a scanner finding runs this BEFORE acting
122
+ * on it, so a buggy or adversarial scanner fails loud instead of silently
123
+ * corrupting (or silently escaping the fold gate in) a path/command.
124
+ * @param {string} text
125
+ * @param {{ index: number, char: string, latinEquivalent: string }} finding
126
+ * @returns {void}
127
+ */
128
+ function assertFinding(text, finding) {
129
+ // Fail loud on a finding that does not match the actual bytes at its offset:
130
+ // a buggy/adversarial scanner reporting a wrong char/index would otherwise
131
+ // silently corrupt the path/command, defeating the deny-rule protection.
132
+ // A negative index is the gap the startsWith guard alone misses: when `char`
133
+ // is a prefix of the text, `startsWith(char, -1)` is true (the offset is
134
+ // clamped to 0), and the slice math below then mangles the string instead of
135
+ // throwing — so range-check the index explicitly first.
136
+ if (!Number.isInteger(finding.index) || finding.index < 0)
137
+ throw new Error(
138
+ `Confusable finding has an out-of-range index ${finding.index}`,
139
+ );
140
+ // An empty `char` makes startsWith("", i) vacuously true for ANY index, so
141
+ // the splice inserts latinEquivalent without consuming a code point — silent
142
+ // insertion-corruption — and describeFolds then crashes on "".codePointAt(0).
143
+ // A finding must name a real matched glyph, so reject it loudly rather than
144
+ // let a buggy/adversarial scanner corrupt the input.
145
+ if (finding.char === "")
146
+ throw new Error(
147
+ `Confusable finding at index ${finding.index} has an empty char`,
148
+ );
149
+ // An ASCII `char` is not a confusable — it is already its own canon. Folding
150
+ // it is a no-op at best, but the fold gate reads a non-ASCII code point's
151
+ // flagged/unflagged status to decide a token's fate, so a scanner claiming an
152
+ // ASCII character is confusable is reporting something the contract says
153
+ // cannot happen. Fail loud rather than act on a finding we cannot interpret.
154
+ if (!hasNonAscii(finding.char))
155
+ throw new Error(
156
+ `Confusable finding at index ${finding.index} names an ASCII char ${JSON.stringify(finding.char)}`,
157
+ );
158
+ if (!text.startsWith(finding.char, finding.index))
159
+ throw new Error(
160
+ `Confusable finding does not match input at index ${finding.index}: expected ${JSON.stringify(finding.char)}`,
161
+ );
162
+ // An empty `latinEquivalent` slips past the ASCII loop below (it never
163
+ // iterates) and would splice the glyph to nothing — silently DELETING a
164
+ // character from a path/command. That is the same class of silent corruption
165
+ // the non-ASCII guard rejects, so fail loud here too rather than let a
166
+ // buggy/adversarial scanner erase input.
167
+ if (finding.latinEquivalent === "")
168
+ throw new Error(
169
+ `Confusable finding for ${JSON.stringify(
170
+ finding.char,
171
+ )} at index ${finding.index} has an empty latinEquivalent`,
172
+ );
173
+ // The replacement must be the ASCII canon the contract promises. A non-ASCII
174
+ // `latinEquivalent` would fold one confusable into ANOTHER look-alike (e.g.
175
+ // Cyrillic а → Cyrillic е), defeating the whole point — the cross-script
176
+ // deny-rule bypass would survive — and silently break the fold-to-ASCII
177
+ // invariant callers rely on, so reject it loudly.
178
+ for (const ch of finding.latinEquivalent)
179
+ if (/** @type {number} */ (ch.codePointAt(0)) > 0x7f)
180
+ throw new Error(
181
+ `Confusable latinEquivalent ${JSON.stringify(
182
+ finding.latinEquivalent,
183
+ )} is not ASCII`,
184
+ );
185
+ }
186
+
187
+ /**
188
+ * True for a code point that ends a token: any ASCII character that is not a
189
+ * letter or digit (space, `/`, `.`, `-`, `_`, quotes, shell metacharacters). A
190
+ * token is therefore a maximal run of ASCII alphanumerics and non-ASCII glyphs.
191
+ * That is not a shell word or a path segment — it is deliberately the SMALLEST
192
+ * unit that still keeps a word of foreign prose whole, so one such word can
193
+ * never switch folding off for the tokens around it.
194
+ * @param {string} ch a single code point
195
+ * @returns {boolean}
196
+ */
197
+ function isTokenBoundary(ch) {
198
+ if (hasNonAscii(ch)) return false;
199
+ const code = ch.charCodeAt(0);
200
+ const isDigit = code >= 0x30 && code <= 0x39;
201
+ const isUpper = code >= 0x41 && code <= 0x5a;
202
+ const isLower = code >= 0x61 && code <= 0x7a;
203
+ return !isDigit && !isUpper && !isLower;
204
+ }
205
+
206
+ /**
207
+ * Keep only the findings whose TOKEN is foldable — see the module header for the
208
+ * two conditions (folds to pure ASCII; is not a lone glyph) and THREAT-MODEL.md
209
+ * for why declining the rest forfeits no enforcement.
210
+ *
211
+ * Every finding is validated before it is judged, so an adversarial scanner's
212
+ * bogus finding throws rather than being quietly dropped by the gate.
213
+ * @param {string} text
214
+ * @param {Array<{ index: number, char: string, latinEquivalent: string }>} findings
215
+ * @returns {Array<{ index: number, char: string, latinEquivalent: string }>}
216
+ */
217
+ export function selectFoldableFindings(text, findings) {
218
+ for (const finding of findings) assertFinding(text, finding);
219
+
220
+ // Every UTF-16 offset a finding covers, not just its start: a scanner is free
221
+ // to report a multi-code-point match, and treating only the first offset as
222
+ // flagged would read the rest as unmapped and reject an otherwise ASCII token.
223
+ const flagged = new Set();
224
+ for (const finding of findings)
225
+ for (let i = 0; i < finding.char.length; i++)
226
+ flagged.add(finding.index + i);
227
+
228
+ // Mark the offsets belonging to a foldable token, so the per-finding lookup
229
+ // below is O(1). An interval scan per finding would be quadratic on a
230
+ // glyph-stuffed command — this runs on every tool call.
231
+ const foldableAt = new Uint8Array(text.length);
232
+ let start = 0;
233
+ let foldable = true;
234
+ let glyphs = 0;
235
+ let anchored = false;
236
+ let index = 0;
237
+ /**
238
+ * Close the token ending at `end`, marking it foldable when it holds no
239
+ * unmapped glyph and is more than a lone glyph, then reset for the next one.
240
+ * @param {number} end
241
+ */
242
+ const flushToken = (end) => {
243
+ if (foldable && (glyphs > 1 || anchored)) foldableAt.fill(1, start, end);
244
+ foldable = true;
245
+ glyphs = 0;
246
+ anchored = false;
247
+ };
248
+ for (const ch of text) {
249
+ if (isTokenBoundary(ch)) {
250
+ flushToken(index);
251
+ start = index + ch.length;
252
+ } else {
253
+ glyphs++;
254
+ if (!hasNonAscii(ch)) anchored = true;
255
+ // An unmapped non-ASCII glyph: this token cannot fold to ASCII.
256
+ else if (!flagged.has(index)) foldable = false;
257
+ }
258
+ index += ch.length;
259
+ }
260
+ flushToken(index);
261
+
262
+ // Every offset the finding covers, not just its first: `char` may span a
263
+ // boundary into a token the gate rejected, and folding it would rewrite that
264
+ // token — the exact mangling this gate exists to prevent.
265
+ return findings.filter((finding) =>
266
+ [...finding.char].every((_, i) => foldableAt[finding.index + i] === 1),
267
+ );
268
+ }
269
+
90
270
  /**
91
271
  * Replace every scan-flagged confusable with its ASCII (latin) equivalent.
92
272
  * `index` is a UTF-16 offset into `text` and `char` is the matched glyph (which
@@ -101,53 +281,10 @@ export function foldConfusables(text, findings) {
101
281
  for (const finding of [...findings].sort(
102
282
  (lhs, rhs) => rhs.index - lhs.index,
103
283
  )) {
104
- // Fail loud on a finding that does not match the actual bytes at its offset:
105
- // a buggy/adversarial scanner reporting a wrong char/index would otherwise
106
- // silently corrupt the path/command, defeating the deny-rule protection.
107
- // A negative index is the gap the startsWith guard alone misses: when `char`
108
- // is a prefix of the text, `startsWith(char, -1)` is true (the offset is
109
- // clamped to 0), and the slice math below then mangles the string instead of
110
- // throwing — so range-check the index explicitly first.
111
- if (!Number.isInteger(finding.index) || finding.index < 0)
112
- throw new Error(
113
- `Confusable finding has an out-of-range index ${finding.index}`,
114
- );
115
- // An empty `char` makes startsWith("", i) vacuously true for ANY index, so
116
- // the slice below inserts latinEquivalent without consuming a code point —
117
- // silent insertion-corruption — and describeFolds then crashes on
118
- // "".codePointAt(0). A finding must name a real matched glyph, so reject it
119
- // loudly rather than let a buggy/adversarial scanner corrupt the input.
120
- if (finding.char === "")
121
- throw new Error(
122
- `Confusable finding at index ${finding.index} has an empty char`,
123
- );
124
- if (!folded.startsWith(finding.char, finding.index))
125
- throw new Error(
126
- `Confusable finding does not match input at index ${finding.index}: expected ${JSON.stringify(finding.char)}`,
127
- );
128
- // An empty `latinEquivalent` slips past the ASCII loop below (it never
129
- // iterates) and would splice the glyph to nothing — silently DELETING a
130
- // character from a path/command. That is the same class of silent corruption
131
- // the non-ASCII guard rejects, so fail loud here too rather than let a
132
- // buggy/adversarial scanner erase input.
133
- if (finding.latinEquivalent === "")
134
- throw new Error(
135
- `Confusable finding for ${JSON.stringify(
136
- finding.char,
137
- )} at index ${finding.index} has an empty latinEquivalent`,
138
- );
139
- // The replacement must be the ASCII canon the contract promises. A non-ASCII
140
- // `latinEquivalent` would fold one confusable into ANOTHER look-alike (e.g.
141
- // Cyrillic а → Cyrillic е), defeating the whole point — the cross-script
142
- // deny-rule bypass would survive — and silently break the fold-to-ASCII
143
- // invariant callers rely on, so reject it loudly.
144
- for (const ch of finding.latinEquivalent)
145
- if (/** @type {number} */ (ch.codePointAt(0)) > 0x7f)
146
- throw new Error(
147
- `Confusable latinEquivalent ${JSON.stringify(
148
- finding.latinEquivalent,
149
- )} is not ASCII`,
150
- );
284
+ // Validate against the partially-folded text: the highest-index-first order
285
+ // leaves every not-yet-spliced offset byte-identical to `text`, so the
286
+ // startsWith check still sees the original glyph at the reported index.
287
+ assertFinding(folded, finding);
151
288
  folded =
152
289
  folded.slice(0, finding.index) +
153
290
  finding.latinEquivalent +
@@ -190,8 +327,13 @@ export function normalizeConfusables(
190
327
  for (const k of candidates) {
191
328
  const { findings } = scan(toolInput[k]);
192
329
  if (findings.length === 0) continue;
193
- updatedInput[k] = foldConfusables(toolInput[k], findings);
194
- normalized.push(`${k} (${describeFolds(findings)})`);
330
+ // Precision gate: fold only what a deny rule could actually match (see
331
+ // selectFoldableFindings). Report the folds APPLIED, not the ones scanned —
332
+ // naming a glyph the model can still see in the field would be a lie.
333
+ const foldable = selectFoldableFindings(toolInput[k], findings);
334
+ if (foldable.length === 0) continue;
335
+ updatedInput[k] = foldConfusables(toolInput[k], foldable);
336
+ normalized.push(`${k} (${describeFolds(foldable)})`);
195
337
  }
196
338
 
197
339
  if (normalized.length === 0) return null;
@@ -12,6 +12,26 @@ export function hasNonAscii(value: string): boolean;
12
12
  * @returns {string}
13
13
  */
14
14
  export function normalizeContext(normalized: string[]): string;
15
+ /**
16
+ * Keep only the findings whose TOKEN is foldable — see the module header for the
17
+ * two conditions (folds to pure ASCII; is not a lone glyph) and THREAT-MODEL.md
18
+ * for why declining the rest forfeits no enforcement.
19
+ *
20
+ * Every finding is validated before it is judged, so an adversarial scanner's
21
+ * bogus finding throws rather than being quietly dropped by the gate.
22
+ * @param {string} text
23
+ * @param {Array<{ index: number, char: string, latinEquivalent: string }>} findings
24
+ * @returns {Array<{ index: number, char: string, latinEquivalent: string }>}
25
+ */
26
+ export function selectFoldableFindings(text: string, findings: Array<{
27
+ index: number;
28
+ char: string;
29
+ latinEquivalent: string;
30
+ }>): Array<{
31
+ index: number;
32
+ char: string;
33
+ latinEquivalent: string;
34
+ }>;
15
35
  /**
16
36
  * Replace every scan-flagged confusable with its ASCII (latin) equivalent.
17
37
  * `index` is a UTF-16 offset into `text` and `char` is the matched glyph (which
@@ -62,13 +82,42 @@ export function normalizeConfusables(tool: string, toolInput: any, { scan, field
62
82
  * an ASCII deny rule, so an attacker could slip a denied path/command past a
63
83
  * filter by spelling it in look-alike code points.
64
84
  *
65
- * Folding is per-character and context-free: every glyph the injected scanner
66
- * flags is replaced with its ASCII (latin) equivalent regardless of its
67
- * neighbours. This deliberately catches an ISOLATED confusable with no ASCII
68
- * anchor (a lone Cyrillic "а" in "/а") that a context-SENSITIVE canonicaliser
69
- * would leave untouched exactly the bypass to close while leaving genuine
70
- * non-confusable non-ASCII (accented Latin, CJK, emoji) alone, since a faithful
71
- * scanner does not flag those.
85
+ * Folding is gated per TOKEN (a maximal run of ASCII alphanumerics and
86
+ * non-ASCII glyphs see isTokenBoundary). A token folds only when both hold:
87
+ *
88
+ * 1. Folding makes it pure ASCII every non-ASCII code point in it is
89
+ * flagged. The bypass requires the folded token to come out byte-equal to
90
+ * an ASCII deny-rule target, so a token left holding an unmapped glyph
91
+ * could never match one anyway; skipping it costs no enforcement.
92
+ * 2. It is more than a lone non-ASCII glyph standing between two boundaries.
93
+ * A one-code-point token is a one-letter foreign word — Russian "с", "о",
94
+ * "у", "а" are among the most frequent words in the language — as readily
95
+ * as it is a disguised argument, and no deny rule targets a single
96
+ * character, so the evidence does not support rewriting it.
97
+ *
98
+ * That still catches the all-confusable disguise ("раѕѕwd" → "passwd") and the
99
+ * anchored one ("pаsswd" → "passwd"), which is exactly the bypass to
100
+ * close, while leaving genuine non-Latin prose intact: "Привет" keeps unmapped
101
+ * П/и/в/т, so its mapped р/е are not folded and the word survives byte-for-byte.
102
+ *
103
+ * WHY THE GATE: without it, folding mangled any Cyrillic/Greek text passing
104
+ * through a command or path — a Russian commit message, issue body, or filename
105
+ * came out transliterated into garbage. A false positive here rewrites content
106
+ * the operator wrote, which this repo weighs as the worse failure.
107
+ *
108
+ * RESIDUAL: a multi-letter foreign word composed ENTIRELY of mapped confusables
109
+ * (the Russian "сор" → "cop") is indistinguishable from a disguised ASCII token
110
+ * by construction, and still folds. Documented in THREAT-MODEL.md, not worked
111
+ * around — the alternatives (a field-level gate, a "looks like a path" shape
112
+ * heuristic) either open a bypass or trade one guess for another.
113
+ *
114
+ * ORDERING: the soundness argument assumes no later layer erases code points
115
+ * from the same field, which would let an unmapped glyph the gate relied on
116
+ * disappear after the decision. Layer 4 runs before sanitizeAuthoredContent on
117
+ * Bash.command; keep it there.
118
+ *
119
+ * Genuine non-confusable non-ASCII (accented Latin, CJK, emoji) is untouched
120
+ * regardless, since a faithful scanner does not flag it.
72
121
  *
73
122
  * The confusable scanner is INJECTED, never imported: the canonical engine
74
123
  * (namespace-guard's vision-weighted map) is a heavy, separately-owned peer.