text-sanctifier 1.3.2 → 1.3.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +349 -332
- package/bin/text-sanctifier.js +753 -753
- package/package.json +5 -1
- package/src/categories.generated.js +223 -223
- package/src/hiddenText.js +120 -119
- package/src/inspectText.js +343 -343
- package/src/sanctifyText.js +506 -506
package/README.md
CHANGED
|
@@ -1,332 +1,349 @@
|
|
|
1
|
-
# text-sanctifier
|
|
2
|
-
|
|
3
|
-
[](https://www.npmjs.com/package/text-sanctifier)
|
|
4
|
-
[](https://www.npmjs.com/package/text-sanctifier)
|
|
5
|
-
[](https://github.com/iWhatty/text-sanctifier)
|
|
8
|
-
[ and the Arabic letter mark (U+061C) used in Trojan Source attacks
|
|
16
|
-
- Purges Unicode tag characters (U+E0000–U+E007F), the "ASCII smuggling" channel for hiding instructions from humans in text an LLM reads, plus variation selectors, Hangul fillers, soft hyphens and invisible operators
|
|
17
|
-
- Locates invisible and control characters by offset, line and column (`inspectTextDetailed`)
|
|
18
|
-
- `text-sanctifier check` / `fix` command line for CI and coding agents, with JSON or SARIF output, exit codes and pre-commit hooks
|
|
19
|
-
- `check --ai-config` finds and decodes hidden text in agent instruction files (`CLAUDE.md`, `AGENTS.md`, Cursor rules, `SKILL.md`, MCP configuration)
|
|
20
|
-
- Normalizes line endings (CRLF, CR, LF) to LF
|
|
21
|
-
- Collapses unwanted spaces and paragraphs
|
|
22
|
-
- Nukes control characters (if enabled)
|
|
23
|
-
- Smart normalization of typographic junk (quotes, dashes, bullets, full-width punctuation)
|
|
24
|
-
- Keyboard-only filtering (retain printable ASCII + full emoji sequences)
|
|
25
|
-
- Preserves ZWJ emoji clusters (👨👩👧👦)
|
|
26
|
-
- Preserves VS16 emoji presentation variants (✌️, ‼️)
|
|
27
|
-
- Configurable via fine-grained flags or ready-made presets
|
|
28
|
-
- Includes strict, loose, and keyboard-only modes
|
|
29
|
-
- Deterministic RegExp usage (no global `lastIndex` state leaks)
|
|
30
|
-
|
|
31
|
-
---
|
|
32
|
-
|
|
33
|
-
## Install
|
|
34
|
-
|
|
35
|
-
```sh
|
|
36
|
-
pnpm add text-sanctifier
|
|
37
|
-
```
|
|
38
|
-
|
|
39
|
-
---
|
|
40
|
-
|
|
41
|
-
## Quick start
|
|
42
|
-
|
|
43
|
-
**Custom config:**
|
|
44
|
-
|
|
45
|
-
```js
|
|
46
|
-
import { summonSanctifier } from 'text-sanctifier';
|
|
47
|
-
|
|
48
|
-
const clean = summonSanctifier({
|
|
49
|
-
purgeInvisibleChars: true,
|
|
50
|
-
purgeEmojis: true,
|
|
51
|
-
collapseSpaces: true,
|
|
52
|
-
collapseNewLines: true,
|
|
53
|
-
preserveParagraphs: true,
|
|
54
|
-
finalTrim: true,
|
|
55
|
-
});
|
|
56
|
-
|
|
57
|
-
const output = clean(rawText);
|
|
58
|
-
```
|
|
59
|
-
|
|
60
|
-
**Strict preset:**
|
|
61
|
-
|
|
62
|
-
```js
|
|
63
|
-
const output = summonSanctifier.strict(rawText);
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
**Loose preset:**
|
|
67
|
-
|
|
68
|
-
```js
|
|
69
|
-
const output = summonSanctifier.loose(rawText);
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
**Keyboard-only (no emojis):**
|
|
73
|
-
|
|
74
|
-
```js
|
|
75
|
-
const output = summonSanctifier.keyboardOnly(userInput);
|
|
76
|
-
```
|
|
77
|
-
|
|
78
|
-
**Keyboard-only (with emojis):**
|
|
79
|
-
|
|
80
|
-
```js
|
|
81
|
-
const output = summonSanctifier.keyboardOnlyEmoji(commentText);
|
|
82
|
-
```
|
|
83
|
-
|
|
84
|
-
---
|
|
85
|
-
|
|
86
|
-
## Security notes
|
|
87
|
-
|
|
88
|
-
- **Not an HTML/XSS sanitizer.** This library normalizes and filters plain text.
|
|
89
|
-
- If you need to render **untrusted content**, render it as text (e.g. `textContent`), not HTML (`innerHTML`).
|
|
90
|
-
- If you need to sanitize **HTML**, use a dedicated HTML sanitizer (e.g. DOMPurify / sanitize-html).
|
|
91
|
-
- Like any text-processing library, extremely large untrusted inputs can be used for CPU/DoS pressure; consider input size limits in high-risk environments.
|
|
92
|
-
|
|
93
|
-
---
|
|
94
|
-
|
|
95
|
-
## API
|
|
96
|
-
|
|
97
|
-
### `summonSanctifier(options?: SanctifyOptions): (text: string) => string`
|
|
98
|
-
|
|
99
|
-
Creates a reusable sanitizer from an option object.
|
|
100
|
-
|
|
101
|
-
### `summonSanctifier.strict`
|
|
102
|
-
|
|
103
|
-
Aggressively purges: emojis, control characters, extra spacing, and newlines.
|
|
104
|
-
|
|
105
|
-
### `summonSanctifier.loose`
|
|
106
|
-
|
|
107
|
-
Gently normalizes spacing and newlines while preserving emojis and paragraphs.
|
|
108
|
-
|
|
109
|
-
### `summonSanctifier.keyboardOnly`
|
|
110
|
-
|
|
111
|
-
Restricts to printable ASCII only (removes emojis).
|
|
112
|
-
|
|
113
|
-
### `summonSanctifier.keyboardOnlyEmoji`
|
|
114
|
-
|
|
115
|
-
Restricts to printable ASCII + full emoji sequences. Preserves ZWJ emoji clusters and emoji presentation variants.
|
|
116
|
-
|
|
117
|
-
### `inspectText(text: string): UnicodeTrashReport`
|
|
118
|
-
|
|
119
|
-
Returns a structural report of control codes, invisible chars, newline styles, and more.
|
|
120
|
-
|
|
121
|
-
```js
|
|
122
|
-
import { inspectText } from 'text-sanctifier';
|
|
123
|
-
|
|
124
|
-
const report = inspectText(input);
|
|
125
|
-
|
|
126
|
-
/*
|
|
127
|
-
{
|
|
128
|
-
hasControlChars: true,
|
|
129
|
-
hasInvisibleChars: true,
|
|
130
|
-
hasMixedNewlines: false,
|
|
131
|
-
newlineStyle: 'LF',
|
|
132
|
-
hasEmojis: true,
|
|
133
|
-
hasNonKeyboardChars: false,
|
|
134
|
-
summary: [
|
|
135
|
-
'Control characters detected.',
|
|
136
|
-
'Invisible Unicode characters detected.',
|
|
137
|
-
'Emojis detected.',
|
|
138
|
-
'Consistent newline style: LF'
|
|
139
|
-
]
|
|
140
|
-
}
|
|
141
|
-
*/
|
|
142
|
-
```
|
|
143
|
-
|
|
144
|
-
Use `inspectText` to preflight text content before rendering, storing, or linting. It's a diagnostic tool to help inform sanitization needs. Pass the report to `getRecommendedSanctifierOptions(report)` to auto-generate config flags for `summonSanctifier()`.
|
|
145
|
-
|
|
146
|
-
### `inspectTextDetailed(text: string, options?: { maxFindings?: number }): DetailedTrashReport`
|
|
147
|
-
|
|
148
|
-
Returns each invisible or control character with its offset, line, column, code point and category, plus per-category counts and a `truncated` flag. Also exported from the browser build (`text-sanctifier/browser`). `inspectText`'s output is unchanged.
|
|
149
|
-
|
|
150
|
-
```js
|
|
151
|
-
import { inspectTextDetailed } from 'text-sanctifier';
|
|
152
|
-
|
|
153
|
-
const { findings, counts, total, truncated } = inspectTextDetailed(input, { maxFindings: 100 });
|
|
154
|
-
|
|
155
|
-
/*
|
|
156
|
-
findings: [
|
|
157
|
-
{ offset: 0, line: 1, column: 1, codePoint: 0xFEFF, category: 'bom', atStart: true },
|
|
158
|
-
{ offset: 42, line: 3, column: 7, codePoint: 0x2067, category: 'bidi_control' },
|
|
159
|
-
]
|
|
160
|
-
counts: { bidi_control: 1, zero_width: 0, nonbreaking_space: 0, bom: 1, nul: 0, c0_control: 0, c1_control: 0,
|
|
161
|
-
invisible_format: 0, hangul_filler: 0, variation_selector: 0, tag: 0 }
|
|
162
|
-
total: 2
|
|
163
|
-
truncated: false
|
|
164
|
-
*/
|
|
165
|
-
```
|
|
166
|
-
|
|
167
|
-
<!-- categories:start (generated from scripts/categories.js by `npm run gen`; do not edit) -->
|
|
168
|
-
| Category | Code points | Removed by |
|
|
169
|
-
| --- | --- | --- |
|
|
170
|
-
| `bidi_control` | U+061C, U+200E, U+200F, U+202A–U+202E, U+2066–U+2069 | `nukeControls`, `purgeInvisibleChars` |
|
|
171
|
-
| `zero_width` | U+200B–U+200D, U+2060 | `purgeInvisibleChars` |
|
|
172
|
-
| `nonbreaking_space` | U+00A0, U+2000–U+200A, U+202F, U+205F, U+3000 (non-breaking and typographic spaces) | `purgeInvisibleChars` |
|
|
173
|
-
| `bom` | U+FEFF (`atStart` is true at offset 0, where a BOM is legitimate) | `purgeInvisibleChars` |
|
|
174
|
-
| `nul` | U+0000 | `nukeControls` |
|
|
175
|
-
| `c0_control` | U+0001–U+0008, U+000B, U+000C, U+000E–U+001F, U+007F (tab, LF and CR are whitespace, not trash) | `nukeControls` |
|
|
176
|
-
| `c1_control` | U+0080–U+009F | `nukeControls` |
|
|
177
|
-
| `invisible_format` | U+00AD (soft hyphen), U+034F (combining grapheme joiner), U+17B4, U+17B5 (Khmer inherent vowels), U+180E (Mongolian vowel separator), U+2061–U+2064 (invisible operators), U+2065 (reserved), U+206A–U+206F (deprecated format characters), U+FFF0–U+FFF8 (reserved), U+1BCA0–U+1BCA3 (shorthand format controls), U+1D173–U+1D17A (musical formatting), U+E0080–U+E00FF (reserved), U+E01F0–U+E0FFF (reserved) (every other Default_Ignorable_Code_Point, which renderers draw as nothing) | `purgeInvisibleChars` |
|
|
178
|
-
| `hangul_filler` | U+115F, U+1160, U+3164, U+FFA0 (letters that render blank) | `purgeInvisibleChars` |
|
|
179
|
-
| `variation_selector` | U+180B–U+180D (Mongolian FVS1-FVS3), U+180F (Mongolian FVS4), U+FE00–U+FE0F (VS1-VS16, including VS16 in emoji), U+E0100–U+E01EF (VS17-VS256) | `purgeInvisibleChars` |
|
|
180
|
-
| `tag` | U+E0000–U+E007F (tag characters, used for "ASCII smuggling" of hidden instructions) | `purgeInvisibleChars` |
|
|
181
|
-
<!-- categories:end -->
|
|
182
|
-
|
|
183
|
-
`codePoint` is the full code point, so a tag character reports `0xE0041`, not a surrogate. `offset` is a 0-based UTF-16 index (for a code point above U+FFFF, that of its high surrogate); `line` and `column` are 1-based, with columns in UTF-16 code units and lines ending at LF, CR or CRLF. Every occurrence is counted; only the first `maxFindings` (default 100, `Infinity` for all) are listed, and `truncated` says when some were not. The scan is one pass with no regex and no copy of the input.
|
|
184
|
-
|
|
185
|
-
**Choosing `maxFindings`:** the scan itself uses almost no memory, but every listed finding is an object, so the cost of `maxFindings` grows with the number of findings, not the size of the input. 16 MB of non-breaking spaces is 16.7 million findings. With a cap of 20, it scans in about 80 ms and uses 0.2 MB. With `Infinity`, expect roughly 100–200 bytes per finding: about 1.5 GB for 8 million. On untrusted input, keep a cap. Use `maxFindings: 0` for a counts-only report: it builds no finding objects, and `counts`, `total` and `truncated` stay exact.
|
|
186
|
-
|
|
187
|
-
### `getNewlineStats(text: string): NewlineStats`
|
|
188
|
-
|
|
189
|
-
Counts each newline style in one pass, with no regex and no copy: `{ crlf, cr, lf, types, mixed }`. `cr` counts CRs not followed by LF, `lf` counts LFs not preceded by CR, `types` lists the styles present in the order `'CRLF'`, `'CR'`, `'LF'`, and `mixed` is true when there is more than one. Also exported from the browser build.
|
|
190
|
-
|
|
191
|
-
```js
|
|
192
|
-
import { getNewlineStats } from 'text-sanctifier';
|
|
193
|
-
|
|
194
|
-
getNewlineStats('a\r\nb\n'); // { crlf: 1, cr: 0, lf: 1, types: ['CRLF', 'LF'], mixed: true }
|
|
195
|
-
```
|
|
196
|
-
|
|
197
|
-
`zero_width` also flags U+200D (zero width joiner) inside emoji ZWJ sequences such as 👨👩👧, so a finding there is not necessarily hidden text. Filter by neighbouring code points if you need to tell them apart. (`purgeInvisibleChars` removes those joiners too, splitting the emoji; `keyboardOnlyEmoji` keeps the sequences whole.) The same holds for `variation_selector`, which flags VS16 (U+FE0F) in emoji such as ✌️ and the ideographic variation selectors in CJK names, and for `tag`, which flags the tag characters inside subdivision flags such as the England flag (U+1F3F4 followed by tags). `purgeInvisibleChars` strips them all; `keyboardOnlyEmoji` keeps VS16. The Mongolian free variation selectors (U+180B–U+180D, U+180F) are also `variation_selector`: they are legitimate in Mongolian text, which `purgeInvisibleChars` can change, so don't purge Mongolian text with it.
|
|
198
|
-
|
|
199
|
-
The categories cover exactly what `nukeControls` and `purgeInvisibleChars` strip. Every code point with the Unicode property Default_Ignorable_Code_Point (characters renderers must draw as nothing) is in a category, and a test checks this against the running Node's Unicode data.
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
.
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
-
|
|
275
|
-
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
-
|
|
288
|
-
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
1
|
+
# text-sanctifier
|
|
2
|
+
|
|
3
|
+
[](https://www.npmjs.com/package/text-sanctifier)
|
|
4
|
+
[](https://www.npmjs.com/package/text-sanctifier)
|
|
5
|
+
[](https://bundlejs.com/?q=text-sanctifier)
|
|
6
|
+
[](https://github.com/iWhatty/text-sanctifier/blob/main/LICENSE)
|
|
7
|
+
[](https://github.com/iWhatty/text-sanctifier)
|
|
8
|
+
[](https://www.npmjs.com/package/text-sanctifier)
|
|
9
|
+
|
|
10
|
+
Brutal text normalizer and invisible Unicode scrubber for modern web projects. ~2.57 KB gzipped.
|
|
11
|
+
|
|
12
|
+
## Features
|
|
13
|
+
|
|
14
|
+
- Purges zero-width Unicode garbage
|
|
15
|
+
- Purges bidi controls, including the isolates (U+2066–U+2069) and the Arabic letter mark (U+061C) used in Trojan Source attacks
|
|
16
|
+
- Purges Unicode tag characters (U+E0000–U+E007F), the "ASCII smuggling" channel for hiding instructions from humans in text an LLM reads, plus variation selectors, Hangul fillers, soft hyphens and invisible operators
|
|
17
|
+
- Locates invisible and control characters by offset, line and column (`inspectTextDetailed`)
|
|
18
|
+
- `text-sanctifier check` / `fix` command line for CI and coding agents, with JSON or SARIF output, exit codes and pre-commit hooks
|
|
19
|
+
- `check --ai-config` finds and decodes hidden text in agent instruction files (`CLAUDE.md`, `AGENTS.md`, Cursor rules, `SKILL.md`, MCP configuration)
|
|
20
|
+
- Normalizes line endings (CRLF, CR, LF) to LF
|
|
21
|
+
- Collapses unwanted spaces and paragraphs
|
|
22
|
+
- Nukes control characters (if enabled)
|
|
23
|
+
- Smart normalization of typographic junk (quotes, dashes, bullets, full-width punctuation)
|
|
24
|
+
- Keyboard-only filtering (retain printable ASCII + full emoji sequences)
|
|
25
|
+
- Preserves ZWJ emoji clusters (👨👩👧👦)
|
|
26
|
+
- Preserves VS16 emoji presentation variants (✌️, ‼️)
|
|
27
|
+
- Configurable via fine-grained flags or ready-made presets
|
|
28
|
+
- Includes strict, loose, and keyboard-only modes
|
|
29
|
+
- Deterministic RegExp usage (no global `lastIndex` state leaks)
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Install
|
|
34
|
+
|
|
35
|
+
```sh
|
|
36
|
+
pnpm add text-sanctifier
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Quick start
|
|
42
|
+
|
|
43
|
+
**Custom config:**
|
|
44
|
+
|
|
45
|
+
```js
|
|
46
|
+
import { summonSanctifier } from 'text-sanctifier';
|
|
47
|
+
|
|
48
|
+
const clean = summonSanctifier({
|
|
49
|
+
purgeInvisibleChars: true,
|
|
50
|
+
purgeEmojis: true,
|
|
51
|
+
collapseSpaces: true,
|
|
52
|
+
collapseNewLines: true,
|
|
53
|
+
preserveParagraphs: true,
|
|
54
|
+
finalTrim: true,
|
|
55
|
+
});
|
|
56
|
+
|
|
57
|
+
const output = clean(rawText);
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
**Strict preset:**
|
|
61
|
+
|
|
62
|
+
```js
|
|
63
|
+
const output = summonSanctifier.strict(rawText);
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
**Loose preset:**
|
|
67
|
+
|
|
68
|
+
```js
|
|
69
|
+
const output = summonSanctifier.loose(rawText);
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
**Keyboard-only (no emojis):**
|
|
73
|
+
|
|
74
|
+
```js
|
|
75
|
+
const output = summonSanctifier.keyboardOnly(userInput);
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
**Keyboard-only (with emojis):**
|
|
79
|
+
|
|
80
|
+
```js
|
|
81
|
+
const output = summonSanctifier.keyboardOnlyEmoji(commentText);
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## Security notes
|
|
87
|
+
|
|
88
|
+
- **Not an HTML/XSS sanitizer.** This library normalizes and filters plain text.
|
|
89
|
+
- If you need to render **untrusted content**, render it as text (e.g. `textContent`), not HTML (`innerHTML`).
|
|
90
|
+
- If you need to sanitize **HTML**, use a dedicated HTML sanitizer (e.g. DOMPurify / sanitize-html).
|
|
91
|
+
- Like any text-processing library, extremely large untrusted inputs can be used for CPU/DoS pressure; consider input size limits in high-risk environments.
|
|
92
|
+
|
|
93
|
+
---
|
|
94
|
+
|
|
95
|
+
## API
|
|
96
|
+
|
|
97
|
+
### `summonSanctifier(options?: SanctifyOptions): (text: string) => string`
|
|
98
|
+
|
|
99
|
+
Creates a reusable sanitizer from an option object.
|
|
100
|
+
|
|
101
|
+
### `summonSanctifier.strict`
|
|
102
|
+
|
|
103
|
+
Aggressively purges: emojis, control characters, extra spacing, and newlines.
|
|
104
|
+
|
|
105
|
+
### `summonSanctifier.loose`
|
|
106
|
+
|
|
107
|
+
Gently normalizes spacing and newlines while preserving emojis and paragraphs.
|
|
108
|
+
|
|
109
|
+
### `summonSanctifier.keyboardOnly`
|
|
110
|
+
|
|
111
|
+
Restricts to printable ASCII only (removes emojis).
|
|
112
|
+
|
|
113
|
+
### `summonSanctifier.keyboardOnlyEmoji`
|
|
114
|
+
|
|
115
|
+
Restricts to printable ASCII + full emoji sequences. Preserves ZWJ emoji clusters and emoji presentation variants.
|
|
116
|
+
|
|
117
|
+
### `inspectText(text: string): UnicodeTrashReport`
|
|
118
|
+
|
|
119
|
+
Returns a structural report of control codes, invisible chars, newline styles, and more.
|
|
120
|
+
|
|
121
|
+
```js
|
|
122
|
+
import { inspectText } from 'text-sanctifier';
|
|
123
|
+
|
|
124
|
+
const report = inspectText(input);
|
|
125
|
+
|
|
126
|
+
/*
|
|
127
|
+
{
|
|
128
|
+
hasControlChars: true,
|
|
129
|
+
hasInvisibleChars: true,
|
|
130
|
+
hasMixedNewlines: false,
|
|
131
|
+
newlineStyle: 'LF',
|
|
132
|
+
hasEmojis: true,
|
|
133
|
+
hasNonKeyboardChars: false,
|
|
134
|
+
summary: [
|
|
135
|
+
'Control characters detected.',
|
|
136
|
+
'Invisible Unicode characters detected.',
|
|
137
|
+
'Emojis detected.',
|
|
138
|
+
'Consistent newline style: LF'
|
|
139
|
+
]
|
|
140
|
+
}
|
|
141
|
+
*/
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
Use `inspectText` to preflight text content before rendering, storing, or linting. It's a diagnostic tool to help inform sanitization needs. Pass the report to `getRecommendedSanctifierOptions(report)` to auto-generate config flags for `summonSanctifier()`.
|
|
145
|
+
|
|
146
|
+
### `inspectTextDetailed(text: string, options?: { maxFindings?: number }): DetailedTrashReport`
|
|
147
|
+
|
|
148
|
+
Returns each invisible or control character with its offset, line, column, code point and category, plus per-category counts and a `truncated` flag. Also exported from the browser build (`text-sanctifier/browser`). `inspectText`'s output is unchanged.
|
|
149
|
+
|
|
150
|
+
```js
|
|
151
|
+
import { inspectTextDetailed } from 'text-sanctifier';
|
|
152
|
+
|
|
153
|
+
const { findings, counts, total, truncated } = inspectTextDetailed(input, { maxFindings: 100 });
|
|
154
|
+
|
|
155
|
+
/*
|
|
156
|
+
findings: [
|
|
157
|
+
{ offset: 0, line: 1, column: 1, codePoint: 0xFEFF, category: 'bom', atStart: true },
|
|
158
|
+
{ offset: 42, line: 3, column: 7, codePoint: 0x2067, category: 'bidi_control' },
|
|
159
|
+
]
|
|
160
|
+
counts: { bidi_control: 1, zero_width: 0, nonbreaking_space: 0, bom: 1, nul: 0, c0_control: 0, c1_control: 0,
|
|
161
|
+
invisible_format: 0, hangul_filler: 0, variation_selector: 0, tag: 0 }
|
|
162
|
+
total: 2
|
|
163
|
+
truncated: false
|
|
164
|
+
*/
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
<!-- categories:start (generated from scripts/categories.js by `npm run gen`; do not edit) -->
|
|
168
|
+
| Category | Code points | Removed by |
|
|
169
|
+
| --- | --- | --- |
|
|
170
|
+
| `bidi_control` | U+061C, U+200E, U+200F, U+202A–U+202E, U+2066–U+2069 | `nukeControls`, `purgeInvisibleChars` |
|
|
171
|
+
| `zero_width` | U+200B–U+200D, U+2060 | `purgeInvisibleChars` |
|
|
172
|
+
| `nonbreaking_space` | U+00A0, U+2000–U+200A, U+202F, U+205F, U+3000 (non-breaking and typographic spaces) | `purgeInvisibleChars` |
|
|
173
|
+
| `bom` | U+FEFF (`atStart` is true at offset 0, where a BOM is legitimate) | `purgeInvisibleChars` |
|
|
174
|
+
| `nul` | U+0000 | `nukeControls` |
|
|
175
|
+
| `c0_control` | U+0001–U+0008, U+000B, U+000C, U+000E–U+001F, U+007F (tab, LF and CR are whitespace, not trash) | `nukeControls` |
|
|
176
|
+
| `c1_control` | U+0080–U+009F | `nukeControls` |
|
|
177
|
+
| `invisible_format` | U+00AD (soft hyphen), U+034F (combining grapheme joiner), U+17B4, U+17B5 (Khmer inherent vowels), U+180E (Mongolian vowel separator), U+2061–U+2064 (invisible operators), U+2065 (reserved), U+206A–U+206F (deprecated format characters), U+FFF0–U+FFF8 (reserved), U+1BCA0–U+1BCA3 (shorthand format controls), U+1D173–U+1D17A (musical formatting), U+E0080–U+E00FF (reserved), U+E01F0–U+E0FFF (reserved) (every other Default_Ignorable_Code_Point, which renderers draw as nothing) | `purgeInvisibleChars` |
|
|
178
|
+
| `hangul_filler` | U+115F, U+1160, U+3164, U+FFA0 (letters that render blank) | `purgeInvisibleChars` |
|
|
179
|
+
| `variation_selector` | U+180B–U+180D (Mongolian FVS1-FVS3), U+180F (Mongolian FVS4), U+FE00–U+FE0F (VS1-VS16, including VS16 in emoji), U+E0100–U+E01EF (VS17-VS256) | `purgeInvisibleChars` |
|
|
180
|
+
| `tag` | U+E0000–U+E007F (tag characters, used for "ASCII smuggling" of hidden instructions) | `purgeInvisibleChars` |
|
|
181
|
+
<!-- categories:end -->
|
|
182
|
+
|
|
183
|
+
`codePoint` is the full code point, so a tag character reports `0xE0041`, not a surrogate. `offset` is a 0-based UTF-16 index (for a code point above U+FFFF, that of its high surrogate); `line` and `column` are 1-based, with columns in UTF-16 code units and lines ending at LF, CR or CRLF. Every occurrence is counted; only the first `maxFindings` (default 100, `Infinity` for all) are listed, and `truncated` says when some were not. The scan is one pass with no regex and no copy of the input.
|
|
184
|
+
|
|
185
|
+
**Choosing `maxFindings`:** the scan itself uses almost no memory, but every listed finding is an object, so the cost of `maxFindings` grows with the number of findings, not the size of the input. 16 MB of non-breaking spaces is 16.7 million findings. With a cap of 20, it scans in about 80 ms and uses 0.2 MB. With `Infinity`, expect roughly 100–200 bytes per finding: about 1.5 GB for 8 million. On untrusted input, keep a cap. Use `maxFindings: 0` for a counts-only report: it builds no finding objects, and `counts`, `total` and `truncated` stay exact.
|
|
186
|
+
|
|
187
|
+
### `getNewlineStats(text: string): NewlineStats`
|
|
188
|
+
|
|
189
|
+
Counts each newline style in one pass, with no regex and no copy: `{ crlf, cr, lf, types, mixed }`. `cr` counts CRs not followed by LF, `lf` counts LFs not preceded by CR, `types` lists the styles present in the order `'CRLF'`, `'CR'`, `'LF'`, and `mixed` is true when there is more than one. Also exported from the browser build.
|
|
190
|
+
|
|
191
|
+
```js
|
|
192
|
+
import { getNewlineStats } from 'text-sanctifier';
|
|
193
|
+
|
|
194
|
+
getNewlineStats('a\r\nb\n'); // { crlf: 1, cr: 0, lf: 1, types: ['CRLF', 'LF'], mixed: true }
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
`zero_width` also flags U+200D (zero width joiner) inside emoji ZWJ sequences such as 👨👩👧, so a finding there is not necessarily hidden text. Filter by neighbouring code points if you need to tell them apart. (`purgeInvisibleChars` removes those joiners too, splitting the emoji; `keyboardOnlyEmoji` keeps the sequences whole.) The same holds for `variation_selector`, which flags VS16 (U+FE0F) in emoji such as ✌️ and the ideographic variation selectors in CJK names, and for `tag`, which flags the tag characters inside subdivision flags such as the England flag (U+1F3F4 followed by tags). `purgeInvisibleChars` strips them all; `keyboardOnlyEmoji` keeps VS16. The Mongolian free variation selectors (U+180B–U+180D, U+180F) are also `variation_selector`: they are legitimate in Mongolian text, which `purgeInvisibleChars` can change, so don't purge Mongolian text with it.
|
|
198
|
+
|
|
199
|
+
The categories cover exactly what `nukeControls` and `purgeInvisibleChars` strip. Every code point with the Unicode property Default_Ignorable_Code_Point (characters renderers must draw as nothing) is in a category, and a test checks this against the running Node's Unicode data.
|
|
200
|
+
|
|
201
|
+
### `text-sanctifier/hidden`: `findHiddenRuns(text)` and `decodeTagText(text)`
|
|
202
|
+
|
|
203
|
+
The run finder behind [`check --ai-config`](#ai-configuration-files), for tools that read text on its way to a model. It has no Node APIs, so it works in a browser too, but it is a separate entry point and not part of the `text-sanctifier/browser` bundle.
|
|
204
|
+
|
|
205
|
+
```js
|
|
206
|
+
import { findHiddenRuns, decodeTagText } from 'text-sanctifier/hidden';
|
|
207
|
+
|
|
208
|
+
const { runs, benign } = findHiddenRuns(text);
|
|
209
|
+
// runs: [{ kind: 'tag', offset, end, count, decoded: 'Ignore the user' }, ...]
|
|
210
|
+
// benign: [[offset, end], ...] England/Scotland/Wales flags, a lone U+FE0E/U+FE0F after a character
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
- `kind` is `tag` (U+E0000–U+E007F), `variation_selector` (U+FE00–U+FE0F, U+E0100–U+E01EF, U+180B–U+180D, U+180F) or `invisible_operator` (U+2061–U+2064). Consecutive characters of one kind form one run.
|
|
214
|
+
- `offset` and `end` are UTF-16 indexes, `[offset, end)`; `count` is in code points. Only tag runs have `decoded`.
|
|
215
|
+
- `decodeTagText` turns U+E0020–U+E007E into the ASCII they shadow and writes any other character as `<U+XXXX>`.
|
|
216
|
+
- Zero-width, bidi and other invisible characters are not runs; use `inspectTextDetailed` for those.
|
|
217
|
+
|
|
218
|
+
---
|
|
219
|
+
|
|
220
|
+
## Command line
|
|
221
|
+
|
|
222
|
+
The package ships a `text-sanctifier` command for CI and coding agents, which cannot see invisible characters in their own tool output. It runs on Node 18+ and has no dependencies.
|
|
223
|
+
|
|
224
|
+
```sh
|
|
225
|
+
npx text-sanctifier check src docs "**/*.md"
|
|
226
|
+
npx text-sanctifier fix --dry-run src
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
### `check [paths...]`
|
|
230
|
+
|
|
231
|
+
Reports every invisible or control character as `path:line:column U+XXXX category`, using the categories of `inspectTextDetailed`:
|
|
232
|
+
|
|
233
|
+
```
|
|
234
|
+
docs/intro.md:12:7 U+200B zero_width
|
|
235
|
+
src/auth.js:40:3 U+202E bidi_control
|
|
236
|
+
2 findings in 2 of 31 files.
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
- `--json` (or `--format json`): one JSON object on stdout, `{ files, total, filesScanned, filesWithFindings, skipped }`. Each entry in `files` is `{ path, ...inspectTextDetailed(text) }`; only files with findings are listed.
|
|
240
|
+
- `--format sarif`: a [SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html) log for code-scanning viewers and editors. See [SARIF output](#sarif-output).
|
|
241
|
+
- `--max-findings N`: findings listed per file (default 100; `all` for no cap). Counts always cover every occurrence.
|
|
242
|
+
- `--strip-bom`: also report a BOM at offset 0, which is skipped by default.
|
|
243
|
+
- `--ai-config`: check agent instruction files for hidden text. See [AI configuration files](#ai-configuration-files).
|
|
244
|
+
|
|
245
|
+
### AI configuration files
|
|
246
|
+
|
|
247
|
+
`check --ai-config` looks for hidden instructions in the files coding agents read: rules files, skill files and MCP configuration. Tag characters, variation selectors and invisible operators can carry text that people reviewing the file cannot see but a model reads.
|
|
248
|
+
|
|
249
|
+
```sh
|
|
250
|
+
npx text-sanctifier check --ai-config
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
```
|
|
254
|
+
skills/review/SKILL.md:1:17 U+E0049 run of 15 tag characters spelling "Ignore the user"
|
|
255
|
+
skills/review/SKILL.md:2:3 U+FE00 run of 3 variation selectors
|
|
256
|
+
.cursor/rules/style.mdc:4:9 U+200B zero_width
|
|
257
|
+
19 findings (2 hidden runs) in 2 of 6 AI configuration files.
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
- With no paths it searches the current directory; directories you name are searched the same way, and files you name are checked whatever they are called. In a git work tree, files `.gitignore` excludes are skipped.
|
|
261
|
+
- Files it looks for, matched case-insensitively: `CLAUDE.md`, `CLAUDE.local.md`, `AGENTS.md`, `AGENT.md`, `GEMINI.md` and `SKILL.md` at any depth; and at the top of the directory, `.claude/**`, `.cursorrules`, `.cursor/rules/**`, `.cursor/mcp.json`, `.windsurfrules`, `.windsurf/rules/**`, `.clinerules` (file or directory), `.roorules`, `.roo/rules/**`, `.github/copilot-instructions.md`, `.github/{instructions,prompts,chatmodes,agents}/**`, `.mcp.json`, `mcp.json` and `.vscode/mcp.json`.
|
|
262
|
+
- Consecutive tag characters, variation selectors or invisible operators are reported once, as a run, at the first character. Tag runs are decoded: U+E0020–U+E007E become the ASCII they shadow, and other tags are shown as `<U+XXXX>`.
|
|
263
|
+
- Every other invisible or control character is reported as in a plain `check`.
|
|
264
|
+
- Two legitimate sequences are left out: the flags of England, Scotland and Wales (U+1F3F4, tag letters, U+E007F), and a single U+FE0E or U+FE0F after another character (text or emoji presentation). A plain `check` still reports both.
|
|
265
|
+
- `--json` adds `hiddenRuns` (`{ kind, offset, line, column, codePoint, count, decoded?, decodedTruncated? }`, at most `--max-findings` per file) and `hiddenRunCount` to each file, and `hiddenRunCount` to the summary. `kind` is `tag`, `variation_selector` or `invisible_operator`; `decoded` is cut at 1000 characters. `total` and `counts` leave out the legitimate sequences.
|
|
266
|
+
|
|
267
|
+
### SARIF output
|
|
268
|
+
|
|
269
|
+
`check --format sarif` (with or without `--ai-config`) prints one SARIF 2.1.0 log with one result per listed finding:
|
|
270
|
+
|
|
271
|
+
- `ruleId` is the category, and `tool.driver.rules` lists every category.
|
|
272
|
+
- `level`: `error` for `bidi_control` and `tag`, `note` for `nonbreaking_space`, `warning` for the rest.
|
|
273
|
+
- Each location has the file (relative to `%SRCROOT%`, the current directory), `startLine`, `startColumn` and `endColumn`. Columns count UTF-16 code units, as in `inspectTextDetailed` (`columnKind: "utf16CodeUnits"`).
|
|
274
|
+
- With `--ai-config`, the message of the first result in a run describes the run and its decoded text.
|
|
275
|
+
- `--max-findings` caps results per file as usual. `runs[0].properties` holds `total`, `filesScanned`, `filesWithFindings`, `truncated` (files with findings left out) and `skipped`.
|
|
276
|
+
|
|
277
|
+
### pre-commit
|
|
278
|
+
|
|
279
|
+
The repository publishes a [pre-commit](https://pre-commit.com) hook manifest, so the check runs locally on the files being committed:
|
|
280
|
+
|
|
281
|
+
```yaml
|
|
282
|
+
# .pre-commit-config.yaml
|
|
283
|
+
repos:
|
|
284
|
+
- repo: https://github.com/iWhatty/text-sanctifier
|
|
285
|
+
rev: v1.3.2 # a release tag
|
|
286
|
+
hooks:
|
|
287
|
+
- id: text-sanctifier # every staged text file
|
|
288
|
+
- id: text-sanctifier-ai-config # agent instruction files, with --ai-config
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
- `text-sanctifier` runs `text-sanctifier check` on staged text files. Use `exclude:` for files that legitimately contain these characters, or `args: [--strip-bom]` and the other `check` options.
|
|
292
|
+
- `text-sanctifier-ai-config` runs `text-sanctifier check --ai-config` on staged files that match the `--ai-config` file list.
|
|
293
|
+
- A hook fails the commit when it finds something (exit code 1). Neither hook changes files; run `text-sanctifier fix` yourself.
|
|
294
|
+
|
|
295
|
+
### `fix [paths...]`
|
|
296
|
+
|
|
297
|
+
Rewrites files in place and reports what it removed.
|
|
298
|
+
|
|
299
|
+
- `--purge-invisible`: `purgeInvisibleChars`
|
|
300
|
+
- `--nuke-controls`: `nukeControls`
|
|
301
|
+
- `--normalize-newlines`: `normalizeNewlines` (CRLF and CR to LF)
|
|
302
|
+
- With none of the three, `fix` uses `--purge-invisible --nuke-controls`, which removes exactly what `check` reports.
|
|
303
|
+
- `--strip-bom`: also remove a BOM at offset 0. It is kept by default.
|
|
304
|
+
- `--dry-run`: write nothing; report what would change.
|
|
305
|
+
- `--json` (or `--format json`): `{ files, dryRun, filesScanned, filesChanged, skipped }`, where each file is `{ path, removed, removedTotal, newlinesNormalized }`.
|
|
306
|
+
|
|
307
|
+
`fix` deletes characters; it does not replace them. A non-breaking space between two words joins them, and emoji lose their zero width joiners and variation selectors. Run `check` or `fix --dry-run` first on prose.
|
|
308
|
+
|
|
309
|
+
### Inputs
|
|
310
|
+
|
|
311
|
+
- Paths can be files, directories or globs (`*`, `**`, `?`, `[...]`, `{a,b}`). Quote globs so the shell passes them through.
|
|
312
|
+
- Inside a git work tree, directories and globs list files with `git ls-files`, so anything `.gitignore` excludes is skipped. Outside git, `.git` and `node_modules` are skipped. A file named explicitly is always read.
|
|
313
|
+
- Binary files (a NUL byte in the first 8000 bytes) and files that are not valid UTF-8 are skipped, and `fix` never writes them. In JSON they appear under `skipped`.
|
|
314
|
+
- `-`, or no paths with piped input, reads stdin. `fix` writes the cleaned text to stdout and the report to stderr.
|
|
315
|
+
|
|
316
|
+
### Exit codes
|
|
317
|
+
|
|
318
|
+
| Code | `check` | `fix` |
|
|
319
|
+
| --- | --- | --- |
|
|
320
|
+
| 0 | nothing found | done (or nothing to do) |
|
|
321
|
+
| 1 | findings | `--dry-run` with changes pending |
|
|
322
|
+
| 2 | usage error, missing path or unreadable file | same |
|
|
323
|
+
|
|
324
|
+
---
|
|
325
|
+
|
|
326
|
+
## Notes
|
|
327
|
+
|
|
328
|
+
### Runtime requirements
|
|
329
|
+
|
|
330
|
+
Requires a modern JavaScript runtime with ES2020+ support:
|
|
331
|
+
|
|
332
|
+
- Node.js 18+ (`engines.node >=18`)
|
|
333
|
+
- Modern evergreen browsers
|
|
334
|
+
|
|
335
|
+
### Package and build info
|
|
336
|
+
|
|
337
|
+
- **Source (`src/`):** ES2020+ ESM modules with JSDoc
|
|
338
|
+
- **ESM only:** no CommonJS build; `require('text-sanctifier')` works only on Node versions with `require(esm)` support (20.19+, 22.12+)
|
|
339
|
+
- **Types:** TypeScript declarations ship in `dist/types`
|
|
340
|
+
- **Browser build (`dist/`):** Minified ESM bundle for `<script type="module">`
|
|
341
|
+
- **Tree-shake friendly:** ships `sideEffects: false`
|
|
342
|
+
- **Zero transpilation:** no built-in polyfills or runtime overhead
|
|
343
|
+
- **Bundler ready:** works with Vite, Rollup, Webpack, Parcel, esbuild
|
|
344
|
+
|
|
345
|
+
---
|
|
346
|
+
|
|
347
|
+
## License
|
|
348
|
+
|
|
349
|
+
Licensed under AGPL-3.0 with WATT3D Additional Terms. See [LICENSE](./LICENSE) and [ADDITIONAL_TERMS.md](./ADDITIONAL_TERMS.md). Commercial AI/model-training use requires compliance with those terms or a separate WATT3D license. © WATT3D.
|