text-sanctifier 1.3.9 → 1.3.10

package/README.md

@@ -7,28 +7,13 @@
 [![stars](https://badgen.net/github/stars/iWhatty/text-sanctifier)](https://github.com/iWhatty/text-sanctifier)
 [![types](https://badgen.net/npm/types/text-sanctifier)](https://www.npmjs.com/package/text-sanctifier)
 
-Brutal text normalizer and invisible Unicode scrubber for modern web projects. The prebuilt browser file stays under <!-- gen:budget:browser -->3 KiB<!-- /gen --> gzipped, a budget `npm run ci` enforces; with a bundler you pay only for what you import.
+Find and remove invisible Unicode in web input, model-bound text and repositories. The prebuilt browser file stays under <!-- gen:budget:browser -->3 KiB<!-- /gen --> gzipped; bundlers include only the imports you use.
 
 ## Features
 
-- Purges zero-width Unicode garbage
-- Purges bidi controls, including the isolates (U+2066–U+2069) and the Arabic letter mark (U+061C) used in Trojan Source attacks
-- Purges Unicode tag characters (U+E0000–U+E007F), the "ASCII smuggling" channel for hiding instructions from humans in text an LLM reads, plus variation selectors, Hangul fillers, soft hyphens and invisible operators
-- Locates invisible and control characters by offset, line and column (`inspectTextDetailed`)
-- Shows hidden characters as visible markers such as `<U+200B>` (`revealInvisible`, `text-sanctifier reveal`)
-- Cleans pasted text and model output without gluing words together: look-alike spaces such as the narrow no-break space (U+202F) become ordinary spaces (`summonSanctifier.pasted`)
-- `text-sanctifier check` / `fix` command line for CI and coding agents, with JSON or SARIF output, exit codes and pre-commit hooks
-- `check --ai-config` finds and decodes hidden text in agent instruction files (`CLAUDE.md`, `AGENTS.md`, Cursor rules, `SKILL.md`, MCP configuration)
-- Normalizes line endings (CRLF, CR, LF) to LF
-- Collapses unwanted spaces and paragraphs
-- Nukes control characters (if enabled)
-- Smart normalization of typographic junk (quotes, dashes, bullets, full-width punctuation)
-- Keyboard-only filtering (retain printable ASCII + full emoji sequences)
-  - Preserves ZWJ emoji clusters (the family emoji, U+1F468 U+200D U+1F469 U+200D U+1F467 U+200D U+1F466)
-  - Preserves VS16 emoji presentation variants (U+270C U+FE0F, U+203C U+FE0F)
-- Configurable via fine-grained flags or ready-made presets
-- Includes strict, loose, and keyboard-only modes
-- Deterministic RegExp usage (no global `lastIndex` state leaks)
+- **Clean:** presets and options for invisible and control characters, including the bidi controls behind Trojan Source attacks and the tag characters behind "ASCII smuggling" to LLMs, plus look-alike spaces, emoji and newlines.
+- **Inspect:** exact offsets, lines, columns and categories; visible markers; context-aware findings and decoded hidden tag text.
+- **CLI:** repository and AI configuration checks, fixes, JSON or SARIF output, exit codes and local pre-commit hooks.
 
 ---
 
@@ -36,65 +21,48 @@ Brutal text normalizer and invisible Unicode scrubber for modern web projects. T
 
 ```sh
 pnpm add text-sanctifier
+# or
+npm i text-sanctifier
 ```
 
----
-
-## Quick start
+## Which import do I use?
 
-**Custom config:**
-
-```js
-import { summonSanctifier } from 'text-sanctifier';
+| Job | Import or command |
+| --- | --- |
+| Sanitize text with presets | `import { sanctifyPasted, sanctifyStrict } from 'text-sanctifier'` |
+| Find and locate characters | `import { inspectTextDetailed } from 'text-sanctifier'` |
+| Emoji-aware findings | `import { inspectTextInContext } from 'text-sanctifier/context'` |
+| Text headed to a model | `import { findHiddenRuns, decodeTagText } from 'text-sanctifier/hidden'` |
+| No build step | `import { sanctifyPasted } from 'text-sanctifier/browser'` (CDN example below) |
+| Repository checks | `npx text-sanctifier check` |
 
-const clean = summonSanctifier({
-  purgeInvisibleChars: true,
-  purgeEmojis: true,
-  collapseSpaces: true,
-  collapseNewLines: true,
-  preserveParagraphs: true,
-  finalTrim: true,
-});
-
-const output = clean(rawText);
-```
-
-**Strict preset:**
-
-```js
-const output = summonSanctifier.strict(rawText);
-```
-
-**Loose preset:**
-
-```js
-const output = summonSanctifier.loose(rawText);
-```
-
-**Keyboard-only (no emojis):**
-
-```js
-const output = summonSanctifier.keyboardOnly(userInput);
-```
+## Quick start
 
-**Keyboard-only (with emojis):**
+### Web
 
 ```js
-const output = summonSanctifier.keyboardOnlyEmoji(commentText);
+import { sanctifyPasted } from 'text-sanctifier';
+const input = 'Hello\u202Fworld\u200B';
+const clean = sanctifyPasted(input); // 'Hello world'
+document.querySelector('#preview').textContent = clean;
 ```
 
-**Pasted text and model output:**
+Without a build step, use the browser file from a CDN:
 
-```js
-const output = summonSanctifier.pasted(clipboardText);
+```html
+<div id="preview"></div>
+<script type="module">
+  import { sanctifyPasted } from 'https://cdn.jsdelivr.net/npm/text-sanctifier/dist/text-sanctifier.min.js';
+  document.querySelector('#preview').textContent = sanctifyPasted('Hello\u200B world');
+</script>
 ```
 
-**Show what is hidden:**
-
-```js
-import { revealInvisible } from 'text-sanctifier';
+### Agents/CI
 
-revealInvisible(text); // 'Hello<U+200B>world'
+```sh
+npx text-sanctifier check src docs
+npx text-sanctifier check --ai-config
+# check exits: 0 clean, 1 findings, 2 usage or I/O error
 ```
 
 ---
@@ -104,7 +72,7 @@ revealInvisible(text); // 'Hello<U+200B>world'
 - **Not an HTML/XSS sanitizer.** This library normalizes and filters plain text.
 - If you need to render **untrusted content**, render it as text (e.g. `textContent`), not HTML (`innerHTML`).
 - If you need to sanitize **HTML**, use a dedicated HTML sanitizer (e.g. DOMPurify / sanitize-html).
-- Like any text-processing library, extremely large untrusted inputs can be used for CPU/DoS pressure; consider input size limits in high-risk environments.
+- Like any text-processing library, extremely large untrusted inputs can be used for CPU/DoS pressure; set input size limits with [`maxLength` and `onMaxLength`](#sanctifyoptions) in high-risk environments. Inspection caps limit output, not input length or scan time.
 
 ---
 
@@ -119,6 +87,36 @@ Two options handle spaces and zero-width characters gently, and both run before
 - `normalizeSpaces`: replace look-alike spaces with an ordinary space. These are U+00A0, U+2000–U+200A, U+202F and U+205F: the `nonbreaking_space` category except U+3000, the ordinary space of CJK text. With `purgeInvisibleChars` alone, a non-breaking space between two words is deleted and the words run together; with `normalizeSpaces` as well, they stay apart.
 - `purgeZeroWidthSpaces`: remove the zero width space U+200B, the word joiner U+2060 and U+FEFF, and nothing else. Emoji joiners, ZWNJ, variation selectors and bidi marks are kept.
 
+### `SanctifyOptions`
+
+Used by `summonSanctifier(options)`. All boolean options default to `false`. Processing follows table order after the input-length check.
+
+| Option | Behavior |
+| --- | --- |
+| `normalizeSpaces` | Replace U+00A0, U+2000-U+200A, U+202F and U+205F with ordinary spaces; keep U+3000. |
+| `purgeZeroWidthSpaces` | Remove only U+200B, U+2060 and U+FEFF. |
+| `purgeInvisibleChars` | Remove characters marked `purgeInvisibleChars` in the category table, including emoji joiners and selectors. |
+| `purgeEmojis` | Remove matched emoji sequences. |
+| `nukeControls` | Remove NUL, non-whitespace C0 controls, DEL, C1 controls and bidi controls; keep tab, LF and CR. |
+| `keyboardOnlyFilter` | Normalize typographic punctuation, then keep printable ASCII, CR/LF and matched emoji sequences; with `purgeEmojis`, keep no emoji. Deletes non-ASCII letters. |
+| `normalizeNewlines` | Convert CRLF and CR to LF. |
+| `trimSpacesAroundNewlines` | Remove spaces and tabs immediately around LF runs. |
+| `collapseNewLines` | Collapse runs of two or more LFs to one, or three or more to two with `preserveParagraphs`. |
+| `preserveParagraphs` | Keep double LF breaks when `collapseNewLines` is enabled; otherwise no effect. |
+| `collapseSpaces` | Collapse runs of ordinary spaces to one; keep tabs and newlines. |
+| `finalTrim` | Apply JavaScript `.trim()` to the result. |
+| `maxLength` | Input cap in UTF-16 code units, checked before processing. Default `Infinity`; negative, non-finite or non-number values mean no cap. |
+| `onMaxLength` | Default `'throw'`: throw `RangeError` above the cap. `'truncate'`: slice input to the cap before processing (may split a surrogate pair). `'noop'`: return original input unchanged. Invalid values use `'throw'`. |
+
+```js
+import { summonSanctifier } from 'text-sanctifier';
+const clean = summonSanctifier({ purgeInvisibleChars: true, maxLength: 10000, onMaxLength: 'throw' });
+```
+
+### Errors
+
+All text-processing exports throw `TypeError` on non-string input, including the preset functions, the sanitizer returned by `summonSanctifier`, `findHiddenRuns` and `decodeTagText`. Inputs are not coerced to strings. Sanitizers also throw `RangeError` when input exceeds `maxLength` with `onMaxLength: 'throw'`.
+
 ### Presets as standalone imports
 
 Each preset is also its own export: `sanctifyStrict`, `sanctifyLoose`, `sanctifyKeyboardOnly`, `sanctifyKeyboardOnlyEmoji` and `sanctifyPasted`. They are the same functions as `summonSanctifier.strict` and the rest. Import one directly and a bundler leaves out `summonSanctifier` and the other presets (about <!-- gen:size:sanctifyPasted -->1.1 KiB<!-- /gen --> gzipped instead of <!-- gen:size:summonSanctifier.pasted -->1.4 KiB<!-- /gen -->).
@@ -131,19 +129,19 @@ const output = sanctifyPasted(clipboardText);
 
 ### `summonSanctifier.strict`
 
-Aggressively purges: emojis, control characters, extra spacing, and newlines.
+Purges invisible characters, emoji and non-whitespace controls. Normalizes CRLF and CR to LF, removes spaces/tabs around LF, collapses blank lines to a single LF, collapses ordinary spaces and trims the ends. Single line breaks remain.
 
 ### `summonSanctifier.loose`
 
-Gently normalizes spacing and newlines while preserving emojis and paragraphs.
+Normalizes spacing and newlines while preserving emoji and paragraph breaks (LF runs collapse to at most two). Keeps zero-width characters such as U+200B, U+200C, U+200D and U+2060; it does not enable `purgeInvisibleChars`.
 
 ### `summonSanctifier.keyboardOnly`
 
-Restricts to printable ASCII only (removes emojis).
+Normalizes typographic punctuation, then keeps printable ASCII and line breaks; removes emoji and non-ASCII letters.
 
 ### `summonSanctifier.keyboardOnlyEmoji`
 
-Restricts to printable ASCII + full emoji sequences. Preserves ZWJ emoji clusters and emoji presentation variants.
+Normalizes typographic punctuation, then keeps printable ASCII, line breaks and matched emoji sequences, including ZWJ clusters and emoji presentation variants. Both keyboard-only presets delete non-ASCII letters: `sanctifyKeyboardOnlyEmoji('caf\u00E9')` returns `'caf'`, not an ASCII transliteration.
 
 ### `summonSanctifier.pasted`
 
@@ -228,7 +226,11 @@ truncated: false
 | `tag` | U+E0000–U+E007F (tag characters, used for "ASCII smuggling" of hidden instructions) | `purgeInvisibleChars` |
 <!-- categories:end -->
 
-`codePoint` is the full code point, so a tag character reports `0xE0041`, not a surrogate. `offset` is a 0-based UTF-16 index (for a code point above U+FFFF, that of its high surrogate); `line` and `column` are 1-based, with columns in UTF-16 code units and lines ending at LF, CR or CRLF. Every occurrence is counted; only the first `maxFindings` (default 100, `Infinity` for all) are listed, and `truncated` says when some were not. The scan is one pass with no regex and no copy of the input.
+`zero_width` also flags U+200D (zero width joiner) inside emoji ZWJ sequences such as U+1F468 U+200D U+1F469 U+200D U+1F467, so a finding there is not necessarily hidden text. Use [`text-sanctifier/context`](#text-sanctifiercontext-inspecttextincontexttext-options) to mark supported legitimate sequences as benign. (`purgeInvisibleChars` removes those joiners too, splitting the emoji; `keyboardOnlyEmoji` keeps the sequences whole.) The same holds for `variation_selector`, which flags VS16 (U+FE0F) in emoji such as U+270C U+FE0F and the ideographic variation selectors in CJK names, and for `tag`, which flags the tag characters inside subdivision flags such as the England flag (U+1F3F4 followed by tags). `purgeInvisibleChars` strips them all; `keyboardOnlyEmoji` keeps VS16. The Mongolian free variation selectors (U+180B–U+180D, U+180F) are also `variation_selector`: they are legitimate in Mongolian text, which `purgeInvisibleChars` can change, so don't purge Mongolian text with it.
+
+The categories cover exactly what `nukeControls` and `purgeInvisibleChars` strip. Every code point with the Unicode property Default_Ignorable_Code_Point (characters renderers must draw as nothing) is in a category, and a test checks this against the running Node's Unicode data.
+
+`codePoint` is the full code point, so a tag character reports `0xE0041`, not a surrogate. `offset` is a 0-based UTF-16 index (for a code point above U+FFFF, that of its high surrogate); `line` and `column` are 1-based, with columns in UTF-16 code units and lines ending at LF, CR or CRLF. Every occurrence is counted; only the first `maxFindings` (default 100, `Infinity` for all; invalid values such as negative numbers, fractions, strings, `NaN` or `null` fall back to 100) are listed, and `truncated` says when some were not. The scan is one pass with no regex and no copy of the input.
 
 **Choosing `maxFindings`:** the scan itself uses almost no memory, but every listed finding is an object, so the cost of `maxFindings` grows with the number of findings, not the size of the input. 16 MB of non-breaking spaces is 16.7 million findings. With a cap of 20, it scans in about 80 ms and uses 0.2 MB (`npm run bench` measures it on your machine). With `Infinity`, expect roughly 100–200 bytes per finding: about 1.5 GB for 8 million. On untrusted input, keep a cap. Use `maxFindings: 0` for a counts-only report: it builds no finding objects, and `counts`, `total` and `truncated` stay exact.
 
@@ -242,10 +244,6 @@ import { getNewlineStats } from 'text-sanctifier';
 getNewlineStats('a\r\nb\n'); // { crlf: 1, cr: 0, lf: 1, types: ['CRLF', 'LF'], mixed: true }
 ```
 
-`zero_width` also flags U+200D (zero width joiner) inside emoji ZWJ sequences such as U+1F468 U+200D U+1F469 U+200D U+1F467, so a finding there is not necessarily hidden text. Filter by neighbouring code points if you need to tell them apart. (`purgeInvisibleChars` removes those joiners too, splitting the emoji; `keyboardOnlyEmoji` keeps the sequences whole.) The same holds for `variation_selector`, which flags VS16 (U+FE0F) in emoji such as U+270C U+FE0F and the ideographic variation selectors in CJK names, and for `tag`, which flags the tag characters inside subdivision flags such as the England flag (U+1F3F4 followed by tags). `purgeInvisibleChars` strips them all; `keyboardOnlyEmoji` keeps VS16. The Mongolian free variation selectors (U+180B–U+180D, U+180F) are also `variation_selector`: they are legitimate in Mongolian text, which `purgeInvisibleChars` can change, so don't purge Mongolian text with it.
-
-The categories cover exactly what `nukeControls` and `purgeInvisibleChars` strip. Every code point with the Unicode property Default_Ignorable_Code_Point (characters renderers must draw as nothing) is in a category, and a test checks this against the running Node's Unicode data.
-
 ### `text-sanctifier/hidden`: `findHiddenRuns(text, options?)` and `decodeTagText(text)`
 
 The run finder behind [`check --ai-config`](#ai-configuration-files), for tools that read text on its way to a model. It has no Node APIs, so it works in a browser too, but it is a separate entry point and not part of the `text-sanctifier/browser` bundle.
@@ -261,7 +259,7 @@ const { runs, benign, counts, total, truncated } = findHiddenRuns(text, { maxRun
 
 - `kind` is `tag` (U+E0000–U+E007F), `variation_selector` (U+FE00–U+FE0F, U+E0100–U+E01EF, U+180B–U+180D, U+180F) `invisible_operator` (U+2061–U+2064) or `reserved_tag_plane` (U+E0080–U+E00FF, U+E01F0–U+E0FFF: unassigned, default ignorable, so they render as nothing and can carry data). Consecutive characters of one kind form one run.
 - `offset` and `end` are UTF-16 indexes, `[offset, end)`; `count` is in code points. `line` and `column` are counted exactly as in `inspectTextDetailed` (1-based, columns in UTF-16 units, CRLF is one line break). Only tag runs have `decoded` and `decodedTruncated`.
-- `maxRuns` caps the `runs` list and, separately, the `benign` list; `counts`, `total` and `benignTotal` count everything, and `truncated` is true when either list was cut. `maxDecodedLength` caps each `decoded` string in UTF-16 units and never cuts a `<U+XXXX>` marker. Both default to no cap, and an invalid value (negative, fractional, a string) also means no cap, so set them explicitly for untrusted input. With both set, memory stays flat on any input: on 16 M characters (32 MB) of hidden text the heap grows by under 1 MB, against hundreds of MB with no caps.
+- `maxRuns` caps the `runs` list and, separately, the `benign` list; `counts`, `total` and `benignTotal` count everything, and `truncated` is true when either list was cut. `maxDecodedLength` caps each `decoded` string in UTF-16 units and never cuts a `<U+XXXX>` marker. Both default to no cap. Only non-negative integers and `Infinity` are valid; invalid values (negative, fractional, a string, `NaN` or `null`) also mean no cap, so set them explicitly for untrusted input. With both set, memory stays flat on any input: on 16 M characters (32 MB) of hidden text the heap grows by under 1 MB, against hundreds of MB with no caps.
 - `decodeTagText` turns U+E0020–U+E007E into the ASCII they shadow and writes any other character as `<U+XXXX>`.
 - Zero-width, bidi and other invisible characters are not runs; use `inspectTextDetailed` for those.
 - `codePointCounts` has the same keys as `counts` and counts the code points in those runs, so long runs weigh more than short ones; `supplementarySelectorCount` counts U+E0100–U+E01EF on their own, apart from the emoji selectors U+FE00–U+FE0F. Both leave out benign spans and stay exact under `maxRuns`.
@@ -299,6 +297,8 @@ U+200B, U+2060, U+FEFF, bidi controls and all other tags are never benign, and n
 The package ships a `text-sanctifier` command for CI and coding agents, which cannot see invisible characters in their own tool output. It runs on Node 18+ and has no dependencies.
 
 ```sh
+npx text-sanctifier --help
+npx text-sanctifier --version
 npx text-sanctifier check src docs "**/*.md"
 npx text-sanctifier fix --dry-run src
 npx text-sanctifier reveal notes.md
@@ -349,6 +349,7 @@ skills/review/SKILL.md:2:3  U+FE00  run of 3 variation selectors
 - `ruleId` is the category, and `tool.driver.rules` lists every category.
 - `level`: `error` for `bidi_control` and `tag`, `note` for `nonbreaking_space`, `warning` for the rest.
 - Each location has the file (relative to `%SRCROOT%`, the current directory), `startLine`, `startColumn` and `endColumn`. Columns count UTF-16 code units, as in `inspectTextDetailed` (`columnKind: "utf16CodeUnits"`).
+- Each result has a `partialFingerprints` hash based on its rule, relative path, line content and occurrence on that line. Adding unrelated lines above does not change it.
 - With `--ai-config`, the message of the first result in a run describes the run and its decoded text.
 - `--max-findings` caps results per file as usual. `runs[0].properties` holds `total`, `filesScanned`, `filesWithFindings`, `truncated` (files with findings left out) and `skipped`.
 
@@ -360,7 +361,7 @@ The repository publishes a [pre-commit](https://pre-commit.com) hook manifest, s
 # .pre-commit-config.yaml
 repos:
   - repo: https://github.com/iWhatty/text-sanctifier
-    rev: v1.3.9  # a release tag
+    rev: v1.3.10  # a release tag
     hooks:
       - id: text-sanctifier            # every staged text file
       - id: text-sanctifier-ai-config  # agent instruction files, with --ai-config
@@ -376,6 +377,7 @@ Rewrites files in place and reports what it replaced and removed.
 
 - `--normalize-spaces`: `normalizeSpaces` (look-alike spaces become ordinary spaces)
 - `--purge-invisible`: `purgeInvisibleChars`
+- `--keep-benign`: preserve contextually legitimate joiners, selectors and flag tags when purging invisible characters. Works with the default cleaning flags too.
 - `--nuke-controls`: `nukeControls`
 - `--normalize-newlines`: `normalizeNewlines` (CRLF and CR to LF)
 - With none of these, `fix` uses `--normalize-spaces --purge-invisible --nuke-controls`: look-alike spaces become ordinary spaces, and everything else `check` reports is removed.
@@ -383,7 +385,7 @@ Rewrites files in place and reports what it replaced and removed.
 - `--dry-run`: write nothing; report what would change.
 - `--json` (or `--format json`): `{ files, dryRun, filesScanned, filesChanged, skipped }`, where each file is `{ path, spacesReplaced, removed, removedTotal, newlinesNormalized }`.
 
-Apart from look-alike spaces, `fix` deletes characters. Emoji lose their zero width joiners and variation selectors, and so does Mongolian text. Run `check` or `fix --dry-run` first on prose. To make hidden characters visible without changing a file, use `reveal`.
+Apart from look-alike spaces, `fix` deletes characters. By default, emoji lose their zero width joiners and variation selectors, and so does Mongolian text. Use `--keep-benign` to preserve recognized legitimate sequences, or run `check` or `fix --dry-run` first on prose. To make hidden characters visible without changing a file, use `reveal`.
 
 ### `reveal [paths...]`
 

package/bin/text-sanctifier.js

@@ -14,10 +14,13 @@ import { readFileSync, writeFileSync, statSync, readdirSync, existsSync } from '
 import { resolve, join, sep, relative } from 'node:path';
 import { spawnSync } from 'node:child_process';
 import { pathToFileURL } from 'node:url';
+import { createHash } from 'node:crypto';
 import { inspectTextDetailed, summonSanctifier, revealInvisible } from '../src/index.js';
 import { getNewlineStats } from '../src/inspectText.js';
 import { SPACE_LOOKALIKES_REGEX } from '../src/sanctifyText.js';
-import { findHiddenRuns } from '../src/hiddenText.js';
+import { scanHiddenSpans } from '../src/hiddenScan.js';
+import { inspectTextInContext } from '../src/context.js';
+import { classifyCodeUnit, classifyAstral } from '../src/categories.generated.js';
 
 const VERSION = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version;
 
@@ -46,6 +49,7 @@ fix options:
   --normalize-spaces   replace look-alike spaces (NBSP, narrow NBSP, typographic
                        spaces) with an ordinary space instead of removing them
   --purge-invisible    remove zero-width, non-breaking space, bidi and BOM characters
+  --keep-benign        keep contextually legitimate joiners and selectors when purging
   --nuke-controls      remove C0/C1 control characters and bidi controls
   --normalize-newlines convert CRLF and CR to LF
   --strip-bom          also remove a BOM at the start of a file (kept by default)
@@ -86,11 +90,12 @@ const BOOLEAN_FLAGS = {
     '--strip-bom': 'stripBom',
     '--normalize-spaces': 'normalizeSpaces',
     '--purge-invisible': 'purgeInvisible',
+    '--keep-benign': 'keepBenign',
     '--nuke-controls': 'nukeControls',
     '--normalize-newlines': 'normalizeNewlines',
     '--dry-run': 'dryRun',
 };
-const FIX_ONLY_FLAGS = new Set(['normalizeSpaces', 'purgeInvisible', 'nukeControls', 'normalizeNewlines', 'dryRun']);
+const FIX_ONLY_FLAGS = new Set(['normalizeSpaces', 'purgeInvisible', 'keepBenign', 'nukeControls', 'normalizeNewlines', 'dryRun']);
 const CHECK_ONLY_FLAGS = new Set(['aiConfig']);
 const FORMATS = new Set(['text', 'json', 'sarif']);
 
@@ -106,6 +111,7 @@ function parseArgs(argv) {
         stripBom: false,
         normalizeSpaces: false,
         purgeInvisible: false,
+        keepBenign: false,
         nukeControls: false,
         normalizeNewlines: false,
         dryRun: false,
@@ -458,46 +464,56 @@ const RUN_NOUNS = { tag: 'tag character', variation_selector: 'variation selecto
  * @param {boolean} stripBom
  */
 function inspectAiConfig(text, maxFindings, stripBom) {
-    const full = inspect(text, Infinity, stripBom);
-    const { runs, benign } = findHiddenRuns(text);
-
-    const kept = [];
-    let b = 0;
-    for (const f of full.findings) {
-        while (b < benign.length && benign[b][1] <= f.offset) b++;
-        if (b < benign.length && benign[b][0] <= f.offset) { full.counts[f.category]--; continue; }
-        kept.push(f);
-    }
-
-    // Every run character is a finding, so each run starts at a kept finding
-    const hiddenRuns = [];
+    const full = inspect(text, 0, stripBom);
+    const findings = [];
     const loose = [];
-    let r = 0;
-    for (const f of kept) {
-        while (r < runs.length && runs[r].end <= f.offset) r++;
-        const run = runs[r];
-        if (!run || f.offset < run.offset) { loose.push(f); continue; }
-        if (f.offset !== run.offset) continue;
-        const entry = { kind: run.kind, offset: run.offset, line: f.line, column: f.column, codePoint: f.codePoint, count: run.count };
+    let looseCount = 0;
+    let position = 0, line = 1, lineStart = 0;
+    const scan = (end, inRun, benign) => {
+        while (position < end) {
+            const offset = position;
+            const codePoint = text.codePointAt(position);
+            position += codePoint > 0xFFFF ? 2 : 1;
+            if (codePoint === 10 || (codePoint === 13 && text.charCodeAt(position) !== 10)) {
+                line++; lineStart = position; continue;
+            }
+            const category = codePoint > 0xFFFF ? classifyAstral(codePoint) : classifyCodeUnit(codePoint);
+            if (!category || (offset === 0 && codePoint === BOM && !stripBom)) continue;
+            if (benign) { full.counts[category]--; full.total--; continue; }
+            if (findings.length < maxFindings || (!inRun && loose.length < maxFindings)) {
+                const f = { offset, line, column: offset - lineStart + 1, codePoint, category };
+                if (category === 'bom') f.atStart = offset === 0;
+                if (findings.length < maxFindings) findings.push(f);
+                if (!inRun && loose.length < maxFindings) loose.push(f);
+            }
+            if (!inRun) looseCount++;
+        }
+    };
+    // Decode beyond the cut point by one marker so slicing matches the original output.
+    const hidden = scanHiddenSpans(text, { maxRuns: maxFindings, maxDecodedLength: MAX_DECODED + 10 }, (span) => {
+        scan(span.offset, false, false);
+        scan(span.end, !span.benign, span.benign);
+    });
+    scan(text.length, false, false);
+    const hiddenRuns = hidden.runs.map((run) => {
+        const entry = { kind: run.kind, offset: run.offset, line: run.line, column: run.column, codePoint: text.codePointAt(run.offset), count: run.count };
         if (run.decoded !== undefined) {
             entry.decoded = run.decoded.slice(0, MAX_DECODED);
-            entry.decodedTruncated = run.decoded.length > MAX_DECODED;
+            entry.decodedTruncated = run.decodedTruncated || run.decoded.length > MAX_DECODED;
         }
-        hiddenRuns.push(entry);
-    }
-
-    const findings = kept.slice(0, maxFindings);
+        return entry;
+    });
     return {
         report: {
             ...full,
             findings,
-            total: kept.length,
-            truncated: kept.length > findings.length,
-            hiddenRuns: hiddenRuns.slice(0, maxFindings),
-            hiddenRunCount: hiddenRuns.length,
+            truncated: full.total > findings.length,
+            hiddenRuns,
+            hiddenRunCount: hidden.total,
         },
         hiddenRuns,
         loose,
+        entryCount: hidden.total + looseCount,
     };
 }
 
@@ -530,6 +546,30 @@ const CATEGORY_RULES = {
 const SARIF_SCHEMA = 'https://json.schemastore.org/sarif-2.1.0.json';
 const INFORMATION_URI = 'https://github.com/iWhatty/text-sanctifier';
 const ABSOLUTE_PATH = /^(?:[a-zA-Z]:)?[\\/]/;
+const fingerprints = new WeakMap();
+
+function fingerprintFindings(label, text, findings, cwd) {
+    const path = label === STDIN_LABEL ? 'stdin' : toSlash(relative(cwd, resolve(cwd, label)));
+    let previousLine = 0, start = 0, end = 0;
+    let occurrences = new Map();
+    const hashOccurrences = new Map();
+    for (const finding of findings) {
+        if (finding.line !== previousLine) {
+            start = finding.offset - finding.column + 1;
+            end = start;
+            while (end < text.length && text.charCodeAt(end) !== 10 && text.charCodeAt(end) !== 13) end++;
+            occurrences = new Map();
+            previousLine = finding.line;
+        }
+        const index = occurrences.get(finding.category) || 0;
+        occurrences.set(finding.category, index + 1);
+        const hash = createHash('sha256').update(JSON.stringify([finding.category, path, text.slice(start, end), index])).digest('hex');
+        const count = (hashOccurrences.get(hash) || 0) + 1;
+        hashOccurrences.set(hash, count);
+        fingerprints.set(finding, count === 1 ? hash :
+            createHash('sha256').update(JSON.stringify([hash, count])).digest('hex'));
+    }
+}
 
 /** @param {string} label */
 function sarifArtifact(label) {
@@ -559,6 +599,7 @@ function sarifLog(files, { cwd, total, filesScanned, skipped, succeeded }) {
             const run = runAt.get(x.offset);
             results.push({
                 ruleId: x.category,
+                partialFingerprints: { 'textSanctifier/v1': fingerprints.get(x) },
                 ruleIndex: ruleIds.indexOf(x.category),
                 level: CATEGORY_RULES[x.category][0],
                 message: { text: `${hex(x.codePoint)} ${x.category}${run ? `, start of a ${describeRun(run)}` : ''}` },
@@ -689,16 +730,18 @@ async function main(argv) {
                 const report = inspect(text, opts.maxFindings, opts.stripBom);
                 total += report.total;
                 if (report.total === 0) return;
+                if (opts.format === 'sarif') fingerprintFindings(label, text, report.findings, cwd);
                 files.push({ path: label, ...report });
                 for (const x of report.findings) lines.push(findingLine(label, x));
                 if (report.truncated) lines.push(`${label}: ${report.total - report.findings.length} more not listed (--max-findings)`);
                 return;
             }
 
-            const { report, hiddenRuns, loose } = inspectAiConfig(text, opts.maxFindings, opts.stripBom);
+            const { report, hiddenRuns, loose, entryCount } = inspectAiConfig(text, opts.maxFindings, opts.stripBom);
             total += report.total;
             hiddenRunCount += report.hiddenRunCount;
             if (report.total === 0) return;
+            if (opts.format === 'sarif') fingerprintFindings(label, text, report.findings, cwd);
             files.push({ path: label, ...report });
             // Text output lists each run once, in place of its characters
             const entries = [
@@ -706,7 +749,7 @@ async function main(argv) {
                 ...loose.map((x) => ({ offset: x.offset, text: findingLine(label, x) })),
             ].sort((a, b) => a.offset - b.offset);
             for (const e of entries.slice(0, opts.maxFindings)) lines.push(e.text);
-            if (entries.length > opts.maxFindings) lines.push(`${label}: ${entries.length - opts.maxFindings} more not listed (--max-findings)`);
+            if (entryCount > opts.maxFindings) lines.push(`${label}: ${entryCount - opts.maxFindings} more not listed (--max-findings)`);
         });
 
         const kind = opts.aiConfig ? 'AI configuration file' : 'file';
@@ -757,7 +800,21 @@ async function main(argv) {
         scanned++;
         // The BOM at offset 0 is set aside so purgeInvisibleChars cannot take it
         const leadingBom = text.charCodeAt(0) === BOM;
-        let cleaned = clean(leadingBom ? text.slice(1) : text);
+        const body = leadingBom ? text.slice(1) : text;
+        let cleaned;
+        if (opts.keepBenign && opts.purgeInvisible) {
+            const benign = inspectTextInContext(body, { maxFindings: Infinity }).findings.filter((f) => f.benign);
+            let start = 0;
+            const parts = [];
+            for (const f of benign) {
+                parts.push(clean(body.slice(start, f.offset)));
+                const end = f.offset + (f.codePoint > 0xFFFF ? 2 : 1);
+                parts.push(body.slice(f.offset, end));
+                start = end;
+            }
+            parts.push(clean(body.slice(start)));
+            cleaned = parts.join('');
+        } else cleaned = clean(body);
         if (leadingBom && !opts.stripBom) cleaned = String.fromCharCode(BOM) + cleaned;
 
         if (input.path === STDIN && !opts.dryRun) process.stdout.write(cleaned);

package/dist/text-sanctifier.min.js

@@ -3,4 +3,4 @@ var y=/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F-\u009F\u061C\u200E\u200F\u2
 `){return e.replace(Q,r)}var re=/[ \t]*(\n+)[ \t]*/g;function ne(e){return e.replace(re,"$1")}var te=/\n{2,}/g,ie=/\n{3,}/g;function oe(e,r){return r?e.replace(ie,`
 
 `):e.replace(te,`
-`)}var ue=/ {2,}/g;function se(e){return e.replace(ue," ")}function le(e){return e.replace(y,"")}function ae(e,r={}){if(typeof e!="string")throw new TypeError("revealInvisible expects a string input.");let n=r&&typeof r.format=="function"?r.format:i=>"<U+"+i.toString(16).toUpperCase().padStart(4,"0")+">",u="",o=0;for(let i of C(e,{maxFindings:1/0}).findings)u+=e.slice(o,i.offset)+n(i.codePoint,i.category),o=i.offset+(i.codePoint>65535?2:1);return u+e.slice(o)}var fe="1.3.9";export{fe as VERSION,b as getNewlineStats,K as getRecommendedSanctifierOptions,j as inspectText,C as inspectTextDetailed,ae as revealInvisible,T as sanctifyKeyboardOnly,N as sanctifyKeyboardOnlyEmoji,w as sanctifyLoose,L as sanctifyPasted,R as sanctifyStrict,W as summonSanctifier};
+`)}var ue=/ {2,}/g;function se(e){return e.replace(ue," ")}function le(e){return e.replace(y,"")}function ae(e,r={}){if(typeof e!="string")throw new TypeError("revealInvisible expects a string input.");let n=r&&typeof r.format=="function"?r.format:i=>"<U+"+i.toString(16).toUpperCase().padStart(4,"0")+">",u="",o=0;for(let i of C(e,{maxFindings:1/0}).findings)u+=e.slice(o,i.offset)+n(i.codePoint,i.category),o=i.offset+(i.codePoint>65535?2:1);return u+e.slice(o)}var fe="1.3.10";export{fe as VERSION,b as getNewlineStats,K as getRecommendedSanctifierOptions,j as inspectText,C as inspectTextDetailed,ae as revealInvisible,T as sanctifyKeyboardOnly,N as sanctifyKeyboardOnlyEmoji,w as sanctifyLoose,L as sanctifyPasted,R as sanctifyStrict,W as summonSanctifier};

package/dist/types/hiddenScan.d.ts

@@ -0,0 +1,30 @@
+/**
+ * Decodes text[start, end) as decodeTagText does, stopping before the output would
+ * pass `limit` UTF-16 units. A `<U+XXXX>` marker is written whole or not at all.
+ * @param {string} text
+ * @param {number} start
+ * @param {number} end
+ * @param {number} limit
+ * @returns {{ decoded: string, truncated: boolean }}
+ */
+export function decodeRange(text: string, start: number, end: number, limit: number): {
+    decoded: string;
+    truncated: boolean;
+};
+/**
+ * @param {string} text
+ * @param {HiddenScanOptions} [options]
+ * @param {(span: {offset: number, end: number, benign: boolean, kind: HiddenRunKind}) => void} [onSpan] - internal streaming visitor
+ * @returns {HiddenTextScan}
+ */
+export function scanHiddenSpans(text: string, options?: HiddenScanOptions, onSpan?: (span: {
+    offset: number;
+    end: number;
+    benign: boolean;
+    kind: HiddenRunKind;
+}) => void): HiddenTextScan;
+export type HiddenRunKind = import("./hiddenText.js").HiddenRunKind;
+export type HiddenRun = import("./hiddenText.js").HiddenRun;
+export type HiddenRunCounts = import("./hiddenText.js").HiddenRunCounts;
+export type HiddenTextScan = import("./hiddenText.js").HiddenTextScan;
+export type HiddenScanOptions = import("./hiddenText.js").HiddenScanOptions;

package/dist/types/hiddenText.d.ts

@@ -1,3 +1,43 @@
+/** @typedef {'tag' | 'variation_selector' | 'invisible_operator' | 'reserved_tag_plane'} HiddenRunKind */
+/**
+ * @typedef {Object} HiddenRun
+ * @property {HiddenRunKind} kind
+ * @property {number} offset - UTF-16 index of the first character
+ * @property {number} end - UTF-16 index just past the last character
+ * @property {number} count - number of code points in the run
+ * @property {number} line - 1-based line of the first character (as in inspectTextDetailed)
+ * @property {number} column - 1-based column of the first character, in UTF-16 units
+ * @property {string} [decoded] - tag runs only: the ASCII the tags spell (see decodeTagText),
+ *   at most `maxDecodedLength` UTF-16 units
+ * @property {boolean} [decodedTruncated] - tag runs only: true when `decoded` stops short of the run
+ */
+/**
+ * @typedef {Object} HiddenRunCounts
+ * @property {number} tag
+ * @property {number} variation_selector
+ * @property {number} invisible_operator
+ * @property {number} reserved_tag_plane
+ */
+/**
+ * @typedef {Object} HiddenTextScan
+ * @property {HiddenRun[]} runs - runs that may hide text, in order, at most `maxRuns` long
+ * @property {Array<[number, number]>} benign - [offset, end) spans of legitimate
+ *   sequences (subdivision flags, a single emoji or text presentation selector), at most `maxRuns` long
+ * @property {HiddenRunCounts} counts - every run per kind, listed or not
+ * @property {HiddenRunCounts} codePointCounts - code points in those runs per kind, listed or not
+ * @property {number} supplementarySelectorCount - code points U+E0100–U+E01EF in those runs, which
+ *   (unlike the emoji selectors U+FE00–U+FE0F) can carry data; included in codePointCounts.variation_selector
+ * @property {number} total - sum of `counts`
+ * @property {number} benignTotal - every benign span, listed or not
+ * @property {boolean} truncated - true when runs or benign spans were left out of the lists
+ */
+/**
+ * @typedef {Object} HiddenScanOptions
+ * @property {number} [maxRuns=Infinity] - cap on listed runs, and separately on listed benign spans
+ *   (non-negative integer or Infinity). Counts stay exact.
+ * @property {number} [maxDecodedLength=Infinity] - cap on each run's `decoded` text, in UTF-16 units.
+ *   For both, any other value (negative, fractional, NaN, a string) means the default, no cap.
+ */
 /**
  * Decodes tag characters to the ASCII they shadow: U+E0020–U+E007E become
  * U+0020–U+007E. Anything else (U+E0001 LANGUAGE TAG, U+E007F CANCEL TAG, unassigned

package/dist/types/version.generated.d.ts

@@ -1,2 +1,2 @@
 /** The installed version of text-sanctifier, as in its package.json. */
-export const VERSION: "1.3.9";
+export const VERSION: "1.3.10";

package/package.json

@@ -1,8 +1,8 @@
 {
   "name": "text-sanctifier",
-  "version": "1.3.9",
+  "version": "1.3.10",
   "type": "module",
-  "description": "A brutal text normalizer and invisible trash scrubber for modern web projects.",
+  "description": "Find and remove invisible Unicode in web input, model-bound text and repositories.",
   "engines": {
     "node": ">=18"
   },

package/src/hiddenScan.js

@@ -0,0 +1,171 @@
+// Internal hidden-span scanner; not a package entry point.
+
+/**
+ * @typedef {import('./hiddenText.js').HiddenRunKind} HiddenRunKind
+ * @typedef {import('./hiddenText.js').HiddenRun} HiddenRun
+ * @typedef {import('./hiddenText.js').HiddenRunCounts} HiddenRunCounts
+ * @typedef {import('./hiddenText.js').HiddenTextScan} HiddenTextScan
+ * @typedef {import('./hiddenText.js').HiddenScanOptions} HiddenScanOptions
+ */
+
+const TAG_FIRST = 0xE0000;
+const TAG_LAST = 0xE007F;
+const CANCEL_TAG = 0xE007F;
+const BLACK_FLAG = 0x1F3F4;
+
+// RGI subdivision flags: U+1F3F4, the tag letters, U+E007F
+const SUBDIVISION_FLAGS = new Set(['gbeng', 'gbsct', 'gbwls']);
+
+/**
+ * @param {number} cp
+ * @returns {HiddenRunKind | null}
+ */
+function kindOf(cp) {
+    if (cp >= TAG_FIRST && cp <= TAG_LAST) return 'tag';
+    if ((cp >= 0xFE00 && cp <= 0xFE0F) || (cp >= 0xE0100 && cp <= 0xE01EF) ||
+        (cp >= 0x180B && cp <= 0x180D) || cp === 0x180F) return 'variation_selector';
+    if (cp >= 0x2061 && cp <= 0x2064) return 'invisible_operator';
+    // Unassigned and default ignorable: U+E0080-U+E00FF and U+E01F0-U+E0FFF
+    if (cp >= 0xE0080 && cp <= 0xE0FFF) return 'reserved_tag_plane';
+    return null;
+}
+
+/** @param {number} cp */
+const hex = (cp) => 'U+' + cp.toString(16).toUpperCase().padStart(4, '0');
+
+/**
+ * Decodes text[start, end) as decodeTagText does, stopping before the output would
+ * pass `limit` UTF-16 units. A `<U+XXXX>` marker is written whole or not at all.
+ * @param {string} text
+ * @param {number} start
+ * @param {number} end
+ * @param {number} limit
+ * @returns {{ decoded: string, truncated: boolean }}
+ */
+export function decodeRange(text, start, end, limit) {
+    let out = '';
+    let i = start;
+    while (i < end) {
+        const cp = /** @type {number} */ (text.codePointAt(i));
+        const piece = cp >= 0xE0020 && cp <= 0xE007E ? String.fromCharCode(cp - TAG_FIRST) : `<${hex(cp)}>`;
+        if (out.length + piece.length > limit) return { decoded: out, truncated: true };
+        out += piece;
+        i += cp > 0xFFFF ? 2 : 1;
+    }
+    return { decoded: out, truncated: false };
+}
+
+// Tested at an index of the text itself, so the check allocates nothing.
+const PICTOGRAPHIC_AT = /\p{Extended_Pictographic}/yu;
+
+/**
+ * True when a presentation selector at `index` follows an emoji-capable character:
+ * an Extended_Pictographic code point, or a keycap base (#, * or a digit) with U+20E3 after.
+ * @param {string} text
+ * @param {number} index
+ * @returns {boolean}
+ */
+function isPresentationBase(text, index) {
+    if (index === 0) return false;
+    const low = text.charCodeAt(index - 1);
+    const p = low >= 0xDC00 && low <= 0xDFFF && index >= 2 &&
+        (text.charCodeAt(index - 2) & 0xFC00) === 0xD800 ? index - 2 : index - 1;
+    const before = text.charCodeAt(p);
+    if ((before === 0x23 || before === 0x2A || (before >= 0x30 && before <= 0x39)) && text.charCodeAt(index + 1) === 0x20E3) return true;
+    PICTOGRAPHIC_AT.lastIndex = p;
+    return PICTOGRAPHIC_AT.test(text);
+}
+
+/**
+ * A cap option: a non-negative integer or Infinity; anything else means no cap.
+ * @param {unknown} value
+ * @returns {number}
+ */
+function cap(value) {
+    return typeof value === 'number' && (value === Infinity || (Number.isInteger(value) && value >= 0))
+        ? value
+        : Infinity;
+}
+
+/**
+ * @param {string} text
+ * @param {HiddenScanOptions} [options]
+ * @param {(span: {offset: number, end: number, benign: boolean, kind: HiddenRunKind}) => void} [onSpan] - internal streaming visitor
+ * @returns {HiddenTextScan}
+ */
+export function scanHiddenSpans(text, options = {}, onSpan) {
+    if (typeof text !== 'string') throw new TypeError('findHiddenRuns expects a string input.');
+    const maxRuns = cap(options && options.maxRuns);
+    const maxDecodedLength = cap(options && options.maxDecodedLength);
+
+    /** @type {HiddenRun[]} */
+    const runs = [];
+    /** @type {Array<[number, number]>} */
+    const benign = [];
+    /** @type {HiddenRunCounts} */
+    const counts = { tag: 0, variation_selector: 0, invisible_operator: 0, reserved_tag_plane: 0 };
+    /** @type {HiddenRunCounts} */
+    const codePointCounts = { tag: 0, variation_selector: 0, invisible_operator: 0, reserved_tag_plane: 0 };
+    let supplementarySelectorCount = 0;
+    let total = 0;
+    let benignTotal = 0;
+    let line = 1;
+    let lineStart = 0;
+
+    let i = 0;
+    while (i < text.length) {
+        const cp = /** @type {number} */ (text.codePointAt(i));
+        const kind = kindOf(cp);
+        if (kind === null) {
+            if (cp === 0x0A || (cp === 0x0D && text.charCodeAt(i + 1) !== 0x0A)) { // CRLF ends at its LF
+                line++;
+                lineStart = i + 1;
+            }
+            i += cp > 0xFFFF ? 2 : 1;
+            continue;
+        }
+
+        const offset = i;
+        let count = 0;
+        let supplementary = 0;
+        let last = cp;
+        while (i < text.length) {
+            const c = /** @type {number} */ (text.codePointAt(i));
+            if (kindOf(c) !== kind) break;
+            last = c;
+            count++;
+            if (c >= 0xE0100 && c <= 0xE01EF) supplementary++;
+            i += c > 0xFFFF ? 2 : 1;
+        }
+        const end = i;
+
+        // A subdivision flag is exactly five tag letters and CANCEL TAG after U+1F3F4.
+        const isBenign = kind === 'tag'
+            ? count === 6 && last === CANCEL_TAG && offset >= 2 && text.codePointAt(offset - 2) === BLACK_FLAG &&
+              SUBDIVISION_FLAGS.has(decodeRange(text, offset, end - 2, Infinity).decoded)
+            : kind === 'variation_selector' && count === 1 && (cp === 0xFE0E || cp === 0xFE0F) &&
+              isPresentationBase(text, offset);
+        if (onSpan) onSpan({ offset, end, benign: isBenign, kind });
+        if (isBenign) {
+            benignTotal++;
+            if (benign.length < maxRuns) benign.push([offset, end]);
+            continue;
+        }
+
+        counts[kind]++;
+        codePointCounts[kind] += count;
+        supplementarySelectorCount += supplementary;
+        total++;
+        if (runs.length < maxRuns) {
+            /** @type {HiddenRun} */
+            const run = { kind, offset, end, line, column: offset - lineStart + 1, count };
+            if (kind === 'tag') {
+                const { decoded, truncated } = decodeRange(text, offset, end, maxDecodedLength);
+                run.decoded = decoded;
+                run.decodedTruncated = truncated;
+            }
+            runs.push(run);
+        }
+    }
+    return { runs, benign, counts, codePointCounts, supplementarySelectorCount, total, benignTotal, truncated: total > runs.length || benignTotal > benign.length };
+}

package/src/hiddenText.js

@@ -9,6 +9,8 @@
 // Plain JavaScript with no Node APIs, so it also runs in a browser. src/index.js does
 // not import this file, so it stays out of the browser bundle and its size budget.
 
+import { scanHiddenSpans, decodeRange } from './hiddenScan.js';
+
 export { VERSION } from './version.generated.js';
 
 /** @typedef {'tag' | 'variation_selector' | 'invisible_operator' | 'reserved_tag_plane'} HiddenRunKind */
@@ -56,31 +58,6 @@ export { VERSION } from './version.generated.js';
  *   For both, any other value (negative, fractional, NaN, a string) means the default, no cap.
  */
 
-const TAG_FIRST = 0xE0000;
-const TAG_LAST = 0xE007F;
-const CANCEL_TAG = 0xE007F;
-const BLACK_FLAG = 0x1F3F4;
-
-// RGI subdivision flags: U+1F3F4, the tag letters, U+E007F
-const SUBDIVISION_FLAGS = new Set(['gbeng', 'gbsct', 'gbwls']);
-
-/**
- * @param {number} cp
- * @returns {HiddenRunKind | null}
- */
-function kindOf(cp) {
-    if (cp >= TAG_FIRST && cp <= TAG_LAST) return 'tag';
-    if ((cp >= 0xFE00 && cp <= 0xFE0F) || (cp >= 0xE0100 && cp <= 0xE01EF) ||
-        (cp >= 0x180B && cp <= 0x180D) || cp === 0x180F) return 'variation_selector';
-    if (cp >= 0x2061 && cp <= 0x2064) return 'invisible_operator';
-    // Unassigned and default ignorable: U+E0080-U+E00FF and U+E01F0-U+E0FFF
-    if (cp >= 0xE0080 && cp <= 0xE0FFF) return 'reserved_tag_plane';
-    return null;
-}
-
-/** @param {number} cp */
-const hex = (cp) => 'U+' + cp.toString(16).toUpperCase().padStart(4, '0');
-
 /**
  * Decodes tag characters to the ASCII they shadow: U+E0020–U+E007E become
  * U+0020–U+007E. Anything else (U+E0001 LANGUAGE TAG, U+E007F CANCEL TAG, unassigned
@@ -89,63 +66,10 @@ const hex = (cp) => 'U+' + cp.toString(16).toUpperCase().padStart(4, '0');
  * @returns {string}
  */
 export function decodeTagText(text) {
+    if (typeof text !== 'string') throw new TypeError('decodeTagText expects a string input.');
     return decodeRange(text, 0, text.length, Infinity).decoded;
 }
 
-/**
- * Decodes text[start, end) as decodeTagText does, stopping before the output would
- * pass `limit` UTF-16 units. A `<U+XXXX>` marker is written whole or not at all.
- * @param {string} text
- * @param {number} start
- * @param {number} end
- * @param {number} limit
- * @returns {{ decoded: string, truncated: boolean }}
- */
-function decodeRange(text, start, end, limit) {
-    let out = '';
-    let i = start;
-    while (i < end) {
-        const cp = /** @type {number} */ (text.codePointAt(i));
-        const piece = cp >= 0xE0020 && cp <= 0xE007E ? String.fromCharCode(cp - TAG_FIRST) : `<${hex(cp)}>`;
-        if (out.length + piece.length > limit) return { decoded: out, truncated: true };
-        out += piece;
-        i += cp > 0xFFFF ? 2 : 1;
-    }
-    return { decoded: out, truncated: false };
-}
-
-// Tested at an index of the text itself, so the check allocates nothing.
-const PICTOGRAPHIC_AT = /\p{Extended_Pictographic}/yu;
-
-/**
- * True when a presentation selector at `index` follows an emoji-capable character:
- * an Extended_Pictographic code point, or a keycap base (#, * or a digit) with U+20E3 after.
- * @param {string} text
- * @param {number} index
- * @returns {boolean}
- */
-function isPresentationBase(text, index) {
-    if (index === 0) return false;
-    const low = text.charCodeAt(index - 1);
-    const p = low >= 0xDC00 && low <= 0xDFFF && index >= 2 &&
-        (text.charCodeAt(index - 2) & 0xFC00) === 0xD800 ? index - 2 : index - 1;
-    const before = text.charCodeAt(p);
-    if ((before === 0x23 || before === 0x2A || (before >= 0x30 && before <= 0x39)) && text.charCodeAt(index + 1) === 0x20E3) return true;
-    PICTOGRAPHIC_AT.lastIndex = p;
-    return PICTOGRAPHIC_AT.test(text);
-}
-
-/**
- * A cap option: a non-negative integer or Infinity; anything else means no cap.
- * @param {unknown} value
- * @returns {number}
- */
-function cap(value) {
-    return typeof value === 'number' && (value === Infinity || (Number.isInteger(value) && value >= 0))
-        ? value
-        : Infinity;
-}
-
 /**
  * Finds runs of tag characters, variation selectors, invisible operators and reserved
  * tag-plane code points (U+E0080–U+E00FF, U+E01F0–U+E0FFF).
@@ -168,76 +92,5 @@ function cap(value) {
  * @returns {HiddenTextScan}
  */
 export function findHiddenRuns(text, options = {}) {
-    const maxRuns = cap(options && options.maxRuns);
-    const maxDecodedLength = cap(options && options.maxDecodedLength);
-
-    /** @type {HiddenRun[]} */
-    const runs = [];
-    /** @type {Array<[number, number]>} */
-    const benign = [];
-    /** @type {HiddenRunCounts} */
-    const counts = { tag: 0, variation_selector: 0, invisible_operator: 0, reserved_tag_plane: 0 };
-    /** @type {HiddenRunCounts} */
-    const codePointCounts = { tag: 0, variation_selector: 0, invisible_operator: 0, reserved_tag_plane: 0 };
-    let supplementarySelectorCount = 0;
-    let total = 0;
-    let benignTotal = 0;
-    let line = 1;
-    let lineStart = 0;
-
-    let i = 0;
-    while (i < text.length) {
-        const cp = /** @type {number} */ (text.codePointAt(i));
-        const kind = kindOf(cp);
-        if (kind === null) {
-            if (cp === 0x0A || (cp === 0x0D && text.charCodeAt(i + 1) !== 0x0A)) { // CRLF ends at its LF
-                line++;
-                lineStart = i + 1;
-            }
-            i += cp > 0xFFFF ? 2 : 1;
-            continue;
-        }
-
-        const offset = i;
-        let count = 0;
-        let supplementary = 0;
-        let last = cp;
-        while (i < text.length) {
-            const c = /** @type {number} */ (text.codePointAt(i));
-            if (kindOf(c) !== kind) break;
-            last = c;
-            count++;
-            if (c >= 0xE0100 && c <= 0xE01EF) supplementary++;
-            i += c > 0xFFFF ? 2 : 1;
-        }
-        const end = i;
-
-        // A subdivision flag is exactly five tag letters and CANCEL TAG after U+1F3F4.
-        const isBenign = kind === 'tag'
-            ? count === 6 && last === CANCEL_TAG && offset >= 2 && text.codePointAt(offset - 2) === BLACK_FLAG &&
-              SUBDIVISION_FLAGS.has(decodeRange(text, offset, end - 2, Infinity).decoded)
-            : kind === 'variation_selector' && count === 1 && (cp === 0xFE0E || cp === 0xFE0F) &&
-              isPresentationBase(text, offset);
-        if (isBenign) {
-            benignTotal++;
-            if (benign.length < maxRuns) benign.push([offset, end]);
-            continue;
-        }
-
-        counts[kind]++;
-        codePointCounts[kind] += count;
-        supplementarySelectorCount += supplementary;
-        total++;
-        if (runs.length < maxRuns) {
-            /** @type {HiddenRun} */
-            const run = { kind, offset, end, line, column: offset - lineStart + 1, count };
-            if (kind === 'tag') {
-                const { decoded, truncated } = decodeRange(text, offset, end, maxDecodedLength);
-                run.decoded = decoded;
-                run.decodedTruncated = truncated;
-            }
-            runs.push(run);
-        }
-    }
-    return { runs, benign, counts, codePointCounts, supplementarySelectorCount, total, benignTotal, truncated: total > runs.length || benignTotal > benign.length };
+    return scanHiddenSpans(text, options);
 }

package/src/version.generated.js

@@ -2,4 +2,4 @@
 // Generated from package.json by scripts/gen-version.js (`npm run gen`); do not edit.
 
 /** The installed version of text-sanctifier, as in its package.json. */
-export const VERSION = '1.3.9';
+export const VERSION = '1.3.10';