text-sanctifier 1.3.7 → 1.3.9

package/README.md

@@ -2,12 +2,12 @@
 
 [![npm](https://badgen.net/npm/v/text-sanctifier)](https://www.npmjs.com/package/text-sanctifier)
 [![downloads](https://badgen.net/npm/dm/text-sanctifier)](https://www.npmjs.com/package/text-sanctifier)
-[![bundle size](https://badgen.net/static/gzip/under%203%20kB/green)](https://bundlejs.com/?q=text-sanctifier)
+[![bundle size](https://badgen.net/static/gzip/under%203%20KiB/green)](https://bundlejs.com/?q=text-sanctifier)
 [![license](https://badgen.net/static/license/AGPL-3.0%20%2B%20terms/blue)](https://github.com/iWhatty/text-sanctifier/blob/main/LICENSE)
 [![stars](https://badgen.net/github/stars/iWhatty/text-sanctifier)](https://github.com/iWhatty/text-sanctifier)
 [![types](https://badgen.net/npm/types/text-sanctifier)](https://www.npmjs.com/package/text-sanctifier)
 
-Brutal text normalizer and invisible Unicode scrubber for modern web projects. ~2.87 KB gzipped.
+Brutal text normalizer and invisible Unicode scrubber for modern web projects. The prebuilt browser file stays under <!-- gen:budget:browser -->3 KiB<!-- /gen --> gzipped, a budget `npm run ci` enforces; with a bundler you pay only for what you import.
 
 ## Features
 
@@ -24,8 +24,8 @@ Brutal text normalizer and invisible Unicode scrubber for modern web projects. ~
 - Nukes control characters (if enabled)
 - Smart normalization of typographic junk (quotes, dashes, bullets, full-width punctuation)
 - Keyboard-only filtering (retain printable ASCII + full emoji sequences)
-  - Preserves ZWJ emoji clusters (👨‍👩‍👧‍👦)
-  - Preserves VS16 emoji presentation variants (✌️, ‼️)
+  - Preserves ZWJ emoji clusters (the family emoji, U+1F468 U+200D U+1F469 U+200D U+1F467 U+200D U+1F466)
+  - Preserves VS16 emoji presentation variants (U+270C U+FE0F, U+203C U+FE0F)
 - Configurable via fine-grained flags or ready-made presets
 - Includes strict, loose, and keyboard-only modes
 - Deterministic RegExp usage (no global `lastIndex` state leaks)
@@ -121,7 +121,7 @@ Two options handle spaces and zero-width characters gently, and both run before
 
 ### Presets as standalone imports
 
-Each preset is also its own export: `sanctifyStrict`, `sanctifyLoose`, `sanctifyKeyboardOnly`, `sanctifyKeyboardOnlyEmoji` and `sanctifyPasted`. They are the same functions as `summonSanctifier.strict` and the rest. Import one directly and a bundler leaves out `summonSanctifier` and the other presets (about 1.1 KB gzipped instead of 1.45 KB).
+Each preset is also its own export: `sanctifyStrict`, `sanctifyLoose`, `sanctifyKeyboardOnly`, `sanctifyKeyboardOnlyEmoji` and `sanctifyPasted`. They are the same functions as `summonSanctifier.strict` and the rest. Import one directly and a bundler leaves out `summonSanctifier` and the other presets (about <!-- gen:size:sanctifyPasted -->1.1 KiB<!-- /gen --> gzipped instead of <!-- gen:size:summonSanctifier.pasted -->1.4 KiB<!-- /gen -->).
 
 ```js
 import { sanctifyPasted } from 'text-sanctifier';
@@ -230,7 +230,7 @@ truncated: false
 
 `codePoint` is the full code point, so a tag character reports `0xE0041`, not a surrogate. `offset` is a 0-based UTF-16 index (for a code point above U+FFFF, that of its high surrogate); `line` and `column` are 1-based, with columns in UTF-16 code units and lines ending at LF, CR or CRLF. Every occurrence is counted; only the first `maxFindings` (default 100, `Infinity` for all) are listed, and `truncated` says when some were not. The scan is one pass with no regex and no copy of the input.
 
-**Choosing `maxFindings`:** the scan itself uses almost no memory, but every listed finding is an object, so the cost of `maxFindings` grows with the number of findings, not the size of the input. 16 MB of non-breaking spaces is 16.7 million findings. With a cap of 20, it scans in about 80 ms and uses 0.2 MB. With `Infinity`, expect roughly 100–200 bytes per finding: about 1.5 GB for 8 million. On untrusted input, keep a cap. Use `maxFindings: 0` for a counts-only report: it builds no finding objects, and `counts`, `total` and `truncated` stay exact.
+**Choosing `maxFindings`:** the scan itself uses almost no memory, but every listed finding is an object, so the cost of `maxFindings` grows with the number of findings, not the size of the input. 16 MB of non-breaking spaces is 16.7 million findings. With a cap of 20, it scans in about 80 ms and uses 0.2 MB (`npm run bench` measures it on your machine). With `Infinity`, expect roughly 100–200 bytes per finding: about 1.5 GB for 8 million. On untrusted input, keep a cap. Use `maxFindings: 0` for a counts-only report: it builds no finding objects, and `counts`, `total` and `truncated` stay exact.
 
 ### `getNewlineStats(text: string): NewlineStats`
 
@@ -242,7 +242,7 @@ import { getNewlineStats } from 'text-sanctifier';
 getNewlineStats('a\r\nb\n'); // { crlf: 1, cr: 0, lf: 1, types: ['CRLF', 'LF'], mixed: true }
 ```
 
-`zero_width` also flags U+200D (zero width joiner) inside emoji ZWJ sequences such as 👨‍👩‍👧, so a finding there is not necessarily hidden text. Filter by neighbouring code points if you need to tell them apart. (`purgeInvisibleChars` removes those joiners too, splitting the emoji; `keyboardOnlyEmoji` keeps the sequences whole.) The same holds for `variation_selector`, which flags VS16 (U+FE0F) in emoji such as ✌️ and the ideographic variation selectors in CJK names, and for `tag`, which flags the tag characters inside subdivision flags such as the England flag (U+1F3F4 followed by tags). `purgeInvisibleChars` strips them all; `keyboardOnlyEmoji` keeps VS16. The Mongolian free variation selectors (U+180B–U+180D, U+180F) are also `variation_selector`: they are legitimate in Mongolian text, which `purgeInvisibleChars` can change, so don't purge Mongolian text with it.
+`zero_width` also flags U+200D (zero width joiner) inside emoji ZWJ sequences such as U+1F468 U+200D U+1F469 U+200D U+1F467, so a finding there is not necessarily hidden text. Filter by neighbouring code points if you need to tell them apart. (`purgeInvisibleChars` removes those joiners too, splitting the emoji; `keyboardOnlyEmoji` keeps the sequences whole.) The same holds for `variation_selector`, which flags VS16 (U+FE0F) in emoji such as U+270C U+FE0F and the ideographic variation selectors in CJK names, and for `tag`, which flags the tag characters inside subdivision flags such as the England flag (U+1F3F4 followed by tags). `purgeInvisibleChars` strips them all; `keyboardOnlyEmoji` keeps VS16. The Mongolian free variation selectors (U+180B–U+180D, U+180F) are also `variation_selector`: they are legitimate in Mongolian text, which `purgeInvisibleChars` can change, so don't purge Mongolian text with it.
 
 The categories cover exactly what `nukeControls` and `purgeInvisibleChars` strip. Every code point with the Unicode property Default_Ignorable_Code_Point (characters renderers must draw as nothing) is in a category, and a test checks this against the running Node's Unicode data.
 
@@ -255,7 +255,7 @@ import { findHiddenRuns, decodeTagText } from 'text-sanctifier/hidden';
 
 const { runs, benign, counts, total, truncated } = findHiddenRuns(text, { maxRuns: 100, maxDecodedLength: 1000 });
 // runs:   [{ kind: 'tag', offset, end, line, column, count, decoded: 'Ignore the user', decodedTruncated: false }, ...]
-// benign: [[offset, end], ...]  England/Scotland/Wales flags, a lone U+FE0E/U+FE0F after a character
+// benign: [[offset, end], ...]  England/Scotland/Wales flags, a lone U+FE0E/U+FE0F after an emoji or in a keycap
 // counts: { tag, variation_selector, invisible_operator, reserved_tag_plane }, every run listed or not; total is their sum
 ```
 
@@ -285,8 +285,12 @@ Benign, and nothing else:
 - one of U+E0100–U+E01EF directly after an ideograph
 - U+180B–U+180D or U+180F directly after a Mongolian letter
 - the tags of the England, Scotland and Wales flags
+- U+200C or U+200D directly after a virama (Devanagari, Bengali, Tamil, Malayalam, Sinhala, Myanmar and the other scripts with one), as IDNA ContextJ (RFC 5892) allows
+- U+200C between two Arabic-script letters, skipping marks, as in Persian. This approximates the RFC's joining-type rule: it also accepts a ZWNJ after a letter that does not join to the left, and none in other joining scripts such as Syriac
 
-U+200B, U+2060, U+FEFF, bidi controls and all other tags are never benign. Joiners in scripts that need them (ZWNJ in Persian, ZWJ in Indic scripts) are not benign yet. It needs Unicode property escapes (Node 10+, current browsers), is not in the browser file, and is about 1.7 KB gzipped on its own.
+U+200B, U+2060, U+FEFF, bidi controls and all other tags are never benign, and neither is U+200D between Arabic letters. It needs Unicode property escapes (Node 10+, current browsers), is not in the browser file, and is about <!-- gen:size:text-sanctifier/context -->2.0 KiB<!-- /gen --> gzipped on its own.
+
+`benignSupplementarySelectorCount` counts the benign ideographic selectors U+E0100–U+E01EF. With `findHiddenRuns` from `text-sanctifier/hidden`, `scan.supplementarySelectorCount - report.benignSupplementarySelectorCount` is the exact number of suspicious ones.
 
 ---
 
@@ -310,7 +314,7 @@ src/auth.js:40:3  U+202E  bidi_control
 2 findings in 2 of 31 files.
 ```
 
-- `--json` (or `--format json`): one JSON object on stdout, `{ files, total, filesScanned, filesWithFindings, skipped }`. Each entry in `files` is `{ path, ...inspectTextDetailed(text) }`; only files with findings are listed.
+- `--json` (or `--format json`): one JSON object on stdout, `{ files, total, filesScanned, filesWithFindings, skipped, notUtf8 }`. Each entry in `files` is `{ path, ...inspectTextDetailed(text) }`; only files with findings are listed.
 - `--format sarif`: a [SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html) log for code-scanning viewers and editors. See [SARIF output](#sarif-output).
 - `--max-findings N`: findings listed per file (default 100; `all` for no cap). Counts always cover every occurrence.
 - `--strip-bom`: also report a BOM at offset 0, which is skipped by default.
@@ -331,11 +335,11 @@ skills/review/SKILL.md:2:3  U+FE00  run of 3 variation selectors
 19 findings (2 hidden runs) in 2 of 6 AI configuration files.
 ```
 
-- With no paths it searches the current directory; directories you name are searched the same way, and files you name are checked whatever they are called. In a git work tree, files `.gitignore` excludes are skipped.
+- With no paths it searches the current directory, even when input is piped (so a CI job without stdin still checks the repository); pass `-` to check stdin; directories you name are searched the same way, and files you name are checked whatever they are called. In a git work tree, files `.gitignore` excludes are skipped.
 - Files it looks for, matched case-insensitively: `CLAUDE.md`, `CLAUDE.local.md`, `AGENTS.md`, `AGENT.md`, `GEMINI.md` and `SKILL.md` at any depth; and at the top of the directory, `.claude/**`, `.cursorrules`, `.cursor/rules/**`, `.cursor/mcp.json`, `.windsurfrules`, `.windsurf/rules/**`, `.clinerules` (file or directory), `.roorules`, `.roo/rules/**`, `.github/copilot-instructions.md`, `.github/{instructions,prompts,chatmodes,agents}/**`, `.mcp.json`, `mcp.json` and `.vscode/mcp.json`.
 - Consecutive tag characters, variation selectors or invisible operators are reported once, as a run, at the first character. Tag runs are decoded: U+E0020–U+E007E become the ASCII they shadow, and other tags are shown as `<U+XXXX>`.
 - Every other invisible or control character is reported as in a plain `check`.
-- Two legitimate sequences are left out: the flags of England, Scotland and Wales (U+1F3F4, tag letters, U+E007F), and a single U+FE0E or U+FE0F after another character (text or emoji presentation). A plain `check` still reports both.
+- Two legitimate sequences are left out: the flags of England, Scotland and Wales (U+1F3F4, tag letters, U+E007F), and a single U+FE0E or U+FE0F (text or emoji presentation) after an emoji (`Extended_Pictographic`) or in a keycap such as 1, U+FE0F, U+20E3. After a letter it is reported: one selector per letter can carry a bit of hidden data. A plain `check` still reports both.
 - `--json` adds `hiddenRuns` (`{ kind, offset, line, column, codePoint, count, decoded?, decodedTruncated? }`, at most `--max-findings` per file) and `hiddenRunCount` to each file, and `hiddenRunCount` to the summary. `kind` is `tag`, `variation_selector`, `invisible_operator` or `reserved_tag_plane`; `decoded` is cut at 1000 characters. `total` and `counts` leave out the legitimate sequences.
 
 ### SARIF output
@@ -356,7 +360,7 @@ The repository publishes a [pre-commit](https://pre-commit.com) hook manifest, s
 # .pre-commit-config.yaml
 repos:
   - repo: https://github.com/iWhatty/text-sanctifier
-    rev: v1.3.7  # a release tag
+    rev: v1.3.9  # a release tag
     hooks:
       - id: text-sanctifier            # every staged text file
       - id: text-sanctifier-ai-config  # agent instruction files, with --ai-config
@@ -394,8 +398,9 @@ git diff | npx text-sanctifier reveal
 
 - Paths can be files, directories or globs (`*`, `**`, `?`, `[...]`, `{a,b}`). Quote globs so the shell passes them through.
 - Inside a git work tree, directories and globs list files with `git ls-files`, so anything `.gitignore` excludes is skipped. Outside git, `.git` and `node_modules` are skipped. A file named explicitly is always read.
-- Binary files (a NUL byte in the first 8000 bytes) and files that are not valid UTF-8 are skipped, and `fix` never writes them. In JSON they appear under `skipped`.
-- `-`, or no paths with piped input, reads stdin. `fix` writes the cleaned text to stdout and the report to stderr.
+- Binary files (a NUL byte in the first 8000 bytes) are skipped and listed under `skipped` in JSON. Stdin and AI configuration files are never treated as binary: a NUL in them is reported as `nul`.
+- Files that are not valid UTF-8 are still checked, with each invalid byte read as U+FFFD, so one bad byte cannot hide the rest of the file. `check` notes them on stderr and lists them under `notUtf8` in JSON; `fix` never writes them and lists them under `skipped`.
+- `-`, or no paths with piped input, reads stdin (with `--ai-config`, only `-` does). `fix` writes the cleaned text to stdout and the report to stderr.
 
 ### Exit codes
 
@@ -422,7 +427,7 @@ Requires a modern JavaScript runtime with ES2020+ support:
 - **ESM only:** no CommonJS build; `require('text-sanctifier')` works only on Node versions with `require(esm)` support (20.19+, 22.12+)
 - **Types:** TypeScript declarations ship in `dist/types`
 - **Browser build (`dist/`):** one minified ESM file with every export, for pages with no build step (`<script type="module">` from a CDN). It is imported as `text-sanctifier/browser`. With a bundler, import `text-sanctifier` instead: the source tree-shakes, and the prebuilt file does not
-- **Tree-shake friendly:** ships `sideEffects: false`; `npm run ci` checks the bundled size of each export on its own (for example `inspectTextDetailed` is under 1 KB gzipped)
+- **Tree-shake friendly:** ships `sideEffects: false`; `npm run ci` checks the bundled size of each export on its own (for example `inspectTextDetailed` is about <!-- gen:size:inspectTextDetailed -->0.9 KiB<!-- /gen --> gzipped)
 - **Zero transpilation:** no built-in polyfills or runtime overhead
 - **Bundler ready:** works with Vite, Rollup, Webpack, Parcel, esbuild
 

package/bin/text-sanctifier.js

@@ -378,18 +378,22 @@ function collectInputs(args, cwd, warn, aiConfig = false) {
 // --- Reading ---
 
 const utf8 = new TextDecoder('utf-8', { fatal: true, ignoreBOM: true });
+const utf8Lenient = new TextDecoder('utf-8', { ignoreBOM: true });
 
 /**
- * Decodes UTF-8, keeping a leading BOM. Returns null for binary or non-UTF-8 data.
+ * Decodes UTF-8, keeping a leading BOM. With `sniff`, data with a NUL in its first
+ * BINARY_SNIFF_BYTES is binary and not decoded. Bytes that are not UTF-8 become U+FFFD
+ * and set `lossy`, so one bad byte cannot hide the rest of a file from `check`.
  * @param {Uint8Array} bytes
- * @returns {{ text: string|null, reason?: string }}
+ * @param {boolean} sniff
+ * @returns {{ text: string|null, reason?: string, lossy?: boolean }}
  */
-function decode(bytes) {
-    if (bytes.subarray(0, BINARY_SNIFF_BYTES).includes(0)) return { text: null, reason: 'binary' };
+function decode(bytes, sniff) {
+    if (sniff && bytes.subarray(0, BINARY_SNIFF_BYTES).includes(0)) return { text: null, reason: 'binary' };
     try {
         return { text: utf8.decode(bytes) };
     } catch {
-        return { text: null, reason: 'not UTF-8' };
+        return { text: utf8Lenient.decode(bytes), lossy: true };
     }
 }
 
@@ -612,6 +616,8 @@ async function main(argv) {
 
     const cwd = process.cwd();
     let paths = opts.paths;
+    // --ai-config with no paths searches the current directory even when stdin is not a
+    // terminal, so a CI job with no stdin still scans the repository; `-` reads stdin.
     if (paths.length === 0 && opts.aiConfig) {
         paths = ['.'];
     } else if (paths.length === 0) {
@@ -627,6 +633,8 @@ async function main(argv) {
     let errors = pathErrors;
     /** @type {Array<{ path: string, reason: string }>} */
     const skipped = [];
+    /** @type {string[]} */
+    const notUtf8 = [];
 
     /**
      * Reads every input in order, calling `visit(label, text, input)` for each decodable one.
@@ -643,12 +651,24 @@ async function main(argv) {
                 errors++;
                 continue;
             }
-            const { text, reason } = decode(bytes);
+            // Files that look binary are skipped. Stdin and AI configuration files are text
+            // by intent, so a NUL in one is a finding, not a reason to skip it.
+            const { text, reason, lossy } = decode(bytes, input.path !== STDIN && !opts.aiConfig);
             if (text === null) {
                 skipped.push({ path: label, reason });
                 if (input.explicit) warn(`${label}: skipped (${reason})`);
                 continue;
             }
+            if (lossy) {
+                // fix never rewrites bytes it could not decode
+                if (opts.command === 'fix') {
+                    skipped.push({ path: label, reason: 'not UTF-8' });
+                    warn(`${label}: skipped (not UTF-8)`);
+                    continue;
+                }
+                notUtf8.push(label);
+                warn(`${label}: not valid UTF-8; checked with invalid bytes read as U+FFFD`);
+            }
             visit(label, text, input);
         }
     };
@@ -691,14 +711,17 @@ async function main(argv) {
 
         const kind = opts.aiConfig ? 'AI configuration file' : 'file';
         if (opts.format === 'json') {
-            const summary = { files, total, filesScanned: scanned, filesWithFindings: files.length, skipped };
+            const summary = { files, total, filesScanned: scanned, filesWithFindings: files.length, skipped, notUtf8 };
             if (opts.aiConfig) summary.hiddenRunCount = hiddenRunCount;
             out(JSON.stringify(summary, null, 2));
         } else if (opts.format === 'sarif') {
             out(JSON.stringify(sarifLog(files, { cwd, total, filesScanned: scanned, skipped, succeeded: errors === 0 }), null, 2));
         } else {
             for (const line of lines) out(line);
-            if (opts.aiConfig && scanned === 0 && skipped.length === 0) out('No AI configuration files found.');
+            if (opts.aiConfig && scanned === 0 && skipped.length === 0) {
+                out('No AI configuration files found.');
+                if (opts.paths.length === 0 && !process.stdin.isTTY) out('To check piped text, pass "-": text-sanctifier check --ai-config -');
+            }
             else if (total === 0) out(`No ${opts.aiConfig ? 'hidden text, ' : ''}invisible or control characters in ${plural(scanned, kind)}.`);
             else out(`${plural(total, 'finding')}${opts.aiConfig ? ` (${plural(hiddenRunCount, 'hidden run')})` : ''} in ${files.length} of ${plural(scanned, kind)}.`);
         }

package/dist/text-sanctifier.min.js

@@ -3,4 +3,4 @@ var y=/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F-\u009F\u061C\u200E\u200F\u2
 `){return e.replace(Q,r)}var re=/[ \t]*(\n+)[ \t]*/g;function ne(e){return e.replace(re,"$1")}var te=/\n{2,}/g,ie=/\n{3,}/g;function oe(e,r){return r?e.replace(ie,`
 
 `):e.replace(te,`
-`)}var ue=/ {2,}/g;function se(e){return e.replace(ue," ")}function le(e){return e.replace(y,"")}function ae(e,r={}){if(typeof e!="string")throw new TypeError("revealInvisible expects a string input.");let n=r&&typeof r.format=="function"?r.format:i=>"<U+"+i.toString(16).toUpperCase().padStart(4,"0")+">",u="",o=0;for(let i of C(e,{maxFindings:1/0}).findings)u+=e.slice(o,i.offset)+n(i.codePoint,i.category),o=i.offset+(i.codePoint>65535?2:1);return u+e.slice(o)}var fe="1.3.7";export{fe as VERSION,b as getNewlineStats,K as getRecommendedSanctifierOptions,j as inspectText,C as inspectTextDetailed,ae as revealInvisible,T as sanctifyKeyboardOnly,N as sanctifyKeyboardOnlyEmoji,w as sanctifyLoose,L as sanctifyPasted,R as sanctifyStrict,W as summonSanctifier};
+`)}var ue=/ {2,}/g;function se(e){return e.replace(ue," ")}function le(e){return e.replace(y,"")}function ae(e,r={}){if(typeof e!="string")throw new TypeError("revealInvisible expects a string input.");let n=r&&typeof r.format=="function"?r.format:i=>"<U+"+i.toString(16).toUpperCase().padStart(4,"0")+">",u="",o=0;for(let i of C(e,{maxFindings:1/0}).findings)u+=e.slice(o,i.offset)+n(i.codePoint,i.category),o=i.offset+(i.codePoint>65535?2:1);return u+e.slice(o)}var fe="1.3.9";export{fe as VERSION,b as getNewlineStats,K as getRecommendedSanctifierOptions,j as inspectText,C as inspectTextDetailed,ae as revealInvisible,T as sanctifyKeyboardOnly,N as sanctifyKeyboardOnlyEmoji,w as sanctifyLoose,L as sanctifyPasted,R as sanctifyStrict,W as summonSanctifier};

package/dist/types/context.d.ts

@@ -3,6 +3,10 @@
  * the ones that belong to legitimate sequences as benign:
  * - U+200D between two emoji (Extended_Pictographic), allowing a skin-tone modifier or
  *   VS16 after the first: an emoji ZWJ sequence
+ * - U+200C or U+200D right after a virama (Devanagari, Bengali, Malayalam, Sinhala and
+ *   the other scripts with a combining class 9 mark), as IDNA ContextJ allows
+ * - U+200C between two Arabic-script letters, nonspacing marks skipped (Persian and
+ *   others); an approximation of IDNA's joining-type rule
  * - U+FE0E or U+FE0F directly after an emoji, or between 0-9, # or * and U+20E3 (keycap)
  * - one of U+E0100–U+E01EF directly after an ideograph: an ideographic variation sequence
  * - U+180B–U+180D or U+180F directly after a Mongolian letter
@@ -53,4 +57,10 @@ export type ContextReport = {
      * - Sum of `benignCounts`.
      */
     benignTotal: number;
+    /**
+     * - Benign U+E0100–U+E01EF (VS17–VS256),
+     * part of `benignCounts.variation_selector`. `supplementarySelectorCount` from findHiddenRuns
+     * minus this is the number of suspicious ones.
+     */
+    benignSupplementarySelectorCount: number;
 };

package/dist/types/hiddenText.d.ts

@@ -13,7 +13,10 @@ export function decodeTagText(text: string): string;
  * legitimate and listed under `benign` instead of `runs`:
  *  - the subdivision flags of England, Scotland and Wales: U+1F3F4 followed by the
  *    tags for "gbeng", "gbsct" or "gbwls" and U+E007F
- *  - a single U+FE0E or U+FE0F (text or emoji presentation) after another character
+ *  - a single U+FE0E or U+FE0F (text or emoji presentation) after an
+ *    Extended_Pictographic character, or between #, * or a digit and U+20E3 (a keycap),
+ *    the same rule as text-sanctifier/context. After a letter it is a run: one selector
+ *    per letter can carry a bit each.
  *
  * `line` and `column` are counted as in inspectTextDetailed. Every run and benign
  * span is counted; only the first `maxRuns` of each are listed. With the caps set,

package/dist/types/sanctifyText.d.ts

@@ -52,7 +52,7 @@ export function sanctifyPasted(text: string): string;
  * Brutal text normalizer and invisible trash scrubber,
  * configurable to kill whatever ghosts you want dead.
  *
- * ⚠️ Note: This is plain-text normalization/filtering — not an HTML/XSS sanitizer.
+ * Note: This is plain-text normalization/filtering — not an HTML/XSS sanitizer.
  *
  * @param {string} text
  * @param {boolean} [purgeInvisibleChars=false] - Remove ZWSP, NBSP, bidi, tags, variation selectors, etc.

package/dist/types/version.generated.d.ts

@@ -1,2 +1,2 @@
 /** The installed version of text-sanctifier, as in its package.json. */
-export const VERSION: "1.3.7";
+export const VERSION: "1.3.9";

package/dist/types/virama.generated.d.ts

@@ -0,0 +1 @@
+export const VIRAMA_AT: RegExp;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "text-sanctifier",
-  "version": "1.3.7",
+  "version": "1.3.9",
   "type": "module",
   "description": "A brutal text normalizer and invisible trash scrubber for modern web projects.",
   "engines": {
@@ -43,15 +43,21 @@
   "types": "./dist/types/index.d.ts",
   "sideEffects": false,
   "scripts": {
-    "gen": "node scripts/gen-categories.js && node scripts/gen-version.js",
-    "gen:check": "node scripts/gen-categories.js --check && node scripts/gen-version.js --check",
+    "gen": "node scripts/gen-categories.js && node scripts/gen-version.js && node scripts/gen-virama.js && node scripts/gen-docs.js",
+    "gen:check": "node scripts/gen-categories.js --check && node scripts/gen-version.js --check && node scripts/gen-virama.js --check && node scripts/gen-docs.js --check",
     "build": "node scripts/build.js",
     "build:types": "tsc -p tsconfig.types.json",
     "test:types": "tsc -p tests/types/tsconfig.json",
-    "test": "node tests/sanctifyText.test.js && node tests/inspectTextDetailed.test.js && node --expose-gc tests/hiddenText.test.js && node --expose-gc tests/context.test.js && node tests/cli.test.js",
+    "test": "node tests/sanctifyText.test.js && node tests/inspectTextDetailed.test.js && node --expose-gc tests/hiddenText.test.js && node --expose-gc tests/context.test.js && node tests/cli.test.js && node tests/release.test.js",
     "test-min": "node tests/sanctifyText.test.min.js",
-    "ci": "npm run gen:check && npm run test && npm run build && npm run size && npm run build:types && npm run test:types && npm run test-min",
-    "size": "node scripts/check-size.js"
+    "check:self": "node bin/text-sanctifier.js check .",
+    "check:names": "node scripts/check-names.js",
+    "ci": "npm run gen:check && npm run check:names && npm run check:self && npm run test && npm run build && npm run size && npm run build:types && npm run test:types && npm run test-min",
+    "size": "node scripts/check-size.js",
+    "bench": "node --expose-gc scripts/bench.js",
+    "release:prep": "node scripts/release.js prep",
+    "release:verify": "node scripts/release.js verify",
+    "hooks:install": "git config core.hooksPath scripts/hooks"
   },
   "keywords": [
     "text",

package/robots.txt

@@ -1,79 +1,79 @@
-# WATT3D — AI scraping policy
-#
-# This file enumerates known AI / LLM training and inference crawlers
-# that the WATT3D AI Training Rider does not consent to. See
-# ./ADDITIONAL_TERMS.md for the legal scope and definitions.
-#
-# Honoring robots.txt is voluntary. The rider applies whether or not
-# a bot respects this file. The enumeration is non-exhaustive — bots
-# not listed here are still bound by the rider if they perform
-# Training Use as defined in ADDITIONAL_TERMS.md §1.
-
-User-agent: GPTBot
-Disallow: /
-
-User-agent: ChatGPT-User
-Disallow: /
-
-User-agent: OAI-SearchBot
-Disallow: /
-
-User-agent: anthropic-ai
-Disallow: /
-
-User-agent: ClaudeBot
-Disallow: /
-
-User-agent: Claude-Web
-Disallow: /
-
-User-agent: Claude-SearchBot
-Disallow: /
-
-User-agent: Google-Extended
-Disallow: /
-
-User-agent: Applebot-Extended
-Disallow: /
-
-User-agent: Meta-ExternalAgent
-Disallow: /
-
-User-agent: Meta-ExternalFetcher
-Disallow: /
-
-User-agent: FacebookBot
-Disallow: /
-
-User-agent: CCBot
-Disallow: /
-
-User-agent: PerplexityBot
-Disallow: /
-
-User-agent: Perplexity-User
-Disallow: /
-
-User-agent: cohere-ai
-Disallow: /
-
-User-agent: Bytespider
-Disallow: /
-
-User-agent: Amazonbot
-Disallow: /
-
-User-agent: Omgilibot
-Disallow: /
-
-User-agent: AI2Bot
-Disallow: /
-
-User-agent: Diffbot
-Disallow: /
-
-# Everyone else: full access. Search engines and human-driven crawlers
-# remain welcome.
-
-User-agent: *
-Allow: /
+# WATT3D — AI scraping policy
+#
+# This file enumerates known AI / LLM training and inference crawlers
+# that the WATT3D AI Training Rider does not consent to. See
+# ./ADDITIONAL_TERMS.md for the legal scope and definitions.
+#
+# Honoring robots.txt is voluntary. The rider applies whether or not
+# a bot respects this file. The enumeration is non-exhaustive — bots
+# not listed here are still bound by the rider if they perform
+# Training Use as defined in ADDITIONAL_TERMS.md §1.
+
+User-agent: GPTBot
+Disallow: /
+
+User-agent: ChatGPT-User
+Disallow: /
+
+User-agent: OAI-SearchBot
+Disallow: /
+
+User-agent: anthropic-ai
+Disallow: /
+
+User-agent: ClaudeBot
+Disallow: /
+
+User-agent: Claude-Web
+Disallow: /
+
+User-agent: Claude-SearchBot
+Disallow: /
+
+User-agent: Google-Extended
+Disallow: /
+
+User-agent: Applebot-Extended
+Disallow: /
+
+User-agent: Meta-ExternalAgent
+Disallow: /
+
+User-agent: Meta-ExternalFetcher
+Disallow: /
+
+User-agent: FacebookBot
+Disallow: /
+
+User-agent: CCBot
+Disallow: /
+
+User-agent: PerplexityBot
+Disallow: /
+
+User-agent: Perplexity-User
+Disallow: /
+
+User-agent: cohere-ai
+Disallow: /
+
+User-agent: Bytespider
+Disallow: /
+
+User-agent: Amazonbot
+Disallow: /
+
+User-agent: Omgilibot
+Disallow: /
+
+User-agent: AI2Bot
+Disallow: /
+
+User-agent: Diffbot
+Disallow: /
+
+# Everyone else: full access. Search engines and human-driven crawlers
+# remain welcome.
+
+User-agent: *
+Allow: /

package/src/context.js

@@ -1,14 +1,16 @@
 // src/context.js
 //
 // inspectTextInContext: inspectTextDetailed plus which findings belong to legitimate
-// sequences (emoji joiners and selectors, ideographic variation sequences, Mongolian
-// selectors, subdivision flags), counted exactly per category. The categories are
+// sequences (emoji joiners and selectors, joiners in Indic and Arabic-script text,
+// ideographic variation sequences, Mongolian selectors, subdivision flags), counted
+// exactly per category. The categories are
 // unchanged; see "Proposal: context-aware findings" in DESIGN.md.
 //
 // Exported as `text-sanctifier/context`. It needs Unicode property escapes in regexes,
 // so it lives apart from the core and the browser file and has its own size budget.
 
 import { inspectTextDetailed } from './inspectText.js';
+import { VIRAMA_AT } from './virama.generated.js';
 
 /** @typedef {import('./inspectText.js').TrashFinding} TrashFinding */
 /** @typedef {import('./inspectText.js').TrashCounts} TrashCounts */
@@ -28,11 +30,16 @@ import { inspectTextDetailed } from './inspectText.js';
  * @property {TrashCounts} benignCounts - Benign occurrences per category, listed or not.
  *   `counts[c] - benignCounts[c]` is the number of suspicious characters in category `c`.
  * @property {number} benignTotal - Sum of `benignCounts`.
+ * @property {number} benignSupplementarySelectorCount - Benign U+E0100–U+E01EF (VS17–VS256),
+ *   part of `benignCounts.variation_selector`. `supplementarySelectorCount` from findHiddenRuns
+ *   minus this is the number of suspicious ones.
  */
 
 // Tested at an index of the text itself, so nothing is allocated per character.
 const PICTOGRAPHIC_AT = /\p{Extended_Pictographic}/yu;
 const IDEOGRAPHIC_AT = /\p{Ideographic}/yu;
+const ARABIC_LETTER_AT = /(?=\p{Script=Arabic})\p{L}/yu;
+const MARK_AT = /\p{Mn}/yu;
 
 // RGI subdivision flags: U+1F3F4, the tag letters, U+E007F
 const SUBDIVISION_FLAGS = ['gbeng', 'gbsct', 'gbwls'];
@@ -80,6 +87,29 @@ function isEmojiJoiner(text, i) {
     return p >= 0 && matchesAt(PICTOGRAPHIC_AT, text, p) && matchesAt(PICTOGRAPHIC_AT, text, i + 1);
 }
 
+/**
+ * IDNA ContextJ (RFC 5892, appendix A.1 and A.2): a U+200C or U+200D is benign right
+ * after a virama, and a U+200C also between two Arabic-script letters, skipping
+ * nonspacing marks on either side. The second rule approximates the RFC's joining types
+ * (L or D before, R or D after), which JavaScript regexes can't express: it also allows
+ * a ZWNJ after a right-joining or non-joining Arabic letter, and none in other joining
+ * scripts such as Syriac or N'Ko.
+ * @param {string} text
+ * @param {number} i - index of the U+200C or U+200D
+ * @param {boolean} zwnj - true for U+200C
+ */
+function isScriptJoiner(text, i, zwnj) {
+    let p = previousStart(text, i);
+    if (p < 0) return false;
+    if (matchesAt(VIRAMA_AT, text, p)) return true;
+    if (!zwnj) return false;
+    while (p >= 0 && matchesAt(MARK_AT, text, p)) p = previousStart(text, p);
+    if (p < 0 || !matchesAt(ARABIC_LETTER_AT, text, p)) return false;
+    let q = i + 1;
+    while (matchesAt(MARK_AT, text, q)) q = MARK_AT.lastIndex;
+    return matchesAt(ARABIC_LETTER_AT, text, q);
+}
+
 /**
  * Length in UTF-16 units of the subdivision-flag tags starting at `i` (just after a
  * U+1F3F4), or 0 when they are not exactly one of the RGI flags, as findHiddenRuns decides.
@@ -106,6 +136,10 @@ function flagTagsAt(text, i) {
  * the ones that belong to legitimate sequences as benign:
  * - U+200D between two emoji (Extended_Pictographic), allowing a skin-tone modifier or
  *   VS16 after the first: an emoji ZWJ sequence
+ * - U+200C or U+200D right after a virama (Devanagari, Bengali, Malayalam, Sinhala and
+ *   the other scripts with a combining class 9 mark), as IDNA ContextJ allows
+ * - U+200C between two Arabic-script letters, nonspacing marks skipped (Persian and
+ *   others); an approximation of IDNA's joining-type rule
  * - U+FE0E or U+FE0F directly after an emoji, or between 0-9, # or * and U+20E3 (keycap)
  * - one of U+E0100–U+E01EF directly after an ideograph: an ideographic variation sequence
  * - U+180B–U+180D or U+180F directly after a Mongolian letter
@@ -132,6 +166,7 @@ export function inspectTextInContext(text, options = {}) {
     const benignCounts = /** @type {TrashCounts} */ ({});
     for (const key of Object.keys(report.counts)) benignCounts[/** @type {keyof TrashCounts} */ (key)] = 0;
     let benignTotal = 0;
+    let benignSupplementarySelectorCount = 0;
     let next = 0; // first listed finding not yet passed
 
     /**
@@ -150,7 +185,9 @@ export function inspectTextInContext(text, options = {}) {
         if (c < 0x180B) continue; // the common case: nothing below can be benign
 
         if (c === 0x200D) {
-            if (isEmojiJoiner(text, i)) markBenign(i, 'zero_width');
+            if (isEmojiJoiner(text, i) || isScriptJoiner(text, i, false)) markBenign(i, 'zero_width');
+        } else if (c === 0x200C) {
+            if (isScriptJoiner(text, i, true)) markBenign(i, 'zero_width');
         } else if (c === 0xFE0E || c === 0xFE0F) {
             const p = previousStart(text, i);
             const before = p < 0 ? -1 : text.charCodeAt(p);
@@ -167,11 +204,14 @@ export function inspectTextInContext(text, options = {}) {
             const low = text.charCodeAt(i + 1);
             if (low >= 0xDD00 && low <= 0xDDEF) { // U+E0100-U+E01EF
                 const p = previousStart(text, i);
-                if (p >= 0 && matchesAt(IDEOGRAPHIC_AT, text, p)) markBenign(i, 'variation_selector');
+                if (p >= 0 && matchesAt(IDEOGRAPHIC_AT, text, p)) {
+                    markBenign(i, 'variation_selector');
+                    benignSupplementarySelectorCount++;
+                }
             }
             if (low >= 0xDC00 && low <= 0xDFFF) i++;
         }
     }
 
-    return { findings, counts: report.counts, total: report.total, truncated: report.truncated, benignCounts, benignTotal };
+    return { findings, counts: report.counts, total: report.total, truncated: report.truncated, benignCounts, benignTotal, benignSupplementarySelectorCount };
 }