text-sanctifier 1.1.1 → 1.3.0

package/README.md

@@ -7,13 +7,15 @@
 [![stars](https://img.shields.io/github/stars/iWhatty/text-sanctifier?style=social)](https://github.com/iWhatty/text-sanctifier)
 [![types](https://img.shields.io/npm/types/text-sanctifier)](https://www.npmjs.com/package/text-sanctifier)
 
-Brutal text normalizer and invisible Unicode scrubber for modern web projects. ~2.17 KB gzipped.
+Brutal text normalizer and invisible Unicode scrubber for modern web projects. ~2.40 KB gzipped.
 
 ## Features
 
 - Purges zero-width Unicode garbage
 - Purges bidi controls, including the isolates (U+2066–U+2069) and the Arabic letter mark (U+061C) used in Trojan Source attacks
+- Purges Unicode tag characters (U+E0000–U+E007F), the "ASCII smuggling" channel for hiding instructions from humans in text an LLM reads, plus variation selectors, Hangul fillers, soft hyphens and invisible operators
 - Locates invisible and control characters by offset, line and column (`inspectTextDetailed`)
+- `text-sanctifier check` / `fix` command line for CI and coding agents, with JSON output and exit codes
 - Normalizes line endings (CRLF, CR, LF) to LF
 - Collapses unwanted spaces and paragraphs
 - Nukes control characters (if enabled)
@@ -154,7 +156,8 @@ findings: [
   { offset: 0, line: 1, column: 1, codePoint: 0xFEFF, category: 'bom', atStart: true },
   { offset: 42, line: 3, column: 7, codePoint: 0x2067, category: 'bidi_control' },
 ]
-counts: { bidi_control: 1, zero_width: 0, nonbreaking_space: 0, bom: 1, nul: 0, c0_control: 0, c1_control: 0 }
+counts: { bidi_control: 1, zero_width: 0, nonbreaking_space: 0, bom: 1, nul: 0, c0_control: 0, c1_control: 0,
+          invisible_format: 0, hangul_filler: 0, variation_selector: 0, tag: 0 }
 total: 2
 truncated: false
 */
@@ -169,15 +172,73 @@ truncated: false
 | `nul` | U+0000 |
 | `c0_control` | U+0001–U+001F except tab, LF and CR; U+007F |
 | `c1_control` | U+0080–U+009F |
+| `invisible_format` | U+00AD (soft hyphen), U+034F, U+180E, U+2061–U+2064 (invisible operators) |
+| `hangul_filler` | U+115F, U+1160, U+3164, U+FFA0 (letters that render blank) |
+| `variation_selector` | U+FE00–U+FE0F, U+E0100–U+E01EF |
+| `tag` | U+E0000–U+E007F (tag characters, used for "ASCII smuggling" of hidden instructions) |
 
-`offset` is a 0-based UTF-16 index; `line` and `column` are 1-based, with columns in UTF-16 code units and lines ending at LF, CR or CRLF. Every occurrence is counted; only the first `maxFindings` (default 100, `Infinity` for all) are listed, and `truncated` says when some were not. The scan is one pass with no regex and no copy of the input.
+`codePoint` is the full code point, so a tag character reports `0xE0041`, not a surrogate. `offset` is a 0-based UTF-16 index (for a code point above U+FFFF, that of its high surrogate); `line` and `column` are 1-based, with columns in UTF-16 code units and lines ending at LF, CR or CRLF. Every occurrence is counted; only the first `maxFindings` (default 100, `Infinity` for all) are listed, and `truncated` says when some were not. The scan is one pass with no regex and no copy of the input.
 
-`zero_width` also flags U+200D (zero width joiner) inside emoji ZWJ sequences such as 👨‍👩‍👧, so a finding there is not necessarily hidden text. Filter by neighbouring code points if you need to tell them apart. (`purgeInvisibleChars` removes those joiners too, splitting the emoji; `keyboardOnlyEmoji` keeps the sequences whole.)
+`zero_width` also flags U+200D (zero width joiner) inside emoji ZWJ sequences such as 👨‍👩‍👧, so a finding there is not necessarily hidden text. Filter by neighbouring code points if you need to tell them apart. (`purgeInvisibleChars` removes those joiners too, splitting the emoji; `keyboardOnlyEmoji` keeps the sequences whole.) The same holds for `variation_selector`, which flags VS16 (U+FE0F) in emoji such as ✌️ and the ideographic variation selectors in CJK names, and for `tag`, which flags the tag characters inside subdivision flags such as the England flag (U+1F3F4 followed by tags). `purgeInvisibleChars` strips them all; `keyboardOnlyEmoji` keeps VS16.
 
 The categories cover exactly what `nukeControls` and `purgeInvisibleChars` strip.
 
 ---
 
+## Command line
+
+The package ships a `text-sanctifier` command for CI and coding agents, which cannot see invisible characters in their own tool output. It runs on Node 18+ and has no dependencies.
+
+```sh
+npx text-sanctifier check src docs "**/*.md"
+npx text-sanctifier fix --dry-run src
+```
+
+### `check [paths...]`
+
+Reports every invisible or control character as `path:line:column  U+XXXX  category`, using the categories of `inspectTextDetailed`:
+
+```
+docs/intro.md:12:7  U+200B  zero_width
+src/auth.js:40:3  U+202E  bidi_control
+2 findings in 2 of 31 files.
+```
+
+- `--json`: one JSON object on stdout, `{ files, total, filesScanned, filesWithFindings, skipped }`. Each entry in `files` is `{ path, ...inspectTextDetailed(text) }`; only files with findings are listed.
+- `--max-findings N`: findings listed per file (default 100; `all` for no cap). Counts always cover every occurrence.
+- `--strip-bom`: also report a BOM at offset 0, which is skipped by default.
+
+### `fix [paths...]`
+
+Rewrites files in place and reports what it removed.
+
+- `--purge-invisible`: `purgeInvisibleChars`
+- `--nuke-controls`: `nukeControls`
+- `--normalize-newlines`: `normalizeNewlines` (CRLF and CR to LF)
+- With none of the three, `fix` uses `--purge-invisible --nuke-controls`, which removes exactly what `check` reports.
+- `--strip-bom`: also remove a BOM at offset 0. It is kept by default.
+- `--dry-run`: write nothing; report what would change.
+- `--json`: `{ files, dryRun, filesScanned, filesChanged, skipped }`, where each file is `{ path, removed, removedTotal, newlinesNormalized }`.
+
+`fix` deletes characters; it does not replace them. A non-breaking space between two words joins them, and emoji lose their zero width joiners and variation selectors. Run `check` or `fix --dry-run` first on prose.
+
+### Inputs
+
+- Paths can be files, directories or globs (`*`, `**`, `?`, `[...]`, `{a,b}`). Quote globs so the shell passes them through.
+- Inside a git work tree, directories and globs list files with `git ls-files`, so anything `.gitignore` excludes is skipped. Outside git, `.git` and `node_modules` are skipped. A file named explicitly is always read.
+- Binary files (a NUL byte in the first 8000 bytes) and files that are not valid UTF-8 are skipped, and `fix` never writes them. In JSON they appear under `skipped`.
+- `-`, or no paths with piped input, reads stdin. `fix` writes the cleaned text to stdout and the report to stderr.
+
+### Exit codes
+
+| Code | `check` | `fix` |
+| --- | --- | --- |
+| 0 | nothing found | done (or nothing to do) |
+| 1 | findings | `--dry-run` with changes pending |
+| 2 | usage error, missing path or unreadable file | same |
+
+---
+
 ## Notes
 
 ### Runtime requirements

package/bin/text-sanctifier.js

@@ -0,0 +1,519 @@
+#!/usr/bin/env node
+// bin/text-sanctifier.js
+//
+// Command-line front end for coding agents and CI:
+//   text-sanctifier check [paths...]   report invisible and control characters
+//   text-sanctifier fix [paths...]     remove them in place
+//
+// Plain Node (>=18), no dependencies. Exit codes: 0 clean, 1 findings (check,
+// or fix --dry-run with changes pending), 2 usage or I/O error.
+
+import { readFileSync, writeFileSync, statSync, readdirSync, existsSync } from 'node:fs';
+import { resolve, join, sep } from 'node:path';
+import { spawnSync } from 'node:child_process';
+import { inspectTextDetailed, summonSanctifier } from '../src/index.js';
+import { getNewlineStats } from '../src/inspectText.js';
+
+const VERSION = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version;
+
+const USAGE = `text-sanctifier ${VERSION}
+
+Usage:
+  text-sanctifier check [options] [paths...]
+  text-sanctifier fix [options] [paths...]
+
+Paths may be files, directories or globs (quote globs so the shell leaves them
+alone). "-", or no paths with piped input, reads stdin. Directories and globs
+skip files ignored by git (inside a git work tree) and skip binary files.
+
+check options:
+  --json               JSON report on stdout
+  --max-findings N     findings listed per file (default 100, "all" for no cap)
+  --strip-bom          also report a BOM at the start of a file
+
+fix options:
+  --purge-invisible    remove zero-width, non-breaking space, bidi and BOM characters
+  --nuke-controls      remove C0/C1 control characters and bidi controls
+  --normalize-newlines convert CRLF and CR to LF
+  --strip-bom          also remove a BOM at the start of a file (kept by default)
+  --dry-run            report what would change; write nothing
+  --json               JSON report
+  With no cleaning flag, fix uses --purge-invisible --nuke-controls.
+  stdin is cleaned to stdout; the report then goes to stderr.
+
+Other:
+  -h, --help           show this help
+  -v, --version        show the version
+
+Exit codes: 0 clean, 1 findings (check) or changes pending (fix --dry-run),
+2 usage or I/O error.
+`;
+
+const BOM = 0xFEFF;
+const STDIN = '-';
+const STDIN_LABEL = '<stdin>';
+const BINARY_SNIFF_BYTES = 8000;
+const WALK_SKIP_DIRS = new Set(['.git', 'node_modules']);
+const CASE_INSENSITIVE_FS = process.platform === 'win32' || process.platform === 'darwin';
+
+class UsageError extends Error {}
+
+
+// --- Arguments ---
+
+const BOOLEAN_FLAGS = {
+    '--json': 'json',
+    '--strip-bom': 'stripBom',
+    '--purge-invisible': 'purgeInvisible',
+    '--nuke-controls': 'nukeControls',
+    '--normalize-newlines': 'normalizeNewlines',
+    '--dry-run': 'dryRun',
+};
+const FIX_ONLY_FLAGS = new Set(['purgeInvisible', 'nukeControls', 'normalizeNewlines', 'dryRun']);
+
+/**
+ * @param {string[]} argv
+ */
+function parseArgs(argv) {
+    const opts = {
+        command: null,
+        paths: [],
+        json: false,
+        stripBom: false,
+        purgeInvisible: false,
+        nukeControls: false,
+        normalizeNewlines: false,
+        dryRun: false,
+        maxFindings: 100,
+        help: false,
+        version: false,
+    };
+    let endOfOptions = false;
+
+    for (let i = 0; i < argv.length; i++) {
+        const arg = argv[i];
+
+        if (endOfOptions || arg === STDIN || !arg.startsWith('-')) {
+            if (opts.command === null && !endOfOptions) opts.command = arg;
+            else opts.paths.push(arg);
+            continue;
+        }
+        if (arg === '--') { endOfOptions = true; continue; }
+        if (arg === '-h' || arg === '--help') { opts.help = true; continue; }
+        if (arg === '-v' || arg === '--version') { opts.version = true; continue; }
+
+        const [name, inline] = arg.split(/=(.*)/s, 2);
+        if (name === '--max-findings') {
+            const value = inline !== undefined ? inline : argv[++i];
+            opts.maxFindings = parseMaxFindings(value);
+            continue;
+        }
+        if (name in BOOLEAN_FLAGS && inline === undefined) {
+            opts[BOOLEAN_FLAGS[name]] = true;
+            continue;
+        }
+        throw new UsageError(`Unknown option: ${arg}`);
+    }
+
+    if (opts.help || opts.version) return opts;
+    if (opts.command !== 'check' && opts.command !== 'fix') {
+        throw new UsageError(opts.command === null ? 'Missing command.' : `Unknown command: ${opts.command}`);
+    }
+    if (opts.command === 'check') {
+        for (const [flag, key] of Object.entries(BOOLEAN_FLAGS)) {
+            if (FIX_ONLY_FLAGS.has(key) && opts[key]) throw new UsageError(`${flag} only applies to fix.`);
+        }
+    }
+    if (opts.command === 'fix' && !opts.purgeInvisible && !opts.nukeControls && !opts.normalizeNewlines) {
+        opts.purgeInvisible = true;
+        opts.nukeControls = true;
+    }
+    return opts;
+}
+
+/** @param {string|undefined} value */
+function parseMaxFindings(value) {
+    if (value === 'all') return Infinity;
+    const n = Number(value);
+    if (value === undefined || value === '' || !Number.isInteger(n) || n < 0) {
+        throw new UsageError(`--max-findings expects a non-negative integer or "all", got ${value === undefined ? 'nothing' : JSON.stringify(value)}.`);
+    }
+    return n;
+}
+
+
+// --- File collection ---
+
+const GLOB_CHARS = /[*?[{]/;
+
+/** @param {string} p */
+const toSlash = (p) => p.split(sep).join('/');
+
+/** @param {string} cwd */
+function isGitWorkTree(cwd) {
+    const r = spawnSync('git', ['rev-parse', '--is-inside-work-tree'], { cwd, encoding: 'utf8' });
+    return !r.error && r.status === 0 && r.stdout.trim() === 'true';
+}
+
+/**
+ * Lists tracked and untracked-but-not-ignored files under one pathspec.
+ * Returns null when git cannot answer (for example a path outside the work tree).
+ * @param {string} cwd
+ * @param {string} pathspec
+ * @returns {string[]|null}
+ */
+function gitListFiles(cwd, pathspec) {
+    const r = spawnSync(
+        'git',
+        ['ls-files', '-z', '--cached', '--others', '--exclude-standard', '--', pathspec],
+        { cwd, encoding: 'utf8', maxBuffer: 1 << 28 }
+    );
+    if (r.error || r.status !== 0) return null;
+    // --cached lists tracked files deleted from the work tree; drop them
+    return r.stdout.split('\0').filter((f) => f && existsSync(resolve(cwd, f)));
+}
+
+/**
+ * Recursive file walk used outside git. Skips .git and node_modules.
+ * @param {string} cwd
+ * @param {string} dir - relative or absolute, as the user typed it
+ * @returns {string[]}
+ */
+function walk(cwd, dir) {
+    const out = [];
+    const stack = [dir];
+    while (stack.length) {
+        const current = stack.pop();
+        let entries;
+        try {
+            entries = readdirSync(resolve(cwd, current), { withFileTypes: true });
+        } catch {
+            continue;
+        }
+        entries.sort((a, b) => (a.name < b.name ? 1 : a.name > b.name ? -1 : 0));
+        for (const e of entries) {
+            const p = current === '.' ? e.name : join(current, e.name);
+            if (e.isDirectory()) {
+                if (!WALK_SKIP_DIRS.has(e.name)) stack.push(p);
+            } else if (e.isFile()) {
+                out.push(toSlash(p));
+            }
+        }
+    }
+    return out.sort();
+}
+
+/**
+ * Glob to RegExp for the non-git fallback: `**`, `*`, `?`, `[...]` and `{a,b}`.
+ * @param {string} glob
+ */
+function globToRegExp(glob) {
+    let re = '';
+    for (let i = 0; i < glob.length; i++) {
+        const c = glob[i];
+        if (c === '*') {
+            if (glob[i + 1] === '*') {
+                const slashAfter = glob[i + 2] === '/';
+                re += slashAfter ? '(?:.*/)?' : '.*';
+                i += slashAfter ? 2 : 1;
+            } else {
+                re += '[^/]*';
+            }
+        } else if (c === '?') {
+            re += '[^/]';
+        } else if (c === '[') {
+            const close = glob.indexOf(']', i + 1);
+            if (close === -1) { re += '\\['; continue; }
+            let body = glob.slice(i + 1, close).replace(/\\/g, '\\\\');
+            if (body[0] === '!') body = '^' + body.slice(1);
+            re += `[${body}]`;
+            i = close;
+        } else if (c === '{') {
+            const close = glob.indexOf('}', i + 1);
+            if (close === -1) { re += '\\{'; continue; }
+            const alts = glob.slice(i + 1, close).split(',').map((a) => a.replace(/[.+^$()|\\]/g, '\\$&'));
+            re += `(?:${alts.join('|')})`;
+            i = close;
+        } else {
+            re += c.replace(/[.+^$()|\\\]}]/g, '\\$&');
+        }
+    }
+    return new RegExp(`^${re}$`);
+}
+
+/**
+ * @param {string} cwd
+ * @param {string} glob
+ */
+function walkGlob(cwd, glob) {
+    const pattern = toSlash(glob).replace(/^\.\//, '');
+    const segments = pattern.split('/');
+    const firstGlob = segments.findIndex((s) => GLOB_CHARS.test(s));
+    const base = segments.slice(0, firstGlob).join('/') || '.';
+    const re = globToRegExp(pattern);
+    return walk(cwd, base).filter((f) => re.test(f));
+}
+
+/**
+ * Expands the path arguments into a de-duplicated input list.
+ * @param {string[]} args
+ * @param {string} cwd
+ * @param {(msg: string) => void} warn
+ * @returns {{ inputs: Array<{ path: string, explicit: boolean }>, errors: number }}
+ */
+function collectInputs(args, cwd, warn) {
+    const inputs = [];
+    const seen = new Set();
+    let errors = 0;
+    let inGit;
+
+    const add = (path, explicit) => {
+        const abs = resolve(cwd, path);
+        const key = path === STDIN ? STDIN : CASE_INSENSITIVE_FS ? abs.toLowerCase() : abs;
+        if (seen.has(key)) return;
+        seen.add(key);
+        inputs.push({ path, explicit });
+    };
+
+    for (const arg of args) {
+        if (arg === STDIN) { add(STDIN, true); continue; }
+
+        let stat = null;
+        try { stat = statSync(resolve(cwd, arg)); } catch { /* not a plain path */ }
+
+        if (stat && stat.isFile()) { add(arg, true); continue; }
+
+        const isGlob = !stat && GLOB_CHARS.test(arg);
+        if (!stat && !isGlob) {
+            warn(`${arg}: no such file or directory`);
+            errors++;
+            continue;
+        }
+
+        if (inGit === undefined) inGit = isGitWorkTree(cwd);
+        const listed =
+            (inGit && gitListFiles(cwd, isGlob ? `:(glob)${toSlash(arg)}` : toSlash(arg))) ||
+            (isGlob ? walkGlob(cwd, arg) : walk(cwd, arg));
+
+        if (listed.length === 0) warn(`${arg}: no files matched`);
+        for (const f of listed) add(f, false);
+    }
+    return { inputs, errors };
+}
+
+
+// --- Reading ---
+
+const utf8 = new TextDecoder('utf-8', { fatal: true, ignoreBOM: true });
+
+/**
+ * Decodes UTF-8, keeping a leading BOM. Returns null for binary or non-UTF-8 data.
+ * @param {Uint8Array} bytes
+ * @returns {{ text: string|null, reason?: string }}
+ */
+function decode(bytes) {
+    if (bytes.subarray(0, BINARY_SNIFF_BYTES).includes(0)) return { text: null, reason: 'binary' };
+    try {
+        return { text: utf8.decode(bytes) };
+    } catch {
+        return { text: null, reason: 'not UTF-8' };
+    }
+}
+
+/** @returns {Promise<Buffer>} */
+function readStdin() {
+    return new Promise((res, rej) => {
+        const chunks = [];
+        process.stdin.on('data', (c) => chunks.push(c));
+        process.stdin.on('end', () => res(Buffer.concat(chunks)));
+        process.stdin.on('error', rej);
+    });
+}
+
+
+// --- check ---
+
+/**
+ * inspectTextDetailed, minus a BOM at offset 0 unless stripBom.
+ * @param {string} text
+ * @param {number} maxFindings
+ * @param {boolean} stripBom
+ */
+function inspect(text, maxFindings, stripBom) {
+    const leadingBom = !stripBom && text.charCodeAt(0) === BOM;
+    const report = inspectTextDetailed(text, { maxFindings: leadingBom ? maxFindings + 1 : maxFindings });
+    if (leadingBom) {
+        report.findings.shift();
+        report.counts.bom--;
+        report.total--;
+        report.truncated = report.total > report.findings.length;
+    }
+    return report;
+}
+
+/** @param {number} cp */
+const hex = (cp) => 'U+' + cp.toString(16).toUpperCase().padStart(4, '0');
+
+/**
+ * @param {Record<string, number>} counts
+ */
+const formatCounts = (counts) =>
+    Object.entries(counts).filter(([, n]) => n > 0).map(([k, n]) => `${k} ${n}`).join(', ');
+
+
+// --- Main ---
+
+/**
+ * @param {string[]} argv
+ * @returns {Promise<number>} exit code
+ */
+async function main(argv) {
+    const out = (s) => process.stdout.write(s + '\n');
+    const err = (s) => process.stderr.write(s + '\n');
+
+    let opts;
+    try {
+        opts = parseArgs(argv);
+    } catch (e) {
+        if (!(e instanceof UsageError)) throw e;
+        err(`text-sanctifier: ${e.message}\nRun "text-sanctifier --help" for usage.`);
+        return 2;
+    }
+    if (opts.help) { out(USAGE); return 0; }
+    if (opts.version) { out(VERSION); return 0; }
+
+    const cwd = process.cwd();
+    let paths = opts.paths;
+    if (paths.length === 0) {
+        if (process.stdin.isTTY) {
+            err('text-sanctifier: no paths given and nothing piped on stdin.\nRun "text-sanctifier --help" for usage.');
+            return 2;
+        }
+        paths = [STDIN];
+    }
+
+    const warn = (msg) => err(`text-sanctifier: ${msg}`);
+    const { inputs, errors: pathErrors } = collectInputs(paths, cwd, warn);
+    let errors = pathErrors;
+    /** @type {Array<{ path: string, reason: string }>} */
+    const skipped = [];
+
+    /**
+     * Reads every input in order, calling `visit(label, text, input)` for each decodable one.
+     * @param {(label: string, text: string, input: { path: string, explicit: boolean }) => void} visit
+     */
+    const forEachText = async (visit) => {
+        for (const input of inputs) {
+            const label = input.path === STDIN ? STDIN_LABEL : input.path;
+            let bytes;
+            try {
+                bytes = input.path === STDIN ? await readStdin() : readFileSync(resolve(cwd, input.path));
+            } catch (e) {
+                warn(`${label}: ${e.code || e.message}`);
+                errors++;
+                continue;
+            }
+            const { text, reason } = decode(bytes);
+            if (text === null) {
+                skipped.push({ path: label, reason });
+                if (input.explicit) warn(`${label}: skipped (${reason})`);
+                continue;
+            }
+            visit(label, text, input);
+        }
+    };
+
+    if (opts.command === 'check') {
+        const files = [];
+        let scanned = 0;
+        let total = 0;
+
+        await forEachText((label, text) => {
+            scanned++;
+            const report = inspect(text, opts.maxFindings, opts.stripBom);
+            total += report.total;
+            if (report.total > 0) files.push({ path: label, ...report });
+        });
+
+        if (opts.json) {
+            out(JSON.stringify({ files, total, filesScanned: scanned, filesWithFindings: files.length, skipped }, null, 2));
+        } else {
+            for (const f of files) {
+                for (const x of f.findings) out(`${f.path}:${x.line}:${x.column}  ${hex(x.codePoint)}  ${x.category}`);
+                if (f.truncated) out(`${f.path}: ${f.total - f.findings.length} more not listed (--max-findings)`);
+            }
+            out(total === 0
+                ? `No invisible or control characters in ${scanned} file${scanned === 1 ? '' : 's'}.`
+                : `${total} finding${total === 1 ? '' : 's'} in ${files.length} of ${scanned} file${scanned === 1 ? '' : 's'}.`);
+        }
+        return errors > 0 ? 2 : total > 0 ? 1 : 0;
+    }
+
+    // fix
+    const clean = summonSanctifier({
+        purgeInvisibleChars: opts.purgeInvisible,
+        nukeControls: opts.nukeControls,
+        normalizeNewlines: opts.normalizeNewlines,
+    });
+    const writesStdout = !opts.dryRun && inputs.some((i) => i.path === STDIN);
+    const report = writesStdout ? err : out;
+    const files = [];
+    let scanned = 0;
+
+    await forEachText((label, text, input) => {
+        scanned++;
+        // The BOM at offset 0 is set aside so purgeInvisibleChars cannot take it
+        const leadingBom = text.charCodeAt(0) === BOM;
+        let cleaned = clean(leadingBom ? text.slice(1) : text);
+        if (leadingBom && !opts.stripBom) cleaned = String.fromCharCode(BOM) + cleaned;
+
+        if (input.path === STDIN && !opts.dryRun) process.stdout.write(cleaned);
+        if (cleaned === text) return;
+
+        const before = inspectTextDetailed(text, { maxFindings: 0 }).counts;
+        const after = inspectTextDetailed(cleaned, { maxFindings: 0 }).counts;
+        const removed = {};
+        let removedTotal = 0;
+        for (const k of Object.keys(before)) {
+            const n = before[k] - (after[k] || 0);
+            if (n > 0) { removed[k] = n; removedTotal += n; }
+        }
+        const nl = getNewlineStats(text);
+        const newlinesNormalized = opts.normalizeNewlines && nl.crlf + nl.cr > 0;
+
+        if (input.path !== STDIN && !opts.dryRun) {
+            try {
+                writeFileSync(resolve(cwd, input.path), cleaned, 'utf8');
+            } catch (e) {
+                warn(`${label}: ${e.code || e.message}`);
+                errors++;
+                return;
+            }
+        }
+        files.push({ path: label, removed, removedTotal, newlinesNormalized });
+    });
+
+    if (opts.json) {
+        report(JSON.stringify({ files, dryRun: opts.dryRun, filesScanned: scanned, filesChanged: files.length, skipped }, null, 2));
+    } else {
+        const verb = opts.dryRun ? 'would fix' : 'fixed';
+        for (const f of files) {
+            const parts = [];
+            if (f.removedTotal > 0) parts.push(`removed ${f.removedTotal} (${formatCounts(f.removed)})`);
+            if (f.newlinesNormalized) parts.push('normalized newlines');
+            report(`${verb} ${f.path}: ${parts.join('; ')}`);
+        }
+        report(files.length === 0
+            ? `Nothing to fix in ${scanned} file${scanned === 1 ? '' : 's'}.`
+            : `${opts.dryRun ? 'Would change' : 'Changed'} ${files.length} of ${scanned} file${scanned === 1 ? '' : 's'}.`);
+    }
+    return errors > 0 ? 2 : opts.dryRun && files.length > 0 ? 1 : 0;
+}
+
+main(process.argv.slice(2)).then(
+    (code) => { process.exitCode = code; },
+    (e) => {
+        process.stderr.write(`text-sanctifier: ${e && e.stack ? e.stack : e}\n`);
+        process.exitCode = 2;
+    }
+);

package/dist/text-sanctifier.min.js

@@ -1,12 +1,14 @@
-function n(a={}){const c=!!a.purgeInvisibleChars,d=!!a.purgeEmojis,f=!!a.nukeControls,g=!!a.keyboardOnlyFilter,l=!!a.normalizeNewlines,h=!!a.trimSpacesAroundNewlines,b=!!a.collapseNewLines,e=!!a.preserveParagraphs,k=!!a.collapseSpaces,m=!!a.finalTrim,p=Number.isFinite(a.maxLength)&&0<=a.maxLength?a.maxLength:Infinity,q="truncate"===a.g||"noop"===a.g||"throw"===a.g?a.g:"throw";return z=>r(z,c,d,f,g,l,h,b,e,k,m,p,q)}n.strict=a=>r(a,!0,!0,!0,!1,!0,!0,!0,!1,!0,!0);
+function n(a={}){const c=!!a.purgeInvisibleChars,e=!!a.purgeEmojis,f=!!a.nukeControls,h=!!a.keyboardOnlyFilter,m=!!a.normalizeNewlines,l=!!a.trimSpacesAroundNewlines,b=!!a.collapseNewLines,d=!!a.preserveParagraphs,g=!!a.collapseSpaces,p=!!a.finalTrim,k=Number.isFinite(a.maxLength)&&0<=a.maxLength?a.maxLength:Infinity,q="truncate"===a.g||"noop"===a.g||"throw"===a.g?a.g:"throw";return z=>r(z,c,e,f,h,m,l,b,d,g,p,k,q)}n.strict=a=>r(a,!0,!0,!0,!1,!0,!0,!0,!1,!0,!0);
 n.loose=a=>r(a,!1,!1,!1,!1,!0,!0,!0,!0,!0,!0);n.keyboardOnlyEmoji=a=>r(a,!1,!1,!1,!0,!0,!0,!1,!1,!1,!0);n.keyboardOnly=a=>r(a,!0,!0,!0,!0,!0,!0,!0,!1,!0,!0);
-function r(a,c=!1,d=!1,f=!1,g=!1,l=!1,h=!1,b=!1,e=!1,k=!1,m=!1,p=Infinity,q="throw"){if("string"!==typeof a)throw new TypeError("sanctifyText expects a string input.");"truncate"!==q&&"noop"!==q&&"throw"!==q&&(q="throw");if(a.length>p)switch(q){case "truncate":a=a.slice(0,p);break;case "noop":return a;default:throw new RangeError(`sanctifyText input length ${a.length} exceeds maxLength ${p}.`);}c&&(a=a.replace(t,""));d&&(a=a.replace(u,""));f&&(a=a.replace(v,""));g&&(a=w(a,d));l&&(a=a.replace(x,"\n"));
-h&&(a=a.replace(y,"$1"));b&&(c=a,a=e?c.replace(A,"\n\n"):c.replace(B,"\n"));k&&(a=a.replace(C," "));return m?a.trim():a}var t=/[\u00A0\u2000-\u200D\u202F\u2060\u3000\uFEFF\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]+/g,D=/[^\x20-\x7E\r\n]+/gu;function w(a,c=!1){a=E(a);return c?a.replace(D,""):a.replace(D,d=>{u.lastIndex=0;if(u.test(d)){u.lastIndex=0;var f="";for(const g of d.matchAll(u))f+=g[0];d=f}else d="";return d})}
-var F=/[\u2018\u2019\u201A\u201B\u2032\u2035]/g,G=/[\u201C\u201D\u201E\u201F\u2033\u2036\u00AB\u00BB]/g,H=/[\u2012\u2013\u2014\u2015\u2212]/g,I=/\u2026/g,J=/[\u2022\u00B7]/g,K=/[\uFF01-\uFF5E]/g;function E(a){return a.replace(F,"'").replace(G,'"').replace(H,"-").replace(I,"...").replace(J,"*").replace(K,c=>String.fromCharCode(c.charCodeAt(0)-65248))}var u;try{u=RegExp("(?:\\p{Extended_Pictographic}(?:\\uFE0F|\\uFE0E)?(?:\\u200D(?:\\p{Extended_Pictographic}|\\w)+)*)","gu")}catch{u=/[\u{2700}-\u{27BF}\u{1F300}-\u{1FAFF}\u{1F3FB}-\u{1F3FF}\u200D\uFE0F\u{1F1E6}-\u{1F1FF}]/gu}
-var x=/\r\n|\r|\n/g,y=/[ \t]*(\n+)[ \t]*/g,B=/\n{2,}/g,A=/\n{3,}/g,C=/ {2,}/g,v=/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F\u0080-\u009F\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]+/g;function L(a,c){a.lastIndex=0;return a.test(c)}
-function M(a){if("string"!==typeof a)throw new TypeError("inspectText expects a string input.");const c=[],d={hasControlChars:!1,hasInvisibleChars:!1,hasMixedNewlines:!1,newlineStyle:null,hasEmojis:!1,hasNonKeyboardChars:!1,summary:c};L(v,a)&&(d.hasControlChars=!0,c.push("Control characters detected."));L(t,a)&&(d.hasInvisibleChars=!0,c.push("Invisible Unicode characters detected."));L(u,a)&&(d.hasEmojis=!0,c.push("Emojis detected."));if("string"!==typeof a)throw new TypeError("getNewlineStats expects a string input.");
-var f=0,g=0;let l=0;for(let k=0,m=a.length;k<m;k++){const p=a.charCodeAt(k);13===p?10===a.charCodeAt(k+1)?(f++,k++):g++:10===p&&l++}f={i:f,h:g,j:l};g=[];0<f.i&&g.push("CRLF");0<f.h&&g.push("CR");0<f.j&&g.push("LF");const {l:h,types:b}={...f,types:g,l:1<g.length};d.hasMixedNewlines=h;d.newlineStyle=h?"Mixed":b[0]||null;d.newlineStyle&&c.push(h?"Mixed newline styles detected.":`Consistent newline style: ${d.newlineStyle}`);a:{a=E(a);if(L(D,a))for(e of a)if(!("\n"===e||"\r"===e||" "<=e&&"~">=e||L(u,
-e))){var e=!0;break a}e=!1}e&&(d.hasNonKeyboardChars=!0,c.push("Non-keyboard characters detected."));return d}
-function N(a,c={}){if("string"!==typeof a)throw new TypeError("inspectTextDetailed expects a string input.");c=c&&c.maxFindings;c=Infinity===c||Number.isInteger(c)&&0<=c?c:100;const d={bidi_control:0,zero_width:0,nonbreaking_space:0,bom:0,nul:0,c0_control:0,c1_control:0},f=[];let g=0,l=1,h=0;for(let e=0,k=a.length;e<k;e++){var b=a.charCodeAt(e);if(32<=b&&127>b)continue;if(10===b){l++;h=e+1;continue}if(13===b){10!==a.charCodeAt(e+1)&&(l++,h=e+1);continue}const m=128>b?0===b?"nul":32>b?9===b||10===
-b||13===b?null:"c0_control":127===b?"c0_control":null:159>=b?"c1_control":160===b?"nonbreaking_space":1564===b?"bidi_control":8192>b?null:8202>=b?"nonbreaking_space":8205>=b?"zero_width":8207>=b||8234<=b&&8238>=b?"bidi_control":8239===b?"nonbreaking_space":8288===b?"zero_width":8294<=b&&8297>=b?"bidi_control":12288===b?"nonbreaking_space":65279===b?"bom":null;null!==m&&(d[m]++,g++,f.length<c&&(b={offset:e,line:l,column:e-h+1,codePoint:b,category:m},"bom"===m&&(b.atStart=0===e),f.push(b)))}return{findings:f,
-counts:d,total:g,truncated:g>f.length}}function O(a){return{purgeInvisibleChars:a.hasInvisibleChars,purgeEmojis:a.hasEmojis,nukeControls:a.hasControlChars,keyboardOnlyFilter:a.hasNonKeyboardChars,normalizeNewlines:a.hasMixedNewlines||"CRLF"===a.newlineStyle||"CR"===a.newlineStyle}}export { n as summonSanctifier, M as inspectText, O as getRecommendedSanctifierOptions, N as inspectTextDetailed };
+function r(a,c=!1,e=!1,f=!1,h=!1,m=!1,l=!1,b=!1,d=!1,g=!1,p=!1,k=Infinity,q="throw"){if("string"!==typeof a)throw new TypeError("sanctifyText expects a string input.");"truncate"!==q&&"noop"!==q&&"throw"!==q&&(q="throw");if(a.length>k)switch(q){case "truncate":a=a.slice(0,k);break;case "noop":return a;default:throw new RangeError(`sanctifyText input length ${a.length} exceeds maxLength ${k}.`);}c&&(a=a.replace(t,""));e&&(a=a.replace(u,""));f&&(a=a.replace(v,""));h&&(a=w(a,e));m&&(a=a.replace(x,"\n"));
+l&&(a=a.replace(y,"$1"));b&&(c=a,a=d?c.replace(A,"\n\n"):c.replace(B,"\n"));g&&(a=a.replace(C," "));return p?a.trim():a}var t=/[\u00a0\u00ad\u034f\u061c\u115f\u1160\u180e\u2000-\u200f\u202a-\u202f\u2060-\u2064\u2066-\u2069\u3000\u3164\ufe00-\ufe0f\ufeff\uffa0\u{E0000}-\u{E007F}\u{E0100}-\u{E01EF}]+/gu,D=/[^\x20-\x7E\r\n]+/gu;
+function w(a,c=!1){a=E(a);return c?a.replace(D,""):a.replace(D,e=>{u.lastIndex=0;if(u.test(e)){u.lastIndex=0;var f="";for(const h of e.matchAll(u))f+=h[0];e=f}else e="";return e})}var F=/[\u2018\u2019\u201A\u201B\u2032\u2035]/g,G=/[\u201C\u201D\u201E\u201F\u2033\u2036\u00AB\u00BB]/g,H=/[\u2012\u2013\u2014\u2015\u2212]/g,I=/\u2026/g,J=/[\u2022\u00B7]/g,K=/[\uFF01-\uFF5E]/g;
+function E(a){return a.replace(F,"'").replace(G,'"').replace(H,"-").replace(I,"...").replace(J,"*").replace(K,c=>String.fromCharCode(c.charCodeAt(0)-65248))}var u;try{u=RegExp("(?:\\p{Extended_Pictographic}(?:\\uFE0F|\\uFE0E)?(?:\\u200D(?:\\p{Extended_Pictographic}|\\w)+)*)","gu")}catch{u=/[\u{2700}-\u{27BF}\u{1F300}-\u{1FAFF}\u{1F3FB}-\u{1F3FF}\u200D\uFE0F\u{1F1E6}-\u{1F1FF}]/gu}var x=/\r\n|\r|\n/g,y=/[ \t]*(\n+)[ \t]*/g,B=/\n{2,}/g,A=/\n{3,}/g,C=/ {2,}/g,v=/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F\u0080-\u009F\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]+/g;
+function L(a,c){a.lastIndex=0;return a.test(c)}
+function M(a){if("string"!==typeof a)throw new TypeError("inspectText expects a string input.");const c=[],e={hasControlChars:!1,hasInvisibleChars:!1,hasMixedNewlines:!1,newlineStyle:null,hasEmojis:!1,hasNonKeyboardChars:!1,summary:c};L(v,a)&&(e.hasControlChars=!0,c.push("Control characters detected."));L(t,a)&&(e.hasInvisibleChars=!0,c.push("Invisible Unicode characters detected."));L(u,a)&&(e.hasEmojis=!0,c.push("Emojis detected."));if("string"!==typeof a)throw new TypeError("getNewlineStats expects a string input.");var f=
+0,h=0;let m=0;for(let g=0,p=a.length;g<p;g++){const k=a.charCodeAt(g);13===k?10===a.charCodeAt(g+1)?(f++,g++):h++:10===k&&m++}f={i:f,h,j:m};h=[];0<f.i&&h.push("CRLF");0<f.h&&h.push("CR");0<f.j&&h.push("LF");const {l,types:b}={...f,types:h,l:1<h.length};e.hasMixedNewlines=l;e.newlineStyle=l?"Mixed":b[0]||null;e.newlineStyle&&c.push(l?"Mixed newline styles detected.":`Consistent newline style: ${e.newlineStyle}`);a:{a=E(a);if(L(D,a))for(d of a)if(!("\n"===d||"\r"===d||" "<=d&&"~">=d||L(u,d))){var d=
+!0;break a}d=!1}d&&(e.hasNonKeyboardChars=!0,c.push("Non-keyboard characters detected."));return e}
+function N(a,c={}){if("string"!==typeof a)throw new TypeError("inspectTextDetailed expects a string input.");c=c&&c.maxFindings;c=Infinity===c||Number.isInteger(c)&&0<=c?c:100;const e={bidi_control:0,zero_width:0,nonbreaking_space:0,bom:0,nul:0,c0_control:0,c1_control:0,invisible_format:0,hangul_filler:0,variation_selector:0,tag:0},f=[];let h=0,m=1,l=0;for(let g=0,p=a.length;g<p;g++){var b=a.charCodeAt(g);if(32<=b&&127>b)continue;if(10===b){m++;l=g+1;continue}if(13===b){10!==a.charCodeAt(g+1)&&(m++,
+l=g+1);continue}const k=g;var d=b;if(56128===b){d=a.charCodeAt(g+1);if(!(56320<=d&&57343>=d))continue;d=65536+(b-55296<<10)+(d-56320);b=917504<=d&&917631>=d?"tag":917760<=d&&917999>=d?"variation_selector":null;g++}else b=128>b?0===b?"nul":32>b?9===b||10===b||13===b?null:"c0_control":127===b?"c0_control":null:159>=b?"c1_control":160===b?"nonbreaking_space":173===b||847===b||6158===b?"invisible_format":1564===b?"bidi_control":4447===b||4448===b?"hangul_filler":8192>b?null:8202>=b?"nonbreaking_space":
+8205>=b?"zero_width":8207>=b||8234<=b&&8238>=b?"bidi_control":8239===b?"nonbreaking_space":8288===b?"zero_width":8289<=b&&8292>=b?"invisible_format":8294<=b&&8297>=b?"bidi_control":12288===b?"nonbreaking_space":12644===b||65440===b?"hangul_filler":65024<=b&&65039>=b?"variation_selector":65279===b?"bom":null;null!==b&&(e[b]++,h++,f.length<c&&(d={offset:k,line:m,column:k-l+1,codePoint:d,category:b},"bom"===b&&(d.atStart=0===k),f.push(d)))}return{findings:f,counts:e,total:h,truncated:h>f.length}}
+function O(a){return{purgeInvisibleChars:a.hasInvisibleChars,purgeEmojis:a.hasEmojis,nukeControls:a.hasControlChars,keyboardOnlyFilter:a.hasNonKeyboardChars,normalizeNewlines:a.hasMixedNewlines||"CRLF"===a.newlineStyle||"CR"===a.newlineStyle}}export { n as summonSanctifier, M as inspectText, O as getRecommendedSanctifierOptions, N as inspectTextDetailed };