text-sanctifier 1.3.10 → 1.3.12

package/README.md

@@ -106,20 +106,22 @@ Used by `summonSanctifier(options)`. All boolean options default to `false`. Pro
 | `collapseSpaces` | Collapse runs of ordinary spaces to one; keep tabs and newlines. |
 | `finalTrim` | Apply JavaScript `.trim()` to the result. |
 | `maxLength` | Input cap in UTF-16 code units, checked before processing. Default `Infinity`; negative, non-finite or non-number values mean no cap. |
-| `onMaxLength` | Default `'throw'`: throw `RangeError` above the cap. `'truncate'`: slice input to the cap before processing (may split a surrogate pair). `'noop'`: return original input unchanged. Invalid values use `'throw'`. |
+| `onMaxLength` | Default `'throw'`: throw `RangeError` above the cap. `'truncate'`: slice input to at most the cap before processing, never through a surrogate pair. `'noop'`: return original input unchanged. Invalid values use `'throw'`. |
 
 ```js
 import { summonSanctifier } from 'text-sanctifier';
 const clean = summonSanctifier({ purgeInvisibleChars: true, maxLength: 10000, onMaxLength: 'throw' });
 ```
 
+Every sanitizer replaces unpaired surrogates with U+FFFD before anything else, so removing a character between two halves cannot join them into a new one.
+
 ### Errors
 
 All text-processing exports throw `TypeError` on non-string input, including the preset functions, the sanitizer returned by `summonSanctifier`, `findHiddenRuns` and `decodeTagText`. Inputs are not coerced to strings. Sanitizers also throw `RangeError` when input exceeds `maxLength` with `onMaxLength: 'throw'`.
 
 ### Presets as standalone imports
 
-Each preset is also its own export: `sanctifyStrict`, `sanctifyLoose`, `sanctifyKeyboardOnly`, `sanctifyKeyboardOnlyEmoji` and `sanctifyPasted`. They are the same functions as `summonSanctifier.strict` and the rest. Import one directly and a bundler leaves out `summonSanctifier` and the other presets (about <!-- gen:size:sanctifyPasted -->1.1 KiB<!-- /gen --> gzipped instead of <!-- gen:size:summonSanctifier.pasted -->1.4 KiB<!-- /gen -->).
+Each preset is also its own export: `sanctifyStrict`, `sanctifyLoose`, `sanctifyKeyboardOnly`, `sanctifyKeyboardOnlyEmoji` and `sanctifyPasted`. They are the same functions as `summonSanctifier.strict` and the rest. Import one directly and a bundler leaves out `summonSanctifier` and the other presets (about <!-- gen:size:sanctifyPasted -->1.1 KiB<!-- /gen --> gzipped instead of <!-- gen:size:summonSanctifier.pasted -->1.5 KiB<!-- /gen -->).
 
 ```js
 import { sanctifyPasted } from 'text-sanctifier';
@@ -187,7 +189,7 @@ const report = inspectText(input);
 */
 ```
 
-Use `inspectText` to preflight text content before rendering, storing, or linting. It's a diagnostic tool to help inform sanitization needs. Pass the report to `getRecommendedSanctifierOptions(report)` to auto-generate config flags for `summonSanctifier()`.
+Use `inspectText` to preflight text content before rendering, storing, or linting. It's a diagnostic tool to help inform sanitization needs. Pass the report to `getRecommendedSanctifierOptions(report)` to auto-generate config flags for `summonSanctifier()`. It turns on what removes trash without losing meaning; `purgeEmojis` and `keyboardOnlyFilter` delete content, so you opt in to those yourself.
 
 ### `inspectTextDetailed(text: string, options?: { maxFindings?: number }): DetailedTrashReport`
 
@@ -299,6 +301,7 @@ The package ships a `text-sanctifier` command for CI and coding agents, which ca
 ```sh
 npx text-sanctifier --help
 npx text-sanctifier --version
+npx text-sanctifier check
 npx text-sanctifier check src docs "**/*.md"
 npx text-sanctifier fix --dry-run src
 npx text-sanctifier reveal notes.md
@@ -314,7 +317,8 @@ src/auth.js:40:3  U+202E  bidi_control
 2 findings in 2 of 31 files.
 ```
 
-- `--json` (or `--format json`): one JSON object on stdout, `{ files, total, filesScanned, filesWithFindings, skipped, notUtf8 }`. Each entry in `files` is `{ path, ...inspectTextDetailed(text) }`; only files with findings are listed.
+- `--json` (or `--format json`): one JSON object on stdout, `{ files, total, filesScanned, filesWithFindings, skipped, excluded, notUtf8, errors }`. Each entry in `files` is `{ path, ...inspectTextDetailed(text) }`; only files with findings are listed. Usage errors also produce JSON on stdout.
+- `--exclude GLOB`: exclude matching paths; repeatable. See [Inputs](#inputs). Not available with `--ai-config`.
 - `--format sarif`: a [SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html) log for code-scanning viewers and editors. See [SARIF output](#sarif-output).
 - `--max-findings N`: findings listed per file (default 100; `all` for no cap). Counts always cover every occurrence.
 - `--strip-bom`: also report a BOM at offset 0, which is skipped by default.
@@ -351,7 +355,7 @@ skills/review/SKILL.md:2:3  U+FE00  run of 3 variation selectors
 - Each location has the file (relative to `%SRCROOT%`, the current directory), `startLine`, `startColumn` and `endColumn`. Columns count UTF-16 code units, as in `inspectTextDetailed` (`columnKind: "utf16CodeUnits"`).
 - Each result has a `partialFingerprints` hash based on its rule, relative path, line content and occurrence on that line. Adding unrelated lines above does not change it.
 - With `--ai-config`, the message of the first result in a run describes the run and its decoded text.
-- `--max-findings` caps results per file as usual. `runs[0].properties` holds `total`, `filesScanned`, `filesWithFindings`, `truncated` (files with findings left out) and `skipped`.
+- `--max-findings` caps results per file as usual. `runs[0].properties` holds `total`, `filesScanned`, `filesWithFindings`, `truncated` (files with findings left out), `skipped` and `excluded`.
 
 ### pre-commit
 
@@ -361,13 +365,13 @@ The repository publishes a [pre-commit](https://pre-commit.com) hook manifest, s
 # .pre-commit-config.yaml
 repos:
   - repo: https://github.com/iWhatty/text-sanctifier
-    rev: v1.3.10  # a release tag
+    rev: v1.3.12  # a release tag
     hooks:
       - id: text-sanctifier            # every staged text file
       - id: text-sanctifier-ai-config  # agent instruction files, with --ai-config
 ```
 
-- `text-sanctifier` runs `text-sanctifier check` on staged text files. Use `exclude:` for files that legitimately contain these characters, or `args: [--strip-bom]` and the other `check` options.
+- `text-sanctifier` runs `text-sanctifier check` on staged text files, as pre-commit defines them: it can drop a file with control bytes near its start before the hook sees it, so scan the repository with `check` as well if that matters to you. Use `exclude:` for files that legitimately contain these characters, or `args: [--strip-bom]` and the other `check` options.
 - `text-sanctifier-ai-config` runs `text-sanctifier check --ai-config` on staged files that match the `--ai-config` file list.
 - A hook fails the commit when it finds something (exit code 1). Neither hook changes files; run `text-sanctifier fix` yourself.
 
@@ -383,7 +387,8 @@ Rewrites files in place and reports what it replaced and removed.
 - With none of these, `fix` uses `--normalize-spaces --purge-invisible --nuke-controls`: look-alike spaces become ordinary spaces, and everything else `check` reports is removed.
 - `--strip-bom`: also remove a BOM at offset 0. It is kept by default.
 - `--dry-run`: write nothing; report what would change.
-- `--json` (or `--format json`): `{ files, dryRun, filesScanned, filesChanged, skipped }`, where each file is `{ path, spacesReplaced, removed, removedTotal, newlinesNormalized }`.
+- `--exclude GLOB`: exclude matching paths; repeatable. See [Inputs](#inputs).
+- `--json` (or `--format json`): `{ files, dryRun, filesScanned, filesChanged, skipped, excluded, errors }`, where each file is `{ path, spacesReplaced, removed, removedTotal, newlinesNormalized }`. Usage errors also produce JSON on stdout.
 
 Apart from look-alike spaces, `fix` deletes characters. By default, emoji lose their zero width joiners and variation selectors, and so does Mongolian text. Use `--keep-benign` to preserve recognized legitimate sequences, or run `check` or `fix --dry-run` first on prose. To make hidden characters visible without changing a file, use `reveal`.
 
@@ -399,10 +404,14 @@ git diff | npx text-sanctifier reveal
 ### Inputs
 
 - Paths can be files, directories or globs (`*`, `**`, `?`, `[...]`, `{a,b}`). Quote globs so the shell passes them through.
-- Inside a git work tree, directories and globs list files with `git ls-files`, so anything `.gitignore` excludes is skipped. Outside git, `.git` and `node_modules` are skipped. A file named explicitly is always read.
-- Binary files (a NUL byte in the first 8000 bytes) are skipped and listed under `skipped` in JSON. Stdin and AI configuration files are never treated as binary: a NUL in them is reported as `nul`.
+- Inside a git work tree, directories and globs list files with `git ls-files`, so anything `.gitignore` excludes is skipped. Outside git, `.git` and `node_modules` are skipped. A file named explicitly is always read, unless `--exclude` matches it.
+- `--exclude GLOB` matches paths relative to the current directory, or any parent directory. Leading `./` and trailing `/` are ignored. `*` does not cross `/`: `*.tar` matches only the top level, while `**/*.tar` matches any depth. Excluded paths are listed in JSON and SARIF and counted in text summaries. Stdin is never excluded. There is no config file, ignore file or inline comment form; scanned text cannot silence the scanner.
+- Binary files are skipped, counted in the summary and listed under `skipped` in JSON. A file counts as binary when its first 8000 bytes hold a NUL and more than 5% control bytes, and it is either invalid UTF-8 or entirely ASCII with at least 8 control bytes. Any non-ASCII byte in valid UTF-8 keeps the file checked, even beyond the sample. A stray NUL or bad byte does not hide a text file. Stdin and AI configuration files are never treated as binary. `fix` is more careful, because it rewrites files: in a directory or glob it also skips any file with a NUL in those bytes, and it cleans such a file only if you name it.
+- UTF-16 files with a BOM are decoded and checked. `fix` never rewrites them and lists them under `skipped`.
 - Files that are not valid UTF-8 are still checked, with each invalid byte read as U+FFFD, so one bad byte cannot hide the rest of the file. `check` notes them on stderr and lists them under `notUtf8` in JSON; `fix` never writes them and lists them under `skipped`.
-- `-`, or no paths with piped input, reads stdin (with `--ai-config`, only `-` does). `fix` writes the cleaned text to stdout and the report to stderr.
+- `-` reads stdin. With no paths, `check` reads piped input, and scans the current directory when stdin is a terminal, the null device or empty. `check --ai-config` always searches the current directory with no paths. `fix` and `reveal` read piped input. `fix` writes the cleaned text to stdout and the report to stderr.
+- A path or glob that matches no files, or a run that checks nothing, exits 2 instead of reporting a clean result. So does `fix` when it refuses stdin or a file you named. Files it skips in a directory or glob don't fail the run.
+- JSON `errors` entries have `path` (or null), `code` and `message`; read and write failures may also include a system `cause`. Codes are `usage`, `not_found`, `no_match`, `read_failed`, `decode_failed`, `write_failed`, `refused` and `nothing_checked`. Exit 2 means `errors` is nonempty.
 
 ### Exit codes
 
@@ -410,7 +419,7 @@ git diff | npx text-sanctifier reveal
 | --- | --- | --- | --- |
 | 0 | nothing found | done (or nothing to do) | done |
 | 1 | findings | `--dry-run` with changes pending | not used |
-| 2 | usage error, missing path or unreadable file | same | same |
+| 2 | usage error, missing or unreadable input, a path matching no files, or nothing checked | same, or an input you named that it refused to rewrite | same as `check` |
 
 ---
 

package/bin/text-sanctifier.js

@@ -10,7 +10,7 @@
 // Plain Node (>=18), no dependencies. Exit codes: 0 clean, 1 findings (check,
 // or fix --dry-run with changes pending), 2 usage or I/O error.
 
-import { readFileSync, writeFileSync, statSync, readdirSync, existsSync } from 'node:fs';
+import { readFileSync, writeFileSync, statSync, fstatSync, readdirSync } from 'node:fs';
 import { resolve, join, sep, relative } from 'node:path';
 import { spawnSync } from 'node:child_process';
 import { pathToFileURL } from 'node:url';
@@ -22,6 +22,10 @@ import { scanHiddenSpans } from '../src/hiddenScan.js';
 import { inspectTextInContext } from '../src/context.js';
 import { classifyCodeUnit, classifyAstral } from '../src/categories.generated.js';
 
+for (const stream of [process.stdout, process.stderr]) {
+    stream.on('error', (e) => { if (e.code !== 'EPIPE') throw e; });
+}
+
 const VERSION = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version;
 
 const USAGE = `text-sanctifier ${VERSION}
@@ -32,12 +36,17 @@ Usage:
   text-sanctifier reveal [paths...]
 
 Paths may be files, directories or globs (quote globs so the shell leaves them
-alone). "-", or no paths with piped input, reads stdin. Directories and globs
-skip files ignored by git (inside a git work tree) and skip binary files.
+alone). "-" reads stdin. With no paths, check reads piped input, or scans the current
+directory when stdin is a terminal, the null device or empty. Fix and reveal read
+piped input. Directories and globs skip files ignored by git (inside
+a git work tree). Binary files are skipped; files that are text apart from a stray
+NUL or bad byte are not, and UTF-16 files with a BOM are decoded. A path or glob that
+matches no files, or a run that checks nothing, exits 2.
 
 check options:
   --format F           text (default), json or sarif (SARIF 2.1.0)
   --json               same as --format json
+  --exclude GLOB       exclude matching paths (repeatable)
   --max-findings N     findings listed per file (default 100, "all" for no cap)
   --strip-bom          also report a BOM at the start of a file
   --ai-config          check agent instruction files (CLAUDE.md, AGENTS.md,
@@ -54,6 +63,7 @@ fix options:
   --normalize-newlines convert CRLF and CR to LF
   --strip-bom          also remove a BOM at the start of a file (kept by default)
   --dry-run            report what would change; write nothing
+  --exclude GLOB       exclude matching paths (repeatable)
   --format F           text (default) or json
   --json               same as --format json
   With no cleaning flag, fix uses --normalize-spaces --purge-invisible --nuke-controls.
@@ -77,6 +87,8 @@ const BOM = 0xFEFF;
 const STDIN = '-';
 const STDIN_LABEL = '<stdin>';
 const BINARY_SNIFF_BYTES = 8000;
+const BINARY_CONTROL_SHARE = 0.05;
+const BINARY_MIN_ASCII_CONTROLS = 8;
 const WALK_SKIP_DIRS = new Set(['.git', 'node_modules']);
 const CASE_INSENSITIVE_FS = process.platform === 'win32' || process.platform === 'darwin';
 
@@ -106,6 +118,7 @@ function parseArgs(argv) {
     const opts = {
         command: null,
         paths: [],
+        excludes: [],
         format: null,
         aiConfig: false,
         stripBom: false,
@@ -134,6 +147,14 @@ function parseArgs(argv) {
         if (arg === '-v' || arg === '--version') { opts.version = true; continue; }
 
         const [name, inline] = arg.split(/=(.*)/s, 2);
+        if (name === '--exclude') {
+            const value = inline !== undefined ? inline : argv[++i];
+            if (!value || value.startsWith('-')) throw new UsageError('--exclude expects a non-empty glob.');
+            try { excludeRegExp(value); }
+            catch (e) { if (e instanceof SyntaxError) throw new UsageError(`Invalid --exclude pattern: ${value}`); throw e; }
+            opts.excludes.push(value);
+            continue;
+        }
         if (name === '--max-findings') {
             const value = inline !== undefined ? inline : argv[++i];
             opts.maxFindings = parseMaxFindings(value);
@@ -160,6 +181,7 @@ function parseArgs(argv) {
         throw new UsageError(opts.command === null ? 'Missing command.' : `Unknown command: ${opts.command}`);
     }
     if (opts.command === 'reveal') {
+        if (opts.excludes.length) throw new UsageError('--exclude does not apply to reveal.');
         for (const [flag, key] of Object.entries(BOOLEAN_FLAGS)) {
             if (opts[key]) throw new UsageError(`${flag} does not apply to reveal.`);
         }
@@ -171,6 +193,7 @@ function parseArgs(argv) {
         if (opts.command === 'check' && FIX_ONLY_FLAGS.has(key) && opts[key]) throw new UsageError(`${flag} only applies to fix.`);
         if (opts.command === 'fix' && CHECK_ONLY_FLAGS.has(key) && opts[key]) throw new UsageError(`${flag} only applies to check.`);
     }
+    if (opts.aiConfig && opts.excludes.length) throw new UsageError('--exclude does not apply to --ai-config.');
     if (opts.format === null) opts.format = 'text';
     if (opts.command === 'fix' && opts.format === 'sarif') throw new UsageError('--format sarif only applies to check.');
     if (opts.command === 'fix' && !opts.normalizeSpaces && !opts.purgeInvisible && !opts.nukeControls && !opts.normalizeNewlines) {
@@ -206,21 +229,36 @@ function isGitWorkTree(cwd) {
 }
 
 /**
- * Lists tracked and untracked-but-not-ignored files under one pathspec.
+ * Lists tracked and untracked-but-not-ignored files under the given pathspecs.
  * Returns null when git cannot answer (for example a path outside the work tree).
  * @param {string} cwd
- * @param {string} pathspec
+ * @param {string[]} pathspecs
  * @returns {string[]|null}
  */
-function gitListFiles(cwd, pathspec) {
+function gitListFiles(cwd, pathspecs) {
     const r = spawnSync(
         'git',
-        ['ls-files', '-z', '--cached', '--others', '--exclude-standard', '--', pathspec],
+        ['ls-files', '-z', '--cached', '--others', '--exclude-standard', '--', ...pathspecs],
         { cwd, encoding: 'utf8', maxBuffer: 1 << 28 }
     );
     if (r.error || r.status !== 0) return null;
-    // --cached lists tracked files deleted from the work tree; drop them
-    return r.stdout.split('\0').filter((f) => f && existsSync(resolve(cwd, f)));
+    // --cached also lists tracked files deleted from the work tree, submodules and
+    // symlinks to directories; keep only what can be read as a file
+    return r.stdout.split('\0').filter((f) => {
+        if (!f) return false;
+        try { return statSync(resolve(cwd, f)).isFile(); } catch { return false; }
+    });
+}
+
+/**
+ * Expands `{a,b}` into one glob per alternative: git pathspecs have no brace syntax.
+ * @param {string} glob
+ * @returns {string[]}
+ */
+function expandBraces(glob) {
+    const m = /\{([^{}]*)\}/.exec(glob);
+    if (!m) return [glob];
+    return m[1].split(',').flatMap((alt) => expandBraces(glob.slice(0, m.index) + alt + glob.slice(m.index + m[0].length)));
 }
 
 /**
@@ -257,7 +295,7 @@ function walk(cwd, dir) {
  * Glob to RegExp for the non-git fallback: `**`, `*`, `?`, `[...]` and `{a,b}`.
  * @param {string} glob
  */
-function globToRegExp(glob) {
+function globToRegExp(glob, flags = '') {
     let re = '';
     for (let i = 0; i < glob.length; i++) {
         const c = glob[i];
@@ -288,9 +326,12 @@ function globToRegExp(glob) {
             re += c.replace(/[.+^$()|\\\]}]/g, '\\$&');
         }
     }
-    return new RegExp(`^${re}$`);
+    return new RegExp(`^${re}$`, flags);
 }
 
+/** @param {string} glob - an --exclude pattern */
+const excludeRegExp = (glob) => globToRegExp(toSlash(glob).replace(/^\.\//, '').replace(/\/$/, ''), CASE_INSENSITIVE_FS ? 'i' : '');
+
 /**
  * @param {string} cwd
  * @param {string} glob
@@ -331,14 +372,13 @@ const isAiConfigPath = (rel) => AI_CONFIG_RES.some((re) => re.test(rel));
  * directories contribute only agent instruction files (AI_CONFIG_GLOBS).
  * @param {string[]} args
  * @param {string} cwd
- * @param {(msg: string) => void} warn
+ * @param {(path: string|null, code: string, message: string) => void} addError
  * @param {boolean} [aiConfig]
- * @returns {{ inputs: Array<{ path: string, explicit: boolean }>, errors: number }}
+ * @returns {Array<{ path: string, explicit: boolean }>}
  */
-function collectInputs(args, cwd, warn, aiConfig = false) {
+function collectInputs(args, cwd, addError, aiConfig = false) {
     const inputs = [];
     const seen = new Set();
-    let errors = 0;
     let inGit;
 
     const add = (path, explicit) => {
@@ -359,25 +399,24 @@ function collectInputs(args, cwd, warn, aiConfig = false) {
 
         const isGlob = !stat && GLOB_CHARS.test(arg);
         if (!stat && !isGlob) {
-            warn(`${arg}: no such file or directory`);
-            errors++;
+            addError(arg, 'not_found', `${arg}: no such file or directory`);
             continue;
         }
 
         if (inGit === undefined) inGit = isGitWorkTree(cwd);
         let listed =
-            (inGit && gitListFiles(cwd, isGlob ? `:(glob)${toSlash(arg)}` : toSlash(arg))) ||
+            (inGit && gitListFiles(cwd, isGlob ? expandBraces(toSlash(arg)).map((g) => `:(glob)${g}`) : [toSlash(arg)])) ||
             (isGlob ? walkGlob(cwd, arg) : walk(cwd, arg));
 
         if (aiConfig && !isGlob) {
             const base = resolve(cwd, arg);
             listed = listed.filter((f) => isAiConfigPath(toSlash(relative(base, resolve(cwd, f)))));
         } else if (listed.length === 0) {
-            warn(`${arg}: no files matched`);
+            addError(arg, 'no_match', `${arg}: no files matched`);
         }
         for (const f of listed) add(f, false);
     }
-    return { inputs, errors };
+    return inputs;
 }
 
 
@@ -385,24 +424,63 @@ function collectInputs(args, cwd, warn, aiConfig = false) {
 
 const utf8 = new TextDecoder('utf-8', { fatal: true, ignoreBOM: true });
 const utf8Lenient = new TextDecoder('utf-8', { ignoreBOM: true });
+const utf16 = new TextDecoder('utf-16le', { ignoreBOM: true });
 
 /**
- * Decodes UTF-8, keeping a leading BOM. With `sniff`, data with a NUL in its first
- * BINARY_SNIFF_BYTES is binary and not decoded. Bytes that are not UTF-8 become U+FFFD
- * and set `lossy`, so one bad byte cannot hide the rest of a file from `check`.
+ * A NUL plus a share of control bytes that text does not have, in the first
+ * BINARY_SNIFF_BYTES. One stray NUL is not enough, or it would hide a file from `check`.
  * @param {Uint8Array} bytes
+ * @param {number} [minControls] - fewest control bytes that count
+ */
+function looksBinary(bytes, minControls = 0) {
+    const sample = bytes.subarray(0, BINARY_SNIFF_BYTES);
+    if (!sample.includes(0)) return false;
+    let controls = 0;
+    for (const b of sample) if (b < 9 || (b > 13 && b < 32)) controls++;
+    return controls >= minControls && controls / sample.length > BINARY_CONTROL_SHARE;
+}
+
+/**
+ * Decodes UTF-8, or UTF-16 with a BOM (setting `wide`), keeping a leading BOM. With
+ * `sniff`, data that looks binary is not decoded if it is not UTF-8, or if it is entirely
+ * ASCII with at least BINARY_MIN_ASCII_CONTROLS control bytes, so a short file with a stray
+ * NUL is still read. Invisible Unicode needs non-ASCII bytes in UTF-8, so ASCII-only data has
+ * only controls to hide. Other bytes that are not UTF-8 become U+FFFD and set `lossy`, so
+ * one bad byte cannot hide the rest of a file.
+ * @param {Buffer} bytes
  * @param {boolean} sniff
- * @returns {{ text: string|null, reason?: string, lossy?: boolean }}
+ * @returns {{ text: string|null, reason?: string, lossy?: boolean, wide?: boolean }}
  */
 function decode(bytes, sniff) {
-    if (sniff && bytes.subarray(0, BINARY_SNIFF_BYTES).includes(0)) return { text: null, reason: 'binary' };
+    if (bytes[0] === 0xFF && bytes[1] === 0xFE) return { text: utf16.decode(bytes), wide: true };
+    if (bytes[0] === 0xFE && bytes[1] === 0xFF) {
+        return { text: utf16.decode(Buffer.from(bytes.subarray(0, bytes.length & ~1)).swap16()), wide: true };
+    }
     try {
-        return { text: utf8.decode(bytes) };
-    } catch {
+        const text = utf8.decode(bytes);
+        if (sniff && looksBinary(bytes, BINARY_MIN_ASCII_CONTROLS) && bytes.every((b) => b < 0x80)) return { text: null, reason: 'binary' };
+        return { text };
+    } catch (e) {
+        if (!(e instanceof TypeError)) throw e; // too large for a string, not bad UTF-8
+        if (sniff && looksBinary(bytes)) return { text: null, reason: 'binary' };
         return { text: utf8Lenient.decode(bytes), lossy: true };
     }
 }
 
+/**
+ * True when stdin may hold input for check with no paths: anything but a terminal,
+ * the null device or an empty file. Pipes must be read to detect empty input;
+ * Windows pipes do not report isFIFO().
+ */
+function stdinHasInput() {
+    try {
+        const stat = fstatSync(0);
+        return !stat.isCharacterDevice() && !(stat.isFile() && stat.size === 0);
+    } catch {
+        return false;
+    }
+}
+
 /** @returns {Promise<Buffer>} */
 function readStdin() {
     return new Promise((res, rej) => {
@@ -581,9 +659,9 @@ function sarifArtifact(label) {
 /**
  * Builds a SARIF 2.1.0 log with one result per listed finding; the rule is the category.
  * @param {Array<{ path: string, findings: Array<{ offset: number, line: number, column: number, codePoint: number, category: string }>, hiddenRuns?: Array<{ offset: number, kind: string, count: number, decoded?: string }>, truncated: boolean }>} files
- * @param {{ cwd: string, total: number, filesScanned: number, skipped: Array<{ path: string, reason: string }>, succeeded: boolean }} info
+ * @param {{ cwd: string, total: number, filesScanned: number, skipped: Array<{ path: string, reason: string }>, excluded: string[], succeeded: boolean }} info
  */
-function sarifLog(files, { cwd, total, filesScanned, skipped, succeeded }) {
+function sarifLog(files, { cwd, total, filesScanned, skipped, excluded, succeeded }) {
     const ruleIds = Object.keys(inspectTextDetailed('', { maxFindings: 0 }).counts);
     const rules = ruleIds.map((id) => ({
         id,
@@ -628,6 +706,7 @@ function sarifLog(files, { cwd, total, filesScanned, skipped, succeeded }) {
                 filesWithFindings: files.length,
                 truncated: files.filter((f) => f.truncated).map((f) => f.path),
                 skipped,
+                excluded,
             },
         }],
     };
@@ -650,28 +729,61 @@ async function main(argv) {
     } catch (e) {
         if (!(e instanceof UsageError)) throw e;
         err(`text-sanctifier: ${e.message}\nRun "text-sanctifier --help" for usage.`);
+        if (argv.some((arg, i) => arg === '--json' || arg === '--format=json' || (arg === '--format' && argv[i + 1] === 'json'))) {
+            out(JSON.stringify({ errors: [{ path: null, code: 'usage', message: e.message }] }));
+        }
         return 2;
     }
     if (opts.help) { out(USAGE); return 0; }
     if (opts.version) { out(VERSION); return 0; }
 
     const cwd = process.cwd();
+    const warn = (msg) => err(`text-sanctifier: ${msg}`);
+    const errors = [];
+    const addError = (path, code, message, cause, emit = true) => {
+        const entry = { path, code, message };
+        if (cause) entry.cause = cause;
+        errors.push(entry);
+        if (emit) warn(message);
+    };
     let paths = opts.paths;
-    // --ai-config with no paths searches the current directory even when stdin is not a
-    // terminal, so a CI job with no stdin still scans the repository; `-` reads stdin.
-    if (paths.length === 0 && opts.aiConfig) {
+    /** @type {Buffer | undefined} */
+    let stdinBytes;
+    // check with no paths reads nonempty stdin once, otherwise scans the current
+    // directory. AI configuration discovery always defaults to the directory.
+    if (paths.length === 0 && opts.command === 'check') {
         paths = ['.'];
+        if (!opts.aiConfig && stdinHasInput()) {
+            try {
+                stdinBytes = await readStdin();
+            } catch (e) {
+                addError(STDIN_LABEL, 'read_failed', `${STDIN_LABEL}: ${e.code || e.message}`, e.code);
+                if (opts.format === 'json') out(JSON.stringify({ errors }));
+                return 2;
+            }
+            if (stdinBytes.length > 0) paths = [STDIN];
+        }
     } else if (paths.length === 0) {
         if (process.stdin.isTTY) {
+            addError(null, 'usage', 'no paths given and nothing piped on stdin.', undefined, false);
             err('text-sanctifier: no paths given and nothing piped on stdin.\nRun "text-sanctifier --help" for usage.');
+            if (opts.format === 'json') out(JSON.stringify({ errors }));
             return 2;
         }
         paths = [STDIN];
     }
 
-    const warn = (msg) => err(`text-sanctifier: ${msg}`);
-    const { inputs, errors: pathErrors } = collectInputs(paths, cwd, warn, opts.aiConfig);
-    let errors = pathErrors;
+    const collected = collectInputs(paths, cwd, addError, opts.aiConfig);
+    const excluded = [];
+    const excludePatterns = opts.excludes.map(excludeRegExp);
+    const inputs = collected.filter((input) => {
+        if (input.path === STDIN) return true;
+        const path = toSlash(relative(cwd, resolve(cwd, input.path)));
+        const parts = path.split('/');
+        const matches = parts.some((_, i) => excludePatterns.some((re) => re.test(parts.slice(0, i + 1).join('/'))));
+        if (matches) excluded.push(input.path);
+        return !matches;
+    });
     /** @type {Array<{ path: string, reason: string }>} */
     const skipped = [];
     /** @type {string[]} */
@@ -682,38 +794,56 @@ async function main(argv) {
      * @param {(label: string, text: string, input: { path: string, explicit: boolean }) => void} visit
      */
     const forEachText = async (visit) => {
+        let visited = 0;
         for (const input of inputs) {
             const label = input.path === STDIN ? STDIN_LABEL : input.path;
             let bytes;
             try {
-                bytes = input.path === STDIN ? await readStdin() : readFileSync(resolve(cwd, input.path));
+                bytes = input.path === STDIN ? (stdinBytes ?? await readStdin()) : readFileSync(resolve(cwd, input.path));
             } catch (e) {
-                warn(`${label}: ${e.code || e.message}`);
-                errors++;
+                addError(label, 'read_failed', `${label}: ${e.code || e.message}`, e.code);
                 continue;
             }
             // Files that look binary are skipped. Stdin and AI configuration files are text
             // by intent, so a NUL in one is a finding, not a reason to skip it.
-            const { text, reason, lossy } = decode(bytes, input.path !== STDIN && !opts.aiConfig);
-            if (text === null) {
+            let decoded;
+            try {
+                decoded = decode(bytes, input.path !== STDIN && !opts.aiConfig);
+            } catch (e) {
+                addError(label, 'decode_failed', `${label}: ${e.message}`);
+                continue;
+            }
+            const { text, lossy, wide } = decoded;
+            let { reason } = decoded;
+            // fix writes UTF-8 in place, so it never rewrites what it could not decode or
+            // what was not UTF-8. A directory scan also leaves any file with a NUL alone;
+            // a file you name is cleaned if it is otherwise UTF-8.
+            if (opts.command === 'fix' && !reason) {
+                if (lossy) reason = 'not UTF-8';
+                else if (wide) reason = 'UTF-16';
+                else if (!input.explicit && bytes.subarray(0, BINARY_SNIFF_BYTES).includes(0)) reason = 'binary';
+            }
+            if (reason) {
                 skipped.push({ path: label, reason });
-                if (input.explicit) warn(`${label}: skipped (${reason})`);
+                if (input.explicit || reason !== 'binary') warn(`${label}: skipped (${reason})`);
+                // Asked to clean this input and did not: that must not look like success
+                if (opts.command === 'fix' && input.explicit) addError(label, 'refused', `${label}: skipped (${reason})`, undefined, false);
                 continue;
             }
             if (lossy) {
-                // fix never rewrites bytes it could not decode
-                if (opts.command === 'fix') {
-                    skipped.push({ path: label, reason: 'not UTF-8' });
-                    warn(`${label}: skipped (not UTF-8)`);
-                    continue;
-                }
                 notUtf8.push(label);
                 warn(`${label}: not valid UTF-8; checked with invalid bytes read as U+FFFD`);
             }
+            visited++;
             visit(label, text, input);
         }
+        // A clean result must mean something was read. Finding no AI configuration files is
+        // an answer; --ai-config reports that itself.
+        if (visited === 0 && errors.length === 0 && !opts.aiConfig) addError(null, 'nothing_checked', 'Nothing checked.', undefined, false);
     };
 
+    const skippedNote = () => (skipped.length > 0 ? `; ${plural(skipped.length, 'file')} skipped` : '') + (excluded.length > 0 ? `; ${plural(excluded.length, 'file')} excluded` : '');
+
     if (opts.command === 'check') {
         const files = [];
         let scanned = 0;
@@ -754,21 +884,22 @@ async function main(argv) {
 
         const kind = opts.aiConfig ? 'AI configuration file' : 'file';
         if (opts.format === 'json') {
-            const summary = { files, total, filesScanned: scanned, filesWithFindings: files.length, skipped, notUtf8 };
+            const summary = { files, total, filesScanned: scanned, filesWithFindings: files.length, skipped, excluded, notUtf8, errors };
             if (opts.aiConfig) summary.hiddenRunCount = hiddenRunCount;
             out(JSON.stringify(summary, null, 2));
         } else if (opts.format === 'sarif') {
-            out(JSON.stringify(sarifLog(files, { cwd, total, filesScanned: scanned, skipped, succeeded: errors === 0 }), null, 2));
+            out(JSON.stringify(sarifLog(files, { cwd, total, filesScanned: scanned, skipped, excluded, succeeded: errors.length === 0 }), null, 2));
         } else {
             for (const line of lines) out(line);
-            if (opts.aiConfig && scanned === 0 && skipped.length === 0) {
+            if (opts.aiConfig && scanned === 0 && skipped.length === 0 && errors.length === 0) {
                 out('No AI configuration files found.');
                 if (opts.paths.length === 0 && !process.stdin.isTTY) out('To check piped text, pass "-": text-sanctifier check --ai-config -');
             }
-            else if (total === 0) out(`No ${opts.aiConfig ? 'hidden text, ' : ''}invisible or control characters in ${plural(scanned, kind)}.`);
-            else out(`${plural(total, 'finding')}${opts.aiConfig ? ` (${plural(hiddenRunCount, 'hidden run')})` : ''} in ${files.length} of ${plural(scanned, kind)}.`);
+            else if (scanned === 0) out(`Nothing checked${skippedNote()}.`);
+            else if (total === 0) out(`No ${opts.aiConfig ? 'hidden text, ' : ''}invisible or control characters in ${plural(scanned, kind)}${skippedNote()}.`);
+            else out(`${plural(total, 'finding')}${opts.aiConfig ? ` (${plural(hiddenRunCount, 'hidden run')})` : ''} in ${files.length} of ${plural(scanned, kind)}${skippedNote()}.`);
         }
-        return errors > 0 ? 2 : total > 0 ? 1 : 0;
+        return errors.length > 0 ? 2 : total > 0 ? 1 : 0;
     }
 
     if (opts.command === 'reveal') {
@@ -781,7 +912,7 @@ async function main(argv) {
             }
             process.stdout.write(revealInvisible(text));
         });
-        return errors > 0 ? 2 : 0;
+        return errors.length > 0 ? 2 : 0;
     }
 
     // fix
@@ -839,8 +970,7 @@ async function main(argv) {
             try {
                 writeFileSync(resolve(cwd, input.path), cleaned, 'utf8');
             } catch (e) {
-                warn(`${label}: ${e.code || e.message}`);
-                errors++;
+                addError(label, 'write_failed', `${label}: ${e.code || e.message}`, e.code);
                 return;
             }
         }
@@ -848,7 +978,7 @@ async function main(argv) {
     });
 
     if (opts.format === 'json') {
-        report(JSON.stringify({ files, dryRun: opts.dryRun, filesScanned: scanned, filesChanged: files.length, skipped }, null, 2));
+        report(JSON.stringify({ files, dryRun: opts.dryRun, filesScanned: scanned, filesChanged: files.length, skipped, excluded, errors }, null, 2));
     } else {
         const verb = opts.dryRun ? 'would fix' : 'fixed';
         for (const f of files) {
@@ -858,11 +988,13 @@ async function main(argv) {
             if (f.newlinesNormalized) parts.push('normalized newlines');
             report(`${verb} ${f.path}: ${parts.join('; ')}`);
         }
-        report(files.length === 0
-            ? `Nothing to fix in ${scanned} file${scanned === 1 ? '' : 's'}.`
-            : `${opts.dryRun ? 'Would change' : 'Changed'} ${files.length} of ${scanned} file${scanned === 1 ? '' : 's'}.`);
+        report(scanned === 0
+            ? `Nothing checked${skippedNote()}.`
+            : files.length === 0
+                ? `Nothing to fix in ${plural(scanned, 'file')}${skippedNote()}.`
+                : `${opts.dryRun ? 'Would change' : 'Changed'} ${files.length} of ${plural(scanned, 'file')}${skippedNote()}.`);
     }
-    return errors > 0 ? 2 : opts.dryRun && files.length > 0 ? 1 : 0;
+    return errors.length > 0 ? 2 : opts.dryRun && files.length > 0 ? 1 : 0;
 }
 
 main(process.argv.slice(2)).then(

package/dist/text-sanctifier.min.js

@@ -1,6 +1,6 @@
-var y=/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F-\u009F\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]+/g,d=/[\u00A0\u00AD\u034F\u061C\u115F\u1160\u17B4\u17B5\u180B-\u180F\u2000-\u200F\u202A-\u202F\u205F-\u206F\u3000\u3164\uFE00-\uFE0F\uFEFF\uFFA0\uFFF0-\uFFF8\u{1BCA0}-\u{1BCA3}\u{1D173}-\u{1D17A}\u{E0000}-\u{E0FFF}]+/gu;function A(e){return e<8192?D(e):v(e)}function D(e){return e<174?e<32?e<11?e<1?"nul":e<9?"c0_control":null:e<13?"c0_control":e<14?null:"c0_control":e<160?e<127?null:e<128?"c0_control":"c1_control":e<161?"nonbreaking_space":e<173?null:"invisible_format":e<4449?e<1564?e<847?null:e<848?"invisible_format":null:e<1565?"bidi_control":e<4447?null:"hangul_filler":e<6155?e<6068?null:e<6070?"invisible_format":null:e<6159?e<6158?"variation_selector":"invisible_format":e<6160?"variation_selector":null}function v(e){return e<8304?e<8240?e<8208?e<8203?"nonbreaking_space":e<8206?"zero_width":"bidi_control":e<8234?null:e<8239?"bidi_control":"nonbreaking_space":e<8289?e<8287?null:e<8288?"nonbreaking_space":"zero_width":e<8294?"invisible_format":e<8298?"bidi_control":"invisible_format":e<65040?e<12644?e<12288?null:e<12289?"nonbreaking_space":null:e<12645?"hangul_filler":e<65024?null:"variation_selector":e<65440?e<65279?null:e<65280?"bom":null:e<65520?e<65441?"hangul_filler":null:e<65529?"invisible_format":null}function I(e){return e<917504?e<113828?e<113824?null:"invisible_format":e<119155?null:e<119163?"invisible_format":null:e<917760?e<917632?"tag":"invisible_format":e<918e3?"variation_selector":e<921600?"invisible_format":null}var g=/[^\x20-\x7E\r\n]+/gu,B=/[\u2018\u2019\u201A\u201B\u2032\u2035]/g,G=/[\u201C\u201D\u201E\u201F\u2033\u2036\u00AB\u00BB]/g,X=/[\u2012\u2013\u2014\u2015\u2212]/g,k=/\u2026/g,O=/[\u2022\u00B7]/g,P=/[\uFF01-\uFF5E]/g;function S(e){return e.replace(B,"'").replace(G,'"').replace(X,"-").replace(k,"...").replace(O,"*").replace(P,r=>String.fromCharCode(r.charCodeAt(0)-65248))}function M(){try{return new RegExp("(?:\\p{Extended_Pictographic}(?:\\uFE0F|\\uFE0E)?(?:\\u200D(?:\\p{Extended_Pictographic}|\\w)+)*)","gu")}catch{return/[\u{2700}-\u{27BF}\u{1F300}-\u{1FAFF}\u{1F3FB}-\u{1F3FF}\u200D\uFE0F\u{1F1E6}-\u{1F1FF}]/gu}}var x=M();function m(e,r){return e.lastIndex=0,e.test(r)}function U(e){let r=S(e);if(!m(g,r))return!1;for(let n of r)if(!(n===`
-`||n==="\r"||n>=" "&&n<="~")&&!m(x,n))return!0;return!1}function j(e){if(typeof e!="string")throw new TypeError("inspectText expects a string input.");let r=[],n={hasControlChars:!1,hasInvisibleChars:!1,hasMixedNewlines:!1,newlineStyle:null,hasEmojis:!1,hasNonKeyboardChars:!1,summary:r},u=(s,c,f)=>{s&&(n[c]=!0,r.push(f))};u(m(y,e),"hasControlChars","Control characters detected."),u(m(d,e),"hasInvisibleChars","Invisible Unicode characters detected."),u(m(x,e),"hasEmojis","Emojis detected.");let{mixed:o,types:i}=b(e);return n.hasMixedNewlines=o,n.newlineStyle=o?"Mixed":i[0]||null,n.newlineStyle&&r.push(o?"Mixed newline styles detected.":`Consistent newline style: ${n.newlineStyle}`),u(U(e),"hasNonKeyboardChars","Non-keyboard characters detected."),n}var z=100;function C(e,r={}){if(typeof e!="string")throw new TypeError("inspectTextDetailed expects a string input.");let n=r&&r.maxFindings,u=n===1/0||Number.isInteger(n)&&n>=0?n:z,o={bidi_control:0,zero_width:0,nonbreaking_space:0,bom:0,nul:0,c0_control:0,c1_control:0,invisible_format:0,hangul_filler:0,variation_selector:0,tag:0},i=[],s=0,c=1,f=0;for(let a=0,h=e.length;a<h;a++){let l=e.charCodeAt(a);if(l>=32&&l<127)continue;if(l===10){c++,f=a+1;continue}if(l===13){e.charCodeAt(a+1)!==10&&(c++,f=a+1);continue}let E=a,F=l,p;if(l>=55296&&l<=56319){let t=e.charCodeAt(a+1);if(!(t>=56320&&t<=57343))continue;F=65536+(l-55296<<10)+(t-56320),p=I(F),a++}else p=A(l);if(p!==null&&(o[p]++,s++,i.length<u)){let t={offset:E,line:c,column:E-f+1,codePoint:F,category:p};p==="bom"&&(t.atStart=E===0),i.push(t)}}return{findings:i,counts:o,total:s,truncated:s>i.length}}function b(e){if(typeof e!="string")throw new TypeError("getNewlineStats expects a string input.");let r=0,n=0,u=0;for(let s=0,c=e.length;s<c;s++){let f=e.charCodeAt(s);f===13?e.charCodeAt(s+1)===10?(r++,s++):n++:f===10&&u++}let o={crlf:r,cr:n,lf:u},i=[];return o.crlf>0&&i.push("CRLF"),o.cr>0&&i.push("CR"),o.lf>0&&i.push("LF"),{...o,types:i,mixed:i.length>1}}function K(e){return{purgeInvisibleChars:e.hasInvisibleChars,purgeEmojis:e.hasEmojis,nukeControls:e.hasControlChars,keyboardOnlyFilter:e.hasNonKeyboardChars,normalizeNewlines:e.hasMixedNewlines||e.newlineStyle==="CRLF"||e.newlineStyle==="CR"}}function H(e={}){let r=!!e.normalizeSpaces,n=!!e.purgeZeroWidthSpaces,u=!!e.purgeInvisibleChars,o=!!e.purgeEmojis,i=!!e.nukeControls,s=!!e.keyboardOnlyFilter,c=!!e.normalizeNewlines,f=!!e.trimSpacesAroundNewlines,a=!!e.collapseNewLines,h=!!e.preserveParagraphs,l=!!e.collapseSpaces,E=!!e.finalTrim,F=Number.isFinite(e.maxLength)&&e.maxLength>=0?e.maxLength:1/0,p=e.onMaxLength==="truncate"||e.onMaxLength==="noop"||e.onMaxLength==="throw"?e.onMaxLength:"throw";return t=>_(t,u,o,i,s,c,f,a,h,l,E,F,p,r,n)}function R(e){return _(e,!0,!0,!0,!1,!0,!0,!0,!1,!0,!0)}function w(e){return _(e,!1,!1,!1,!1,!0,!0,!0,!0,!0,!0)}function N(e){return _(e,!1,!1,!1,!0,!0,!0,!1,!1,!1,!0)}function T(e){return _(e,!0,!0,!0,!0,!0,!0,!0,!1,!0,!0)}function L(e){return _(e,!1,!1,!1,!1,!0,!1,!1,!1,!1,!1,1/0,"throw",!0,!0)}var W=Object.assign(H,{strict:R,loose:w,keyboardOnlyEmoji:N,keyboardOnly:T,pasted:L});function _(e,r=!1,n=!1,u=!1,o=!1,i=!1,s=!1,c=!1,f=!1,a=!1,h=!1,l=1/0,E="throw",F=!1,p=!1){if(typeof e!="string")throw new TypeError("sanctifyText expects a string input.");if(E!=="truncate"&&E!=="noop"&&E!=="throw"&&(E="throw"),e.length>l)switch(E){case"truncate":e=e.slice(0,l);break;case"noop":return e;case"throw":default:throw new RangeError(`sanctifyText input length ${e.length} exceeds maxLength ${l}.`)}let t=e;return F&&(t=t.replace(J," ")),p&&(t=t.replace(V,"")),r&&(t=Z(t)),n&&(t=q(t)),u&&(t=le(t)),o&&(t=Y(t,n)),i&&(t=ee(t)),s&&(t=ne(t)),c&&(t=oe(t,f)),a&&(t=se(t)),h?t.trim():t}var J=/[\u00A0\u2000-\u200A\u202F\u205F]/g,V=/[\u200B\u2060\uFEFF]/g;function Z(e){return e.replace(d,"")}function $(e){if(x.lastIndex=0,!x.test(e))return"";x.lastIndex=0;let r="";for(let n of e.matchAll(x))r+=n[0];return r}function Y(e,r=!1){let n=S(e);return r?n.replace(g,""):n.replace(g,u=>$(u))}function q(e){return e.replace(x,"")}var Q=/\r\n|\r|\n/g;function ee(e,r=`
-`){return e.replace(Q,r)}var re=/[ \t]*(\n+)[ \t]*/g;function ne(e){return e.replace(re,"$1")}var te=/\n{2,}/g,ie=/\n{3,}/g;function oe(e,r){return r?e.replace(ie,`
+var y=/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F-\u009F\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]+/g,S=/[\u00A0\u00AD\u034F\u061C\u115F\u1160\u17B4\u17B5\u180B-\u180F\u2000-\u200F\u202A-\u202F\u205F-\u206F\u3000\u3164\uFE00-\uFE0F\uFEFF\uFFA0\uFFF0-\uFFF8\u{1BCA0}-\u{1BCA3}\u{1D173}-\u{1D17A}\u{E0000}-\u{E0FFF}]+/gu;function I(e){return e<8192?v(e):B(e)}function v(e){return e<174?e<32?e<11?e<1?"nul":e<9?"c0_control":null:e<13?"c0_control":e<14?null:"c0_control":e<160?e<127?null:e<128?"c0_control":"c1_control":e<161?"nonbreaking_space":e<173?null:"invisible_format":e<4449?e<1564?e<847?null:e<848?"invisible_format":null:e<1565?"bidi_control":e<4447?null:"hangul_filler":e<6155?e<6068?null:e<6070?"invisible_format":null:e<6159?e<6158?"variation_selector":"invisible_format":e<6160?"variation_selector":null}function B(e){return e<8304?e<8240?e<8208?e<8203?"nonbreaking_space":e<8206?"zero_width":"bidi_control":e<8234?null:e<8239?"bidi_control":"nonbreaking_space":e<8289?e<8287?null:e<8288?"nonbreaking_space":"zero_width":e<8294?"invisible_format":e<8298?"bidi_control":"invisible_format":e<65040?e<12644?e<12288?null:e<12289?"nonbreaking_space":null:e<12645?"hangul_filler":e<65024?null:"variation_selector":e<65440?e<65279?null:e<65280?"bom":null:e<65520?e<65441?"hangul_filler":null:e<65529?"invisible_format":null}function b(e){return e<917504?e<113828?e<113824?null:"invisible_format":e<119155?null:e<119163?"invisible_format":null:e<917760?e<917632?"tag":"invisible_format":e<918e3?"variation_selector":e<921600?"invisible_format":null}var g=/[^\x20-\x7E\r\n]+/gu,G=/[\u2018\u2019\u201A\u201B\u2032\u2035]/g,X=/[\u201C\u201D\u201E\u201F\u2033\u2036\u00AB\u00BB]/g,M=/[\u2012\u2013\u2014\u2015\u2212]/g,O=/\u2026/g,k=/[\u2022\u00B7]/g,P=/[\uFF01-\uFF5E]/g;function d(e){return e.replace(G,"'").replace(X,'"').replace(M,"-").replace(O,"...").replace(k,"*").replace(P,r=>String.fromCharCode(r.charCodeAt(0)-65248))}function U(){try{return new RegExp("(?:\\p{Extended_Pictographic}(?:\\uFE0F|\\uFE0E)?(?:\\u200D(?:\\p{Extended_Pictographic}|\\w)+)*)","gu")}catch{return/[\u{2700}-\u{27BF}\u{1F300}-\u{1FAFF}\u{1F3FB}-\u{1F3FF}\u200D\uFE0F\u{1F1E6}-\u{1F1FF}]/gu}}var p=U();function m(e,r){return e.lastIndex=0,e.test(r)}function z(e){let r=d(e);if(!m(g,r))return!1;for(let n of r)if(!(n===`
+`||n==="\r"||n>=" "&&n<="~")&&!m(p,n))return!0;return!1}function j(e){if(typeof e!="string")throw new TypeError("inspectText expects a string input.");let r=[],n={hasControlChars:!1,hasInvisibleChars:!1,hasMixedNewlines:!1,newlineStyle:null,hasEmojis:!1,hasNonKeyboardChars:!1,summary:r},u=(l,c,f)=>{l&&(n[c]=!0,r.push(f))};u(m(y,e),"hasControlChars","Control characters detected."),u(m(S,e),"hasInvisibleChars","Invisible Unicode characters detected."),u(m(p,e),"hasEmojis","Emojis detected.");let{mixed:o,types:i}=R(e);return n.hasMixedNewlines=o,n.newlineStyle=o?"Mixed":i[0]||null,n.newlineStyle&&r.push(o?"Mixed newline styles detected.":`Consistent newline style: ${n.newlineStyle}`),u(z(e),"hasNonKeyboardChars","Non-keyboard characters detected."),n}var K=100;function A(e,r={}){if(typeof e!="string")throw new TypeError("inspectTextDetailed expects a string input.");let n=r&&r.maxFindings,u=n===1/0||Number.isInteger(n)&&n>=0?n:K,o={bidi_control:0,zero_width:0,nonbreaking_space:0,bom:0,nul:0,c0_control:0,c1_control:0,invisible_format:0,hangul_filler:0,variation_selector:0,tag:0},i=[],l=0,c=1,f=0;for(let a=0,h=e.length;a<h;a++){let s=e.charCodeAt(a);if(s>=32&&s<127)continue;if(s===10){c++,f=a+1;continue}if(s===13){e.charCodeAt(a+1)!==10&&(c++,f=a+1);continue}let x=a,F=s,E;if(s>=55296&&s<=56319){let t=e.charCodeAt(a+1);if(!(t>=56320&&t<=57343))continue;F=65536+(s-55296<<10)+(t-56320),E=b(F),a++}else E=I(s);if(E!==null&&(o[E]++,l++,i.length<u)){let t={offset:x,line:c,column:x-f+1,codePoint:F,category:E};E==="bom"&&(t.atStart=x===0),i.push(t)}}return{findings:i,counts:o,total:l,truncated:l>i.length}}function R(e){if(typeof e!="string")throw new TypeError("getNewlineStats expects a string input.");let r=0,n=0,u=0;for(let l=0,c=e.length;l<c;l++){let f=e.charCodeAt(l);f===13?e.charCodeAt(l+1)===10?(r++,l++):n++:f===10&&u++}let o={crlf:r,cr:n,lf:u},i=[];return o.crlf>0&&i.push("CRLF"),o.cr>0&&i.push("CR"),o.lf>0&&i.push("LF"),{...o,types:i,mixed:i.length>1}}function H(e){return{normalizeSpaces:e.hasInvisibleChars,purgeInvisibleChars:e.hasInvisibleChars,purgeEmojis:!1,nukeControls:e.hasControlChars,keyboardOnlyFilter:!1,normalizeNewlines:e.hasMixedNewlines||e.newlineStyle==="CRLF"||e.newlineStyle==="CR"}}function W(e={}){let r=!!e.normalizeSpaces,n=!!e.purgeZeroWidthSpaces,u=!!e.purgeInvisibleChars,o=!!e.purgeEmojis,i=!!e.nukeControls,l=!!e.keyboardOnlyFilter,c=!!e.normalizeNewlines,f=!!e.trimSpacesAroundNewlines,a=!!e.collapseNewLines,h=!!e.preserveParagraphs,s=!!e.collapseSpaces,x=!!e.finalTrim,F=Number.isFinite(e.maxLength)&&e.maxLength>=0?e.maxLength:1/0,E=e.onMaxLength==="truncate"||e.onMaxLength==="noop"||e.onMaxLength==="throw"?e.onMaxLength:"throw";return t=>_(t,u,o,i,l,c,f,a,h,s,x,F,E,r,n)}function w(e){return _(e,!0,!0,!0,!1,!0,!0,!0,!1,!0,!0)}function N(e){return _(e,!1,!1,!1,!1,!0,!0,!0,!0,!0,!0)}function T(e){return _(e,!1,!1,!1,!0,!0,!0,!1,!1,!1,!0)}function L(e){return _(e,!0,!0,!0,!0,!0,!0,!0,!1,!0,!0)}function D(e){return _(e,!1,!1,!1,!1,!0,!1,!1,!1,!1,!1,1/0,"throw",!0,!0)}var J=Object.assign(W,{strict:w,loose:N,keyboardOnlyEmoji:T,keyboardOnly:L,pasted:D});function _(e,r=!1,n=!1,u=!1,o=!1,i=!1,l=!1,c=!1,f=!1,a=!1,h=!1,s=1/0,x="throw",F=!1,E=!1){if(typeof e!="string")throw new TypeError("sanctifyText expects a string input.");if(e.length>s)switch(x){case"truncate":{let C=e.charCodeAt(s-1);e=e.slice(0,C>=55296&&C<=56319?s-1:s)}break;case"noop":return e;default:throw new RangeError(`sanctifyText input length ${e.length} exceeds maxLength ${s}.`)}let t=e.replace($,"\uFFFD");return F&&(t=t.replace(V," ")),E&&(t=t.replace(Z,"")),r&&(t=Y(t)),n&&(t=ee(t)),u&&(t=fe(t)),o&&(t=Q(t,n)),i&&(t=ne(t)),l&&(t=ie(t)),c&&(t=se(t,f)),a&&(t=ae(t)),h?t.trim():t}var V=/[\u00A0\u2000-\u200A\u202F\u205F]/g,$=/[\uD800-\uDFFF]/gu,Z=/[\u200B\u2060\uFEFF]/g;function Y(e){return e.replace(S,"")}function q(e){if(p.lastIndex=0,!p.test(e))return"";p.lastIndex=0;let r="";for(let n of e.matchAll(p))r+=n[0];return r}function Q(e,r=!1){let n=d(e);return r?n.replace(g,""):n.replace(g,u=>q(u))}function ee(e){return e.replace(p,"")}var re=/\r\n|\r|\n/g;function ne(e,r=`
+`){return e.replace(re,r)}var te=/[ \t]*(\n+)[ \t]*|([ \t]+)/g;function ie(e){return e.replace(te,"$1$2")}var oe=/\n{2,}/g,ue=/\n{3,}/g;function se(e,r){return r?e.replace(ue,`
 
-`):e.replace(te,`
-`)}var ue=/ {2,}/g;function se(e){return e.replace(ue," ")}function le(e){return e.replace(y,"")}function ae(e,r={}){if(typeof e!="string")throw new TypeError("revealInvisible expects a string input.");let n=r&&typeof r.format=="function"?r.format:i=>"<U+"+i.toString(16).toUpperCase().padStart(4,"0")+">",u="",o=0;for(let i of C(e,{maxFindings:1/0}).findings)u+=e.slice(o,i.offset)+n(i.codePoint,i.category),o=i.offset+(i.codePoint>65535?2:1);return u+e.slice(o)}var fe="1.3.10";export{fe as VERSION,b as getNewlineStats,K as getRecommendedSanctifierOptions,j as inspectText,C as inspectTextDetailed,ae as revealInvisible,T as sanctifyKeyboardOnly,N as sanctifyKeyboardOnlyEmoji,w as sanctifyLoose,L as sanctifyPasted,R as sanctifyStrict,W as summonSanctifier};
+`):e.replace(oe,`
+`)}var le=/ {2,}/g;function ae(e){return e.replace(le," ")}function fe(e){return e.replace(y,"")}function ce(e,r={}){if(typeof e!="string")throw new TypeError("revealInvisible expects a string input.");let n=r&&typeof r.format=="function"?r.format:i=>"<U+"+i.toString(16).toUpperCase().padStart(4,"0")+">",u="",o=0;for(let i of A(e,{maxFindings:1/0}).findings)u+=e.slice(o,i.offset)+n(i.codePoint,i.category),o=i.offset+(i.codePoint>65535?2:1);return u+e.slice(o)}var Ee="1.3.12";export{Ee as VERSION,R as getNewlineStats,H as getRecommendedSanctifierOptions,j as inspectText,A as inspectTextDetailed,ce as revealInvisible,L as sanctifyKeyboardOnly,T as sanctifyKeyboardOnlyEmoji,N as sanctifyLoose,D as sanctifyPasted,w as sanctifyStrict,J as summonSanctifier};

package/dist/types/inspectText.d.ts

@@ -43,7 +43,10 @@ export function inspectTextDetailed(text: string, options?: InspectDetailedOptio
  */
 export function getNewlineStats(text: string): NewlineStats;
 /**
- * Creates defaultOptions for summonSanctifier based on inspectText result
+ * Creates defaultOptions for summonSanctifier based on inspectText result.
+ * Recommends only what removes trash without losing meaning: non-ASCII text and emoji are
+ * content, so `purgeEmojis` and `keyboardOnlyFilter` stay opt-in, and look-alike spaces
+ * become ordinary spaces so that words do not run together.
  * @param {!UnicodeTrashReport} report
  * @return {!SanctifyOptions}
  */

package/dist/types/sanctifyText.d.ts

@@ -68,7 +68,7 @@ export function sanctifyPasted(text: string): string;
  * @param {number} [maxLength=Infinity] - Hard input length cap (UTF-16 code units via `text.length`).
  * @param {'throw'|'truncate'|'noop'} [onMaxLength='throw'] - Behavior when `text.length` exceeds `maxLength`:
  *   - `'throw'`: throw a `RangeError`
- *   - `'truncate'`: slice to `maxLength` before processing
+ *   - `'truncate'`: slice to at most `maxLength` before processing, never through a surrogate pair
  *   - `'noop'`: return the original input unchanged (skips processing)
  * @param {boolean} [normalizeSpaces=false] - Replace look-alike spaces with an ordinary space. Runs first,
  *   so `purgeInvisibleChars` no longer joins the words around them.

package/dist/types/version.generated.d.ts

@@ -1,2 +1,2 @@
 /** The installed version of text-sanctifier, as in its package.json. */
-export const VERSION: "1.3.10";
+export const VERSION: "1.3.12";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "text-sanctifier",
-  "version": "1.3.10",
+  "version": "1.3.12",
   "type": "module",
   "description": "Find and remove invisible Unicode in web input, model-bound text and repositories.",
   "engines": {
@@ -48,15 +48,17 @@
     "build": "node scripts/build.js",
     "build:types": "tsc -p tsconfig.types.json",
     "test:types": "tsc -p tests/types/tsconfig.json",
-    "test": "node tests/sanctifyText.test.js && node tests/inspectTextDetailed.test.js && node --expose-gc tests/hiddenText.test.js && node --expose-gc tests/context.test.js && node tests/cli.test.js && node tests/release.test.js",
+    "test": "node tests/sanctifyText.test.js && node tests/inspectTextDetailed.test.js && node --expose-gc tests/hiddenText.test.js && node --expose-gc tests/context.test.js && node tests/cli.test.js && node tests/release.test.js && node tests/names.test.js && node tests/docs.test.js && node tests/repo.test.js",
     "test-min": "node tests/sanctifyText.test.min.js",
     "check:self": "node bin/text-sanctifier.js check .",
     "check:names": "node scripts/check-names.js",
-    "ci": "npm run gen:check && npm run check:names && npm run check:self && npm run test && npm run build && npm run size && npm run build:types && npm run test:types && npm run test-min",
+    "check:repo": "node scripts/check-repo.js",
+    "ci": "npm run gen:check && npm run check:names && npm run check:repo && npm run check:self && npm run test && npm run build && npm run size && npm run build:types && npm run test:types && npm run test-min",
     "size": "node scripts/check-size.js",
     "bench": "node --expose-gc scripts/bench.js",
     "release:prep": "node scripts/release.js prep",
     "release:verify": "node scripts/release.js verify",
+    "prepublishOnly": "node scripts/release.js check-publish",
     "hooks:install": "git config core.hooksPath scripts/hooks"
   },
   "keywords": [
@@ -83,7 +85,7 @@
     "test": "tests"
   },
   "devDependencies": {
-    "esbuild": "^0.25.3",
-    "typescript": "^5.9.3"
+    "esbuild": "0.25.12",
+    "typescript": "5.9.3"
   }
 }

package/src/inspectText.js

@@ -322,16 +322,20 @@ export function getNewlineStats(text) {
 }
 
 /**
- * Creates defaultOptions for summonSanctifier based on inspectText result
+ * Creates defaultOptions for summonSanctifier based on inspectText result.
+ * Recommends only what removes trash without losing meaning: non-ASCII text and emoji are
+ * content, so `purgeEmojis` and `keyboardOnlyFilter` stay opt-in, and look-alike spaces
+ * become ordinary spaces so that words do not run together.
  * @param {!UnicodeTrashReport} report
  * @return {!SanctifyOptions}
  */
 export function getRecommendedSanctifierOptions(report) {
     return {
+        normalizeSpaces: report.hasInvisibleChars,
         purgeInvisibleChars: report.hasInvisibleChars,
-        purgeEmojis: report.hasEmojis,
+        purgeEmojis: false,
         nukeControls: report.hasControlChars,
-        keyboardOnlyFilter: report.hasNonKeyboardChars,
+        keyboardOnlyFilter: false,
         normalizeNewlines:
             report.hasMixedNewlines || report.newlineStyle === 'CRLF' || report.newlineStyle === 'CR',
     };

package/src/sanctifyText.js

@@ -294,7 +294,7 @@ export const summonSanctifier = /* @__PURE__ */ Object.assign(createSanctifier,
  * @param {number} [maxLength=Infinity] - Hard input length cap (UTF-16 code units via `text.length`).
  * @param {'throw'|'truncate'|'noop'} [onMaxLength='throw'] - Behavior when `text.length` exceeds `maxLength`:
  *   - `'throw'`: throw a `RangeError`
- *   - `'truncate'`: slice to `maxLength` before processing
+ *   - `'truncate'`: slice to at most `maxLength` before processing, never through a surrogate pair
  *   - `'noop'`: return the original input unchanged (skips processing)
  * @param {boolean} [normalizeSpaces=false] - Replace look-alike spaces with an ordinary space. Runs first,
  *   so `purgeInvisibleChars` no longer joins the words around them.
@@ -324,26 +324,27 @@ export function sanctifyText(
         throw new TypeError('sanctifyText expects a string input.');
     }
 
-    if (onMaxLength !== 'truncate' && onMaxLength !== 'noop' && onMaxLength !== 'throw') {
-        onMaxLength = 'throw';
-    }
-
     if (text.length > maxLength) {
         switch (onMaxLength) {
             case 'truncate':
-                text = text.slice(0, maxLength);
+                {
+                    // Cut before a surrogate pair, not through it
+                    const last = text.charCodeAt(maxLength - 1);
+                    text = text.slice(0, last >= 0xD800 && last <= 0xDBFF ? maxLength - 1 : maxLength);
+                }
                 break;
             case 'noop':
                 return text;
-            case 'throw':
-            default:
+            default: // 'throw', and any invalid value
                 throw new RangeError(
                     `sanctifyText input length ${text.length} exceeds maxLength ${maxLength}.`
                 );
         }
     }
 
-    let cleaned = text;
+    // Unpaired surrogates become U+FFFD first, so removing a character between two
+    // halves cannot join them into a new character, such as a hidden tag.
+    let cleaned = text.replace(LONE_SURROGATES_REGEX, '\uFFFD');
 
     // Look-alike spaces become ordinary spaces, before anything can delete them
     if (normalizeSpaces) cleaned = cleaned.replace(SPACE_LOOKALIKES_REGEX, ' ');
@@ -389,6 +390,9 @@ export function sanctifyText(
  */
 export const SPACE_LOOKALIKES_REGEX = /[\u00A0\u2000-\u200A\u202F\u205F]/g;
 
+/** With the `u` flag a surrogate pair is one character, so this matches only unpaired halves. */
+const LONE_SURROGATES_REGEX = /[\uD800-\uDFFF]/gu;
+
 /** Zero-width space, word joiner and BOM: invisible and never needed in plain text. */
 export const ZERO_WIDTH_SPACES_REGEX = /[\u200B\u2060\uFEFF]/g;
 
@@ -497,9 +501,11 @@ function normalizeNewlineChars(text, normalized = '\n') {
  * @param {string} text
  * @returns {string}
  */
-const TRIM_SPACES_AROUND_NEWLINES_REGEX = /[ \t]*(\n+)[ \t]*/g;
+// The second branch consumes a run with no newline after it, so matching resumes past the
+// run instead of retrying from each of its characters, which took quadratic time.
+const TRIM_SPACES_AROUND_NEWLINES_REGEX = /[ \t]*(\n+)[ \t]*|([ \t]+)/g;
 function trimSpacesAroundNewlineChars(text) {
-    return text.replace(TRIM_SPACES_AROUND_NEWLINES_REGEX, '$1');
+    return text.replace(TRIM_SPACES_AROUND_NEWLINES_REGEX, '$1$2');
 }
 
 

package/src/version.generated.js

@@ -2,4 +2,4 @@
 // Generated from package.json by scripts/gen-version.js (`npm run gen`); do not edit.
 
 /** The installed version of text-sanctifier, as in its package.json. */
-export const VERSION = '1.3.10';
+export const VERSION = '1.3.12';

package/src/index.d.ts

@@ -1,22 +0,0 @@
-import { inspectText } from './inspectText.js';
-import { inspectTextDetailed } from './inspectText.js';
-import { getNewlineStats } from './inspectText.js';
-import { getRecommendedSanctifierOptions } from './inspectText.js';
-import { summonSanctifier } from './sanctifyText.js';
-import { sanctifyStrict } from './sanctifyText.js';
-import { sanctifyLoose } from './sanctifyText.js';
-import { sanctifyKeyboardOnlyEmoji } from './sanctifyText.js';
-import { sanctifyKeyboardOnly } from './sanctifyText.js';
-import { sanctifyPasted } from './sanctifyText.js';
-import { revealInvisible } from './reveal.js';
-import { VERSION } from './version.generated.js';
-export { inspectText, inspectTextDetailed, getNewlineStats, getRecommendedSanctifierOptions, summonSanctifier, sanctifyStrict, sanctifyLoose, sanctifyKeyboardOnlyEmoji, sanctifyKeyboardOnly, sanctifyPasted, revealInvisible, VERSION };
-export type UnicodeTrashReport = import('./inspectText.js').UnicodeTrashReport;
-export type TrashCategory = import('./inspectText.js').TrashCategory;
-export type TrashFinding = import('./inspectText.js').TrashFinding;
-export type TrashCounts = import('./inspectText.js').TrashCounts;
-export type DetailedTrashReport = import('./inspectText.js').DetailedTrashReport;
-export type InspectDetailedOptions = import('./inspectText.js').InspectDetailedOptions;
-export type NewlineStats = import('./inspectText.js').NewlineStats;
-export type SanctifyOptions = import('./sanctifyText.js').SanctifyOptions;
-export type RevealOptions = import('./reveal.js').RevealOptions;