text-sanctifier 1.3.11 → 1.3.13

package/README.md

@@ -261,6 +261,7 @@ const { runs, benign, counts, total, truncated } = findHiddenRuns(text, { maxRun
 
 - `kind` is `tag` (U+E0000–U+E007F), `variation_selector` (U+FE00–U+FE0F, U+E0100–U+E01EF, U+180B–U+180D, U+180F) `invisible_operator` (U+2061–U+2064) or `reserved_tag_plane` (U+E0080–U+E00FF, U+E01F0–U+E0FFF: unassigned, default ignorable, so they render as nothing and can carry data). Consecutive characters of one kind form one run.
 - `offset` and `end` are UTF-16 indexes, `[offset, end)`; `count` is in code points. `line` and `column` are counted exactly as in `inspectTextDetailed` (1-based, columns in UTF-16 units, CRLF is one line break). Only tag runs have `decoded` and `decodedTruncated`.
+- `sequence` marks a lone selector in a form `text-sanctifier/context` calls benign: `'ideographic'` for U+E0100–U+E01EF after an ideograph, `'mongolian'` for U+180B–U+180D or U+180F after a Mongolian letter. These stay in `runs` because they can carry data, up to a byte per ideograph, so a run with `sequence` is usually ordinary text, but many of them can be a message. U+FE0E and U+FE0F have no such room and go under `benign`.
 - `maxRuns` caps the `runs` list and, separately, the `benign` list; `counts`, `total` and `benignTotal` count everything, and `truncated` is true when either list was cut. `maxDecodedLength` caps each `decoded` string in UTF-16 units and never cuts a `<U+XXXX>` marker. Both default to no cap. Only non-negative integers and `Infinity` are valid; invalid values (negative, fractional, a string, `NaN` or `null`) also mean no cap, so set them explicitly for untrusted input. With both set, memory stays flat on any input: on 16 M characters (32 MB) of hidden text the heap grows by under 1 MB, against hundreds of MB with no caps.
 - `decodeTagText` turns U+E0020–U+E007E into the ASCII they shadow and writes any other character as `<U+XXXX>`.
 - Zero-width, bidi and other invisible characters are not runs; use `inspectTextDetailed` for those.
@@ -317,7 +318,8 @@ src/auth.js:40:3  U+202E  bidi_control
 2 findings in 2 of 31 files.
 ```
 
-- `--json` (or `--format json`): one JSON object on stdout, `{ files, total, filesScanned, filesWithFindings, skipped, notUtf8 }`. Each entry in `files` is `{ path, ...inspectTextDetailed(text) }`; only files with findings are listed.
+- `--json` (or `--format json`): one JSON object on stdout, `{ files, total, filesScanned, filesWithFindings, skipped, excluded, notUtf8, errors }`. Each entry in `files` is `{ path, ...inspectTextDetailed(text) }`; only files with findings are listed. Usage errors also produce JSON on stdout.
+- `--exclude GLOB`: exclude matching paths; repeatable. See [Inputs](#inputs). Not available with `--ai-config`.
 - `--format sarif`: a [SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html) log for code-scanning viewers and editors. See [SARIF output](#sarif-output).
 - `--max-findings N`: findings listed per file (default 100; `all` for no cap). Counts always cover every occurrence.
 - `--strip-bom`: also report a BOM at offset 0, which is skipped by default.
@@ -354,7 +356,7 @@ skills/review/SKILL.md:2:3  U+FE00  run of 3 variation selectors
 - Each location has the file (relative to `%SRCROOT%`, the current directory), `startLine`, `startColumn` and `endColumn`. Columns count UTF-16 code units, as in `inspectTextDetailed` (`columnKind: "utf16CodeUnits"`).
 - Each result has a `partialFingerprints` hash based on its rule, relative path, line content and occurrence on that line. Adding unrelated lines above does not change it.
 - With `--ai-config`, the message of the first result in a run describes the run and its decoded text.
-- `--max-findings` caps results per file as usual. `runs[0].properties` holds `total`, `filesScanned`, `filesWithFindings`, `truncated` (files with findings left out) and `skipped`.
+- `--max-findings` caps results per file as usual. `runs[0].properties` holds `total`, `filesScanned`, `filesWithFindings`, `truncated` (files with findings left out), `skipped` and `excluded`.
 
 ### pre-commit
 
@@ -364,7 +366,7 @@ The repository publishes a [pre-commit](https://pre-commit.com) hook manifest, s
 # .pre-commit-config.yaml
 repos:
   - repo: https://github.com/iWhatty/text-sanctifier
-    rev: v1.3.11  # a release tag
+    rev: v1.3.13  # a release tag
     hooks:
       - id: text-sanctifier            # every staged text file
       - id: text-sanctifier-ai-config  # agent instruction files, with --ai-config
@@ -386,7 +388,8 @@ Rewrites files in place and reports what it replaced and removed.
 - With none of these, `fix` uses `--normalize-spaces --purge-invisible --nuke-controls`: look-alike spaces become ordinary spaces, and everything else `check` reports is removed.
 - `--strip-bom`: also remove a BOM at offset 0. It is kept by default.
 - `--dry-run`: write nothing; report what would change.
-- `--json` (or `--format json`): `{ files, dryRun, filesScanned, filesChanged, skipped }`, where each file is `{ path, spacesReplaced, removed, removedTotal, newlinesNormalized }`.
+- `--exclude GLOB`: exclude matching paths; repeatable. See [Inputs](#inputs).
+- `--json` (or `--format json`): `{ files, dryRun, filesScanned, filesChanged, skipped, excluded, errors }`, where each file is `{ path, spacesReplaced, removed, removedTotal, newlinesNormalized }`. Usage errors also produce JSON on stdout.
 
 Apart from look-alike spaces, `fix` deletes characters. By default, emoji lose their zero width joiners and variation selectors, and so does Mongolian text. Use `--keep-benign` to preserve recognized legitimate sequences, or run `check` or `fix --dry-run` first on prose. To make hidden characters visible without changing a file, use `reveal`.
 
@@ -402,12 +405,14 @@ git diff | npx text-sanctifier reveal
 ### Inputs
 
 - Paths can be files, directories or globs (`*`, `**`, `?`, `[...]`, `{a,b}`). Quote globs so the shell passes them through.
-- Inside a git work tree, directories and globs list files with `git ls-files`, so anything `.gitignore` excludes is skipped. Outside git, `.git` and `node_modules` are skipped. A file named explicitly is always read.
-- Binary files are skipped, counted in the summary and listed under `skipped` in JSON. A file counts as binary when it is not valid UTF-8 and its first 8000 bytes hold a NUL and more than 5% control bytes, so a stray NUL or bad byte does not hide a text file: the NUL is reported as `nul`. Stdin and AI configuration files are never treated as binary. `fix` is more careful, because it rewrites files: in a directory or glob it also skips any file with a NUL in those bytes, and it cleans such a file only if you name it.
+- Inside a git work tree, directories and globs list files with `git ls-files`, so anything `.gitignore` excludes is skipped. Outside git, `.git` and `node_modules` are skipped. A file named explicitly is always read, unless `--exclude` matches it.
+- `--exclude GLOB` matches paths relative to the current directory, or any parent directory. Leading `./` and trailing `/` are ignored. `*` does not cross `/`: `*.tar` matches only the top level, while `**/*.tar` matches any depth. Excluded paths are listed in JSON and SARIF and counted in text summaries. Stdin is never excluded. There is no config file, ignore file or inline comment form; scanned text cannot silence the scanner.
+- Binary files are skipped, counted in the summary and listed under `skipped` in JSON. A file counts as binary when its first 8000 bytes hold a NUL and more than 5% control bytes, and it is either invalid UTF-8 or entirely ASCII with at least 8 control bytes. Any non-ASCII byte in valid UTF-8 keeps the file checked, even beyond the sample. A stray NUL or bad byte does not hide a text file. Stdin and AI configuration files are never treated as binary. `fix` is more careful, because it rewrites files: in a directory or glob it also skips any file with a NUL in those bytes, and it cleans such a file only if you name it.
 - UTF-16 files with a BOM are decoded and checked. `fix` never rewrites them and lists them under `skipped`.
 - Files that are not valid UTF-8 are still checked, with each invalid byte read as U+FFFD, so one bad byte cannot hide the rest of the file. `check` notes them on stderr and lists them under `notUtf8` in JSON; `fix` never writes them and lists them under `skipped`.
 - `-` reads stdin. With no paths, `check` reads piped input, and scans the current directory when stdin is a terminal, the null device or empty. `check --ai-config` always searches the current directory with no paths. `fix` and `reveal` read piped input. `fix` writes the cleaned text to stdout and the report to stderr.
 - A path or glob that matches no files, or a run that checks nothing, exits 2 instead of reporting a clean result. So does `fix` when it refuses stdin or a file you named. Files it skips in a directory or glob don't fail the run.
+- JSON `errors` entries have `path` (or null), `code` and `message`; read and write failures may also include a system `cause`. Codes are `usage`, `not_found`, `no_match`, `read_failed`, `decode_failed`, `write_failed`, `refused` and `nothing_checked`. Exit 2 means `errors` is nonempty.
 
 ### Exit codes
 

package/bin/text-sanctifier.js

@@ -22,6 +22,10 @@ import { scanHiddenSpans } from '../src/hiddenScan.js';
 import { inspectTextInContext } from '../src/context.js';
 import { classifyCodeUnit, classifyAstral } from '../src/categories.generated.js';
 
+for (const stream of [process.stdout, process.stderr]) {
+    stream.on('error', (e) => { if (e.code !== 'EPIPE') throw e; });
+}
+
 const VERSION = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version;
 
 const USAGE = `text-sanctifier ${VERSION}
@@ -42,6 +46,7 @@ matches no files, or a run that checks nothing, exits 2.
 check options:
   --format F           text (default), json or sarif (SARIF 2.1.0)
   --json               same as --format json
+  --exclude GLOB       exclude matching paths (repeatable)
   --max-findings N     findings listed per file (default 100, "all" for no cap)
   --strip-bom          also report a BOM at the start of a file
   --ai-config          check agent instruction files (CLAUDE.md, AGENTS.md,
@@ -58,6 +63,7 @@ fix options:
   --normalize-newlines convert CRLF and CR to LF
   --strip-bom          also remove a BOM at the start of a file (kept by default)
   --dry-run            report what would change; write nothing
+  --exclude GLOB       exclude matching paths (repeatable)
   --format F           text (default) or json
   --json               same as --format json
   With no cleaning flag, fix uses --normalize-spaces --purge-invisible --nuke-controls.
@@ -82,6 +88,7 @@ const STDIN = '-';
 const STDIN_LABEL = '<stdin>';
 const BINARY_SNIFF_BYTES = 8000;
 const BINARY_CONTROL_SHARE = 0.05;
+const BINARY_MIN_ASCII_CONTROLS = 8;
 const WALK_SKIP_DIRS = new Set(['.git', 'node_modules']);
 const CASE_INSENSITIVE_FS = process.platform === 'win32' || process.platform === 'darwin';
 
@@ -111,6 +118,7 @@ function parseArgs(argv) {
     const opts = {
         command: null,
         paths: [],
+        excludes: [],
         format: null,
         aiConfig: false,
         stripBom: false,
@@ -139,6 +147,14 @@ function parseArgs(argv) {
         if (arg === '-v' || arg === '--version') { opts.version = true; continue; }
 
         const [name, inline] = arg.split(/=(.*)/s, 2);
+        if (name === '--exclude') {
+            const value = inline !== undefined ? inline : argv[++i];
+            if (!value || value.startsWith('-')) throw new UsageError('--exclude expects a non-empty glob.');
+            try { excludeRegExp(value); }
+            catch (e) { if (e instanceof SyntaxError) throw new UsageError(`Invalid --exclude pattern: ${value}`); throw e; }
+            opts.excludes.push(value);
+            continue;
+        }
         if (name === '--max-findings') {
             const value = inline !== undefined ? inline : argv[++i];
             opts.maxFindings = parseMaxFindings(value);
@@ -165,6 +181,7 @@ function parseArgs(argv) {
         throw new UsageError(opts.command === null ? 'Missing command.' : `Unknown command: ${opts.command}`);
     }
     if (opts.command === 'reveal') {
+        if (opts.excludes.length) throw new UsageError('--exclude does not apply to reveal.');
         for (const [flag, key] of Object.entries(BOOLEAN_FLAGS)) {
             if (opts[key]) throw new UsageError(`${flag} does not apply to reveal.`);
         }
@@ -176,6 +193,7 @@ function parseArgs(argv) {
         if (opts.command === 'check' && FIX_ONLY_FLAGS.has(key) && opts[key]) throw new UsageError(`${flag} only applies to fix.`);
         if (opts.command === 'fix' && CHECK_ONLY_FLAGS.has(key) && opts[key]) throw new UsageError(`${flag} only applies to check.`);
     }
+    if (opts.aiConfig && opts.excludes.length) throw new UsageError('--exclude does not apply to --ai-config.');
     if (opts.format === null) opts.format = 'text';
     if (opts.command === 'fix' && opts.format === 'sarif') throw new UsageError('--format sarif only applies to check.');
     if (opts.command === 'fix' && !opts.normalizeSpaces && !opts.purgeInvisible && !opts.nukeControls && !opts.normalizeNewlines) {
@@ -277,7 +295,7 @@ function walk(cwd, dir) {
  * Glob to RegExp for the non-git fallback: `**`, `*`, `?`, `[...]` and `{a,b}`.
  * @param {string} glob
  */
-function globToRegExp(glob) {
+function globToRegExp(glob, flags = '') {
     let re = '';
     for (let i = 0; i < glob.length; i++) {
         const c = glob[i];
@@ -308,9 +326,12 @@ function globToRegExp(glob) {
             re += c.replace(/[.+^$()|\\\]}]/g, '\\$&');
         }
     }
-    return new RegExp(`^${re}$`);
+    return new RegExp(`^${re}$`, flags);
 }
 
+/** @param {string} glob - an --exclude pattern */
+const excludeRegExp = (glob) => globToRegExp(toSlash(glob).replace(/^\.\//, '').replace(/\/$/, ''), CASE_INSENSITIVE_FS ? 'i' : '');
+
 /**
  * @param {string} cwd
  * @param {string} glob
@@ -351,14 +372,13 @@ const isAiConfigPath = (rel) => AI_CONFIG_RES.some((re) => re.test(rel));
  * directories contribute only agent instruction files (AI_CONFIG_GLOBS).
  * @param {string[]} args
  * @param {string} cwd
- * @param {(msg: string) => void} warn
+ * @param {(path: string|null, code: string, message: string) => void} addError
  * @param {boolean} [aiConfig]
- * @returns {{ inputs: Array<{ path: string, explicit: boolean }>, errors: number }}
+ * @returns {Array<{ path: string, explicit: boolean }>}
  */
-function collectInputs(args, cwd, warn, aiConfig = false) {
+function collectInputs(args, cwd, addError, aiConfig = false) {
     const inputs = [];
     const seen = new Set();
-    let errors = 0;
     let inGit;
 
     const add = (path, explicit) => {
@@ -379,8 +399,7 @@ function collectInputs(args, cwd, warn, aiConfig = false) {
 
         const isGlob = !stat && GLOB_CHARS.test(arg);
         if (!stat && !isGlob) {
-            warn(`${arg}: no such file or directory`);
-            errors++;
+            addError(arg, 'not_found', `${arg}: no such file or directory`);
             continue;
         }
 
@@ -393,12 +412,11 @@ function collectInputs(args, cwd, warn, aiConfig = false) {
             const base = resolve(cwd, arg);
             listed = listed.filter((f) => isAiConfigPath(toSlash(relative(base, resolve(cwd, f)))));
         } else if (listed.length === 0) {
-            warn(`${arg}: no files matched`);
-            errors++;
+            addError(arg, 'no_match', `${arg}: no files matched`);
         }
         for (const f of listed) add(f, false);
     }
-    return { inputs, errors };
+    return inputs;
 }
 
 
@@ -412,19 +430,23 @@ const utf16 = new TextDecoder('utf-16le', { ignoreBOM: true });
  * A NUL plus a share of control bytes that text does not have, in the first
  * BINARY_SNIFF_BYTES. One stray NUL is not enough, or it would hide a file from `check`.
  * @param {Uint8Array} bytes
+ * @param {number} [minControls] - fewest control bytes that count
  */
-function looksBinary(bytes) {
+function looksBinary(bytes, minControls = 0) {
     const sample = bytes.subarray(0, BINARY_SNIFF_BYTES);
     if (!sample.includes(0)) return false;
     let controls = 0;
     for (const b of sample) if (b < 9 || (b > 13 && b < 32)) controls++;
-    return controls / sample.length > BINARY_CONTROL_SHARE;
+    return controls >= minControls && controls / sample.length > BINARY_CONTROL_SHARE;
 }
 
 /**
  * Decodes UTF-8, or UTF-16 with a BOM (setting `wide`), keeping a leading BOM. With
- * `sniff`, data that is not UTF-8 and looks binary is not decoded. Other bytes that are
- * not UTF-8 become U+FFFD and set `lossy`, so one bad byte cannot hide the rest of a file.
+ * `sniff`, data that looks binary is not decoded if it is not UTF-8, or if it is entirely
+ * ASCII with at least BINARY_MIN_ASCII_CONTROLS control bytes, so a short file with a stray
+ * NUL is still read. Invisible Unicode needs non-ASCII bytes in UTF-8, so ASCII-only data has
+ * only controls to hide. Other bytes that are not UTF-8 become U+FFFD and set `lossy`, so
+ * one bad byte cannot hide the rest of a file.
  * @param {Buffer} bytes
  * @param {boolean} sniff
  * @returns {{ text: string|null, reason?: string, lossy?: boolean, wide?: boolean }}
@@ -435,7 +457,9 @@ function decode(bytes, sniff) {
         return { text: utf16.decode(Buffer.from(bytes.subarray(0, bytes.length & ~1)).swap16()), wide: true };
     }
     try {
-        return { text: utf8.decode(bytes) };
+        const text = utf8.decode(bytes);
+        if (sniff && looksBinary(bytes, BINARY_MIN_ASCII_CONTROLS) && bytes.every((b) => b < 0x80)) return { text: null, reason: 'binary' };
+        return { text };
     } catch (e) {
         if (!(e instanceof TypeError)) throw e; // too large for a string, not bad UTF-8
         if (sniff && looksBinary(bytes)) return { text: null, reason: 'binary' };
@@ -635,9 +659,9 @@ function sarifArtifact(label) {
 /**
  * Builds a SARIF 2.1.0 log with one result per listed finding; the rule is the category.
  * @param {Array<{ path: string, findings: Array<{ offset: number, line: number, column: number, codePoint: number, category: string }>, hiddenRuns?: Array<{ offset: number, kind: string, count: number, decoded?: string }>, truncated: boolean }>} files
- * @param {{ cwd: string, total: number, filesScanned: number, skipped: Array<{ path: string, reason: string }>, succeeded: boolean }} info
+ * @param {{ cwd: string, total: number, filesScanned: number, skipped: Array<{ path: string, reason: string }>, excluded: string[], succeeded: boolean }} info
  */
-function sarifLog(files, { cwd, total, filesScanned, skipped, succeeded }) {
+function sarifLog(files, { cwd, total, filesScanned, skipped, excluded, succeeded }) {
     const ruleIds = Object.keys(inspectTextDetailed('', { maxFindings: 0 }).counts);
     const rules = ruleIds.map((id) => ({
         id,
@@ -682,6 +706,7 @@ function sarifLog(files, { cwd, total, filesScanned, skipped, succeeded }) {
                 filesWithFindings: files.length,
                 truncated: files.filter((f) => f.truncated).map((f) => f.path),
                 skipped,
+                excluded,
             },
         }],
     };
@@ -704,6 +729,9 @@ async function main(argv) {
     } catch (e) {
         if (!(e instanceof UsageError)) throw e;
         err(`text-sanctifier: ${e.message}\nRun "text-sanctifier --help" for usage.`);
+        if (argv.some((arg, i) => arg === '--json' || arg === '--format=json' || (arg === '--format' && argv[i + 1] === 'json'))) {
+            out(JSON.stringify({ errors: [{ path: null, code: 'usage', message: e.message }] }));
+        }
         return 2;
     }
     if (opts.help) { out(USAGE); return 0; }
@@ -711,6 +739,13 @@ async function main(argv) {
 
     const cwd = process.cwd();
     const warn = (msg) => err(`text-sanctifier: ${msg}`);
+    const errors = [];
+    const addError = (path, code, message, cause, emit = true) => {
+        const entry = { path, code, message };
+        if (cause) entry.cause = cause;
+        errors.push(entry);
+        if (emit) warn(message);
+    };
     let paths = opts.paths;
     /** @type {Buffer | undefined} */
     let stdinBytes;
@@ -722,21 +757,33 @@ async function main(argv) {
             try {
                 stdinBytes = await readStdin();
             } catch (e) {
-                warn(`${STDIN_LABEL}: ${e.code || e.message}`);
+                addError(STDIN_LABEL, 'read_failed', `${STDIN_LABEL}: ${e.code || e.message}`, e.code);
+                if (opts.format === 'json') out(JSON.stringify({ errors }));
                 return 2;
             }
             if (stdinBytes.length > 0) paths = [STDIN];
         }
     } else if (paths.length === 0) {
         if (process.stdin.isTTY) {
+            addError(null, 'usage', 'no paths given and nothing piped on stdin.', undefined, false);
             err('text-sanctifier: no paths given and nothing piped on stdin.\nRun "text-sanctifier --help" for usage.');
+            if (opts.format === 'json') out(JSON.stringify({ errors }));
             return 2;
         }
         paths = [STDIN];
     }
 
-    const { inputs, errors: pathErrors } = collectInputs(paths, cwd, warn, opts.aiConfig);
-    let errors = pathErrors;
+    const collected = collectInputs(paths, cwd, addError, opts.aiConfig);
+    const excluded = [];
+    const excludePatterns = opts.excludes.map(excludeRegExp);
+    const inputs = collected.filter((input) => {
+        if (input.path === STDIN) return true;
+        const path = toSlash(relative(cwd, resolve(cwd, input.path)));
+        const parts = path.split('/');
+        const matches = parts.some((_, i) => excludePatterns.some((re) => re.test(parts.slice(0, i + 1).join('/'))));
+        if (matches) excluded.push(input.path);
+        return !matches;
+    });
     /** @type {Array<{ path: string, reason: string }>} */
     const skipped = [];
     /** @type {string[]} */
@@ -754,8 +801,7 @@ async function main(argv) {
             try {
                 bytes = input.path === STDIN ? (stdinBytes ?? await readStdin()) : readFileSync(resolve(cwd, input.path));
             } catch (e) {
-                warn(`${label}: ${e.code || e.message}`);
-                errors++;
+                addError(label, 'read_failed', `${label}: ${e.code || e.message}`, e.code);
                 continue;
             }
             // Files that look binary are skipped. Stdin and AI configuration files are text
@@ -764,8 +810,7 @@ async function main(argv) {
             try {
                 decoded = decode(bytes, input.path !== STDIN && !opts.aiConfig);
             } catch (e) {
-                warn(`${label}: ${e.message}`);
-                errors++;
+                addError(label, 'decode_failed', `${label}: ${e.message}`);
                 continue;
             }
             const { text, lossy, wide } = decoded;
@@ -782,7 +827,7 @@ async function main(argv) {
                 skipped.push({ path: label, reason });
                 if (input.explicit || reason !== 'binary') warn(`${label}: skipped (${reason})`);
                 // Asked to clean this input and did not: that must not look like success
-                if (opts.command === 'fix' && input.explicit) errors++;
+                if (opts.command === 'fix' && input.explicit) addError(label, 'refused', `${label}: skipped (${reason})`, undefined, false);
                 continue;
             }
             if (lossy) {
@@ -794,10 +839,10 @@ async function main(argv) {
         }
         // A clean result must mean something was read. Finding no AI configuration files is
         // an answer; --ai-config reports that itself.
-        if (visited === 0 && errors === 0 && !opts.aiConfig) errors++;
+        if (visited === 0 && errors.length === 0 && !opts.aiConfig) addError(null, 'nothing_checked', 'Nothing checked.', undefined, false);
     };
 
-    const skippedNote = () => (skipped.length > 0 ? `; ${plural(skipped.length, 'file')} skipped` : '');
+    const skippedNote = () => (skipped.length > 0 ? `; ${plural(skipped.length, 'file')} skipped` : '') + (excluded.length > 0 ? `; ${plural(excluded.length, 'file')} excluded` : '');
 
     if (opts.command === 'check') {
         const files = [];
@@ -839,14 +884,14 @@ async function main(argv) {
 
         const kind = opts.aiConfig ? 'AI configuration file' : 'file';
         if (opts.format === 'json') {
-            const summary = { files, total, filesScanned: scanned, filesWithFindings: files.length, skipped, notUtf8 };
+            const summary = { files, total, filesScanned: scanned, filesWithFindings: files.length, skipped, excluded, notUtf8, errors };
             if (opts.aiConfig) summary.hiddenRunCount = hiddenRunCount;
             out(JSON.stringify(summary, null, 2));
         } else if (opts.format === 'sarif') {
-            out(JSON.stringify(sarifLog(files, { cwd, total, filesScanned: scanned, skipped, succeeded: errors === 0 }), null, 2));
+            out(JSON.stringify(sarifLog(files, { cwd, total, filesScanned: scanned, skipped, excluded, succeeded: errors.length === 0 }), null, 2));
         } else {
             for (const line of lines) out(line);
-            if (opts.aiConfig && scanned === 0 && skipped.length === 0) {
+            if (opts.aiConfig && scanned === 0 && skipped.length === 0 && errors.length === 0) {
                 out('No AI configuration files found.');
                 if (opts.paths.length === 0 && !process.stdin.isTTY) out('To check piped text, pass "-": text-sanctifier check --ai-config -');
             }
@@ -854,7 +899,7 @@ async function main(argv) {
             else if (total === 0) out(`No ${opts.aiConfig ? 'hidden text, ' : ''}invisible or control characters in ${plural(scanned, kind)}${skippedNote()}.`);
             else out(`${plural(total, 'finding')}${opts.aiConfig ? ` (${plural(hiddenRunCount, 'hidden run')})` : ''} in ${files.length} of ${plural(scanned, kind)}${skippedNote()}.`);
         }
-        return errors > 0 ? 2 : total > 0 ? 1 : 0;
+        return errors.length > 0 ? 2 : total > 0 ? 1 : 0;
     }
 
     if (opts.command === 'reveal') {
@@ -867,7 +912,7 @@ async function main(argv) {
             }
             process.stdout.write(revealInvisible(text));
         });
-        return errors > 0 ? 2 : 0;
+        return errors.length > 0 ? 2 : 0;
     }
 
     // fix
@@ -925,8 +970,7 @@ async function main(argv) {
             try {
                 writeFileSync(resolve(cwd, input.path), cleaned, 'utf8');
             } catch (e) {
-                warn(`${label}: ${e.code || e.message}`);
-                errors++;
+                addError(label, 'write_failed', `${label}: ${e.code || e.message}`, e.code);
                 return;
             }
         }
@@ -934,7 +978,7 @@ async function main(argv) {
     });
 
     if (opts.format === 'json') {
-        report(JSON.stringify({ files, dryRun: opts.dryRun, filesScanned: scanned, filesChanged: files.length, skipped }, null, 2));
+        report(JSON.stringify({ files, dryRun: opts.dryRun, filesScanned: scanned, filesChanged: files.length, skipped, excluded, errors }, null, 2));
     } else {
         const verb = opts.dryRun ? 'would fix' : 'fixed';
         for (const f of files) {
@@ -950,7 +994,7 @@ async function main(argv) {
                 ? `Nothing to fix in ${plural(scanned, 'file')}${skippedNote()}.`
                 : `${opts.dryRun ? 'Would change' : 'Changed'} ${files.length} of ${plural(scanned, 'file')}${skippedNote()}.`);
     }
-    return errors > 0 ? 2 : opts.dryRun && files.length > 0 ? 1 : 0;
+    return errors.length > 0 ? 2 : opts.dryRun && files.length > 0 ? 1 : 0;
 }
 
 main(process.argv.slice(2)).then(

package/dist/text-sanctifier.min.js

@@ -3,4 +3,4 @@ var y=/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F-\u009F\u061C\u200E\u200F\u2
 `){return e.replace(re,r)}var te=/[ \t]*(\n+)[ \t]*|([ \t]+)/g;function ie(e){return e.replace(te,"$1$2")}var oe=/\n{2,}/g,ue=/\n{3,}/g;function se(e,r){return r?e.replace(ue,`
 
 `):e.replace(oe,`
-`)}var le=/ {2,}/g;function ae(e){return e.replace(le," ")}function fe(e){return e.replace(y,"")}function ce(e,r={}){if(typeof e!="string")throw new TypeError("revealInvisible expects a string input.");let n=r&&typeof r.format=="function"?r.format:i=>"<U+"+i.toString(16).toUpperCase().padStart(4,"0")+">",u="",o=0;for(let i of A(e,{maxFindings:1/0}).findings)u+=e.slice(o,i.offset)+n(i.codePoint,i.category),o=i.offset+(i.codePoint>65535?2:1);return u+e.slice(o)}var Ee="1.3.11";export{Ee as VERSION,R as getNewlineStats,H as getRecommendedSanctifierOptions,j as inspectText,A as inspectTextDetailed,ce as revealInvisible,L as sanctifyKeyboardOnly,T as sanctifyKeyboardOnlyEmoji,N as sanctifyLoose,D as sanctifyPasted,w as sanctifyStrict,J as summonSanctifier};
+`)}var le=/ {2,}/g;function ae(e){return e.replace(le," ")}function fe(e){return e.replace(y,"")}function ce(e,r={}){if(typeof e!="string")throw new TypeError("revealInvisible expects a string input.");let n=r&&typeof r.format=="function"?r.format:i=>"<U+"+i.toString(16).toUpperCase().padStart(4,"0")+">",u="",o=0;for(let i of A(e,{maxFindings:1/0}).findings)u+=e.slice(o,i.offset)+n(i.codePoint,i.category),o=i.offset+(i.codePoint>65535?2:1);return u+e.slice(o)}var Ee="1.3.13";export{Ee as VERSION,R as getNewlineStats,H as getRecommendedSanctifierOptions,j as inspectText,A as inspectTextDetailed,ce as revealInvisible,L as sanctifyKeyboardOnly,T as sanctifyKeyboardOnlyEmoji,N as sanctifyLoose,D as sanctifyPasted,w as sanctifyStrict,J as summonSanctifier};

package/dist/types/hiddenScan.d.ts

@@ -25,6 +25,7 @@ export function scanHiddenSpans(text: string, options?: HiddenScanOptions, onSpa
 }) => void): HiddenTextScan;
 export type HiddenRunKind = import("./hiddenText.js").HiddenRunKind;
 export type HiddenRun = import("./hiddenText.js").HiddenRun;
+export type HiddenRunSequence = import("./hiddenText.js").HiddenRunSequence;
 export type HiddenRunCounts = import("./hiddenText.js").HiddenRunCounts;
 export type HiddenTextScan = import("./hiddenText.js").HiddenTextScan;
 export type HiddenScanOptions = import("./hiddenText.js").HiddenScanOptions;

package/dist/types/hiddenText.d.ts

@@ -1,4 +1,5 @@
 /** @typedef {'tag' | 'variation_selector' | 'invisible_operator' | 'reserved_tag_plane'} HiddenRunKind */
+/** @typedef {'ideographic' | 'mongolian'} HiddenRunSequence */
 /**
  * @typedef {Object} HiddenRun
  * @property {HiddenRunKind} kind
@@ -10,6 +11,9 @@
  * @property {string} [decoded] - tag runs only: the ASCII the tags spell (see decodeTagText),
  *   at most `maxDecodedLength` UTF-16 units
  * @property {boolean} [decodedTruncated] - tag runs only: true when `decoded` stops short of the run
+ * @property {HiddenRunSequence} [sequence] - a lone selector in a form text-sanctifier/context calls
+ *   benign: U+E0100–U+E01EF after an ideograph, or U+180B–U+180D or U+180F after a Mongolian letter.
+ *   Still a run, because these can carry data: up to a byte per ideograph.
  */
 /**
  * @typedef {Object} HiddenRunCounts
@@ -57,6 +61,7 @@ export function decodeTagText(text: string): string;
  *    Extended_Pictographic character, or between #, * or a digit and U+20E3 (a keycap),
  *    the same rule as text-sanctifier/context. After a letter it is a run: one selector
  *    per letter can carry a bit each.
+ * Lone ideographic and Mongolian selectors stay runs, marked with `sequence`.
  *
  * `line` and `column` are counted as in inspectTextDetailed. Every run and benign
  * span is counted; only the first `maxRuns` of each are listed. With the caps set,
@@ -70,6 +75,7 @@ export function decodeTagText(text: string): string;
 export function findHiddenRuns(text: string, options?: HiddenScanOptions): HiddenTextScan;
 export { VERSION } from "./version.generated.js";
 export type HiddenRunKind = "tag" | "variation_selector" | "invisible_operator" | "reserved_tag_plane";
+export type HiddenRunSequence = "ideographic" | "mongolian";
 export type HiddenRun = {
     kind: HiddenRunKind;
     /**
@@ -101,6 +107,12 @@ export type HiddenRun = {
      * - tag runs only: true when `decoded` stops short of the run
      */
     decodedTruncated?: boolean;
+    /**
+     * - a lone selector in a form text-sanctifier/context calls
+     * benign: U+E0100–U+E01EF after an ideograph, or U+180B–U+180D or U+180F after a Mongolian letter.
+     * Still a run, because these can carry data: up to a byte per ideograph.
+     */
+    sequence?: HiddenRunSequence;
 };
 export type HiddenRunCounts = {
     tag: number;

package/dist/types/version.generated.d.ts

@@ -1,2 +1,2 @@
 /** The installed version of text-sanctifier, as in its package.json. */
-export const VERSION: "1.3.11";
+export const VERSION: "1.3.13";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "text-sanctifier",
-  "version": "1.3.11",
+  "version": "1.3.13",
   "type": "module",
   "description": "Find and remove invisible Unicode in web input, model-bound text and repositories.",
   "engines": {

package/src/hiddenScan.js

@@ -3,6 +3,7 @@
 /**
  * @typedef {import('./hiddenText.js').HiddenRunKind} HiddenRunKind
  * @typedef {import('./hiddenText.js').HiddenRun} HiddenRun
+ * @typedef {import('./hiddenText.js').HiddenRunSequence} HiddenRunSequence
  * @typedef {import('./hiddenText.js').HiddenRunCounts} HiddenRunCounts
  * @typedef {import('./hiddenText.js').HiddenTextScan} HiddenTextScan
  * @typedef {import('./hiddenText.js').HiddenScanOptions} HiddenScanOptions
@@ -76,6 +77,32 @@ function isPresentationBase(text, index) {
     return PICTOGRAPHIC_AT.test(text);
 }
 
+const IDEOGRAPHIC_AT = /\p{Ideographic}/yu;
+
+/**
+ * The legitimate sequence a lone selector at `index` could belong to, by the same rules
+ * as text-sanctifier/context. It stays a run: unlike U+FE0E and U+FE0F, these selectors
+ * have enough values to spell data, up to a byte per ideograph.
+ * @param {string} text
+ * @param {number} index
+ * @param {number} cp - the selector
+ * @returns {HiddenRunSequence | null}
+ */
+function sequenceOf(text, index, cp) {
+    if (cp >= 0xE0100 && cp <= 0xE01EF) {
+        if (index === 0) return null;
+        const low = text.charCodeAt(index - 1);
+        IDEOGRAPHIC_AT.lastIndex = low >= 0xDC00 && low <= 0xDFFF && index >= 2 &&
+            (text.charCodeAt(index - 2) & 0xFC00) === 0xD800 ? index - 2 : index - 1;
+        return IDEOGRAPHIC_AT.test(text) ? 'ideographic' : null;
+    }
+    if (cp > 0x180F) return null;
+    const c = text.charCodeAt(index - 1);
+    return (c >= 0x1820 && c <= 0x1878) || (c >= 0x1880 && c <= 0x1884) || (c >= 0x1887 && c <= 0x18A8) || c === 0x18AA
+        ? 'mongolian'
+        : null;
+}
+
 /**
  * A cap option: a non-negative integer or Infinity; anything else means no cap.
  * @param {unknown} value
@@ -163,6 +190,9 @@ export function scanHiddenSpans(text, options = {}, onSpan) {
                 const { decoded, truncated } = decodeRange(text, offset, end, maxDecodedLength);
                 run.decoded = decoded;
                 run.decodedTruncated = truncated;
+            } else if (kind === 'variation_selector' && count === 1) {
+                const sequence = sequenceOf(text, offset, cp);
+                if (sequence) run.sequence = sequence;
             }
             runs.push(run);
         }

package/src/hiddenText.js

@@ -15,6 +15,8 @@ export { VERSION } from './version.generated.js';
 
 /** @typedef {'tag' | 'variation_selector' | 'invisible_operator' | 'reserved_tag_plane'} HiddenRunKind */
 
+/** @typedef {'ideographic' | 'mongolian'} HiddenRunSequence */
+
 /**
  * @typedef {Object} HiddenRun
  * @property {HiddenRunKind} kind
@@ -26,6 +28,9 @@ export { VERSION } from './version.generated.js';
  * @property {string} [decoded] - tag runs only: the ASCII the tags spell (see decodeTagText),
  *   at most `maxDecodedLength` UTF-16 units
  * @property {boolean} [decodedTruncated] - tag runs only: true when `decoded` stops short of the run
+ * @property {HiddenRunSequence} [sequence] - a lone selector in a form text-sanctifier/context calls
+ *   benign: U+E0100–U+E01EF after an ideograph, or U+180B–U+180D or U+180F after a Mongolian letter.
+ *   Still a run, because these can carry data: up to a byte per ideograph.
  */
 
 /**
@@ -81,6 +86,7 @@ export function decodeTagText(text) {
  *    Extended_Pictographic character, or between #, * or a digit and U+20E3 (a keycap),
  *    the same rule as text-sanctifier/context. After a letter it is a run: one selector
  *    per letter can carry a bit each.
+ * Lone ideographic and Mongolian selectors stay runs, marked with `sequence`.
  *
  * `line` and `column` are counted as in inspectTextDetailed. Every run and benign
  * span is counted; only the first `maxRuns` of each are listed. With the caps set,

package/src/version.generated.js

@@ -2,4 +2,4 @@
 // Generated from package.json by scripts/gen-version.js (`npm run gen`); do not edit.
 
 /** The installed version of text-sanctifier, as in its package.json. */
-export const VERSION = '1.3.11';
+export const VERSION = '1.3.13';