text-sanctifier 1.3.4 → 1.3.6

package/README.md

@@ -7,7 +7,7 @@
 [![stars](https://badgen.net/github/stars/iWhatty/text-sanctifier)](https://github.com/iWhatty/text-sanctifier)
 [![types](https://badgen.net/npm/types/text-sanctifier)](https://www.npmjs.com/package/text-sanctifier)
 
-Brutal text normalizer and invisible Unicode scrubber for modern web projects. ~2.57 KB gzipped.
+Brutal text normalizer and invisible Unicode scrubber for modern web projects. ~2.87 KB gzipped.
 
 ## Features
 
@@ -15,6 +15,8 @@ Brutal text normalizer and invisible Unicode scrubber for modern web projects. ~
 - Purges bidi controls, including the isolates (U+2066–U+2069) and the Arabic letter mark (U+061C) used in Trojan Source attacks
 - Purges Unicode tag characters (U+E0000–U+E007F), the "ASCII smuggling" channel for hiding instructions from humans in text an LLM reads, plus variation selectors, Hangul fillers, soft hyphens and invisible operators
 - Locates invisible and control characters by offset, line and column (`inspectTextDetailed`)
+- Shows hidden characters as visible markers such as `<U+200B>` (`revealInvisible`, `text-sanctifier reveal`)
+- Cleans pasted text and model output without gluing words together: look-alike spaces such as the narrow no-break space (U+202F) become ordinary spaces (`summonSanctifier.pasted`)
 - `text-sanctifier check` / `fix` command line for CI and coding agents, with JSON or SARIF output, exit codes and pre-commit hooks
 - `check --ai-config` finds and decodes hidden text in agent instruction files (`CLAUDE.md`, `AGENTS.md`, Cursor rules, `SKILL.md`, MCP configuration)
 - Normalizes line endings (CRLF, CR, LF) to LF
@@ -81,6 +83,20 @@ const output = summonSanctifier.keyboardOnly(userInput);
 const output = summonSanctifier.keyboardOnlyEmoji(commentText);
 ```
 
+**Pasted text and model output:**
+
+```js
+const output = summonSanctifier.pasted(clipboardText);
+```
+
+**Show what is hidden:**
+
+```js
+import { revealInvisible } from 'text-sanctifier';
+
+revealInvisible(text); // 'Hello<U+200B>world'
+```
+
 ---
 
 ## Security notes
@@ -98,6 +114,21 @@ const output = summonSanctifier.keyboardOnlyEmoji(commentText);
 
 Creates a reusable sanitizer from an option object.
 
+Two options handle spaces and zero-width characters gently, and both run before the others:
+
+- `normalizeSpaces`: replace look-alike spaces with an ordinary space. These are U+00A0, U+2000–U+200A, U+202F and U+205F: the `nonbreaking_space` category except U+3000, the ordinary space of CJK text. With `purgeInvisibleChars` alone, a non-breaking space between two words is deleted and the words run together; with `normalizeSpaces` as well, they stay apart.
+- `purgeZeroWidthSpaces`: remove the zero width space U+200B, the word joiner U+2060 and U+FEFF, and nothing else. Emoji joiners, ZWNJ, variation selectors and bidi marks are kept.
+
+### Presets as standalone imports
+
+Each preset is also its own export: `sanctifyStrict`, `sanctifyLoose`, `sanctifyKeyboardOnly`, `sanctifyKeyboardOnlyEmoji` and `sanctifyPasted`. They are the same functions as `summonSanctifier.strict` and the rest. Import one directly and a bundler leaves out `summonSanctifier` and the other presets (about 1.1 KB gzipped instead of 1.45 KB).
+
+```js
+import { sanctifyPasted } from 'text-sanctifier';
+
+const output = sanctifyPasted(clipboardText);
+```
+
 ### `summonSanctifier.strict`
 
 Aggressively purges: emojis, control characters, extra spacing, and newlines.
@@ -114,6 +145,23 @@ Restricts to printable ASCII only (removes emojis).
 
 Restricts to printable ASCII + full emoji sequences. Preserves ZWJ emoji clusters and emoji presentation variants.
 
+### `summonSanctifier.pasted`
+
+For text pasted from documents, chat apps and PDFs, and for model output. Some models put narrow no-break spaces (U+202F) and other look-alike spaces into their answers. This preset uses `normalizeSpaces`, `purgeZeroWidthSpaces` and `normalizeNewlines`: look-alike spaces become ordinary spaces, zero-width spaces and word joiners are removed, and CRLF and CR become LF. It keeps emoji joiners, variation selectors, bidi marks, indentation, blank lines and leading and trailing whitespace, so it is safe on code and Markdown.
+
+### `revealInvisible(text: string, options?: RevealOptions): string`
+
+Replaces every character `inspectTextDetailed` would report with a visible marker and leaves everything else alone, including tabs, newlines and emoji. Use it to show hidden text in logs, diffs, review tools and agent tool output. Also in the browser build.
+
+```js
+import { revealInvisible } from 'text-sanctifier';
+
+revealInvisible('a\u200Bb\u{E0041}');                    // 'a<U+200B>b<U+E0041>'
+revealInvisible('a\u202Eb', { format: (codePoint, category) => `[${category}]` }); // 'a[bidi_control]b'
+```
+
+The default marker is `<U+XXXX>` with at least four uppercase hex digits. Every hidden character is replaced, with no cap like `maxFindings`. A literal `<U+200B>` already in the text looks the same as a marker; pass `format` to choose markers that can't occur in your input.
+
 ### `inspectText(text: string): UnicodeTrashReport`
 
 Returns a structural report of control codes, invisible chars, newline styles, and more.
@@ -198,22 +246,25 @@ getNewlineStats('a\r\nb\n'); // { crlf: 1, cr: 0, lf: 1, types: ['CRLF', 'LF'],
 
 The categories cover exactly what `nukeControls` and `purgeInvisibleChars` strip. Every code point with the Unicode property Default_Ignorable_Code_Point (characters renderers must draw as nothing) is in a category, and a test checks this against the running Node's Unicode data.
 
-### `text-sanctifier/hidden`: `findHiddenRuns(text)` and `decodeTagText(text)`
+### `text-sanctifier/hidden`: `findHiddenRuns(text, options?)` and `decodeTagText(text)`
 
 The run finder behind [`check --ai-config`](#ai-configuration-files), for tools that read text on its way to a model. It has no Node APIs, so it works in a browser too, but it is a separate entry point and not part of the `text-sanctifier/browser` bundle.
 
 ```js
 import { findHiddenRuns, decodeTagText } from 'text-sanctifier/hidden';
 
-const { runs, benign } = findHiddenRuns(text);
-// runs:   [{ kind: 'tag', offset, end, count, decoded: 'Ignore the user' }, ...]
+const { runs, benign, counts, total, truncated } = findHiddenRuns(text, { maxRuns: 100, maxDecodedLength: 1000 });
+// runs:   [{ kind: 'tag', offset, end, line, column, count, decoded: 'Ignore the user', decodedTruncated: false }, ...]
 // benign: [[offset, end], ...]  England/Scotland/Wales flags, a lone U+FE0E/U+FE0F after a character
+// counts: { tag, variation_selector, invisible_operator, reserved_tag_plane }, every run listed or not; total is their sum
 ```
 
-- `kind` is `tag` (U+E0000–U+E007F), `variation_selector` (U+FE00–U+FE0F, U+E0100–U+E01EF, U+180B–U+180D, U+180F) or `invisible_operator` (U+2061–U+2064). Consecutive characters of one kind form one run.
-- `offset` and `end` are UTF-16 indexes, `[offset, end)`; `count` is in code points. Only tag runs have `decoded`.
+- `kind` is `tag` (U+E0000–U+E007F), `variation_selector` (U+FE00–U+FE0F, U+E0100–U+E01EF, U+180B–U+180D, U+180F) `invisible_operator` (U+2061–U+2064) or `reserved_tag_plane` (U+E0080–U+E00FF, U+E01F0–U+E0FFF: unassigned, default ignorable, so they render as nothing and can carry data). Consecutive characters of one kind form one run.
+- `offset` and `end` are UTF-16 indexes, `[offset, end)`; `count` is in code points. `line` and `column` are counted exactly as in `inspectTextDetailed` (1-based, columns in UTF-16 units, CRLF is one line break). Only tag runs have `decoded` and `decodedTruncated`.
+- `maxRuns` caps the `runs` list and, separately, the `benign` list; `counts`, `total` and `benignTotal` count everything, and `truncated` is true when either list was cut. `maxDecodedLength` caps each `decoded` string in UTF-16 units and never cuts a `<U+XXXX>` marker. Both default to no cap, and an invalid value (negative, fractional, a string) also means no cap, so set them explicitly for untrusted input. With both set, memory stays flat on any input: on 16 M characters (32 MB) of hidden text the heap grows by under 1 MB, against hundreds of MB with no caps.
 - `decodeTagText` turns U+E0020–U+E007E into the ASCII they shadow and writes any other character as `<U+XXXX>`.
 - Zero-width, bidi and other invisible characters are not runs; use `inspectTextDetailed` for those.
+- `VERSION` is exported here and from the package root: the installed version, as in `package.json`.
 
 ---
 
@@ -224,6 +275,7 @@ The package ships a `text-sanctifier` command for CI and coding agents, which ca
 ```sh
 npx text-sanctifier check src docs "**/*.md"
 npx text-sanctifier fix --dry-run src
+npx text-sanctifier reveal notes.md
 ```
 
 ### `check [paths...]`
@@ -262,7 +314,7 @@ skills/review/SKILL.md:2:3  U+FE00  run of 3 variation selectors
 - Consecutive tag characters, variation selectors or invisible operators are reported once, as a run, at the first character. Tag runs are decoded: U+E0020–U+E007E become the ASCII they shadow, and other tags are shown as `<U+XXXX>`.
 - Every other invisible or control character is reported as in a plain `check`.
 - Two legitimate sequences are left out: the flags of England, Scotland and Wales (U+1F3F4, tag letters, U+E007F), and a single U+FE0E or U+FE0F after another character (text or emoji presentation). A plain `check` still reports both.
-- `--json` adds `hiddenRuns` (`{ kind, offset, line, column, codePoint, count, decoded?, decodedTruncated? }`, at most `--max-findings` per file) and `hiddenRunCount` to each file, and `hiddenRunCount` to the summary. `kind` is `tag`, `variation_selector` or `invisible_operator`; `decoded` is cut at 1000 characters. `total` and `counts` leave out the legitimate sequences.
+- `--json` adds `hiddenRuns` (`{ kind, offset, line, column, codePoint, count, decoded?, decodedTruncated? }`, at most `--max-findings` per file) and `hiddenRunCount` to each file, and `hiddenRunCount` to the summary. `kind` is `tag`, `variation_selector`, `invisible_operator` or `reserved_tag_plane`; `decoded` is cut at 1000 characters. `total` and `counts` leave out the legitimate sequences.
 
 ### SARIF output
 
@@ -282,7 +334,7 @@ The repository publishes a [pre-commit](https://pre-commit.com) hook manifest, s
 # .pre-commit-config.yaml
 repos:
   - repo: https://github.com/iWhatty/text-sanctifier
-    rev: v1.3.4  # a release tag
+    rev: v1.3.6  # a release tag
     hooks:
       - id: text-sanctifier            # every staged text file
       - id: text-sanctifier-ai-config  # agent instruction files, with --ai-config
@@ -294,17 +346,27 @@ repos:
 
 ### `fix [paths...]`
 
-Rewrites files in place and reports what it removed.
+Rewrites files in place and reports what it replaced and removed.
 
+- `--normalize-spaces`: `normalizeSpaces` (look-alike spaces become ordinary spaces)
 - `--purge-invisible`: `purgeInvisibleChars`
 - `--nuke-controls`: `nukeControls`
 - `--normalize-newlines`: `normalizeNewlines` (CRLF and CR to LF)
-- With none of the three, `fix` uses `--purge-invisible --nuke-controls`, which removes exactly what `check` reports.
+- With none of these, `fix` uses `--normalize-spaces --purge-invisible --nuke-controls`: look-alike spaces become ordinary spaces, and everything else `check` reports is removed.
 - `--strip-bom`: also remove a BOM at offset 0. It is kept by default.
 - `--dry-run`: write nothing; report what would change.
-- `--json` (or `--format json`): `{ files, dryRun, filesScanned, filesChanged, skipped }`, where each file is `{ path, removed, removedTotal, newlinesNormalized }`.
+- `--json` (or `--format json`): `{ files, dryRun, filesScanned, filesChanged, skipped }`, where each file is `{ path, spacesReplaced, removed, removedTotal, newlinesNormalized }`.
+
+Apart from look-alike spaces, `fix` deletes characters. Emoji lose their zero width joiners and variation selectors, and so does Mongolian text. Run `check` or `fix --dry-run` first on prose. To make hidden characters visible without changing a file, use `reveal`.
+
+### `reveal [paths...]`
 
-`fix` deletes characters; it does not replace them. A non-breaking space between two words joins them, and emoji lose their zero width joiners and variation selectors. Run `check` or `fix --dry-run` first on prose.
+Prints each input with every character `check` would report replaced by a marker such as `<U+200B>` (see `revealInvisible`). Everything else, including newlines, is printed as it is, and files are never changed. Several inputs are printed one after another under `==> path <==` headers. `reveal` takes no options and exits 0 unless an input can't be read.
+
+```sh
+npx text-sanctifier reveal AGENTS.md
+git diff | npx text-sanctifier reveal
+```
 
 ### Inputs
 
@@ -315,11 +377,11 @@ Rewrites files in place and reports what it removed.
 
 ### Exit codes
 
-| Code | `check` | `fix` |
-| --- | --- | --- |
-| 0 | nothing found | done (or nothing to do) |
-| 1 | findings | `--dry-run` with changes pending |
-| 2 | usage error, missing path or unreadable file | same |
+| Code | `check` | `fix` | `reveal` |
+| --- | --- | --- | --- |
+| 0 | nothing found | done (or nothing to do) | done |
+| 1 | findings | `--dry-run` with changes pending | not used |
+| 2 | usage error, missing path or unreadable file | same | same |
 
 ---
 
@@ -337,8 +399,8 @@ Requires a modern JavaScript runtime with ES2020+ support:
 - **Source (`src/`):** ES2020+ ESM modules with JSDoc
 - **ESM only:** no CommonJS build; `require('text-sanctifier')` works only on Node versions with `require(esm)` support (20.19+, 22.12+)
 - **Types:** TypeScript declarations ship in `dist/types`
-- **Browser build (`dist/`):** Minified ESM bundle for `<script type="module">`
-- **Tree-shake friendly:** ships `sideEffects: false`
+- **Browser build (`dist/`):** one minified ESM file with every export, for pages with no build step (`<script type="module">` from a CDN). It is imported as `text-sanctifier/browser`. With a bundler, import `text-sanctifier` instead: the source tree-shakes, and the prebuilt file does not
+- **Tree-shake friendly:** ships `sideEffects: false`; `npm run ci` checks the bundled size of each export on its own (for example `inspectTextDetailed` is under 1 KB gzipped)
 - **Zero transpilation:** no built-in polyfills or runtime overhead
 - **Bundler ready:** works with Vite, Rollup, Webpack, Parcel, esbuild
 

package/bin/text-sanctifier.js

@@ -5,6 +5,7 @@
 //   text-sanctifier check [paths...]   report invisible and control characters
 //   text-sanctifier check --ai-config  report hidden text in agent instruction files
 //   text-sanctifier fix [paths...]     remove them in place
+//   text-sanctifier reveal [paths...]  print the text with hidden characters shown as <U+XXXX>
 //
 // Plain Node (>=18), no dependencies. Exit codes: 0 clean, 1 findings (check,
 // or fix --dry-run with changes pending), 2 usage or I/O error.
@@ -13,8 +14,9 @@ import { readFileSync, writeFileSync, statSync, readdirSync, existsSync } from '
 import { resolve, join, sep, relative } from 'node:path';
 import { spawnSync } from 'node:child_process';
 import { pathToFileURL } from 'node:url';
-import { inspectTextDetailed, summonSanctifier } from '../src/index.js';
+import { inspectTextDetailed, summonSanctifier, revealInvisible } from '../src/index.js';
 import { getNewlineStats } from '../src/inspectText.js';
+import { SPACE_LOOKALIKES_REGEX } from '../src/sanctifyText.js';
 import { findHiddenRuns } from '../src/hiddenText.js';
 
 const VERSION = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version;
@@ -24,6 +26,7 @@ const USAGE = `text-sanctifier ${VERSION}
 Usage:
   text-sanctifier check [options] [paths...]
   text-sanctifier fix [options] [paths...]
+  text-sanctifier reveal [paths...]
 
 Paths may be files, directories or globs (quote globs so the shell leaves them
 alone). "-", or no paths with piped input, reads stdin. Directories and globs
@@ -40,6 +43,8 @@ check options:
                        for those files, and tag text is decoded
 
 fix options:
+  --normalize-spaces   replace look-alike spaces (NBSP, narrow NBSP, typographic
+                       spaces) with an ordinary space instead of removing them
   --purge-invisible    remove zero-width, non-breaking space, bidi and BOM characters
   --nuke-controls      remove C0/C1 control characters and bidi controls
   --normalize-newlines convert CRLF and CR to LF
@@ -47,9 +52,15 @@ fix options:
   --dry-run            report what would change; write nothing
   --format F           text (default) or json
   --json               same as --format json
-  With no cleaning flag, fix uses --purge-invisible --nuke-controls.
+  With no cleaning flag, fix uses --normalize-spaces --purge-invisible --nuke-controls.
   stdin is cleaned to stdout; the report then goes to stderr.
 
+reveal:
+  Prints each input with every character check would report replaced by a
+  marker such as <U+200B>; everything else, including newlines, is unchanged.
+  Several inputs are printed one after another under "==> path <==" headers.
+  Takes no options. Exit code 0 unless an input can't be read.
+
 Other:
   -h, --help           show this help
   -v, --version        show the version
@@ -73,12 +84,13 @@ class UsageError extends Error {}
 const BOOLEAN_FLAGS = {
     '--ai-config': 'aiConfig',
     '--strip-bom': 'stripBom',
+    '--normalize-spaces': 'normalizeSpaces',
     '--purge-invisible': 'purgeInvisible',
     '--nuke-controls': 'nukeControls',
     '--normalize-newlines': 'normalizeNewlines',
     '--dry-run': 'dryRun',
 };
-const FIX_ONLY_FLAGS = new Set(['purgeInvisible', 'nukeControls', 'normalizeNewlines', 'dryRun']);
+const FIX_ONLY_FLAGS = new Set(['normalizeSpaces', 'purgeInvisible', 'nukeControls', 'normalizeNewlines', 'dryRun']);
 const CHECK_ONLY_FLAGS = new Set(['aiConfig']);
 const FORMATS = new Set(['text', 'json', 'sarif']);
 
@@ -92,6 +104,7 @@ function parseArgs(argv) {
         format: null,
         aiConfig: false,
         stripBom: false,
+        normalizeSpaces: false,
         purgeInvisible: false,
         nukeControls: false,
         normalizeNewlines: false,
@@ -137,16 +150,25 @@ function parseArgs(argv) {
     }
 
     if (opts.help || opts.version) return opts;
-    if (opts.command !== 'check' && opts.command !== 'fix') {
+    if (opts.command !== 'check' && opts.command !== 'fix' && opts.command !== 'reveal') {
         throw new UsageError(opts.command === null ? 'Missing command.' : `Unknown command: ${opts.command}`);
     }
+    if (opts.command === 'reveal') {
+        for (const [flag, key] of Object.entries(BOOLEAN_FLAGS)) {
+            if (opts[key]) throw new UsageError(`${flag} does not apply to reveal.`);
+        }
+        if (opts.format !== null) throw new UsageError('--format does not apply to reveal.');
+        if (opts.maxFindings !== 100) throw new UsageError('--max-findings does not apply to reveal.');
+        return opts;
+    }
     for (const [flag, key] of Object.entries(BOOLEAN_FLAGS)) {
         if (opts.command === 'check' && FIX_ONLY_FLAGS.has(key) && opts[key]) throw new UsageError(`${flag} only applies to fix.`);
         if (opts.command === 'fix' && CHECK_ONLY_FLAGS.has(key) && opts[key]) throw new UsageError(`${flag} only applies to check.`);
     }
     if (opts.format === null) opts.format = 'text';
     if (opts.command === 'fix' && opts.format === 'sarif') throw new UsageError('--format sarif only applies to check.');
-    if (opts.command === 'fix' && !opts.purgeInvisible && !opts.nukeControls && !opts.normalizeNewlines) {
+    if (opts.command === 'fix' && !opts.normalizeSpaces && !opts.purgeInvisible && !opts.nukeControls && !opts.normalizeNewlines) {
+        opts.normalizeSpaces = true;
         opts.purgeInvisible = true;
         opts.nukeControls = true;
     }
@@ -421,7 +443,7 @@ const plural = (n, word) => `${n} ${word}${n === 1 ? '' : 's'}`;
 // --- check --ai-config ---
 
 const MAX_DECODED = 1000;
-const RUN_NOUNS = { tag: 'tag character', variation_selector: 'variation selector', invisible_operator: 'invisible operator' };
+const RUN_NOUNS = { tag: 'tag character', variation_selector: 'variation selector', invisible_operator: 'invisible operator', reserved_tag_plane: 'reserved plane-14 code point' };
 
 /**
  * inspect() for agent instruction files: drops findings that belong to legitimate
@@ -683,8 +705,22 @@ async function main(argv) {
         return errors > 0 ? 2 : total > 0 ? 1 : 0;
     }
 
+    if (opts.command === 'reveal') {
+        const many = inputs.length > 1;
+        let first = true;
+        await forEachText((label, text) => {
+            if (many) {
+                process.stdout.write(`${first ? '' : '\n'}==> ${label} <==\n`);
+                first = false;
+            }
+            process.stdout.write(revealInvisible(text));
+        });
+        return errors > 0 ? 2 : 0;
+    }
+
     // fix
     const clean = summonSanctifier({
+        normalizeSpaces: opts.normalizeSpaces,
         purgeInvisibleChars: opts.purgeInvisible,
         nukeControls: opts.nukeControls,
         normalizeNewlines: opts.normalizeNewlines,
@@ -706,6 +742,10 @@ async function main(argv) {
 
         const before = inspectTextDetailed(text, { maxFindings: 0 }).counts;
         const after = inspectTextDetailed(cleaned, { maxFindings: 0 }).counts;
+        // Look-alike spaces that became ordinary spaces were replaced, not removed
+        SPACE_LOOKALIKES_REGEX.lastIndex = 0;
+        const spacesReplaced = opts.normalizeSpaces ? (text.match(SPACE_LOOKALIKES_REGEX) || []).length : 0;
+        before.nonbreaking_space -= spacesReplaced;
         const removed = {};
         let removedTotal = 0;
         for (const k of Object.keys(before)) {
@@ -724,7 +764,7 @@ async function main(argv) {
                 return;
             }
         }
-        files.push({ path: label, removed, removedTotal, newlinesNormalized });
+        files.push({ path: label, spacesReplaced, removed, removedTotal, newlinesNormalized });
     });
 
     if (opts.format === 'json') {
@@ -733,6 +773,7 @@ async function main(argv) {
         const verb = opts.dryRun ? 'would fix' : 'fixed';
         for (const f of files) {
             const parts = [];
+            if (f.spacesReplaced > 0) parts.push(`replaced ${plural(f.spacesReplaced, 'look-alike space')}`);
             if (f.removedTotal > 0) parts.push(`removed ${f.removedTotal} (${formatCounts(f.removed)})`);
             if (f.newlinesNormalized) parts.push('normalized newlines');
             report(`${verb} ${f.path}: ${parts.join('; ')}`);

package/dist/text-sanctifier.min.js

@@ -1,15 +1,6 @@
-var n=/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F-\u009F\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]+/g,q=/[\u00A0\u00AD\u034F\u061C\u115F\u1160\u17B4\u17B5\u180B-\u180F\u2000-\u200F\u202A-\u202F\u205F-\u206F\u3000\u3164\uFE00-\uFE0F\uFEFF\uFFA0\uFFF0-\uFFF8\u{1BCA0}-\u{1BCA3}\u{1D173}-\u{1D17A}\u{E0000}-\u{E0FFF}]+/gu;
-function t(a={}){const c=!!a.purgeInvisibleChars,d=!!a.purgeEmojis,f=!!a.nukeControls,h=!!a.keyboardOnlyFilter,g=!!a.normalizeNewlines,l=!!a.trimSpacesAroundNewlines,b=!!a.collapseNewLines,e=!!a.preserveParagraphs,k=!!a.collapseSpaces,r=!!a.finalTrim,m=Number.isFinite(a.maxLength)&&0<=a.maxLength?a.maxLength:Infinity,p="truncate"===a.g||"noop"===a.g||"throw"===a.g?a.g:"throw";return A=>u(A,c,d,f,h,g,l,b,e,k,r,m,p)}t.strict=a=>u(a,!0,!0,!0,!1,!0,!0,!0,!1,!0,!0);
-t.loose=a=>u(a,!1,!1,!1,!1,!0,!0,!0,!0,!0,!0);t.keyboardOnlyEmoji=a=>u(a,!1,!1,!1,!0,!0,!0,!1,!1,!1,!0);t.keyboardOnly=a=>u(a,!0,!0,!0,!0,!0,!0,!0,!1,!0,!0);
-function u(a,c=!1,d=!1,f=!1,h=!1,g=!1,l=!1,b=!1,e=!1,k=!1,r=!1,m=Infinity,p="throw"){if("string"!==typeof a)throw new TypeError("sanctifyText expects a string input.");"truncate"!==p&&"noop"!==p&&"throw"!==p&&(p="throw");if(a.length>m)switch(p){case "truncate":a=a.slice(0,m);break;case "noop":return a;default:throw new RangeError(`sanctifyText input length ${a.length} exceeds maxLength ${m}.`);}c&&(a=a.replace(q,""));d&&(a=a.replace(v,""));f&&(a=a.replace(n,""));h&&(a=w(a,d));g&&(a=a.replace(x,"\n"));
-l&&(a=a.replace(y,"$1"));b&&(c=a,a=e?c.replace(z,"\n\n"):c.replace(B,"\n"));k&&(a=a.replace(C," "));return r?a.trim():a}var D=/[^\x20-\x7E\r\n]+/gu;function w(a,c=!1){a=E(a);return c?a.replace(D,""):a.replace(D,d=>{v.lastIndex=0;if(v.test(d)){v.lastIndex=0;var f="";for(const h of d.matchAll(v))f+=h[0];d=f}else d="";return d})}
-var F=/[\u2018\u2019\u201A\u201B\u2032\u2035]/g,G=/[\u201C\u201D\u201E\u201F\u2033\u2036\u00AB\u00BB]/g,H=/[\u2012\u2013\u2014\u2015\u2212]/g,I=/\u2026/g,J=/[\u2022\u00B7]/g,K=/[\uFF01-\uFF5E]/g;function E(a){return a.replace(F,"'").replace(G,'"').replace(H,"-").replace(I,"...").replace(J,"*").replace(K,c=>String.fromCharCode(c.charCodeAt(0)-65248))}var v;try{v=RegExp("(?:\\p{Extended_Pictographic}(?:\\uFE0F|\\uFE0E)?(?:\\u200D(?:\\p{Extended_Pictographic}|\\w)+)*)","gu")}catch{v=/[\u{2700}-\u{27BF}\u{1F300}-\u{1FAFF}\u{1F3FB}-\u{1F3FF}\u200D\uFE0F\u{1F1E6}-\u{1F1FF}]/gu}
-var x=/\r\n|\r|\n/g,y=/[ \t]*(\n+)[ \t]*/g,B=/\n{2,}/g,z=/\n{3,}/g,C=/ {2,}/g;function L(a,c){a.lastIndex=0;return a.test(c)}
-function M(a){if("string"!==typeof a)throw new TypeError("inspectText expects a string input.");const c=[],d={hasControlChars:!1,hasInvisibleChars:!1,hasMixedNewlines:!1,newlineStyle:null,hasEmojis:!1,hasNonKeyboardChars:!1,summary:c};L(n,a)&&(d.hasControlChars=!0,c.push("Control characters detected."));L(q,a)&&(d.hasInvisibleChars=!0,c.push("Invisible Unicode characters detected."));L(v,a)&&(d.hasEmojis=!0,c.push("Emojis detected."));const {mixed:f,types:h}=N(a);d.hasMixedNewlines=f;d.newlineStyle=
-f?"Mixed":h[0]||null;d.newlineStyle&&c.push(f?"Mixed newline styles detected.":`Consistent newline style: ${d.newlineStyle}`);a:{a=E(a);if(L(D,a))for(g of a)if(!("\n"===g||"\r"===g||" "<=g&&"~">=g||L(v,g))){var g=!0;break a}g=!1}g&&(d.hasNonKeyboardChars=!0,c.push("Non-keyboard characters detected."));return d}
-function O(a,c={}){if("string"!==typeof a)throw new TypeError("inspectTextDetailed expects a string input.");c=c&&c.maxFindings;c=Infinity===c||Number.isInteger(c)&&0<=c?c:100;const d={bidi_control:0,zero_width:0,nonbreaking_space:0,bom:0,nul:0,c0_control:0,c1_control:0,invisible_format:0,hangul_filler:0,variation_selector:0,tag:0},f=[];let h=0,g=1,l=0;for(let k=0,r=a.length;k<r;k++){var b=a.charCodeAt(k);if(32<=b&&127>b)continue;if(10===b){g++;l=k+1;continue}if(13===b){10!==a.charCodeAt(k+1)&&(g++,
-l=k+1);continue}const m=k;var e=b;if(55296<=b&&56319>=b){e=a.charCodeAt(k+1);if(!(56320<=e&&57343>=e))continue;e=65536+(b-55296<<10)+(e-56320);b=917504>e?113828>e?113824>e?null:"invisible_format":119155>e?null:119163>e?"invisible_format":null:917760>e?917632>e?"tag":"invisible_format":918E3>e?"variation_selector":921600>e?"invisible_format":null;k++}else b=8192>b?174>b?32>b?11>b?1>b?"nul":9>b?"c0_control":null:13>b?"c0_control":14>b?null:"c0_control":160>b?127>b?null:128>b?"c0_control":"c1_control":
-161>b?"nonbreaking_space":173>b?null:"invisible_format":4449>b?1564>b?847>b?null:848>b?"invisible_format":null:1565>b?"bidi_control":4447>b?null:"hangul_filler":6155>b?6068>b?null:6070>b?"invisible_format":null:6159>b?6158>b?"variation_selector":"invisible_format":6160>b?"variation_selector":null:8304>b?8240>b?8208>b?8203>b?"nonbreaking_space":8206>b?"zero_width":"bidi_control":8234>b?null:8239>b?"bidi_control":"nonbreaking_space":8289>b?8287>b?null:8288>b?"nonbreaking_space":"zero_width":8294>b?
-"invisible_format":8298>b?"bidi_control":"invisible_format":65040>b?12644>b?12288>b?null:12289>b?"nonbreaking_space":null:12645>b?"hangul_filler":65024>b?null:"variation_selector":65440>b?65279>b?null:65280>b?"bom":null:65520>b?65441>b?"hangul_filler":null:65529>b?"invisible_format":null;null!==b&&(d[b]++,h++,f.length<c&&(e={offset:m,line:g,column:m-l+1,codePoint:e,category:b},"bom"===b&&(e.atStart=0===m),f.push(e)))}return{findings:f,counts:d,total:h,truncated:h>f.length}}
-function N(a){if("string"!==typeof a)throw new TypeError("getNewlineStats expects a string input.");var c=0;let d=0,f=0;for(let h=0,g=a.length;h<g;h++){const l=a.charCodeAt(h);13===l?10===a.charCodeAt(h+1)?(c++,h++):d++:10===l&&f++}a={crlf:c,cr:d,lf:f};c=[];0<a.crlf&&c.push("CRLF");0<a.cr&&c.push("CR");0<a.lf&&c.push("LF");return{...a,types:c,mixed:1<c.length}}
-function P(a){return{purgeInvisibleChars:a.hasInvisibleChars,purgeEmojis:a.hasEmojis,nukeControls:a.hasControlChars,keyboardOnlyFilter:a.hasNonKeyboardChars,normalizeNewlines:a.hasMixedNewlines||"CRLF"===a.newlineStyle||"CR"===a.newlineStyle}}export { t as summonSanctifier, M as inspectText, P as getRecommendedSanctifierOptions, O as inspectTextDetailed, N as getNewlineStats };
+var y=/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F-\u009F\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]+/g,d=/[\u00A0\u00AD\u034F\u061C\u115F\u1160\u17B4\u17B5\u180B-\u180F\u2000-\u200F\u202A-\u202F\u205F-\u206F\u3000\u3164\uFE00-\uFE0F\uFEFF\uFFA0\uFFF0-\uFFF8\u{1BCA0}-\u{1BCA3}\u{1D173}-\u{1D17A}\u{E0000}-\u{E0FFF}]+/gu;function A(e){return e<8192?D(e):v(e)}function D(e){return e<174?e<32?e<11?e<1?"nul":e<9?"c0_control":null:e<13?"c0_control":e<14?null:"c0_control":e<160?e<127?null:e<128?"c0_control":"c1_control":e<161?"nonbreaking_space":e<173?null:"invisible_format":e<4449?e<1564?e<847?null:e<848?"invisible_format":null:e<1565?"bidi_control":e<4447?null:"hangul_filler":e<6155?e<6068?null:e<6070?"invisible_format":null:e<6159?e<6158?"variation_selector":"invisible_format":e<6160?"variation_selector":null}function v(e){return e<8304?e<8240?e<8208?e<8203?"nonbreaking_space":e<8206?"zero_width":"bidi_control":e<8234?null:e<8239?"bidi_control":"nonbreaking_space":e<8289?e<8287?null:e<8288?"nonbreaking_space":"zero_width":e<8294?"invisible_format":e<8298?"bidi_control":"invisible_format":e<65040?e<12644?e<12288?null:e<12289?"nonbreaking_space":null:e<12645?"hangul_filler":e<65024?null:"variation_selector":e<65440?e<65279?null:e<65280?"bom":null:e<65520?e<65441?"hangul_filler":null:e<65529?"invisible_format":null}function I(e){return e<917504?e<113828?e<113824?null:"invisible_format":e<119155?null:e<119163?"invisible_format":null:e<917760?e<917632?"tag":"invisible_format":e<918e3?"variation_selector":e<921600?"invisible_format":null}var g=/[^\x20-\x7E\r\n]+/gu,B=/[\u2018\u2019\u201A\u201B\u2032\u2035]/g,G=/[\u201C\u201D\u201E\u201F\u2033\u2036\u00AB\u00BB]/g,X=/[\u2012\u2013\u2014\u2015\u2212]/g,k=/\u2026/g,O=/[\u2022\u00B7]/g,P=/[\uFF01-\uFF5E]/g;function S(e){return e.replace(B,"'").replace(G,'"').replace(X,"-").replace(k,"...").replace(O,"*").replace(P,r=>String.fromCharCode(r.charCodeAt(0)-65248))}function M(){try{return new RegExp("(?:\\p{Extended_Pictographic}(?:\\uFE0F|\\uFE0E)?(?:\\u200D(?:\\p{Extended_Pictographic}|\\w)+)*)","gu")}catch{return/[\u{2700}-\u{27BF}\u{1F300}-\u{1FAFF}\u{1F3FB}-\u{1F3FF}\u200D\uFE0F\u{1F1E6}-\u{1F1FF}]/gu}}var x=M();function m(e,r){return e.lastIndex=0,e.test(r)}function U(e){let r=S(e);if(!m(g,r))return!1;for(let n of r)if(!(n===`
+`||n==="\r"||n>=" "&&n<="~")&&!m(x,n))return!0;return!1}function j(e){if(typeof e!="string")throw new TypeError("inspectText expects a string input.");let r=[],n={hasControlChars:!1,hasInvisibleChars:!1,hasMixedNewlines:!1,newlineStyle:null,hasEmojis:!1,hasNonKeyboardChars:!1,summary:r},u=(s,c,f)=>{s&&(n[c]=!0,r.push(f))};u(m(y,e),"hasControlChars","Control characters detected."),u(m(d,e),"hasInvisibleChars","Invisible Unicode characters detected."),u(m(x,e),"hasEmojis","Emojis detected.");let{mixed:o,types:i}=b(e);return n.hasMixedNewlines=o,n.newlineStyle=o?"Mixed":i[0]||null,n.newlineStyle&&r.push(o?"Mixed newline styles detected.":`Consistent newline style: ${n.newlineStyle}`),u(U(e),"hasNonKeyboardChars","Non-keyboard characters detected."),n}var z=100;function C(e,r={}){if(typeof e!="string")throw new TypeError("inspectTextDetailed expects a string input.");let n=r&&r.maxFindings,u=n===1/0||Number.isInteger(n)&&n>=0?n:z,o={bidi_control:0,zero_width:0,nonbreaking_space:0,bom:0,nul:0,c0_control:0,c1_control:0,invisible_format:0,hangul_filler:0,variation_selector:0,tag:0},i=[],s=0,c=1,f=0;for(let a=0,h=e.length;a<h;a++){let l=e.charCodeAt(a);if(l>=32&&l<127)continue;if(l===10){c++,f=a+1;continue}if(l===13){e.charCodeAt(a+1)!==10&&(c++,f=a+1);continue}let E=a,F=l,p;if(l>=55296&&l<=56319){let t=e.charCodeAt(a+1);if(!(t>=56320&&t<=57343))continue;F=65536+(l-55296<<10)+(t-56320),p=I(F),a++}else p=A(l);if(p!==null&&(o[p]++,s++,i.length<u)){let t={offset:E,line:c,column:E-f+1,codePoint:F,category:p};p==="bom"&&(t.atStart=E===0),i.push(t)}}return{findings:i,counts:o,total:s,truncated:s>i.length}}function b(e){if(typeof e!="string")throw new TypeError("getNewlineStats expects a string input.");let r=0,n=0,u=0;for(let s=0,c=e.length;s<c;s++){let f=e.charCodeAt(s);f===13?e.charCodeAt(s+1)===10?(r++,s++):n++:f===10&&u++}let o={crlf:r,cr:n,lf:u},i=[];return o.crlf>0&&i.push("CRLF"),o.cr>0&&i.push("CR"),o.lf>0&&i.push("LF"),{...o,types:i,mixed:i.length>1}}function K(e){return{purgeInvisibleChars:e.hasInvisibleChars,purgeEmojis:e.hasEmojis,nukeControls:e.hasControlChars,keyboardOnlyFilter:e.hasNonKeyboardChars,normalizeNewlines:e.hasMixedNewlines||e.newlineStyle==="CRLF"||e.newlineStyle==="CR"}}function H(e={}){let r=!!e.normalizeSpaces,n=!!e.purgeZeroWidthSpaces,u=!!e.purgeInvisibleChars,o=!!e.purgeEmojis,i=!!e.nukeControls,s=!!e.keyboardOnlyFilter,c=!!e.normalizeNewlines,f=!!e.trimSpacesAroundNewlines,a=!!e.collapseNewLines,h=!!e.preserveParagraphs,l=!!e.collapseSpaces,E=!!e.finalTrim,F=Number.isFinite(e.maxLength)&&e.maxLength>=0?e.maxLength:1/0,p=e.onMaxLength==="truncate"||e.onMaxLength==="noop"||e.onMaxLength==="throw"?e.onMaxLength:"throw";return t=>_(t,u,o,i,s,c,f,a,h,l,E,F,p,r,n)}function R(e){return _(e,!0,!0,!0,!1,!0,!0,!0,!1,!0,!0)}function w(e){return _(e,!1,!1,!1,!1,!0,!0,!0,!0,!0,!0)}function N(e){return _(e,!1,!1,!1,!0,!0,!0,!1,!1,!1,!0)}function T(e){return _(e,!0,!0,!0,!0,!0,!0,!0,!1,!0,!0)}function L(e){return _(e,!1,!1,!1,!1,!0,!1,!1,!1,!1,!1,1/0,"throw",!0,!0)}var W=Object.assign(H,{strict:R,loose:w,keyboardOnlyEmoji:N,keyboardOnly:T,pasted:L});function _(e,r=!1,n=!1,u=!1,o=!1,i=!1,s=!1,c=!1,f=!1,a=!1,h=!1,l=1/0,E="throw",F=!1,p=!1){if(typeof e!="string")throw new TypeError("sanctifyText expects a string input.");if(E!=="truncate"&&E!=="noop"&&E!=="throw"&&(E="throw"),e.length>l)switch(E){case"truncate":e=e.slice(0,l);break;case"noop":return e;case"throw":default:throw new RangeError(`sanctifyText input length ${e.length} exceeds maxLength ${l}.`)}let t=e;return F&&(t=t.replace(J," ")),p&&(t=t.replace(V,"")),r&&(t=Z(t)),n&&(t=q(t)),u&&(t=le(t)),o&&(t=Y(t,n)),i&&(t=ee(t)),s&&(t=ne(t)),c&&(t=oe(t,f)),a&&(t=se(t)),h?t.trim():t}var J=/[\u00A0\u2000-\u200A\u202F\u205F]/g,V=/[\u200B\u2060\uFEFF]/g;function Z(e){return e.replace(d,"")}function $(e){if(x.lastIndex=0,!x.test(e))return"";x.lastIndex=0;let r="";for(let n of e.matchAll(x))r+=n[0];return r}function Y(e,r=!1){let n=S(e);return r?n.replace(g,""):n.replace(g,u=>$(u))}function q(e){return e.replace(x,"")}var Q=/\r\n|\r|\n/g;function ee(e,r=`
+`){return e.replace(Q,r)}var re=/[ \t]*(\n+)[ \t]*/g;function ne(e){return e.replace(re,"$1")}var te=/\n{2,}/g,ie=/\n{3,}/g;function oe(e,r){return r?e.replace(ie,`
+
+`):e.replace(te,`
+`)}var ue=/ {2,}/g;function se(e){return e.replace(ue," ")}function le(e){return e.replace(y,"")}function ae(e,r={}){if(typeof e!="string")throw new TypeError("revealInvisible expects a string input.");let n=r&&typeof r.format=="function"?r.format:i=>"<U+"+i.toString(16).toUpperCase().padStart(4,"0")+">",u="",o=0;for(let i of C(e,{maxFindings:1/0}).findings)u+=e.slice(o,i.offset)+n(i.codePoint,i.category),o=i.offset+(i.codePoint>65535?2:1);return u+e.slice(o)}var fe="1.3.6";export{fe as VERSION,b as getNewlineStats,K as getRecommendedSanctifierOptions,j as inspectText,C as inspectTextDetailed,ae as revealInvisible,T as sanctifyKeyboardOnly,N as sanctifyKeyboardOnlyEmoji,w as sanctifyLoose,L as sanctifyPasted,R as sanctifyStrict,W as summonSanctifier};

package/dist/types/hiddenText.d.ts

@@ -7,17 +7,26 @@
  */
 export function decodeTagText(text: string): string;
 /**
- * Finds runs of tag characters, variation selectors and invisible operators.
+ * Finds runs of tag characters, variation selectors, invisible operators and reserved
+ * tag-plane code points (U+E0080–U+E00FF, U+E01F0–U+E0FFF).
  * Consecutive characters of the same kind form one run. Two kinds of sequence are
  * legitimate and listed under `benign` instead of `runs`:
  *  - the subdivision flags of England, Scotland and Wales: U+1F3F4 followed by the
  *    tags for "gbeng", "gbsct" or "gbwls" and U+E007F
  *  - a single U+FE0E or U+FE0F (text or emoji presentation) after another character
+ *
+ * `line` and `column` are counted as in inspectTextDetailed. Every run and benign
+ * span is counted; only the first `maxRuns` of each are listed. With the caps set,
+ * memory stays flat however large the input is. One pass over the string, no copy.
+ * Set both caps for untrusted input: without them, results grow with the input.
+ *
  * @param {string} text
+ * @param {HiddenScanOptions} [options]
  * @returns {HiddenTextScan}
  */
-export function findHiddenRuns(text: string): HiddenTextScan;
-export type HiddenRunKind = "tag" | "variation_selector" | "invisible_operator";
+export function findHiddenRuns(text: string, options?: HiddenScanOptions): HiddenTextScan;
+export { VERSION } from "./version.generated.js";
+export type HiddenRunKind = "tag" | "variation_selector" | "invisible_operator" | "reserved_tag_plane";
 export type HiddenRun = {
     kind: HiddenRunKind;
     /**
@@ -33,18 +42,65 @@ export type HiddenRun = {
      */
     count: number;
     /**
-     * - tag runs only: the ASCII the tags spell (see decodeTagText)
+     * - 1-based line of the first character (as in inspectTextDetailed)
+     */
+    line: number;
+    /**
+     * - 1-based column of the first character, in UTF-16 units
+     */
+    column: number;
+    /**
+     * - tag runs only: the ASCII the tags spell (see decodeTagText),
+     * at most `maxDecodedLength` UTF-16 units
      */
     decoded?: string;
+    /**
+     * - tag runs only: true when `decoded` stops short of the run
+     */
+    decodedTruncated?: boolean;
+};
+export type HiddenRunCounts = {
+    tag: number;
+    variation_selector: number;
+    invisible_operator: number;
+    reserved_tag_plane: number;
 };
 export type HiddenTextScan = {
     /**
-     * - runs that may hide text, in order
+     * - runs that may hide text, in order, at most `maxRuns` long
      */
     runs: HiddenRun[];
     /**
      * - [offset, end) spans of legitimate
-     * sequences (subdivision flags, a single emoji or text presentation selector)
+     * sequences (subdivision flags, a single emoji or text presentation selector), at most `maxRuns` long
      */
     benign: Array<[number, number]>;
+    /**
+     * - every run per kind, listed or not
+     */
+    counts: HiddenRunCounts;
+    /**
+     * - sum of `counts`
+     */
+    total: number;
+    /**
+     * - every benign span, listed or not
+     */
+    benignTotal: number;
+    /**
+     * - true when runs or benign spans were left out of the lists
+     */
+    truncated: boolean;
+};
+export type HiddenScanOptions = {
+    /**
+     * - cap on listed runs, and separately on listed benign spans
+     * (non-negative integer or Infinity). Counts stay exact.
+     */
+    maxRuns?: number;
+    /**
+     * - cap on each run's `decoded` text, in UTF-16 units.
+     * For both, any other value (negative, fractional, NaN, a string) means the default, no cap.
+     */
+    maxDecodedLength?: number;
 };

package/dist/types/index.d.ts

@@ -6,9 +6,17 @@ export type DetailedTrashReport = import("./inspectText.js").DetailedTrashReport
 export type InspectDetailedOptions = import("./inspectText.js").InspectDetailedOptions;
 export type NewlineStats = import("./inspectText.js").NewlineStats;
 export type SanctifyOptions = import("./sanctifyText.js").SanctifyOptions;
+export type RevealOptions = import("./reveal.js").RevealOptions;
 import { inspectText } from './inspectText.js';
 import { inspectTextDetailed } from './inspectText.js';
 import { getNewlineStats } from './inspectText.js';
 import { getRecommendedSanctifierOptions } from './inspectText.js';
 import { summonSanctifier } from './sanctifyText.js';
-export { inspectText, inspectTextDetailed, getNewlineStats, getRecommendedSanctifierOptions, summonSanctifier };
+import { sanctifyStrict } from './sanctifyText.js';
+import { sanctifyLoose } from './sanctifyText.js';
+import { sanctifyKeyboardOnlyEmoji } from './sanctifyText.js';
+import { sanctifyKeyboardOnly } from './sanctifyText.js';
+import { sanctifyPasted } from './sanctifyText.js';
+import { revealInvisible } from './reveal.js';
+import { VERSION } from './version.generated.js';
+export { inspectText, inspectTextDetailed, getNewlineStats, getRecommendedSanctifierOptions, summonSanctifier, sanctifyStrict, sanctifyLoose, sanctifyKeyboardOnlyEmoji, sanctifyKeyboardOnly, sanctifyPasted, revealInvisible, VERSION };

package/dist/types/reveal.d.ts

@@ -0,0 +1,26 @@
+/** @typedef {import('./inspectText.js').TrashCategory} TrashCategory */
+/**
+ * @typedef {Object} RevealOptions
+ * @property {(codePoint: number, category: TrashCategory) => string} [format] - Text shown in place of
+ *   each hidden character. Default: `<U+XXXX>` (at least four uppercase hex digits).
+ */
+/**
+ * Makes hidden characters visible: every character `inspectTextDetailed` would report is
+ * replaced by a marker such as `<U+200B>`, and everything else is kept as it is,
+ * including tabs, newlines and emoji. Use it to show hidden text in logs, diffs, review
+ * tools and agent tool output. A literal `<U+200B>` already in the text looks the same as
+ * a marker; pass `format` to choose markers that can't occur in your input.
+ *
+ * @param {string} text
+ * @param {RevealOptions} [options]
+ * @returns {string}
+ */
+export function revealInvisible(text: string, options?: RevealOptions): string;
+export type TrashCategory = import("./inspectText.js").TrashCategory;
+export type RevealOptions = {
+    /**
+     * - Text shown in place of
+     * each hidden character. Default: `<U+XXXX>` (at least four uppercase hex digits).
+     */
+    format?: (codePoint: number, category: TrashCategory) => string;
+};