@link-foundation/sensitive-data-sanitizer 0.19.2 → 0.20.0

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.20.0
+
+### Minor Changes
+
+- befe673: Match the strongest features of competing secret and PII tools. Add national, tax, bank and device identifiers from Presidio, OpenRedaction, DataFog and Google; 203 vendored Betterleaks rules; offline CRC32 checks for GitHub and npm tokens (`tokenChecksum`, `tokenWithChecksum`); escape-run, HTML-tag and UTF-16 decoding; `allowList`, `inlineAllow` and `minConfidenceByType`; mask tuning and deterministic `encrypt` with `decrypt`; `createPatternDetector`, `createIdentifierDetector`, `identifierChecks` and the read-only `nationalRules` table; SARIF output (`toSarif`, `scan --format sarif`), multi-path `scan` and pre-commit hooks. Detect an IPv4 address followed by a full stop. Add a competitor comparison with measured results on 853 competitor test cases.
+
+## 0.19.3
+
+### Patch Changes
+
+- 6bdaf11: Recognize additional inflected special-category attributes and unfamiliar values after explicit category cues in strict mode. Require personal context for ambiguous English attributes so operational logs and JSON statuses stay intact.
+
 ## 0.19.2
 
 ### Patch Changes

package/README.md

@@ -1,6 +1,6 @@
 # Sensitive Data Sanitizer
 
-An offline JavaScript library, detector framework, and CLI for redacting credentials and personal data from logs, AI sessions, and text files, including logs with invalid UTF-8 bytes. It combines required Secretlint and 221 bundled Gitleaks text rules with contextual credentials, default multilingual name/PII recognition, known values and bounded encoding detection. Optional local Gitleaks, TruffleHog, and Presidio detectors extend the union.
+An offline JavaScript library, detector framework, and CLI for redacting credentials and personal data from logs, AI sessions, and text files, including logs with invalid UTF-8 bytes. It combines required Secretlint, 221 bundled Gitleaks text rules and 203 further Betterleaks rules with contextual credentials, default multilingual name/PII recognition, known values and bounded encoding detection. Optional local Gitleaks, TruffleHog, and Presidio detectors extend the union.
 
 Library calls without a profile replace every actionable finding completely with `[REDACTED]`. Personal findings below the default confidence threshold of 0.5 remain inspectable without replacing ordinary log numbers; credentials always qualify. Publication methods verify full redaction and block output if a required detector fails or finds a residual. Optional typed transformations, compatible partial masks and encoding preservation are described in the API. Findings contain offsets and rule metadata; they omit matched values and source excerpts.
 
@@ -131,6 +131,7 @@ console.log(result.text); // password=[REDACTED] [REDACTED]
 printf 'password=short\n' | sensitive-data-sanitizer redact -
 sensitive-data-sanitizer redact input.txt --in-place
 sensitive-data-sanitizer scan logs --config /private/policy.json
+sensitive-data-sanitizer scan src/a.js src/b.js --format sarif > report.sarif
 sensitive-data-sanitizer redact input.txt --gitleaks --trufflehog --output safe.txt
 sensitive-data-sanitizer redact input.txt --presidio /path/presidio-bridge.py --model en_core_web_sm --language en
 sensitive-data-sanitizer redact large-session.jsonl --jsonl --stream --output safe-session.jsonl
@@ -148,6 +149,8 @@ sensitive-data-sanitizer --help
 
 Defaults: 32 MiB whole-text input bytes, 100,000 findings and 10,000 files. `--stream` permits 1 GiB total with plain-text records up to the total input limit or 8 MiB JSONL records, bounded workers and atomic publication. `--worker-timeout-ms N` sets the worker inactivity/per-batch deadline (default 60000 ms); verified file progress resets the deadline, so large files can run longer overall. `--max-bytes` sets the byte limit; JSON config sets the other sanitizer limits. `--paranoid` adds entropy heuristics and increases false positives. `--native-only` explicitly disables Secretlint.
 
+`scan` accepts several paths and `--format sarif` emits SARIF 2.1.0 for code-scanning viewers. The repository's `.pre-commit-hooks.yaml` provides `sensitive-data-sanitizer` (credentials) and `sensitive-data-sanitizer-pii` hooks for [pre-commit](https://pre-commit.com). UTF-16 files with a byte order mark are read and written back in their own encoding.
+
 ## Configuration
 
 For AI sessions, `--jsonl` preserves JSON records and declares `id`, `type`,
@@ -195,6 +198,8 @@ Keep policies containing real private values outside Git. Built-in exact public
 }
 ```
 
+Reviewed false positives can be kept with `allowList` (whole-value strings or anchored regexes) or, opt-in, with `inlineAllow: true`, which honours `gitleaks:allow`, `pragma: allowlist secret` and similar comments; credentials stay redacted unless `allowCredentialDisable: true` is also set. `minConfidenceByType` sets per-type thresholds, the `mask` transformation accepts `char`, `ignore`, `ignoreClasses`, `count` and `fromEnd`, and `{mode: 'encrypt', key}` produces deterministic AES-256-GCM tokens that `decrypt(text, key)` restores. See [the API](docs/API.md#allow-lists-suppression-comments-and-template-values) and [the competitor comparison](docs/COMPARISON.md).
+
 Custom policies preserve exact reviewed `PERSON`, `ORGANIZATION` and `EMAIL` detections. Built-in public resolvers/documentation IPs are also exempt. Credential detections always win, including a public email used as a password or authorization value. Evidence is an explicit caller decision and does not make every occurrence harmless.
 
 ## Extend detection
@@ -218,6 +223,8 @@ const sanitizer = createSanitizer({
 
 Install external binaries and language models separately. TruffleHog runs with verification and updates disabled. The Presidio example checks for an installed model and never downloads it at analysis time. Each selected detector is required, including during residual verification. See [the API and adapter contract](docs/API.md), [combined scanners](examples/combined-scanners.mjs), and [the local NER bridge](examples/presidio-bridge.py).
 
+`createPatternDetector([{type, patterns, denyList, context, validate}])` adds Presidio-style pattern recognizers or Google-style custom infoTypes with hotword boosts, and `createIdentifierDetector` adds national identifiers with checksum validators; see [declarative pattern detectors](docs/API.md#declarative-pattern-detectors).
+
 The framework also accepts Gitleaks, TruffleHog, detect-secrets, Presidio, or explicitly typed offset reports. Cloud reports can be converted locally; the package itself makes no cloud requests.
 
 For CLI name recognition, `--presidio PATH` selects a local Python JSON bridge. `--python COMMAND` selects its interpreter, and `--model` / `--language` select an already installed model. These options require the bridge and participate in final-output verification.

package/bin/sensitive-data-sanitizer.js

@@ -6,9 +6,11 @@ import { createSanitizer } from '../src/engines.js';
 import { failure } from '../src/detection.js';
 import {
   readTextFile,
+  readTextInput,
   atomicWrite,
   listTextCandidates,
-  decodeText,
+  decodeTextInput,
+  encodeTextOutput,
 } from '../src/files.js';
 import { auditGitHistory } from '../src/history.js';
 import { rewriteGitHistory } from '../src/history-rewrite.js';
@@ -17,7 +19,7 @@ import {
   knownSecretsFromEnv,
   knownSecretsFromGitHubConfig,
 } from '../src/known.js';
-import { encodeLogText } from '../src/log-bytes.js';
+import { toSarif } from '../src/sarif.js';
 import { createWikidataVerifier } from '../src/public-verifier.js';
 import { streamCommand } from './stream.js';
 import { sessionStructuralFields } from '../src/structured.js';
@@ -31,7 +33,8 @@ import {
 const help = `Usage: sensitive-data-sanitizer <redact|scan|restore|history> [path|-] [options]
 
 redact    Sanitize one text log or stdin; publish only after verification
-scan      Report metadata for a file, directory, or stdin (never source text)
+scan      Report metadata for files, directories, or stdin (never source text);
+          several paths may follow scan (for example staged files)
 restore   Restore placeholders in a file using --map PATH (private output)
 history   Audit local reachable Git blobs, commit/tag metadata, and reflogs
 history rewrite SOURCE --output DIRECTORY [--apply]
@@ -67,6 +70,7 @@ history rewrite SOURCE --output DIRECTORY [--apply]
 --no-map            Discard placeholder mapping after full anonymization
 --placeholder-format TEMPLATE  Default {Type}-{n}; also [{TYPE}_{n}]
 --map-format FORMAT json or lino (otherwise inferred from .lino extension)
+--format FORMAT     scan report: json (default) or sarif (SARIF 2.1.0)
 --map-credentials   Explicitly keep credentials in the private restore map
 --max-record-bytes N Maximum record bytes (text: total limit; JSONL: 8388608)
 --hive-mask         Opt-in first/last 3 mask for values longer than 12 characters
@@ -129,8 +133,9 @@ function parseArgs(argv) {
     ['--map', 'map'],
     ['--map-format', 'mapFormat'],
     ['--placeholder-format', 'placeholderFormat'],
+    ['--format', 'format'],
   ]);
-  let hasPath = false;
+  const paths = [];
   for (let i = rewrite ? 2 : 1; i < argv.length; i++) {
     const arg = argv[i];
     if (boolean.has(arg)) {
@@ -140,13 +145,18 @@ function parseArgs(argv) {
         throw failure('ERR_ARGUMENT');
       }
       config[valued.get(arg)] = argv[++i];
-    } else if (!hasPath && !arg.startsWith('--')) {
-      config.path = arg;
-      hasPath = true;
+    } else if (!arg.startsWith('--')) {
+      paths.push(arg);
     } else {
       throw failure('ERR_ARGUMENT');
     }
   }
+  // Only scan accepts several paths (for example staged files from a hook).
+  if (paths.length > 1 && (command !== 'scan' || paths.includes('-'))) {
+    throw failure('ERR_ARGUMENT');
+  }
+  config.path = paths[0] ?? '-';
+  config.paths = paths.length ? paths : ['-'];
   validateArgs(config);
   return config;
 }
@@ -194,7 +204,11 @@ function validateFeatureArgs(config) {
 
 function validateStructuredArgs(config) {
   validateStreamArgs(config);
-  if (config.twoPass && !config.stream) {
+  if (
+    (config.twoPass && !config.stream) ||
+    (config.format !== undefined &&
+      (config.command !== 'scan' || !['json', 'sarif'].includes(config.format)))
+  ) {
     throw failure('ERR_ARGUMENT');
   }
   if ((config.json && config.jsonl) || (config.stream && config.json)) {
@@ -249,7 +263,7 @@ async function stdinText(stream, maxBytes, options) {
     }
     chunks.push(buffer);
   }
-  return decodeText(Buffer.concat(chunks), options);
+  return decodeTextInput(Buffer.concat(chunks), options);
 }
 
 export async function runCli(
@@ -266,9 +280,7 @@ export async function runCli(
       return 0;
     }
     if (['--version', '-v'].includes(argv[0])) {
-      stdout.write(
-        `${JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version}\n`
-      );
+      stdout.write(`${packageVersion()}\n`);
       return 0;
     }
     const config = parseArgs(argv);
@@ -422,24 +434,20 @@ function inputText(path, config, stdin) {
   };
   return path === '-'
     ? stdinText(stdin, config.maxBytes, options)
-    : readTextFile(path, config.maxBytes, options);
+    : readTextInput(path, config.maxBytes, options);
 }
 
 async function redactCommand(config, engine, { stdin, stdout }) {
-  const result = await engine.sanitize(
-    await inputText(config.path, config, stdin)
-  );
+  const input = await inputText(config.path, config, stdin);
+  const result = await engine.sanitize(input.text);
   await saveMap(config);
+  const output = encodeTextOutput(result.text, input.encoding);
   if (config.output || config.inPlace) {
-    await atomicWrite(
-      config.output ?? config.path,
-      encodeLogText(result.text),
-      {
-        replace: Boolean(config.inPlace),
-      }
-    );
+    await atomicWrite(config.output ?? config.path, output, {
+      replace: Boolean(config.inPlace),
+    });
   } else {
-    stdout.write(encodeLogText(result.text));
+    stdout.write(output);
   }
   return 0;
 }
@@ -464,8 +472,10 @@ async function historyCommand(config, engine, { stdout }) {
 }
 
 async function scanCommand(config, engine, { stdin, stdout }) {
-  const files =
-    config.path === '-' ? ['-'] : await listTextCandidates(config.path);
+  const files = [];
+  for (const path of config.paths) {
+    files.push(...(path === '-' ? ['-'] : await listTextCandidates(path)));
+  }
   const results = [],
     skipped = [];
   for (const file of files) {
@@ -473,7 +483,9 @@ async function scanCommand(config, engine, { stdin, stdout }) {
     try {
       results.push({
         path,
-        findings: await engine.inspect(await inputText(file, config, stdin)),
+        findings: await engine.inspect(
+          (await inputText(file, config, stdin)).text
+        ),
       });
     } catch (error) {
       if (['ERR_BINARY', 'ERR_ENCODING', 'ERR_LIMIT'].includes(error.code)) {
@@ -484,7 +496,10 @@ async function scanCommand(config, engine, { stdin, stdout }) {
     }
   }
   const complete = skipped.length === 0;
-  stdout.write(`${JSON.stringify({ complete, results, skipped }, null, 2)}\n`);
+  const report = { complete, results, skipped };
+  stdout.write(
+    `${JSON.stringify(config.format === 'sarif' ? toSarif(report, { version: packageVersion() }) : report, null, 2)}\n`
+  );
   return !complete ? 2 : results.some((r) => r.findings.length) ? 1 : 0;
 }
 
@@ -495,6 +510,12 @@ if (
   process.exitCode = await runCli(process.argv.slice(2));
 }
 
+function packageVersion() {
+  return JSON.parse(
+    readFileSync(new URL('../package.json', import.meta.url), 'utf8')
+  ).version;
+}
+
 function defaultByteLimit(config) {
   return config.stream ? 1073741824 : 33554432;
 }

package/docs/API.md

@@ -16,8 +16,12 @@ Import the ESM root entry; complete declarations are in `src/index.d.ts`. Inputs
 | `createSyncSanitizer(options)`         | Reusable synchronous `inspect` / `sanitize` with the same policies                                           |
 | `compileKnown(...lists)`               | Copy, freeze, merge and precompile caller-known personal values                                              |
 | `auditGitHistory(repository, options)` | Read-only audit of local reachable text objects                                                              |
+| `decrypt(text, key, options)`          | Restore values from `[ENCRYPTED:TYPE:…]` tokens written by the `encrypt` transformation                      |
+| `createPatternDetector(specs, opts)`   | Declarative regex, deny-list, hotword and validator detector                                                 |
+| `createIdentifierDetector(rules)`      | Detector for caller-supplied national identifier rules (`nationalRules` shape)                               |
+| `toSarif(report, options)`             | Convert a scan report or findings to SARIF 2.1.0 without values                                              |
 
-`redactions` counts merged replacement spans. `findings` can contain several detector hits for one replacement. Findings have `{start, end, type, category, rule, line, column}`; no text, quote, match, entropy score, provider details, or raw diagnostic is copied. Debugging is off by default; `debug(event)` receives value-free counts, rule timings, residual rounds and registry event metadata (see `DebugEvent` in the TypeScript declarations).
+`redactions` counts merged replacement spans. `findings` can contain several detector hits for one replacement. Findings have `{start, end, type, category, rule, line, column}`; no text, quote, match, entropy score, provider details, or raw diagnostic is copied. The optional `strict-attributes` debug event reports only detected and suppressed match counts. Debugging is off by default; `debug(event)` receives value-free counts, rule timings, residual rounds and registry event metadata (see `DebugEvent` in the TypeScript declarations).
 
 ## Options
 
@@ -242,6 +246,7 @@ Without a profile, full redaction is the library default. `transformation` selec
 | `fake`              | Keyed realistic names, valid dates/MRZ/checksummed numbers and reserved contact destinations                           |
 | `date-shift`        | Personal ISO dates; finite integer `days`, UTC calendar arithmetic; invalid dates become opaque                        |
 | `bucket`            | Personal safe integers; positive integer `size`, inclusive numeric interval                                            |
+| `encrypt`           | Personal data only; deterministic AES-256-GCM token reversible with `decrypt` and the key (see below)                  |
 
 All keyed modes require a key of at least 16 UTF-8 bytes. `format-preserving` is a non-reversible shape pseudonym, not NIST FF1 encryption; it does not preserve checksum validity, country-specific alphabets or guarantee uniqueness. Partial masks, date shifts and buckets intentionally retain information. Keep pseudonym keys outside logs. Residual verification checks the corresponding fully redacted representation; opted-in disclosures can remain recognizable in the transformed output. Use default redaction for publication that requires opaque output.
 
@@ -368,7 +373,7 @@ The optional ESLint plugin exports `require-sanitized-output` from the package's
 
 ## Native documented entity catalogs
 
-`entityCatalogs` exposes source URLs, source-page SHA256 and documented names for Google (213), Azure (176) and AWS (36). Their canonical uppercase underscore labels are recognized natively, with complete value spans, independent of cloud services. The package also implements selected automatic formats and checksum algorithms. Catalog vocabulary support does not reproduce proprietary NER, image/document inspection, regional identity validators, cloud risk analysis or their accuracy. See the coverage table for that distinction.
+`entityCatalogs` exposes source URLs, source-page SHA256 and documented names for Google (214), Azure (180) and AWS (36). Their canonical uppercase underscore labels are recognized natively, with complete value spans, independent of cloud services. The package also implements selected automatic formats and checksum algorithms. Catalog vocabulary support does not reproduce proprietary NER, image/document inspection, regional identity validators, cloud risk analysis or their accuracy. See the coverage table for that distinction.
 
 ## Two-pass session publication
 
@@ -742,3 +747,189 @@ one or both languages. Each invocation is limited to 1 Mi UTF-16 units; larger
 inputs throw `ERR_LIMIT`. `debug` is off by default and emits counts only. Core
 imports never load the optional entry or NLP.js; leaving it off adds no optional
 runtime/module cost.
+
+## Allow lists, suppression comments and template values
+
+`allowList` keeps detected values that the caller has reviewed as safe, as in
+Presidio's `allow_list`, Google's exclusion rules with
+`MATCHING_TYPE_FULL_MATCH` and DataFog's `allowlist_patterns`. A string must
+equal the whole detected value; a `RegExp` must match the whole value (it is
+anchored internally, `g`/`y` flags are ignored). A partial match never shortens
+a finding. At most 10 000 entries; anything else is `ERR_CONFIG`.
+
+```js
+sanitize('Contact support@example.org or alice@private.test', {
+  allowList: ['support@example.org', /.+@example\.org/],
+});
+// → 'Contact support@example.org or [REDACTED]'
+```
+
+`inlineAllow: true` honours the suppression comments of other scanners on the
+same line: `gitleaks:allow`, `betterleaks:allow`, `kingfisher:allow`/`ignore`,
+`trufflehog:ignore`, `sanitizer:allow`, detect-secrets/ripsecrets
+`pragma: allowlist secret` and SonarQube `NOSONAR`. A line containing
+`pragma: allowlist nextline secret` suppresses findings on the next line. It is
+off by default because anyone who can write to a log could otherwise disable
+redaction.
+
+Both controls follow the same safety rules:
+
+- declared `knownPersonal`/`knownSecrets` values, confirmed personal values and
+  structured `JSON_VALUE` denials are never exempted;
+- credentials are exempted only together with `allowCredentialDisable: true`;
+- they apply to text, structured and asynchronous paths alike.
+
+Built-in credential rules also skip values that consist only of a template or
+mask: `${PASSWORD}`, `$${PYPI_PASSWORD}`, `$DB_PASS`, `$1`, `<password>`,
+`<Openverse API token>`, `[PASSWORD]`, `****`, `XXXXXXXX`, `...`, optionally
+after `Bearer`/`Basic`/`token` and on either side of a `user:password` colon.
+A mixed value such as `alice:${DB_PASSWORD}` or `<x>abc123…` is still redacted.
+Findings from custom detectors are never dropped this way. The value-free debug
+event `{event: 'template-value', rule}` reports each skipped match.
+
+## Per-type thresholds
+
+`minConfidenceByType` maps finding types to a threshold that overrides
+`minConfidence` for those types only, as in Presidio score thresholds or
+Google's per-infoType likelihood:
+
+```js
+sanitize(text, { minConfidence: 0.5, minConfidenceByType: { PERSON: 0.8 } });
+```
+
+Values must be numbers from 0 to 1. Credentials are never filtered by a
+threshold.
+
+## Character masks
+
+The `mask` transformation accepts Presidio/Google-style tuning in addition to
+`keepStart`, `keepEnd`, `minLength` and `marker`:
+
+| Option          | Meaning                                                        |
+| --------------- | -------------------------------------------------------------- |
+| `char`          | One masking character (default `*`)                            |
+| `ignore`        | Characters that are never masked, e.g. `'-@.'`                 |
+| `ignoreClasses` | Any of `digits`, `upper`, `lower`, `punctuation`, `whitespace` |
+| `count`         | Mask at most this many characters                              |
+| `fromEnd`       | Mask the last `count` characters instead of the first          |
+
+```js
+sanitize('Card 4111-1111-1111-1111', {
+  transformation: { mode: 'mask', char: '#', ignore: '-', count: 12 },
+});
+// → 'Card ####-####-####-1111'
+```
+
+`marker` cannot be combined with the tuning options. A value with nothing left
+to mask is fully redacted. Masks apply to personal data only.
+
+## Deterministic encryption
+
+`{mode: 'encrypt', key}` replaces each personal value with
+`[ENCRYPTED:TYPE:…]`, comparable to Google's `CryptoDeterministicConfig` and
+Presidio's `encrypt` operator. The token is AES-256-GCM with a synthetic IV
+(HMAC of type and value), so equal values of one type produce equal tokens and
+joins or counts still work. Keys are at least 16 UTF-8 bytes and are expanded
+with HKDF-SHA256.
+
+```js
+import { sanitize, decrypt } from '@link-foundation/sensitive-data-sanitizer';
+const key = process.env.SANITIZER_KEY;
+const { text } = sanitize('Mail alice@private.test', {
+  transformation: { mode: 'encrypt', key },
+});
+decrypt(text, key); // → 'Mail alice@private.test'
+```
+
+`decrypt(text, key, {json: true})` escapes restored values inside JSON string
+literals. A token that fails authentication throws `ERR_MAP`; a non-string text
+throws `ERR_INPUT`. Credentials are still fully redacted and never encrypted.
+Anyone holding the key can recover the values, so treat encrypted output as
+pseudonymised, not anonymous.
+
+## Declarative pattern detectors
+
+`createPatternDetector(specs, {id})` builds a synchronous Detector from
+Presidio-style pattern recognizers or Google custom infoTypes without writing
+code:
+
+```js
+import {
+  createPatternDetector,
+  sanitize,
+} from '@link-foundation/sensitive-data-sanitizer';
+const detector = createPatternDetector([
+  {
+    type: 'MEDICAL_RECORD',
+    category: 'personal',
+    patterns: [{ regex: /\bMRN-\d{6}\b/g, confidence: 0.4 }],
+    denyList: ['Mr. Bean'],
+    context: ['mrn', 'medical record'],
+    contextWindow: { before: 64, after: 16 },
+    contextBoost: 0.35,
+    validate: (value) => value !== 'MRN-000000',
+  },
+]);
+sanitize('medical record MRN-123456', { detectors: [detector] });
+```
+
+| Field              | Meaning                                                                                        |
+| ------------------ | ---------------------------------------------------------------------------------------------- |
+| `type`, `category` | Uppercase type; `personal` or `credential`                                                     |
+| `rule`             | Lowercase rule id (default `custom-pattern`)                                                   |
+| `confidence`       | Base confidence (default 0.6)                                                                  |
+| `patterns`         | Up to 64 `RegExp`, strings or `{regex, confidence, group}` (a capture group narrows the span)  |
+| `denyList`         | Whole-word values, case-insensitive unless `caseSensitive: true`; `denyListConfidence` (1)     |
+| `context`          | Hotwords (strings or RegExp) searched in `contextWindow` (number or `{before, after}`, ≤ 4096) |
+| `contextBoost`     | Added when a hotword is present (default 0.35, floor 0.4); a negative value lowers confidence  |
+| `validate`         | `false` drops a match; a number sets its confidence                                            |
+
+Malformed specifications throw `ERR_CONFIG`.
+
+`createIdentifierDetector(rules, {id})` adapts rules with the shape of the
+exported `nationalRules` table (`{type, pattern, context, checksum, plausible?,
+valid?}`) so callers can add or replace national identifiers. `identifierChecks`
+exposes the checksum and format validators (Luhn, mod-11, mod-97, Verhoeff and
+the national algorithms) for reuse; a valid checksum never proves issuance.
+
+## Offline token checksums
+
+GitHub (`ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_`) and npm (`npm_`) tokens end in
+six base62 characters holding the CRC32 of the 30 characters after the first
+`_`. As in Kingfisher and osv-scalibr, the native `github`/`npm` rules and the
+vendored Gitleaks/Betterleaks rules give a checksum-valid token confidence
+0.99. A token whose checksum does not match keeps its ordinary confidence and
+is still redacted, because truncated or retyped tokens remain sensitive. No
+network request is made. The exported `tokenChecksum(value)` returns `true`,
+`false` or `undefined` (not a checksummed format), and
+`tokenWithChecksum(prefix, body)` builds checksum-valid test fixtures at
+runtime so repositories need not store token literals.
+
+## SARIF reports
+
+`toSarif(report, {version, path})` converts a CLI `scan` report, an array of
+`{path, findings}` or a findings array into SARIF 2.1.0 for GitHub code
+scanning and other CI viewers. Rules are keyed by `finding.rule`; credential
+findings are `error` and personal findings `warning`. Regions carry
+`charOffset`, `charLength`, `startLine` and `startColumn` in UTF-16 code units.
+Skipped files become tool execution notifications. Values are never included.
+
+```sh
+sensitive-data-sanitizer scan --format sarif src > results.sarif
+```
+
+`--format` accepts `json` (default) or `sarif` and is only valid for `scan`.
+
+CLI `scan` accepts several paths, so a CI step or a
+[pre-commit](https://pre-commit.com) hook can pass the staged files directly.
+The repository ships `.pre-commit-hooks.yaml` with two hooks:
+`sensitive-data-sanitizer` (credentials only) and
+`sensitive-data-sanitizer-pii` (credentials and personal data).
+
+```yaml
+repos:
+  - repo: https://github.com/link-foundation/sensitive-data-sanitizer
+    rev: v0.20.0
+    hooks:
+      - id: sensitive-data-sanitizer
+```

package/docs/COVERAGE.md

@@ -6,7 +6,7 @@ Coverage is a union of maintained detectors, explicit known values, context and
 
 The native provider supplements in `src/rules.js` cover GitHub, GitLab, OpenAI, Anthropic, AWS access IDs, Google API/OAuth, Slack, Stripe, SendGrid, Twilio, npm, PyPI, Telegram, Discord, Hugging Face, Shopify, Databricks, Square, Mailchimp, DigitalOcean, Vault, Grafana, Linear, Notion, Vercel, and JWT-shaped values. Prefix matches also protect selected truncated tokens; they do not validate a token with a provider.
 
-The default native engine bundles 221 text rules from Gitleaks 8.30.1 (222 upstream rules; the path-only PKCS12 rule is outside text detection), compiled unchanged with RE2/WASM or RE2JS for inputs above 4 MiB. Dense rule matches switch to RE2JS after at most eight WASM matches per view, bounding repeated whole-input copies without truncating secrets or changing anchors. The MIT license, commit and sync script are retained in `src/vendor/gitleaks`. Keyword prefilters and configured entropy thresholds run natively; upstream comment/path allowlists cannot suppress a sanitizer finding. Quoted hex/base64 entropy detection supplements those rules. The default asynchronous engine additionally uses the pinned Secretlint recommended scanners, excluding its comment-filter rule. Embedded disable directives in untrusted text cannot suppress findings. Optional Gitleaks and TruffleHog invoke their installed detector sets, and custom detectors can add formats. Service counts are not a measure of comparative recall.
+The default native engine bundles 221 text rules from Gitleaks 8.30.1 (222 upstream rules; the path-only PKCS12 rule is outside text detection), compiled unchanged with RE2/WASM or RE2JS for inputs above 1 MiB (re2-wasm has a fixed 16 MiB heap shared by every compiled rule and copied subject). Dense rule matches switch to RE2JS after at most eight WASM matches per view, bounding repeated whole-input copies without truncating secrets or changing anchors. The MIT license, commit and sync script are retained in `src/vendor/gitleaks`. Because Gitleaks is no longer updated, 203 further rules from its maintained successor Betterleaks 1.10.0 (ids absent from the Gitleaks set, such as Groq, Cerebras, xAI, Vercel, Supabase and Cloudflare v2 tokens) run the same way as `betterleaks/<id>` findings, with their MIT license, provenance and `scripts/sync-betterleaks.py` in `src/vendor/betterleaks`. Their entropy floors and secret-only `matchesAny`/`containsAny` filters are honoured; BPE token-ratio and nearby-line filters, live validation, validation-only components and the context-dependent `generic-*` rules are not, so these rules can only report more than Betterleaks, not less. Keyword prefilters and configured entropy thresholds run natively; upstream comment/path allowlists cannot suppress a sanitizer finding. Quoted hex/base64 entropy detection supplements those rules. The default asynchronous engine additionally uses the pinned Secretlint recommended scanners, excluding its comment-filter rule. Embedded disable directives in untrusted text cannot suppress findings. Optional Gitleaks and TruffleHog invoke their installed detector sets, and custom detectors can add formats. Service counts are not a measure of comparative recall.
 
 Context rules recognize credential labels with bounded key affixes, JSON and nested escaped JSON, YAML/env/INI assignments, shell flags, XML values, URL userinfo, authorization headers, cookies, webhook paths, and complete or truncated private-key PEM blocks. Low-entropy contextual values such as `password=1234` are protected. Generic token counters, booleans/null/language literals and ordinary Git hashes are excluded to retain useful logs. Supply literal known values for ambiguous credentials or unusual key names.
 
@@ -185,7 +185,7 @@ records standards, alternatives and regression coverage for each feature.
 | Local subprocess      | Default 30 seconds and 10 MiB report/stdout cap; failures block output                                |
 | Git objects           | 10,000 objects, 10 MiB per object and 100 MiB total by default                                        |
 
-Supported decoding includes canonical base64/base64url, line-wrapped base64, hex, percent, HTML entities, UTF-8 byte escapes and JSON string escapes, with fatal UTF-8 checks for decoded payloads. Plain log files and streams preserve invalid UTF-8 bytes reversibly while scanning surrounding valid text; structured JSON/JSONL and configuration remain strict. Opt-in encoding preservation round-trips sanitized payloads; nested escaped JSON retains its structure by default. Arbitrarily split fragments, unlimited nesting, compressed/encrypted payloads, malformed encodings, archives, PDFs and OCR/images are outside this text profile. Whole-input/candidate limits fail; oversized textual percent/base64 runs receive audited complete-run redaction. Unsupported representations may remain undetected. `decode: false` intentionally reduces coverage. A caller needing another format should decode it through a bounded preprocessing adapter and retain correct original spans.
+Supported decoding includes canonical base64/base64url, line-wrapped base64, hex, percent, HTML entities, UTF-8 byte escapes and JSON string escapes, with fatal UTF-8 checks for decoded payloads. The detection view additionally decodes `\uXXXX`, `\u{X}`, `\x{X}`, `\UXXXXXXXX` and `%uXXXX` escapes, runs of four or more `U+XXXX` code points, CSS escape runs (`\74 \6f \6b`), drops NUL bytes of UTF-16 text read as single bytes, zero-width and bidirectional format characters and ANSI colour sequences, removes inline HTML formatting tags (`<em>`, `<span class=…>`) and turns block tags into line breaks, while findings keep their original offsets. A finding inside a `U+` or CSS escape run redacts the whole run, so no undecodable fragment of it remains. The CLI `redact` and `scan` commands also read UTF-16LE/BE files that start with a byte order mark and write redacted output back in the same encoding; BOM-less UTF-16 and stream input must be UTF-8. Plain log files and streams preserve invalid UTF-8 bytes reversibly while scanning surrounding valid text; structured JSON/JSONL and configuration remain strict. Opt-in encoding preservation round-trips sanitized payloads; nested escaped JSON retains its structure by default. Arbitrarily split fragments, unlimited nesting, compressed/encrypted payloads, malformed encodings, archives, PDFs and OCR/images are outside this text profile. Whole-input/candidate limits fail; oversized textual percent/base64 runs receive audited complete-run redaction. Unsupported representations may remain undetected. `decode: false` intentionally reduces coverage. A caller needing another format should decode it through a bounded preprocessing adapter and retain correct original spans.
 
 Default text mode can replace JSON numeric values with string markers and change syntax/types. Opt-in structured JSON/JSONL mode scans string keys/values separately, preserves primitive types, and rejects malformed output or key collisions. Numeric personal values need a separate caller schema policy. Existing quoted JSON strings and escaped-string fixtures retain their delimiters. Review sanitized artifacts for their intended consumer.
 
@@ -193,7 +193,7 @@ An unchanged result is not proof that text contains no sensitive information. Th
 
 ## Entity catalogs and optional modes
 
-Documented Google/Azure/AWS catalog names (213/176/36) are recognized as explicit native labels, including canonical underscore and original camel-case forms. Generic DATE is excluded so ordinary headers/calendar dates remain intact; personal birth-date context still qualifies. This vocabulary is distinct from automatic vendor NLP/image/cloud-service features. Typed transformations and confidence are described in [API.md](API.md); full redaction remains the default. Public Wikidata names, known role/domain emails and public resolver/documentation IPs use explicit offline knowledge and heuristics. Opt-in network lookup does not establish identity; private overrides and credential priority remain authoritative.
+Documented Google/Azure/AWS catalog names (214/180/36) are recognized as explicit native labels, including canonical underscore and original camel-case forms. Generic DATE is excluded so ordinary headers/calendar dates remain intact; personal birth-date context still qualifies. This vocabulary is distinct from automatic vendor NLP/image/cloud-service features. Typed transformations and confidence are described in [API.md](API.md); full redaction remains the default. Public Wikidata names, known role/domain emails and public resolver/documentation IPs use explicit offline knowledge and heuristics. Opt-in network lookup does not establish identity; private overrides and credential priority remain authoritative.
 
 Large text logs use bounded record streaming and worker file publication. Plain-text records inherit the configurable total limit (1 GiB by default), so memory grows with the largest held record. The whole-string default is 32 Mi UTF-16 units. `credentialsOnly: true` preserves personal/account details across all finding sources while retaining credential detection and verification. The review corpus benchmark is a finite, synthetic regression comparison; [method, settings and results](../experiments/benchmark/README.md) explain its scope. It does not demonstrate general market-wide superiority.
 
@@ -476,7 +476,25 @@ proper-name mode. Romashka Inc keeps its existing legal-form treatment.
 `strict: true` independently opts into personal attributes (gender, family,
 ethnicity, religion, health, criminal/political status) and bare dates, ages,
 passport/mobile/postal shapes. It is intended for stored personal prose and can
-remove ordinary uses of these words. It preserves ISO timestamps, work durations
+remove ordinary uses of unambiguous category words. Bounded Russian case/stem
+rules cover disability, ethnicity, religion, diagnoses, intimate life and criminal
+status, including `инвалидом`, `армянин`, `еврейка`, `казах`, `христианин`, `иудей`,
+`эпилепсией`, `астма`, `онкологию`, `туберкулез`, `психиатра`, `депрессия`,
+`гепатит`, `гей` and `отбывал наказание`. English coverage includes `epilepsy`,
+`gay` and `Armenian`. Explicit category cues such as `по национальности X`,
+`исповедует X`, `болеет X`, `на учете у X`, `diagnosed with X` and `ethnicity: X`
+protect one following word (including hyphenated words) up to 64 characters,
+including unfamiliar values outside the lexicon. These cues stay on one line.
+Ambiguous English `disabled`, `single`, `man`, `men`, `russian`
+and `asian` require an adjacent personal cue (pronoun/person role, personal
+label, or following person role). Thus `he is disabled`, `candidate is single`
+and `gender: male` redact, while `the module is disabled`, `single run finished`,
+`single-flight lock acquired`, `man page missing`, `locale russian selected`
+and `asian region endpoint` remain intact. Bare operational JSON statuses
+`disabled`, `single` and `Russian` also remain intact; explicit personal labels
+and caller-declared known personal values still redact. A cue in another clause
+or string value does not make an ambiguous word personal.
+It preserves ISO timestamps, work durations
 (including `Стаж: 5 лет`, `работал там 5 лет`, and duration cues from issue 87),
 and parenthesized durations after known technologies such as `Kubernetes (5 лет)`
 and `Power BI (5 лет)`.