@link-foundation/sensitive-data-sanitizer 0.19.3 → 0.21.0

package/docs/API.md

@@ -1,6 +1,6 @@
 # Library and detector contract
 
-Import the ESM root entry; complete declarations are in `src/index.d.ts`. Inputs are strings, offsets are UTF-16 code units, starts are zero-based, and ends are exclusive. Returned line and column numbers are one-based; columns also count UTF-16 units.
+Import the root entry with `import` or `require`; complete declarations are in `src/index.d.ts`. Inputs are strings, offsets are UTF-16 code units, starts are zero-based, and ends are exclusive. Returned line and column numbers are one-based; columns also count UTF-16 units.
 
 ## Methods
 
@@ -16,9 +16,38 @@ Import the ESM root entry; complete declarations are in `src/index.d.ts`. Inputs
 | `createSyncSanitizer(options)`         | Reusable synchronous `inspect` / `sanitize` with the same policies                                           |
 | `compileKnown(...lists)`               | Copy, freeze, merge and precompile caller-known personal values                                              |
 | `auditGitHistory(repository, options)` | Read-only audit of local reachable text objects                                                              |
+| `decrypt(text, key, options)`          | Restore values from `[ENCRYPTED:TYPE:…]` tokens written by the `encrypt` transformation                      |
+| `createPatternDetector(specs, opts)`   | Declarative regex, deny-list, hotword and validator detector                                                 |
+| `createIdentifierDetector(rules)`      | Detector for caller-supplied national identifier rules (`nationalRules` shape)                               |
+| `toSarif(report, options)`             | Convert a scan report or findings to SARIF 2.1.0 without values                                              |
 
 `redactions` counts merged replacement spans. `findings` can contain several detector hits for one replacement. Findings have `{start, end, type, category, rule, line, column}`; no text, quote, match, entropy score, provider details, or raw diagnostic is copied. The optional `strict-attributes` debug event reports only detected and suppressed match counts. Debugging is off by default; `debug(event)` receives value-free counts, rule timings, residual rounds and registry event metadata (see `DebugEvent` in the TypeScript declarations).
 
+### Credentials-only entry
+
+`@link-foundation/sensitive-data-sanitizer/credentials` exports `inspect`,
+`sanitize`, `redact`, `createSanitizer`, `createSyncSanitizer`, `REDACTED`,
+`entropy`, `builtInRules`, `tokenChecksum`, `tokenWithChecksum` and the
+`knownSecretsFrom*` helpers. It always runs with `credentialsOnly: true` and
+does not load the personal-data detectors, faker or libphonenumber, so it starts
+about three times faster than the root entry (about 0.25 s against 0.75 s on
+Node.js 26). Use it for loggers that must mask every line from the first write:
+
+```js
+const {
+  createSyncSanitizer,
+  knownSecretsFromEnv,
+} = require('@link-foundation/sensitive-data-sanitizer/credentials');
+const mask = createSyncSanitizer({ knownSecrets: knownSecretsFromEnv() });
+console.log(mask.sanitize('Authorization: Bearer abc123def456ghi789').text);
+// Authorization: Bearer [REDACTED]
+```
+
+Options that need the personal-data modules, `credentialsOnly: false`,
+`mode: 'fake'` and `properNames`, fail with `ERR_CONFIG`. Known personal values
+(`knownPersonal`) still work. Both entries offer `import`, `require` (Node.js
+22.12 or later) and `default` export conditions.
+
 ## Options
 
 ### Replace only known values
@@ -67,6 +96,31 @@ Unicode letters, marks, digits and underscores bound those tokens. Ordinary
 technical names such as GPT-4, ISO-9001 and Windows-10 are not reserved. The map's format cannot change
 on reuse. A malformed map or ambiguous exact restoration blocks output.
 
+`existing: 'keep'` on the placeholder transformation accepts input that already
+contains tokens of the chosen namespace, for example the output of an earlier
+pass or a model answer that quotes tokens. Those tokens stay as they are and are
+never replaced, faked or treated as names; new values of a type are numbered
+after the highest kept number of that type:
+
+```js
+const policy = { transformation: { mode: 'placeholder', existing: 'keep' } };
+sanitize('Person-1 встретил Анну Смирнову', policy).text;
+// Person-1 встретил Person-2
+```
+
+A second pass over sanitized text is then a fixed point. The default
+`existing: 'error'` keeps the `ERR_PLACEHOLDER_COLLISION` check; other values and
+a mix with per-type `transformations` are `ERR_CONFIG`. A kept token restores to
+itself, so the map still reconstructs every newly replaced value exactly.
+
+`reservedPlaceholderTypes()` returns the sorted, frozen list of every type a
+built-in rule can turn into a token. `isPlaceholderToken(text, {format?, types?})`
+checks whether `text` is one whole token of the namespace (`Person-1`,
+`UsBankNumber-12`, `<PERSON_3>` with `format: '<{TYPE}_{n}>'`); `types` adds
+custom types such as those of `createPatternDetector`. Both are also exported by
+the credentials-only entry. The namespace pattern is compiled once per format and
+cached.
+
 `createRestoreMap({format?})` creates an empty reusable map. Pass a map as
 `transformation.map` to continue a corpus; the operation updates that same
 object. A `createSanitizer()` engine also keeps its registry across calls.
@@ -147,6 +201,13 @@ JSON keys remain unchanged by automatic personal detection in all transformation
 modes. Explicit known values and credentials retain priority. Parseability and
 duplicate-key checks remain enforced. Plain redact/fake policies retain their
 existing metadata/key behavior. Use `structured` for valid structured input.
+A whole JSON value replaced by a structured rule (`sensitiveFields`, or a
+personal field such as `age` replaced with its JSON type) becomes one JSON
+string token typed by its field: `{"notes":"любой текст"}` reads
+`{"notes":"Sensitive-1"}` and `{"age":34}` reads `{"age":"Age-1"}`. Such findings
+carry `valueType` when the field has a personal type. `restore()` brings back
+the original JSON value, including numbers and objects. Other modes keep
+`"[REDACTED]"`.
 The normal encoding limits and credential redaction policy continue to apply.
 
 CLI: `redact FILE --placeholder --map private.json` creates or continues a map;
@@ -165,7 +226,7 @@ new document recognizers; `documentEntities: false` turns them off explicitly.
 | `scripts`                | `['Cyrillic']`; Unicode Script aliases that activate the prose rule; `[]` disables only script activation                                                        |
 | `profile`                | Off; `'publication'` sets a 0.3 threshold and identity masking; `'documents'` enables document entities                                                          |
 | `knownOnly`              | `false`; disable all automatic detection (including credentials, Secretlint and custom detectors); use explicit findings, known values and reused map identities |
-| `knownSecrets`           | `[]`; exact literal credential values, regardless of length or format                                                                                            |
+| `knownSecrets`           | `[]`; exact literal credential values, regardless of length or format, plus their standalone hex (either case) and base64/base64url spellings                    |
 | `knownPersonal`          | `[]`; `{type, value}` values with whole-word Unicode case-insensitive matching and bounded Russian forms; list parts to match them alone                         |
 | `publicEntities`         | `[]`; exact PERSON/ORGANIZATION/EDUCATION/LOCATION/EMAIL exception with HTTPS evidence and review date                                                           |
 | `findings`               | `[]`; validated detections for the original input, supplied by the caller                                                                                        |
@@ -192,7 +253,7 @@ new document recognizers; `documentEntities: false` turns them off explicitly.
 | `strict`                 | `false`; aggressive stored-log personal attributes and bare values                                                                                               |
 | `structuralFields`       | `[]`; declared property names suppress weak name/ID/entropy heuristics in their string values                                                                    |
 
-Unknown options and invalid limits are errors. `knownSecretsFromEnv(env, names?)` discovers credential-named variables (excluding auth configuration names) when names are omitted, or uses an explicit selection, deduplicating nonempty values. It never invokes shell commands or reads auth files. `personalVariants(entries)` generates numeric and ISO/day-first date spellings; name case variants are already handled by detection. Neither helper invents transliterations.
+Unknown options and invalid limits are errors. `knownSecretsFromEnv(env, names?, {minLength?})` discovers credential-named variables (excluding configuration names) when names are omitted, or uses an explicit selection, deduplicating nonempty values (or values of at least `minLength` characters). It never invokes shell commands or reads auth files. `personalVariants(entries)` generates numeric and ISO/day-first date spellings; name case variants are already handled by detection. Neither helper invents transliterations.
 
 ## Required detectors
 
@@ -242,6 +303,7 @@ Without a profile, full redaction is the library default. `transformation` selec
 | `fake`              | Keyed realistic names, valid dates/MRZ/checksummed numbers and reserved contact destinations                           |
 | `date-shift`        | Personal ISO dates; finite integer `days`, UTC calendar arithmetic; invalid dates become opaque                        |
 | `bucket`            | Personal safe integers; positive integer `size`, inclusive numeric interval                                            |
+| `encrypt`           | Personal data only; deterministic AES-256-GCM token reversible with `decrypt` and the key (see below)                  |
 
 All keyed modes require a key of at least 16 UTF-8 bytes. `format-preserving` is a non-reversible shape pseudonym, not NIST FF1 encryption; it does not preserve checksum validity, country-specific alphabets or guarantee uniqueness. Partial masks, date shifts and buckets intentionally retain information. Keep pseudonym keys outside logs. Residual verification checks the corresponding fully redacted representation; opted-in disclosures can remain recognizable in the transformed output. Use default redaction for publication that requires opaque output.
 
@@ -255,7 +317,7 @@ Generated replacements carry `faked: true`. A private bounded registry recognize
 
 Faking is pseudonymisation: a private key and retained ages, relations and structure can permit linking. Keep the key out of published data. See [the realistic fake example](../examples/realistic-fakes.mjs) and [the complete requirement/research record](case-studies/issue-24/REQUIREMENTS.md).
 
-`preserveEncoding: true` sanitizes decoded payloads with the same required engine union, re-encodes them and verifies decoding reproduces the sanitized content. Supported formats include base64/base64url, hex, percent, numeric HTML and UTF-8 byte escapes. Escaped JSON string content is sanitized and re-encoded even under default redaction so nested session JSON stays parseable. Malformed or unsupported encodings block opted-in preservation. Known encoding candidates and nesting are bounded.
+`preserveEncoding: true` sanitizes decoded payloads with the same required engine union, re-encodes them and verifies decoding reproduces the sanitized content. Supported formats include base64/base64url, hex, percent, numeric HTML and UTF-8 byte escapes. Escaped JSON string content is sanitized and re-encoded even under default redaction so nested session JSON stays parseable. Wrapped base64 and hex keep their line width and line separator (`\n` or `\r\n`), and a wrapped run is reported as one finding. When the sanitized payload cannot be re-encoded in a form the decoder recognizes (for example hex shorter than 32 digits), the encoded text is masked as it stands rather than blocking the input. Known encoding candidates and nesting are bounded.
 
 ## Identity policy and structured sessions
 
@@ -303,9 +365,15 @@ status identifiers. Explicit `knownPersonal` values override these heuristics.
 
 Oversized encoded runs receive a conservative complete-run `ENCODED_LIMIT`
 redaction rather than aborting the record. Decoding probes are capped at 4,096
-characters; full decoding stays capped at 8,192 characters. Whole-input,
-candidate-count, record and worker limits still apply. This fallback can remove
-large non-sensitive encoded outputs; its finding makes that decision auditable.
+characters; full decoding stays capped at 8,192 characters. A multi-line run is
+treated as one wrapped payload only when its probe decodes as text or its lines
+share one base64 width (a multiple of four, standard alphabet, not plain hex);
+lists of identifiers or commit hashes are scanned line by line. Up to 4,096
+textually decodable candidates are scanned per 64 KiB window of the input;
+further candidates in a denser window receive `ENCODED_LIMIT` redactions rather
+than an `ERR_LIMIT` for the whole text. Whole-input, record and worker limits
+still apply. This fallback can remove large non-sensitive encoded outputs; its
+finding makes that decision auditable.
 
 ## Streaming and worker publication
 
@@ -368,7 +436,7 @@ The optional ESLint plugin exports `require-sanitized-output` from the package's
 
 ## Native documented entity catalogs
 
-`entityCatalogs` exposes source URLs, source-page SHA256 and documented names for Google (213), Azure (176) and AWS (36). Their canonical uppercase underscore labels are recognized natively, with complete value spans, independent of cloud services. The package also implements selected automatic formats and checksum algorithms. Catalog vocabulary support does not reproduce proprietary NER, image/document inspection, regional identity validators, cloud risk analysis or their accuracy. See the coverage table for that distinction.
+`entityCatalogs` exposes source URLs, source-page SHA256 and documented names for Google (214), Azure (180) and AWS (36). Their canonical uppercase underscore labels are recognized natively, with complete value spans, independent of cloud services. The package also implements selected automatic formats and checksum algorithms. Catalog vocabulary support does not reproduce proprietary NER, image/document inspection, regional identity validators, cloud risk analysis or their accuracy. See the coverage table for that distinction.
 
 ## Two-pass session publication
 
@@ -446,12 +514,28 @@ findings retain the normal engine union, confidence rules and verification.
 This keeps EMAIL/account-ID values without per-value `publicEntities` entries.
 The default policy continues to protect personal data.
 
-`knownSecretsFromEnv(env = process.env, names?)` discovers nonempty exact values
-when `names` is omitted, using
-`/(?:^|_)(?:TOKEN|(?:API|ACCESS|PRIVATE)?KEY|SECRET|PASSWORD|PASSWD|CREDENTIALS?|AUTH|PAT)(?:_|$)/i`.
-This covers `MY_SERVICE_TOKEN`, `CLAUDE_CODE_OAUTH_TOKEN`, `CODEX_API_KEY`,
-`OPENCODE_API_KEY`, `DASHSCOPE_API_KEY`, `HF_TOKEN` and `TELEGRAM_BOT_TOKEN`.
-Automatic discovery excludes names ending in `AUTH_TRANSPORT`, `AUTH_MODE`, `AUTH_METHOD`, `AUTH_TYPE`, `AUTH_URL`, `AUTH_URI`, `AUTH_HOST`, `AUTH_HOSTNAME`, `AUTH_PATH`, `AUTH_ENABLED`, `AUTH_DISABLED`, `AUTH_PORT`, `AUTH_PROTOCOL` or `AUTH_SCHEME` (including prefixed forms such as `HIVE_MIND_GIT_AUTH_TRANSPORT`). Credential names such as `AUTH_TOKEN` still contribute their exact values, including short/low-entropy values; no value-shape filter can reliably distinguish passwords from configuration. An explicit array bypasses the name exclusions; `[]` selects nothing.
+`knownSecretsFromEnv(env = process.env, names?, { minLength = 1 }?)` discovers
+nonempty exact values when `names` is omitted. A name is selected when it has a
+credential keyword between `_` boundaries
+(`/(?:^|_)(?:TOKEN|(?:API|ACCESS|PRIVATE)?KEY|SECRET|PASSWORD|PASSWD|CREDENTIALS?|O?AUTH|PAT|COOKIE)(?:_|$)/i`)
+or ends in `PASSWORD`, `PASSWD`, `TOKEN` or `COOKIE` without one. This covers
+`MY_SERVICE_TOKEN`, `CLAUDE_CODE_OAUTH_TOKEN`, `CODEX_API_KEY`,
+`OPENCODE_API_KEY`, `DASHSCOPE_API_KEY`, `HF_TOKEN`, `TELEGRAM_BOT_TOKEN`,
+`PGPASSWORD`, `GITHUBTOKEN`, `OAUTH` and `SESSION_COOKIE`.
+Automatic discovery excludes names whose last segment describes configuration
+rather than a credential: `_TRANSPORT`, `_MODE`, `_METHOD`, `_TYPE`, `_URL`,
+`_URI`, `_HOST`, `_HOSTNAME`, `_PATH`, `_ENABLED`, `_DISABLED`, `_PORT`,
+`_PROTOCOL`, `_SCHEME`, `_GUARD`, `_COUNT`, `_LIMIT`, `_FILE`, `_FILENAME`,
+`_DIR`, `_DIRECTORY`, `_NAME` and `_ID` (for example `HIVE_MIND_GIT_AUTH_TRANSPORT`,
+`HIVE_MIND_ROUTER_AUTH_GUARD`, `MAX_TOKEN_COUNT`, `SSH_KEY_PATH`, `SECRET_NAME`
+and `AWS_ACCESS_KEY_ID`), plus `GOOGLE_APPLICATION_CREDENTIALS` (a file path) and
+`GIT_CONFIG_KEY_<n>` (the name of a git setting). `GIT_CONFIG_VALUE_<n>` is
+selected when its `GIT_CONFIG_KEY_<n>` names a header, password, token, secret,
+cookie file or auth setting, such as `http.extraheader`. Credential names such as
+`AUTH_TOKEN` and `AUTH_MODE_TOKEN` still contribute their exact values, including
+short/low-entropy values; no value-shape filter can reliably distinguish passwords
+from configuration. Pass `minLength` to skip shorter values. An explicit array
+bypasses the name exclusions; `[]` selects nothing.
 These values are loaded only when the helper or CLI `--env-secrets` is requested.
 
 `await knownSecretsFromGitHubConfig({path?})` reads `oauth_token` and `api_token`
@@ -462,6 +546,9 @@ symlinks, aliases, files above 1 MiB and read failures produce value-free
 `ERR_AUTH`. YAML quoting is parsed with the `yaml` package, with alias expansion
 disabled. CLI `--gh-config` adds these exact values to `knownSecrets`.
 `knownSecretsFromGitHubAuth` / `--gh-auth` remain available independently.
+A `hosts.yml` that uses YAML aliases or is a symlink stays a hard `ERR_AUTH`
+failure, so a caller relying on `--gh-config` never runs with a silently
+incomplete list.
 
 Plain log streams and CLI file/stdin/output/in-place paths preserve invalid
 UTF-8 bytes exactly. Valid Unicode keeps its normal meaning for detection.
@@ -554,9 +641,33 @@ also applies independently within serialized JSON strings, to the existing
 bounded decoding depth. Findings do not seed document-wide name propagation.
 Repeated redaction is stable. Reusing a placeholder map for the same original
 produces stable tokens and exact restoration; the existing placeholder-namespace
-collision checks still reject input that already contains typed placeholders.
+collision checks still reject input that already contains typed placeholders
+unless the transformation sets `existing: 'keep'`.
+
+Capitalized words that identify nobody stay readable, so placeholder output keeps
+its sentence: pronouns (`Вас`, `Я`), role and label cues (`Кандидат`, `Сотрудник`,
+the default and configured `roles` and `personalLabels`) anywhere, and greetings,
+function words and verbs (past tense `Звонила`, `Перенёс`, `Работал` before a
+lowercase word; present `Работаю`, `Рекомендует`) at the start of a sentence:
 
-This is a structural heuristic, so capitalized ordinary words also disappear.
+```js
+sanitize('Кандидат Петров Иван Сергеевич. Звонила Анна Смирнова', {
+  properNames: 'capitalized',
+  transformation: { mode: 'placeholder', map: false },
+}).text;
+// Кандидат Person-1. Звонила Person-2
+```
+
+A word that is also a given-name form (`Вера`, `Любовь`) never counts as such a
+word, and a bare masculine `-л` form before a capitalized word may be a surname
+(`Шагал Марк`), so both are still replaced. A compact organization, location,
+school or person span found by the document rules (`ООО «Ромашка»`, `в Казани`)
+replaces the proper-name fragments inside it as one typed value
+(`Organization-1`, `Location-1`); spans with tabs or line breaks, or longer than
+160 characters, keep the separate fragments. A bare oblique surname (`Иванову`)
+links to the one person whose name contains it.
+
+Other capitalized ordinary words still disappear; this is a structural heuristic.
 Lowercase organizations without employment context stay; the document rules
 replace a lowercase employer after `работал в` / `работала в`. Complete contact
 and identifier findings take priority over overlapping proper-name fragments.
@@ -641,6 +752,13 @@ cues. Inputs allow at most 256 nonempty strings of 128 code units per list, with
 no line breaks/NUL. Unknown lists and malformed replacement records fail with
 `ERR_CONFIG`. Compilation uses a bounded 64-configuration cache.
 
+`personalAttributes: ['вепс', 'старовер']` adds special-category stems to the
+`strict` rule (it has no effect without `strict: true`). Each word matches as a
+stem followed by up to eight letters, so `вепска` and `староверами` match, and the
+words of a multi-word entry may be separated by spaces or tabs. Matches are
+reported as `PERSONAL_ATTRIBUTE` with the `strict-attribute` rule. The default
+list is empty.
+
 ## Callback transformations and stages
 
 ```js
@@ -742,3 +860,224 @@ one or both languages. Each invocation is limited to 1 Mi UTF-16 units; larger
 inputs throw `ERR_LIMIT`. `debug` is off by default and emits counts only. Core
 imports never load the optional entry or NLP.js; leaving it off adds no optional
 runtime/module cost.
+
+## Allow lists, suppression comments and template values
+
+`allowList` keeps detected values that the caller has reviewed as safe, as in
+Presidio's `allow_list`, Google's exclusion rules with
+`MATCHING_TYPE_FULL_MATCH` and DataFog's `allowlist_patterns`. A string must
+equal the whole detected value; a `RegExp` must match the whole value (it is
+anchored internally, `g`/`y` flags are ignored). A partial match never shortens
+a finding. At most 10 000 entries; anything else is `ERR_CONFIG`.
+
+```js
+sanitize('Contact support@example.org or alice@private.test', {
+  allowList: ['support@example.org', /.+@example\.org/],
+});
+// → 'Contact support@example.org or [REDACTED]'
+```
+
+`inlineAllow: true` honours the suppression comments of other scanners on the
+same line: `gitleaks:allow`, `betterleaks:allow`, `kingfisher:allow`/`ignore`,
+`trufflehog:ignore`, `sanitizer:allow`, detect-secrets/ripsecrets
+`pragma: allowlist secret` and SonarQube `NOSONAR`. A line containing
+`pragma: allowlist nextline secret` suppresses findings on the next line. It is
+off by default because anyone who can write to a log could otherwise disable
+redaction.
+
+Both controls follow the same safety rules:
+
+- declared `knownPersonal`/`knownSecrets` values, confirmed personal values and
+  structured `JSON_VALUE` denials are never exempted;
+- credentials are exempted only together with `allowCredentialDisable: true`;
+- they apply to text, structured and asynchronous paths alike.
+
+Built-in credential rules also skip values that consist only of a template or
+mask: `${PASSWORD}`, `$${PYPI_PASSWORD}`, `$DB_PASS`, `$1`, `<password>`,
+`<Openverse API token>`, `[PASSWORD]`, `****`, `XXXXXXXX`, `...`, optionally
+after `Bearer`/`Basic`/`token` and on either side of a `user:password` colon.
+A mixed value such as `alice:${DB_PASSWORD}` or `<x>abc123…` is still redacted.
+Findings from custom detectors are never dropped this way. The value-free debug
+event `{event: 'template-value', rule}` reports each skipped match.
+
+### Exempting findings with a hook
+
+`exempt(finding, text) => boolean` keeps findings that the caller's own policy
+knows are not sensitive: a configuration flag, a token counter, a version
+range, a value another tool already masked. It runs after detection and before
+transformation, once per finding. The finding carries `value` and the offsets
+into `text`. Return `true` to keep the span unchanged and `false` to transform
+it. Any other result, including `undefined` from a forgotten `return`, is
+`ERR_CONFIG`.
+
+```js
+createSyncSanitizer({
+  credentialsOnly: true,
+  // The key before the value decides; offsets point into `text`.
+  exempt: (finding, text) =>
+    /FEATURE_TOKEN=$/.test(text.slice(0, finding.start)),
+}).sanitize(
+  'FEATURE_TOKEN=keep-this-value-123 other_token=keep-this-value-123'
+);
+// → 'FEATURE_TOKEN=keep-this-value-123 other_token=[REDACTED]'
+```
+
+Unlike a callback transformation that returns the original value, which fails
+with `ERR_RESIDUAL`, an exempted span is excluded from residual verification
+by span, not by value. Verification of the published text calls the hook again
+for every finding in the output, so the same value elsewhere is still caught
+unless the hook keeps that span too. The hook is explicit code, so it can keep
+credential findings without `allowCredentialDisable`. Declared
+`knownSecrets`/`knownPersonal` values, confirmed personal values and structured
+`JSON_VALUE` denials are never offered to it.
+
+`createSanitizer` awaits a hook that returns a `Promise`.
+`createSyncSanitizer` and `sanitize()` need a synchronous hook. A `Promise`
+there is `ERR_CONFIG`.
+
+## Per-type thresholds
+
+`minConfidenceByType` maps finding types to a threshold that overrides
+`minConfidence` for those types only, as in Presidio score thresholds or
+Google's per-infoType likelihood:
+
+```js
+sanitize(text, { minConfidence: 0.5, minConfidenceByType: { PERSON: 0.8 } });
+```
+
+Values must be numbers from 0 to 1. Credentials are never filtered by a
+threshold.
+
+## Character masks
+
+The `mask` transformation accepts Presidio/Google-style tuning in addition to
+`keepStart`, `keepEnd`, `minLength` and `marker`:
+
+| Option          | Meaning                                                        |
+| --------------- | -------------------------------------------------------------- |
+| `char`          | One masking character (default `*`)                            |
+| `ignore`        | Characters that are never masked, e.g. `'-@.'`                 |
+| `ignoreClasses` | Any of `digits`, `upper`, `lower`, `punctuation`, `whitespace` |
+| `count`         | Mask at most this many characters                              |
+| `fromEnd`       | Mask the last `count` characters instead of the first          |
+
+```js
+sanitize('Card 4111-1111-1111-1111', {
+  transformation: { mode: 'mask', char: '#', ignore: '-', count: 12 },
+});
+// → 'Card ####-####-####-1111'
+```
+
+`marker` cannot be combined with the tuning options. A value with nothing left
+to mask is fully redacted. Masks apply to personal data only.
+
+## Deterministic encryption
+
+`{mode: 'encrypt', key}` replaces each personal value with
+`[ENCRYPTED:TYPE:…]`, comparable to Google's `CryptoDeterministicConfig` and
+Presidio's `encrypt` operator. The token is AES-256-GCM with a synthetic IV
+(HMAC of type and value), so equal values of one type produce equal tokens and
+joins or counts still work. Keys are at least 16 UTF-8 bytes and are expanded
+with HKDF-SHA256.
+
+```js
+import { sanitize, decrypt } from '@link-foundation/sensitive-data-sanitizer';
+const key = process.env.SANITIZER_KEY;
+const { text } = sanitize('Mail alice@private.test', {
+  transformation: { mode: 'encrypt', key },
+});
+decrypt(text, key); // → 'Mail alice@private.test'
+```
+
+`decrypt(text, key, {json: true})` escapes restored values inside JSON string
+literals. A token that fails authentication throws `ERR_MAP`; a non-string text
+throws `ERR_INPUT`. Credentials are still fully redacted and never encrypted.
+Anyone holding the key can recover the values, so treat encrypted output as
+pseudonymised, not anonymous.
+
+## Declarative pattern detectors
+
+`createPatternDetector(specs, {id})` builds a synchronous Detector from
+Presidio-style pattern recognizers or Google custom infoTypes without writing
+code:
+
+```js
+import {
+  createPatternDetector,
+  sanitize,
+} from '@link-foundation/sensitive-data-sanitizer';
+const detector = createPatternDetector([
+  {
+    type: 'MEDICAL_RECORD',
+    category: 'personal',
+    patterns: [{ regex: /\bMRN-\d{6}\b/g, confidence: 0.4 }],
+    denyList: ['Mr. Bean'],
+    context: ['mrn', 'medical record'],
+    contextWindow: { before: 64, after: 16 },
+    contextBoost: 0.35,
+    validate: (value) => value !== 'MRN-000000',
+  },
+]);
+sanitize('medical record MRN-123456', { detectors: [detector] });
+```
+
+| Field              | Meaning                                                                                        |
+| ------------------ | ---------------------------------------------------------------------------------------------- |
+| `type`, `category` | Uppercase type; `personal` or `credential`                                                     |
+| `rule`             | Lowercase rule id (default `custom-pattern`)                                                   |
+| `confidence`       | Base confidence (default 0.6)                                                                  |
+| `patterns`         | Up to 64 `RegExp`, strings or `{regex, confidence, group}` (a capture group narrows the span)  |
+| `denyList`         | Whole-word values, case-insensitive unless `caseSensitive: true`; `denyListConfidence` (1)     |
+| `context`          | Hotwords (strings or RegExp) searched in `contextWindow` (number or `{before, after}`, ≤ 4096) |
+| `contextBoost`     | Added when a hotword is present (default 0.35, floor 0.4); a negative value lowers confidence  |
+| `validate`         | `false` drops a match; a number sets its confidence                                            |
+
+Malformed specifications throw `ERR_CONFIG`.
+
+`createIdentifierDetector(rules, {id})` adapts rules with the shape of the
+exported `nationalRules` table (`{type, pattern, context, checksum, plausible?,
+valid?}`) so callers can add or replace national identifiers. `identifierChecks`
+exposes the checksum and format validators (Luhn, mod-11, mod-97, Verhoeff and
+the national algorithms) for reuse; a valid checksum never proves issuance.
+
+## Offline token checksums
+
+GitHub (`ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_`) and npm (`npm_`) tokens end in
+six base62 characters holding the CRC32 of the 30 characters after the first
+`_`. As in Kingfisher and osv-scalibr, the native `github`/`npm` rules and the
+vendored Gitleaks/Betterleaks rules give a checksum-valid token confidence
+0.99. A token whose checksum does not match keeps its ordinary confidence and
+is still redacted, because truncated or retyped tokens remain sensitive. No
+network request is made. The exported `tokenChecksum(value)` returns `true`,
+`false` or `undefined` (not a checksummed format), and
+`tokenWithChecksum(prefix, body)` builds checksum-valid test fixtures at
+runtime so repositories need not store token literals.
+
+## SARIF reports
+
+`toSarif(report, {version, path})` converts a CLI `scan` report, an array of
+`{path, findings}` or a findings array into SARIF 2.1.0 for GitHub code
+scanning and other CI viewers. Rules are keyed by `finding.rule`; credential
+findings are `error` and personal findings `warning`. Regions carry
+`charOffset`, `charLength`, `startLine` and `startColumn` in UTF-16 code units.
+Skipped files become tool execution notifications. Values are never included.
+
+```sh
+sensitive-data-sanitizer scan --format sarif src > results.sarif
+```
+
+`--format` accepts `json` (default) or `sarif` and is only valid for `scan`.
+
+CLI `scan` accepts several paths, so a CI step or a
+[pre-commit](https://pre-commit.com) hook can pass the staged files directly.
+The repository ships `.pre-commit-hooks.yaml` with two hooks:
+`sensitive-data-sanitizer` (credentials only) and
+`sensitive-data-sanitizer-pii` (credentials and personal data).
+
+```yaml
+repos:
+  - repo: https://github.com/link-foundation/sensitive-data-sanitizer
+    rev: v0.20.0
+    hooks:
+      - id: sensitive-data-sanitizer
+```

package/docs/COVERAGE.md

@@ -6,9 +6,9 @@ Coverage is a union of maintained detectors, explicit known values, context and
 
 The native provider supplements in `src/rules.js` cover GitHub, GitLab, OpenAI, Anthropic, AWS access IDs, Google API/OAuth, Slack, Stripe, SendGrid, Twilio, npm, PyPI, Telegram, Discord, Hugging Face, Shopify, Databricks, Square, Mailchimp, DigitalOcean, Vault, Grafana, Linear, Notion, Vercel, and JWT-shaped values. Prefix matches also protect selected truncated tokens; they do not validate a token with a provider.
 
-The default native engine bundles 221 text rules from Gitleaks 8.30.1 (222 upstream rules; the path-only PKCS12 rule is outside text detection), compiled unchanged with RE2/WASM or RE2JS for inputs above 4 MiB. Dense rule matches switch to RE2JS after at most eight WASM matches per view, bounding repeated whole-input copies without truncating secrets or changing anchors. The MIT license, commit and sync script are retained in `src/vendor/gitleaks`. Keyword prefilters and configured entropy thresholds run natively; upstream comment/path allowlists cannot suppress a sanitizer finding. Quoted hex/base64 entropy detection supplements those rules. The default asynchronous engine additionally uses the pinned Secretlint recommended scanners, excluding its comment-filter rule. Embedded disable directives in untrusted text cannot suppress findings. Optional Gitleaks and TruffleHog invoke their installed detector sets, and custom detectors can add formats. Service counts are not a measure of comparative recall.
+The default native engine bundles 221 text rules from Gitleaks 8.30.1 (222 upstream rules; the path-only PKCS12 rule is outside text detection), compiled unchanged with RE2/WASM or RE2JS for inputs above 1 MiB (re2-wasm has a fixed 16 MiB heap shared by every compiled rule and copied subject). Dense rule matches switch to RE2JS after at most eight WASM matches per view, bounding repeated whole-input copies without truncating secrets or changing anchors. The MIT license, commit and sync script are retained in `src/vendor/gitleaks`. Because Gitleaks is no longer updated, 203 further rules from its maintained successor Betterleaks 1.10.0 (ids absent from the Gitleaks set, such as Groq, Cerebras, xAI, Vercel, Supabase and Cloudflare v2 tokens) run the same way as `betterleaks/<id>` findings, with their MIT license, provenance and `scripts/sync-betterleaks.py` in `src/vendor/betterleaks`. Their entropy floors and secret-only `matchesAny`/`containsAny` filters are honoured; BPE token-ratio and nearby-line filters, live validation, validation-only components and the context-dependent `generic-*` rules are not, so these rules can only report more than Betterleaks, not less. Keyword prefilters and configured entropy thresholds run natively; upstream comment/path allowlists cannot suppress a sanitizer finding. Quoted hex/base64 entropy detection supplements those rules. The default asynchronous engine additionally uses the pinned Secretlint recommended scanners, excluding its comment-filter rule. Embedded disable directives in untrusted text cannot suppress findings. Optional Gitleaks and TruffleHog invoke their installed detector sets, and custom detectors can add formats. Service counts are not a measure of comparative recall.
 
-Context rules recognize credential labels with bounded key affixes, JSON and nested escaped JSON, YAML/env/INI assignments, shell flags, XML values, URL userinfo, authorization headers, cookies, webhook paths, and complete or truncated private-key PEM blocks. Low-entropy contextual values such as `password=1234` are protected. Generic token counters, booleans/null/language literals and ordinary Git hashes are excluded to retain useful logs. Supply literal known values for ambiguous credentials or unusual key names.
+Context rules recognize credential labels with bounded key affixes, JSON and nested escaped JSON, YAML/env/INI assignments, Ruby/Perl/PHP hash rockets before a quoted literal (`'password' => '…'`), shell flags including plural credential names (`--access-tokens`, but not `--max-tokens`), XML values (also with attributes and camelCase element names), values that start with markup (`token: <script>…`; a bare `<placeholder>` stays), all-caps CI/test credential markers that contain a digit (`SYNTHETIC_TOKEN_ALPHA_123456`, rule `marker-token`), URL userinfo, authorization headers, cookies, webhook paths, and complete or truncated private-key PEM blocks. Low-entropy contextual values such as `password=1234` are protected; a number or boolean under a configuration name (`AUTH_MODE=2`, `ROUTER_AUTH_GUARD=0`) is a setting. Generic token counters, booleans/null/language literals and ordinary Git hashes are excluded to retain useful logs. Supply literal known values for ambiguous credentials or unusual key names.
 
 Broader unquoted entropy scanning is opt-in (`paranoid: true`), uses Shannon entropy ≥4.2 on bounded candidates, and exempts ordinary hex hashes and common tool identifiers. Quoted hex/base64 and entropy checks belonging to bundled provider rules run by default. Entropy has both false positives and false negatives; contextual detection and known-secret dictionaries remain necessary.
 
@@ -185,7 +185,7 @@ records standards, alternatives and regression coverage for each feature.
 | Local subprocess      | Default 30 seconds and 10 MiB report/stdout cap; failures block output                                |
 | Git objects           | 10,000 objects, 10 MiB per object and 100 MiB total by default                                        |
 
-Supported decoding includes canonical base64/base64url, line-wrapped base64, hex, percent, HTML entities, UTF-8 byte escapes and JSON string escapes, with fatal UTF-8 checks for decoded payloads. Plain log files and streams preserve invalid UTF-8 bytes reversibly while scanning surrounding valid text; structured JSON/JSONL and configuration remain strict. Opt-in encoding preservation round-trips sanitized payloads; nested escaped JSON retains its structure by default. Arbitrarily split fragments, unlimited nesting, compressed/encrypted payloads, malformed encodings, archives, PDFs and OCR/images are outside this text profile. Whole-input/candidate limits fail; oversized textual percent/base64 runs receive audited complete-run redaction. Unsupported representations may remain undetected. `decode: false` intentionally reduces coverage. A caller needing another format should decode it through a bounded preprocessing adapter and retain correct original spans.
+Supported decoding includes canonical base64/base64url, line-wrapped base64, hex, percent, HTML entities, UTF-8 byte escapes and JSON string escapes, with fatal UTF-8 checks for decoded payloads. The detection view additionally decodes `\uXXXX`, `\u{X}`, `\x{X}`, `\UXXXXXXXX` and `%uXXXX` escapes, runs of four or more `U+XXXX` code points, CSS escape runs (`\74 \6f \6b`), drops NUL bytes of UTF-16 text read as single bytes, zero-width and bidirectional format characters and ANSI colour sequences, removes inline HTML formatting tags (`<em>`, `<span class=…>`) and turns block tags into line breaks, while findings keep their original offsets. A finding inside a `U+` or CSS escape run redacts the whole run, so no undecodable fragment of it remains. The CLI `redact` and `scan` commands also read UTF-16LE/BE files that start with a byte order mark and write redacted output back in the same encoding; BOM-less UTF-16 and stream input must be UTF-8. Plain log files and streams preserve invalid UTF-8 bytes reversibly while scanning surrounding valid text; structured JSON/JSONL and configuration remain strict. Opt-in encoding preservation round-trips sanitized payloads; nested escaped JSON retains its structure by default. Arbitrarily split fragments, unlimited nesting, compressed/encrypted payloads, malformed encodings, archives, PDFs and OCR/images are outside this text profile. Whole-input/candidate limits fail; oversized textual percent/base64 runs receive audited complete-run redaction. Unsupported representations may remain undetected. `decode: false` intentionally reduces coverage. A caller needing another format should decode it through a bounded preprocessing adapter and retain correct original spans.
 
 Default text mode can replace JSON numeric values with string markers and change syntax/types. Opt-in structured JSON/JSONL mode scans string keys/values separately, preserves primitive types, and rejects malformed output or key collisions. Numeric personal values need a separate caller schema policy. Existing quoted JSON strings and escaped-string fixtures retain their delimiters. Review sanitized artifacts for their intended consumer.
 
@@ -193,7 +193,7 @@ An unchanged result is not proof that text contains no sensitive information. Th
 
 ## Entity catalogs and optional modes
 
-Documented Google/Azure/AWS catalog names (213/176/36) are recognized as explicit native labels, including canonical underscore and original camel-case forms. Generic DATE is excluded so ordinary headers/calendar dates remain intact; personal birth-date context still qualifies. This vocabulary is distinct from automatic vendor NLP/image/cloud-service features. Typed transformations and confidence are described in [API.md](API.md); full redaction remains the default. Public Wikidata names, known role/domain emails and public resolver/documentation IPs use explicit offline knowledge and heuristics. Opt-in network lookup does not establish identity; private overrides and credential priority remain authoritative.
+Documented Google/Azure/AWS catalog names (214/180/36) are recognized as explicit native labels, including canonical underscore and original camel-case forms. Generic DATE is excluded so ordinary headers/calendar dates remain intact; personal birth-date context still qualifies. This vocabulary is distinct from automatic vendor NLP/image/cloud-service features. Typed transformations and confidence are described in [API.md](API.md); full redaction remains the default. Public Wikidata names, known role/domain emails and public resolver/documentation IPs use explicit offline knowledge and heuristics. Opt-in network lookup does not establish identity; private overrides and credential priority remain authoritative.
 
 Large text logs use bounded record streaming and worker file publication. Plain-text records inherit the configurable total limit (1 GiB by default), so memory grows with the largest held record. The whole-string default is 32 Mi UTF-16 units. `credentialsOnly: true` preserves personal/account details across all finding sources while retaining credential detection and verification. The review corpus benchmark is a finite, synthetic regression comparison; [method, settings and results](../experiments/benchmark/README.md) explain its scope. It does not demonstrate general market-wide superiority.
 
@@ -480,17 +480,27 @@ remove ordinary uses of unambiguous category words. Bounded Russian case/stem
 rules cover disability, ethnicity, religion, diagnoses, intimate life and criminal
 status, including `инвалидом`, `армянин`, `еврейка`, `казах`, `христианин`, `иудей`,
 `эпилепсией`, `астма`, `онкологию`, `туберкулез`, `психиатра`, `депрессия`,
-`гепатит`, `гей` and `отбывал наказание`. English coverage includes `epilepsy`,
-`gay` and `Armenian`. Explicit category cues such as `по национальности X`,
+`гепатит`, `гей` and `отбывал наказание`, plus nationalities (`узбечка`,
+`таджик`, `чеченец`, `грузинка`, `азербайджанец`, `цыганка`, `кореянка`),
+religions (`протестант`), health (`инсульт`, `шизофрения`, `слабослышащий`,
+`колясочник`, `передвигается на коляске`, `аутизм`), sexuality (`бисексуал`,
+`трансгендер`), convictions (`сидел за кражу`) and trade-union membership
+(`член профсоюза`). English coverage includes `epilepsy`, `gay`, `Armenian`,
+`autism`, `transgender`, `wheelchair user`, `trade union member` and
+`served time for`. Country, region and language words (`Узбекистан`, `Чувашия`,
+`грузинский язык`) are not attributes. `vocabulary.personalAttributes` adds caller
+stems. Explicit category cues such as `по национальности X`,
 `исповедует X`, `болеет X`, `на учете у X`, `diagnosed with X` and `ethnicity: X`
 protect one following word (including hyphenated words) up to 64 characters,
 including unfamiliar values outside the lexicon. These cues stay on one line.
-Ambiguous English `disabled`, `single`, `man`, `men`, `russian`
-and `asian` require an adjacent personal cue (pronoun/person role, personal
+Ambiguous English `disabled`, `single`, `man`, `men`, `russian`,
+`asian`, `uzbek`, `korean`, `chinese`, `georgian`, `kazakh`, `roma` and
+`union member` require an adjacent personal cue (pronoun/person role, personal
 label, or following person role). Thus `he is disabled`, `candidate is single`
 and `gender: male` redact, while `the module is disabled`, `single run finished`,
 `single-flight lock acquired`, `man page missing`, `locale russian selected`
-and `asian region endpoint` remain intact. Bare operational JSON statuses
+`asian region endpoint`, `Korean locale enabled` and `union member list`
+remain intact. Bare operational JSON statuses
 `disabled`, `single` and `Russian` also remain intact; explicit personal labels
 and caller-declared known personal values still redact. A cue in another clause
 or string value does not make an ambiguous word personal.