@link-foundation/sensitive-data-sanitizer 0.0.0-stage → 0.12.0

package/docs/API.md

@@ -0,0 +1,336 @@
+# Library and detector contract
+
+Import the ESM root entry; complete declarations are in `src/index.d.ts`. Inputs are strings, offsets are UTF-16 code units, starts are zero-based, and ends are exclusive. Returned line and column numbers are one-based; columns also count UTF-16 units.
+
+## Methods
+
+| Method                                 | Behavior                                                                                                     |
+| -------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
+| `createSanitizer(options)`             | Asynchronous native + required Secretlint + selected detector union                                          |
+| `engine.inspect(text)`                 | Detect, validate, deduplicate, apply public policy, and return metadata                                      |
+| `engine.sanitize(text)`                | Detect, union overlapping spans, mask completely, verify every engine, return `{text, findings, redactions}` |
+| `engine.sanitizeJsonl(text)`           | Batched structured JSONL sanitization using the same options and required engines                            |
+| `inspect(text, options)`               | Synchronous native detection and caller findings                                                             |
+| `sanitize(text, options)`              | Synchronous native redaction and native residual verification                                                |
+| `redact(text, findings, options)`      | Apply validated ranges without automatic detection/verification                                              |
+| `auditGitHistory(repository, options)` | Read-only audit of local reachable text objects                                                              |
+
+`redactions` counts merged replacement spans. `findings` can contain several detector hits for one replacement. Findings have `{start, end, type, category, rule, line, column}`; no text, quote, match, entropy score, provider details, or raw diagnostic is copied. Debugging is off by default; `debug(event)` receives only `{event: 'detected', count}`.
+
+## Options
+
+| Option             | Default / meaning                                                                             |
+| ------------------ | --------------------------------------------------------------------------------------------- |
+| `profile`          | Off; `'publication'` sets a 0.3 personal threshold, identity masking and strict fake policy   |
+| `knownSecrets`     | `[]`; exact literal credential values, regardless of length or format                         |
+| `knownPersonal`    | `[]`; `{type, value}` literals with Unicode case-insensitive matching                         |
+| `publicEntities`   | `[]`; exact PERSON/ORGANIZATION/EMAIL exception with HTTPS source and review date             |
+| `findings`         | `[]`; validated detections for the original input, supplied by the caller                     |
+| `detectors`        | `[]`; additional required callbacks in the asynchronous engine                                |
+| `secretlint`       | `true` in the asynchronous engine                                                             |
+| `decode`           | `true`; bounded base64/base64url, hex and percent decoding                                    |
+| `paranoid`         | `false`; entropy-based supplemental credential heuristic                                      |
+| `maxInputLength`   | 33,554,432 UTF-16 units                                                                       |
+| `maxFindings`      | 100,000 detector hits before deduplication                                                    |
+| `minConfidence`    | 0.5; personal findings below this score remain visible to inspect but are not replaced        |
+| `threshold`        | Alias for minConfidence; both options must agree when supplied                                |
+| `debug`            | Optional trusted callback for counts only                                                     |
+| `identityMask`     | `false`; identity types keep 2+2 Unicode characters with fixed `***`, or redact below five    |
+| `fakeIdentity`     | Off; `'specimen-and-synthetic'` retains explicitly fake identity values                       |
+| `fakeValues`       | `[]`; exact caller-reviewed identity values to retain, regardless of fakeIdentity             |
+| `structured`       | Off; `'json'` or `'jsonl'` validates and sanitizes string keys/values separately              |
+| `structuralFields` | `[]`; declared property names suppress weak name/ID/entropy heuristics in their string values |
+
+Unknown options and invalid limits are errors. `knownSecretsFromEnv(env, names?)` discovers credential-named variables when names are omitted, or uses an explicit selection, deduplicating nonempty values. It never invokes shell commands or reads auth files. `personalVariants(entries)` generates numeric and ISO/day-first date spellings; name case variants are already handled by detection. Neither helper invents transliterations.
+
+## Required detectors
+
+A detector supplies `detect(text)`, returning an array of `{start, end, type, category, rule}` or a Promise of that array. Every range must refer to the exact input passed to that invocation. `category` is `credential` or `personal`; types are uppercase identifiers, rules are bounded lowercase identifiers. Throw if analysis is incomplete. The framework invokes engines on original input, normalized input when changed, bounded decoded views, and finally the sanitized output. Decoded matches redact the entire original encoded run. Malformed findings and engine errors block publication with generic error codes.
+
+Callbacks, policies, binaries, model packages, and their rule identifiers are trusted code/configuration. A malicious adapter can omit findings, leak input through its own logging/network calls, or never resolve. Local subprocess helpers use 30-second termination and 10 MiB output caps; arbitrary callback time budgets belong to the embedding application. Pin binary/model versions in deployments. Do not convert a failed scan into an empty result.
+
+## Report adapters
+
+- `fromGitleaks(text, report)` matches every literal `Secret` occurrence and supported decoded representations.
+- `fromTrufflehog(text, report)` maps `Raw`/`RawV2` to every literal or bounded decoded occurrence. Unmapped raw values are errors.
+- `fromDetectSecrets(text, baseline, filename)` redacts each flagged complete line because hashed baselines contain no recoverable secret span.
+- `fromPresidio(text, results)` converts Python Unicode codepoint ranges, mapping standard PII types and credential passwords.
+- `fromOffsetReport(text, entries, {unit, rule})` accepts explicitly declared `utf16`, `codepoint`, or UTF-8 `byte` ranges. Entries supply category/type. Unknown units and partial UTF-8 character boundaries fail.
+
+`codePointRange()` and `byteRange()` expose the same conversions. Extra provider fields are discarded. Reject incomplete/paginated/truncated reports in the caller before adaptation.
+
+For [Google findings](https://docs.cloud.google.com/java/docs/reference/google-cloud-dlp/latest/com.google.privacy.dlp.v2.Location), convert `location.byteRange` numeric strings to numbers and use `unit: 'byte'`; map infoTypes to appropriate category/type. Use only a report for the exact single text item, not offsets relative to a table cell. For [Azure](https://learn.microsoft.com/en-us/rest/api/language/analyze-text/analyze-text/analyze-text?view=rest-language-analyze-text-2026-05-01), request `stringIndexType: 'Utf16CodeUnit'` and use `offset` through `offset + length` with `unit: 'utf16'`. AWS's [PII entity documentation](https://docs.aws.amazon.com/comprehend/latest/APIReference/API_PiiEntity.html) describes character offsets; establish non-ASCII/end-boundary behavior for the SDK/report version before choosing an explicit unit. There is no guessed cloud offset default.
+
+The native package does not send text to these services. An application that chooses remote detection controls its own consent, regions, credentials, and retention.
+
+## Transformations and confidence
+
+Findings also carry `confidence` (0–1) and `likelihood` (`VERY_UNLIKELY` through `VERY_LIKELY`). Native scores describe rule strength; they are not probabilities calibrated across engines. External detectors can supply their own validated scores.
+
+Bare checksum coincidences generally score 0.35. Identity labels raise confidence; specific national structures can supply stronger evidence independently. Timestamp/elapsed metadata and plausible epoch numbers are negative evidence. `minConfidence` defaults to 0.5, applies to personal findings in every detection/rendering/verification path, and never weakens credential protection. Set it to 0.3 to replace bare checksum matches. Generic calendar dates remain unchanged unless personal/birth context identifies them.
+
+`profile: 'publication'` sets `minConfidence: 0.3`, `identityMask: true` and
+`fakeIdentity: false` unless explicitly overridden. It retains required engines,
+decoding, input limits and full-redaction residual verification. Both synchronous
+and async APIs accept it; CLI `redact` uses it by default. Set `identityMask: false`
+for complete identity replacement. A supplied `threshold` is honored, and
+conflicting threshold aliases still fail configuration validation. Formatted
+checksum-valid SNILS and 12-digit personal INN score 0.99 even without this
+profile; bare 10-digit organizational INN remains 0.35. See the effective
+[national-ID defaults](COVERAGE.md#national-id-defaults).
+
+Without a profile, full redaction is the library default. `transformation` selects a global mode; `transformations` maps finding types to overrides. Credentials receive full redaction except for explicit `hive-mask` or `fake` with `credentials: true`:
+
+| Mode                | Configuration and behavior                                                                                             |
+| ------------------- | ---------------------------------------------------------------------------------------------------------------------- |
+| `redact`            | Complete `[REDACTED]` replacement                                                                                      |
+| `hive-mask`         | First/last three Unicode characters with `…` for values longer than 12; shorter values remain opaque                   |
+| `mask`              | Personal data only; `keepStart`/`keepEnd` reveal a bounded prefix/suffix and mask the rest                             |
+| `pseudonym`         | Personal data only; keyed HMAC-SHA256, type-separated deterministic opaque identifier                                  |
+| `format-preserving` | Personal data only; keyed deterministic character-class pseudonym, preserving punctuation and ASCII letter/digit shape |
+| `fake`              | Keyed realistic names, valid dates/MRZ/checksummed numbers and reserved contact destinations                           |
+| `date-shift`        | Personal ISO dates; finite integer `days`, UTC calendar arithmetic; invalid dates become opaque                        |
+| `bucket`            | Personal safe integers; positive integer `size`, inclusive numeric interval                                            |
+
+All keyed modes require a key of at least 16 UTF-8 bytes. `format-preserving` is a non-reversible shape pseudonym, not NIST FF1 encryption; it does not preserve checksum validity, country-specific alphabets or guarantee uniqueness. Partial masks, date shifts and buckets intentionally retain information. Keep pseudonym keys outside logs. Residual verification checks the corresponding fully redacted representation; opted-in disclosures can remain recognizable in the transformed output. Use default redaction for publication that requires opaque output.
+
+`{mode: 'fake', key}` infers name locale, script and gender from the supported gazetteer/datasets and recognizable surname/patronymic endings. Independent HMAC-seeded Faker instances make results independent of processing order. Russian Cyrillic/transliterated spelling, name order and filename separators share component identities. Ambiguous names can use a fallback locale; this is not universal name/gender inference. Dates use one keyed backward offset of 1–365 days and preserve ISO, slash and dot formats. Complete TD1/TD2/TD3 MRZs regenerate individual/composite checks and match separately faked name/date fields. Issuer/nationality remain original unless `mrzCountry: 'UTO'` is supplied. Unsupported/damaged formats fall back to full redaction.
+
+Implemented national checksums, IBANs and payment-card Luhn checks remain valid. Phones retain calling code, length and separators; NANP uses 555-0100–0199 and UK mobile uses 07700 900000–900999. Email/domain outputs use example.com/.test. Other national identities and phone numbers have no universal reserved range: generated arithmetic does not prove that a value is unassigned.
+
+Generated replacements carry `faked: true`. A private bounded registry recognizes values produced by the same transformation context as `kept: 'fake'` during later inspection/verification. Caller-supplied flags and arbitrary FAKE-looking strings grant no exemption. A fresh sanitizer does not inherit that registry; reviewed persisted fake identity values can be configured through `fakeValues`. Credentials remain fully redacted unless `credentials: true` requests provider-shaped values containing FAKE and punctuation that violates token grammar. Generated personal values cannot exempt credential spans.
+
+Faking is pseudonymisation: a private key and retained ages, relations and structure can permit linking. Keep the key out of published data. See [the realistic fake example](../examples/realistic-fakes.mjs) and [the complete requirement/research record](case-studies/issue-24/REQUIREMENTS.md).
+
+`preserveEncoding: true` sanitizes decoded payloads with the same required engine union, re-encodes them and verifies decoding reproduces the sanitized content. Supported formats include base64/base64url, hex, percent, numeric HTML and UTF-8 byte escapes. Escaped JSON string content is sanitized and re-encoded even under default redaction so nested session JSON stays parseable. Malformed or unsupported encodings block opted-in preservation. Known encoding candidates and nesting are bounded.
+
+## Identity policy and structured sessions
+
+`identityMask: true` applies `{mode: 'mask', keepStart: 2, keepEnd: 2,
+marker: '***', minLength: 5}` to document/travel/national-ID types, including
+catalog passport types. Explicit per-type/global transformations take precedence.
+`mask.marker` is an optional fixed string; omitting it preserves length-based
+asterisks. `mask.minLength` defaults to five for identity types and zero for other
+types. Length and retained boundaries count Unicode characters.
+
+`fakeIdentity: 'specimen-and-synthetic'` recognizes ICAO Utopia/Eriksson specimens,
+national specimen markers, repeated document digits and at least six consecutive
+ascending/descending document digits. `fakeValues` supplies exact reviewed
+identity values independently of that option. Kept findings carry `kept: 'fake'`
+and do not count as redactions. The same value check applies during verification.
+Words such as test/example/fake do not grant exemptions. Credentials from any
+engine and overlapping known secrets always win. Both policies are opt-in.
+
+```js
+const sessionSanitizer = createSanitizer({
+  structured: 'jsonl',
+  structuralFields: ['id', 'type', 'name', 'description'],
+});
+const result = await sessionSanitizer.sanitizeJsonl(sessionText);
+```
+
+Structured mode scans string keys and values in bounded batches, supplies field
+context, retains original source offsets/formatting, and verifies parseable
+output. Numbers, booleans and null remain their original types. Declarations
+apply to property names at every nesting level, including string array elements
+inside that property. Declare only fields your application treats as metadata:
+`name` may also contain a person's name. Known private values, explicit provider
+credentials, password/auth context, document formats and other strong findings
+remain protected in declared fields. Custom required detectors still run.
+Escaped nested JSON retains encoding. Invalid JSON and key collisions block
+publication with `ERR_JSON` / `ERR_JSON_COLLISION`.
+
+Inferred personal values propagate only inside JSON string values. They never
+rewrite keys, numbers, booleans, null or structural punctuation. Direct findings
+on sensitive non-identifier keys and explicitly known private keys remain
+protected. Personal name labels require plausible name words; lowercase code
+identifiers, product/tool words and punctuation do not establish a person.
+Place labels require natural place text rather than code expressions or lowercase
+status identifiers. Explicit `knownPersonal` values override these heuristics.
+
+Oversized encoded runs receive a conservative complete-run `ENCODED_LIMIT`
+redaction rather than aborting the record. Decoding probes are capped at 4,096
+characters; full decoding stays capped at 8,192 characters. Whole-input,
+candidate-count, record and worker limits still apply. This fallback can remove
+large non-sensitive encoded outputs; its finding makes that decision auditable.
+
+## Streaming and worker publication
+
+`sanitizeStream(source, options)` accepts iterable bytes/strings and yields sanitized records. It holds incomplete lines, quoted multiline values, PEM/PuTTY blocks, netrc triplets and wrapped base64 groups across chunks. Defaults: 256 KiB batch and 1 GiB total input. Plain-text records inherit the total input limit unless `maxRecordBytes` is set explicitly; JSONL retains its 8 MiB record default. A record exceeding an explicit `maxRecordBytes` fails. Incomplete records are held whole, so memory grows with the largest held record. Boundary scans occur at geometrically increasing sizes to avoid repeated scans of long lines. The generator may have yielded earlier verified records when a later record fails. Use `sanitizeStreamToFile` for atomic publication of the whole stream.
+
+For sessions above the whole-input limit, use `sanitizeStream(source, {structured: 'jsonl'})` alongside `engine.sanitizeJsonl(text)`. JSONL defaults to 8 MiB per record, supports arbitrary UTF-8 chunk boundaries and CRLF, and scans bounded record batches with two persistent workers in input order. `workers` selects 1–16; each worker has a 60-second job deadline. Node requests a 256 MiB old-generation heap and a 4 MiB stack; parent V8 sizing flags can override heap limits. Bun ignores worker resourceLimits, so its byte/queue limits and deadlines still apply but its worker heap is not capped. Worker failures and consumer cancellation terminate all workers. `workers: 1` runs in process; callbacks/custom sanitizers default to one and cannot be combined with parallel workers. Deno uses its in-process fallback because its Node compatibility lacks resource-limited worker threads.
+
+Each stream carries confirmed names, MRZ components and strong personal findings
+into later batches, including bounded Cyrillic/passport/GOST spelling aliases.
+Workers receive ordered private registry snapshots. `maxRegistryValues` defaults
+to 100,000 combined confirmed aliases and prior-token digests; exhaustion fails
+with `ERR_LIMIT`. Prior unredacted word tokens and weak inspected identity spans
+are retained as per-stream keyed HMAC digests, not plaintext history. Repeated
+tokens are deduplicated before hashing. A late confirmation matching this history
+fails with `ERR_LATE_PERSONAL`; this conservative component check can block
+ordinary words reused in a later personal name, username or address. Use `sanitizeStreamToFile` or CLI streaming for
+atomic publication: an unfinished generator can already have yielded earlier
+records and must not be published directly. Registries are never shared between
+streams. `knownPersonal` remains available for caller-confirmed private values;
+`confirmedPersonal` supplies boundary-matched initial values when needed.
+
+```js
+import { createReadStream } from 'node:fs';
+import { sanitizeStreamToFile } from '@link-foundation/sensitive-data-sanitizer';
+await sanitizeStreamToFile(
+  createReadStream('large-session.jsonl'),
+  'safe.jsonl',
+  {
+    profile: 'publication',
+    structured: 'jsonl',
+    workers: 2,
+  }
+);
+```
+
+`sanitizeFileToFile` rejects source symlinks and writes private temporary output, publishing only after complete verification. `sanitizeFileBounded` additionally uses a worker with a 60-second deadline and requests a 256 MiB old-generation heap and 4 MiB stack on Node (Bun ignores those resourceLimits); `workerHeapMb` and `workerTimeoutMs` are configurable. Worker failure removes its private staging directory before returning. Limits bound package work, not arbitrary external detectors. Callbacks supplied through `sanitizer` run in process; set `worker: false` explicitly to disable isolation. Pass serializable options through `sanitizerOptions` for workers. Memory limits do not include all native/WASM allocation.
+
+```js
+import { sanitizeFileBounded } from '@link-foundation/sensitive-data-sanitizer';
+await sanitizeFileBounded('session.jsonl', 'session.safe.jsonl', {
+  profile: 'publication',
+  maxTotalBytes: 1024 * 1024 * 1024,
+  maxRecordBytes: 1024 * 1024,
+});
+```
+
+The CLI supports `redact FILE --stream --output NEWFILE` and stdin streaming. Stdout uses a private spool and receives nothing until the entire scan succeeds. `--max-record-bytes` controls held records; `--max-bytes` defaults to 1 GiB with `--stream`. `--json` / `--jsonl`, `--hive-mask`, `--preserve-encoding` and `--gh-auth` are explicit opt-ins. CLI redact uses the publication profile and identity masking by default; configure `identityMask: false` for full redaction.
+
+## Public verification and publication helpers
+
+Built-in Wikidata names/organizations, resolver/documentation IPs and role emails on known organization domains are enabled by default; `publicKnowledge: false` disables those heuristics. `knownPersonal` forces private treatment. `publicEntities` remains an audited exact override; credentials always outrank exemptions.
+
+`createWikidataVerifier(options)` provides optional asynchronous `verifyPublic`; the CLI enables it only with `--verify-public`. It sends eligible PERSON/ORGANIZATION names to Wikidata, requires an exact label and a public-role description, caches bounded results, and blocks output on network failure. Name collisions remain possible: this is a public-name heuristic, not proof of a person's identity. Explicit private literals and sensitive relationship contexts prevent automatic exemptions. Offline mode makes no network requests.
+
+`knownSecretsFromGitHubAuth({command, hostname, timeoutMs})` explicitly invokes local `gh auth token` and returns its nonempty exact value without diagnostics. It is never called automatically; failed authentication blocks opted-in scans. `knownSecretsFromEnv` remains separate and reads credential-named environment values or an explicit name selection.
+
+`sanitizePayload(payload, {sanitizer})` sanitizes JSON payload values with the required union while retaining JSON structure. `createSentryBeforeSend(options)` returns a Sentry hook that drops the event (`null`) if sanitization fails. `createOutboundSanitizer(send, options)` calls the supplied transport only after successful sanitization. These helpers do not send data themselves.
+
+The optional ESLint plugin exports `require-sanitized-output` from the package's `eslint-plugin` subpath. It checks configured outbound sinks and common GitHub CLI body/title arguments, including `--body-file`, templates and argv arrays. Configure `sinks` and `sanitizers` for your application. It is a conservative syntactic aid, not a complete data-flow proof; runtime verification remains required.
+
+## Native documented entity catalogs
+
+`entityCatalogs` exposes source URLs, source-page SHA256 and documented names for Google (213), Azure (176) and AWS (36). Their canonical uppercase underscore labels are recognized natively, with complete value spans, independent of cloud services. The package also implements selected automatic formats and checksum algorithms. Catalog vocabulary support does not reproduce proprietary NER, image/document inspection, regional identity validators, cloud risk analysis or their accuracy. See the coverage table for that distinction.
+
+## Two-pass session publication
+
+Pass `twoPass: true` to `sanitizeStreamToFile`, `sanitizeFileToFile` or
+`sanitizeFileBounded` when personal values may be confirmed after their first
+appearance. Pass 1 reads the source once into a private temporary input file and
+collects confirmed values with every configured detector. Pass 2 reads that
+snapshot and redacts with the complete registry, including in parallel JSONL
+workers. Publication is atomic; success and failure remove the input spool and
+unfinished output. Existing targets remain intact on failure. Raw input is
+stored with mode `0600`; budget disk space for both input and sanitized output.
+
+```js
+await sanitizeStreamToFile(createReadStream('session.jsonl'), 'safe.jsonl', {
+  structured: 'jsonl',
+  profile: 'publication',
+  twoPass: true,
+  workers: 4,
+});
+```
+
+The default remains one pass. `twoPass` is supported only by atomic file APIs;
+`sanitizeStream` rejects it with `ERR_CONFIG`. The CLI equivalent is
+`redact session.jsonl --stream --jsonl --two-pass --output safe.jsonl`, including
+stdin and `--in-place`. Record, input-byte and registry bounds still apply;
+discovery retains confirmed values only and does not retain prior-token history.
+Required detectors run during collection and final publication verification.
+
+Confirmation values are trimmed before normalization in both passes. Registry
+matches use the `confirmed-personal` finding rule and do not seed new values
+from their matched spans. Pass 2 redacts any additional confirmation without
+raising `ERR_LATE_PERSONAL`; it still enforces registry limits and residual
+verification. Optional `debug` reports a `two-pass-confirmation` event containing
+only the entity type when that happens. One-pass streaming retains its late
+confirmation guard.
+
+Personal label values stop at escaped line breaks and quotes and are capped at
+512 UTF-16 units. Phone and email labels require matching value shapes. Inferred
+propagation seeds are capped at 256 units; longer detected spans are still
+redacted but do not enter the propagation registry. `debug` reports a
+`propagation-skipped` event containing only the type and reason when this happens.
+
+Credential label values stay inside their logical line and enclosing JSON
+string: they stop at escaped line breaks, escape sequences and the closing quote,
+and never continue into the next JSON key. A qualifier such as `token for
+staging:` needs a separator on the same line, verbs such as `is` must be whole
+words, and generic labels such as `token`, `sig` or `secret` followed by a verb
+need a generated-looking value, so prose such as `the token is valid` remains
+text. A credential covering a whole escaped JSON run is redacted as one value.
+
+Residual verification redacts the fully masked output again when any engine
+still finds an actionable value, for example when raw and decoded layers
+disagree about a span. Up to three additional rounds run; each reports a
+`residual-redaction` debug event containing only the round number. Output that
+does not converge, or whose structure becomes invalid, still fails closed with
+`ERR_RESIDUAL` or the matching structured error.
+Ordinary identifier keys retain their spelling unless explicitly configured as
+private. Document names require a filename extension or path plus name-shaped
+words or a known given name. Known CI service accounts in home paths and the
+GitHub runner workspace are preserved; caller-confirmed private values retain
+precedence.
+
+Valid JSON lone surrogate escapes (such as `\ud83d`) remain escaped and round-trip
+through redaction alongside emoji. Malformed UTF-8 bytes in structured JSONL still block publication
+with `ERR_ENCODING`. For JSONL streams, the error includes a one-based `record`,
+a zero-based absolute byte `offset`, and a zero-based `recordOffset`, identifying
+the start of the malformed sequence without exposing its contents. These
+positions survive bounded file workers and appear in CLI diagnostics.
+
+## Shared tool logs and byte preservation
+
+`credentialsOnly: true` filters all personal findings, including supplied,
+known, inferred, decoded, external-detector and structured findings. Credential
+findings retain the normal engine union, confidence rules and verification.
+This keeps EMAIL/account-ID values without per-value `publicEntities` entries.
+The default policy continues to protect personal data.
+
+`knownSecretsFromEnv(env = process.env, names?)` discovers nonempty exact values
+when `names` is omitted, using
+`/(?:^|_)(?:TOKEN|(?:API|ACCESS|PRIVATE)?KEY|SECRET|PASSWORD|PASSWD|CREDENTIALS?|AUTH|PAT)(?:_|$)/i`.
+This covers `MY_SERVICE_TOKEN`, `CLAUDE_CODE_OAUTH_TOKEN`, `CODEX_API_KEY`,
+`OPENCODE_API_KEY`, `DASHSCOPE_API_KEY`, `HF_TOKEN` and `TELEGRAM_BOT_TOKEN`.
+An explicit array keeps the previous selection behavior; `[]` selects nothing.
+These values are loaded only when the helper or CLI `--env-secrets` is requested.
+
+`await knownSecretsFromGitHubConfig({path?})` reads `oauth_token` and `api_token`
+strings from every host/account in a local YAML file. The default follows gh's
+`GH_CONFIG_DIR`, `XDG_CONFIG_HOME`, Windows `AppData`, then `~/.config/gh`
+precedence and appends `hosts.yml`. A missing file returns `[]`; malformed YAML,
+symlinks, aliases, files above 1 MiB and read failures produce value-free
+`ERR_AUTH`. YAML quoting is parsed with the `yaml` package, with alias expansion
+disabled. CLI `--gh-config` adds these exact values to `knownSecrets`.
+`knownSecretsFromGitHubAuth` / `--gh-auth` remain available independently.
+
+Plain log streams and CLI file/stdin/output/in-place paths preserve invalid
+UTF-8 bytes exactly. Valid Unicode keeps its normal meaning for detection.
+String APIs can use a reversible view:
+
+```js
+const view = decodeLogBytes(inputBytes);
+const result = await createSanitizer({ credentialsOnly: true }).sanitize(view);
+const outputBytes = encodeLogText(result.text);
+```
+
+`decodeLogBytes(Uint8Array)` represents each invalid byte with an unpaired
+U+DC80–U+DCFF surrogate; `encodeLogText(string)` restores those bytes. Use the
+encoder when collecting `sanitizeStream` string output containing such bytes.
+Ordinary UTF-8 string encoding would replace them and lose information.
+`sanitizeStreamToFile` and `sanitizeFileBounded` perform the encoding themselves,
+including workers and both passes. NUL bytes still identify binary input and
+block publication. JSON/JSONL, scanner protocol files and configuration retain
+strict UTF-8 validation; malformed bytes cannot represent valid JSON text.
+
+Default detection keeps RFC 1918, loopback, link-local and IPv6 local addresses,
+recognized auth-mode metadata, npm dependency version declarations, Actions
+permission values and literal credential references. `publicKnowledge: false`
+restores IP detection; explicit `knownSecrets` and `knownPersonal` retain
+priority. No shell command or workflow expression is executed.

package/docs/COVERAGE.md

@@ -0,0 +1,136 @@
+# Detection coverage and limits
+
+Coverage is a union of maintained detectors, explicit known values, context and conservative native rules. Detection is heuristic; there is no claim of perfect recall, universal anonymity, or superiority over every competitor.
+
+## Credential coverage
+
+The native provider supplements in `src/rules.js` cover GitHub, GitLab, OpenAI, Anthropic, AWS access IDs, Google API/OAuth, Slack, Stripe, SendGrid, Twilio, npm, PyPI, Telegram, Discord, Hugging Face, Shopify, Databricks, Square, Mailchimp, DigitalOcean, Vault, Grafana, Linear, Notion, Vercel, and JWT-shaped values. Prefix matches also protect selected truncated tokens; they do not validate a token with a provider.
+
+The default native engine bundles 221 text rules from Gitleaks 8.30.1 (222 upstream rules; the path-only PKCS12 rule is outside text detection), compiled unchanged with RE2/WASM or RE2JS for inputs above 4 MiB. The MIT license, commit and sync script are retained in `src/vendor/gitleaks`. Keyword prefilters and configured entropy thresholds run natively; upstream comment/path allowlists cannot suppress a sanitizer finding. Quoted hex/base64 entropy detection supplements those rules. The default asynchronous engine additionally uses the pinned Secretlint recommended scanners, excluding its comment-filter rule. Embedded disable directives in untrusted text cannot suppress findings. Optional Gitleaks and TruffleHog invoke their installed detector sets, and custom detectors can add formats. Service counts are not a measure of comparative recall.
+
+Context rules recognize credential labels with bounded key affixes, JSON and nested escaped JSON, YAML/env/INI assignments, shell flags, XML values, URL userinfo, authorization headers, cookies, webhook paths, and complete or truncated private-key PEM blocks. Low-entropy contextual values such as `password=1234` are protected. Generic token counters, booleans/null/language literals and ordinary Git hashes are excluded to retain useful logs. Supply literal known values for ambiguous credentials or unusual key names.
+
+Broader unquoted entropy scanning is opt-in (`paranoid: true`), uses Shannon entropy ≥4.2 on bounded candidates, and exempts ordinary hex hashes and common tool identifiers. Quoted hex/base64 and entropy checks belonging to bundled provider rules run by default. Entropy has both false positives and false negatives; contextual detection and known-secret dictionaries remain necessary.
+
+## Personal data and languages
+
+Native rules cover international/Unicode emails, international and selected national-format phones, Luhn-valid payment cards, US SSNs, IPv4/IPv6, checksum-validated IBAN/CPF/Spanish DNI/US NPI, Indian PAN, UK NI, wallet shapes, MACs, messenger handles, home usernames, labelled PII and selected postal address grammars. A bundled 17-locale name gazetteer supplies default free-text detection, including selected inflections/transliterations. Each locale has credential prose and eight labelled PII fixtures. This finite dictionary/grammar is not unrestricted statistical NER; unfamiliar names and unsupported national formats require known values or additional recognizers.
+
+Credential labels have fixtures for 17 language variants:
+
+| Language            | Example credential label |
+| ------------------- | ------------------------ |
+| English             | password                 |
+| Portuguese          | senha                    |
+| Spanish             | contraseña               |
+| French              | mot_de_passe             |
+| German              | passwort                 |
+| Russian             | пароль                   |
+| Simplified Chinese  | 密码                     |
+| Traditional Chinese | 密碼                     |
+| Japanese            | パスワード               |
+| Korean              | 비밀번호                 |
+| Arabic              | كلمة المرور              |
+| Hebrew              | סיסמה                    |
+| Hindi               | पासवर्ड                  |
+| Turkish             | şifre                    |
+| Indonesian          | kata_sandi               |
+| Vietnamese          | mật_khẩu                 |
+| Thai                | รหัสผ่าน                 |
+
+These are tested label patterns, not 17 complete NER language models. `src/locales.js` and `src/rules.js` define the exact vocabulary. Unicode NFKC, combining marks, common invisible/bidi controls, ANSI escapes and JSON Unicode escapes are projected to a detection view with original offset mapping. Original delimiters/content outside redactions remain intact. NFKC does not equate every homoglyph or transliteration.
+
+The optional [Presidio bridge](../examples/presidio-bridge.py) accepts `--language` and `--model` for separately installed spaCy models. [Presidio's multilingual guide](https://presidio.dataprivacystack.org/tutorial/05_languages/) explains that models and recognizers must be configured per language. Installing one English model does not provide every language. There is no runtime model download. Offline public classification uses bundled sourced entries and documented role/domain heuristics. Callers can add reviewed exact policies or explicitly enable Wikidata verification; known-private values and credential overlaps always retain private treatment.
+
+## Identity and additional credential formats
+
+Native MRZ extraction covers TD1/TD2/TD3 and MRV-A/MRV-B, including standalone
+name/data fragments and bounded whitespace reflow. Check digits raise confidence;
+invalid checks do not allow MRZ through. Country passport shapes include Russian
+domestic/international, US/UK, German, Indian, Korean and Italian documents.
+Ambiguous bare nine-digit values receive lower confidence and redact under the publication profile.
+Travel labels protect PNR/booking references, 13-digit tickets and visa numbers.
+Birth-date vocabulary supports numeric and English/Russian month-name dates.
+
+Native country types: UK_NHS, KR_RRN, IT_FISCAL_CODE, SG_NRIC_FIN, PL_PESEL,
+SE_PERSONNUMMER, ZA_ID_NUMBER, AU_TFN, CA_SIN, RU_SNILS and RU_INN. Checks validate
+format arithmetic, not issuance; modern Korean RRNs may use randomized suffixes,
+so damaged/non-legacy checks still receive contextual or shape coverage. Bare
+checksum-only numeric formats receive low confidence (usually 0.35) and are
+inspectable but unchanged under the default 0.5 personal threshold. Specific
+structures and identity/health context supply stronger evidence. Card detection
+requires a supported network prefix and length as well as Luhn; timestamp
+contexts and epoch-shaped numbers are negatives.
+Messenger/call context adds RU/VN/TH/ID national trunk-prefix phone shapes.
+
+Confirmed names and strong personal values propagate within a document and
+across later streaming batches, including filename/path occurrences and bounded
+Cyrillic/ICAO/GOST/common spelling aliases. Russian patronymics, capitalized or
+all-caps Slavic surname pairs in either order, and two/three-part names adjacent
+to document filename words supplement the gazetteer. Late confirmations and
+registry exhaustion block completed stream publication; see [API.md](API.md).
+
+### National-ID defaults
+
+Scores below apply to valid reported shapes outside timestamp/order metadata.
+Default API personal threshold is 0.5; publication profile/CLI redact use 0.3.
+Identity/health labels generally raise checksum-valid scores to 0.99 and damaged
+checks to 0.9. These scores indicate rule strength, not issuance or probability.
+
+| Type/shape                                    | Bare score | API default  | Publication profile |
+| --------------------------------------------- | ---------- | ------------ | ------------------- |
+| UK_NHS, spaced or compact                     | 0.35       | Inspect only | Replace/mask        |
+| KR_RRN, plausible date and checksum           | 0.99       | Replace      | Replace/mask        |
+| IT_FISCAL_CODE, alphanumeric structure        | 0.99       | Replace      | Replace/mask        |
+| SG_NRIC_FIN, prefix and check letter          | 0.99       | Replace      | Replace/mask        |
+| PL_PESEL, valid checksum                      | 0.35       | Inspect only | Replace/mask        |
+| SE_PERSONNUMMER, valid formatted checksum     | 0.35       | Inspect only | Replace/mask        |
+| ZA_ID_NUMBER, valid checksum                  | 0.35       | Inspect only | Replace/mask        |
+| AU_TFN, valid spaced/compact checksum         | 0.35       | Inspect only | Replace/mask        |
+| CA_SIN, valid spaced/compact checksum         | 0.35       | Inspect only | Replace/mask        |
+| RU_SNILS, `ddd-ddd-ddd dd` / `ddd ddd ddd dd` | 0.99       | Replace      | Replace/mask        |
+| RU_SNILS, compact checksum only               | 0.35       | Inspect only | Replace/mask        |
+| RU_INN, 12 digits/two checks                  | 0.99       | Replace      | Replace/mask        |
+| RU_INN, 10 digits/one check                   | 0.35       | Inspect only | Replace/mask        |
+
+Russian SNILS contexts include СНИЛС, пенсионное, страховое свидетельство and
+страховой номер. English passport contexts include series and number, passport
+series and document number. Unlabelled Russian passport shapes retain their
+0.4 score and are protected by the publication profile. Numeric-noise exclusions
+continue to preserve timestamps, order IDs and routine tool identifiers.
+
+Credential rules add complete netrc triplets, NuGet oy2 keys, all eight WordPress
+key/salt constants and complete/truncated PuTTY v2/v3 private blocks. Streaming
+holds netrc/PuTTY blocks across chunks. Placeholder exclusions do not override
+known secrets. [The requirement matrix](case-studies/repository-issue-13/REQUIREMENTS.md)
+records standards, alternatives and regression coverage for each feature.
+
+## Bounds and unsupported inputs
+
+| Boundary              | Behavior                                                                                              |
+| --------------------- | ----------------------------------------------------------------------------------------------------- |
+| Input                 | 32 MiB CLI bytes / 32 Mi UTF-16 library units by default; configurable                                |
+| Normalized projection | At most 18 times input length for Unicode normalization expansion; input limits apply                 |
+| Encoded candidates    | 8,192 raw characters per decoded run; audited redaction for longer runs; 4,096 candidates, two layers |
+| Directory             | 10,000 files; `.git` and `node_modules` omitted                                                       |
+| Findings              | 100,000 hits; exceeding the limit blocks analysis                                                     |
+| Local subprocess      | Default 30 seconds and 10 MiB report/stdout cap; failures block output                                |
+| Git objects           | 10,000 objects, 10 MiB per object and 100 MiB total by default                                        |
+
+Supported decoding includes canonical base64/base64url, line-wrapped base64, hex, percent, HTML entities, UTF-8 byte escapes and JSON string escapes, with fatal UTF-8 checks for decoded payloads. Plain log files and streams preserve invalid UTF-8 bytes reversibly while scanning surrounding valid text; structured JSON/JSONL and configuration remain strict. Opt-in encoding preservation round-trips sanitized payloads; nested escaped JSON retains its structure by default. Arbitrarily split fragments, unlimited nesting, compressed/encrypted payloads, malformed encodings, archives, PDFs and OCR/images are outside this text profile. Whole-input/candidate limits fail; oversized textual percent/base64 runs receive audited complete-run redaction. Unsupported representations may remain undetected. `decode: false` intentionally reduces coverage. A caller needing another format should decode it through a bounded preprocessing adapter and retain correct original spans.
+
+Default text mode can replace JSON numeric values with string markers and change syntax/types. Opt-in structured JSON/JSONL mode scans string keys/values separately, preserves primitive types, and rejects malformed output or key collisions. Numeric personal values need a separate caller schema policy. Existing quoted JSON strings and escaped-string fixtures retain their delimiters. Review sanitized artifacts for their intended consumer.
+
+An unchanged result is not proof that text contains no sensitive information. The original sensitive input, private config, scanner reports, temporary files during execution, backups, screenshots, and remote caches remain separate handling concerns. POSIX permissions do not create equivalent Windows ACL guarantees or secure memory/disk erasure. In-place replacement creates a new file; it does not preserve ownership metadata, ACLs, or guarantee crash durability of the directory entry.
+
+## Entity catalogs and optional modes
+
+Documented Google/Azure/AWS catalog names (213/176/36) are recognized as explicit native labels, including canonical underscore and original camel-case forms. Generic DATE is excluded so ordinary headers/calendar dates remain intact; personal birth-date context still qualifies. This vocabulary is distinct from automatic vendor NLP/image/cloud-service features. Typed transformations and confidence are described in [API.md](API.md); full redaction remains the default. Public Wikidata names, known role/domain emails and public resolver/documentation IPs use explicit offline knowledge and heuristics. Opt-in network lookup does not establish identity; private overrides and credential priority remain authoritative.
+
+Large text logs use bounded record streaming and worker file publication. Plain-text records inherit the configurable total limit (1 GiB by default), so memory grows with the largest held record. The whole-string default is 32 Mi UTF-16 units. `credentialsOnly: true` preserves personal/account details across all finding sources while retaining credential detection and verification. The review corpus benchmark is a finite, synthetic regression comparison; [method, settings and results](../experiments/benchmark/README.md) explain its scope. It does not demonstrate general market-wide superiority.
+
+Structured JSONL streaming supports 8 MiB records and ordered record workers;
+[issue 24 evidence](case-studies/issue-24/README.md) records the 7 MiB regression
+and reproducible throughput measurement. Realistic fake mode preserves supported
+scripts/formats/checksums and uses reserved contact ranges where available.
+Generated identities are pseudonyms; valid arithmetic does not prove non-issuance.

package/docs/HISTORY-REMEDIATION.md

@@ -0,0 +1,41 @@
+# Auditing and remediating Git history
+
+Deleting a file in the current checkout does not remove its older blobs. The sanitizer's `history` command reads all local refs and reflog-reachable objects, scanning blobs, commit authors/messages and annotated tag metadata. It emits object IDs and safe findings only, and never changes refs or pushes.
+
+```sh
+sensitive-data-sanitizer history /path/to/audit-clone --config /private/policy.json
+```
+
+Exit 1 means findings; exit 2 means incomplete analysis or failure. Trees are traversed by Git but their filename bytes are not sanitized. Binary/non-UTF-8 objects are skipped with `complete: false`. Input/object/count limits prevent treating a partial audit as success. Unreachable objects without refs/reflogs, un-fetched branches, Git LFS storage, PR refs not fetched locally, submodules, release artifacts and remote mirrors are outside that local scope. A local report cannot certify erasure on GitHub.
+
+## Reviewed rewrite procedure
+
+1. Revoke/rotate exposed credentials immediately. Removing their text does not undo exposure.
+2. Inventory affected branches, tags, PR refs, authors/messages, filenames and external artifacts. Keep a separate offline restricted backup of the original mirror and the private value list. Do not refresh the only backup after a rewrite.
+3. Create a fresh disposable clone for rewriting with the maintained [git-filter-repo](https://github.com/newren/git-filter-repo) tool. Record its version and follow its fresh-clone safety requirements. Install it separately; this package does not install or invoke a rewrite automatically.
+4. Create mode-0600 replacement and optional mailmap files outside the repository, using the tool's documented literal syntax. Include known date/phone/name spellings and case/transliteration variants that require removal. Values containing delimiters or newlines require appropriately reviewed callback logic; do not interpolate them into shell commands.
+5. Run the following only in the disposable clone, with reviewed private inputs. Text replacements handle blobs; message replacements and a mailmap address different kinds of metadata. Path filtering requires separate decisions.
+
+   ```sh
+   git filter-repo --sensitive-data-removal --replace-text /private/replacements.txt \
+     --replace-message /private/replacements.txt --mailmap /private/mailmap
+   ```
+
+6. Audit complete rewritten commit snapshots, messages and tags with the sanitizer and independent scanners. Inspect selected revisions, run the repository's tests, and check that intended public values and structure survive. A scan of only added diff lines cannot prove historical removal.
+7. Compare refs and commit mappings against the immutable backup. Remove retained original refs from the _disposable cleaned clone_ according to git-filter-repo documentation; keep the backup private. Re-audit the cleaned clone, including reflogs. Partial rewrites that retain old refs can still expose old objects.
+8. Have the repository owner review the actual rewritten refs, then coordinate publication and collaborator recloning. GitHub branch protections, PR references, caches, forks, Actions artifacts and other clones may require separate cleanup. Follow [GitHub's sensitive-data removal guide](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository), including Support assistance where applicable. Do not issue an unreviewed blanket force push.
+
+The automated regression creates a repository with a credential-bearing file, commits its deletion, and confirms the audit still finds the old blob and author email while leaving refs unchanged. It tests local discovery, not destructive rewriting or remote erasure.
+
+## Automated clone-only preview and rewrite
+
+`history rewrite SOURCE --output NEW_DIRECTORY` creates a new private mirror clone with `--no-local`, audits reachable objects and reflogs, and writes private `replace-text.txt`, `mailmap`, `replacements.json` and callback files for review. The original repository is unchanged. These preview files contain original private values: keep the directory private and remove it after review. The clone's origin remote is removed; no operation pushes.
+
+Add `--apply` to rewrite only the fresh clone using locally installed `git-filter-repo` (tested with 2.47.0). Use `--filter-repo PATH` to select its executable. Exact original-object callbacks sanitize complete blob contents, commit/tag messages, author/committer/tagger identities, filenames and refnames. They avoid replacement-file newline/regex ambiguity. Path/ref collisions fail rather than dropping objects. After filtering, the command audits again, checks names and refs, runs `git fsck --full`, removes private plans and writes a metadata-only manifest. An incomplete audit, binary object, remaining finding or missing tool fails and removes the new directory. A pre-existing destination is never reused or deleted.
+
+```sh
+sensitive-data-sanitizer history rewrite ./repository --output ./private-preview
+sensitive-data-sanitizer history rewrite ./repository --output ./private-rewritten --apply
+```
+
+The library exports `rewriteGitHistory(source, destination, options)` with `apply: false` by default and the same audit limits. A rewritten history changes object IDs and invalidates signatures. Review the clone before any separate repository replacement or remote push. Rotate affected credentials; deleting history cannot revoke a leaked secret or clean other clones. The disposable integration experiment in `experiments/history-rewrite-integration.mjs` verifies deleted blobs, messages, identities, paths and tags with the actual rewrite tool.

package/examples/combined-scanners.mjs

@@ -0,0 +1,23 @@
+import {
+  createSanitizer,
+  createGitleaksDetector,
+  createTrufflehogDetector,
+  createPresidioDetector,
+} from '../src/index.js';
+import { fileURLToPath, URL } from 'node:url';
+
+// Install these optional local scanners before running this example.
+// All selected detectors are required: an unavailable scanner blocks output.
+const sanitizer = createSanitizer({
+  detectors: [
+    createGitleaksDetector(),
+    createTrufflehogDetector(),
+    createPresidioDetector({
+      args: [fileURLToPath(new URL('./presidio-bridge.py', import.meta.url))],
+    }),
+  ],
+});
+const result = await sanitizer.sanitize(
+  'name: Jane Doe; password=short; jane@example.org'
+);
+console.log(result.text);

package/examples/presidio-bridge.py

@@ -0,0 +1,32 @@
+"""Local stdin/stdout adapter. Install Presidio and models before use.
+
+Example: python -m pip install presidio-analyzer
+         python -m spacy download en_core_web_lg
+Configure NlpEngineProvider here for other installed language models.
+This script does not download models or contact a remote analysis service.
+"""
+import argparse
+import json
+import sys
+
+from presidio_analyzer import AnalyzerEngine
+from presidio_analyzer.nlp_engine import NlpEngineProvider
+import spacy
+
+parser = argparse.ArgumentParser()
+parser.add_argument('--language', default='en')
+parser.add_argument('--model', default='en_core_web_lg')
+options = parser.parse_args()
+if not spacy.util.is_package(options.model):
+    raise SystemExit('Install the selected model before running this offline bridge.')
+
+text = sys.stdin.read(10 * 1024 * 1024 + 1)
+if len(text) > 10 * 1024 * 1024:
+    raise SystemExit(2)
+nlp = NlpEngineProvider(nlp_configuration={
+    'nlp_engine_name': 'spacy',
+    'models': [{'lang_code': options.language, 'model_name': options.model}],
+}).create_engine()
+analyzer = AnalyzerEngine(nlp_engine=nlp, supported_languages=[options.language])
+results = analyzer.analyze(text=text, language=options.language)
+json.dump([{"start":r.start,"end":r.end,"entity_type":r.entity_type} for r in results], sys.stdout)

package/examples/realistic-fakes.mjs

@@ -0,0 +1,18 @@
+import { randomBytes } from 'node:crypto';
+import { createSanitizer } from '../src/index.js';
+
+// Persist a private key outside published logs when stable identities are needed.
+const key = process.env.SANITIZER_FAKE_KEY ?? randomBytes(32).toString('hex');
+const sanitizer = createSanitizer({ transformation: { mode: 'fake', key } });
+const input = [
+  'Applicant: Marina Kovaleva',
+  'born on 07/12/1985',
+  'PNR K7QWZP; phone +1 (212) 234-5678',
+  'email marina@private.org; password=short-private',
+].join('\n');
+const result = await sanitizer.sanitize(input);
+console.log(result.text);
+console.log(
+  'Generated fake findings:',
+  result.findings.filter((f) => f.faked).length
+);