@link-foundation/sensitive-data-sanitizer 0.19.3 → 0.21.0

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.21.0
+
+### Minor Changes
+
+- d1d7a22: Resolve the sanitizer requests of issues 130–141. Placeholder mode accepts `transformation.existing: 'keep'` for a second pass over sanitized text, types whole JSON values (`Sensitive-1`, `Age-1`), and keeps pronouns, cue words and sentence-initial greetings and verbs readable beside `Person-n`; compact organization and location spans become one typed token. `reservedPlaceholderTypes()` and `isPlaceholderToken()` expose the namespace. `strict` covers more nationality, religion, health, sexuality, conviction and trade-union wording, and `vocabulary.personalAttributes` adds caller stems. A credentials-only entry (`@link-foundation/sensitive-data-sanitizer/credentials`) starts about three times faster, and both entries support `require`. Credential context stays readable and masks stay stable; configuration settings are kept and an `exempt(finding, text)` hook can keep findings by policy. New detections cover CI credential markers, XML attributes, hash rockets, markup values and plural credential flags. Wrapped base64 and hex keep their layout; candidates are bounded per 64 KiB window; `knownSecrets` also match their hex and base64 spellings. Environment secret discovery selects `PGPASSWORD`-style names and cookies, skips more configuration names, and accepts `minLength`.
+
+## 0.20.0
+
+### Minor Changes
+
+- befe673: Match the strongest features of competing secret and PII tools. Add national, tax, bank and device identifiers from Presidio, OpenRedaction, DataFog and Google; 203 vendored Betterleaks rules; offline CRC32 checks for GitHub and npm tokens (`tokenChecksum`, `tokenWithChecksum`); escape-run, HTML-tag and UTF-16 decoding; `allowList`, `inlineAllow` and `minConfidenceByType`; mask tuning and deterministic `encrypt` with `decrypt`; `createPatternDetector`, `createIdentifierDetector`, `identifierChecks` and the read-only `nationalRules` table; SARIF output (`toSarif`, `scan --format sarif`), multi-path `scan` and pre-commit hooks. Detect an IPv4 address followed by a full stop. Add a competitor comparison with measured results on 853 competitor test cases.
+
 ## 0.19.3
 
 ### Patch Changes

package/README.md

@@ -1,6 +1,6 @@
 # Sensitive Data Sanitizer
 
-An offline JavaScript library, detector framework, and CLI for redacting credentials and personal data from logs, AI sessions, and text files, including logs with invalid UTF-8 bytes. It combines required Secretlint and 221 bundled Gitleaks text rules with contextual credentials, default multilingual name/PII recognition, known values and bounded encoding detection. Optional local Gitleaks, TruffleHog, and Presidio detectors extend the union.
+An offline JavaScript library, detector framework, and CLI for redacting credentials and personal data from logs, AI sessions, and text files, including logs with invalid UTF-8 bytes. It combines required Secretlint, 221 bundled Gitleaks text rules and 203 further Betterleaks rules with contextual credentials, default multilingual name/PII recognition, known values and bounded encoding detection. Optional local Gitleaks, TruffleHog, and Presidio detectors extend the union.
 
 Library calls without a profile replace every actionable finding completely with `[REDACTED]`. Personal findings below the default confidence threshold of 0.5 remain inspectable without replacing ordinary log numbers; credentials always qualify. Publication methods verify full redaction and block output if a required detector fails or finds a residual. Optional typed transformations, compatible partial masks and encoding preservation are described in the API. Findings contain offsets and rule metadata; they omit matched values and source excerpts.
 
@@ -20,6 +20,12 @@ two-pass publication:
 sensitive-data-sanitizer redact session.jsonl --stream --jsonl --two-pass --output safe.jsonl
 ```
 
+For loggers that only mask credentials, import the
+[credentials-only entry](docs/API.md#credentials-only-entry)
+`@link-foundation/sensitive-data-sanitizer/credentials`: it skips the
+personal-data modules and starts about three times faster. Both entries also
+support `require`.
+
 For synchronous log hooks, `createSyncSanitizer` supports custom detectors and
 verified callback transformations. Rule/type controls, caller vocabulary,
 structured field selectors and `compileKnown` configure reusable policies; see
@@ -89,6 +95,13 @@ Known Russian names match in all six cases and in any word order, with `ё`/`е`
 spelling variants and initials (`с Петровым Данилом Сергеевичем`, `Петров Д. С.`).
 A part of a longer name is matched alone only when it is listed itself.
 
+With `properNames: 'capitalized'`, placeholder output keeps the sentence around a
+name (`Кандидат Person-1`, `Меня зовут Person-1`, `Благодарим Вас, Person-1`), and
+`transformation.existing: 'keep'` lets a second pass over sanitized text keep its
+`Type-n` tokens and number new values after them. `isPlaceholderToken` and
+`reservedPlaceholderTypes` expose the namespace; see the
+[second-pass example](examples/second-pass-placeholders.mjs).
+
 For irreversible output, choose `transformation: { mode: 'placeholder', map:
 false }`: no restore map is returned or written. Detection limits still apply
 in either mode. See the [placeholder API](docs/API.md#reversible-placeholders),
@@ -131,6 +144,7 @@ console.log(result.text); // password=[REDACTED] [REDACTED]
 printf 'password=short\n' | sensitive-data-sanitizer redact -
 sensitive-data-sanitizer redact input.txt --in-place
 sensitive-data-sanitizer scan logs --config /private/policy.json
+sensitive-data-sanitizer scan src/a.js src/b.js --format sarif > report.sarif
 sensitive-data-sanitizer redact input.txt --gitleaks --trufflehog --output safe.txt
 sensitive-data-sanitizer redact input.txt --presidio /path/presidio-bridge.py --model en_core_web_sm --language en
 sensitive-data-sanitizer redact large-session.jsonl --jsonl --stream --output safe-session.jsonl
@@ -148,6 +162,8 @@ sensitive-data-sanitizer --help
 
 Defaults: 32 MiB whole-text input bytes, 100,000 findings and 10,000 files. `--stream` permits 1 GiB total with plain-text records up to the total input limit or 8 MiB JSONL records, bounded workers and atomic publication. `--worker-timeout-ms N` sets the worker inactivity/per-batch deadline (default 60000 ms); verified file progress resets the deadline, so large files can run longer overall. `--max-bytes` sets the byte limit; JSON config sets the other sanitizer limits. `--paranoid` adds entropy heuristics and increases false positives. `--native-only` explicitly disables Secretlint.
 
+`scan` accepts several paths and `--format sarif` emits SARIF 2.1.0 for code-scanning viewers. The repository's `.pre-commit-hooks.yaml` provides `sensitive-data-sanitizer` (credentials) and `sensitive-data-sanitizer-pii` hooks for [pre-commit](https://pre-commit.com). UTF-16 files with a byte order mark are read and written back in their own encoding.
+
 ## Configuration
 
 For AI sessions, `--jsonl` preserves JSON records and declares `id`, `type`,
@@ -195,6 +211,8 @@ Keep policies containing real private values outside Git. Built-in exact public
 }
 ```
 
+Reviewed false positives can be kept with `allowList` (whole-value strings or anchored regexes) or, opt-in, with `inlineAllow: true`, which honours `gitleaks:allow`, `pragma: allowlist secret` and similar comments; credentials stay redacted unless `allowCredentialDisable: true` is also set. `minConfidenceByType` sets per-type thresholds, the `mask` transformation accepts `char`, `ignore`, `ignoreClasses`, `count` and `fromEnd`, and `{mode: 'encrypt', key}` produces deterministic AES-256-GCM tokens that `decrypt(text, key)` restores. See [the API](docs/API.md#allow-lists-suppression-comments-and-template-values) and [the competitor comparison](docs/COMPARISON.md).
+
 Custom policies preserve exact reviewed `PERSON`, `ORGANIZATION` and `EMAIL` detections. Built-in public resolvers/documentation IPs are also exempt. Credential detections always win, including a public email used as a password or authorization value. Evidence is an explicit caller decision and does not make every occurrence harmless.
 
 ## Extend detection
@@ -218,6 +236,8 @@ const sanitizer = createSanitizer({
 
 Install external binaries and language models separately. TruffleHog runs with verification and updates disabled. The Presidio example checks for an installed model and never downloads it at analysis time. Each selected detector is required, including during residual verification. See [the API and adapter contract](docs/API.md), [combined scanners](examples/combined-scanners.mjs), and [the local NER bridge](examples/presidio-bridge.py).
 
+`createPatternDetector([{type, patterns, denyList, context, validate}])` adds Presidio-style pattern recognizers or Google-style custom infoTypes with hotword boosts, and `createIdentifierDetector` adds national identifiers with checksum validators; see [declarative pattern detectors](docs/API.md#declarative-pattern-detectors).
+
 The framework also accepts Gitleaks, TruffleHog, detect-secrets, Presidio, or explicitly typed offset reports. Cloud reports can be converted locally; the package itself makes no cloud requests.
 
 For CLI name recognition, `--presidio PATH` selects a local Python JSON bridge. `--python COMMAND` selects its interpreter, and `--model` / `--language` select an already installed model. These options require the bridge and participate in final-output verification.
@@ -241,7 +261,7 @@ See [contributing](docs/CONTRIBUTING.md). `add`, `multiply` and `delay` live in
 
 The universal app's preview images regenerate on pushes to the default branch. Run `npm run example:web:preview-images` to regenerate them locally; see [the template guide](docs/TEMPLATE-PIPELINE.md) for setup.
 
-Automatic `--env-secrets` discovery skips authentication configuration names such as `GIT_AUTH_TRANSPORT`, `AUTH_MODE` and `AUTH_METHOD`. Explicit `knownSecretsFromEnv(env, names)` selections still include their exact values.
+Automatic `--env-secrets` discovery also selects `PGPASSWORD`-style suffixes and cookies, and skips configuration names such as `GIT_AUTH_TRANSPORT`, `AUTH_MODE`, `*_AUTH_GUARD`, `MAX_TOKEN_COUNT`, `SSH_KEY_PATH`, `SECRET_NAME`, `AWS_ACCESS_KEY_ID` and `GIT_CONFIG_KEY_<n>`; `knownSecretsFromEnv(env, undefined, { minLength: 8 })` also skips short values. Explicit `knownSecretsFromEnv(env, names)` selections still include their exact values.
 
 For shared tool logs, `--credentials-only --env-secrets --gh-config` preserves
 public account details while redacting credential patterns and exact local

package/bin/sensitive-data-sanitizer.js

@@ -1,4 +1,5 @@
 #!/usr/bin/env node
+import '../src/personal-features.js';
 import { readFileSync, realpathSync } from 'node:fs';
 import { fileURLToPath, URL } from 'node:url';
 import { resolve } from 'node:path';
@@ -6,9 +7,11 @@ import { createSanitizer } from '../src/engines.js';
 import { failure } from '../src/detection.js';
 import {
   readTextFile,
+  readTextInput,
   atomicWrite,
   listTextCandidates,
-  decodeText,
+  decodeTextInput,
+  encodeTextOutput,
 } from '../src/files.js';
 import { auditGitHistory } from '../src/history.js';
 import { rewriteGitHistory } from '../src/history-rewrite.js';
@@ -17,7 +20,7 @@ import {
   knownSecretsFromEnv,
   knownSecretsFromGitHubConfig,
 } from '../src/known.js';
-import { encodeLogText } from '../src/log-bytes.js';
+import { toSarif } from '../src/sarif.js';
 import { createWikidataVerifier } from '../src/public-verifier.js';
 import { streamCommand } from './stream.js';
 import { sessionStructuralFields } from '../src/structured.js';
@@ -31,7 +34,8 @@ import {
 const help = `Usage: sensitive-data-sanitizer <redact|scan|restore|history> [path|-] [options]
 
 redact    Sanitize one text log or stdin; publish only after verification
-scan      Report metadata for a file, directory, or stdin (never source text)
+scan      Report metadata for files, directories, or stdin (never source text);
+          several paths may follow scan (for example staged files)
 restore   Restore placeholders in a file using --map PATH (private output)
 history   Audit local reachable Git blobs, commit/tag metadata, and reflogs
 history rewrite SOURCE --output DIRECTORY [--apply]
@@ -67,6 +71,7 @@ history rewrite SOURCE --output DIRECTORY [--apply]
 --no-map            Discard placeholder mapping after full anonymization
 --placeholder-format TEMPLATE  Default {Type}-{n}; also [{TYPE}_{n}]
 --map-format FORMAT json or lino (otherwise inferred from .lino extension)
+--format FORMAT     scan report: json (default) or sarif (SARIF 2.1.0)
 --map-credentials   Explicitly keep credentials in the private restore map
 --max-record-bytes N Maximum record bytes (text: total limit; JSONL: 8388608)
 --hive-mask         Opt-in first/last 3 mask for values longer than 12 characters
@@ -129,8 +134,9 @@ function parseArgs(argv) {
     ['--map', 'map'],
     ['--map-format', 'mapFormat'],
     ['--placeholder-format', 'placeholderFormat'],
+    ['--format', 'format'],
   ]);
-  let hasPath = false;
+  const paths = [];
   for (let i = rewrite ? 2 : 1; i < argv.length; i++) {
     const arg = argv[i];
     if (boolean.has(arg)) {
@@ -140,13 +146,18 @@ function parseArgs(argv) {
         throw failure('ERR_ARGUMENT');
       }
       config[valued.get(arg)] = argv[++i];
-    } else if (!hasPath && !arg.startsWith('--')) {
-      config.path = arg;
-      hasPath = true;
+    } else if (!arg.startsWith('--')) {
+      paths.push(arg);
     } else {
       throw failure('ERR_ARGUMENT');
     }
   }
+  // Only scan accepts several paths (for example staged files from a hook).
+  if (paths.length > 1 && (command !== 'scan' || paths.includes('-'))) {
+    throw failure('ERR_ARGUMENT');
+  }
+  config.path = paths[0] ?? '-';
+  config.paths = paths.length ? paths : ['-'];
   validateArgs(config);
   return config;
 }
@@ -194,7 +205,11 @@ function validateFeatureArgs(config) {
 
 function validateStructuredArgs(config) {
   validateStreamArgs(config);
-  if (config.twoPass && !config.stream) {
+  if (
+    (config.twoPass && !config.stream) ||
+    (config.format !== undefined &&
+      (config.command !== 'scan' || !['json', 'sarif'].includes(config.format)))
+  ) {
     throw failure('ERR_ARGUMENT');
   }
   if ((config.json && config.jsonl) || (config.stream && config.json)) {
@@ -249,7 +264,7 @@ async function stdinText(stream, maxBytes, options) {
     }
     chunks.push(buffer);
   }
-  return decodeText(Buffer.concat(chunks), options);
+  return decodeTextInput(Buffer.concat(chunks), options);
 }
 
 export async function runCli(
@@ -266,9 +281,7 @@ export async function runCli(
       return 0;
     }
     if (['--version', '-v'].includes(argv[0])) {
-      stdout.write(
-        `${JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version}\n`
-      );
+      stdout.write(`${packageVersion()}\n`);
       return 0;
     }
     const config = parseArgs(argv);
@@ -422,24 +435,20 @@ function inputText(path, config, stdin) {
   };
   return path === '-'
     ? stdinText(stdin, config.maxBytes, options)
-    : readTextFile(path, config.maxBytes, options);
+    : readTextInput(path, config.maxBytes, options);
 }
 
 async function redactCommand(config, engine, { stdin, stdout }) {
-  const result = await engine.sanitize(
-    await inputText(config.path, config, stdin)
-  );
+  const input = await inputText(config.path, config, stdin);
+  const result = await engine.sanitize(input.text);
   await saveMap(config);
+  const output = encodeTextOutput(result.text, input.encoding);
   if (config.output || config.inPlace) {
-    await atomicWrite(
-      config.output ?? config.path,
-      encodeLogText(result.text),
-      {
-        replace: Boolean(config.inPlace),
-      }
-    );
+    await atomicWrite(config.output ?? config.path, output, {
+      replace: Boolean(config.inPlace),
+    });
   } else {
-    stdout.write(encodeLogText(result.text));
+    stdout.write(output);
   }
   return 0;
 }
@@ -464,8 +473,10 @@ async function historyCommand(config, engine, { stdout }) {
 }
 
 async function scanCommand(config, engine, { stdin, stdout }) {
-  const files =
-    config.path === '-' ? ['-'] : await listTextCandidates(config.path);
+  const files = [];
+  for (const path of config.paths) {
+    files.push(...(path === '-' ? ['-'] : await listTextCandidates(path)));
+  }
   const results = [],
     skipped = [];
   for (const file of files) {
@@ -473,7 +484,9 @@ async function scanCommand(config, engine, { stdin, stdout }) {
     try {
       results.push({
         path,
-        findings: await engine.inspect(await inputText(file, config, stdin)),
+        findings: await engine.inspect(
+          (await inputText(file, config, stdin)).text
+        ),
       });
     } catch (error) {
       if (['ERR_BINARY', 'ERR_ENCODING', 'ERR_LIMIT'].includes(error.code)) {
@@ -484,7 +497,10 @@ async function scanCommand(config, engine, { stdin, stdout }) {
     }
   }
   const complete = skipped.length === 0;
-  stdout.write(`${JSON.stringify({ complete, results, skipped }, null, 2)}\n`);
+  const report = { complete, results, skipped };
+  stdout.write(
+    `${JSON.stringify(config.format === 'sarif' ? toSarif(report, { version: packageVersion() }) : report, null, 2)}\n`
+  );
   return !complete ? 2 : results.some((r) => r.findings.length) ? 1 : 0;
 }
 
@@ -495,6 +511,12 @@ if (
   process.exitCode = await runCli(process.argv.slice(2));
 }
 
+function packageVersion() {
+  return JSON.parse(
+    readFileSync(new URL('../package.json', import.meta.url), 'utf8')
+  ).version;
+}
+
 function defaultByteLimit(config) {
   return config.stream ? 1073741824 : 33554432;
 }