@link-foundation/sensitive-data-sanitizer 0.21.2 → 0.21.4

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.21.4
+
+### Patch Changes
+
+- 7454024: Preserve canonical short base64/base64url Hive masks and escaped JSON log structure through verified sanitization. Reinspect bounded escaped JSON views with the configured exemption policy and detectors, while retaining exact known-secret detection and fail-closed handling at inspection limits.
+
+## 0.21.3
+
+### Patch Changes
+
+- ee1ec36: Preserve harmless wrapped base64 binary within a bounded complete inspection, while retaining embedded credential detection and fail-closed handling of uninspected oversized payloads. Add regression coverage for complete JSON re-encoding after surrounding prose through sync, async and streaming sanitization.
+
 ## 0.21.2
 
 ### Patch Changes

package/docs/API.md

@@ -317,7 +317,7 @@ Generated replacements carry `faked: true`. A private bounded registry recognize
 
 Faking is pseudonymisation: a private key and retained ages, relations and structure can permit linking. Keep the key out of published data. See [the realistic fake example](../examples/realistic-fakes.mjs) and [the complete requirement/research record](case-studies/issue-24/REQUIREMENTS.md).
 
-`preserveEncoding: true` sanitizes decoded payloads with the same required engine union, re-encodes them and verifies decoding reproduces the sanitized content. Supported formats include base64/base64url, hex, percent, numeric HTML and UTF-8 byte escapes. Escaped JSON string content is sanitized and re-encoded even under default redaction so nested session JSON stays parseable. Wrapped base64 and hex keep their line width and line separator (`\n` or `\r\n`), and a wrapped run is reported as one finding. When the sanitized payload cannot be re-encoded in a form the decoder recognizes (for example hex shorter than 32 digits), the encoded text is masked as it stands rather than blocking the input. Known encoding candidates and nesting are bounded.
+`preserveEncoding: true` sanitizes decoded payloads with the same required engine union, re-encodes them and verifies decoding reproduces the sanitized content. Supported formats include base64/base64url, hex, percent, numeric HTML and UTF-8 byte escapes. Escaped JSON string content is sanitized and re-encoded even under default redaction so nested session JSON stays parseable. Wrapped base64 and hex keep their line width and line separator (`\n` or `\r\n`), and a wrapped run is reported as one finding. When the sanitized payload cannot be re-encoded in a form the decoder recognizes (for example hex shorter than 32 digits), the encoded text is masked as it stands rather than blocking the input. Known encoding candidates and nesting are bounded. Canonical base64/base64url encodings of complete Hive masks (including short forms such as `Z2hv4oCmOTk5`, encoding `gho…999`) remain intact on repeated passes. Arbitrary decoded text, malformed encodings and literal text appended to a mask receive no mask exemption; exact known secrets still take priority.
 
 ## Identity policy and structured sessions
 
@@ -365,7 +365,12 @@ status identifiers. Explicit `knownPersonal` values override these heuristics.
 
 Oversized encoded runs receive a conservative complete-run `ENCODED_LIMIT`
 redaction rather than aborting the record. Decoding probes are capped at 4,096
-characters; full decoding stays capped at 8,192 characters. A multi-line run is
+characters; ordinary text decoding stays capped at 8,192 characters. Uniform
+wrapped binary base64 up to 256 KiB of encoded input is decoded completely
+and its embedded text is inspected. Harmless binary bytes are preserved; a
+sensitive binary payload is masked as a whole because a text re-encoding
+cannot preserve its bytes. Larger uninspected wrapped binary runs still
+receive `ENCODED_LIMIT`. A multi-line run is
 treated as one wrapped payload only when its probe decodes as text or its lines
 share one base64 width (a multiple of four, standard alphabet, not plain hex);
 lists of identifiers or commit hashes are scanned line by line. Up to 4,096
@@ -946,6 +951,13 @@ credential findings without `allowCredentialDisable`. Declared
 `knownSecrets`/`knownPersonal` values, confirmed personal values and structured
 `JSON_VALUE` denials are never offered to it.
 
+For escaped JSON findings up to 8192 characters, the engine independently
+inspects the decoded view with the same configured policy and detectors before
+keeping a clean view. Hooks in that inspection receive decoded text and its
+offsets. Additional inspection stops after three levels; malformed, oversized,
+and deeper views retain normal fail-closed handling. The optional value-free
+`exempt-decoded-view` debug event reports a clean view.
+
 `createSanitizer` awaits a hook that returns a `Promise`.
 `createSyncSanitizer` and `sanitize()` need a synchronous hook. A `Promise`
 there is `ERR_CONFIG`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-foundation/sensitive-data-sanitizer",
-  "version": "0.21.2",
+  "version": "0.21.4",
   "description": "Offline sensitive-text sanitization with credential, multilingual personal-data and scanner union detection",
   "type": "module",
   "main": "src/index.js",

package/src/credentials.js

@@ -1,4 +1,26 @@
 import { collectMatches } from './detection.js';
+import { Buffer } from 'node:buffer';
+
+// Re-encoded Hive output can be shorter than the general decoder threshold.
+// Recognize only a whole canonical base64/base64url spelling of a complete
+// mask; arbitrary decodable text and masks glued to literal text stay sensitive.
+function encodedMask(value) {
+  if (
+    value.length > 64 ||
+    !/^(?:[A-Za-z0-9+/]+|[A-Za-z0-9_-]+)={0,2}$/.test(value)
+  ) {
+    return false;
+  }
+  const bytes = Buffer.from(value, 'base64url');
+  const canonical = bytes.toString('base64');
+  const standard = value.replaceAll('-', '+').replaceAll('_', '/');
+  const decoded = bytes.toString('utf8');
+  return (
+    (standard === canonical || standard === canonical.replace(/=+$/, '')) &&
+    Buffer.from(decoded).equals(bytes) &&
+    /^(?:\[REDACTED\]|[^\s…]{0,3}…[^\s…]{0,3})$/u.test(decoded)
+  );
+}
 
 export function credentialPlaceholder(value) {
   return (
@@ -59,7 +81,10 @@ export function templatedCredential(text, finding) {
   const sentenceEnd = /^[.,;:!?]+(?:\s|$)/.test(
     text.slice(finding.start + value.length)
   );
-  return (!/[\w$%+/=.-]/.test(after) || sentenceEnd) && templatedValue(value);
+  return (
+    (!/[\w$%+/=.-]/.test(after) || sentenceEnd) &&
+    (templatedValue(value) || encodedMask(value))
+  );
 }
 // Only variable references and the documented gh lookup are exempt. Arbitrary
 // shell commands and Actions expressions can contain literal credentials.

package/src/encoded.js

@@ -7,6 +7,9 @@ import { jsonStringRuns } from './json-string-runs.js';
 // remaining candidates as `encoded-limit` runs, without blocking the text.
 const WINDOW = 65536;
 const WINDOW_CANDIDATES = 4096;
+// Inspect a complete bounded binary payload, including any embedded text.
+// Ordinary text detection retains its 8 KiB cap.
+const WRAPPED_BINARY_LIMIT = 256 * 1024;
 
 const WRAPPED =
   /(?<![A-Za-z0-9_+/-])[A-Za-z0-9_+/-]{16,128}(?:\r?\n[A-Za-z0-9_+/-]{1,128})+={0,2}(?![A-Za-z0-9_+/-])/g;
@@ -56,7 +59,8 @@ function* matchRuns(match, wrapped, scan) {
   const start = match.index;
   const end = start + raw.length;
   const limited = { start, end, limited: true, depth: scan.depth };
-  if (raw.length > 8192) {
+  const candidates = runCandidates(raw, wrapped);
+  if (!candidates) {
     if (isLimitedTextRun(raw, wrapped)) {
       scan.covered.push(limited);
       yield limited;
@@ -64,7 +68,7 @@ function* matchRuns(match, wrapped, scan) {
     return;
   }
   const window = Math.floor(start / WINDOW);
-  for (const candidate of decodeCandidates(raw)) {
+  for (const candidate of candidates) {
     const { text: decoded, encoding } = candidate;
     const key = `${start}:${raw.length}:${decoded}`;
     if (scan.seen.has(key) || !decodedText(raw, decoded)) {
@@ -91,11 +95,48 @@ function* matchRuns(match, wrapped, scan) {
     };
     yield run;
     for (const inner of decodedRuns(decoded, scan.depth + 1)) {
-      yield { ...run, text: inner.text, depth: inner.depth };
+      yield inner.limited
+        ? { ...limited, depth: inner.depth }
+        : { ...run, text: inner.text, depth: inner.depth };
     }
   }
 }
 
+function runCandidates(raw, wrapped) {
+  if (raw.length <= 8192) {
+    return decodeCandidates(raw);
+  }
+  const binary = wrapped && boundedWrappedBinary(raw);
+  return binary ? [binary] : undefined;
+}
+
+function boundedWrappedBinary(raw) {
+  if (raw.length > WRAPPED_BINARY_LIMIT || !isWrappedBinary(raw)) {
+    return false;
+  }
+  const bytes = base64Bytes(raw.replace(/\s/g, ''));
+  if (!bytes) {
+    return false;
+  }
+  try {
+    const decoded = new TextDecoder('utf-8', { fatal: true }).decode(bytes);
+    if (decodedText(raw, decoded)) {
+      return undefined;
+    }
+  } catch {
+    /* Binary content still needs an embedded-text detection view. */
+  }
+  return {
+    encoding: 'binary',
+    // Preserve text offsets within the view while separating binary controls
+    // and invalid UTF-8. This view is never used to reconstruct binary bytes.
+    text: bytes
+      .toString('utf8')
+      // eslint-disable-next-line no-control-regex -- Replace binary controls in the text detection view.
+      .replace(/[\x00-\x08\x0e-\x1f\x7f\ufffd]/g, ' '),
+  };
+}
+
 function candidateMatches(text, pattern) {
   return pattern ? text.matchAll(pattern) : jsonStringRuns(text);
 }
@@ -211,11 +252,8 @@ function decodeCandidates(raw, partial) {
   } else {
     const compact = raw.replace(/\s/g, '');
     // Decode canonical base64 text only.
-    const bytes = Buffer.from(compact, 'base64url');
-    if (
-      bytes.toString('base64url') ===
-      compact.replace(/=+$/, '').replaceAll('+', '-').replaceAll('/', '_')
-    ) {
+    const bytes = base64Bytes(compact);
+    if (bytes) {
       try {
         candidates.push({
           encoding:
@@ -248,7 +286,18 @@ function decodeCandidates(raw, partial) {
   return candidates;
 }
 
+function base64Bytes(compact) {
+  const bytes = Buffer.from(compact, 'base64url');
+  return bytes.toString('base64url') ===
+    compact.replace(/=+$/, '').replaceAll('+', '-').replaceAll('/', '_')
+    ? bytes
+    : undefined;
+}
+
 export function encodeRun(run, text) {
+  if (run.encoding === 'binary') {
+    return undefined;
+  }
   const bytes = Buffer.from(text, 'utf8');
   let encoded;
   if (run.encoding === 'json-content') {

package/src/engines.js

@@ -18,7 +18,7 @@ import { projectText } from './projection.js';
 import { decodedRuns } from './encoded.js';
 import { hasCustomTransforms } from './transforms.js';
 import { builtInEnabled } from './policy.js';
-import { exemptFindingsAsync } from './exempt.js';
+import { exemptFindingsAsync, escapedExemptionView } from './exempt.js';
 import {
   initializePlaceholders,
   placeholderKnown,
@@ -235,6 +235,59 @@ function configuredEngines(options) {
   return engines;
 }
 
+async function screenDecodedFindings(text, findings, options, depth, screened) {
+  const retained = [];
+  for (const finding of findings) {
+    const view = escapedExemptionView(text, finding, depth);
+    const clean =
+      view !== undefined &&
+      !(
+        await screened(
+          view,
+          [],
+          true,
+          { ...options, findings: [], structured: undefined },
+          undefined,
+          depth + 1
+        )
+      ).length;
+    if (clean) {
+      options.debug?.({ event: 'exempt-decoded-view', rule: finding.rule });
+    } else {
+      retained.push(finding);
+    }
+  }
+  return retained;
+}
+
+function createScreened(analyze, options) {
+  // The caller's `exempt` hook screens the findings of the text it sees,
+  // including the verification passes over the published output.
+  return async function screened(
+    text,
+    initial,
+    plain,
+    scopedOptions = options,
+    nativeFindings,
+    exemptionDepth = 0
+  ) {
+    const findings = await exemptFindingsAsync(
+      text,
+      await analyze(text, initial, plain, scopedOptions, nativeFindings),
+      scopedOptions.exempt
+    );
+    return scopedOptions.exempt
+      ? screenDecodedFindings(
+          text,
+          findings,
+          scopedOptions,
+          exemptionDepth,
+          screened
+        )
+      : findings;
+  };
+}
+
 export function createSanitizer(options = {}) {
   options = applyProfile(options);
   validateOptions(options);
@@ -281,21 +334,7 @@ export function createSanitizer(options = {}) {
       : detected;
   }
 
-  // The caller's `exempt` hook screens the findings of the text it sees,
-  // including the verification passes over the published output.
-  async function screened(
-    text,
-    initial,
-    plain,
-    scopedOptions = options,
-    nativeFindings
-  ) {
-    return exemptFindingsAsync(
-      text,
-      await analyze(text, initial, plain, scopedOptions, nativeFindings),
-      scopedOptions.exempt
-    );
-  }
+  const screened = createScreened(analyze, options);
 
   const sanitizer = {
     inspect: guarded(screened),

package/src/exempt.js

@@ -16,6 +16,30 @@ const declared = new Set([
 const offered = (finding) =>
   !declared.has(finding.rule) && finding.type !== 'JSON_VALUE';
 
+// Only bounded escaped JSON views qualify for an independent policy check.
+// Keep the decoder's 8192-character bound and cap recursive checks at three;
+// malformed, oversized and other encoded findings retain normal handling.
+export function escapedExemptionView(text, finding, depth) {
+  if (
+    depth >= 3 ||
+    finding.type !== 'ENCODED_SENSITIVE' ||
+    !['encoded', 'encoded-engine'].includes(finding.rule) ||
+    finding.end - finding.start > 8192
+  ) {
+    return undefined;
+  }
+  const raw = text.slice(finding.start, finding.end);
+  if (!raw.includes('\\')) {
+    return undefined;
+  }
+  try {
+    const decoded = JSON.parse(`"${raw}"`);
+    return decoded !== raw ? decoded : undefined;
+  } catch {
+    return undefined;
+  }
+}
+
 function view(text, finding) {
   return Object.freeze({
     ...finding,

package/src/index.d.ts

@@ -125,6 +125,8 @@ export interface SanitizerOptions {
    * Verification of the output consults it again per span, so the same value
    * elsewhere is still caught. Declared knownSecrets/knownPersonal/
    * confirmedPersonal values and JSON_VALUE denials are never offered.
+   * Bounded escaped JSON findings are independently inspected with the same
+   * policy (up to three levels); hooks there receive decoded text/offsets.
    * createSyncSanitizer and sanitize() require a synchronous hook.
    */
   exempt?: (
@@ -154,6 +156,7 @@ export type DebugEvent =
   | { event: 'document-experience' | 'proper-name-prose'; count: number }
   | { event: 'residual-redaction'; round: number }
   | { event: 'template-value'; rule: string }
+  | { event: 'exempt-decoded-view'; rule: string }
   | { event: 'two-pass-confirmation'; type: string }
   | { event: 'propagation-skipped'; reason: 'length'; type: string }
   | { event: 'gitleaks-rule'; rule: string; length: number; ms: number };

package/src/sanitizer.js

@@ -28,7 +28,11 @@ import { isIdentityType } from './identity-types.js';
 import { rejectRestoreMap } from './restore-map.js';
 import { allowListed, validateAllowList } from './allow-list.js';
 import { templatedCredential } from './credentials.js';
-import { exemptFindings, validateExempt } from './exempt.js';
+import {
+  exemptFindings,
+  escapedExemptionView,
+  validateExempt,
+} from './exempt.js';
 import { builtInRules } from './rule-catalog.js';
 import {
   initializePlaceholders,
@@ -455,16 +459,31 @@ function locations(text, findings) {
   });
 }
 
-export function inspect(text, options = {}) {
+export function inspect(text, options = {}, exemptionDepth = 0) {
   try {
     const exempt = options?.exempt;
     if (!exempt) {
       return inspectText(text, options);
     }
-    // Nested views (structured values, decoded runs) are screened once, at
-    // the offsets of the text the caller sees.
+    // Screen source offsets first, then independently inspect escaped JSON
+    // with the same policy. A clean decoded view must not become a stale
+    // whole-view finding that rendering would replace with [REDACTED].
     const findings = inspectText(text, { ...options, exempt: undefined });
-    return exemptFindings(text, findings, exempt);
+    return exemptFindings(text, findings, exempt).filter((finding) => {
+      const view = escapedExemptionView(text, finding, exemptionDepth);
+      if (view === undefined) {
+        return true;
+      }
+      const clean = !inspect(
+        view,
+        { ...options, findings: [], structured: undefined },
+        exemptionDepth + 1
+      ).length;
+      if (clean) {
+        options.debug?.({ event: 'exempt-decoded-view', rule: finding.rule });
+      }
+      return !clean;
+    });
   } catch (error) {
     throw internalFailure(error);
   }
@@ -542,6 +561,14 @@ function inspectText(text, options = {}) {
   if (projection.changed) {
     nativeDetect(text, options, emit);
     detectSync(text, options, emit);
+  } else if (
+    (options.knownSecrets ?? []).some(
+      (value, index) => value !== nativeOptions.knownSecrets[index]
+    )
+  ) {
+    // Encoded exact bytes must remain detectable even when the declared
+    // secret normalizes (for example the ellipsis in a known Hive mask).
+    nativeDetect(text, { ...options, knownOnly: true }, emit);
   }
   nativeDetect(projection.text, nativeOptions, mapped);
   detectSync(projection.text, nativeOptions, mapped);