@link-foundation/sensitive-data-sanitizer 0.21.3 → 0.21.5

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.21.5
+
+### Patch Changes
+
+- e61a218: Preserve Authorization and Proxy-Authorization scheme words when overlapping credential heuristics match exactly the scheme before a separate payload. Continue masking payloads, bare credentials, and exact known secrets.
+
+## 0.21.4
+
+### Patch Changes
+
+- 7454024: Preserve canonical short base64/base64url Hive masks and escaped JSON log structure through verified sanitization. Reinspect bounded escaped JSON views with the configured exemption policy and detectors, while retaining exact known-secret detection and fail-closed handling at inspection limits.
+
 ## 0.21.3
 
 ### Patch Changes

package/docs/API.md

@@ -317,7 +317,7 @@ Generated replacements carry `faked: true`. A private bounded registry recognize
 
 Faking is pseudonymisation: a private key and retained ages, relations and structure can permit linking. Keep the key out of published data. See [the realistic fake example](../examples/realistic-fakes.mjs) and [the complete requirement/research record](case-studies/issue-24/REQUIREMENTS.md).
 
-`preserveEncoding: true` sanitizes decoded payloads with the same required engine union, re-encodes them and verifies decoding reproduces the sanitized content. Supported formats include base64/base64url, hex, percent, numeric HTML and UTF-8 byte escapes. Escaped JSON string content is sanitized and re-encoded even under default redaction so nested session JSON stays parseable. Wrapped base64 and hex keep their line width and line separator (`\n` or `\r\n`), and a wrapped run is reported as one finding. When the sanitized payload cannot be re-encoded in a form the decoder recognizes (for example hex shorter than 32 digits), the encoded text is masked as it stands rather than blocking the input. Known encoding candidates and nesting are bounded.
+`preserveEncoding: true` sanitizes decoded payloads with the same required engine union, re-encodes them and verifies decoding reproduces the sanitized content. Supported formats include base64/base64url, hex, percent, numeric HTML and UTF-8 byte escapes. Escaped JSON string content is sanitized and re-encoded even under default redaction so nested session JSON stays parseable. Wrapped base64 and hex keep their line width and line separator (`\n` or `\r\n`), and a wrapped run is reported as one finding. When the sanitized payload cannot be re-encoded in a form the decoder recognizes (for example hex shorter than 32 digits), the encoded text is masked as it stands rather than blocking the input. Known encoding candidates and nesting are bounded. Canonical base64/base64url encodings of complete Hive masks (including short forms such as `Z2hv4oCmOTk5`, encoding `gho…999`) remain intact on repeated passes. Arbitrary decoded text, malformed encodings and literal text appended to a mask receive no mask exemption; exact known secrets still take priority.
 
 ## Identity policy and structured sessions
 
@@ -579,7 +579,7 @@ permission values and literal credential references. `publicKnowledge: false`
 restores IP detection; explicit `knownSecrets` and `knownPersonal` retain
 priority. No shell command or workflow expression is executed.
 
-Authorization headers (including Proxy-Authorization) redact the scheme and one credential token. Whitespace, C0 controls (including NUL), DEL, quotes, backslashes and commas end the value, preserving subsequent log text in shell and escaped JSON prompts. Matching brackets opened within a token are included in its redaction; unmatched closing brackets remain part of the surrounding log. Already-redacted tokens and known credential references remain exempt after an optional scheme, including `<token>` placeholders. The debug callback receives value-free `gitleaks-rule` events with rule ID, input length and elapsed milliseconds; tracing is off by default.
+Authorization headers (including Proxy-Authorization) preserve the scheme and redact the credential payload. Overlapping heuristic findings that cover exactly a scheme followed by a separate nonempty payload are ignored, including Gitleaks generic API-key findings on `SharedAccessSignature`. Bare credentials without a scheme, payload findings and exact `knownSecrets` matches remain subject to redaction. Whitespace, C0 controls (including NUL), DEL, quotes, backslashes and commas end the value, preserving subsequent log text in shell and escaped JSON prompts. Matching brackets opened within a token are included in its redaction; unmatched closing brackets remain part of the surrounding log. Already-redacted tokens and known credential references remain exempt after an optional scheme, including `<token>` placeholders. The debug callback receives value-free `gitleaks-rule` events with rule ID, input length and elapsed milliseconds; tracing is off by default.
 
 ## Caller-declared sensitive fields
 
@@ -951,6 +951,13 @@ credential findings without `allowCredentialDisable`. Declared
 `knownSecrets`/`knownPersonal` values, confirmed personal values and structured
 `JSON_VALUE` denials are never offered to it.
 
+For escaped JSON findings up to 8192 characters, the engine independently
+inspects the decoded view with the same configured policy and detectors before
+keeping a clean view. Hooks in that inspection receive decoded text and its
+offsets. Additional inspection stops after three levels; malformed, oversized,
+and deeper views retain normal fail-closed handling. The optional value-free
+`exempt-decoded-view` debug event reports a clean view.
+
 `createSanitizer` awaits a hook that returns a `Promise`.
 `createSyncSanitizer` and `sanitize()` need a synchronous hook. A `Promise`
 there is `ERR_CONFIG`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-foundation/sensitive-data-sanitizer",
-  "version": "0.21.3",
+  "version": "0.21.5",
   "description": "Offline sensitive-text sanitization with credential, multilingual personal-data and scanner union detection",
   "type": "module",
   "main": "src/index.js",

package/src/context.js

@@ -198,8 +198,8 @@ export function authorizationRanges(text, start) {
   if (reference) {
     return [{ start, end: start + reference[0].length }];
   }
-  const scheme = /^[a-z][a-z0-9_-]*[ \t]+/i.exec(text.slice(start));
-  const tokenStart = start + (scheme?.[0].length ?? 0);
+  const scheme = authorizationScheme(text, start);
+  const tokenStart = scheme?.payloadStart ?? start;
   const params = scheme ? authParams(text, tokenStart) : undefined;
   if (params) {
     return params;
@@ -208,6 +208,25 @@ export function authorizationRanges(text, start) {
   const token = text.slice(tokenStart, end);
   return usableAuthorization(token) ? [{ start: tokenStart, end }] : [];
 }
+
+// A scheme is structural only when a separate, nonempty payload follows it.
+// Share this boundary with span refinement so overlapping vendor detections
+// cannot mask the scheme that the Authorization parser deliberately preserves.
+export function authorizationScheme(text, start) {
+  const opening = /^(\\*)(["'])/.exec(text.slice(start));
+  start += opening?.[0].length ?? 0;
+  const scheme = /^([a-z][a-z0-9_-]*)[ \t]+/i.exec(text.slice(start));
+  if (!scheme) {
+    return undefined;
+  }
+  const payloadStart = start + scheme[0].length;
+  const char = text[payloadStart];
+  return char &&
+    !authorizationDelimiter(char, text.charCodeAt(payloadStart)) &&
+    !/[}\]>]/.test(char)
+    ? { start, end: start + scheme[1].length, payloadStart }
+    : undefined;
+}
 function authorizationEnd(text, tokenStart) {
   const brackets = [];
   let end = tokenStart;

package/src/credential-spans.js

@@ -1,4 +1,5 @@
 import { maskedValue } from './credentials.js';
+import { authorizationScheme } from './context.js';
 
 // Credential findings cover the secret, not the readable context around it:
 // PEM markers, URL hosts and user names, cookie names and the rest of a
@@ -159,7 +160,22 @@ function pemMarkers(text, findings) {
 
 export function refineCredentialSpans(text, findings) {
   const markers = pemMarkers(text, findings);
+  const schemes = new Map();
+  for (const header of text.matchAll(
+    /(?<![\w-])(?:Proxy-)?Authorization(?:\\*["'])?[ \t]*:[ \t]*/gi
+  )) {
+    const scheme = authorizationScheme(text, header.index + header[0].length);
+    if (scheme) {
+      schemes.set(scheme.start, scheme.end);
+    }
+  }
   const narrow = findings
+    .filter(
+      (finding) =>
+        finding.category !== 'credential' ||
+        finding.rule === 'known-secret' ||
+        schemes.get(finding.start) !== finding.end
+    )
     .filter(
       (finding) =>
         finding.category === 'credential' ||

package/src/credentials.js

@@ -1,4 +1,26 @@
 import { collectMatches } from './detection.js';
+import { Buffer } from 'node:buffer';
+
+// Re-encoded Hive output can be shorter than the general decoder threshold.
+// Recognize only a whole canonical base64/base64url spelling of a complete
+// mask; arbitrary decodable text and masks glued to literal text stay sensitive.
+function encodedMask(value) {
+  if (
+    value.length > 64 ||
+    !/^(?:[A-Za-z0-9+/]+|[A-Za-z0-9_-]+)={0,2}$/.test(value)
+  ) {
+    return false;
+  }
+  const bytes = Buffer.from(value, 'base64url');
+  const canonical = bytes.toString('base64');
+  const standard = value.replaceAll('-', '+').replaceAll('_', '/');
+  const decoded = bytes.toString('utf8');
+  return (
+    (standard === canonical || standard === canonical.replace(/=+$/, '')) &&
+    Buffer.from(decoded).equals(bytes) &&
+    /^(?:\[REDACTED\]|[^\s…]{0,3}…[^\s…]{0,3})$/u.test(decoded)
+  );
+}
 
 export function credentialPlaceholder(value) {
   return (
@@ -59,7 +81,10 @@ export function templatedCredential(text, finding) {
   const sentenceEnd = /^[.,;:!?]+(?:\s|$)/.test(
     text.slice(finding.start + value.length)
   );
-  return (!/[\w$%+/=.-]/.test(after) || sentenceEnd) && templatedValue(value);
+  return (
+    (!/[\w$%+/=.-]/.test(after) || sentenceEnd) &&
+    (templatedValue(value) || encodedMask(value))
+  );
 }
 // Only variable references and the documented gh lookup are exempt. Arbitrary
 // shell commands and Actions expressions can contain literal credentials.

package/src/engines.js

@@ -18,7 +18,7 @@ import { projectText } from './projection.js';
 import { decodedRuns } from './encoded.js';
 import { hasCustomTransforms } from './transforms.js';
 import { builtInEnabled } from './policy.js';
-import { exemptFindingsAsync } from './exempt.js';
+import { exemptFindingsAsync, escapedExemptionView } from './exempt.js';
 import {
   initializePlaceholders,
   placeholderKnown,
@@ -235,6 +235,59 @@ function configuredEngines(options) {
   return engines;
 }
 
+async function screenDecodedFindings(text, findings, options, depth, screened) {
+  const retained = [];
+  for (const finding of findings) {
+    const view = escapedExemptionView(text, finding, depth);
+    const clean =
+      view !== undefined &&
+      !(
+        await screened(
+          view,
+          [],
+          true,
+          { ...options, findings: [], structured: undefined },
+          undefined,
+          depth + 1
+        )
+      ).length;
+    if (clean) {
+      options.debug?.({ event: 'exempt-decoded-view', rule: finding.rule });
+    } else {
+      retained.push(finding);
+    }
+  }
+  return retained;
+}
+
+function createScreened(analyze, options) {
+  // The caller's `exempt` hook screens the findings of the text it sees,
+  // including the verification passes over the published output.
+  return async function screened(
+    text,
+    initial,
+    plain,
+    scopedOptions = options,
+    nativeFindings,
+    exemptionDepth = 0
+  ) {
+    const findings = await exemptFindingsAsync(
+      text,
+      await analyze(text, initial, plain, scopedOptions, nativeFindings),
+      scopedOptions.exempt
+    );
+    return scopedOptions.exempt
+      ? screenDecodedFindings(
+          text,
+          findings,
+          scopedOptions,
+          exemptionDepth,
+          screened
+        )
+      : findings;
+  };
+}
+
 export function createSanitizer(options = {}) {
   options = applyProfile(options);
   validateOptions(options);
@@ -281,21 +334,7 @@ export function createSanitizer(options = {}) {
       : detected;
   }
 
-  // The caller's `exempt` hook screens the findings of the text it sees,
-  // including the verification passes over the published output.
-  async function screened(
-    text,
-    initial,
-    plain,
-    scopedOptions = options,
-    nativeFindings
-  ) {
-    return exemptFindingsAsync(
-      text,
-      await analyze(text, initial, plain, scopedOptions, nativeFindings),
-      scopedOptions.exempt
-    );
-  }
+  const screened = createScreened(analyze, options);
 
   const sanitizer = {
     inspect: guarded(screened),

package/src/exempt.js

@@ -16,6 +16,30 @@ const declared = new Set([
 const offered = (finding) =>
   !declared.has(finding.rule) && finding.type !== 'JSON_VALUE';
 
+// Only bounded escaped JSON views qualify for an independent policy check.
+// Keep the decoder's 8192-character bound and cap recursive checks at three;
+// malformed, oversized and other encoded findings retain normal handling.
+export function escapedExemptionView(text, finding, depth) {
+  if (
+    depth >= 3 ||
+    finding.type !== 'ENCODED_SENSITIVE' ||
+    !['encoded', 'encoded-engine'].includes(finding.rule) ||
+    finding.end - finding.start > 8192
+  ) {
+    return undefined;
+  }
+  const raw = text.slice(finding.start, finding.end);
+  if (!raw.includes('\\')) {
+    return undefined;
+  }
+  try {
+    const decoded = JSON.parse(`"${raw}"`);
+    return decoded !== raw ? decoded : undefined;
+  } catch {
+    return undefined;
+  }
+}
+
 function view(text, finding) {
   return Object.freeze({
     ...finding,

package/src/index.d.ts

@@ -125,6 +125,8 @@ export interface SanitizerOptions {
    * Verification of the output consults it again per span, so the same value
    * elsewhere is still caught. Declared knownSecrets/knownPersonal/
    * confirmedPersonal values and JSON_VALUE denials are never offered.
+   * Bounded escaped JSON findings are independently inspected with the same
+   * policy (up to three levels); hooks there receive decoded text/offsets.
    * createSyncSanitizer and sanitize() require a synchronous hook.
    */
   exempt?: (
@@ -154,6 +156,7 @@ export type DebugEvent =
   | { event: 'document-experience' | 'proper-name-prose'; count: number }
   | { event: 'residual-redaction'; round: number }
   | { event: 'template-value'; rule: string }
+  | { event: 'exempt-decoded-view'; rule: string }
   | { event: 'two-pass-confirmation'; type: string }
   | { event: 'propagation-skipped'; reason: 'length'; type: string }
   | { event: 'gitleaks-rule'; rule: string; length: number; ms: number };

package/src/sanitizer.js

@@ -28,7 +28,11 @@ import { isIdentityType } from './identity-types.js';
 import { rejectRestoreMap } from './restore-map.js';
 import { allowListed, validateAllowList } from './allow-list.js';
 import { templatedCredential } from './credentials.js';
-import { exemptFindings, validateExempt } from './exempt.js';
+import {
+  exemptFindings,
+  escapedExemptionView,
+  validateExempt,
+} from './exempt.js';
 import { builtInRules } from './rule-catalog.js';
 import {
   initializePlaceholders,
@@ -455,16 +459,31 @@ function locations(text, findings) {
   });
 }
 
-export function inspect(text, options = {}) {
+export function inspect(text, options = {}, exemptionDepth = 0) {
   try {
     const exempt = options?.exempt;
     if (!exempt) {
       return inspectText(text, options);
     }
-    // Nested views (structured values, decoded runs) are screened once, at
-    // the offsets of the text the caller sees.
+    // Screen source offsets first, then independently inspect escaped JSON
+    // with the same policy. A clean decoded view must not become a stale
+    // whole-view finding that rendering would replace with [REDACTED].
     const findings = inspectText(text, { ...options, exempt: undefined });
-    return exemptFindings(text, findings, exempt);
+    return exemptFindings(text, findings, exempt).filter((finding) => {
+      const view = escapedExemptionView(text, finding, exemptionDepth);
+      if (view === undefined) {
+        return true;
+      }
+      const clean = !inspect(
+        view,
+        { ...options, findings: [], structured: undefined },
+        exemptionDepth + 1
+      ).length;
+      if (clean) {
+        options.debug?.({ event: 'exempt-decoded-view', rule: finding.rule });
+      }
+      return !clean;
+    });
   } catch (error) {
     throw internalFailure(error);
   }
@@ -542,6 +561,14 @@ function inspectText(text, options = {}) {
   if (projection.changed) {
     nativeDetect(text, options, emit);
     detectSync(text, options, emit);
+  } else if (
+    (options.knownSecrets ?? []).some(
+      (value, index) => value !== nativeOptions.knownSecrets[index]
+    )
+  ) {
+    // Encoded exact bytes must remain detectable even when the declared
+    // secret normalizes (for example the ellipsis in a known Hive mask).
+    nativeDetect(text, { ...options, knownOnly: true }, emit);
   }
   nativeDetect(projection.text, nativeOptions, mapped);
   detectSync(projection.text, nativeOptions, mapped);