@link-foundation/sensitive-data-sanitizer 0.19.1 → 0.19.3

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.19.3
+
+### Patch Changes
+
+- 6bdaf11: Recognize additional inflected special-category attributes and unfamiliar values after explicit category cues in strict mode. Require personal context for ambiguous English attributes so operational logs and JSON statuses stay intact.
+
+## 0.19.2
+
+### Patch Changes
+
+- 4afc88d: Read `Mr`, `Ms` and `Dr` as person cues only when capitalized, so lowercase `ms` (milliseconds) and `dr` before a capitalized log word stay. Treat compound JSON keys ending in a personal field, such as `recruiterDisplayName`, `authorFullName` and `resumePdfOriginalFileName`, as person fields.
+
 ## 0.19.1
 
 ### Patch Changes

package/docs/API.md

@@ -17,7 +17,7 @@ Import the ESM root entry; complete declarations are in `src/index.d.ts`. Inputs
 | `compileKnown(...lists)`               | Copy, freeze, merge and precompile caller-known personal values                                              |
 | `auditGitHistory(repository, options)` | Read-only audit of local reachable text objects                                                              |
 
-`redactions` counts merged replacement spans. `findings` can contain several detector hits for one replacement. Findings have `{start, end, type, category, rule, line, column}`; no text, quote, match, entropy score, provider details, or raw diagnostic is copied. Debugging is off by default; `debug(event)` receives value-free counts, rule timings, residual rounds and registry event metadata (see `DebugEvent` in the TypeScript declarations).
+`redactions` counts merged replacement spans. `findings` can contain several detector hits for one replacement. Findings have `{start, end, type, category, rule, line, column}`; no text, quote, match, entropy score, provider details, or raw diagnostic is copied. The optional `strict-attributes` debug event reports only detected and suppressed match counts. Debugging is off by default; `debug(event)` receives value-free counts, rule timings, residual rounds and registry event metadata (see `DebugEvent` in the TypeScript declarations).
 
 ## Options
 

package/docs/COVERAGE.md

@@ -370,6 +370,11 @@ this explicit vocabulary: `firstName`, `lastName`, `middleName`, `patronymic`,
 sensitive in full, including ISO birth timestamps. Thus `first_name` and
 `firstName`, or `birth_date` and `birthDate`, follow the same policy. These are
 schema assumptions: `contact` or `company` can hold operational data too.
+Person fields also match as a word-boundary suffix of a compound key, so
+`recruiterDisplayName`, `author_full_name` and `resumePdfOriginalFileName` are
+person fields. The same suffix rule applies to `vocabulary.personalFields`.
+Other types match only the whole key (`maxAge` and `fileName` stay), and a
+bare `name` is not in the list.
 Existing native labels still apply. Unlisted URL-named fields have ordinary
 content detection; arbitrary names do not imply sensitivity. Declare them with
 [`sensitiveFields`](API.md#caller-declared-sensitive-fields) to replace complete
@@ -471,7 +476,25 @@ proper-name mode. Romashka Inc keeps its existing legal-form treatment.
 `strict: true` independently opts into personal attributes (gender, family,
 ethnicity, religion, health, criminal/political status) and bare dates, ages,
 passport/mobile/postal shapes. It is intended for stored personal prose and can
-remove ordinary uses of these words. It preserves ISO timestamps, work durations
+remove ordinary uses of unambiguous category words. Bounded Russian case/stem
+rules cover disability, ethnicity, religion, diagnoses, intimate life and criminal
+status, including `инвалидом`, `армянин`, `еврейка`, `казах`, `христианин`, `иудей`,
+`эпилепсией`, `астма`, `онкологию`, `туберкулез`, `психиатра`, `депрессия`,
+`гепатит`, `гей` and `отбывал наказание`. English coverage includes `epilepsy`,
+`gay` and `Armenian`. Explicit category cues such as `по национальности X`,
+`исповедует X`, `болеет X`, `на учете у X`, `diagnosed with X` and `ethnicity: X`
+protect one following word (including hyphenated words) up to 64 characters,
+including unfamiliar values outside the lexicon. These cues stay on one line.
+Ambiguous English `disabled`, `single`, `man`, `men`, `russian`
+and `asian` require an adjacent personal cue (pronoun/person role, personal
+label, or following person role). Thus `he is disabled`, `candidate is single`
+and `gender: male` redact, while `the module is disabled`, `single run finished`,
+`single-flight lock acquired`, `man page missing`, `locale russian selected`
+and `asian region endpoint` remain intact. Bare operational JSON statuses
+`disabled`, `single` and `Russian` also remain intact; explicit personal labels
+and caller-declared known personal values still redact. A cue in another clause
+or string value does not make an ambiguous word personal.
+It preserves ISO timestamps, work durations
 (including `Стаж: 5 лет`, `работал там 5 лет`, and duration cues from issue 87),
 and parenthesized durations after known technologies such as `Kubernetes (5 лет)`
 and `Power BI (5 лет)`.
@@ -542,6 +565,9 @@ corpus, unbounded input, parallel builds or external analysis service.
 
 Document rules recognize a capitalized Latin name after `candidate`, `recruiter`,
 `applicant`, `Dear`, `Hi`, `Mr`/`Mr.`, `Ms`/`Ms.`, `Dr`/`Dr.`, or `message from`.
+Honorifics count only when capitalized: lowercase `ms` (milliseconds), `mr` and
+`dr` are not cues, and `Ms` right after a number is read as a unit, so
+`took 30 ms Retry scheduled` and `see dr Drive mount` stay unchanged.
 Only the name is replaced; punctuation and the cue remain. These rules work
 without enabling Latin script matching. Finite generic heading nouns such as
 `Stage` and `Status` are preserved: `Candidate Stage changed` and

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-foundation/sensitive-data-sanitizer",
-  "version": "0.19.1",
+  "version": "0.19.3",
   "description": "Offline sensitive-text sanitization with credential, multilingual personal-data and scanner union detection",
   "type": "module",
   "main": "src/index.js",

package/src/document-fields.js

@@ -25,7 +25,24 @@ const types = new Map(
   )
 );
 export function documentFieldType(field) {
-  return types.get(normalizeFieldName(field));
+  const type = types.get(normalizeFieldName(field));
+  if (type) {
+    return type;
+  }
+  // Compound person keys (`recruiterDisplayName`, `author_full_name`) end in
+  // a listed person field. Only PERSON entries extend this way, so generic
+  // suffixes such as `maxAge` or `homeSchool` keep their exact-match meaning,
+  // and a bare `name` is never listed.
+  return fieldSuffixes(field).some((suffix) => types.get(suffix) === 'PERSON')
+    ? 'PERSON'
+    : undefined;
+}
+// Word-boundary suffixes of a key, normalized; the whole key comes first.
+export function fieldSuffixes(field) {
+  const words = field
+    .split(/[\s_.-]+|(?<=[\p{Ll}\p{N}])(?=\p{Lu})|(?<=\p{Lu})(?=\p{Lu}\p{Ll})/u)
+    .filter(Boolean);
+  return words.map((_, i) => words.slice(i).join('').toLowerCase());
 }
 export const normalizeFieldName = (field) =>
   field.replace(/[_-]/g, '').toLowerCase();

package/src/document-person-cues.js

@@ -4,8 +4,11 @@ import { collectMatches } from './detection.js';
 // capitalized-word matching for the rest of an English log or document.
 const boundary = '[\\p{L}\\p{M}\\p{N}_]';
 const name = "[A-Z][a-z]{1,47}(?:['’-][A-Z]?[a-z]{1,47})?";
+// Honorifics count only when capitalized: lowercase `ms` is the millisecond
+// unit in logs and lowercase `dr` is usually a word or path fragment. A
+// capitalized `Ms` right after a number is still a unit (`took 30 Ms Retry`).
 const cue =
-  '(?:[Cc]andidate|[Rr]ecruiter|[Aa]pplicant|[Dd]ear|[Hh]i|[Mm]r\\.?|[Mm]s\\.?|[Dd]r\\.?|[Mm]essage[ \\t]+from)';
+  '(?:[Cc]andidate|[Rr]ecruiter|[Aa]pplicant|[Dd]ear|[Hh]i|Mr\\.?|(?<!\\d[ \\t]*)Ms\\.?|Dr\\.?|[Mm]essage[ \\t]+from)';
 const pattern = new RegExp(
   `(?<!${boundary})${cue}[ \\t]+(${name})(?!${boundary})`,
   'gu'

package/src/index.d.ts

@@ -118,6 +118,7 @@ export interface SanitizerOptions {
 }
 /** Value-free diagnostics; enabled only when a debug callback is supplied. */
 export type DebugEvent =
+  | { event: 'strict-attributes'; detected: number; suppressed: number }
   | { event: 'detected'; count: number }
   | { event: 'document-experience' | 'proper-name-prose'; count: number }
   | { event: 'residual-redaction'; round: number }

package/src/strict-attributes.js

@@ -0,0 +1,71 @@
+import { collectMatches } from './detection.js';
+
+const attributes =
+  /(?<![\p{L}\p{N}_])(?:женщин[аыуеой]{1,2}|мужчин[аыуеой]{1,2}|замужем|женат(?:а|ы)?|(?:один|одна|двое|трое|четверо|пятеро|\d{1,2})[ \t]+(?:детей|реб[её]нок|реб[её]нка)|русск(?:ая|ий|ого|ой|ие)|татар(?:ин|ка|ы)|православн(?:ый|ая|ого|ой)|мусульман(?:ин|ка)|инвалидност[ьи](?:[ \t]+\d[ \t]+группы)?|беременн(?:а|ая|ость)|судимость(?:[ \t]+погашена)?|член[ \t]+партии|woman|women|male|female|married|divorced|pregnant|pregnancy|disability|christian|muslim|jewish|orthodox|hispanic|criminal[ \t]+record|party[ \t]+member|(?:one|two|three|\d{1,2})[ \t]+children)(?![\p{L}\p{N}_])/giu;
+const additionalAttributes =
+  /(?<![\p{L}\p{N}_])(?:диабет(?:ом|а|у|е)?|ВИЧ(?:-положительн(?:ый|ая|ого|ой))?|болен|больна|больны|католик(?:ом|а|и)?|католичк(?:а|ой|и)|буддист(?:ка|кой|ом|а|ы)?|атеист(?:ка|кой|ом|а|ы)?|украин(?:ец|ка|цы|цем|кой)|белорус(?:ка|кой|ом|а|ы)?|несудим(?:а|ы)?|судим(?:а|ы)?|член[ \t]+политической[ \t]+партии|холост(?:а|ы)?|развед[её]н(?:а|ы)?|в[ \t]+разводе|diabetes|diabetic|HIV(?:[ \t-]+positive)?|cancer|buddhist|catholic|atheist|ukrainian|belarusian|previous[ \t]+conviction|member[ \t]+of[ \t]+(?:a[ \t]+)?(?:political[ \t]+)?party|unmarried|widowed)(?![\p{L}\p{N}_])/giu;
+
+// Bounded suffixes cover Russian cases, plurals and adjectival forms without
+// turning a stem into an unbounded substring search. Category cues below cover
+// unfamiliar diagnoses, nationalities and religions outside this lexicon.
+const categoryForms =
+  /(?<![\p{L}\p{N}_])(?:(?:диабет|католик|католич|буддист|атеист|белорус|православн|беременн|судим)[\p{L}]{0,8}|татар(?:ин(?:а|у|ом|е)?|к[аиуе]|кой|ы|ам|ами|ах)?|мусульман(?:ин(?:а|у|ом|е)?|к[аиуе]|кой|е|ам|ами|ах)?|украин(?:ец|ца|цу|цем|це|цы|цев|цам|цами|цах|к[аиуе]|кой)|инвалид(?:а|у|ом|е|ы|ов|ам|ами|ах|к[аиу]|кой)?|армян(?:ин|ина|ину|ином|ине|е|ам|ами|ах|ка|ки|ке|ку|кой)|евре(?:й|я|ю|ем|и|ев|ям|ями|ях|йк[аиуе]|йкой)|казах(?:а|у|ом|е|и|ов|ам|ами|ах|к[аиуе]|кой)?|христиан(?:ин|ина|ину|ином|ине|е|ам|ами|ах|к[аиуе]|кой)|иуде(?:й|я|ю|ем|и|ев|ям|ями|ях|йк[аиуе]|йкой)|ислам[\p{L}]{0,5}|эпилепси[\p{L}]{1,5}|астм[\p{L}]{1,5}|онкологи[\p{L}]{1,5}|туберкул[её]з[\p{L}]{0,5}|психиатр[\p{L}]{0,6}|депресси[\p{L}]{1,5}|гепатит[\p{L}]{0,5}|гомосексуал[\p{L}]{0,7}|лесбиян[\p{L}]{1,5}|ге(?:й|я|ю|ем|и|ев|ям|ями|ях)|нетрадиционн[\p{L}]{2,3}[ \t]+ориентаци[\p{L}]{1,3}|отбыва(?:л[аи]?|ет|ют|вш[\p{L}]{2,3})[ \t]+наказани[\p{L}]{1,3}|ВИЧ-инфицирован[\p{L}]{0,4}|epilepsy|asthma|tuberculosis|hepatitis|depression|gay|lesbian|homosexual|bisexual|armenian)(?![\p{L}\p{N}_])/giu;
+const categoryCue =
+  /(?<![\p{L}\p{N}_])(?:по[ \t]+национальности|национальность|этничность|вероисповедание|религия|диагноз|исповеду(?:ет|ют|ю|ешь)|боле(?:ет|ют|ю|ешь)|страда(?:ет|ют|ю|ешь)[ \t]+от|на[ \t]+уч[её]те[ \t]+у|diagnosed[ \t]+with|suffers[ \t]+from|nationality|ethnicity|religion|diagnosis)(?:["']?[ \t]*[:=][ \t]*|[ \t]+)["']?([\p{L}][\p{L}\p{M}-]{0,63})(?![\p{L}\p{M}\p{N}_])/giu;
+const ambiguous =
+  /(?<![\p{L}\p{N}_])(?:disabled|single|man|men|russian|asian)(?![\p{L}\p{N}_])/giu;
+
+function personalContext(text, match) {
+  const prefix = text.slice(Math.max(0, match.index - 128), match.index);
+  const suffix = text.slice(
+    match.index + match[0].length,
+    match.index + match[0].length + 64
+  );
+  // These continuations remain technical even after a pronoun ("it is a
+  // single-flight lock"). No document-wide cue can authorize another clause.
+  if (
+    /^(?:-flight\b|[ \t]+(?:page|pages|run|worker|thread|lock|region|endpoint|locale|module|feature|flag)\b)/iu.test(
+      suffix
+    )
+  ) {
+    return false;
+  }
+  return (
+    /(?<![\p{L}\p{N}_])(?:i|he|she|we|they|candidate|applicant|patient|person|employee)(?:[ \t]+(?:am|is|are|was|were|identifies[ \t]+as))?(?:[ \t]+(?:a|an))?[ \t]+$/iu.test(
+      prefix
+    ) ||
+    /(?<![\p{L}\p{N}_])(?:gender|sex|nationality|ethnicity|marital[ \t_-]*status|disability)(?:["'][ \t]*)?[ \t]*[:=][ \t]*["']?$/iu.test(
+      prefix
+    ) ||
+    /^[ \t]+(?:candidate|applicant|patient|person|people|employee)(?![\p{L}\p{N}_])/iu.test(
+      suffix
+    )
+  );
+}
+
+export function detectStrictAttributes(text, emit, options) {
+  let detected = 0,
+    suppressed = 0;
+  const details = {
+    type: 'PERSONAL_ATTRIBUTE',
+    rule: 'strict-attribute',
+    category: 'personal',
+    confidence: 0.9,
+  };
+  const report = (finding) => {
+    detected++;
+    emit(finding);
+  };
+  for (const pattern of [attributes, additionalAttributes, categoryForms]) {
+    collectMatches(text, pattern, report, details);
+  }
+  collectMatches(text, categoryCue, report, details, 1);
+  collectMatches(text, ambiguous, report, details, 0, (_value, match) => {
+    const accepted = personalContext(text, match);
+    if (!accepted) {
+      suppressed++;
+    }
+    return accepted;
+  });
+  options.debug?.({ event: 'strict-attributes', detected, suppressed });
+}

package/src/strict.js

@@ -3,12 +3,9 @@ import { months } from './personal-context.js';
 import { isDocumentTechnology } from './document-experience.js';
 import { placeholderValues } from './placeholders.js';
 import { generatedValues } from './fake.js';
+import { detectStrictAttributes } from './strict-attributes.js';
 
 const boundary = '[\\p{L}\\p{N}_]';
-const attributes =
-  /(?<![\p{L}\p{N}_])(?:женщин[аыуеой]{1,2}|мужчин[аыуеой]{1,2}|замужем|женат(?:а|ы)?|(?:один|одна|двое|трое|четверо|пятеро|\d{1,2})[ \t]+(?:детей|реб[её]нок|реб[её]нка)|русск(?:ая|ий|ого|ой|ие)|татар(?:ин|ка|ы)|православн(?:ый|ая|ого|ой)|мусульман(?:ин|ка)|инвалидност[ьи](?:[ \t]+\d[ \t]+группы)?|беременн(?:а|ая|ость)|судимость(?:[ \t]+погашена)?|член[ \t]+партии|woman|women|man|men|male|female|married|divorced|pregnant|pregnancy|disabled|disability|christian|muslim|jewish|orthodox|russian|hispanic|asian|criminal[ \t]+record|party[ \t]+member|(?:one|two|three|\d{1,2})[ \t]+children)(?![\p{L}\p{N}_])/giu;
-const additionalAttributes =
-  /(?<![\p{L}\p{N}_])(?:диабет(?:ом|а|у|е)?|ВИЧ(?:-положительн(?:ый|ая|ого|ой))?|болен|больна|больны|католик(?:ом|а|и)?|католичк(?:а|ой|и)|буддист(?:ка|кой|ом|а|ы)?|атеист(?:ка|кой|ом|а|ы)?|украин(?:ец|ка|цы|цем|кой)|белорус(?:ка|кой|ом|а|ы)?|несудим(?:а|ы)?|судим(?:а|ы)?|член[ \t]+политической[ \t]+партии|холост(?:а|ы)?|развед[её]н(?:а|ы)?|в[ \t]+разводе|diabetes|diabetic|HIV(?:[ \t-]+positive)?|cancer|buddhist|catholic|atheist|ukrainian|belarusian|previous[ \t]+conviction|member[ \t]+of[ \t]+(?:a[ \t]+)?(?:political[ \t]+)?party|single|unmarried|widowed)(?![\p{L}\p{N}_])/giu;
 
 function isDuration(text, match, replacements) {
   // Keep cues on the same clause; an age on the next line is independent.
@@ -57,14 +54,7 @@ export function detectStrict(text, emit, options) {
   if (!options.strict) {
     return;
   }
-  for (const pattern of [attributes, additionalAttributes]) {
-    collectMatches(text, pattern, emit, {
-      type: 'PERSONAL_ATTRIBUTE',
-      rule: 'strict-attribute',
-      category: 'personal',
-      confidence: 0.9,
-    });
-  }
+  detectStrictAttributes(text, emit, options);
   const date = new RegExp(
     `(?<!${boundary})(?:\\d{4}-\\d{2}-\\d{2}|\\d{1,2}[./]\\d{1,2}[./]\\d{4}|\\d{1,2}[ \\t]+(?:${months})[ \\t]+\\d{4})(?!${boundary}|[Tt]\\d)`,
     'giu'

package/src/vocabulary.js

@@ -1,6 +1,6 @@
 import { failure, collectMatches } from './detection.js';
 import { wordPattern } from './personal-variants.js';
-import { normalizeFieldName } from './document-fields.js';
+import { fieldSuffixes, normalizeFieldName } from './document-fields.js';
 
 export const defaultVocabulary = Object.freeze(
   Object.fromEntries(
@@ -170,7 +170,8 @@ export function vocabularyField(field, options) {
   if (!vocabulary) {
     return undefined;
   }
-  return vocabulary.fields.has(normalizeFieldName(field))
+  return vocabulary.fields.has(normalizeFieldName(field)) ||
+    fieldSuffixes(field).some((suffix) => vocabulary.fields.has(suffix))
     ? true
     : options.vocabulary.personalFields?.replace
       ? false