@link-foundation/sensitive-data-sanitizer 0.13.0 → 0.14.0

package/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 0.14.0
+
+### Minor Changes
+
+- d5b2aac: Add known-only reversible sanitization, Unicode known-value boundaries and bounded Russian entity case matching. Correct placeholder namespace collisions, Russian document entities and fake-name roles, and credential prose false positives. Report irreversible replacements in placeholder results.
+
 ## 0.13.0
 
 ### Minor Changes

package/README.md

@@ -55,6 +55,11 @@ sensitive-data-sanitizer restore safe.txt --map private.lino --output restored.t
 sensitive-data-sanitizer redact cv.txt --profile documents --placeholder --no-map
 ```
 
+`knownOnly: true` (CLI `--known-only`) limits replacement to `knownPersonal`,
+`knownSecrets` and reused map identities, preserving fictional examples.
+Placeholder results report `irreversibleRedactions`; a nonzero count means some
+replacements, such as credentials excluded from the map, cannot be restored.
+
 For irreversible output, choose `transformation: { mode: 'placeholder', map:
 false }`: no restore map is returned or written. Detection limits still apply
 in either mode. See the [placeholder API](docs/API.md#reversible-placeholders),

package/bin/sensitive-data-sanitizer.js

@@ -42,6 +42,7 @@ history rewrite SOURCE --output DIRECTORY [--apply]
 --in-place          Atomically replace the input file (redact only)
 --native-only       Disable Secretlint; selected local detectors remain required
 --credentials-only  Preserve personal/account details; redact credentials only
+--known-only        Replace only knownPersonal/knownSecrets and reused map values
 --env-secrets       Include values of all credential-named environment variables
 --gh-config         Include tokens from the local gh hosts.yml credential file
 --paranoid          Add entropy detection (can redact innocent identifiers)
@@ -88,6 +89,7 @@ function parseArgs(argv) {
     ['--in-place', 'inPlace'],
     ['--native-only', 'nativeOnly'],
     ['--credentials-only', 'credentialsOnly'],
+    ['--known-only', 'knownOnly'],
     ['--env-secrets', 'envSecrets'],
     ['--gh-config', 'ghConfig'],
     ['--paranoid', 'paranoid'],
@@ -339,6 +341,9 @@ async function loadOptions(config) {
 
 async function featureOptions(config, options) {
   await localCredentials(config, options);
+  if (config.knownOnly) {
+    options.knownOnly = true;
+  }
   if (config.credentialsOnly) {
     options.credentialsOnly = true;
   }

package/docs/API.md

@@ -19,6 +19,24 @@ Import the ESM root entry; complete declarations are in `src/index.d.ts`. Inputs
 
 ## Options
 
+### Replace only known values
+
+Use `knownOnly: true` to preserve fictional examples while replacing selected
+private values. This applies to sync/async, JSON/JSONL, streams and fake mode.
+It skips automatic credential detection too; supply `knownSecrets` explicitly
+for credentials that must be removed. `credentialsOnly` can further exclude
+personal values. CLI: `--known-only` with a `--config` containing the known lists.
+
+```js
+const result = sanitize('Анна Смирнова; example Jane Doe', {
+  knownOnly: true,
+  knownPersonal: [{ type: 'PERSON', value: 'Анна Смирнова' }],
+  transformation: { mode: 'placeholder' },
+});
+// result.text: 'Person-1; example Jane Doe'
+restore(result.text, result.map); // exact original
+```
+
 ### Reversible placeholders
 
 `transformation: {mode: 'placeholder'}` emits typed tokens such as `Person-1`,
@@ -42,7 +60,9 @@ are left alone.
 `format` is a template with exactly one type field (`{Type}`, `{TYPE}` or
 `{type}`) and one `{n}` field. Examples: `{Type}-{n}`, `[{TYPE}_{n}]`,
 `{{{type}_{n}}}`. Sanitization refuses input containing any token in the chosen
-namespace, including numbers not allocated yet. The map's format cannot change
+namespace for built-in and supplied type labels, including numbers not allocated yet.
+Unicode letters, marks, digits and underscores bound those tokens. Ordinary
+technical names such as GPT-4, ISO-9001 and Windows-10 are not reserved. The map's format cannot change
 on reuse. A malformed map or ambiguous exact restoration blocks output.
 
 `createRestoreMap({format?})` creates an empty reusable map. Pass a map as
@@ -68,13 +88,20 @@ is plaintext personal data, not encrypted storage; protect it as the original.
 
 `credentials: true` explicitly opts credentials into `Credential-n` entries.
 Otherwise they remain `[REDACTED]` and cannot be recovered, including through
-occurrence metadata. `map: false` returns only the anonymized text and ordinary
+occurrence metadata. Placeholder results include `irreversibleRedactions`, the
+number of replacements excluded from the restore map (including mixed policies).
+Check that this is zero before relying on exact restoration. A positive count
+means `restore()` cannot reconstruct all of the input. `map: false` returns only the anonymized text and ordinary
 value-free finding metadata. The temporary numbering registry is discarded
 after an operation (or the complete stream); no map file is created. Residual verification
 still runs, including every required engine for `createSanitizer`.
 
 Name components, unique initial-plus-surname forms, existing transliteration
 aliases and selected Russian `-ов`/`-ев` inflections share an entity token.
+Explicit `knownPersonal` and reused map entries also match bounded Russian noun,
+adjective, feminine surname and given-name case endings, two-part person names
+in reverse order, and organization cores without legal forms/quotes. Inflected
+forms consume whole Unicode words; minimum stem lengths limit overmatching.
 This is conservative heuristic identity linking, not general coreference:
 ambiguous surnames/initials and unrelated people are not intentionally merged.
 Every replaced occurrence retains its exact original surface privately. With
@@ -99,28 +126,29 @@ derived model answer. Input, output and map paths must be separate. Maps are
 saved separately before stdout publication. `--profile documents` enables the
 new document recognizers; `documentEntities: false` turns them off explicitly.
 
-| Option             | Default / meaning                                                                                       |
-| ------------------ | ------------------------------------------------------------------------------------------------------- |
-| `documentEntities` | Off; bounded organization, education and location rules; enabled by the documents profile               |
-| `profile`          | Off; `'publication'` sets a 0.3 threshold and identity masking; `'documents'` enables document entities |
-| `knownSecrets`     | `[]`; exact literal credential values, regardless of length or format                                   |
-| `knownPersonal`    | `[]`; `{type, value}` literals with Unicode case-insensitive matching                                   |
-| `publicEntities`   | `[]`; exact PERSON/ORGANIZATION/EDUCATION/LOCATION/EMAIL exception with HTTPS evidence and review date  |
-| `findings`         | `[]`; validated detections for the original input, supplied by the caller                               |
-| `detectors`        | `[]`; additional required callbacks in the asynchronous engine                                          |
-| `secretlint`       | `true` in the asynchronous engine                                                                       |
-| `decode`           | `true`; bounded base64/base64url, hex and percent decoding                                              |
-| `paranoid`         | `false`; entropy-based supplemental credential heuristic                                                |
-| `maxInputLength`   | 33,554,432 UTF-16 units                                                                                 |
-| `maxFindings`      | 100,000 detector hits before deduplication                                                              |
-| `minConfidence`    | 0.5; personal findings below this score remain visible to inspect but are not replaced                  |
-| `threshold`        | Alias for minConfidence; both options must agree when supplied                                          |
-| `debug`            | Optional trusted callback for value-free counts, timing and event metadata                              |
-| `identityMask`     | `false`; identity types keep 2+2 Unicode characters with fixed `***`, or redact below five              |
-| `fakeIdentity`     | Off; `'specimen-and-synthetic'` retains explicitly fake identity values                                 |
-| `fakeValues`       | `[]`; exact caller-reviewed identity values to retain, regardless of fakeIdentity                       |
-| `structured`       | Off; `'json'` or `'jsonl'` validates and sanitizes string keys/values separately                        |
-| `structuralFields` | `[]`; declared property names suppress weak name/ID/entropy heuristics in their string values           |
+| Option             | Default / meaning                                                                                                                                           |
+| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `documentEntities` | Off; bounded organization, education and location rules; enabled by the documents profile                                                                   |
+| `profile`          | Off; `'publication'` sets a 0.3 threshold and identity masking; `'documents'` enables document entities                                                     |
+| `knownOnly`        | `false`; disable all automatic detection (including credentials, Secretlint and custom detectors); use only explicit known values and reused map identities |
+| `knownSecrets`     | `[]`; exact literal credential values, regardless of length or format                                                                                       |
+| `knownPersonal`    | `[]`; `{type, value}` values with whole-word Unicode case-insensitive matching and bounded Russian case forms                                               |
+| `publicEntities`   | `[]`; exact PERSON/ORGANIZATION/EDUCATION/LOCATION/EMAIL exception with HTTPS evidence and review date                                                      |
+| `findings`         | `[]`; validated detections for the original input, supplied by the caller                                                                                   |
+| `detectors`        | `[]`; additional required callbacks in the asynchronous engine                                                                                              |
+| `secretlint`       | `true` in the asynchronous engine                                                                                                                           |
+| `decode`           | `true`; bounded base64/base64url, hex and percent decoding                                                                                                  |
+| `paranoid`         | `false`; entropy-based supplemental credential heuristic                                                                                                    |
+| `maxInputLength`   | 33,554,432 UTF-16 units                                                                                                                                     |
+| `maxFindings`      | 100,000 detector hits before deduplication                                                                                                                  |
+| `minConfidence`    | 0.5; personal findings below this score remain visible to inspect but are not replaced                                                                      |
+| `threshold`        | Alias for minConfidence; both options must agree when supplied                                                                                              |
+| `debug`            | Optional trusted callback for value-free counts, timing and event metadata                                                                                  |
+| `identityMask`     | `false`; identity types keep 2+2 Unicode characters with fixed `***`, or redact below five                                                                  |
+| `fakeIdentity`     | Off; `'specimen-and-synthetic'` retains explicitly fake identity values                                                                                     |
+| `fakeValues`       | `[]`; exact caller-reviewed identity values to retain, regardless of fakeIdentity                                                                           |
+| `structured`       | Off; `'json'` or `'jsonl'` validates and sanitizes string keys/values separately                                                                            |
+| `structuralFields` | `[]`; declared property names suppress weak name/ID/entropy heuristics in their string values                                                               |
 
 Unknown options and invalid limits are errors. `knownSecretsFromEnv(env, names?)` discovers credential-named variables (excluding auth configuration names) when names are omitted, or uses an explicit selection, deduplicating nonempty values. It never invokes shell commands or reads auth files. `personalVariants(entries)` generates numeric and ISO/day-first date spellings; name case variants are already handled by detection. Neither helper invents transliterations.
 

package/docs/COVERAGE.md

@@ -29,6 +29,15 @@ remains available under those profiles.
 | `LOCATION`     | Exact city/region entries from a bounded offline vocabulary                                                                           | 0.60       |
 | `LOCATION`     | Bounded proper names after location labels, including based in, located in, relocated from, г., город and locale equivalents          | 0.75       |
 
+Russian document rules also accept legal-form quoted names with lowercase words,
+digits and hyphens (up to 160 characters), organization Group/Групп/Holding/Холдинг
+markers, adjectival institution names with lowercase connectors and optional
+`имени` tails, education acronyms in context, residence labels and city/country
+phrasing. A known given name after `Меня зовут` or `С уважением` scores 0.70.
+Detected organization cores propagate within a document and across confirmed
+stream records. Russian regional cities include Ярославль and Кострома; the
+vocabulary remains finite. Default profiles do not enable these document rules.
+
 These confidence values indicate heuristic strength, not calibrated
 probabilities. `minConfidence` still controls replacement. Findings retain
 their confidence even below threshold. `publicEntities` supports exact reviewed
@@ -41,7 +50,7 @@ Spanish, French, German, Portuguese, Russian, Japanese, Simplified/Traditional
 Chinese, Korean, Arabic, Hebrew, Hindi, Turkish, Indonesian, Vietnamese and
 Thai each have list-form employer, school and city fixtures. Proper-name
 rules are finite (up to six capitalized words, or four words in scripts without
-case); they do not implement grammatical parsing. Common lowercase connectors,
+case); they do not implement grammatical parsing. Other lowercase connectors,
 unusual typography, unsupported legal forms and unfamiliar institutions can be
 missed. Legal-form words and uncased scripts can also yield false positives.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-foundation/sensitive-data-sanitizer",
-  "version": "0.13.0",
+  "version": "0.14.0",
   "description": "Offline sensitive-text sanitization with credential, multilingual personal-data and scanner union detection",
   "type": "module",
   "main": "src/index.js",

package/src/context.js

@@ -8,6 +8,7 @@ import { plausiblePerson } from './names.js';
 // unrelated words such as passport. Verb forms cover the supported locales.
 export const credentialVerbs = [
   'is',
+  'are',
   'was',
   'equals',
   'es',
@@ -45,12 +46,11 @@ export const credentialVerbs = [
   'คือ',
 ];
 const verbs = credentialVerbs.map(escapePattern).join('|');
-const key = `(?:[\\p{L}\\p{N}]+[_.-])*(?:auth|authorization)(?:[_.-][\\p{L}\\p{N}]+)*|[\\p{L}\\p{N}_.-]{0,32}(?:${labels.credential
+const label = labels.credential
   .filter((word) => !['auth', 'authorization'].includes(word))
   .map(escapePattern)
-  .join(
-    '|'
-  )})[\\p{L}\\p{N}_.-]{0,32}|(?:[\\p{L}\\p{N}_.-]{1,32}[_-])?(?:pass|pw)(?:[_-][\\p{L}\\p{N}_.-]{1,32})?`;
+  .join('|');
+const key = `(?:[\\p{L}\\p{N}]+[_.-])*(?:auth|authorization|PGPASSWORD|${label}|pass|pw)(?:[_.-][\\p{L}\\p{N}]+)*`;
 // A qualifier such as "for staging" stays on one logical line: it never
 // crosses an escaped line break, an escape sequence or an enclosing quote.
 // Verbs are whole words, so "request" does not end in the verb "est".
@@ -60,8 +60,6 @@ const credentialPrefix = new RegExp(
 );
 // Bare generic labels are common English nouns. After a verb they need a
 // value that looks generated; "the token is valid" is prose, not a secret.
-const genericLabel =
-  /^(?:token|sig|sas|secret|auth|authorization|cookie|секрет|токен)$/iu;
 function generatedWord(text, range) {
   const word = /^[^\s]+/u
     .exec(text.slice(range.start, range.end))?.[0]
@@ -170,7 +168,7 @@ export function authorizationRange(text, start) {
   for (; end < text.length; end++) {
     const char = text[end];
     const code = text.charCodeAt(end);
-    if (code < 32 || code === 127 || /[\s"'\\,]/.test(char)) {
+    if (authorizationDelimiter(char, code)) {
       break;
     }
     if (char === '<' && end !== tokenStart && !brackets.length) {
@@ -185,7 +183,13 @@ export function authorizationRange(text, start) {
     }
   }
   const token = text.slice(tokenStart, end);
-  return !token || credentialPlaceholder(token) ? undefined : { start, end };
+  return usableAuthorization(token) ? { start, end } : undefined;
+}
+function usableAuthorization(token) {
+  return Boolean(token) && !credentialPlaceholder(token);
+}
+function authorizationDelimiter(char, code) {
+  return code < 32 || code === 127 || /[\s"'\\,]/.test(char);
 }
 function quotedRange(text, start, opening, maxLength) {
   const depth = opening[1].length,
@@ -323,6 +327,13 @@ function backslashes(text, index) {
   return count;
 }
 
+function sourceReference(label, tail) {
+  return (
+    /\.(?:[cm]?[jt]sx?|py|go|rs|c|cpp|h|java|cs|rb|sh)$/iu.test(label) &&
+    /^\d+(?:[):]|\b)/u.test(tail)
+  );
+}
+
 function detectCredentials(text, emit) {
   const details = { category: 'credential', type: 'SECRET', rule: 'context' };
   for (const match of text.matchAll(credentialPrefix)) {
@@ -341,9 +352,12 @@ function detectCredentials(text, emit) {
     let range = /^(?:proxy-)?authorization$/i.test(match[1])
       ? authorizationRange(text, start)
       : valueRange(text, start, mode);
-    if (range && mode === 'prose' && genericLabel.test(match[1])) {
+    if (range && mode === 'prose') {
       range = generatedWord(text, range);
     }
+    if (sourceReference(match[1], text.slice(start))) {
+      continue;
+    }
     if (range) {
       emitValue(text, range.start, range.end, emit, details, match[1]);
     }

package/src/detection.js

@@ -13,6 +13,7 @@ import {
   credentialPlaceholder,
 } from './credentials.js';
 import { localAddress } from './public.js';
+import { detectKnownPersonal } from './personal-variants.js';
 import { detectDocumentEntities } from './document-entities.js';
 
 export const REDACTED = '[REDACTED]';
@@ -133,6 +134,29 @@ function detectCookies(text, emit) {
 
 export function nativeDetect(text, options, emit) {
   const credential = { type: 'SECRET', category: 'credential' };
+  for (const value of options.knownSecrets ?? []) {
+    collectMatches(text, new RegExp(escapePattern(value), 'g'), emit, {
+      ...credential,
+      rule: 'known-secret',
+    });
+  }
+  detectKnownPersonal(
+    text,
+    options.knownPersonal,
+    'known-personal',
+    emit,
+    options
+  );
+  detectKnownPersonal(
+    text,
+    options.confirmedPersonal,
+    'confirmed-personal',
+    emit,
+    options
+  );
+  if (options.knownOnly) {
+    return;
+  }
   detectGitleaks(
     text,
     emit,
@@ -143,31 +167,6 @@ export function nativeDetect(text, options, emit) {
   for (const [rule, pattern] of serviceRules) {
     collectMatches(text, pattern, emit, { ...credential, rule });
   }
-  for (const value of options.knownSecrets ?? []) {
-    collectMatches(text, new RegExp(escapePattern(value), 'g'), emit, {
-      ...credential,
-      rule: 'known-secret',
-    });
-  }
-  for (const entry of options.knownPersonal ?? []) {
-    collectMatches(text, new RegExp(escapePattern(entry.value), 'giu'), emit, {
-      type: entry.type,
-      category: 'personal',
-      rule: 'known-personal',
-    });
-  }
-  for (const entry of options.confirmedPersonal ?? []) {
-    const pattern = new RegExp(
-      `(?<![\\p{L}\\p{N}])${personalPattern(entry.value)}(?![\\p{L}\\p{N}])`,
-      'giu'
-    );
-    collectMatches(text, pattern, emit, {
-      type: entry.type,
-      category: 'personal',
-      rule: 'confirmed-personal',
-      confidence: 0.9,
-    });
-  }
   detectContext(text, emit, options.credentialsOnly);
   if (!options.credentialsOnly) {
     detectPersonal(text, emit);

package/src/document-entities.js

@@ -1,4 +1,6 @@
 import { collectMatches, escapePattern } from './detection.js';
+import { firstNames } from './names.js';
+import { organizationCore, personalRegex } from './personal-variants.js';
 import { documentVocabulary } from './document-vocabulary.js';
 
 const words = (column) =>
@@ -14,7 +16,7 @@ const latinName = `${proper}(?:[ \\t]+${proper}){0,5}`;
 const uncased =
   '[\\p{Script=Han}\\p{Script=Hiragana}\\p{Script=Katakana}\\p{Script=Hangul}\\p{Script=Arabic}\\p{Script=Hebrew}\\p{Script=Devanagari}\\p{Script=Thai}\\p{M}]{2,48}';
 const name = `(?:${latinName}|${uncased}(?:[ \\t]+${uncased}){0,3})`;
-const quoted = `(?:[«“「『"]${name}[»”」』"]|${name})`;
+const quoted = `(?:«[^«»\\r\\n]{1,160}»|“[^“”\\r\\n]{1,160}”|"[^"\\r\\n]{1,160}"|「[^「」\\r\\n]{1,160}」|『[^『』\\r\\n]{1,160}』|${name})`;
 const boundary = '[\\p{L}\\p{M}\\p{N}_]';
 const organizations = new RegExp(
   `(?<!${boundary})(?:(?:${legal})[ \\t]*${quoted}|${quoted}[ \\t]*(?:${legal}))(?!${boundary})`,
@@ -33,11 +35,38 @@ const labelledPlace = new RegExp(
   'gu'
 );
 
+const ruWord = '[А-ЯЁа-яё][А-ЯЁа-яё-]{1,47}';
+const ruSchool = new RegExp(
+  `(?<!${boundary})[А-ЯЁ][а-яё]{2,30}(?:-[А-ЯЁа-яё][а-яё]{1,30})?(?:[ \t]+[а-яё-]{2,40}){0,4}[ \t]+(?:университет|институт|колледж|школа|академия)(?:[ \t]+имени[ \t]+${ruWord}(?:[ \t]+[А-ЯЁ][а-яё-]{2,40}){0,2})?(?!${boundary})`,
+  'gu'
+);
+const ruAcronym =
+  /(?<![\p{L}\p{N}_])(?:НИУ[ \t]+[А-ЯЁ]{2,12}|(?:МГУ|СПбГУ|МГТУ|РГГУ|ВШЭ)(?=[, \t]+(?:магистратура|бакалавриат|университет|обучение)))(?![\p{L}\p{N}_])/gu;
+const ruResidence =
+  /(?:[Гг]ород|[Пп]роживаю|[Пп]роживание|[Мм]есто жительства)[ \t]*:[ \t]*([А-ЯЁ][а-яё-]{2,47})|([А-ЯЁ][а-яё-]{2,47})(?=,[ \t]*(?:готов[а]?[ \t]+к[ \t]+переезду|Россия|Беларусь|Казахстан|Украина))/gu;
+const ruIntroduction = new RegExp(
+  `(?:[Мм]еня зовут|[Сс] уважением)[, \t]+(${firstNames.ru.split(' ').join('|')})(?!${boundary})`,
+  'gu'
+);
+const groupOrganization = new RegExp(
+  `(?<!${boundary})${latinName}[ \t]+(?:Групп|Group|Холдинг|Holding)(?!${boundary})`,
+  'gu'
+);
+
 export function detectDocumentEntities(text, emit, options) {
   if (!options.documentEntities) {
     return;
   }
+  const found = [];
+  const capture = (finding) => {
+    found.push(finding);
+    emit(finding);
+  };
   for (const [type, rule, pattern, group, confidence] of [
+    ['ORGANIZATION', 'organization-group', groupOrganization, 0, 0.7],
+    ['EDUCATION', 'education-russian', ruSchool, 0, 0.65],
+    ['EDUCATION', 'education-acronym', ruAcronym, 0, 0.65],
+    ['PERSON', 'document-introduction', ruIntroduction, 1, 0.7],
     ['ORGANIZATION', 'organization-legal-form', organizations, 0, 0.7],
     ['EDUCATION', 'education-type', schools, 0, 0.65],
     ['LOCATION', 'location-gazetteer', places, 0, 0.6],
@@ -46,9 +75,36 @@ export function detectDocumentEntities(text, emit, options) {
     collectMatches(
       text,
       pattern,
-      emit,
+      capture,
       { type, rule, category: 'personal', confidence },
       group
     );
   }
+  for (const match of text.matchAll(ruResidence)) {
+    const value = match[1] ?? match[2];
+    const start = match.index + match[0].lastIndexOf(value);
+    emit({
+      start,
+      end: start + value.length,
+      type: 'LOCATION',
+      rule: 'location-residence',
+      category: 'personal',
+      confidence: 0.75,
+    });
+  }
+  // Organization cores are confirmed only by a bounded legal-form finding.
+  for (const finding of found.filter((f) => f.type === 'ORGANIZATION')) {
+    const core = organizationCore(text.slice(finding.start, finding.end));
+    if (core.length < 3 || core.length > 256) {
+      continue;
+    }
+    for (const match of text.matchAll(personalRegex(core, 'ORGANIZATION'))) {
+      emit({
+        ...finding,
+        start: match.index,
+        end: match.index + match[0].length,
+        rule: 'organization-core',
+      });
+    }
+  }
 }

package/src/document-vocabulary.js

@@ -35,7 +35,7 @@ export const documentVocabulary = [
     'ru',
     'ООО|АО|ПАО|ИП|ЗАО',
     'Университет|университет|Институт|институт|Школа|школа',
-    'Москва|Санкт-Петербург|Казань|Татарстан',
+    'Москва|Санкт-Петербург|Казань|Татарстан|Ярославль|Кострома|Новосибирск|Екатеринбург|Нижний Новгород|Самара|Омск|Челябинск|Ростов-на-Дону|Уфа|Красноярск|Пермь|Воронеж|Волгоград|Саратов|Тюмень|Тольятти|Ижевск|Барнаул|Иркутск|Хабаровск|Владивосток|Махачкала|Томск|Оренбург|Кемерово|Рязань|Астрахань|Пенза|Липецк|Киров|Чебоксары|Тула|Калининград|Курск|Улан-Удэ|Ставрополь|Сочи|Тверь|Брянск|Белгород|Владимир|Архангельск|Калуга|Смоленск|Вологда|Саранск|Орёл',
   ],
   ['ja', '株式会社|有限会社|合同会社', '大学|学院|学校', '東京|大阪|北海道'],
   ['zh', '有限公司|股份有限公司', '大学|学院|学校', '北京|上海|广东'],

package/src/engines.js

@@ -162,8 +162,12 @@ export function createSanitizer(options = {}) {
     }
   }
   const engines =
-    options.secretlint === false ? [] : [{ detect: secretlintDetect }];
-  engines.push(...(options.detectors ?? []));
+    options.secretlint === false || options.knownOnly
+      ? []
+      : [{ detect: secretlintDetect }];
+  if (!options.knownOnly) {
+    engines.push(...(options.detectors ?? []));
+  }
   let prepared;
   async function analyze(
     text,

package/src/entities.js

@@ -36,6 +36,13 @@ for (const [vendor, catalog] of Object.entries(entityCatalogs)) {
     entries.set(name.toUpperCase(), entry);
   }
 }
+export const catalogTypes = () => [
+  ...new Set(
+    [...entries.values()]
+      .filter((entry) => entry.category === 'personal')
+      .map((entry) => (entry.type === 'DATE_OF_BIRTH' ? 'DATE' : entry.type))
+  ),
+];
 // All documented names are native labelled recognizers; automatic format
 // recognizers are separate and do not claim the vendors' model behavior.
 const prefix = new RegExp(

package/src/fake-names.js

@@ -146,6 +146,58 @@ function scriptLocale(value) {
 }
 export function fakeName(value, key) {
   indexNames();
+  const words = value.match(/[\p{L}\p{M}]+/gu) ?? [];
+  const roles = new Map();
+  let nameSex;
+  if (
+    /\p{Script=Cyrillic}/u.test(value) &&
+    words.length >= 2 &&
+    words.length <= 3
+  ) {
+    const middle = words.findIndex((w) =>
+      /(?:ович|евич|ична|овна|евна)$/iu.test(w)
+    );
+    const surname = words.findIndex(
+      (w, i) =>
+        i !== middle &&
+        /(?:ов|ев|ин|ский|ова|ева|ина|ская)$/iu.test(w) &&
+        !given.has(latinName(w))
+    );
+    const genderWord = words[middle >= 0 ? middle : surname];
+    if (genderWord) {
+      nameSex = /(?:ова|ева|ина|ская|овна|евна|ична)$/iu.test(genderWord)
+        ? 'female'
+        : 'male';
+    }
+    if (middle > 0) {
+      words.forEach((w, i) =>
+        roles.set(
+          w,
+          i === middle
+            ? 'middleName'
+            : i === middle - 1
+              ? 'firstName'
+              : 'lastName'
+        )
+      );
+    } else if (surname >= 0) {
+      words.forEach((w, i) =>
+        roles.set(w, i === surname ? 'lastName' : 'firstName')
+      );
+    } else {
+      // Ambiguous spans receive a complete name with the same token count.
+      words.forEach((w, i) =>
+        roles.set(
+          w,
+          i === 0
+            ? 'firstName'
+            : i === words.length - 1
+              ? 'lastName'
+              : 'middleName'
+        )
+      );
+    }
+  }
   return value.replace(/[\p{L}\p{M}]+/gu, (word) => {
     for (const locale of ['ja', 'zh', 'ko']) {
       const family = firstNames[locale]
@@ -158,7 +210,7 @@ export function fakeName(value, key) {
         );
       }
     }
-    return fakeWord(word, key);
+    return fakeWord(word, key, roles.get(word), undefined, nameSex);
   });
 }
 function wordProfile(word, forcedRole, forcedLocale) {
@@ -188,12 +240,14 @@ function wordProfile(word, forcedRole, forcedLocale) {
         : 'lastName');
   return { canonical, locale, sex, role };
 }
-function fakeWord(word, key, forcedRole, forcedLocale) {
-  const { canonical, locale, sex, role } = wordProfile(
-    word,
-    forcedRole,
-    forcedLocale
-  );
+function fakeWord(word, key, forcedRole, forcedLocale, forcedSex) {
+  const {
+    canonical,
+    locale,
+    sex: detectedSex,
+    role,
+  } = wordProfile(word, forcedRole, forcedLocale);
+  const sex = forcedSex ?? detectedSex;
   const faker = new Faker({ locale: [locales[locale] ?? en, en] });
   faker.seed(
     createHmac('sha256', key)

package/src/index.d.ts

@@ -42,6 +42,8 @@ export interface Detector {
 export interface SanitizerOptions {
   /** Preserve all personal/account data; credential findings still redact. */
   credentialsOnly?: boolean;
+  /** Replace only knownPersonal/knownSecrets and reused map values; skip automatic detectors. */
+  knownOnly?: boolean;
   profile?: 'publication' | 'documents';
   /** Opt in to bounded employer, school, city/region heuristics. */
   documentEntities?: boolean;
@@ -97,6 +99,8 @@ export interface SanitizedText {
   findings: Finding[];
   /** Number of disjoint replacements after overlapping detections are merged. */
   redactions: number;
+  /** Placeholder-mode replacements absent from the restore map (e.g. credentials). */
+  irreversibleRedactions?: number;
   /** Private personal data. Absent for map:false and non-placeholder modes. */
   map?: RestoreMap;
 }

package/src/keywords.js

@@ -27,10 +27,27 @@ export function keywordMatcher(words) {
       queue.push(child);
     }
   }
+  // At the root, characters that cannot start any keyword need no state
+  // transitions. A native search skips large padding runs without Map work
+  // per character. Escape UTF-16 units so metacharacters retain literal meaning.
+  const starters = new RegExp(
+    `[${[...nodes[0].next.keys()]
+      .map((char) => `\\u${char.charCodeAt(0).toString(16).padStart(4, '0')}`)
+      .join('')}]`,
+    'g'
+  );
   return (text) => {
     const found = new Set();
     let state = 0;
     for (let i = 0; i < text.length; i++) {
+      if (!state && !nodes[0].output.length && !nodes[0].next.has(text[i])) {
+        starters.lastIndex = i + 1;
+        const next = starters.exec(text);
+        if (!next) {
+          break;
+        }
+        i = next.index;
+      }
       const char = text[i];
       while (state && !nodes[state].next.has(char)) {
         state = nodes[state].fail;

package/src/names.js

@@ -12,7 +12,7 @@ export const firstNames = {
   jaLatn: 'Yamada Sato Suzuki Takahashi Tanaka Ito Watanabe Nakamura Kobayashi',
   zhLatn:
     'Zhang Wang Li Zhao Liu Chen Yang Huang Zhou Wu Xu Sun Hu Zhu Gao Lin',
-  ru: 'Иван Пётр Петр Сергей Александр Алексей Дмитрий Михаил Николай Андрей Владимир Павел Виктор Олег Борис Евгений Василий Юрий Марина Анна Мария Елена Ольга Наталья Ирина Светлана Татьяна Екатерина Анастасия Юлия Ивану Ивана Иваном Сергея Сергею Александра Александру Анне Анну Елены',
+  ru: 'Данил Максим Алина Иван Пётр Петр Сергей Александр Алексей Дмитрий Михаил Николай Андрей Владимир Павел Виктор Олег Борис Евгений Василий Юрий Марина Анна Мария Елена Ольга Наталья Ирина Светлана Татьяна Екатерина Анастасия Юлия Ивану Ивана Иваном Сергея Сергею Александра Александру Анне Анну Елены',
   ja: '山田 佐藤 鈴木 高橋 田中 伊藤 渡辺 渡邊 中村 小林 加藤 吉田 松本 山本 井上 木村 林 清水 斎藤 斉藤',
   zh: '张 王 李 赵 刘 陈 杨 黄 周 吴 徐 孙 胡 朱 高 林 何 郭 马 罗 張 趙 劉 陳 楊 黃 吳 孫 馬 羅',
   ko: '김 이 박 최 정 강 조 윤 장 임 한 오 서 신 권 황 안 송 전 홍 유 고 문 양 손 배 백',

package/src/personal-variants.js

@@ -0,0 +1,94 @@
+import { escapePattern, failure } from './detection.js';
+
+const boundary = '[\\p{L}\\p{M}\\p{N}_]';
+export function organizationCore(value) {
+  return value
+    .replace(/^(?:ООО|АО|ПАО|ИП|ЗАО)[ \t]+/u, '')
+    .replace(/^[«“"]|[»”"]$/gu, '')
+    .trim();
+}
+// Only caller-confirmed Russian words receive finite case alternatives. A
+// minimum three-letter stem avoids turning short names into broad matches.
+function wordPattern(word) {
+  const endings = [
+    [/(?:ова|ева|ина)$/u, 1, '(?:а|ой|ою|у|е|ы)'],
+    [/(?:ая)$/u, 2, '(?:ая|ой|ою|ую)'],
+    [
+      /(?:ий|ый|ой)$/u,
+      2,
+      '(?:ий|ый|ой|ого|его|ому|ему|им|ым|ом|ем|ия|ию|ием|ии)',
+    ],
+    [/(?:ов|ев|ин)$/u, 0, '(?:а|у|ым|ом|е|ой)?'],
+    [/а$/u, 1, '(?:а|ы|и|е|у|ой|ою)'],
+    [/я$/u, 1, '(?:я|и|е|ю|ей|ею)'],
+    [/ь$/u, 1, '(?:ь|я|ю|ем|и)'],
+  ];
+  const lower = word.toLowerCase();
+  if (/^[а-яё]+$/u.test(lower)) {
+    for (const [ending, remove, suffix] of endings) {
+      if (ending.test(lower) && word.length - remove >= 3) {
+        return escapePattern(remove ? word.slice(0, -remove) : word) + suffix;
+      }
+    }
+  }
+  if (/^[а-яё]{4,}[бвгджзклмнпрстфхцчшщ]$/u.test(lower)) {
+    return `${escapePattern(word)}(?:а|у|ом|е|ы|ов|ам|ами|ах)?`;
+  }
+  return escapePattern(word);
+}
+export function personalPatterns(value, type) {
+  if (value.length > 256) {
+    // Large explicit values still support literal matching without expansion.
+    return [escapePattern(value)];
+  }
+  const values = new Set([value]);
+  if (type === 'ORGANIZATION') {
+    const core = organizationCore(value);
+    if (core.length >= 3) {
+      values.add(core);
+    }
+  }
+  const patterns = new Set();
+  for (const surface of values) {
+    const parts = surface.split(/([ \t]+)/u);
+    const pattern = parts
+      .map((part, i) => (i % 2 ? '[ \\t]+' : wordPattern(part)))
+      .join('');
+    patterns.add(pattern);
+    if (type === 'PERSON' && parts.length === 3) {
+      patterns.add(
+        [wordPattern(parts[2]), wordPattern(parts[0])].join('[ \\t]+')
+      );
+    }
+  }
+  return [...patterns];
+}
+export function personalRegex(value, type) {
+  return new RegExp(
+    `(?<!${boundary})(?:${personalPatterns(value, type).join('|')})(?!${boundary})`,
+    'giu'
+  );
+}
+export function samePersonal(value, original, type) {
+  return personalPatterns(original, type).some((pattern) =>
+    new RegExp(`^(?:${pattern})$`, 'iu').test(value)
+  );
+}
+export function detectKnownPersonal(text, entries, rule, emit, options) {
+  let count = 0;
+  for (const entry of entries ?? []) {
+    for (const match of text.matchAll(personalRegex(entry.value, entry.type))) {
+      if (++count > (options.maxFindings ?? 100000)) {
+        throw failure('ERR_LIMIT');
+      }
+      emit({
+        start: match.index,
+        end: match.index + match[0].length,
+        type: entry.type,
+        category: 'personal',
+        rule,
+        confidence: rule === 'known-personal' ? 0.85 : 0.9,
+      });
+    }
+  }
+}

package/src/personal.js

@@ -179,7 +179,12 @@ const russianFull =
   /(?<![\p{L}])(?:[А-ЯЁ][а-яё]{2,30}[ \t]+){2}[А-ЯЁ][а-яё]{2,30}(?:вич|вна|ична)|(?<![\p{L}])(?:[А-ЯЁ][а-яё]{2,30}[ \t]+){2}(?:оглы|кызы)(?![\p{L}])/gu;
 export function detectPersonal(text, emit) {
   detectDocumentNames(text, emit);
-  for (const pattern of [latinPairs, slavicPairs, russianFull]) {
+  for (const pattern of [
+    latinPairs,
+    slavicPairs,
+    russianFull,
+    /(?<![\p{L}])(?:[А-ЯЁ][а-яё]{2,30}[ \t]+)[А-ЯЁ][а-яё]{2,30}(?:вич|вна|ична)[ \t]+[А-ЯЁ][а-яё]{2,30}(?![\p{L}])/gu,
+  ]) {
     collectMatches(
       text,
       pattern,

package/src/placeholder-types.js

@@ -0,0 +1,33 @@
+import { catalogTypes } from './entities.js';
+
+// Native unlabelled types supplement the shared vendor-label catalog.
+export const reservedPlaceholderTypes = () => [
+  ...catalogTypes(),
+  'ADDRESS',
+  'AU_TFN',
+  'BRAZIL_CPF_NUMBER',
+  'CREDENTIAL',
+  'CREDIT_CARD',
+  'DATE',
+  'EDUCATION',
+  'EMAIL',
+  'ENCODED_SENSITIVE',
+  'IBAN',
+  'ID',
+  'IP_ADDRESS',
+  'KR_RRN',
+  'LOCATION',
+  'MAC_ADDRESS',
+  'ORGANIZATION',
+  'PASSPORT_MRZ',
+  'PASSPORT_NUMBER',
+  'PERSON',
+  'PHONE',
+  'PL_PESEL',
+  'RU_INN',
+  'RU_SNILS',
+  'SG_NRIC_FIN',
+  'STATE',
+  'USERNAME',
+  'US_HEALTHCARE_NPI',
+];

package/src/placeholders.js

@@ -1,4 +1,6 @@
 import { failure, REDACTED, escapePattern } from './detection.js';
+import { reservedPlaceholderTypes } from './placeholder-types.js';
+import { samePersonal } from './personal-variants.js';
 import { nameAliases } from './transliteration.js';
 import { createHash } from 'node:crypto';
 import { jsonStringLiterals } from './json-string-runs.js';
@@ -106,6 +108,13 @@ function aliases(value, type) {
   ];
 }
 function related(value, entry) {
+  if (
+    (entry.values ?? []).some((original) =>
+      samePersonal(value, original, entry.type)
+    )
+  ) {
+    return true;
+  }
   const candidates = aliases(value, entry.type);
   if (candidates.some((alias) => entry.aliases.includes(alias))) {
     return true;
@@ -139,7 +148,7 @@ function related(value, entry) {
   return false;
 }
 export function placeholderNameFindings(text, findings, options) {
-  if (!hasPlaceholders(options)) {
+  if (!hasPlaceholders(options) || options.knownOnly) {
     return findings;
   }
   const people = [
@@ -199,12 +208,23 @@ export function preparePlaceholderText(text, findings, options) {
   if (!state || !hasPlaceholders(options)) {
     return;
   }
-  // Reserve the entire namespace, not only the numbers about to be allocated.
-  const template = state.map.format;
+  const types = new Set([
+    ...reservedPlaceholderTypes(),
+    ...findings.map((f) =>
+      f.category === 'credential' ? 'CREDENTIAL' : f.type
+    ),
+    ...(options.knownPersonal ?? []).map((e) => e.type),
+    ...state.map.entries.map((e) => e.type),
+  ]);
+  const [before, after] = state.map.format.split('{n}');
+  const tokens = [...types].map(
+    (type) =>
+      `${escapePattern(
+        placeholderToken(type, 1, before)
+      )}[1-9][0-9]*${escapePattern(placeholderToken(type, 1, after))}`
+  );
   const pattern = new RegExp(
-    escapePattern(template)
-      .replace(/\\\{(?:Type|TYPE|type)\\\}/, '[A-Za-z][A-Za-z0-9_]*')
-      .replace('\\{n\\}', '[1-9][0-9]*'),
+    `(?<![\\p{L}\\p{M}\\p{N}_])(?:${tokens.join('|')})(?![\\p{L}\\p{M}\\p{N}_])`,
     'u'
   );
   if (namespaceCollision(text, pattern)) {
@@ -255,6 +275,15 @@ function canonicalPerson(value, people) {
   );
   return distinct.size === 1 ? [...distinct.values()][0] : value;
 }
+function canonicalKnown(value, type, options, state) {
+  const known = (options.knownPersonal ?? []).filter(
+    (e) => e.type === type && samePersonal(value, e.value, type)
+  );
+  if (known.length === 1) {
+    return known[0].value;
+  }
+  return type === 'PERSON' ? canonicalPerson(value, state.people ?? []) : value;
+}
 export function placeholderReplacement(value, finding, transform, options) {
   if (finding.category === 'credential' && !transform.credentials) {
     return REDACTED;
@@ -270,8 +299,7 @@ export function placeholderReplacement(value, finding, transform, options) {
       : finding.type === 'DATE_OF_BIRTH'
         ? 'DATE'
         : finding.type;
-  const canonical =
-    type === 'PERSON' ? canonicalPerson(value, state.people ?? []) : value;
+  const canonical = canonicalKnown(value, type, options, state);
   const candidates = map.entries.filter(
     (entry) => entry.type === type && related(canonical, entry)
   );

package/src/projection.js

@@ -27,8 +27,12 @@ export function projectText(input) {
     !/\x1b\[|\\u[\da-fA-F]{4}|[\u200b-\u200f\u202a-\u202e\u2060-\u206f\ufeff]/u.test(
       input
     ) &&
-    input.normalize('NFKC') === input &&
-    !/[^0-9]/.test(input.replace(/[^\p{Nd}]/gu, ''))
+    // ASCII already has normalized letters and digits. Avoid constructing a
+    // second multi-megabyte string just to check its decimal characters.
+    // eslint-disable-next-line no-control-regex -- Check the ASCII range.
+    (!/[^\x00-\x7f]/.test(input) ||
+      (input.normalize('NFKC') === input &&
+        !/[^0-9]/.test(input.replace(/[^\p{Nd}]/gu, ''))))
   ) {
     return { text: input, changed: false };
   }

package/src/propagation.js

@@ -1,5 +1,6 @@
 import { personalPattern, maxPropagationLength, failure } from './detection.js';
 import { actionable } from './confidence.js';
+import { organizationCore } from './personal-variants.js';
 import { nameAliases } from './transliteration.js';
 
 function mrzSeeds(value, finding, seeds) {
@@ -32,11 +33,13 @@ function decodedValue(value) {
 
 function eligibleSeed(finding, options) {
   return (
+    !options.knownOnly &&
     finding.category === 'personal' &&
     finding.type !== 'ENCODED_SENSITIVE' &&
     !['confirmed-personal', 'document-propagation'].includes(finding.rule) &&
     actionable(finding, options) &&
-    (finding.confidence >= 0.85 ||
+    ((options.documentEntities && finding.type === 'ORGANIZATION') ||
+      finding.confidence >= 0.85 ||
       [
         'PERSON',
         'PASSPORT_NUMBER',
@@ -47,6 +50,14 @@ function eligibleSeed(finding, options) {
       ].includes(finding.type))
   );
 }
+function organizationSeed(value, finding, seeds, options) {
+  if (options.documentEntities && finding.type === 'ORGANIZATION') {
+    const core = organizationCore(value);
+    if (core.length >= 3) {
+      seeds.set(core, finding);
+    }
+  }
+}
 // Only confirmed values seed propagation. A low-score numeric coincidence
 // must never turn every log occurrence into a high-score identity.
 export function confirmedSeeds(text, findings, options) {
@@ -71,6 +82,7 @@ export function confirmedSeeds(text, findings, options) {
       mrzSeeds(value, finding, seeds);
     } else if (value.length >= 2) {
       seeds.set(value, finding);
+      organizationSeed(value, finding, seeds, options);
       if (finding.type === 'PERSON') {
         for (const name of value.match(/\p{L}{2,}/gu) ?? []) {
           seeds.set(name, finding);

package/src/sanitizer.js

@@ -45,6 +45,7 @@ const publicTypes = new Set([
 ]);
 const optionNames = new Set([
   'credentialsOnly',
+  'knownOnly',
   'knownSecrets',
   'knownPersonal',
   'confirmedPersonal',
@@ -166,6 +167,7 @@ export function validateOptions(options = {}) {
   for (const key of [
     'paranoid',
     'credentialsOnly',
+    'knownOnly',
     'decode',
     'secretlint',
     'publicKnowledge',
@@ -695,6 +697,7 @@ export function redactResult(
   const parts = [];
   const faked = [];
   const occurrences = [];
+  let irreversibleRedactions = 0;
   let outputLength = 0;
   let cursor = 0;
   for (const span of spans) {
@@ -723,6 +726,12 @@ export function redactResult(
         raw: text.slice(span.start, span.end),
       });
     }
+    if (
+      hasPlaceholders(options) &&
+      !retainedReplacement(transform, 'placeholder', span, replacement)
+    ) {
+      irreversibleRedactions++;
+    }
     outputLength += replacement.length;
     if (retainedReplacement(transform, 'fake', span, replacement)) {
       faked.push(span);
@@ -731,7 +740,11 @@ export function redactResult(
   }
   parts.push(text.slice(cursor));
   const result = finishPlaceholders(
-    { text: parts.join(''), redactions: spans.length },
+    {
+      text: parts.join(''),
+      redactions: spans.length,
+      ...(hasPlaceholders(options) ? { irreversibleRedactions } : {}),
+    },
     occurrences,
     options
   );
@@ -861,6 +874,13 @@ export function redactAgain(result, findings, options, round, rerun) {
   const merge = (next) => ({
     ...next,
     redactions: result.redactions + next.redactions,
+    ...(hasPlaceholders(options)
+      ? {
+          irreversibleRedactions:
+            (result.irreversibleRedactions ?? 0) +
+            (next.irreversibleRedactions ?? 0),
+        }
+      : {}),
     findings: auditFaked(findings, options, result),
   });
   const next = rerun({ ...options, findings: [] }, round + 1);