@link-foundation/sensitive-data-sanitizer 0.13.0 → 0.15.0

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.15.0
+
+### Minor Changes
+
+- 7e977ac: Match known Russian names in all six cases and both word orders, with `ё`/`е` variants, initials and surname-plus-given forms. Give all case forms of one known person, organization, school or location one declined fake identity with natural names and an optional `pools` override. Keep `@handle/email` and `Name-99` organization spans whole so placeholders stay restorable and are never glued to digits. Detect employers in CV experience lines of the `documents` profile.
+
+## 0.14.0
+
+### Minor Changes
+
+- d5b2aac: Add known-only reversible sanitization, Unicode known-value boundaries and bounded Russian entity case matching. Correct placeholder namespace collisions, Russian document entities and fake-name roles, and credential prose false positives. Report irreversible replacements in placeholder results.
+
 ## 0.13.0
 
 ### Minor Changes

package/README.md

@@ -55,6 +55,15 @@ sensitive-data-sanitizer restore safe.txt --map private.lino --output restored.t
 sensitive-data-sanitizer redact cv.txt --profile documents --placeholder --no-map
 ```
 
+`knownOnly: true` (CLI `--known-only`) limits replacement to `knownPersonal`,
+`knownSecrets` and reused map identities, preserving fictional examples.
+Placeholder results report `irreversibleRedactions`; a nonzero count means some
+replacements, such as credentials excluded from the map, cannot be restored.
+
+Known Russian names match in all six cases and in any word order, with `ё`/`е`
+spelling variants and initials (`с Петровым Данилом Сергеевичем`, `Петров Д. С.`).
+A part of a longer name is matched alone only when it is listed itself.
+
 For irreversible output, choose `transformation: { mode: 'placeholder', map:
 false }`: no restore map is returned or written. Detection limits still apply
 in either mode. See the [placeholder API](docs/API.md#reversible-placeholders),
@@ -139,7 +148,9 @@ See [the API](docs/API.md) and [all issue 13 requirements and research](docs/cas
 `transformation: {mode: 'fake', key}` generates deterministic names, valid
 dates/MRZs/checksummed IDs and reserved email/domain destinations. Credentials
 remain fully redacted by default. Keep the key private: this preserves relations
-and structure and is pseudonymisation. See [realistic fakes](examples/realistic-fakes.mjs)
+and structure and is pseudonymisation. Russian mentions of one known person or
+entity share one fake identity declined to match the source (`Петров … С Петровым`
+becomes `Баранов … С Барановым`), and `pools` overrides the entity names. See [realistic fakes](examples/realistic-fakes.mjs)
 and [the complete issue 24 requirements, research and solution plans](docs/case-studies/issue-24/REQUIREMENTS.md).
 
 Keep policies containing real private values outside Git. Built-in exact public knowledge and narrowly scoped role/domain heuristics are enabled offline. Explicit private literals override public exemptions. Keep custom public policies reviewed against evidence; `--verify-public` separately opts into Wikidata name lookup:

package/bin/sensitive-data-sanitizer.js

@@ -42,6 +42,7 @@ history rewrite SOURCE --output DIRECTORY [--apply]
 --in-place          Atomically replace the input file (redact only)
 --native-only       Disable Secretlint; selected local detectors remain required
 --credentials-only  Preserve personal/account details; redact credentials only
+--known-only        Replace only knownPersonal/knownSecrets and reused map values
 --env-secrets       Include values of all credential-named environment variables
 --gh-config         Include tokens from the local gh hosts.yml credential file
 --paranoid          Add entropy detection (can redact innocent identifiers)
@@ -88,6 +89,7 @@ function parseArgs(argv) {
     ['--in-place', 'inPlace'],
     ['--native-only', 'nativeOnly'],
     ['--credentials-only', 'credentialsOnly'],
+    ['--known-only', 'knownOnly'],
     ['--env-secrets', 'envSecrets'],
     ['--gh-config', 'ghConfig'],
     ['--paranoid', 'paranoid'],
@@ -339,6 +341,9 @@ async function loadOptions(config) {
 
 async function featureOptions(config, options) {
   await localCredentials(config, options);
+  if (config.knownOnly) {
+    options.knownOnly = true;
+  }
   if (config.credentialsOnly) {
     options.credentialsOnly = true;
   }

package/docs/API.md

@@ -19,6 +19,24 @@ Import the ESM root entry; complete declarations are in `src/index.d.ts`. Inputs
 
 ## Options
 
+### Replace only known values
+
+Use `knownOnly: true` to preserve fictional examples while replacing selected
+private values. This applies to sync/async, JSON/JSONL, streams and fake mode.
+It skips automatic credential detection too; supply `knownSecrets` explicitly
+for credentials that must be removed. `credentialsOnly` can further exclude
+personal values. CLI: `--known-only` with a `--config` containing the known lists.
+
+```js
+const result = sanitize('Анна Смирнова; example Jane Doe', {
+  knownOnly: true,
+  knownPersonal: [{ type: 'PERSON', value: 'Анна Смирнова' }],
+  transformation: { mode: 'placeholder' },
+});
+// result.text: 'Person-1; example Jane Doe'
+restore(result.text, result.map); // exact original
+```
+
 ### Reversible placeholders
 
 `transformation: {mode: 'placeholder'}` emits typed tokens such as `Person-1`,
@@ -42,7 +60,9 @@ are left alone.
 `format` is a template with exactly one type field (`{Type}`, `{TYPE}` or
 `{type}`) and one `{n}` field. Examples: `{Type}-{n}`, `[{TYPE}_{n}]`,
 `{{{type}_{n}}}`. Sanitization refuses input containing any token in the chosen
-namespace, including numbers not allocated yet. The map's format cannot change
+namespace for built-in and supplied type labels, including numbers not allocated yet.
+Unicode letters, marks, digits and underscores bound those tokens. Ordinary
+technical names such as GPT-4, ISO-9001 and Windows-10 are not reserved. The map's format cannot change
 on reuse. A malformed map or ambiguous exact restoration blocks output.
 
 `createRestoreMap({format?})` creates an empty reusable map. Pass a map as
@@ -68,15 +88,32 @@ is plaintext personal data, not encrypted storage; protect it as the original.
 
 `credentials: true` explicitly opts credentials into `Credential-n` entries.
 Otherwise they remain `[REDACTED]` and cannot be recovered, including through
-occurrence metadata. `map: false` returns only the anonymized text and ordinary
+occurrence metadata. Placeholder results include `irreversibleRedactions`, the
+number of replacements excluded from the restore map (including mixed policies).
+Check that this is zero before relying on exact restoration. A positive count
+means `restore()` cannot reconstruct all of the input. `map: false` returns only the anonymized text and ordinary
 value-free finding metadata. The temporary numbering registry is discarded
 after an operation (or the complete stream); no map file is created. Residual verification
 still runs, including every required engine for `createSanitizer`.
 
 Name components, unique initial-plus-surname forms, existing transliteration
 aliases and selected Russian `-ов`/`-ев` inflections share an entity token.
+Explicit `knownPersonal` and reused map entries also match bounded Russian noun,
+adjective, feminine surname and given-name case endings, two-part person names
+in reverse order, and organization cores without legal forms/quotes. Inflected
+forms consume whole Unicode words; minimum stem lengths limit overmatching.
 This is conservative heuristic identity linking, not general coreference:
 ambiguous surnames/initials and unrelated people are not intentionally merged.
+
+A listed Russian PERSON value also matches all six grammatical cases, including
+the instrumental (`Петровым Данилом Сергеевичем`), every word order, `ё`/`е`
+spelling differences in either direction, upper case, and initials with or
+without spaces in both positions and all cases (`Петрову Д. С.`, `Д.С. Петрова`).
+The contiguous name parts of a listed three-part name match together
+(`Петров Данил`, `Данил Сергеевич`, `с Петровым Данилом`). A part on its own does
+not: a bare `Данил` or `Сергеевич` is too ambiguous to imply from a longer value,
+so list each part you want matched alone as its own `knownPersonal` entry. Listing
+parts next to the full name keeps one token and one fake identity.
 Every replaced occurrence retains its exact original surface privately. With
 one identical token, a derived answer cannot encode the original grammatical
 case; it receives the canonical spelling. Two distinct originals that produce
@@ -99,28 +136,29 @@ derived model answer. Input, output and map paths must be separate. Maps are
 saved separately before stdout publication. `--profile documents` enables the
 new document recognizers; `documentEntities: false` turns them off explicitly.
 
-| Option             | Default / meaning                                                                                       |
-| ------------------ | ------------------------------------------------------------------------------------------------------- |
-| `documentEntities` | Off; bounded organization, education and location rules; enabled by the documents profile               |
-| `profile`          | Off; `'publication'` sets a 0.3 threshold and identity masking; `'documents'` enables document entities |
-| `knownSecrets`     | `[]`; exact literal credential values, regardless of length or format                                   |
-| `knownPersonal`    | `[]`; `{type, value}` literals with Unicode case-insensitive matching                                   |
-| `publicEntities`   | `[]`; exact PERSON/ORGANIZATION/EDUCATION/LOCATION/EMAIL exception with HTTPS evidence and review date  |
-| `findings`         | `[]`; validated detections for the original input, supplied by the caller                               |
-| `detectors`        | `[]`; additional required callbacks in the asynchronous engine                                          |
-| `secretlint`       | `true` in the asynchronous engine                                                                       |
-| `decode`           | `true`; bounded base64/base64url, hex and percent decoding                                              |
-| `paranoid`         | `false`; entropy-based supplemental credential heuristic                                                |
-| `maxInputLength`   | 33,554,432 UTF-16 units                                                                                 |
-| `maxFindings`      | 100,000 detector hits before deduplication                                                              |
-| `minConfidence`    | 0.5; personal findings below this score remain visible to inspect but are not replaced                  |
-| `threshold`        | Alias for minConfidence; both options must agree when supplied                                          |
-| `debug`            | Optional trusted callback for value-free counts, timing and event metadata                              |
-| `identityMask`     | `false`; identity types keep 2+2 Unicode characters with fixed `***`, or redact below five              |
-| `fakeIdentity`     | Off; `'specimen-and-synthetic'` retains explicitly fake identity values                                 |
-| `fakeValues`       | `[]`; exact caller-reviewed identity values to retain, regardless of fakeIdentity                       |
-| `structured`       | Off; `'json'` or `'jsonl'` validates and sanitizes string keys/values separately                        |
-| `structuralFields` | `[]`; declared property names suppress weak name/ID/entropy heuristics in their string values           |
+| Option             | Default / meaning                                                                                                                                           |
+| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `documentEntities` | Off; bounded organization, education and location rules; enabled by the documents profile                                                                   |
+| `profile`          | Off; `'publication'` sets a 0.3 threshold and identity masking; `'documents'` enables document entities                                                     |
+| `knownOnly`        | `false`; disable all automatic detection (including credentials, Secretlint and custom detectors); use only explicit known values and reused map identities |
+| `knownSecrets`     | `[]`; exact literal credential values, regardless of length or format                                                                                       |
+| `knownPersonal`    | `[]`; `{type, value}` values with whole-word Unicode case-insensitive matching and bounded Russian forms; list parts to match them alone                    |
+| `publicEntities`   | `[]`; exact PERSON/ORGANIZATION/EDUCATION/LOCATION/EMAIL exception with HTTPS evidence and review date                                                      |
+| `findings`         | `[]`; validated detections for the original input, supplied by the caller                                                                                   |
+| `detectors`        | `[]`; additional required callbacks in the asynchronous engine                                                                                              |
+| `secretlint`       | `true` in the asynchronous engine                                                                                                                           |
+| `decode`           | `true`; bounded base64/base64url, hex and percent decoding                                                                                                  |
+| `paranoid`         | `false`; entropy-based supplemental credential heuristic                                                                                                    |
+| `maxInputLength`   | 33,554,432 UTF-16 units                                                                                                                                     |
+| `maxFindings`      | 100,000 detector hits before deduplication                                                                                                                  |
+| `minConfidence`    | 0.5; personal findings below this score remain visible to inspect but are not replaced                                                                      |
+| `threshold`        | Alias for minConfidence; both options must agree when supplied                                                                                              |
+| `debug`            | Optional trusted callback for value-free counts, timing and event metadata                                                                                  |
+| `identityMask`     | `false`; identity types keep 2+2 Unicode characters with fixed `***`, or redact below five                                                                  |
+| `fakeIdentity`     | Off; `'specimen-and-synthetic'` retains explicitly fake identity values                                                                                     |
+| `fakeValues`       | `[]`; exact caller-reviewed identity values to retain, regardless of fakeIdentity                                                                           |
+| `structured`       | Off; `'json'` or `'jsonl'` validates and sanitizes string keys/values separately                                                                            |
+| `structuralFields` | `[]`; declared property names suppress weak name/ID/entropy heuristics in their string values                                                               |
 
 Unknown options and invalid limits are errors. `knownSecretsFromEnv(env, names?)` discovers credential-named variables (excluding auth configuration names) when names are omitted, or uses an explicit selection, deduplicating nonempty values. It never invokes shell commands or reads auth files. `personalVariants(entries)` generates numeric and ISO/day-first date spellings; name case variants are already handled by detection. Neither helper invents transliterations.
 
@@ -177,6 +215,8 @@ All keyed modes require a key of at least 16 UTF-8 bytes. `format-preserving` is
 
 `{mode: 'fake', key}` infers name locale, script and gender from the supported gazetteer/datasets and recognizable surname/patronymic endings. Independent HMAC-seeded Faker instances make results independent of processing order. Russian Cyrillic/transliterated spelling, name order and filename separators share component identities. Ambiguous names can use a fallback locale; this is not universal name/gender inference. Dates use one keyed backward offset of 1–365 days and preserve ISO, slash and dot formats. Complete TD1/TD2/TD3 MRZs regenerate individual/composite checks and match separately faked name/date fields. Issuer/nationality remain original unless `mrzCountry: 'UTO'` is supplied. Unsupported/damaged formats fall back to full redaction.
 
+Russian mentions of one person, organization, school or location share one fake identity. The fake follows the source word order, initials, capitalization, grammatical case and, for people, sex: `Петров пришёл. Позвонили Петрову. С Петровым.` becomes `Баранов пришёл. Позвонили Баранову. С Барановым.`, and `ООО «Ромашка» … в Ромашке` becomes `ООО «Импульс» … в Импульсе`. Declension uses finite offline rules for the generated vocabulary. The grammatical case comes from the source surface and the preceding preposition, so a source preposition stays as written even where the fake would call for another spelling (`о Осипове`). Organizations, schools and locations draw from an authored Russian or Latin vocabulary according to the script of the original, skip any name that is itself private in the text, and take a numeric suffix (`Заря-2`) once the vocabulary is exhausted. Each transformation object remembers its entity names so that two entities never share one, which makes a name depend on first appearance when two entities start on the same vocabulary entry; reuse the transformation object across documents to keep names stable, and note that this memory is not shared across worker threads or processes. `pools: {ORGANIZATION: [...], LOCATION: [...], EDUCATION: [...]}` on the fake transformation replaces a vocabulary with caller-reviewed non-empty names. Autodetected mentions are linked as in placeholder mode: a mention that appears only in oblique forms can remain a separate entity unless the value is listed in `knownPersonal`.
+
 Implemented national checksums, IBANs and payment-card Luhn checks remain valid. Phones retain calling code, length and separators; NANP uses 555-0100–0199 and UK mobile uses 07700 900000–900999. Email/domain outputs use example.com/.test. Other national identities and phone numbers have no universal reserved range: generated arithmetic does not prove that a value is unassigned.
 
 Generated replacements carry `faked: true`. A private bounded registry recognizes values produced by the same transformation context as `kept: 'fake'` during later inspection/verification. Caller-supplied flags and arbitrary FAKE-looking strings grant no exemption. A fresh sanitizer does not inherit that registry; reviewed persisted fake identity values can be configured through `fakeValues`. Credentials remain fully redacted unless `credentials: true` requests provider-shaped values containing FAKE and punctuation that violates token grammar. Generated personal values cannot exempt credential spans.

package/docs/COVERAGE.md

@@ -29,22 +29,43 @@ remains available under those profiles.
 | `LOCATION`     | Exact city/region entries from a bounded offline vocabulary                                                                           | 0.60       |
 | `LOCATION`     | Bounded proper names after location labels, including based in, located in, relocated from, г., город and locale equivalents          | 0.75       |
 
+Russian document rules also accept legal-form quoted names with lowercase words,
+digits and hyphens (up to 160 characters), organization Group/Групп/Holding/Холдинг
+markers, adjectival institution names with lowercase connectors and optional
+`имени` tails, education acronyms in context, residence labels and city/country
+phrasing. A known given name after `Меня зовут` or `С уважением` scores 0.70.
+Detected organization cores propagate within a document and across confirmed
+stream records. Russian regional cities include Ярославль and Кострома; the
+vocabulary remains finite. Default profiles do not enable these document rules.
+
 These confidence values indicate heuristic strength, not calibrated
 probabilities. `minConfidence` still controls replacement. Findings retain
 their confidence even below threshold. `publicEntities` supports exact reviewed
 exceptions for all three types; `knownPersonal` always forces removal of a
 listed private value. All types support placeholder mode; fake mode uses stable,
-visibly synthetic organization/school/location labels.
+plausible organization/school/location names from an authored vocabulary or
+caller-supplied `pools`, declined to match Russian source mentions. These names
+are plausible and can coincide with real entities that the text does not mention.
 
 `src/document-vocabulary.js` is the complete authored vocabulary: English,
 Spanish, French, German, Portuguese, Russian, Japanese, Simplified/Traditional
 Chinese, Korean, Arabic, Hebrew, Hindi, Turkish, Indonesian, Vietnamese and
 Thai each have list-form employer, school and city fixtures. Proper-name
 rules are finite (up to six capitalized words, or four words in scripts without
-case); they do not implement grammatical parsing. Common lowercase connectors,
+case); they do not implement grammatical parsing. Other lowercase connectors,
 unusual typography, unsupported legal forms and unfamiliar institutions can be
 missed. Legal-form words and uncased scripts can also yield false positives.
 
+The `documents` profile also reads employers from CV experience lines anchored on
+a date range: `Title, Employer (2019-05 – 2021-03)`, `Title — Employer, 2019–2021`
+and `Title (Employer), 2019–2021`. Ranges use years, `YYYY-MM`, `MM.YYYY`, full
+dates or `month YYYY`, with `по настоящее время`, `н.в.`, `present` and similar
+open ends. Confidence is 0.60, and the employer then propagates through the
+document like other organizations. The default profile ignores these lines. Lines
+with more fields (`Title, Employer, City (dates)`) and titles containing a comma
+are not targeted and can leave the employer in place; date-anchored prose with
+the same shape can be flagged.
+
 The gazetteer has only the cities/regions explicitly listed in that module,
 including Springfield and Riverton. It is not a worldwide geocoder. Location
 labels can find unlisted proper names; unlabelled places outside the vocabulary

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-foundation/sensitive-data-sanitizer",
-  "version": "0.13.0",
+  "version": "0.15.0",
   "description": "Offline sensitive-text sanitization with credential, multilingual personal-data and scanner union detection",
   "type": "module",
   "main": "src/index.js",

package/src/context.js

@@ -8,6 +8,7 @@ import { plausiblePerson } from './names.js';
 // unrelated words such as passport. Verb forms cover the supported locales.
 export const credentialVerbs = [
   'is',
+  'are',
   'was',
   'equals',
   'es',
@@ -45,12 +46,11 @@ export const credentialVerbs = [
   'คือ',
 ];
 const verbs = credentialVerbs.map(escapePattern).join('|');
-const key = `(?:[\\p{L}\\p{N}]+[_.-])*(?:auth|authorization)(?:[_.-][\\p{L}\\p{N}]+)*|[\\p{L}\\p{N}_.-]{0,32}(?:${labels.credential
+const label = labels.credential
   .filter((word) => !['auth', 'authorization'].includes(word))
   .map(escapePattern)
-  .join(
-    '|'
-  )})[\\p{L}\\p{N}_.-]{0,32}|(?:[\\p{L}\\p{N}_.-]{1,32}[_-])?(?:pass|pw)(?:[_-][\\p{L}\\p{N}_.-]{1,32})?`;
+  .join('|');
+const key = `(?:[\\p{L}\\p{N}]+[_.-])*(?:auth|authorization|PGPASSWORD|${label}|pass|pw)(?:[_.-][\\p{L}\\p{N}]+)*`;
 // A qualifier such as "for staging" stays on one logical line: it never
 // crosses an escaped line break, an escape sequence or an enclosing quote.
 // Verbs are whole words, so "request" does not end in the verb "est".
@@ -60,8 +60,6 @@ const credentialPrefix = new RegExp(
 );
 // Bare generic labels are common English nouns. After a verb they need a
 // value that looks generated; "the token is valid" is prose, not a secret.
-const genericLabel =
-  /^(?:token|sig|sas|secret|auth|authorization|cookie|секрет|токен)$/iu;
 function generatedWord(text, range) {
   const word = /^[^\s]+/u
     .exec(text.slice(range.start, range.end))?.[0]
@@ -170,7 +168,7 @@ export function authorizationRange(text, start) {
   for (; end < text.length; end++) {
     const char = text[end];
     const code = text.charCodeAt(end);
-    if (code < 32 || code === 127 || /[\s"'\\,]/.test(char)) {
+    if (authorizationDelimiter(char, code)) {
       break;
     }
     if (char === '<' && end !== tokenStart && !brackets.length) {
@@ -185,7 +183,13 @@ export function authorizationRange(text, start) {
     }
   }
   const token = text.slice(tokenStart, end);
-  return !token || credentialPlaceholder(token) ? undefined : { start, end };
+  return usableAuthorization(token) ? { start, end } : undefined;
+}
+function usableAuthorization(token) {
+  return Boolean(token) && !credentialPlaceholder(token);
+}
+function authorizationDelimiter(char, code) {
+  return code < 32 || code === 127 || /[\s"'\\,]/.test(char);
 }
 function quotedRange(text, start, opening, maxLength) {
   const depth = opening[1].length,
@@ -323,6 +327,13 @@ function backslashes(text, index) {
   return count;
 }
 
+function sourceReference(label, tail) {
+  return (
+    /\.(?:[cm]?[jt]sx?|py|go|rs|c|cpp|h|java|cs|rb|sh)$/iu.test(label) &&
+    /^\d+(?:[):]|\b)/u.test(tail)
+  );
+}
+
 function detectCredentials(text, emit) {
   const details = { category: 'credential', type: 'SECRET', rule: 'context' };
   for (const match of text.matchAll(credentialPrefix)) {
@@ -341,9 +352,12 @@ function detectCredentials(text, emit) {
     let range = /^(?:proxy-)?authorization$/i.test(match[1])
       ? authorizationRange(text, start)
       : valueRange(text, start, mode);
-    if (range && mode === 'prose' && genericLabel.test(match[1])) {
+    if (range && mode === 'prose') {
       range = generatedWord(text, range);
     }
+    if (sourceReference(match[1], text.slice(start))) {
+      continue;
+    }
     if (range) {
       emitValue(text, range.start, range.end, emit, details, match[1]);
     }

package/src/detection.js

@@ -13,6 +13,7 @@ import {
   credentialPlaceholder,
 } from './credentials.js';
 import { localAddress } from './public.js';
+import { detectKnownPersonal } from './personal-variants.js';
 import { detectDocumentEntities } from './document-entities.js';
 
 export const REDACTED = '[REDACTED]';
@@ -133,6 +134,29 @@ function detectCookies(text, emit) {
 
 export function nativeDetect(text, options, emit) {
   const credential = { type: 'SECRET', category: 'credential' };
+  for (const value of options.knownSecrets ?? []) {
+    collectMatches(text, new RegExp(escapePattern(value), 'g'), emit, {
+      ...credential,
+      rule: 'known-secret',
+    });
+  }
+  detectKnownPersonal(
+    text,
+    options.knownPersonal,
+    'known-personal',
+    emit,
+    options
+  );
+  detectKnownPersonal(
+    text,
+    options.confirmedPersonal,
+    'confirmed-personal',
+    emit,
+    options
+  );
+  if (options.knownOnly) {
+    return;
+  }
   detectGitleaks(
     text,
     emit,
@@ -143,31 +167,6 @@ export function nativeDetect(text, options, emit) {
   for (const [rule, pattern] of serviceRules) {
     collectMatches(text, pattern, emit, { ...credential, rule });
   }
-  for (const value of options.knownSecrets ?? []) {
-    collectMatches(text, new RegExp(escapePattern(value), 'g'), emit, {
-      ...credential,
-      rule: 'known-secret',
-    });
-  }
-  for (const entry of options.knownPersonal ?? []) {
-    collectMatches(text, new RegExp(escapePattern(entry.value), 'giu'), emit, {
-      type: entry.type,
-      category: 'personal',
-      rule: 'known-personal',
-    });
-  }
-  for (const entry of options.confirmedPersonal ?? []) {
-    const pattern = new RegExp(
-      `(?<![\\p{L}\\p{N}])${personalPattern(entry.value)}(?![\\p{L}\\p{N}])`,
-      'giu'
-    );
-    collectMatches(text, pattern, emit, {
-      type: entry.type,
-      category: 'personal',
-      rule: 'confirmed-personal',
-      confidence: 0.9,
-    });
-  }
   detectContext(text, emit, options.credentialsOnly);
   if (!options.credentialsOnly) {
     detectPersonal(text, emit);
@@ -241,11 +240,29 @@ function detectAuthorization(text, emit, credential) {
   }
 }
 
+// `/` and `|` are valid local-part characters, so `@handle/jane@example.com`
+// reads as one address that starts after `@`. A mention keeps its own finding;
+// any other leading `@` joins the address so that it cannot survive as a handle.
+function emailAfterMention(text, finding) {
+  if (
+    text[finding.start - 1] !== '@' ||
+    /[\p{L}\p{N}_.\]]/u.test(text[finding.start - 2] ?? '')
+  ) {
+    return finding;
+  }
+  const handle = /^[\p{L}\p{N}_]{2,32}[/|]/u.exec(
+    text.slice(finding.start, finding.end)
+  );
+  return handle
+    ? { ...finding, start: finding.start + handle[0].length }
+    : { ...finding, start: finding.start - 1 };
+}
+
 function detectPersonalFormats(text, options, emit) {
   collectMatches(
     text,
     /(?<![\p{L}\p{N}_.!#$%&'*+/?^`{|}~-])[\p{L}\p{N}_.!#$%&'*+/?^`{|}~-]+@[\p{L}\p{N}](?:[\p{L}\p{N}.-]{0,252})\.[\p{L}]{2,63}(?![\p{L}\p{N}_-])/gu,
-    emit,
+    (finding) => emit(emailAfterMention(text, finding)),
     { type: 'EMAIL', category: 'personal', rule: 'email' }
   );
   collectMatches(

package/src/document-entities.js

@@ -1,5 +1,8 @@
 import { collectMatches, escapePattern } from './detection.js';
+import { firstNames } from './names.js';
+import { organizationCore, personalRegex } from './personal-variants.js';
 import { documentVocabulary } from './document-vocabulary.js';
+import { months } from './personal-context.js';
 
 const words = (column) =>
   [...new Set(documentVocabulary.flatMap((row) => row[column].split('|')))]
@@ -14,7 +17,7 @@ const latinName = `${proper}(?:[ \\t]+${proper}){0,5}`;
 const uncased =
   '[\\p{Script=Han}\\p{Script=Hiragana}\\p{Script=Katakana}\\p{Script=Hangul}\\p{Script=Arabic}\\p{Script=Hebrew}\\p{Script=Devanagari}\\p{Script=Thai}\\p{M}]{2,48}';
 const name = `(?:${latinName}|${uncased}(?:[ \\t]+${uncased}){0,3})`;
-const quoted = `(?:[«“「『"]${name}[»”」』"]|${name})`;
+const quoted = `(?:«[^«»\\r\\n]{1,160}»|“[^“”\\r\\n]{1,160}”|"[^"\\r\\n]{1,160}"|「[^「」\\r\\n]{1,160}」|『[^『』\\r\\n]{1,160}』|${name})`;
 const boundary = '[\\p{L}\\p{M}\\p{N}_]';
 const organizations = new RegExp(
   `(?<!${boundary})(?:(?:${legal})[ \\t]*${quoted}|${quoted}[ \\t]*(?:${legal}))(?!${boundary})`,
@@ -33,22 +36,104 @@ const labelledPlace = new RegExp(
   'gu'
 );
 
+const ruWord = '[А-ЯЁа-яё][А-ЯЁа-яё-]{1,47}';
+const ruSchool = new RegExp(
+  `(?<!${boundary})[А-ЯЁ][а-яё]{2,30}(?:-[А-ЯЁа-яё][а-яё]{1,30})?(?:[ \t]+[а-яё-]{2,40}){0,4}[ \t]+(?:университет|институт|колледж|школа|академия)(?:[ \t]+имени[ \t]+${ruWord}(?:[ \t]+[А-ЯЁ][а-яё-]{2,40}){0,2})?(?!${boundary})`,
+  'gu'
+);
+const ruAcronym =
+  /(?<![\p{L}\p{N}_])(?:НИУ[ \t]+[А-ЯЁ]{2,12}|(?:МГУ|СПбГУ|МГТУ|РГГУ|ВШЭ)(?=[, \t]+(?:магистратура|бакалавриат|университет|обучение)))(?![\p{L}\p{N}_])/gu;
+const ruResidence =
+  /(?:[Гг]ород|[Пп]роживаю|[Пп]роживание|[Мм]есто жительства)[ \t]*:[ \t]*([А-ЯЁ][а-яё-]{2,47})|([А-ЯЁ][а-яё-]{2,47})(?=,[ \t]*(?:готов[а]?[ \t]+к[ \t]+переезду|Россия|Беларусь|Казахстан|Украина))/gu;
+const ruIntroduction = new RegExp(
+  `(?:[Мм]еня зовут|[Сс] уважением)[, \t]+(${firstNames.ru.split(' ').join('|')})(?!${boundary})`,
+  'gu'
+);
+const groupOrganization = new RegExp(
+  `(?<!${boundary})${latinName}[ \t]+(?:Групп|Group|Холдинг|Holding)(?!${boundary})`,
+  'gu'
+);
+// CV experience lines: the date range anchors the employer slot, so prose with
+// a comma is untouched. Titles are not required to be lowercase because English
+// ones are often title case (`Senior QA Engineer`).
+const capitalized = (word) => word[0].toUpperCase() + word.slice(1);
+const month = [
+  ...new Set(months.split('|').flatMap((word) => [word, capitalized(word)])),
+].join('|');
+const rangeStart = `(?:(?:19|20)\\d{2}(?:-\\d{2}){0,2}|\\d{2}[./](?:\\d{2}\\.)?\\d{4}|(?:${month})\\.?[ \\t]+(?:19|20)\\d{2})`;
+const rangeOpen = ['по настоящее время', 'настоящее время', 'сейчас', 'н.в.']
+  .concat(['н/в', 'present', 'now', 'current'])
+  .flatMap((word) => [word, capitalized(word)])
+  .map(escapePattern)
+  .join('|');
+const range = `${rangeStart}(?:[ \\t]*[-–—][ \\t]*|[ \\t]+(?:по|to)[ \\t]+)(?:${rangeStart}|${rangeOpen})(?!${boundary})`;
+const titleWord = '[\\p{L}\\p{N}][\\p{L}\\p{N}/+#.&-]{0,39}';
+const lineTitle = `^[ \\t]*(?:[-–—•*·][ \\t]+|\\d{1,2}[.)][ \\t]+)?${titleWord}(?:[ \\t]+${titleWord}){0,5}`;
+const employerWord = "[\\p{L}\\p{M}\\p{N}&'’.+-]{1,40}";
+const employer = `«[^«»\\r\\n]{1,160}»|"[^"\\r\\n]{1,160}"|“[^“”\\r\\n]{1,160}”|(?:\\p{Lu}|\\d{1,4}-?\\p{L})${employerWord}?(?:[ \\t]+${employerWord}){0,5}`;
+const experienceLine = new RegExp(
+  `${lineTitle}(?:,[ \\t]+|[ \\t]+[-–—][ \\t]+)(${employer})(?:[ \\t]*\\(${range}\\)|[ \\t]*[,–—-][ \\t]*${range})`,
+  'gmu'
+);
+const experienceParenthesis = new RegExp(
+  `${lineTitle}[ \\t]*\\((${employer})\\)(?:,?[ \\t]*${range}|[ \\t]*\\(${range}\\))`,
+  'gmu'
+);
+
 export function detectDocumentEntities(text, emit, options) {
   if (!options.documentEntities) {
     return;
   }
+  const found = [];
+  const capture = (finding) => {
+    found.push(finding);
+    emit(finding);
+  };
   for (const [type, rule, pattern, group, confidence] of [
+    ['ORGANIZATION', 'organization-group', groupOrganization, 0, 0.7],
+    ['EDUCATION', 'education-russian', ruSchool, 0, 0.65],
+    ['EDUCATION', 'education-acronym', ruAcronym, 0, 0.65],
+    ['PERSON', 'document-introduction', ruIntroduction, 1, 0.7],
     ['ORGANIZATION', 'organization-legal-form', organizations, 0, 0.7],
     ['EDUCATION', 'education-type', schools, 0, 0.65],
     ['LOCATION', 'location-gazetteer', places, 0, 0.6],
     ['LOCATION', 'location-label', labelledPlace, 1, 0.75],
+    ['ORGANIZATION', 'organization-experience', experienceLine, 1, 0.6],
+    ['ORGANIZATION', 'organization-experience', experienceParenthesis, 1, 0.6],
   ]) {
     collectMatches(
       text,
       pattern,
-      emit,
+      capture,
       { type, rule, category: 'personal', confidence },
       group
     );
   }
+  for (const match of text.matchAll(ruResidence)) {
+    const value = match[1] ?? match[2];
+    const start = match.index + match[0].lastIndexOf(value);
+    emit({
+      start,
+      end: start + value.length,
+      type: 'LOCATION',
+      rule: 'location-residence',
+      category: 'personal',
+      confidence: 0.75,
+    });
+  }
+  // Organization cores are confirmed only by a bounded legal-form finding.
+  for (const finding of found.filter((f) => f.type === 'ORGANIZATION')) {
+    const core = organizationCore(text.slice(finding.start, finding.end));
+    if (core.length < 3 || core.length > 256) {
+      continue;
+    }
+    for (const match of text.matchAll(personalRegex(core, 'ORGANIZATION'))) {
+      emit({
+        ...finding,
+        start: match.index,
+        end: match.index + match[0].length,
+        rule: 'organization-core',
+      });
+    }
+  }
 }

package/src/document-vocabulary.js

@@ -35,7 +35,7 @@ export const documentVocabulary = [
     'ru',
     'ООО|АО|ПАО|ИП|ЗАО',
     'Университет|университет|Институт|институт|Школа|школа',
-    'Москва|Санкт-Петербург|Казань|Татарстан',
+    'Москва|Санкт-Петербург|Казань|Татарстан|Ярославль|Кострома|Новосибирск|Екатеринбург|Нижний Новгород|Самара|Омск|Челябинск|Ростов-на-Дону|Уфа|Красноярск|Пермь|Воронеж|Волгоград|Саратов|Тюмень|Тольятти|Ижевск|Барнаул|Иркутск|Хабаровск|Владивосток|Махачкала|Томск|Оренбург|Кемерово|Рязань|Астрахань|Пенза|Липецк|Киров|Чебоксары|Тула|Калининград|Курск|Улан-Удэ|Ставрополь|Сочи|Тверь|Брянск|Белгород|Владимир|Архангельск|Калуга|Смоленск|Вологда|Саранск|Орёл',
   ],
   ['ja', '株式会社|有限会社|合同会社', '大学|学院|学校', '東京|大阪|北海道'],
   ['zh', '有限公司|股份有限公司', '大学|学院|学校', '北京|上海|广东'],

package/src/engines.js

@@ -162,8 +162,12 @@ export function createSanitizer(options = {}) {
     }
   }
   const engines =
-    options.secretlint === false ? [] : [{ detect: secretlintDetect }];
-  engines.push(...(options.detectors ?? []));
+    options.secretlint === false || options.knownOnly
+      ? []
+      : [{ detect: secretlintDetect }];
+  if (!options.knownOnly) {
+    engines.push(...(options.detectors ?? []));
+  }
   let prepared;
   async function analyze(
     text,

package/src/entities.js

@@ -36,6 +36,13 @@ for (const [vendor, catalog] of Object.entries(entityCatalogs)) {
     entries.set(name.toUpperCase(), entry);
   }
 }
+export const catalogTypes = () => [
+  ...new Set(
+    [...entries.values()]
+      .filter((entry) => entry.category === 'personal')
+      .map((entry) => (entry.type === 'DATE_OF_BIRTH' ? 'DATE' : entry.type))
+  ),
+];
 // All documented names are native labelled recognizers; automatic format
 // recognizers are separate and do not claim the vendors' model behavior.
 const prefix = new RegExp(