@link-foundation/sensitive-data-sanitizer 0.16.0 → 0.16.2

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.16.2
+
+### Patch Changes
+
+- Make CI security and validation gates fail on incomplete evidence, recover transient release push failures, preserve validated release dependencies, and prevent quadratic scanning of PuTTY headers, travel identifiers and postal-address whitespace.
+
+  Preserve durations, stack frames, empty document fields and local bind addresses; protect personal URL paths and additional document forms. Reduce repeated label-pattern construction and document structured JSON per-call costs.
+
+## 0.16.1
+
+### Patch Changes
+
+- 4dfe2b7: Scan standalone and embedded JSON string values with their own line boundaries. Extend the documents profile's contact, identity, name, attribute, employer, school and residence rules, and preserve ordinary relative URLs, operational timestamps and context-free SHA-1-shaped values consistently.
+
 ## 0.16.0
 
 ### Minor Changes

package/docs/COVERAGE.md

@@ -202,3 +202,138 @@ Structured JSONL streaming supports 8 MiB records and ordered record workers;
 and reproducible throughput measurement. Realistic fake mode preserves supported
 scripts/formats/checksums and uses reserved contact ranges where available.
 Generated identities are pseudonyms; valid arithmetic does not prove non-issuance.
+
+## Document forms reviewed in issues 77–83
+
+Structured JSON/JSONL scans each decoded string at its own line start as well as
+with its field label. This includes array elements, multiline strings and up to
+two embedded JSON container layers. Both the synchronous API and asynchronous
+engine union use these views. Source offsets include original JSON escapes;
+unchanged values and whitespace remain byte-identical. Numeric primitives still
+require a caller schema policy. `debug` can observe `structured-value` events
+with finding counts, without logging the string contents.
+
+The `documents` profile additionally recognizes:
+
+- Russian mobile numbers with dashed, dotted, spaced or parenthesized forms,
+  including ten-digit `9xx` forms and domestic `8` prefixes; `тел.`, `моб.` and
+  WhatsApp labels; `wa.me` contact links.
+- Obfuscated email separators `(at)`, `[at]`, `собака`, and English `dot`/`at`.
+  This is a finite spelling grammar, not arbitrary email deobfuscation.
+- Personal paths on LinkedIn (`/in/`), GitHub, VK, Instagram and Telegram,
+  including scheme-less links; Telegram/телеграм/tg/skype/ник labels without `@`.
+  A provider root such as `https://github.com/` remains unchanged. These rules
+  cannot distinguish a person from an organization account on the same service.
+- Russian passport series/number phrases, ОГРНИП, subdivision codes, dated
+  ОВД/УВД/МВД issuer phrases, Russian/English driving licence labels, ОМС/полис,
+  Russian account numbers and numbered diplomas. These contextual patterns do
+  not validate issuance; digits are protected even with invalid checksums.
+- Numbered schools, gymnasiums, lyceums and colleges; school acronyms after
+  Образование/ВУЗ/Университет/окончил/окончила, with optional `им. Баумана`-style
+  suffixes. Acronym detection is contextual rather than a complete school list.
+- Surname/initial shapes in Cyrillic and selected Latin Slavic forms, known
+  given-name plus initial, Russian surname/given-name pairs using finite case
+  forms and lower case, and surnames after г-жа/господин/госпожа.
+- Birth years with д.р./г.р./год рождения/родился/родилась/Born in context;
+  Russian age expressions and Пол labels, English Sex, and gender beside age.
+  Bare age-in-years expressions are ambiguous and remain unchanged without
+  affirmative personal-age context; durations are preserved.
+- Labelled citizenship, marital status, children, nationality, religion,
+  disability and criminal record, including the reported English labels.
+- Company/employer/workplace labels, capitalized company/holding/bank names,
+  employment prose and year-range employer lines; residence/metro/district
+  labels and contextual six-digit postal codes.
+- Russian street-plus-house/flat structures at line start or after
+  `проживает по адресу`. Public-place prose such as
+  `Музей находится на ул. Ленина, д. 5, кв. 12` stays unchanged. Public addresses
+  presented alone have the same shape as personal addresses and may be flagged;
+  inspect and review standalone public addresses before publication.
+
+Explicit limits accepted for the ambiguous examples in #82/#83:
+`Спасибо, Анна!`, `Анна, добрый день!`, `Менеджер Анна` and `Лютик, менеджер`
+remain unchanged without known values or additional context. Bare given names
+and unmarked employer/title fragments cannot reliably establish identity.
+`зарплата 150 000 руб.` and `Salary: 150000` also remain unchanged: salary is
+outside the native document rules. Supply `knownPersonal` or an additional
+recognizer when these fields must be removed. Initial/name rules do not cover
+all surnames, languages, cases or arbitrary lower-case names.
+
+Operational value policy (#79): a single-slash relative URL with no query,
+fragment, userinfo or explicit u/in/user/profile/member/home/people path stays unchanged
+under `url`; other detectors and known values still apply. An ISO-style
+`YYYY-MM-DDTHH:mm:ss[.fraction](Z|±HH:mm)` timestamp under `time` stays unchanged.
+This exemption is syntactic, not date validation. Userinfo, token-bearing URLs
+and personal paths still redact. Context-free 40-character hexadecimal values
+are treated as SHA-1-shaped identifiers in both text and structured modes,
+including quoted values. This does not prove they are harmless: actual bare
+40-hex secrets require `knownSecrets` or credential context. Credential labels,
+provider-specific rules and required external scanners remain authoritative.
+
+The [complete requirement inventory and alternatives](case-studies/issue-84/README.md)
+links every reported text row to its regression case.
+
+## Document context refinements (issues 87–88)
+
+Age requires a label (`возраст 41`, `Age 41`), personal pronoun (`мне 29 лет`),
+birthday verb (`исполнилось 34`) or gender followed by age (`Женщина, 34 года`).
+Bare `34 года` remains unchanged as an ambiguous limit. Work experience, tenure,
+contract lengths and elapsed/future durations stay unchanged. `Дети: нет` and
+`Гражданство: не указано` are treated as empty fields; negative disability and
+criminal-record statements remain sensitive attributes.
+
+Additional document forms include `контактный телефон` and `сот.` phone labels,
+parenthesized `(собака)` email separators, `телега`/`Zoom` messenger labels,
+dated `ТП УФМС`/`УФМС` passport issuers, `р/с` account numbers, `г-н` surname
+titles, `1985 года рождения` and `г.р. 1985`, abbreviated/single-letter gender
+values (`Пол: Ж`, `Пол: М`, `пол: жен.`, `пол муж.`), `гражданин РФ`,
+`есть двое детей`, `проживаю в Перми`, and English `Employer`, `Company` and
+`Education` labels. A capitalized Cyrillic word followed by two initials is
+recognized regardless of surname ending (`Цой В. Р.`, `Ким Д. С.`,
+`Шевченко Т. Г.`). This structural rule can flag non-name text with the same
+shape; it does not provide a complete surname dictionary.
+
+JSON escape boundaries cannot start an obfuscated-email local part, preserving
+stack newlines and frames after IP:port. Relative `/u/<name>` and `/in/<name>`
+URLs are personal paths under `url`, like the existing user/profile paths.
+Exact `0.0.0.0` is an operational unspecified bind address, like loopback/local
+addresses. `publicKnowledge: false` and explicit private known values still
+redact it.
+
+## Structured JSON cost (issue 86)
+
+Structured sanitization scans labelled raw strings and each decoded string with
+its own line boundaries, then verifies sanitized output. This preserves detection
+in arrays, CV fragments and embedded JSON, but adds fixed work per string/key.
+Costs depend on string count, enabled engines, content, runtime and hardware;
+line length alone cannot predict synchronous main-thread delay.
+
+A finite serial measurement on Linux x64, Node 26.13.0, using the documents
+profile and structured JSON, produced these median milliseconds per `sanitize`
+call (100 warm-ups, three 100-call trials):
+
+| Fixture                                   | 0.16.0 | 0.16.1 | This change |
+| ----------------------------------------- | -----: | -----: | ----------: |
+| Short JSON (89 characters)                |   1.21 |   8.44 |        4.67 |
+| Exact issue reproduction (264 characters) |   1.81 |  19.71 |       11.15 |
+| Longer synthetic JSON (9,396 characters)  |  10.04 |  37.88 |       28.32 |
+
+The short and longer synthetic rows approximate the issue's unpublished
+100/9,453-character inputs; only the 264-character input was provided in full.
+These results are evidence for this container, not a universal latency guarantee
+or an Apple Silicon comparison. The issue reporter measured 0.16.1 at
+2.31/5.61/34.85 ms on their short/264-character/9 KiB rows and 1.10 ms for the
+264-character text-mode control. The remaining structured overhead is **above
+2x** of 0.16.0 here; documenting that cost fulfills the issue's alternative
+acceptance criterion. No detection view is removed to meet a timing target.
+
+Personal-label patterns are now constructed once instead of for every scan, and
+redundant explicit RegExp cloning is avoided when `matchAll` supplies isolation.
+Reproduce with `node --max-old-space-size=256 experiments/issue-89/benchmark.mjs`
+and optionally pass a released checkout's `src/index.js`. For latency-sensitive
+logging, measure the real workload before synchronously sanitizing every line;
+the existing JSONL record-worker API/CLI can move record scanning off the main
+thread (see [API.md](API.md)). External engines and more sensitive values can add
+further work. These numbers do not describe `createSanitizer` with Secretlint.
+
+The [complete issue 89 requirement inventory](case-studies/issue-89/README.md)
+records every row, root cause, alternative and regression mapping.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@link-foundation/sensitive-data-sanitizer",
-  "version": "0.16.0",
+  "version": "0.16.2",
   "description": "Offline sensitive-text sanitization with credential, multilingual personal-data and scanner union detection",
   "type": "module",
   "main": "src/index.js",
@@ -92,7 +92,7 @@
   },
   "devDependencies": {
     "@changesets/cli": "^3.0.3",
-    "eslint": "^9.38.0",
+    "eslint": "^10.13.0",
     "eslint-config-prettier": "^10.1.8",
     "eslint-plugin-prettier": "^5.5.4",
     "husky": "^9.1.7",
@@ -101,7 +101,10 @@
     "mrz": "^5.0.2",
     "prettier": "^3.6.2",
     "secretlint": "13.0.7",
-    "test-anywhere": "^0.8.48"
+    "test-anywhere": "^0.8.48",
+    "@eslint/js": "^10.0.1",
+    "command-stream": "2.4.0",
+    "lino-arguments": "0.4.0"
   },
   "lint-staged": {
     "*.{js,mjs,cjs}": [
@@ -112,6 +115,12 @@
     "*.md": [
       "prettier --write",
       "prettier --check"
+    ],
+    "*.sh": [
+      "pipx run --spec shellcheck-py==0.11.0.1 shellcheck"
+    ],
+    ".husky/*": [
+      "pipx run --spec shellcheck-py==0.11.0.1 shellcheck"
     ]
   },
   "dependencies": {
@@ -124,5 +133,11 @@
     "re2-wasm": "1.0.2",
     "re2js": "^2.9.0",
     "yaml": "^2.9.1"
+  },
+  "allowScripts": {
+    "node-pty": false
+  },
+  "overrides": {
+    "minimatch@>=7 <11": "^11.0.4"
   }
 }

package/src/context.js

@@ -228,6 +228,9 @@ function quotedRange(text, start, opening, maxLength) {
 }
 
 function plausibleLabel(value, type) {
+  if (operationalValue(value, type)) {
+    return false;
+  }
   if (type === 'PERSON') {
     return plausiblePerson(value.replace(/[,;\d].*$/s, '').trim());
   }
@@ -240,6 +243,20 @@ function plausibleLabel(value, type) {
   return true;
 }
 
+function operationalValue(value, type) {
+  return (
+    (type === 'URL' &&
+      /^\/(?!\/)[^\s?@#]*$/.test(value) &&
+      !/(?:^|\/)(?:u|in|users?|profiles?|people|members?|home)\/[^/]+/i.test(
+        value
+      )) ||
+    (type === 'TIME' &&
+      /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})$/.test(
+        value
+      ))
+  );
+}
+
 // Shared by native labels and the vendor catalogs. Code expressions and
 // line escapes are not personal field values, even when a key is familiar.
 export function personalValueAllowed(text, range, type, labelStart) {
@@ -376,19 +393,24 @@ function detectCredentials(text, emit) {
   }
 }
 
-function detectPersonalLabels(text, emit) {
-  for (const [type, words] of Object.entries(labels)) {
-    if (type === 'credential') {
-      continue;
-    }
+const personalPrefixes = Object.entries(labels)
+  .filter(([type]) => type !== 'credential')
+  .map(([type, words]) => {
     const separator =
       type === 'ID'
         ? `(?:[:=：]|(?:${verbs})(?![\\p{L}\\p{N}_]))?[ \\t]+|[:=：][ \\t]*`
         : `(?:[:=：]|(?:${verbs})(?![\\p{L}\\p{N}_]))[ \\t]*`;
-    const prefix = new RegExp(
-      `(?<![\\p{L}\\p{N}_])(?:\\\\*["'])?(?:${words.map(escapePattern).join('|')})(?![\\p{L}\\p{N}_])(?:\\\\*["'])?[ \\t]*(?:${separator})`,
-      'giu'
-    );
+    return [
+      type,
+      new RegExp(
+        `(?<![\\p{L}\\p{N}_])(?:\\\\*["'])?(?:${words.map(escapePattern).join('|')})(?![\\p{L}\\p{N}_])(?:\\\\*["'])?[ \\t]*(?:${separator})`,
+        'giu'
+      ),
+    ];
+  });
+
+function detectPersonalLabels(text, emit) {
+  for (const [type, prefix] of personalPrefixes) {
     for (const match of text.matchAll(prefix)) {
       const identity = type === 'ID';
       const suffix = identity

package/src/credentials.js

@@ -45,10 +45,26 @@ export function detectCredentialFormats(text, emit) {
     (value) =>
       !credentialPlaceholder(value) && value !== 'put your unique phrase here'
   );
-  collectMatches(
-    text,
-    /PuTTY-User-Key-File-[23]:[^\r\n]*(?:\r\n|\r|\n)[\s\S]*?(?:Private-MAC:[ \t]*[a-fA-F0-9]+|$)/g,
-    emit,
-    details('putty-private-key')
-  );
+  detectPuttyPrivateKeys(text, emit);
+}
+
+// Scan each consumed range once. A regex combining an unbounded header line
+// and a lazy body rescans every repeated header when the newline is missing.
+function detectPuttyPrivateKeys(text, emit) {
+  const headers = /PuTTY-User-Key-File-[23]:/g;
+  const lineEnd = /[\r\n]/g;
+  const mac = /Private-MAC:[ \t]*[a-fA-F0-9]+/g;
+  let header;
+  while ((header = headers.exec(text))) {
+    lineEnd.lastIndex = headers.lastIndex;
+    const newline = lineEnd.exec(text);
+    if (!newline) {
+      break;
+    }
+    mac.lastIndex = newline.index + 1;
+    const terminator = mac.exec(text);
+    const end = terminator ? mac.lastIndex : text.length;
+    emit({ start: header.index, end, ...details('putty-private-key') });
+    headers.lastIndex = end;
+  }
 }

package/src/detection.js

@@ -58,11 +58,14 @@ export function collectMatches(
   group = 0,
   accept = () => true
 ) {
-  // Clone expressions so concurrent callers never share lastIndex.
-  const pattern = new RegExp(
-    regex.source,
-    regex.flags.includes('g') ? regex.flags : `${regex.flags}g`
-  );
+  // matchAll clones global expressions and leaves their lastIndex untouched.
+  const pattern =
+    regex.global && regex.lastIndex === 0
+      ? regex
+      : new RegExp(
+          regex.source,
+          regex.global ? regex.flags : `${regex.flags}g`
+        );
   for (const match of text.matchAll(pattern)) {
     const raw = match[group];
     const value = details.category === 'personal' ? raw?.trim() : raw;
@@ -185,7 +188,9 @@ export function nativeDetect(text, options, emit) {
     emit,
     { ...credential, rule: 'quoted-entropy' },
     1,
-    (value) => entropy(value) >= (/^[a-f0-9]+$/i.test(value) ? 3 : 4.5)
+    (value) =>
+      !/^[a-f0-9]{40}$/i.test(value) &&
+      entropy(value) >= (/^[a-f0-9]+$/i.test(value) ? 3 : 4.5)
   );
   collectMatches(
     text,

package/src/document-entities.js

@@ -3,6 +3,7 @@ import { firstNames } from './names.js';
 import { organizationCore, personalRegex } from './personal-variants.js';
 import { documentVocabulary } from './document-vocabulary.js';
 import { months } from './personal-context.js';
+import { detectDocumentPatterns } from './document-patterns.js';
 
 const words = (column) =>
   [...new Set(documentVocabulary.flatMap((row) => row[column].split('|')))]
@@ -87,6 +88,7 @@ export function detectDocumentEntities(text, emit, options) {
   if (!options.documentEntities) {
     return;
   }
+  detectDocumentPatterns(text, emit);
   const found = [];
   const capture = (finding) => {
     found.push(finding);

package/src/document-patterns.js

@@ -0,0 +1,245 @@
+import { firstNames } from './names.js';
+import { isNameForm } from './russian-names.js';
+
+const boundary = '[\\p{L}\\p{N}_]';
+const word = '[А-ЯЁ][а-яё-]{1,47}';
+const surname =
+  '[А-ЯЁа-яё][А-ЯЁа-яё-]{1,28}(?:ов|ев|ин|ын|ск|цк)(?:а|ой|ою|у|ым|ом|е|ы|и|их|ий|ая|ого|ому|ую|ие)?';
+const initials = '(?:[А-ЯЁA-Z]\\.[ \\t]*){1,2}';
+const latinSurname = '[A-Z][a-z]{1,28}(?:ova|eva|ov|ev|in|ina|sky|ski|skaya)';
+const given = [
+  ...new Set(Object.values(firstNames).flatMap((s) => s.split(' '))),
+].join('|');
+const nameInitials = new RegExp(
+  `(?<!${boundary})(?:${word}[ \\t]+(?:[А-ЯЁ]\\.[ \\t]*){2}|${surname}[ \\t]+${initials}|${initials}(?:${surname}|${latinSurname})|${latinSurname}[ \\t]+${initials}|(?:${given})[ \\t]+[А-ЯЁA-Z]\\.)(?!${boundary})`,
+  'gu'
+);
+const namePairs = new RegExp(
+  `(?<!${boundary})(${surname})[ \\t]+([А-ЯЁа-яё]{2,32})|(?<!${boundary})([А-ЯЁа-яё]{2,32})[ \\t]+(${surname})(?!${boundary})`,
+  'gu'
+);
+// Every rule is finite and requires either an identifying structure or a label.
+// Public locations/ambiguous bare words are not inferred from arbitrary prose.
+const rules = [
+  [
+    'PHONE',
+    'document-phone',
+    /(?<![\p{L}\p{N}_])(?:[78][ .-]?)?9\d{2}(?:[ .-]\d{3}[ .-]\d{2}[ .-]\d{2})(?!\d)|(?<![\p{L}\p{N}_])8[ .-]?\(9\d{2}\)[ .-]?\d{3}[ .-]?\d{2}[ .-]?\d{2}(?!\d)/gu,
+    0,
+  ],
+  [
+    'PHONE',
+    'document-phone-label',
+    /(?<![\p{L}\p{N}_])(?:(?:контактный[ \t]+)?телефон|тел\.?|моб\.?|сот\.?|WhatsApp)(?:[ \t]*:[ \t]*|[ \t]+)((?:[78][ () .-]*)?9\d{2}[ () .-]*\d{3}[ .-]*\d{2}[ .-]*\d{2})(?!\d)/giu,
+    1,
+  ],
+  [
+    'EMAIL',
+    'obfuscated-email',
+    /(?<![\p{L}\p{N}_.+\\-])[\p{L}\p{N}_+-]{1,64}(?:(?:\.|[ \t]{1,8}dot[ \t]{1,8})[\p{L}\p{N}_+-]{1,64}){0,4}(?:[ \t]{0,8}\((?:at|собака)\)[ \t]{0,8}|[ \t]{0,8}\[at\][ \t]{0,8}|[ \t]{1,8}(?:at|собака)[ \t]{1,8})[\p{L}\p{N}-]{1,63}(?:(?:\.|[ \t]{1,8}dot[ \t]{1,8})[\p{L}\p{N}-]{1,63}){1,8}(?![\p{L}\p{N}_])/giu,
+    0,
+  ],
+  [
+    'USERNAME',
+    'personal-profile-url',
+    /(?<![\p{L}\p{N}_./-])(?:https?:\/\/)?(?:www\.)?(?:linkedin\.com\/in|github\.com|vk\.com|instagram\.com|t\.me|wa\.me)\/[\p{L}\p{N}_][\p{L}\p{N}_.-]{1,100}(?:\/[\p{L}\p{N}_.-]{1,100}){0,4}(?![\p{L}\p{N}_])/giu,
+    0,
+  ],
+  [
+    'USERNAME',
+    'messenger-label',
+    /(?<![\p{L}\p{N}_])(?:Telegram|телеграм|телега|tg|skype|Zoom|ник)(?:[ \t]*:[ \t]*|[ \t]+)(@?[a-z][a-z\d_.-]{2,63})(?![\p{L}\p{N}_])/giu,
+    1,
+  ],
+  [
+    'PASSPORT_NUMBER',
+    'passport-series',
+    /(?<![\p{L}\p{N}_])(?:серия|series)[ \t]+(\d{2}[ \t]?\d{2}[ \t]+(?:№|No|номер|number|#)[ \t]*\d{6})(?!\d)/giu,
+    1,
+  ],
+  [
+    'ID',
+    'identity-number',
+    /(?<![\p{L}\p{N}_])(?:ОГРНИП[ \t]+(\d{15})|код подразделения[ \t]+(\d{3}-\d{3})|(?:водительское удостоверение|в\/у|Driver's licen[cs]e)[ \t]+(\d{2}[ \t]?\d{2}[ \t]?\d{6})|(?:ОМС|полис)[ \t]+(\d{16})|(?:сч[её]т|р\/с)[ \t]+(\d{20})|диплом[ \t]+(?:№|No|#)[ \t]*(\d{6}))(?!\d)/giu,
+    -1,
+  ],
+  [
+    'ID',
+    'passport-issuer',
+    /(?<![\p{L}\p{N}_])выдан[ \t]+((?:ОВД|УВД|МВД|(?:ТП[ \t]+)?УФМС)[ \t]+[^\r\n";|]{1,120}?\d{2}\.\d{2}\.\d{4})(?!\d)/giu,
+    1,
+  ],
+  [
+    'EDUCATION',
+    'numbered-school',
+    /(?<![\p{L}\p{N}_])(?:школа|гимназия|лицей|колледж)[ \t]*(?:№|No|#|:)[ \t]*\d{1,4}(?!\d)/giu,
+    0,
+  ],
+  [
+    'EDUCATION',
+    'school-acronym-context',
+    /(?<![\p{L}\p{N}_])(?:образование|ВУЗ|университет|окончил[аи]?)(?:[ \t]*:[ \t]*|[ \t]+)([А-ЯЁ]{2,12}(?:[ \t]+им\.[ \t]+[А-ЯЁ][а-яё-]{2,40})?)(?![\p{L}\p{N}_])/giu,
+    1,
+  ],
+  [
+    'EDUCATION',
+    'education-label',
+    /(?<![\p{L}\p{N}_])Education[ \t]*:[ \t]*([A-Z][A-Za-z.-]{1,47})(?![\p{L}\p{N}_])/gu,
+    1,
+  ],
+  [
+    'PERSONAL_ATTRIBUTE',
+    'profile-prose',
+    /(?<![\p{L}\p{N}_])(?:граждан(?:ин|ка)[ \t]+([А-ЯЁ]{2,4})|есть[ \t]+((?:один|одна|двое|трое|четверо|пятеро|\d{1,2})[ \t]+(?:детей|реб[её]нок|реб[её]нка)))(?![\p{L}\p{N}_])/gu,
+    -1,
+  ],
+  ['PERSON', 'name-initials', nameInitials, 0],
+  [
+    'PERSON',
+    'name-title',
+    new RegExp(
+      `(?<!${boundary})(?:г-жа|г-н|господин|госпожа)[ \\t]+(${word})(?!${boundary})`,
+      'gu'
+    ),
+    1,
+  ],
+  [
+    'DATE_OF_BIRTH',
+    'birth-year',
+    /(?<![\p{L}\p{N}_])(?:д\.р\.|г\.р\.|год рождения|родил(?:ся|ась)(?:[ \t]+в)?|Born(?:[ \t]+in)?)[ \t]*:?[ \t]*((?:19|20)\d{2})(?!\d)|(?<!\d)((?:19|20)\d{2})(?=[ \t]+(?:г\.р\.|года[ \t]+рождения)(?![\p{L}\p{N}_]))/giu,
+    -1,
+  ],
+  [
+    'AGE',
+    'age-context',
+    /(?<![\p{L}\p{N}_])(?:(?:возраст|Age)(?:[ \t]*:[ \t]*|[ \t]+)|(?:мне|исполнилось)[ \t]+)(\d{1,3})(?!\d)|(?<![\p{L}\p{N}_])(?:Женщина|Мужчина),[ \t]*(\d{1,3})(?=[ \t]+(?:года|год|лет)(?![\p{L}\p{N}_]))/giu,
+    -1,
+  ],
+  [
+    'GENDER',
+    'gender-context',
+    /(?<![\p{L}\p{N}_])(?:Пол|Sex)(?:[ \t]*:[ \t]*|[ \t]+)(женский|мужской|жен\.|муж\.|Ж|М|male|female)(?![\p{L}\p{N}_])|(?<![\p{L}\p{N}_])(Женщина|Мужчина)(?=,[ \t]*\d{1,3}[ \t]+(?:года|год|лет))/giu,
+    -1,
+  ],
+  [
+    'PERSONAL_ATTRIBUTE',
+    'profile-attribute',
+    /(?<![\p{L}\p{N}_])(?:Гражданство|Citizenship|Семейное положение|Marital status|Дети|национальность|Вероисповедание|инвалидность|судимость)[ \t]*:[ \t]*([^\r\n";|]{1,160})/giu,
+    1,
+  ],
+  [
+    'ORGANIZATION',
+    'employer-label',
+    new RegExp(
+      `(?<!${boundary})(?:[Кк]омпания|[Рр]аботодатель|[Мм]есто работы|[Cc]ompany|[Ee]mployer)[ \\t]*:[ \\t]*([\\p{Lu}][\\p{L}\\p{M}.-]{1,47})(?!${boundary})`,
+      'gu'
+    ),
+    1,
+  ],
+  [
+    'ORGANIZATION',
+    'employer-prose',
+    new RegExp(
+      `(?<!${boundary})(?:компания|работал[аи]?[ \\t]+в|холдинг|[Бб]анк)[ \\t]+(«[^«»\\r\\n]{1,120}»|${word})(?!${boundary})`,
+      'gu'
+    ),
+    1,
+  ],
+  [
+    'ORGANIZATION',
+    'employer-years',
+    new RegExp(
+      `(?:^|\\n)[ \\t]*(?:19|20)\\d{2}[–—-](?:19|20)\\d{2}[ \\t]+(${word})(?!${boundary})`,
+      'gu'
+    ),
+    1,
+  ],
+  [
+    'LOCATION',
+    'residence-prose',
+    new RegExp(
+      `(?<!${boundary})(?:(?:живу|проживаю)[ \\t]+в|из|м\\.|метро|район)[ \\t]+(${word})(?!${boundary})`,
+      'gu'
+    ),
+    1,
+  ],
+  [
+    'LOCATION',
+    'postal-code',
+    /(?<![\p{L}\p{N}_])индекс[ \t]+(\d{6})(?!\d)|(?<!\d)(\d{6})(?=,[ \t]*[А-ЯЁ][а-яё-]{2,47})/gu,
+    -1,
+  ],
+];
+
+export function detectDocumentPatterns(text, emit) {
+  for (const [type, rule, pattern, group] of rules) {
+    for (const match of text.matchAll(pattern)) {
+      const value =
+        group === -1
+          ? match.slice(1).find((part) => part !== undefined)
+          : match[group].trim();
+      if (
+        !documentValueAllowed(rule, value) ||
+        (rule === 'profile-attribute' &&
+          /^Дети[ \t]*:/iu.test(match[0]) &&
+          /^нет$/iu.test(value))
+      ) {
+        continue;
+      }
+      const start =
+        match.index + (group === 0 ? 0 : match[0].lastIndexOf(value));
+      emit({
+        start,
+        end: start + value.length,
+        type,
+        rule,
+        category: 'personal',
+        confidence: group === -1 ? 0.85 : 0.75,
+      });
+    }
+  }
+  for (const match of text.matchAll(namePairs)) {
+    if (isNameForm(match[2] ?? match[3])) {
+      emit({
+        start: match.index,
+        end: match.index + match[0].length,
+        type: 'PERSON',
+        rule: 'name-case',
+        category: 'personal',
+        confidence: 0.8,
+      });
+    }
+  }
+  // An address at line start or in explicit residence context is personal.
+  // Public-place prose with a street address does not supply that context.
+  const street =
+    /(?:^|\n)[ \t]*((?:ул\.|улица)[ \t]+[А-ЯЁ][а-яё-]{1,40},[ \t]*(?:(?:д\.|дом)[ \t]*)?\d{1,5}(?:[ \t]*,[ \t]*кв\.[ \t]*\d{1,5})?)|проживает по адресу[ \t]+((?:ул\.|улица)[ \t]+[А-ЯЁ][а-яё-]{1,40},[ \t]*(?:(?:д\.|дом)[ \t]*)?\d{1,5}(?:[ \t]*,[ \t]*кв\.[ \t]*\d{1,5})?)/gu;
+  for (const match of text.matchAll(street)) {
+    const value = match[1] ?? match[2];
+    const start = match.index + match[0].lastIndexOf(value);
+    emit({
+      start,
+      end: start + value.length,
+      type: 'ADDRESS',
+      rule: 'document-street',
+      category: 'personal',
+      confidence: 0.85,
+    });
+  }
+}
+
+function documentValueAllowed(rule, value) {
+  return (
+    Boolean(value) &&
+    value !== '[REDACTED]' &&
+    !(
+      rule === 'school-acronym-context' &&
+      !/^[А-ЯЁ]{2,12}(?![А-ЯЁа-яё])/.test(value)
+    ) &&
+    !(
+      rule === 'profile-attribute' &&
+      /^(?:не указан[оаы]?|not specified|unknown|n\/a)$/iu.test(value)
+    ) &&
+    !(rule === 'messenger-label' && /^(?:недоступен|unavailable)$/i.test(value))
+  );
+}

package/src/engines.js

@@ -24,6 +24,7 @@ import {
 } from './placeholders.js';
 import {
   structuredBatches,
+  structuredValueViews,
   mapStructuredFindings,
   verifyStructured,
 } from './structured.js';
@@ -143,6 +144,21 @@ async function analyzeStructured(
     )) {
       findings.push(finding);
     }
+    for (const view of structuredValueViews(batch, options)) {
+      const detected = await analyze(
+        view.text,
+        [],
+        true,
+        { ...options, structured: undefined },
+        nativeFindings === undefined ? undefined : []
+      );
+      for (const finding of detected) {
+        const mapped = view.map(finding);
+        if (mapped) {
+          findings.push(mapped);
+        }
+      }
+    }
   }
   // Native structured inspection validates, locates and reapplies the
   // same fake-value/public policy to source offsets from all engines.

package/src/identity.js

@@ -181,11 +181,11 @@ function detectPassports(text, emit) {
   });
 }
 const booking =
-  /(?<![\p{L}\p{N}_])(?:booking(?:[ _-]+(?:reference|code))?|reservation(?:[ _-]+code)?|confirmation(?:[ _-]+code)?|бронь|record[ _-]+locator|PNR|код брони|номер брони|бронирование)[ \t]*[:=：]?[ \t]*[`"']?([A-Z0-9]{4,8})(?![\p{L}\p{N}_])/giu;
+  /(?<![\p{L}\p{N}_])(?:booking(?:[ _-]+(?:reference|code))?|reservation(?:[ _-]+code)?|confirmation(?:[ _-]+code)?|бронь|record[ _-]+locator|PNR|код брони|номер брони|бронирование)[ \t]*(?:[:=：][ \t]*)?[`"']?([A-Z0-9]{4,8})(?![\p{L}\p{N}_])/giu;
 const ticket =
-  /(?<![\p{L}\p{N}_])(?:e[ -]?ticket|ticket[ _-]*(?:number|no)?|номер билета|электронный билет)[ \t]*[:=：]?[ \t]*[`"']?(\d{3}[ -]?\d{10})(?![\p{L}\p{N}_])/giu;
+  /(?<![\p{L}\p{N}_])(?:e[ -]?ticket|ticket[ _-]*(?:number|no)?|номер билета|электронный билет)[ \t]*(?:[:=：][ \t]*)?[`"']?(\d{3}[ -]?\d{10})(?![\p{L}\p{N}_])/giu;
 const visa =
-  /(?<![\p{L}\p{N}_])(?:e[ -]?visa|visa|номер визы|виза)(?:[ \t_-]+(?:registration|application|number|no|code|номер|код)){0,3}[ \t]*[:=：]?[ \t]*[`"']?([A-Z0-9][A-Z0-9-]{4,31})(?![\p{L}\p{N}_])/giu;
+  /(?<![\p{L}\p{N}_])(?:e[ -]?visa|visa|номер визы|виза)(?:[ \t_-]+(?:registration|application|number|no|code|номер|код)){0,3}[ \t]*(?:[:=：][ \t]*)?[`"']?([A-Z0-9][A-Z0-9-]{4,31})(?![\p{L}\p{N}_])/giu;
 const placeholder =
   /^(?:NONE|UNKNOWN|NULL|UNDEFINED|REDACTED|EXAMPLE|SAMPLE|PENDING|HOTEL|BOOKING|REFERENCE|REGISTRATION|APPLICATION|NUMBER|CODE|FAILED|ERROR|HTTP|OK|CANCELLED|CONFIRMED|SUCCESS|READY|ACTIVE|CLOSED)$/i;
 function detectTravel(text, emit) {

package/src/personal.js

@@ -224,7 +224,7 @@ export function detectPersonal(text, emit) {
   );
   collectMatches(
     text,
-    /(?<![\p{L}\p{N}])\d{1,6}[A-Za-z]?[ \t]+(?:[\p{Lu}][\p{L}.'-]*[ \t]+){1,5}(?:Street|St|Road|Rd|Avenue|Ave|Lane|Ln|Drive|Dr|Boulevard|Blvd|Way|Straße|Strasse|Rue|Calle|Rua)\b(?:,[ \t]+(?:[^\r\n<>]{1,80}?[ \t]+(?:[A-Z]{1,2}\d[A-Z\d]?[ \t]+\d[A-Z]{2}|\d{5}(?:-\d{4})?)\b|(?:[\p{Lu}][\p{Ll}.'-]+[ \t]*){1,4}))?/gu,
+    /(?<![\p{L}\p{N}])\d{1,6}[A-Za-z]?[ \t]+(?:[\p{Lu}][\p{L}.'-]*[ \t]+){1,5}(?:Street|St|Road|Rd|Avenue|Ave|Lane|Ln|Drive|Dr|Boulevard|Blvd|Way|Straße|Strasse|Rue|Calle|Rua)\b(?:,[ \t]+(?:(?:[^ \t\r\n<>][^\r\n<>]{0,79}?[ \t]+)?(?:[A-Z]{1,2}\d[A-Z\d]?[ \t]+\d[A-Z]{2}|\d{5}(?:-\d{4})?)\b|(?:[\p{Lu}][\p{Ll}.'-]+[ \t]*){1,4}))?/gu,
     emit,
     details('ADDRESS', 'postal-address')
   );

package/src/public.js

@@ -144,6 +144,7 @@ export function localAddress(value) {
   }
   const [a, b] = value.split('.').map(Number);
   return (
+    value === '0.0.0.0' ||
     a === 127 ||
     a === 10 ||
     (a === 172 && b >= 16 && b <= 31) ||

package/src/sanitizer.js

@@ -31,6 +31,7 @@ import {
 } from './placeholders.js';
 import {
   structuredBatches,
+  inspectStructuredValues,
   mapStructuredFindings,
   preserveIdentifierKeys,
   structuredValueRanges,
@@ -392,8 +393,8 @@ function inspectText(text, options = {}) {
   validateOptions(options);
   validateInput(text, options);
   if (options.structured) {
-    const findings = structuredBatches(text, options).flatMap((batch) =>
-      mapStructuredFindings(
+    const findings = structuredBatches(text, options).flatMap((batch) => [
+      ...mapStructuredFindings(
         batch,
         inspect(batch.text, {
           ...options,
@@ -401,8 +402,11 @@ function inspectText(text, options = {}) {
           findings: [],
         }),
         options
-      )
-    );
+      ),
+      ...inspectStructuredValues(batch, options, (value) =>
+        inspect(value, { ...options, structured: undefined, findings: [] })
+      ),
+    ]);
     return inspectFindingsOnly(text, findings, options);
   }
   const findings = [];

package/src/structured.js

@@ -280,3 +280,87 @@ export function verifyStructured(text, options) {
     }
   }
 }
+
+// Scan decoded strings at their own line start, retaining a map through every
+// JSON escape. Both native and asynchronous engines consume these same views.
+export function* structuredValueViews(
+  batch,
+  options,
+  depth = 0,
+  outerMap = (f) => f
+) {
+  for (const entry of batch.entries) {
+    const value = JSON.parse(entry.raw);
+    const offsets = stringOffsets(entry.raw);
+    const map = (finding) => {
+      const raw = {
+        ...finding,
+        start: offsets ? offsets.starts[finding.start] : finding.start + 1,
+        end: offsets ? offsets.ends[finding.end - 1] : finding.end + 1,
+      };
+      if (
+        preserveStructuredEntry(
+          entry,
+          {
+            ...raw,
+            start: entry.batchStart + raw.start,
+            end: entry.batchStart + raw.end,
+          },
+          batch.text,
+          options
+        )
+      ) {
+        return undefined;
+      }
+      return outerMap({
+        ...raw,
+        start: entry.start + raw.start,
+        end: entry.start + raw.end,
+      });
+    };
+    yield { text: value, map };
+    // Match the encoded detector's finite depth; arbitrary nesting is a limit.
+    if (depth < 2 && /^[ \t\r\n]*[[{]/.test(value)) {
+      let batches;
+      try {
+        batches = structuredBatches(value, { ...options, structured: 'json' });
+      } catch (error) {
+        if (error.code !== 'ERR_JSON') {
+          throw error;
+        }
+        batches = [];
+      }
+      for (const nested of batches) {
+        yield* structuredValueViews(nested, options, depth + 1, map);
+      }
+    }
+  }
+}
+export function inspectStructuredValues(batch, options, inspectValue) {
+  const result = [];
+  for (const view of structuredValueViews(batch, options)) {
+    const findings = inspectValue(view.text);
+    options.debug?.({ event: 'structured-value', count: findings.length });
+    for (const finding of findings) {
+      const mapped = view.map(finding);
+      if (mapped) {
+        result.push(mapped);
+      }
+    }
+  }
+  return result;
+}
+function stringOffsets(raw) {
+  if (!raw.includes('\\')) {
+    return undefined;
+  }
+  const starts = [],
+    ends = [];
+  for (let i = 1; i < raw.length - 1; ) {
+    const start = i;
+    i += raw[i] === '\\' ? (raw[i + 1] === 'u' ? 6 : 2) : 1;
+    starts.push(start);
+    ends.push(i);
+  }
+  return { starts, ends };
+}