@doccloak/core 0.12.1 → 0.13.0

package/CHANGELOG.md

@@ -4,6 +4,339 @@ All notable changes to `@doccloak/core` are documented here. The format is
 based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the
 project follows [Semantic Versioning](https://semver.org/).
 
+## [0.13.0] - 2026-09-25
+
+PDF-to-PDF redaction that keeps the text layer (new `@doccloak/core/pdf`
+entry point, see "Added" below), cancelable detection, and detection fixes
+found while testing real PDFs: Polish landline phones, catch-all numbers
+that swallowed a PESEL and a phone, regex spans without surrounding
+whitespace, placeholder fitting, OCR tokens split by a lost character,
+kerned punctuation after a redacted value, and overlapping detections that
+dropped part of a value.
+Host-facing additions are optional (`signal` on detect, `cancelDetect` in
+the worker protocol, the PDF entry point); nothing a 0.12.x host uses was
+removed. The new PDF dependencies are pinned exactly like the others:
+`@cantoo/pdf-lib` 2.11.1, `@cantoo/fontkit` 2.0.12, `pdfjs-dist` 6.3.289.
+
+### Found by the first CI corpus run on real LibreOffice PDFs (T233)
+
+- **PDF writer: kerned punctuation after a redacted value.** LibreOffice
+  writes `[(... Müller) 40 (, wohnhaft)] TJ`: the comma is pulled 0.48 pt
+  into the value's span. When a narrower placeholder closed the slack, the
+  rest of the line moved left but the comma, starting just before the end
+  of the removed span, counted as "left of the gap" and kept its old x,
+  landing inside "wohnhaft" ("wohn,haft", "[PHO,NE_1]"). A kept glyph now
+  moves with the text after a gap when it starts past the gap's middle; a
+  TJ that really jumps back still draws its text before the gap.
+- **Overlap resolution keeps what a losing entity covers past the winner.**
+  `resolveOverlaps` dropped the loser whole. In a PDF exported from a
+  spreadsheet a clipped cell is glued to the next one ("31-147
+  Krakók.wisniewska@firma.pl"): the model took "31-147 Krakók", the e-mail
+  rule the whole token, and ".wisniewska@firma.pl" stayed in clear. Now the
+  rest of a regex match is re-run through its rule and kept where it still
+  matches (the e-mail does; "before 2025" left over from a bogus
+  "00 before 2025" date does not), and the rest of a model span is kept
+  without its edge punctuation. Detection never covers less than before.
+- Tests: the corpus matches needles in PDFs across the line breaks a PDF
+  export puts inside a value (a space turned newline, a long e-mail broken
+  in a narrow cell, "Pawlak-" / "Dudek"), so wrapped values are replaced
+  and asserted absent instead of being counted as present and left alone;
+  values that LibreOffice Calc clips at a cell border are listed under
+  `optionalFor.pdf` in two seeds. PDF outputs of tests that redact only
+  some values on purpose (the desktop fixtures, the office invoice) are
+  written as `pdf-*`, so the CI poppler step opens them without grepping
+  the corpus needles.
+
+### OCR image redaction covers both halves of a token OCR split (T232)
+
+- Tesseract can drop a character inside a token and return two words: in
+  the kit's scanned letter "t.zielinski@zielinski-kancelaria.pl" came back
+  as "t" + "zielinski@zielinski-kancelaria.pl." with the dot's pixels in the
+  second box. The e-mail rule matched the second word only and boxes are
+  drawn per OCR word, so the "t" stayed visible (with "jan.kowalski@..." it
+  would be a first name). `selectRedactionBoxes` now spreads a selection to
+  same-line neighbours that `findSplitTokens` (exported) marks as one token:
+  the gap between their boxes is at most 0.3 of the line's median word gap
+  (height-based fallback on short lines; touching boxes included), and
+  boxes taller than 1.6x the line's median height (OCR noise such as a
+  26 px "w" on an 11 px line) never join. On seven OCR runs (the scan plus
+  kit pages 01/02 at 1000/1700/2500 px, about 1 150 words) exactly one pair
+  joins: the split e-mail. An ink test for a lost glyph left in a
+  normal-width gap was tried and dropped: Tesseract often cuts a letter's
+  edge out of its own box, which reads the same ("w terminie", "nr wpisu").
+  `renderRedactedImage`'s signature is unchanged.
+
+### A catch-all regex span yields to the specific detections it overlaps (T230)
+
+- `universal:long_number` (confidence 0.5) admits whitespace, so a PESEL cell
+  followed by a phone cell without "+48" ("90050598761 512 987 654") was one
+  23-character span that overlap resolution preferred (same start, longer)
+  over PESEL 0.9 + PHONE 0.8: the row came out as a single [SSN] and the
+  phone was "not detected". `detectEntities` now runs `splitCatchAlls`
+  first: a regex entity at or below confidence 0.5 that overlaps specific
+  detections (confidence >= 0.8, ML or regex) is replaced by its leftovers,
+  which are re-run through the same rule (an unknown long number next to a
+  phone is still caught); the specific spans then win. Catch-alls nothing
+  specific overlaps are untouched, and so is every other overlap (the
+  leak-averse "earlier start, longer span" order is unchanged).
+  Tests: tests/regex/catch-all-split.test.ts. No corpus expectation changed.
+
+### Polish landlines without +48; stricter postal-code-plus-city (T227)
+
+- `regex:pl:phone` only knew the 3-3-3 mobile grouping, so Polish landlines
+  written without the country code ("22 555 01 02", "(22) 555-01-02",
+  "71 777 88 99") fell through to `universal:long_number` and came out as
+  [SSN_n] under region pl. The rule now also accepts the 2-3-2-2 landline
+  grouping with an optional bracketed area code, with or without +48, and
+  the "+" of a "+48" prefix is part of the match (the mobile branch used to
+  start at "48"). A PESEL, a NIP, a bank account and a date still do not
+  match; an 11-digit number starting with 48 is no longer taken for a phone.
+- `regex:universal:postal_city` matched "05/2026 Maria" (date-like prefix)
+  and "111 222 przegląd" (lowercase word) and stole the start of a phone or
+  a person name. The code must now stand at the start of the line or after
+  whitespace (so nothing glued to a digit, "/", "." or "-" matches), a
+  4-2 / 4-4 hyphenated shape ("2025-03", "2024-2025") is skipped and the
+  city must start with an uppercase letter. "00-950 Warszawa", "75008 Paris"
+  and "1010 Wien" still match; a code glued to an opening bracket
+  ("(75008 Paris)") no longer does through this rule (country packs still
+  cover it). The rule stays lookbehind-free (pattern dialect lint) by
+  consuming the leading whitespace, which `runRule` trims (T226).
+- Web regression corpus `universal.json` case `u-03-generic-identifiers`:
+  "987654321 remains" was a baked-in postal_city false positive; the number
+  is now PHONE "987654321" via `regex:pl:phone` (0.8 beats
+  `universal:long_number` 0.5 and the other 9-digit rules on the same span).
+
+### Regex matches never carry whitespace (T226)
+
+- The IBAN rule ended its match on a word boundary after a class that admits
+  whitespace, so "PL61 1090 1014 0000 0712 1981 2874 w terminie" matched
+  with the trailing space and the redaction produced "[IBAN_1]w terminie"
+  (text, DOCX and PDF alike). The pattern now ends on a letter or digit, and
+  `runRule` trims whitespace at both ends of every regex match as a general
+  safety net (start/end adjusted; a validator sees the trimmed value).
+
+### Placeholder fit: move before shrink (T225)
+
+- The planner now moves the rest of the line before it shrinks a placeholder:
+  room up to the next column and up to the page's text edge is used at full
+  size first; only the remainder is shrunk (floor raised from 65 % to 75 %)
+  and condensed (80 %). Short words on left-aligned lines ("NIP", "Kraków")
+  no longer get half-size placeholders; lines that already reach the margin
+  behave as before. `fit.minSizeRatio` default is 0.75. The page's text edge
+  is a hard limit (content boxes are clipped by Chrome and Word), and of a
+  value wrapped over several lines the widest part now carries the
+  placeholder (before: always the first part).
+
+### Cancelable detection (T222)
+
+- `DetectionProvider.detect(text, onProgress?, signal?)`: the three built-in
+  providers check the signal between inference chunks and reject with the new
+  `DetectionAbortedError` (also thrown by `engine.detect` for a signal aborted
+  before or after the ML pass; it used to be a plain `Error('Detection aborted')`).
+- Worker protocol: new `cancelDetect { requestId }` request; the cancelled
+  call answers `detectError { aborted: true }`. `connectEngine().detect` with
+  an aborted signal now sends `cancelDetect` and rejects with
+  `DetectionAbortedError` once the host has acknowledged, so a rejection
+  means the worker is idle again (before, it rejected at once and the host
+  kept computing).
+
+PDF-to-PDF redaction that keeps the text layer (T206-T218, founder decision
+2026-09-24 reversing the 2026-08-30 desktop-only split).
+
+### Added
+
+- New entry point **`@doccloak/core/pdf`**: `readPdf(file, { assets })` and
+  `writeAnonymizedPdfWithReport(extraction, replacements, values, options)`.
+  The redacted glyphs are cut out of the content streams, the placeholder is
+  written back as real text in the document's own font when that font can
+  show it (codes proven present in the subset), else in a metrics-compatible
+  Liberation face (SIL OFL, shipped under `fonts/liberation`) embedded as a
+  subset. Placeholders are fitted to the redacted span: same size when it
+  fits, then shrunk to 65 %, then condensed to 80 % width, then the rest of
+  the line moves right by the overflow; a narrow placeholder closes the slack
+  so no hole is left. Every later show in the positioning scope is
+  re-anchored absolutely, so residual glyph positions cannot encode the
+  removed text (USENIX 2022 "Story Beyond the Eye").
+- Fonts: simple Type1/TrueType/Type3 (Standard/WinAnsi/MacRoman/Differences,
+  ToUnicode), composite Type0 with Identity-H/V or embedded CMaps (Chrome,
+  Word, macOS output), standard-14 metrics for non-embedded fonts.
+- Two independent extractors: our decoder and pdf.js. A page whose text
+  differs between them, or whose fonts cannot be read safely, is rasterized
+  (pixels blacked out) when it carries a redaction, and listed in
+  `rasterizedPages`. Text of such pages comes from pdf.js so detection still
+  sees it.
+- Output is rebuilt as a fresh single-revision document: Info, XMP,
+  outlines, forms, annotations, attachments, JavaScript, named destinations,
+  structure tree, page labels and the incremental-update history of the
+  input never reach the output (reported in `removed`); ToUnicode maps of
+  edited fonts are trimmed to the codes still used; ActualText/Alt marked
+  content is stripped; dates normalised to 2000-01-01, producer `DocCloak`.
+- Verification after every write: re-extraction with both extractors plus a
+  byte scan of every decoded stream and string (UTF-8, Latin-1, UTF-16 BE/LE)
+  and of the removed code sequences; any trace throws `PdfVerifyError` and
+  nothing is exported.
+- `office.ts`: `officeFileKind` returns `'pdf'`; `analyzeOfficeFile` and
+  `redactOfficeFile` handle PDFs through a lazy import of the PDF module
+  (`options.pdf` carries assets, password, fit policy); results gain
+  `removed` and `rasterizedPages`.
+- `UnredactablePart.kind` gains `'scanned-page'` and `'undecodable-text'`.
+- Dependencies: `@cantoo/pdf-lib`, `@cantoo/fontkit`, `pdfjs-dist`. Hosts
+  serve the pdf.js worker, its CMaps and standard fonts, and the Liberation
+  TTFs same-origin (see README, "PDF").
+
+### Second review (T219, same release)
+
+- Layer zero searches the flat text in the verifier's normal form (whitespace,
+  soft hyphens and zero-width characters dropped, ligatures spelled out, case
+  folded), so a value wrapped over a line end ("Jan" / "Kowalski" on the next
+  line), spaced differently by the layout or set with a ligature is redacted
+  instead of blocking the export with `PdfVerifyError`. Segments of a value
+  on other lines close up over the removed glyphs.
+- A fallback font needed inside a form XObject that has no `/Resources` of
+  its own goes into the page resources (where viewers resolve it); duplicated
+  forms with identical bytes all receive it.
+- Lexer: inline image data ends the way pdf.js finds it (`EI` need not follow
+  whitespace, must be followed by ASCII; DCT data ends at its EOI marker,
+  ASCIIHex at `>`, ASCII85 at `~>`; `/L` honoured); an image without an end
+  makes the rest of the stream unsafe instead of a silent opaque operator;
+  arrays/dicts nested deeper than 64 are unsafe instead of a stack overflow;
+  `--5` reads as -5 (pdf.js, Acrobat); raw CR inside literal strings is kept.
+- CMap parser tolerates what pdf.js tolerates: a missing `endcmap`, a bfrange
+  destination array shorter than the range, bounds of different byte lengths.
+  Rewritten ToUnicode maps keep the source codespace and skip empty
+  destinations. Glyph list gains the AGL combining-mark names.
+- `readPdf` enforces the byte cap before pdf.js parses the file. Files that
+  carry only an owner password (print/copy restrictions) open without one.
+- Fonts: a placeholder reuses a simple font only through its encoding (glyph
+  names), never through ToUnicode, and only for codes whose ToUnicode entry
+  agrees with the glyph, so the page shows what the text layer says; composite
+  placeholders take the code byte length from the encoding CMap (Identity-H:
+  2), not from the ToUnicode codespace; the lowest code wins ties in the
+  reverse map (space is 32, not 160); an empty ToUnicode destination falls
+  back to the encoding; TrueType fonts without `/Encoding` default the way
+  pdf.js does (Standard when Nonsymbolic, WinAnsi without the flag, MacRoman
+  for symbolic non-embedded); a symbolic embedded font our decoder cannot
+  read is rasterized on edit (pdf.js reads it), no longer refused; direct
+  font dictionaries no longer share a cache entry; right-to-left pages are
+  compared with pdf.js as a bag of characters (pdf.js hands them back in
+  visual order).
+- Style matching: weight and slant words count only as whole words
+  ("Kobold", "Digital", "Hospital" are not bold/italic), `/StemV` is no
+  longer read as a weight (Chrome writes 130+ for regular Georgia), Computer
+  Modern / Nimbus / Cascadia names map to their families.
+- Verifier: the removed-code-sequence pass only checks the complete code
+  sequence of a whole value (a fragment of a value split over two shows
+  legitimately occurs elsewhere) and deduplicates sequences; layer zero and
+  the placement planner are linear in the number of replacements (a 200-page
+  file with 32 000 replacements writes in under 2 s instead of 90 s).
+- Layout: text pushed right by an overflowing placeholder stops before the
+  next column (the placeholder gives up its size floors rather than overlap);
+  a continuation line that starts with the tail of a wrapped value closes up
+  as a whole, justified word gaps included.
+- `fitPlaceholder` rejects non-finite input and clamps a policy that would
+  enlarge the placeholder.
+
+### Third review (T220, same release)
+
+- Text that lives in tiling pattern cells and ExtGState soft-mask groups is
+  walked like form XObjects: it reaches detection, is edited in place, and
+  never shares a line with page text (it was invisible to both extractors).
+- Glyphs are kept in reading order along the baseline: a value shown back to
+  front, or words drawn from the right, reads as "Jan Kowalski" in the text a
+  detector sees; overlapping glyphs keep content order.
+- The pdf.js cross-check no longer replaces our text when pdf.js read less
+  (a font it rejects, pattern text) or misread a font we decoded completely;
+  such pages keep our text and are rasterized on edit with our glyph
+  geometry for the black boxes (pdf.js item widths can be wrong). A page
+  whose only reading comes from pdf.js as glyph ids, controls, U+FFFD or
+  private-use characters is unredactable instead of "fine".
+- Fonts: a placeholder never reuses a code the font advances by zero;
+  /Differences names without metrics (/.notdef, /gNN) advance like the base
+  encoding's glyph and blank glyphs are not "unreadable"; symbolic embedded
+  TrueType fonts without /Encoding decode through MacRoman (pdf.js); glyph
+  text is folded at decode time (fi -> fi, Kangxi radicals -> ideographs).
+- Matching: layer zero, the verifier's text passes and its byte passes share
+  one search (whitespace, hyphens and ligatures ignored, case folded) that
+  requires token boundaries: "Nowak" is not "Nowakowski", "kid" is not /Kids;
+  a value hyphenated at a line end is still found. The verifier also scans
+  name objects and, for multi-word or numeric values, font programs.
+- Layout: two placeholders on one line, or a value touching another in one
+  show, each get their placeholder and their shift; consecutive TJ
+  adjustments add up; the room before the next column accounts for shifts
+  already planned; a placeholder never goes below 4 pt (a warning reports an
+  unavoidable overlap); text on a rotated baseline crossing a line no longer
+  drags that line's cells.
+- Output hygiene: XMP on any copied object, /PieceInfo, font descriptor
+  family/charset strings and layer names are removed from the output (and
+  the orphaned objects from the file); ToUnicode maps are rewritten for every
+  edited font, including fonts whose edited shows were deleted whole.
+- Files with only an owner password open; Type3 fonts never take standard-14
+  metrics.
+
+### Fourth review (T221, same release)
+
+- Randomized invariants (`tests/pdf-fuzz.test.ts`, seeded generator of
+  layouts with out-of-order text, kerning, wrapped and touching values,
+  ligatures, diacritics, fallback faces; 2000-document campaigns) found and
+  fixed: narrow glyphs drawn back to front filed after their neighbour;
+  shifts inside a TJ applied by array order instead of position (a jump-back
+  adjustment moved the wrong text, even off the page); overlapping runs
+  losing their word separator; two values touching in one show
+  over-estimating the room before the next column; layer zero handling a
+  shorter value before a longer one that contains it (the surname stayed);
+  verifier byte passes counting a Latin-1 letter (ó, ü) or a kerning number
+  as a token boundary ("Nowak" refused because of "Nowaków"); the
+  code-sequence pass without boundaries (now single-byte codes at token
+  boundaries only).
+- Second real-world corpus (44 files: Acrobat, iText, Aspose, Scribus,
+  TCPDF, wkhtmltopdf, pdfTeX/xdvipdfmx, LibreOffice/OpenOffice, ArcGIS,
+  PDF 2.0 samples): pdfTeX Type1 fonts without /Encoding are decoded through
+  the font program's cleartext encoding (LaTeX documents are edited in place
+  instead of rasterized); a rasterized vertical-writing page is covered whole
+  (the value stayed legible before); the verifier scans only string operands
+  of content streams (a house number equal to a font size no longer refuses
+  the export), recognises font programs by their magic bytes (TrueType table
+  tags in the fallback subset are not values), and ignores a value that only
+  survives inside its own placeholder ("We" in "[WE]"); raster boxes on
+  pdf.js-read pages cover the value plus one glyph each side instead of the
+  whole item; scanned pages with a hidden OCR layer are described as such;
+  consecutive unredactable pages collapse into one entry ("pages 2-22").
+- Robustness (byte mutations, hostile structures): page-tree errors and
+  every pdf.js failure surface as typed errors; a page pdf.js cannot read
+  fails verification closed; rasterization failures are typed; replacement
+  ranges outside the text are refused instead of clamped to the whole
+  document; caps on decoded content per page (32 MB), form XObject draws per
+  page (20 000) and glyphs per document (1.5 million, runs weighted) turn
+  Flate bombs and exponential XObject graphs into `too-large` or a refused
+  page, and pdf.js is never asked about such a page; every pdf.js page
+  operation has a 60 s deadline (effective where pdf.js runs in a worker);
+  layer zero, the placement planner (baseline index) and the byte scan
+  (native string search) are linear again for thousands of replacements.
+- Verifier: pages both extractors read identically are settled by our own
+  pass; pdf.js is consulted only for pages with issues (its spacing
+  heuristics reported "Jan" inside a kerned "Janusz"), and its items are read
+  in baseline order (words drawn from the right no longer spell a value); the
+  removed-code-sequence pass runs only when our extractor cannot re-read the
+  output.
+- pdf.js never registers document fonts with the browser's FontFace API:
+  reading a 100-font file in Chromium dropped from 4.2 s to 0.8 s and the
+  export is no longer dominated by font round trips.
+- Web: a damaged PDF gets its own message (the docx one told users to open
+  the file in Word); a failed upload while another file is loaded shows its
+  message.
+
+### Tests
+
+- 15 synthetic PDF fixtures (standard-14, Type0 Liberation, TJ kerning,
+  split shows, short names, form XObject, hidden text, ActualText,
+  annotations, AcroForm, outlines/Info/XMP, embedded file, incremental
+  update, object streams, rotated page with rise/Tz/Tw), the desktop's 10
+  standard-14 fixtures and a Chrome/Skia Type0 fixture; `assertNoTrace`
+  understands PDF (inflated streams, object streams, strings, UTF-16 BE).
+- Fixture zips no longer carry implicit folder entries: JSZip stamped them
+  with the current time, so the "is deterministic" check failed when two
+  builds straddled a 2-second boundary (seen once under full-suite load).
+
 ## [0.12.1] - 2026-09-24
 
 First run of the real-file corpus job (LibreOffice-generated docx, doc and

package/NOTICE

@@ -3,3 +3,8 @@ Copyright 2026 Witold Lojek
 
 This product includes software developed as part of the DocCloak project
 (https://github.com/WLojek).
+
+This product bundles the Liberation Fonts (fonts/liberation/), Copyright 2012
+Red Hat, Inc., licensed under the SIL Open Font License, Version 1.1
+(see fonts/liberation/LICENSE). The fonts are embedded as subsets into
+redacted PDF output; the Reserved Font Name "Liberation" is kept.

package/README.md

@@ -59,7 +59,10 @@ ONNX Runtime also needs its WASM binaries at runtime; point
 ## API overview
 
 - **Engine** - `createEngine(env, initialSettings?, options?)` returns a
-  `DocCloakEngine`: `detect(text, signal?, onProgress?)`, `preload()`,
+  `DocCloakEngine`: `detect(text, signal?, onProgress?)` (an aborted
+  `signal` stops the model at the next inference chunk and rejects with
+  `DetectionAbortedError`; over the worker protocol the client sends
+  `cancelDetect` and rejects once the host has stopped), `preload()`,
   `switchProvider(id)`, `getSettings()` / `updateSettings(patch)`,
   `onDownloadProgress` / `onDetectionProgress`, `release()`. Settings persist
   through `env.kv`. `options.autoLoad` (default `true`) lets `detect()`
@@ -165,6 +168,89 @@ streams: { data, objectPool, macros } }` without throwing, so a host can show
 the same consent card for a .doc whose `ObjectPool` (embedded objects) or
 `Macros` (VBA) storage will be copied unchanged.
 
+## PDF: redaction that keeps the text layer
+
+`@doccloak/core/pdf` redacts PDFs without rasterizing them. The glyphs of a
+detected value are cut out of the content stream; the placeholder is written
+back as real text, in the document's own font whenever that font can show it
+(every character must map to a code whose glyph is proven present), else in
+a metrics-compatible Liberation face (Sans/Serif/Mono, matched by the font's
+style flags and name) embedded as a subset. Everything else on the page
+stays live text.
+
+```ts
+import { readPdf, writeAnonymizedPdfWithReport } from '@doccloak/core/pdf';
+
+const assets = {
+  pdfjsWorkerUrl: '/pdf/pdf.worker.mjs',       // node_modules/pdfjs-dist/legacy/build/pdf.worker.mjs
+  cMapUrl: '/pdf/cmaps/',                       // node_modules/pdfjs-dist/cmaps/
+  standardFontDataUrl: '/pdf/standard_fonts/',  // node_modules/pdfjs-dist/standard_fonts/
+  fontsUrl: '/pdf/fonts/',                      // node_modules/@doccloak/core/fonts/liberation/
+};
+const extraction = await readPdf(file, { assets });     // extraction.plainText -> detect
+const entities = await engine.detect(extraction.plainText);
+const replacements = entities.map((e) => ({ start: e.start, end: e.end, replacement: session.anonymize(e.value, e.type) }));
+const { blob, warnings, removed, rasterizedPages } = await writeAnonymizedPdfWithReport(
+  extraction, replacements, session.getEntries(), { assets },
+);
+```
+
+Or go through `redactOfficeFile(file, { session, detect, pdf: { assets } })`,
+which now accepts `.pdf` next to `.docx` and `.xlsx`.
+
+**Placeholder fitting.** A placeholder rarely has the width of what it
+replaces. The writer keeps the original size when the placeholder fits the
+gap. Otherwise it first moves the rest of the line right, as far as the next
+column and the page's text edge (the farthest line end on the page, a proxy
+for the right margin) allow; only what still does not fit is shrunk down to
+75 % of the size, then condensed down to 80 % of its width (`Tz`), so a
+short word on a left-aligned line gets a full-size placeholder and a line
+that already reaches the margin (right-aligned, justified, a table cell)
+gets a smaller one (`fit` option: `minSizeRatio`, `minHorizontalScale`,
+`allowOverflow`). The page's text edge is a hard limit (Chrome and Word clip
+their content boxes, so text pushed past the margin would be cut); against
+it, as against a column, the placeholder gives up its floors (never below
+4 pt). Of a value wrapped over several lines, the widest part carries the
+placeholder and the other parts are excised, so "ul." at a line end never
+has to hold "[ADDRESS_1]". A placeholder narrower than the gap closes the slack, so
+"[PERSON_1], PESEL [ID_1], ..." reads without holes. Text on other lines and
+text separated by a tab-sized gap on the same line never moves. Every other
+occurrence of a session value is redacted too (layer zero, as for docx),
+matched with whitespace, soft hyphens and ligatures ignored: a name wrapped
+over a line end gets its placeholder on the first line, and the next line
+closes up over the removed part.
+
+**Safety.** Every show after an edited one in the same positioning scope is
+re-anchored with an absolute text matrix computed from the original
+geometry, so the surviving glyphs cannot encode what was removed. Text in
+form XObjects, tiling pattern cells and soft-mask groups is walked too, and
+glyphs are read in baseline order whatever the content order. Two
+independent extractors read every page (our decoder and pdf.js): a page
+whose text differs, or whose fonts cannot be read safely (CJK predefined
+CMaps, unreadable ToUnicode, vertical writing, unparsable content), is rasterized
+when it carries a redaction and blacked out in the pixels; such pages are
+listed in `rasterizedPages`. After writing, the output is re-read by both
+extractors and byte-scanned (every decoded stream and string, UTF-8, Latin-1,
+UTF-16 BE/LE, plus the removed code sequences); any trace throws
+`PdfVerifyError` and nothing is exported.
+
+**What is removed.** The output is rebuilt as a fresh single-revision file:
+document properties (`/Info`), XMP, bookmarks, interactive forms,
+annotations (comments, links, widgets), attachments, JavaScript, named
+destinations, page labels and the tagged-PDF structure tree are dropped and
+listed in `removed`; the input's incremental-update history cannot survive.
+ToUnicode maps of edited fonts are trimmed to the codes still used,
+`ActualText`/`Alt` marked-content properties are stripped, dates are set to
+2000-01-01 and the producer to `DocCloak`. Optional-content (layer)
+configuration is dropped as well, so content of hidden layers becomes
+visible (it is redacted like everything else).
+
+**Refusals.** Encrypted files (`'encrypted'`, pass `password`), files over
+50 MB or 500 pages (`'too-large'`), non-PDF bytes (`'invalid-package'`).
+Scanned pages (image without a text layer) and text no decoder can read are
+reported in `unredactable` (`'scanned-page'`, `'undecodable-text'`) and gate
+the export behind `allowUnredactable`, like Office packages.
+
 ## What is removed or normalised in Office packages
 
 Independently of what was detected, every exported .docx / .xlsx has:
@@ -200,6 +286,41 @@ Independently of what was detected, every exported .docx / .xlsx has:
 
 ## Known limits
 
+**PDF**
+
+- The width of a redacted region stays observable: the placeholder (or the
+  closed slack) tells how wide the original text was. This is inherent to
+  every layout-preserving redaction; only rasterizing or re-typesetting the
+  paragraph would hide it.
+- Subset fonts keep their glyph repertoire (which characters occurred
+  somewhere in the document), not the removed text.
+- Pages with predefined CJK CMaps, unreadable ToUnicode maps, vertical
+  writing or text our decoder cannot confirm against pdf.js are rasterized
+  when they carry a redaction (their text layer is gone on that page).
+  Type3 fonts are edited like other simple fonts (their text comes from the
+  encoding names); a placeholder is never drawn with a Type3 font unless the
+  font itself showed every character of it. Digital signatures are
+  invalidated by any change.
+- Images are copied unchanged; text inside pictures is not OCR-scanned.
+- Forms, annotations, bookmarks and attachments are dropped, not redacted.
+- Right-to-left text: pdf.js reports it in visual order, so the cross-check
+  compares the characters of such pages as a set; detection runs on our
+  logical-order text.
+- Placeholders are set in the document's font or a Liberation face: a
+  placeholder character neither can show (CJK, emoji) is written as `?`
+  and reported in `warnings`.
+- A page whose text only pdf.js can read, and only as glyph ids, controls or
+  replacement characters (fonts without a usable encoding or ToUnicode), is
+  reported as unredactable: no detector could find a value in it.
+- Limits: 50 MB, 500 pages, 1.5 million glyphs (`maxBytes`, `maxPages`,
+  `maxGlyphs`), 32 MB of decoded content and 20 000 form draws per page.
+  Memory: the glyph model of the whole file is held during write and verify
+  (a 200-page text-heavy file peaks around 0.8 GB of heap in Node).
+- A page whose content would make a viewer work forever (an exponential
+  chain of form XObjects, a decompression bomb) is never handed to pdf.js and
+  cannot be rasterized; pdf.js page operations time out after 60 s where it
+  runs in a worker (in Node's fake worker a hostile file can still block).
+
 **Office packages (.docx, .xlsx)**
 
 - Images inside documents (`media/*`) are copied unchanged; text in pictures
@@ -207,8 +328,9 @@ Independently of what was detected, every exported .docx / .xlsx has:
 - Embedded objects (`embeddings/*`), VBA projects, `altChunk` HTML fragments,
   external data connections, external links and printer settings cannot be
   redacted. They are reported in `extraction.unredactable` and exported
-  verbatim only with `allowUnredactable: true` (see above). Removing them is
-  planned for 0.13.0 (T199), gated on the real-file corpus (T196).
+  verbatim only with `allowUnredactable: true` (see above). Removing them
+  (instead of copying them) is planned for a later release (T199), gated on
+  the real-file corpus (T196); 0.13.0 keeps the report-and-ask behaviour.
 - Spreadsheets: sheet names with an entity are reported in `warnings`, not
   renamed; numeric and date cells, formula literals, defined names and data
   validation texts are not extracted yet (T176).
@@ -268,7 +390,7 @@ interface CoreEnv {
   fetch: typeof fetch;            // injectable for tests/proxies
   wasm: { paths: string; numThreads?: number };  // onnxruntime-web asset location
   buildTokenizer?(tokenizerJson: unknown, tokenizerConfig: unknown): unknown | Promise<unknown>;
-  /** @deprecated since 0.12.0, removed in 0.13.0; used only when buildTokenizer is absent */
+  /** @deprecated since 0.12.0, removal planned for 0.14.0; used only when buildTokenizer is absent */
   loadTokenizer?(hfModelId: string): Promise<unknown>;
   hardware?: HardwareHints;       // deviceMemoryGB / isMobile, feeds provider default
   persistStorage?: () => Promise<boolean>;  // e.g. navigator.storage.persist()
@@ -302,11 +424,11 @@ With `buildTokenizer` the host never contacts the network for tokenizers, a
 warm `modelCache` works offline, and the only URLs fetched on a cold start
 are the three pinned `resolve/<commit>/...` files of the active provider.
 
-`loadTokenizer(hfModelId)` (0.11 and earlier) is still honoured when
-`buildTokenizer` is absent, with a one-time deprecation warning. It lets the
-host library probe the mutable `resolve/main` branch and keep an unverified
-second cache, so migrate before 0.13.0, where `buildTokenizer` becomes
-required and `loadTokenizer` is removed.
+`loadTokenizer(hfModelId)` (0.11 and earlier) is still honoured in 0.13.0
+when `buildTokenizer` is absent, with a one-time deprecation warning. It
+lets the host library probe the mutable `resolve/main` branch and keep an
+unverified second cache, so migrate now: the removal is planned for 0.14.0,
+where `buildTokenizer` becomes required.
 
 ### autoLoad and ModelNotLoadedError
 

package/dist/dom/errors.d.ts

@@ -29,7 +29,7 @@ export declare function isUnsupportedDocumentError(err: unknown): err is Unsuppo
 export interface UnredactablePart {
     /** Zip path of the part, e.g. 'word/embeddings/oleObject1.bin' */
     part: string;
-    kind: 'embedded-object' | 'macros' | 'html-chunk' | 'external-data' | 'printer-settings' | 'unknown';
+    kind: 'embedded-object' | 'macros' | 'html-chunk' | 'external-data' | 'printer-settings' | 'unknown' | 'scanned-page' | 'undecodable-text';
     /** Human-readable label for host UIs, e.g. 'embedded Excel sheet' */
     label: string;
 }

package/dist/dom/ocr.d.ts

@@ -85,22 +85,48 @@ interface BlockLike {
  * Exported for tests.
  */
 export declare function buildTextFromBlocks(blocks: BlockLike[]): OcrExtraction;
+type PixelBox = {
+    x0: number;
+    y0: number;
+    x1: number;
+    y1: number;
+};
+/**
+ * For each pair of consecutive OCR words, whether they are really one token
+ * that OCR split by dropping a character. Tesseract reads the dot in
+ * "t.zielinski@firma.pl" as a space and returns "t" + "zielinski@firma.pl",
+ * with the dot's pixels inside the second box; the e-mail rule then matches
+ * only the second word, and without this the "t" would stay visible. Such a
+ * split leaves a sliver between the boxes, far narrower than a real space: a
+ * pair on the same line is one token when its gap is at most
+ * TIGHT_GAP_SHARE of the line's typical word gap (boxes that touch or
+ * overlap included). Boxes much taller than the line (MAX_JOIN_HEIGHT_RATIO)
+ * are OCR noise and never join, so the fix cannot paint a bar across
+ * neighbouring lines.
+ *
+ * Not covered: a lost glyph that stays in a normal-width gap (seen only on
+ * low-resolution images, around 120 dpi). Telling it from a glyph edge that
+ * Tesseract cut off its own box needs pixels and misfired on real scans
+ * ("w terminie", "nr wpisu"), so gap width alone decides.
+ *
+ * Returns an array of length words.length - 1: entry i joins words i and i+1.
+ * Exported for tests.
+ */
+export declare function findSplitTokens(words: OcrWord[]): boolean[];
 /**
  * Pick the bounding boxes of every OCR word that overlaps one of the given
  * character ranges. A word partially covered by a range is fully redacted
- * (over-redaction is safer than leaking half an identifier).
+ * (over-redaction is safer than leaking half an identifier). A selected
+ * word also pulls in the same-line neighbours OCR split off by dropping a
+ * character (findSplitTokens), so "t" + "zielinski@firma.pl" is redacted
+ * as one token.
  *
  * Exported for tests.
  */
 export declare function selectRedactionBoxes(words: OcrWord[], ranges: {
     start: number;
     end: number;
-}[]): {
-    x0: number;
-    y0: number;
-    x1: number;
-    y1: number;
-}[];
+}[]): PixelBox[];
 /**
  * Draw black boxes over the given character ranges and return the redacted
  * image as a PNG blob.