@doccloak/core 0.11.0 → 0.12.0

package/CHANGELOG.md

@@ -4,6 +4,125 @@ All notable changes to `@doccloak/core` are documented here. The format is
 based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the
 project follows [Semantic Versioning](https://semver.org/).
 
+## [0.12.0] - 2026-09-24
+
+Security remediation release (audit `SECURITY_REPORT-2026-09.md`, plan
+`documentation/security-remediation-plan-2026-09-v2.md`). Every change is
+fail-closed: what Core cannot redact is refused or reported, never copied in
+silence.
+
+### Breaking for hosts
+
+- **`CoreEnv.buildTokenizer(tokenizerJson, tokenizerConfig)` (T185):**
+  providers now download `tokenizer.json` and `tokenizer_config.json` from
+  their pinned commit through `fetchModelBlob` (SHA-256 and size verified,
+  cached in `doccloak-models`, resumable) and hand the parsed JSON to the
+  host, which constructs the tokenizer (`new PreTrainedTokenizer(json,
+  config)` on the web). `CoreEnv.loadTokenizer(hfModelId)` still works in
+  0.12.0 as a deprecated fallback (one console warning) and is removed in
+  0.13.0. The `AutoTokenizer.from_pretrained` probe to `resolve/main` on
+  every start is gone, so a fully cached model loads with zero requests.
+- **`redactOfficeFile` throws `StaleAnalysisError` (T179)** when an entity
+  no longer matches the fresh extraction, instead of silently dropping it
+  and exporting the value. Pass `onMismatch: 'drop'` for the old behaviour.
+- **Files with unredactable parts are refused by the writers (T177)** unless
+  `allowUnredactable: true` is passed (`writeAnonymizedDocx*`,
+  `writeAnonymizedXlsx*`, `redactOfficeFile`). `readDocx` / `readXlsx`
+  report them in `extraction.unredactable` (`part`, `kind`, `label`) so a
+  host can ask the user first; when allowed, `result.warnings` names every
+  part copied verbatim.
+
+### Added
+
+- **Package-wide value scrub, "layer zero" (T172):** after the offset
+  replacements, every known session value (entries, variants, name tokens)
+  is replaced in every text node and attribute of every XML part of the
+  package (`charts`, `diagrams`, `customXml`, `pivotCache`, `docProps`,
+  rels and more), with a safety rule that never touches identifiers,
+  geometry or layout attributes. `writeAnonymizedDocxWithReport` /
+  `writeAnonymizedXlsxWithReport` return `{ blob, warnings }`.
+- **New text parts and units (T174, T175):** charts (`c:v` caches, titles),
+  SmartArt (`dgm:t`), DrawingML text (`a:t`), `customXml/item*.xml` leaf
+  text, the whole `word/glossary/**` set, `numbering.xml` level text,
+  `webSettings.xml` frame titles, `w:delInstrText`, OMML `m:t`; attribute
+  units for field instruction arguments, content-control aliases, drawing
+  alt text and bookmark names (renamed bookmarks keep `REF`, `PAGEREF`,
+  `NOTEREF`, `HYPERLINK \l` and anchors valid). Tabs, breaks and
+  non-breaking hyphens are emitted as separators, text boxes are separated
+  from their host paragraph, `mc:Fallback` twins are redacted too, and a
+  replacement never crosses a paragraph break.
+- **Part policy (T177):** `classifyPart` tables for docx and xlsx (text /
+  structural / unredactable / unknown), `describeUnredactableParts`,
+  200 MB unpacked-size guard (`UnsupportedDocumentError` code
+  `'too-large'`).
+- **`UnsupportedDocumentError`** with codes `unrecognized-namespace`,
+  `invalid-package`, `unredactable-parts`, `too-large`, `fast-saved`,
+  `encrypted`; `extraction.empty`, `extraction.warnings`.
+- **Engine (T185):** `createEngine(env, factories, { autoLoad: false })` makes
+  `detect()` throw `ModelNotLoadedError` until the host calls `preload()`
+  (hosts implement consent above Core); `serveEngine` validates every
+  message and answers unknown or malformed ones with an `error` frame;
+  verification markers carry `sha256:size` and are re-checked against the
+  cached blob size.
+- **Session (T182, T183):** `registerDocumentValues`, `lastAmbiguous()` /
+  `getAmbiguousValues()`; surrogates are never a value (or a name token)
+  that occurs in the document; single-pass exact restore with script-aware
+  word boundaries; literal `[TYPE_N]` tokens in the input never collide
+  with generated ones.
+- **Tests:** `assertNoTrace` helper scanning every part and the raw archive
+  in UTF-8, latin1 and UTF-16LE, 33 synthetic Office / .doc fixtures, a
+  real-file corpus generated by LibreOffice in CI with an independent
+  extraction check, a 30-document end-to-end session suite, adversarial
+  regex inputs.
+
+### Changed
+
+- Regex rules are enabled by default (`regexEnabled` defaults to `true` when neither the host nor a saved setting says otherwise). Structured identifiers no longer depend on the model alone in a fresh profile (T203).
+- ML spans are trimmed of surrounding punctuation before merging: a whitespace-split provider such as BardS.ai returned `Anna Nowak,` and the redacted text lost the comma (T204).
+
+- **Strict OOXML is supported (T172):** transitional and
+  `purl.oclc.org` namespaces are both read; a main part without a
+  recognised body is refused (`unrecognized-namespace`) instead of being
+  exported untouched.
+- **Regex detection runs per line (T180)** (rules with `multiline: true`
+  run on the whole text), which removes the multi-minute stalls on long
+  prose; postal/city rules end on a letter and use a Unicode look-ahead
+  instead of `\b` (T181).
+- **False-positive filter (T181):** CJK / Thai values of 2 characters and
+  Latin PERSON values of 2 characters (`Li`, `Ng`, `J.`) are kept; a short
+  stoplist drops titles (`Mr`, `Dr`, `Jr`...). Propagation uses Unicode
+  boundaries, exact offsets after `İ`, exact case for terms of 2
+  characters, and skips lowercase hits of common-word names (`bill`,
+  `mark`, `will`...).
+- **Restore (T182):** `renameLabel` rejects empty and colliding labels;
+  `importEntries` / `deserialize` validate every entry and cap at 10 000;
+  `personTokens` is linear. Surrogate pools no longer contain common
+  English words or names of 3 letters or fewer.
+- **Metadata (T179):** `contentStatus`, `identifier`, `created`,
+  `modified`, `lastPrinted`, `revision`, `version`, document statistics,
+  `TitlesOfParts` / `HeadingPairs` are blanked, zeroed or normalised;
+  comment and revision dates are set to 2000-01-01; `w:rsid*` attributes
+  are removed; ZIP entries are dated 1980-01-01 with no comments.
+- **Always scrubbed without detection (T174):** hyperlink tooltips,
+  `mailto:` relationship targets, `w:docVars`, content-control tags without
+  data binding.
+- **Legacy .doc hardening (T178, security report 2026-09 H5, H6, M5, L2):**
+  `readDocText` and `writeAnonymizedDoc` now throw `UnsupportedDocumentError`
+  (`code: 'fast-saved'`) for incrementally saved files (`fComplex`, or
+  `cQuickSaves > 0` on Word 97 files) and (`code: 'encrypted'`) for
+  `fEncrypted` / `fObfuscated` files instead of returning partial or
+  ciphertext output. `writeAnonymizedDoc` takes an optional
+  `valueReplacements` list (same shape as `writeAnonymizedDocx`) and destroys
+  every remaining byte occurrence of each original value and its 4+ character
+  tokens, any case, in `WordDocument` (UTF-16LE), `Table` (UTF-16LE and
+  cp1252) and `Data` (UTF-16LE), length-preserving. `SttbfRMark`,
+  `GrpXstAtnOwners`, `SttbfBkmk`, `SttbSavedBy` and `AutosaveSource` are
+  scrubbed alongside `SttbfAssoc`. New `inspectDoc(buffer)` reports the
+  refusal flags and the presence of `Data`, `ObjectPool` and `Macros`
+  streams (the latter two are still copied through; see README "Known
+  limits: legacy .doc"). `UnsupportedDocumentError` is also exported from
+  the main entry.
+
 ## [0.11.0] - 2026-09-23
 
 ### Changed
@@ -98,13 +217,13 @@ project follows [Semantic Versioning](https://semver.org/).
   PEM private-key blocks, JWTs, connection strings with embedded
   credentials, `secret=`-style assignments and high-entropy tokens, and
   a GB driving-licence rule. Credentials never receive realistic
-  stand-ins — even in surrogate mode they always fall back to typed
+  stand-ins, even in surrogate mode they always fall back to typed
   placeholders, since a same-shape fake key still reads as a live
   credential.
 
 - **Tolerant restore-token matching (T098):** new side-effect-free
   `@doccloak/core/restore-tokens` export that restores placeholders LLM
-  replies have mangled — case changes (`[person_1]`), separator swaps
+  replies have mangled, case changes (`[person_1]`), separator swaps
   (`[PERSON 1]`), markdown wrapping (`**[PERSON_1]**`), dropped brackets
   and stray edge punctuation. Restoration only happens on an unambiguous
   canonical hit; anything ambiguous is left exactly as the model wrote

package/README.md

@@ -38,14 +38,19 @@ the engine downloads them from Hugging Face and stores them in the
 `CoreEnv.modelCache` you provide (the browser Cache Storage in the reference
 hosts):
 
-| Provider | Model | Download |
+| Provider id | Model | Download |
 | --- | --- | --- |
-| `gliner` | GLiNER PII Edge (multi-language, custom labels) | ~65 MB |
-| `bardsai` | BardS.ai EU PII (24 EU languages, 35 PII types) | ~279 MB |
-
-`pickDefaultProvider(hardware)` chooses `gliner` on mobile or low-memory
-devices and `bardsai` otherwise. Download progress is surfaced via
-`engine.onDownloadProgress(cb)`.
+| `gliner` | GLiNER PII Small (multi-language, custom labels) | ~83 MB |
+| `gliner-base` | GLiNER PII Base (often better for English and dates, custom labels) | ~197 MB |
+| `bardsai` | BardS.ai EU PII (24+ EU languages, 35 PII types; labeled "Legacy" in the web app) | ~279 MB |
+
+`pickDefaultProvider(hardware)` chooses `gliner` on mobile devices or when
+`deviceMemoryGB` is 4 or less, and `bardsai` otherwise. Download progress is
+surfaced via `engine.onDownloadProgress(cb)`. Every model and both of its
+tokenizer files are fetched from an immutable `resolve/<commit>` URL and
+verified against a pinned SHA-256 and byte size; the pins are listed in
+[documentation/model-provenance.md](../documentation/model-provenance.md)
+of the DocCloak workspace.
 
 ONNX Runtime also needs its WASM binaries at runtime; point
 `CoreEnv.wasm.paths` at wherever your host serves the `onnxruntime-web`
@@ -57,23 +62,198 @@ ONNX Runtime also needs its WASM binaries at runtime; point
   `DocCloakEngine`: `detect(text, signal?, onProgress?)`, `preload()`,
   `switchProvider(id)`, `getSettings()` / `updateSettings(patch)`,
   `onDownloadProgress` / `onDetectionProgress`, `release()`. Settings persist
-  through `env.kv`.
+  through `env.kv`. `options.autoLoad` (default `true`) lets `detect()`
+  download and initialise the model on its own; with `autoLoad: false` the
+  engine never downloads implicitly: `detect()` before a completed
+  `preload()` (or `switchProvider()`) rejects with `ModelNotLoadedError`
+  (a preload still in flight is awaited). Use `false` when model downloads
+  are gated behind user consent and call `preload()` from the consent flow.
 - **Detection pipeline** - `detectEntities`, `detectWithRegex`,
   `resolveOverlaps`, `filterFalsePositives`, `propagateEntities` for hosts
   that compose their own flow; `ALL_REGEX_RULES` and `REGEX_REGIONS` describe
   the built-in regex rule set (universal + per-country rules).
 - **Session state** - `AnonymizationSession` maps detected entities to stable
-  placeholders (labeled or blanked) and back.
-- **Documents** (`/dom`) - `readDocx` / `writeAnonymizedDocx`, legacy `.doc`
-  text extraction (`readDocText` on the main entry), OCR helpers
+  placeholders (labeled, blanked or surrogate) and back. In surrogate mode
+  `registerDocumentValues(valuesOrEntities)` tells the session which values
+  occur in the document so no surrogate ever equals a real value from it
+  (`anonymizeText` does this automatically); `getAmbiguousValues()` lists the
+  values whose person-variant match was a tie and therefore got a new
+  placeholder instead of being merged with an existing person.
+- **Documents** (`/dom`) - `readDocx` / `writeAnonymizedDocx` /
+  `writeAnonymizedDocxWithReport`, `readXlsx` / `writeAnonymizedXlsx`,
+  `analyzeOfficeFile` / `redactOfficeFile`, legacy `.doc` support on the main
+  entry (`readDocText`, `writeAnonymizedDoc`, `inspectDoc`), OCR helpers
   (`loadImageToCanvas`, `recognizeCanvas`, `renderRedactedImage`) built on
-  tesseract.js.
+  tesseract.js. See "Document API: refusals, warnings and consent" below for
+  the error and result shapes.
 - **Worker protocol** (`/worker-protocol`) - `serveEngine(engine, port)` on
   the side that owns the engine, `connectEngine(port)` on the UI side.
   `PortLike` is satisfied by thin wrappers over `Worker`, `MessagePort` or
   `chrome.runtime.Port`.
 - **Model loader** - `fetchModelBlob` / `evictModelFromCache` implement
-  cache-first model downloads with progress and retry (`retryAsync`).
+  cache-first model downloads with progress, resume, retry and SHA-256
+  verification (`ModelIntegrityError` on a mismatch; the entry is evicted).
+  Verified cache entries carry a marker `${sha256}:${size}` bound to the blob
+  size, so a cache hit whose size changed is re-hashed instead of trusted.
+  Each provider also pins its two tokenizer files (`GLINER_TOKENIZER_FILES`,
+  `GLINER_BASE_TOKENIZER_FILES`, `BARDSAI_TOKENIZER_FILES`: immutable
+  `resolve/<commit>` URL, sha256, size); `loadPinnedTokenizer` fetches them
+  through the same path and hands the parsed JSON to `env.buildTokenizer`.
+
+### Document API: refusals, warnings and consent
+
+The readers and writers fail closed: a file the engine cannot redact
+completely is refused with a typed error, or reported so the host can ask the
+user, never exported under a false "redacted" label.
+
+**`UnsupportedDocumentError`** (main entry and `/dom`) carries a stable
+`code` and a `details: string[]` list for diagnostics. Map `code` to your own
+localized message with a one-line remedy:
+
+| `code` | Thrown by | Meaning | Remedy to show |
+| --- | --- | --- | --- |
+| `'unrecognized-namespace'` | `readDocx`, `readXlsx` | The main part uses an XML namespace the reader does not know (neither the transitional nor the strict OOXML set), so text could be missed | Open in Word/Excel and save again as .docx/.xlsx |
+| `'invalid-package'` | `readDocx`, `readXlsx` | The zip or a required XML part cannot be parsed | Same |
+| `'too-large'` | `readDocx`, `readXlsx` | The unpacked size of the package exceeds 200 MB (zip-bomb and memory guard) | Split the document |
+| `'unredactable-parts'` | `writeAnonymizedDocx*`, `writeAnonymizedXlsx*`, `redactOfficeFile` | The package contains parts whose content cannot be redacted and the host did not pass `allowUnredactable: true` | Ask the user (see below) |
+| `'fast-saved'` | `readDocText`, `writeAnonymizedDoc` | Legacy .doc saved incrementally: deleted text lives outside the piece table where no detector sees it | Open in Word, Save As (or save as .docx) |
+| `'encrypted'` | `readDocText`, `writeAnonymizedDoc` | Legacy .doc is encrypted or XOR-obfuscated | Remove the password |
+
+**Extraction results.** `readDocx` and `readXlsx` return, next to
+`plainText` and the offset mappings:
+
+- `unredactable: UnredactablePart[]` - parts the writer cannot redact, each
+  with `part` (zip path), `kind` (`'embedded-object' | 'macros' |
+  'html-chunk' | 'external-data' | 'printer-settings' | 'unknown'`) and a
+  human `label` ("embedded Excel sheet"). The reader never throws for these;
+  the writer refuses with `'unredactable-parts'` unless the host passes
+  `{ allowUnredactable: true }`, in which case those parts are copied
+  verbatim and one warning per part is returned. Only pass it after the user
+  has been told which parts stay unchanged (informed consent); a host that
+  does not ask keeps the fail-closed default. Layer zero still scrubs every
+  known value from the XML parts of the package either way.
+- `warnings: string[]` - parts that could not be parsed and were therefore
+  skipped by the layer-zero scrub, and similar non-fatal findings (a sheet
+  name containing an entity, for example). Show them to the user.
+- `empty: boolean` - the document contains no text units at all; show
+  "nothing to redact" instead of exporting.
+
+**Write results.** `writeAnonymizedDocxWithReport` /
+`writeAnonymizedXlsxWithReport` return `{ blob, warnings }`; the plain
+`writeAnonymizedDocx` / `writeAnonymizedXlsx` return the `Blob` only.
+
+**One-call path.** `analyzeOfficeFile(file, detect)` reads the package,
+runs your detector and exposes `unredactable[]` and `warnings[]` next to the
+entities. `redactOfficeFile(file, analysis, session, options)` applies the
+session's replacements and writes the file:
+
+- `options.onMismatch` (`'throw' | 'drop'`, default `'throw'`) decides what
+  happens when an entity of the analysis no longer matches the text at its
+  offsets (the document was re-read or edited in between): `'throw'` rejects
+  with `StaleAnalysisError`, which lists the stale entities, so a stale
+  analysis never yields a leaking "success"; `'drop'` skips those entities
+  and reports them in `result.warnings`. `isStaleAnalysisError(err)` works
+  across realms.
+- `options.allowUnredactable` (default `false`) is required to export a file
+  with embedded objects, macros, HTML chunks, external data or printer
+  settings; `result.warnings` then names each part copied verbatim.
+
+All of these are exported from `@doccloak/core/dom`.
+
+**Legacy .doc.** `inspectDoc(buffer)` returns `{ encrypted, fastSaved,
+streams: { data, objectPool, macros } }` without throwing, so a host can show
+the same consent card for a .doc whose `ObjectPool` (embedded objects) or
+`Macros` (VBA) storage will be copied unchanged.
+
+## What is removed or normalised in Office packages
+
+Independently of what was detected, every exported .docx / .xlsx has:
+
+- **Removed:** `cp:revision`, `cp:version`; every `w:rsid*` attribute in the
+  content parts and `w:rsids` in `word/settings.xml`; the thumbnail
+  (`docProps/thumbnail.*` with its relationship and content-type override);
+  the attached template, mail-merge settings and document variables
+  (`w:docVars`) in `word/settings.xml`; hyperlink tooltips; content-control
+  tags without a data binding; `c:externalData` links in charts (the chart
+  keeps its cached data).
+- **Blanked:** `creator`, `lastModifiedBy`, `title`, `subject`,
+  `description`, `keywords`, `category`, `contentStatus`, `identifier`
+  (`docProps/core.xml`); `Company`, `Manager`, `HyperlinkBase`, `Template`
+  and every heading / sheet / defined-name title in `TitlesOfParts` and
+  `HeadingPairs` (`docProps/app.xml`); custom property values (names kept,
+  `docProps/custom.xml`); the comment-author registry (`word/people.xml`)
+  and `w:author` / `w:initials` on comments and tracked changes.
+- **Normalised:** `dcterms:created`, `dcterms:modified` and `cp:lastPrinted`
+  to `2000-01-01T00:00:00Z`; `w:date` on comments, insertions, deletions and
+  property-change records to `2000-01-01T00:00:00Z` (the only visible
+  effect: comment and revision dates in the downloaded file); page, word,
+  character, line and paragraph counts and `TotalTime` to 0; every ZIP entry
+  dated `1980-01-01` with entry and archive comments removed, so the output
+  no longer reveals which parts were touched; `mailto:` relationship targets
+  replaced by `mailto:redacted@example.invalid`.
+- **Layer zero:** every value of the session (with its variants and name
+  tokens of 4+ characters) is scrubbed from the text nodes and attribute
+  values of every XML, `.rels` and `.vml` part of the package (charts,
+  diagrams, customXml, glossary, docProps, pivot caches), never touching
+  element names, `[Content_Types].xml`, relationship `Type`s, ids or numeric
+  layout attributes.
+
+## Known limits
+
+**Office packages (.docx, .xlsx)**
+
+- Images inside documents (`media/*`) are copied unchanged; text in pictures
+  is not OCR-scanned. Hosts can run the OCR helpers on extracted images.
+- Embedded objects (`embeddings/*`), VBA projects, `altChunk` HTML fragments,
+  external data connections, external links and printer settings cannot be
+  redacted. They are reported in `extraction.unredactable` and exported
+  verbatim only with `allowUnredactable: true` (see above). Removing them is
+  planned for 0.13.0 (T199), gated on the real-file corpus (T196).
+- Spreadsheets: sheet names with an entity are reported in `warnings`, not
+  renamed; numeric and date cells, formula literals, defined names and data
+  validation texts are not extracted yet (T176).
+- A bookmark rename keeps the identifier valid (`PERSON_1`, never
+  `[PERSON_1]`); field instructions are replaced only inside their quoted
+  arguments and bookmark tokens.
+
+**Legacy .doc**
+
+`writeAnonymizedDoc(buffer, replacements, valueReplacements?)` rewrites the
+piece table for main-text replacements, overwrites subdocument ranges in
+place, space-fills the OLE property sets and the Table string tables
+(`SttbfAssoc`, `SttbfRMark`, `GrpXstAtnOwners`, `SttbfBkmk`, `SttbSavedBy`,
+`AutosaveSource`) and then destroys every remaining byte occurrence of each
+original value (the value, its whitespace tokens of 4+ characters, any case)
+without moving a single structure: `WordDocument` and `Data` are scanned as
+UTF-16LE, `Table` as UTF-16LE and cp1252. What it does not do:
+
+- **Fast-saved files are refused** (`code: 'fast-saved'`; `fComplex`, or
+  `cQuickSaves > 0` on Word 97 files). Deleted text of an incremental save
+  lives outside the piece table, so detection never sees it. Ask the user to
+  open the file in Word and Save As (or save as .docx).
+- **Encrypted or XOR-obfuscated files are refused** (`code: 'encrypted'`).
+- **cp1252 remnants in `Data` and `WordDocument` are not searched.** Hyperlink
+  field data and OLE previews store text as UTF-16; a cp1252 search would hit
+  bytes inside embedded pictures and corrupt them.
+- **`ObjectPool/*` (embedded OLE objects), `Macros/*` (VBA) and `\x01CompObj`
+  are copied through unchanged.** Call `inspectDoc(buffer)` and warn when
+  `streams.objectPool` or `streams.macros` is `true`.
+- A token that also occurs inside a replacement (for example `Person` in
+  `[PERSON_1]`) is not scrubbed on its own so placeholders survive; the full
+  original value always is.
+
+**Detection and restore**
+
+- No model catches every name; hosts should keep a review step and let users
+  add manual entities or dictionary terms.
+- Regex rules run per line since 0.12.0 (except rules flagged `multiline`),
+  so a pattern split across a line break (postal code on one line, city on
+  the next) is not matched as one entity.
+- Surrogate mode preserves structure by design: name token count, gender and
+  locale; the phone country prefix; the date format and a constant per-session
+  day shift (so one known date reveals the shift for all dates); the IBAN
+  country and letter positions; the email shape. Surrogate restore matches
+  by value and is best-effort; placeholder mode is exact.
 
 ## Host integration: CoreEnv
 
@@ -84,10 +264,12 @@ engine talks to the outside world:
 ```ts
 interface CoreEnv {
   kv: KVStore;                    // async string KV for settings persistence
-  modelCache: BlobCache;          // blob cache for downloaded model files
+  modelCache: BlobCache;          // blob cache for downloaded model + tokenizer files
   fetch: typeof fetch;            // injectable for tests/proxies
   wasm: { paths: string; numThreads?: number };  // onnxruntime-web asset location
-  loadTokenizer(hfModelId: string): Promise<unknown>;  // e.g. @huggingface/transformers
+  buildTokenizer?(tokenizerJson: unknown, tokenizerConfig: unknown): unknown | Promise<unknown>;
+  /** @deprecated since 0.12.0, removed in 0.13.0; used only when buildTokenizer is absent */
+  loadTokenizer?(hfModelId: string): Promise<unknown>;
   hardware?: HardwareHints;       // deviceMemoryGB / isMobile, feeds provider default
   persistStorage?: () => Promise<boolean>;  // e.g. navigator.storage.persist()
 }
@@ -96,11 +278,62 @@ interface CoreEnv {
 `memoryKV()` and `memoryBlobCache()` provide in-memory implementations for
 tests and short-lived Node processes.
 
+### buildTokenizer
+
+Core downloads the tokenizer files itself. Every provider pins
+`tokenizer.json` and `tokenizer_config.json` at the same immutable commit as
+its ONNX model, with a SHA-256 and byte size, and fetches them through
+`fetchModelBlob` (resume, retry, `modelCache`, verification marker,
+`ModelIntegrityError`). The host only turns the two verified, parsed objects
+into a tokenizer instance:
+
+```ts
+import { PreTrainedTokenizer } from '@huggingface/transformers';
+
+buildTokenizer: (tokenizerJson, tokenizerConfig) =>
+  new PreTrainedTokenizer(tokenizerJson as never, tokenizerConfig as never),
+```
+
+Pick the class from `tokenizerConfig.tokenizer_class` if your library needs
+a specific subclass (`PreTrainedTokenizerFast` for GLiNER PII Small,
+`DebertaV2Tokenizer` for GLiNER PII Base, `XLMRobertaTokenizer` for BardS.ai);
+`PreTrainedTokenizer` handles all three in `@huggingface/transformers`.
+With `buildTokenizer` the host never contacts the network for tokenizers, a
+warm `modelCache` works offline, and the only URLs fetched on a cold start
+are the three pinned `resolve/<commit>/...` files of the active provider.
+
+`loadTokenizer(hfModelId)` (0.11 and earlier) is still honoured when
+`buildTokenizer` is absent, with a one-time deprecation warning. It lets the
+host library probe the mutable `resolve/main` branch and keep an unverified
+second cache, so migrate before 0.13.0, where `buildTokenizer` becomes
+required and `loadTokenizer` is removed.
+
+### autoLoad and ModelNotLoadedError
+
+```ts
+import { createEngine, ModelNotLoadedError } from '@doccloak/core';
+
+const engine = createEngine(env, undefined, { autoLoad: false });
+try {
+  await engine.detect(text);
+} catch (err) {
+  if (err instanceof ModelNotLoadedError) {
+    // ask the user for consent, then:
+    await engine.preload();
+  }
+}
+```
+
+With `autoLoad: false` no code path of the engine downloads a model unless
+the host calls `preload()` or `switchProvider()`; `ModelNotLoadedError`
+carries the `providerId` that needs loading. The default (`autoLoad: true`)
+keeps the 0.11 behaviour where `detect()` loads on demand.
+
 ### Web app quickstart
 
 ```ts
 import { createEngine, type CoreEnv } from '@doccloak/core';
-import { AutoTokenizer } from '@huggingface/transformers';
+import { PreTrainedTokenizer } from '@huggingface/transformers';
 
 const env: CoreEnv = {
   kv: {
@@ -118,7 +351,7 @@ const env: CoreEnv = {
   },
   fetch: (...args) => fetch(...args),
   wasm: { paths: '/', numThreads: 1 },  // serve onnxruntime-web *.wasm under /
-  loadTokenizer: (hfModelId) => AutoTokenizer.from_pretrained(hfModelId),
+  buildTokenizer: (json, config) => new PreTrainedTokenizer(json as never, config as never),
   hardware: { isMobile: /Mobile/i.test(navigator.userAgent) },
 };
 
@@ -190,14 +423,14 @@ minimal setup:
 
 ```ts
 import { createEngine, memoryKV, memoryBlobCache } from '@doccloak/core';
-import { AutoTokenizer } from '@huggingface/transformers';
+import { PreTrainedTokenizer } from '@huggingface/transformers';
 
 const engine = createEngine({
   kv: memoryKV(),
   modelCache: memoryBlobCache(),
   fetch: (...args) => fetch(...args),
   wasm: { paths: 'node_modules/onnxruntime-web/dist/', numThreads: 1 },
-  loadTokenizer: (hfModelId) => AutoTokenizer.from_pretrained(hfModelId),
+  buildTokenizer: (json, config) => new PreTrainedTokenizer(json as never, config as never),
 });
 await engine.ready;
 console.log(await engine.detect('Contact jane.doe@example.com or +48 601 123 456.'));

package/dist/doc.d.ts

@@ -1,8 +1,53 @@
+import { type ValueReplacement } from './docx.js';
+/** Result of inspectDoc. */
+export interface DocInspection {
+    /** fEncrypted or fObfuscated is set: readDocText / writeAnonymizedDoc refuse the file. */
+    encrypted: boolean;
+    /** fComplex is set (or Word 97 cQuickSaves > 0): readDocText / writeAnonymizedDoc refuse the file. */
+    fastSaved: boolean;
+    /** Streams that are copied through the export without being redacted. */
+    streams: {
+        /** Data stream (field data such as hyperlinks, pictures, OLE previews). Scanned and scrubbed in UTF-16LE only. */
+        data: boolean;
+        /** ObjectPool storage (embedded OLE documents). Not scrubbed. */
+        objectPool: boolean;
+        /** Macros storage (VBA project). Not scrubbed. */
+        macros: boolean;
+    };
+}
+/**
+ * Cheap look at a legacy .doc without parsing the piece table: which refusal
+ * flags are set and which non-text streams the file carries. Hosts use it to
+ * show a warning next to the export (ObjectPool / Macros are copied through
+ * unchanged, see writeAnonymizedDoc) or to explain a refusal up front.
+ * Throws only when the file is not a .doc at all.
+ */
+export declare function inspectDoc(buffer: ArrayBuffer | Uint8Array): DocInspection;
+/**
+ * Write a redacted copy of a legacy .doc.
+ *
+ * `replacements` are offsets into the text returned by readDocText. Main-text
+ * ranges are swapped for the replacement string through a rewritten piece
+ * table; ranges in subdocuments (footnotes, headers, comments) are overwritten
+ * in place. `valueReplacements` (same shape as writeAnonymizedDocx) add
+ * original values that were not located by offset, e.g. session entries.
+ *
+ * Every original value (from both sources) is then destroyed wherever else it
+ * still occurs as bytes, without moving any structure (see scrubStreamBytes).
+ *
+ * Refuses (UnsupportedDocumentError) fast-saved and encrypted files, see
+ * assertSupportedFib.
+ *
+ * Copied through untouched: ObjectPool (embedded OLE objects), Macros (VBA)
+ * and \x01CompObj. TODO(T196, T199): delete them with CFB.utils.cfb_del once
+ * the open-in-Word / LibreOffice experiment proves the result still opens.
+ * Until then hosts should call inspectDoc and warn when they are present.
+ */
 export declare function writeAnonymizedDoc(buffer: ArrayBuffer, replacements: Array<{
     start: number;
     end: number;
     replacement: string;
-}>): Promise<Blob>;
+}>, valueReplacements?: ValueReplacement[]): Promise<Blob>;
 /**
  * Extract plain text from a legacy .doc (OLE2/Compound File Binary) file.
  * Parses the FIB and piece table to correctly extract text from both
@@ -11,3 +56,25 @@ export declare function writeAnonymizedDoc(buffer: ArrayBuffer, replacements: Ar
  * in those regions is visible to detection.
  */
 export declare function readDocText(buffer: ArrayBuffer): string;
+/**
+ * Derive the byte-scrub needles from the original values: each value itself,
+ * each whitespace-separated token and each token stripped of surrounding
+ * punctuation, keeping those of at least MIN_SCRUB_NEEDLE characters.
+ * Matching is case-insensitive (see scrubStreamBytes), which covers every
+ * case variant (as written, lower, UPPER, Capitalised) in one pass.
+ *
+ * A needle that occurs (case-insensitively) inside one of the `replacements`
+ * is dropped: a token such as "Person" from "Leaky Person" would otherwise
+ * blank the "[PERSON_1]" placeholder, and a surrogate that keeps a word of
+ * the original ("Smith & Co" -> "Nowak & Co") would lose it. The full
+ * original value is never dropped.
+ */
+export declare function buildScrubNeedles(originals: string[], replacements?: string[]): string[];
+/**
+ * Overwrite every case-insensitive occurrence of `needle` in `bytes` with
+ * spaces of the same encoding. UTF-16LE: units are read at every byte
+ * offset (strings in the Table stream are not always 2-byte aligned) and
+ * replaced by 0x20 0x00; cp1252: bytes replaced by 0x20. Returns the number
+ * of hits. Stream length never changes.
+ */
+export declare function scrubStreamBytes(bytes: Uint8Array, needle: string, encoding: 'utf16le' | 'cp1252'): number;