euredact 0.3.9 → 0.5.0

package/README.md

@@ -34,13 +34,19 @@ console.log(result.detections);
 ## Features
 
 - **31 European countries** with country-specific patterns
-- **25+ PII entity types:** national IDs, IBANs, phone numbers, email, VAT
+- **41 PII entity types** (30 from the rules engine, 11 from the
+  [cloud tier](#cloud-tier)): national IDs, IBANs, phone numbers, email, VAT
   numbers, license plates, credit cards, BIC/SWIFT, VIN, IMEI, GPS coordinates,
   UUIDs, social handles, IP/IPv6, MAC addresses, secrets/API keys, and more
 - **Secret/API key detection:** known-prefix patterns for AWS, GitHub, Stripe,
   OpenAI, Slack, JWT, SendGrid, plus Shannon entropy-based detection for generic
   high-entropy tokens near context keywords
 - **Custom patterns:** register your own regex patterns for domain-specific PII
+- **Reversible tokenization:** `tokenize: true` swaps values for `EMAIL_K7Q2`-style
+  tokens and `restore()` puts them back — for prompts that go to an LLM and
+  come back
+- **Allowlist:** values that are never redacted, such as your own organisation's
+  name and addresses, per call or per instance
 - **Checksum validation:** IBAN mod-97, Luhn (credit cards/IMEI), and 30+
   country-specific validators
 - **Priority-aware deduplication:** validated (checksum, corroborated by the
@@ -65,6 +71,26 @@ console.log(result.detections);
 
 ## API Reference
 
+### Which option do I need?
+
+| I want to… | Use |
+|---|---|
+| redact a document with no further setup | `redact(text)` — all 31 countries, `[ENTITY_TYPE]` output |
+| restrict scope to the countries I operate in | `{ countries: ["NL", "BE"] }` |
+| keep detection wide but resolve ambiguity | `{ countryHint: ["DE"] }` |
+| send a prompt to an LLM and restore the reply | `{ tokenize: true }`, then [`restore()`](#restoretext-tokens) |
+| keep relationships visible across a whole session | `{ referentialIntegrity: true }` |
+| never redact my own company name or addresses | `{ allowlist: [...] }`, or `new EuRedact({ allowlist })` |
+| never redact anything at my own domain | `{ allowlistDomains: ["acme.be"] }` |
+| audit what the allowlist kept in the document | `result.exempted` |
+| catch person names, employers, job titles, diagnoses | `redactAsync(text, { mode: "cloud" })` — see [Cloud tier](#cloud-tier) |
+| include dates of birth | `{ detectDates: true }` |
+| redact a document too large for one call | `context` + `chunkOffset` — see [Chunked documents](#chunked-documents) |
+| detect an identifier the engine does not know | [`addCustomPattern()`](#addcustompatternname-pattern) |
+| isolate tenants from each other | one `new EuRedact()` each |
+| free PII held in memory | `clear()` |
+
+
 ### Module-Level Functions
 
 #### `redact(text, options?)`
@@ -77,25 +103,68 @@ Main entry point. Detects and redacts PII in the given text.
 
 ```ts
 interface RedactOptions {
-  countries?: string[] | null;    // Scope. Flags anything attributed elsewhere as
-                                  // outOfScope. Never gates what is looked for.
-  countryHint?: string[] | null;  // A prior only — resolves ambiguity without
-                                  // narrowing scope or flagging anything
-  context?: DocumentContext | null; // Share country evidence across the chunks
-                                  // of one document (see "Chunked documents")
-  chunkOffset?: number;           // Where this chunk starts in the document
-  referentialIntegrity?: boolean; // Replace with consistent labels (default: false)
-  detectDates?: boolean;          // Include DOB/date-of-death detections (default: false)
-  cache?: boolean;                // Enable result caching (default: true)
+  countries?: string[] | null;      // scope (see below); null = all 31
+  countryHint?: string[] | null;    // a prior only; does not narrow scope
+  context?: DocumentContext | null; // share evidence across chunks
+  chunkOffset?: number;             // where this chunk starts
+  mode?: string;                    // "rules" (default) | "cloud"
+  referentialIntegrity?: boolean;   // consistent labels, EMAIL_1
+  tokenize?: boolean;               // reversible tokens, EMAIL_K7Q2
+  allowlist?: string[] | null;      // values never redacted
+  allowlistDomains?: string[] | null; // domains never redacted (EMAIL/URL)
+  detectDates?: boolean;            // include DOB / date of death
+  cache?: boolean;                  // reuse results for identical input
 }
 ```
 
+**What is looked for**
+
+| Parameter | Default | Description |
+|---|---|---|
+| `text` | — | Input text to scan. Longer than `maxInputLength` throws. |
+| `countries` | `null` | ISO 3166-1 alpha-2 codes that define **scope**. `null` loads all 31. This never gates what is looked for: a detection attributed elsewhere is flagged `outOfScope`, never dropped. A bare string throws `TypeError`. See [Country codes](#country-codes). |
+| `countryHint` | `null` | A **prior only**. Resolves an ambiguous value without narrowing scope. See [Country Hints](#country-hints). |
+| `detectDates` | `false` | Include `DOB` and `DATE_OF_DEATH`. Off by default: a bare date without keyword or structural context is deferred to the cloud tier. |
+
+**How the output looks** — pick at most one of the first three.
+
+| Parameter | Default | Description |
+|---|---|---|
+| *(none)* | — | Default: each span becomes `[ENTITY_TYPE]`. |
+| `referentialIntegrity` | `false` | Consistent label per distinct value (`EMAIL_1`), persisting **on the instance** across calls. See [Referential Integrity](#referential-integrity). |
+| `tokenize` | `false` | Reversible token per value (`EMAIL_K7Q2`), with the mapping in `result.tokens` for [`restore()`](#restoretext-tokens). Unique to the **call**. Cannot be combined with `referentialIntegrity`. See [Reversible tokenization](#reversible-tokenization). |
+| `allowlist` | `null` | Values never to redact, whole-span and case-insensitive, merged with the instance's list. Structured identifiers (IBAN, phone, VAT, …) also match across spacing and hyphenation. Applies to cloud-tier types too. See [Allowlist](#allowlist). |
+| `allowlistDomains` | `null` | Domains whose addresses are never redacted, e.g. `["acme.be"]`. Applies to `EMAIL` and `URL` only, and covers subdomains. See [Allowlist](#allowlist). |
+
+**Long documents and tiers**
+
 | Parameter | Default | Description |
 |---|---|---|
-| `countries` | `null` | ISO 3166-1 alpha-2 codes to restrict detection. `null` loads all 31 countries. |
-| `referentialIntegrity` | `false` | Replace PII with consistent labels instead of entity-type labels. |
-| `detectDates` | `false` | Include date-of-birth and date-of-death detections. Off by default. |
-| `cache` | `true` | Cache results for identical inputs. |
+| `mode` | `"rules"` | `"rules"` runs locally and synchronously. `"cloud"` must go through [`redactAsync()`](#redactasynctext-options) — `redact()` is synchronous and a network call cannot be. See [Cloud tier](#cloud-tier). |
+| `context` | `null` | Share country evidence across the chunks of one document. Pass the same `DocumentContext` to every chunk. Disables the cache. See [Chunked documents](#chunked-documents). |
+| `chunkOffset` | `0` | Where this chunk starts in the whole document. Only rebases spans recorded in `context`; returned detections stay relative to `text`. |
+| `cache` | `true` | Reuse the result for an identical input and configuration. |
+
+TypeScript has no `coref` option; the Python SDK accepts one as a reserved
+no-op.
+
+#### `redactAsync(text, options?)`
+
+```ts
+function redactAsync(text: string, options?: RedactOptions): Promise<RedactResult>;
+```
+
+The asynchronous entry point, and the only way to reach the [cloud
+tier](#cloud-tier). With `mode: "rules"` (the default) it resolves immediately
+with exactly what `redact()` returns, so a caller that may or may not use the
+cloud tier can keep one code path:
+
+```ts
+const result = await redactAsync(text, { countries: ["BE"], mode: "cloud" });
+```
+
+Takes the same `RedactOptions` as `redact()`. Options the service cannot honour
+throw rather than being ignored — see [Cloud tier](#cloud-tier).
 
 #### `redactBatch(texts, options?)`
 
@@ -115,6 +184,16 @@ function addCustomPattern(name: string, pattern: string): void;
 Register a custom regex pattern. Matches are reported with `name` as the entity
 type. See [Custom Patterns](#custom-patterns) below.
 
+#### `restore(text, tokens)`
+
+```ts
+function restore(text: string, tokens: Record<string, string>): string;
+```
+
+Put the original values back into text that derives from a `tokenize: true`
+result — typically an LLM's reply to the tokenized prompt. `tokens` is
+`result.tokens`. See [Reversible tokenization](#reversible-tokenization).
+
 #### `availableCountries()`
 
 ```ts
@@ -141,8 +220,25 @@ console.log(result.redactedText);
 // "See [CASE_REF] for details"
 ```
 
-The `EuRedact` class exposes: `redact()`, `redactBatch()`, and
-`addCustomPattern()`.
+The `EuRedact` class exposes `redact()`, `redactAsync()`, `redactBatch()`,
+`clear()` and `addCustomPattern()`.
+
+```ts
+new EuRedact({
+  maxInputLength?: number,   // default 10_485_760 (~10 MB)
+  allowlist?: string[] | null,
+})
+```
+
+| Parameter | Default | Description |
+|---|---|---|
+| `maxInputLength` | `10_485_760` | Longest document `redact()` accepts, in characters. Above it, it throws — split the input or raise the ceiling. |
+| `allowlist` | `null` | Values never to redact, for every call on this instance. Merged with the per-call `allowlist`. See [Allowlist](#allowlist). |
+| `allowlistDomains` | `null` | Domains never to redact, for every call on this instance. |
+
+The result cache, referential-integrity labels and custom patterns are all per
+instance, which is what makes one instance per tenant the right default.
+`clear()` releases the cache and the label mapping.
 
 ### Return Types
 
@@ -154,6 +250,8 @@ interface RedactResult {
   detections: Detection[];    // All PII spans found
   source: string;             // Detection backend ("rules")
   degraded: boolean;          // True if the engine fell back to a simpler mode
+  tokens: Record<string, string>; // token -> original value; only with tokenize: true
+  exempted: Exemption[];          // spans the allowlist kept, with the rule that matched
 }
 ```
 
@@ -210,6 +308,7 @@ interface RedactResult {
   evidence: CountryEvidence[];                // every signal, with the span behind it
   detectionMode: string;                      // "declared" if countries was passed,
                                               // "inferred" otherwise
+  tokens: Record<string, string>;             // token -> original value; only with tokenize: true
 }
 ```
 
@@ -218,7 +317,7 @@ interface RedactResult {
 String enum with all supported PII categories:
 
 ```
-NAME              ADDRESS           BANK_ACCOUNT      BIC
+PERSON_NAME       ADDRESS           BANK_ACCOUNT      BIC
 CREDIT_CARD       PHONE             EMAIL             DOB
 DATE_OF_DEATH     NATIONAL_ID       SSN               TAX_ID
 PASSPORT          DRIVERS_LICENSE   RESIDENCE_PERMIT  LICENSE_PLATE
@@ -226,6 +325,13 @@ VIN               VAT               POSTAL_CODE       IP_ADDRESS
 IPV6_ADDRESS      MAC_ADDRESS       HEALTH_INSURANCE  HEALTHCARE_PROVIDER
 CHAMBER_OF_COMMERCE  IMEI          GPS_COORDINATES   UUID
 SOCIAL_HANDLE     SECRET            INTERNAL_ID       OTHER
+
+Cloud tier only — the rule engine never emits these, because there is no shape
+to match on. That is precisely why the model exists:
+
+ORGANISATION_NAME JOB_TITLE         MEDICAL_CONDITION SENSITIVE_ATTRIBUTE
+BIOMETRIC_REF     FINANCIAL_AMOUNT  QUASI_IDENTIFIER  CREDENTIAL
+URL
 ```
 
 `INTERNAL_ID` — an employee, badge or customer number tied to a person — is
@@ -264,6 +370,67 @@ masked is unchanged either way.
 
 For custom patterns, `entityType` is a plain string (e.g. `"EMPLOYEE_ID"`).
 
+## Cloud tier
+
+> **Status: private alpha.** The cloud tier is in closed testing — it is **not**
+> in public beta and is not generally available. Keys are issued to alpha
+> participants only, and the request/response surface may still change between
+> releases. The rules engine below is unaffected and is the supported path:
+> `mode="rules"` is the default and carries no alpha caveat.
+
+The rule engine catches what has a shape: IBANs, national IDs, phone numbers,
+anything with a checksum. It cannot catch what does not — a person's name, an
+employer, a diagnosis, a job title. The cloud tier adds a fine-tuned model
+asked only *what did the rules miss?*
+
+```ts
+import { configure, redactAsync } from "euredact";
+
+configure({ apiKey: "erk_..." });               // or set EUREDACT_API_KEY
+const result = await redactAsync(text, { countries: ["BE"], mode: "cloud" });
+
+result.source;                                   // "cloud"
+result.detections.map(d => [d.entityType, d.text]);
+// [["PERSON_NAME", "Bas Verhoeven"], ["PHONE", "+32 ..."]]
+```
+
+**`redact()` is synchronous and cannot serve the cloud tier**, so
+`redact(text, { mode: "cloud" })` throws and names `redactAsync`. It never
+falls back to rules-only output: a caller who believes names and diagnoses were
+checked, and ships a document that only had its phone numbers masked, is the
+one failure this library must not have.
+
+`redactAsync` with `mode: "rules"` resolves immediately with exactly what
+`redact()` returns, so a caller that may or may not use the cloud tier can hold
+one code path.
+
+Retries carry an `Idempotency-Key`, so a retry after a timeout cannot bill
+twice. `Retry-After` is obeyed. A document that outlives the service's sync
+window is polled transparently. Oversized input rejects with `TooLargeError`
+(413): the service refuses it rather than chunking, because the model has never
+seen a chunk boundary.
+
+Options the service cannot honour reject rather than being ignored: multiple
+`countries`, `countryHint`, `context`/`chunkOffset` and `referentialIntegrity`.
+`tokenize`, `allowlist` and `allowlistDomains` are honoured: the SDK applies them to the spans the
+service returns and rebuilds the text from those.
+
+The package stays **zero-dependency** — the client uses the platform's own
+`fetch`. Node 18+ provides one; on Node 16 the rules engine is unaffected and a
+`fetchImpl` can be supplied.
+
+## `NAME` is now `PERSON_NAME`
+
+The canonical type name is `PERSON_NAME`; `NAME` is a legacy alias, exactly as
+`IBAN` aliases `BANK_ACCOUNT`. The placeholder written into redacted text is
+`[PERSON_NAME]`.
+
+Nothing could have depended on the old value: the type is cloud-only and the
+cloud tier was stubbed until this release, so it was never emitted. Code
+matching the *string* `"NAME"` should be updated; `LEGACY_TYPE_ALIASES`
+publishes the mapping, and `STREET_ADDRESS` → `ADDRESS` and
+`NATIONALITY_ETHNICITY` → `SENSITIVE_ATTRIBUTE` are recognised the same way.
+
 ## Country codes
 
 `countries` accepts **ISO 3166-1 alpha-2** codes. The two EU/VAT spellings are
@@ -314,8 +481,9 @@ redact("Rekening: BE68 5390 0754 7034", { countries: ["AT"] });
 ### Which country a value belongs to
 
 Because every pattern runs, the same digits often match several countries'
-schemes. 36.6% of national-ID values in our corpus validate under more than one
-country's checksum, so the digits alone cannot decide it — the *document* does.
+schemes. Of the national-ID values in our corpus that pass any country's checksum,
+34.7% pass more than one country's (32,827 of 94,528), so the digits alone
+cannot decide it — the *document* does.
 
 The engine infers the document's countries from entities that carry their
 country in the string, then uses that to resolve the ambiguity:
@@ -549,6 +717,120 @@ console.log(result.redactedText);
 // "BSN NATIONAL_ID_1 en later weer NATIONAL_ID_1"
 ```
 
+## Reversible tokenization
+
+`tokenize: true` is for text that has to come back. Each value is replaced by a
+token that names its type and nothing else, and the result carries the mapping
+that turns tokens back into values:
+
+```ts
+import { redact, restore } from "euredact";
+
+const prompt = "Write an email to Joren at joren.janssens@euredact.be or call +32 475 12 34 56 about the invoice.";
+const result = redact(prompt, { countries: ["BE"], tokenize: true });
+console.log(result.redactedText);
+// "Write an email to Joren at EMAIL_K7Q2 or call PHONE_P4RT about the invoice."
+console.log(result.tokens);
+// { EMAIL_K7Q2: "joren.janssens@euredact.be", PHONE_P4RT: "+32 475 12 34 56" }
+
+const reply = await callYourLlm(result.redactedText);  // sees tokens, never the values
+console.log(restore(reply, result.tokens));
+// the reply, with the real address and number back in it
+```
+
+The suffixes are random; yours will differ. With the [cloud tier](#cloud-tier)
+the name is tokenized too (`PERSON_NAME_W3NB`), since person names have no
+shape for the rules tier to match on.
+
+Within one call the same value gets the same token, so a prompt that names
+someone twice still reads as one person. Across calls it gets a different
+token: nothing is retained on the instance, `result.tokens` is the only copy,
+and two tokenized documents never reveal that they share a value. That is the
+opposite retention model from `referentialIntegrity`, which is why the two
+cannot be combined. `redactBatch` tokenizes each text on its own.
+
+A token is `TYPE_` plus four characters from `ABCDEFGHJKLMNPQRSTUVWXYZ23456789`
+(no vowels, no `0`/`1`/`I`/`O`). Tokens are kept clear of any token-shaped
+string already in the document, so an LLM's reply to a tokenized prompt can
+itself be redacted without `restore()` putting the wrong value back.
+`restore()` replaces every occurrence, including a token an LLM glued to other
+characters (`EMAIL_P4RTs`) — leaving a token behind is the worse failure.
+
+Works in cloud mode via `redactAsync`: the SDK rebuilds the text from the spans
+the service returns, which is exactly what the service built its own output
+from.
+
+## Allowlist
+
+Values a caller declares are not PII to them — their own organisation's name,
+their own addresses. Set it per call, on the instance, or both; the two merge:
+
+```ts
+import { EuRedact } from "euredact";
+
+const sdk = new EuRedact({ allowlist: ["ACME NV", "info@acme.be"] });
+
+const text = "ACME NV: mail info@acme.be or jan@acme.be about IBAN NL91 ABNA 0417 1643 00.";
+console.log(sdk.redact(text, { countries: ["NL"] }).redactedText);
+// "ACME NV: mail info@acme.be or [EMAIL] about IBAN [BANK_ACCOUNT]."
+
+console.log(sdk.redact(text, { countries: ["NL"], allowlist: ["jan@acme.be"] }).redactedText);
+// "ACME NV: mail info@acme.be or jan@acme.be about IBAN [BANK_ACCOUNT]."
+```
+
+Matching is whole-span and case-insensitive. A bare string
+(`allowlist: "ACME NV"`) throws `TypeError` rather than being iterated into
+single letters that exempt nothing. Works in cloud mode via `redactAsync` —
+including on types only the model finds, such as `ORGANISATION_NAME` — and
+together with `tokenize`.
+
+### Spacing and punctuation
+
+For **structured identifiers** separators are presentational, so an allowlisted
+value matches however the document writes it:
+
+```ts
+const sdk = new EuRedact({ allowlist: ["NL91ABNA0417164300"] });
+sdk.redact("Pay to NL91 ABNA 0417 1643 00.", { countries: ["NL"] }).redactedText;
+// 'Pay to NL91 ABNA 0417 1643 00.'   — exempt, despite the spacing
+```
+
+Applies to `BANK_ACCOUNT`, `BIC`, `CREDIT_CARD`, `PHONE`, `VAT`, `NATIONAL_ID`,
+`SSN`, `TAX_ID`, `PASSPORT`, `DRIVERS_LICENSE`, `RESIDENCE_PERMIT`,
+`HEALTH_INSURANCE`, `CHAMBER_OF_COMMERCE`, `IMEI` and `VIN`. It widens the
+*spelling*, never the *scope*: a different account is still redacted. Free-text
+types stay literal, because `jan.devries@acme.be` and `jandevries@acme.be` are
+different mailboxes at most providers.
+
+### Exempting a whole domain
+
+```ts
+const sdk = new EuRedact({ allowlistDomains: ["acme.be"] });
+sdk.redact("Mail jan@acme.be or piet@acme.be", { countries: ["NL"] }).redactedText;
+// 'Mail jan@acme.be or piet@acme.be'
+```
+
+`EMAIL` and `URL` only, covering subdomains (`mail.acme.be`). The match is on a
+label boundary, so `acme.be` does **not** exempt `evilacme.be`. Entries may be
+written `acme.be`, `@acme.be` or `.acme.be`.
+
+There are deliberately **no wildcards**. The allowlist is the only option that
+turns redaction *off*, so an over-broad entry fails toward under-redaction and
+does so silently.
+
+### What was exempted
+
+```ts
+const r = redact("Mail jan@acme.be", { countries: ["NL"], allowlistDomains: ["acme.be"] });
+for (const e of r.exempted) console.log(e.entityType, e.text, e.rule, e.ruleKind);
+// EMAIL jan@acme.be acme.be domain
+```
+
+`Exemption` carries `entityType`, `start`, `end`, `text`, the `rule` that
+matched as you wrote it, and `ruleKind` (`"value"` or `"domain"`). An exempted
+span is absent from `detections`, so `exempted` is the only record that it was
+found at all.
+
 ## Architecture
 
 ```
@@ -623,18 +905,24 @@ Two tiers are less obvious than they look:
 
 ## Performance
 
-Measured on one core, all 31 countries loaded, `detectDates: true`. Cost scales
-with document length, so the input size is stated rather than averaged away.
+Measured on one core (Apple Silicon M3 Pro, Node 22.12), all 31 countries
+loaded, `detectDates: true`, cache off, over two cohorts sampled evenly from the
+corpus: 3,000 distinct short records and 611 distinct real documents. Median per
+document, 10th–90th percentile in brackets:
 
 | Input | Latency | Throughput |
 |---|---:|---:|
-| Short record (186 chars) | 155 µs | 6,462 docs/s |
-| Real document (3,424 chars) | 1.56 ms | 643 docs/s |
+| Short record (~190 chars) | 154 µs <br><sub>89 – 237</sub> | 6,494 docs/s |
+| Real document (~3,450 chars) | 1.43 ms <br><sub>1.04 – 1.96</sub> | 699 docs/s |
+
+Cost tracks identifier density rather than length: the per-candidate cue and
+suppressor checks dominate, so a form-like document dense in national IDs costs
+several times an ordinary chat log of the same size.
 
 No optional accelerator is needed or offered. The Python SDK ships an
 `[fast]` extra (RE2 / Aho-Corasick) because CPython's regex engine is the
 bottleneck there; V8's has literal prefilters that make it unnecessary here.
-Measured on the same 611 documents, this SDK runs about 3× faster than the
+Measured on the same 611 documents, this SDK runs about 6–7× faster than the
 accelerated Python path, so adding a native addon — and with it the loss of
 bundler, edge-runtime and Deno compatibility — would buy nothing.
 

package/dist/cjs/cache.js

@@ -1,6 +1,7 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.ResultCache = void 0;
+exports.resultChars = resultChars;
 const node_crypto_1 = require("node:crypto");
 // Default ceiling on retained characters, not entries. A count-only cap of
 // 1024 entries says nothing about memory: 1024 cached 1 MB documents is a
@@ -11,6 +12,10 @@ function resultChars(result) {
     let chars = result.redactedText.length;
     for (const d of result.detections)
         chars += d.text.length;
+    for (const [token, value] of Object.entries(result.tokens))
+        chars += token.length + value.length;
+    for (const e of result.exempted)
+        chars += e.text.length + e.rule.length;
     return chars;
 }
 class ResultCache {