mask-privacy 2.0.0 → 3.1.0

package/README.md

@@ -87,6 +87,12 @@ Mask prevents the misidentification of real data as tokens by using universally
 
 This prefix-based approach ensures that the SDK does not inadvertently process valid PII as an existing token.
 
+Additional collision-proof prefixes for international identifiers:
+* Turkish TCID tokens use the `990000` prefix (no valid Kimlik number starts with `99`).
+* Saudi NID tokens use the `100000` prefix (length-constrained to avoid overlap with real IDs).
+* UAE Emirates ID tokens use the `784-0000-` prefix (zeroed sub-fields are structurally invalid).
+* IBAN tokens zero the check digits (`XX00...`), which always fails ISO 7064 Mod-97 verification.
+
 ### 4. Enterprise Async Support
 Mask is built from the ground up for high-concurrency Node.js environments. All core operations are asynchronous and promised-based. Calling `encode()`, `decode()`, or `scanAndTokenize()` allows your event loop to remain unblocked while handling PII tokenization tasks.
 
@@ -127,19 +133,97 @@ Performance-sensitive deployments utilize the built-in `LocalTransformersScanner
 ### 7. Sub-string Detokenization
 Mask includes the ability to detokenize PII embedded within larger text blocks (like email bodies or chat messages). `detokenizeText()` uses high-performance regex to find and restore all tokens within a paragraph before they hit your tools.
 
-## Installation and Setup
+## Multilingual PII Detection (Waterfall Pipeline)
+
+Mask is built for the global enterprise. While many privacy tools are English-centric, the TypeScript SDK implements a **3-Tier Waterfall Detection** strategy designed for high-performance PII detection across 8 major languages using local ONNX models.
+
+### Supported Language Matrix
+
+Mask provides first-class support for the following languages:
+
+| Language | Code | Tier 0 (DLP) | Tier 2 (NLP Engine) |
+| :--- | :--- | :--- | :--- |
+| **English** | `en` | ✅ Full | DistilBERT (Simple) |
+| **Spanish** | `es` | ✅ Full | BERT Multilingual |
+| **French** | `fr` | ✅ Full | BERT Multilingual |
+| **German** | `de` | ✅ Full | BERT Multilingual |
+| **Turkish** | `tr` | ✅ Full | BERT Multilingual |
+| **Arabic** | `ar` | ✅ Full | BERT Multilingual |
+| **Japanese** | `ja` | ✅ Full | BERT Multilingual |
+| **Chinese** | `zh` | ✅ Full | BERT Multilingual |
+
+### How the Waterfall Works: The Excising Mechanism
+
+To maintain high performance, the TypeScript SDK does not simply run three separate scans. It uses a **Sequential Mutation** strategy:
+
+1.  **Tier 0 & 1 (The Scouts):** The SDK first runs the high-speed DLP and Regex engines synchronously in the main thread.
+2.  **Immediate Tokenization:** Any PII found by these tiers is **immediately replaced** by a token in the string buffer.
+3.  **Tier 2 (The Heavy Infantry):** The expensive NLP engine (Transformers.js) only scans the *remaining* text. Because the PII has already been "excised" (cut out and replaced with tokens), the NLP engine doesn't waste compute on data already identified.
+4.  **Bypass Logic:** All tiers are "token-aware." If a scan encounter a string that is already a Mask token, it skips it entirely, preventing redundant processing or "double-tokenization."
+
+---
+
+### Configuration & Environment Variables
+
+Configure your multilingual environment using standard variables. These are parsed at runtime when the `LocalTransformersScanner` is initialized.
+
+| Variable | Default | Description |
+| :--- | :--- | :--- |
+| `MASK_LANGUAGES` | `en` | Comma-separated list of languages (e.g., `en,es,fr,ar`). |
+| `MASK_NLP_MODEL` | *(varies)* | Override the default model (e.g., `Xenova/bert-base-multilingual-cased-ner-hrl`). |
+| `MASK_MODEL_CACHE_DIR` | `~/.cache` | Local directory for storing serialized ONNX models. |
+| `MASK_NLP_MAX_WORKERS` | `4` | Number of worker processes/threads for NLP analysis. |
+| `MASK_NLP_TIMEOUT_SECONDS` | `60` | Max seconds for a scan before timing out. |
+
+---
+
+### Automatic Model Management
+
+The TypeScript SDK manages AI models automatically via the **Transformers.js** runtime. 
+
+#### 1. Automatic Downloads
+When you set `MASK_LANGUAGES` to include non-English languages, the scanner will automatically download the multilingual BERT model from Hugging Face on the first execution and cache it locally.
+
+#### 2. Pre-Warming (Performance)
+Upon initialization, the `LocalTransformersScanner` starts a worker pool and "pre-warms" the models. This ensures that the first real request doesn't suffer from high "cold-start" latency.
 
-Install the core SDK via npm:
+#### 3. Air-Gapped / Offline Environments
+For high-security environments, you can pre-cache models. Run this script in your build pipeline:
 ```bash
-npm install mask-privacy
+# Set a custom cache directory
+export MASK_MODEL_CACHE_DIR="./models"
+
+# Run a dummy scan to trigger the download
+node -e "require('mask-privacy').getScanner().scanAndTokenize('John Doe')"
+
+# Bundle the './models' folder with your container
 ```
 
-Add peer dependencies depending on your infrastructure:
+---
+
+### Performance & Latency Benchmarks
+
+*Measured on 4-vCPU 8GB RAM Instance (Node.js 20+)*
+
+| Tier | Engine | Avg. Latency | Rationale |
+| :--- | :--- | :--- | :--- |
+| Tier 0 | DLP (Heuristic) | ~2ms | Main-thread synchronous regex |
+| Tier 1 | Regex (Deterministic) | ~1ms | Main-thread synchronous regex |
+| Tier 2 | Transformers (Local) | 300ms - 800ms | Offloaded to Worker Threads (Piscina) |
+
+**Total Overhead:** Usually **< 400ms** for standard chat lengths. Mask uses an **Excising Mechanism** to ensure that text already identified in Tiers 0/1 is removed from the NLP buffer, significantly accelerating the heavy Transformer inference.
+
+---
+
+### Installing AI Models (Production Ready)
+The TypeScript SDK manages AI models automatically via **Transformers.js**. For production air-gapped environments or to avoid "cold-start" latency, we recommend using the pre-caching CLI:
+
 ```bash
-npm install ioredis          # For Redis vaults
-npm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb # For DynamoDB vaults
-npm install memjs            # For Memcached vaults
-npm install @huggingface/transformers # Required for local NLP scanning
+npm install @huggingface/transformers # Required extra
+
+# Pre-cache models for your required languages
+export MASK_LANGUAGES="en,es,fr"
+npx mask-privacy cache-models
 ```
 
 ### Framework Support
@@ -152,6 +236,24 @@ Mask supports major AI frameworks via built-in hooks:
 
 Before running your agents, Mask requires an encryption key and a vault backend selection.
 
+#### Where to set these?
+Select the method that best fits your deployment:
+
+1.  **In a `.env` file (Recommended)**: Create a file in your project root.
+    ```env
+    MASK_LANGUAGES="es,en"
+    MASK_ENCRYPTION_KEY="your-key"
+    ```
+    Then load it using a library like `dotenv`.
+2.  **In your Terminal**:
+    *   **Bash**: `export MASK_LANGUAGES="es,en"`
+    *   **PowerShell**: `$env:MASK_LANGUAGES="es,en"`
+3.  **Directly in TypeScript/Node.js**:
+    ```typescript
+    process.env.MASK_LANGUAGES = "es,en";
+    // Ensure this happens BEFORE initializing the MaskClient
+    ```
+
 #### 1. Configure Keys
 By default, Mask reads from environment variables.
 ```bash
@@ -181,15 +283,24 @@ export MASK_DYNAMODB_REGION=us-east-1
 export MASK_MEMCACHED_HOST=localhost
 export MASK_MEMCACHED_PORT=11211
 
-#### 4. Security Enforcement
-# Enable strict mode to refuse startup without MASK_ENCRYPTION_KEY
-export MASK_STRICT_PROD=true
+#### 4. Security Guardrails: Fail-Shut by Default
 
-# Configure Blind Index Salt (Optional)
-export MASK_BLIND_INDEX_SALT="custom-salt-here"
+To prevent accidental data leakage, Mask defaults to a **Fail-Shut** strategy. If the Vault or Key Provider is unreachable, the SDK will throw a `MaskVaultConnectionError`.
 
 > [!IMPORTANT]
-> **Security Warning:** In production, you **must** change the default `MASK_BLIND_INDEX_SALT`. Using the default salt makes your blind indices vulnerable to pre-computed hash (rainbow table) attacks across different SDK installations.
+> **Environment Modes:**
+> - **Production (Default):** Fail-Shut enabled. Strictly protects PII.
+> - **Development:** Set `MASK_ENV=dev` to allow "Fail-Open" behavior (PII is returned as-is if the vault fails).
+
+#### 5. Model Pre-caching CLI
+
+For production air-gapped environments or to avoid "cold-start" latency, use the model pre-caching tool:
+
+```bash
+# Cache English and Spanish models
+export MASK_MODEL_CACHE_DIR="./models"
+npx ts-node src/cli.ts cache-models --languages en,es
+```
 
 # Configure MemoryVault cleanup aggressiveness (default: 0.01)
 export MASK_VAULT_CLEANUP_FREQUENCY=0.05

package/dist/index.d.mts

@@ -14,6 +14,7 @@ type EncodeOptions = {
     ttl?: number;
     searchBuckets?: ('year' | 'month' | 'day' | 'numeric')[];
     searchBucketSize?: number;
+    entityType?: string;
 };
 /**
  * Tokenise rawText, encrypt it, store in vault, return the FPE token.
@@ -49,9 +50,40 @@ declare function looksLikeToken(value: string | any): boolean;
 /** Clear the cached master key. Useful in tests. */
 declare function resetMasterKey(): void;
 /**
- * Return a **deterministic**, format-preserving token for rawText.
+ * Return a **deterministic**, format-preserving token for rawText using its entityType.
+ */
+declare function generateFPEToken(rawText: string, entityType?: string): Promise<string>;
+
+/**
+ * Span Resolution Engine — Sweep-Line Overlap Resolver (TypeScript).
+ *
+ * All detection tiers now return Span objects instead of mutating the text.
+ * resolveOverlaps() chooses the winning span in every conflicting region,
+ * and reconstruct() rebuilds the string exactly once.
+ */
+interface Span {
+    start: number;
+    end: number;
+    entityType: string;
+    originalValue: string;
+    confidence: number;
+    method: string;
+    language?: string;
+    maskedValue?: string;
+}
+
+/**
+ * Entity Detection Scanner — Tiered Waterfall Pipeline.
+ *
+ * Scans unstructured text to identify PII (Emails, Phones, SSNs, Credit Cards,
+ * Names) and replaces them in-place with Format-Preserving Encryption (FPE)
+ * tokens.
+ *
+ * Detection Architecture (Waterfall):
+ *   Tier 0 — DLP Heuristic: Multilingual, 50+ types, checksum validators
+ *   Tier 1 — Deterministic: Regex + Checksum  (fast, provable, auditable)
+ *   Tier 2 — Probabilistic: Local NLP via Transformers (catches names/orgs)
  */
-declare function generateFPEToken(rawText: string): Promise<string>;
 
 declare class BaseScanner {
     protected _supportedEntities: string[];
@@ -61,18 +93,23 @@ declare class BaseScanner {
     protected static _luhnChecksum(ccNumber: string): boolean;
     /** Validate a US ABA routing number using the checksum algorithm. */
     protected static _abaChecksum(routingNumber: string): boolean;
-    protected _tier1Regex(text: string, encodeFn: (val: string) => Promise<string>, boostEntities: Set<string>, aggressive: boolean, confidenceThreshold: number): Promise<[string, any[]]>;
-    protected _tier2Nlp(text: string, encodeFn: (val: string) => Promise<string>, boostEntities: Set<string>, aggressive: boolean, confidenceThreshold: number): Promise<[string, any[]]>;
+    protected _tier0CollectSpans(text: string, confidenceThreshold: number): Promise<Span[]>;
+    /** Backward-compat wrapper — collects spans then single-pass encodes. */
+    protected _tier0Dlp(text: string, encodeFn: (val: string, options?: any) => Promise<string>, confidenceThreshold: number): Promise<[string, any[]]>;
+    protected _tier1CollectSpans(text: string, boostEntities: Set<string>, aggressive: boolean, confidenceThreshold: number): Promise<Span[]>;
+    /** Backward-compat wrapper. */
+    protected _tier1Regex(text: string, encodeFn: (val: string, options?: any) => Promise<string>, boostEntities: Set<string>, aggressive: boolean, confidenceThreshold: number): Promise<[string, any[]]>;
+    protected _tier2Nlp(text: string, encodeFn: (val: string, options?: any) => Promise<string>, boostEntities: Set<string>, aggressive: boolean, confidenceThreshold: number): Promise<[string, any[]]>;
     protected _resolveBoost(context?: string | null): Set<string>;
     scanAndTokenize(text: string, options?: {
-        encodeFn?: (val: string) => Promise<string>;
+        encodeFn?: (val: string, options?: any) => Promise<string>;
         pipeline?: string[];
         confidenceThreshold?: number;
         context?: string | null;
         aggressive?: boolean;
     }): Promise<string>;
     scanAndReturnEntities(text: string, options?: {
-        encodeFn?: (val: string) => Promise<string>;
+        encodeFn?: (val: string, options?: any) => Promise<string>;
         pipeline?: string[];
         confidenceThreshold?: number;
         context?: string | null;
@@ -106,7 +143,7 @@ declare class LocalTransformersScanner extends BaseScanner {
      * Map Transformer entity types to Mask internal entity types.
      */
     private _mapEntityType;
-    protected _tier2Nlp(text: string, encodeFn: (val: string) => Promise<string>, boostEntities: Set<string>, aggressive: boolean, confidenceThreshold: number): Promise<[string, any[]]>;
+    protected _tier2Nlp(text: string, encodeFn: (val: string, options?: any) => Promise<string>, boostEntities: Set<string>, aggressive: boolean, confidenceThreshold: number): Promise<[string, any[]]>;
     /**
      * Merges sub-tokens and entities of the same type while precisely tracking
      * offsets in the original text.
@@ -265,6 +302,183 @@ declare class MaskClient {
     close(): Promise<void>;
 }
 
+/**
+ * Language Context Resolver — Unicode-block heuristic for multilingual DLP.
+ *
+ * Examines the character distribution of an input buffer to infer the
+ * dominant script / language.  The resolved language tag is consumed by
+ * the DLPPatternRegistry to prioritise locale-specific regex groups.
+ *
+ * Supported language tags:
+ *   en — English (default / Latin-only fallback)
+ *   es — Spanish
+ *   fr — French
+ *   de — German
+ *   tr — Turkish
+ *   ar — Arabic
+ *   zh — Chinese
+ *   ja — Japanese
+ */
+type LanguageTag = "en" | "es" | "fr" | "de" | "tr" | "ar" | "zh" | "ja";
+interface LanguageBreakdown {
+    language: LanguageTag;
+    breakdown: Record<string, number>;
+}
+/**
+ * Determine the dominant language of a text buffer.
+ *
+ * The resolver is stateless and safe for concurrent use.
+ *
+ * @example
+ * ```ts
+ * const resolver = new LanguageContextResolver();
+ * const tag = resolver.resolve("Merhaba, TC Kimlik Numaram 12345678901");
+ * // tag === "tr"
+ * ```
+ */
+declare class LanguageContextResolver {
+    /** Minimum number of script-specific characters required. */
+    private readonly charThreshold;
+    constructor(charThreshold?: number);
+    /** Return an ISO-639-1 language tag for the given text. Falls back to "en". */
+    resolve(text: string): LanguageTag;
+    /** Return the language tag together with per-script hit counts. */
+    resolveWithDetail(text: string): LanguageBreakdown;
+}
+
+/**
+ * DLP Pattern Registry — Centralised catalogue of 50+ sensitive-data signatures.
+ *
+ * Each entry bundles a compiled regex, a list of proximity keywords (used by
+ * the scorer for context boosting), a base leakage-risk probability, and an
+ * optional hard-validator tag that tells the DLPValidationEngine which
+ * checksum to run after the initial pattern match.
+ *
+ * Patterns are organised into SensitiveCategory groups so that callers can
+ * selectively load only the groups relevant to their compliance scope.
+ */
+
+declare enum SensitiveCategory {
+    FINANCIAL = "FINANCIAL",
+    CONTACT = "CONTACT",
+    PERSONAL = "PERSONAL",
+    HEALTHCARE = "HEALTHCARE",
+    IDENTITY_US = "IDENTITY_US",
+    IDENTITY_INTL = "IDENTITY_INTL",
+    VEHICLE = "VEHICLE",
+    CORPORATE = "CORPORATE"
+}
+interface PatternDescriptor {
+    compiledRe: RegExp;
+    proximityTerms: ReadonlySet<string>;
+    baseRisk: number;
+    category: SensitiveCategory;
+    validatorTag: string | null;
+    isHighEntropy: boolean;
+    supportedLocales: string[];
+}
+/**
+ * Immutable catalogue of sensitive-data regex signatures.
+ */
+declare class DLPPatternRegistry {
+    private readonly catalogue;
+    private readonly localeCategoryRegexMap;
+    constructor(loadGroups?: ReadonlySet<SensitiveCategory>);
+    get typeNames(): string[];
+    /** Yield [typeName, descriptor] pairs. */
+    iterDescriptors(): IterableIterator<[string, PatternDescriptor]>;
+    descriptorFor(typeName: string): PatternDescriptor | undefined;
+    namePatternsFor(lang: LanguageTag | string): RegExp[];
+    addressPatternsFor(lang: LanguageTag | string): RegExp[];
+    getCategoryRegexesMap(locale?: string): Map<string, {
+        re: RegExp;
+        typeOrder: string[];
+    }>;
+    getCategoryTypeMap(categoryName: string, locale?: string): string[];
+    private compileForLocale;
+    private buildCatalogue;
+}
+
+/**
+ * Run the appropriate hard-validator for a given validator tag.
+ *
+ * @example
+ * ```ts
+ * const engine = new DLPValidationEngine();
+ * const passed = engine.run("luhn", "4111111111111111");
+ * ```
+ */
+declare class DLPValidationEngine {
+    /**
+     * Execute the validator identified by tag.
+     *
+     * @returns `true` — value passed checksum → confidence override.
+     * @returns `false` — value failed → confidence penalty.
+     * @returns `null` — no validator registered for tag.
+     */
+    run(tag: string | null | undefined, rawValue: string): boolean | null;
+    /** Return all registered validator tag names. */
+    static availableTags(): string[];
+}
+
+/**
+ * DLP Confidence Scorer — Proximity-weighted scoring for sensitive data matches.
+ *
+ * Combines three independent signals into a single confidence value:
+ * 1. Base risk — intrinsic leakage probability of the data type.
+ * 2. Proximity boost — logarithmic-decay bonus for each context keyword
+ *    found near the match within a configurable window.
+ * 3. Validator override — hard-validator pass forces confidence to 0.99.
+ *
+ * The scorer is stateless and safe for concurrent use.
+ */
+interface ScorerConfig {
+    contextWindow?: number;
+    keywordBoost?: number;
+    validatorOverride?: number;
+    maxConfidence?: number;
+    penaltyFactor?: number;
+}
+interface ScoreInput {
+    baseRisk: number;
+    matchStart: number;
+    matchEnd: number;
+    fullText: string;
+    proximityTerms: ReadonlySet<string>;
+    validatorPassed: boolean | null;
+}
+/**
+ * Calculate a weighted confidence score for a single regex hit.
+ *
+ * @example
+ * ```ts
+ * const scorer = new DLPConfidenceScorer();
+ * const score = scorer.score({
+ *   baseRisk: 0.92,
+ *   matchStart: 10,
+ *   matchEnd: 21,
+ *   fullText: "TC Kimlik No: 10000000146",
+ *   proximityTerms: new Set(["kimlik", "tc"]),
+ *   validatorPassed: true,
+ * });
+ * // score === 0.99 (validator override)
+ * ```
+ */
+declare class DLPConfidenceScorer {
+    private readonly window;
+    private readonly kwBoost;
+    private readonly valOverride;
+    private readonly ceil;
+    private readonly penalty;
+    constructor(overrides?: ScorerConfig);
+    /**
+     * Compute the final confidence for one candidate match.
+     *
+     * @returns Confidence in [0.0, maxConfidence].
+     */
+    score(input: ScoreInput): number;
+}
+
 /**
  * Mask Privacy SDK
  * Just-In-Time Privacy Middleware for AI Agents.
@@ -298,4 +512,4 @@ declare function ascanAndTokenize(text: string, options?: {
  */
 declare function secureTool(...args: any[]): any;
 
-export { BaseScanner, LocalTransformersScanner, MaskClient, MaskDecryptionError, MaskError, MaskNLPTimeout, MaskSecurityError, MaskVaultConnectionError, PresidioScanner, VERSION, adecode, adetokenizeText, aencode, ascanAndTokenize, decode, detectEntitiesWithConfidence, detokenizeText, encode, generateFPEToken, getScanner, getVault, looksLikeToken, resetMasterKey, secureTool };
+export { BaseScanner, DLPConfidenceScorer, DLPPatternRegistry, DLPValidationEngine, LanguageContextResolver, LocalTransformersScanner, MaskClient, MaskDecryptionError, MaskError, MaskNLPTimeout, MaskSecurityError, MaskVaultConnectionError, PresidioScanner, SensitiveCategory, VERSION, adecode, adetokenizeText, aencode, ascanAndTokenize, decode, detectEntitiesWithConfidence, detokenizeText, encode, generateFPEToken, getScanner, getVault, looksLikeToken, resetMasterKey, secureTool };