mask-privacy 1.0.2 → 3.0.0

package/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Mask AI Solutions
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/README.md

@@ -27,9 +27,9 @@ Instead of trusting the LLM to safeguard plain-text data, the system strictly en
 
 This guarantees that the LLM can orchestrate workflows involving sensitive data without ever actually exposing the raw data to the model or its remote provider logs. 
 
-Additionally, we solve two critical sub-issues to make this enterprise-ready:
-1. **The Statefulness Trap**: Traditional "vaults" break down in multi-node Kubernetes environments. We support pluggable distributed vaults (Redis, DynamoDB, Memcached) so detokenization state is instantly shared across all your horizontally scaled pods.
-2. **The Schema Trap**: Strict downstream tools will crash if handed a random token. We use Format-Preserving Tokenization backed by an encrypted vault to generate tokens that retain the exact format of the original data (Emails, US Phones, SSNs, 16-digit Credit Cards, 9-digit Routing Numbers). Tokens look like real data; the real values are stored encrypted and retrieved via the vault.
+Additionally, the SDK addresses two technical considerations for production use:
+1. **Distributed State Management**: Traditional "vaults" may lose state in multi-node environments. Pluggable backends (Redis, DynamoDB, Memcached) ensure detokenization state is shared across all pods.
+2. **Schema Compatibility**: Downstream tools frequently require specific formats. Format-Preserving Tokenization (Emails, US Phones, SSNs, etc.) generates tokens that retain the format of original data.
 
 ### How We Handle Data (Local-First by Default)
 
@@ -49,7 +49,7 @@ The Data Plane is the open-source, transparent, auditable runtime execution laye
 *   **JIT Cryptography Engine:** The core pre-tool decryption and post-tool encryption hooks that intercept and mutate data in-flight.
 *   **Format-Preserving Tokenization Router:** Ensures downstream databases and strict schemas don't break when handed a token. Tokens look like real data; the real values are stored encrypted and retrieved via the vault.
 *   **Pluggable Distributed Vaults:** Support for enterprise-native caching layers (Redis, DynamoDB, Memcached) to ensure horizontally-scaled edge agents have synchronized access to detokenization mapping.
-*   **Local Audit Logger:** An asynchronous AuditLogger that buffers privacy events to a local SQLite database and emits structured JSON logs for SIEM ingestion.
+*   **Local Audit Logger:** An asynchronous AuditLogger that buffers privacy events in memory and emits structured JSON logs to stdout for SIEM ingestion.
 
 ---
 
@@ -79,37 +79,151 @@ const client = new MaskClient({
 const safeToken = await client.encode("user@tenant.com");
 ```
 
-### 3. Heuristic Safety mathematically guaranteed
-It is catastrophic if an SDK misidentifies a user's *real* SSN as a "token" and accidentally passes it in plaintext to an LLM. Mask's `looksLikeToken()` heuristic algorithm strictly uses universally invalid prefixes. 
-* SSN tokens always begin with `000` (The Social Security Administration has never issued an Area Number of 000).
+### 3. Collision Avoidance
+Mask prevents the misidentification of real data as tokens by using universally invalid prefixes for token generation:
+* SSN tokens always begin with `000` (The Social Security Administration does not issue Area Numbers of 000).
 * Routing tokens always begin with `0000` (The Federal Reserve valid range starts at 01).
 * Credit Card tokens use the `4000-0000-0000` Visa reserved test BIN. 
-By generating statistically impossible tokens, Mask guarantees it will never accidentally swallow real PII.
+
+This prefix-based approach ensures that the SDK does not inadvertently process valid PII as an existing token.
+
+Additional collision-proof prefixes for international identifiers:
+* Turkish TCID tokens use the `990000` prefix (no valid Kimlik number starts with `99`).
+* Saudi NID tokens use the `100000` prefix (length-constrained to avoid overlap with real IDs).
+* UAE Emirates ID tokens use the `784-0000-` prefix (zeroed sub-fields are structurally invalid).
+* IBAN tokens zero the check digits (`XX00...`), which always fails ISO 7064 Mod-97 verification.
 
 ### 4. Enterprise Async Support
 Mask is built from the ground up for high-concurrency Node.js environments. All core operations are asynchronous and promised-based. Calling `encode()`, `decode()`, or `scanAndTokenize()` allows your event loop to remain unblocked while handling PII tokenization tasks.
 
-### 5. Pluggable Key Providers (AWS KMS / HashiCorp Vault)
-For zero-trust environments, `MASK_ENCRYPTION_KEY` no longer needs to live in a static environment variable. Developers can fetch secrets dynamically from AWS KMS, Azure Key Vault, or HashiCorp Vault at runtime and inject them into the `CryptoEngine`.
+### 5. Pluggable Key Management (Enterprise KMS)
+For zero-trust environments, `MASK_ENCRYPTION_KEY` can be managed outside of static environment variables. Mask supports a pluggable `BaseKeyProvider` architecture that allows you to fetch secrets dynamically from dedicated Key Management Services.
+
+#### Supported Providers
+*   **EnvKeyProvider (Default)**: Reads from `MASK_ENCRYPTION_KEY` and `MASK_MASTER_KEY`.
+*   **AwsKmsKeyProvider (Stub)**: Placeholder for AWS KMS. Requires implementation of `@aws-sdk/client-kms`.
+*   **AzureKeyVaultProvider (Stub)**: Placeholder for Azure Key Vault. Requires implementation of `@azure/keyvault-keys`.
+*   **HashiCorpVaultProvider (Stub)**: Placeholder for HashiCorp Vault.
+
+> [!NOTE]
+> All KMS stub providers are designed for **Fail-Shut** operation. If you attempt to use a stub provider that is not yet implemented, the SDK will throw an `Error` rather than fall back to insecure defaults.
 
-### 6. Remote NLP Scanning
-Performance-sensitive deployments can now offload the NLP models (like spaCy or Presidio) to a centralized service using the `RemotePresidioScanner`. This permits "lightweight" edge workers (e.g., AWS Lambda) to run Mask with near-zero memory footprint.
+#### Example: Implementing a Custom Provider
+If you need to support a specific KMS today, you can easily implement the `BaseKeyProvider` interface:
+
+```typescript
+import { BaseKeyProvider, setKeyProvider } from 'mask-privacy/core/key_provider';
+
+class MyCustomKmsProvider extends BaseKeyProvider {
+  async getEncryptionKey(): Promise<string> {
+    // Logic to fetch from your KMS
+    return "your-secret-key";
+  }
+  async getMasterKey(): Promise<string> {
+    return "your-master-key";
+  }
+}
+
+setKeyProvider(new MyCustomKmsProvider());
+```
+
+### 6. Local NLP Scanning (Default)
+Performance-sensitive deployments utilize the built-in `LocalTransformersScanner` by default. This uses HuggingFace Transformers to run PII detection locally within your Node.js process, eliminating the need for external NLP services.
 
 ### 7. Sub-string Detokenization
 Mask includes the ability to detokenize PII embedded within larger text blocks (like email bodies or chat messages). `detokenizeText()` uses high-performance regex to find and restore all tokens within a paragraph before they hit your tools.
 
-## Installation and Setup
+## Multilingual PII Detection (Waterfall Pipeline)
+
+Mask is built for the global enterprise. While many privacy tools are English-centric, the TypeScript SDK implements a **3-Tier Waterfall Detection** strategy designed for high-performance PII detection across 8 major languages using local ONNX models.
+
+### Supported Language Matrix
+
+Mask provides first-class support for the following languages:
+
+| Language | Code | Tier 0 (DLP) | Tier 2 (NLP Engine) |
+| :--- | :--- | :--- | :--- |
+| **English** | `en` | ✅ Full | DistilBERT (Simple) |
+| **Spanish** | `es` | ✅ Full | BERT Multilingual |
+| **French** | `fr` | ✅ Full | BERT Multilingual |
+| **German** | `de` | ✅ Full | BERT Multilingual |
+| **Turkish** | `tr` | ✅ Full | BERT Multilingual |
+| **Arabic** | `ar` | ✅ Full | BERT Multilingual |
+| **Japanese** | `ja` | ✅ Full | BERT Multilingual |
+| **Chinese** | `zh` | ✅ Full | BERT Multilingual |
+
+### How the Waterfall Works: The Excising Mechanism
+
+To maintain high performance, the TypeScript SDK does not simply run three separate scans. It uses a **Sequential Mutation** strategy:
+
+1.  **Tier 0 & 1 (The Scouts):** The SDK first runs the high-speed DLP and Regex engines synchronously in the main thread.
+2.  **Immediate Tokenization:** Any PII found by these tiers is **immediately replaced** by a token in the string buffer.
+3.  **Tier 2 (The Heavy Infantry):** The expensive NLP engine (Transformers.js) only scans the *remaining* text. Because the PII has already been "excised" (cut out and replaced with tokens), the NLP engine doesn't waste compute on data already identified.
+4.  **Bypass Logic:** All tiers are "token-aware." If a scan encounter a string that is already a Mask token, it skips it entirely, preventing redundant processing or "double-tokenization."
+
+---
+
+### Configuration & Environment Variables
+
+Configure your multilingual environment using standard variables. These are parsed at runtime when the `LocalTransformersScanner` is initialized.
+
+| Variable | Default | Description |
+| :--- | :--- | :--- |
+| `MASK_LANGUAGES` | `en` | Comma-separated list of languages (e.g., `en,es,fr,ar`). |
+| `MASK_NLP_MODEL` | *(varies)* | Override the default model (e.g., `Xenova/bert-base-multilingual-cased-ner-hrl`). |
+| `MASK_MODEL_CACHE_DIR` | `~/.cache` | Local directory for storing serialized ONNX models. |
+| `MASK_NLP_MAX_WORKERS` | `4` | Number of worker processes/threads for NLP analysis. |
+| `MASK_NLP_TIMEOUT_SECONDS` | `60` | Max seconds for a scan before timing out. |
+
+---
+
+### Automatic Model Management
+
+The TypeScript SDK manages AI models automatically via the **Transformers.js** runtime. 
 
-Install the core SDK via npm:
+#### 1. Automatic Downloads
+When you set `MASK_LANGUAGES` to include non-English languages, the scanner will automatically download the multilingual BERT model from Hugging Face on the first execution and cache it locally.
+
+#### 2. Pre-Warming (Performance)
+Upon initialization, the `LocalTransformersScanner` starts a worker pool and "pre-warms" the models. This ensures that the first real request doesn't suffer from high "cold-start" latency.
+
+#### 3. Air-Gapped / Offline Environments
+For high-security environments, you can pre-cache models. Run this script in your build pipeline:
 ```bash
-npm install mask-privacy
+# Set a custom cache directory
+export MASK_MODEL_CACHE_DIR="./models"
+
+# Run a dummy scan to trigger the download
+node -e "require('mask-privacy').getScanner().scanAndTokenize('John Doe')"
+
+# Bundle the './models' folder with your container
 ```
 
-Add peer dependencies depending on your infrastructure:
+---
+
+### Performance & Latency Benchmarks
+
+*Measured on 4-vCPU 8GB RAM Instance (Node.js 20+)*
+
+| Tier | Engine | Avg. Latency | Rationale |
+| :--- | :--- | :--- | :--- |
+| Tier 0 | DLP (Heuristic) | ~2ms | Main-thread synchronous regex |
+| Tier 1 | Regex (Deterministic) | ~1ms | Main-thread synchronous regex |
+| Tier 2 | Transformers (Local) | 300ms - 800ms | Offloaded to Worker Threads (Piscina) |
+
+**Total Overhead:** Usually **< 400ms** for standard chat lengths. Mask uses an **Excising Mechanism** to ensure that text already identified in Tiers 0/1 is removed from the NLP buffer, significantly accelerating the heavy Transformer inference.
+
+---
+
+### Installing AI Models (Production Ready)
+The TypeScript SDK manages AI models automatically via **Transformers.js**. For production air-gapped environments or to avoid "cold-start" latency, we recommend using the pre-caching CLI:
+
 ```bash
-npm install ioredis          # For Redis vaults
-npm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb # For DynamoDB vaults
-npm install memjs            # For Memcached vaults
+npm install @huggingface/transformers # Required extra
+
+# Pre-cache models for your required languages
+export MASK_LANGUAGES="en,es,fr"
+npx mask-privacy cache-models
 ```
 
 ### Framework Support
@@ -122,6 +236,24 @@ Mask supports major AI frameworks via built-in hooks:
 
 Before running your agents, Mask requires an encryption key and a vault backend selection.
 
+#### Where to set these?
+Select the method that best fits your deployment:
+
+1.  **In a `.env` file (Recommended)**: Create a file in your project root.
+    ```env
+    MASK_LANGUAGES="es,en"
+    MASK_ENCRYPTION_KEY="your-key"
+    ```
+    Then load it using a library like `dotenv`.
+2.  **In your Terminal**:
+    *   **Bash**: `export MASK_LANGUAGES="es,en"`
+    *   **PowerShell**: `$env:MASK_LANGUAGES="es,en"`
+3.  **Directly in TypeScript/Node.js**:
+    ```typescript
+    process.env.MASK_LANGUAGES = "es,en";
+    // Ensure this happens BEFORE initializing the MaskClient
+    ```
+
 #### 1. Configure Keys
 By default, Mask reads from environment variables.
 ```bash
@@ -150,6 +282,28 @@ export MASK_DYNAMODB_REGION=us-east-1
 # For Memcached:
 export MASK_MEMCACHED_HOST=localhost
 export MASK_MEMCACHED_PORT=11211
+
+#### 4. Security Guardrails: Fail-Shut by Default
+
+To prevent accidental data leakage, Mask defaults to a **Fail-Shut** strategy. If the Vault or Key Provider is unreachable, the SDK will throw a `MaskVaultConnectionError`.
+
+> [!IMPORTANT]
+> **Environment Modes:**
+> - **Production (Default):** Fail-Shut enabled. Strictly protects PII.
+> - **Development:** Set `MASK_ENV=dev` to allow "Fail-Open" behavior (PII is returned as-is if the vault fails).
+
+#### 5. Model Pre-caching CLI
+
+For production air-gapped environments or to avoid "cold-start" latency, use the model pre-caching tool:
+
+```bash
+# Cache English and Spanish models
+export MASK_MODEL_CACHE_DIR="./models"
+npx ts-node src/cli.ts cache-models --languages en,es
+```
+
+# Configure MemoryVault cleanup aggressiveness (default: 0.01)
+export MASK_VAULT_CLEANUP_FREQUENCY=0.05
 ```
 
 ---
@@ -236,11 +390,14 @@ The SDK includes a thread-safe, asynchronous AuditLogger built-in.
 
 As your agents encrypt and decrypt data, the logger buffers these privacy events (Action, Agent, TTL). **Raw PII is never logged.** 
 
-Audit events are stored locally in a SQLite database (`.mask_audit.db`) and flushed to stdout as structured JSON. Pipe them into your existing Datadog or Splunk agents for SOC2/HIPAA compliance reporting.
+Audit events are buffered in memory and flushed periodically to stdout as structured JSON. Pipe these logs into your existing Datadog or Splunk agents for SOC2/HIPAA compliance reporting.
+
+The `AuditLogger` includes graceful shutdown hooks (`SIGTERM`, `SIGINT`) to ensure all buffered events are flushed before the process exits.
+
+To prevent memory issues in high-volume environments, the buffer size can be capped:
 
-To disable the local SQLite buffer:
 ```bash
-export MASK_DISABLE_AUDIT_DB=true
+export MASK_AUDIT_MAX_BUFFER_SIZE=5000
 ```
 
 ## License