rehydra 0.1.0

package/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2025 ELAN Languages
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

package/README.md

@@ -0,0 +1,615 @@
+# Rehydra
+
+![License](https://img.shields.io/github/license/rehydra-ai/rehydra)
+![Issues](https://img.shields.io/github/issues/rehydra-ai/rehydra)
+[![codecov](https://codecov.io/github/rehydra-ai/rehydra/graph/badge.svg?token=WX5RI0ZZJG)](https://codecov.io/github/rehydra-ai/rehydra)
+
+On-device PII anonymization module for high-privacy AI workflows. Detects and replaces Personally Identifiable Information (PII) with placeholder tags while maintaining an encrypted mapping for later rehydration.
+
+**Works in Node.js, Bun, and browsers** - zero server-side dependencies required.
+
+## Features
+
+- **Structured PII Detection**: Regex-based detection for emails, phones, IBANs, credit cards, IPs, URLs
+- **Soft PII Detection**: ONNX-powered NER model for names, organizations, locations (auto-downloads on first use if enabled)
+- **Semantic Enrichment**: AI/MT-friendly tags with gender/location attributes for better translations
+- **Secure PII Mapping**: AES-256-GCM encrypted storage of original PII values
+- **Cross-Platform**: Works identically in Node.js, Bun, and browsers
+- **Configurable Policies**: Customizable detection rules, thresholds, and allowlists
+- **Validation & Leak Scanning**: Built-in validation and optional leak detection
+
+## Installation
+
+### Node.js / Bun
+
+```bash
+npm install rehydra
+```
+
+### Browser (with bundler)
+
+```bash
+npm install rehydra onnxruntime-web
+```
+
+### Browser (without bundler)
+
+```html
+<script type="module">
+  // Import directly from your dist folder or CDN
+  import { createAnonymizer } from './node_modules/rehydra/dist/index.js';
+  
+  // onnxruntime-web is automatically loaded from CDN when needed
+</script>
+```
+
+## Quick Start
+
+### Regex-Only Mode (No Downloads Required)
+
+For structured PII like emails, phones, IBANs, credit cards:
+
+```typescript
+import { anonymizeRegexOnly } from 'rehydra';
+
+const result = await anonymizeRegexOnly(
+  'Contact john@example.com or call +49 30 123456. IBAN: DE89370400440532013000'
+);
+
+console.log(result.anonymizedText);
+// "Contact <PII type="EMAIL" id="1"/> or call <PII type="PHONE" id="2"/>. IBAN: <PII type="IBAN" id="3"/>"
+```
+
+### Full Mode with NER (Detects Names, Organizations, Locations)
+
+The NER model is automatically downloaded on first use (~280 MB for quantized):
+
+```typescript
+import { createAnonymizer } from 'rehydra';
+
+const anonymizer = createAnonymizer({
+  ner: { 
+    mode: 'quantized',  // or 'standard' for full model (~1.1 GB)
+    onStatus: (status) => console.log(status),
+  }
+});
+
+await anonymizer.initialize();  // Downloads model if needed
+
+const result = await anonymizer.anonymize(
+  'Hello John Smith from Acme Corp in Berlin!'
+);
+
+console.log(result.anonymizedText);
+// "Hello <PII type="PERSON" id="1"/> from <PII type="ORG" id="2"/> in <PII type="LOCATION" id="3"/>!"
+
+// Clean up when done
+await anonymizer.dispose();
+```
+
+### With Semantic Enrichment
+
+Add gender and location scope for better machine translation:
+
+```typescript
+import { createAnonymizer } from 'rehydra';
+
+const anonymizer = createAnonymizer({
+  ner: { mode: 'quantized' },
+  semantic: { 
+    enabled: true,  // Downloads ~12 MB of semantic data on first use
+    onStatus: (status) => console.log(status),
+  }
+});
+
+await anonymizer.initialize();
+
+const result = await anonymizer.anonymize(
+  'Hello Maria Schmidt from Berlin!'
+);
+
+console.log(result.anonymizedText);
+// "Hello <PII type="PERSON" gender="female" id="1"/> from <PII type="LOCATION" scope="city" id="2"/>!"
+```
+
+## Example: Translation Workflow (Anonymize → Translate → Rehydrate)
+
+The full workflow for privacy-preserving translation:
+
+```typescript
+import { 
+  createAnonymizer, 
+  decryptPIIMap, 
+  rehydrate,
+  InMemoryKeyProvider 
+} from 'rehydra';
+
+// 1. Create a key provider (required to decrypt later)
+const keyProvider = new InMemoryKeyProvider();
+
+// 2. Create anonymizer with key provider
+const anonymizer = createAnonymizer({
+  ner: { mode: 'quantized' },
+  keyProvider: keyProvider
+});
+
+await anonymizer.initialize();
+
+// 3. Anonymize before translation
+const original = 'Hello John Smith from Acme Corp in Berlin!';
+const result = await anonymizer.anonymize(original);
+
+console.log(result.anonymizedText);
+// "Hello <PII type="PERSON" id="1"/> from <PII type="ORG" id="2"/> in <PII type="LOCATION" id="3"/>!"
+
+// 4. Translate (or do other AI workloads that preserve placeholders)
+const translated = await yourAIWorkflow(result.anonymizedText, { from: 'en', to: 'de' });
+// "Hallo <PII type="PERSON" id="1"/> von <PII type="ORG" id="2"/> in <PII type="LOCATION" id="3"/>!"
+
+// 5. Decrypt the PII map using the same key
+const encryptionKey = await keyProvider.getKey();
+const piiMap = await decryptPIIMap(result.piiMap, encryptionKey);
+
+// 6. Rehydrate - replace placeholders with original values
+const rehydrated = rehydrate(translated, piiMap);
+
+console.log(rehydrated);
+// "Hallo John Smith von Acme Corp in Berlin!"
+
+// 7. Clean up
+await anonymizer.dispose();
+```
+
+### Key Points
+
+- **Save the encryption key** - You need the same key to decrypt the PII map
+- **Placeholders are XML-like** - Most translation services preserve them automatically
+- **PII stays local** - Original values never leave your system during translation
+
+## API Reference
+
+### Configuration Options
+
+```typescript
+import { createAnonymizer, InMemoryKeyProvider } from 'rehydra';
+
+const anonymizer = createAnonymizer({
+  // NER configuration
+  ner: {
+    mode: 'quantized',              // 'standard' | 'quantized' | 'disabled' | 'custom'
+    autoDownload: true,             // Auto-download model if not present
+    onStatus: (status) => {},       // Status messages callback
+    onDownloadProgress: (progress) => {
+      console.log(`${progress.file}: ${progress.percent}%`);
+    },
+    
+    // For 'custom' mode only:
+    modelPath: './my-model.onnx',
+    vocabPath: './vocab.txt',
+  },
+  
+  // Semantic enrichment (adds gender/scope attributes)
+  semantic: {
+    enabled: true,                  // Enable MT-friendly attributes
+    autoDownload: true,             // Auto-download semantic data (~12 MB)
+    onStatus: (status) => {},
+    onDownloadProgress: (progress) => {},
+  },
+  
+  // Encryption key provider
+  keyProvider: new InMemoryKeyProvider(),
+  
+  // Custom policy (optional)
+  defaultPolicy: { /* see Policy section */ },
+});
+
+await anonymizer.initialize();
+```
+
+### NER Modes
+
+| Mode | Description | Size | Auto-Download |
+|------|-------------|------|---------------|
+| `'disabled'` | No NER, regex only | 0 | N/A |
+| `'quantized'` | Smaller model, ~95% accuracy | ~280 MB | Yes |
+| `'standard'` | Full model, best accuracy | ~1.1 GB | Yes |
+| `'custom'` | Your own ONNX model | Varies | No |
+
+### Main Functions
+
+#### `createAnonymizer(config?)`
+
+Creates a reusable anonymizer instance:
+
+```typescript
+const anonymizer = createAnonymizer({
+  ner: { mode: 'quantized' }
+});
+
+await anonymizer.initialize();
+const result = await anonymizer.anonymize('text');
+await anonymizer.dispose();
+```
+
+#### `anonymize(text, locale?, policy?)`
+
+One-off anonymization (regex-only by default):
+
+```typescript
+import { anonymize } from 'rehydra';
+
+const result = await anonymize('Contact test@example.com');
+```
+
+#### `anonymizeWithNER(text, nerConfig, policy?)`
+
+One-off anonymization with NER:
+
+```typescript
+import { anonymizeWithNER } from 'rehydra';
+
+const result = await anonymizeWithNER(
+  'Hello John Smith',
+  { mode: 'quantized' }
+);
+```
+
+#### `anonymizeRegexOnly(text, policy?)`
+
+Fast regex-only anonymization:
+
+```typescript
+import { anonymizeRegexOnly } from 'rehydra';
+
+const result = await anonymizeRegexOnly('Card: 4111111111111111');
+```
+
+### Rehydration Functions
+
+#### `decryptPIIMap(encryptedMap, key)`
+
+Decrypts the PII map for rehydration:
+
+```typescript
+import { decryptPIIMap } from 'rehydra';
+
+const piiMap = await decryptPIIMap(result.piiMap, encryptionKey);
+// Returns Map<string, string> where key is "PERSON:1" and value is "John Smith"
+```
+
+#### `rehydrate(text, piiMap)`
+
+Replaces placeholders with original values:
+
+```typescript
+import { rehydrate } from 'rehydra';
+
+const original = rehydrate(translatedText, piiMap);
+```
+
+### Result Structure
+
+```typescript
+interface AnonymizationResult {
+  // Text with PII replaced by placeholder tags
+  anonymizedText: string;
+  
+  // Detected entities (without original text for safety)
+  entities: Array<{
+    type: PIIType;
+    id: number;
+    start: number;
+    end: number;
+    confidence: number;
+    source: 'REGEX' | 'NER';
+  }>;
+  
+  // Encrypted PII mapping (for later rehydration)
+  piiMap: {
+    ciphertext: string;  // Base64
+    iv: string;          // Base64
+    authTag: string;     // Base64
+  };
+  
+  // Processing statistics
+  stats: {
+    countsByType: Record<PIIType, number>;
+    totalEntities: number;
+    processingTimeMs: number;
+    modelVersion: string;
+    leakScanPassed?: boolean;
+  };
+}
+```
+
+## Supported PII Types
+
+| Type | Description | Detection | Semantic Attributes |
+|------|-------------|-----------|---------------------|
+| `EMAIL` | Email addresses | Regex | - |
+| `PHONE` | Phone numbers (international) | Regex | - |
+| `IBAN` | International Bank Account Numbers | Regex + Checksum | - |
+| `BIC_SWIFT` | Bank Identifier Codes | Regex | - |
+| `CREDIT_CARD` | Credit card numbers | Regex + Luhn | - |
+| `IP_ADDRESS` | IPv4 and IPv6 addresses | Regex | - |
+| `URL` | Web URLs | Regex | - |
+| `CASE_ID` | Case/ticket numbers | Regex (configurable) | - |
+| `CUSTOMER_ID` | Customer identifiers | Regex (configurable) | - |
+| `PERSON` | Person names | NER | `gender` (male/female/neutral) |
+| `ORG` | Organization names | NER | - |
+| `LOCATION` | Location/place names | NER | `scope` (city/country/region) |
+| `ADDRESS` | Physical addresses | NER | - |
+| `DATE_OF_BIRTH` | Dates of birth | NER | - |
+
+## Configuration
+
+### Anonymization Policy
+
+```typescript
+import { createAnonymizer, PIIType } from 'rehydra';
+
+const anonymizer = createAnonymizer({
+  ner: { mode: 'quantized' },
+  defaultPolicy: {
+    // Which PII types to detect
+    enabledTypes: new Set([PIIType.EMAIL, PIIType.PHONE, PIIType.PERSON]),
+    
+    // Confidence thresholds per type (0.0 - 1.0)
+    confidenceThresholds: new Map([
+      [PIIType.PERSON, 0.8],
+      [PIIType.EMAIL, 0.5],
+    ]),
+    
+    // Terms to never treat as PII
+    allowlistTerms: new Set(['Customer Service', 'Help Desk']),
+    
+    // Enable semantic enrichment (gender/scope)
+    enableSemanticMasking: true,
+    
+    // Enable leak scanning on output
+    enableLeakScan: true,
+  },
+});
+```
+
+### Custom Recognizers
+
+Add domain-specific patterns:
+
+```typescript
+import { createCustomIdRecognizer, PIIType, createAnonymizer } from 'rehydra';
+
+const customRecognizer = createCustomIdRecognizer([
+  {
+    name: 'Order Number',
+    pattern: /\bORD-[A-Z0-9]{8}\b/g,
+    type: PIIType.CASE_ID,
+  },
+]);
+
+const anonymizer = createAnonymizer();
+anonymizer.getRegistry().register(customRecognizer);
+```
+
+## Data & Model Storage
+
+Models and semantic data are cached locally for offline use.
+
+### Node.js Cache Locations
+
+| Data | macOS | Linux | Windows |
+|------|-------|-------|---------|
+| NER Models | `~/Library/Caches/rehydra/models/` | `~/.cache/rehydra/models/` | `%LOCALAPPDATA%/rehydra/models/` |
+| Semantic Data | `~/Library/Caches/rehydra/semantic-data/` | `~/.cache/rehydra/semantic-data/` | `%LOCALAPPDATA%/rehydra/semantic-data/` |
+
+### Browser Cache
+
+In browsers, data is stored using:
+- **IndexedDB**: For semantic data and smaller files
+- **Origin Private File System (OPFS)**: For large model files (~280 MB)
+
+Data persists across page reloads and browser sessions.
+
+### Manual Data Management
+
+```typescript
+import { 
+  // Model management
+  isModelDownloaded, 
+  downloadModel, 
+  clearModelCache,
+  listDownloadedModels,
+  
+  // Semantic data management
+  isSemanticDataDownloaded,
+  downloadSemanticData,
+  clearSemanticDataCache,
+} from 'rehydra';
+
+// Check if model is downloaded
+const hasModel = await isModelDownloaded('quantized');
+
+// Manually download model with progress
+await downloadModel('quantized', (progress) => {
+  console.log(`${progress.file}: ${progress.percent}%`);
+});
+
+// Check semantic data
+const hasSemanticData = await isSemanticDataDownloaded();
+
+// List downloaded models
+const models = await listDownloadedModels();
+
+// Clear caches
+await clearModelCache('quantized');  // or clearModelCache() for all
+await clearSemanticDataCache();
+```
+
+## Encryption & Security
+
+The PII map is encrypted using **AES-256-GCM** via the Web Crypto API (works in both Node.js and browsers).
+
+### Key Providers
+
+```typescript
+import { 
+  InMemoryKeyProvider,    // For development/testing
+  ConfigKeyProvider,      // For production with pre-configured key
+  KeyProvider,            // Interface for custom implementations
+  generateKey,
+} from 'rehydra';
+
+// Development: In-memory key (generates random key, lost on page refresh)
+const devKeyProvider = new InMemoryKeyProvider();
+
+// Production: Pre-configured key
+// Generate key: openssl rand -base64 32
+const keyBase64 = process.env.PII_ENCRYPTION_KEY;  // or read from config
+const prodKeyProvider = new ConfigKeyProvider(keyBase64);
+
+// Custom: Implement KeyProvider interface
+class SecureKeyProvider implements KeyProvider {
+  async getKey(): Promise<Uint8Array> {
+    // Retrieve from secure storage, HSM, keychain, etc.
+    return await getKeyFromSecureStorage();
+  }
+}
+```
+
+### Security Best Practices
+
+- **Never log the raw PII map** - Always use encrypted storage
+- **Persist the encryption key securely** - Use platform keystores (iOS Keychain, Android Keystore, etc.)
+- **Rotate keys** - Implement key rotation for long-running applications
+- **Enable leak scanning** - Catch any missed PII in output
+
+## Browser Usage
+
+The library works seamlessly in browsers without any special configuration.
+
+### Basic Browser Example
+
+```html
+<!DOCTYPE html>
+<html>
+<head>
+  <title>PII Anonymization</title>
+</head>
+<body>
+  <script type="module">
+    import { 
+      createAnonymizer, 
+      InMemoryKeyProvider,
+      decryptPIIMap,
+      rehydrate
+    } from './node_modules/rehydra/dist/index.js';
+    
+    async function demo() {
+      // Create anonymizer
+      const keyProvider = new InMemoryKeyProvider();
+      const anonymizer = createAnonymizer({
+        ner: { 
+          mode: 'quantized',
+          onStatus: (s) => console.log('NER:', s),
+          onDownloadProgress: (p) => console.log(`Download: ${p.percent}%`)
+        },
+        semantic: { enabled: true },
+        keyProvider
+      });
+      
+      // Initialize (downloads models on first use)
+      await anonymizer.initialize();
+      
+      // Anonymize
+      const result = await anonymizer.anonymize(
+        'Contact Maria Schmidt at maria@example.com in Berlin.'
+      );
+      
+      console.log('Anonymized:', result.anonymizedText);
+      // "Contact <PII type="PERSON" gender="female" id="1"/> at <PII type="EMAIL" id="2"/> in <PII type="LOCATION" scope="city" id="3"/>."
+      
+      // Rehydrate
+      const key = await keyProvider.getKey();
+      const piiMap = await decryptPIIMap(result.piiMap, key);
+      const original = rehydrate(result.anonymizedText, piiMap);
+      
+      console.log('Rehydrated:', original);
+      
+      await anonymizer.dispose();
+    }
+    
+    demo().catch(console.error);
+  </script>
+</body>
+</html>
+```
+
+### Browser Notes
+
+- **First-use downloads**: NER model (~280 MB) and semantic data (~12 MB) are downloaded on first use
+- **ONNX runtime**: Automatically loaded from CDN if not bundled
+- **Offline support**: After initial download, everything works offline
+- **Storage**: Uses IndexedDB and OPFS - data persists across sessions
+
+## Bun Support
+
+This library works with [Bun](https://bun.sh). Since `onnxruntime-node` is a native Node.js addon, Bun uses `onnxruntime-web`:
+
+```bash
+bun add rehydra onnxruntime-web
+```
+
+Usage is identical - the library auto-detects the runtime.
+
+## Performance
+
+| Component | Time (2K chars) | Notes |
+|-----------|-----------------|-------|
+| Regex pass | ~5 ms | All regex recognizers |
+| NER inference | ~100-150 ms | Quantized model |
+| Semantic enrichment | ~1-2 ms | After data loaded |
+| Total pipeline | ~150-200 ms | Full anonymization |
+
+| Model | Size | First-Use Download |
+|-------|------|-------------------|
+| Quantized | ~280 MB | ~30s on fast connection |
+| Standard | ~1.1 GB | ~2min on fast connection |
+| Semantic Data | ~12 MB | ~5s on fast connection |
+
+## Requirements
+
+| Environment | Version | Notes |
+|-------------|---------|-------|
+| Node.js | >= 18.0.0 | Uses native `onnxruntime-node` |
+| Bun | >= 1.0.0 | Requires `onnxruntime-web` |
+| Browsers | Chrome 86+, Firefox 89+, Safari 15.4+, Edge 86+ | Uses OPFS for model storage |
+
+## Development
+
+```bash
+# Install dependencies
+npm install
+
+# Run tests
+npm test
+
+# Build
+npm run build
+
+# Lint
+npm run lint
+```
+
+### Building Custom Models
+
+For development or custom models:
+
+```bash
+# Requires Python 3.8+
+npm run setup:ner              # Standard model
+npm run setup:ner:quantized    # Quantized model
+```
+
+## License
+
+MIT

package/dist/crypto/index.d.ts

@@ -0,0 +1,6 @@
+/**
+ * Crypto Module
+ * Exports encryption utilities for PII map
+ */
+export * from './pii-map-crypto.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/crypto/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/crypto/index.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,cAAc,qBAAqB,CAAC"}

package/dist/crypto/index.js

@@ -0,0 +1,6 @@
+/**
+ * Crypto Module
+ * Exports encryption utilities for PII map
+ */
+export * from './pii-map-crypto.js';
+//# sourceMappingURL=index.js.map

package/dist/crypto/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/crypto/index.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,cAAc,qBAAqB,CAAC"}