@privacyscrubber/sdk 2.2.5 → 2.2.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE.md +1 -1
- package/README.md +195 -32
- package/cli.js +120 -4
- package/examples/langchain-rag.js +81 -0
- package/examples/llamaindex-rag.js +81 -0
- package/examples/vector-guard-rag.js +137 -0
- package/index.js +26 -7
- package/package.json +2 -1
- package/ps-license-manager.js +1 -1
- package/ps-pii-engine.cjs +1 -1
- package/scrubber-core.cjs +1 -1
package/LICENSE.md
CHANGED
|
@@ -7,7 +7,7 @@ Subject to the terms and conditions of this Agreement, BrandMeWeb grants you a p
|
|
|
7
7
|
|
|
8
8
|
### 2. Commercial & Enterprise Production Deployment
|
|
9
9
|
Deploying this SDK in production servers, cloud microservices, customer-facing applications, RAG pipelines, or autonomous AI agent clusters beyond Free Tier quotas requires a valid commercial license:
|
|
10
|
-
- **Developer Pro**:
|
|
10
|
+
- **Developer Pro**: Unlimited internal backend microservice nodes ($299/mo or $2,990/yr).
|
|
11
11
|
- **Enterprise Air-Gapped**: Up to 5 production nodes, AWS Nitro Enclaves / SCIF offline deployment, full 30 Specialized Industry Profiles, SOC 2 Evidence Binder ($12,000/yr).
|
|
12
12
|
|
|
13
13
|
Licensing and key activation: https://privacyscrubber.com/pricing
|
package/README.md
CHANGED
|
@@ -13,6 +13,8 @@
|
|
|
13
13
|
> **In-Process Consumer Privacy Defense & Pre-Emptive Interception Engine**
|
|
14
14
|
> 100% In-Memory Execution. Zero Third-Party Subprocessors. Zero Data Loss. Sub-Millisecond Latency.
|
|
15
15
|
|
|
16
|
+
> ⭐ **Support Zero-Trust Open Source:** If `@privacyscrubber/sdk` protects your pipeline from data leaks, please **[Star this repository](https://github.com/moxno/privacyscrubber-mcp)** or run `gh repo star moxno/privacyscrubber-mcp` in your terminal!
|
|
17
|
+
|
|
16
18
|
The official Node.js / TypeScript programmatic SDK for [PrivacyScrubber.com](https://privacyscrubber.com). Acting as an autonomous in-process consumer data fiduciary, `@privacyscrubber/sdk` intercepts and neutralizes consumer PII and infrastructure secrets at your application boundary before transmission to external AI models or persistent storage—enforcing GDPR, CCPA/CPRA, and HIPAA Safe Harbor compliance without introducing third-party subprocessors or sacrificing LLM reasoning fidelity.
|
|
17
19
|
|
|
18
20
|
---
|
|
@@ -37,7 +39,7 @@ The official Node.js / TypeScript programmatic SDK for [PrivacyScrubber.com](htt
|
|
|
37
39
|
| **Runtime Footprint** | **~150KB (0 dependencies)** | ~500MB (Python + spaCy) | Cloud SDK | Cloud SDK |
|
|
38
40
|
| **OpenAI 1-Line Drop-in** | **Yes (`wrapOpenAI`)** | Custom wrapper required | Custom pipeline | Custom pipeline |
|
|
39
41
|
| **DevOps Secrets Scanning** | **Built-in (AWS, JWT, DBs)** | Custom regex rules | Custom detectors | Custom classifiers |
|
|
40
|
-
| **Cost Predictability** | **Free Tier / Flat $
|
|
42
|
+
| **Cost Predictability** | **Free Tier / Flat $299/mo** | DevOps server maintenance | Pay-per-GB cloud fees | Pay-per-unit API fees |
|
|
41
43
|
|
|
42
44
|
> 📊 **Full Technical Benchmark**: Read the in-depth latency analysis, cold-start profiles, and architecture deep-dive in [docs/benchmarks/sdk-vs-presidio.md](https://github.com/moxno/privacyscrubber-mcp/blob/main/docs/benchmarks/sdk-vs-presidio.md).
|
|
43
45
|
|
|
@@ -45,7 +47,7 @@ The official Node.js / TypeScript programmatic SDK for [PrivacyScrubber.com](htt
|
|
|
45
47
|
|
|
46
48
|
| The Risk You Eliminate | Real-World Incident Cost | The @privacyscrubber/sdk Solution |
|
|
47
49
|
| :--- | :--- | :--- |
|
|
48
|
-
| **1 Leaked AWS Key in Cloud LLM Logs** | **$18,400+** (Emergency key rotation, forensic audit, incident response) | **$0 / $
|
|
50
|
+
| **1 Leaked AWS Key in Cloud LLM Logs** | **$18,400+** (Emergency key rotation, forensic audit, incident response) | **$0 / $299 flat**: Masked to `[AWS_KEY_1]` in local RAM before leaving your node |
|
|
49
51
|
| **50k Customer Records in Vector DB** | **$4.45M** (Average data breach cost, IBM Security Report) | **<1ms latency**: Sanitize streams on-the-fly before embedding vectors |
|
|
50
52
|
| **Cloud DLP Egress Fees & Latency** | **$1,500–$4,000/mo** + 350ms added prompt delay | **0 egress bytes**: Pure in-memory execution, 0 network dependencies |
|
|
51
53
|
| **Third-Party Subprocessor Vendor Audit** | **$25,000+** per enterprise vendor review (SOC 2, DPA, Legal) | **0 Subprocessors**: No third party touches data; zero security review backlog |
|
|
@@ -64,7 +66,7 @@ Backend engineering and InfoSec review boards consistently ask four core archite
|
|
|
64
66
|
Generate signed SHA-256 session integrity receipts for enterprise SIEM ingestion (Splunk, Datadog, Elastic) directly within your backend microservice, proving regulatory compliance (GDPR Art. 25/32, HIPAA § 164.514, SOC 2 CC6.6) with zero telemetry leaving your VPC.
|
|
65
67
|
|
|
66
68
|
> 📚 **Academic & Standards Track SSOT**:
|
|
67
|
-
> - **IETF Specification**: [draft-sibiryakov-ztds-protocol-
|
|
69
|
+
> - **IETF Specification**: [draft-sibiryakov-ztds-protocol-01](https://datatracker.ietf.org/doc/draft-sibiryakov-ztds-protocol/) — Revision 01 · 2026-09-23 · 13 pages
|
|
68
70
|
> - **CERN / Zenodo Foundation**: [DOI 10.5281/zenodo.22058770](https://doi.org/10.5281/zenodo.22058770)
|
|
69
71
|
> - **Center for Open Science (OSF) Benchmark**: [DOI 10.17605/OSF.IO/5BYJF](https://doi.org/10.17605/OSF.IO/5BYJF)
|
|
70
72
|
> - **Law Archive Treatise**: [osf.io/preprints/lawarchive/4wc86/](https://osf.io/preprints/lawarchive/4wc86/)
|
|
@@ -229,10 +231,41 @@ await pipeline(readableLlmStream, restorer, clientResponseStream);
|
|
|
229
231
|
|
|
230
232
|
---
|
|
231
233
|
|
|
232
|
-
### 4. LangChain (LCEL
|
|
234
|
+
### 4. LangChain (LCEL Chains & `createDocumentTransformer`)
|
|
235
|
+
|
|
236
|
+
PrivacyScrubber provides dedicated higher-order components for both LCEL generation chains and RAG pre-ingestion document transformation:
|
|
233
237
|
|
|
234
|
-
|
|
238
|
+
#### 4a. RAG Document Ingestion Transformer (`createDocumentTransformer` / `createLangChainDocumentTransformer`)
|
|
239
|
+
Sanitizes `pageContent` and sensitive metadata keys across LangChain `Document` arrays prior to vector embedding. Automatically tracks session token mappings for downstream answer re-hydration. Zero network egress (<1ms per document):
|
|
235
240
|
|
|
241
|
+
```typescript
|
|
242
|
+
import { createDocumentTransformer } from '@privacyscrubber/sdk';
|
|
243
|
+
|
|
244
|
+
const transformer = createDocumentTransformer({
|
|
245
|
+
profile: 'General',
|
|
246
|
+
sensitiveMetadataKeys: ['account_owner', 'submitter_email'],
|
|
247
|
+
attachTelemetry: true,
|
|
248
|
+
attachTokenMap: false // Avoid persisting token maps into vector storage
|
|
249
|
+
});
|
|
250
|
+
|
|
251
|
+
// Compatible with LangChain document pipelines
|
|
252
|
+
const rawDocs = [
|
|
253
|
+
{
|
|
254
|
+
pageContent: "Account Executive John Doe (john.doe@enterprise.com) closed deal for Acme Corp.",
|
|
255
|
+
metadata: { docId: 'doc_101', account_owner: 'john.doe@enterprise.com' }
|
|
256
|
+
}
|
|
257
|
+
];
|
|
258
|
+
|
|
259
|
+
// Transform documents before vector store ingestion
|
|
260
|
+
const sanitizedDocs = await transformer.transformDocuments(rawDocs);
|
|
261
|
+
// or: const sanitizedDocs = await transformer.invoke(rawDocs);
|
|
262
|
+
|
|
263
|
+
// After LLM generates an answer from retrieved sanitized documents, re-hydrate in memory:
|
|
264
|
+
const { restoredText } = transformer.restore(rawLlmAnswer);
|
|
265
|
+
```
|
|
266
|
+
> Complete runnable recipe: [`npm-sdk/examples/langchain-rag.js`](./examples/langchain-rag.js).
|
|
267
|
+
|
|
268
|
+
#### 4b. Composable LCEL Generation Chains & Middleware
|
|
236
269
|
```typescript
|
|
237
270
|
import { ChatOpenAI } from '@langchain/openai';
|
|
238
271
|
import { createLangChainTransform, PrivacyScrubberEngine } from '@privacyscrubber/sdk';
|
|
@@ -258,15 +291,39 @@ const result = await secureChain("Debug AWS_KEY=AKIAIOSFODNN7EXAMPLE for user al
|
|
|
258
291
|
|
|
259
292
|
---
|
|
260
293
|
|
|
261
|
-
### 5. LlamaIndex.TS (RAG Node
|
|
294
|
+
### 5. LlamaIndex.TS (RAG Node Transform & `createLlamaIndexTransform`)
|
|
295
|
+
|
|
296
|
+
Prevent irreversible PII poisoning in vector stores and comply with GDPR Article 17 (Right to be Forgotten) by transforming documents and nodes prior to indexing:
|
|
262
297
|
|
|
263
|
-
|
|
298
|
+
#### 5a. Native Node Transform (`createLlamaIndexTransform`)
|
|
299
|
+
Transforms `TextNode` and `Document` objects with seamless support for `.text`, getter/setter patterns (`getContent()` / `setContent()`), and metadata tagging:
|
|
264
300
|
|
|
301
|
+
```typescript
|
|
302
|
+
import { createLlamaIndexTransform } from '@privacyscrubber/sdk';
|
|
303
|
+
|
|
304
|
+
const transform = createLlamaIndexTransform({
|
|
305
|
+
profile: 'General',
|
|
306
|
+
attachTelemetry: true
|
|
307
|
+
});
|
|
308
|
+
|
|
309
|
+
// Transform single node or array of nodes/documents
|
|
310
|
+
const sanitizedNodes = transform.transform([
|
|
311
|
+
{
|
|
312
|
+
text: "Patient Alice Brown (555-839-2001, alice.brown@health-secure.org) visited Cardiology.",
|
|
313
|
+
metadata: { doc_id: 'ehr_84920' }
|
|
314
|
+
}
|
|
315
|
+
]);
|
|
316
|
+
|
|
317
|
+
// After LlamaIndex synthesizer returns an answer, re-hydrate in memory:
|
|
318
|
+
const { restoredText } = transform.restore(synthesizedAnswer);
|
|
319
|
+
```
|
|
320
|
+
> Complete runnable recipe: [`npm-sdk/examples/llamaindex-rag.js`](./examples/llamaindex-rag.js).
|
|
321
|
+
|
|
322
|
+
#### 5b. Functional Document Pre-Ingestion Mapping
|
|
265
323
|
```typescript
|
|
266
324
|
import { Document } from 'llamaindex';
|
|
267
325
|
import { sanitize } from '@privacyscrubber/sdk';
|
|
268
326
|
|
|
269
|
-
// Sanitize documents before vector index ingestion (HIPAA & GDPR safe)
|
|
270
327
|
export function sanitizeDocumentsForIndex(docs: Document[]): Document[] {
|
|
271
328
|
return docs.map(doc => {
|
|
272
329
|
const { scrubbedText, telemetry } = sanitize(doc.text, {
|
|
@@ -288,27 +345,59 @@ export function sanitizeDocumentsForIndex(docs: Document[]): Document[] {
|
|
|
288
345
|
|
|
289
346
|
---
|
|
290
347
|
|
|
291
|
-
### 6.
|
|
348
|
+
### 6. Universal Vector Database Ingestion Guard (`createVectorIngestionGuard`)
|
|
292
349
|
|
|
293
|
-
Storing raw PII inside vector embeddings is irreversible and
|
|
350
|
+
Storing raw PII inside vector embeddings (Chroma, Pinecone, Qdrant, Weaviate, Milvus, pgvector) is irreversible and creates severe compliance liability. `createVectorIngestionGuard` provides columnar batch sanitization, query token alignment, and transparent vector store proxy wrapping:
|
|
294
351
|
|
|
295
|
-
```
|
|
296
|
-
import {
|
|
352
|
+
```typescript
|
|
353
|
+
import { createVectorIngestionGuard } from '@privacyscrubber/sdk';
|
|
297
354
|
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
355
|
+
const guard = createVectorIngestionGuard({
|
|
356
|
+
profile: 'General',
|
|
357
|
+
sensitiveMetadataKeys: ['contractor_email', 'billing_contact'],
|
|
358
|
+
attachTelemetry: true
|
|
359
|
+
});
|
|
360
|
+
|
|
361
|
+
// 1. Chroma Columnar Ingestion ({ ids, documents, metadatas })
|
|
362
|
+
const cleanChromaBatch = guard.sanitizeRecords({
|
|
363
|
+
ids: ['doc-1', 'doc-2'],
|
|
364
|
+
documents: [
|
|
365
|
+
"Audit for Michael Scott (mscott@dundermifflin.com), SSN 000-44-5555.",
|
|
366
|
+
"Branch notes for Jim Halpert (jhalpert@dundermifflin.com)."
|
|
367
|
+
],
|
|
368
|
+
metadatas: [
|
|
369
|
+
{ branch: "Scranton", contractor_email: "temp@auditor.com" },
|
|
370
|
+
{ branch: "Scranton" }
|
|
371
|
+
]
|
|
372
|
+
});
|
|
373
|
+
|
|
374
|
+
// 2. Pinecone / Qdrant Record Array Ingestion
|
|
375
|
+
const cleanRecords = guard.sanitizeRecords([
|
|
376
|
+
{
|
|
377
|
+
id: 'vec-1',
|
|
378
|
+
text: "Employee Pam Beesly: pamb@office.com, phone 555-432-1098",
|
|
379
|
+
metadata: { role: "Office Admin" }
|
|
380
|
+
},
|
|
381
|
+
{
|
|
382
|
+
id: 'vec-2',
|
|
383
|
+
payload: {
|
|
384
|
+
text: "Escalation for Dwight Schrute (dschrute@schrutefarms.com)",
|
|
385
|
+
region: "PA"
|
|
386
|
+
}
|
|
387
|
+
}
|
|
388
|
+
]);
|
|
389
|
+
|
|
390
|
+
// 3. Search Query Token Alignment
|
|
391
|
+
// Aligns user search query terms to identical ZTDS tokens in the vector space
|
|
392
|
+
const { query: alignedQuery } = guard.sanitizeQuery("Find agreements for mscott@dundermifflin.com");
|
|
393
|
+
|
|
394
|
+
// 4. Transparent Vector Store Proxy Wrapping
|
|
395
|
+
// Intercepts addDocuments/upsert and similaritySearch/query automatically
|
|
396
|
+
const guardedStore = guard.wrapVectorStore(nativeVectorStore);
|
|
397
|
+
await guardedStore.addDocuments(rawDocs); // writes sanitized tokens only
|
|
398
|
+
const results = await guardedStore.similaritySearch("mscott@dundermifflin.com"); // re-hydrates in RAM
|
|
311
399
|
```
|
|
400
|
+
> Complete runnable recipe: [`npm-sdk/examples/vector-guard-rag.js`](./examples/vector-guard-rag.js).
|
|
312
401
|
|
|
313
402
|
---
|
|
314
403
|
|
|
@@ -423,6 +512,46 @@ test('pipeline prompt does not leak PII to third-party LLM', () => {
|
|
|
423
512
|
assert.ok(!scrubbedText.includes(mockUserData.email));
|
|
424
513
|
assert.ok(!scrubbedText.includes(mockUserData.apiKey));
|
|
425
514
|
});
|
|
515
|
+
```
|
|
516
|
+
|
|
517
|
+
---
|
|
518
|
+
|
|
519
|
+
### 10. Python, CrewAI & AutoGen Interop (Zero-Dependency HTTP Sidecar)
|
|
520
|
+
|
|
521
|
+
Running a Python agent fleet (CrewAI, LangGraph, AutoGen)? You can run `@privacyscrubber/sdk` as an air-gapped local sidecar daemon with sub-millisecond in-memory execution and zero cloud hops:
|
|
522
|
+
|
|
523
|
+
```bash
|
|
524
|
+
# Start local sidecar daemon (in local RAM, zero external network egress):
|
|
525
|
+
npx @privacyscrubber/sdk sidecar --port 3100
|
|
526
|
+
```
|
|
527
|
+
|
|
528
|
+
```python
|
|
529
|
+
# In your Python CrewAI / LangGraph Agent tool:
|
|
530
|
+
import requests
|
|
531
|
+
|
|
532
|
+
class PrivacyScrubberTool:
|
|
533
|
+
def sanitize(self, raw_prompt: str) -> tuple[str, dict]:
|
|
534
|
+
res = requests.post("http://127.0.0.1:3100/sanitize", json={
|
|
535
|
+
"text": raw_prompt,
|
|
536
|
+
"profile": "Dev"
|
|
537
|
+
}).json()
|
|
538
|
+
return res["scrubbedText"], res["tokenMap"]
|
|
539
|
+
|
|
540
|
+
def restore(self, agent_response: str, token_map: dict) -> str:
|
|
541
|
+
res = requests.post("http://127.0.0.1:3100/restore", json={
|
|
542
|
+
"text": agent_response,
|
|
543
|
+
"tokenMap": token_map
|
|
544
|
+
}).json()
|
|
545
|
+
return res["restoredText"]
|
|
546
|
+
```
|
|
547
|
+
|
|
548
|
+
> 🏢 **Enterprise & Production Licensing:**
|
|
549
|
+
> - **Self-Serve Developer SDK ($299/mo flat or $2,990/yr):** Unlimited internal backend nodes, microservices, and RAG pipelines. [Get SDK License](https://privacyscrubber.com/pricing?tier=sdk&utm_source=sdk_readme)
|
|
550
|
+
> - **PrivacyScrubber TEAMS ($99/mo flat):** Unlimited team seats, centralized policy enforcement, encrypted session handoff. [Deploy TEAMS](https://privacyscrubber.com/teams?utm_source=sdk_readme)
|
|
551
|
+
> - **Enterprise Air-Gapped License:** On-premise source code distribution, zero-network custom models. [Contact Enterprise](https://privacyscrubber.com/enterprise?utm_source=sdk_readme)
|
|
552
|
+
|
|
553
|
+
---
|
|
554
|
+
|
|
426
555
|
### 11. Client-Side Website Form Interceptor (`@privacyscrubber/sdk/browser`)
|
|
427
556
|
|
|
428
557
|
Intercept customer support forms, loan applications, and lead generation inputs inside the browser DOM before network transmission. Dispatches safe surrogate tokens to public CRMs (Zendesk, HubSpot) or LLMs while isolating the encrypted `tokenMap` in your private database:
|
|
@@ -476,7 +605,7 @@ console.log(result.scrubbedText);
|
|
|
476
605
|
|
|
477
606
|
PrivacyScrubber works free out-of-the-box with standard developer limits:
|
|
478
607
|
- **Free Tier ($0)**: Up to 15,000 characters per request for General profile; 5,000 characters free quota for specialized industry profiles. Built-in quota for local dev and CI testing without credit card or sign-up.
|
|
479
|
-
- **Developer SDK License ($
|
|
608
|
+
- **Developer SDK License ($299/mo or $2,990/yr)**: Unlimited character throughput across all 30 specialized compliance profiles, unrestricted RAG & microservice batch pipelines, zero quota limits.
|
|
480
609
|
|
|
481
610
|
### Activating Your Commercial License
|
|
482
611
|
|
|
@@ -500,7 +629,7 @@ const result = sanitize(largeTextPayload, {
|
|
|
500
629
|
```
|
|
501
630
|
|
|
502
631
|
To acquire a commercial Developer SDK license:
|
|
503
|
-
|
|
632
|
+
[PrivacyScrubber Developer Licensing](https://privacyscrubber.com/pricing/?utm_source=npm_sdk&utm_medium=readme&utm_campaign=dev_upsell)
|
|
504
633
|
|
|
505
634
|
---
|
|
506
635
|
|
|
@@ -523,10 +652,44 @@ To acquire a commercial Developer SDK license:
|
|
|
523
652
|
- ⭐ **GitHub Repository**: [moxno/privacyscrubber-mcp](https://github.com/moxno/privacyscrubber-mcp) (Star us on GitHub!)
|
|
524
653
|
- 🧪 **DLP Latency Benchmark**: [https://privacyscrubber.com/dlp-speed-test/](https://privacyscrubber.com/dlp-speed-test/)
|
|
525
654
|
|
|
526
|
-
###
|
|
527
|
-
|
|
528
|
-
-
|
|
529
|
-
|
|
655
|
+
### Cookbooks & Integration Recipes (27 Frameworks & Architecture Guides)
|
|
656
|
+
|
|
657
|
+
Explore production-ready integration guides with complete copy-pasteable TypeScript/Node.js snippets, empirical latency benchmarks, and compliance evidence:
|
|
658
|
+
|
|
659
|
+
#### AI Frameworks & RAG Ingestion
|
|
660
|
+
- [LangChain.js PII Redaction: In-Memory Zero-Trust Sanitization (<1ms)](https://privacyscrubber.com/solutions/dev/langchain-pii-masking-nodejs/)
|
|
661
|
+
- [LlamaIndex TS Zero-Egress Sanitization: Pre-Prompt In-Memory Masking](https://privacyscrubber.com/solutions/dev/llamaindex-zero-egress-sanitization/)
|
|
662
|
+
- [Vercel AI SDK Stream Sanitizer: Zero-Egress In-Memory Masking](https://privacyscrubber.com/solutions/dev/vercel-ai-sdk-stream-sanitizer/)
|
|
663
|
+
- [Pinecone Vector Upsert PII Redaction: In-Memory Metadata Masking](https://privacyscrubber.com/solutions/dev/pinecone-vector-upsert-pii-redaction/)
|
|
664
|
+
- [OpenAI Node.js SDK PII Wrapper: 1-Line In-Memory Interceptor](https://privacyscrubber.com/solutions/dev/openai-node-wrapper-pii-masking/)
|
|
665
|
+
- [Anthropic Claude Node.js SDK Sanitizer: Sub-Millisecond PII Masking](https://privacyscrubber.com/solutions/dev/anthropic-claude-node-sdk-sanitizer/)
|
|
666
|
+
- [ChromaDB Local PII Sanitizer: In-Memory Pre-Embedding Redaction](https://privacyscrubber.com/solutions/dev/chromadb-local-pii-sanitizer/)
|
|
667
|
+
- [Qdrant Vector Metadata Scrubber: Zero-Cloud-Egress Ingestion](https://privacyscrubber.com/solutions/dev/qdrant-vector-metadata-scrubber/)
|
|
668
|
+
- [AI Agent Tool Calling Sanitizer: Prevent Terminal & Secrets Leakage](https://privacyscrubber.com/solutions/dev/crewai-autogen-agent-tool-sanitizer/)
|
|
669
|
+
- [Reverse Unscrubbing: Restoring Deterministic PII Tokens in RAM](https://privacyscrubber.com/solutions/dev/reverse-unscrub-llm-response-tokens/)
|
|
670
|
+
|
|
671
|
+
#### Backend Web Frameworks & Microservices
|
|
672
|
+
- [Fastify PII Redaction Middleware: High-Throughput In-Memory Sanitization](https://privacyscrubber.com/solutions/dev/fastify-pii-redaction-middleware/)
|
|
673
|
+
- [Express.js In-Memory PII Masking: Zero-Latency Middleware](https://privacyscrubber.com/solutions/dev/express-in-memory-pii-masking/)
|
|
674
|
+
- [NestJS PII Sanitization Interceptor: Clean Architecture DLP](https://privacyscrubber.com/solutions/dev/nestjs-pii-sanitization-guard/)
|
|
675
|
+
- [Pino & Winston Logger PII Scrubber: In-Memory Log Anonymization](https://privacyscrubber.com/solutions/dev/pino-winston-logger-pii-scrubber/)
|
|
676
|
+
- [Kafka Stream PII Sanitization Worker: High-Throughput Node.js Microservice](https://privacyscrubber.com/solutions/dev/kafka-stream-pii-sanitization-worker/)
|
|
677
|
+
- [ETL & Data Warehouse PII Sanitizer: In-Memory Node.js Ingestion Guard](https://privacyscrubber.com/solutions/dev/dbt-snowflake-etl-pii-sanitizer/)
|
|
678
|
+
|
|
679
|
+
#### Cloud DLP & Presidio Displacement
|
|
680
|
+
- [Microsoft Presidio Alternative for Node.js: 0.8ms In-Memory vs 40ms Docker](https://privacyscrubber.com/solutions/dev/microsoft-presidio-alternative-nodejs/)
|
|
681
|
+
- [Google Cloud DLP Alternative for Node.js: Local In-Memory Sanitization](https://privacyscrubber.com/solutions/dev/google-cloud-dlp-local-in-memory-alternative/)
|
|
682
|
+
- [AWS Comprehend PII Alternative: In-Memory Latency Reduction (<1ms)](https://privacyscrubber.com/solutions/dev/aws-comprehend-pii-latency-reduction/)
|
|
683
|
+
- [Microsoft Presidio Docker vs In-Memory ZTDS: Technical Latency Benchmark](https://privacyscrubber.com/solutions/dev/presidio-docker-vs-ztds-wasm-benchmark/)
|
|
684
|
+
|
|
685
|
+
#### RegTech, FinTech & Healthcare Compliance
|
|
686
|
+
- [HIPAA Safe Harbor Prompt Masking for Node.js: 18 Identifiers Redacted in RAM](https://privacyscrubber.com/solutions/dev/hipaa-safe-harbor-prompt-masking-nodejs/)
|
|
687
|
+
- [ISO 20022 XML Payment Sanitizer: In-Memory SWIFT & IBAN Redaction](https://privacyscrubber.com/solutions/dev/iso-20022-xml-payment-sanitizer-nodejs/)
|
|
688
|
+
- [HL7 FHIR JSON PII Redaction for Node.js: Clinical AI Data Guard](https://privacyscrubber.com/solutions/dev/hl7-fhir-json-pii-redaction-nodejs/)
|
|
689
|
+
- [AWS & JWT Secrets Sanitizer for Backend Logs: Stop Cloud Credential Leaks](https://privacyscrubber.com/solutions/dev/aws-jwt-secrets-sanitizer-backend-logs/)
|
|
690
|
+
- [GDPR Article 25 Data Minimization in RAG: Engineering Compliance Guide](https://privacyscrubber.com/solutions/dev/gdpr-article-25-rag-data-minimization/)
|
|
691
|
+
- [Stripe & PCI DSS 4.0 Credit Card PAN Scrubber in Node.js (<1ms)](https://privacyscrubber.com/solutions/dev/stripe-credit-card-pan-scrubber-nodejs/)
|
|
692
|
+
- [SOC 2 Type II LLM Data Masking SDK: Automated Evidence Generation](https://privacyscrubber.com/solutions/dev/soc2-type-2-llm-data-masking-sdk/)
|
|
530
693
|
|
|
531
694
|
---
|
|
532
695
|
|
|
@@ -551,5 +714,5 @@ The Zero-Trust Data Sanitization (ZTDS) architecture, in-memory deterministic to
|
|
|
551
714
|
- **Specification Title:** *The Zero-Trust Data Sanitization (ZTDS) Protocol for Frontier Artificial Intelligence Ingestion*
|
|
552
715
|
- **Standards Track:** IETF Internet Standard Track
|
|
553
716
|
- **IETF Datatracker:** [https://datatracker.ietf.org/doc/draft-sibiryakov-ztds-protocol/](https://datatracker.ietf.org/doc/draft-sibiryakov-ztds-protocol/)
|
|
554
|
-
- **Archive Plaintext:** [https://www.ietf.org/archive/id/draft-sibiryakov-ztds-protocol-
|
|
717
|
+
- **Archive Plaintext:** [https://www.ietf.org/archive/id/draft-sibiryakov-ztds-protocol-01.txt](https://www.ietf.org/archive/id/draft-sibiryakov-ztds-protocol-01.txt)
|
|
555
718
|
|
package/cli.js
CHANGED
|
@@ -12,6 +12,7 @@
|
|
|
12
12
|
import fs from 'node:fs';
|
|
13
13
|
import path from 'node:path';
|
|
14
14
|
import crypto from 'node:crypto';
|
|
15
|
+
import http from 'node:http';
|
|
15
16
|
import { fileURLToPath } from 'node:url';
|
|
16
17
|
import { sanitize, restore, generateProcurementMemo, StreamingDetokenizer } from './index.js';
|
|
17
18
|
|
|
@@ -53,6 +54,116 @@ if (args.includes('memo') || args.includes('--memo') || args.includes('--procure
|
|
|
53
54
|
process.exit(0);
|
|
54
55
|
}
|
|
55
56
|
|
|
57
|
+
// ── Sidecar / Server Mode: Lightweight Local HTTP & Unix Socket Daemon ──
|
|
58
|
+
if (args.includes('sidecar') || args.includes('--sidecar') || args.includes('serve') || args.includes('--serve')) {
|
|
59
|
+
let port = 3100;
|
|
60
|
+
const portIdx = args.indexOf('--port') !== -1 ? args.indexOf('--port') : args.indexOf('-p');
|
|
61
|
+
if (portIdx !== -1 && args[portIdx + 1]) {
|
|
62
|
+
port = parseInt(args[portIdx + 1], 10);
|
|
63
|
+
}
|
|
64
|
+
let socketPath = null;
|
|
65
|
+
const sockIdx = args.indexOf('--socket');
|
|
66
|
+
if (sockIdx !== -1 && args[sockIdx + 1]) {
|
|
67
|
+
socketPath = args[sockIdx + 1];
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
const server = http.createServer((req, res) => {
|
|
71
|
+
// CORS headers for local loopback
|
|
72
|
+
res.setHeader('Access-Control-Allow-Origin', '*');
|
|
73
|
+
res.setHeader('Access-Control-Allow-Methods', 'GET, POST, OPTIONS');
|
|
74
|
+
res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization');
|
|
75
|
+
|
|
76
|
+
if (req.method === 'OPTIONS') {
|
|
77
|
+
res.writeHead(204);
|
|
78
|
+
res.end();
|
|
79
|
+
return;
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
if (req.method === 'GET' && (req.url === '/' || req.url === '/health')) {
|
|
83
|
+
res.writeHead(200, { 'Content-Type': 'application/json' });
|
|
84
|
+
res.end(JSON.stringify({ status: 'ok', engine: 'ZTDS', tier: 'Developer SDK', version: '2.2.6' }));
|
|
85
|
+
return;
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
if (req.method === 'POST') {
|
|
89
|
+
let bodyStr = '';
|
|
90
|
+
req.on('data', chunk => {
|
|
91
|
+
bodyStr += chunk;
|
|
92
|
+
if (bodyStr.length > 50 * 1024 * 1024) {
|
|
93
|
+
res.writeHead(413, { 'Content-Type': 'application/json' });
|
|
94
|
+
res.end(JSON.stringify({ error: 'Payload Too Large' }));
|
|
95
|
+
req.destroy();
|
|
96
|
+
}
|
|
97
|
+
});
|
|
98
|
+
|
|
99
|
+
req.on('end', () => {
|
|
100
|
+
try {
|
|
101
|
+
const payload = bodyStr ? JSON.parse(bodyStr) : {};
|
|
102
|
+
if (req.url === '/scrub' || req.url === '/sanitize') {
|
|
103
|
+
const text = payload.text || '';
|
|
104
|
+
const opts = payload.options || { profile: payload.profile || 'General', detectSecrets: true };
|
|
105
|
+
const start = process.hrtime.bigint();
|
|
106
|
+
const result = sanitize(text, opts);
|
|
107
|
+
const end = process.hrtime.bigint();
|
|
108
|
+
const durationUs = Number(end - start) / 1000;
|
|
109
|
+
|
|
110
|
+
res.writeHead(200, { 'Content-Type': 'application/json' });
|
|
111
|
+
res.end(JSON.stringify({
|
|
112
|
+
sanitizedText: result.scrubbedText,
|
|
113
|
+
tokenMap: result.tokenMap,
|
|
114
|
+
telemetry: {
|
|
115
|
+
...result.telemetry,
|
|
116
|
+
latencyUs: durationUs,
|
|
117
|
+
latencyMs: (durationUs / 1000).toFixed(3)
|
|
118
|
+
}
|
|
119
|
+
}));
|
|
120
|
+
return;
|
|
121
|
+
}
|
|
122
|
+
|
|
123
|
+
if (req.url === '/unscrub' || req.url === '/restore') {
|
|
124
|
+
const text = payload.text || '';
|
|
125
|
+
const tokenMap = payload.tokenMap || {};
|
|
126
|
+
const { restoredText, restoredCount } = restore(text, tokenMap);
|
|
127
|
+
res.writeHead(200, { 'Content-Type': 'application/json' });
|
|
128
|
+
res.end(JSON.stringify({ restoredText, restoredCount }));
|
|
129
|
+
return;
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
res.writeHead(404, { 'Content-Type': 'application/json' });
|
|
133
|
+
res.end(JSON.stringify({ error: 'Endpoint Not Found. Supported: /scrub, /unscrub, /health' }));
|
|
134
|
+
} catch (err) {
|
|
135
|
+
res.writeHead(400, { 'Content-Type': 'application/json' });
|
|
136
|
+
res.end(JSON.stringify({ error: 'Invalid JSON payload', message: err.message }));
|
|
137
|
+
}
|
|
138
|
+
});
|
|
139
|
+
return;
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
res.writeHead(405, { 'Content-Type': 'application/json' });
|
|
143
|
+
res.end(JSON.stringify({ error: 'Method Not Allowed' }));
|
|
144
|
+
});
|
|
145
|
+
|
|
146
|
+
if (socketPath) {
|
|
147
|
+
if (fs.existsSync(socketPath)) {
|
|
148
|
+
try { fs.unlinkSync(socketPath); } catch (_) {}
|
|
149
|
+
}
|
|
150
|
+
server.listen(socketPath, () => {
|
|
151
|
+
console.log(`[ZTDS Sidecar] Listening on Unix socket: ${socketPath}`);
|
|
152
|
+
console.log(`[ZTDS Sidecar] Ready for Python/Go local IPC with <1ms in-memory execution.`);
|
|
153
|
+
});
|
|
154
|
+
} else {
|
|
155
|
+
server.listen(port, '127.0.0.1', () => {
|
|
156
|
+
console.log(`[ZTDS Sidecar] HTTP Daemon listening on http://127.0.0.1:${port}`);
|
|
157
|
+
console.log(`[ZTDS Sidecar] Endpoints: POST /scrub | POST /unscrub | GET /health`);
|
|
158
|
+
console.log(`[ZTDS Sidecar] Latency: <1.2ms (Zero network hops, local loopback)`);
|
|
159
|
+
});
|
|
160
|
+
}
|
|
161
|
+
} else {
|
|
162
|
+
runCli();
|
|
163
|
+
}
|
|
164
|
+
|
|
165
|
+
function runCli() {
|
|
166
|
+
|
|
56
167
|
// ── Demo Command: Live In-Memory Benchmark & Zero-Trust Inspection ──
|
|
57
168
|
if (args.includes('demo') || args.includes('--demo')) {
|
|
58
169
|
const sampleInput = `DevOps Incident Report #4829
|
|
@@ -94,7 +205,8 @@ ${res.scrubbedText.split('\n').map(l => `║ ${l.slice(0, 62).padEnd(64)}║`).
|
|
|
94
205
|
║ NEXT STEPS & PROCUREMENT: ║
|
|
95
206
|
║ * Copy CISO Approval Memo : npx @privacyscrubber/sdk memo ║
|
|
96
207
|
║ * CI/CD PR Stamp : cat prompt.txt | sdk --verify-ci ║
|
|
97
|
-
║ *
|
|
208
|
+
║ * Star on GitHub : github.com/moxno/privacyscrubber-mcp║
|
|
209
|
+
║ * Unlimited SDK ($299/mo) : https://privacyscrubber.com/pricing ║
|
|
98
210
|
╚${border}╝
|
|
99
211
|
`);
|
|
100
212
|
process.exit(0);
|
|
@@ -109,10 +221,12 @@ Usage:
|
|
|
109
221
|
npx @privacyscrubber/sdk --verify-ci <fileOrText> [options]
|
|
110
222
|
npx @privacyscrubber/sdk memo [options]
|
|
111
223
|
npx @privacyscrubber/sdk demo
|
|
224
|
+
npx @privacyscrubber/sdk sidecar [--port 3100] [--socket /path/to.sock]
|
|
112
225
|
|
|
113
226
|
Commands:
|
|
114
227
|
memo Generate executive CISO & DPO procurement memo
|
|
115
228
|
demo Run live in-memory benchmark (<0.1ms latency check)
|
|
229
|
+
sidecar, serve Run zero-dependency local HTTP/Unix daemon (<1ms in-memory)
|
|
116
230
|
|
|
117
231
|
Options:
|
|
118
232
|
--verify-ci Generate CI/CD PR verification stamp (audit receipt)
|
|
@@ -136,7 +250,7 @@ Examples:
|
|
|
136
250
|
cat stream.txt | npx @privacyscrubber/sdk --restore --stream --token-map tokens.json
|
|
137
251
|
|
|
138
252
|
Commercial Licensing:
|
|
139
|
-
Developer SDK ($
|
|
253
|
+
Developer SDK ($299/mo flat): https://privacyscrubber.com/pricing?tier=sdk&src=cli_help
|
|
140
254
|
Teams ($99/mo flat): https://privacyscrubber.com/pricing?tier=teams&src=cli_help
|
|
141
255
|
`);
|
|
142
256
|
process.exit(0);
|
|
@@ -203,7 +317,7 @@ function handleExecution(text) {
|
|
|
203
317
|
|
|
204
318
|
if (isVerifyCi) {
|
|
205
319
|
if (!text || !text.trim()) {
|
|
206
|
-
console.log(`> **Verified by PrivacyScrubber (ZTDS Standard)**\n> * **Status:** PASSED · 0 PII / 0 Secrets detected in prompt pipeline (Empty payload)\n> * **Execution:** Local RAM (<1ms) · 0 Network Egress Bytes\n> * **Compliance Spec:** [Zero-Trust Data Sanitization (ZTDS)](https://privacyscrubber.com/solutions/engineering/zero-trust-ai-data-sanitization?src=ci_stamp) | [Developer SDK ($
|
|
320
|
+
console.log(`> **Verified by PrivacyScrubber (ZTDS Standard)**\n> * **Status:** PASSED · 0 PII / 0 Secrets detected in prompt pipeline (Empty payload)\n> * **Execution:** Local RAM (<1ms) · 0 Network Egress Bytes\n> * **Compliance Spec:** [Zero-Trust Data Sanitization (ZTDS)](https://privacyscrubber.com/solutions/engineering/zero-trust-ai-data-sanitization?src=ci_stamp) | ⭐ [Star on GitHub](https://github.com/moxno/privacyscrubber-mcp) | [Developer SDK ($299/mo)](https://privacyscrubber.com/pricing?tier=sdk&src=ci_stamp)`);
|
|
207
321
|
process.exit(0);
|
|
208
322
|
}
|
|
209
323
|
|
|
@@ -216,7 +330,7 @@ function handleExecution(text) {
|
|
|
216
330
|
> * **Status:** PASSED · ${maskedCount} item(s) protected (${telemetry?.riskLevel || 'CLEAN'})
|
|
217
331
|
> * **Execution:** Local RAM (<1ms) · 0 Network Egress Bytes
|
|
218
332
|
> * **Audit Hash:** \`${hash}\` (Zero-Trust Verified)
|
|
219
|
-
> * **Compliance Spec:** [Zero-Trust Data Sanitization (ZTDS)](https://privacyscrubber.com/solutions/engineering/zero-trust-ai-data-sanitization?src=ci_stamp) | [Developer SDK ($
|
|
333
|
+
> * **Compliance Spec:** [Zero-Trust Data Sanitization (ZTDS)](https://privacyscrubber.com/solutions/engineering/zero-trust-ai-data-sanitization?src=ci_stamp) | ⭐ [Star on GitHub](https://github.com/moxno/privacyscrubber-mcp) | [Developer SDK ($299/mo)](https://privacyscrubber.com/pricing?tier=sdk&src=ci_stamp)
|
|
220
334
|
`;
|
|
221
335
|
console.log(stamp.trim());
|
|
222
336
|
process.exit(0);
|
|
@@ -275,3 +389,5 @@ if (inputArg) {
|
|
|
275
389
|
console.error(`Error: No input provided. Use: npx @privacyscrubber/sdk <fileOrText> or pipe stdin.`);
|
|
276
390
|
process.exit(1);
|
|
277
391
|
}
|
|
392
|
+
}
|
|
393
|
+
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* @privacyscrubber/sdk - LangChain RAG Ingestion & Synthesis Recipe
|
|
3
|
+
*
|
|
4
|
+
* Demonstrates:
|
|
5
|
+
* 1. Pre-ingestion document transformation with createDocumentTransformer.
|
|
6
|
+
* 2. Neutralizing PII in pageContent and sensitive metadata keys (e.g. author, user_id).
|
|
7
|
+
* 3. Preserving vector embedding semantic fidelity with deterministic tokens.
|
|
8
|
+
* 4. In-memory re-hydration of synthesized LLM answers with zero data leakage.
|
|
9
|
+
*
|
|
10
|
+
* Usage:
|
|
11
|
+
* node npm-sdk/examples/langchain-rag.js
|
|
12
|
+
*/
|
|
13
|
+
|
|
14
|
+
import { createDocumentTransformer } from '../index.js';
|
|
15
|
+
|
|
16
|
+
async function main() {
|
|
17
|
+
console.log('=== PrivacyScrubber SDK: LangChain RAG Recipe ===\n');
|
|
18
|
+
|
|
19
|
+
// 1. Initialize the LangChain Document Transformer
|
|
20
|
+
// Zero network egress: operates 100% in local process memory (<1ms per document)
|
|
21
|
+
const transformer = createDocumentTransformer({
|
|
22
|
+
profile: 'General',
|
|
23
|
+
sensitiveMetadataKeys: ['account_owner', 'submitter_email'],
|
|
24
|
+
attachTelemetry: true,
|
|
25
|
+
attachTokenMap: false // Do not leak token mappings into the vector database
|
|
26
|
+
});
|
|
27
|
+
|
|
28
|
+
// 2. Sample raw documents loaded from CRM / internal ticket logs
|
|
29
|
+
const rawDocs = [
|
|
30
|
+
{
|
|
31
|
+
pageContent: 'Account Executive John Doe (john.doe@enterprise.com) finalized deal for client Acme Corp under SSN 000-12-3456.',
|
|
32
|
+
metadata: {
|
|
33
|
+
docId: 'doc_101',
|
|
34
|
+
department: 'Enterprise Sales',
|
|
35
|
+
account_owner: 'john.doe@enterprise.com',
|
|
36
|
+
source: 'Salesforce'
|
|
37
|
+
}
|
|
38
|
+
},
|
|
39
|
+
{
|
|
40
|
+
pageContent: 'Technical escalation for customer Jane Smith. Direct contact phone is +1-555-019-2834, email janes@corp.org.',
|
|
41
|
+
metadata: {
|
|
42
|
+
docId: 'doc_102',
|
|
43
|
+
department: 'Customer Success',
|
|
44
|
+
submitter_email: 'janes@corp.org',
|
|
45
|
+
source: 'Zendesk'
|
|
46
|
+
}
|
|
47
|
+
}
|
|
48
|
+
];
|
|
49
|
+
|
|
50
|
+
console.log('--- 1. Raw Documents Before Sanitization ---');
|
|
51
|
+
console.log(JSON.stringify(rawDocs, null, 2));
|
|
52
|
+
|
|
53
|
+
// 3. Transform documents before vector embedding / storage
|
|
54
|
+
const sanitizedDocs = await transformer.transformDocuments(rawDocs);
|
|
55
|
+
|
|
56
|
+
console.log('\n--- 2. Sanitized Documents (Safe for Vector DB Ingestion) ---');
|
|
57
|
+
console.log(JSON.stringify(sanitizedDocs, null, 2));
|
|
58
|
+
|
|
59
|
+
// 4. Verify zero cleartext PII exists in the documents to be embedded
|
|
60
|
+
const containsRawEmail = sanitizedDocs.some(d => d.pageContent.includes('john.doe@enterprise.com'));
|
|
61
|
+
const containsRawSsn = sanitizedDocs.some(d => d.pageContent.includes('000-12-3456'));
|
|
62
|
+
console.log(`\nZero Cleartext Email Verified: ${!containsRawEmail}`);
|
|
63
|
+
console.log(`Zero Cleartext SSN Verified: ${!containsRawSsn}`);
|
|
64
|
+
|
|
65
|
+
// 5. Simulate an LLM synthesizing an answer over retrieved tokenized chunks
|
|
66
|
+
// The LLM operates on the tokens without knowing the customer's true identity:
|
|
67
|
+
const mockLlmResponse = 'Based on document [doc_101], [NAME_1] ([EMAIL_1]) successfully closed the deal for Acme Corp. Contract verified under tax ID [ID_1].';
|
|
68
|
+
console.log('\n--- 3. LLM Answer Synthesized over Vector Space ---');
|
|
69
|
+
console.log(`Raw LLM Output: "${mockLlmResponse}"`);
|
|
70
|
+
|
|
71
|
+
// 6. Restore the answer in local memory for the authorized end user
|
|
72
|
+
const { restoredText } = transformer.restore(mockLlmResponse);
|
|
73
|
+
console.log('\n--- 4. Re-hydrated Answer (Local RAM Only) ---');
|
|
74
|
+
console.log(`Final Response: "${restoredText}"`);
|
|
75
|
+
|
|
76
|
+
// 7. Reset session when switching security context / tenant
|
|
77
|
+
transformer.resetSession();
|
|
78
|
+
console.log('\nSession token map purged. Memory footprint clean.');
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
main().catch(console.error);
|