@link-foundation/sensitive-data-sanitizer 0.0.0-stage → 0.12.0

package/README.md

@@ -1,3 +1,185 @@
-# Temporary Holding Version
+# Sensitive Data Sanitizer
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+An offline JavaScript library, detector framework, and CLI for redacting credentials and personal data from logs, AI sessions, and text files, including logs with invalid UTF-8 bytes. It combines required Secretlint and 221 bundled Gitleaks text rules with contextual credentials, default multilingual name/PII recognition, known values and bounded encoding detection. Optional local Gitleaks, TruffleHog, and Presidio detectors extend the union.
+
+Library calls without a profile replace every actionable finding completely with `[REDACTED]`. Personal findings below the default confidence threshold of 0.5 remain inspectable without replacing ordinary log numbers; credentials always qualify. Publication methods verify full redaction and block output if a required detector fails or finds a residual. Optional typed transformations, compatible partial masks and encoding preservation are described in the API. Findings contain offsets and rule metadata; they omit matched values and source excerpts.
+
+For publishing sessions, use `createSanitizer({profile: 'publication'})`:
+the personal threshold is 0.3, identity values use the fixed first/last-two
+`***` mask, and fake-identity exemptions remain disabled. CLI `redact` defaults
+to this profile. Explicit configuration such as `identityMask: false` selects
+complete redaction. Credentials always receive complete redaction unless an
+explicit credential transformation is selected.
+
+Detection has limits: the default 17-locale name gazetteer has finite vocabulary; unfamiliar names need an additional NER model or known-personal dictionary. New token formats, unsupported encodings, and unconfigured languages can escape detection. Successful verification means the enabled detectors found no residual, not that the input is universally anonymous. See [coverage and limits](docs/COVERAGE.md) and the [issue #1 case study](docs/case-studies/issue-1/README.md).
+
+For real session files whose personal values are confirmed later, use atomic
+two-pass publication:
+
+```sh
+sensitive-data-sanitizer redact session.jsonl --stream --jsonl --two-pass --output safe.jsonl
+```
+
+## Quick Start
+
+Sanitizer APIs require Node.js 22.13 or later. Template arithmetic utilities are available under `@link-foundation/sensitive-data-sanitizer/legacy`; the existing universal example uses that browser-compatible entry. From this checkout:
+
+```sh
+npm ci
+node bin/sensitive-data-sanitizer.js redact session.jsonl --output session.safe.jsonl
+node bin/sensitive-data-sanitizer.js scan ./logs
+node bin/sensitive-data-sanitizer.js history ./repository
+```
+
+The npm package is `@link-foundation/sensitive-data-sanitizer`; the installed executable is `sensitive-data-sanitizer`. This PR prepares its release through Changesets.
+
+First npm publication currently requires owner setup; see the
+[verified release blockers and exact setup steps](docs/case-studies/issue-32/REQUIREMENTS.md#exact-npm-owner-steps).
+
+```js
+import {
+  createSanitizer,
+  knownSecretsFromEnv,
+} from '@link-foundation/sensitive-data-sanitizer';
+
+const sanitizer = createSanitizer({
+  profile: 'publication',
+  knownSecrets: knownSecretsFromEnv(), // Explicitly include local credentials.
+});
+const result = await sanitizer.sanitize('password=short person@example.org');
+console.log(result.text); // password=[REDACTED] [REDACTED]
+```
+
+`createSanitizer()` requires the pinned Secretlint recommended rules to succeed. The synchronous `sanitize()` and `inspect()` functions use the native rules only. `redact()` applies supplied findings without a residual scan. Prefer the asynchronous sanitizer for publication.
+
+## CLI
+
+```sh
+printf 'password=short\n' | sensitive-data-sanitizer redact -
+sensitive-data-sanitizer redact input.txt --in-place
+sensitive-data-sanitizer scan logs --config /private/policy.json
+sensitive-data-sanitizer redact input.txt --gitleaks --trufflehog --output safe.txt
+sensitive-data-sanitizer redact input.txt --presidio /path/presidio-bridge.py --model en_core_web_sm --language en
+sensitive-data-sanitizer redact large-session.jsonl --jsonl --stream --output safe-session.jsonl
+sensitive-data-sanitizer history rewrite ./repository --output ./private-preview
+sensitive-data-sanitizer --help
+```
+
+`--output` creates a new file without clobbering an existing one. `--in-place` replaces the input atomically after verification. Intermediate and output files use mode `0600` on POSIX. Symlinks and NUL-containing binary input are refused. Plain logs preserve invalid UTF-8 bytes exactly; JSON/JSONL and configuration files require valid UTF-8. Directory scans omit `.git` and `node_modules`; skipped binary/malformed-structured/oversized files make the report incomplete. Stdout is written after the complete operation succeeds.
+
+| Exit | Meaning                                                                  |
+| ---- | ------------------------------------------------------------------------ |
+| 0    | Redaction succeeded, or a complete scan found nothing                    |
+| 1    | A complete scan/history audit found sensitive data                       |
+| 2    | Error, detector failure, or incomplete scan; redaction output is blocked |
+
+Defaults: 32 MiB whole-text input bytes, 100,000 findings and 10,000 files. `--stream` permits 1 GiB total with plain-text records up to the total input limit or 8 MiB JSONL records, bounded workers and atomic publication. `--max-bytes` sets the byte limit; JSON config sets the other sanitizer limits. `--paranoid` adds entropy heuristics and increases false positives. `--native-only` explicitly disables Secretlint.
+
+## Configuration
+
+For AI sessions, `--jsonl` preserves JSON records and declares `id`, `type`,
+`name`, and `description` as metadata. `--json` handles a single JSON document;
+use JSONL for streaming. Known private values and credential formats still
+override declarations. Set `structuralFields` in private config to choose your
+own property names, or use an empty array to scan every field normally.
+The library provides `structured: 'json' | 'jsonl'` and
+`engine.sanitizeJsonl(text)` with an empty structural policy by default.
+
+For input above the whole-text limit, pass an iterable to
+`sanitizeStream(source, {structured: 'jsonl', profile: 'publication'})` or use `sanitizeStreamToFile` for
+atomic output. JSONL defaults to two ordered record workers. See [the streaming
+example](examples/stream-jsonl.mjs).
+
+Native coverage includes tolerant passport MRZ fragments, country passport
+formats, PNR/ticket/visa identifiers, contextual birth dates, national IDs and
+messenger phone numbers. `identityMask: true` (the publication/CLI redact default) uses
+2+2 characters with a fixed `***` marker. Opt-in fake identity policies retain
+audited specimen/synthetic values while credentials remain protected.
+See [the API](docs/API.md) and [all issue 13 requirements and research](docs/case-studies/repository-issue-13/REQUIREMENTS.md).
+
+`transformation: {mode: 'fake', key}` generates deterministic names, valid
+dates/MRZs/checksummed IDs and reserved email/domain destinations. Credentials
+remain fully redacted by default. Keep the key private: this preserves relations
+and structure and is pseudonymisation. See [realistic fakes](examples/realistic-fakes.mjs)
+and [the complete issue 24 requirements, research and solution plans](docs/case-studies/issue-24/REQUIREMENTS.md).
+
+Keep policies containing real private values outside Git. Built-in exact public knowledge and narrowly scoped role/domain heuristics are enabled offline. Explicit private literals override public exemptions. Keep custom public policies reviewed against evidence; `--verify-public` separately opts into Wikidata name lookup:
+
+```json
+{
+  "knownSecrets": [],
+  "knownPersonal": [{ "type": "PERSON", "value": "Private Person" }],
+  "publicEntities": [
+    {
+      "type": "EMAIL",
+      "value": "contact@example.org",
+      "source": "https://example.org/contact",
+      "reviewedAt": "2026-10-07"
+    }
+  ]
+}
+```
+
+Custom policies preserve exact reviewed `PERSON`, `ORGANIZATION` and `EMAIL` detections. Built-in public resolvers/documentation IPs are also exempt. Credential detections always win, including a public email used as a password or authorization value. Evidence is an explicit caller decision and does not make every occurrence harmless.
+
+## Extend detection
+
+```js
+import {
+  createSanitizer,
+  createGitleaksDetector,
+  createTrufflehogDetector,
+  createPresidioDetector,
+} from '@link-foundation/sensitive-data-sanitizer';
+
+const sanitizer = createSanitizer({
+  detectors: [
+    createGitleaksDetector(),
+    createTrufflehogDetector(),
+    createPresidioDetector({ args: ['/absolute/path/presidio-bridge.py'] }),
+  ],
+});
+```
+
+Install external binaries and language models separately. TruffleHog runs with verification and updates disabled. The Presidio example checks for an installed model and never downloads it at analysis time. Each selected detector is required, including during residual verification. See [the API and adapter contract](docs/API.md), [combined scanners](examples/combined-scanners.mjs), and [the local NER bridge](examples/presidio-bridge.py).
+
+The framework also accepts Gitleaks, TruffleHog, detect-secrets, Presidio, or explicitly typed offset reports. Cloud reports can be converted locally; the package itself makes no cloud requests.
+
+For CLI name recognition, `--presidio PATH` selects a local Python JSON bridge. `--python COMMAND` selects its interpreter, and `--model` / `--language` select an already installed model. These options require the bridge and participate in final-output verification.
+
+## Git history
+
+`history` audits reachable blobs, commit messages/authors, annotated tags, and reflog-reachable objects without changing refs. It can find deleted files. It reports object hashes and safe findings, and refuses limits or reports skipped binary objects as incomplete. The `history rewrite SOURCE --output NEW_DIRECTORY` command previews private replacement/mailmap files in a fresh mirror clone; add `--apply` to rewrite only that clone and re-audit blobs, metadata, paths and refs. It never pushes. Remote caches, forks, artifacts and unreachable objects require separate remediation; see the [history guide](docs/HISTORY-REMEDIATION.md).
+
+## Contributing
+
+```sh
+npm test
+npm run check
+bun test --timeout 30000
+deno test --allow-read --allow-env
+```
+
+See [contributing](docs/CONTRIBUTING.md). `add`, `multiply` and `delay` live in the `legacy` subpath, with the `sensitive-data-sanitizer-legacy` CLI and retained universal app; inherited pipeline instructions are in [the template guide](docs/TEMPLATE-PIPELINE.md).
+
+### Auto-regenerated preview screenshots
+
+The universal app's preview images regenerate on pushes to the default branch. Run `npm run example:web:preview-images` to regenerate them locally; see [the template guide](docs/TEMPLATE-PIPELINE.md) for setup.
+
+For shared tool logs, `--credentials-only --env-secrets --gh-config` preserves
+public account details while redacting credential patterns and exact local
+secret values. Environment discovery matches credential-related variable names,
+including AI-provider keys; `--gh-config` reads every host/account token from the
+local GitHub CLI `hosts.yml`. Both local sources are opt-in. Literal references
+such as `${{ secrets.NPM_TOKEN }}` and `$(gh auth token)` remain readable; actual
+credential values are still redacted.
+
+Long plain-text records are held until a safe boundary, including credentials
+longer than a 64 KiB input chunk. Memory grows with the largest held record;
+set `--max-record-bytes` for a smaller deployment budget. Crossing an explicit
+record or total limit blocks atomic publication. See [the tool-log integration
+example](examples/shared-tool-logs.mjs) and [API byte handling](docs/API.md).
+
+## License
+
+Released under the [Unlicense](LICENSE). External components keep their own licenses; see [component provenance](docs/case-studies/issue-1/COMPETITORS.md).

package/bin/legacy-example-package-name.js

@@ -0,0 +1,95 @@
+#!/usr/bin/env node
+
+import { readFileSync, realpathSync } from 'node:fs';
+import { fileURLToPath, URL } from 'node:url';
+
+import { add, multiply } from '../src/legacy.js';
+
+const COMMANDS = {
+  add,
+  multiply,
+};
+
+function usage() {
+  return [
+    'Usage: sensitive-data-sanitizer-legacy <command> <left> <right>',
+    '',
+    'Commands:',
+    '  add       Add two numbers',
+    '  multiply  Multiply two numbers',
+    '',
+    'Options:',
+    '  --help     Show this help',
+    '  --version  Show package version',
+  ].join('\n');
+}
+
+function readVersion() {
+  const packageUrl = new URL('../package.json', import.meta.url);
+  const packageJson = JSON.parse(readFileSync(packageUrl, 'utf8'));
+  return packageJson.version;
+}
+
+function parseNumber(value, label) {
+  const parsed = Number(value);
+
+  if (!Number.isFinite(parsed)) {
+    throw new Error(`${label} must be a finite number`);
+  }
+
+  return parsed;
+}
+
+export function runCli(
+  argv,
+  { stderr = console.error, stdout = console.log } = {}
+) {
+  const [command, left, right] = argv;
+
+  if (!command || command === '--help' || command === '-h') {
+    stdout(usage());
+    return 0;
+  }
+
+  if (command === '--version' || command === '-v') {
+    stdout(readVersion());
+    return 0;
+  }
+
+  const operation = COMMANDS[command];
+
+  if (!operation) {
+    stderr(`Unknown command: ${command}`);
+    stderr(usage());
+    return 1;
+  }
+
+  try {
+    const leftNumber = parseNumber(left, 'left');
+    const rightNumber = parseNumber(right, 'right');
+    stdout(String(operation(leftNumber, rightNumber)));
+    return 0;
+  } catch (error) {
+    stderr(error.message);
+    return 1;
+  }
+}
+
+function isCliEntryPoint() {
+  if (!process.argv[1]) {
+    return false;
+  }
+
+  try {
+    return (
+      realpathSync(process.argv[1]) ===
+      realpathSync(fileURLToPath(import.meta.url))
+    );
+  } catch {
+    return false;
+  }
+}
+
+if (isCliEntryPoint()) {
+  process.exitCode = runCli(process.argv.slice(2));
+}