agent-sanitizer 2.0.3 → 2.2.0

package/README.md

@@ -162,6 +162,27 @@ await rehydrateRedacted("Edit", toolInput, {
 }); // { updatedInput, context } | { deny } | null — a deny never exposes a secret
 ```
 
+The credential-noun vocabulary — the words that make an identifier name a secret —
+is published as data so a consumer with its own matcher derives it from one list
+instead of forking one. Each noun carries the `uses` it is valid for: `env-name`
+for a matcher that inspects a variable NAME only, `field-value` for one that
+redacts whatever follows `noun = ` (a broad noun there mangles ordinary text, so
+`key` and `pat` are name-only).
+
+```js
+import { createRequire } from "node:module";
+const vocabulary = createRequire(import.meta.url)(
+  "agent-sanitizer/credential-names",
+);
+vocabulary.nouns; // [{ parts: ["api", "key"], uses: ["env-name", "field-value"] }, …]
+```
+
+```python
+from agent_sanitizer.secrets import credential_name_segments
+
+credential_name_segments()  # ("API_KEY", "APIKEY", "ACCESS_KEY", …) — rendered for a NAME matcher
+```
+
 ## Limits
 
 The CLI (and the worker that backs the Python client) rejects any single request

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-sanitizer",
-  "version": "2.0.3",
+  "version": "2.2.0",
   "description": "Defend an agent against hidden-content injection: strip payload-capable invisible Unicode and ANSI, splice out human-invisible HTML, and flag data-exfil URLs in untrusted text before any model sees it.",
   "type": "module",
   "repository": {
@@ -119,10 +119,12 @@
     "./rehydrate": {
       "types": "./types/rehydrate.d.mts",
       "default": "./src/rehydrate.mjs"
-    }
+    },
+    "./credential-names": "./python/agent_sanitizer/secrets/data/credential-names.json"
   },
   "files": [
     "src/*.mjs",
+    "python/agent_sanitizer/secrets/data/credential-names.json",
     "bin/sanitize-cli.mjs",
     "types",
     "LICENSE",

package/python/README.md

@@ -0,0 +1,115 @@
+# agent-sanitizer (Python client)
+
+A thin Python bridge to the [`agent-sanitizer`](https://github.com/AlexanderMattTurner/agent-sanitizer)
+Node.js CLI. The sanitization logic has a single source of truth — the
+JavaScript in `src/` — so this package shells out to the CLI rather than
+re-implementing it, giving a Python pipeline byte-identical verdicts with no
+second implementation to keep in sync.
+
+## Requirements
+
+- **Node.js (>= 22) on `PATH`.** The sanitizer is JavaScript; something has to
+  run it. There is deliberately no pure-Python fallback.
+
+That's it — `pip install agent-sanitizer` and, with Node available, you're
+ready to go. The wheel ships a self-contained, single-file build of the CLI
+(the `src/` logic and its npm dependencies bundled into one `.mjs` at release
+time), so there is no separate JavaScript checkout to clone and no environment
+variable to set. The bundle is a versioned build artifact from `src/`, not a
+hand-maintained port, so it can't drift from the JS.
+
+### Optional: point at your own JS checkout
+
+Set `AGENT_SANITIZER_CLI` to a checkout's `bin/sanitize-cli.mjs` to override the
+bundled CLI (e.g. to run against unreleased `src/` changes). When the module is
+imported directly from a repo checkout, the source CLI is found automatically.
+
+## Usage
+
+```python
+from agent_sanitizer import sanitize
+
+result = sanitize("untrusted text", html=True)
+print(result.cleaned, result.found, result.warnings)
+```
+
+See the package docstring for the full set of entry points (`sanitize_text`,
+`classify_prompt`, `scan_instruction_files`, `clean_file`, and the long-lived
+`Sanitizer` worker).
+
+## Secret redaction (`[secrets]` extra)
+
+The base install is dependency-free. The optional `secrets` extra adds a
+pure-Python secret-redaction engine under `agent_sanitizer.secrets` —
+detect-secrets plus custom detectors, benign-value skipping, cross-line
+reassembly, PEM collapse, and exact-match redaction of caller-supplied env-var
+values. Unlike the sanitizer above it needs **no Node.js**; its only dependency
+is `detect-secrets`, pulled in by the extra:
+
+```bash
+pip install "agent-sanitizer[secrets]"
+```
+
+Every detect-secrets import lives inside this subpackage, so a plain
+`import agent_sanitizer` never touches it. The engine shares the parent
+package's invisible-character SSOT (`agent_sanitizer.invisible`) rather
+than forking it — a fork would be a silent security regression.
+
+### In-process
+
+```python
+from agent_sanitizer.secrets import RedactorConfig, redact, redact_map
+
+redacted, found = redact("aws_key = AKIAIOSFODNN7EXAMPLE")
+# -> ("aws_key = [REDACTED: AWS Access Key]", ["AWS Access Key"])
+
+# Pass provider/host secret values in — config is supplied, never discovered.
+cfg = RedactorConfig(host_cred_vars={"GH_TOKEN": "ghp_realtokenvalue123"})
+redact("tok=ghp_realtokenvalue123", cfg)  # -> ("tok=[REDACTED: GH_TOKEN]", ["GH_TOKEN"])
+```
+
+### The rehydration map contract
+
+`redact_map(text, config)` returns a lossless, two-way view:
+
+```python
+{"text": "<redacted>",
+ "pairs": [{"placeholder": "[REDACTED: …]", "original": "<secret>", "start": <int>}],
+ "found": ["<type>", ...]}
+```
+
+Substituting each `pair["original"]` at its `start` reconstructs the input
+byte-for-byte. `start` is a **Unicode code-point** offset into `text` (the field
+is `start`, not `start_offset`). U+E000 / U+E001 are reserved placeholder
+sentinels: input already containing either is refused with
+`{"unmappable": "input contains reserved sentinel characters"}` (fail closed)
+rather than producing an ambiguous map.
+
+### Daemon vs. one-shot
+
+detect-secrets caches a process-global `secret_type → plugin` mapping, so a fresh
+one-shot call must re-register the plugin set every time — slow under load. Two
+console scripts (installed with the extra) cover both modes:
+
+- **`agent-secret-redactor`** — one-shot: reads text on stdin, writes the
+  redaction result as JSON on stdout. Re-registers plugins per call.
+- **`agent-secret-redactor-daemon <socket-path>`** — long-lived Unix-socket
+  server that configures the plugin set **once** at startup (a warm-up scan
+  primes the cache before it binds, so a bound socket means ready). Wire protocol
+  both directions: a 4-byte big-endian length prefix then that many bytes of
+  UTF-8 JSON. Request `{"text", "map", "web_ingress", "env_secrets"}`; response is
+  the one-shot result, JSON `null` when nothing is redacted, or `{"error": …}` on
+  a scan failure.
+
+Both console scripts import the engine on startup, so they fail loud if the
+`[secrets]` extra (hence `detect-secrets`) is not installed — fail closed, never
+a silent no-op.
+
+## Versioning
+
+This package is versioned in lockstep with the npm
+[`agent-sanitizer`](https://www.npmjs.com/package/agent-sanitizer):
+each release publishes both at the same version from the same commit, and the
+wheel bundles `src/` at exactly that version. So `pip install
+agent-sanitizer==X.Y.Z` and `npm i agent-sanitizer@X.Y.Z` are the
+same underlying logic.

package/python/agent_sanitizer/secrets/data/credential-names.json

@@ -0,0 +1,29 @@
+{
+  "$comment": "The credential-noun vocabulary: the words that make an identifier name a secret. Published so every consumer derives its own matcher from ONE list — a newly recognized noun reaches them all through a version bump instead of N hand edits. Read it from Python via agent_sanitizer.secrets (credential_name_segments / credential_field_name_patterns / non_secret_name_segments) or from JavaScript via the npm subpath export `agent-sanitizer/credential-names`. `parts` are the lowercase words of the noun; a consumer renders them for its own matcher (underscore-joined `API_KEY` and bare-joined `APIKEY` for an env-var NAME, `api[_-]?key` for a `field = value` regex). `uses` says which matcher may use the noun, because the two are not interchangeable: `env-name` matches a variable NAME and never inspects a value, so a broad noun there costs nothing, while `field-value` redacts whatever follows `noun = ` and a broad noun there mangles ordinary text — `key = <20 chars>` is a false-positive flood, so `key`, `pat`, `credential`, `credentials`, `secrets` and `passphrase` are env-name only. `nonSecretSuffixes` are the trailing words that make a credential-shaped name hold a NON-secret (a key's identifier, the public half of a keypair), which a consumer must not redact. Every part is restricted to a-z0-9 so it carries no regex metacharacter; the accessors enforce that and fail closed on a violation, an empty list, or an unknown `uses` value. It sits inside the Python package because a wheel can only ship data under its package directory, while npm's `files`/`exports` can name any path — so ONE physical file backs both ecosystems and there is no copy to drift.",
+  "nouns": [
+    { "parts": ["api", "key"], "uses": ["env-name", "field-value"] },
+    { "parts": ["access", "key"], "uses": ["env-name", "field-value"] },
+    { "parts": ["access", "token"], "uses": ["env-name", "field-value"] },
+    { "parts": ["secret", "key"], "uses": ["env-name", "field-value"] },
+    { "parts": ["client", "secret"], "uses": ["env-name", "field-value"] },
+    { "parts": ["private", "key"], "uses": ["env-name", "field-value"] },
+    { "parts": ["auth", "token"], "uses": ["env-name", "field-value"] },
+    { "parts": ["auth", "key"], "uses": ["env-name", "field-value"] },
+    { "parts": ["authorization"], "uses": ["env-name", "field-value"] },
+    { "parts": ["password"], "uses": ["env-name", "field-value"] },
+    { "parts": ["passwd"], "uses": ["env-name", "field-value"] },
+    { "parts": ["passphrase"], "uses": ["env-name"] },
+    { "parts": ["bearer"], "uses": ["env-name", "field-value"] },
+    { "parts": ["secret"], "uses": ["env-name", "field-value"] },
+    { "parts": ["secrets"], "uses": ["env-name"] },
+    { "parts": ["credential"], "uses": ["env-name"] },
+    { "parts": ["credentials"], "uses": ["env-name"] },
+    { "parts": ["token"], "uses": ["env-name", "field-value"] },
+    { "parts": ["pat"], "uses": ["env-name"] },
+    { "parts": ["key"], "uses": ["env-name"] }
+  ],
+  "nonSecretSuffixes": [
+    ["key", "id"],
+    ["public", "key"]
+  ]
+}