@vitrinka/redact 0.1.0

package/spec/REDACTION-SPEC.md

@@ -0,0 +1,176 @@
+# vitrinka recorder redaction — engine specification
+
+This document defines the redaction semantics every vitrinka capture client
+must implement, in any language. The TypeScript engine in this package is the
+reference implementation; `vectors.json` (same directory) is the conformance
+suite — **an implementation is conformant when it passes every vector**. The
+vitrinka server applies the same semantics at ingest as the load-bearing
+backstop, so client-side redaction is defense in depth: a conformant client
+never lets a secret leave the device, and a broken one still cannot store one.
+
+An implementation MAY redact **more** than required (superset matching is
+explicitly encouraged), except where a vector's `exact` field demands
+byte-identical output — those cases pin down *over*-redaction bugs (benign
+data must survive).
+
+## The policy document
+
+Clients fetch the workspace policy at session start from
+`GET /api/v1/recorder/policy` (same PAT-gated auth as the ingest surface):
+
+```jsonc
+{
+  "extraHeaders":  ["X-Tenant-Secret"],   // extra header names to scrub
+  "extraBodyKeys": ["internalId"],        // extra body keys, matched normalized
+  "patterns":      ["vk_[A-Za-z0-9]+"],   // regexes; every match → "[redacted]"
+  "maskAllText":   false,                  // DOM recorders mask ALL text;
+                                           // pixel recorders degrade to blur
+  "fullFidelity":  false                   // self-host escape hatch — no-op
+}
+```
+
+Rules:
+
+- **Fail closed.** A failed/slow/absent fetch means the built-in defaults —
+  never capture-everything. The fetch must never reject; defaults apply until
+  it resolves. Capture may start before it resolves only in states that are
+  already safe (DOM recording starts with inputs masked).
+- `patterns` served to clients are pre-filtered server-side to a
+  backtracking-safe subset, but each pattern still compiles inside its own
+  try/catch: a bad pattern is skipped (and logged), the rest keep working.
+- `fullFidelity: true` disables all scrubbing. Clients only ever *honor* the
+  field (the server refuses to serve it unless the deployment allows it);
+  clients never default to it.
+
+## Normalization
+
+`norm(name)` = lowercase, then remove every `-` and `_`. All name matching
+below is on normalized names, so `card_number` ≡ `cardNumber` ≡ `Card-Number`.
+
+## Default rule set
+
+Header names (values replaced with `[redacted]`): `authorization`,
+`proxy-authorization`, `cookie`, `set-cookie`, `x-api-key`, `x-auth-token`,
+`x-csrf-token`, `x-amz-security-token`, plus any name whose normalized form
+**ends with** `apikey` or `token`, plus `extraHeaders`.
+
+Body keys (values replaced wholesale, recursively): `password`, `passwd`,
+`secret`, `token`, `access_token`, `refresh_token`, `id_token`,
+`authorization`, `api_key`, `apikey`, `client_secret`, `card`, `card_number`,
+`cvv`, `cvc`, `pin`, `ssn`, plus `extraBodyKeys`.
+
+URL parameters: every sensitive body key, plus the `apikey`/`token` suffix
+rule (`?access_token=…`, `?sas_token=…`, `?my_api_key=…`).
+
+The reference engine additionally **token-matches** names (splitting
+camelCase/snake/kebab into words) against a secret vocabulary (`auth`, `otp`,
+`pin`, `cvv`, `iban`, `ssn`, `credential`, …) and secret phrases
+(`privatekey`, `sessionid`, `cardnumber`, …), so `X-Dev-Auth-Secret` and
+`otpCode` are caught without being listed. Ports SHOULD implement this; the
+vectors only require the normalized-set matching above.
+
+## Surfaces
+
+Every capture surface routes through the matching transform. The replacement
+marker is always the string `[redacted]` — a marker, not deletion, so replay
+still shows the field existed.
+
+### Headers
+
+Sensitive names lose their value; benign values pass through the extra
+patterns. Values are capped (reference: 1024 chars/value, 8 KiB budget per
+map, overflow marked with a `"…": "(truncated)"` entry).
+
+### Bodies
+
+Dispatch on shape and declared content type:
+
+1. **JSON** (first non-space char `{`/`[`), within the structural size limit:
+   parse, walk recursively (bounded depth; beyond it, subtree → `[redacted]`).
+   Sensitive-keyed values replaced wholesale; string values under URL-ish keys
+   (`url`, `href`, `uri`, `location`) get URL scrubbing; every other string
+   passes the extra patterns.
+2. **JSON that fails to parse or exceeds the parse limit** (truncated at a
+   client cap — the cap must NEVER become a bypass): a key-pair fallback regex
+   rewrites `"<sensitive-key>": <value>` pairs, tolerating a missing closing
+   quote; then patterns.
+3. **`application/x-www-form-urlencoded`**: split pairs on **both `&` and
+   `;`** and scrub key-wise, THEN run the free-text scan over the result.
+   Never a platform query parser — `;` handling is the known bypass class
+   (see below). The content-type path must be a SUPERSET of the shapeless
+   free-text path, never a replacement: a benign-keyed value can carry a raw
+   credential the key scrub cannot see (`client_assertion=eyJ…`).
+4. **`multipart/form-data`**: parse parts by the boundary; a part whose
+   `Content-Disposition` form name is sensitive loses its value; other part
+   values (file parts included) get the free-text scan (a part value can be a
+   header line, a bare JWT, or embedded JSON carrying sensitive keys);
+   rebuild with the same boundary. If the body won't parse (truncated, no
+   boundary): FAIL CLOSED — omit the body with a marker. Free-text scanning
+   cannot see multipart structure (a `name="password"` part's raw value sits
+   on its own line), so a partial scan would leak exactly the fields the key
+   scrub exists for.
+5. **Anything else**: free-text scanning (reference behavior, see below) and
+   the extra patterns. At minimum the patterns must run.
+
+**Order with capping**: redact JSON *before* any size cap (capping first
+makes it invalid JSON and silently downgrades it to the weaker fallback);
+cap free text first, then scrub the slice.
+
+### URLs
+
+Scrub the query AND the fragment (implicit-grant OAuth returns tokens as
+`#access_token=…`). Split pairs on **both `&` and `;`** — never
+`URLSearchParams`/`url.ParseQuery`, which mishandle `;` and turn one
+semicolon into a full bypass. Do not trust platform URL parsers to accept
+every captured URL: an unparseable URL still scrubs everything after the
+first `?`/`#` (the reference implementation is string-split end to end).
+A fully benign URL comes back **byte-identical** (no re-encoding churn).
+
+A benign-named parameter whose *decoded* value embeds a secret pair
+(`?next=%2Fcb%3Ftoken%3D…`) also loses its value (reference behavior;
+decoding for inspection fails closed, escape-by-escape, bounded double-decode).
+Hash-route fragments nest a query inside one pair (`#/reset?token=…` yields
+the key `/reset?token`): the part after the last `?` in a pair's key MUST be
+inspected as a name too, or set-matched keys (including policy extras) evade
+the scrub. Truncated-JSON fallback keys MUST be classified with JSON string
+escapes decoded (`pass\\u0077ord` ≡ `password`).
+
+### Free text (reference engine; ports SHOULD follow)
+
+Header-shaped lines (`Key: value`) with sensitive names lose the rest of the
+line; `Authorization`-family names mask mid-line too; standalone scheme
+credentials (`Bearer …`, `Basic …`, `Digest realm=…`) and bare JWTs
+(`eyJ…`) are masked; generic `key=value` / `"key": value` pairs with
+sensitive keys lose their values. Then patterns.
+
+### DOM capture (rrweb-style recorders)
+
+`maskDirectives(rules)`:
+
+- default: `maskAllInputs: true` — input values never recorded. This is the
+  fail-closed *starting* state: recording must begin masked even if policy
+  delivery races.
+- `maskAllText` ⇒ `maskAllText: true` + `maskTextSelector: "*"`.
+- `fullFidelity` ⇒ both off.
+
+### Pixel capture (screenshot recorders)
+
+Screenshots carry real rendered text — there is no DOM to mask. Under
+`maskAllText` (without `fullFidelity`) `pixelPolicy(rules)` = `blur`: degrade
+resolution until text is unreadable while layout survives. Otherwise `none`.
+
+## Event shape invariants
+
+- Recorded event kinds never change (`net` for Expo, `request` for the
+  extension — stored sessions carry them forever).
+- Existing body caps (64 KiB in both kit clients) stay as they are.
+
+## Conformance
+
+Run every case in `vectors.json`. Case surfaces map to:
+`headerName` → the sensitive-header predicate; `headers` → the header map
+transform; `body` → the body transform (with `contentType`); `url` → the URL
+transform. Assertions: `exact` (byte-identical output), `mustNotContain`,
+`mustContain`, `outputIsJson`, `redactedKeys`/`keptEntries`/
+`valueMustNotContain` for header maps. The `policy` field, when present, is
+the policy to compile for that case.

package/spec/vectors.json

@@ -0,0 +1,294 @@
+{
+  "version": 1,
+  "description": "Language-agnostic conformance vectors for the vitrinka recorder redaction engine, ported from the server engine's table tests (vitrinka internal/redact/redact_test.go). Any implementation, in any language, must pass every case. Semantics: REDACTED = \"[redacted]\"; an implementation MAY redact more than required (superset matching), except where `exact` demands byte-identical output.",
+  "cases": [
+    {
+      "name": "default sensitive header names",
+      "surface": "headerName",
+      "sensitive": true,
+      "inputs": [
+        "Authorization", "authorization", "AUTHORIZATION",
+        "Proxy-Authorization", "Cookie", "Set-Cookie",
+        "X-Api-Key", "x-api-key", "X_API_KEY",
+        "X-Auth-Token", "X-CSRF-Token", "X-Amz-Security-Token",
+        "X-Access-Token", "X-Github-Token", "Api-Key", "ApiKey", "My-Service-Api-Key"
+      ]
+    },
+    {
+      "name": "benign header names stay benign",
+      "surface": "headerName",
+      "sensitive": false,
+      "inputs": ["Content-Type", "Accept", "X-Request-Id", "User-Agent", "ETag"]
+    },
+    {
+      "name": "header map: sensitive values redacted, benign kept",
+      "surface": "headers",
+      "policy": { "extraHeaders": ["X-Tenant-Secret"] },
+      "input": {
+        "Authorization": "Bearer sk-live-12345",
+        "Cookie": "session=abcdef",
+        "X-Tenant-Secret": "shh",
+        "Content-Type": "application/json"
+      },
+      "redactedKeys": ["Authorization", "Cookie", "X-Tenant-Secret"],
+      "keptEntries": { "Content-Type": "application/json" }
+    },
+    {
+      "name": "JSON body: nested objects and arrays scrub key-wise",
+      "surface": "body",
+      "contentType": "application/json",
+      "input": "{\"user\":{\"name\":\"lukas\",\"password\":\"hunter2\",\"cards\":[{\"card_number\":\"4111111111111111\",\"cvv\":\"123\",\"label\":\"main\"}]},\"items\":[{\"token\":\"tok_x\"},{\"note\":\"ok\"}],\"accessToken\":\"at\"}",
+      "mustNotContain": ["hunter2", "4111111111111111", "\"123\"", "tok_x", "\"at\""],
+      "mustContain": ["lukas", "main", "ok"],
+      "outputIsJson": true
+    },
+    {
+      "name": "form-encoded body scrubs key-wise",
+      "surface": "body",
+      "contentType": "application/x-www-form-urlencoded; charset=utf-8",
+      "input": "username=lukas&password=hunter2&remember=1",
+      "mustNotContain": ["hunter2"],
+      "mustContain": ["username=lukas", "remember=1"]
+    },
+    {
+      "name": "semicolon separator never bypasses: query",
+      "surface": "url",
+      "input": "https://app.example.com/cb?a=1;access_token=SEMI-SECRET",
+      "mustNotContain": ["SEMI-SECRET"],
+      "mustContain": ["a=1"]
+    },
+    {
+      "name": "semicolon separator never bypasses: query, secret first",
+      "surface": "url",
+      "input": "https://app.example.com/cb?access_token=SEMI-SECRET;x=1",
+      "mustNotContain": ["SEMI-SECRET"]
+    },
+    {
+      "name": "semicolon separator never bypasses: mixed separators",
+      "surface": "url",
+      "input": "https://app.example.com/cb?a=1;access_token=SEMI-SECRET&b=2",
+      "mustNotContain": ["SEMI-SECRET"],
+      "mustContain": ["a=1"]
+    },
+    {
+      "name": "semicolon separator never bypasses: fragment",
+      "surface": "url",
+      "input": "https://app.example.com/cb#access_token=FRAG-SECRET;type=bearer",
+      "mustNotContain": ["FRAG-SECRET"],
+      "mustContain": ["type=bearer"]
+    },
+    {
+      "name": "semicolon separator never bypasses: form body",
+      "surface": "body",
+      "contentType": "application/x-www-form-urlencoded",
+      "input": "y=1;password=FORM-SECRET",
+      "mustNotContain": ["FORM-SECRET"],
+      "mustContain": ["y=1"]
+    },
+    {
+      "name": "benign semicolon query stays byte-identical",
+      "surface": "url",
+      "input": "https://app.example.com/list?a=1;b=2",
+      "exact": "https://app.example.com/list?a=1;b=2"
+    },
+    {
+      "name": "unparseable URL still scrubs query",
+      "surface": "url",
+      "input": "https://app.example.com/a\u0000b?token=CTRL-SECRET&ok=1",
+      "mustNotContain": ["CTRL-SECRET"],
+      "mustContain": ["ok=1"]
+    },
+    {
+      "name": "unparseable URL still scrubs fragment",
+      "surface": "url",
+      "input": "https://app.example.com/a\u0000b#access_token=CTRL-FRAG",
+      "mustNotContain": ["CTRL-FRAG"]
+    },
+    {
+      "name": "unparseable URL still scrubs semicolon pairs",
+      "surface": "url",
+      "input": "https://app.example.com/a\u0000b?x=1;token=CTRL-SEMI",
+      "mustNotContain": ["CTRL-SEMI"],
+      "mustContain": ["x=1"]
+    },
+    {
+      "name": "plain prose body left alone",
+      "surface": "body",
+      "contentType": "text/plain",
+      "input": "plain text mentioning password words but not a form",
+      "exact": "plain text mentioning password words but not a form"
+    },
+    {
+      "name": "multipart/form-data: sensitive field loses its value, benign parts survive",
+      "surface": "body",
+      "contentType": "multipart/form-data; boundary=b123",
+      "input": "--b123\r\nContent-Disposition: form-data; name=\"username\"\r\n\r\nlukas\r\n--b123\r\nContent-Disposition: form-data; name=\"password\"\r\n\r\nMULTI-SECRET\r\n--b123\r\nContent-Disposition: form-data; name=\"avatar\"; filename=\"a.png\"\r\nContent-Type: image/png\r\n\r\nPNGBYTES\r\n--b123--\r\n",
+      "mustNotContain": ["MULTI-SECRET"],
+      "mustContain": ["[redacted]", "lukas", "PNGBYTES", "--b123"]
+    },
+    {
+      "name": "multipart/form-data: policy extra body keys apply (normalized match)",
+      "surface": "body",
+      "policy": { "extraBodyKeys": ["internalId"] },
+      "contentType": "multipart/form-data; boundary=b123",
+      "input": "--b123\r\nContent-Disposition: form-data; name=\"internal_id\"\r\n\r\nEXTRA-SECRET\r\n--b123--\r\n",
+      "mustNotContain": ["EXTRA-SECRET"]
+    },
+    {
+      "name": "truncated multipart FAILS CLOSED — the body is omitted, never half-scanned",
+      "surface": "body",
+      "contentType": "multipart/form-data; boundary=b123",
+      "input": "--b123\r\nContent-Disposition: form-data; name=\"password\"\r\n\r\nTRUNC-SECRET\r\n--b123\r\nContent-Disposition: form-data; name=\"avatar\"; filename=\"a.png\"\r\nContent-Type: image/png\r\n\r\nPNGBYTES\r\n--b1",
+      "mustNotContain": ["TRUNC-SECRET", "PNGBYTES"]
+    },
+    {
+      "name": "hash-route fragment: nested query key is inspected (set-only key)",
+      "surface": "url",
+      "policy": { "extraBodyKeys": ["internalId"] },
+      "input": "https://app.example.com/#/reset?internal_id=NESTED-SECRET",
+      "mustNotContain": ["NESTED-SECRET"]
+    },
+    {
+      "name": "hash-route fragment: nested magic-link token is scrubbed",
+      "surface": "url",
+      "input": "https://app.example.com/#/reset?token=FRAG-ROUTE-SECRET",
+      "mustNotContain": ["FRAG-ROUTE-SECRET"]
+    },
+    {
+      "name": "truncated JSON: escaped key spellings cannot evade the fallback",
+      "surface": "body",
+      "contentType": "application/json",
+      "input": "{\"note\":\"x\",\"pass\\u0077ord\":\"ESCAPED-LEAK\",\"blob\":\"aaaaaaaaaaaaaaaaaaaaaaaaaa",
+      "mustNotContain": ["ESCAPED-LEAK"]
+    },
+    {
+      "name": "form body: benign-keyed JWT credential is still text-scanned (CT path is a superset)",
+      "surface": "body",
+      "contentType": "application/x-www-form-urlencoded",
+      "input": "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&client_assertion=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJhZG1pbiJ9.SFlKc2VjcmV0U2ln",
+      "mustNotContain": ["eyJhbGciOiJIUzI1NiJ9"],
+      "mustContain": ["grant_type="]
+    },
+    {
+      "name": "multipart: benign part carrying a Bearer header line is text-scanned",
+      "surface": "body",
+      "contentType": "multipart/form-data; boundary=b123",
+      "input": "--b123\r\nContent-Disposition: form-data; name=\"payload\"\r\n\r\nAuthorization: Bearer sk_live_ABC123SECRET\r\n--b123--\r\n",
+      "mustNotContain": ["sk_live_ABC123SECRET"]
+    },
+    {
+      "name": "multipart: benign part carrying embedded JSON with a sensitive key is scanned",
+      "surface": "body",
+      "contentType": "multipart/form-data; boundary=b123",
+      "input": "--b123\r\nContent-Disposition: form-data; name=\"metadata\"\r\n\r\n{\"access_token\":\"sk_live_SECRET\"}\r\n--b123--\r\n",
+      "mustNotContain": ["sk_live_SECRET"]
+    },
+    {
+      "name": "truncated JSON still scrubs known keys (cap is never a bypass)",
+      "surface": "body",
+      "contentType": "application/json",
+      "input": "{\"note\":\"x\",\"password\":\"hunter2\",\"refresh_token\":\"rt-abc\",\"blob\":\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+      "mustNotContain": ["hunter2", "rt-abc"]
+    },
+    {
+      "name": "URL: OAuth code callback keeps state, loses access_token",
+      "surface": "url",
+      "input": "https://app.example.com/cb?code=abc&access_token=AT-1&state=xyz",
+      "mustNotContain": ["AT-1"],
+      "mustContain": ["state=xyz", "/cb"]
+    },
+    {
+      "name": "URL: SAS token suffix rule",
+      "surface": "url",
+      "input": "https://acct.blob.core.windows.net/c/b.png?sv=1&sig=SAS&sas_token=SAS2",
+      "mustNotContain": ["SAS2"],
+      "mustContain": ["sv=1"]
+    },
+    {
+      "name": "URL: magic link token",
+      "surface": "url",
+      "input": "https://app.example.com/magic?token=MAGIC",
+      "mustNotContain": ["MAGIC"]
+    },
+    {
+      "name": "URL: implicit-grant fragment token",
+      "surface": "url",
+      "input": "https://app.example.com/cb#access_token=FRAG&type=bearer",
+      "mustNotContain": ["FRAG"],
+      "mustContain": ["type=bearer"]
+    },
+    {
+      "name": "benign URL stays byte-identical: encoded query",
+      "surface": "url",
+      "input": "https://api.example.com/v1/items?page=2&q=caf%C3%A9",
+      "exact": "https://api.example.com/v1/items?page=2&q=caf%C3%A9"
+    },
+    {
+      "name": "benign URL stays byte-identical: relative path",
+      "surface": "url",
+      "input": "/relative/path?ok=1",
+      "exact": "/relative/path?ok=1"
+    },
+    {
+      "name": "benign URL stays byte-identical: SPA fragment route",
+      "surface": "url",
+      "input": "https://app.example.com/#/spa/route",
+      "exact": "https://app.example.com/#/spa/route"
+    },
+    {
+      "name": "url-keyed strings inside JSON bodies get URL scrubbing",
+      "surface": "body",
+      "contentType": "application/json",
+      "input": "{\"callback\":{\"href\":\"https://x.example/cb?api_key=KEY-1\"}}",
+      "mustNotContain": ["KEY-1"]
+    },
+    {
+      "name": "policy extensions: extra body key (normalized) + inline pattern",
+      "surface": "body",
+      "policy": { "extraBodyKeys": ["internalId"], "patterns": ["vk[a-z]?_[A-Za-z0-9]{8,}"] },
+      "contentType": "application/json",
+      "input": "{\"internal_id\":\"sek\",\"note\":\"key vkb_abcdefgh12 inline\"}",
+      "mustNotContain": ["sek", "vkb_abcdefgh12"]
+    },
+    {
+      "name": "policy patterns run over benign header values",
+      "surface": "headers",
+      "policy": { "patterns": ["vk[a-z]?_[A-Za-z0-9]{8,}"] },
+      "input": { "X-Debug": "saw vkb_abcdefgh12 here" },
+      "redactedKeys": [],
+      "valueMustNotContain": { "X-Debug": ["vkb_abcdefgh12"] }
+    },
+    {
+      "name": "a bad regex pattern is skipped, defaults still apply",
+      "surface": "body",
+      "policy": { "patterns": ["("] },
+      "contentType": "application/json",
+      "input": "{\"password\":\"hunter2\",\"note\":\"ok\"}",
+      "mustNotContain": ["hunter2"],
+      "mustContain": ["ok"]
+    },
+    {
+      "name": "fullFidelity: headers pass through untouched",
+      "surface": "headers",
+      "policy": { "fullFidelity": true },
+      "input": { "Authorization": "Bearer x" },
+      "redactedKeys": [],
+      "keptEntries": { "Authorization": "Bearer x" }
+    },
+    {
+      "name": "fullFidelity: body passes through untouched",
+      "surface": "body",
+      "policy": { "fullFidelity": true },
+      "contentType": "application/json",
+      "input": "{\"password\":\"hunter2\"}",
+      "exact": "{\"password\":\"hunter2\"}"
+    },
+    {
+      "name": "fullFidelity: URL passes through untouched",
+      "surface": "url",
+      "policy": { "fullFidelity": true },
+      "input": "https://app.example.com/cb?access_token=AT-1",
+      "exact": "https://app.example.com/cb?access_token=AT-1"
+    }
+  ]
+}