@nurix/etna 0.15.6 → 0.15.8

package/data/harnesses/discover/rules/ingestion.md

@@ -0,0 +1,551 @@
+# Ingestion Contract — Capture Shape · Provenance · Ethics · Session Playbook
+
+The reference knowledge for **Ingestion** (`discover` harness, Mode 2). Every dimension capture obeys the
+contract in §1–8 so (a) the **evaluation** mode can rely on a stable shape and weight each claim by
+*how it was obtained*, and (b) a future agent can reproduce or extend any capture. §9 carries the
+deepest, gated dimension's method — the logged-in browser-session playbook — which the `session`
+agent follows.
+
+This contract generalizes the source playbook's three-format contract (`captures/*.md` narrative +
+`raw/*.json` verbatim + `curls.md`) to *any* dimension.
+
+> **Contract style = "lighter".** Provenance ("how it was captured") lives in the
+> **`_summary.md` frontmatter**, not a separate `manifest.yaml`. There is no parallel YAML
+> machine-spine — everything stays markdown-with-frontmatter. Evaluation reads the frontmatter
+> to weight claims; self-correction reads it to detect under-delivery.
+
+---
+
+## 1. Folder layout
+
+```
+research/<target>/dimensions/
+├── _shared/           ← cross-dimension seam artifacts, ONE physical copy (see §6)
+│   ├── api-path-catalog.md
+│   └── feature-flags.md
+└── <dimension>/
+    ├── _summary.md    ← decoded narrative + provenance frontmatter (human + machine read this)
+    └── raw/           ← verbatim artifacts, DIMENSION-PRIVATE (the reproducer reads these)
+```
+
+Richer dimensions inherit the playbook contract (extra files, same `_summary.md` provenance):
+- **session**: `dimensions/session/{_summary.md, captures/p*.md, raw/*.json, curls.md}`
+- **wire-capture**: `dimensions/wire-capture/{_summary.md, captures/wire-narrative.md, raw/flows.json, curls.md}`
+
+Two layout rules the seam and the "absence is a finding" principle depend on:
+- **Shared-seam files live in `dimensions/_shared/`, one physical copy each — never under a single
+  dimension's `raw/`** (§6). Everything else under a dimension's `raw/` is **dimension-private by
+  default**; only the §6 allowlist (`api-path-catalog.md`, `feature-flags.md`) is shared.
+- **A folded or absent dimension still leaves a record.** When a dimension is folded into another or
+  is absent, write a one-paragraph `_summary.md` with `status: folded` (naming the fold target) or
+  `status: absent` (+ the cheap evidence). "An absent dimension is a finding" must be **materialized
+  as a file**, never a silent drop.
+
+---
+
+## 2. `_summary.md` — required shape (with provenance frontmatter)
+
+```markdown
+---
+dimension: <codebase|docs|packages|api|website|community|session|deployed-client-bundle|infra-backend-fingerprint|wire-capture|distribution-artifacts>
+target: <slug>
+status: complete | partial | blocked
+# --- provenance (the "how it was captured" layer evaluation weights on) ---
+access_grade_used: <source:full | runtime:reachable | auth:have-login | presence:rich | ...>
+method: <the ACTUAL extraction method — see §3 vocabulary>
+completeness_pct: <0-100, honest self-estimate of reachable surface captured>
+confidence: high | medium | low      # how much to TRUST what was captured (see §3)
+captured_at: <YYYY-MM-DD>
+sources: [<urls / paths actually consulted>]
+gaps: [<structured — each is a thing evaluation must NOT treat as covered>]
+---
+
+# <Target> — <Dimension> capture
+
+## Method
+Exactly what was done, which tools, what was skipped and why. (The prose behind the frontmatter `method`.)
+
+## Findings
+Ordered, concrete observations — real values (package names, versions, endpoint paths, file
+paths, page URLs, prices). Tables liberally. The load-bearing section.
+
+## Inferences
+What the findings imply. Mark inference vs. fact.
+
+## Open questions
+What couldn't be captured and why. Never fabricate to fill a gap.
+
+## Artifacts
+Index of `raw/` files, one line each.
+```
+
+Keep `status` and `confidence` honest — they drive the entire evaluation weighting.
+
+---
+
+## 3. Method & confidence vocabulary (drives evaluation weighting)
+
+`method` is one of: `clone-and-map`, `source-map-reassembly`, `registry-metadata`,
+`openapi-verbatim`, `graphql-introspection`, `crawl-clip`, `docs-reconstructed`,
+`bundle-string-mine`, `dns-ct-fingerprint`, `wire-tap-browser`, `wire-har`, `wire-mitmproxy`,
+`binary-extract` (asar/apk/crx), `session-playbook`, `sdk-harness`, `inferred`.
+
+**Confidence is a function of method** (evaluation relies on this mapping):
+- **high** — verbatim machine artifacts: `openapi-verbatim`, `graphql-introspection`, `registry-metadata`, `clone-and-map`, `binary-extract`.
+- **medium** — `crawl-clip`, `docs-reconstructed`, `bundle-string-mine`, `dns-ct-fingerprint`, `wire-har`.
+- **low** — `inferred`, `session-playbook` Pass-1-only, `wire-tap-browser` shapes-only, any wire-drift observation without a spec.
+
+---
+
+## 4. `raw/*` — required shape
+
+Structured captures (API responses, registry metadata, schemas) → JSON with a `_meta` header:
+```json
+{
+  "_meta": { "source": "...", "captured_at": "YYYY-MM-DD", "kind": "...", "byte_length": 0, "redactions": ["field → <REDACTED:...>"] },
+  "body": { /* verbatim, PII + secrets redacted */ },
+  "_decoded": { "field": "meaning + observed values" },
+  "_observation": "optional — what's load-bearing"
+}
+```
+Prose captures (doc/marketing pages, structure maps, catalogs) → markdown with a metadata line:
+`<!-- source: <url-or-path> · captured_at: YYYY-MM-DD · method: <method> -->`
+
+**One file per logical artifact**, named by what it is (`endpoint-catalog.md`, `react-exports.md`,
+`pricing.md`, `structure-map.md`).
+
+---
+
+## 5. Size discipline — reduce, don't dump
+
+Large artifacts (a 3 MB bundle, a 5 MB response, a 700-page sitemap) are **digested**, not pasted:
+the schema/shape, enum distributions, one redacted representative sample, counts. A `raw/` file is
+readable, not a memory dump. Set `digested: true` context in the artifact's `_meta`.
+
+---
+
+## 6. The shared-surface seam rule (bundle ↔ session ↔ wire-capture)
+
+`deployed-client-bundle` (static unauth SPA), `session` (authenticated browser runtime), and
+`wire-capture` (non-browser client) can all surface the same **API path catalog** and **feature
+flags**. To keep them consumable instead of triplicated **N times** (a real run forked
+`api-path-catalog.md` up to 7×, and "two copies agree" then mis-reads as corroboration):
+
+- **One physical file, one canonical location, append-only.** The seam files live at
+  `dimensions/_shared/api-path-catalog.md` and `dimensions/_shared/feature-flags.md` — **Ingestion
+  seeds the two empty files (with a header) once, before the collector fan-out.** Every contributing
+  dimension **appends a `source:`-tagged section** (`source: bundle | session | wire | distribution`)
+  and **never creates its own copy**; each collector's `_summary.md` Artifacts list **links** the shared
+  file rather than duplicating it. **Shared-seam allowlist:** these two filenames are the *only*
+  shared/single-physical artifacts — every other `raw/*` is dimension-private.
+- Evaluation owns the **API-path diff** (an *adaptive N-way* diff over whichever runtime sources were
+  actually run — see evaluation template 3) in `data-model-api-surface.md`. A path that appears in
+  `bundle` but never in `session`/`wire` is a dormant/unshipped route — a finding, not noise.
+
+> **Same artifact ⇒ ONE source (never independent corroboration).** Two dimensions that string-mine
+> the *same* static asset (one bundle / one spec / one binary) are **one source** for the promotion
+> rule — never a confidence band-bump — and their divergent counts are a **reported range, not a
+> flagged conflict.** Name **one owner dimension per asset**: when `source:none` and no machine spec
+> is served, `deployed-client-bundle` **owns** the app-API bundle mine, and `api` is re-scoped to the
+> *published/partner* spec only (or marked `partial` if none is served) — the others **cite the owner
+> by path and do not re-mine**.
+
+> **Published API ≠ the app's own API.** A product's developer/REST/MCP surface (`api`/`docs`/`packages`
+> dims) is frequently *not* the API its own app runs on (`session`/`bundle` dims) — different protocol,
+> host, and operation set. Always diff the two: operations the app uses but the public API omits, **and**
+> public operations the app never calls, are both findings. The published surface is the integration
+> story; the runtime surface is the real architecture. When a published API exists, Evaluation emits
+> this split as a *named* output, not inline prose (evaluation template 3).
+
+---
+
+## 7. Ethics & redaction (non-negotiable — binds EVERY dimension)
+
+Violating them voids the work. Rules 2–3 are enforced by the **runnable redaction layer in §7.0** —
+install it inside the tap wrapper so no secret can reach `raw/` in the first place.
+
+1. **Never enter credentials.** The user signs into any logged-in surface beforehand.
+2. **Storage: shape, not values.** No raw cookies/tokens/JWT-signatures. Decode JWT *structure* with identity claims (`sub`,`email`,`phone`,`user_id`,`uid`,`sid`) redacted.
+3. **Never save secrets.** Before any `raw/` write, redact every credential-shaped field → `<REDACTED:...>` + log in `_meta.redactions`. Field patterns: `api_key`,`*_secret`,`client_secret`,`webhook_secret`,`signing_*`,`password`,`private_key`,`access_token`/`refresh_token` in bodies, vendor-named secrets. Value patterns: `sk_live_…`/`pk_live_…` (Stripe), `sk-…` (OpenAI), `xox[bp]-…` (Slack), `AKIA…` (AWS), `gh[ps]_…` (GitHub), `AIza…` (Google), `eyJ…` (JWT), opaque base64 ≥32 chars in a credential field. Over-redact when unsure. **The executable scrubbers that enforce this rule are in §7.0 below** (`SCRUB` / `redactSecrets` / `ULTRA`); the full keyboard layer lives in `tradecraft.md`.
+4. **Confirm before state changes** — anything that creates state, costs money, or is visible to others (even a draft toggle). State the exact action + estimated cost.
+5. **No destructive actions.** Document the absence; move on.
+6. **No bypassing safety harnesses** — redact, never evade.
+7. **No exfiltration.** Captures stay local under `research/<target>/`.
+8. **Respect TOS & robots.** For spec-writing / integration / competitive analysis — not credential theft, rate-limit bypass, or scraping.
+9. **One browser-driving dimension at a time.** Chrome MCP is a shared singleton. Only the gated `session`/`wire-capture` dimensions drive the browser; parallel non-browser collectors use WebFetch/curl, or — if a page genuinely needs a real browser — create their **own** tab and never touch the session tab. Never run two browser-driving collectors concurrently.
+10. **Verify an absence before asserting it.** A "host is NXDOMAIN / dimension is absent / endpoint 404s everywhere" claim that would *contradict another dimension* must be re-checked by a **second method** (alternate resolver, direct HTTP probe, second source) before it is written as fact. A single-resolver/single-probe negative is recorded as *tentative*, never as a cross-dimension directive. (An HTTP 4xx/5xx means the host **resolves** — present, not absent.) **Feedback-channel / changelog absence (`community`): a 404 on the marketing host's `/changelog` is NOT proof of "no public feedback loop"** — before asserting it, probe the common alternate homes: `feedback.`/`changelog.`/`updates.`/`roadmap.` subdomains, hosted boards (FeatureOS/Canny/Featurebase), a status-page "announcements" tab, and `<link rel="alternate" type="application/rss+xml">` autodiscovery on the docs/blog. Only after those miss is the closed-feedback cap (evaluation) warranted; a wrong-host 404 alone leaves it *tentative*.
+11. **Bounded live-token use (read-only, in-page only).** To fill a Pass-1 capture gap you MAY read the live token from its store and replay a **read-only** request with it — *only* if the token is used **in-page / in-process only**, is **never written to `raw/`, never returned, never logged**, and the replay changes no state. This is the single permitted touch of a live credential; rules 2–3 (shape-not-values, never-save-secrets) still bind everything written to disk.
+
+### 7.0 Redaction — runnable (the executable layer behind rule 3)
+
+These are **executable guardrails**, not prose policy. Install them **inside the tap/fetch wrapper** so every captured string is scrubbed *before it reaches the harness or disk* — credentials never reach `raw/`, and the browser MCP's content-block (`[BLOCKED: Cookie/query string data]`) never fires on a legitimate capture. Run `SCRUB(value)` on every string returned from a `javascript_tool` capture; run `redactSecrets(body, redactions)` on every object before any `Write` of `raw/*.json`; fall back to `ULTRA` only when the harness still blocks output.
+
+#### 1. `SCRUB` — the value-shape scrubber (run on every captured string)
+
+Combine into one `SCRUB` function and apply to every string returned from `javascript_tool`. It neutralises PII + provider-credential value shapes regardless of field name (catches credentials embedded in string concatenations like a deep-link URL or a log line):
+
+```js
+const SCRUB = (s) => String(s||'')
+  .replace(/eyJ[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+/g, '<jwt>')
+  .replace(/[A-Za-z0-9+/]{40,}={0,2}/g, '<b64>')
+  .replace(/data:image\/[^"]+/g, '<data-image>')
+  .replace(/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g, '<email>')
+  .replace(/[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}/g, '<uuid>')
+  .replace(/[a-f0-9]{32,}/g, '<hex>')
+  .replace(/(user|org|ins|sess|cli)_[A-Za-z0-9]{20,}/g, '.$1id.')   // Clerk-style IDs
+  .replace(/\+91\d{10}/g, '<phone-in>')                            // Indian phone numbers
+  .replace(/\+1\d{10}/g, '<phone-us>')                             // US phone numbers
+  .replace(/\b\d{10,15}\b/g, '<phone>')                            // generic numeric phone
+  .replace(/[?&]([a-z_]+)=([^&\s"]+)/gi, (_,k,v) => '?'+k+'=<v>');
+```
+
+The regex table behind `SCRUB` (PII value shapes + known provider-key prefixes), so a new run can extend it:
+
+| Pattern | Replacement | Catches |
+|---------|-------------|---------|
+| `eyJ[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+` | `<jwt>` | JWT tokens (3-segment base64) |
+| `[A-Za-z0-9+/]{40,}={0,2}` | `<b64>` | Long base64 |
+| `data:image/[^"]+` | `<data-image>` | Inline image data URIs |
+| `[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}` | `<uuid>` | UUIDs |
+| `[a-f0-9]{32,}` | `<hex>` | Long hex strings (hashes) |
+| `(user\|org\|ins\|sess\|cli)_[A-Za-z0-9]{20,}` | `.<group>id.` | Clerk-style IDs |
+| `\+91\d{10}` | `<phone-in>` | Indian phone numbers |
+| `\+1\d{10}` | `<phone-us>` | US phone numbers |
+| `\b\d{10,15}\b` | `<phone>` | Generic numeric phone |
+| `\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b` | `<email>` | Email addresses |
+| `[?&]([a-z_]+)=([^&\s"]+)` | `?$1=<v>` | Query string values |
+| `sk_(live\|test)_[A-Za-z0-9]{16,}` | `<REDACTED:STRIPE_SK>` | Stripe secret key |
+| `(rk\|pk)_(live\|test)_[A-Za-z0-9]{16,}` | `<REDACTED:STRIPE_KEY>` | Stripe restricted / publishable key |
+| `sk-(proj-)?[A-Za-z0-9_\-]{20,}` | `<REDACTED:OPENAI>` | OpenAI API key (incl. project-scoped) |
+| `xox[bpoars]-[A-Za-z0-9-]{10,}` | `<REDACTED:SLACK>` | Slack token |
+| `\b(AKIA\|ASIA)[A-Z0-9]{16}\b` | `<REDACTED:AWS_KEY_ID>` | AWS access key ID (long-lived + temporary) |
+| `gh[pousr]_[A-Za-z0-9]{30,}` | `<REDACTED:GITHUB>` | GitHub PAT / OAuth / refresh / server token |
+| `AIza[0-9A-Za-z_\-]{35}` | `<REDACTED:GOOGLE>` | Google API key |
+
+#### 2. `redactSecrets` — the field-name walker (run before every `raw/*.json` write)
+
+`SCRUB` catches known value *shapes*; some providers use opaque base64 tokens with **no recognisable prefix**, so you also need a recursive walker that redacts by **key name** regardless of value shape. Run both — `SCRUB` on captured strings, `redactSecrets` on the parsed object before any `Write`:
+
+```js
+const SECRET_KEY_RE = /(api[_-]?key|apikey|secret|password|passphrase|pwd|client[_-]?secret|webhook[_-]?secret|signing[_-]?(?:key|secret)|private[_-]?key|service[_-]?account[_-]?key|access[_-]?token|refresh[_-]?token|bearer[_-]?token|auth[_-]?token)$/i;
+
+function redactSecrets(obj, redactions) {
+  if (obj === null || typeof obj !== 'object') return obj;
+  if (Array.isArray(obj)) return obj.map(v => redactSecrets(v, redactions));
+  const out = {};
+  for (const [k, v] of Object.entries(obj)) {
+    if (typeof v === 'string' && SECRET_KEY_RE.test(k) && v.length > 0) {
+      redactions.push(`${k} → <REDACTED:SECRET> (was ${v.length} chars)`);
+      out[k] = '<REDACTED:SECRET>';
+    } else {
+      out[k] = redactSecrets(v, redactions);
+    }
+  }
+  return out;
+}
+
+// Use:
+const redactions = [];
+const safeBody = redactSecrets(rawBody, redactions);
+// Then write { _meta: { ..., redactions }, body: safeBody, _decoded_fields: {...} }
+```
+
+At run time, append the **current session's** own identifiers (your user UUID, email, display name,
+org slug) as explicit `.replace(...)` lines in a final pass over the serialized output — but **never
+commit those literal values into this file.** The generic UUID / email / phone / Clerk-ID patterns
+above already neutralise them; the explicit pass is a belt-and-suspenders step that stays local to the
+run.
+
+For a logged-in session, also redact: the user's display name, the user's avatar URL, any organisation slug (`org_slug` claim) the user might want hidden, and phone numbers at any masking level. Over-redaction is cheap; a leaked partner API key is not.
+
+#### 3. `ULTRA` — the last-resort scrubber (when the harness still blocks output)
+
+When `SCRUB` isn't enough (e.g. the body has agent-generated content with patterns that match the blocklist), drop to a regex that strips everything except readable ASCII:
+
+```js
+const ULTRA = (s) => String(s||'').replace(/[^a-zA-Z0-9 _.,:!?\-/]/g,'.');
+```
+
+You lose JSON braces and special characters, but you keep the **structural information** (field names, enum values, lengths) — useful for getting any output at all when the body is irretrievably full of blocked patterns.
+
+#### 4. PII / value-shape coverage (what these patterns neutralise)
+
+- **JWTs** — 3-segment base64 `eyJ…` anywhere (body fields, not just `Authorization`).
+- **Phone numbers** — Indian `+91` (10 digits), US `+1` (10 digits), and generic 10–15-digit numeric.
+- **Emails** — `local@domain.tld` (and the user's own specific address explicitly).
+- **UUIDs** — canonical `8-4-4-4-12` hex (and the user's own specific user UUID explicitly).
+- **Clerk-style IDs** — `user_` / `org_` / `ins_` / `sess_` / `cli_` + 20+ chars.
+- **Query-string values** — every `?k=v` / `&k=v` value collapsed to `?k=<v>`.
+- **Long base64 / long hex** — `≥40`-char base64 and `≥32`-char hex (hashes, opaque tokens).
+- **Inline image data URIs** — `data:image/…`.
+- **Provider key prefixes** — Stripe (`sk_`/`rk_`/`pk_`), OpenAI (`sk-`/`sk-proj-`), Slack (`xox*`), AWS (`AKIA`/`ASIA`), GitHub (`gh[pousr]_`), Google (`AIza`).
+
+#### 5. Maintenance loop
+
+When a run uncovers a new key or value shape (a regional BSP / telephony token, a niche analytics SDK key, an internal-tool API, an opaquely-named credential field), **add the regex to the `SCRUB` table here** (and the field name to `SECRET_KEY_RE` if it's name-based) — the layer only stays a real guardrail if every new shape is folded back in.
+
+### 7.1 wire-capture & active-interception ethics (the strictest tier)
+
+The `wire-capture` dimension climbs a **3-rung ladder, lowest privilege first**, and records the
+chosen rung in `_meta`:
+
+1. **Browser network tap (default, unauth).** Chrome `read_network_requests` + the in-page
+   fetch/XHR interceptor on the **public** surface only. Capture request/response *shapes*; redact
+   in the wrapper before disk. No login. → `method: wire-tap-browser`.
+2. **HAR export.** Export DevTools HAR of the unauth SPA's XHRs, digest to an endpoint catalog,
+   scrub `Authorization`/`Cookie`/secret-shaped values. → `method: wire-har`.
+3. **mitmproxy / intercepting proxy — GATED, opt-in, user-run.** ONLY first-party traffic the
+   **user owns** (their own desktop/mobile/CLI client), ONLY with explicit chat authorization, ONLY
+   TOS-checked. The harness NEVER stands up a proxy against a third party's infra unattended; the
+   *user* installs the CA and routes their client — the agent only ingests exported flows. →
+   `method: wire-mitmproxy`. Same explicit-authorization loop as session Pass-2.
+
+> **Rung 3+ — the first-party SDK-harness lane (the deepest method for a closed runtime that PUBLISHES
+> an SDK).** When the product ships an official client SDK (npm / PyPI / Maven / …) and the *user* can
+> provision credentials (free tier / trial), the deepest no-source method is not to intercept the
+> *vendor's* app but to **scaffold a minimal throw-away app with the product's OWN SDK + the user's keys,
+> run it, and instrument the client you control**: inject fetch/WS/EventSource taps at `document_start`
+> (so there is **no interceptor-timing problem** — you wrap the constructors before any connection
+> opens), observe every request/response shape, and — with the user routing it through *their own*
+> mitmproxy — the fully-decoded wire. Because you **own** the client, there is no third-party-traffic
+> concern and no missed on-load socket. **Gated exactly like rung 3 / session Pass-2:** the *user*
+> provisions the account + keys and runs any proxy; the **agent never self-signs-up and never enters
+> credentials**, only scaffolds + instruments, and confirms any cost/quota first. → `method: sdk-harness`
+> (a first-party runtime lane; confidence = what is actually observed, typically high since you control
+> the client). Especially powerful for `runtime:reachable + source:none` targets whose published SDK is
+> the only public mirror of the runtime — and it independently corroborates the `api`/`packages` lanes.
+
+> **Never fold a browser-rung into a non-browser (curl/WebFetch) collector.** Rung 1 *is* a browser tap,
+> and the browser is a singleton owned by `session`/`wire-capture` — routing it to a curl collector
+> produces zero `source: wire` rows (a real run did exactly this). When `wire-capture` is **folded**,
+> the rung-1 tap is the **in-page interceptor that `session` already installs**: `session` appends its
+> observed paths to `dimensions/_shared/api-path-catalog.md` as `source: wire`. A folded `wire-capture`
+> still leaves a one-paragraph `_summary.md` (`status: folded`, naming the fold target) per §1 —
+> "folded" is recorded, never silently dropped.
+
+> Wireshark / packet capture is explicitly **not** used: for HTTPS it yields only TLS ciphertext,
+> and a packet sweep of a third party is the bright line this harness will not cross.
+
+### 7.2 distribution-artifacts ethics
+Downloading a public binary (CRX/`.asar`/`.ipa`/`.apk`) and reading its manifest/strings is fine.
+But binary **decompilation** can cross from "inspect" to "decompile in violation of EULA":
+the agent reads the artifact's own license/EULA first and records a restrictive one as an Open
+question rather than silently decompiling.
+
+---
+
+## 8. Completeness check
+A capture is complete when: `_summary.md` exists with full provenance frontmatter + honest `status`;
+every Finding is backed by a `raw/` artifact or inline value; `raw/` artifacts have `_meta` +
+decoding (large ones digested); redaction pass run; `gaps` recorded (never fabricated).
+
+**Summary-first + bounded, visible fan-out (never go silent long enough to look dead).** Write a
+`status: partial` `_summary.md` with whatever is mined so far **before** any long-tail fetch loop
+(downloading many chunks, crawling many pages, deep pagination), then enrich it in place and flip to
+`complete`. **Bound** every fan-out — cap it by count / total bytes / wall-time (e.g. ≤40 items or
+≤90 s) and record the remainder as a `gap` rather than fetching exhaustively; emit a progress note per
+batch. A collector that fetches everything before writing anything can run many times longer than its
+siblings and read as hung to the orchestrator, forcing a needless salvage. Partial-but-finalized beats
+complete-but-silent.
+
+**Capability gap = a checkpoint, not a silent fallback.** When a *preferred* capture capability is found
+**unavailable at runtime** — a CDP/remote-debugging port is closed, Chrome / the browser MCP is unreachable,
+a gated proxy or tool can't be stood up — such that you would proceed on a **materially weaker** method
+(e.g. source-only or a digest instead of a live wire tap), **pause and confirm with the user before
+continuing.** State plainly: what is unavailable, the degraded method you would fall back to, and the
+coverage that would be lost. This is distinct from a *planned* method pre-grade (e.g. `openapi-verbatim` →
+`docs-reconstructed` when no spec is served), which is **recorded, not confirmed** — the checkpoint is for an
+*infrastructure/capability* gap discovered mid-run, where silently degrading hides a real coverage loss
+behind a confident-looking capture. (A real run found the CDP port closed and silently fell back to a
+browser-MCP tap; it worked, but the degradation should have been surfaced first.)
+
+---
+
+## 9. The session dimension — logged-in browser-session playbook
+
+The method for the **session** dimension: discovering the *authenticated runtime* of a
+product by driving it through a browser (Chrome MCP) and instrumenting it. It produces a
+section-by-section understanding of a product's wire protocol, auth, and state model **without
+source-code access**. The `session` agent follows this section.
+
+This is the deepest and most sensitive dimension. The ethics in §7 above bind hardest here.
+
+### 9.1 Prerequisites (verify before anything else — all three non-negotiable)
+
+| # | Precondition | How to verify |
+|---|--------------|---------------|
+| 1 | Target is **pre-logged-in** in Chrome | open the target URL in the connected tab → an authenticated surface (a dashboard, not a sign-in page). The agent never enters credentials. |
+| 2 | **Claude Chrome extension** installed | `mcp__claude-in-chrome__tabs_context_mcp` returns live tabs |
+| 3 | **Chrome CDP** connected | the same call returns stable tab IDs |
+
+If any fails, return `blocked` naming the missing precondition.
+
+### 9.2 The three-pass strategy
+
+A single linear sweep wastes context on things you can read for free and wastes credits on
+actions whose response shape you could infer. Use three passes.
+
+**Pass 1 — read-only mining (no money, no state change).** If the user already has data (a
+project, conversation, account, order), mine it first. Hit every list/state/history endpoint with
+the existing token; capture verbatim bodies. This yields ~70% of the spec for free. Mine, by role:
+identity (`/me`, `/account`, `/workspaces`), list (the canonical taxonomy), single-resource state
+(server-injected vs client-sent fields), history/event-log (a whole trajectory in one call),
+runtime/preview state, per-resource billing, account billing, feature flags/cohort, **the account's
+`entitlements`/`permissions` arrays** (the single richest data-model artifact — `entitlements`/`features`
+enumerate every gated feature *including ones your plan hides*; `permissions` are CRUD-verbs-per-entity,
+so the verb set enumerates the domain entity set — `X_CREATE/VIEW/EDIT/DELETE` ⇒ entity `X`; mine these
+first), integration
+catalog (`/mcp/`, `/integrations/`), and schema/capabilities (`/openapi.json`, `/.well-known/*`,
+`/page-metadata`).
+
+**Pass 2 — fresh-run validation (with explicit cost/state-change confirmation).** Only after
+Pass 1, and only with explicit user OK stating the exact action + cheapest mode + estimated cost.
+Catches what Pass 1 can't: the submit/write request body, live state-machine transitions, the
+completion event, server-side resolution of shorthand, ID/slug-generation, the real iteration
+endpoint, and the full realtime event catalog. **Legitimate to skip** if the user opts out — leave
+write-side shapes as open questions; never fabricate them.
+
+> **Record the write-side decision as a machine flag, not re-derived prose.** A Pass-1-only run leaves
+> the *entire* write/generate/launch surface unobserved — feature *existence/gating* is solid, but
+> feature *execution behavior* rests on marketing/docs and must never be promoted to fact (this caveat
+> otherwise gets re-hedged across 3–4 rollups every cost-gated run). So the session `_summary.md`
+> frontmatter carries `write_side_observed: <true|false>` and `pass2: <not-applicable | offered-pending
+> | declined-by-user | deferred | done>` (this flag is the load-bearing part — it lets Evaluation render
+> the write-side gap *once*). When it would help a later run resume, optionally add
+> `captures/pass2-followup.md` listing the **exact cheapest-mode write actions + est. cost** that would
+> close the gap. Discovery seeds the `pass2:` decision in `00-recon-plan.md` for any `auth:have-login`
+> target.
+
+**Pass 3 — synthesis.** Walk the captures, pull each finding into the dimension's `_summary.md`,
+note the cleverest design choices, defaults imposed, performance characteristics, and open
+questions.
+
+### 9.3 The P0 → P6 phases
+
+The same phases apply whether the target is an AI builder, analytics dashboard, CRM, or collab tool:
+
+| Phase | Focus |
+|-------|-------|
+| **P0 Surface** | routes, bundles, storage shape, the API host(s), the auth tell; build the **coverage checklist** from any route/op catalog a sibling declared (§9.4) |
+| **P1 Auth** | token type + issuer + storage location + refresh path; tenancy model; **disambiguate the transport by omission** (replay one authed read with each credential mechanism dropped) — and never test auth with a schema-only field |
+| **P2 Core action** | the primary write — the submit/create body and its server resolution |
+| **P3 Streaming** | the realtime transport (WS / SSE / WebChannel / Pusher / none) + event catalog; **fingerprint binary frames by signature, never log them as opaque** (§9.4) |
+| **P4 Runtime/state** | sandbox/preview/build/deploy state; the live result surface |
+| **P5 Iteration** | the follow-up/edit endpoint (often surprisingly named) + any auto-iteration loop |
+| **P6 Synthesis** | reconcile into the dimension's `_summary.md` + `captures/` + `curls.md` |
+
+### 9.4 Runtime taps (Pass-1 instrumentation, run in-page)
+
+> **The runnable taps + the ~27-entry Chrome-MCP gotcha catalogue + the dead-end recovery table + the
+> Apollo read-side recipe + the fingerprint tables live in `tradecraft.md`** — paste them, do
+> not re-derive. The prose below is the *why*; that file is the *keyboard layer*. The §7.0 redaction
+> scrubbers (`SCRUB`/`redactSecrets`/`ULTRA`) install inside these tap wrappers.
+
+> **Install the in-page interceptor FIRST, then move section-to-section WITHOUT a full reload.**
+> CDP `read_network_requests` is unreliable for an already-loaded single-page app: it starts tracking
+> only when first called and silently misses requests already in flight or served from cache (seen on a
+> real run where only a third-party analytics beacon was captured while the app's own API calls went
+> unseen). Treat it as a *supplement*; the in-page interceptor is the primary capture. A full-page
+> `navigate` wipes the interceptor, so traverse the app using whatever it exposes — **clicking in-app
+> nav works on any stack**; or call the client router if one is reachable (`history.pushState`+`popstate`,
+> React Router / Vue Router / Angular Router, or a globally-exposed router such as `window.next.router`
+> on Next.js). The principle is framework-agnostic; the specific call is just whatever that app ships.
+
+- **WebSocket / EventSource / fetch+XHR interceptors** — wrap the constructors/methods to log
+  URLs, frames, and bodies (shapes, not secrets — redact in the wrapper before anything reaches
+  disk). Add a `SCRUB(value)` step inside the fetch wrapper so credentials never reach the harness.
+  **Never log a binary frame as opaque `<binary>`.** Any realtime/streaming/collab layer (token
+  streams, CRDT/OT edits, presence, telemetry, game or market state) may ship `ArrayBuffer`/`Blob`
+  frames; for each *distinct* frame capture its **byte-length + a hex dump of the first ~16 bytes** —
+  enough to fingerprint the wire format (protobuf, MessagePack, CBOR, Yjs/CRDT, gRPC-web, or a
+  text/topic-prefixed channel protocol) without recording content. **Decode the signature as ASCII
+  first** — many "binary" protocols are framed text (STOMP, Phoenix, Centrifugo, a `+topic` channel
+  prefix), so a legible prefix names the protocol in one step before a binary-codec guess. The frame
+  *signature* is the finding; the payload is not.
+  - **Realtime-on-load capture (the connection that opens before you can wrap it).** The "install the
+    interceptor FIRST" rule assumes you are *already on* the target surface; it does **not** cover a
+    realtime connection that fires during the **initial load of a page you `navigate` to** (or a
+    lazy-loaded/`iframe`d live demo). A post-load interceptor **cannot** wrap a `WebSocket`/`EventSource`
+    constructed before injection, and an `offline`→`online`/`focus` reconnect-trigger may spawn no new
+    connection on such a demo — so frame capture comes back empty (a real run got 0 frames on a
+    collab-demo this way). When that happens: (a) capture the **CDP-visible auth/token-mint fetch** that
+    precedes the socket (it names the realtime host/handshake); (b) **the technique that works —
+    client-side-navigate within the SPA** (a router `<Link>` click, not a full reload): the in-page
+    wrapper **survives client-side nav**, and the next route **mounts a fresh room → a new socket through
+    your wrapper** (this captures the handshake + frames where `offline`/`online` reconnect-triggers
+    don't — a real run decoded Liveblocks' v8 JSON protocol this way after a 0-frame first pass); (c) if a
+    raw **CDP `Network.webSocketFrameReceived`** tap is reachable, it captures frames protocol-level with
+    no injection-timing problem at all (the browser MCP's HTTP-only `read_network_requests` does **not**
+    surface WS frames — that gap is what forces (b)); and (d) fall back to the `codebase`/`distribution`
+    lanes for the wire format — record it as a *folded* gap, not a silent miss, since the protocol is then
+    sourced independently rather than live-observed. **Decode captured frames against any in-repo protocol
+    enums** (message-code / op-code tables) so the live wire *validates* the source read.
+- **Storage shape inspector** — enumerate localStorage / sessionStorage / cookies / IndexedDB
+  **key names + lengths**, never raw values.
+- **JWT decoder** — decode the token *structure* (alg, claim names, exp) with identity claims
+  (`sub`, `email`, `phone`, `user_id`, `uid`, `sid`) redacted.
+- **Auth-channel disambiguation.** Token *location* is itself a tell: not in localStorage ⇒ an httpOnly
+  cookie **or** an SDK store (IndexedDB — Firebase/Auth0/Supabase/Cognito keep tokens there, which is why
+  a "no JWT in localStorage" result is a finding, not a dead end). Determine the *required* transport by
+  **omission**: replay one harmless authed read — using a field that needs the authed context (`viewer`/
+  `me`), **never** a schema-only field (GraphQL `__typename`, a health route) that resolves
+  unauthenticated — with each candidate credential dropped (cookie-only, then header-only); whichever
+  response comes back unauthorized names the channel the app actually requires.
+- If a realtime SDK is loaded (Pusher, Ably, Socket.IO, Centrifugo, Phoenix, …), bind a global listener
+  to capture the event/channel catalog.
+- **Third-party-SDK fingerprint** (cross-checks `infra-backend-fingerprint`). Storage key names + the
+  network host list name the embedded vendor stack — auth (Firebase/Auth0), flags (LaunchDarkly/Statsig),
+  support (Intercom/Zendesk/Ada), analytics (Segment/Amplitude/Datadog), payments (Stripe). Record them;
+  they corroborate the sub-processor list *independently* (a different dimension → promotes confidence).
+- **Drive op coverage from any catalog a sibling declared.** If `deployed-client-bundle` (or `api`/`docs`)
+  recovered a declared **route / endpoint / operation catalog** — a REST path table, a GraphQL operation
+  set, an RPC/tool method list — consume it as a **coverage checklist**: walk the routes/actions that
+  trigger the un-observed entries so the declared-vs-observed diff (§6) reflects a real walk, not just the
+  landing page.
+- **Server-side-read (RSC) check — don't mis-record "no read API."** In a modern Next.js App-Router /
+  React-Server-Components app (and similar SSR/streaming stacks), the initial **read** data is fetched
+  *server-side* with the session and streamed — so the client wire shows only mutations, asset/media
+  fetches, telemetry, and token-mints, and **no client-side data-read XHRs fire on load even though the
+  UI is fully populated.** Before recording "read API not observed," check for the server-side path: a
+  document/`?_rsc=` response with `text/x-component`, embedded `__next_f`/flight payloads, `_next/data`
+  JSON, or server actions. When reads are server-resolved, say so — and note that a sibling
+  bundle's app-own *path count* then **overstates the client read surface** (fold this into the
+  published-vs-app-own diff, §6, rather than reporting a phantom "unused routes" gap). (Writes can be
+  off-main-thread too — a generate POST issued from a **Web Worker** won't appear in a main-thread fetch
+  tap; corroborate via CDP / Performance resource timing.)
+
+### 9.5 The capture contract (richer than a standard dimension capture)
+
+Under `research/<target>/dimensions/session/`:
+
+- **`captures/p*.md`** — per-phase narrative for the human: frontmatter (title, summary, scope,
+  status), **Method** (taps run, actions taken), **Findings** (concrete values — URLs, fields,
+  status codes), **Inferences**, **Diff vs reference**, **Open questions**. Decoded, opinionated;
+  don't paste raw bodies here.
+- **`raw/*.json`** — verbatim endpoint captures for the reproducer: a `_meta` header
+  (endpoint, auth-shape, captured_at, status, byte_length, redactions) + `body` (PII + secrets
+  redacted) + `_decoded_fields`. Digest bodies > 50 KB into a structured summary.
+- **`curls.md`** — reproducible curl per endpoint, the auth model documented, a variables block,
+  and an endpoint-catalog table.
+- **Visual / surface map** — `captures/screens/` (**preferred when available**): one annotated
+  screenshot per primary surface (each major nav section), redact visible PII before saving. **When
+  screenshot capture is unavailable** (e.g. the runtime lacks an image tool, or the app holds a
+  persistent connection that never reaches page-idle), a **textual `captures/surface-map.md` is the
+  accepted substitute** and satisfies this visual-map requirement — and the **visual-coverage gap is
+  recorded in the `gaps:` frontmatter** so Evaluation reads a *known constraint*, not a silent miss.
+- **`captures/pass2-followup.md`** (optional, when Pass 2 is deferred/declined and it would help a
+  later run resume) — the exact cheapest-mode write/generate actions + estimated cost that would close
+  the write-side gap (§9.2). The `write_side_observed`/`pass2` frontmatter flags are mandatory; this
+  file is the convenience.
+- **`_summary.md`** — the standard dimension bridge file (provenance frontmatter per §2) pointing
+  into the above, so Evaluation treats session like any other dimension. `method: session-playbook`,
+  `confidence: low` for Pass-1-only/inferred. **Session frontmatter additionally carries
+  `write_side_observed: <true|false>` and `pass2: <not-applicable | offered-pending | declined-by-user
+  | deferred | done>`** (§9.2) so Evaluation renders the write-side gap once, machine-readably.
+
+### 9.6 Session ethics (binds hardest here — see §7 above)
+
+Never enter credentials; storage shape not values; redact every secret **inside the tap wrapper**
+before it reaches disk; **confirm before any Pass-2 state change or cost** (exact action + exact
+cost); no destructive actions; no bypassing the harness (redact, never evade); captures stay local;
+respect TOS. The provenance/redaction rules in §1–8 apply to everything written here.