@nurix/etna 0.15.6 → 0.15.8

package/data/harnesses/re/rules/re-ingestion.md

@@ -1,327 +0,0 @@
-# Ingestion Contract — Dump Shape · Provenance · Ethics · Session Playbook
-
-The reference knowledge for **Ingestion** (`re` harness, Mode 2). Every dimension dump obeys the
-contract in §1–8 so (a) the **evaluation** mode can rely on a stable shape and weight each claim by
-*how it was obtained*, and (b) a future agent can reproduce or extend any dump. §9 carries the
-deepest, gated dimension's method — the logged-in browser-session playbook — which the `session`
-agent follows.
-
-This contract generalizes the source playbook's three-format contract (`captures/*.md` narrative +
-`raw/*.json` verbatim + `curls.md`) to *any* dimension.
-
-> **Contract style = "lighter".** Provenance ("how it was dumped") lives in the
-> **`_summary.md` frontmatter**, not a separate `manifest.yaml`. There is no parallel YAML
-> machine-spine — everything stays markdown-with-frontmatter. Evaluation reads the frontmatter
-> to weight claims; self-correction reads it to detect under-delivery.
-
----
-
-## 1. Folder layout
-
-```
-research/<target>/dimensions/<dimension>/
-├── _summary.md        ← decoded narrative + provenance frontmatter (human + machine read this)
-└── raw/               ← verbatim artifacts (the reproducer reads these)
-```
-
-Richer dimensions inherit the playbook contract (extra files, same `_summary.md` provenance):
-- **session**: `dimensions/session/{_summary.md, captures/p*.md, raw/*.json, curls.md}`
-- **wire-capture**: `dimensions/wire-capture/{_summary.md, captures/wire-narrative.md, raw/flows.json, curls.md}`
-
----
-
-## 2. `_summary.md` — required shape (with provenance frontmatter)
-
-```markdown
----
-dimension: <codebase|docs|packages|api|website|community|session|deployed-client-bundle|infra-backend-fingerprint|wire-capture|distribution-artifacts>
-target: <slug>
-status: complete | partial | blocked
-# --- provenance (the "how it was dumped" layer evaluation weights on) ---
-access_grade_used: <source:full | runtime:reachable | auth:have-login | presence:rich | ...>
-method: <the ACTUAL extraction method — see §3 vocabulary>
-completeness_pct: <0-100, honest self-estimate of reachable surface captured>
-confidence: high | medium | low      # how much to TRUST what was captured (see §3)
-captured_at: <YYYY-MM-DD>
-sources: [<urls / paths actually consulted>]
-gaps: [<structured — each is a thing evaluation must NOT treat as covered>]
----
-
-# <Target> — <Dimension> dump
-
-## Method
-Exactly what was done, which tools, what was skipped and why. (The prose behind the frontmatter `method`.)
-
-## Findings
-Ordered, concrete observations — real values (package names, versions, endpoint paths, file
-paths, page URLs, prices). Tables liberally. The load-bearing section.
-
-## Inferences
-What the findings imply. Mark inference vs. fact.
-
-## Open questions
-What couldn't be captured and why. Never fabricate to fill a gap.
-
-## Artifacts
-Index of `raw/` files, one line each.
-```
-
-Keep `status` and `confidence` honest — they drive the entire evaluation weighting.
-
----
-
-## 3. Method & confidence vocabulary (drives evaluation weighting)
-
-`method` is one of: `clone-and-map`, `source-map-reassembly`, `registry-metadata`,
-`openapi-verbatim`, `graphql-introspection`, `crawl-clip`, `docs-reconstructed`,
-`bundle-string-mine`, `dns-ct-fingerprint`, `wire-tap-browser`, `wire-har`, `wire-mitmproxy`,
-`binary-extract` (asar/apk/crx), `session-playbook`, `inferred`.
-
-**Confidence is a function of method** (evaluation relies on this mapping):
-- **high** — verbatim machine artifacts: `openapi-verbatim`, `graphql-introspection`, `registry-metadata`, `clone-and-map`, `binary-extract`.
-- **medium** — `crawl-clip`, `docs-reconstructed`, `bundle-string-mine`, `dns-ct-fingerprint`, `wire-har`.
-- **low** — `inferred`, `session-playbook` Pass-1-only, `wire-tap-browser` shapes-only, any wire-drift observation without a spec.
-
----
-
-## 4. `raw/*` — required shape
-
-Structured captures (API responses, registry metadata, schemas) → JSON with a `_meta` header:
-```json
-{
-  "_meta": { "source": "...", "captured_at": "YYYY-MM-DD", "kind": "...", "byte_length": 0, "redactions": ["field → <REDACTED:...>"] },
-  "body": { /* verbatim, PII + secrets redacted */ },
-  "_decoded": { "field": "meaning + observed values" },
-  "_observation": "optional — what's load-bearing"
-}
-```
-Prose captures (doc/marketing pages, structure maps, catalogs) → markdown with a metadata line:
-`<!-- source: <url-or-path> · captured_at: YYYY-MM-DD · method: <method> -->`
-
-**One file per logical artifact**, named by what it is (`endpoint-catalog.md`, `react-exports.md`,
-`pricing.md`, `structure-map.md`).
-
----
-
-## 5. Size discipline — reduce, don't dump
-
-Large artifacts (a 3 MB bundle, a 5 MB response, a 700-page sitemap) are **digested**, not pasted:
-the schema/shape, enum distributions, one redacted representative sample, counts. A `raw/` file is
-readable, not a memory dump. Set `digested: true` context in the artifact's `_meta`.
-
----
-
-## 6. The shared-surface seam rule (bundle ↔ session ↔ wire-capture)
-
-`deployed-client-bundle` (static unauth SPA), `session` (authenticated browser runtime), and
-`wire-capture` (non-browser client) can all surface the same **API path catalog** and **feature
-flags**. To keep them consumable instead of triplicated:
-
-- Use **one canonical artifact name** across all three: `raw/api-path-catalog.md` and
-  `raw/feature-flags.md`, each row carrying a `source: bundle | session | wire` column.
-- Evaluation owns the **three-way diff** (static-declared vs browser-observed vs nonbrowser-observed)
-  in `data-model-api-surface.md`. A path that appears in `bundle` but never in `session`/`wire` is a
-  dormant/unshipped route — a finding, not noise.
-
-> **Published API ≠ the app's own API.** A product's developer/REST/MCP surface (`api`/`docs`/`packages`
-> dims) is frequently *not* the API its own app runs on (`session`/`bundle` dims) — different protocol,
-> host, and operation set. Always diff the two: operations the app uses but the public API omits, **and**
-> public operations the app never calls, are both findings. The published surface is the integration
-> story; the runtime surface is the real architecture.
-
----
-
-## 7. Ethics & redaction (non-negotiable — binds EVERY dimension)
-
-Carried from the source playbook (`setup.md` §6). Violating them voids the work.
-
-1. **Never enter credentials.** The user signs into any logged-in surface beforehand.
-2. **Storage: shape, not values.** No raw cookies/tokens/JWT-signatures. Decode JWT *structure* with identity claims (`sub`,`email`,`phone`,`user_id`,`uid`,`sid`) redacted.
-3. **Never save secrets.** Before any `raw/` write, redact every credential-shaped field → `<REDACTED:...>` + log in `_meta.redactions`. Field patterns: `api_key`,`*_secret`,`client_secret`,`webhook_secret`,`signing_*`,`password`,`private_key`,`access_token`/`refresh_token` in bodies, vendor-named secrets. Value patterns: `sk_live_…`/`pk_live_…` (Stripe), `sk-…` (OpenAI), `xox[bp]-…` (Slack), `AKIA…` (AWS), `gh[ps]_…` (GitHub), `AIza…` (Google), `eyJ…` (JWT), opaque base64 ≥32 chars in a credential field. Over-redact when unsure.
-4. **Confirm before state changes** — anything that creates state, costs money, or is visible to others (even a draft toggle). State the exact action + estimated cost.
-5. **No destructive actions.** Document the absence; move on.
-6. **No bypassing safety harnesses** — redact, never evade.
-7. **No exfiltration.** Captures stay local under `research/<target>/`.
-8. **Respect TOS & robots.** For spec-writing / integration / competitive analysis — not credential theft, rate-limit bypass, or scraping.
-9. **One browser-driving dimension at a time.** Chrome MCP is a shared singleton. Only the gated `session`/`wire-capture` dimensions drive the browser; parallel non-browser dumpers use WebFetch/curl, or — if a page genuinely needs a real browser — create their **own** tab and never touch the session tab. Never run two browser-driving dumpers concurrently.
-10. **Verify an absence before asserting it.** A "host is NXDOMAIN / dimension is absent / endpoint 404s everywhere" claim that would *contradict another dimension* must be re-checked by a **second method** (alternate resolver, direct HTTP probe, second source) before it is written as fact. A single-resolver/single-probe negative is recorded as *tentative*, never as a cross-dimension directive. (An HTTP 4xx/5xx means the host **resolves** — present, not absent.)
-11. **Bounded live-token use (read-only, in-page only).** To fill a Pass-1 capture gap you MAY read the live token from its store and replay a **read-only** request with it — *only* if the token is used **in-page / in-process only**, is **never written to `raw/`, never returned, never logged**, and the replay changes no state. This is the single permitted touch of a live credential; rules 2–3 (shape-not-values, never-save-secrets) still bind everything written to disk.
-
-### 7.1 wire-capture & active-interception ethics (the strictest tier)
-
-The `wire-capture` dimension climbs a **3-rung ladder, lowest privilege first**, and records the
-chosen rung in `_meta`:
-
-1. **Browser network tap (default, unauth).** Chrome `read_network_requests` + the in-page
-   fetch/XHR interceptor on the **public** surface only. Capture request/response *shapes*; redact
-   in the wrapper before disk. No login. → `method: wire-tap-browser`.
-2. **HAR export.** Export DevTools HAR of the unauth SPA's XHRs, digest to an endpoint catalog,
-   scrub `Authorization`/`Cookie`/secret-shaped values. → `method: wire-har`.
-3. **mitmproxy / intercepting proxy — GATED, opt-in, user-run.** ONLY first-party traffic the
-   **user owns** (their own desktop/mobile/CLI client), ONLY with explicit chat authorization, ONLY
-   TOS-checked. The harness NEVER stands up a proxy against a third party's infra unattended; the
-   *user* installs the CA and routes their client — the agent only ingests exported flows. →
-   `method: wire-mitmproxy`. Same explicit-authorization loop as session Pass-2.
-
-> Wireshark / packet capture is explicitly **not** used: for HTTPS it yields only TLS ciphertext,
-> and a packet sweep of a third party is the bright line this harness will not cross.
-
-### 7.2 distribution-artifacts ethics
-Downloading a public binary (CRX/`.asar`/`.ipa`/`.apk`) and reading its manifest/strings is fine.
-But binary **decompilation** can cross from "inspect" to "reverse-engineer in violation of EULA":
-the agent reads the artifact's own license/EULA first and records a restrictive one as an Open
-question rather than silently decompiling.
-
----
-
-## 8. Completeness check
-A dump is complete when: `_summary.md` exists with full provenance frontmatter + honest `status`;
-every Finding is backed by a `raw/` artifact or inline value; `raw/` artifacts have `_meta` +
-decoding (large ones digested); redaction pass run; `gaps` recorded (never fabricated).
-
-**Summary-first + bounded, visible fan-out (never go silent long enough to look dead).** Write a
-`status: partial` `_summary.md` with whatever is mined so far **before** any long-tail fetch loop
-(downloading many chunks, crawling many pages, deep pagination), then enrich it in place and flip to
-`complete`. **Bound** every fan-out — cap it by count / total bytes / wall-time (e.g. ≤40 items or
-≤90 s) and record the remainder as a `gap` rather than fetching exhaustively; emit a progress note per
-batch. A dumper that fetches everything before writing anything can run many times longer than its
-siblings and read as hung to the orchestrator, forcing a needless salvage. Partial-but-finalized beats
-complete-but-silent.
-
----
-
-## 9. The session dimension — logged-in browser-session playbook
-
-The method for the **session** dimension: reverse-engineering the *authenticated runtime* of a
-product by driving it through a browser (Chrome MCP) and instrumenting it. It produces a
-section-by-section understanding of a product's wire protocol, auth, and state model **without
-source-code access**. The `session` agent follows this section.
-
-This is the deepest and most sensitive dimension. The ethics in §7 above bind hardest here.
-
-### 9.1 Prerequisites (verify before anything else — all three non-negotiable)
-
-| # | Precondition | How to verify |
-|---|--------------|---------------|
-| 1 | Target is **pre-logged-in** in Chrome | open the target URL in the connected tab → an authenticated surface (a dashboard, not a sign-in page). The agent never enters credentials. |
-| 2 | **Claude Chrome extension** installed | `mcp__claude-in-chrome__tabs_context_mcp` returns live tabs |
-| 3 | **Chrome CDP** connected | the same call returns stable tab IDs |
-
-If any fails, return `blocked` naming the missing precondition.
-
-### 9.2 The three-pass strategy
-
-A single linear sweep wastes context on things you can read for free and wastes credits on
-actions whose response shape you could infer. Use three passes.
-
-**Pass 1 — read-only mining (no money, no state change).** If the user already has data (a
-project, conversation, account, order), mine it first. Hit every list/state/history endpoint with
-the existing token; capture verbatim bodies. This yields ~70% of the spec for free. Mine, by role:
-identity (`/me`, `/account`, `/workspaces`), list (the canonical taxonomy), single-resource state
-(server-injected vs client-sent fields), history/event-log (a whole trajectory in one call),
-runtime/preview state, per-resource billing, account billing, feature flags/cohort, **the account's
-`entitlements`/`permissions` arrays** (the single richest data-model artifact — `entitlements`/`features`
-enumerate every gated feature *including ones your plan hides*; `permissions` are CRUD-verbs-per-entity,
-so the verb set enumerates the domain entity set — `X_CREATE/VIEW/EDIT/DELETE` ⇒ entity `X`; mine these
-first), integration
-catalog (`/mcp/`, `/integrations/`), and schema/capabilities (`/openapi.json`, `/.well-known/*`,
-`/page-metadata`).
-
-**Pass 2 — fresh-run validation (with explicit cost/state-change confirmation).** Only after
-Pass 1, and only with explicit user OK stating the exact action + cheapest mode + estimated cost.
-Catches what Pass 1 can't: the submit/write request body, live state-machine transitions, the
-completion event, server-side resolution of shorthand, ID/slug-generation, the real iteration
-endpoint, and the full realtime event catalog. **Legitimate to skip** if the user opts out — leave
-write-side shapes as open questions; never fabricate them.
-
-**Pass 3 — synthesis.** Walk the captures, pull each finding into the dimension's `_summary.md`,
-note the cleverest design choices, defaults imposed, performance characteristics, and open
-questions.
-
-### 9.3 The P0 → P6 phases
-
-The same phases apply whether the target is an AI builder, analytics dashboard, CRM, or collab tool:
-
-| Phase | Focus |
-|-------|-------|
-| **P0 Surface** | routes, bundles, storage shape, the API host(s), the auth tell; build the **coverage checklist** from any route/op catalog a sibling declared (§9.4) |
-| **P1 Auth** | token type + issuer + storage location + refresh path; tenancy model; **disambiguate the transport by omission** (replay one authed read with each credential mechanism dropped) — and never test auth with a schema-only field |
-| **P2 Core action** | the primary write — the submit/create body and its server resolution |
-| **P3 Streaming** | the realtime transport (WS / SSE / WebChannel / Pusher / none) + event catalog; **fingerprint binary frames by signature, never log them as opaque** (§9.4) |
-| **P4 Runtime/state** | sandbox/preview/build/deploy state; the live result surface |
-| **P5 Iteration** | the follow-up/edit endpoint (often surprisingly named) + any auto-iteration loop |
-| **P6 Synthesis** | reconcile into the dimension's `_summary.md` + `captures/` + `curls.md` |
-
-### 9.4 Runtime taps (Pass-1 instrumentation, run in-page)
-
-> **Install the in-page interceptor FIRST, then move section-to-section WITHOUT a full reload.**
-> CDP `read_network_requests` is unreliable for an already-loaded single-page app: it starts tracking
-> only when first called and silently misses requests already in flight or served from cache (seen on a
-> real run where only a third-party analytics beacon was captured while the app's own API calls went
-> unseen). Treat it as a *supplement*; the in-page interceptor is the primary capture. A full-page
-> `navigate` wipes the interceptor, so traverse the app using whatever it exposes — **clicking in-app
-> nav works on any stack**; or call the client router if one is reachable (`history.pushState`+`popstate`,
-> React Router / Vue Router / Angular Router, or a globally-exposed router such as `window.next.router`
-> on Next.js). The principle is framework-agnostic; the specific call is just whatever that app ships.
-
-- **WebSocket / EventSource / fetch+XHR interceptors** — wrap the constructors/methods to log
-  URLs, frames, and bodies (shapes, not secrets — redact in the wrapper before anything reaches
-  disk). Add a `SCRUB(value)` step inside the fetch wrapper so credentials never reach the harness.
-  **Never log a binary frame as opaque `<binary>`.** Any realtime/streaming/collab layer (token
-  streams, CRDT/OT edits, presence, telemetry, game or market state) may ship `ArrayBuffer`/`Blob`
-  frames; for each *distinct* frame capture its **byte-length + a hex dump of the first ~16 bytes** —
-  enough to fingerprint the wire format (protobuf, MessagePack, CBOR, Yjs/CRDT, gRPC-web, or a
-  text/topic-prefixed channel protocol) without recording content. **Decode the signature as ASCII
-  first** — many "binary" protocols are framed text (STOMP, Phoenix, Centrifugo, a `+topic` channel
-  prefix), so a legible prefix names the protocol in one step before a binary-codec guess. The frame
-  *signature* is the finding; the payload is not.
-- **Storage shape inspector** — enumerate localStorage / sessionStorage / cookies / IndexedDB
-  **key names + lengths**, never raw values.
-- **JWT decoder** — decode the token *structure* (alg, claim names, exp) with identity claims
-  (`sub`, `email`, `phone`, `user_id`, `uid`, `sid`) redacted.
-- **Auth-channel disambiguation.** Token *location* is itself a tell: not in localStorage ⇒ an httpOnly
-  cookie **or** an SDK store (IndexedDB — Firebase/Auth0/Supabase/Cognito keep tokens there, which is why
-  a "no JWT in localStorage" result is a finding, not a dead end). Determine the *required* transport by
-  **omission**: replay one harmless authed read — using a field that needs the authed context (`viewer`/
-  `me`), **never** a schema-only field (GraphQL `__typename`, a health route) that resolves
-  unauthenticated — with each candidate credential dropped (cookie-only, then header-only); whichever
-  response comes back unauthorized names the channel the app actually requires.
-- If a realtime SDK is loaded (Pusher, Ably, Socket.IO, Centrifugo, Phoenix, …), bind a global listener
-  to capture the event/channel catalog.
-- **Third-party-SDK fingerprint** (cross-checks `infra-backend-fingerprint`). Storage key names + the
-  network host list name the embedded vendor stack — auth (Firebase/Auth0), flags (LaunchDarkly/Statsig),
-  support (Intercom/Zendesk/Ada), analytics (Segment/Amplitude/Datadog), payments (Stripe). Record them;
-  they corroborate the sub-processor list *independently* (a different dimension → promotes confidence).
-- **Drive op coverage from any catalog a sibling declared.** If `deployed-client-bundle` (or `api`/`docs`)
-  recovered a declared **route / endpoint / operation catalog** — a REST path table, a GraphQL operation
-  set, an RPC/tool method list — consume it as a **coverage checklist**: walk the routes/actions that
-  trigger the un-observed entries so the declared-vs-observed diff (§6) reflects a real walk, not just the
-  landing page.
-
-### 9.5 The capture contract (richer than a standard dimension dump)
-
-Under `research/<target>/dimensions/session/`:
-
-- **`captures/p*.md`** — per-phase narrative for the human: frontmatter (title, summary, scope,
-  status), **Method** (taps run, actions taken), **Findings** (concrete values — URLs, fields,
-  status codes), **Inferences**, **Diff vs reference**, **Open questions**. Decoded, opinionated;
-  don't paste raw bodies here.
-- **`raw/*.json`** — verbatim endpoint captures for the reproducer: a `_meta` header
-  (endpoint, auth-shape, captured_at, status, byte_length, redactions) + `body` (PII + secrets
-  redacted) + `_decoded_fields`. Digest bodies > 50 KB into a structured summary.
-- **`curls.md`** — reproducible curl per endpoint, the auth model documented, a variables block,
-  and an endpoint-catalog table.
-- **`captures/screens/`** (recommended) — one annotated screenshot per primary surface (each major nav
-  section), plus a short tour recording if useful. The fastest "what does it actually do" artifact for a
-  human reader, complementing the wire teardown — and the place the visual product map lives. Redact any
-  visible PII before saving.
-- **`_summary.md`** — the standard dimension bridge file (provenance frontmatter per §2) pointing
-  into the above, so Evaluation treats session like any other dimension. `method: session-playbook`,
-  `confidence: low` for Pass-1-only/inferred.
-
-### 9.6 Session ethics (binds hardest here — see §7 above)
-
-Never enter credentials; storage shape not values; redact every secret **inside the tap wrapper**
-before it reaches disk; **confirm before any Pass-2 state change or cost** (exact action + exact
-cost); no destructive actions; no bypassing the harness (redact, never evade); captures stay local;
-respect TOS. The provenance/redaction rules in §1–8 apply to everything written here.