@nurix/etna 0.15.2 → 0.15.4

package/data/harnesses/re/rules/re-ingestion.md

@@ -0,0 +1,266 @@
+# Ingestion Contract — Dump Shape · Provenance · Ethics · Session Playbook
+
+The reference knowledge for **Ingestion** (`re` harness, Mode 2). Every dimension dump obeys the
+contract in §1–8 so (a) the **evaluation** mode can rely on a stable shape and weight each claim by
+*how it was obtained*, and (b) a future agent can reproduce or extend any dump. §9 carries the
+deepest, gated dimension's method — the logged-in browser-session playbook — which the `session`
+agent follows.
+
+This contract generalizes the source playbook's three-format contract (`captures/*.md` narrative +
+`raw/*.json` verbatim + `curls.md`) to *any* dimension.
+
+> **Contract style = "lighter".** Provenance ("how it was dumped") lives in the
+> **`_summary.md` frontmatter**, not a separate `manifest.yaml`. There is no parallel YAML
+> machine-spine — everything stays markdown-with-frontmatter. Evaluation reads the frontmatter
+> to weight claims; self-correction reads it to detect under-delivery.
+
+---
+
+## 1. Folder layout
+
+```
+research/<target>/dimensions/<dimension>/
+├── _summary.md        ← decoded narrative + provenance frontmatter (human + machine read this)
+└── raw/               ← verbatim artifacts (the reproducer reads these)
+```
+
+Richer dimensions inherit the playbook contract (extra files, same `_summary.md` provenance):
+- **session**: `dimensions/session/{_summary.md, captures/p*.md, raw/*.json, curls.md}`
+- **wire-capture**: `dimensions/wire-capture/{_summary.md, captures/wire-narrative.md, raw/flows.json, curls.md}`
+
+---
+
+## 2. `_summary.md` — required shape (with provenance frontmatter)
+
+```markdown
+---
+dimension: <codebase|docs|packages|api|website|community|session|deployed-client-bundle|infra-backend-fingerprint|wire-capture|distribution-artifacts>
+target: <slug>
+status: complete | partial | blocked
+# --- provenance (the "how it was dumped" layer evaluation weights on) ---
+access_grade_used: <source:full | runtime:reachable | auth:have-login | presence:rich | ...>
+method: <the ACTUAL extraction method — see §3 vocabulary>
+completeness_pct: <0-100, honest self-estimate of reachable surface captured>
+confidence: high | medium | low      # how much to TRUST what was captured (see §3)
+captured_at: <YYYY-MM-DD>
+sources: [<urls / paths actually consulted>]
+gaps: [<structured — each is a thing evaluation must NOT treat as covered>]
+---
+
+# <Target> — <Dimension> dump
+
+## Method
+Exactly what was done, which tools, what was skipped and why. (The prose behind the frontmatter `method`.)
+
+## Findings
+Ordered, concrete observations — real values (package names, versions, endpoint paths, file
+paths, page URLs, prices). Tables liberally. The load-bearing section.
+
+## Inferences
+What the findings imply. Mark inference vs. fact.
+
+## Open questions
+What couldn't be captured and why. Never fabricate to fill a gap.
+
+## Artifacts
+Index of `raw/` files, one line each.
+```
+
+Keep `status` and `confidence` honest — they drive the entire evaluation weighting.
+
+---
+
+## 3. Method & confidence vocabulary (drives evaluation weighting)
+
+`method` is one of: `clone-and-map`, `source-map-reassembly`, `registry-metadata`,
+`openapi-verbatim`, `graphql-introspection`, `crawl-clip`, `docs-reconstructed`,
+`bundle-string-mine`, `dns-ct-fingerprint`, `wire-tap-browser`, `wire-har`, `wire-mitmproxy`,
+`binary-extract` (asar/apk/crx), `session-playbook`, `inferred`.
+
+**Confidence is a function of method** (evaluation relies on this mapping):
+- **high** — verbatim machine artifacts: `openapi-verbatim`, `graphql-introspection`, `registry-metadata`, `clone-and-map`, `binary-extract`.
+- **medium** — `crawl-clip`, `docs-reconstructed`, `bundle-string-mine`, `dns-ct-fingerprint`, `wire-har`.
+- **low** — `inferred`, `session-playbook` Pass-1-only, `wire-tap-browser` shapes-only, any wire-drift observation without a spec.
+
+---
+
+## 4. `raw/*` — required shape
+
+Structured captures (API responses, registry metadata, schemas) → JSON with a `_meta` header:
+```json
+{
+  "_meta": { "source": "...", "captured_at": "YYYY-MM-DD", "kind": "...", "byte_length": 0, "redactions": ["field → <REDACTED:...>"] },
+  "body": { /* verbatim, PII + secrets redacted */ },
+  "_decoded": { "field": "meaning + observed values" },
+  "_observation": "optional — what's load-bearing"
+}
+```
+Prose captures (doc/marketing pages, structure maps, catalogs) → markdown with a metadata line:
+`<!-- source: <url-or-path> · captured_at: YYYY-MM-DD · method: <method> -->`
+
+**One file per logical artifact**, named by what it is (`endpoint-catalog.md`, `react-exports.md`,
+`pricing.md`, `structure-map.md`).
+
+---
+
+## 5. Size discipline — reduce, don't dump
+
+Large artifacts (a 3 MB bundle, a 5 MB response, a 700-page sitemap) are **digested**, not pasted:
+the schema/shape, enum distributions, one redacted representative sample, counts. A `raw/` file is
+readable, not a memory dump. Set `digested: true` context in the artifact's `_meta`.
+
+---
+
+## 6. The shared-surface seam rule (bundle ↔ session ↔ wire-capture)
+
+`deployed-client-bundle` (static unauth SPA), `session` (authenticated browser runtime), and
+`wire-capture` (non-browser client) can all surface the same **API path catalog** and **feature
+flags**. To keep them consumable instead of triplicated:
+
+- Use **one canonical artifact name** across all three: `raw/api-path-catalog.md` and
+  `raw/feature-flags.md`, each row carrying a `source: bundle | session | wire` column.
+- Evaluation owns the **three-way diff** (static-declared vs browser-observed vs nonbrowser-observed)
+  in `data-model-api-surface.md`. A path that appears in `bundle` but never in `session`/`wire` is a
+  dormant/unshipped route — a finding, not noise.
+
+---
+
+## 7. Ethics & redaction (non-negotiable — binds EVERY dimension)
+
+Carried from the source playbook (`setup.md` §6). Violating them voids the work.
+
+1. **Never enter credentials.** The user signs into any logged-in surface beforehand.
+2. **Storage: shape, not values.** No raw cookies/tokens/JWT-signatures. Decode JWT *structure* with identity claims (`sub`,`email`,`phone`,`user_id`,`uid`,`sid`) redacted.
+3. **Never save secrets.** Before any `raw/` write, redact every credential-shaped field → `<REDACTED:...>` + log in `_meta.redactions`. Field patterns: `api_key`,`*_secret`,`client_secret`,`webhook_secret`,`signing_*`,`password`,`private_key`,`access_token`/`refresh_token` in bodies, vendor-named secrets. Value patterns: `sk_live_…`/`pk_live_…` (Stripe), `sk-…` (OpenAI), `xox[bp]-…` (Slack), `AKIA…` (AWS), `gh[ps]_…` (GitHub), `AIza…` (Google), `eyJ…` (JWT), opaque base64 ≥32 chars in a credential field. Over-redact when unsure.
+4. **Confirm before state changes** — anything that creates state, costs money, or is visible to others (even a draft toggle). State the exact action + estimated cost.
+5. **No destructive actions.** Document the absence; move on.
+6. **No bypassing safety harnesses** — redact, never evade.
+7. **No exfiltration.** Captures stay local under `research/<target>/`.
+8. **Respect TOS & robots.** For spec-writing / integration / competitive analysis — not credential theft, rate-limit bypass, or scraping.
+
+### 7.1 wire-capture & active-interception ethics (the strictest tier)
+
+The `wire-capture` dimension climbs a **3-rung ladder, lowest privilege first**, and records the
+chosen rung in `_meta`:
+
+1. **Browser network tap (default, unauth).** Chrome `read_network_requests` + the in-page
+   fetch/XHR interceptor on the **public** surface only. Capture request/response *shapes*; redact
+   in the wrapper before disk. No login. → `method: wire-tap-browser`.
+2. **HAR export.** Export DevTools HAR of the unauth SPA's XHRs, digest to an endpoint catalog,
+   scrub `Authorization`/`Cookie`/secret-shaped values. → `method: wire-har`.
+3. **mitmproxy / intercepting proxy — GATED, opt-in, user-run.** ONLY first-party traffic the
+   **user owns** (their own desktop/mobile/CLI client), ONLY with explicit chat authorization, ONLY
+   TOS-checked. The harness NEVER stands up a proxy against a third party's infra unattended; the
+   *user* installs the CA and routes their client — the agent only ingests exported flows. →
+   `method: wire-mitmproxy`. Same explicit-authorization loop as session Pass-2.
+
+> Wireshark / packet capture is explicitly **not** used: for HTTPS it yields only TLS ciphertext,
+> and a packet sweep of a third party is the bright line this harness will not cross.
+
+### 7.2 distribution-artifacts ethics
+Downloading a public binary (CRX/`.asar`/`.ipa`/`.apk`) and reading its manifest/strings is fine.
+But binary **decompilation** can cross from "inspect" to "reverse-engineer in violation of EULA":
+the agent reads the artifact's own license/EULA first and records a restrictive one as an Open
+question rather than silently decompiling.
+
+---
+
+## 8. Completeness check
+A dump is complete when: `_summary.md` exists with full provenance frontmatter + honest `status`;
+every Finding is backed by a `raw/` artifact or inline value; `raw/` artifacts have `_meta` +
+decoding (large ones digested); redaction pass run; `gaps` recorded (never fabricated).
+
+---
+
+## 9. The session dimension — logged-in browser-session playbook
+
+The method for the **session** dimension: reverse-engineering the *authenticated runtime* of a
+product by driving it through a browser (Chrome MCP) and instrumenting it. It produces a
+section-by-section understanding of a product's wire protocol, auth, and state model **without
+source-code access**. The `session` agent follows this section.
+
+This is the deepest and most sensitive dimension. The ethics in §7 above bind hardest here.
+
+### 9.1 Prerequisites (verify before anything else — all three non-negotiable)
+
+| # | Precondition | How to verify |
+|---|--------------|---------------|
+| 1 | Target is **pre-logged-in** in Chrome | open the target URL in the connected tab → an authenticated surface (a dashboard, not a sign-in page). The agent never enters credentials. |
+| 2 | **Claude Chrome extension** installed | `mcp__claude-in-chrome__tabs_context_mcp` returns live tabs |
+| 3 | **Chrome CDP** connected | the same call returns stable tab IDs |
+
+If any fails, return `blocked` naming the missing precondition.
+
+### 9.2 The three-pass strategy
+
+A single linear sweep wastes context on things you can read for free and wastes credits on
+actions whose response shape you could infer. Use three passes.
+
+**Pass 1 — read-only mining (no money, no state change).** If the user already has data (a
+project, conversation, account, order), mine it first. Hit every list/state/history endpoint with
+the existing token; capture verbatim bodies. This yields ~70% of the spec for free. Mine, by role:
+identity (`/me`, `/account`, `/workspaces`), list (the canonical taxonomy), single-resource state
+(server-injected vs client-sent fields), history/event-log (a whole trajectory in one call),
+runtime/preview state, per-resource billing, account billing, feature flags/cohort, integration
+catalog (`/mcp/`, `/integrations/`), and schema/capabilities (`/openapi.json`, `/.well-known/*`,
+`/page-metadata`).
+
+**Pass 2 — fresh-run validation (with explicit cost/state-change confirmation).** Only after
+Pass 1, and only with explicit user OK stating the exact action + cheapest mode + estimated cost.
+Catches what Pass 1 can't: the submit/write request body, live state-machine transitions, the
+completion event, server-side resolution of shorthand, ID/slug-generation, the real iteration
+endpoint, and the full realtime event catalog. **Legitimate to skip** if the user opts out — leave
+write-side shapes as open questions; never fabricate them.
+
+**Pass 3 — synthesis.** Walk the captures, pull each finding into the dimension's `_summary.md`,
+note the cleverest design choices, defaults imposed, performance characteristics, and open
+questions.
+
+### 9.3 The P0 → P6 phases
+
+The same phases apply whether the target is an AI builder, analytics dashboard, CRM, or collab tool:
+
+| Phase | Focus |
+|-------|-------|
+| **P0 Surface** | routes, bundles, storage shape, the API host(s), the auth tell |
+| **P1 Auth** | token type + issuer + storage location + refresh path; tenancy model |
+| **P2 Core action** | the primary write — the submit/create body and its server resolution |
+| **P3 Streaming** | the realtime transport (WS / SSE / WebChannel / Pusher / none) + event catalog |
+| **P4 Runtime/state** | sandbox/preview/build/deploy state; the live result surface |
+| **P5 Iteration** | the follow-up/edit endpoint (often surprisingly named) + any auto-iteration loop |
+| **P6 Synthesis** | reconcile into the dimension's `_summary.md` + `captures/` + `curls.md` |
+
+### 9.4 Runtime taps (Pass-1 instrumentation, run in-page)
+
+- **WebSocket / EventSource / fetch+XHR interceptors** — wrap the constructors/methods to log
+  URLs, frames, and bodies (shapes, not secrets — redact in the wrapper before anything reaches
+  disk). Add a `SCRUB(value)` step inside the fetch wrapper so credentials never reach the harness.
+- **Storage shape inspector** — enumerate localStorage / sessionStorage / cookies / IndexedDB
+  **key names + lengths**, never raw values.
+- **JWT decoder** — decode the token *structure* (alg, claim names, exp) with identity claims
+  (`sub`, `email`, `phone`, `user_id`, `uid`, `sid`) redacted.
+- If a Pusher/realtime SDK is loaded, bind a global listener to capture the event catalog.
+
+### 9.5 The capture contract (richer than a standard dimension dump)
+
+Under `research/<target>/dimensions/session/`:
+
+- **`captures/p*.md`** — per-phase narrative for the human: frontmatter (title, summary, scope,
+  status), **Method** (taps run, actions taken), **Findings** (concrete values — URLs, fields,
+  status codes), **Inferences**, **Diff vs reference**, **Open questions**. Decoded, opinionated;
+  don't paste raw bodies here.
+- **`raw/*.json`** — verbatim endpoint captures for the reproducer: a `_meta` header
+  (endpoint, auth-shape, captured_at, status, byte_length, redactions) + `body` (PII + secrets
+  redacted) + `_decoded_fields`. Digest bodies > 50 KB into a structured summary.
+- **`curls.md`** — reproducible curl per endpoint, the auth model documented, a variables block,
+  and an endpoint-catalog table.
+- **`_summary.md`** — the standard dimension bridge file (provenance frontmatter per §2) pointing
+  into the above, so Evaluation treats session like any other dimension. `method: session-playbook`,
+  `confidence: low` for Pass-1-only/inferred.
+
+### 9.6 Session ethics (binds hardest here — see §7 above)
+
+Never enter credentials; storage shape not values; redact every secret **inside the tap wrapper**
+before it reaches disk; **confirm before any Pass-2 state change or cost** (exact action + exact
+cost); no destructive actions; no bypassing the harness (redact, never evade); captures stay local;
+respect TOS. The provenance/redaction rules in §1–8 apply to everything written here.

package/package.json

@@ -1,7 +1,7 @@
 {
     "name": "@nurix/etna",
-    "version": "0.15.2",
-    "description": "Bootstrap a project with a Nurix harness — installs a Claude Code plugin from the nurixlabs/skills marketplace and drops the harness's CLAUDE.md at the project root.",
+    "version": "0.15.4",
+    "description": "Bootstrap a project with a Nurix harness — copies the harness's CLAUDE.md, skills, agents, rules, and hooks directly into the project from a bundled payload (no marketplace fetch, no network).",
     "type": "module",
     "bin": {
         "etna": "bin/etna.js"