@nurix/etna 0.19.40 → 0.19.42

package/data/harnesses/discover/rules/tooling.md

@@ -0,0 +1,205 @@
+# Tooling Reference — Host Capability · Tiers · The Bright Line
+
+The reference for **Step 0** (`discover` harness, run once per corpus folder before Mode 1). Capture depth
+is bounded by host tooling most machines lack. This file carries the **tier lists** (what to recommend),
+the **ethics classification** (clean · gated · avoid), the **per-dimension gap map** (what a bare host
+cannot do), and the **install caveats** that have actually bitten.
+
+**Nothing here is ever installed automatically.** The harness detects what is present, recommends only what
+is missing, and records the result in `.claude/discover-tooling.json` so it never re-prompts in the same
+folder. The harness recommends; the user decides. A user who declines proceeds to Mode 1 on whatever is
+present — a missing capability is re-surfaced at the Mode 2 capability-gap checkpoint (`ingestion.md` §8),
+never silently degraded.
+
+A host almost always arrives strong on languages and package managers (node, pnpm, python, uv, cargo, gh,
+jq, ripgrep) and **bare on capture/analysis tooling**. That gap is the difference between a shallow capture
+and a deep one.
+
+---
+
+## The three classes (this is an ethics classification, not a convenience one)
+
+| Class | Meaning | Rule |
+| --- | --- | --- |
+| **clean** | passive / local-only; always in-bounds | recommend freely, offer the one-paste block |
+| **gated** | in-bounds **only** first-party / user-owned / opt-in — the user installs the CA, routes *their own* client, owns the keys | surface the command, **never run it for the user** |
+| **avoid** | crosses the harness bright line (active scanning, exploitation, third-party interception) | never recommend; listed so it is not added by reflex |
+
+The harness is for **spec-writing / integration / competitive analysis** — passive or first-party only,
+respecting TOS and robots (`ingestion.md` §7). No tool changes that; the gated ones simply need the user in
+the loop, exactly as `session` Pass-2 and `wire-capture` rung 3 do.
+
+---
+
+## Why this file matters more since the §5 rewrite
+
+`ingestion.md` §5 now requires **verbatim dumping** — the artifact lands on disk whole, and the digest is
+written *beside* it, never *instead* of it. The obvious objection is "a 6 MB spec is unusable." **These
+tools are the answer to that objection.** They are what make a verbatim corpus queryable, which is what
+makes dumping-not-digesting practical:
+
+| Instead of digesting… | Dump verbatim, then |
+| --- | --- |
+| a 6 MB OpenAPI spec → a hand-typed endpoint table | `curl -o raw/spec.json` → **spectral** to catalog, **oasdiff** to diff, **gron** to grep |
+| a 9 MB minified bundle → prose about the routes | `curl -o raw/bundle.js` → **webcrack** / **source-map-explorer** to read, **ast-grep** to query |
+| a HAR → a narrative of the calls | keep `raw/flows.har` → **gron** to flatten and grep |
+| a docs site → distilled sections only | keep the page sources → **trafilatura** / **pandoc** to render |
+| a package's exports → string-mined prose | `npm view --json` → **typescript + attw + publint**, **griffe** for Python |
+
+**A digest is a lossy compression you cannot undo; a verbatim artifact plus a query tool is lossless.**
+Recommend Tier 1 on that basis, not as a nice-to-have.
+
+---
+
+## Tier 1 — clean, install now (broad leverage)
+
+Highest-consensus, always in-bounds. Offer as one paste:
+
+```bash
+# Homebrew (CLI analysis + data-wrangling + web→text)
+brew install gron pandoc ast-grep scc universal-ctags oasdiff fd yq dasel monolith tokei sevenzip
+# Rust
+cargo install htmlq
+# Python (via uv — pipx is typically absent; uv tool install is the path)
+uv tool install trafilatura && uv tool install griffe
+# Node global (JS/TS surface + bundle de-obfuscation + spec lint)
+npm i -g webcrack source-map-explorer @arethetypeswrong/cli publint @electron/asar typescript @stoplight/spectral-cli
+# Headless browser for JS-rendered docs/marketing (clean — the user's own headless instance)
+npm i -g playwright && npx playwright install chromium
+```
+
+| Tool | Unlocks | Dimensions served |
+| --- | --- | --- |
+| **gron** | flatten any JSON → greppable/diffable `path = value` (specs, captures, manifests, HAR, configs) | **7** (codebase, api, session, bundle, infra, wire, cross-cutting) |
+| **trafilatura** | boilerplate-stripped main content + dates from HTML → clean markdown; feed/sitemap discovery | **4** (docs, website, community, infra) |
+| **pandoc** | deterministic HTML→GFM (headings, tables, code fences) at scale | 3 (docs, website, community) |
+| **ast-grep** | tree-sitter structural search/count across many files + langs (call sites, routes, exports) — catches what regex misses | 2 (codebase, bundle) |
+| **oasdiff** | structured breaking-vs-non-breaking OpenAPI diff — makes the published-vs-app-own diff machine-checked | 2 (api, cross-cutting) |
+| **webcrack** | un-minify / un-webpack / de-obfuscate SPA & Electron JS into readable modules | 2 (bundle, distribution) |
+| **scc** | instant LOC-by-language + complexity census of a fresh clone (JSON) — the codebase opening move | 1 (codebase) |
+| **universal-ctags** | real multi-lang JSON symbol index (the host's BSD ctags is broken for this) | 1 (codebase) |
+| **htmlq** | surgical CSS-selector extraction (pricing cards, feature tables) instead of regex | 1 (website) |
+| **source-map-explorer** | reassemble a shipped `.map` → original `src/` tree (the `source-map-reassembly` method) | 1 (bundle) |
+| **spectral** | validate/resolve an OpenAPI/AsyncAPI spec → structured endpoint/security/`$ref` catalog | 1 (api) |
+| **griffe** | extract a PyPI package's full public API (the `.d.ts` equivalent for Python) | 1 (packages) |
+| **@arethetypeswrong/cli + typescript + publint** | parse the real `.d.ts`/exports type graph + grade the publish surface (verbatim, not string-mined) | 1 (packages) |
+| **@electron/asar** | extract Electron `app.asar` → JS module tree (the distribution step the playbook names but cannot otherwise run) | 1 (distribution) |
+| **fd · yq · dasel · monolith · tokei · sevenzip** | gitignore-aware file discovery · YAML/XML/TOML query · RSS/XML→JSON · single-file page archive · LOC cross-check · multi-format extract | several |
+
+---
+
+## Tier 2 — gated, install when a run needs them (first-party / passive only)
+
+The capture lanes. In-bounds **only** for the user's own session, the user's own first-party clients, or
+passive sources. **Surface these; never run them for the user.**
+
+```bash
+brew install mitmproxy websocat grpcurl subfinder tlsx testssl jadx apktool
+npm i -g chrome-remote-interface get-graphql-schema
+# one-time, USER-RUN: expose a CDP port on the user's own Chrome (closes the WS-frame blind spot)
+#   open -na "Google Chrome" --args --remote-debugging-port=9222
+# mobile-only, the user's own device + explicit opt-in:  uv tool install frida-tools
+```
+
+| Tool | Unlocks | Ethics note |
+| --- | --- | --- |
+| **mitmproxy** | decode full HTTPS + WebSocket flows of any **first-party** client (CLI/desktop/mobile/SDK) — the `wire-capture` rung-3 keystone (`ingestion.md` §7.1) | user installs the CA and routes *their own* client only |
+| **chrome-remote-interface** | raw CDP → `Network.webSocketFrameReceived`/`Created`, SSE, full bodies — **captures the on-load WS frames the browser MCP's HTTP-only `read_network_requests` drops** | the user's own logged-in session |
+| **websocat** | curl-for-WebSockets: connect / frame / replay / hexdump a realtime endpoint outside the browser | first-party endpoint |
+| **grpcurl** | list services + invoke RPCs via gRPC server reflection → readable op/message catalog | first-party endpoint |
+| **get-graphql-schema** | introspect a first-party GraphQL endpoint → full SDL | first-party endpoint |
+| **subfinder · tlsx · testssl.sh** | passive CT/DNS subdomain breadth + structured TLS posture (vs hand-rolled crt.sh + openssl text) | passive aggregation; respect rate limits |
+| **jadx · apktool** | decompile Android `.apk`/`.dex` → Java + decode `AndroidManifest`/resources (capability map) | public binary; read the EULA first (`ingestion.md` §7.2) |
+| **frida-tools** | cert-pinning bypass to capture **the user's own** mobile client's wire | the user's own device + explicit opt-in only |
+
+> **The single biggest unlock for a realtime target** is `chrome-remote-interface` + a Chrome
+> `--remote-debugging-port` + `websocat` + `mitmproxy` together: they give `session`/`wire-capture` a real
+> WebSocket/SSE capture path instead of a client-side-remount workaround. The browser MCP's
+> `read_network_requests` is **HTTP-only** — that is *why* a realtime tap comes back empty until the remount
+> trick (`tradecraft.md` gotcha #13, `ingestion.md` §9.4). A CDP port removes the injection-timing problem
+> entirely rather than working around it.
+>
+> (Origin: Emergent — with no CDP port open, the run fell back to in-page taps. That worked, but a
+> post-boot `window.WebSocket` wrapper is structurally blind to a library socket opened at boot, and the
+> prior run published a false negative because of it.)
+
+---
+
+## AVOID — never recommend (this is the bright line)
+
+Out-of-bounds for a spec-writing / competitive-analysis harness. Listed so they are not added by reflex:
+
+- **`nmap`, `masscan`, `naabu`** — active port sweeps / service-version probing of remote hosts.
+- **`nuclei`** — template-based remote vulnerability scanning.
+- **`sqlmap`** — injection / exploitation.
+- **`amass` (active mode)** — active DNS brute-forcing. Its *passive* mode overlaps subfinder; prefer subfinder.
+- **`mitmproxy` or `frida` against a third party**, **`socat` as a public relay** — intercepting traffic the
+  user does not own is the line this harness will not cross.
+
+These stay absent **by design**. The recon ethics depend on it, and a tool absent from the host is a
+stronger guarantee than a rule telling the agent not to use it.
+
+---
+
+## Per-dimension gap map (what a bare host cannot do)
+
+| Dimension | Biggest capability gap on a bare host | Top addition(s) |
+| --- | --- | --- |
+| **codebase** | no LOC census, no AST search; the host's BSD ctags is broken for this | scc · ast-grep · universal-ctags (+ fd) |
+| **docs** | no HTML/MDX→md, no JS render → client-rendered docs come back empty | pandoc · trafilatura · playwright |
+| **packages** | the real `.d.ts`/exports surface is string-mined, not parsed | typescript + attw + publint · griffe (Py) · yq |
+| **api** | can fetch a spec but cannot lint / diff / introspect / exercise it | spectral · oasdiff · get-graphql-schema · grpcurl · websocat |
+| **website** | no JS render, no clean table/text extraction → misses pricing and feature grids | trafilatura · htmlq · playwright |
+| **community** | no clean changelog/blog→text, no RSS→JSON | trafilatura · pandoc · dasel |
+| **session** | **structurally blind to WebSocket/SSE frames** (HTTP-only MCP, no CDP port) | chrome-remote-interface · mitmproxy · websocat |
+| **deployed-client-bundle** | a minified/obfuscated blob stays opaque; no source-map reassembly | webcrack · source-map-explorer · ast-grep · gron |
+| **infra-backend-fingerprint** | shallow CT coverage, TLS posture by hand, sub-processor pages scraped raw | subfinder · tlsx · trafilatura (passive only) |
+| **wire-capture** | the keystone (mitmproxy) plus every non-browser replay/decode client is absent | mitmproxy · websocat · grpcurl |
+| **distribution-artifacts** | only CRX/Mach-O readable; no Electron or Android decoders | @electron/asar · jadx · apktool · webcrack · sevenzip |
+| **external-reputation / hiring-intel** | bot-walled review and ATS pages return 403 to plain fetch | playwright (own headless instance) · trafilatura · dasel |
+| **cross-cutting (synthesis)** | no structured diff for the N-way seam reconciliation; no out-of-band WS capture | gron · oasdiff · chrome-remote-interface · yq |
+
+---
+
+## Known install caveats (each of these has actually bitten)
+
+- **`webcrack` needs Node 22.** Its hard dependency `isolated-vm` is a native (node-gyp) addon that fails to
+  build against Node 26, and `--omit=optional` does not help — the dep is non-optional. Run it under a Node
+  22 LTS shim (`npx -y webcrack@latest …` in a Node-22 env), or lean on the already-clean
+  `source-map-explorer` (the preferred `source-map-reassembly` path whenever the SPA ships `.map` files)
+  plus `ast-grep`. De-obfuscating a **no-maps, minified** bundle is the only gap this leaves.
+- **`chrome-remote-interface` is a library, not a CLI.** It installs as a Node module and is invoked from a
+  script, not a shell binary. **Detect it with `npm ls -g chrome-remote-interface`, not `command -v`** — a
+  `command -v` probe reports it missing when it is installed, and the run then wrongly records a capability
+  gap.
+- **`pipx` is typically absent; use `uv tool install`** for Python CLIs (uv is present). Simpler than adding pipx.
+- **`playwright` is clean** — the user's own headless instance. It unblocks JS-rendered docs and marketing
+  for `docs` + `website` **without contending for the gated `session` Chrome**, which matters because the
+  browser is a singleton (`ingestion.md` §7 rule 9).
+
+---
+
+## How Step 0 uses this reference
+
+Once per corpus folder, before Mode 1. If `.claude/discover-tooling.json` exists, tooling was already
+reviewed here — **skip straight to Mode 1 and never re-prompt.** If it is absent:
+
+1. Read this file — the tier lists, the classification, the AVOID list.
+2. **Detect before recommending** — `command -v <tool>` per tool, and `npm ls -g <pkg>` for the Node
+   libraries that ship no binary (see the caveat above). Detection only; nothing is installed at this step.
+3. **Recommend only what is missing**, grouped **clean** (offer the Tier-1 one-paste block) vs **gated**
+   (surface the command, never run it). Never suggest anything on the AVOID list. If nothing is missing,
+   say so in one line.
+4. Write the state file so the harness never asks again in this folder:
+   ```json
+   { "checked": "<ISO date>", "present": ["gron", "pandoc"], "missing": ["mitmproxy"], "recommended": true }
+   ```
+
+The state file is **host-local** (tools live on the machine, not the repo) — gitignore it in a shared
+folder; delete it to force a fresh check after installing a batch. **This step never blocks a run.**
+
+> **Maintenance loop.** When a run hits a capability wall — a format nothing on the host can read, a
+> capture lane that needed a tool that was not listed, an install that failed in a new way — fold it back
+> in here: the tool into the right tier with its ethics class, or the failure into the caveats. This file
+> only stays useful if every new wall is recorded (the same loop `ingestion.md` §7.0 carries for redaction
+> patterns).

package/data/harnesses/discover/rules/tradecraft.md

@@ -69,12 +69,25 @@ The runnable in-page instrumentation `session`/`wire-capture` install to mine th
       return origOpen.apply(this, arguments);
     };
     XMLHttpRequest.prototype.send = function (body) {
-      window.__fetchTap.push({
+      const rec = {
         kind: "xhr",
         url: this.__url,
         method: this.__method,
-        body: typeof body === "string" ? body.slice(0, 600) : "[non-string]",
+        // clamp MATCHES the fetch tap (8000) — see the clamp-parity note below
+        body: typeof body === "string" ? body.slice(0, 8000) : "[non-string]",
         at: Date.now(),
+      };
+      window.__fetchTap.push(rec);
+      this.addEventListener("load", () => {
+        try {
+          rec.status = this.status;
+          rec.respCt = (this.getResponseHeader("content-type") || "").split(";")[0];
+          const t = this.responseText;
+          if (typeof t === "string") {
+            rec.respLen = t.length;
+            rec.resp = t.slice(0, 12000);
+          }
+        } catch (_) {}
       });
       return origSend.apply(this, arguments);
     };
@@ -87,6 +100,105 @@ The runnable in-page instrumentation `session`/`wire-capture` install to mine th
 
 ---
 
+### 1b. The clipboard-to-local channel (unlimited size, bypasses BOTH content filters)
+
+**When to use:** any captured body that exceeds the `javascript_tool` display cap (as low as ~1.1 KB) or
+that contains blocklisted shapes (query strings, long base64, many UUIDs) which would return
+`[BLOCKED: Cookie/query string data]`. Paging a 90 KB body at ~1 KB a slice costs ~90 round trips; this
+costs one. It is also immune to Chrome's silent multi-download block.
+
+**Requires document focus** — click the page once before calling it, or `execCommand('copy')` returns `false`.
+
+```js
+window.__CP = (obj) => {
+  const ta = document.createElement("textarea");
+  ta.value = typeof obj === "string" ? obj : JSON.stringify(obj, null, 1);
+  ta.style.cssText = "position:fixed;left:0;top:0;width:100px;height:60px;opacity:0.01;z-index:2147483647";
+  document.body.appendChild(ta); ta.focus(); ta.select();
+  ta.setSelectionRange(0, ta.value.length);
+  const ok = document.execCommand("copy"); ta.remove();
+  return "copied=" + ok + " len=" + ta.value.length;   // returns only a length — never the payload
+};
+```
+
+Then read it locally — the payload never passes through the model's context or the content filter:
+
+```bash
+pbpaste > capture.json      # macOS
+xclip -o  > capture.json    # Linux
+```
+
+**Scrub before copying**, not after: build the object with `__SCRUB` first, then `window.__CP(pack)`. The
+clipboard is local, so this stays inside the no-exfiltration rule (ingestion §7 rule 7).
+
+> (Origin: Emergent, second run — the prior run recorded router model-ids as "masked `[variant]`" and 6 of
+> 77 agent rows as display-cap truncated. Both were artifacts of the *display* layer, not the capture: the
+> same values came back verbatim through the clipboard channel. **The display layer is not the capture
+> layer.**)
+
+> **Clamp parity is load-bearing — the XHR branch must match the fetch branch (Mode-5, Emergent).** Many
+> SPAs call their own API through **axios/XHR, not `fetch`** (Emergent does). An XHR branch clamped smaller
+> than the fetch branch silently truncates exactly the payload you care about most — a real run lost the
+> primary write body (`POST /jobs/v0/submit-queue/`) to a 600-char clamp and had to re-wrap mid-capture.
+> Keep both at 8000 (request) / 12000 (response).
+>
+> **Re-wrapping XHR mid-run is safe; re-wrapping `fetch` is not.** The double-wrap hazard is specific to
+> `fetch` + `response.clone()` (gotcha §2.9): a second body interceptor breaks the inner wrapper's clone.
+> XHR has no clone semantics, so installing a second, wider XHR wrapper (under a fresh sentinel such as
+> `window.__xhr2`) is a legitimate mid-run recovery when you discover the first clamp was too small.
+
+---
+
+### 1c. Pre-shutter redaction overlay (run before EVERY screenshot)
+
+**When to use:** any `save_to_disk` capture (`ingestion.md` §5.5). Redaction must happen **in the DOM before
+the shutter** — a leaked pixel is undetectable afterwards, because the §7.0 4c literal sweep is text-only and
+`grep` cannot read a PNG.
+
+Idempotent; re-run it after every navigation and before every capture, since the elements needing redaction
+change as you move. Extend `PII` with the current session's own identifiers at run time (never commit them).
+
+```js
+(() => {
+  if (!document.getElementById("__redact_style")) {
+    const st = document.createElement("style");
+    st.id = "__redact_style";
+    st.textContent = `.__redacted{background:#111 !important;color:transparent !important;
+      border-radius:3px !important;text-shadow:none !important}
+      .__redacted *{visibility:hidden !important}
+      .__blurred{filter:blur(14px) !important}`;
+    document.head.appendChild(st);
+  }
+  // 1. account identity — extend per run (name / email / org / workspace slug)
+  const PII = /REPLACE_WITH_RUN_IDENTIFIERS/i;
+  let n = 0;
+  document.querySelectorAll("body *").forEach(el => {
+    if (el.children.length === 0 && el.textContent && PII.test(el.textContent)) {
+      el.classList.add("__redacted"); n++;
+    }
+  });
+  // 2. THIRD-PARTY SURFACES — the corner-of-the-frame trap. Blur wholesale; do not enumerate their text.
+  const THIRD_PARTY = [
+    '[class*="intercom" i]','[id*="intercom" i]','iframe[src*="intercom"]',
+    '[class*="crisp" i]','[id*="crisp" i]','[class*="drift" i]','[class*="zendesk" i]',
+    '[class*="beacon" i]','[class*="helpscout" i]','[class*="freshchat" i]',
+    '[class*="toast" i]','[class*="notification" i]','[role="alert"]','[role="status"]',
+    '[class*="presence" i]','[class*="avatar-group" i]','[class*="online" i]'
+  ];
+  document.querySelectorAll(THIRD_PARTY.join(",")).forEach(el => { el.classList.add("__blurred"); n++; });
+  // 3. avatars anywhere
+  document.querySelectorAll('img[src*="avatar"],img[src*="googleusercontent"],[class*="avatar" i] img')
+    .forEach(el => { el.classList.add("__blurred"); n++; });
+  return "redacted_nodes=" + n;
+})()
+```
+
+> **Then LOOK at the saved image before asserting it is clean.** The overlay handles what you predicted;
+> the frame may hold what you did not. Record `redacted: <what>` in `screens/_index.md` — never a blanket
+> "no PII in the pixels" claim you did not verify by reading the file (`ingestion.md` §5.5 origin note).
+
+---
+
 ### 2. WebSocket constructor wrapper (with binary-frame signature handling)
 
 **When to use:** any realtime/collab/streaming layer carried over WebSocket. Catches only sockets created **after** install — a library already initialised at page-boot (Pusher/Ably) is invisible here; use tap 4 instead. For an on-load socket, client-side-nav to a fresh room so a new socket is constructed through your wrapper (ingestion §9.4).
@@ -286,6 +398,8 @@ The Chrome-MCP browser-driving layer has ~27 recurring failure modes; every one
 27. **Probe Bearer vs cookie vs raw-token explicitly** — a JS-readable token cookie does NOT imply cookie-auth; on Creatify and Omneky the API rejects the raw cookie (`credentials:'include'` → `Failed to fetch` under `ACAO:*`) and accepts only an explicit `Authorization: Bearer` → probe all three (Bearer header / cookie / raw-token) to pin the real header model before mining. _(§2.26)_
 28. **Dynamic/runtime-constructed realtime hub URLs can't be recovered by bundle regex** — AdCreative's SignalR hub path is built at runtime in `initializeHub` with no static literal → record realtime _presence_ + transport as confirmed, the URL + message contract as open (needs Pass 2). Don't report "no realtime" just because grep found no socket URL. _(§2.27 — AdCreative)_
 
+29. **`javascript_tool` does NOT await promises — an `async` IIFE returns `{}`** — the tool stringifies the pending Promise, so `(async () => { const r = await fetch(...); return r; })()` yields a bare `{}` that looks identical to a content block or an empty result → **never `await` at the top level.** Use fire-and-forget + poll: fire the requests storing results on `window.__X`, return a literal like `"fired"`, then poll `window.__X` in a second call. Three captures were silently lost to this before it was diagnosed. _(§2.28 — Emergent, second run)_
+
 ### The escape snippets (paste verbatim)
 
 **Fire-and-forget poll — CDP `Runtime.evaluate` times out at ~45s; a fetch that holds the connection open longer (long-poll, slow stream) must be fired, then polled (§1.10):**

package/data/harnesses/etna/harness.json

@@ -1,7 +1,7 @@
 {
   "harness": "etna",
   "name": "etna",
-  "version": "1.3.6",
+  "version": "1.3.7",
   "files": [
     ".prettierignore",
     ".prettierrc.json",

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@nurix/etna",
-    "version": "0.19.40",
+    "version": "0.19.42",
     "description": "Bootstrap a project with a Nurix harness — copies the harness's CLAUDE.md, skills, agents, rules, and hooks directly into the project from a bundled payload (no marketplace fetch, no network).",
     "type": "module",
     "bin": {
@@ -41,5 +41,5 @@
         "agent",
         "plugin"
     ],
-    "etnaFingerprint": "a4419d09d415"
+    "etnaFingerprint": "e47ee7aac58b"
 }

package/data/harnesses/discover/tooling-recommendations.md

@@ -1,119 +0,0 @@
-# Tooling recommendations — what to install to improve capture & synthesis
-
-> The `discover` harness reads this file on **first run** (Step 0 in [`CLAUDE.md`](CLAUDE.md)): it detects which of these tools are already on the host, recommends only the **missing** ones (the gated ones only with the user in the loop), and records the result in `.claude/discover-tooling.json` so it never re-prompts in this folder. Nothing here is installed automatically — the harness recommends, the user decides.
-
-The host almost always arrives strong on languages and package managers (node, pnpm, python, uv, cargo, gh, jq, ripgrep) but **bare on capture/analysis tooling** — the tools below are what turn a shallow capture into a deep one.
-
-## How to read this
-
-- **clean** — passive / local-only; always in-bounds. Install freely.
-- **gated** — fine, but only **first-party / user-owned / opt-in** (the user installs the CA, routes _their own_ client, owns the keys). mitmproxy, websocat, CDP, subfinder/tlsx, frida on the user's own device.
-- **avoid** — crosses the harness bright line (active scanning / exploitation / third-party packet capture). Listed so they're _not_ added by reflex.
-
-The harness is for **spec-writing / integration / competitive analysis** — passive or first-party only, respect TOS/robots. No tool below changes that; the gated ones just need the user in the loop.
-
----
-
-## Tier 1 — install now (clean, broad leverage)
-
-The highest-consensus, always-in-bounds tools. One paste:
-
-```bash
-# Homebrew (CLI analysis + data-wrangling + web→text)
-brew install gron pandoc ast-grep scc universal-ctags oasdiff fd yq dasel monolith tokei sevenzip
-# Rust
-cargo install htmlq
-# Python (via uv — pipx is absent; uv tool install is the path)
-uv tool install trafilatura && uv tool install griffe
-# Node global (JS/TS surface + bundle de-obfuscation + spec lint)
-npm i -g webcrack source-map-explorer @arethetypeswrong/cli publint @electron/asar typescript @stoplight/spectral-cli
-# Headless browser for JS-rendered docs/marketing (clean — the user's own headless instance)
-npm i -g playwright && npx playwright install chromium
-```
-
-| Tool | Unlocks | Dimensions served |
-| --- | --- | --- |
-| **gron** | flatten any JSON → greppable/diffable `path = value` (specs, captures, manifests, HAR, configs) | **7** (codebase, api, session, bundle, infra, wire, cross-cutting) |
-| **trafilatura** | boilerplate-stripped main-content + dates from HTML → clean markdown; feed/sitemap discovery | **4** (docs, website, community, infra) |
-| **pandoc** | deterministic HTML→GFM (headings, tables, code fences) at scale | 3 (docs, website, community) |
-| **ast-grep** | tree-sitter structural search/count across many files+langs (call sites, routes, exports) — catches what regex misses | 2 (codebase, bundle) |
-| **oasdiff** | structured breaking-vs-non-breaking OpenAPI diff — makes the published-vs-app-own diff machine-checked | 2 (api, cross-cutting) |
-| **webcrack** | un-minify / un-webpack / de-obfuscate SPA & Electron JS into readable modules | 2 (bundle, distribution) |
-| **scc** | instant LOC-by-language + complexity census of a fresh clone (JSON) — the codebase opening move | 1 (codebase) |
-| **universal-ctags** | real multi-lang JSON symbol index (the host's BSD ctags is broken for this) | 1 (codebase) |
-| **htmlq** | surgical CSS-selector extraction (pricing cards, feature tables) instead of regex | 1 (website) |
-| **source-map-explorer** | reassemble shipped `.map` → original `src/` tree (the `source-map-reassembly` method) | 1 (bundle) |
-| **spectral** | validate/resolve an OpenAPI/AsyncAPI spec → structured endpoint/security/$ref catalog | 1 (api) |
-| **griffe** | extract a PyPI package's full public API (the `.d.ts`-equivalent for Python) | 1 (packages) |
-| **@arethetypeswrong/cli + typescript + publint** | parse the real `.d.ts`/exports type graph + grade the publish surface (verbatim, not string-mined) | 1 (packages) |
-| **@electron/asar** | extract Electron `app.asar` → JS module tree (the distribution step the playbook names but can't run) | 1 (distribution) |
-| **fd / yq / dasel / monolith / tokei / sevenzip** | gitignore-aware file discovery · YAML/XML/TOML query · RSS/XML→JSON · single-file page archive · LOC cross-check · multi-format extract | several |
-
----
-
-## Tier 2 — install when a run needs them (gated: first-party / passive-care)
-
-The capture lanes. In-bounds **only** for the user's own session / first-party clients / passive sources.
-
-```bash
-brew install mitmproxy websocat grpcurl subfinder tlsx testssl jadx apktool
-npm i -g chrome-remote-interface get-graphql-schema
-# one-time, user-run: expose a CDP port on the user's own Chrome (closes the WS-frame blind spot)
-#   open -na "Google Chrome" --args --remote-debugging-port=9222
-# mobile-only, the user's own device + opt-in:  uv tool install frida-tools
-```
-
-| Tool | Unlocks | Ethics note |
-| --- | --- | --- |
-| **mitmproxy** | decode full HTTPS + WebSocket flows of any **first-party** client (CLI/desktop/mobile/SDK) — wire-capture rung 3 keystone | user installs CA + routes _their own_ client only |
-| **chrome-remote-interface** | raw CDP client → `Network.webSocketFrameReceived`/`Created`, SSE, full bodies — **captures the on-load WS frames the browser MCP's HTTP-only `read_network_requests` drops** | the user's own logged-in session |
-| **websocat** | curl-for-WebSockets: connect/frame/replay/hexdump a realtime endpoint outside the browser | first-party endpoint |
-| **grpcurl** | list services + invoke RPCs via gRPC server reflection → readable op/message catalog | first-party endpoint |
-| **get-graphql-schema** | introspect a first-party GraphQL endpoint → full SDL | first-party endpoint |
-| **subfinder / tlsx / testssl.sh** | passive CT/DNS subdomain breadth + structured TLS posture (vs hand-rolled crt.sh + openssl text) | passive aggregation; respect rate limits |
-| **jadx / apktool** | decompile Android `.apk`/`.dex` → Java + decode `AndroidManifest`/resources (capability map) | public binary, read EULA |
-| **frida-tools** | cert-pinning bypass to capture **the user's own** mobile client's wire | the user's own device + opt-in only |
-
-**The single biggest unlock** for realtime targets: `chrome-remote-interface` + a Chrome `--remote-debugging-port` + `websocat` + `mitmproxy` together give `session`/`wire-capture` a real WebSocket/SSE capture path, instead of a client-side-remount workaround. The browser-MCP `read_network_requests` is HTTP-only — that's _why_ a realtime tap comes back empty until the remount trick.
-
----
-
-## AVOID — do NOT install (crosses the harness bright line)
-
-Out-of-bounds for a spec-writing/competitive-analysis harness — listed so they aren't added by reflex:
-
-- **`nmap`, `masscan`, `naabu`** — active port sweeps / service-version probing of remote hosts.
-- **`nuclei`** — template-based remote vuln scanning.
-- **`sqlmap`** — injection/exploitation.
-- **`amass` (active mode)** — active DNS brute-forcing (its _passive_ mode overlaps subfinder; prefer subfinder).
-- **third-party `mitmproxy`, `frida` on devices the user doesn't own, `socat` as a public relay** — third-party traffic interception is the line this harness will not cross.
-
-These stay absent by design; the recon ethics depend on it.
-
----
-
-## Per-dimension view (gap → top pick)
-
-| Dimension | Biggest capability gap on a bare host | Top addition(s) |
-| --- | --- | --- |
-| **codebase** | no LOC census, no AST search; BSD ctags is broken | scc · ast-grep · universal-ctags (+ fd) |
-| **docs** | no HTML/MDX→md, no JS-render → client-rendered docs come back empty | pandoc · trafilatura · playwright |
-| **packages** | the real `.d.ts`/exports surface is string-mined, not parsed | typescript+attw+publint · griffe (Py) · yq |
-| **api** | can fetch a spec but can't lint/diff/introspect/exercise it | spectral · oasdiff · get-graphql-schema · grpcurl · websocat |
-| **website** | no JS-render, no clean table/text extraction → misses pricing/feature grids | trafilatura · htmlq · playwright |
-| **community** | no clean changelog/blog→text, no RSS→JSON | trafilatura · pandoc · dasel |
-| **session** | **structurally blind to WebSocket/SSE frames** (HTTP-only MCP, no CDP port) | chrome-remote-interface · mitmproxy · websocat |
-| **deployed-client-bundle** | minified/obfuscated blob stays opaque; no source-map reassembly | webcrack · source-map-explorer · ast-grep · gron |
-| **infra-backend-fingerprint** | shallow CT coverage, TLS posture by hand, sub-processor pages scraped raw | subfinder · tlsx · trafilatura (passive only) |
-| **wire-capture** | keystone mitmproxy + all non-browser replay/decode clients absent | mitmproxy · websocat · grpcurl |
-| **distribution-artifacts** | only CRX/Mach-O readable; no Electron/Android decoders | @electron/asar · jadx · apktool · webcrack · sevenzip |
-| **cross-cutting (synthesis)** | no structured diff for the N-way seam reconciliation; no out-of-band WS capture | gron · oasdiff · chrome-remote-interface · yq |
-
----
-
-## Known install caveats
-
-- **`webcrack` needs Node 22.** Its hard dependency `isolated-vm` is a native (node-gyp) addon that fails to build against Node 26 (and `--omit=optional` doesn't help — the dep is non-optional). If it won't install, run it under a Node 22 LTS shim (`npx -y webcrack@latest …` in a Node-22 env), or lean on the already-clean `source-map-explorer` (the preferred `source-map-reassembly` path whenever the SPA ships `.map` files) + `ast-grep`. De-obfuscation of a _no-maps, minified_ bundle is the only gap this leaves.
-- **`chrome-remote-interface` is a library, not a CLI** — it installs as a Node module and is invoked from a script, not a shell binary. Detect it with `npm ls -g chrome-remote-interface`, not `command -v`.
-- **`pipx` is typically absent; use `uv tool install`** for Python CLIs (uv is present) — simpler than adding pipx.
-- **`playwright` is clean** (the user's own headless instance) and unblocks JS-rendered docs/marketing for `docs`+`website` without contending for the gated `session` Chrome.