@nurix/etna 0.15.7 → 0.15.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/etna.js +64 -13
- package/data/harness-index.json +7 -5
- package/data/harnesses/code/CLAUDE.md +3 -7
- package/data/harnesses/code/agents/harness-manager.md +4 -4
- package/data/harnesses/code/agents/maintenance.md +1 -1
- package/data/harnesses/code/harness.json +4 -3
- package/data/harnesses/code/hooks/hooks.json +3 -17
- package/data/harnesses/code/hooks/prompt-recall.js +157 -0
- package/data/harnesses/code/hooks/session-reflect.js +283 -0
- package/data/harnesses/code/skills/distiller/SKILL.md +62 -60
- package/data/harnesses/code/skills/distiller/references/docs-mode.md +21 -21
- package/data/harnesses/code/skills/distiller/references/method.md +41 -41
- package/data/harnesses/code/skills/documentation/references/templates.md +20 -27
- package/data/harnesses/code-fable/CLAUDE.md +138 -0
- package/data/harnesses/code-fable/agents/brainstorm.md +35 -0
- package/data/harnesses/code-fable/agents/change-review.md +249 -0
- package/data/harnesses/code-fable/agents/harness-manager.md +193 -0
- package/data/harnesses/code-fable/agents/maintenance.md +123 -0
- package/data/harnesses/code-fable/agents/performance-review.md +218 -0
- package/data/harnesses/code-fable/agents/spec-mode.md +56 -0
- package/data/harnesses/code-fable/harness.json +7 -0
- package/data/harnesses/code-fable/hooks/hooks.json +27 -0
- package/data/harnesses/code-fable/hooks/lib/redact.js +37 -0
- package/data/harnesses/code-fable/hooks/prompt-recall.js +157 -0
- package/data/harnesses/code-fable/hooks/session-reflect.js +283 -0
- package/data/harnesses/code-fable/rules/general.md +96 -0
- package/data/harnesses/code-fable/rules/nomenclature.md +34 -0
- package/data/harnesses/code-fable/skills/boundary/SKILL.md +262 -0
- package/data/harnesses/code-fable/skills/boundary/references/boundary-taxonomy.md +734 -0
- package/data/harnesses/code-fable/skills/brainstorm/README.md +136 -0
- package/data/harnesses/code-fable/skills/brainstorm/SKILL.md +135 -0
- package/data/harnesses/code-fable/skills/brainstorm/harvest.md +26 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/backlog.md +39 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/build.md +37 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/decisions.md +29 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/doctrine.md +31 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/domain-model.md +25 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/glossary.md +19 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/milestones.md +43 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/non-goals.md +21 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/plan-doc.md +61 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/questions.md +35 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/references.md +28 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/rejected.md +29 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/state.md +8 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/tech.md +23 -0
- package/data/harnesses/code-fable/skills/brainstorm/templates/thesis.md +21 -0
- package/data/harnesses/code-fable/skills/distiller/SKILL.md +219 -0
- package/data/harnesses/code-fable/skills/distiller/references/docs-mode.md +526 -0
- package/data/harnesses/code-fable/skills/distiller/references/method.md +166 -0
- package/data/harnesses/code-fable/skills/documentation/SKILL.md +127 -0
- package/data/harnesses/code-fable/skills/documentation/references/examples.md +193 -0
- package/data/harnesses/code-fable/skills/documentation/references/style-guide.md +288 -0
- package/data/harnesses/code-fable/skills/documentation/references/templates.md +942 -0
- package/data/harnesses/code-fable/skills/workflow-mapper/SKILL.md +430 -0
- package/data/harnesses/code-fable/skills/workflow-mapper/references/patterns.md +214 -0
- package/data/harnesses/{re → discover}/CLAUDE.md +48 -33
- package/data/harnesses/{re → discover}/agents/api.md +3 -3
- package/data/harnesses/{re → discover}/agents/codebase.md +3 -3
- package/data/harnesses/{re → discover}/agents/community.md +3 -3
- package/data/harnesses/{re → discover}/agents/deployed-client-bundle.md +5 -5
- package/data/harnesses/{re → discover}/agents/distribution-artifacts.md +4 -4
- package/data/harnesses/{re → discover}/agents/docs.md +3 -3
- package/data/harnesses/{re → discover}/agents/infra-backend-fingerprint.md +2 -2
- package/data/harnesses/{re → discover}/agents/packages.md +3 -3
- package/data/harnesses/{re → discover}/agents/session.md +19 -11
- package/data/harnesses/{re → discover}/agents/website.md +3 -3
- package/data/harnesses/{re → discover}/agents/wire-capture.md +9 -7
- package/data/harnesses/discover/harness.json +7 -0
- package/data/harnesses/{re/rules/re-discovery.md → discover/rules/discovery.md} +54 -17
- package/data/harnesses/{re/rules/re-evaluation.md → discover/rules/evaluation.md} +12 -1
- package/data/harnesses/{re/rules/re-ingestion.md → discover/rules/ingestion.md} +192 -23
- package/data/harnesses/discover/rules/self-correction.md +177 -0
- package/data/harnesses/discover/rules/tradecraft.md +532 -0
- package/data/harnesses/hr/CLAUDE.md +91 -0
- package/data/harnesses/hr/harness.json +7 -0
- package/data/harnesses/hr/skills/interview-eval/SKILL.md +118 -0
- package/data/harnesses/hr/skills/interview-eval/TEMPLATE.md +124 -0
- package/data/harnesses/hr/skills/interview-eval/extract_session.py +418 -0
- package/data/harnesses/hr/skills/interview-eval/rubric.md +219 -0
- package/data/harnesses/kb-agi/harness.json +1 -0
- package/package.json +1 -1
- package/data/harnesses/code/hooks/post-commit.js +0 -136
- package/data/harnesses/code/hooks/post-tool-failure.js +0 -136
- package/data/harnesses/code/hooks/session-stop.js +0 -207
- package/data/harnesses/re/harness.json +0 -6
|
@@ -1,19 +1,32 @@
|
|
|
1
1
|
# Discovery Reference — Scope Gate · Access Vector · Dimension Catalog
|
|
2
2
|
|
|
3
|
-
The reference knowledge for **Discovery** (`
|
|
4
|
-
Discovery's steps in the order they run:
|
|
3
|
+
The reference knowledge for **Discovery** (`discover` harness, Mode 1). It carries all three of
|
|
4
|
+
Discovery's steps in the order they run (preceded by corpus-root resolution, Step 00 below):
|
|
5
5
|
|
|
6
6
|
- **Part A — Scope-feasibility gate** (Step 0): resolve the target's identity and emit a verdict
|
|
7
7
|
so the run never wastes effort on an infeasible target.
|
|
8
8
|
- **Part B — Access vector** (Step 1): grade the four-axis capability vector that decides which
|
|
9
9
|
dimensions are reachable and which method each uses.
|
|
10
10
|
- **Part C — Dimension catalog** (Step 2): the canonical list of dimensions, their detection
|
|
11
|
-
signals,
|
|
11
|
+
signals, collection strategy, and owning agent.
|
|
12
12
|
|
|
13
13
|
`ingestion` also reads **Part C** to dispatch the right agent with the right method.
|
|
14
14
|
|
|
15
15
|
---
|
|
16
16
|
|
|
17
|
+
# Step 00 — Resolve the corpus root (before the gate)
|
|
18
|
+
|
|
19
|
+
Before Part A, decide **where this run writes** — the harness picks intelligently, it does not hardcode `research/`:
|
|
20
|
+
|
|
21
|
+
1. **Reuse an existing discovery corpus** if the project has one — search for any folder containing a prior run's `*/00-scope-verdict.md` (commonly `research/`, but also `recon/`, `discovery/`, `teardowns/`, `analysis/`, or a folder the user names).
|
|
22
|
+
2. If several exist, prefer the one the user points at, else the most recently written.
|
|
23
|
+
3. If none exists and the user hasn't named one, default to `research/` at the repo root.
|
|
24
|
+
4. **Confirm the chosen root in one line** before writing `00-scope-verdict.md`.
|
|
25
|
+
|
|
26
|
+
All `research/<target>/` paths in the harness are the *default form*, relative to the resolved `<corpus>/`.
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
17
30
|
# Part A — Scope-feasibility gate (Step 0)
|
|
18
31
|
|
|
19
32
|
Run this **before** any dimension probing. It resolves the target's identity and emits one of four
|
|
@@ -152,6 +165,14 @@ dimensions). Rules:
|
|
|
152
165
|
|
|
153
166
|
Discovery records the resolved vector in the `access_grade` frontmatter of `00-recon-plan.md`.
|
|
154
167
|
|
|
168
|
+
> **Hypotheses are not facts (seed them tagged, never echo them).** Any *derived inference* Discovery
|
|
169
|
+
> makes beyond the four graded axes — e.g. "this `localStorage` key implies a GraphQL backend", "this
|
|
170
|
+
> host is the realtime endpoint" — is recorded in `00-recon-plan.md` **as a labelled hypothesis**, never
|
|
171
|
+
> as a plan fact, and is tagged `verify:may-be-stale` when seeded to a collector (CLAUDE.md Mode 2) so
|
|
172
|
+
> Ingestion **re-derives** rather than echoes it. A wrong inference seeded as framing propagates: a real
|
|
173
|
+
> run's `apolloAnonId ⇒ GraphQL` guess was repeated (hedged) by two captures before the `session` dimension
|
|
174
|
+
> refuted it (the backend was RSC + Server Actions; the key was an Apollo.io tracker).
|
|
175
|
+
|
|
155
176
|
---
|
|
156
177
|
|
|
157
178
|
# Part C — The dimension catalog (Step 2)
|
|
@@ -164,7 +185,7 @@ Discovery records the resolved vector in the `access_grade` frontmatter of `00-r
|
|
|
164
185
|
| 4 | **api** | `api` | runtime/presence | OpenAPI/GraphQL → endpoint catalog (+ **MCP/tool-catalog fold-in**) |
|
|
165
186
|
| 5 | **website** | `website` | presence | crawl public pages → features, pricing, positioning |
|
|
166
187
|
| 6 | **community** | `community` | presence | first-party changelog/issues/discord themes |
|
|
167
|
-
| 7 | **session** | `session` | auth:have-login | delegate to the session playbook (§9 of `
|
|
188
|
+
| 7 | **session** | `session` | auth:have-login | delegate to the session playbook (§9 of `ingestion`: 3-pass, P0–P6) |
|
|
168
189
|
| 8 | **deployed-client-bundle** | `deployed-client-bundle` | runtime:reachable | unauth SPA: route table, API-path catalog, flags, env, source maps |
|
|
169
190
|
| 9 | **infra-backend-fingerprint** | `infra-backend-fingerprint` | runtime/presence | DNS, cert-transparency, CDN/cloud, headers (+ **sub-processor/trust fold-in**) |
|
|
170
191
|
| 10 | **wire-capture** | `wire-capture` | runtime (source:none) | the 3-rung ladder for non-browser/no-source clients (gated) |
|
|
@@ -177,7 +198,7 @@ Discovery records the resolved vector in the `access_grade` frontmatter of `00-r
|
|
|
177
198
|
> issues/roadmap/changelog/forum — feedback funnelled to a gated channel), flag `external-reputation`
|
|
178
199
|
> as **recommended-for-this-run** in `00-recon-plan.md`, so the user-sentiment blind spot is a surfaced
|
|
179
200
|
> choice rather than a silent gap (the `community` user-pain claims are then capped at tentative —
|
|
180
|
-
>
|
|
201
|
+
> evaluation closed-feedback cap).
|
|
181
202
|
|
|
182
203
|
An **absent** dimension is a finding — record it with its evidence; never silently drop it.
|
|
183
204
|
|
|
@@ -188,24 +209,39 @@ See git history / the source playbook for the full strategy text; in brief:
|
|
|
188
209
|
packages + internal dep hierarchy, return a per-package work-list for the main-session fan-out.
|
|
189
210
|
- **docs** — prefer in-repo docs, then `llms.txt`/`llms-full.txt`, then crawl; organize by section.
|
|
190
211
|
- **packages** — per public package: registry metadata, `exports` map + `.d.ts` type surface, health.
|
|
212
|
+
**Registry-ownership cheap-test (grade ownership before calling an SDK "partial/unverified").** Read
|
|
213
|
+
the registry `author`/`maintainers`/`summary`/`project_urls`/`Homepage` fields and check the
|
|
214
|
+
dependency set for a generator fingerprint (Stainless: `httpx`/`pydantic`/`anyio`/`distro`/`sniffio`
|
|
215
|
+
+ in-file "generated by Stainless" headers; Speakeasy; OpenAPI-Generator). An `author`/`summary` that
|
|
216
|
+
says "official … library for the X API" + a vendor-domain `project_urls` confirms **official**
|
|
217
|
+
ownership at **high** confidence even when the *source* repo is private — the published tarball IS the
|
|
218
|
+
public-API mirror (a high-confidence `registry-metadata` anchor), so it is **not** "partial." Tag a
|
|
219
|
+
genuinely third-party/community package (different author/scope) as such, but don't down-grade an
|
|
220
|
+
official-but-private-source SDK to "unverified."
|
|
191
221
|
- **api** — fetch the spec verbatim, build an endpoint catalog (auth, envelope, pagination, webhooks).
|
|
192
222
|
**Grade the method from a RETRIEVED artifact, never a label.** Before grading `openapi-verbatim`,
|
|
193
223
|
actually fetch the candidate spec URL and confirm a **parseable JSON/YAML body** — a docs page merely
|
|
194
224
|
*titled* "OpenAPI" with no served `spec.json` (or a `/swagger` that 403s) is `docs-reconstructed`
|
|
195
225
|
(capped medium), and Discovery must pre-grade it that way so the plan matches reality. **A
|
|
196
226
|
sibling-declared host/op/route count is a coverage checklist (a floor), never a ceiling** — every
|
|
197
|
-
|
|
227
|
+
collector expands from sibling-mined hosts/routes and records the predicted-vs-found delta.
|
|
198
228
|
**NEW fold-in:** also probe MCP/tool surfaces — `/.well-known/mcp`, an MCP server manifest, a
|
|
199
229
|
function-calling tool schema, an LLM "actions" file — and write `raw/tool-catalog.md` (an MCP tool
|
|
200
|
-
list *is* an API catalog; it lives here, not in its own dimension).
|
|
230
|
+
list *is* an API catalog; it lives here, not in its own dimension). **Unauth MCP-probe recipe (when
|
|
231
|
+
the tool catalog itself is auth-gated):** a `GET /mcp` → `405` body often names the transport
|
|
232
|
+
(e.g. "stateless; only POST is supported" = streamable-HTTP); `GET /sse` presence/absence tells
|
|
233
|
+
legacy-SSE vs streamable; and `/.well-known/oauth-authorization-server` + `/.well-known/oauth-protected-resource/{name}`
|
|
234
|
+
reveal the **OAuth/PKCE model + issuer** (e.g. a `jwks_uri` on the auth vendor names Clerk/Auth0/etc.)
|
|
235
|
+
— all without a token. Capture transport + auth + issuer from these, reconstruct the tool list from
|
|
236
|
+
docs/SDK, and mark the verbatim `tools/list` **partial** (OAuth-gated).
|
|
201
237
|
- **website** — crawl home/features/pricing/customers/integrations; extract positioning, pricing
|
|
202
238
|
matrix, target market.
|
|
203
239
|
- **community** — *first-party only*: changelog cadence + direction, issue/discussion themes, blog.
|
|
204
240
|
- **session** — the deepest, opt-in, user-gated dimension; delegates to the session playbook in
|
|
205
|
-
`
|
|
241
|
+
`ingestion` §9 (3-pass, P0–P6, captures/raw/curls). For any `auth:have-login`/`auth:claimed`
|
|
206
242
|
target, Discovery **seeds the Pass-2 decision** (`pass2: offered-pending | declined-by-user |
|
|
207
243
|
deferred`) into `00-recon-plan.md`, so a Pass-1-only (read-only) session is a *recorded choice*, not a
|
|
208
|
-
silent omission of the write surface (
|
|
244
|
+
silent omission of the write surface (ingestion §9.2).
|
|
209
245
|
|
|
210
246
|
## 8 — deployed-client-bundle — `deployed-client-bundle`
|
|
211
247
|
|
|
@@ -220,12 +256,12 @@ bootstrap-config paths (`/config`, `/backend/config/*`, `/env.json`, a runtime-c
|
|
|
220
256
|
200 here is both a method pre-grade and a possible unauth config/secret leak (a finding).
|
|
221
257
|
|
|
222
258
|
**Strategy:** fetch the main bundle(s); string-mine the **route table**, the **API-path catalog**
|
|
223
|
-
(→ append to the shared `dimensions/_shared/api-path-catalog.md`, `source: bundle` —
|
|
259
|
+
(→ append to the shared `dimensions/_shared/api-path-catalog.md`, `source: bundle` — ingestion §6),
|
|
224
260
|
**feature flags** (→ shared `dimensions/_shared/feature-flags.md`), embedded **env config** (`VITE_*`,
|
|
225
261
|
`NEXT_PUBLIC_*`, public keys — field-names only, redact values), and reassemble **source maps** if
|
|
226
262
|
shipped (`method: source-map-reassembly`, confidence high). Digest huge bundles, don't paste. This
|
|
227
263
|
dimension **owns** the app-API bundle mine — other dimensions cite its catalog by path, never re-mine
|
|
228
|
-
the same bundle (
|
|
264
|
+
the same bundle (ingestion §6).
|
|
229
265
|
|
|
230
266
|
**Artifacts:** `_summary.md` + `raw/{route-table.md, env-config.md, bundle-map.md}` + appends to the
|
|
231
267
|
shared `dimensions/_shared/{api-path-catalog.md, feature-flags.md}`.
|
|
@@ -253,15 +289,16 @@ what hosts it *might* call; the capture says what it *actually* calls.
|
|
|
253
289
|
**Detection:** a non-browser client exists (from distribution-artifacts) OR an unauth SPA makes
|
|
254
290
|
structured XHRs worth a tap, AND the appropriate ethics rung is authorized.
|
|
255
291
|
|
|
256
|
-
**Strategy:** the **3-rung ladder** (lowest privilege first) — see `
|
|
292
|
+
**Strategy:** the **3-rung ladder** (lowest privilege first) — see `ingestion` §7.1:
|
|
257
293
|
1. browser network tap (unauth, default) → `wire-tap-browser`;
|
|
258
294
|
2. HAR export → `wire-har`;
|
|
259
|
-
3. mitmproxy (**gated, opt-in, user-run, first-party traffic only**) → `wire-mitmproxy
|
|
295
|
+
3. mitmproxy (**gated, opt-in, user-run, first-party traffic only**) → `wire-mitmproxy`;
|
|
296
|
+
4. **first-party SDK-harness** (when the product publishes an official SDK + the *user* can provision keys) — scaffold + instrument a throw-away client you own (`document_start` taps, full wire) → `sdk-harness` (`ingestion` §7.1). The deepest lane for a `runtime:reachable + source:none` product whose SDK is the only public mirror of the runtime; gated like rung 3.
|
|
260
297
|
|
|
261
298
|
**Fold note (common path):** when there is no separate non-browser client, wire-capture **folds into
|
|
262
|
-
`session`** (which owns the browser singleton), **never into a curl/WebFetch
|
|
263
|
-
|
|
264
|
-
|
|
299
|
+
`session`** (which owns the browser singleton), **never into a curl/WebFetch collector** (a non-browser
|
|
300
|
+
collector cannot run rung-1's in-page tap, and routing it there yields zero `source: wire` rows —
|
|
301
|
+
ingestion §7.1). The folded rung-1 *is* the interceptor `session` installs; its observed paths
|
|
265
302
|
append to the shared `dimensions/_shared/api-path-catalog.md` as `source: wire`. A folded wire-capture
|
|
266
303
|
still leaves a one-paragraph `_summary.md` (`status: folded`, naming the fold target) — never a silent
|
|
267
304
|
drop.
|
|
@@ -283,7 +320,7 @@ capability map). **Electron** — `npx asar extract app.asar` → map JS modules
|
|
|
283
320
|
route/API/flag catalog from the desktop client (often richer than web). **mobile** — `Info.plist`/
|
|
284
321
|
`AndroidManifest` hosts, ATS exceptions, embedded-SDK list from strings (**redact any embedded
|
|
285
322
|
keys**). **CLI** — help text/subcommands = an API catalog. Read the artifact's EULA first (see
|
|
286
|
-
`
|
|
323
|
+
`ingestion` §7.2); never silently decompile a restrictively-licensed binary.
|
|
287
324
|
|
|
288
325
|
**Artifacts:** `_summary.md` + `raw/{extension-manifest.md, electron-asar-map.md, mobile-strings.md, cli-catalog.md}` (whichever apply) + appends to shared `dimensions/_shared/api-path-catalog.md` (source: distribution).
|
|
289
326
|
|
|
@@ -27,13 +27,19 @@ agreeing is still one source. A bundle-declared route + a session-observed route
|
|
|
27
27
|
one spec / one binary) are **one source** — never a band-bump — even when they report different
|
|
28
28
|
counts; the divergent counts are a **reported range, not a flagged conflict** (`deployed-client-bundle`
|
|
29
29
|
278 vs `website` 361 over one bundle is one finding with a range, not corroboration and not a conflict).
|
|
30
|
+
**OSS distribution clients are an independent corroboration lane.** An API-surface claim resting on a
|
|
31
|
+
single *third-party* wrapper (a community MCP server, an unofficial SDK) is tentative — but a
|
|
32
|
+
**first-party open-source integration client** (`distribution-artifacts`: a Unity/Figma/CLI plugin or
|
|
33
|
+
published SDK source) that calls the same routes is **independent of both marketing and the third-party
|
|
34
|
+
wrapper**, and corroborates it → promotes. (A legacy/granular route family seen only in a third-party
|
|
35
|
+
MCP becomes a real finding once the official OSS client calls it too.)
|
|
30
36
|
|
|
31
37
|
**Closed-feedback confidence cap.** When the `community` dimension finds **no first-party feedback
|
|
32
38
|
loop** (no public issues / roadmap / changelog / forum — feedback funnelled to a gated channel), its
|
|
33
39
|
user-pain / most-requested-feature claims are **inference-from-changelog only → capped at
|
|
34
40
|
`low`/tentative**, never promoted on changelog frequency alone. The surface that *would* corroborate
|
|
35
41
|
user sentiment is `external-reputation` (a deferred dimension) — flag it as recommended-for-this-run
|
|
36
|
-
when this cap fires (see
|
|
42
|
+
when this cap fires (see discovery Part C deferred-dimension trigger).
|
|
37
43
|
|
|
38
44
|
**Write-side-unobserved cap.** A `session` dimension carrying `write_side_observed: false` (a
|
|
39
45
|
Pass-1-only run) has observed feature *existence/gating* but **not** feature *execution behavior*. Any
|
|
@@ -129,3 +135,8 @@ prose, not via a separate `claims.yaml`.)
|
|
|
129
135
|
4. Tag every claim with provenance; weight per the rules above; flag every disagreement; record
|
|
130
136
|
every gap. A confident doc built on one low-confidence dimension is worse than an honest
|
|
131
137
|
"single-source, unverified."
|
|
138
|
+
|
|
139
|
+
> **Classifying a raw observation** — to turn a storage-key prefix, an auth tell, an envelope shape,
|
|
140
|
+
> or a realtime frame into a *named* stack finding, use the fingerprint tables in
|
|
141
|
+
> `tradecraft.md` (storage→stack, auth-provider, API-envelope, realtime-trap). They name what
|
|
142
|
+
> a finding *is*; this file governs how confidently to *state* it.
|
|
@@ -1,15 +1,15 @@
|
|
|
1
|
-
# Ingestion Contract —
|
|
1
|
+
# Ingestion Contract — Capture Shape · Provenance · Ethics · Session Playbook
|
|
2
2
|
|
|
3
|
-
The reference knowledge for **Ingestion** (`
|
|
3
|
+
The reference knowledge for **Ingestion** (`discover` harness, Mode 2). Every dimension capture obeys the
|
|
4
4
|
contract in §1–8 so (a) the **evaluation** mode can rely on a stable shape and weight each claim by
|
|
5
|
-
*how it was obtained*, and (b) a future agent can reproduce or extend any
|
|
5
|
+
*how it was obtained*, and (b) a future agent can reproduce or extend any capture. §9 carries the
|
|
6
6
|
deepest, gated dimension's method — the logged-in browser-session playbook — which the `session`
|
|
7
7
|
agent follows.
|
|
8
8
|
|
|
9
9
|
This contract generalizes the source playbook's three-format contract (`captures/*.md` narrative +
|
|
10
10
|
`raw/*.json` verbatim + `curls.md`) to *any* dimension.
|
|
11
11
|
|
|
12
|
-
> **Contract style = "lighter".** Provenance ("how it was
|
|
12
|
+
> **Contract style = "lighter".** Provenance ("how it was captured") lives in the
|
|
13
13
|
> **`_summary.md` frontmatter**, not a separate `manifest.yaml`. There is no parallel YAML
|
|
14
14
|
> machine-spine — everything stays markdown-with-frontmatter. Evaluation reads the frontmatter
|
|
15
15
|
> to weight claims; self-correction reads it to detect under-delivery.
|
|
@@ -50,7 +50,7 @@ Two layout rules the seam and the "absence is a finding" principle depend on:
|
|
|
50
50
|
dimension: <codebase|docs|packages|api|website|community|session|deployed-client-bundle|infra-backend-fingerprint|wire-capture|distribution-artifacts>
|
|
51
51
|
target: <slug>
|
|
52
52
|
status: complete | partial | blocked
|
|
53
|
-
# --- provenance (the "how it was
|
|
53
|
+
# --- provenance (the "how it was captured" layer evaluation weights on) ---
|
|
54
54
|
access_grade_used: <source:full | runtime:reachable | auth:have-login | presence:rich | ...>
|
|
55
55
|
method: <the ACTUAL extraction method — see §3 vocabulary>
|
|
56
56
|
completeness_pct: <0-100, honest self-estimate of reachable surface captured>
|
|
@@ -60,7 +60,7 @@ sources: [<urls / paths actually consulted>]
|
|
|
60
60
|
gaps: [<structured — each is a thing evaluation must NOT treat as covered>]
|
|
61
61
|
---
|
|
62
62
|
|
|
63
|
-
# <Target> — <Dimension>
|
|
63
|
+
# <Target> — <Dimension> capture
|
|
64
64
|
|
|
65
65
|
## Method
|
|
66
66
|
Exactly what was done, which tools, what was skipped and why. (The prose behind the frontmatter `method`.)
|
|
@@ -88,7 +88,7 @@ Keep `status` and `confidence` honest — they drive the entire evaluation weigh
|
|
|
88
88
|
`method` is one of: `clone-and-map`, `source-map-reassembly`, `registry-metadata`,
|
|
89
89
|
`openapi-verbatim`, `graphql-introspection`, `crawl-clip`, `docs-reconstructed`,
|
|
90
90
|
`bundle-string-mine`, `dns-ct-fingerprint`, `wire-tap-browser`, `wire-har`, `wire-mitmproxy`,
|
|
91
|
-
`binary-extract` (asar/apk/crx), `session-playbook`, `inferred`.
|
|
91
|
+
`binary-extract` (asar/apk/crx), `session-playbook`, `sdk-harness`, `inferred`.
|
|
92
92
|
|
|
93
93
|
**Confidence is a function of method** (evaluation relies on this mapping):
|
|
94
94
|
- **high** — verbatim machine artifacts: `openapi-verbatim`, `graphql-introspection`, `registry-metadata`, `clone-and-map`, `binary-extract`.
|
|
@@ -133,13 +133,13 @@ flags**. To keep them consumable instead of triplicated **N times** (a real run
|
|
|
133
133
|
|
|
134
134
|
- **One physical file, one canonical location, append-only.** The seam files live at
|
|
135
135
|
`dimensions/_shared/api-path-catalog.md` and `dimensions/_shared/feature-flags.md` — **Ingestion
|
|
136
|
-
seeds the two empty files (with a header) once, before the
|
|
136
|
+
seeds the two empty files (with a header) once, before the collector fan-out.** Every contributing
|
|
137
137
|
dimension **appends a `source:`-tagged section** (`source: bundle | session | wire | distribution`)
|
|
138
|
-
and **never creates its own copy**; each
|
|
138
|
+
and **never creates its own copy**; each collector's `_summary.md` Artifacts list **links** the shared
|
|
139
139
|
file rather than duplicating it. **Shared-seam allowlist:** these two filenames are the *only*
|
|
140
140
|
shared/single-physical artifacts — every other `raw/*` is dimension-private.
|
|
141
141
|
- Evaluation owns the **API-path diff** (an *adaptive N-way* diff over whichever runtime sources were
|
|
142
|
-
actually run — see
|
|
142
|
+
actually run — see evaluation template 3) in `data-model-api-surface.md`. A path that appears in
|
|
143
143
|
`bundle` but never in `session`/`wire` is a dormant/unshipped route — a finding, not noise.
|
|
144
144
|
|
|
145
145
|
> **Same artifact ⇒ ONE source (never independent corroboration).** Two dimensions that string-mine
|
|
@@ -155,26 +155,135 @@ flags**. To keep them consumable instead of triplicated **N times** (a real run
|
|
|
155
155
|
> host, and operation set. Always diff the two: operations the app uses but the public API omits, **and**
|
|
156
156
|
> public operations the app never calls, are both findings. The published surface is the integration
|
|
157
157
|
> story; the runtime surface is the real architecture. When a published API exists, Evaluation emits
|
|
158
|
-
> this split as a *named* output, not inline prose (
|
|
158
|
+
> this split as a *named* output, not inline prose (evaluation template 3).
|
|
159
159
|
|
|
160
160
|
---
|
|
161
161
|
|
|
162
162
|
## 7. Ethics & redaction (non-negotiable — binds EVERY dimension)
|
|
163
163
|
|
|
164
|
-
|
|
164
|
+
Violating them voids the work. Rules 2–3 are enforced by the **runnable redaction layer in §7.0** —
|
|
165
|
+
install it inside the tap wrapper so no secret can reach `raw/` in the first place.
|
|
165
166
|
|
|
166
167
|
1. **Never enter credentials.** The user signs into any logged-in surface beforehand.
|
|
167
168
|
2. **Storage: shape, not values.** No raw cookies/tokens/JWT-signatures. Decode JWT *structure* with identity claims (`sub`,`email`,`phone`,`user_id`,`uid`,`sid`) redacted.
|
|
168
|
-
3. **Never save secrets.** Before any `raw/` write, redact every credential-shaped field → `<REDACTED:...>` + log in `_meta.redactions`. Field patterns: `api_key`,`*_secret`,`client_secret`,`webhook_secret`,`signing_*`,`password`,`private_key`,`access_token`/`refresh_token` in bodies, vendor-named secrets. Value patterns: `sk_live_…`/`pk_live_…` (Stripe), `sk-…` (OpenAI), `xox[bp]-…` (Slack), `AKIA…` (AWS), `gh[ps]_…` (GitHub), `AIza…` (Google), `eyJ…` (JWT), opaque base64 ≥32 chars in a credential field. Over-redact when unsure.
|
|
169
|
+
3. **Never save secrets.** Before any `raw/` write, redact every credential-shaped field → `<REDACTED:...>` + log in `_meta.redactions`. Field patterns: `api_key`,`*_secret`,`client_secret`,`webhook_secret`,`signing_*`,`password`,`private_key`,`access_token`/`refresh_token` in bodies, vendor-named secrets. Value patterns: `sk_live_…`/`pk_live_…` (Stripe), `sk-…` (OpenAI), `xox[bp]-…` (Slack), `AKIA…` (AWS), `gh[ps]_…` (GitHub), `AIza…` (Google), `eyJ…` (JWT), opaque base64 ≥32 chars in a credential field. Over-redact when unsure. **The executable scrubbers that enforce this rule are in §7.0 below** (`SCRUB` / `redactSecrets` / `ULTRA`); the full keyboard layer lives in `tradecraft.md`.
|
|
169
170
|
4. **Confirm before state changes** — anything that creates state, costs money, or is visible to others (even a draft toggle). State the exact action + estimated cost.
|
|
170
171
|
5. **No destructive actions.** Document the absence; move on.
|
|
171
172
|
6. **No bypassing safety harnesses** — redact, never evade.
|
|
172
173
|
7. **No exfiltration.** Captures stay local under `research/<target>/`.
|
|
173
174
|
8. **Respect TOS & robots.** For spec-writing / integration / competitive analysis — not credential theft, rate-limit bypass, or scraping.
|
|
174
|
-
9. **One browser-driving dimension at a time.** Chrome MCP is a shared singleton. Only the gated `session`/`wire-capture` dimensions drive the browser; parallel non-browser
|
|
175
|
-
10. **Verify an absence before asserting it.** A "host is NXDOMAIN / dimension is absent / endpoint 404s everywhere" claim that would *contradict another dimension* must be re-checked by a **second method** (alternate resolver, direct HTTP probe, second source) before it is written as fact. A single-resolver/single-probe negative is recorded as *tentative*, never as a cross-dimension directive. (An HTTP 4xx/5xx means the host **resolves** — present, not absent.)
|
|
175
|
+
9. **One browser-driving dimension at a time.** Chrome MCP is a shared singleton. Only the gated `session`/`wire-capture` dimensions drive the browser; parallel non-browser collectors use WebFetch/curl, or — if a page genuinely needs a real browser — create their **own** tab and never touch the session tab. Never run two browser-driving collectors concurrently.
|
|
176
|
+
10. **Verify an absence before asserting it.** A "host is NXDOMAIN / dimension is absent / endpoint 404s everywhere" claim that would *contradict another dimension* must be re-checked by a **second method** (alternate resolver, direct HTTP probe, second source) before it is written as fact. A single-resolver/single-probe negative is recorded as *tentative*, never as a cross-dimension directive. (An HTTP 4xx/5xx means the host **resolves** — present, not absent.) **Feedback-channel / changelog absence (`community`): a 404 on the marketing host's `/changelog` is NOT proof of "no public feedback loop"** — before asserting it, probe the common alternate homes: `feedback.`/`changelog.`/`updates.`/`roadmap.` subdomains, hosted boards (FeatureOS/Canny/Featurebase), a status-page "announcements" tab, and `<link rel="alternate" type="application/rss+xml">` autodiscovery on the docs/blog. Only after those miss is the closed-feedback cap (evaluation) warranted; a wrong-host 404 alone leaves it *tentative*.
|
|
176
177
|
11. **Bounded live-token use (read-only, in-page only).** To fill a Pass-1 capture gap you MAY read the live token from its store and replay a **read-only** request with it — *only* if the token is used **in-page / in-process only**, is **never written to `raw/`, never returned, never logged**, and the replay changes no state. This is the single permitted touch of a live credential; rules 2–3 (shape-not-values, never-save-secrets) still bind everything written to disk.
|
|
177
178
|
|
|
179
|
+
### 7.0 Redaction — runnable (the executable layer behind rule 3)
|
|
180
|
+
|
|
181
|
+
These are **executable guardrails**, not prose policy. Install them **inside the tap/fetch wrapper** so every captured string is scrubbed *before it reaches the harness or disk* — credentials never reach `raw/`, and the browser MCP's content-block (`[BLOCKED: Cookie/query string data]`) never fires on a legitimate capture. Run `SCRUB(value)` on every string returned from a `javascript_tool` capture; run `redactSecrets(body, redactions)` on every object before any `Write` of `raw/*.json`; fall back to `ULTRA` only when the harness still blocks output.
|
|
182
|
+
|
|
183
|
+
#### 1. `SCRUB` — the value-shape scrubber (run on every captured string)
|
|
184
|
+
|
|
185
|
+
Combine into one `SCRUB` function and apply to every string returned from `javascript_tool`. It neutralises PII + provider-credential value shapes regardless of field name (catches credentials embedded in string concatenations like a deep-link URL or a log line):
|
|
186
|
+
|
|
187
|
+
```js
|
|
188
|
+
const SCRUB = (s) => String(s||'')
|
|
189
|
+
.replace(/eyJ[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+/g, '<jwt>')
|
|
190
|
+
.replace(/[A-Za-z0-9+/]{40,}={0,2}/g, '<b64>')
|
|
191
|
+
.replace(/data:image\/[^"]+/g, '<data-image>')
|
|
192
|
+
.replace(/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g, '<email>')
|
|
193
|
+
.replace(/[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}/g, '<uuid>')
|
|
194
|
+
.replace(/[a-f0-9]{32,}/g, '<hex>')
|
|
195
|
+
.replace(/(user|org|ins|sess|cli)_[A-Za-z0-9]{20,}/g, '.$1id.') // Clerk-style IDs
|
|
196
|
+
.replace(/\+91\d{10}/g, '<phone-in>') // Indian phone numbers
|
|
197
|
+
.replace(/\+1\d{10}/g, '<phone-us>') // US phone numbers
|
|
198
|
+
.replace(/\b\d{10,15}\b/g, '<phone>') // generic numeric phone
|
|
199
|
+
.replace(/[?&]([a-z_]+)=([^&\s"]+)/gi, (_,k,v) => '?'+k+'=<v>');
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
The regex table behind `SCRUB` (PII value shapes + known provider-key prefixes), so a new run can extend it:
|
|
203
|
+
|
|
204
|
+
| Pattern | Replacement | Catches |
|
|
205
|
+
|---------|-------------|---------|
|
|
206
|
+
| `eyJ[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+\.[A-Za-z0-9_\-]+` | `<jwt>` | JWT tokens (3-segment base64) |
|
|
207
|
+
| `[A-Za-z0-9+/]{40,}={0,2}` | `<b64>` | Long base64 |
|
|
208
|
+
| `data:image/[^"]+` | `<data-image>` | Inline image data URIs |
|
|
209
|
+
| `[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}` | `<uuid>` | UUIDs |
|
|
210
|
+
| `[a-f0-9]{32,}` | `<hex>` | Long hex strings (hashes) |
|
|
211
|
+
| `(user\|org\|ins\|sess\|cli)_[A-Za-z0-9]{20,}` | `.<group>id.` | Clerk-style IDs |
|
|
212
|
+
| `\+91\d{10}` | `<phone-in>` | Indian phone numbers |
|
|
213
|
+
| `\+1\d{10}` | `<phone-us>` | US phone numbers |
|
|
214
|
+
| `\b\d{10,15}\b` | `<phone>` | Generic numeric phone |
|
|
215
|
+
| `\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b` | `<email>` | Email addresses |
|
|
216
|
+
| `[?&]([a-z_]+)=([^&\s"]+)` | `?$1=<v>` | Query string values |
|
|
217
|
+
| `sk_(live\|test)_[A-Za-z0-9]{16,}` | `<REDACTED:STRIPE_SK>` | Stripe secret key |
|
|
218
|
+
| `(rk\|pk)_(live\|test)_[A-Za-z0-9]{16,}` | `<REDACTED:STRIPE_KEY>` | Stripe restricted / publishable key |
|
|
219
|
+
| `sk-(proj-)?[A-Za-z0-9_\-]{20,}` | `<REDACTED:OPENAI>` | OpenAI API key (incl. project-scoped) |
|
|
220
|
+
| `xox[bpoars]-[A-Za-z0-9-]{10,}` | `<REDACTED:SLACK>` | Slack token |
|
|
221
|
+
| `\b(AKIA\|ASIA)[A-Z0-9]{16}\b` | `<REDACTED:AWS_KEY_ID>` | AWS access key ID (long-lived + temporary) |
|
|
222
|
+
| `gh[pousr]_[A-Za-z0-9]{30,}` | `<REDACTED:GITHUB>` | GitHub PAT / OAuth / refresh / server token |
|
|
223
|
+
| `AIza[0-9A-Za-z_\-]{35}` | `<REDACTED:GOOGLE>` | Google API key |
|
|
224
|
+
|
|
225
|
+
#### 2. `redactSecrets` — the field-name walker (run before every `raw/*.json` write)
|
|
226
|
+
|
|
227
|
+
`SCRUB` catches known value *shapes*; some providers use opaque base64 tokens with **no recognisable prefix**, so you also need a recursive walker that redacts by **key name** regardless of value shape. Run both — `SCRUB` on captured strings, `redactSecrets` on the parsed object before any `Write`:
|
|
228
|
+
|
|
229
|
+
```js
|
|
230
|
+
const SECRET_KEY_RE = /(api[_-]?key|apikey|secret|password|passphrase|pwd|client[_-]?secret|webhook[_-]?secret|signing[_-]?(?:key|secret)|private[_-]?key|service[_-]?account[_-]?key|access[_-]?token|refresh[_-]?token|bearer[_-]?token|auth[_-]?token)$/i;
|
|
231
|
+
|
|
232
|
+
function redactSecrets(obj, redactions) {
|
|
233
|
+
if (obj === null || typeof obj !== 'object') return obj;
|
|
234
|
+
if (Array.isArray(obj)) return obj.map(v => redactSecrets(v, redactions));
|
|
235
|
+
const out = {};
|
|
236
|
+
for (const [k, v] of Object.entries(obj)) {
|
|
237
|
+
if (typeof v === 'string' && SECRET_KEY_RE.test(k) && v.length > 0) {
|
|
238
|
+
redactions.push(`${k} → <REDACTED:SECRET> (was ${v.length} chars)`);
|
|
239
|
+
out[k] = '<REDACTED:SECRET>';
|
|
240
|
+
} else {
|
|
241
|
+
out[k] = redactSecrets(v, redactions);
|
|
242
|
+
}
|
|
243
|
+
}
|
|
244
|
+
return out;
|
|
245
|
+
}
|
|
246
|
+
|
|
247
|
+
// Use:
|
|
248
|
+
const redactions = [];
|
|
249
|
+
const safeBody = redactSecrets(rawBody, redactions);
|
|
250
|
+
// Then write { _meta: { ..., redactions }, body: safeBody, _decoded_fields: {...} }
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
At run time, append the **current session's** own identifiers (your user UUID, email, display name,
|
|
254
|
+
org slug) as explicit `.replace(...)` lines in a final pass over the serialized output — but **never
|
|
255
|
+
commit those literal values into this file.** The generic UUID / email / phone / Clerk-ID patterns
|
|
256
|
+
above already neutralise them; the explicit pass is a belt-and-suspenders step that stays local to the
|
|
257
|
+
run.
|
|
258
|
+
|
|
259
|
+
For a logged-in session, also redact: the user's display name, the user's avatar URL, any organisation slug (`org_slug` claim) the user might want hidden, and phone numbers at any masking level. Over-redaction is cheap; a leaked partner API key is not.
|
|
260
|
+
|
|
261
|
+
#### 3. `ULTRA` — the last-resort scrubber (when the harness still blocks output)
|
|
262
|
+
|
|
263
|
+
When `SCRUB` isn't enough (e.g. the body has agent-generated content with patterns that match the blocklist), drop to a regex that strips everything except readable ASCII:
|
|
264
|
+
|
|
265
|
+
```js
|
|
266
|
+
const ULTRA = (s) => String(s||'').replace(/[^a-zA-Z0-9 _.,:!?\-/]/g,'.');
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
You lose JSON braces and special characters, but you keep the **structural information** (field names, enum values, lengths) — useful for getting any output at all when the body is irretrievably full of blocked patterns.
|
|
270
|
+
|
|
271
|
+
#### 4. PII / value-shape coverage (what these patterns neutralise)
|
|
272
|
+
|
|
273
|
+
- **JWTs** — 3-segment base64 `eyJ…` anywhere (body fields, not just `Authorization`).
|
|
274
|
+
- **Phone numbers** — Indian `+91` (10 digits), US `+1` (10 digits), and generic 10–15-digit numeric.
|
|
275
|
+
- **Emails** — `local@domain.tld` (and the user's own specific address explicitly).
|
|
276
|
+
- **UUIDs** — canonical `8-4-4-4-12` hex (and the user's own specific user UUID explicitly).
|
|
277
|
+
- **Clerk-style IDs** — `user_` / `org_` / `ins_` / `sess_` / `cli_` + 20+ chars.
|
|
278
|
+
- **Query-string values** — every `?k=v` / `&k=v` value collapsed to `?k=<v>`.
|
|
279
|
+
- **Long base64 / long hex** — `≥40`-char base64 and `≥32`-char hex (hashes, opaque tokens).
|
|
280
|
+
- **Inline image data URIs** — `data:image/…`.
|
|
281
|
+
- **Provider key prefixes** — Stripe (`sk_`/`rk_`/`pk_`), OpenAI (`sk-`/`sk-proj-`), Slack (`xox*`), AWS (`AKIA`/`ASIA`), GitHub (`gh[pousr]_`), Google (`AIza`).
|
|
282
|
+
|
|
283
|
+
#### 5. Maintenance loop
|
|
284
|
+
|
|
285
|
+
When a run uncovers a new key or value shape (a regional BSP / telephony token, a niche analytics SDK key, an internal-tool API, an opaquely-named credential field), **add the regex to the `SCRUB` table here** (and the field name to `SECRET_KEY_RE` if it's name-based) — the layer only stays a real guardrail if every new shape is folded back in.
|
|
286
|
+
|
|
178
287
|
### 7.1 wire-capture & active-interception ethics (the strictest tier)
|
|
179
288
|
|
|
180
289
|
The `wire-capture` dimension climbs a **3-rung ladder, lowest privilege first**, and records the
|
|
@@ -191,8 +300,23 @@ chosen rung in `_meta`:
|
|
|
191
300
|
*user* installs the CA and routes their client — the agent only ingests exported flows. →
|
|
192
301
|
`method: wire-mitmproxy`. Same explicit-authorization loop as session Pass-2.
|
|
193
302
|
|
|
194
|
-
> **
|
|
195
|
-
>
|
|
303
|
+
> **Rung 3+ — the first-party SDK-harness lane (the deepest method for a closed runtime that PUBLISHES
|
|
304
|
+
> an SDK).** When the product ships an official client SDK (npm / PyPI / Maven / …) and the *user* can
|
|
305
|
+
> provision credentials (free tier / trial), the deepest no-source method is not to intercept the
|
|
306
|
+
> *vendor's* app but to **scaffold a minimal throw-away app with the product's OWN SDK + the user's keys,
|
|
307
|
+
> run it, and instrument the client you control**: inject fetch/WS/EventSource taps at `document_start`
|
|
308
|
+
> (so there is **no interceptor-timing problem** — you wrap the constructors before any connection
|
|
309
|
+
> opens), observe every request/response shape, and — with the user routing it through *their own*
|
|
310
|
+
> mitmproxy — the fully-decoded wire. Because you **own** the client, there is no third-party-traffic
|
|
311
|
+
> concern and no missed on-load socket. **Gated exactly like rung 3 / session Pass-2:** the *user*
|
|
312
|
+
> provisions the account + keys and runs any proxy; the **agent never self-signs-up and never enters
|
|
313
|
+
> credentials**, only scaffolds + instruments, and confirms any cost/quota first. → `method: sdk-harness`
|
|
314
|
+
> (a first-party runtime lane; confidence = what is actually observed, typically high since you control
|
|
315
|
+
> the client). Especially powerful for `runtime:reachable + source:none` targets whose published SDK is
|
|
316
|
+
> the only public mirror of the runtime — and it independently corroborates the `api`/`packages` lanes.
|
|
317
|
+
|
|
318
|
+
> **Never fold a browser-rung into a non-browser (curl/WebFetch) collector.** Rung 1 *is* a browser tap,
|
|
319
|
+
> and the browser is a singleton owned by `session`/`wire-capture` — routing it to a curl collector
|
|
196
320
|
> produces zero `source: wire` rows (a real run did exactly this). When `wire-capture` is **folded**,
|
|
197
321
|
> the rung-1 tap is the **in-page interceptor that `session` already installs**: `session` appends its
|
|
198
322
|
> observed paths to `dimensions/_shared/api-path-catalog.md` as `source: wire`. A folded `wire-capture`
|
|
@@ -204,14 +328,14 @@ chosen rung in `_meta`:
|
|
|
204
328
|
|
|
205
329
|
### 7.2 distribution-artifacts ethics
|
|
206
330
|
Downloading a public binary (CRX/`.asar`/`.ipa`/`.apk`) and reading its manifest/strings is fine.
|
|
207
|
-
But binary **decompilation** can cross from "inspect" to "
|
|
331
|
+
But binary **decompilation** can cross from "inspect" to "decompile in violation of EULA":
|
|
208
332
|
the agent reads the artifact's own license/EULA first and records a restrictive one as an Open
|
|
209
333
|
question rather than silently decompiling.
|
|
210
334
|
|
|
211
335
|
---
|
|
212
336
|
|
|
213
337
|
## 8. Completeness check
|
|
214
|
-
A
|
|
338
|
+
A capture is complete when: `_summary.md` exists with full provenance frontmatter + honest `status`;
|
|
215
339
|
every Finding is backed by a `raw/` artifact or inline value; `raw/` artifacts have `_meta` +
|
|
216
340
|
decoding (large ones digested); redaction pass run; `gaps` recorded (never fabricated).
|
|
217
341
|
|
|
@@ -220,15 +344,26 @@ decoding (large ones digested); redaction pass run; `gaps` recorded (never fabri
|
|
|
220
344
|
(downloading many chunks, crawling many pages, deep pagination), then enrich it in place and flip to
|
|
221
345
|
`complete`. **Bound** every fan-out — cap it by count / total bytes / wall-time (e.g. ≤40 items or
|
|
222
346
|
≤90 s) and record the remainder as a `gap` rather than fetching exhaustively; emit a progress note per
|
|
223
|
-
batch. A
|
|
347
|
+
batch. A collector that fetches everything before writing anything can run many times longer than its
|
|
224
348
|
siblings and read as hung to the orchestrator, forcing a needless salvage. Partial-but-finalized beats
|
|
225
349
|
complete-but-silent.
|
|
226
350
|
|
|
351
|
+
**Capability gap = a checkpoint, not a silent fallback.** When a *preferred* capture capability is found
|
|
352
|
+
**unavailable at runtime** — a CDP/remote-debugging port is closed, Chrome / the browser MCP is unreachable,
|
|
353
|
+
a gated proxy or tool can't be stood up — such that you would proceed on a **materially weaker** method
|
|
354
|
+
(e.g. source-only or a digest instead of a live wire tap), **pause and confirm with the user before
|
|
355
|
+
continuing.** State plainly: what is unavailable, the degraded method you would fall back to, and the
|
|
356
|
+
coverage that would be lost. This is distinct from a *planned* method pre-grade (e.g. `openapi-verbatim` →
|
|
357
|
+
`docs-reconstructed` when no spec is served), which is **recorded, not confirmed** — the checkpoint is for an
|
|
358
|
+
*infrastructure/capability* gap discovered mid-run, where silently degrading hides a real coverage loss
|
|
359
|
+
behind a confident-looking capture. (A real run found the CDP port closed and silently fell back to a
|
|
360
|
+
browser-MCP tap; it worked, but the degradation should have been surfaced first.)
|
|
361
|
+
|
|
227
362
|
---
|
|
228
363
|
|
|
229
364
|
## 9. The session dimension — logged-in browser-session playbook
|
|
230
365
|
|
|
231
|
-
The method for the **session** dimension:
|
|
366
|
+
The method for the **session** dimension: discovering the *authenticated runtime* of a
|
|
232
367
|
product by driving it through a browser (Chrome MCP) and instrumenting it. It produces a
|
|
233
368
|
section-by-section understanding of a product's wire protocol, auth, and state model **without
|
|
234
369
|
source-code access**. The `session` agent follows this section.
|
|
@@ -301,6 +436,11 @@ The same phases apply whether the target is an AI builder, analytics dashboard,
|
|
|
301
436
|
|
|
302
437
|
### 9.4 Runtime taps (Pass-1 instrumentation, run in-page)
|
|
303
438
|
|
|
439
|
+
> **The runnable taps + the ~27-entry Chrome-MCP gotcha catalogue + the dead-end recovery table + the
|
|
440
|
+
> Apollo read-side recipe + the fingerprint tables live in `tradecraft.md`** — paste them, do
|
|
441
|
+
> not re-derive. The prose below is the *why*; that file is the *keyboard layer*. The §7.0 redaction
|
|
442
|
+
> scrubbers (`SCRUB`/`redactSecrets`/`ULTRA`) install inside these tap wrappers.
|
|
443
|
+
|
|
304
444
|
> **Install the in-page interceptor FIRST, then move section-to-section WITHOUT a full reload.**
|
|
305
445
|
> CDP `read_network_requests` is unreliable for an already-loaded single-page app: it starts tracking
|
|
306
446
|
> only when first called and silently misses requests already in flight or served from cache (seen on a
|
|
@@ -322,6 +462,24 @@ The same phases apply whether the target is an AI builder, analytics dashboard,
|
|
|
322
462
|
first** — many "binary" protocols are framed text (STOMP, Phoenix, Centrifugo, a `+topic` channel
|
|
323
463
|
prefix), so a legible prefix names the protocol in one step before a binary-codec guess. The frame
|
|
324
464
|
*signature* is the finding; the payload is not.
|
|
465
|
+
- **Realtime-on-load capture (the connection that opens before you can wrap it).** The "install the
|
|
466
|
+
interceptor FIRST" rule assumes you are *already on* the target surface; it does **not** cover a
|
|
467
|
+
realtime connection that fires during the **initial load of a page you `navigate` to** (or a
|
|
468
|
+
lazy-loaded/`iframe`d live demo). A post-load interceptor **cannot** wrap a `WebSocket`/`EventSource`
|
|
469
|
+
constructed before injection, and an `offline`→`online`/`focus` reconnect-trigger may spawn no new
|
|
470
|
+
connection on such a demo — so frame capture comes back empty (a real run got 0 frames on a
|
|
471
|
+
collab-demo this way). When that happens: (a) capture the **CDP-visible auth/token-mint fetch** that
|
|
472
|
+
precedes the socket (it names the realtime host/handshake); (b) **the technique that works —
|
|
473
|
+
client-side-navigate within the SPA** (a router `<Link>` click, not a full reload): the in-page
|
|
474
|
+
wrapper **survives client-side nav**, and the next route **mounts a fresh room → a new socket through
|
|
475
|
+
your wrapper** (this captures the handshake + frames where `offline`/`online` reconnect-triggers
|
|
476
|
+
don't — a real run decoded Liveblocks' v8 JSON protocol this way after a 0-frame first pass); (c) if a
|
|
477
|
+
raw **CDP `Network.webSocketFrameReceived`** tap is reachable, it captures frames protocol-level with
|
|
478
|
+
no injection-timing problem at all (the browser MCP's HTTP-only `read_network_requests` does **not**
|
|
479
|
+
surface WS frames — that gap is what forces (b)); and (d) fall back to the `codebase`/`distribution`
|
|
480
|
+
lanes for the wire format — record it as a *folded* gap, not a silent miss, since the protocol is then
|
|
481
|
+
sourced independently rather than live-observed. **Decode captured frames against any in-repo protocol
|
|
482
|
+
enums** (message-code / op-code tables) so the live wire *validates* the source read.
|
|
325
483
|
- **Storage shape inspector** — enumerate localStorage / sessionStorage / cookies / IndexedDB
|
|
326
484
|
**key names + lengths**, never raw values.
|
|
327
485
|
- **JWT decoder** — decode the token *structure* (alg, claim names, exp) with identity claims
|
|
@@ -344,8 +502,19 @@ The same phases apply whether the target is an AI builder, analytics dashboard,
|
|
|
344
502
|
set, an RPC/tool method list — consume it as a **coverage checklist**: walk the routes/actions that
|
|
345
503
|
trigger the un-observed entries so the declared-vs-observed diff (§6) reflects a real walk, not just the
|
|
346
504
|
landing page.
|
|
347
|
-
|
|
348
|
-
|
|
505
|
+
- **Server-side-read (RSC) check — don't mis-record "no read API."** In a modern Next.js App-Router /
|
|
506
|
+
React-Server-Components app (and similar SSR/streaming stacks), the initial **read** data is fetched
|
|
507
|
+
*server-side* with the session and streamed — so the client wire shows only mutations, asset/media
|
|
508
|
+
fetches, telemetry, and token-mints, and **no client-side data-read XHRs fire on load even though the
|
|
509
|
+
UI is fully populated.** Before recording "read API not observed," check for the server-side path: a
|
|
510
|
+
document/`?_rsc=` response with `text/x-component`, embedded `__next_f`/flight payloads, `_next/data`
|
|
511
|
+
JSON, or server actions. When reads are server-resolved, say so — and note that a sibling
|
|
512
|
+
bundle's app-own *path count* then **overstates the client read surface** (fold this into the
|
|
513
|
+
published-vs-app-own diff, §6, rather than reporting a phantom "unused routes" gap). (Writes can be
|
|
514
|
+
off-main-thread too — a generate POST issued from a **Web Worker** won't appear in a main-thread fetch
|
|
515
|
+
tap; corroborate via CDP / Performance resource timing.)
|
|
516
|
+
|
|
517
|
+
### 9.5 The capture contract (richer than a standard dimension capture)
|
|
349
518
|
|
|
350
519
|
Under `research/<target>/dimensions/session/`:
|
|
351
520
|
|