@nurix/etna 0.19.40 → 0.19.42

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (38) hide show
  1. package/data/harness-index.json +1 -1
  2. package/data/harnesses/code/harness.json +1 -1
  3. package/data/harnesses/delivery/harness.json +1 -1
  4. package/data/harnesses/discover/CLAUDE.md +50 -2
  5. package/data/harnesses/discover/agents/api.md +24 -0
  6. package/data/harnesses/discover/agents/codebase.md +24 -0
  7. package/data/harnesses/discover/agents/community.md +24 -0
  8. package/data/harnesses/discover/agents/deployed-client-bundle.md +24 -0
  9. package/data/harnesses/discover/agents/distribution-artifacts.md +24 -0
  10. package/data/harnesses/discover/agents/docs.md +27 -1
  11. package/data/harnesses/discover/agents/infra-backend-fingerprint.md +24 -0
  12. package/data/harnesses/discover/agents/packages.md +24 -0
  13. package/data/harnesses/discover/agents/session.md +40 -0
  14. package/data/harnesses/discover/agents/website.md +24 -0
  15. package/data/harnesses/discover/agents/wire-capture.md +40 -0
  16. package/data/harnesses/discover/codex/agents/api.toml +24 -0
  17. package/data/harnesses/discover/codex/agents/codebase.toml +24 -0
  18. package/data/harnesses/discover/codex/agents/community.toml +24 -0
  19. package/data/harnesses/discover/codex/agents/deployed-client-bundle.toml +24 -0
  20. package/data/harnesses/discover/codex/agents/distribution-artifacts.toml +24 -0
  21. package/data/harnesses/discover/codex/agents/docs.toml +27 -1
  22. package/data/harnesses/discover/codex/agents/infra-backend-fingerprint.toml +24 -0
  23. package/data/harnesses/discover/codex/agents/packages.toml +24 -0
  24. package/data/harnesses/discover/codex/agents/session.toml +40 -0
  25. package/data/harnesses/discover/codex/agents/website.toml +24 -0
  26. package/data/harnesses/discover/codex/agents/wire-capture.toml +40 -0
  27. package/data/harnesses/discover/harness.json +3 -4
  28. package/data/harnesses/discover/rules/calibration.md +219 -0
  29. package/data/harnesses/discover/rules/cartography-coverage.md +69 -1
  30. package/data/harnesses/discover/rules/discovery.md +59 -1
  31. package/data/harnesses/discover/rules/evaluation.md +13 -0
  32. package/data/harnesses/discover/rules/ingestion.md +248 -7
  33. package/data/harnesses/discover/rules/self-correction.md +6 -2
  34. package/data/harnesses/discover/rules/tooling.md +205 -0
  35. package/data/harnesses/discover/rules/tradecraft.md +116 -2
  36. package/data/harnesses/etna/harness.json +1 -1
  37. package/package.json +2 -2
  38. package/data/harnesses/discover/tooling-recommendations.md +0 -119
@@ -55,4 +55,28 @@ Then write the per-type raw files that apply — `raw/{extension-manifest.md, el
55
55
  Return a **one-paragraph headline** of what the shipped client reveals — its declared hosts, permission surface, embedded vendor SDKs, and how much richer (or not) it is than the web app. The API paths and feature flags you appended feed the **seam rule (§6)**: your `dimensions/_shared/api-path-catalog.md` rows carry `source: distribution` so evaluation can run its adaptive diff against `deployed-client-bundle` (`source: bundle`), `session`, and `wire-capture`. The hosts and SDK list cross-check **infra-backend-fingerprint** (a host the binary calls that DNS/CT didn't surface is a finding); the runtime hosts also seed **wire-capture** targets (the binary says what it _might_ call; the capture says what it _does_).
56
56
 
57
57
  **Discipline:** EULA before extraction — restrictive license ⇒ Open question, never silent decompile. Manifests/strings only; no protected-logic reversing. Redact every embedded key before any `raw/` write (§7 rule 3). Digest large bundles (§5). A missing artifact type is a recorded gap, not a guess.
58
+
59
+ ---
60
+
61
+ ## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
62
+
63
+ **MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
64
+ the manifests verbatim (`manifest.json`, `Info.plist`, `AndroidManifest.xml`) and the extracted file listing.
65
+
66
+ Three rules bind this, and they override any instinct to summarise:
67
+
68
+ 1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
69
+ count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
70
+ medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
71
+ the citation will outlive its evidence.
72
+ 2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
73
+ `_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
74
+ essay, not a capture.
75
+ 3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
76
+ straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
77
+ bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
78
+ discard evidence.
79
+
80
+ **Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
81
+ regardless of how good the prose is.
58
82
  '''
@@ -32,5 +32,31 @@ You collect a target's docs into `research/<target>/dimensions/docs/`. Read `.cl
32
32
 
33
33
  A one-paragraph headline (the product's documented surface in a sentence) + the section list with page counts + anything that overlaps the api/packages dimensions (flag it so synthesis can cross-check).
34
34
 
35
- **Discipline:** distill, don't mirror. A 300-page docs site becomes a navigable map + the load-bearing content, not 300 verbatim files. Record paywalled/gated docs as Open questions.
35
+ **Discipline:** distill **in addition to**, never instead of. Save each captured page's source form under `raw/pages/` (md or html, as fetched) — that is the evidence layer — and build the navigable map + load-bearing content as the narrative layer beside it (ingestion §5.1). A distilled section whose source page was never saved is unverifiable by the next run (§5.3).
36
+
37
+ **Do not mirror the SITE:** A 300-page docs site becomes a navigable map + the load-bearing content, not 300 verbatim files. Record paywalled/gated docs as Open questions.
38
+
39
+ ---
40
+
41
+ ## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
42
+
43
+ **MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
44
+ each captured page's source form under `raw/pages/` (md or html, as fetched).
45
+
46
+ Three rules bind this, and they override any instinct to summarise:
47
+
48
+ 1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
49
+ count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
50
+ medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
51
+ the citation will outlive its evidence.
52
+ 2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
53
+ `_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
54
+ essay, not a capture.
55
+ 3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
56
+ straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
57
+ bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
58
+ discard evidence.
59
+
60
+ **Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
61
+ regardless of how good the prose is.
36
62
  '''
@@ -61,4 +61,28 @@ Return a **one-paragraph headline** of the reconstructed backend — origin clou
61
61
  **Discipline:** read-only public recon — never scan, brute-force, or hit a non-public host; respect crt.sh/WHOIS rate limits and back off; redact any credential-shaped value before writing (contract §7 rule 3 — DSNs, verification tokens, embedded keys surfaced in headers/TXT). An unresolvable host or a missing sub-processor page is an Open question, not a guess.
62
62
 
63
63
  **Verify an absence before asserting it (contract §7 rule 10).** Before reporting any host as **NXDOMAIN / absent** — especially `api.<target>` or any host a sibling dimension (bundle / session / wire) reports actually hitting — re-resolve via a **second resolver** (`dig @1.1.1.1 <host>`, `dig @8.8.8.8 <host>`) **and** a direct `curl -sI https://<host>`. An HTTP 4xx/5xx response means the host **resolves** → report it **present**, not absent. A single negative resolver result is recorded as _tentative_, never stated as fact and never used to instruct sibling dimensions. Likewise, an _inferred_ backend engine (e.g. "looks like Hasura/Supabase from the CNAME") is phrased as an inference and yields to a runtime-observing dimension (`session`/`deployed-client-bundle`) that names the engine directly.
64
+
65
+ ---
66
+
67
+ ## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
68
+
69
+ **MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
70
+ the raw `dig`, `crt.sh` and HTTP header responses as returned; the decoded table is the narrative layer beside them.
71
+
72
+ Three rules bind this, and they override any instinct to summarise:
73
+
74
+ 1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
75
+ count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
76
+ medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
77
+ the citation will outlive its evidence.
78
+ 2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
79
+ `_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
80
+ essay, not a capture.
81
+ 3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
82
+ straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
83
+ bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
84
+ discard evidence.
85
+
86
+ **Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
87
+ regardless of how good the prose is.
64
88
  '''
@@ -27,4 +27,28 @@ You collect a target's installable artifacts into `research/<target>/dimensions/
27
27
  A one-paragraph headline (the package family in a sentence) + the package table + which packages are the core public API (so synthesis can anchor the SDK surface). Flag overlap with the docs api-reference and the codebase dimensions for cross-check.
28
28
 
29
29
  **Discipline:** the exports map + type defs are the load-bearing artifact — prioritize them over download-count trivia. Digest huge `.d.ts` to signatures. Redact any registry tokens.
30
+
31
+ ---
32
+
33
+ ## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
34
+
35
+ **MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
36
+ per package: the registry metadata JSON (`npm view <pkg> --json > raw/<pkg>.registry.json`, PyPI `/pypi/<pkg>/json`) and the `.d.ts` / exports surface as fetched.
37
+
38
+ Three rules bind this, and they override any instinct to summarise:
39
+
40
+ 1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
41
+ count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
42
+ medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
43
+ the citation will outlive its evidence.
44
+ 2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
45
+ `_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
46
+ essay, not a capture.
47
+ 3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
48
+ straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
49
+ bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
50
+ discard evidence.
51
+
52
+ **Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
53
+ regardless of how good the prose is.
30
54
  '''
@@ -38,4 +38,44 @@ Under `research/<target>/dimensions/session/`:
38
38
  A one-paragraph headline (the auth model + transport + the single most load-bearing wire finding) + the phase coverage + which passes ran. Flag drift between the _observed_ wire surface and the _published_ api/packages surface — that drift is the highest-value synthesis input. If Pass 2 was needed but not authorized, list the open write-side questions; never fabricate request bodies.
39
39
 
40
40
  **Discipline:** ethics first, always. Redact before writing. Confirm before any state change. No destructive actions. No exfiltration. A gap is an Open question, never a guess.
41
+
42
+ ---
43
+
44
+ ## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
45
+
46
+ **MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
47
+ ≥3 verbatim envelopes as `raw/*.json` — the identity read, one canonical list read, and the primary write (request + response) — per ingestion §8.
48
+
49
+ Three rules bind this, and they override any instinct to summarise:
50
+
51
+ 1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
52
+ count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
53
+ medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
54
+ the citation will outlive its evidence.
55
+ 2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
56
+ `_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
57
+ essay, not a capture.
58
+ 3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
59
+ straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
60
+ bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
61
+ discard evidence.
62
+
63
+ **Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
64
+ regardless of how good the prose is.
65
+
66
+ ## Visual capture (ingestion §5.5) — screenshot generously
67
+
68
+ You drive a real browser, so you carry the visual-evidence obligation. **Capture continuously, not
69
+ one-image-per-surface:** every surface, plus every materially different state within it — empty vs
70
+ populated, modal/drawer open, dropdown expanded, error and validation states, permission/plan walls,
71
+ loading/in-flight, and the before/after pair around any authorized state change.
72
+
73
+ This is **not** scored — `screenshot_coverage` is a narrow Mode-5 floor. The beneficiary is the human
74
+ reading this corpus weeks later for a purpose nobody anticipated. The product's past state is the one thing
75
+ a later run genuinely cannot reproduce.
76
+
77
+ **Redact in the DOM before the shutter** (`tradecraft.md` §1c) — a leaked pixel is undetectable afterwards.
78
+ **Check the corners of the frame**, not just the content: support-chat widgets, toasts, presence avatars and
79
+ member lists routinely carry *another human's* name and message. Then look at the saved image, and record
80
+ `redacted: <what>` in `screens/_index.md` rather than a blanket zero-PII claim.
41
81
  '''
@@ -33,4 +33,28 @@ You collect a target's public-facing site into `research/<target>/dimensions/web
33
33
  A one-paragraph headline (the positioning in a sentence) + the pricing tiers + the headline feature claims. Flag claims that should be cross-checked against the codebase/docs (marketing often over-claims) so synthesis can reconcile.
34
34
 
35
35
  **Discipline:** quote positioning verbatim; capture pricing precisely (numbers + limits); separate _claims_ from verified capability — the reconciliation happens in synthesis. Respect robots.txt and rate limits; this is read-only public crawling, not scraping at scale.
36
+
37
+ ---
38
+
39
+ ## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
40
+
41
+ **MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
42
+ the raw HTML or extracted text per captured page, and the pricing table as structured data.
43
+
44
+ Three rules bind this, and they override any instinct to summarise:
45
+
46
+ 1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
47
+ count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
48
+ medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
49
+ the citation will outlive its evidence.
50
+ 2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
51
+ `_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
52
+ essay, not a capture.
53
+ 3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
54
+ straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
55
+ bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
56
+ discard evidence.
57
+
58
+ **Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
59
+ regardless of how good the prose is.
36
60
  '''
@@ -42,4 +42,44 @@ Then the captures + raw artifacts (per `discovery` Part C (wire-capture)):
42
42
  Return a **one-paragraph headline** of what the wire reveals — the real hosts contacted, the envelope/auth scheme, and the rung used (so the reader knows how much to trust it). Then a compact table of observed endpoints. **Flag the seam rule (§6):** every `api-path-catalog.md` row is `source: wire`, so evaluation can diff _actually-called_ (wire) against _statically-declared_ (`bundle` / `distribution-artifacts`) and _browser-observed_ (`session`) — a host or path the binary declares but the wire never hits is a finding, not noise; the inverse (called but undeclared) is a stronger one. If only rung 1 was authorized, say so and name what rungs 2/3 would unlock so the main session can request authorization.
43
43
 
44
44
  **Discipline:** lowest privilege first; never self-escalate the ladder; never install a CA or tap third-party traffic; never run Wireshark; redact every body before disk and log it in `_meta`. An unauthorized higher rung is an Open question, not an excuse to guess what the wire would show.
45
+
46
+ ---
47
+
48
+ ## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
49
+
50
+ **MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
51
+ the flow log itself (HAR or the tap's JSON export), not only the decoded narrative.
52
+
53
+ Three rules bind this, and they override any instinct to summarise:
54
+
55
+ 1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
56
+ count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
57
+ medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
58
+ the citation will outlive its evidence.
59
+ 2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
60
+ `_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
61
+ essay, not a capture.
62
+ 3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
63
+ straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
64
+ bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
65
+ discard evidence.
66
+
67
+ **Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
68
+ regardless of how good the prose is.
69
+
70
+ ## Visual capture (ingestion §5.5) — screenshot generously
71
+
72
+ You drive a real browser, so you carry the visual-evidence obligation. **Capture continuously, not
73
+ one-image-per-surface:** every surface, plus every materially different state within it — empty vs
74
+ populated, modal/drawer open, dropdown expanded, error and validation states, permission/plan walls,
75
+ loading/in-flight, and the before/after pair around any authorized state change.
76
+
77
+ This is **not** scored — `screenshot_coverage` is a narrow Mode-5 floor. The beneficiary is the human
78
+ reading this corpus weeks later for a purpose nobody anticipated. The product's past state is the one thing
79
+ a later run genuinely cannot reproduce.
80
+
81
+ **Redact in the DOM before the shutter** (`tradecraft.md` §1c) — a leaked pixel is undetectable afterwards.
82
+ **Check the corners of the frame**, not just the content: support-chat widgets, toasts, presence avatars and
83
+ member lists routinely carry *another human's* name and message. Then look at the saved image, and record
84
+ `redacted: <what>` in `screens/_index.md` rather than a blanket zero-PII claim.
45
85
  '''
@@ -1,12 +1,11 @@
1
1
  {
2
2
  "harness": "discover",
3
3
  "name": "discover",
4
- "version": "0.1.12",
4
+ "version": "0.1.14",
5
5
  "files": [
6
6
  ".prettierignore",
7
7
  ".prettierrc.json",
8
- ".vscode/settings.json",
9
- "tooling-recommendations.md"
8
+ ".vscode/settings.json"
10
9
  ],
11
- "digest": "48356bdb7940"
10
+ "digest": "4ab9f67f0d67"
12
11
  }
@@ -0,0 +1,219 @@
1
+ # Calibration Reference — Blind Re-runs · Disposition · The Fleet Ledger
2
+
3
+ The reference for the harness's **only external signal**. Everything else in this harness is self-assessment:
4
+ collectors grade their own `confidence` and `completeness_pct`, Evaluation weights those self-grades, and
5
+ Mode 5 scores the result against a checklist the same system authored. That loop can converge on a corpus
6
+ that is confidently, well-provenanced, and wrong.
7
+
8
+ **Calibration is the one procedure that can tell you whether any of it is true.** It re-derives claims a
9
+ prior run published, blind, and records what survived.
10
+
11
+ > **The measured position, as of 2026-07-28.** One target has ever been re-run (`emergent`, 2026-06-25 →
12
+ > 2026-07-28). Of 16 claims re-tested: **7 refuted, 4 corrected, 5 confirmed** — a 44% flat-falsification
13
+ > rate — and **4 of the 7 refutations came from lanes the prior run had graded `confidence: high`.** The
14
+ > prior run had self-scored **88/100 "strong"** and `stopped_on: converged`.
15
+ >
16
+ > That is **n=1**. It is the entire empirical basis of this harness. Every rule in `.claude/rules/` is
17
+ > currently a hypothesis with n=0 on any target it did not come from. The purpose of this file is to make
18
+ > n grow.
19
+
20
+ ---
21
+
22
+ ## Two tiers — pick the one that will actually happen
23
+
24
+ A metric too expensive to run leaves `n` at 1 forever. The cheap tier is the important one.
25
+
26
+ | Tier | Cadence | Cost | What it measures |
27
+ | --- | --- | --- | --- |
28
+ | **A — Claim spot-check** | monthly | ~1 session, no full ingestion | the refutation rate, and whether `confidence` tracks truth |
29
+ | **B — Full blind re-run** | quarterly, or when a rule set changes materially | a complete run | everything Tier A does, plus coverage drift and whether new rules changed behaviour |
30
+
31
+ **Tier A is the default. Do not skip it waiting for capacity to do Tier B** — a monthly spot-check of 15–20
32
+ claims across one target produces the regression that matters, and produces it twelve times a year.
33
+
34
+ ---
35
+
36
+ ## Selection — rotate, never choose
37
+
38
+ **The target is selected by rotation, not by judgment.** Take the least-recently-calibrated target in the
39
+ corpus root; break ties by oldest `captured_at`. Record the selection rule alongside the result.
40
+
41
+ A human (or a run) choosing which target to re-check will choose the one it feels good about. That single
42
+ substitution destroys the metric, because the sample stops being representative of the fleet and starts
43
+ being representative of the harness's confidence — which is the very thing under test.
44
+
45
+ ---
46
+
47
+ ## THE BLIND RULE (this is what makes it evidence)
48
+
49
+ **Capture first. Read the prior corpus only after your own observation is written down.**
50
+
51
+ 1. Select the target. **Do not open its corpus.**
52
+ 2. Draw the claim sample (below) — a list of *questions*, not answers. Someone or something other than the
53
+ re-deriving agent extracts the questions, or the questions are extracted mechanically from claim
54
+ anchors, so the sample carries no prior conclusions.
55
+ 3. **Re-derive each claim from the live product**, from scratch, recording what you observe and the method.
56
+ 4. **Only then** open the prior corpus and diff.
57
+
58
+ > **This deliberately overrides `discovery.md` Step 00 for calibration runs.** Step 00 tells a run to find a
59
+ > prior corpus, restore it, and reconcile against it — correct for a normal re-run, where the prior corpus
60
+ > is a genuine corroboration lane and re-deriving everything from zero is waste. **It is wrong here.** A run
61
+ > that reads the prior claim first will anchor on it: it will look for confirmation, find it, and report a
62
+ > confirmation rate that measures its own suggestibility. Anchoring is the specific failure this protocol
63
+ > exists to avoid, so for a calibration run Step 00 is deferred until after step 3, and the run records
64
+ > `blind: true`.
65
+ >
66
+ > If a run cannot honestly attest that it re-derived before reading, it must record `blind: false` — and
67
+ > the result is then a **consistency check, not a calibration**, and does not enter the ledger's refutation
68
+ > rate.
69
+
70
+ **Sampling the claims.** 15–20 per target, drawn to make the regression computable, not to be flattering:
71
+
72
+ - **Stratify by the prior run's stated `confidence`** — roughly equal numbers of `high`, `medium`, `low`.
73
+ Without this the regression cannot be computed, and the regression is the point.
74
+ - **Include the load-bearing ones** — claims a README headline rests on, and claims a reader would act on
75
+ (pricing, limits, guarantees, auth model, what the product can and cannot do).
76
+ - **Include at least 3 negative claims** ("no public SDK", "no realtime layer", "not supported"). Negatives
77
+ are the harness's demonstrated weak spot: **5 of the 7 refutations in the one data point were negatives.**
78
+ - **Exclude** anything whose truth cannot change and cannot be re-checked (a historical funding round).
79
+
80
+ ---
81
+
82
+ ## Disposition — five values, and the fifth is the one the old schema lacked
83
+
84
+ Every sampled claim resolves to exactly one:
85
+
86
+ | Disposition | Meaning | What it says about the harness |
87
+ | --- | --- | --- |
88
+ | **confirmed** | independently re-derived, same conclusion | the method worked |
89
+ | **corrected** | right direction, wrong detail (a count, a version, a name) | the method worked; the reading was imprecise |
90
+ | **refuted** | the claim was **wrong when it was made** | **the method failed — this is the signal** |
91
+ | **stale** | the claim was **true when made; the product changed** | the method worked; the corpus has a shelf life |
92
+ | **unverifiable** | cannot be re-checked now (surface gone, plan changed, gated) | no signal either way; excluded from the rate |
93
+
94
+ > **`refuted` and `stale` are the distinction the original artifact had no slot for, and they mean opposite
95
+ > things.** A high `refuted` rate means *the instrument is unreliable* and the rules need work. A high
96
+ > `stale` rate means *the instrument is fine and the corpus decays* — which calls for shorter shelf-lives
97
+ > and `volatility` tagging, not methodology changes. Conflating them makes the harness fix the wrong
98
+ > problem. When you cannot tell which it is — no dated evidence either way — record **`refuted`**. The
99
+ > conservative reading is the one that keeps the harness honest about itself.
100
+ >
101
+ > (Origin: `emergent/00-prior-run-reconciliation.md` — a genuinely good artifact whose section A had to be
102
+ > titled *"REFUTED (prior claim was wrong, not merely stale)"* precisely because the schema gave it nowhere
103
+ > else to put the distinction.)
104
+
105
+ **The refutation rate** = `refuted ÷ (confirmed + corrected + refuted + stale)`. `unverifiable` is excluded
106
+ from the denominator, and its count is reported so a shrinking sample is visible.
107
+
108
+ ---
109
+
110
+ ## The regression — the actual question
111
+
112
+ For each calibration, record the disposition **against the prior run's stated confidence for that claim**:
113
+
114
+ | prior confidence | n | confirmed | corrected | refuted | stale | refutation rate |
115
+ | --- | --- | --- | --- | --- | --- | --- |
116
+ | high | | | | | | |
117
+ | medium | | | | | | |
118
+ | low | | | | | | |
119
+
120
+ **If `high` claims are refuted at a similar rate to `low` claims, the `confidence` field carries no signal
121
+ and `ingestion.md` §3's method→confidence table is decorative.** That is the single most consequential thing
122
+ this harness could learn about itself, and it is unknowable without this table.
123
+
124
+ Also record refutation rate **by `method`** once n is large enough — that tells you which of the 17 methods
125
+ in §3 are over-graded, which is directly actionable (a method's band gets lowered).
126
+
127
+ ---
128
+
129
+ ## Artifacts
130
+
131
+ ### Per calibration — `research/<target>/00-prior-run-reconciliation.md`
132
+
133
+ Formalizes the shape `emergent` invented. **This file is part of the output contract** whenever a prior
134
+ corpus exists.
135
+
136
+ ```markdown
137
+ ---
138
+ target: <slug>
139
+ reconciled_at: <YYYY-MM-DD>
140
+ prior_run_date: <YYYY-MM-DD>
141
+ prior_run_commit: <sha>
142
+ prior_run_score: <n>
143
+ tier: A | B # spot-check or full blind re-run
144
+ blind: true | false # false ⇒ consistency check only, excluded from the rate
145
+ selection_rule: rotation # or: named-by-user (record it either way)
146
+ sampled: <n>
147
+ confirmed: <n>
148
+ corrected: <n>
149
+ refuted: <n>
150
+ stale: <n>
151
+ unverifiable: <n>
152
+ refutation_rate: <0-100>
153
+ by_confidence: # the regression — REQUIRED
154
+ high: { n: <n>, refuted: <n> }
155
+ medium: { n: <n>, refuted: <n> }
156
+ low: { n: <n>, refuted: <n> }
157
+ ---
158
+
159
+ ## A. REFUTED — wrong when made (claim · what was observed · lane that refuted it · prior confidence)
160
+ ## B. STALE — true when made, changed (claim · what changed · evidence of the change being NEW)
161
+ ## C. CORRECTED — right, imprecise
162
+ ## D. CONFIRMED — independently re-derived
163
+ ## E. UNVERIFIABLE — and why
164
+ ## F. What this says about the METHOD (which lanes/methods produced the refutations, and whether a
165
+ §3 confidence band should be lowered)
166
+ ```
167
+
168
+ ### Fleet-level — `research/_calibration/ledger.md`
169
+
170
+ **The metric itself.** One row per calibration, accumulating across targets and time. This is the file to
171
+ read when asking "does this harness work?" It is never rewritten, only appended to.
172
+
173
+ ---
174
+
175
+ ## What a result CHANGES (otherwise this is just bookkeeping)
176
+
177
+ A calibration that produces a number and no action is theatre. Each outcome routes:
178
+
179
+ | Finding | Action |
180
+ | --- | --- |
181
+ | A **method** shows a high refutation rate | lower its band in `ingestion.md` §3, and record the calibration as the origin |
182
+ | **Negative claims** refute disproportionately | tighten the absence rules (`ingestion.md` §7 rule 10, `discovery.md` Part B) — the `blind_to:` direction |
183
+ | `high` refutes ≈ as often as `low` | **the confidence field is not carrying signal** — this is a structural finding about the harness, and it goes to the user, not into a rule tweak |
184
+ | A **rule added after the prior run** would have prevented a refutation | evidence the improvement loop works — record it, it is the only such evidence that exists |
185
+ | A refutation recurs **after** a rule was added to prevent it | evidence a rule did **not** change behaviour; escalate to the user rather than adding another rule |
186
+ | High **stale** rate | shorten shelf-life expectations and tag volatile claims; **not** a methodology problem |
187
+
188
+ ---
189
+
190
+ ## How to run one (Tier A, the monthly default)
191
+
192
+ 1. **Select** by rotation from `research/_calibration/ledger.md`. Record the target and the rule.
193
+ 2. **Extract the sample** mechanically — pull claim anchors from the target's `evaluation/*.md`, stratify by
194
+ the cited confidence, and write them as **questions with the answers stripped**.
195
+ 3. **Re-derive blind.** Do not open the corpus. Use whatever lanes the access vector allows; record method
196
+ and observation per claim.
197
+ 4. **Diff and disposition** each claim into the five values.
198
+ 5. **Write** `00-prior-run-reconciliation.md` (append a new dated section if one already exists — never
199
+ overwrite a prior calibration).
200
+ 6. **Append one row** to the fleet ledger, and update its running totals.
201
+ 7. **Route** every finding through the table above. Anything that becomes a rule change goes through the
202
+ normal Mode-5 approval gate.
203
+
204
+ **Cost/state discipline is unchanged:** a calibration is read-only unless the user authorizes otherwise; a
205
+ claim that can only be re-derived by spending money or mutating state is recorded `unverifiable` with the
206
+ reason, never quietly exercised.
207
+
208
+ ---
209
+
210
+ ## How the harness uses this reference
211
+
212
+ Calibration is a **cross-run process**, not a mode of a single run — nothing in Modes 1–5 triggers it, and it
213
+ must not be folded into them (a run cannot calibrate itself; that is the circularity it exists to break).
214
+ Mode 5 reads the fleet ledger as *context* when weighing a proposal — a rule contradicted by calibration
215
+ data should not be promoted — but Mode 5 never writes to it.
216
+
217
+ > **Do not attach calibration to the Mode-5 score.** The moment `refutation_rate` becomes a rubric input,
218
+ > the run acquires an interest in the number, and the one uncorrupted signal in the system is gone. It
219
+ > measures the **harness**, not the run. Publish it; never grade with it.
@@ -33,6 +33,40 @@ differently:
33
33
  | **edition/plan-gated** | not in this tenant's edition/plan (e.g. a trial) | a recorded gap — **not** re-walked (re-walking can't find what isn't shipped here) |
34
34
  | **roadmap-not-shipped** | marketed but not yet a shipped capability | a recorded gap |
35
35
  | **marketing over-claim** | the claim overstates what the product does | an `evaluation/product-features.md` **over-claim flag** |
36
+ | **not-yet-reachable** | it exists and is shipped, but the surface **does not render at all** until a state change (deploy / connect / upgrade / first-run) that this run did not perform | the **gated re-walk queue** — NOT the read-only auto-loop |
37
+
38
+ ### `not-yet-reachable` vs `deeper-than-looked` — the distinction that matters
39
+
40
+ These two look identical in a coverage matrix and route **oppositely**, so get it right:
41
+
42
+ - **`deeper-than-looked`** — the surface **is in the DOM right now**; the run just didn't navigate to it.
43
+ More read-only walking finds it. Routes to the **read-only re-walk queue**, which Mode 5's auto-loop
44
+ works for free.
45
+ - **`not-yet-reachable`** — the surface **is not in the product yet.** No amount of navigation reveals it,
46
+ because the affordance is created by a state change. Routes to a **separate gated queue** that the
47
+ auto-loop must **not** touch: closing it costs money, mutates state, or needs a third-party consent, so
48
+ it goes through the explicit-confirmation gate (ingestion §7 rule 4) or is recorded as a standing gap.
49
+
50
+ **Mis-filing a `not-yet-reachable` row as `deeper-than-looked` is a Cartography defect**, because it sends
51
+ the auto-loop hunting for something that does not exist, and it reads as "we were sloppy" when the honest
52
+ statement is "the product's IA is state-dependent and this run stayed inside its authorization."
53
+
54
+ **A run that never performs the state change cannot map a state-dependent IA — say so rather than
55
+ scoring it as a coverage miss.** Note it beside the scorecard the same way `experiential_gated` annotates
56
+ a policy deferral: it documents, it does not inflate.
57
+
58
+ > (Origin: Emergent — a deployment executed mid-run **created** two whole surfaces: the entire *Manage
59
+ > Publishing* panel with five sub-tabs, and the `db_tool` database-console affordance, which opened a
60
+ > separately-hosted plugin on a second auth system. Neither exists in the toolbar of an undeployed project.
61
+ > The pre-deploy coverage doc had filed the publish family under *deeper-than-looked* and queued it for a
62
+ > read-only re-walk that could never have succeeded. Note also the **two-factor gate**: the cohort flag
63
+ > `db_tool: true` was necessary but **not sufficient** — an entitlement flag reading `true` does not mean
64
+ > the surface is reachable.)
65
+
66
+ **Where this bites generally:** any PaaS or marketplace with deploy/publish-gated management surfaces; any
67
+ product whose integrations render only after an OAuth connect; any tier-gated admin console; any
68
+ first-run/onboarding-gated dashboard. Treat an entitlement flag as **necessary, not sufficient** — check
69
+ whether the *state* it depends on has also been reached.
36
70
 
37
71
  ---
38
72
 
@@ -50,6 +84,7 @@ coverage_scorecard:
50
84
  ia_surfaces_mapped: <N> # surface cards written in information-architecture.md
51
85
  ia_nav_complete: <true|false> # every top-level nav destination has a surface card
52
86
  screenshot_coverage: <0-100> # primary surfaces with a screenshot ÷ all primary surfaces
87
+ features_not_yet_reachable: <N> # shipped, but behind a state change this run did not perform
53
88
  ---
54
89
  ## Coverage matrix
55
90
  | Feature (claimed) | Source | Located? | Route · depth | Walked? | Edition-gated? | Confidence | Note |
@@ -57,6 +92,9 @@ coverage_scorecard:
57
92
  | <feature> | website,docs | ✅ | #/…/x · D2 | ❌ not-walked | — | high | promoted on site, 2 levels deep, not exercised this run |
58
93
  | <feature> | website(AI) | ⚠ edition? | — | ❌ | maybe-not-in-trial | low | marketed flagship, not located in this tenant |
59
94
  ## Located-but-not-walked (the within-run re-walk queue — Mode 5 walks these read-only, no cost)
95
+ ## Not-yet-reachable (the GATED queue — shipped surfaces behind a deploy/connect/upgrade this run
96
+ did not perform. Mode 5 must NOT auto-walk these; each names the exact state
97
+ change that would unlock it and its cost/consent class)
60
98
  ## Claimed-but-not-located (each row dispositioned per the taxonomy above — NEVER left blank)
61
99
  ## Buried flagships (marketing-promoted features sitting ≥D3 — the positioning-vs-product gap,
62
100
  drawn from information-architecture.md's promoted-vs-buried headline)
@@ -69,6 +107,10 @@ coverage_scorecard:
69
107
  experiential / IA coverage axis (`self-correction.md` rubric, axis 2, max 20) is reproducible:
70
108
 
71
109
  - `feature_location_rate` = `100 × features_located ÷ features_claimed`.
110
+ - **`not-yet-reachable` rows stay in the `features_claimed` denominator** — they are real, shipped,
111
+ unlocated capabilities and hiding them would inflate the rate. But they are **dispositioned**, so they
112
+ never draw the −2 un-dispositioned penalty, and the scorecard carries `features_not_yet_reachable` so a
113
+ reader can tell a *policy/authorization* boundary from a *thoroughness* failure.
72
114
  - `flow_coverage` = `100 × (primary journeys diagrammed in ux-flows.md ÷ journeys identified)`.
73
115
  - `screenshot_coverage` = `100 × (primary surfaces with a screenshot ÷ all primary surfaces)`.
74
116
  - `ia_nav_complete` = every D0 nav destination has a surface card.
@@ -81,6 +123,13 @@ un-dispositioned. The `located-but-not-walked` list is the re-walk queue Mode 5
81
123
 
82
124
  ## Screenshots — `dimensions/session/captures/screens/`
83
125
 
126
+ > **The scorecard metric is a FLOOR, not the capture rule.** `screenshot_coverage` asks a narrow question —
127
+ > did each *primary surface* get one image — because Mode 5 needs something countable. **The actual capture
128
+ > obligation is `ingestion.md` §5.5: screenshot generously, including states within a surface, for the human
129
+ > who reads this corpus later.** Satisfying this metric and stopping is the failure mode; the metric is
130
+ > nearly always already met by a real walk. Nothing here caps how much you capture, and extra images never
131
+ > hurt the score.
132
+
84
133
  One annotated screenshot per **primary** surface (each D0 nav destination + each materially distinct
85
134
  detail / editor surface). **Redact visible PII before saving** (user names, emails, customer data,
86
135
  partner identifiers). Name `<nn>-<surface>.png`; write a one-line `screens/_index.md` mapping each file
@@ -89,6 +138,25 @@ partner identifiers). Name `<nn>-<surface>.png`; write a one-line `screens/_inde
89
138
  **Screenshots are the Mode-4 deliverable that the session playbook's textual surface-map only
90
139
  _substitutes_ for** (ingestion §9.5) — capture them live here; fall back to the textual map (and record
91
140
  the `screenshot_coverage` gap) only when the runtime genuinely cannot screenshot (no image tool, or the
92
- app never reaches page-idle because of a persistent connection). The screenshots live under the
141
+ app never reaches page-idle because of a persistent connection).
142
+
143
+ > **Re-test the capability before recording `screenshot_coverage: 0` — an environment limitation claimed
144
+ > by a PRIOR run is `verify:`, never inherited.** Take one screenshot with `save_to_disk: true` in the
145
+ > current session and record the literal tool result. Only an actual failure in *this* session justifies
146
+ > the textual fallback. (Origin: Emergent — a run recorded `screenshot_coverage: 0` because "`save_to_disk`
147
+ > returned no persistable path in this environment", and the harness codified that as an accepted
148
+ > fallback. In the same environment it works and returns a real path; the next run captured 13 images. An
149
+ > unverified environment claim had become a standing excuse that suppressed the most heavily-weighted
150
+ > rubric axis.)
151
+ >
152
+ > **Validate the coverage doc by READING THE FILE, not by trusting a delegated agent's return.** Because
153
+ > Mode 5 scores this file's `coverage_scorecard` frontmatter directly, a **stale** `feature-coverage.md`
154
+ > is worse than a missing one — it scores silently, against the *previous* run's numbers. Assert (a) the
155
+ > file's mtime is newer than the dispatch and (b) its frontmatter matches this run's facts
156
+ > (`write_side_observed`, `screenshot_coverage`, the credit figure). (Origin: Emergent, second run — a
157
+ > sub-agent delivered its IA doc and silently dropped this one; the untouched Pass-1 file would have
158
+ > scored a full write-side run at `screenshot_coverage: 0` / `write_side_observed: false`.)
159
+
160
+ The screenshots live under the
93
161
  `session` dimension (it owns the browser captures); `information-architecture.md`'s surface cards
94
162
  reference them by path.