@nurix/etna 0.19.40 → 0.19.42
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/data/harness-index.json +1 -1
- package/data/harnesses/code/harness.json +1 -1
- package/data/harnesses/delivery/harness.json +1 -1
- package/data/harnesses/discover/CLAUDE.md +50 -2
- package/data/harnesses/discover/agents/api.md +24 -0
- package/data/harnesses/discover/agents/codebase.md +24 -0
- package/data/harnesses/discover/agents/community.md +24 -0
- package/data/harnesses/discover/agents/deployed-client-bundle.md +24 -0
- package/data/harnesses/discover/agents/distribution-artifacts.md +24 -0
- package/data/harnesses/discover/agents/docs.md +27 -1
- package/data/harnesses/discover/agents/infra-backend-fingerprint.md +24 -0
- package/data/harnesses/discover/agents/packages.md +24 -0
- package/data/harnesses/discover/agents/session.md +40 -0
- package/data/harnesses/discover/agents/website.md +24 -0
- package/data/harnesses/discover/agents/wire-capture.md +40 -0
- package/data/harnesses/discover/codex/agents/api.toml +24 -0
- package/data/harnesses/discover/codex/agents/codebase.toml +24 -0
- package/data/harnesses/discover/codex/agents/community.toml +24 -0
- package/data/harnesses/discover/codex/agents/deployed-client-bundle.toml +24 -0
- package/data/harnesses/discover/codex/agents/distribution-artifacts.toml +24 -0
- package/data/harnesses/discover/codex/agents/docs.toml +27 -1
- package/data/harnesses/discover/codex/agents/infra-backend-fingerprint.toml +24 -0
- package/data/harnesses/discover/codex/agents/packages.toml +24 -0
- package/data/harnesses/discover/codex/agents/session.toml +40 -0
- package/data/harnesses/discover/codex/agents/website.toml +24 -0
- package/data/harnesses/discover/codex/agents/wire-capture.toml +40 -0
- package/data/harnesses/discover/harness.json +3 -4
- package/data/harnesses/discover/rules/calibration.md +219 -0
- package/data/harnesses/discover/rules/cartography-coverage.md +69 -1
- package/data/harnesses/discover/rules/discovery.md +59 -1
- package/data/harnesses/discover/rules/evaluation.md +13 -0
- package/data/harnesses/discover/rules/ingestion.md +248 -7
- package/data/harnesses/discover/rules/self-correction.md +6 -2
- package/data/harnesses/discover/rules/tooling.md +205 -0
- package/data/harnesses/discover/rules/tradecraft.md +116 -2
- package/data/harnesses/etna/harness.json +1 -1
- package/package.json +2 -2
- package/data/harnesses/discover/tooling-recommendations.md +0 -119
|
@@ -55,4 +55,28 @@ Then write the per-type raw files that apply — `raw/{extension-manifest.md, el
|
|
|
55
55
|
Return a **one-paragraph headline** of what the shipped client reveals — its declared hosts, permission surface, embedded vendor SDKs, and how much richer (or not) it is than the web app. The API paths and feature flags you appended feed the **seam rule (§6)**: your `dimensions/_shared/api-path-catalog.md` rows carry `source: distribution` so evaluation can run its adaptive diff against `deployed-client-bundle` (`source: bundle`), `session`, and `wire-capture`. The hosts and SDK list cross-check **infra-backend-fingerprint** (a host the binary calls that DNS/CT didn't surface is a finding); the runtime hosts also seed **wire-capture** targets (the binary says what it _might_ call; the capture says what it _does_).
|
|
56
56
|
|
|
57
57
|
**Discipline:** EULA before extraction — restrictive license ⇒ Open question, never silent decompile. Manifests/strings only; no protected-logic reversing. Redact every embedded key before any `raw/` write (§7 rule 3). Digest large bundles (§5). A missing artifact type is a recorded gap, not a guess.
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
|
|
62
|
+
|
|
63
|
+
**MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
|
|
64
|
+
the manifests verbatim (`manifest.json`, `Info.plist`, `AndroidManifest.xml`) and the extracted file listing.
|
|
65
|
+
|
|
66
|
+
Three rules bind this, and they override any instinct to summarise:
|
|
67
|
+
|
|
68
|
+
1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
|
|
69
|
+
count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
|
|
70
|
+
medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
|
|
71
|
+
the citation will outlive its evidence.
|
|
72
|
+
2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
|
|
73
|
+
`_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
|
|
74
|
+
essay, not a capture.
|
|
75
|
+
3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
|
|
76
|
+
straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
|
|
77
|
+
bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
|
|
78
|
+
discard evidence.
|
|
79
|
+
|
|
80
|
+
**Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
|
|
81
|
+
regardless of how good the prose is.
|
|
58
82
|
'''
|
|
@@ -32,5 +32,31 @@ You collect a target's docs into `research/<target>/dimensions/docs/`. Read `.cl
|
|
|
32
32
|
|
|
33
33
|
A one-paragraph headline (the product's documented surface in a sentence) + the section list with page counts + anything that overlaps the api/packages dimensions (flag it so synthesis can cross-check).
|
|
34
34
|
|
|
35
|
-
**Discipline:** distill
|
|
35
|
+
**Discipline:** distill **in addition to**, never instead of. Save each captured page's source form under `raw/pages/` (md or html, as fetched) — that is the evidence layer — and build the navigable map + load-bearing content as the narrative layer beside it (ingestion §5.1). A distilled section whose source page was never saved is unverifiable by the next run (§5.3).
|
|
36
|
+
|
|
37
|
+
**Do not mirror the SITE:** A 300-page docs site becomes a navigable map + the load-bearing content, not 300 verbatim files. Record paywalled/gated docs as Open questions.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
|
|
42
|
+
|
|
43
|
+
**MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
|
|
44
|
+
each captured page's source form under `raw/pages/` (md or html, as fetched).
|
|
45
|
+
|
|
46
|
+
Three rules bind this, and they override any instinct to summarise:
|
|
47
|
+
|
|
48
|
+
1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
|
|
49
|
+
count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
|
|
50
|
+
medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
|
|
51
|
+
the citation will outlive its evidence.
|
|
52
|
+
2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
|
|
53
|
+
`_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
|
|
54
|
+
essay, not a capture.
|
|
55
|
+
3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
|
|
56
|
+
straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
|
|
57
|
+
bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
|
|
58
|
+
discard evidence.
|
|
59
|
+
|
|
60
|
+
**Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
|
|
61
|
+
regardless of how good the prose is.
|
|
36
62
|
'''
|
|
@@ -61,4 +61,28 @@ Return a **one-paragraph headline** of the reconstructed backend — origin clou
|
|
|
61
61
|
**Discipline:** read-only public recon — never scan, brute-force, or hit a non-public host; respect crt.sh/WHOIS rate limits and back off; redact any credential-shaped value before writing (contract §7 rule 3 — DSNs, verification tokens, embedded keys surfaced in headers/TXT). An unresolvable host or a missing sub-processor page is an Open question, not a guess.
|
|
62
62
|
|
|
63
63
|
**Verify an absence before asserting it (contract §7 rule 10).** Before reporting any host as **NXDOMAIN / absent** — especially `api.<target>` or any host a sibling dimension (bundle / session / wire) reports actually hitting — re-resolve via a **second resolver** (`dig @1.1.1.1 <host>`, `dig @8.8.8.8 <host>`) **and** a direct `curl -sI https://<host>`. An HTTP 4xx/5xx response means the host **resolves** → report it **present**, not absent. A single negative resolver result is recorded as _tentative_, never stated as fact and never used to instruct sibling dimensions. Likewise, an _inferred_ backend engine (e.g. "looks like Hasura/Supabase from the CNAME") is phrased as an inference and yields to a runtime-observing dimension (`session`/`deployed-client-bundle`) that names the engine directly.
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
|
|
68
|
+
|
|
69
|
+
**MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
|
|
70
|
+
the raw `dig`, `crt.sh` and HTTP header responses as returned; the decoded table is the narrative layer beside them.
|
|
71
|
+
|
|
72
|
+
Three rules bind this, and they override any instinct to summarise:
|
|
73
|
+
|
|
74
|
+
1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
|
|
75
|
+
count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
|
|
76
|
+
medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
|
|
77
|
+
the citation will outlive its evidence.
|
|
78
|
+
2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
|
|
79
|
+
`_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
|
|
80
|
+
essay, not a capture.
|
|
81
|
+
3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
|
|
82
|
+
straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
|
|
83
|
+
bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
|
|
84
|
+
discard evidence.
|
|
85
|
+
|
|
86
|
+
**Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
|
|
87
|
+
regardless of how good the prose is.
|
|
64
88
|
'''
|
|
@@ -27,4 +27,28 @@ You collect a target's installable artifacts into `research/<target>/dimensions/
|
|
|
27
27
|
A one-paragraph headline (the package family in a sentence) + the package table + which packages are the core public API (so synthesis can anchor the SDK surface). Flag overlap with the docs api-reference and the codebase dimensions for cross-check.
|
|
28
28
|
|
|
29
29
|
**Discipline:** the exports map + type defs are the load-bearing artifact — prioritize them over download-count trivia. Digest huge `.d.ts` to signatures. Redact any registry tokens.
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
|
|
34
|
+
|
|
35
|
+
**MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
|
|
36
|
+
per package: the registry metadata JSON (`npm view <pkg> --json > raw/<pkg>.registry.json`, PyPI `/pypi/<pkg>/json`) and the `.d.ts` / exports surface as fetched.
|
|
37
|
+
|
|
38
|
+
Three rules bind this, and they override any instinct to summarise:
|
|
39
|
+
|
|
40
|
+
1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
|
|
41
|
+
count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
|
|
42
|
+
medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
|
|
43
|
+
the citation will outlive its evidence.
|
|
44
|
+
2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
|
|
45
|
+
`_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
|
|
46
|
+
essay, not a capture.
|
|
47
|
+
3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
|
|
48
|
+
straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
|
|
49
|
+
bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
|
|
50
|
+
discard evidence.
|
|
51
|
+
|
|
52
|
+
**Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
|
|
53
|
+
regardless of how good the prose is.
|
|
30
54
|
'''
|
|
@@ -38,4 +38,44 @@ Under `research/<target>/dimensions/session/`:
|
|
|
38
38
|
A one-paragraph headline (the auth model + transport + the single most load-bearing wire finding) + the phase coverage + which passes ran. Flag drift between the _observed_ wire surface and the _published_ api/packages surface — that drift is the highest-value synthesis input. If Pass 2 was needed but not authorized, list the open write-side questions; never fabricate request bodies.
|
|
39
39
|
|
|
40
40
|
**Discipline:** ethics first, always. Redact before writing. Confirm before any state change. No destructive actions. No exfiltration. A gap is an Open question, never a guess.
|
|
41
|
+
|
|
42
|
+
---
|
|
43
|
+
|
|
44
|
+
## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
|
|
45
|
+
|
|
46
|
+
**MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
|
|
47
|
+
≥3 verbatim envelopes as `raw/*.json` — the identity read, one canonical list read, and the primary write (request + response) — per ingestion §8.
|
|
48
|
+
|
|
49
|
+
Three rules bind this, and they override any instinct to summarise:
|
|
50
|
+
|
|
51
|
+
1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
|
|
52
|
+
count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
|
|
53
|
+
medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
|
|
54
|
+
the citation will outlive its evidence.
|
|
55
|
+
2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
|
|
56
|
+
`_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
|
|
57
|
+
essay, not a capture.
|
|
58
|
+
3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
|
|
59
|
+
straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
|
|
60
|
+
bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
|
|
61
|
+
discard evidence.
|
|
62
|
+
|
|
63
|
+
**Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
|
|
64
|
+
regardless of how good the prose is.
|
|
65
|
+
|
|
66
|
+
## Visual capture (ingestion §5.5) — screenshot generously
|
|
67
|
+
|
|
68
|
+
You drive a real browser, so you carry the visual-evidence obligation. **Capture continuously, not
|
|
69
|
+
one-image-per-surface:** every surface, plus every materially different state within it — empty vs
|
|
70
|
+
populated, modal/drawer open, dropdown expanded, error and validation states, permission/plan walls,
|
|
71
|
+
loading/in-flight, and the before/after pair around any authorized state change.
|
|
72
|
+
|
|
73
|
+
This is **not** scored — `screenshot_coverage` is a narrow Mode-5 floor. The beneficiary is the human
|
|
74
|
+
reading this corpus weeks later for a purpose nobody anticipated. The product's past state is the one thing
|
|
75
|
+
a later run genuinely cannot reproduce.
|
|
76
|
+
|
|
77
|
+
**Redact in the DOM before the shutter** (`tradecraft.md` §1c) — a leaked pixel is undetectable afterwards.
|
|
78
|
+
**Check the corners of the frame**, not just the content: support-chat widgets, toasts, presence avatars and
|
|
79
|
+
member lists routinely carry *another human's* name and message. Then look at the saved image, and record
|
|
80
|
+
`redacted: <what>` in `screens/_index.md` rather than a blanket zero-PII claim.
|
|
41
81
|
'''
|
|
@@ -33,4 +33,28 @@ You collect a target's public-facing site into `research/<target>/dimensions/web
|
|
|
33
33
|
A one-paragraph headline (the positioning in a sentence) + the pricing tiers + the headline feature claims. Flag claims that should be cross-checked against the codebase/docs (marketing often over-claims) so synthesis can reconcile.
|
|
34
34
|
|
|
35
35
|
**Discipline:** quote positioning verbatim; capture pricing precisely (numbers + limits); separate _claims_ from verified capability — the reconciliation happens in synthesis. Respect robots.txt and rate limits; this is read-only public crawling, not scraping at scale.
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
|
|
40
|
+
|
|
41
|
+
**MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
|
|
42
|
+
the raw HTML or extracted text per captured page, and the pricing table as structured data.
|
|
43
|
+
|
|
44
|
+
Three rules bind this, and they override any instinct to summarise:
|
|
45
|
+
|
|
46
|
+
1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
|
|
47
|
+
count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
|
|
48
|
+
medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
|
|
49
|
+
the citation will outlive its evidence.
|
|
50
|
+
2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
|
|
51
|
+
`_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
|
|
52
|
+
essay, not a capture.
|
|
53
|
+
3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
|
|
54
|
+
straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
|
|
55
|
+
bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
|
|
56
|
+
discard evidence.
|
|
57
|
+
|
|
58
|
+
**Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
|
|
59
|
+
regardless of how good the prose is.
|
|
36
60
|
'''
|
|
@@ -42,4 +42,44 @@ Then the captures + raw artifacts (per `discovery` Part C (wire-capture)):
|
|
|
42
42
|
Return a **one-paragraph headline** of what the wire reveals — the real hosts contacted, the envelope/auth scheme, and the rung used (so the reader knows how much to trust it). Then a compact table of observed endpoints. **Flag the seam rule (§6):** every `api-path-catalog.md` row is `source: wire`, so evaluation can diff _actually-called_ (wire) against _statically-declared_ (`bundle` / `distribution-artifacts`) and _browser-observed_ (`session`) — a host or path the binary declares but the wire never hits is a finding, not noise; the inverse (called but undeclared) is a stronger one. If only rung 1 was authorized, say so and name what rungs 2/3 would unlock so the main session can request authorization.
|
|
43
43
|
|
|
44
44
|
**Discipline:** lowest privilege first; never self-escalate the ladder; never install a CA or tap third-party traffic; never run Wireshark; redact every body before disk and log it in `_meta`. An unauthorized higher rung is an Open question, not an excuse to guess what the wire would show.
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
## Dump manifest (ingestion §5.3–5.4) — evidence before narrative
|
|
49
|
+
|
|
50
|
+
**MUST land in `raw/` as verbatim, redacted files before this dimension may be `status: complete`:**
|
|
51
|
+
the flow log itself (HAR or the tap's JSON export), not only the decoded narrative.
|
|
52
|
+
|
|
53
|
+
Three rules bind this, and they override any instinct to summarise:
|
|
54
|
+
|
|
55
|
+
1. **If you cite it, dump it (§5.3).** Every value that reaches your `_summary.md` — a version, a path, a
|
|
56
|
+
count, a price, a field name — must be traceable to a file on disk. **The session is not a storage
|
|
57
|
+
medium:** a number you extracted from a response you did not save is unverifiable by the next run, and
|
|
58
|
+
the citation will outlive its evidence.
|
|
59
|
+
2. **Digest IN ADDITION, never INSTEAD (§5.1).** `raw/` is the evidence layer and may be unreadable;
|
|
60
|
+
`_summary.md` is the narrative layer. A readable digest whose source artifact was never written is an
|
|
61
|
+
essay, not a capture.
|
|
62
|
+
3. **Size is a routing problem (§5.2).** Never read a large artifact into context to save it — pipe it
|
|
63
|
+
straight to disk (`curl -o`, `| gzip >`, or the clipboard channel for in-page captures). If you must
|
|
64
|
+
bound an unbounded source, record the bound in `_meta.sampling`. Context budget is never a reason to
|
|
65
|
+
discard evidence.
|
|
66
|
+
|
|
67
|
+
**Self-check before returning:** count the verbatim files in `raw/`. Zero ⇒ the dimension is `partial`,
|
|
68
|
+
regardless of how good the prose is.
|
|
69
|
+
|
|
70
|
+
## Visual capture (ingestion §5.5) — screenshot generously
|
|
71
|
+
|
|
72
|
+
You drive a real browser, so you carry the visual-evidence obligation. **Capture continuously, not
|
|
73
|
+
one-image-per-surface:** every surface, plus every materially different state within it — empty vs
|
|
74
|
+
populated, modal/drawer open, dropdown expanded, error and validation states, permission/plan walls,
|
|
75
|
+
loading/in-flight, and the before/after pair around any authorized state change.
|
|
76
|
+
|
|
77
|
+
This is **not** scored — `screenshot_coverage` is a narrow Mode-5 floor. The beneficiary is the human
|
|
78
|
+
reading this corpus weeks later for a purpose nobody anticipated. The product's past state is the one thing
|
|
79
|
+
a later run genuinely cannot reproduce.
|
|
80
|
+
|
|
81
|
+
**Redact in the DOM before the shutter** (`tradecraft.md` §1c) — a leaked pixel is undetectable afterwards.
|
|
82
|
+
**Check the corners of the frame**, not just the content: support-chat widgets, toasts, presence avatars and
|
|
83
|
+
member lists routinely carry *another human's* name and message. Then look at the saved image, and record
|
|
84
|
+
`redacted: <what>` in `screens/_index.md` rather than a blanket zero-PII claim.
|
|
45
85
|
'''
|
|
@@ -1,12 +1,11 @@
|
|
|
1
1
|
{
|
|
2
2
|
"harness": "discover",
|
|
3
3
|
"name": "discover",
|
|
4
|
-
"version": "0.1.
|
|
4
|
+
"version": "0.1.14",
|
|
5
5
|
"files": [
|
|
6
6
|
".prettierignore",
|
|
7
7
|
".prettierrc.json",
|
|
8
|
-
".vscode/settings.json"
|
|
9
|
-
"tooling-recommendations.md"
|
|
8
|
+
".vscode/settings.json"
|
|
10
9
|
],
|
|
11
|
-
"digest": "
|
|
10
|
+
"digest": "4ab9f67f0d67"
|
|
12
11
|
}
|
|
@@ -0,0 +1,219 @@
|
|
|
1
|
+
# Calibration Reference — Blind Re-runs · Disposition · The Fleet Ledger
|
|
2
|
+
|
|
3
|
+
The reference for the harness's **only external signal**. Everything else in this harness is self-assessment:
|
|
4
|
+
collectors grade their own `confidence` and `completeness_pct`, Evaluation weights those self-grades, and
|
|
5
|
+
Mode 5 scores the result against a checklist the same system authored. That loop can converge on a corpus
|
|
6
|
+
that is confidently, well-provenanced, and wrong.
|
|
7
|
+
|
|
8
|
+
**Calibration is the one procedure that can tell you whether any of it is true.** It re-derives claims a
|
|
9
|
+
prior run published, blind, and records what survived.
|
|
10
|
+
|
|
11
|
+
> **The measured position, as of 2026-07-28.** One target has ever been re-run (`emergent`, 2026-06-25 →
|
|
12
|
+
> 2026-07-28). Of 16 claims re-tested: **7 refuted, 4 corrected, 5 confirmed** — a 44% flat-falsification
|
|
13
|
+
> rate — and **4 of the 7 refutations came from lanes the prior run had graded `confidence: high`.** The
|
|
14
|
+
> prior run had self-scored **88/100 "strong"** and `stopped_on: converged`.
|
|
15
|
+
>
|
|
16
|
+
> That is **n=1**. It is the entire empirical basis of this harness. Every rule in `.claude/rules/` is
|
|
17
|
+
> currently a hypothesis with n=0 on any target it did not come from. The purpose of this file is to make
|
|
18
|
+
> n grow.
|
|
19
|
+
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## Two tiers — pick the one that will actually happen
|
|
23
|
+
|
|
24
|
+
A metric too expensive to run leaves `n` at 1 forever. The cheap tier is the important one.
|
|
25
|
+
|
|
26
|
+
| Tier | Cadence | Cost | What it measures |
|
|
27
|
+
| --- | --- | --- | --- |
|
|
28
|
+
| **A — Claim spot-check** | monthly | ~1 session, no full ingestion | the refutation rate, and whether `confidence` tracks truth |
|
|
29
|
+
| **B — Full blind re-run** | quarterly, or when a rule set changes materially | a complete run | everything Tier A does, plus coverage drift and whether new rules changed behaviour |
|
|
30
|
+
|
|
31
|
+
**Tier A is the default. Do not skip it waiting for capacity to do Tier B** — a monthly spot-check of 15–20
|
|
32
|
+
claims across one target produces the regression that matters, and produces it twelve times a year.
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## Selection — rotate, never choose
|
|
37
|
+
|
|
38
|
+
**The target is selected by rotation, not by judgment.** Take the least-recently-calibrated target in the
|
|
39
|
+
corpus root; break ties by oldest `captured_at`. Record the selection rule alongside the result.
|
|
40
|
+
|
|
41
|
+
A human (or a run) choosing which target to re-check will choose the one it feels good about. That single
|
|
42
|
+
substitution destroys the metric, because the sample stops being representative of the fleet and starts
|
|
43
|
+
being representative of the harness's confidence — which is the very thing under test.
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## THE BLIND RULE (this is what makes it evidence)
|
|
48
|
+
|
|
49
|
+
**Capture first. Read the prior corpus only after your own observation is written down.**
|
|
50
|
+
|
|
51
|
+
1. Select the target. **Do not open its corpus.**
|
|
52
|
+
2. Draw the claim sample (below) — a list of *questions*, not answers. Someone or something other than the
|
|
53
|
+
re-deriving agent extracts the questions, or the questions are extracted mechanically from claim
|
|
54
|
+
anchors, so the sample carries no prior conclusions.
|
|
55
|
+
3. **Re-derive each claim from the live product**, from scratch, recording what you observe and the method.
|
|
56
|
+
4. **Only then** open the prior corpus and diff.
|
|
57
|
+
|
|
58
|
+
> **This deliberately overrides `discovery.md` Step 00 for calibration runs.** Step 00 tells a run to find a
|
|
59
|
+
> prior corpus, restore it, and reconcile against it — correct for a normal re-run, where the prior corpus
|
|
60
|
+
> is a genuine corroboration lane and re-deriving everything from zero is waste. **It is wrong here.** A run
|
|
61
|
+
> that reads the prior claim first will anchor on it: it will look for confirmation, find it, and report a
|
|
62
|
+
> confirmation rate that measures its own suggestibility. Anchoring is the specific failure this protocol
|
|
63
|
+
> exists to avoid, so for a calibration run Step 00 is deferred until after step 3, and the run records
|
|
64
|
+
> `blind: true`.
|
|
65
|
+
>
|
|
66
|
+
> If a run cannot honestly attest that it re-derived before reading, it must record `blind: false` — and
|
|
67
|
+
> the result is then a **consistency check, not a calibration**, and does not enter the ledger's refutation
|
|
68
|
+
> rate.
|
|
69
|
+
|
|
70
|
+
**Sampling the claims.** 15–20 per target, drawn to make the regression computable, not to be flattering:
|
|
71
|
+
|
|
72
|
+
- **Stratify by the prior run's stated `confidence`** — roughly equal numbers of `high`, `medium`, `low`.
|
|
73
|
+
Without this the regression cannot be computed, and the regression is the point.
|
|
74
|
+
- **Include the load-bearing ones** — claims a README headline rests on, and claims a reader would act on
|
|
75
|
+
(pricing, limits, guarantees, auth model, what the product can and cannot do).
|
|
76
|
+
- **Include at least 3 negative claims** ("no public SDK", "no realtime layer", "not supported"). Negatives
|
|
77
|
+
are the harness's demonstrated weak spot: **5 of the 7 refutations in the one data point were negatives.**
|
|
78
|
+
- **Exclude** anything whose truth cannot change and cannot be re-checked (a historical funding round).
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## Disposition — five values, and the fifth is the one the old schema lacked
|
|
83
|
+
|
|
84
|
+
Every sampled claim resolves to exactly one:
|
|
85
|
+
|
|
86
|
+
| Disposition | Meaning | What it says about the harness |
|
|
87
|
+
| --- | --- | --- |
|
|
88
|
+
| **confirmed** | independently re-derived, same conclusion | the method worked |
|
|
89
|
+
| **corrected** | right direction, wrong detail (a count, a version, a name) | the method worked; the reading was imprecise |
|
|
90
|
+
| **refuted** | the claim was **wrong when it was made** | **the method failed — this is the signal** |
|
|
91
|
+
| **stale** | the claim was **true when made; the product changed** | the method worked; the corpus has a shelf life |
|
|
92
|
+
| **unverifiable** | cannot be re-checked now (surface gone, plan changed, gated) | no signal either way; excluded from the rate |
|
|
93
|
+
|
|
94
|
+
> **`refuted` and `stale` are the distinction the original artifact had no slot for, and they mean opposite
|
|
95
|
+
> things.** A high `refuted` rate means *the instrument is unreliable* and the rules need work. A high
|
|
96
|
+
> `stale` rate means *the instrument is fine and the corpus decays* — which calls for shorter shelf-lives
|
|
97
|
+
> and `volatility` tagging, not methodology changes. Conflating them makes the harness fix the wrong
|
|
98
|
+
> problem. When you cannot tell which it is — no dated evidence either way — record **`refuted`**. The
|
|
99
|
+
> conservative reading is the one that keeps the harness honest about itself.
|
|
100
|
+
>
|
|
101
|
+
> (Origin: `emergent/00-prior-run-reconciliation.md` — a genuinely good artifact whose section A had to be
|
|
102
|
+
> titled *"REFUTED (prior claim was wrong, not merely stale)"* precisely because the schema gave it nowhere
|
|
103
|
+
> else to put the distinction.)
|
|
104
|
+
|
|
105
|
+
**The refutation rate** = `refuted ÷ (confirmed + corrected + refuted + stale)`. `unverifiable` is excluded
|
|
106
|
+
from the denominator, and its count is reported so a shrinking sample is visible.
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
## The regression — the actual question
|
|
111
|
+
|
|
112
|
+
For each calibration, record the disposition **against the prior run's stated confidence for that claim**:
|
|
113
|
+
|
|
114
|
+
| prior confidence | n | confirmed | corrected | refuted | stale | refutation rate |
|
|
115
|
+
| --- | --- | --- | --- | --- | --- | --- |
|
|
116
|
+
| high | | | | | | |
|
|
117
|
+
| medium | | | | | | |
|
|
118
|
+
| low | | | | | | |
|
|
119
|
+
|
|
120
|
+
**If `high` claims are refuted at a similar rate to `low` claims, the `confidence` field carries no signal
|
|
121
|
+
and `ingestion.md` §3's method→confidence table is decorative.** That is the single most consequential thing
|
|
122
|
+
this harness could learn about itself, and it is unknowable without this table.
|
|
123
|
+
|
|
124
|
+
Also record refutation rate **by `method`** once n is large enough — that tells you which of the 17 methods
|
|
125
|
+
in §3 are over-graded, which is directly actionable (a method's band gets lowered).
|
|
126
|
+
|
|
127
|
+
---
|
|
128
|
+
|
|
129
|
+
## Artifacts
|
|
130
|
+
|
|
131
|
+
### Per calibration — `research/<target>/00-prior-run-reconciliation.md`
|
|
132
|
+
|
|
133
|
+
Formalizes the shape `emergent` invented. **This file is part of the output contract** whenever a prior
|
|
134
|
+
corpus exists.
|
|
135
|
+
|
|
136
|
+
```markdown
|
|
137
|
+
---
|
|
138
|
+
target: <slug>
|
|
139
|
+
reconciled_at: <YYYY-MM-DD>
|
|
140
|
+
prior_run_date: <YYYY-MM-DD>
|
|
141
|
+
prior_run_commit: <sha>
|
|
142
|
+
prior_run_score: <n>
|
|
143
|
+
tier: A | B # spot-check or full blind re-run
|
|
144
|
+
blind: true | false # false ⇒ consistency check only, excluded from the rate
|
|
145
|
+
selection_rule: rotation # or: named-by-user (record it either way)
|
|
146
|
+
sampled: <n>
|
|
147
|
+
confirmed: <n>
|
|
148
|
+
corrected: <n>
|
|
149
|
+
refuted: <n>
|
|
150
|
+
stale: <n>
|
|
151
|
+
unverifiable: <n>
|
|
152
|
+
refutation_rate: <0-100>
|
|
153
|
+
by_confidence: # the regression — REQUIRED
|
|
154
|
+
high: { n: <n>, refuted: <n> }
|
|
155
|
+
medium: { n: <n>, refuted: <n> }
|
|
156
|
+
low: { n: <n>, refuted: <n> }
|
|
157
|
+
---
|
|
158
|
+
|
|
159
|
+
## A. REFUTED — wrong when made (claim · what was observed · lane that refuted it · prior confidence)
|
|
160
|
+
## B. STALE — true when made, changed (claim · what changed · evidence of the change being NEW)
|
|
161
|
+
## C. CORRECTED — right, imprecise
|
|
162
|
+
## D. CONFIRMED — independently re-derived
|
|
163
|
+
## E. UNVERIFIABLE — and why
|
|
164
|
+
## F. What this says about the METHOD (which lanes/methods produced the refutations, and whether a
|
|
165
|
+
§3 confidence band should be lowered)
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
### Fleet-level — `research/_calibration/ledger.md`
|
|
169
|
+
|
|
170
|
+
**The metric itself.** One row per calibration, accumulating across targets and time. This is the file to
|
|
171
|
+
read when asking "does this harness work?" It is never rewritten, only appended to.
|
|
172
|
+
|
|
173
|
+
---
|
|
174
|
+
|
|
175
|
+
## What a result CHANGES (otherwise this is just bookkeeping)
|
|
176
|
+
|
|
177
|
+
A calibration that produces a number and no action is theatre. Each outcome routes:
|
|
178
|
+
|
|
179
|
+
| Finding | Action |
|
|
180
|
+
| --- | --- |
|
|
181
|
+
| A **method** shows a high refutation rate | lower its band in `ingestion.md` §3, and record the calibration as the origin |
|
|
182
|
+
| **Negative claims** refute disproportionately | tighten the absence rules (`ingestion.md` §7 rule 10, `discovery.md` Part B) — the `blind_to:` direction |
|
|
183
|
+
| `high` refutes ≈ as often as `low` | **the confidence field is not carrying signal** — this is a structural finding about the harness, and it goes to the user, not into a rule tweak |
|
|
184
|
+
| A **rule added after the prior run** would have prevented a refutation | evidence the improvement loop works — record it, it is the only such evidence that exists |
|
|
185
|
+
| A refutation recurs **after** a rule was added to prevent it | evidence a rule did **not** change behaviour; escalate to the user rather than adding another rule |
|
|
186
|
+
| High **stale** rate | shorten shelf-life expectations and tag volatile claims; **not** a methodology problem |
|
|
187
|
+
|
|
188
|
+
---
|
|
189
|
+
|
|
190
|
+
## How to run one (Tier A, the monthly default)
|
|
191
|
+
|
|
192
|
+
1. **Select** by rotation from `research/_calibration/ledger.md`. Record the target and the rule.
|
|
193
|
+
2. **Extract the sample** mechanically — pull claim anchors from the target's `evaluation/*.md`, stratify by
|
|
194
|
+
the cited confidence, and write them as **questions with the answers stripped**.
|
|
195
|
+
3. **Re-derive blind.** Do not open the corpus. Use whatever lanes the access vector allows; record method
|
|
196
|
+
and observation per claim.
|
|
197
|
+
4. **Diff and disposition** each claim into the five values.
|
|
198
|
+
5. **Write** `00-prior-run-reconciliation.md` (append a new dated section if one already exists — never
|
|
199
|
+
overwrite a prior calibration).
|
|
200
|
+
6. **Append one row** to the fleet ledger, and update its running totals.
|
|
201
|
+
7. **Route** every finding through the table above. Anything that becomes a rule change goes through the
|
|
202
|
+
normal Mode-5 approval gate.
|
|
203
|
+
|
|
204
|
+
**Cost/state discipline is unchanged:** a calibration is read-only unless the user authorizes otherwise; a
|
|
205
|
+
claim that can only be re-derived by spending money or mutating state is recorded `unverifiable` with the
|
|
206
|
+
reason, never quietly exercised.
|
|
207
|
+
|
|
208
|
+
---
|
|
209
|
+
|
|
210
|
+
## How the harness uses this reference
|
|
211
|
+
|
|
212
|
+
Calibration is a **cross-run process**, not a mode of a single run — nothing in Modes 1–5 triggers it, and it
|
|
213
|
+
must not be folded into them (a run cannot calibrate itself; that is the circularity it exists to break).
|
|
214
|
+
Mode 5 reads the fleet ledger as *context* when weighing a proposal — a rule contradicted by calibration
|
|
215
|
+
data should not be promoted — but Mode 5 never writes to it.
|
|
216
|
+
|
|
217
|
+
> **Do not attach calibration to the Mode-5 score.** The moment `refutation_rate` becomes a rubric input,
|
|
218
|
+
> the run acquires an interest in the number, and the one uncorrupted signal in the system is gone. It
|
|
219
|
+
> measures the **harness**, not the run. Publish it; never grade with it.
|
|
@@ -33,6 +33,40 @@ differently:
|
|
|
33
33
|
| **edition/plan-gated** | not in this tenant's edition/plan (e.g. a trial) | a recorded gap — **not** re-walked (re-walking can't find what isn't shipped here) |
|
|
34
34
|
| **roadmap-not-shipped** | marketed but not yet a shipped capability | a recorded gap |
|
|
35
35
|
| **marketing over-claim** | the claim overstates what the product does | an `evaluation/product-features.md` **over-claim flag** |
|
|
36
|
+
| **not-yet-reachable** | it exists and is shipped, but the surface **does not render at all** until a state change (deploy / connect / upgrade / first-run) that this run did not perform | the **gated re-walk queue** — NOT the read-only auto-loop |
|
|
37
|
+
|
|
38
|
+
### `not-yet-reachable` vs `deeper-than-looked` — the distinction that matters
|
|
39
|
+
|
|
40
|
+
These two look identical in a coverage matrix and route **oppositely**, so get it right:
|
|
41
|
+
|
|
42
|
+
- **`deeper-than-looked`** — the surface **is in the DOM right now**; the run just didn't navigate to it.
|
|
43
|
+
More read-only walking finds it. Routes to the **read-only re-walk queue**, which Mode 5's auto-loop
|
|
44
|
+
works for free.
|
|
45
|
+
- **`not-yet-reachable`** — the surface **is not in the product yet.** No amount of navigation reveals it,
|
|
46
|
+
because the affordance is created by a state change. Routes to a **separate gated queue** that the
|
|
47
|
+
auto-loop must **not** touch: closing it costs money, mutates state, or needs a third-party consent, so
|
|
48
|
+
it goes through the explicit-confirmation gate (ingestion §7 rule 4) or is recorded as a standing gap.
|
|
49
|
+
|
|
50
|
+
**Mis-filing a `not-yet-reachable` row as `deeper-than-looked` is a Cartography defect**, because it sends
|
|
51
|
+
the auto-loop hunting for something that does not exist, and it reads as "we were sloppy" when the honest
|
|
52
|
+
statement is "the product's IA is state-dependent and this run stayed inside its authorization."
|
|
53
|
+
|
|
54
|
+
**A run that never performs the state change cannot map a state-dependent IA — say so rather than
|
|
55
|
+
scoring it as a coverage miss.** Note it beside the scorecard the same way `experiential_gated` annotates
|
|
56
|
+
a policy deferral: it documents, it does not inflate.
|
|
57
|
+
|
|
58
|
+
> (Origin: Emergent — a deployment executed mid-run **created** two whole surfaces: the entire *Manage
|
|
59
|
+
> Publishing* panel with five sub-tabs, and the `db_tool` database-console affordance, which opened a
|
|
60
|
+
> separately-hosted plugin on a second auth system. Neither exists in the toolbar of an undeployed project.
|
|
61
|
+
> The pre-deploy coverage doc had filed the publish family under *deeper-than-looked* and queued it for a
|
|
62
|
+
> read-only re-walk that could never have succeeded. Note also the **two-factor gate**: the cohort flag
|
|
63
|
+
> `db_tool: true` was necessary but **not sufficient** — an entitlement flag reading `true` does not mean
|
|
64
|
+
> the surface is reachable.)
|
|
65
|
+
|
|
66
|
+
**Where this bites generally:** any PaaS or marketplace with deploy/publish-gated management surfaces; any
|
|
67
|
+
product whose integrations render only after an OAuth connect; any tier-gated admin console; any
|
|
68
|
+
first-run/onboarding-gated dashboard. Treat an entitlement flag as **necessary, not sufficient** — check
|
|
69
|
+
whether the *state* it depends on has also been reached.
|
|
36
70
|
|
|
37
71
|
---
|
|
38
72
|
|
|
@@ -50,6 +84,7 @@ coverage_scorecard:
|
|
|
50
84
|
ia_surfaces_mapped: <N> # surface cards written in information-architecture.md
|
|
51
85
|
ia_nav_complete: <true|false> # every top-level nav destination has a surface card
|
|
52
86
|
screenshot_coverage: <0-100> # primary surfaces with a screenshot ÷ all primary surfaces
|
|
87
|
+
features_not_yet_reachable: <N> # shipped, but behind a state change this run did not perform
|
|
53
88
|
---
|
|
54
89
|
## Coverage matrix
|
|
55
90
|
| Feature (claimed) | Source | Located? | Route · depth | Walked? | Edition-gated? | Confidence | Note |
|
|
@@ -57,6 +92,9 @@ coverage_scorecard:
|
|
|
57
92
|
| <feature> | website,docs | ✅ | #/…/x · D2 | ❌ not-walked | — | high | promoted on site, 2 levels deep, not exercised this run |
|
|
58
93
|
| <feature> | website(AI) | ⚠ edition? | — | ❌ | maybe-not-in-trial | low | marketed flagship, not located in this tenant |
|
|
59
94
|
## Located-but-not-walked (the within-run re-walk queue — Mode 5 walks these read-only, no cost)
|
|
95
|
+
## Not-yet-reachable (the GATED queue — shipped surfaces behind a deploy/connect/upgrade this run
|
|
96
|
+
did not perform. Mode 5 must NOT auto-walk these; each names the exact state
|
|
97
|
+
change that would unlock it and its cost/consent class)
|
|
60
98
|
## Claimed-but-not-located (each row dispositioned per the taxonomy above — NEVER left blank)
|
|
61
99
|
## Buried flagships (marketing-promoted features sitting ≥D3 — the positioning-vs-product gap,
|
|
62
100
|
drawn from information-architecture.md's promoted-vs-buried headline)
|
|
@@ -69,6 +107,10 @@ coverage_scorecard:
|
|
|
69
107
|
experiential / IA coverage axis (`self-correction.md` rubric, axis 2, max 20) is reproducible:
|
|
70
108
|
|
|
71
109
|
- `feature_location_rate` = `100 × features_located ÷ features_claimed`.
|
|
110
|
+
- **`not-yet-reachable` rows stay in the `features_claimed` denominator** — they are real, shipped,
|
|
111
|
+
unlocated capabilities and hiding them would inflate the rate. But they are **dispositioned**, so they
|
|
112
|
+
never draw the −2 un-dispositioned penalty, and the scorecard carries `features_not_yet_reachable` so a
|
|
113
|
+
reader can tell a *policy/authorization* boundary from a *thoroughness* failure.
|
|
72
114
|
- `flow_coverage` = `100 × (primary journeys diagrammed in ux-flows.md ÷ journeys identified)`.
|
|
73
115
|
- `screenshot_coverage` = `100 × (primary surfaces with a screenshot ÷ all primary surfaces)`.
|
|
74
116
|
- `ia_nav_complete` = every D0 nav destination has a surface card.
|
|
@@ -81,6 +123,13 @@ un-dispositioned. The `located-but-not-walked` list is the re-walk queue Mode 5
|
|
|
81
123
|
|
|
82
124
|
## Screenshots — `dimensions/session/captures/screens/`
|
|
83
125
|
|
|
126
|
+
> **The scorecard metric is a FLOOR, not the capture rule.** `screenshot_coverage` asks a narrow question —
|
|
127
|
+
> did each *primary surface* get one image — because Mode 5 needs something countable. **The actual capture
|
|
128
|
+
> obligation is `ingestion.md` §5.5: screenshot generously, including states within a surface, for the human
|
|
129
|
+
> who reads this corpus later.** Satisfying this metric and stopping is the failure mode; the metric is
|
|
130
|
+
> nearly always already met by a real walk. Nothing here caps how much you capture, and extra images never
|
|
131
|
+
> hurt the score.
|
|
132
|
+
|
|
84
133
|
One annotated screenshot per **primary** surface (each D0 nav destination + each materially distinct
|
|
85
134
|
detail / editor surface). **Redact visible PII before saving** (user names, emails, customer data,
|
|
86
135
|
partner identifiers). Name `<nn>-<surface>.png`; write a one-line `screens/_index.md` mapping each file
|
|
@@ -89,6 +138,25 @@ partner identifiers). Name `<nn>-<surface>.png`; write a one-line `screens/_inde
|
|
|
89
138
|
**Screenshots are the Mode-4 deliverable that the session playbook's textual surface-map only
|
|
90
139
|
_substitutes_ for** (ingestion §9.5) — capture them live here; fall back to the textual map (and record
|
|
91
140
|
the `screenshot_coverage` gap) only when the runtime genuinely cannot screenshot (no image tool, or the
|
|
92
|
-
app never reaches page-idle because of a persistent connection).
|
|
141
|
+
app never reaches page-idle because of a persistent connection).
|
|
142
|
+
|
|
143
|
+
> **Re-test the capability before recording `screenshot_coverage: 0` — an environment limitation claimed
|
|
144
|
+
> by a PRIOR run is `verify:`, never inherited.** Take one screenshot with `save_to_disk: true` in the
|
|
145
|
+
> current session and record the literal tool result. Only an actual failure in *this* session justifies
|
|
146
|
+
> the textual fallback. (Origin: Emergent — a run recorded `screenshot_coverage: 0` because "`save_to_disk`
|
|
147
|
+
> returned no persistable path in this environment", and the harness codified that as an accepted
|
|
148
|
+
> fallback. In the same environment it works and returns a real path; the next run captured 13 images. An
|
|
149
|
+
> unverified environment claim had become a standing excuse that suppressed the most heavily-weighted
|
|
150
|
+
> rubric axis.)
|
|
151
|
+
>
|
|
152
|
+
> **Validate the coverage doc by READING THE FILE, not by trusting a delegated agent's return.** Because
|
|
153
|
+
> Mode 5 scores this file's `coverage_scorecard` frontmatter directly, a **stale** `feature-coverage.md`
|
|
154
|
+
> is worse than a missing one — it scores silently, against the *previous* run's numbers. Assert (a) the
|
|
155
|
+
> file's mtime is newer than the dispatch and (b) its frontmatter matches this run's facts
|
|
156
|
+
> (`write_side_observed`, `screenshot_coverage`, the credit figure). (Origin: Emergent, second run — a
|
|
157
|
+
> sub-agent delivered its IA doc and silently dropped this one; the untouched Pass-1 file would have
|
|
158
|
+
> scored a full write-side run at `screenshot_coverage: 0` / `write_side_observed: false`.)
|
|
159
|
+
|
|
160
|
+
The screenshots live under the
|
|
93
161
|
`session` dimension (it owns the browser captures); `information-architecture.md`'s surface cards
|
|
94
162
|
reference them by path.
|