@tyroneross/build-loop 0.36.0 → 0.43.0

package/skills/handoff/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: handoff
 description: "Compose a complete, durable build-loop handoff document from the current run state, and optionally launch a fresh session with it injected. Use when crossing a context boundary (context limit, planned restart, worktree GC). Triggers: 'hand off', 'handoff', 'new session', 'context limit', 'restart', 'fresh session', '/build-loop:compose-handoff'."
-user-invocable: true
+user-invocable: false
 ---
 
 <!-- SPDX-FileCopyrightText: 2025-2026 Tyrone Ross, Jr <46267523+tyroneross@users.noreply.github.com> | SPDX-License-Identifier: Apache-2.0 -->
@@ -18,20 +18,154 @@ then (optionally) launch a fresh session in the STABLE checkout with that docume
 - A worktree about to be GC'd — extract state before the folder disappears.
 - Handing off work to a peer agent or co-developer.
 
+## The seven content classes a handoff must carry
+
+Derived empirically (2026-07-26): a generated handoff was cold-read by fresh agents at
+three model tiers, scored, repaired, and re-tested. Every failure fell into one of seven
+classes. A handoff missing any one of them produces an agent that can *restate* the work
+but cannot *act* on it. Ranked by how much damage the omission causes.
+
+| # | Class | The question it answers | Failure when missing |
+|---|-------|------------------------|----------------------|
+| 1 | **Orientation** | What is this product, for whom, on what stack? | Reviewer inferred "Apple app" only from `xcodebuild` in a gotcha. Everything downstream is guesswork. |
+| 2 | **Constraints / non-goals** | What must I NOT do? | The single most-cited gap. This is the class truncation kills first, because non-goals sit at the END of an intent file. |
+| 3 | **Landmines** | What will bite me on my FIRST action? | Stale run-id markers, set push-holds, emptied state blocks. Each fires before any real work begins. |
+| 4 | **Authorization** | Am I allowed to just start? | Absent this, an agent picks the most visible queue item — which may be another run's unpaid audit debt. |
+| 5 | **Verification recipe** | How do I build/test, and what must I never run? | Without it an agent reaches for the obvious tool and gets a false green. |
+| 6 | **Open decisions** | What is genuinely undecided, who owns it, what decides it? | A dangling "may have been wrong" with no owner and no criterion is unresolvable by the next agent. |
+| 7 | **Provenance** | How trustworthy is this record itself? | A reconstructed or overridden record read as ground truth is worse than no record. |
+
+Two cross-cutting rules learned the same way:
+
+- **Never truncate a class-2 or class-6 section.** Both live at the end of their source
+  files, so any line cap removes exactly the text that carries the constraint.
+- **Counts are claims.** A queue count that silently caps (or counts a derived `INDEX.md`
+  instead of `items/`) understates open work. Titles alone are not enough either — carry
+  the frontmatter fields that decide whether an item is safe to pick up
+  (`status`, `classify`, `judgment_verdict`, `owed_layers`, `blocked_by`).
+
+## Recall and confidence are different failures with different fixes
+
+This is the finding that mattered most, and it is not obvious: **an agent can score full
+marks on recall and still refuse to act.**
+
+Measured on one document across four rounds. Haiku answered 10/10 factual questions
+correctly while rating its own confidence **2/5** — it had every fact and would not touch
+the repo. Adding more facts would not have helped, because facts were never the gap.
+
+| Failure | Caused by | Symptom | Fix |
+|---|---|---|---|
+| **Recall** | Omission — truncation, silent caps, missing orientation | "NOT IN DOC", wrong counts, guessing the tech stack | Put the content in (classes 1–5) |
+| **Confidence** | Unresolved implication — a flag with no interpretation | Correct answers, low self-rating, "I'd have to read the code first" | State what each flag *means for the reader* (classes 6–7) |
+
+Concretely, what moved Haiku from 2/5 to 5/5 was not new information. It was:
+
+- Queue items carrying `judgment_verdict` / `classify` / `owed_layers`, so "Judgment owed —
+  bl-…-codex-299759" became something a reader could triage instead of fear.
+- The open decision gaining an **owner** and a **criterion for deciding**, so it read as
+  assigned rather than abandoned.
+- The override gaining an explicit **cost** ("runs zero tests") and **bound** (what it does
+  and does not invalidate), so it read as disclosed rather than alarming.
+
+**Rule: every warning in a handoff must carry its own interpretation.** A flag without a
+"so what" transfers anxiety, not information — and a cautious reader responds by doing
+nothing, which is the exact outcome the handoff exists to prevent.
+
+## How to test a handoff (the cold-read protocol)
+
+Do not self-assess a handoff. Dispatch a fresh agent whose ONLY artifact is the document
+path, tell it to open nothing else, and score it. Brief it and the test is worthless.
+
+Run the same fixed question set at **two model tiers**, because they detect different
+defects:
+
+| Tier | Detects | Why |
+|---|---|---|
+| **Weakest available** (Haiku) | Unresolved implications | It will not infer past an ambiguity; low confidence with correct answers pinpoints class 6/7 gaps |
+| **Strongest available** (Opus) | Internal contradictions | It cross-references sections and finds claims that cannot both be true |
+
+Ask for `CONFIDENCE (1-5)`, `GAPS`, and `CONTRADICTIONS` explicitly, and instruct the
+agent to answer "NOT IN DOC" rather than guess — otherwise general knowledge silently
+fills holes and the doc scores better than it deserves. Tell it to be harsh; a comfortable
+review is a useless one.
+
+Question set that surfaced every defect found (adapt the nouns, keep the shapes): what is
+this product and for whom · what platforms and storage · what was just done and is it
+finished · what builds/tests it and what must you never run · what is <the largest open
+item> and is it done · name three forbidden things · how many queue items and which are
+audit debt · did this ship through a normal gate · what decision is owed, who owns it,
+what decides it · which field proves no new run started and which must you not read · why
+is this record untrustworthy · what is the trap when searching for callers · are you
+authorized to start · what must you run after adding a file · who wrote this record and
+when relative to the work.
+
+**Expect the strong tier to challenge your evidence, not just your prose.** On the final
+round Opus accepted every fact and still flagged that a "three platforms build clean"
+claim was overstated — the third platform's target excluded the directory the changed
+files lived in, so its green proved the change could not break it, not that it was
+verified. That correction came from the review, not from the author.
+
 ## What it composes
 
-The handoff doc has eight fixed sections (always the same order; absent data renders as "n/a"):
+Nine fixed sections (always the same order; absent data renders as "n/a"):
 
 | # | Section | Source |
 |---|---------|--------|
-| 1 | North Star (intent) | `.build-loop/intent.md` |
-| 2 | Current Goal | `.build-loop/goal.md` |
+| 1 | North Star (intent) — incl. Orientation + glossary | `.build-loop/intent.md` (inlined WHOLE) |
+| 2 | Current Goal — incl. open decisions | `.build-loop/goal.md` (inlined WHOLE) |
 | 3 | Phase + Live Checklist | `.build-loop/state.json` (execution + runs[]) |
-| 4 | Git State | `git status` + `git log` |
+| 4 | Git State | `git status` + `git log` (working-tree listing capped — see below) |
 | 5 | Queues | `followup/`, `backlog/`, `ux-queue/`, `issues/` |
 | 6 | Gotchas / Lessons | `.build-loop/feedback.md` |
 | 7 | Last Run Summary | `state.json.runs[-1]` |
-| 8 | Resume Instructions | generated (workdir, phase context) |
+| 8 | Landmines | detected (stale `.current-run-id`, `.push-hold`, crash marker, …) |
+| 9 | Resume Instructions | generated (workdir, phase context) |
+
+### §4 working-tree listing is capped at 40 paths
+
+Above 40 dirty files, §4 lists the paths in the **smallest** top-level groups and
+collapses the rest into a count-by-top-level-path table, then prints a
+`git add -A` warning. Reproduced 2026-07-25 on atomize-ai: ~2,981 pre-existing
+dirty tooling files (`.navgator/`, `.build-loop/`, `.bookmark/`, `.rally/`) made §4
+2,999 of the document's 3,214 lines — 93% — burying North Star, Goal, Landmines,
+and Resume Instructions past 3,000 lines of cache paths. A handoff exists for when
+context is scarce, so paying 3,000 lines for it inverts the tool's purpose.
+
+The cap is on **enumeration only**: every dirty file is still counted in the table.
+Smallest-group-first is what keeps it useful — `git status` sorts by path, so a
+head-40 would have emitted 40 `.bookmark/` cache paths and cut every source edit.
+
+`--full-git` restores the raw per-file listing when it is genuinely wanted.
+
+## Delivery is not the same as writing it
+
+A handoff that exists only on disk has not been handed off. Two things are required
+every time, and neither is optional because the file was committed.
+
+**1. State the absolute path in the reply.** Not "handoff written", not a repo-relative
+path. The person picking this up is in a different terminal, often a different repo, and
+frequently is not a person. Print the full path:
+
+```
+/Users/<user>/dev/git-folder/<repo>/HANDOFF.md
+```
+
+**2. Post it to Rally Point.** A handoff invisible to the coordination substrate cannot be
+picked up by a peer agent, which is the case it most exists for.
+
+```bash
+rally say fact --tool "<your-tool-id>" \
+  --subject "handoff: <one line, what state the work is in>" \
+  --evidence "file:<absolute path>" --json
+```
+
+The same applies to a retrospective. Those land in the memory store rather than the repo,
+so they are unfindable from the working directory unless the path is stated.
+
+**Why this is a rule and not a preference.** The seven content classes below exist so a
+cold reader can resume without the author. All of that work is wasted if the reader cannot
+locate the document. The most common failure is not a thin handoff; it is a good handoff
+nobody found.
 
 ## Usage — no flag (emit doc)
 
@@ -45,7 +179,27 @@ Prints the handoff doc to stdout. Pipe to a file or share directly.
 python3 ${CLAUDE_PLUGIN_ROOT}/scripts/handoff --workdir "$PWD" --output handoff.md
 ```
 
-Writes to a file instead.
+Writes to a file instead, and prints where it landed **plus a browsable URL**:
+
+```
+Handoff written to /path/projects/<proj>/handoffs/2026-08-25-lane.md
+  url: https://github.com/<owner>/<repo>/blob/main/projects/<proj>/handoffs/2026-08-25-lane.md
+  (live)
+```
+
+A handoff is written so someone ELSE can pick the work up, and an absolute local
+path names a file on one machine — useless to a peer session, a ticket, or
+another host. The URL is the part that travels.
+
+Two states are distinguished, because a link to unpushed content is worse than
+no link: `(live)` means the upstream holds this exact content; `(NOT PUSHED YET
+— commit and push before sharing)` means it does not. An untracked file reports
+NOT PUSHED, not live — `git diff @{u} -- <path>` exits 0 for a path git has
+never seen, so the check confirms the upstream actually has the path first.
+
+When no github/gitlab origin resolves for that path, it prints the `file://` URI
+and says the doc is local-only. It never guesses a URL: a wrong link sends the
+reader to a 404 or to a stale copy of a different file.
 
 ```bash
 python3 ${CLAUDE_PLUGIN_ROOT}/scripts/handoff --workdir "$PWD" --json
@@ -53,6 +207,13 @@ python3 ${CLAUDE_PLUGIN_ROOT}/scripts/handoff --workdir "$PWD" --json
 
 Emits a JSON envelope `{document, sources, errors, ts}` for programmatic use.
 
+```bash
+python3 ${CLAUDE_PLUGIN_ROOT}/scripts/handoff --workdir "$PWD" --full-git
+```
+
+Lists every dirty path in §4 instead of capping at 40 and summarizing the rest.
+Combines with any of the flags above.
+
 ## Usage — `--launch` (fresh session)
 
 The command surface (`/build-loop:compose-handoff --launch`) handles this. The skill provides
@@ -84,7 +245,7 @@ new session starts.
 `scripts/handoff/__main__.py` reads `.build-loop/` using only `json`, `pathlib`,
 and `subprocess` from the standard library — no new dependencies. It does NOT
 re-implement state parsing; it reads `state.json` directly at the same paths the
-orchestrator already writes. Tests: `scripts/handoff/test_handoff.py` (13 tests).
+orchestrator already writes. Tests: `scripts/handoff/test_handoff.py`.
 
 ## Host-agnostic design
 

package/skills/ibr-bridge/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: build-loop:ibr-bridge
-description: Routing bridge to the IBR plugin for UI visual verification. Build-loop prefers IBR `scan` / `scan_macos` when the IBR plugin is installed; otherwise falls back to build-loop's own `native-ax-driver` / `ui-validator`. Never falls back to nm/strings. NOT for API registration (use `api-registry-bridge`) or PRD grounding (use `prd-bridge`).
+description: Route rendered UI design updates, comparisons, and audits to the IBR plugin when installed, falling back to build-loop's own driver otherwise. Use when a build changes, compares, or audits a rendered UI and that change needs visual evidence before it can close. Not for registering a new API (use `api-registry-bridge`) or PRD grounding (use `prd-bridge`).
 version: 0.3.0
 user-invocable: false
 ---
@@ -19,6 +19,7 @@ At Phase 1 Assess, the orchestrator reads `state.json.availablePlugins.ibr` (pop
 
 | Build-loop need | Primary (when `availablePlugins.ibr == true`) | Fallback (when IBR absent) |
 |---|---|---|
+| UI design update / comparison / audit | IBR `snapshot` + `compare`, `match`, or the narrowest platform `scan` | `ui-validator` / native AX / simulator or browser screenshots + static scanner |
 | Web UI verify | IBR `scan` against the dev-server route | `ui-validator` agent + browser/screenshot tooling |
 | macOS UI verify | IBR `scan_macos` against the running `.app` | `native-ax-driver` (`skills/native-ax-driver/`) pid-anchored AX-tree + screenshot |
 | iOS sim UI verify | IBR `scan` against the booted sim (when supported) | `xcrun simctl io booted screenshot` + `idb ui` for interaction |
@@ -61,6 +62,8 @@ The bridge returns results in the same envelope shape regardless of which route
 
 The orchestrator forwards `verification` and `artifacts` to the BL-1 gate envelope as the chunk's `verification` and `evidence_paths` fields.
 
+The binding automatic trigger and viewer boundary are defined once in `../../references/ibr-ui-verification-policy.md`.
+
 ## Status: 0.3.0 (2026-06-04)
 
 - Reversed the explicit-only default. IBR is now the preferred primary when present; build-loop-owned `native-ax-driver` / `ui-validator` is the always-available fallback.

package/skills/knowledge/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: knowledge
-description: Canonical build-loop-memory framework. Use when the user asks to "record a decision", "log an ADR", "write an MADR", "capture this choice", "regenerate the decisions index", "validate knowledge", "migrate feedback to decisions", or "recall <topic>". ALSO the read-only review surface (review mode): "review my decisions", "show review queue", "check decision rot", "list open conflicts", "find stale procedures", or `/knowledge:review`. Active durable writes go to `~/dev/git-folder/build-loop-memory`; legacy `.episodic/` paths are migration/archive inputs only.
-user-invocable: true
+description: Canonical build-loop-memory framework. Use when the user asks to "record a decision", "log an ADR", "write an MADR", "capture this choice", "regenerate the decisions index", "validate knowledge", "migrate feedback to decisions", or "recall <topic>". ALSO the read-only review surface (review mode): "review my decisions", "show review queue", "check decision rot", "list open conflicts", "find stale procedures", or `/knowledge:review`. Active durable writes go to the resolved build-loop-memory root (override via `$BUILD_LOOP_MEMORY_ROOT`; per-user default otherwise — see `scripts/_paths.py:memory_store_root()`); legacy `.episodic/` paths are migration/archive inputs only.
+user-invocable: false
 when_to_use: |
   - User wants to record a substantive choice with rationale
   - User asks to regenerate canonical decision indexes or validate decision files
@@ -12,7 +12,7 @@ when_to_use: |
   - Auto-capture (Phase 3) and consolidation (Phase 4) are NOT yet
     implemented; this skill covers Phase 1 (manual + scripted) and
     Phase 2 (Postgres + pgvector retrieval) only.
-namespace: ~/dev/git-folder/build-loop-memory/ (canonical), .build-loop/events.jsonl (repo-local runtime), legacy .episodic/ (migration/archive only)
+namespace: resolved build-loop-memory root ($BUILD_LOOP_MEMORY_ROOT override, else per-user default — canonical), .build-loop/events.jsonl (repo-local runtime), legacy .episodic/ (migration/archive only)
 ---
 
 <!-- SPDX-FileCopyrightText: 2025-2026 Tyrone Ross, Jr <46267523+tyroneross@users.noreply.github.com> | SPDX-License-Identifier: Apache-2.0 -->
@@ -20,15 +20,16 @@ namespace: ~/dev/git-folder/build-loop-memory/ (canonical), .build-loop/events.j
 # Knowledge — Canonical Build-Loop Memory (Phases 1 + 2)
 
 This skill is the entrypoint for the four-memory-types framework. The
-full design lives at
-`~/dev/research/topics/repo-episodic-memory-framework/repo-episodic-memory-framework.md`
-(see §11–§14 for the four-memory-type taxonomy, extraction pipeline,
-and Postgres schema). Read it before making structural changes.
+full design lives in "Repo Episodic Memory Framework" (private research
+note — substance summarized here; see §11–§14 for the four-memory-type
+taxonomy, extraction pipeline, and Postgres schema). This skill's
+`scripts/` are the executable source of truth for structural changes;
+the research note is background reading, not a required dependency.
 
 ## What lives where
 
 ```
-~/dev/git-folder/build-loop-memory/
+<build-loop-memory root>/           # resolved by scripts/_paths.py:memory_store_root()
 ├── projects/<project>/decisions/   # canonical MADR decisions + INDEX.md
 ├── projects/<project>/lessons/     # project-specific lessons
 ├── lessons/                        # cross-project lessons
@@ -72,7 +73,7 @@ relevant prior memory rather than reading INDEX.md wholesale. See
 2. Run `write_decision.py` with the required flags. The script:
    - Allocates the next sequential ID (zero-padded 4-digit).
    - Writes the MADR to
-     `~/dev/git-folder/build-loop-memory/projects/<project>/decisions/<canonical-id>.md` using
+     `<build-loop-memory root>/projects/<project>/decisions/<canonical-id>.md` (root resolved per `scripts/_paths.py:memory_store_root()`) using
      `skills/knowledge/templates/madr-minimal.md` as the body
      scaffold (filled from CLI flags).
    - Regenerates the canonical decisions `INDEX.md`.
@@ -106,12 +107,23 @@ the current version.
 `write_decision.py` calls the validator as a pre-write gate; you can
 also run it standalone over the whole tree.
 
-## Postgres connection
+## Postgres connection (optional — Phase 2 only)
 
-DB-side scripts read connection from
-`~/.config/agent-memory/connection.env` (DATABASE_URL=
-postgresql://tyroneross@localhost:5432/agent_memory). Per-project schema:
-this repo uses `build_loop_memory`. The schema name is configurable via
+The entire Postgres/pgvector/Ollama-embedding stack is **optional**. Phase 1
+(file-only decisions) works with zero DB configuration; `write_decision.py`'s
+DB write is best-effort and never fails the file write (see "DB errors do NOT
+fail the file write" above). Skip this section entirely if you only want
+Phase 1.
+
+When you do want Phase 2 retrieval, DB-side scripts resolve a connection
+string via `scripts/_db_url.py:resolve_db_url()`, in order:
+
+1. `$BUILD_LOOP_DATABASE_URL`
+2. `$DATABASE_URL`
+3. `~/.config/agent-memory/connection.env` (`DATABASE_URL=postgresql://$USER@localhost:5432/agent_memory` is a reasonable local default to put there)
+
+Per-project schema defaults to `$AGENT_MEMORY_SCHEMA` or `personal_memory`;
+this repo uses `build_loop_memory`. The schema name is also configurable via
 the `--schema` flag on each DB-aware script.
 
 ## Review mode (read-only)

package/skills/knowledge/references/review-mode.md

@@ -7,9 +7,8 @@
 framework (Working / Episodic / Semantic / Procedural). It does NOT modify
 any data; it lists what humans need to decide on.
 
-The full design lives at
-`~/dev/research/topics/repo-episodic-memory-framework/repo-episodic-memory-framework.md`
-(see §11–§14).
+The full design lives in "Repo Episodic Memory Framework" (private
+research note — substance summarized here; see §11–§14).
 
 Companion scripts: `scripts/knowledge_review.py` (aggregates all four sections into a
 markdown report), `scripts/detect_decision_rot.py` (rot section),

package/skills/knowledge/templates/madr-minimal.md

@@ -15,7 +15,7 @@ project: project-name
 tool: claude-code
 model: claude-opus-4-7
 task_category: feature
-author: tyroneross
+author: your-username
 source: manual
 related_runs: []
 related_decisions: []

package/skills/mcp-builder/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: mcp-builder
-description: Use when building, packaging, or debugging an MCP server, adding MCP tools to a plugin, or working on .mcp.json, transport, or bundling. Pair with plugin-builder when the plugin exposes MCP tools.
+description: Build, package, or debug an MCP server and its tools. Use when the user is adding MCP tools, or working on `.mcp.json`, transport, or bundling. Not for scaffolding the plugin around it (use `plugin-builder`; pair the two when a plugin exposes MCP tools).
 user-invocable: false
 ---
 

package/skills/model-bakeoff/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: model-bakeoff
-description: Use to run a controlled multi-model bake-off — have N models (e.g. Opus 4.8, Sonnet 5.0, GPT-5.5) each independently diagnose→plan→execute the SAME bounded change in isolated git worktrees, then deterministically re-score their committed code on a fresh server, merge the best-of (grafting distinct wins from the others), and repeat per change. Triggers on "bake-off", "compare models on this task", "which model is best at", "run the same change across models and score them". NOT for choosing a model tier/segment for one agent — use `model-tiering`.
+description: Run the same bounded change across multiple models in parallel, then deterministically score and merge the best result. Use when the user says "bake-off", "compare models on this task", "which model is best at", or "A/B test these models". Not for picking a single model tier for one agent (use `model-tiering`).
 user-invocable: false
 ---
 
@@ -8,22 +8,51 @@ user-invocable: false
 
 Run a fair, evidence-based competition where several models each solve the *same* change end-to-end, then merge the best result. One orchestrator (this session) coordinates; contestants are single agents (measure the model, not a multi-agent loop). Repeat per change, accumulating merges on one experiment branch.
 
+## Validation contract (define BEFORE any dispatch)
+
+Fix the grading before running — a metric invented after seeing outputs bends to them.
+
+1. **Metric + rubric + judge, up front.** Write the exact success metric, the rubric, and the pass/fail threshold BEFORE dispatch. Name the **independent judge** (a non-contestant model, e.g. Fable) for subjective dims. No post-hoc scoring.
+2. **Reproduce the baseline FAILURE first.** To prove an intervention (profile/scaffold/fix) makes a model succeed, first REPRODUCE the failure deterministically. n=1 anecdotal failures don't generalize — a task where the control already passes has no fix to validate. (2026-07-09: "kimi fails multi-crate Rust" did NOT reproduce on synthetic 2-crate tasks — the control passed both a 3-file crate and a 2-crate cross-crate task via the default profile; the original sandbox-dogfood failure was task-specific, not a general ceiling.)
+3. **Task at the failure boundary.** If every arm passes, the task is too easy → no signal to separate arms. Calibrate difficulty until the control fails.
+4. **A control arm.** Always include the default-profile / no-intervention (and, for harness work, the no-harness one-shot) arm to isolate the intervention's marginal value.
+
+## Model + thinking identity (mandatory data contract)
+
+A benchmark measures a **model at a thinking level and mode**, not a model name alone. Use Benchmark Lab's existing `abc-comparison/v2` artifact; do not create a Build Loop-only benchmark schema. Every arm adds:
+
+- `model`: exact provider model/version, separate from the host or harness.
+- `effort`: exact effective provider/runtime label, or `unknown`.
+- `effort_normalized`: `none|minimal|low|medium|high|xhigh|max|ultra|unknown`.
+- `effort_provenance`: `source_document|experiment_config|runtime_log|unknown`.
+- `effort_evidence`: source page/table/footnote for published data, or exact config/command/log evidence for our test.
+- `mode`: `single_agent|multi_agent|adaptive|unknown`.
+
+These are additive arm fields on the canonical artifact, following the existing field-observation contract. Benchmark Lab owns validation, normalization, and aggregation, but its current converter does not retain these additive fields. Preserve the raw artifact as source evidence until that Lab-owned extension lands; ingest acceptance alone is not proof of retention. Never infer effort from a model default or translate provider labels without evidence. If a source omits row-level effort, record `unknown`, cite the omission, and mark the comparison effort-confounded/directional. `ultra` or another multi-agent mode is a distinct treatment arm, never a higher single-agent score.
+
+## When scoring: check the CODE and the OUTPUT, and distrust the rig
+
+- **Check the produced code AND its oracle output — never "it ran."** Read the code; run the oracle; parse the RIGHT signal. (2026-07-09: an auto-grep matched the lib unit-test line "0 passed" and mislabeled a 2/5-passing control as FAIL; multi-binary `cargo test` needs pass/fail SUMMED across binaries, not first-match. Always confirm the specific test binary that carries the assertions.)
+- **Separate the measurement apparatus from the thing measured.** The **rig** = everything that is NOT the model or the code under test: the grading script, the exact command + flags you invoke, the tooling environment (installed binaries, language toolchain, PATH), and the fixtures/scaffold. A **rig bug** is a defect in that apparatus that yields a false pass or a false fail wrongly blamed on the model. Before trusting any `0` or any FAIL, confirm the rig produced a *valid run*: the command parsed, the tools were present, the oracle actually executed against real output. Three rig bugs on 2026-07-08, each first mis-scored as a model failure: (a) `pytest` absent from the grading env → a contestant looped on a false-negative it could never clear; (b) `harness swarm --segments` expects a manifest FILE, not an integer count → the treatment arm produced 0 files twice on a malformed command; (c) an auto-grep read the wrong `cargo test` binary's summary line → a 2-of-5-passing run was labeled FAIL. None were the model; all were the rig.
+
 ## Roster & dispatch (verified handles)
 - Opus 4.8 → `Agent(model: "opus")`; Sonnet 5.0 → `Agent(model: "sonnet")` (`sonnet` = latest, NOT 4.x — older Sonnets have no clean subagent handle).
-- GPT-5.5 → Codex MCP `mcp__codex__codex` with `model: "gpt-5.5"`, `config: {model_reasoning_effort: "xhigh", sandbox_workspace_write:{network_access:true}}`, `approval-policy: "never"`, `sandbox: "workspace-write"`. (Check `~/.codex/config.toml` for the exact model id; `-codex` suffixes fail on ChatGPT-account Codex.)
+- Codex/OpenAI → read the exact model id and requested effort from the experiment manifest; pass both explicitly (for example `model: "gpt-5.6-terra"`, `config: {model_reasoning_effort: "high"}`). Capture the effective runtime config in the arm log; do not rely on `~/.codex/config.toml` defaults or guess `-codex` suffixes.
+- Claude effort → use a host surface that can set the requested effort. If the available dispatch primitive cannot set it, record the effective value as `unknown`; do not label the arm `xhigh` from intent alone.
 - Independent judge: prefer a NON-contestant model (e.g. Fable). If unavailable, the orchestrator scores subjective dims with over-cited evidence + a stated caveat, and leans on deterministic dims.
 
 ## Per-change protocol
 1. **Baseline:** branch the experiment off clean `origin/main` (not a dirty/active branch). Confirm no concurrent session collides.
-2. **Scaffold** one worktree per contestant off the experiment branch HEAD: `git worktree add -b bakeoff/<Cn>-<model> <path> <branch>`; `npm ci` (or lockfile-equiv) per worktree; copy `.env.local`.
-3. **Brief** (IDENTICAL for all): give the *symptom* + acceptance criteria + a fair equal entrypoint pointer — WITHHOLD the diagnosis (that's what's scored). Add repo guardrails (see below).
-4. **Dispatch** all contestants in parallel (Agent arms `run_in_background: true`; Codex arm blocks the turn — fine, the others run concurrently).
-5. **Commit stranded Codex work:** Codex's sandbox usually cannot write an external worktree's `.git` (`index.lock: Operation not permitted`). The orchestrator commits it: `git -C <worktree> add -A && git commit`. (RESULT.md is often gitignored → `git add -f`.)
-6. **Score deterministically** (don't trust self-reports — re-run each contestant's committed code):
+2. **Manifest:** pre-register each arm's exact model/version, requested thinking level, mode, dispatch surface, and evidence path in the run manifest. Pre-create the `abc-comparison/v2` arm fields above; update `effort` and `effort_evidence` from effective runtime evidence before scoring.
+3. **Scaffold** one worktree per contestant off the experiment branch HEAD: `git worktree add -b bakeoff/<Cn>-<model> <path> <branch>`; `npm ci` (or lockfile-equiv) per worktree; copy `.env.local`.
+4. **Brief** (IDENTICAL for all): give the *symptom* + acceptance criteria + a fair equal entrypoint pointer — WITHHOLD the diagnosis (that's what's scored). Add repo guardrails (see below).
+5. **Dispatch** all contestants in parallel (Agent arms `run_in_background: true`; Codex arm blocks the turn — fine, the others run concurrently). Write the effective thinking setting and its runtime evidence back to the run manifest and observation arm before scoring.
+6. **Commit stranded Codex work:** Codex's sandbox usually cannot write an external worktree's `.git` (`index.lock: Operation not permitted`). The orchestrator commits it: `git -C <worktree> add -A && git commit`. (RESULT.md is often gitignored → `git add -f`.)
+7. **Score deterministically** (don't trust self-reports — re-run each contestant's committed code):
    - Objective dims computed in code: typecheck, build, test, betterer, + the change-specific success criterion run on a fresh server. Weight these highest.
    - Subjective dims (diagnosis depth, plan, code quality, intent fidelity) by the judge reading diffs + captured outputs.
-7. **Scorecard** (rubric below) → **merge** best base onto the experiment branch, **grafting** distinct wins from the others (each graft: fixes a real gap the winner has, is isolable from the loser's *harmful* parts, verified by re-running). Document graft rationale.
-8. **Re-verify the merged result**, regenerate coverage, commit. Then next change.
+8. **Scorecard** (rubric below) → **merge** best base onto the experiment branch, **grafting** distinct wins from the others (each graft: fixes a real gap the winner has, is isolable from the loser's *harmful* parts, verified by re-running). Document graft rationale.
+9. **Re-verify the merged result**, regenerate coverage, validate the canonical observation through Benchmark Lab when available, commit. Then next change.
 
 ## Rubric (max 50; tune weights per task)
 success-criteria attainment ×3 (objective) · build/typecheck/test/betterer ×2 (objective) · diagnosis accuracy ×2 · plan quality ×1 · code quality+scope ×1 · intent fidelity ×1. Objective dims dominate; the LLM judge is confined to subjective dims.
@@ -41,4 +70,13 @@ success-criteria attainment ×3 (objective) · build/typecheck/test/betterer ×2
 Work ONLY in your worktree; no edits outside it; no deploy/push; no `--no-verify`; no DDL against the shared DB; unique per-model test-user id + id-scoped cleanup; verify by RUNNING (name the exact verification mechanism: curl the endpoint, CDP virtual authenticator for WebAuthn, function-level test for cleaners, screenshots for UI).
 
 ## Output
-Per stage: a `SCORECARD.md` (rubric table + verdict + merge/graft rationale). At the end: a consolidated `RESULTS.md` (cross-stage scoreboard, per-model performance pattern, where multi-model merge beat any single model, scoring-integrity caveats).
+Per stage: a `SCORECARD.md` (rubric table + verdict + merge/graft rationale). At the end:
+
+- `RESULTS.md`: cross-stage scoreboard, each arm labeled `model_id @ thinking level/mode`, per-model performance pattern, where multi-model merge beat any single model, and scoring-integrity caveats.
+- `OBSERVATION.json`: canonical `abc-comparison/v2` rounds with each arm's exact model,
+  thinking fields, result metrics, and provenance. This is the machine-readable source of
+  truth; Benchmark Lab is the sole normalization and aggregation owner.
+
+Preserve the raw JSON whether or not Benchmark Lab ingestion is available. Until the Lab
+converter and storage retain these fields, do not use successful ingest as evidence that
+thinking provenance survived, and do not aggregate results across thinking settings.