loki-mode 10.5.0 → 10.5.2

package/docs/v10/MEDIUM-ANALYSIS.md

@@ -178,4 +178,169 @@ Re-running the failed tasks uses `eval/loki10/run.sh` with a separate `--out` pe
   - The results are published in this file as the "after" table.
 - Budget: 60 min of eval, then a 15 min code change. Tier: HIGH (it changes the default flow).
 
+## After (E-98f)
+
+Eval run 2026-09-28, worktree `agent-a340b90dfe1a90ce0`, branch `slice-E-98f`,
+harness_sha `3abb3ac6b5cdcb84467f247c56f15eba964b5b77-dirty` (dirty only from
+`loki-ts/dist/cockpit.js`'s non-deterministic build output; source is
+`main` with E-98a..e merged, dist rebuilt in this branch). `bin/loki
+--version` = v10.4.1, `manifest.jsonl` `arm_binary` is this worktree's own
+`bin/loki` (the nocascade arm through a `LOKI_EVAL_LOKI_BIN` shim, see
+below), never a global install. Model `claude-opus-5-5`, the same 7 `pub-*`
+medium tasks, 900s cap, `--parallel 3`, fresh clone per run, n=3 per arm (21
+task-runs each). Raw is not re-run (per instruction); both prior raw
+measurements are carried forward for comparison.
+
+**nocascade note:** `LOKI_E10_CASCADE` is not in `eval/loki10/harness.py`'s
+`V10_ENGINE_ENV_ALLOWLIST` (only `LOKI_E10_PLAN/WALL/WALL_TIER/CAP_S/
+INVOKER/DASHBOARD_PORT` are listed), so it is dropped by `arm_env`'s blanket
+`LOKI_*` scrub before reaching the arm. Worked around without a harness code
+change: `LOKI_EVAL_LOKI_BIN` pointed at a one-line shim
+(`LOKI_E10_CASCADE=0 exec <worktree>/bin/loki "$@"`) that sets the var in
+the process `harness.py` execs, outside the scrub. Verified live: 21/21
+nocascade implement sessions ran `session.started` with
+`"model":"claude-opus-5-5"`; 21/21 default and nowall implement sessions ran
+on `"model":"sonnet"`. nowall verified live too: 21/21 nowall runs show a
+wall `stage.skipped` with `"reason":"LOKI_E10_WALL=0"`; 0/21 default or
+nocascade runs show that skip. This allowlist gap should be closed
+(`eval/loki10/harness.py`, add `LOKI_E10_CASCADE`) as a harness follow-up; it
+was not fixed here (eval-only, no code changes).
+
+**Auth incident, not concurrency.** r2 for all three arms was lost in full
+(21/21 rows `auth_unavailable`) and had to be re-run. Cause: `harness.py`'s
+per-row check, `arm_auth(cap + AUTH_MARGIN_S)` = `arm_auth(1020)`, refuses to
+start a row unless the keychain OAuth access token has at least 1020s (17
+min) left. Claude Code does not refresh that token until it is itself close
+to expiry (observed: still only ~415s left immediately after running
+`claude --print` as the operator, i.e. the refresh did not fire early), so
+across a batch of many-minute arm sessions the token's remaining lifetime
+can fall under 1020s well before it actually expires, and every row started
+after that point is refused until the operator's own use of `claude`
+eventually triggers a real refresh. The r1 `pub-werkzeug-3271` miss (last
+task started in each arm) had the same cause; it was retried once
+`arm_auth(1020)` and `arm_auth(3600)` both passed (`expiresAt` about 8h out
+by then). This is an auth-lifecycle fact of the harness/environment, not
+evidence of contention from running 3 arms concurrently (9-12 sessions at
+once, load 2.5-4.7 of a 14-core box throughout). No rate-limit signal (429/
+`rate_limit`/`overloaded`/529) was found in any arm's stdout/stderr logs, or
+in the one apparent hit inside an `arm_stdout.log` (a `seal` receipt hash
+substring, `5963429c...`, a false positive).
+
+**Dedupe rule.** Each result file can hold more than one attempt per task
+(an `auth_unavailable`/`interrupted` attempt from a lost run, followed by a
+real attempt after re-running that same r-file's `--out`). Rows are deduped
+to exactly one per (arm, run-file, task): the latest `status: ok` attempt if
+one exists in that run-file, else its latest attempt overall (by `started`,
+falling back to file order). This never collapses across r1/r2/r3, since
+those are separate files/reps, and it matches EV-15's own "pool the real
+rows, one per task per run" method, made explicit here because unlike EV-15
+this run needed retries. Rows dropped by this rule (superseded
+`auth_unavailable`/`interrupted` attempts, all from the r2 auth incident
+above, plus each arm's r1 werkzeug-3271 retry): 22 for default, 15 for
+nowall, 15 for nocascade. Each arm's final pooled table has exactly 21 rows,
+7 tasks times 3 runs, matching `--tasks` times n=3.
+
+### Before (raw, carried forward)
+
+| source | arm | completed | rate | cost per completed | p50 / p90 |
+|---|---|---|---|---|---|
+| EV-14 (docs/v10/METRICS.md; raw result files lost, METRICS.md entry is the record) | raw `claude -p` | 12/14 | 85.7% | $0.5119 | 70s / 239s |
+| EV-15 (this doc, section 1/4) | raw `claude -p` | 10/14 | 71.4% | $0.5085 | 56s / 104s |
+| EV-15 (this doc, section 1/4) | v10 default knobs | 9/14 | 64.3% | >= $0.788 (corrected lower bound, section 4) | 128s / 330s |
+
+### After (E-98f, n=3, 21 task-runs per arm)
+
+| arm | completed | rate | cost per completed | null-cost rows | p50 / p90 |
+|---|---|---|---|---|---|
+| default | 15/21 | 71.4% | n/a | 10/21 (lower bound $0.6958) | 209s / 457s |
+| `LOKI_E10_WALL=0` (nowall) | 16/21 | 76.2% | n/a | 2/21 (lower bound $0.786) | 214s / 500s |
+| `LOKI_E10_CASCADE=0` (nocascade) | 15/21 | 71.4% | n/a | 11/21 (lower bound $0.3233, unreliable at this null rate) | 138s / 218s |
+
+p50/p90 were measured at up to 12 concurrent arm sessions (3 arms times
+`--parallel 3`, plus single-task retries) against EV-15's 3; not apples to
+apples with the before rows. The decision rule below does not use latency.
+
+Null-cost rows persist well above zero in every arm despite E-98e's fix
+being present and firing correctly at the iteration level. Root cause,
+confirmed by reading both files for one null row
+(`e98f-engine/default-r1/pub-werkzeug-3121.../.loki/metrics/`): the killed
+wall iteration's `efficiency/iteration-2.json` does carry a real,
+non-zero `cost_usd` (`0.0628`) from E-98e, but with `"cost_source":
+"partial-stream"`. `eval/loki10/harness.py`'s `provider_cost()` (line
+~698) sums a row's cost only if every one of its iteration files has
+`cost_source == "provider"` exactly; `"partial-stream"` fails that check by
+design, so the entire row's `cost_usd` is `null` even though the money was
+correctly recorded. This is not a bug in E-98e or a bug in the harness
+reading the wrong field; it is the harness's deliberate "only trust a
+provider-reported number" policy colliding with E-98e's necessarily
+estimated number for a killed session, and it fires on any stage kill, not
+only Wall's: nowall's 2 null rows both trace to `implement` itself hitting
+its own stage limit (`stage.failed implement {"reason":"limit"}`), since
+Wall is skipped there and has nothing left to kill. This explains the
+per-arm null counts directionally: default and nocascade run Wall for real
+(more stage-limit kills, more nulls: 10/21 and 11/21) while nowall skips it
+(fewer kills, fewer nulls: 2/21). Cost per completed is therefore `n/a`
+under the harness's own rule (every evaluated row must have a cost) for all
+three arms; the lower bounds above sum only the rows with a `provider`
+cost and likely understate the true figure (see MEDIUM-ANALYSIS section 4's
+own correction of EV-15's lower bound for the same reason). A harness
+follow-up (not made here, eval-only): accept `cost_source in ("provider",
+"partial-stream")` in `provider_cost()`, or report the two sums separately.
+
+**Misses**, all three arms: only `pub-werkzeug-3105` (hidden tests failed
+in all 9 of 9 attempts across the three arms) and `pub-werkzeug-3271`
+(missed in 8 of 9: 7 hidden-tests-failed plus 1 no-branch-pushed, that one
+also `hidden_pass: false` so a genuine miss, not EV-15 raw r2's "correct fix
+never pushed" pattern; nowall r3 completed it) -- the same two
+task-difficulty misses EV-15 found shared with raw (section 1). Unlike
+EV-15, `pub-jinja-1413` completed in all 9 attempts across all three arms
+(0 misses): the E-98a/E-98b/E-98c fixes (project interpreter, spec_conflict
+still verifies, append-only existing tests) removed the jinja regression
+that was EV-15's only pure v10-side loss.
+
+**Fix-effect counts** (from `events.jsonl`, filtered to the engine slot of
+each `status: ok` row; 21 files per arm):
+
+| arm | pytest not_run / total | ruff not_run / total | fix rounds started | runs with >=1 fix round | spec_conflict runs | spec_conflict, verify started | spec_conflict, verify completed |
+|---|---|---|---|---|---|---|---|
+| default | 0/80 | 11/25 | 5 | 3 | 1 | 1 | 1 |
+| nowall | 0/72 | 16/24 | 6 | 5 | 2 | 2 | 1 (1 verify killed at its own limit) |
+| nocascade | 0/58 | 19/22 | 3 | 2 | 1 | 1 | 1 |
+
+pytest `not_run` is 0 of 80/72/58 in every arm, and every one of those
+checks ran on the `project` interpreter (0 on `system`): E-98a's interpreter
+resolution holds up at n=21 per arm, and this is the evidence the jinja
+regression's root cause (a test run against the installed package instead
+of `src/`) is closed. ruff `not_run` is high but not a resolution failure:
+every `not_run` reason is "ruff not found on PATH" (genuinely absent from
+that task's own venv, not a wrong-interpreter miss), and every ruff check
+that did find a binary resolved it to the `project` interpreter, same as
+pytest (14 of 25 for default, 8 of 24 nowall, 3 of 22 nocascade). E-98a's
+resolver (commit `cabf16ef`, verify.ts:148) does cover ruff; most task repos
+here simply do not ship ruff in their own environment. Every `spec_conflict`
+implement exit started a real `verify` stage in all three arms (E-98b;
+unlike EV-15's 0 of 5 verify starts). Of the 4 spec_conflict runs, verify
+itself then completed in 3; the 4th (nowall) had verify killed at its own
+limit (`stage.failed verify {"reason":"limit"}`), so "spec_conflict reaching
+verify" in the table below counts verify *starting*, and the completed/
+failed split is called out per arm.
+
+### Decision
+
+Rule (founder, 2026-09-28): the chosen arm must complete at or above raw's
+best measured medium rate, EV-14's 85.7% (12/14, 18/21 at this n), and its
+cost per completed must be at or below raw's EV-15 $0.5085.
+
+| arm | vs EV-14 85.7% (18/21) | vs EV-15 71.4% (15/21) | vs EV-15 $0.5085 |
+|---|---|---|---|
+| default | fails (15/21) | ties exactly (15/21) | fails on its own lower bound ($0.6958); true cost n/a |
+| nowall | fails (16/21) | passes (16/21) | fails on its own lower bound ($0.786); true cost n/a |
+| nocascade | fails (15/21) | ties exactly (15/21) | undetermined: lower bound $0.3233 is below the bar, but 11/21 rows (52%) are uncosted, too many to trust the bound |
+
+No arm reaches 18/21 (85.7%): nowall is highest at 16/21 (76.2%), default
+and nocascade both at 15/21 (71.4%, an exact tie with EV-15's raw rate). No
+arm's cost per completed is a clean number under the harness's own rule
+(`n/a` in all three; see the null-cost cause above). **No arm meets the
+founder's rule. `sizing.ts` is left unchanged.**
+
 Order: a, b, c, d and e in parallel (their file sets do not overlap), then f. The "after" table for the founder is the E-98f default arm next to section 1 above.

package/docs/v10/METRICS.md

@@ -126,6 +126,14 @@ critical path and reviewed immediately.
 
 ## Usage (hourly, from scripts/usage-governor.py)
 
+### 2026-09-28T21:15Z
+- 5h window: uncalibrated, 5,183,443 output tokens since 2026-09-28T16:15Z (uncalibrated)
+- Weekly: uncalibrated, 47,364,676 output tokens, 43.7h to reset (uncalibrated)
+- Last hour: 477,429 output tokens; 17 active engineers; 24,479 per engineer; Chief of Staff 61,285
+- Max engineers next hour: uncalibrated (no plan reading on file)
+- Output tokens by model (all scanned transcripts): claude-sonnet-5 30,642,455 (42.1%), claude-opus-5-5 16,713,247 (23.0%), claude-opus-4-8 14,720,953 (20.2%), claude-opus-4-7 4,678,300 (6.4%), claude-opus-5 4,489,139 (6.2%), claude-fable-5 1,254,167 (1.7%), claude-opus-4-6 203,492 (0.3%), claude-haiku-4-5-20251001 6,565 (0.0%)
+- Output tokens by role (all scanned transcripts): workflow-agent 43,102,887, chief-of-staff 29,573,766, subagent 31,665
+
 ## CI: Tests workflow shell-tests sharding (S-81, supersedes S-70)
 
 S-70 resharded shell-tests 4 -> 8 with a plain `idx % n` split and measured
@@ -334,3 +342,37 @@ since `arm_stdout.log` can hold env), so there are no harness result files
 tracked in git to commit, matching every earlier EV entry in this file. Only
 this METRICS.md section is committed. The raw result files above remain on
 disk in this worktree at `eval/loki10/results/ev14-medium-{raw,v10}-r{1,2}/`.
+
+## Medium tier A/B (E-98f)
+
+E-98f (2026-09-28, branch `slice-E-98f`, harness_sha
+`3abb3ac6b5cdcb84467f247c56f15eba964b5b77-dirty`, v10.4.1, `claude-opus-5-5`,
+the same 7 `pub-*` medium tasks, 900s cap, `--parallel 3`, fresh clone per
+run) measured three knob arms at n=3 (21 task-runs each) against E-98a..e
+merged: default knobs, `LOKI_E10_WALL=0` (nowall) and `LOKI_E10_CASCADE=0`
+(nocascade, via a `LOKI_EVAL_LOKI_BIN` shim since that var is not in
+`harness.py`'s `V10_ENGINE_ENV_ALLOWLIST`; see `docs/v10/MEDIUM-ANALYSIS.md`
+"After (E-98f)" for the full method and the auth-lifecycle incident that
+lost and required re-running every arm's r2). Raw was not re-run; both
+prior raw measurements are carried forward. Dedupe rule: one row per (arm,
+run-file, task), the latest `status: ok` attempt if one exists else the
+latest attempt overall, never collapsing across the 3 reps (full rule and
+per-arm rationale in MEDIUM-ANALYSIS.md). Rows dropped by this rule: 22
+(default), 15 (nowall), 15 (nocascade), all superseded `auth_unavailable`/
+`interrupted` attempts from the auth incident; each arm's pooled table below
+has exactly 21 rows (7 tasks times 3 runs).
+
+| source | arm | completed | rate | cost per completed | p50 / p90 |
+|---|---|---|---|---|---|
+| EV-14 | raw `claude -p` | 12/14 | 85.7% | $0.5119 | 70s / 239s |
+| EV-15 | raw `claude -p` | 10/14 | 71.4% | $0.5085 | 56s / 104s |
+| EV-15 | v10 default knobs | 9/14 | 64.3% | >= $0.788 (corrected lower bound) | 128s / 330s |
+| E-98f | v10 default knobs | 15/21 | 71.4% | n/a (10/21 null-cost rows; lower bound $0.6958) | 209s / 457s |
+| E-98f | v10 `LOKI_E10_WALL=0` | 16/21 | 76.2% | n/a (2/21 null-cost rows; lower bound $0.786) | 214s / 500s |
+| E-98f | v10 `LOKI_E10_CASCADE=0` | 15/21 | 71.4% | n/a (11/21 null-cost rows; lower bound $0.3233, unreliable) | 138s / 218s |
+
+Decision rule (founder, 2026-09-28): the chosen arm must complete at or
+above EV-14's 85.7% and cost at or below EV-15's $0.5085 per completed. No
+E-98f arm reaches 85.7% (nowall highest at 76.2%), and no arm's cost per
+completed is a clean number (`n/a` in all three, see the null-cost note in
+MEDIUM-ANALYSIS.md). No arm meets the rule; `sizing.ts` is unchanged.

package/docs/v10/PROGRESS.md

@@ -1,5 +1,24 @@
 # Progress
 
+## 2026-09-28T21:44Z: v10.5.1 published; E-98f says no flip; D41 plan and D42 rulings
+- v10.5.1: cut 21:27:45Z (d697ea4c: E-110 pre-push scans every path, METRICS usage snapshot), Release 36486250082 success, publish-npm 21:36:44Z, npm latest 10.5.1 gitHead d697ea4c, body 9 lines from release-notes.sh (no backfill).
+- E-98f (merged eca3e8ef): n=3 x 21 medium task-runs per arm. default 15/21 (71.4%) p50 209s; WALL=0 16/21 (76.2%) p50 214s; CASCADE=0 15/21 p50 138s; raw 12/14 (EV-14) and 10/14 (EV-15) at p50 56s. No arm has a clean cost (null rows 10/2/11, harness rejects partial-stream cost). E-98 fixes hold: pytest not_run 0/80, spec_conflict reaches verify, jinja-1413 9/9. No flip.
+- D41 (founder 21:22Z, 534b2a6e): balanced scorecard per model tier (completion, cost per completed, p50, all at or better than raw); flip only when green on small and medium. Architect plan docs/v10/SCORECARD-PLAN.md (7f671f5c), 16 S41 cards in 5 waves. D42 (CTO, 923648cd): e10ext/ approved with a 1,500-line cap and no verdict logic; S41-14 (Wall in parallel) deferred for lack of a speed win; S41-16 confirmed P0 (Wall red-on-base vacuous when python is missing: exit 127 counted as failed); M-13 must bind normalizers into the oracle seal.
+- Security: E-114 (CI gitleaks) rejected twice by opus on real bypasses (pushed .gitleaks.toml; then event.before chained from an ungated push); r2 takes the base from the last release tag. main has no branch protection (gh api returns 404); raised with the founder.
+- Merged since v10.5.1: DEP-05/06/07, E-90 (42 rows marked released), E-116, E-117, E-120 (reason in receipt, injection-safe). Building: S41-01, S41-02, S41-16, M-13 r4, E-114 r2.
+
+## 2026-09-28T21:18Z: founder 17:10 directives actioned
+- Usage in METRICS.md: scripts/metrics-usage-append.py (343c087b) renders scripts/usage-governor.py --json (4.6s) under "## Usage (hourly ...)"; first snapshot 21:15Z (5h window 5,183,443 output tokens, weekly 47,364,676, last hour 477,429 across 17 active engineers, 24,479 per engineer, Chief of Staff 61,285). Session cron 494e26d8 appends hourly at :07.
+- Staffing to 8: the governor cannot set a ceiling yet (max engineers "uncalibrated": no 5h or weekly plan reading on file), so the founder default of 8 applies. Building: E-98f, E-110 (r2 in opus re-review), E-114, E-90, plus DEP-05, DEP-06, DEP-07, M-13 r2 dispatched 21:18Z. Held with reasons: G-04 cloud fan-out (needs a calibrated governor), EV-9 (after E-98f), E-87 (release gate, rejected once), M-14+ (depend on M-13). E-85 parked (conflicts with fingerprint-only allowlisting).
+- E-98f: running since 20:20Z, most arms at run 3 (result files under ~/loki-ci-logs/eval/e98f-*); flip bar set to raw's best 12/14 (85.7%) and at or below $0.5085 per completed; results go to METRICS.md and MEDIUM-ANALYSIS.md.
+
+## 2026-09-28T21:01Z: v10.4.1 (30afbd07) and v10.5.0 (6498effe) published; red main a4623675 fixed forward by 024b947c in 11 min
+- v10.4.1: cut 20:14:04Z (30afbd07), publish-npm 20:19:57Z, npm latest 10.4.1 gitHead 30afbd07; body was the placeholder (E-88 not yet on origin), backfilled from CHANGELOG (release-notes.sh rc=0, 10 lines).
+- Red main: train a4623675 (pushed 20:30:47Z) failed Tests shard 4/8, tests/test-release-notes.sh T9l/T9n fixtures 34 bytes on Linux (yes piped to head under pipefail; passes 41/0 on macOS). P0 E-113 dispatched 20:39Z, Tech Lead APPROVE (fixture bytes identical: 132034/98034/224089), pushed 20:42:15Z as 024b947c; Tests and Bun Parity success 20:48:31Z.
+- v10.5.0 (minor): cut 20:49:19Z (6498effe), Release 36481891650 success, publish-npm 20:55:23Z (cut to publish 6m04s). Contents: E-98a..e medium fixes (verify on the project interpreter, seal never VERIFIED on a system interpreter, spec_conflict verified, Python repo map, killed-session cost), E-106 SDK pin drift, E-88 release-notes gate, E-108 debugId, internal guards. First release with the release-notes gate on origin; body check pending in the watcher.
+- Governor: E-109 merged (cold 12.9s, warm 4.9s from 115s); pulse now reads opus share 3.1% of output last hour. Budget burn stays UNKNOWN until a founder plan reading is recorded.
+- Next: E-98f (A/B medium eval, n=3 x 3 arms) decides the v10 small+medium flip; E-110 (pre-push scans every path) building.
+
 ## 2026-09-28T20:17Z: v10.4.0 blocked by gitleaks, re-released as v10.4.1; E-98 fixes landing; drift audit (turn 576)
 - 19:40 slot, late: v10.4.0 cut 19:50:24Z (7895a698: G-01, M-08, E-101, E-102, E-103). Release 36475005937 FAILED at required-ci: Security Audit gitleaks "leaks found: 3", all in bbe83c7a:tests/test-eval-archive.sh, synthetic E-101 redaction fixtures (AKIAABCD..., sk-ant-planted..., a token string). Not published; no tag.
 - P0 response (20:02Z to 20:14Z): exact fingerprints allowlisted with a reason (6358e0a6, local gitleaks on d811e5c3..HEAD: no leaks found); Tests and Bun Parity green on it; v10.4.1 cut 20:14:04Z (30afbd07) carrying the 10.4.0 notes. Watcher on Release, npm and body. Guard E-110 (HIGH): pre-push scans every pushed commit and path, fixtures built at runtime. Root cause: pre-push ran gitleaks only for eval fixture pushes while CI scans everything.

package/docs/v10/SCORECARD-PLAN.md

@@ -0,0 +1,250 @@
+# D41 scorecard plan: slices for items 1 and 2
+
+Architect plan (opus) for D41. Base: local main 9869f217, after E-98f. Cards S41-01 to S41-16. Items 3 to 5 are sketched at the end.
+
+## 0. Starting facts (each one cites a file or a command)
+
+- **E-98f (MEDIUM-ANALYSIS.md "After"), 7 medium tasks, n=3, 21 runs per arm.**
+  - default 15/21, p50 209s; `LOKI_E10_WALL=0` 16/21, p50 214s; `LOKI_E10_CASCADE=0` 15/21, p50 138s. Raw: 10/14 (EV-15) and 12/14 (EV-14), p50 56s to 70s.
+  - Null-cost rows are 10, 2 and 11 of 21, so no arm has a cost figure. Every miss is werkzeug-3105 or werkzeug-3271. Speed is the largest gap: v10 p50 is 2.5 to 4x raw.
+- **Three harness defects block the scorecard.**
+  - (a) `provider_cost()` (harness.py ~698) accepts only `cost_source == "provider"`. E-98e writes `"partial-stream"` for killed sessions (runner/budget.ts:502), so any stage kill nulls the whole row.
+  - (b) `V10_ENGINE_ENV_ALLOWLIST` (harness.py:95-98) lacks `LOKI_E10_CASCADE`, and `arm_env` drops every other `LOKI_*`. E-98f needed a bin shim.
+  - (c) `arm_auth(cap + 120)` refuses a row once the keychain token has under 1020s left. Claude Code refreshes only near expiry, so every r2 row was lost once.
+- **D33 core:** 4,964 lines at 9869f217, counted as `budget.test.ts` counts (`split("\n").length` over engine10 minus modernize/). Headroom is 35 lines.
+- **Prefix today:** E-65 measured 7,136 tokens on sonnet for the engine10 session shape (6 tools, `claude_code` preset, no append; runner/providers.ts:488-492). There is no per-stage first-turn figure yet (S41-04).
+- **Where the cost goes (MEDIUM-ANALYSIS section 4):** implement cache reads are 9.23M against raw's 4.79M. Plan, Wall and intake are 31% of spend. Plan runs on opus in the eval: `tier: "fast"` has no pin, and the override sets FAST to opus (session.ts:44-49).
+- **Agent SDK 0.3.283** has a PostToolUse `updatedToolOutput` that "replaces the tool output before it is sent to the model" and "works for all tools" (sdk.d.ts:2674-2681). Trimming has a real mechanism.
+
+## 1. Prerequisite ruling (CTO, before S41-05 merges)
+
+Core cannot absorb items 1 and 2: the new logic is about 600 lines against 35 lines of headroom. Proposal:
+- New modules go in `loki-ts/src/e10ext/`, outside engine10/, with a 1,500-line cap added to `budget.test.ts` next to modernize/.
+- Core may import e10ext/. e10ext/ may not import stages/.
+- Nothing that decides a verdict leaves core: Seal, the authoritative verify and the Wall seal stay in engine10/. e10ext/ only picks which diff reaches them and which model runs.
+- Because verify and Seal re-check whatever is chosen, a wrong pick costs accuracy but can never produce a false seal.
+- A D-entry records the ruling. If the CTO refuses, S41-05, 08, 10, 12 and 13 put their modules under engine10/ and offset every added line.
+
+## 2. Fixed metric definitions (identical before and after)
+
+- **Rows:** harness `dedupe` applied per rep file (it keys on model, harness_sha, task and arm, so run over concatenated reps it would merge r1 to r3), then E-98f's rule: one row per (arm label, rep file, task), the latest `status: ok` row, else the latest row. Arm label = `<arm>@<model>`. A task that is `task_invalid` anywhere is dropped from every arm.
+- **Completion** = completed / evaluated, using the harness's own `completed` and `is_evaluated`.
+- **Cost per completed** = sum of `cost_usd` over evaluated rows / completed rows.
+  - Provider and partial-stream dollars both count; the table shows the partial-stream share of spend.
+  - Any evaluated row still null makes the figure n/a, and n/a is always red.
+- **p50** = nearest-rank p50 of `time_to_pr_s` over completed rows (harness `nearest_rank`).
+- **Tokens per completed, by type:** input, cache read, cache write and output, each summed over evaluated rows / completed. Reported, not graded.
+- **Comparability.** The tool refuses to mark a pair unless all of these hold:
+  - the same task set with the same n per task;
+  - the same `--parallel` and cap;
+  - arms ran back to back, never concurrently;
+  - reused raw rows have an equal `git hash-object eval/loki10/harness.py`. Both hashes are printed.
+- **Marks.** Completion green if Loki >= raw; cost green if Loki <= raw; p50 green if Loki <= raw. The tier verdict is green only if all three are green. The headline pair, loki-sonnet against raw-opus, is green on cost only at Loki <= 0.5x raw.
+- **METRICS.md columns:** tier, pair, n, the three Loki/raw pairs with marks, verdict, tokens by type, partial share, date, harness hash, command.
+
+## 3. How the harness picks the model per arm today (item B)
+
+- **Knob: `LOKI_EVAL_MODEL`** (harness.py:1264).
+  - Unset, it takes the first `tier: "planning"` entry in providers/model_catalog.json. Today that is `claude-opus-5-5`, but `claude-opus-5` is also planning-tier, so the default depends on catalog order.
+  - The value must have a `cli_aliases` entry (harness.py:1267).
+- **raw-claude arm:** `--model <id>` (harness.py:1137).
+- **v10 arm:** `LOKI_MODEL_OVERRIDE=<id>` plus `LOKI_SESSION_MODEL=<alias>` (harness.py:523-525).
+  - session.ts:44-49 copies the override into PLANNING, DEVELOPMENT and FAST. The E-64 cascade then pins implement to `wallModel()`, sonnet by default (sizing.ts).
+  - Fix escalation targets `ctx.model` (fix.ts), so on loki-sonnet nothing can escalate today. S41-08 adds `LOKI_E10_TOP_MODEL`.
+- **Arms, all with pinned ids:** raw-sonnet = `LOKI_EVAL_MODEL=claude-sonnet-5 --arm raw-claude`; raw-opus = `LOKI_EVAL_MODEL=claude-opus-5-5 --arm raw-claude`; loki-sonnet = `LOKI_EVAL_MODEL=claude-sonnet-5 --arm v10` (plus `LOKI_E10_TOP_MODEL=opus` after S41-08); loki-opus = `LOKI_EVAL_MODEL=claude-opus-5-5 --arm v10` (the pair for raw-opus).
+
+## 4. Definitions for item 1
+
+**Deterministic failure.** A verify output has one when some check `c` meets all six conditions:
+1. `c.result == "fail"`: in verify.ts `runCheck`, it failed and then failed its immediate rerun. `flaky` (fail then pass) and `not_run` (missing tool, timeout, abort) never count.
+2. `c.name` is a test check, not `lint:*` and not `select-tests` (the same test as fix.ts `isTestFailure`).
+3. `c.interpreter != "system"`.
+4. The test file of `c` existed at baseSha or is a sealed Wall file. A test the attempt wrote itself never counts.
+5. `c.exit_code` is not a pytest non-test exit (2, 3, 4, or 5 = no tests collected). S41-08 records `exit_code` (+2 core lines). Without it, collection errors would buy top-model rounds.
+6. It reproduced: the same `c.name` failed in the previous verify after one cheap fix round, or it failed in both parallel attempts.
+
+Only then does a fix round run on the top model. With `MAX_FIX_ROUNDS = 2` (types.ts:25), a single attempt escalates in round 2 at the earliest. When both attempts failed the same check, escalation comes in round 1.
+
+**Attempt selection (two attempts).**
+- **Shared set S** = sealed Wall tests plus `impacted(testmap, changed_A plus changed_B)`, keeping only tests that exist at baseSha.
+  - Each attempt is scored in the primary tree, in turn: apply its diff, run S, reset (S41-12).
+  - An attempt is never scored in its own worktree: that worktree has no untracked task `.venv`, and a shared editable venv would test attempt A's `src/`.
+- **Disqualify an attempt when:** its session was killed or ended in error; its diff is empty (without already_done evidence); or its diff edits or deletes an existing test function or a Wall file (seal.ts weakened-test rule).
+- **Rank lexicographically:**
+  1. Wall passes, descending.
+  2. Deterministic-fail count in S (conditions 1 to 3 and 5), ascending.
+  3. Pass count in S on the project interpreter, descending.
+  4. Flaky count, ascending.
+  5. Lint fails, ascending.
+  6. Diff size (added plus deleted non-blank lines), ascending.
+  7. Attempt index, ascending.
+- **Early accept:** the first finished attempt with S non-empty and every S and Wall check `pass` is accepted, and the other session is killed. The killed session is still priced (partial-stream).
+- **Gate:** two attempts only for size `normal` when the predicted S (Wall enabled, or impacted tests from named or relevant files) is non-empty. With no selector there is no second attempt.
+- **Fallback:** if both attempts are disqualified, keep A. Verify and Seal decide as usual.
+
+## 5. Slice cards
+
+Each card lists: goal; files; Wall check (red first, then the eval measurement); tier and budget; dependencies; net core lines.
+
+### Wave 1 (start now: measurement, harness fixes, moat P0; 7 cards, disjoint files)
+
+**S41-01 Harness: cost, knobs, row fields.**
+- Goal:
+  - (a) `provider_cost` accepts `cost_source` of `provider` or `partial-stream` and records `cost_partial_usd`.
+  - (b) The allowlist gains `LOKI_E10_CASCADE`, `_TOP_MODEL`, `_ATTEMPTS`, `_TRIM`, `_PREFIX`, `_CONTEXT` and `_WALL_PARALLEL` now, so no later card touches harness.py.
+  - Rows gain: `tier` (from `_task_tier`); `tokens{input, cache_read, cache_write, output}` (raw from `arm_stdout` usage, v10 from `cost` events with a `by_stage` split); `first_turn_prompt_tokens` from result-cost files when present; and `escalations` and `attempts` from events.
+- Files: eval/loki10/harness.py, eval/loki10/test-harness.sh.
+- Wall check:
+  - Red first: a fixture efficiency dir with one partial-stream record gives `cost_usd` None before the change and a number after. `LOKI_E10_CASCADE=0` reaches `arm_env` output.
+  - Positive controls: pricing the preserved E-98f null row (e98f-engine/default-r1/pub-werkzeug-3121, `iteration-2.json` usd 0.0628) gives non-null, and re-pricing all three preserved E-98f arms from e98f-engine gives the first full cost table (deliverable; no eval spend). It decides whether "cheapest capable model" survives: nocascade's unreliable lower bound ($0.32) sits below default's ($0.70). The token harvest over ev15-raw-claude-r1/r2 and ev15-engine reproduces 4.79M raw and 9.23M v10-implement cache reads (MEDIUM-ANALYSIS section 4) to within 1%.
+- HIGH (measuring instrument); 30 min. Dependencies: none. Core: 0.
+
+**S41-02 Scorecard tool.**
+- Goal: `eval/loki10/scorecard <label=results.jsonl>... [--append docs/v10/METRICS.md]` prints one table per tier, per section 2.
+- Files: eval/loki10/scorecard (python; imports harness `dedupe` and `nearest_rank`), eval/loki10/test-scorecard.sh.
+- Wall check:
+  - Red first: synthetic rows cover every mark colour, n/a-is-red, and refusal on unequal task sets or harness hashes.
+  - Positive control: replaying the preserved E-98f files reproduces 15/21 at 209s, 16/21 at 214s and 15/21 at 138s. Replaying EV-15 reproduces raw 10/14 and v10 9/14.
+- MEDIUM; 30 min. Dependencies: none (reads the S41-01 fields when present). Core: 0.
+
+**S41-03 Arm runner plus auth guard (fixes c).**
+- Goal: `eval/loki10/scorecard-run.sh --tier T --n N --arms raw-sonnet,raw-opus,loki-sonnet,loki-opus --out DIR`.
+  - Runs arms back to back with pinned ids (section 3), the same `--parallel` for all, and one `--out` per rep.
+  - Before each arm and rep it reads only the keychain `expiresAt`, never the token. If fewer than cap x ceil(tasks/parallel) + 600 s remain, it waits until expiry is under 300s, runs one operator `claude -p ok --model claude-haiku-4-5` to force a refresh, and re-reads.
+  - Documents the arms in README.md.
+- Files: eval/loki10/scorecard-run.sh, eval/loki10/test-scorecard-run.sh, eval/loki10/README.md.
+- Prefers `ANTHROPIC_API_KEY` when set (`arm_auth` already takes it first). The under-300s refresh threshold is unverified (E-98f saw no refresh at about 415s), so the card starts with one live check of `expiresAt` before and after the operator call.
+- Wall check: red first, a stub `security` and `claude` with expiry 900s shows the wrapper waits, refreshes and never starts run.sh early. A dry run prints the 4 pinned commands.
+- HIGH (auth); 30 min. Dependencies: none. Core: 0.
+
+**S41-04 First-turn prefix measurement.**
+- Goal: the SDK stream parser records the first assistant message's `input + cache_creation + cache_read` as `first_turn_prompt_tokens` in the result-cost file.
+- Files: loki-ts/src/runner/sdk_stream_parser.ts and its test under loki-ts/tests/runner/.
+- Wall check: red first, a 3-message fixture stream gives the first message's sum, not the total. Then one loki-sonnet run of 1 small and 1 medium task records the per-stage prefix in METRICS.md.
+- LOW; 15 min. Dependencies: none. Core: 0.
+
+**S41-05 Attempt-selection rule (pure function).**
+- Goal: `selectAttempt(attempts, S, wall)` in e10ext/select.ts, implementing section 4 exactly. Adds the e10ext cap to budget.test.ts.
+- Files: loki-ts/src/e10ext/select.ts, loki-ts/tests/e10ext/select.test.ts, loki-ts/tests/engine10/budget.test.ts.
+- Wall check: red first, one test per rank key, per disqualifier, for early accept and for "attempt-authored test ignored". Mutating any key order turns a test red.
+- MEDIUM; 30 min. Dependencies: the section 1 ruling. Core: 0.
+
+**S41-12 Tree swap helper.**
+- Goal: `snapshotDiff`, `resetToBase` and `applyDiff` in e10ext/treeswap.ts, for the primary tree.
+  - Covers tracked and untracked files; excludes `.loki/`, `.venv/`, `venv/` and the attempts dir.
+  - Snapshots are in memory or under runDir, never `git stash` (the stash stack is shared across worktrees).
+  - Attempt worktrees live under `<runDir>/attempts/` and are removed by exact path.
+- Files: loki-ts/src/e10ext/treeswap.ts and its test.
+- Wall check: red first, a round trip over a fixture with a modified, a new and a deleted file plus a venv restores the tree byte for byte, and `git status` is clean after reset. A mutation that skips untracked files goes red.
+- HIGH (it feeds what Seal sees); 45 min. Dependencies: none. Core: 0.
+
+**S41-16 P0: Wall red-on-base is vacuous without `python` (confirmed).**
+- Found: wall.ts `RealBaseTestRunner` runs `RUNNER_CMD.pytest = "python -m pytest -q <files>"` through `/bin/sh`; any throw counts as `fail`. On this host `command -v python` finds nothing and `/bin/sh -c 'python -m pytest ...'` exits 127, so every pytest Wall test "fails on base" by launch error: red-first is vacuous and the clean-base `already_satisfied` path can never fire. It also skips the task venv (E-98a's verify defect, still live in Wall).
+- Goal: the base runner uses verify `runnerCmd` (project interpreter first); a missing tool or exit 127 is `not_run`, which refuses the seal (NOT PROVEN), never counts as red. Delete the `RUNNER_CMD` table.
+- Files: loki-ts/src/engine10/stages/wall.ts and its test.
+- Wall check: red first, with no `python` on PATH and a `.venv` shim, a Wall test that passes on base is sealed today and refused after; with no interpreter at all the result is not_run. Eval: medium n=2, report Wall seal count against E-98f.
+- HIGH (moat); 30 min. Dependencies: none. Core: -3.
+
+### Wave 2 (after S41-01 to S41-04 merge; 5 cards)
+
+**S41-06 Baseline scorecard (eval only).**
+- Goal: run small and medium, n=2, all 4 arms through S41-03, and append the "before" tables through S41-02. This is the reference for every later card.
+- Files: docs/v10/METRICS.md.
+- Wall check: every row has a non-null cost, or the table says n/a with the row ids. Raw rows are reused later only while the harness hash matches.
+- MEDIUM; 90 min of eval. Dependencies: S41-01 to S41-04. Core: 0.
+
+**S41-07 Plan on the cheap model.**
+- Goal: with the cascade on, plan pins `model: cascadeImplementModel()`, the same pin Wall and intake already use (wall.ts:139, already_done.ts:177).
+- Files: loki-ts/src/engine10/stages/plan.ts and its test.
+- Wall check: red first, a session-options test shows the plan model is sonnet under `LOKI_MODEL_OVERRIDE=claude-opus-5-5`. Eval: loki-opus on medium, n=2; plan cost per run down at least 60% against S41-06, completion not lower.
+- MEDIUM; 15 min. Dependencies: S41-06 for the eval. Core: +1.
+
+**S41-08 Top model and deterministic-failure escalation.**
+- Goal: `LOKI_E10_TOP_MODEL` (an alias, default `opus`, through `resolveModelAlias`) replaces `ctx.model` as the escalation target. `isDeterministicFailure(prevVerify, verify, baseTests, wallPaths)` in e10ext/escalate.ts implements section 4. verify.ts records `exit_code` per check.
+- Files: loki-ts/src/e10ext/escalate.ts, loki-ts/src/engine10/stages/fix.ts, loki-ts/src/engine10/stages/verify.ts, and their tests.
+- Wall check:
+  - Red first: flaky, not_run, lint, system-interpreter, exit 5 and attempt-authored failures never escalate. A reproduced project-interpreter fail escalates in round 2, and loki-sonnet reaches opus where today it cannot.
+  - Eval: loki-sonnet on medium, n=2; report the escalation rate and the cost share of escalated rounds.
+- HIGH (verify); 45 min. Dependencies: S41-06. Core: +4.
+
+**S41-09 Lean stable prefix (flag).**
+- Goal: `LOKI_E10_PREFIX=lean` gives engine10 sessions a short fixed string system prompt instead of the `claude_code` preset. The tool list is unchanged.
+- Files: loki-ts/src/runner/providers.ts and its test.
+- Wall check:
+  - Red first: the options test shows `systemPrompt` is a string under the flag and the preset without it.
+  - Eval: `first_turn_prompt_tokens` down at least 40% on every stage; completion on medium, n=2, not below S41-06.
+  - Kill criterion: if S41-04 shows the prefix is under 10% of implement cache reads, close with the measurement and build nothing.
+- MEDIUM; 30 min. Dependencies: S41-04. Core: 0.
+
+**S41-14 Wall in parallel with implement (item 4, promoted for speed).**
+- Goal:
+  - FLOW becomes `intake, plan, [wall, implement], verify`.
+  - Wall still writes only in its temp dir. Its copy-in and its red-on-base check run after implement completes: snapshot, reset to base, run the Wall tests (they must fail), restore.
+  - The `already_satisfied` short-circuit moves to that post-implement check.
+- Files: loki-ts/src/engine10/machine.ts, loki-ts/src/engine10/stages/wall.ts, and their tests.
+- Wall check: red first, implement's `stage.started` precedes Wall's `stage.completed`, sealed sha256 values are unchanged, and a Wall test passing on base is still refused after the move. Eval: medium p50 not worse than S41-06, completion not lower, and report Wall's share of wall-clock. E-98f argues against a big win: nowall p50 214s against default 209s, nocascade (opus implement) 138s, so the time is in implement, not Wall.
+- HIGH (moat: the red-first Wall); 45 min. Dependencies: S41-12, S41-16. Core: +6.
+
+### Wave 3 (2 cards)
+
+**S41-10 Relevant context and a static-first brief.**
+- Goal: the implement and fix briefs:
+  - put the fixed rules first, as a block that is byte-identical across stages and runs;
+  - replace the first 200 repo paths (`repoMapText`) with up to 20 relevant files: plan `relevant_files`, else `selectRelevantFiles` over the tree-keyed cached map (cache.ts `repomap-<tree>.json`);
+  - add their impacted tests, with one exact command per test from verify `runnerCmd` (project interpreter included).
+- Files: loki-ts/src/e10ext/context.ts, loki-ts/src/engine10/stages/implement.ts, loki-ts/src/engine10/stages/fix.ts, and their tests.
+- Wall check: red first, the brief for a 2,000-file fixture is under 3 KB and names `.venv/bin/python -m pytest -q <test>`. Eval: loki-sonnet on medium, n=2; implement cache reads per completed down at least 30% against S41-06, completion not lower.
+- MEDIUM; 30 min. Dependencies: S41-08 (fix.ts). Core: +1.
+
+**S41-11 Rule-based trajectory trimming (flag).**
+- Goal: `LOKI_E10_TRIM=1` adds a PostToolUse hook that returns `updatedToolOutput`. It only shortens new results; rewriting history would break the cache prefix and is out of scope.
+- Rules:
+  - A Bash result over 200 lines keeps the first 40 and the last 120, plus a `[loki trimmed N lines]` marker.
+  - A Read over 400 lines keeps the first 400.
+  - A Grep over 100 matches keeps the first 100.
+  - From tool call 25 on, the limits halve.
+  - A Bash command that exits nonzero always keeps its last 120 lines.
+- Files: loki-ts/src/runner/trim.ts, loki-ts/src/runner/providers.ts, and their tests.
+- Wall check: red first, each rule, the nonzero-exit tail guarantee, and unchanged pass-through when the flag is off. Eval: implement cache reads per completed down at least 25% on medium, n=2, completion not lower.
+- MEDIUM; 30 min. Dependencies: S41-09 (providers.ts). Core: 0.
+
+### Wave 4 (1 card)
+
+**S41-13 Two parallel attempts (flag).**
+- Goal: with `LOKI_E10_ATTEMPTS=2`, implement runs 2 cheap-model sessions concurrently when the section 4 gate allows.
+  - A runs in repoDir; B runs in a detached worktree at baseSha under runDir.
+  - Each is scored in the primary tree through S41-12. S41-05 picks the winner, which is applied to repoDir. Verify and Seal then run as today.
+  - A check that failed in both attempts feeds S41-08 condition 6.
+- Files: loki-ts/src/e10ext/attempts.ts, loki-ts/src/engine10/stages/implement.ts (hook), and their tests.
+- Wall check:
+  - Red first: fake sessions cover early accept, full ranking, the both-disqualified fallback, and gate-off with no selector. For a run with a killed attempt, the sum of `cost` events equals the harness `cost_usd`.
+  - Eval: loki-sonnet on medium, n=3 (E-98f's n); completion at least 18/21 (raw's best rate, EV-14), and the tier verdict not red on cost.
+- HIGH; 45 min. Dependencies: S41-05, S41-08, S41-10, S41-12. Core: +6.
+
+### Wave 5 (1 card)
+
+**S41-15 After scorecard and defaults.**
+- Goal: rerun S41-06 with every flag. Flip a flag's default in `sizing.ts` only when the tier verdict with it is at least as green as without it, on both small and medium.
+- Files: loki-ts/src/engine10/sizing.ts, docs/v10/METRICS.md.
+- Wall check: the before and after tables sit side by side in METRICS.md. The D41 flip happens only on green for small and medium. The headline row (loki-sonnet against raw-opus) is reported either way.
+- HIGH (default flow); 90 min of eval plus 15 min of code. Dependencies: all of the above. Core: +3.
+
+**Core total:** -3 +1 +4 +6 +1 +6 +3 = +18 of 35 headroom. A card over its estimate offsets the excess in the same slice (D33). **Wave counts:** W1 7, W2 5, W3 2, W4 1, W5 1 (16 cards). Only S41-06 and S41-15 are eval-only.
+
+## 6. Risks
+
+- **Seal accuracy (top risk).** S41-12, S41-13 and S41-14 change which tree Seal sees and when the Wall is proven red, so a treeswap bug could seal an unverified tree. Guards: verify and Seal always re-run on the final primary tree; Seal's sha256 check on Wall files is unchanged; red-on-base stays mandatory, only later; each card carries a mutation test that must go red.
+- **Wall red-on-base is vacuous today (S41-16).** Until it merges, Wall seals on this host prove nothing about red-first; S41-14 must not move the check before S41-16 lands.
+- **Cost measurement.** Partial-stream dollars are estimates: the share is shown, and over 20% is flagged. Attempts and killed sessions write cost from two cwds; S41-13's event-sum check guards this. Without S41-01, every null row keeps cost red forever.
+- **Attempts can raise cost.** Implement is already 69% of v10 spend. Two attempts pay off only if S41-10 and S41-11 cut per-attempt reads first and early accept often kills B. S41-15 keeps the flag off unless the verdict says otherwise.
+- **p50 comparability.** E-98f ran up to 12 concurrent sessions against EV-15's 3. The scorecard refuses unequal `--parallel`, arms run back to back, and keychain waits fall outside the timed rows.
+- **Noise.** At n=21 one run flips a mark; the table prints n. **Escalation starvation:** condition 6 delays opus to round 2; the eval reports the share of round-2 wins that escalation bought.
+- **Accuracy regressions from trimming and a lean prefix.** Both are flags with completion floors in their Wall checks. Trimming never cuts a failing command's tail.
+
+## 7. Later waves (items 3 to 5, sketches only)
+
+**Item 3, repo memory.** Extend cache.ts's per-repo dir (maps already keyed by tree) with the verified build and test command per runner (the one that ran on the project interpreter), the flaky list (`recordFlaky` exists), the top failure causes (`topFailures` exists, but only the brief reads it) and conventions mined from accepted diffs. Inject one fixed-size block into S41-10's static-first brief. Measure cold against warm: the same task twice, cache wiped then kept; warm must be cheaper and faster on the scorecard.
+
+**Item 4, speed.** Wall in parallel is S41-14. Still to do: cache dependency install by lockfile hash (venv or node_modules restored by `sha256(lockfile)`), keep fast verify to impacted tests only with a cap on the changed-test-files list, and pipeline plan into implement on the lean path. Target: p50 at or below raw on every tier.
+
+**Item 5, backlog throughput.** `loki <label|milestone>` fetches N issues (fetch_issue.ts) and runs one supervisor per issue, each in its own worktree with a single writer, opening one PR per issue. Total spend and concurrency are bounded by the D39 usage governor. Measure issues per hour and cost per merged PR against raw run serially.