loki-mode 9.78.0 → 9.80.1

package/docs/v10/METRICS.md

@@ -217,3 +217,29 @@ remains the only cap on this step. Guarded by
 ## Over-budget stops
 
 - 2026-09-27T21:33Z: batch 8 workflow wf_ff8d537c-40e stopped. Its last two reviewers (S-191, S-195) ran about 55 minutes against the 30-minute reviewer budget with no verdict (pulse AGENT_OVER_BUDGET at 21:31Z: "S-191 building MEDIUM (55.9 min, budget 30 min)"). Re-dispatched as two standalone reviewers with 20 and 25 minute scopes. Load average was 15 to 17 during the stall.
+
+## Loki 10 eval: first real arm numbers (2026-09-27, smoke, not the gate)
+
+Raw `claude -p` arm (EV-4), model claude-opus-5-5, 900s cap, 3 public tasks, harness from the EV-1 branch with EV-3 isolation (auth via keychain access token; no credential in any result file):
+
+| task | completed | time to PR | cost (provider-reported) |
+|---|---|---|---|
+| pub-more-itertools-1192 | yes | 20s | $0.1266 |
+| pub-click-2877 | yes | 34s | $0.1861 |
+| pub-humanize-152 | yes | 45s | $0.2704 |
+
+`summarize --markdown`: 3/3 completed, p50 34s, p90 45s, $0.1944 per completed task, 0 capped. This is a 3-task smoke, not the 29-task gate. It sets the bar the v10 fast lane must meet on completion and cost per completed task (D29), which is far tighter than the 5-minute p50 target.
+
+## Loki 10 eval: raw claude -p arm, full 29 tasks (2026-09-27 23:05-23:16Z)
+
+Model claude-opus-5-5, harness 84c22568, 900s cap, --parallel 3, EV-3 isolation. Results: ~/loki-ci-logs/eval-raw-claude-20260927T231653Z/.
+
+| Arm | Completed | Rate | p50 time to PR | p90 time to PR | Cost per completed | Cost measured | Capped |
+|---|---|---|---|---|---|---|---|
+| raw-claude | 27/29 | 93.1% | 39s | 68s | $0.2363 | 29/29 | 0 |
+
+Misses: pub-click-3059 (hidden tests failed: wrong metavar bracketing), pub-humanize-174 (hidden tests failed: rounding boundary cases). Total provider-reported spend $6.38.
+
+## Loki 10 engine: first real run (E-14 smoke, not the gate)
+
+pub-more-itertools-1192 through the E-14 glue entry, real claude, --no-pr: 30s wall (intake 0s, plan 16s, implement 12s, verify 3s, commit and seal under 1s), hidden test passes, $0.4486 over 2 sessions (plan $0.2264, implement $0.2223), verdict PARTIAL only because ruff is not installed. Raw claude on the same task: 15s, $0.1290. The planner session costs as much as the implement session.

package/docs/v10/PROGRESS.md

@@ -1248,3 +1248,31 @@ sample.
 - Eval numbers so far: none (no arm has run yet).
 - ETA for v10.0.0: thin path end to end targeted for about 01:00 UTC; gate decision at 03:00 UTC.
 - Top blocker: E-01 (shared types) gates phase B; augmentiq has only 3 issues, so the augmentiq share of the eval is 1 task, not 5.
+
+## 2026-09-27T22:57Z: Loki 10 engine progress
+- On main (`git ls-remote origin refs/heads/main` = 694866fc): engine wave 1 E-01..E-13 except E-14, plus E-11 (both halves), E-16, E-20; the eval harness EV-1 with EV-3 isolation and the 29 EV-2 tasks (EV-7 nonce fixes). tests/engine10 plus spawn guard 185 pass, 0 fail; eval/loki10/test-harness.sh 77 passed.
+- Eval baseline: a no-op arm over all 29 tasks graded every task not completed, none task_invalid (108s). No real arm numbers yet: EV-4 (raw claude -p, 3 tasks) and EV-5 (legacy, 1 task) are running.
+- In flight: E-14 end to end plus the first real v10 run on pub-more-itertools-1192; reworks E-15, E-17, E-18, E-21, E-22; E-19, E-23, E-TG.
+- Incidents fixed forward: two orphan loki test runs over 24 hours (stopped by PID; E-00 guard merged); main CI red on Coverage from engine spawns without env (72423684, 19128f6b).
+- ETA for v10.0.0: first real v10 task within the next hour; gate decision at 03:00 UTC.
+- Top blocker: E-14 integration (first time the modules run together).
+
+## 2026-09-27T23:03Z: first real eval numbers
+- Raw claude -p arm, 3 public tasks: 3/3 completed, p50 34s, p90 45s, $0.1944 per completed task (docs/v10/METRICS.md). Legacy arm and v10 arm not yet measured.
+- Implication for the gate: v10 must match raw completion and cost per completed task. Any extra model call (planner, Wall author, fix round) adds cost against a $0.19 baseline, so the fast lane must skip stages a small task does not need.
+
+## 2026-09-27T23:27Z: Loki 10 progress
+- Raw claude -p full arm: 27/29 (93.1%), p50 39s, p90 68s, $0.2363 per completed task (docs/v10/METRICS.md).
+- v10 first real task (E-14 glue entry, commit 9d9ff7d2): fixed in 30s, hidden test result "1 passed, exit 0", $0.45 (plan $0.23 plus implement $0.22; `cost-summary.py --json` fully_measured true) against raw $0.13 on the same task.
+- On main (`git ls-remote origin refs/heads/main` = 44731e0f): every engine slice except E-14's glue, plus E-41 local-origin push; v9.79.0 tagged 23:12:55.
+- In flight: E-42 (real entry end to end, Rule of Two for the pr stage, harness v10 arm on one task), E-43 cost knobs, E-44 output fixes, E-33..E-35, EV-5 legacy full arm.
+- ETA: first harness-scored v10 task by about 00:15Z; full v10 arm by about 01:30Z; gate decision 03:00Z.
+- Top blocker: cost. Plan plus implement is about 2x raw per task on this sample; the gate requires v10 cost per completed task at or below raw ($0.2363).
+
+## 2026-09-28T00:11Z: Loki 10 progress
+- On main (`git ls-remote origin refs/heads/main` = addda547): E-42 real entry end to end (run.ts glue deleted), E-44 output fixes, agent SDK 0.3.283 (9db2a4ed), E-51 eval PR path regression, and train E12 (23b20e6f): E-33 gate report, EV-6 publish script, E-38 legacy arm pin, E-39 interrupt/resume, E-40 live PR smoke. Train checks: engine10 349 pass 0 fail, tsc exit 0, test-harness 79/0, test-gate-report 34/0.
+- SDK route: claude-opus-5-5 now runs through the SDK (probe: `success OK`); 0.3.267 rejected it. The same probe cost $0.33 for a one-word reply, so fixed per-call overhead, not task work, drives v10 cost.
+- Legacy arm (EV-5): 3 results so far (2 hidden pass), each at the 900s cap; runner alive at parallel 3, ETA about 02:15Z. Incident: 6 then 2 legacy watchdog loops escaped the harness timeout group (ppid 1, 30 min); stopped by PID; guard slice EV-10.
+- In flight (wave E13, 12 builders): E-45 cost path rework, E-36 preflight (plus E-37), E-32 rebase, E-34 docs refresh, E-52..E-56, E-58 (makes E-47 green), E-59 (makes E-50 green), EV-10.
+- ETA: E-45 merge about 00:45Z, EV-8 5-task cost check right after, EV-9 full v10 arm about 01:00Z to 01:45Z, gate decision 03:00Z.
+- Top blocker: cost per completed task (raw $0.2363); E-45 plus the per-call overhead finding decide it.