loki-mode 10.5.17 → 10.5.20

package/docs/v10/METRICS.md

@@ -534,3 +534,39 @@ profiling implement's own internal turn-by-turn timing (not available from
 
 Command: `python3 eval/loki10/stage-profile.py
 ~/loki-ci-logs/eval/e98f-engine/{default,nocascade,nowall}-r{1,2,3}`
+
+## D50 model-lift baseline (small tier)
+
+Measured 2026-09-30 on main c5eaddb0 (harness_sha c5eaddb0, clean; loki-ts deps via `bun install --frozen-lockfile`), claude 2.1.286, 1 rep, 900s cap, EV-3 isolation, provider-reported cost only, hidden tests decide completion. Models from providers/model_catalog.json: claude-haiku-4-5 and claude-sonnet-5. Opus arms not run. The Loki arm is `v10` at default knobs (LOKI_ENGINE=v10, same model pinned via LOKI_EVAL_MODEL).
+
+Subset: the small tier has 29 tasks; 4 arms x 29 would be 116 runs, over the 40-run budget, so the first 10 small-tier tasks by id are used (40 runs total): aiq-52-searchbar, pub-click-2877, pub-click-3059, pub-click-3487, pub-click-3572, pub-humanize-152, pub-humanize-174, pub-humanize-333, pub-jsonschema-1389, pub-markupsafe-417. aiq-52-searchbar is an `expected_outcome: no_change_needed` task. n=10 per arm, 1 rep: every rate has a wide interval (one task is 10 points); this is a baseline, not a gate. Machine load average was 24-37 during the run (more than 2 concurrent arms never ran); wall times are inflated and noisy.
+
+| arm | attempted | completed | rate | median wall, all runs | median wall, completed | total cost | cost per completed | timed out |
+|---|---|---|---|---|---|---|---|---|
+| raw haiku-4-5 | 10 | 4 | 40% | 103.2s | 100.5s | $2.3124 | $0.5781 | 0 |
+| Loki+haiku (v10) | 10 | 7 | 70% | 123.0s | 127.7s | $3.4649 (1 run unmeasured) | $0.4950 | 0 |
+| raw sonnet-5 | 10 | 9 | 90% | 111.6s | 118.0s | $4.9239 | $0.5471 | 0 |
+| Loki+sonnet (v10) | 10 | 5 | 50% | 92.9s | 128.8s | $3.7278 (1 run unmeasured) | $0.7456 | 0 |
+
+Cost per completed = total spend of all 10 runs (failures included) / completed. The 2 unmeasured v10 runs are the aiq-52-searchbar runs that exited in about 1s before any model call (no spend, no provider cost record).
+
+Model lift (D50 targets):
+
+| comparison | completion | cost per completed | time (median, all runs) | verdict |
+|---|---|---|---|---|
+| Loki+haiku vs raw haiku | 70% vs 40% (+30 points, 3 tasks) | $0.4950 vs $0.5781 (-14%) | 123.0s vs 103.2s (1.19x) | lift, within 1.2x |
+| Loki+sonnet vs raw sonnet | 50% vs 90% (-40 points, 4 tasks) | $0.7456 vs $0.5471 (+36%) | 92.9s vs 111.6s (0.83x) | LOSS: below raw X |
+| Loki+haiku vs raw sonnet (cross) | 70% vs 90% (-20 points) | $0.4950 vs $0.5471 (-10%); total $3.46 vs $4.92 (-30%) | 123.0s vs 111.6s | target (>= raw sonnet) NOT met |
+| Loki+sonnet vs raw opus | not run | | | |
+
+Failures by category (every failed or non-completed cell listed; no run timed out or was capped; all rows status ok):
+- raw haiku (6): hidden tests failed with a PR pushed: pub-click-2877, pub-click-3059, pub-humanize-174, pub-humanize-333, pub-jsonschema-1389. aiq-52-searchbar: hidden regression test passed but it pushed a PR on a no-change task (wrong outcome).
+- raw sonnet (1): hidden tests failed with a PR: pub-click-3059.
+- Loki+haiku (3): aiq-52-searchbar: engine refused to start in 1.3s, "dirty tracked tree: M frontend/package-lock.json" (task `setup` npm install rewrote a tracked file; no model call). pub-humanize-174: outcome BLOCKED (spec conflict claimed against existing tests), no PR. pub-humanize-333: PR opened, hidden tests failed.
+- Loki+sonnet (5): aiq-52-searchbar: same dirty-tree refusal in 0.7s. pub-click-2877: outcome ALREADY_SATISFIED claimed, no PR, hidden tests fail (false already-done). pub-humanize-174: FAILED "empty diff without an already_done marker", no PR. pub-click-3059 and pub-humanize-333: PR opened, hidden tests failed.
+
+Findings: (1) the v10 dirty-tree refusal is a harness-vs-engine defect that costs both Loki arms the aiq task (the `no_change_needed` outcome is unreachable when `setup` dirties a tracked file); not fixed in this slice. (2) Loki+sonnet loses to raw sonnet mainly through no-PR outcomes (ALREADY_SATISFIED false positive, empty diff) and BLOCKED, not through slow runs. (3) Per D50 a slice that lowers lift is dropped; this baseline is the bar.
+
+Reproduce (per arm; a run-owned temp is created by run.sh; needs `cd loki-ts && bun install --frozen-lockfile` for the v10 arm):
+`LOKI_NO_BROWSER=1 LOKI_EVAL_MAX_LOAD=80 LOKI_EVAL_MODEL=<claude-haiku-4-5|claude-sonnet-5> eval/loki10/run.sh --arm <raw-claude|v10> --tasks aiq-52-searchbar,pub-click-2877,pub-click-3059,pub-click-3487,pub-click-3572,pub-humanize-152,pub-humanize-174,pub-humanize-333,pub-jsonschema-1389,pub-markupsafe-417 --parallel 1 --out ~/loki-ci-logs/d50-<arm>-<model>`
+Raw rows: ~/loki-ci-logs/d50-{raw-haiku,raw-sonnet,v10-haiku,v10-sonnet}/results.jsonl.

package/docs/v10/PROGRESS.md

@@ -1705,3 +1705,45 @@ The swarm is stopped: .loki/V10-STOP exists, the loop has no wake-ups, and the c
 - Trains: train/24 1f3c9da1 RED (E-142 board sweep: E-161 row had a `||`, fixed in 7cfd7399); train/25 b1d2ecbd superseded (same row); train/26 6d288543 = local main d449a419+ (A-117, E-157 r4, E-159 r2, D49-auto, D51-A3, D51-A4, D48-r10, D48-r6, board fix) in CI. When green: ff origin/main to 6d288543, cut 10.5.17; D49-auto then auto-promotes later releases.
 - In review: A-118 r3 b48adeacc (opus). Rework (fresh agents): A-121b r3 (sys.path guard), A-119b r3 (diff-line asserts, scoped exclude), D51-A12 r2 (Host allowlist vs DNS rebinding). Built, unreviewed or unpushed: D48-r1 gaming matrix de71397dd (needs a CTO call on the pre-red case and a duration row), D48-r2 b073d50a (opus review pending), E-162 6b9738fdc (pushed; sonnet review pending), E-163 678f2162e (unpushed; sonnet review pending). Building: D48-flip, D50 baseline eval, DOC-01, DOC-02.
 - Rules: slice pushes go from the main checkout only (the agent pre-push guard blocks agents); cancel queued slice CI for merged slices; any slice adding tests/*.sh runs the registration, E-00, shard-coverage and shellcheck guards before merge; no `|` or `||` inside BOARD cells.
+
+## 2026-10-01T03:51Z train/26 red, drops, train/27 cut; D50 baseline
+- train/26 6d288543 red: (1) E-133 map guard, committed maps had ../../ sources from a symlinked node_modules build: dist rebuilt in main f370d9aa0 (test-release-dist-guard 13/0); (2) D48-r6 doctor: caveman unsuppressed claude subcall, spawn-env guard (spawnSync without env), --fix test fails on CI: DROPPED (reverts 14ada6db2, 1ca5abd6a). A-119b r3 merge reverted 7a2467d7f (core 5024 over cap; re-land as A-119c with code out of core and add-based weakening closed).
+- A-121b r3 merged 8714c988f (one-line write keeps core 4999; verify_cmd/log_seal/budget/roundtrip 55/0, quick-receipt-order 26/0). train/27 8714c988 pushed (guards: shellcheck 0, v10-ops 67/0, shard-coverage 19/0).
+- D50 baseline merged 871ff5dc7 (METRICS.md): raw haiku 4/10, Loki+haiku 7/10 (cost/completed -14%, 1.19x time), raw sonnet 9/10, Loki+sonnet 5/10 (LOSS). Defect E-164 (lockfile dirtied by setup blocks v10 start) building.
+- Rework: A-118 r4 (3 narrow cases), D51-A12 r3 (health exempt, deploy hosts, 27 tests). DOC-01 a13ca60be done, unreviewed.
+
+## 2026-10-01T04:29Z D48 report (deadline 04:35Z) and release state
+- Releases: v10.5.17 published (Release success; npm 10.5.17 gitHead 78d8713b) and AUTO-PROMOTED by promote.yml (event workflow_run, success): dist-tags latest=10.5.17, next=10.5.17. First D49 auto-promotion.
+- train/28 red (D51-A12 host check 403 in non-dashboard tests; E-164 spawn without env): both dropped, fixed (D51-A12 r4 af58ceead, E-164 r2 be5b70070) and re-landed on main. train/29 red on Python 3.10 only (A-118 treats any pyproject edit as config when tomllib is missing): A-118 dropped (bb9658a15), r5 in progress. train/30 bb9658a1 (D51-A12 UI, E-164, DOC-02 doc sweep) in CI.
+- D48 rows (acceptance test status, honest):
+  1 gaming matrix: NOT DONE. Test built (D48-r1 de71397dd) but not merged; 20 expected-fail cases remain (A-117 shipped; A-118 r5 and A-119c still open; pre-red case needs a CTO call).
+  2 portable receipt: PARTIAL. A-117 (tamper detection incl. tail truncation) shipped in 10.5.17; `loki keys export` + `loki verify --pubkey` built (b073d50a), not yet reviewed or merged.
+  3 quiet and fast (<=8 lines, <=1.5x raw time): NOT DONE (not measured; D50 baseline shows Loki+haiku 1.19x raw haiku time).
+  4 commits only the fix on node/pytest/go repos: NOT STARTED.
+  5 exit ladder + --json schema: NOT DONE (A-118 r5 in progress; schema not started).
+  6 doctor --fix and <2s: NOT DONE (built, dropped from train/26 on CI: unsuppressed claude subcall, spawn env, test failing on CI).
+  7-9 v10 default engine + non-null cost: NOT DONE (flip slice built? builder has not reported).
+  10 loki-seal installable from the repo marketplace: PASS (shipped in 10.5.17, on latest; clean-HOME install verified).
+- D51 Phase A on latest (10.5.17): `loki backlog` + loki.yaml + Slack (v10 path). Pending on train/30: bare `loki` opens the UI with PAT onboarding and backlog complete-all.
+- D50: baseline in METRICS.md (Loki+haiku 7/10 vs raw haiku 4/10; Loki+sonnet 5/10 vs raw sonnet 9/10, a LOSS). E-164 (setup-dirtied lockfile) fixed.
+- Pacing: from 04:35Z the governor paces to about 0.48%/h of the week (about 7 engineers).
+
+## 2026-10-01T04:33Z LEAD HANDOFF STATE (paced mode from 04:35Z, about 4 engineers)
+- npm: latest=next=10.5.17 (78d8713b, auto-promoted). origin/main=78d8713b. train/30 bb9658a1 in CI (D51-A12 UI, E-164, DOC-02); when green: ff main, cut 10.5.18 (D49 auto-promotes).
+- Local main ahead of train/30: A-118 re-landed with r5 (cdc8213df, 3.10 pyproject fallback), loki-seal README install (f8ba34a34). Next train after train/30.
+- Built, not merged: D48-r1 gaming matrix de71397dd (CTO call on pre-red), D48-r2 keys export/--pubkey b073d50a (opus review), E-162 slice CI tier-down 6b9738fdc (pushed, review), E-163 live /usage governor 678f2162e (review), DOC-01 r2 README (agent finishing), D48-r6 doctor --fix (dropped; needs: caveman suppression, spawn env, CI test fix).
+- Open rows: A-119c (add-based helper weakening, keep core under cap), A-121c (CTO), A-134c, E-160 (CTO), E-161 guard, E-165 (P0 guard: mirofish test launches a live build), E-166 Jira wiring, D48 rows 3-9, D50 harness items, D51 Phase B/C.
+- Rules this session learned: re-land a dropped slice by reverting the revert, then merging the fix; rebuild dist in the main checkout after any merge touching loki-ts (no symlinked node_modules); no `|` or `||` in BOARD cells; run the registration, shard, spawn-env, dist-guard and docs-drift checks before every train.
+
+## 2026-10-01T04:58Z P0: v10.5.18 release blocked by CodeQL
+- v10.5.18 (981d9734) Release run FAILED at required-ci: Security Audit failed on CodeQL alert 607 py/command-line-injection (critical) at dashboard/api_start.py:120, plus path-injection 604-606 (lines 69, 75), all from D51-A12 onboarding (provider name from the request reaching subprocess and a file path). Not published; npm latest/next stay 10.5.17.
+- Fix forward (main already contains it): slice-P0-codeql (constant allowlists for the provider binary and secret file names) in progress; then train with it, then re-cut the release.
+- train/31 731a335d is green on Tests/Parity/Coverage but its Security Audit fails on the same alert; it will not release until the fix lands.
+- D48-flip rejected by opus (bare loki verify still legacy after a default v10 run: FAILED run reads VERIFIED); r2 in progress.
+
+## 2026-10-01T05:22Z v10.5.19 cut (CodeQL fix forward), train/33 pushed
+- train/32 aad61f9a green: Tests, Bun Parity, Coverage (baseline), Security Audit all completed success (watcher bs3ybh02p). main fast-forwarded to aad61f9a.
+- v10.5.19 release commit 47050d65 pushed by SHA (parent == origin/main; --check-clean rc=0). It carries the unpublished 10.5.18 content plus the CodeQL 604-607 fix, A-118 r5 and DOC-01. Release run and auto-promote watcher: bqimqxkgo.
+- Merged on local main: D48-flip r3 (opus APPROVE: FAILED receipt exits 4 with or without --allow-unsigned; dispatch 56/0; gate --stub 0 failed), A-134b r4 (sonnet APPROVE; registration rc=0, shard rc=0, own test 6/0), E-162 (opus APPROVE as CTO call).
+- train/33 63db8710 pushed. Guards: registration rc=0, shard-coverage rc=0 (E2e Features row 19 -> 20 to keep the five heaviest suites in distinct shards), v10-ops rc=0, run-shellcheck rc=0, docs-cli-drift rc=0, release-dist-guard rc=0, budget.test 24 pass.
+- REJECTED, fix rounds building: E-163 (live /usage timeout has no process-group kill), D48-r2 (unsigned receipt exits 0 under --pubkey; merge conflicts with the exit-4 change).

package/docs/walkthrough/README.md

@@ -11,16 +11,10 @@ Interactive HTML walkthroughs demonstrating Loki Mode's capabilities. Each file
 | `comparison.html` | **Feature Comparison** -- Side-by-side comparison of Loki Mode vs alternative AI build tools with interactive feature matrix | 26 KB |
 | `gallery.html` | **Project Gallery** -- Showcase of 24 example projects built with Loki Mode across SaaS, CLI, mobile, and enterprise categories | 37 KB |
 | `video-placeholder.html` | **Build Process Animation** -- CSS-animated walkthrough of the build process with terminal replay and phase transitions | 25 KB |
-| `build-demo.html` | **Full Build Video Demo** -- Replay of a complete SaaS dashboard build showing all 12 iterations with timeline and progress tracking | 18 KB |
-| `ide-demo.html` | **Purple Lab IDE Demo** -- Interactive mockup of the browser-based Purple Lab IDE with file explorer, editor, and Loki Mode status panel | 17 KB |
-| `enterprise-demo.html` | **Enterprise Features Demo** -- Audit logging, RBAC, security architecture, compliance, and governance capabilities for organizations | 13 KB |
-| `provider-race-demo.html` | **Multi-Provider Race Demo** -- Animated race comparing Claude, Codex, and Gemini building the same app simultaneously | 17 KB |
-| `memory-demo.html` | **Memory System Demo** -- Interactive exploration of episodic, semantic, and procedural memory layers with progressive disclosure | 19 KB |
-| `dashboard-demo.html` | **Dashboard Monitoring Demo** -- Real-time monitoring dashboard mockup with build log, agent status, task queue, and KPI tracking | 17 KB |
 
 ## Hub Page
 
-Open `hub.html` for a landing page that links to all walkthroughs with descriptions and categories.
+Open `hub.html` for a landing page that links to the walkthroughs with descriptions and categories.
 
 ## How to View