loki-mode 11.3.0 → 11.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -650
- package/SKILL.md +4 -4
- package/VERSION +1 -1
- package/autonomy/completion-council.sh +2 -2
- package/autonomy/context-tracker.py +4 -4
- package/autonomy/council-v2.sh +1 -1
- package/autonomy/grill.sh +1 -1
- package/autonomy/lib/done-recognition.sh +1 -1
- package/autonomy/lib/prd-enrich.sh +1 -1
- package/autonomy/lib/voter-agents.sh +1 -1
- package/autonomy/loki +99 -75
- package/autonomy/run.sh +30 -18
- package/autonomy/sandbox.sh +1 -1
- package/autonomy/tui.sh +6 -5
- package/autonomy/verify.sh +1 -1
- package/bin/loki +166 -59
- package/completions/_loki +2 -0
- package/completions/loki.bash +2 -2
- package/dashboard/__init__.py +1 -1
- package/dashboard/server.py +25 -31
- package/docs/ARCHITECTURE.md +337 -0
- package/docs/BRANDING.md +124 -0
- package/docs/CLI-REFERENCE.md +7 -1
- package/docs/COMPONENTS.md +447 -0
- package/docs/DEPLOYMENT.md +9 -9
- package/docs/DOCKER.md +469 -0
- package/docs/INSTALLATION.md +67 -1045
- package/docs/KNOWN-ISSUES.md +11 -35
- package/docs/TESTING.md +161 -0
- package/docs/air-gapped.md +1 -2
- package/docs/alternative-installations.md +14 -24
- package/docs/architecture/DECISIONS.md +333 -0
- package/docs/audit-logging.md +7 -7
- package/docs/authentication.md +3 -3
- package/docs/authorization.md +1 -1
- package/docs/certification/04-production-deployment/lab.md +6 -6
- package/docs/certification/04-production-deployment/lesson.md +4 -4
- package/docs/certification/README.md +1 -1
- package/docs/control-plane-container.md +2 -2
- package/docs/dev/real-run.md +34 -0
- package/docs/dev/release-checklist.md +2 -2
- package/docs/exit-codes.md +3 -3
- package/docs/git-workflow.md +3 -3
- package/docs/metrics.md +37 -55
- package/docs/network-security.md +1 -1
- package/docs/siem-integration.md +5 -5
- package/docs/v10/CLI-MODERN.md +4 -3
- package/docs/v10/DECISIONS.md +44 -0
- package/docs/v10/DEPS.md +22 -22
- package/docs/v10/ENGINE.md +1 -1
- package/docs/v10/FAILURE-CLASSES.md +118 -1
- package/docs/v10/FOUNDER-QUEUE.md +1 -0
- package/docs/v10/GUARDS.md +13 -0
- package/docs/v10/LEGACY-REMOVAL.md +4 -6
- package/docs/v10/METRICS.md +2 -2
- package/docs/v10/MODERNIZE.md +1 -1
- package/docs/v10/OPERATING-MODEL.md +2 -1
- package/docs/v10/PROGRESS.md +29 -0
- package/docs/v11/COST-HALF.md +352 -0
- package/docs/v11/MARKET-SLICES.md +315 -0
- package/docs/v11/REGISTRIES.md +730 -0
- package/loki-ts/data/model-pricing.json +52 -24
- package/loki-ts/dist/loki.js +1036 -957
- package/mcp/__init__.py +1 -1
- package/package.json +1 -1
- package/packages/control-plane/dist/server.js +2467 -736
- package/plugins/loki-mode/.claude-plugin/plugin.json +1 -1
- package/src/protocols/mcp-client.js +6 -2
- package/src/protocols/mcp-server.js +13 -3
- package/tools/audit-docs.py +1 -1
- package/web-app/dist/assets/{AdminPage-D_e45uTw.js → AdminPage-CMkHNJbF.js} +1 -1
- package/web-app/dist/assets/{Avatar-T3nIpoaf.js → Avatar-BJyjV7Zi.js} +1 -1
- package/web-app/dist/assets/{Badge-BKqb59pP.js → Badge-D572l0uq.js} +1 -1
- package/web-app/dist/assets/{Button-DjVlkh8_.js → Button-T4BnOZgl.js} +1 -1
- package/web-app/dist/assets/{CockpitPage-DXJgLGmh.js → CockpitPage-Cj_GFsNn.js} +1 -1
- package/web-app/dist/assets/{ComparePage-BRP_eVc_.js → ComparePage-A8m4ZXak.js} +1 -1
- package/web-app/dist/assets/{ErrorBoundary-CLjpGLjC.js → ErrorBoundary-hugCg5yg.js} +1 -1
- package/web-app/dist/assets/{EvidenceReceiptPanel-DX-vb4Xy.js → EvidenceReceiptPanel-CuJKDC2G.js} +1 -1
- package/web-app/dist/assets/{GitHubIssuesPanel-214HVHtX.js → GitHubIssuesPanel-BYGKjarC.js} +1 -1
- package/web-app/dist/assets/{GitHubPRsPanel-BEiZKc4t.js → GitHubPRsPanel-c7sFIJlf.js} +1 -1
- package/web-app/dist/assets/{HomePage-VvMHS5ga.js → HomePage-A1MIkG2P.js} +2 -2
- package/web-app/dist/assets/{LoginPage-5WD-v7n3.js → LoginPage-8KQMRhtR.js} +1 -1
- package/web-app/dist/assets/{MagicPage-DUQT_Jye.js → MagicPage-C05QW2Lx.js} +1 -1
- package/web-app/dist/assets/{MetricsPage-lga-z79v.js → MetricsPage-Oukv5hme.js} +1 -1
- package/web-app/dist/assets/{NotFoundPage-CGhBX3G1.js → NotFoundPage-DKIwsawz.js} +1 -1
- package/web-app/dist/assets/{ProjectPage-CdzF2GzZ.js → ProjectPage-DwFRPLCG.js} +3 -3
- package/web-app/dist/assets/{ProjectsPage-DRG64N8s.js → ProjectsPage-CgY4YYDX.js} +1 -1
- package/web-app/dist/assets/{SettingsPage-5fzW9Frb.js → SettingsPage-BpfcSjUA.js} +1 -1
- package/web-app/dist/assets/{ShowcasePage-DrTdxO4y.js → ShowcasePage-BqBakC4K.js} +1 -1
- package/web-app/dist/assets/{SystemSettingsPage-C9Mqs6Rd.js → SystemSettingsPage-CZLu5VB0.js} +1 -1
- package/web-app/dist/assets/{TeamsPage-CUGPdSWo.js → TeamsPage-D55S1jKz.js} +1 -1
- package/web-app/dist/assets/{TemplatesPage-CNraDY0F.js → TemplatesPage-D3SydvxN.js} +1 -1
- package/web-app/dist/assets/{TerminalOutput-BIu_6Pea.js → TerminalOutput-hNg8G1-e.js} +1 -1
- package/web-app/dist/assets/{activity-Bk-kgqaf.js → activity-D2C0VagO.js} +1 -1
- package/web-app/dist/assets/{bell-BXmUdmo8.js → bell-CMvYi36w.js} +1 -1
- package/web-app/dist/assets/{bot-CiuzXxUR.js → bot-BvtQ6lmo.js} +1 -1
- package/web-app/dist/assets/{check-BEzjXppd.js → check-B13xkXed.js} +1 -1
- package/web-app/dist/assets/{chevron-left-AYWCXHf8.js → chevron-left-emu6r49E.js} +1 -1
- package/web-app/dist/assets/{circle-alert-BNg2uSRq.js → circle-alert-CExlwjuy.js} +1 -1
- package/web-app/dist/assets/{clock-DTj0kkGT.js → clock-D7Uas3oL.js} +1 -1
- package/web-app/dist/assets/{cloud-x4nTDBLT.js → cloud-CkB4S86C.js} +1 -1
- package/web-app/dist/assets/{code-xml-BVh_MLrk.js → code-xml-SL9IIWIn.js} +1 -1
- package/web-app/dist/assets/{database-DdfBaowG.js → database-BQFkd7t1.js} +1 -1
- package/web-app/dist/assets/{dollar-sign-XuYFmAOY.js → dollar-sign-qHmnNYsU.js} +1 -1
- package/web-app/dist/assets/{file-code-corner-D4cRnmeL.js → file-code-corner-C2YUDP0K.js} +1 -1
- package/web-app/dist/assets/{file-plus-CyoVlHc_.js → file-plus-C_ooIkDe.js} +1 -1
- package/web-app/dist/assets/{globe-Y6IK2RPW.js → globe-CQ-GL485.js} +1 -1
- package/web-app/dist/assets/{hammer-BKzr9VZI.js → hammer-U05L_D7V.js} +1 -1
- package/web-app/dist/assets/{index-B20cSx5G.js → index-CooHd7I-.js} +3 -3
- package/web-app/dist/assets/{layers-DebNOhGS.js → layers-Buo17rjW.js} +1 -1
- package/web-app/dist/assets/{loader-circle-DyGbrEgd.js → loader-circle-15vK3oV1.js} +1 -1
- package/web-app/dist/assets/{lock-BKPpyt20.js → lock-BeXLebuJ.js} +1 -1
- package/web-app/dist/assets/{package-egF5Wi4y.js → package-MVh_6-6R.js} +1 -1
- package/web-app/dist/assets/{plus-OiEgKIc-.js → plus-Dh_nd9vS.js} +1 -1
- package/web-app/dist/assets/{refresh-cw-CzKvrONq.js → refresh-cw-C9cBJEAT.js} +1 -1
- package/web-app/dist/assets/{rotate-ccw-Bu2mII0p.js → rotate-ccw-D3PAtzST.js} +1 -1
- package/web-app/dist/assets/{scroll-text-CSkiyax8.js → scroll-text-DZCoXQjo.js} +1 -1
- package/web-app/dist/assets/{server-DlPthJFw.js → server-OCOLI5J_.js} +1 -1
- package/web-app/dist/assets/{shield-alert-CbpHWQQX.js → shield-alert-oVhmf1_h.js} +1 -1
- package/web-app/dist/assets/{thumbs-up-DgGpZe-Q.js → thumbs-up-B-kbX4vN.js} +1 -1
- package/web-app/dist/assets/{trash-2-BV3GlCB-.js → trash-2-EL6moveW.js} +1 -1
- package/web-app/dist/assets/{trending-down-Y4TJtXhD.js → trending-down-Cl3kjpsx.js} +1 -1
- package/web-app/dist/assets/{trending-up-DsDNoHZY.js → trending-up-DST7I6M2.js} +1 -1
- package/web-app/dist/assets/{upload-gkThrc_I.js → upload-CUOT780G.js} +1 -1
- package/web-app/dist/assets/{usePolling-jpNSeNAe.js → usePolling-Djmt90eb.js} +1 -1
- package/web-app/dist/assets/{user-3Zez1WOK.js → user-BRBohA6J.js} +1 -1
- package/web-app/dist/index.html +1 -1
- package/docs/ARCHITECTURE-OVERVIEW.md +0 -209
- package/docs/AUTONOMI-ECOSYSTEM.md +0 -107
- package/docs/COCKPIT-SPEC.md +0 -72
- package/docs/COCKPIT-UX-MANDATE.md +0 -53
- package/docs/COMPETITIVE-ANALYSIS.md +0 -303
- package/docs/COMPETITIVE-NEXT-10.md +0 -316
- package/docs/COMPETITIVE-RALPH-ZEROSHOT-2026-07-08.md +0 -81
- package/docs/DASHBOARD-9.12-EVIDENCE.md +0 -97
- package/docs/DASHBOARD-ARCHITECTURE.md +0 -425
- package/docs/DASHBOARD_V2_CHECKLIST.md +0 -243
- package/docs/DEMOS.md +0 -63
- package/docs/ENTERPRISE-IDENTITY-ROADMAP.md +0 -206
- package/docs/EVIDENCE-RECEIPT-BUILD-APPLICABILITY.md +0 -79
- package/docs/FEAT-PRDREUSE-DOCKER-PLAN.md +0 -144
- package/docs/GATE-FAILURE-TRIAGE.md +0 -254
- package/docs/HANDOFF-2026-08-03.md +0 -439
- package/docs/LOOP-CANDIDATE-PROPOSAL-v1.md +0 -167
- package/docs/LOOP-HARNESS-AUDIT.md +0 -684
- package/docs/MERGE-DEDUP-MAP.md +0 -420
- package/docs/MERGE-ROUTE-MAP.md +0 -139
- package/docs/MODEL-REGISTRY-PLAN.md +0 -155
- package/docs/OUTCOME-FRONTIER.md +0 -536
- package/docs/P2-SPEC-ROBUSTNESS-PLAN.md +0 -192
- package/docs/PLANS-INDEX.md +0 -32
- package/docs/PRD-REUSE-DONE-RECOGNITION-PLAN.md +0 -536
- package/docs/PROMPT-ABLATION-RESULT.md +0 -97
- package/docs/R4-TRUST-TRAJECTORY-DESIGN.md +0 -127
- package/docs/R5-AUTO-WIKI-DESIGN.md +0 -137
- package/docs/R6-ROLLBACK-CHECKPOINT-PLAN.md +0 -107
- package/docs/R7-ZERO-CONFIG-FIRST-RUN-PLAN.md +0 -137
- package/docs/R8-SHAREABLE-TEAM-ASSETS-PLAN.md +0 -129
- package/docs/R9-OPEN-CORE-HOOKS-PLAN.md +0 -113
- package/docs/RARV-C-100X-PLAN.md +0 -286
- package/docs/RARV-C-LOOP-EFFICIENCY-PLAN.md +0 -470
- package/docs/SHAREABLE-PROOF-PLAN.md +0 -194
- package/docs/SONNET5-DEFAULT-PLAN.md +0 -157
- package/docs/SPEED-AND-FIRST-PASS-PLAN.md +0 -281
- package/docs/STALE-STATE-AUDIT.md +0 -174
- package/docs/SYNERGY-TASKS.md +0 -143
- package/docs/TASKLIST-ACCURACY-PLAN.md +0 -355
- package/docs/TOP-100-BACKLOG.md +0 -159
- package/docs/V8-ACCEPTANCE-TRIAGE-2026-07-24.md +0 -114
- package/docs/V8-COMPLEXITY-AUDIT-2026-07-24.md +0 -45
- package/docs/V8-MAJOR-RELEASE-PLAN.md +0 -137
- package/docs/V8-OVERNIGHT-PLAN-2026-07-25.md +0 -82
- package/docs/V8-SDK-RESEARCH-RAW.md +0 -129
- package/docs/WELCOME-OPENER-PLAN.md +0 -106
- package/docs/adoption-baseline-2026-07-31.md +0 -84
- package/docs/architecture/bmad-integration-epic.md +0 -271
- package/docs/architecture/bmad-integration-review.md +0 -86
- package/docs/architecture/bmad-integration-validation.md +0 -249
- package/docs/architecture/bmad-loki-voice-agent-council-analysis.md +0 -61
- package/docs/auto-claude-comparison.md +0 -250
- package/docs/bug-fixes/agent-01-cli-fixes.md +0 -101
- package/docs/bug-fixes/agent-02-purplelab-fixes.md +0 -88
- package/docs/bug-fixes/agent-03-dashboard-fixes.md +0 -119
- package/docs/bug-fixes/agent-04-memory-fixes.md +0 -105
- package/docs/bug-fixes/agent-05-provider-fixes.md +0 -86
- package/docs/bug-fixes/agent-06-integration-fixes.md +0 -101
- package/docs/bug-fixes/agent-07-dash-run-fixes.md +0 -101
- package/docs/bug-fixes/agent-08-docker-fixes.md +0 -164
- package/docs/bug-fixes/agent-09-e2e-build-fixes.md +0 -69
- package/docs/bug-fixes/agent-10-e2e-fullstack-fixes.md +0 -102
- package/docs/bug-fixes/agent-11-e2e-session-fixes.md +0 -70
- package/docs/bug-fixes/agent-12-scenario-fixes.md +0 -120
- package/docs/bug-fixes/agent-13-enterprise-fixes.md +0 -143
- package/docs/bug-fixes/agent-14-uat-newuser-fixes.md +0 -88
- package/docs/bug-fixes/agent-15-uat-poweruser-fixes.md +0 -132
- package/docs/bug-fixes/agent-19-code-review.md +0 -316
- package/docs/bug-fixes/agent-20-architecture-review.md +0 -331
- package/docs/competitive/bolt-new-analysis.md +0 -579
- package/docs/competitive/emergence-others-analysis.md +0 -605
- package/docs/competitive/replit-lovable-analysis.md +0 -622
- package/docs/dashboard-guide.md +0 -487
- package/docs/enterprise/migration.md +0 -265
- package/docs/history/one-run-audit.md +0 -304
- package/docs/plans/FORGE-AUTONOMOUS-QUEUE.md +0 -312
- package/docs/plans/ULTRAPLAN-FORGE-BAAS.md +0 -612
- package/docs/plans/UT2-6-LSP-DIAGNOSTIC-BROADCAST.md +0 -394
- package/docs/prd-purple-lab-ide.md +0 -152
- package/docs/prd-purple-lab-platform-v2.md +0 -223
- package/docs/prd-purple-lab-platform.md +0 -319
- package/docs/retrospectives/v7.5.15-fleet-postmortem.md +0 -325
- package/docs/retrospectives/v7.5.15-honesty-audit.md +0 -136
- package/docs/retrospectives/v7.5.15-llm-failure-modes.md +0 -49
- package/docs/show-hn-post.md +0 -46
- package/docs/test-scenarios/enterprise-scenarios.md +0 -732
- package/docs/thick2thin.md +0 -173
|
@@ -1,157 +0,0 @@
|
|
|
1
|
-
# Sonnet 5 Default - Implementation Plan
|
|
2
|
-
|
|
3
|
-
Status: SHIPPED (v7.104.0; kept as design history). Worktree: `../loki-sonnet5-wt` (branch `feat/sonnet5-default`).
|
|
4
|
-
Board decision: `strategy/roadmap/13-BOARD-2026-06-30.md`. Verified facts: `reference-sonnet5-model-ids` memory.
|
|
5
|
-
Release class: MINOR (new default execution model for every user). Full local-ci + council 3/3 + release ladder.
|
|
6
|
-
|
|
7
|
-
## 1. Goal (founder directive, 2026-06-30)
|
|
8
|
-
|
|
9
|
-
Make Sonnet 5 the DEFAULT execution model for Loki runs. Advisor stays Opus (opt-in / when needed).
|
|
10
|
-
User can set all tiers to haiku|sonnet|opus from the dashboard at start, and hot-switch mid-run without
|
|
11
|
-
interrupting executions. Protect the "never lies about done" trust moat: the switch must NOT weaken the
|
|
12
|
-
verdict.
|
|
13
|
-
|
|
14
|
-
## 2. Ground truth (verified against source 2026-06-30)
|
|
15
|
-
|
|
16
|
-
- Live model resolver: `resolve_model_for_tier()` (providers/claude.sh:390) reads
|
|
17
|
-
`PROVIDER_MODEL_{PLANNING,DEVELOPMENT,FAST}` (claude.sh:65-67) which chain from `CLAUDE_DEFAULT_*`
|
|
18
|
-
(claude.sh:55-57). Today: PLANNING=opus, DEVELOPMENT=opus (comment "was sonnet"), FAST=sonnet.
|
|
19
|
-
- `get_provider_tier_param` case block (run.sh:~2447) with hardcoded `development) echo opus` is the
|
|
20
|
-
INERT legacy fallback, reachable only when claude.sh is unsourced (guard run.sh:2414). Flip anyway
|
|
21
|
-
for consistency.
|
|
22
|
-
- Trust-gate code-review reviewers: `_dispatch_reviewer()` (run.sh:~10148) sets
|
|
23
|
-
`_rv_argv=("--dangerously-skip-permissions")` and deliberately passes NO `--model` -> reviewers run
|
|
24
|
-
on the ACCOUNT DEFAULT model = non-deterministic, config-dependent. The in-code comment (run.sh:~10164)
|
|
25
|
-
explicitly says a future `--model` here "must be pinned to opus, never fable" (Fable refuses cyber
|
|
26
|
-
content, ends -p turn with stop_reason refusal). Bun route mirrors this: quality_gates.ts:28 + :1223.
|
|
27
|
-
- Hot-switch mid-run EXISTS: `.loki/state/model-override` read each iteration (run.sh:16007-16070),
|
|
28
|
-
written by dashboard `POST /api/session/model` (server.py:2852), allowlist haiku|sonnet|opus|fable,
|
|
29
|
-
clamped by `loki_apply_max_tier_clamp` (claude.sh:347). Fresh `claude -p` per iteration = clean switch.
|
|
30
|
-
- Sonnet 5 = alias `claude-sonnet-5` (CLI resolves aliases to latest; loki passes alias verbatim at
|
|
31
|
-
claude.sh:334). Opus 4.8 = `claude-opus-4-8`. Pricing (server.py:6174, run.sh:4847/12153): sonnet
|
|
32
|
-
$3/$15, opus $5/$25, haiku $1/$5, fable $10/$50 - all correct. model_catalog.json has stale IDs
|
|
33
|
-
opus-4-7, sonnet-4-6.
|
|
34
|
-
- Parity-locked: `tests/test-model-override.sh` + dashboard estimator (`_clamp_to_max_tier`, `_max_tier`)
|
|
35
|
-
are byte-locked to the resolver. Flipping defaults MOVES the quote -> expected values must update.
|
|
36
|
-
- Dashboard-ui: no start-time model picker (grep empty). mid-run switcher UI also appears absent
|
|
37
|
-
(POST endpoint exists, no UI consumer found).
|
|
38
|
-
|
|
39
|
-
## 3. Tier -> model mapping AFTER the change
|
|
40
|
-
|
|
41
|
-
| Tier (RARV phase) | Today | After |
|
|
42
|
-
|--------------------------|--------------------------|----------------------------------------|
|
|
43
|
-
| planning (REASON) | opus | sonnet (Opus opt-in via advisor path) |
|
|
44
|
-
| development (ACT/REFLECT)| opus | sonnet |
|
|
45
|
-
| fast (VERIFY) | sonnet | sonnet (haiku via LOKI_ALLOW_HAIKU) |
|
|
46
|
-
| trust gate (council/review)| account default (float)| account default (UNCHANGED); opus via OPT-IN `LOKI_ADVISOR_MODEL` |
|
|
47
|
-
| advisor / reviewer pin | (does not exist) | `LOKI_ADVISOR_MODEL` (default unset = today's behavior; set=opus) |
|
|
48
|
-
| Opus-for-execution | default | opt-in (dashboard "all opus" / env) |
|
|
49
|
-
|
|
50
|
-
### CORRECTION (advisor review 2026-06-30) - D2 is OPT-IN, not always-on
|
|
51
|
-
|
|
52
|
-
The original board D2 ("always pin reviewers to --model opus") is REJECTED as written, for four reasons:
|
|
53
|
-
1. NOT causally forced by the Sonnet flip: `_dispatch_reviewer` reads neither `tier_param` nor
|
|
54
|
-
`CLAUDE_DEFAULT_*`, so flipping the builder default does NOT change the reviewer model. The judge
|
|
55
|
-
floating with the account default is a PRE-EXISTING condition, not something this PR exposes.
|
|
56
|
-
2. UNCOSTED: pinning 3 reviewers + devil's-advocate + security-sentinel to Opus ADDS Opus-priced tokens
|
|
57
|
-
on top of the builder savings. Opus ~1.7x sonnet. This could eat or exceed the "~40% cheaper" story.
|
|
58
|
-
Net cost is UNKNOWN until we measure reviewer-token vs builder-token volume for a typical `loki start`.
|
|
59
|
-
3. CONTRADICTS the founder's words: "advisor will be opus IF NEEDED OR USER OPTED" = conditional/opt-in.
|
|
60
|
-
Always-on removes the implicit cost control users have today (account default) - opposite of the
|
|
61
|
-
adoption/cost intent.
|
|
62
|
-
4. "Deterministic verdict" is the wrong framing: LLM reviewers are never deterministic; this is a
|
|
63
|
-
consistency/quality argument, not determinism.
|
|
64
|
-
|
|
65
|
-
Moat guarantee (corrected): default behavior of the trust gate is UNCHANGED by this release (no
|
|
66
|
-
cost regression, no surprise). Users who want a strong, consistent Opus judge set `LOKI_ADVISOR_MODEL=opus`
|
|
67
|
-
(or the dashboard "advisor: Opus" toggle), which pins the reviewer/override-judge to Opus. This is the
|
|
68
|
-
founder's "opus if needed or user opted", made real and opt-in. Lane B and Lane C COLLAPSE into this one
|
|
69
|
-
opt-in advisor path.
|
|
70
|
-
|
|
71
|
-
## 4. Per-file change list (disjoint lanes for parallel principal engineers)
|
|
72
|
-
|
|
73
|
-
### Lane A - Execution default flip (providers/claude.sh) [engineer 1]
|
|
74
|
-
- claude.sh:56 `CLAUDE_DEFAULT_DEVELOPMENT="opus"` -> `"sonnet"`.
|
|
75
|
-
- claude.sh:55 `CLAUDE_DEFAULT_PLANNING="opus"` -> `"sonnet"` (see open question Q1 - founder may want
|
|
76
|
-
planning to stay opus; design so this one line is a trivial revert).
|
|
77
|
-
- claude.sh:57 FAST stays sonnet.
|
|
78
|
-
- Trace + document the clamp ripple: `loki_apply_max_tier_clamp` (claude.sh:347) resolves planning/fable
|
|
79
|
-
DOWN to `PROVIDER_MODEL_DEVELOPMENT`. With dev=sonnet, `LOKI_MAX_TIER=sonnet` now genuinely pins
|
|
80
|
-
sonnet (previously it clamped to opus). Update the comment block at claude.sh:383-386 which currently
|
|
81
|
-
documents the opposite. This is a BEHAVIOR clarification, verify against the parity ports (Lane E).
|
|
82
|
-
- run.sh:~2447 legacy fallback `development) echo opus` -> `sonnet` (inert but consistent).
|
|
83
|
-
|
|
84
|
-
### Lane B+C (COLLAPSED) - OPT-IN advisor/reviewer Opus pin [engineer 2]
|
|
85
|
-
- Introduce `LOKI_ADVISOR_MODEL` (unset default = TODAY'S behavior, no change; set to `opus` = pin).
|
|
86
|
-
- Bash: `_dispatch_reviewer()` (run.sh:~10148): IF `LOKI_ADVISOR_MODEL` is a valid allowlisted alias,
|
|
87
|
-
add `("--model" "$LOKI_ADVISOR_MODEL")` to `_rv_argv`; else leave unset (current behavior). Honor the
|
|
88
|
-
existing comment (opus ok, never fable - validate the alias is not fable). The override/architect paths
|
|
89
|
-
(which rewrite tier_param, NOT this dispatch) still cannot touch it.
|
|
90
|
-
- Bun: `quality_gates.ts` reviewer dispatch (near :1223): mirror the same opt-in read. BOTH routes match.
|
|
91
|
-
- Update the "reviewers deliberately do NOT pass --model" comments to: default no --model; opt-in pin
|
|
92
|
-
via LOKI_ADVISOR_MODEL; rationale = consistent/strong judge when the user wants it (NOT determinism).
|
|
93
|
-
- Devil's-advocate + security-sentinel inherit the opt-in pin (same dispatch).
|
|
94
|
-
- Dashboard: an "Advisor model" toggle (default: account default; option: Opus) writes LOKI_ADVISOR_MODEL
|
|
95
|
-
into the run env / a state file the dispatch reads. This IS the founder's "opus if needed or user opted".
|
|
96
|
-
- DO NOT make this always-on. Default behavior of the trust gate is unchanged (no cost regression).
|
|
97
|
-
|
|
98
|
-
### Lane D - Dashboard model selection [engineer 4] (follows frontend-ui-engineering skill)
|
|
99
|
-
- Backend: confirm `POST /api/session/model` (server.py:2852) writes `.loki/state/model-override`
|
|
100
|
-
with allowlist haiku|sonnet|opus|fable (already does). Add start-time selection: `start_build`
|
|
101
|
-
(server.py:3234) accepts an optional `model` that writes the override file BEFORE launch (so iteration
|
|
102
|
-
1 picks it up). Validate against the same allowlist + clamp.
|
|
103
|
-
- Frontend (legacy-ui/src): add an accessible model selector (haiku/sonnet/opus) on the start/new-run
|
|
104
|
-
surface AND a mid-run switcher that calls POST /api/session/model. Real design-system tokens, keyboard
|
|
105
|
-
accessible, no AI-aesthetic. Show current effective model + that switch applies at next iteration.
|
|
106
|
-
Rebuild legacy-ui-static per CLAUDE.md release step 2.
|
|
107
|
-
|
|
108
|
-
### Lane E - Catalog IDs, pricing note, parity tests [engineer 5]
|
|
109
|
-
- model_catalog.json: opus-4-7 -> claude-opus-4-8 (3 occurrences), sonnet-4-6 -> claude-sonnet-5
|
|
110
|
-
(5 occurrences). Keep the alias->tier structure.
|
|
111
|
-
- Pricing: LEAVE the cost NUMBERS at standing $3/$15 for sonnet (over-estimate is the safe direction;
|
|
112
|
-
avoids date-expiry logic that silently breaks after Aug 31 2026). ADD a one-line intro-price NOTE in
|
|
113
|
-
the dashboard pricing display / label ("Sonnet 5 intro $2/$10 through Aug 31 2026") - display only,
|
|
114
|
-
not a computed rate. Update labels "Sonnet (latest)" -> "Sonnet 5".
|
|
115
|
-
- codex gpt-5.3-codex $1.50/$12: UNVERIFIED external number. WebFetch OpenAI pricing to confirm or leave
|
|
116
|
-
with a "verify" note; do not propagate a guess as fact.
|
|
117
|
-
- Confirm gemini fully absent from pricing (deprecated v7.5.18).
|
|
118
|
-
- Parity tests: update `tests/test-model-override.sh` expected values (dev tier now sonnet). Update
|
|
119
|
-
dashboard estimator expected quotes. Any Bun parity test for the resolver. Run the parity matrix.
|
|
120
|
-
|
|
121
|
-
## 5. Test / parity changes enumerated
|
|
122
|
-
- tests/test-model-override.sh: dev-tier resolution + LOKI_MAX_TIER=sonnet clamp expectations flip.
|
|
123
|
-
- Dashboard estimator parity test: per-iteration quote changes (opus->sonnet dev cost).
|
|
124
|
-
- New: a test asserting the trust-gate reviewer dispatch includes `--model opus` on BOTH routes
|
|
125
|
-
(grep/behavioral) - locks the moat guarantee so a future edit can't silently un-pin.
|
|
126
|
-
- New: start_build with model param writes a valid override; invalid model rejected.
|
|
127
|
-
- CLI-invariance: existing `loki verify` / `loki start` still run (no crash); default model differs
|
|
128
|
-
(that IS the intended change - do NOT assert byte-identical default output here).
|
|
129
|
-
|
|
130
|
-
## 6. Calibration protocol (CEO gate - before ship)
|
|
131
|
-
- Pick 5-10 representative specs from benchmarks/ or internal fixtures spanning easy->hard.
|
|
132
|
-
- Run each on Sonnet-default vs current Opus-default (real model calls; NOT in CI - documented manual
|
|
133
|
-
step). Record: iterations-to-council-APPROVE, BLOCK count, wall-clock, and NET cost (builder tokens +
|
|
134
|
-
reviewer tokens, so the "~40% cheaper" claim is verified NET, not just on the builder tier). Append to
|
|
135
|
-
a calibration log under artifacts/sonnet5-calibration/.
|
|
136
|
-
- SHIP criterion: Sonnet-default does not materially increase iterations-to-APPROVE or BLOCK-rate on
|
|
137
|
-
hard specs AND the net cost genuinely drops. If iterations/BLOCK regress: keep the pre-built escape
|
|
138
|
-
hatch (planning->opus one-line revert; dev stays sonnet) and re-measure. Never silently keep Opus -
|
|
139
|
-
make the escalation legible. If net cost does NOT drop as claimed, correct the marketing claim to the
|
|
140
|
-
measured number - never ship an unverified "~40% cheaper".
|
|
141
|
-
|
|
142
|
-
## 7. Release (per CLAUDE.md)
|
|
143
|
-
- Version bump all files (MINOR). Loud CHANGELOG entry: "Default execution model changed Opus -> Sonnet 5.
|
|
144
|
-
~40% cheaper builds. Trust-gate reviewers pinned to Opus (deterministic verdict). Override any model
|
|
145
|
-
from the dashboard." Never claim "same accuracy" (no controlled study).
|
|
146
|
-
- bash scripts/local-ci.sh (must be green). Council 3/3 (2 Opus + 1 Sonnet reviewers that RUN tests).
|
|
147
|
-
- Rebuild dashboard frontend (Lane D). Merge worktree to main, tag via CI, verify 4 channels.
|
|
148
|
-
- NEVER run a loki build from the worktree (self-delete trap). Verify engine changes by function
|
|
149
|
-
extraction to scratch repos.
|
|
150
|
-
|
|
151
|
-
## 8. Open questions for the founder (do NOT block starting)
|
|
152
|
-
- Q1: Should PLANNING tier also default to Sonnet (full flatten), or stay Opus (architecture quality)?
|
|
153
|
-
Plan flattens both per plain reading; Lane A makes planning a one-line revert either way.
|
|
154
|
-
- Q2: "advisor" scope - is it the completion-council override judge (recommended, minimal) or a
|
|
155
|
-
distinct advisory role you envision elsewhere?
|
|
156
|
-
- Q3: Sonnet 5 intro pricing - display-only note (planned) vs actually bill the $2/$10 in estimates
|
|
157
|
-
(rejected: date-expiry bug risk). Confirm display-only is acceptable.
|
|
@@ -1,281 +0,0 @@
|
|
|
1
|
-
# Fastest First-Pass Completion: measured plan
|
|
2
|
-
|
|
3
|
-
Goal (founder, 2026-08-01): **fastest first-pass full completion** -- highest
|
|
4
|
-
quality output, no waiting through a second iteration, faster than Cursor /
|
|
5
|
-
Cognition / Replit on the axis a user actually feels.
|
|
6
|
-
|
|
7
|
-
This plan is built on measurement first, competitor research second, and it
|
|
8
|
-
DISCARDS the intuition I started with.
|
|
9
|
-
|
|
10
|
-
---
|
|
11
|
-
|
|
12
|
-
## 1. What the competition actually does (researched, not assumed)
|
|
13
|
-
|
|
14
|
-
| Vendor | Speed mechanism | Reachable for us? |
|
|
15
|
-
|---|---|---|
|
|
16
|
-
| **Cursor** | Composer: in-house MoE model, ~250 tok/s (~4x GPT-5/Sonnet). MXFP8 training, no post-train quant. Compaction-in-the-loop RL cuts context errors 50%. | **No.** Custom frontier model. |
|
|
17
|
-
| **Cognition** | SWE-1.6 at ~950 tok/s; SWE-bench 51.5%. | **No.** Custom model. |
|
|
18
|
-
| **Replit** | Agent 3: ~2 min to visual preview, ~10 min to first app version. Design Mode <2 min. 2-3x gains across 2025. | **Partly.** Their edge is product/lifecycle, not model. |
|
|
19
|
-
|
|
20
|
-
**The strategic conclusion.** Cursor and Cognition bought speed by TRAINING
|
|
21
|
-
THEIR OWN MODELS. We cannot copy that and should stop pretending the gap is
|
|
22
|
-
closable that way. Replit's edge is time-to-first-visible-thing, which IS
|
|
23
|
-
reachable, because it is a product and lifecycle property.
|
|
24
|
-
|
|
25
|
-
The industry-wide numbers that matter for us, from the latency research:
|
|
26
|
-
|
|
27
|
-
- **Tool execution is 35-61% of total agent request time.** Harness, not model.
|
|
28
|
-
- Parallel tool calling / speculative execution deliver **2-5x latency
|
|
29
|
-
reduction while preserving correctness**.
|
|
30
|
-
- Prompt caching turns repeated-prefix attention from O(n^2) to O(n).
|
|
31
|
-
|
|
32
|
-
That 35-61% is our lane. It is won by architecture, and it does not require a
|
|
33
|
-
frontier lab.
|
|
34
|
-
|
|
35
|
-
## 2. What we actually cost, MEASURED (this is the surprise)
|
|
36
|
-
|
|
37
|
-
Real `stage_complete` events from builds on this machine (n=35):
|
|
38
|
-
|
|
39
|
-
| stage | n | median s | max s | total s |
|
|
40
|
-
|---|---:|---:|---:|---:|
|
|
41
|
-
| **code_review** | 3 | **281** | **504** | **1055** |
|
|
42
|
-
| static_analysis | 5 | 5 | 14 | 26 |
|
|
43
|
-
| security_scan | 5 | 1 | 2 | 4 |
|
|
44
|
-
| lsp_diagnostics | 4 | 1 | 1 | 2 |
|
|
45
|
-
| test_suite | 4 | 0 | 1 | 1 |
|
|
46
|
-
| mutation_integrity | 4 | 0 | 1 | 1 |
|
|
47
|
-
| mock_integrity | 4 | 0 | 0 | 0 |
|
|
48
|
-
| doc_coverage | 3 | 0 | 0 | 0 |
|
|
49
|
-
| magic_debate | 3 | 0 | 0 | 0 |
|
|
50
|
-
| **TOTAL** | | | | **1089** |
|
|
51
|
-
|
|
52
|
-
**`code_review` is 97% of all measured gate time.** Everything else combined is
|
|
53
|
-
34 seconds.
|
|
54
|
-
|
|
55
|
-
### Two intuitions this killed
|
|
56
|
-
|
|
57
|
-
**First:** I was about to propose parallelizing the seven sequential gates at
|
|
58
|
-
`run.sh:22616-22980`. They ARE sequential and mostly independent. The change
|
|
59
|
-
would have been clean, defensible, and worth **about 20 seconds**.
|
|
60
|
-
|
|
61
|
-
**Second, and worse:** I then proposed parallelizing the reviewer council --
|
|
62
|
-
which was **already parallel**. I had grepped the wrong line range, got no
|
|
63
|
-
match, and treated that silence as evidence. A plan whose headline item was a
|
|
64
|
-
no-op.
|
|
65
|
-
|
|
66
|
-
Both were caught by going back to data instead of trusting the story. The
|
|
67
|
-
standing rule for this document:
|
|
68
|
-
|
|
69
|
-
> **No optimization ships without a before-number from this table, and no
|
|
70
|
-
> claim about how the code behaves ships without reading the code that does
|
|
71
|
-
> it.**
|
|
72
|
-
|
|
73
|
-
An absent grep match is not evidence of absence -- it is evidence the grep did
|
|
74
|
-
not match.
|
|
75
|
-
|
|
76
|
-
### Where the real time goes
|
|
77
|
-
|
|
78
|
-
Code review dominates, and the driver is **council size**, not sequential
|
|
79
|
-
dispatch (see the correction in P0 -- my first reading of this was wrong).
|
|
80
|
-
|
|
81
|
-
Measured: 3 reviewers finish in 31s; 6-7 reviewers take 177-502s. The council
|
|
82
|
-
is already concurrent, so what grows is the max-of-N tail plus contention on a
|
|
83
|
-
single provider.
|
|
84
|
-
|
|
85
|
-
This explains the founder's "21 minutes for a simple GitHub issue" better than
|
|
86
|
-
any other measurement taken today: a scoped issue was drawing a 7-member
|
|
87
|
-
council containing two overlapping security reviewers.
|
|
88
|
-
|
|
89
|
-
---
|
|
90
|
-
|
|
91
|
-
## 3. The plan, ranked by measured seconds returned
|
|
92
|
-
|
|
93
|
-
### P0 -- CORRECTED: the council is ALREADY parallel. The cost is its SIZE.
|
|
94
|
-
|
|
95
|
-
**This section originally claimed reviewers ran sequentially and proposed
|
|
96
|
-
parallelizing them. That was wrong, and the correction is recorded here rather
|
|
97
|
-
than quietly edited out.**
|
|
98
|
-
|
|
99
|
-
`run_code_review` forks every reviewer with `) &` (run.sh:14803), collects PIDs,
|
|
100
|
-
and `wait`s on each (run.sh:14880). It has been parallel all along. My first
|
|
101
|
-
grep searched the wrong line range, returned nothing, and I read that silence as
|
|
102
|
-
proof of absence -- the same mistake class this codebase has been punishing all
|
|
103
|
-
session.
|
|
104
|
-
|
|
105
|
-
**What the data actually says.** Pairing `code_review_start` with
|
|
106
|
-
`code_review_complete` across every recorded review:
|
|
107
|
-
|
|
108
|
-
| reviewers | seconds | verdict |
|
|
109
|
-
|---:|---:|---|
|
|
110
|
-
| 7 | 502 | 0 pass / 7 fail |
|
|
111
|
-
| 7 | 280 | 1 pass / 6 fail |
|
|
112
|
-
| 6 | 177 | 0 pass / 5 fail |
|
|
113
|
-
| **3** | **31** | 2 pass / 1 fail |
|
|
114
|
-
|
|
115
|
-
**3 reviewers = 31s. 7 reviewers = 280-502s.** Roughly 2x the council for 9-16x
|
|
116
|
-
the wall clock. Since dispatch is already concurrent, that superlinearity is not
|
|
117
|
-
the count itself -- it is that a larger council pulls in slower reviewers and
|
|
118
|
-
contends for the same provider, so the max-of-N tail dominates.
|
|
119
|
-
|
|
120
|
-
**And the 7-member council is partly redundant:**
|
|
121
|
-
|
|
122
|
-
```
|
|
123
|
-
architecture-strategist, maintainer-mergeability,
|
|
124
|
-
security-sentinel, review-security, <- TWO security reviewers
|
|
125
|
-
performance-oracle, eng-qa, dependency-analyst
|
|
126
|
-
```
|
|
127
|
-
|
|
128
|
-
`security-sentinel` and `review-security` overlap. `eng-qa` and
|
|
129
|
-
`dependency-analyst` are appended agents, not part of the sized battery.
|
|
130
|
-
|
|
131
|
-
**The real P0, in priority order:**
|
|
132
|
-
|
|
133
|
-
1. **Deduplicate overlapping reviewers.** Two security reviewers on one diff is
|
|
134
|
-
paying the tail cost twice for one signal. Collapse by mandate, not by name.
|
|
135
|
-
2. **Cap the effective council for scoped changes.** The tier map is
|
|
136
|
-
`{simple: 2, standard: 2, complex: 4}` specialists + 2 mandatory. Appended
|
|
137
|
-
agents bypass that sizing entirely, which is how 4 becomes 7. Bound the
|
|
138
|
-
TOTAL, not just the specialist slots.
|
|
139
|
-
3. **Bound the tail, not the mean.** One slow reviewer sets the whole council's
|
|
140
|
-
latency. A per-reviewer deadline that records a non-vote (never a silent
|
|
141
|
-
pass) converts a 502s worst case into a bounded one.
|
|
142
|
-
|
|
143
|
-
Expected: the 6-7 member case moves toward the measured 3-member behaviour
|
|
144
|
-
(31s) for scoped work, while `complex` keeps its deeper battery.
|
|
145
|
-
|
|
146
|
-
**Fail-safe direction is load-bearing:** a dropped or deadlined reviewer must
|
|
147
|
-
count as a NON-VOTE exactly as today, never as a pass. Shrinking a council must
|
|
148
|
-
never be able to manufacture approval. Guard with a test asserting the verdict
|
|
149
|
-
is identical for the same fixture at any council size, and that a deadlined
|
|
150
|
-
reviewer never contributes a PASS.
|
|
151
|
-
|
|
152
|
-
### P1 -- Do not run the full council on iteration 1 of a scoped change
|
|
153
|
-
|
|
154
|
-
The 8-gate council exists to catch regressions in a large build. On a scoped
|
|
155
|
-
GitHub-issue fix, running a 3-reviewer council before the change is even
|
|
156
|
-
verified is spending 281s to review something a test run would disprove in 1s.
|
|
157
|
-
|
|
158
|
-
- Order: cheap deterministic gates FIRST (test_suite, static_analysis,
|
|
159
|
-
lsp_diagnostics -- 6s combined), council only if those pass.
|
|
160
|
-
- A failing test means the council would have rejected anyway; running it first
|
|
161
|
-
is strictly wasted wall-clock.
|
|
162
|
-
- Expected: on a failing first pass, ~281s saved outright.
|
|
163
|
-
|
|
164
|
-
### P2 -- Time-to-first-signal (the Replit lesson)
|
|
165
|
-
|
|
166
|
-
Replit shows a visual preview in ~2 minutes. We show nothing until an iteration
|
|
167
|
-
completes. Even when our total time is competitive, the FELT time is worse.
|
|
168
|
-
|
|
169
|
-
- Emit a first-signal event as soon as the agent's first file write lands.
|
|
170
|
-
- `.loki/app-runner/first-preview.json` already exists (write-once, bash route)
|
|
171
|
-
-- surface it, and extend to the non-preview case as "first artifact".
|
|
172
|
-
- This is perception, not throughput, and it is cheap.
|
|
173
|
-
|
|
174
|
-
### P3 -- First-pass correctness (the actual "no second iteration" ask)
|
|
175
|
-
|
|
176
|
-
Research finding: success now hinges on **specification quality and dynamic
|
|
177
|
-
context**, not static upfront planning. Notably, auto-generated context files
|
|
178
|
-
REDUCED success ~3% while human-written ones improved it ~4%, both raising cost
|
|
179
|
-
20%+. So "generate a big context file" is measurably the wrong move.
|
|
180
|
-
|
|
181
|
-
What the evidence supports:
|
|
182
|
-
- ACE (ICLR 2026): generate -> reflect -> curate as an evolving context loop:
|
|
183
|
-
**+10.6% on coding benchmarks**.
|
|
184
|
-
- We already have the seam: `LOKI_INJECT_FINDINGS` feeds structured per-finding
|
|
185
|
-
records into the next iteration.
|
|
186
|
-
- The lever is making iteration 1's prompt carry what iteration 2 would have
|
|
187
|
-
learned -- which the existing first-pass-excellence work began and which
|
|
188
|
-
measured 2.8x cheaper with correctness held.
|
|
189
|
-
|
|
190
|
-
**Do not** add a static generated context file. The data says it hurts.
|
|
191
|
-
|
|
192
|
-
### P4 -- Prompt-cache discipline (already partly built, verify it holds)
|
|
193
|
-
|
|
194
|
-
`[CACHE_BREAKPOINT]` splits a cache-stable prefix from a volatile tail. Cache
|
|
195
|
-
reads price at 0.1x input. Any always-on instruction added to the wrong side
|
|
196
|
-
busts the cache every iteration.
|
|
197
|
-
|
|
198
|
-
- Add a regression test asserting no volatile content crosses the breakpoint.
|
|
199
|
-
- This is a cost lever primarily, and a TTFT lever secondarily.
|
|
200
|
-
|
|
201
|
-
---
|
|
202
|
-
|
|
203
|
-
## 4. What we do NOT do
|
|
204
|
-
|
|
205
|
-
- **Do not train a model.** Cursor and Cognition's speed is bought with custom
|
|
206
|
-
MoE models at 250-950 tok/s. That is not a gap we close with harness work,
|
|
207
|
-
and claiming otherwise would be dishonest.
|
|
208
|
-
- **Do not parallelize the seven cheap gates as a headline.** Measured worth:
|
|
209
|
-
~20s. Do it opportunistically inside P1's reordering, never sold as the win.
|
|
210
|
-
- **Do not add static generated context files.** Measured -3% success, +20%
|
|
211
|
-
cost.
|
|
212
|
-
|
|
213
|
-
## 4b. Shipped, and VERIFIED ON THE REAL CODE PATH
|
|
214
|
-
|
|
215
|
-
P0-P2 shipped in v8.33.0, v8.34.0 and v8.35.0. Each is default-OFF, so none
|
|
216
|
-
changes behaviour until switched on.
|
|
217
|
-
|
|
218
|
-
The claim "the knob works" was verified by extracting run.sh's ACTUAL selector
|
|
219
|
-
heredoc and running it -- not by re-implementing it, which is the mistake that
|
|
220
|
-
let three mutations survive across v8.28.0/v8.34.0/v8.35.0:
|
|
221
|
-
|
|
222
|
-
```
|
|
223
|
-
cap=0 -> 6 reviewers: architecture-strategist, maintainer-mergeability,
|
|
224
|
-
security-sentinel, test-coverage-auditor,
|
|
225
|
-
performance-oracle, dependency-analyst
|
|
226
|
-
cap=4 -> 4 reviewers: architecture-strategist, maintainer-mergeability,
|
|
227
|
-
security-sentinel, test-coverage-auditor
|
|
228
|
-
cap=3 -> 3 reviewers: architecture-strategist, maintainer-mergeability,
|
|
229
|
-
security-sentinel
|
|
230
|
-
```
|
|
231
|
-
|
|
232
|
-
Two things this proves that a unit test could not:
|
|
233
|
-
|
|
234
|
-
1. The cap reaches the real selector through the real env, at every level.
|
|
235
|
-
2. **The safety property holds in production code**: `architecture-strategist`
|
|
236
|
-
and `maintainer-mergeability` -- the mandatory pair -- survive at cap=3,
|
|
237
|
-
cap=4 and uncapped. Only the appended tail is trimmed. A cap can shrink the
|
|
238
|
-
council but can never remove a mandate.
|
|
239
|
-
|
|
240
|
-
For the skip knob, `gate_failures` (declared run.sh:22697) and the skip decision
|
|
241
|
-
(run.sh:22962) were confirmed to sit in the SAME function scope
|
|
242
|
-
(`run_autonomous`, run.sh:20842), so the skip genuinely observes the accumulated
|
|
243
|
-
failures rather than an empty shadow.
|
|
244
|
-
|
|
245
|
-
**Mapping to measured wall clock.** From the table in section 2: 6 reviewers
|
|
246
|
-
took 177s and 3 took 31s. Capping a complex-tier council from 6 to 3 therefore
|
|
247
|
-
targets the 31s regime for scoped work while `complex` keeps its deeper battery
|
|
248
|
-
when uncapped.
|
|
249
|
-
|
|
250
|
-
**What is still NOT proven:** an end-to-end scoped-issue run with the knobs on.
|
|
251
|
-
That requires a real paid build, and until it is measured, the numbers above are
|
|
252
|
-
selector behaviour plus historical stage timings -- not a demonstrated
|
|
253
|
-
end-to-end improvement. Do not report it as one.
|
|
254
|
-
|
|
255
|
-
## 5. How we know it worked
|
|
256
|
-
|
|
257
|
-
Every item ships with a before/after from the same `stage_complete` telemetry
|
|
258
|
-
that produced the table above. The acceptance number for the founder's
|
|
259
|
-
complaint:
|
|
260
|
-
|
|
261
|
-
> a scoped GitHub issue completes in **under 5 minutes**, first pass,
|
|
262
|
-
> with the council verdict intact.
|
|
263
|
-
|
|
264
|
-
Current measured critical path for that case is dominated by 281s of
|
|
265
|
-
sequential code review. P0 + P1 target exactly that.
|
|
266
|
-
|
|
267
|
-
## 6. Honest competitive position after this plan
|
|
268
|
-
|
|
269
|
-
- **vs Cursor** on raw interactive latency: still behind. Different category.
|
|
270
|
-
- **vs Replit** on time-to-first-visible: reachable with P2.
|
|
271
|
-
- **vs Cognition** on trust: ahead, and that is the moat -- the Evidence
|
|
272
|
-
Receipt, now that the gates behind it actually run (v8.24.0-v8.31.0).
|
|
273
|
-
|
|
274
|
-
The wedge is **verified-and-fast**, not fastest-in-absolute. We should say that
|
|
275
|
-
plainly rather than claim a speed crown we cannot hold.
|
|
276
|
-
|
|
277
|
-
---
|
|
278
|
-
|
|
279
|
-
Sources: Cursor Composer blog, VentureBeat Composer coverage, Cognition
|
|
280
|
-
SWE-1.6 reporting, Replit Agent 3 reviews, Zylos speculative-execution/parallel
|
|
281
|
-
tool-calling research, ACE (ICLR 2026), 2026 context-engineering surveys.
|
|
@@ -1,174 +0,0 @@
|
|
|
1
|
-
# Stale-State Audit
|
|
2
|
-
|
|
3
|
-
Audit for siblings of the `loki.pgid` session-killer (fixed in 4792b521).
|
|
4
|
-
|
|
5
|
-
**The shape being hunted:** a file recording a PID, PGID, port, lock, or session
|
|
6
|
-
id, written by one run, NOT removed on abnormal exit, later TRUSTED by another
|
|
7
|
-
run after the OS recycled the identifier.
|
|
8
|
-
|
|
9
|
-
**Method:** grep for the write, find its `rm -f`, check whether a trap covers
|
|
10
|
-
INT/TERM/crash, then check whether the reader proves the record is still
|
|
11
|
-
current. Every candidate was measured on this host rather than judged by
|
|
12
|
-
reading, because the pgid finding was credible on evidence (155h/202h orphans),
|
|
13
|
-
not on the code looking wrong.
|
|
14
|
-
|
|
15
|
-
Scope: read-only across the repo; fix applied to one file (`autonomy/run.sh`).
|
|
16
|
-
|
|
17
|
-
---
|
|
18
|
-
|
|
19
|
-
## Measured state of this host
|
|
20
|
-
|
|
21
|
-
Run from the repo root on 2026-08-08:
|
|
22
|
-
|
|
23
|
-
| File | Contents | Live? | Age |
|
|
24
|
-
|---|---|---|---|
|
|
25
|
-
| `~/.loki/dashboard/dashboard.pid` | absent | n/a | n/a |
|
|
26
|
-
| `.loki/dashboard/dashboard.pid` | `87992` | **DEAD** | **mtime Jul 31 (8 days)** |
|
|
27
|
-
| `$TMPDIR/loki-local-ci.lock` | `61876` | LIVE (real local-ci) | current |
|
|
28
|
-
| `.loki/app-runner/app.pid` | absent | n/a | n/a |
|
|
29
|
-
|
|
30
|
-
The dashboard pid file is the same measured shape as the pgid orphans: a dead
|
|
31
|
-
identifier, days old, still on disk, still trusted by a code path that kills.
|
|
32
|
-
|
|
33
|
-
---
|
|
34
|
-
|
|
35
|
-
## Findings, ranked by blast radius
|
|
36
|
-
|
|
37
|
-
### FINDING 1 (HIGH -- fixed here): shared dashboard pid is killed unverified
|
|
38
|
-
|
|
39
|
-
**Evidence.** `autonomy/run.sh:1597-1600` reads a pid from
|
|
40
|
-
`~/.loki/dashboard/dashboard.pid` and sends it `kill` then `kill -9` with no
|
|
41
|
-
liveness and no identity check.
|
|
42
|
-
|
|
43
|
-
Removal of that file happens ONLY on explicit stop paths -- `run.sh:16608`,
|
|
44
|
-
`:16658`, `:17399`, and `autonomy/loki:7078-7079`. **No trap covers a crash or
|
|
45
|
-
Ctrl+C.** That is precisely why the measured copy in this checkout is 8 days
|
|
46
|
-
stale with a dead pid.
|
|
47
|
-
|
|
48
|
-
**Reachable:** yes. Called from `cleanup` on both stop paths (`run.sh:25311`,
|
|
49
|
-
`:25384`).
|
|
50
|
-
|
|
51
|
-
**Blast radius: the worst class.** The file lives under `~/.loki`, so it is
|
|
52
|
-
machine-global -- the victim need not belong to this project. A recycled pid
|
|
53
|
-
names an unrelated live process and receives `kill -9`.
|
|
54
|
-
|
|
55
|
-
**Why `kill -0` would not have fixed it:** a recycled pid IS alive, so a
|
|
56
|
-
liveness check passes. This is the identical insufficiency the `loki.pgid`
|
|
57
|
-
self-check had. The guard has to check IDENTITY.
|
|
58
|
-
|
|
59
|
-
**Fix applied.** `_loki_pid_looks_like_dashboard()` at `autonomy/run.sh:1526`,
|
|
60
|
-
gating the kill at `:1597`. It mirrors `_app_runner_pid_is_ours`
|
|
61
|
-
(`app-runner.sh:242`) and **fails OPEN**: when `ps` reports nothing we signal
|
|
62
|
-
exactly as before, so a legitimate dashboard is never left running by this
|
|
63
|
-
check. The only behavior change is refusing to kill a process positively
|
|
64
|
-
identified as not a dashboard.
|
|
65
|
-
|
|
66
|
-
Deliberately NOT done: adding a trap to remove the pid file on crash. That is a
|
|
67
|
-
larger change across the dashboard lifecycle, and the identity guard already
|
|
68
|
-
makes a stale file harmless at the point where it does damage. The stale file
|
|
69
|
-
still being present is untidy, not dangerous, once the killer verifies.
|
|
70
|
-
|
|
71
|
-
**Test:** `tests/test-stale-dashboard-pid.sh`, 6/6, mutation-verified below.
|
|
72
|
-
|
|
73
|
-
### FINDING 2 (MEDIUM -- reported, not fixed): `app_runner_stop` group-kills unverified
|
|
74
|
-
|
|
75
|
-
**Evidence.** `app-runner.sh:1407` falls back to reading `app.pid` from disk,
|
|
76
|
-
then `:1456` sends `kill -TERM "-$_APP_RUNNER_PID"` -- a **process-group**
|
|
77
|
-
signal -- without an identity check.
|
|
78
|
-
|
|
79
|
-
The repo already has the right guard: `_app_runner_pid_is_ours`
|
|
80
|
-
(`app-runner.sh:242`). It is called at `:1657` and `:1829` but **not** on the
|
|
81
|
-
stop path. `app-runner.sh` has **no trap at all** (`grep -n "trap " ` returns
|
|
82
|
-
nothing), so `app.pid` survives a crash exactly like the pgid file did.
|
|
83
|
-
|
|
84
|
-
**Blast radius:** higher per-hit than Finding 1 (a group signal reaches a whole
|
|
85
|
-
tree) but narrower reach: the file is project-local (`.loki/app-runner/`), not
|
|
86
|
-
machine-global, and was absent on this host. Not fixed because it is a third
|
|
87
|
-
file and the lead scoped this task to one; it needs its own change adding
|
|
88
|
-
`_app_runner_pid_is_ours` to the stop path.
|
|
89
|
-
|
|
90
|
-
### FINDING 3 (LOW -- reported, not fixed): local-ci lock guard is a no-op under `/bin/bash`
|
|
91
|
-
|
|
92
|
-
**Evidence.** `scripts/local-ci.sh:104`:
|
|
93
|
-
|
|
94
|
-
```
|
|
95
|
-
trap '[ "$$" = "$_lci_owner" ] && rm -f "$_lci_lock" 2>/dev/null || true' EXIT
|
|
96
|
-
```
|
|
97
|
-
|
|
98
|
-
The scar comment above it (`:99-103`) correctly diagnoses that a bare EXIT trap
|
|
99
|
-
fires in every subshell and that a finishing child deleted the parent's lock.
|
|
100
|
-
**But `$$` does not change in a bash subshell, so this guard does not
|
|
101
|
-
discriminate.** Verified:
|
|
102
|
-
|
|
103
|
-
```
|
|
104
|
-
$ bash -c 'p=$$; ( [ "$$" = "$p" ] && echo same )'
|
|
105
|
-
same
|
|
106
|
-
```
|
|
107
|
-
|
|
108
|
-
The correct discriminators are `BASHPID` (bash 4+) or `BASH_SUBSHELL` (works in
|
|
109
|
-
3.2). Measured on this host: the script is `#!/usr/bin/env bash` which resolves
|
|
110
|
-
to **bash 5.3**, where `BASHPID` is available. But `/bin/bash` here is **3.2**,
|
|
111
|
-
where `BASHPID` is unset and `BASH_SUBSHELL` is the version-safe choice.
|
|
112
|
-
|
|
113
|
-
**Blast radius: benign** by the lead's own criterion. Worst case is two
|
|
114
|
-
concurrent local-ci runs starving each other into phantom failures -- costly in
|
|
115
|
-
time and trust, but it kills nothing. Left unfixed as out of scope; the one-line
|
|
116
|
-
change is `[ "${BASH_SUBSHELL:-0}" = 0 ]`.
|
|
117
|
-
|
|
118
|
-
Note on the sibling fix already shipped: `_loki_remove_pgid_file`
|
|
119
|
-
(`run.sh:25189`) uses `${BASHPID:-$$}`. Under bash 3.2 that collapses to `$$`
|
|
120
|
-
and the guard degrades to the no-op. **The degradation direction is safe** -- a
|
|
121
|
-
subshell deletes the pgid file early, the reap then finds no file and skips, so
|
|
122
|
-
an orphan survives and nobody gets killed. `run.sh` is also `#!/usr/bin/env
|
|
123
|
-
bash` (5.3 here), so this is a portability note, not a live defect. Worth
|
|
124
|
-
knowing: the mutation test for that guard asserts *source text* contains
|
|
125
|
-
`BASHPID`, so it proves the code is present, not that it behaves correctly on a
|
|
126
|
-
3.2 host.
|
|
127
|
-
|
|
128
|
-
---
|
|
129
|
-
|
|
130
|
-
## Already guarded -- do not re-audit
|
|
131
|
-
|
|
132
|
-
Checked and found correct. Listed explicitly so this ground is not covered
|
|
133
|
-
twice.
|
|
134
|
-
|
|
135
|
-
- **`autonomy/lib/lock.sh:56-67`** -- `_loki_lock_is_stale` requires the
|
|
136
|
-
sentinel PID to be dead **AND** mtime > 30s. Both conditions, not either.
|
|
137
|
-
Correct.
|
|
138
|
-
- **`cleanup_orphan_pids` (`run.sh:2300+`)** -- reaps only on liveness AND
|
|
139
|
-
parent death AND (for wrappers) idle-past-budget AND no live engine child.
|
|
140
|
-
Also self-skips `$$`. Correct, and notably stricter than the pgid reap was.
|
|
141
|
-
- **`_app_runner_pid_is_ours` (`app-runner.sh:242`)** -- a real identity token
|
|
142
|
-
captured post-exec, failing open on a missing token so a live app is never
|
|
143
|
-
falsely killed. Correct where it is called; see Finding 2 for where it is not.
|
|
144
|
-
- **`_app_runner_collect_descendants` (`app-runner.sh:289`)** -- refuses pid
|
|
145
|
-
0/1 and walks parent-child links from our own pid only, so it structurally
|
|
146
|
-
cannot signal outside our subtree. Correct.
|
|
147
|
-
- **`status.ts:266-270` and the bash status reader (`loki:5092`)** -- both do
|
|
148
|
-
`os.kill(pid, 0)` before reporting. A wrong answer only mislabels a URL in
|
|
149
|
-
status output. Benign even when stale.
|
|
150
|
-
- **The `CLEAR`/`KEEP` registry check (`run.sh:1545-1560`)** -- correctly
|
|
151
|
-
refuses to tear the shared dashboard down while any other project holds a
|
|
152
|
-
live pid. This gates Finding 1's call site; the defect was the missing check
|
|
153
|
-
on the pid itself, not this decision.
|
|
154
|
-
|
|
155
|
-
---
|
|
156
|
-
|
|
157
|
-
## Mutation verification (Finding 1's fix)
|
|
158
|
-
|
|
159
|
-
Each guard was reverted individually; the test must go red, and only on its own
|
|
160
|
-
assertion.
|
|
161
|
-
|
|
162
|
-
| Mutation | Before | After | Assertion killed |
|
|
163
|
-
|---|---|---|---|
|
|
164
|
-
| Identity check downgraded to bare `kill -0` | 6 pass / 0 fail | 5 / 1 | "a live non-dashboard process is refused" |
|
|
165
|
-
| Call site ungated (guard present but unused) | 6 / 0 | 5 / 1 | "the kill is gated on the identity guard" |
|
|
166
|
-
| `pid > 1` check removed | 6 / 0 | 5 / 1 | "pid 0/1, empty and malformed inputs refused" |
|
|
167
|
-
|
|
168
|
-
Restored after each. Final: `bash -n autonomy/run.sh` clean,
|
|
169
|
-
`tests/test-stale-dashboard-pid.sh` 6/6, `tests/test-pgid-stale-reap.sh` 8/8
|
|
170
|
-
(unaffected).
|
|
171
|
-
|
|
172
|
-
The first mutation is the important one: it replaces the identity check with
|
|
173
|
-
exactly the insufficient guard (`kill -0`) that a reviewer would most likely
|
|
174
|
-
propose, and the test catches it.
|