loki-mode 11.74.2 → 11.75.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/SKILL.md +2 -3
- package/VERSION +1 -1
- package/autonomy/completion-council.sh +10 -10
- package/autonomy/council-v2.sh +4 -4
- package/autonomy/grill.sh +1 -1
- package/autonomy/lib/browser-open.sh +1 -1
- package/autonomy/lib/chat.sh +1 -1
- package/autonomy/lib/cockpit-render.sh +1 -1
- package/autonomy/lib/done-recognition.sh +4 -4
- package/autonomy/lib/efficiency_cost.py +1 -1
- package/autonomy/lib/engine10-push.sh +2 -2
- package/autonomy/lib/prd-enrich.sh +1 -1
- package/autonomy/lib/proof-check.sh +1 -1
- package/autonomy/lib/proof-generator.py +3 -3
- package/autonomy/lib/proof-pr.sh +1 -1
- package/autonomy/lib/skill-link-heal.sh +1 -1
- package/autonomy/lib/voter-agents.sh +1 -1
- package/autonomy/loki +26 -37
- package/autonomy/prd-checklist.sh +1 -1
- package/autonomy/receipt_jwt.py +1 -1
- package/autonomy/run.sh +28 -28
- package/autonomy/trigger-schedule.py +6 -414
- package/autonomy/tui.sh +1 -1
- package/autonomy/verify.sh +4 -4
- package/completions/loki.bash +0 -5
- package/dashboard/__init__.py +1 -1
- package/docs/ARCHITECTURE.md +1 -2
- package/docs/CLI-REFERENCE.md +1 -1
- package/docs/COMPONENTS.md +1 -1
- package/docs/INSTALLATION.md +1 -1
- package/docs/ROUTING-EVIDENCE.md +1 -2
- package/docs/V10-RESEARCH.md +1 -1
- package/docs/dev/architecture-reference.md +3 -7
- package/docs/dev/project-structure.md +1 -4
- package/docs/dev/release-checklist.md +13 -31
- package/docs/research-2026-07/OBSERVABILITY-TOOL-SPEC.md +1 -1
- package/docs/v11/MARKET-SLICES.md +1 -1
- package/docs/v11/REGISTRIES.md +1 -1
- package/docs/v11/ROUTER-1.md +2 -2
- package/loki-ts/dist/loki.js +2 -2
- package/mcp/__init__.py +1 -1
- package/package.json +2 -2
- package/plugins/loki-mode/.claude-plugin/plugin.json +1 -1
- package/skills/extending.md +2 -3
- package/skills/factory-operations.md +11 -18
- package/skills/release-cadence.md +2 -2
- package/web-app/dist/assets/{AdminPage-jkjilfbl.js → AdminPage-xKgtIhk2.js} +1 -1
- package/web-app/dist/assets/{Avatar-bd5JWkSJ.js → Avatar-Q14AHox9.js} +1 -1
- package/web-app/dist/assets/{Badge-DVRit2Xw.js → Badge-yk5M-lip.js} +1 -1
- package/web-app/dist/assets/{Button-BNAB-pVZ.js → Button-ynznk0At.js} +1 -1
- package/web-app/dist/assets/{CockpitPage-DpG29h_w.js → CockpitPage-C8B765za.js} +1 -1
- package/web-app/dist/assets/{ComparePage-Dasg5V41.js → ComparePage-DqzmWKnf.js} +1 -1
- package/web-app/dist/assets/{ErrorBoundary-DJojse6b.js → ErrorBoundary-DdBPd_Lq.js} +1 -1
- package/web-app/dist/assets/{EvidenceReceiptPanel-Blxw7_u1.js → EvidenceReceiptPanel-jb4Tp38b.js} +1 -1
- package/web-app/dist/assets/{GitHubIssuesPanel-BBvSFj80.js → GitHubIssuesPanel-xJQiLcxs.js} +1 -1
- package/web-app/dist/assets/{GitHubPRsPanel-BCIKeSBg.js → GitHubPRsPanel-84W3Dhk_.js} +1 -1
- package/web-app/dist/assets/{HomePage-C7d__oNG.js → HomePage-St1Xl2tu.js} +2 -2
- package/web-app/dist/assets/{LoginPage-BXljL3yY.js → LoginPage-jRt0NUHb.js} +1 -1
- package/web-app/dist/assets/{MagicPage-Xjkts_yE.js → MagicPage-CQLFe7LR.js} +1 -1
- package/web-app/dist/assets/{MetricsPage-kQvVp-j2.js → MetricsPage-zzX9eTC5.js} +1 -1
- package/web-app/dist/assets/{NotFoundPage-D-4ZlEaO.js → NotFoundPage-DDu023SY.js} +1 -1
- package/web-app/dist/assets/{ProjectPage-DfzR5RXP.js → ProjectPage-D60UURaH.js} +3 -3
- package/web-app/dist/assets/{ProjectsPage-DFDsWzkV.js → ProjectsPage-DqVsKxlt.js} +1 -1
- package/web-app/dist/assets/{SettingsPage-CLPIoYT3.js → SettingsPage-G7pPKvo0.js} +1 -1
- package/web-app/dist/assets/{ShowcasePage-PZKkyGeY.js → ShowcasePage-CrU_aKgS.js} +1 -1
- package/web-app/dist/assets/{SystemSettingsPage-BsBlcnzo.js → SystemSettingsPage-ClKGQFsB.js} +1 -1
- package/web-app/dist/assets/{TeamsPage-BSjgNhoU.js → TeamsPage-B530vSNe.js} +1 -1
- package/web-app/dist/assets/{TemplatesPage-W2KeAzbg.js → TemplatesPage-COmv_Cqd.js} +1 -1
- package/web-app/dist/assets/{TerminalOutput-BjroTiCC.js → TerminalOutput-CufG9YzF.js} +1 -1
- package/web-app/dist/assets/{activity-BXZ5YYes.js → activity-Ct1Updr6.js} +1 -1
- package/web-app/dist/assets/{bell-D1urp5pa.js → bell-rcvhQuGr.js} +1 -1
- package/web-app/dist/assets/{bot-Chedf2bF.js → bot-DzeqEI-b.js} +1 -1
- package/web-app/dist/assets/{check-DhaCuOCV.js → check-zacnZHyC.js} +1 -1
- package/web-app/dist/assets/{chevron-left-D7BpQyA6.js → chevron-left-Lxnkl8Qg.js} +1 -1
- package/web-app/dist/assets/{circle-alert-DnGvQ_AQ.js → circle-alert-DoqSdQdB.js} +1 -1
- package/web-app/dist/assets/{clock-CqfxWhHM.js → clock-B7bJcVIC.js} +1 -1
- package/web-app/dist/assets/{cloud-BKxp5AWB.js → cloud-DdA9Bu6q.js} +1 -1
- package/web-app/dist/assets/{code-xml-BDQ6GBtt.js → code-xml-CAZEZ9T8.js} +1 -1
- package/web-app/dist/assets/{database-BmpEx1y_.js → database-Da2GkEIN.js} +1 -1
- package/web-app/dist/assets/{dollar-sign-DYNY9R_C.js → dollar-sign-DjDiG-37.js} +1 -1
- package/web-app/dist/assets/{file-code-corner-BJi0IBFd.js → file-code-corner-CSL-Jij-.js} +1 -1
- package/web-app/dist/assets/{file-plus-Dqyb2bcQ.js → file-plus-02DRA8pV.js} +1 -1
- package/web-app/dist/assets/{globe-DPh4RWfM.js → globe-CGyB90kK.js} +1 -1
- package/web-app/dist/assets/{hammer-B3xnuoA1.js → hammer-BaQB96TG.js} +1 -1
- package/web-app/dist/assets/{index-BpX8DcTr.js → index-lcoI1_Kk.js} +3 -3
- package/web-app/dist/assets/{layers-BRCmyCW8.js → layers-BWJBvoeg.js} +1 -1
- package/web-app/dist/assets/{loader-circle-n07Pky0M.js → loader-circle-CdcI0FeS.js} +1 -1
- package/web-app/dist/assets/{lock-BWqOuQmS.js → lock-BBkw5NJx.js} +1 -1
- package/web-app/dist/assets/{package-BJlB_Z-Z.js → package-BpPRO3LM.js} +1 -1
- package/web-app/dist/assets/{plus-oH17BZu2.js → plus-D7XbyK29.js} +1 -1
- package/web-app/dist/assets/{refresh-cw-CPQ476LL.js → refresh-cw-DLG3hyAi.js} +1 -1
- package/web-app/dist/assets/{rotate-ccw-7hLvOcnS.js → rotate-ccw-Bqau4R_S.js} +1 -1
- package/web-app/dist/assets/{scroll-text-BCmLauQn.js → scroll-text-BtIL3cNk.js} +1 -1
- package/web-app/dist/assets/{server-DfTfvW84.js → server-tZCmZ4yH.js} +1 -1
- package/web-app/dist/assets/{shield-alert-CaeGRq19.js → shield-alert-K5JXzFKg.js} +1 -1
- package/web-app/dist/assets/{thumbs-up-Dz-l2GPC.js → thumbs-up-CHCVyJdw.js} +1 -1
- package/web-app/dist/assets/{trash-2-B-OsDmSo.js → trash-2-DncNxVq0.js} +1 -1
- package/web-app/dist/assets/{trending-down-CXTFOxzq.js → trending-down-D8Bp0azX.js} +1 -1
- package/web-app/dist/assets/{trending-up-FQEdqr3v.js → trending-up-AixETdKR.js} +1 -1
- package/web-app/dist/assets/{upload-CacWSNTg.js → upload-VCbjFn2H.js} +1 -1
- package/web-app/dist/assets/{usePolling-cIjZ9qd2.js → usePolling-CQvKD2Jj.js} +1 -1
- package/web-app/dist/assets/{user-DAouWFHi.js → user-SK-YguQ6.js} +1 -1
- package/web-app/dist/index.html +1 -1
- package/autonomy/company-seats.json +0 -82
- package/docs/COMPETITIVE-INTEL-2026-09.md +0 -444
- package/docs/FOUNDER-DECISIONS-TRUST-SEMANTICS-2026-08-01.md +0 -145
- package/docs/FOUNDER-STATUS-AND-DECISIONS-2026-07-01.md +0 -132
- package/docs/LOKI-10-BUILD-PROMPT.md +0 -350
- package/docs/STRATEGY-2026-2028.md +0 -124
- package/docs/V10-VISION.md +0 -161
- package/docs/v10/ADOPTION-PLAN.md +0 -375
- package/docs/v10/ARCHITECT-CUTS.md +0 -744
- package/docs/v10/BACKLOG.md +0 -231
- package/docs/v10/BOARD.md +0 -1403
- package/docs/v10/CONTROL-PLANE.md +0 -120
- package/docs/v10/CONTROL.md +0 -36
- package/docs/v10/CP-ASK-PLAN.md +0 -85
- package/docs/v10/CP-ENTERPRISE-UI.md +0 -537
- package/docs/v10/D51-PHASE-B.md +0 -120
- package/docs/v10/D61-SPEED.md +0 -105
- package/docs/v10/D63-CARDS.md +0 -354
- package/docs/v10/DECISIONS.md +0 -915
- package/docs/v10/E2E-D65.md +0 -123
- package/docs/v10/ENGINE-LAWS.md +0 -163
- package/docs/v10/ENGINE-MEASURE.md +0 -69
- package/docs/v10/ENGINE.md +0 -743
- package/docs/v10/FAILURE-CLASSES.md +0 -2026
- package/docs/v10/FC21B-FC22-PLAN.md +0 -107
- package/docs/v10/FIRST-RUN-GATE.md +0 -39
- package/docs/v10/FOUNDER-QUEUE.md +0 -29
- package/docs/v10/GAP-RESEARCH-1003.md +0 -32
- package/docs/v10/GUARDS.md +0 -883
- package/docs/v10/L0-WAVE1.md +0 -81
- package/docs/v10/LEGACY-REMOVAL.md +0 -408
- package/docs/v10/MEDIUM-ANALYSIS.md +0 -346
- package/docs/v10/METRICS.md +0 -645
- package/docs/v10/MODERNIZE.md +0 -437
- package/docs/v10/OPENCODE-PARITY-PLAN.md +0 -84
- package/docs/v10/OPERATING-MODEL.md +0 -60
- package/docs/v10/PLAN-V12.md +0 -896
- package/docs/v10/PROGRESS.md +0 -2735
- package/docs/v10/RELEASE-11.md +0 -77
- package/docs/v10/RETRO-2026.md +0 -650
- package/docs/v10/ROADMAP-D63.md +0 -35
- package/docs/v10/SCALE.md +0 -140
- package/docs/v10/SCORECARD-PLAN.md +0 -250
- package/docs/v10/SWARM-PROMPT.md +0 -98
- package/docs/v10/SWARM.md +0 -293
- package/docs/v10/cp-redesign/PARITY.md +0 -188
- package/docs/v10/cp-redesign/legacy/INDEX.md +0 -1167
- package/docs/v10/roadmap-10/01-measurement.md +0 -113
- package/docs/v10/roadmap-10/02-cost.md +0 -75
- package/docs/v10/roadmap-10/03-speed.md +0 -65
- package/docs/v10/roadmap-10/04-first-run.md +0 -61
- package/docs/v10/roadmap-10/05-deploy.md +0 -61
- package/docs/v10/roadmap-10/06-stability.md +0 -64
- package/docs/v10/roadmap-10/07-quality.md +0 -57
- package/docs/v10/roadmap-10/08-proof.md +0 -48
- package/docs/v10/roadmap-10/09-safety.md +0 -74
- package/docs/v10/roadmap-10/README.md +0 -1464
- package/skills/sdlc-fleet.md +0 -170
- package/tools/cockpit/SPEC.md +0 -275
- package/tools/cockpit/index.html +0 -146
- package/tools/cockpit/roi-basis.json +0 -33
- package/tools/cockpit/server.py +0 -946
|
@@ -1,113 +0,0 @@
|
|
|
1
|
-
# 01 Measurement (enabler for every other score)
|
|
2
|
-
|
|
3
|
-
Why: every cost, speed and quality claim so far is either unmeasured or refuted. C-F1 run1 reported 1.81x but the ledger showed about 12.3x raw. Run4 had Loki at 1.41x MORE than lean raw on 6 tasks (CI 1.21-1.68), NOT PROVEN on 8. Medium cost is null on 10, 2 and 11 of 21 rows. The seeded-defect catch rate has never been measured. The newest public SWE-bench result is from 2026-01.
|
|
4
|
-
|
|
5
|
-
Rule: no score in README.md moves without a result file cited here.
|
|
6
|
-
|
|
7
|
-
## MEAS-1 Bench rows carry version, profile and one cost truth
|
|
8
|
-
|
|
9
|
-
Story: as the CTO, I need every bench row to say which build and profile produced it and what it really cost, so a claim cannot mix versions or cost sources.
|
|
10
|
-
|
|
11
|
-
Acceptance:
|
|
12
|
-
- Every Loki row in `benchmarks/bench` output has non-null `loki_version` and `cost_profile` (dry-run fixture test).
|
|
13
|
-
- `cost_claim.py` mixed-version refusal stays red-first in a test.
|
|
14
|
-
- On a 6-task run, the list of rows where |events cost - reported cost| > 5% is empty; the ledger total is cross-checked against receipt `cost.usd`.
|
|
15
|
-
- The real engine rc is stored per row (today `rc` is null; `attribution.py` `run_loki` about line 455-490). Stdout and stderr are kept per row.
|
|
16
|
-
|
|
17
|
-
Todo:
|
|
18
|
-
- [ ] Stamp loki_version and cost_profile in `benchmarks/bench/adapters/loki*.py` and `attribution.py`
|
|
19
|
-
- [ ] Store real rc and keep stdout/stderr in `adapters/_base.py` run_cli
|
|
20
|
-
- [ ] Reconcile events cost, efficiency and ledger in `attribution.py` and `loki-ts/src/engine10/cost.ts`
|
|
21
|
-
- [ ] Red-first fixture tests for all three
|
|
22
|
-
|
|
23
|
-
## MEAS-2 Bench Wall uses the task interpreter
|
|
24
|
-
|
|
25
|
-
Story: as the bench, Loki rows must be able to reach VERIFIED, otherwise Loki pays for Wall and gets no proof.
|
|
26
|
-
|
|
27
|
-
Acceptance:
|
|
28
|
-
- On the run4 task set, 0 rows log "ran under the system interpreter".
|
|
29
|
-
- Loki verdict VERIFIED on >= 80% of accepted rows.
|
|
30
|
-
- Bench seeds that ship no test still yield a Loki-executed check (or the bench records why PARTIAL is correct).
|
|
31
|
-
|
|
32
|
-
Todo:
|
|
33
|
-
- [ ] Wall RUNNER_CMD resolves the task venv interpreter (`loki-ts/src/engine10/stages/wall.ts`)
|
|
34
|
-
- [ ] Bench venv setup exposes `python` on PATH
|
|
35
|
-
- [ ] Regression fixture: bench row with venv goes VERIFIED
|
|
36
|
-
|
|
37
|
-
## MEAS-3 C-F1 run5
|
|
38
|
-
|
|
39
|
-
Story: as the founder, I want one trustworthy cost and time comparison against raw Claude Code.
|
|
40
|
-
|
|
41
|
-
Acceptance:
|
|
42
|
-
- Arms: raw-configured, raw-lean, raw-bare, loki, loki-lean. n=3. sonnet-5-5. Cap set. 20 tasks including coord-1 and longspec-1 group tasks.
|
|
43
|
-
- 0 UNMEASURED raw-bare cells.
|
|
44
|
-
- Group tasks: 3 of 3 Loki rows have a non-null summed group cost and do not exit in 0.2s.
|
|
45
|
-
- One verdict line per arm per floor appended to `docs/v10/METRICS.md` C-F1 section.
|
|
46
|
-
|
|
47
|
-
Todo:
|
|
48
|
-
- [ ] Measure raw-bare floor in the attribution floor runner
|
|
49
|
-
- [ ] Re-measure group cells on >= v11.17.16 (USER/LOGNAME fix)
|
|
50
|
-
- [ ] Grow task set from 8 to 20 (mix of small, medium, group)
|
|
51
|
-
- [ ] Run, append verdicts, link result dir
|
|
52
|
-
|
|
53
|
-
## MEAS-4 Medium A/B with cost never null
|
|
54
|
-
|
|
55
|
-
Acceptance:
|
|
56
|
-
- `provider_cost()` accepts partial-stream cost; null-cost rows 0 of N on rerun (`eval/loki10/harness.py` about line 698).
|
|
57
|
-
- All 14 medium tasks, n=3, arms run back to back at equal `--parallel`.
|
|
58
|
-
- Bootstrap CI reported per arm (S41-17).
|
|
59
|
-
|
|
60
|
-
Todo:
|
|
61
|
-
- [ ] Partial-stream cost accepted
|
|
62
|
-
- [ ] Run all 14 tasks, n=3
|
|
63
|
-
- [ ] Record in METRICS.md with CI
|
|
64
|
-
|
|
65
|
-
## MEAS-5 Seeded-defect catch rate
|
|
66
|
-
|
|
67
|
-
Story: as a buyer, I want to know how often Seal says VERIFIED on broken code.
|
|
68
|
-
|
|
69
|
-
Acceptance:
|
|
70
|
-
- A runner executes real agent sessions against the 100-case corpus and writes `benchmarks/seeded-defects/results/wrong-pass.json`.
|
|
71
|
-
- Top tier vs floor tier wrong-pass reported; P4 `floor-does-not-raise-wrong-pass` case PASS.
|
|
72
|
-
- METRICS.md False-SEALED row filled (target 0 of 100).
|
|
73
|
-
|
|
74
|
-
Todo:
|
|
75
|
-
- [ ] Runner script over the corpus
|
|
76
|
-
- [ ] Top vs floor runs
|
|
77
|
-
- [ ] METRICS row and moat case flip
|
|
78
|
-
|
|
79
|
-
## MEAS-6 Public benchmarks
|
|
80
|
-
|
|
81
|
-
Story: SWE-bench Verified is saturated (about 96%); buyers now read Terminal-Bench 2.0 and SWE-bench Pro.
|
|
82
|
-
|
|
83
|
-
Acceptance:
|
|
84
|
-
- SWE-bench Pro pilot (119 instances) run with Loki and raw Claude Code; results JSON committed under `benchmarks/results/swebench-pro-pilot/`.
|
|
85
|
-
- Terminal-Bench 2.0 adapter and one full run, results committed.
|
|
86
|
-
- SWE-bench pinned subset marked verified=true or replaced; recall@8 recorded.
|
|
87
|
-
|
|
88
|
-
Todo:
|
|
89
|
-
- [ ] Run SWE-bench Pro pilot both arms
|
|
90
|
-
- [ ] Terminal-Bench 2.0 adapter plus run
|
|
91
|
-
- [ ] Verify pinned SWE-bench ids, publish recall@8
|
|
92
|
-
|
|
93
|
-
## MEAS-7 Time to first sealed PR, real provider, nightly
|
|
94
|
-
|
|
95
|
-
Acceptance:
|
|
96
|
-
- Nightly job runs `scripts/first-run-gate.sh --real --entry start-empty` with a token secret and fails above budget.
|
|
97
|
-
- Stub `--entry start-empty` also runs in nightly.
|
|
98
|
-
- METRICS.md row "time to first sealed PR" has a dated number; p50 <= 120s.
|
|
99
|
-
|
|
100
|
-
Todo:
|
|
101
|
-
- [ ] Add stub start-empty entry to `.github/workflows/nightly.yml`
|
|
102
|
-
- [ ] Add real-provider entry with secret, guarded so forks skip
|
|
103
|
-
- [ ] METRICS row
|
|
104
|
-
|
|
105
|
-
## MEAS-8 Public scoreboard
|
|
106
|
-
|
|
107
|
-
Acceptance:
|
|
108
|
-
- One generator reads the result files from MEAS-3..7 and renders the scoreboard; numbers in README.md and METRICS.md match it (test).
|
|
109
|
-
- Every number shows date, version, n and CI.
|
|
110
|
-
|
|
111
|
-
Todo:
|
|
112
|
-
- [ ] Generator (extend the existing summarize tooling, do not add a parallel one)
|
|
113
|
-
- [ ] Drift test README vs METRICS vs generator
|
|
@@ -1,75 +0,0 @@
|
|
|
1
|
-
# 02 Cost (target: at least 2x cheaper than Claude Code, proven)
|
|
2
|
-
|
|
3
|
-
Now 6. Winner today: Aider on raw spend; Cursor Composer at about $0.07 per task.
|
|
4
|
-
|
|
5
|
-
Why we are not 10:
|
|
6
|
-
- Run4: Loki 1.41x more than lean raw on 6 tasks; pooled 1.11x NOT PROVEN with acceptance 0.750 vs 0.958.
|
|
7
|
-
- Plan and Wall run as fixed sessions even on small tasks: run4 sized all 31 tasks normal (19 "no test runner detected", 12 over 600 chars), `plan_skipped=false` everywhere. Overhead stages are 31% of spend on medium; the planner costs as much as implement.
|
|
8
|
-
- Implement cache reads 1.9x raw (9.23M vs 4.79M) because the brief gives no test to run, so the session explores.
|
|
9
|
-
- Best measured point is lean config 0.68x raw (opus, single arm, no CI). The lean knobs are built but OFF.
|
|
10
|
-
|
|
11
|
-
Bar for 10: `cost_claim` reports PROVEN_AT(0.5) against all 3 floors, n >= 3, one version, ledger-checked, with acceptance >= floor.
|
|
12
|
-
|
|
13
|
-
Depends on: 01 MEAS-1, MEAS-2, MEAS-3.
|
|
14
|
-
|
|
15
|
-
## COST-A Small-task lean path actually triggers
|
|
16
|
-
|
|
17
|
-
Story: as a user with a small fix, I pay for one implement session plus verify, not Plan plus Wall plus implement.
|
|
18
|
-
|
|
19
|
-
Acceptance:
|
|
20
|
-
- `plan_skipped=true` on >= 60% of C-F1 tasks.
|
|
21
|
-
- Acceptance >= floor on the same run.
|
|
22
|
-
- "no test runner detected" no longer forces normal size when the repo has a discoverable test command.
|
|
23
|
-
|
|
24
|
-
Todo:
|
|
25
|
-
- [ ] Revisit `SMALL_NO_RUNNER` and `MAX_CHARS` defaults in `loki-ts/src/engine10/sizing.ts` behind a measured loki-lean arm
|
|
26
|
-
- [ ] Better runner detection (feeds sizing)
|
|
27
|
-
- [ ] Flip default only after MEAS-3 shows acceptance held
|
|
28
|
-
|
|
29
|
-
## COST-B Focused implement brief
|
|
30
|
-
|
|
31
|
-
Story: the implementer gets the up-to-20 relevant files and the exact test command, so it does not explore.
|
|
32
|
-
|
|
33
|
-
Acceptance:
|
|
34
|
-
- Implement cache reads per completed down >= 30% on medium, n=2.
|
|
35
|
-
- Completion not lower.
|
|
36
|
-
|
|
37
|
-
Todo:
|
|
38
|
-
- [ ] Brief builder in `loki-ts/src/e10ext/context.ts`
|
|
39
|
-
- [ ] Wire into `stages/implement.ts` (Captain edit if hot)
|
|
40
|
-
- [ ] Paid A/B
|
|
41
|
-
|
|
42
|
-
## COST-C Flip measured-OFF knobs
|
|
43
|
-
|
|
44
|
-
| Knob | Flip when |
|
|
45
|
-
|---|---|
|
|
46
|
-
| `--cost lean` / `LOKI_COST_PROFILE` | loki-lean arm CI upper bound <= 0.5 and acceptance >= floor |
|
|
47
|
-
| `LOKI_IMPLEMENT_HAIKU_FIRST` | same arm plus escalation rate recorded |
|
|
48
|
-
| `LOKI_E10_TRIM` | implement cache reads down >= 25% on medium, completion not lower |
|
|
49
|
-
| `LOKI_E10_ATTEMPTS=2` | medium completion >= 18/21, cost not red |
|
|
50
|
-
|
|
51
|
-
Todo:
|
|
52
|
-
- [ ] One paid arm per knob, results in METRICS.md
|
|
53
|
-
- [ ] Flip each knob that passes, in its own release
|
|
54
|
-
|
|
55
|
-
## COST-D Cost correctness followups
|
|
56
|
-
|
|
57
|
-
Todo:
|
|
58
|
-
- [ ] Alias duplicate flags: make cost flag aliases resolve to one value and reject conflicting duplicates
|
|
59
|
-
- [ ] `autonomy/run.sh` about line 7626: BUDGET_LIMIT ".5" handled like the TS `parseCapUsd` fix (TEN-COST-59); run.sh and goal_score.ts change together
|
|
60
|
-
- [ ] Price table negative-price test (reject negative or NaN prices)
|
|
61
|
-
- [ ] TEN-COST-59 sibling sweep: every other dollar parser accepts ".5" and "5."
|
|
62
|
-
- [ ] BUDGET_BURN pulse miscount: pulse counts 128-132 "active" engineers when the real number is single digits; fix the counter in `scripts/v10-pulse.sh`
|
|
63
|
-
|
|
64
|
-
Acceptance: each item has a FAILURE-CLASSES row, a red-first test, and a sibling sweep note.
|
|
65
|
-
|
|
66
|
-
## COST-E Publish the claim
|
|
67
|
-
|
|
68
|
-
Acceptance:
|
|
69
|
-
- PROVEN_AT(0.5) line in METRICS.md with run dir, version, n, CI.
|
|
70
|
-
- Scoreboard (MEAS-8) shows it.
|
|
71
|
-
- Receipt `cost.usd` matches ledger within 5% on every row used.
|
|
72
|
-
|
|
73
|
-
Todo:
|
|
74
|
-
- [ ] Run after COST-A..C land
|
|
75
|
-
- [ ] Publish
|
|
@@ -1,65 +0,0 @@
|
|
|
1
|
-
# 03 Speed
|
|
2
|
-
|
|
3
|
-
Now 7. Winner today: Cursor Composer 2.5 Fast (about 6.7 min average per task), Codex.
|
|
4
|
-
|
|
5
|
-
Why we are not 10:
|
|
6
|
-
- Medium p50 209s default vs raw 56-70s (2.5-4x). Small run4 p50 31.9s vs 24.3s (1.31x).
|
|
7
|
-
- Implement is 72-86% of p50; Wall is 23-29% and is killed at its 90s limit in 9-11 of 21 runs; implement starts within 1ms of Wall closing.
|
|
8
|
-
- Fix rounds add 18.9-35.7s at p90.
|
|
9
|
-
- Pre-model overhead (cold bun child per session) is not measured.
|
|
10
|
-
|
|
11
|
-
Bar for 10: TEN-SPEED-01 paired time ratio CI lower bound >= 1.0 on small; medium p50 <= raw at equal parallelism.
|
|
12
|
-
|
|
13
|
-
Owns `loki-ts/src/engine10/stages/wall.ts` for this program.
|
|
14
|
-
|
|
15
|
-
## SPD-A Wall off the critical path
|
|
16
|
-
|
|
17
|
-
Story: implement starts immediately; Wall runs beside it and never blocks it.
|
|
18
|
-
|
|
19
|
-
Acceptance:
|
|
20
|
-
- `LOKI_E10_WALL_CONCURRENT` default on after HIGH review (S41-14).
|
|
21
|
-
- Medium p50 down >= 20% vs 209s at equal `--parallel`.
|
|
22
|
-
- Wall still red-on-base; completion not lower.
|
|
23
|
-
- Wall limit kills <= 2 of 21; Wall share of p50 < 15%.
|
|
24
|
-
|
|
25
|
-
Todo:
|
|
26
|
-
- [ ] HIGH review S41-14 (opus adversarial)
|
|
27
|
-
- [ ] Measured arm, flip default (`machine.ts`)
|
|
28
|
-
- [ ] `wallLimitS` tuning or skip-on-haiku-timeout in `sizing.ts`
|
|
29
|
-
|
|
30
|
-
## SPD-B Pre-model overhead
|
|
31
|
-
|
|
32
|
-
Acceptance:
|
|
33
|
-
- `pre_model` p50 reported per arm on eval rows (`eval/loki10/harness.py`, `stage-profile.py`).
|
|
34
|
-
- p50 < 2s warm.
|
|
35
|
-
|
|
36
|
-
Todo:
|
|
37
|
-
- [ ] Timer in harness
|
|
38
|
-
- [ ] Warm-process or socket reuse if over 2s
|
|
39
|
-
|
|
40
|
-
## SPD-C Parallel verify and base jobs
|
|
41
|
-
|
|
42
|
-
Acceptance:
|
|
43
|
-
- Paid A/B of `LOKI_E10_VERIFY_JOBS=4` and `LOKI_E10_BASE_JOBS=4`: verify plus Wall base time down >= 30%, 0 new flakes over 3 reps.
|
|
44
|
-
- Defaults flipped only if it passes.
|
|
45
|
-
|
|
46
|
-
Todo:
|
|
47
|
-
- [ ] A/B run
|
|
48
|
-
- [ ] Flip in `stages/verify.ts` and `util/check_result.ts`
|
|
49
|
-
|
|
50
|
-
## SPD-D Speed tier tasks
|
|
51
|
-
|
|
52
|
-
Acceptance:
|
|
53
|
-
- `eval/loki10/tasks/spd-*` exist (D61 slice 17).
|
|
54
|
-
- `LOKI_SPEED=0/1` A/B: parallel wall <= 0.5x sequential on decomposable tasks, tokens not higher.
|
|
55
|
-
|
|
56
|
-
Todo:
|
|
57
|
-
- [ ] Author spd tasks
|
|
58
|
-
- [ ] A/B run, METRICS row
|
|
59
|
-
|
|
60
|
-
## SPD-E Equal-parallelism scorecard rule
|
|
61
|
-
|
|
62
|
-
Acceptance: `eval/loki10/summarize` refuses a pool with mixed `--parallel` (red-first test) and runs arms back to back.
|
|
63
|
-
|
|
64
|
-
Todo:
|
|
65
|
-
- [ ] Guard plus test
|
|
@@ -1,61 +0,0 @@
|
|
|
1
|
-
# 04 First run and adoption
|
|
2
|
-
|
|
3
|
-
Now 8. Winner today: Lovable (2-3 minutes, zero install) for greenfield; Cursor and Claude Code for existing repos. Note the founder scoped Loki as "not a vibe coding tool" (FAILURE-CLASSES.md); we compete on existing-repo first success, and offer a zero-install try path, not a vibe builder.
|
|
4
|
-
|
|
5
|
-
Why we are not 10:
|
|
6
|
-
- Time to first sealed PR has never been measured with a real provider; real gate attempts died in 1-3s with "Not logged in" in a clean HOME.
|
|
7
|
-
- Host needs node, python3, jq, git and curl plus one provider, and `loki setup` is a separate step.
|
|
8
|
-
- MCP deeplink config has no env, so a non-TTY launch without `LOKI_MCP_AUTO_BOOTSTRAP=1` exits 2 (from code reading).
|
|
9
|
-
- git identity missing stays fatal; empty-folder auto-init fails silently without it.
|
|
10
|
-
- v10 preflight fatals emit no `first_run_blocked` telemetry.
|
|
11
|
-
- Measured: stub `loki quick` 111s; real 97s vs 10s raw.
|
|
12
|
-
|
|
13
|
-
Bar for 10: clean machine to first sealed PR p50 <= 120s, measured nightly (MEAS-7), with zero dead ends: every block prints exactly one actionable line.
|
|
14
|
-
|
|
15
|
-
## FR-A One command to first sealed PR
|
|
16
|
-
|
|
17
|
-
Story: `npm i -g loki-mode && loki start "<task>"` in a repo works with no separate setup.
|
|
18
|
-
|
|
19
|
-
Acceptance:
|
|
20
|
-
- `loki start` in a clean HOME prints exactly one actionable line per missing tool and offers the fix.
|
|
21
|
-
- Gate `--entry start-empty` passes with no `loki setup` call.
|
|
22
|
-
- On a TTY, missing git identity is offered as a repo-local set (never global); empty-folder test passes with no global identity.
|
|
23
|
-
|
|
24
|
-
Todo:
|
|
25
|
-
- [ ] Fold setup checks into start preflight (`start.ts`, `doctor_fix.ts`)
|
|
26
|
-
- [ ] git identity offer (`preflight.ts`, `first_run_repo.ts`)
|
|
27
|
-
- [ ] Gate test
|
|
28
|
-
|
|
29
|
-
## FR-B Fewer host requirements
|
|
30
|
-
|
|
31
|
-
Acceptance: doctor marks jq and curl optional unless the bash route is used; the v10 gate passes with jq and curl off PATH. Clean macOS python3 without `cryptography` does not block the v10 path.
|
|
32
|
-
|
|
33
|
-
Todo:
|
|
34
|
-
- [ ] `doctor.ts`, `setup-wizard.sh` requirement tiers
|
|
35
|
-
- [ ] Gate run with tools removed from PATH
|
|
36
|
-
|
|
37
|
-
## FR-C Every block is telemetered and named
|
|
38
|
-
|
|
39
|
-
Acceptance: each v10 preflight fatal emits `first_run_blocked` with its fixed value (not_a_repo, git_identity, not_logged_in, no_provider, disk). One test asserts one event per value.
|
|
40
|
-
|
|
41
|
-
Todo:
|
|
42
|
-
- [ ] TS telemetry helper call in `engine10/preflight.ts`
|
|
43
|
-
- [ ] Test
|
|
44
|
-
|
|
45
|
-
## FR-D IDE and MCP one-click works
|
|
46
|
-
|
|
47
|
-
Acceptance: README deeplink and `loki mcp install` entries carry `env: {LOKI_MCP_AUTO_BOOTSTRAP: "1"}`; a clean-HOME non-TTY launch answers `tools/list` without exit 2.
|
|
48
|
-
|
|
49
|
-
Todo:
|
|
50
|
-
- [ ] `README.md` deeplinks, `autonomy/mcp-launch.sh`
|
|
51
|
-
- [ ] Clean-HOME launch test
|
|
52
|
-
|
|
53
|
-
## FR-E Zero-install try path
|
|
54
|
-
|
|
55
|
-
Story: a prospect sees a real sealed run in under 3 minutes without installing anything.
|
|
56
|
-
|
|
57
|
-
Acceptance: a hosted or browser path (extend `loki tour` / Control Plane, do not build a new product) replays or runs a sample and shows `loki verify` passing on the receipt. Time from link to verified receipt < 3 min.
|
|
58
|
-
|
|
59
|
-
Todo:
|
|
60
|
-
- [ ] Decide host (Control Plane or static page with the browser verifier from PRF-B)
|
|
61
|
-
- [ ] Build and measure
|
|
@@ -1,61 +0,0 @@
|
|
|
1
|
-
# 05 Deploy
|
|
2
|
-
|
|
3
|
-
Now 7.5. Winner today: Replit (code plus database rollback), Vercel plus Neon (database branch per preview), Lovable and Bolt for one-click.
|
|
4
|
-
|
|
5
|
-
What exists: `loki deploy` (bash only) prints by default, `--execute` behind a VERIFIED receipt gate; Vercel, Netlify, Fly.io, Cloudflare; health checks; auto rollback on Vercel and Fly only; canary; preview/staging destinations (Vercel only); env-file push with names only in records; domain with TLS proof (Vercel, Fly); signed `loki.deploy/1` record.
|
|
6
|
-
|
|
7
|
-
Why we are not 10: no database setup, no per-PR preview URL, no auto rollback on Netlify or Cloudflare, no continuous health watch, deploy not on the Bun route, Vercel secrets on argv (`-e K=V` visible in `ps`).
|
|
8
|
-
|
|
9
|
-
Bar for 10: every PR from `loki start --deploy` carries a live preview URL with health; failed health rolls back on all 4 providers; optional DB provisioning; signed record for each step.
|
|
10
|
-
|
|
11
|
-
## DEP-A Per-PR preview URL
|
|
12
|
-
|
|
13
|
-
Acceptance: `loki start --deploy --destination preview` puts the preview URL and health status in the PR body for Vercel and Netlify; stub-CLI test asserts the PR body.
|
|
14
|
-
|
|
15
|
-
Todo:
|
|
16
|
-
- [ ] `cmd_deploy` preview mode for Netlify
|
|
17
|
-
- [ ] PR body wiring in `engine10/stages/pr.ts`
|
|
18
|
-
- [ ] Test
|
|
19
|
-
|
|
20
|
-
## DEP-B Auto rollback on all providers
|
|
21
|
-
|
|
22
|
-
Acceptance: Netlify uses its API restore; Cloudflare uses `wrangler rollback`. Stub-CLI test shows FAILED_HEALTH, rollback, post_rollback_health recorded, exit 5.
|
|
23
|
-
|
|
24
|
-
Todo:
|
|
25
|
-
- [ ] Netlify rollback
|
|
26
|
-
- [ ] Cloudflare rollback
|
|
27
|
-
- [ ] Tests
|
|
28
|
-
|
|
29
|
-
## DEP-C Database provisioning hook
|
|
30
|
-
|
|
31
|
-
Acceptance: `loki deploy --db neon|supabase` provisions a DB (or a branch per preview on Neon), writes `DATABASE_URL` through the env-file path, records names only, refuses without CLI auth.
|
|
32
|
-
|
|
33
|
-
Todo:
|
|
34
|
-
- [ ] Neon branch per preview
|
|
35
|
-
- [ ] Supabase project
|
|
36
|
-
- [ ] Rollback story for schema (document limits honestly)
|
|
37
|
-
|
|
38
|
-
## DEP-D Health watch
|
|
39
|
-
|
|
40
|
-
Acceptance: `loki status --probe --watch N` (or Control Plane cron) detects a regression and opens an alert or rollback suggestion; test with a fake server that turns 500.
|
|
41
|
-
|
|
42
|
-
Todo:
|
|
43
|
-
- [ ] Watch mode in `status.ts`
|
|
44
|
-
- [ ] Test
|
|
45
|
-
|
|
46
|
-
## DEP-E Deploy on the Bun route
|
|
47
|
-
|
|
48
|
-
Acceptance: `commands/deploy.ts` registered; print-only output byte-equal to bash (parity test); `--execute` gate tests ported.
|
|
49
|
-
|
|
50
|
-
Todo:
|
|
51
|
-
- [ ] Port print-only
|
|
52
|
-
- [ ] Port execute gate
|
|
53
|
-
- [ ] Remove the bash path once parity holds (extend, never duplicate)
|
|
54
|
-
|
|
55
|
-
## DEP-F Secrets never on argv
|
|
56
|
-
|
|
57
|
-
Acceptance: Vercel env push uses stdin or a file descriptor; a test asserts no secret value appears in the spawned argv.
|
|
58
|
-
|
|
59
|
-
Todo:
|
|
60
|
-
- [ ] Fix Vercel env push
|
|
61
|
-
- [ ] Sweep other providers
|
|
@@ -1,64 +0,0 @@
|
|
|
1
|
-
# 06 Stability
|
|
2
|
-
|
|
3
|
-
Now 7.5. Winner today: no clear leader; the bar is a public status page plus sustained green.
|
|
4
|
-
|
|
5
|
-
Why we are not 10:
|
|
6
|
-
- Main Tests red about 10 hours on 2026-10-17 (28 of last 30 failed); over 200 runs since 10-02, 87 success, 106 failure; longest green streak 23.
|
|
7
|
-
- Failures: structural line budgets (engine10 core 5015 >= 5000), moat P9 allowtoken-only-fetch-callers, L2 destructive registry, 7 FC-01 resolver and multi-root verify tests, stub e2e "worker killed by the supervisor backstop", shell shards 3, 4, 5.
|
|
8
|
-
- Releases ship while Tests is red by decision (D102); caps break when parallel slices merge (FC-60) and nothing enforces them after merge.
|
|
9
|
-
- Latest nightly read success while every test job was skipped (same-SHA skip).
|
|
10
|
-
|
|
11
|
-
Bar for 10: 24 hours of consecutive green on main, red-to-green under 1 hour, merged-train guards on every push, public status page.
|
|
12
|
-
|
|
13
|
-
## STAB-1 Main green now
|
|
14
|
-
|
|
15
|
-
Todo:
|
|
16
|
-
- [ ] Engine10 core under 4950 with headroom (fix/e10-core-budget branch, mapPool into util/check_result.ts)
|
|
17
|
-
- [ ] P9 allowToken pin fixed (FC-25c a), moat P9 PROVEN
|
|
18
|
-
- [ ] L2 destructive git call registry current
|
|
19
|
-
- [ ] 7 FC-01 resolver / multi-root verify tests green
|
|
20
|
-
- [ ] Supervisor backstop kills in stub e2e diagnosed (FAILURE-CLASSES row first)
|
|
21
|
-
- [ ] Shell shards 3, 4, 5 green
|
|
22
|
-
|
|
23
|
-
Acceptance: three consecutive green Tests runs on main, each cited by run id.
|
|
24
|
-
|
|
25
|
-
## STAB-2 Merged-train guard job
|
|
26
|
-
|
|
27
|
-
Story: a cap that each slice passes alone but the merged main breaks is caught within minutes.
|
|
28
|
-
|
|
29
|
-
Acceptance: on every main push, a fast job runs budget, L0, moat P9 and structural checks; on red it opens a BOARD P0 row within 5 minutes (fixture test).
|
|
30
|
-
|
|
31
|
-
Todo:
|
|
32
|
-
- [ ] Job in `.github/workflows/tests.yml` (extend, no new workflow)
|
|
33
|
-
- [ ] Auto P0 row writer
|
|
34
|
-
|
|
35
|
-
## STAB-3 Nightly honesty
|
|
36
|
-
|
|
37
|
-
Acceptance: a nightly where every test job was skipped concludes neutral or failure, never success.
|
|
38
|
-
|
|
39
|
-
Todo:
|
|
40
|
-
- [ ] `.github/workflows/nightly.yml` conclusion rule plus test
|
|
41
|
-
|
|
42
|
-
## STAB-4 Green streak and time-to-fix in pulse
|
|
43
|
-
|
|
44
|
-
Acceptance: pulse prints current green streak and `MAIN_RED_MINUTES`; VIOLATION when red over 60 minutes.
|
|
45
|
-
|
|
46
|
-
Todo:
|
|
47
|
-
- [ ] `scripts/v10-pulse.sh`
|
|
48
|
-
- [ ] Fix BUDGET_BURN active-engineer miscount (see 02 COST-D)
|
|
49
|
-
|
|
50
|
-
## STAB-5 Public status page
|
|
51
|
-
|
|
52
|
-
Acceptance: a page shows main CI streak, last release, npm dist-tags and incident history, generated from the same data as pulse.
|
|
53
|
-
|
|
54
|
-
Todo:
|
|
55
|
-
- [ ] Generator plus publish step
|
|
56
|
-
|
|
57
|
-
## STAB-6 Repo hygiene
|
|
58
|
-
|
|
59
|
-
Todo:
|
|
60
|
-
- [ ] Triage 295 unmerged local branches (delete only those whose work is on main or parked)
|
|
61
|
-
- [ ] Fix BOARD unparsed rows Q17-05b, Q18-03e, A-112b
|
|
62
|
-
- [ ] METRICS.md:45 moat count and ENGINE-MEASURE.md refreshed for v10
|
|
63
|
-
- [ ] Drift audit cadence kept (pulse DRIFT AUDIT DUE)
|
|
64
|
-
- [ ] Remove stale local tag mismatch (local v11.17.53 vs npm next)
|
|
@@ -1,57 +0,0 @@
|
|
|
1
|
-
# 07 Delivered quality and autonomy
|
|
2
|
-
|
|
3
|
-
Now 7.5. Winner today: Claude Code and Codex; Terminal-Bench 2.0 about 82% is the public bar; SWE-bench Verified is saturated (about 96%) and no longer differentiates.
|
|
4
|
-
|
|
5
|
-
Why we are not 10:
|
|
6
|
-
- Small tier (29 tasks, opus): raw 27/29 at $0.2363 p50 39s; v10 default 26/29 at $0.3946 p50 85.5s. Lean 27/29 at $0.1616 p50 40s but not default.
|
|
7
|
-
- Medium: no arm meets 85.7% completion at <= $0.5085 per completed. pub-werkzeug-3105 and 3271 fail in all arms.
|
|
8
|
-
- D50: Loki+sonnet 50% vs raw sonnet 90% (false ALREADY_SATISFIED, empty diff, BLOCKED); 2 of 4 losses real on rerun.
|
|
9
|
-
- Moat: P3 Wall (3 cases) and P4 model freedom (2 cases) pending; P9 regressed on main.
|
|
10
|
-
- A-119b: weakening via helper file or snapshot still ends VERIFIED.
|
|
11
|
-
|
|
12
|
-
Bar for 10: medium completion >= 85.7% at <= $0.5085 per completed with CI; Loki beats raw at every model tier; moat 9 of 9; public Terminal-Bench 2.0 and SWE-bench Pro numbers at or above raw Claude Code.
|
|
13
|
-
|
|
14
|
-
## QUAL-A Beat raw on medium
|
|
15
|
-
|
|
16
|
-
Acceptance: MEAS-4 run shows completion >= 85.7% and cost < $0.5085 per completed, CI reported.
|
|
17
|
-
|
|
18
|
-
Todo:
|
|
19
|
-
- [ ] Root-cause werkzeug-3105 and 3271 (FAILURE-CLASSES rows)
|
|
20
|
-
- [ ] Apply COST-B brief and SPD-A Wall changes, re-measure
|
|
21
|
-
|
|
22
|
-
## QUAL-B Close the D50 sonnet loss
|
|
23
|
-
|
|
24
|
-
Acceptance: Loki+sonnet >= raw sonnet at n=3 on the D50 set; humanize-174 and aiq-52 pass.
|
|
25
|
-
|
|
26
|
-
Todo:
|
|
27
|
-
- [ ] False ALREADY_SATISFIED guard in intake
|
|
28
|
-
- [ ] Empty-diff retry rule in implement
|
|
29
|
-
- [ ] Re-run D50
|
|
30
|
-
|
|
31
|
-
## QUAL-C Moat 9 of 9
|
|
32
|
-
|
|
33
|
-
Todo:
|
|
34
|
-
- [ ] P3: check-author step, `--freeze` interface, implementer prompt no longer authors checks (3 cases PASS)
|
|
35
|
-
- [ ] P4: high and small tiers resolve to distinct models (`three-setups-resolve` PASS)
|
|
36
|
-
- [ ] P4: floor-does-not-raise-wrong-pass PASS (needs MEAS-5)
|
|
37
|
-
- [ ] P9 stays PROVEN (STAB-1)
|
|
38
|
-
- [ ] Moat suite runs on every main push, not only full-suite (feeds STAB-2)
|
|
39
|
-
|
|
40
|
-
Acceptance: `bash tests/moat/run.sh` reports 9 of 9 PROVEN, `pending.txt` empty of P3/P4.
|
|
41
|
-
|
|
42
|
-
## QUAL-D Close A-119b weakening holes
|
|
43
|
-
|
|
44
|
-
Acceptance: A-119b repro cases (helper `eq=()=>{}`, snapshot weakening) end FAILED or PARTIAL with rc != 0.
|
|
45
|
-
|
|
46
|
-
Todo:
|
|
47
|
-
- [ ] `engine10/stages/verify.ts` detection
|
|
48
|
-
- [ ] Moat case for each repro
|
|
49
|
-
|
|
50
|
-
## QUAL-E Public benchmark numbers
|
|
51
|
-
|
|
52
|
-
Acceptance: MEAS-6 results published on the scoreboard with Loki and raw Claude Code side by side.
|
|
53
|
-
|
|
54
|
-
Todo:
|
|
55
|
-
- [ ] Terminal-Bench 2.0 score
|
|
56
|
-
- [ ] SWE-bench Pro pilot score
|
|
57
|
-
- [ ] Q18-03e followups: jsonc, toml, import-attributes and symlink cases in the project model
|
|
@@ -1,48 +0,0 @@
|
|
|
1
|
-
# 08 Proof and trust
|
|
2
|
-
|
|
3
|
-
Now 9. Winner today: Loki. No competitor signs a record binding spec, tests and result. This is the moat; the work here is making it verifiable by a stranger.
|
|
4
|
-
|
|
5
|
-
What exists: VERIFIED needs agent-independent proof by default (D104, D105); Ed25519 receipts signed by default; offline `loki verify` (hash, schema, events prefix and seq, JWT attestation, sealed line, drift; `--export-dsse`, `--url` with `.well-known/loki-provenance.json`, `--require-exogenous`); CycloneDX AI-BOM; `loki tour`; gate badge; loki-seal plugin; 67 moat cases.
|
|
6
|
-
|
|
7
|
-
Why we are not 10:
|
|
8
|
-
- Trust root is a self-generated local key; a third party needs the JWK out of band; unknown kid reads UNCHECKED; no transparency log; no published key directory.
|
|
9
|
-
- No browser verifier; `vendor/autonomi-verify/embed.js` is a closed prebuilt bundle.
|
|
10
|
-
- Not shown that the receipt records proof opt-outs (`LOKI_EXOGENOUS_PROOF=0`, `LOKI_PROOF_REQUIRE_EXOGENOUS=0`, `LOKI_VERIFY_ALLOW_UNSIGNED=1`); check `seal.ts` first.
|
|
11
|
-
- No published catch rate (MEAS-5); no independent audit.
|
|
12
|
-
|
|
13
|
-
Bar for 10: anyone can verify any Loki receipt in a browser with no install, against a public log, and read an independent audit and a published catch rate.
|
|
14
|
-
|
|
15
|
-
## PRF-A Third-party verifiable trust root
|
|
16
|
-
|
|
17
|
-
Acceptance:
|
|
18
|
-
- Receipts optionally logged to a transparency log (Sigstore Rekor); `loki verify` checks inclusion.
|
|
19
|
-
- Public key directory (per user or org) discoverable from the receipt kid.
|
|
20
|
-
- Unknown kid with a log entry verifies as VERIFIED (logged), not UNCHECKED.
|
|
21
|
-
|
|
22
|
-
Todo:
|
|
23
|
-
- [ ] Rekor upload on seal (opt-in, then default after review)
|
|
24
|
-
- [ ] Inclusion check in `verify_cmd.ts`
|
|
25
|
-
- [ ] Key directory format plus lookup
|
|
26
|
-
|
|
27
|
-
## PRF-B Browser verifier and open source
|
|
28
|
-
|
|
29
|
-
Acceptance: a static page verifies a dropped receipt fully client side (same checks as `loki verify`); verifier source is open and built in CI; the vendor bundle is either open-sourced or replaced.
|
|
30
|
-
|
|
31
|
-
Todo:
|
|
32
|
-
- [ ] Shared verify core compiled for browser
|
|
33
|
-
- [ ] Page (also serves FR-E)
|
|
34
|
-
- [ ] Decide on `vendor/autonomi-verify`
|
|
35
|
-
|
|
36
|
-
## PRF-C Receipts record every opt-out
|
|
37
|
-
|
|
38
|
-
Acceptance: every proof-weakening env var that was active is listed in the signed receipt and shown by `loki verify --explain`; test per variable.
|
|
39
|
-
|
|
40
|
-
Todo:
|
|
41
|
-
- [ ] Read `seal.ts` and confirm current behavior
|
|
42
|
-
- [ ] Record and display
|
|
43
|
-
|
|
44
|
-
## PRF-D Published catch rate and audit
|
|
45
|
-
|
|
46
|
-
Todo:
|
|
47
|
-
- [ ] Publish MEAS-5 catch rate on the scoreboard
|
|
48
|
-
- [ ] Commission an independent audit of the receipt chain, `verify_cmd` and the threat model; publish the report and fix list
|
|
@@ -1,74 +0,0 @@
|
|
|
1
|
-
# 09 Safety and security
|
|
2
|
-
|
|
3
|
-
Now 8.5. Bar set by: Codex (offline agent phase, secrets stripped before the agent runs, domain allowlist), Claude Code (OS sandbox: Seatbelt on macOS, bubblewrap on Linux), Copilot (default security scans, restricted push branches).
|
|
4
|
-
|
|
5
|
-
Why we are not 10:
|
|
6
|
-
- `loki share` uploads a PUBLIC gist by default from `loki report` output with no redact call (`autonomy/loki` cmd_report / share). Live leak risk.
|
|
7
|
-
- `redact.ts` misses secret words mid-name: `AWS_SECRET_ACCESS_KEY`, `PGPASSWORD`, `MYSQL_PWD`, `DB_PASS`, `PRIVATE_KEY`, `SECRET_KEY_BASE`, `client_secret_value`; unterminated `password="...`; the `=>` form. Python `proof_redact.py` catches them (parity gap).
|
|
8
|
-
- URL userinfo ReDoS (2.8s on 100KB). SECURITY-26 fix `fb2408cfc` was BLOCKED: the added `\b` lets hash-glued URLs leak userinfo.
|
|
9
|
-
- Default agent mode is bypassPermissions; engine10 has no sandbox; the bash denylist is literal (`rm -r -f /`, `find / -delete`, `shutil.rmtree` pass).
|
|
10
|
-
- `policyGate` covers only the PR action and allows when no policy file exists.
|
|
11
|
-
- npm publish has no `--provenance`, uses a long-lived `NPM_TOKEN`; actions mostly tag-pinned.
|
|
12
|
-
- SARIF and `redact.ts` keep separate pattern lists; SARIF never gates.
|
|
13
|
-
|
|
14
|
-
Owns `loki-ts/src/util/redact.ts` for this program. All tests build secrets by string concatenation so gitleaks does not flag them. HIGH tier: opus adversarial reviewer on every item.
|
|
15
|
-
|
|
16
|
-
## SAFE-1 Share is private and redacted
|
|
17
|
-
|
|
18
|
-
Acceptance: `loki share` defaults to a secret gist, requires `--public` explicitly, and runs output through redaction; test with seeded secrets shows none uploaded.
|
|
19
|
-
|
|
20
|
-
Todo:
|
|
21
|
-
- [ ] Flip default, add redact call
|
|
22
|
-
- [ ] Test
|
|
23
|
-
|
|
24
|
-
## SAFE-2 Redaction parity and ReDoS
|
|
25
|
-
|
|
26
|
-
Todo:
|
|
27
|
-
- [ ] Rework SECURITY-26 on main: URL userinfo scheme `/([a-z][a-z0-9+.-]{0,63}:\/\/)[^\s/?#@]+@/gi` (no `\b`), regression tests for `"sha256:"+h+"https://admin:...@db"`, `h+"postgres://u:...@db"`, `"x".repeat(80)+"://u:p@h"`; ReDoS test limit 250ms
|
|
28
|
-
- [ ] N2: unterminated quoted values redacted
|
|
29
|
-
- [ ] N3: mid-name keywords (AWS_SECRET_ACCESS_KEY, PGPASSWORD, MYSQL_PWD, DB_PASS, PRIVATE_KEY, SECRET_KEY_BASE)
|
|
30
|
-
- [ ] N4: `=>` assignment form
|
|
31
|
-
- [ ] Parity test: the same corpus through `redact.ts` and `proof_redact.py` gives the same masked set
|
|
32
|
-
|
|
33
|
-
Acceptance: parity test green; adversarial 100KB and 1MB inputs linear; opus HIGH reviewer APPROVE.
|
|
34
|
-
|
|
35
|
-
## SAFE-3 Sandbox by default on engine10
|
|
36
|
-
|
|
37
|
-
Acceptance: engine10 agent runs inside the OS sandbox (Seatbelt / bubblewrap, or the provider's own sandbox) with write access limited to the worktree; `bypassPermissions` only inside the sandbox; denylist tests cover `rm -r -f /`, `find / -delete`, `python -c shutil.rmtree`.
|
|
38
|
-
|
|
39
|
-
Todo:
|
|
40
|
-
- [ ] Sandbox wrapper in runner
|
|
41
|
-
- [ ] Shape-based denylist (not literal strings)
|
|
42
|
-
- [ ] Escape tests
|
|
43
|
-
|
|
44
|
-
## SAFE-4 Egress allowlist and offline phase
|
|
45
|
-
|
|
46
|
-
Acceptance: optional offline agent phase after dependency install; domain allowlist enforced; secrets stripped from the agent env before the agent runs (spawn env guard extended).
|
|
47
|
-
|
|
48
|
-
Todo:
|
|
49
|
-
- [ ] Allowlist config plus enforcement
|
|
50
|
-
- [ ] Offline phase flag
|
|
51
|
-
- [ ] Env strip test
|
|
52
|
-
|
|
53
|
-
## SAFE-5 Policy gate everywhere
|
|
54
|
-
|
|
55
|
-
Acceptance: `policyGate` runs for commit, push, deploy and network actions; missing check.js fails closed; tests per action.
|
|
56
|
-
|
|
57
|
-
Todo:
|
|
58
|
-
- [ ] Extend `util/run_outputs.ts` call sites
|
|
59
|
-
|
|
60
|
-
## SAFE-6 Supply chain
|
|
61
|
-
|
|
62
|
-
Acceptance: `npm publish --provenance` via OIDC (`id-token: write`), no long-lived NPM_TOKEN; all actions SHA-pinned; SLSA provenance for the tarball; `npm view loki-mode` shows provenance.
|
|
63
|
-
|
|
64
|
-
Todo:
|
|
65
|
-
- [ ] release.yml publish job
|
|
66
|
-
- [ ] Pin actions by SHA (script plus guard)
|
|
67
|
-
|
|
68
|
-
## SAFE-7 One pattern list
|
|
69
|
-
|
|
70
|
-
Acceptance: SARIF export and `redact.ts` read one shared pattern source; SARIF can gate with `--fail-on`; drift test.
|
|
71
|
-
|
|
72
|
-
Todo:
|
|
73
|
-
- [ ] Shared module, delete the duplicate list
|
|
74
|
-
- [ ] Gate flag
|