loki-mode 11.74.2 → 11.75.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/SKILL.md +2 -3
- package/VERSION +1 -1
- package/autonomy/completion-council.sh +10 -10
- package/autonomy/council-v2.sh +4 -4
- package/autonomy/grill.sh +1 -1
- package/autonomy/lib/browser-open.sh +1 -1
- package/autonomy/lib/chat.sh +1 -1
- package/autonomy/lib/cockpit-render.sh +1 -1
- package/autonomy/lib/done-recognition.sh +4 -4
- package/autonomy/lib/efficiency_cost.py +1 -1
- package/autonomy/lib/engine10-push.sh +2 -2
- package/autonomy/lib/prd-enrich.sh +1 -1
- package/autonomy/lib/proof-check.sh +1 -1
- package/autonomy/lib/proof-generator.py +3 -3
- package/autonomy/lib/proof-pr.sh +1 -1
- package/autonomy/lib/skill-link-heal.sh +1 -1
- package/autonomy/lib/voter-agents.sh +1 -1
- package/autonomy/loki +26 -37
- package/autonomy/prd-checklist.sh +1 -1
- package/autonomy/receipt_jwt.py +1 -1
- package/autonomy/run.sh +28 -28
- package/autonomy/trigger-schedule.py +6 -414
- package/autonomy/tui.sh +1 -1
- package/autonomy/verify.sh +4 -4
- package/completions/loki.bash +0 -5
- package/dashboard/__init__.py +1 -1
- package/docs/ARCHITECTURE.md +1 -2
- package/docs/CLI-REFERENCE.md +1 -1
- package/docs/COMPONENTS.md +1 -1
- package/docs/INSTALLATION.md +1 -1
- package/docs/ROUTING-EVIDENCE.md +1 -2
- package/docs/V10-RESEARCH.md +1 -1
- package/docs/dev/architecture-reference.md +3 -7
- package/docs/dev/project-structure.md +1 -4
- package/docs/dev/release-checklist.md +13 -31
- package/docs/research-2026-07/OBSERVABILITY-TOOL-SPEC.md +1 -1
- package/docs/v11/MARKET-SLICES.md +1 -1
- package/docs/v11/REGISTRIES.md +1 -1
- package/docs/v11/ROUTER-1.md +2 -2
- package/loki-ts/dist/loki.js +2 -2
- package/mcp/__init__.py +1 -1
- package/package.json +2 -2
- package/plugins/loki-mode/.claude-plugin/plugin.json +1 -1
- package/skills/extending.md +2 -3
- package/skills/factory-operations.md +11 -18
- package/skills/release-cadence.md +2 -2
- package/web-app/dist/assets/{AdminPage-jkjilfbl.js → AdminPage-xKgtIhk2.js} +1 -1
- package/web-app/dist/assets/{Avatar-bd5JWkSJ.js → Avatar-Q14AHox9.js} +1 -1
- package/web-app/dist/assets/{Badge-DVRit2Xw.js → Badge-yk5M-lip.js} +1 -1
- package/web-app/dist/assets/{Button-BNAB-pVZ.js → Button-ynznk0At.js} +1 -1
- package/web-app/dist/assets/{CockpitPage-DpG29h_w.js → CockpitPage-C8B765za.js} +1 -1
- package/web-app/dist/assets/{ComparePage-Dasg5V41.js → ComparePage-DqzmWKnf.js} +1 -1
- package/web-app/dist/assets/{ErrorBoundary-DJojse6b.js → ErrorBoundary-DdBPd_Lq.js} +1 -1
- package/web-app/dist/assets/{EvidenceReceiptPanel-Blxw7_u1.js → EvidenceReceiptPanel-jb4Tp38b.js} +1 -1
- package/web-app/dist/assets/{GitHubIssuesPanel-BBvSFj80.js → GitHubIssuesPanel-xJQiLcxs.js} +1 -1
- package/web-app/dist/assets/{GitHubPRsPanel-BCIKeSBg.js → GitHubPRsPanel-84W3Dhk_.js} +1 -1
- package/web-app/dist/assets/{HomePage-C7d__oNG.js → HomePage-St1Xl2tu.js} +2 -2
- package/web-app/dist/assets/{LoginPage-BXljL3yY.js → LoginPage-jRt0NUHb.js} +1 -1
- package/web-app/dist/assets/{MagicPage-Xjkts_yE.js → MagicPage-CQLFe7LR.js} +1 -1
- package/web-app/dist/assets/{MetricsPage-kQvVp-j2.js → MetricsPage-zzX9eTC5.js} +1 -1
- package/web-app/dist/assets/{NotFoundPage-D-4ZlEaO.js → NotFoundPage-DDu023SY.js} +1 -1
- package/web-app/dist/assets/{ProjectPage-DfzR5RXP.js → ProjectPage-D60UURaH.js} +3 -3
- package/web-app/dist/assets/{ProjectsPage-DFDsWzkV.js → ProjectsPage-DqVsKxlt.js} +1 -1
- package/web-app/dist/assets/{SettingsPage-CLPIoYT3.js → SettingsPage-G7pPKvo0.js} +1 -1
- package/web-app/dist/assets/{ShowcasePage-PZKkyGeY.js → ShowcasePage-CrU_aKgS.js} +1 -1
- package/web-app/dist/assets/{SystemSettingsPage-BsBlcnzo.js → SystemSettingsPage-ClKGQFsB.js} +1 -1
- package/web-app/dist/assets/{TeamsPage-BSjgNhoU.js → TeamsPage-B530vSNe.js} +1 -1
- package/web-app/dist/assets/{TemplatesPage-W2KeAzbg.js → TemplatesPage-COmv_Cqd.js} +1 -1
- package/web-app/dist/assets/{TerminalOutput-BjroTiCC.js → TerminalOutput-CufG9YzF.js} +1 -1
- package/web-app/dist/assets/{activity-BXZ5YYes.js → activity-Ct1Updr6.js} +1 -1
- package/web-app/dist/assets/{bell-D1urp5pa.js → bell-rcvhQuGr.js} +1 -1
- package/web-app/dist/assets/{bot-Chedf2bF.js → bot-DzeqEI-b.js} +1 -1
- package/web-app/dist/assets/{check-DhaCuOCV.js → check-zacnZHyC.js} +1 -1
- package/web-app/dist/assets/{chevron-left-D7BpQyA6.js → chevron-left-Lxnkl8Qg.js} +1 -1
- package/web-app/dist/assets/{circle-alert-DnGvQ_AQ.js → circle-alert-DoqSdQdB.js} +1 -1
- package/web-app/dist/assets/{clock-CqfxWhHM.js → clock-B7bJcVIC.js} +1 -1
- package/web-app/dist/assets/{cloud-BKxp5AWB.js → cloud-DdA9Bu6q.js} +1 -1
- package/web-app/dist/assets/{code-xml-BDQ6GBtt.js → code-xml-CAZEZ9T8.js} +1 -1
- package/web-app/dist/assets/{database-BmpEx1y_.js → database-Da2GkEIN.js} +1 -1
- package/web-app/dist/assets/{dollar-sign-DYNY9R_C.js → dollar-sign-DjDiG-37.js} +1 -1
- package/web-app/dist/assets/{file-code-corner-BJi0IBFd.js → file-code-corner-CSL-Jij-.js} +1 -1
- package/web-app/dist/assets/{file-plus-Dqyb2bcQ.js → file-plus-02DRA8pV.js} +1 -1
- package/web-app/dist/assets/{globe-DPh4RWfM.js → globe-CGyB90kK.js} +1 -1
- package/web-app/dist/assets/{hammer-B3xnuoA1.js → hammer-BaQB96TG.js} +1 -1
- package/web-app/dist/assets/{index-BpX8DcTr.js → index-lcoI1_Kk.js} +3 -3
- package/web-app/dist/assets/{layers-BRCmyCW8.js → layers-BWJBvoeg.js} +1 -1
- package/web-app/dist/assets/{loader-circle-n07Pky0M.js → loader-circle-CdcI0FeS.js} +1 -1
- package/web-app/dist/assets/{lock-BWqOuQmS.js → lock-BBkw5NJx.js} +1 -1
- package/web-app/dist/assets/{package-BJlB_Z-Z.js → package-BpPRO3LM.js} +1 -1
- package/web-app/dist/assets/{plus-oH17BZu2.js → plus-D7XbyK29.js} +1 -1
- package/web-app/dist/assets/{refresh-cw-CPQ476LL.js → refresh-cw-DLG3hyAi.js} +1 -1
- package/web-app/dist/assets/{rotate-ccw-7hLvOcnS.js → rotate-ccw-Bqau4R_S.js} +1 -1
- package/web-app/dist/assets/{scroll-text-BCmLauQn.js → scroll-text-BtIL3cNk.js} +1 -1
- package/web-app/dist/assets/{server-DfTfvW84.js → server-tZCmZ4yH.js} +1 -1
- package/web-app/dist/assets/{shield-alert-CaeGRq19.js → shield-alert-K5JXzFKg.js} +1 -1
- package/web-app/dist/assets/{thumbs-up-Dz-l2GPC.js → thumbs-up-CHCVyJdw.js} +1 -1
- package/web-app/dist/assets/{trash-2-B-OsDmSo.js → trash-2-DncNxVq0.js} +1 -1
- package/web-app/dist/assets/{trending-down-CXTFOxzq.js → trending-down-D8Bp0azX.js} +1 -1
- package/web-app/dist/assets/{trending-up-FQEdqr3v.js → trending-up-AixETdKR.js} +1 -1
- package/web-app/dist/assets/{upload-CacWSNTg.js → upload-VCbjFn2H.js} +1 -1
- package/web-app/dist/assets/{usePolling-cIjZ9qd2.js → usePolling-CQvKD2Jj.js} +1 -1
- package/web-app/dist/assets/{user-DAouWFHi.js → user-SK-YguQ6.js} +1 -1
- package/web-app/dist/index.html +1 -1
- package/autonomy/company-seats.json +0 -82
- package/docs/COMPETITIVE-INTEL-2026-09.md +0 -444
- package/docs/FOUNDER-DECISIONS-TRUST-SEMANTICS-2026-08-01.md +0 -145
- package/docs/FOUNDER-STATUS-AND-DECISIONS-2026-07-01.md +0 -132
- package/docs/LOKI-10-BUILD-PROMPT.md +0 -350
- package/docs/STRATEGY-2026-2028.md +0 -124
- package/docs/V10-VISION.md +0 -161
- package/docs/v10/ADOPTION-PLAN.md +0 -375
- package/docs/v10/ARCHITECT-CUTS.md +0 -744
- package/docs/v10/BACKLOG.md +0 -231
- package/docs/v10/BOARD.md +0 -1403
- package/docs/v10/CONTROL-PLANE.md +0 -120
- package/docs/v10/CONTROL.md +0 -36
- package/docs/v10/CP-ASK-PLAN.md +0 -85
- package/docs/v10/CP-ENTERPRISE-UI.md +0 -537
- package/docs/v10/D51-PHASE-B.md +0 -120
- package/docs/v10/D61-SPEED.md +0 -105
- package/docs/v10/D63-CARDS.md +0 -354
- package/docs/v10/DECISIONS.md +0 -915
- package/docs/v10/E2E-D65.md +0 -123
- package/docs/v10/ENGINE-LAWS.md +0 -163
- package/docs/v10/ENGINE-MEASURE.md +0 -69
- package/docs/v10/ENGINE.md +0 -743
- package/docs/v10/FAILURE-CLASSES.md +0 -2026
- package/docs/v10/FC21B-FC22-PLAN.md +0 -107
- package/docs/v10/FIRST-RUN-GATE.md +0 -39
- package/docs/v10/FOUNDER-QUEUE.md +0 -29
- package/docs/v10/GAP-RESEARCH-1003.md +0 -32
- package/docs/v10/GUARDS.md +0 -883
- package/docs/v10/L0-WAVE1.md +0 -81
- package/docs/v10/LEGACY-REMOVAL.md +0 -408
- package/docs/v10/MEDIUM-ANALYSIS.md +0 -346
- package/docs/v10/METRICS.md +0 -645
- package/docs/v10/MODERNIZE.md +0 -437
- package/docs/v10/OPENCODE-PARITY-PLAN.md +0 -84
- package/docs/v10/OPERATING-MODEL.md +0 -60
- package/docs/v10/PLAN-V12.md +0 -896
- package/docs/v10/PROGRESS.md +0 -2735
- package/docs/v10/RELEASE-11.md +0 -77
- package/docs/v10/RETRO-2026.md +0 -650
- package/docs/v10/ROADMAP-D63.md +0 -35
- package/docs/v10/SCALE.md +0 -140
- package/docs/v10/SCORECARD-PLAN.md +0 -250
- package/docs/v10/SWARM-PROMPT.md +0 -98
- package/docs/v10/SWARM.md +0 -293
- package/docs/v10/cp-redesign/PARITY.md +0 -188
- package/docs/v10/cp-redesign/legacy/INDEX.md +0 -1167
- package/docs/v10/roadmap-10/01-measurement.md +0 -113
- package/docs/v10/roadmap-10/02-cost.md +0 -75
- package/docs/v10/roadmap-10/03-speed.md +0 -65
- package/docs/v10/roadmap-10/04-first-run.md +0 -61
- package/docs/v10/roadmap-10/05-deploy.md +0 -61
- package/docs/v10/roadmap-10/06-stability.md +0 -64
- package/docs/v10/roadmap-10/07-quality.md +0 -57
- package/docs/v10/roadmap-10/08-proof.md +0 -48
- package/docs/v10/roadmap-10/09-safety.md +0 -74
- package/docs/v10/roadmap-10/README.md +0 -1464
- package/skills/sdlc-fleet.md +0 -170
- package/tools/cockpit/SPEC.md +0 -275
- package/tools/cockpit/index.html +0 -146
- package/tools/cockpit/roi-basis.json +0 -33
- package/tools/cockpit/server.py +0 -946
|
@@ -1,82 +0,0 @@
|
|
|
1
|
-
[
|
|
2
|
-
{
|
|
3
|
-
"name": "chief-of-staff",
|
|
4
|
-
"cron_expr": "1,16,31,46 * * * *",
|
|
5
|
-
"action": "seat chief-of-staff",
|
|
6
|
-
"args": [],
|
|
7
|
-
"enabled": true,
|
|
8
|
-
"last_run": null
|
|
9
|
-
},
|
|
10
|
-
{
|
|
11
|
-
"name": "release-reliability-lead",
|
|
12
|
-
"cron_expr": "4,19,34,49 * * * *",
|
|
13
|
-
"action": "seat release-reliability-lead",
|
|
14
|
-
"args": [],
|
|
15
|
-
"enabled": true,
|
|
16
|
-
"last_run": null
|
|
17
|
-
},
|
|
18
|
-
{
|
|
19
|
-
"name": "verification-scientist",
|
|
20
|
-
"cron_expr": "7,37 * * * *",
|
|
21
|
-
"action": "seat verification-scientist",
|
|
22
|
-
"args": [],
|
|
23
|
-
"enabled": true,
|
|
24
|
-
"last_run": null
|
|
25
|
-
},
|
|
26
|
-
{
|
|
27
|
-
"name": "platform-lead",
|
|
28
|
-
"cron_expr": "11,41 * * * *",
|
|
29
|
-
"action": "seat platform-lead",
|
|
30
|
-
"args": [],
|
|
31
|
-
"enabled": true,
|
|
32
|
-
"last_run": null
|
|
33
|
-
},
|
|
34
|
-
{
|
|
35
|
-
"name": "security-lead",
|
|
36
|
-
"cron_expr": "14,44 * * * *",
|
|
37
|
-
"action": "seat security-lead",
|
|
38
|
-
"args": [],
|
|
39
|
-
"enabled": true,
|
|
40
|
-
"last_run": null
|
|
41
|
-
},
|
|
42
|
-
{
|
|
43
|
-
"name": "cto",
|
|
44
|
-
"cron_expr": "22 * * * *",
|
|
45
|
-
"action": "seat cto",
|
|
46
|
-
"args": [],
|
|
47
|
-
"enabled": true,
|
|
48
|
-
"last_run": null
|
|
49
|
-
},
|
|
50
|
-
{
|
|
51
|
-
"name": "developer-experience-lead",
|
|
52
|
-
"cron_expr": "27 * * * *",
|
|
53
|
-
"action": "seat developer-experience-lead",
|
|
54
|
-
"args": [],
|
|
55
|
-
"enabled": true,
|
|
56
|
-
"last_run": null
|
|
57
|
-
},
|
|
58
|
-
{
|
|
59
|
-
"name": "product-growth-lead",
|
|
60
|
-
"cron_expr": "33 * * * *",
|
|
61
|
-
"action": "seat product-growth-lead",
|
|
62
|
-
"args": [],
|
|
63
|
-
"enabled": true,
|
|
64
|
-
"last_run": null
|
|
65
|
-
},
|
|
66
|
-
{
|
|
67
|
-
"name": "research-harness-scientist",
|
|
68
|
-
"cron_expr": "39 * * * *",
|
|
69
|
-
"action": "seat research-harness-scientist",
|
|
70
|
-
"args": [],
|
|
71
|
-
"enabled": true,
|
|
72
|
-
"last_run": null
|
|
73
|
-
},
|
|
74
|
-
{
|
|
75
|
-
"name": "research-outcomes-scientist",
|
|
76
|
-
"cron_expr": "52 * * * *",
|
|
77
|
-
"action": "seat research-outcomes-scientist",
|
|
78
|
-
"args": [],
|
|
79
|
-
"enabled": true,
|
|
80
|
-
"last_run": null
|
|
81
|
-
}
|
|
82
|
-
]
|
|
@@ -1,444 +0,0 @@
|
|
|
1
|
-
# Competitive intel, 30 days to 2026-09-14
|
|
2
|
-
|
|
3
|
-
Window: **2026-08-15 to 2026-09-14**. Every claim below carries a URL or a
|
|
4
|
-
file:line. Anything without one is in the UNVERIFIED section at the bottom and
|
|
5
|
-
is deliberately excluded from the ranked list.
|
|
6
|
-
|
|
7
|
-
**Prior document.** `docs/COMPETITIVE-NEXT-10.md` was written 2026-09-09 and is
|
|
8
|
-
the same shape of deliverable (ranked build list vs factory.ai and 8090, with
|
|
9
|
-
already-exists verdicts). This document does NOT supersede it. It covers what is
|
|
10
|
-
new since it, and every ranked item below states whether it is a NEXT-10 item or
|
|
11
|
-
not. Re-proposing a NEXT-10 open item as a fresh discovery would be this repo's
|
|
12
|
-
signature failure committed at the document level rather than the item level.
|
|
13
|
-
|
|
14
|
-
**Evidence labelling used throughout:**
|
|
15
|
-
|
|
16
|
-
- `[measured]` - a command run against this repo on 2026-09-14, output quoted.
|
|
17
|
-
- `[sourced: URL]` - a page actually fetched on 2026-09-14.
|
|
18
|
-
- `vendor-claimed` - the vendor's own page or blog. Evidence that a CLAIM exists,
|
|
19
|
-
not that the capability works.
|
|
20
|
-
- `UNVERIFIED` - could not be cited. Never ranked.
|
|
21
|
-
|
|
22
|
-
---
|
|
23
|
-
|
|
24
|
-
## 1. What genuinely shipped in the window (2026-08-15 to 2026-09-14)
|
|
25
|
-
|
|
26
|
-
### Factory.ai
|
|
27
|
-
|
|
28
|
-
**Aug 27, 2026 - "What it Takes for Coding Agents to Complete Large Software
|
|
29
|
-
Tasks"** [sourced: https://factory.ai/news/what-it-takes-for-coding-agents-to-complete-large-software-tasks]
|
|
30
|
-
|
|
31
|
-
The most strategically relevant thing any competitor published in this window.
|
|
32
|
-
Factory argues a single agent validating its own work stops early with "much of
|
|
33
|
-
the outcome absent", and reports that splitting the run into orchestrator /
|
|
34
|
-
implementer / validator separated by an information barrier they call **"the
|
|
35
|
-
wall"** produces large gains. The validator authors an independent measurement
|
|
36
|
-
instrument BEFORE implementation begins, so the implementer cannot tailor work to
|
|
37
|
-
visible test cases.
|
|
38
|
-
|
|
39
|
-
Numbers, all **vendor-claimed**, on ProgramBench (partial-credit behavioral
|
|
40
|
-
parity, not pass/fail):
|
|
41
|
-
|
|
42
|
-
| Subject | Single agent | With the wall |
|
|
43
|
-
|---|---|---|
|
|
44
|
-
| GDAL | 36% | 90% |
|
|
45
|
-
| 7-Zip | 54% | 95% |
|
|
46
|
-
| DuckDB | 34% | 80% |
|
|
47
|
-
| Fable 5 median, 24 tasks | 56.7% | 89.3% |
|
|
48
|
-
| Kimi K3 median | 45.1% | 75.4% |
|
|
49
|
-
| GPT-5.6-Sol median | 48.6% | 66.2% |
|
|
50
|
-
|
|
51
|
-
Note on reading these: ProgramBench's own paper reports that "none fully resolve
|
|
52
|
-
any task" [sourced: https://arxiv.org/pdf/2605.03546]. That is NOT a
|
|
53
|
-
contradiction of Factory's figures - ProgramBench scores partial behavioral
|
|
54
|
-
parity, so a high median partial score and zero fully-resolved tasks are
|
|
55
|
-
compatible. Treat the table as vendor-claimed on a partial-credit metric.
|
|
56
|
-
|
|
57
|
-
**Sep 1, 2026 - CLI v0.209.0** [sourced: https://docs.factory.ai/changelog/release-notes]
|
|
58
|
-
Default model for new sessions became GPT-5.6 Sol. Session archiving.
|
|
59
|
-
|
|
60
|
-
**Aug 17 - Sep 1, 2026 - changelog band** [sourced: same]
|
|
61
|
-
Roughly twenty releases. Read in bulk, it is overwhelmingly **UI and session
|
|
62
|
-
polish**, not capability: diff line wrapping, sidebar width persistence, session
|
|
63
|
-
search speed, model-list search box, voice dictation accuracy, markdown tables,
|
|
64
|
-
mermaid rendering. Three items have substance: `droid doctor` for connectivity
|
|
65
|
-
diagnostics (Aug 25), the explorer droid gaining every read-only tool (Aug 18),
|
|
66
|
-
and org admins being able to disable image generation (Aug 21).
|
|
67
|
-
|
|
68
|
-
**Sep 9, 2026 - Factory listed on Claude Marketplace**
|
|
69
|
-
[sourced: https://factory.ai/news/claude-marketplace]
|
|
70
|
-
Enterprises can spend existing Anthropic commitments on Factory, removing a
|
|
71
|
-
separate procurement cycle. This is a **distribution and procurement** move, not
|
|
72
|
-
an engineering one. It is ranked separately below for that reason.
|
|
73
|
-
|
|
74
|
-
### Frontier models
|
|
75
|
-
|
|
76
|
-
**Sep 3, 2026 - GPT-6 Astra** [sourced: https://openai.com/index/gpt-6-astra/,
|
|
77
|
-
https://simonwillison.net/2026/Sep/3/gpt6-astra/]. Closed multimodal reasoning
|
|
78
|
-
model; headline upgrade is native computer use.
|
|
79
|
-
|
|
80
|
-
**Claude Fable 5.1** - released in the window per
|
|
81
|
-
[sourced: https://patmcguinness.substack.com/p/claude-fable-51-gpt-6-astra-and-the].
|
|
82
|
-
Exact date not established from a primary Anthropic source; see UNVERIFIED.
|
|
83
|
-
|
|
84
|
-
**Meta Muse Spark 1.3** - named in the same roundup as one of five models pushing
|
|
85
|
-
efficiency at lower cost. Secondary source only; see UNVERIFIED.
|
|
86
|
-
|
|
87
|
-
### Independent / community signal
|
|
88
|
-
|
|
89
|
-
**Verification is now the named bottleneck.** Multiple September HN digests
|
|
90
|
-
converge on the same sentence: the constraint is no longer generation speed but
|
|
91
|
-
verification capacity
|
|
92
|
-
[sourced: https://www.developersdigest.tech/blog/what-hacker-news-gets-right-about-ai-coding-agents-2026].
|
|
93
|
-
This is the market moving toward this repo's stated wedge, not away from it.
|
|
94
|
-
|
|
95
|
-
**Docket** (github.com/yielab/docket) [sourced: https://github.com/yielab/docket]
|
|
96
|
-
is the closest direct competitor to the receipt. It gives every AI-written commit
|
|
97
|
-
a hash-chained JSONL audit log with a `docket audit verify` command. It is
|
|
98
|
-
self-labelled `v0.2.0-beta.2` and warns "Expect breaking changes between beta
|
|
99
|
-
releases"; no public star count was visible on the page I fetched, and the fetch
|
|
100
|
-
could not establish repo age or last-commit date, so **treat its adoption as
|
|
101
|
-
unmeasured rather than zero**. The durable point is architectural, and does not
|
|
102
|
-
depend on adoption: verification is hash-chain only, with **no cryptographic
|
|
103
|
-
signatures and no offline third-party verification**. Its own docs concede "an
|
|
104
|
-
operator able to delete all docket state can erase both the log and its backup."
|
|
105
|
-
|
|
106
|
-
---
|
|
107
|
-
|
|
108
|
-
## 2. Context: important, but OUTSIDE the 30-day window
|
|
109
|
-
|
|
110
|
-
Kept separate so nothing here is mistaken for a recent development.
|
|
111
|
-
|
|
112
|
-
- **Apr 1, 2026 - Legacy-Bench** [sourced: https://factory.ai/news/legacy-bench,
|
|
113
|
-
https://github.com/factory-ai/legacy-bench]. Apache-2.0, 19 stars. COBOL 46%,
|
|
114
|
-
Java 7 32%, BASIC/C89/Fortran/Assembly 5-6% each. Pass rates **16.9% to 42.5%**
|
|
115
|
-
across 12 model-agent combos, against >70% for the same models on
|
|
116
|
-
Terminal-Bench 2 and SWE-bench Verified. Droid + GPT-5.3-Codex tops it at
|
|
117
|
-
42.5%. Ten sample tasks are public; the full set requires contacting Factory.
|
|
118
|
-
Factory states "We welcome evaluation submissions."
|
|
119
|
-
- **Jun 30, 2026 - "Building to the Test: Coding Agents Deliver What You Check,
|
|
120
|
-
Not What You Requested"** [sourced: https://arxiv.org/pdf/2606.28430]. Ma,
|
|
121
|
-
Kereopa-Yorke, Schultz. Independent, non-vendor evidence for exactly the
|
|
122
|
-
failure mode Factory's "wall" post claims to fix.
|
|
123
|
-
- **Jun 1, 2026 - Factory Router** [sourced: https://factory.ai/news/factory-router].
|
|
124
|
-
"cuts token spend by 20-25%", private research preview, measured on
|
|
125
|
-
Terminal-Bench 2 and Legacy-Bench against an Opus 4.7 baseline.
|
|
126
|
-
- **Aug 13, 2026 - Agent Effectiveness** [sourced: https://factory.ai/news/agent-effectiveness].
|
|
127
|
-
Thirty-two days out, so just outside. **"Currently in Private Preview"** - a
|
|
128
|
-
claim of a product, not a shipped one. Measures cycle time, work intent, and
|
|
129
|
-
attribution of sessions to shipped artifacts.
|
|
130
|
-
- **Jan 23, 2026 - Signals** [sourced: https://factory.ai/news/factory-signals].
|
|
131
|
-
Shipped and running daily; friction detection that auto-files Linear tickets
|
|
132
|
-
and assigns them to Droid.
|
|
133
|
-
- **Sep 25, 2025 - Terminal-Bench 58.75%** [sourced: https://factory.ai/news/terminal-bench].
|
|
134
|
-
Self-reported by Factory, run on their own Ubuntu machines.
|
|
135
|
-
- **Aug 13, 2026 - DeepSeek V4-Pro GA and `deepseek-harness`** (MIT, everything a
|
|
136
|
-
plugin) [sourced: https://www.digitalapplied.com/blog/deepseek-harness-open-source-agent-framework-2026].
|
|
137
|
-
Secondary source; the post itself notes the package is `v0.1.0-rc.5` with no
|
|
138
|
-
GitHub release behind it.
|
|
139
|
-
|
|
140
|
-
### 8090.ai
|
|
141
|
-
|
|
142
|
-
**Nothing shipped in the window that I could cite.** The site sells
|
|
143
|
-
Software Factory (an "AI-native SDLC control plane") and 8090 Enterprise
|
|
144
|
-
[sourced: https://www.8090.ai/]. Most recent datable events are the EY.ai PDLC
|
|
145
|
-
launch (Mar 2026) and a $135M Series A led by Salesforce Ventures (Jun 2026)
|
|
146
|
-
[sourced: https://siliconangle.com/2026/06/29/ai-software-development-startup-8090-nabs-135m-funding-round/].
|
|
147
|
-
Absence of evidence reported as a finding, not filled with a guess.
|
|
148
|
-
|
|
149
|
-
**CORRECTION to a prior repo audit, and it goes the inconvenient way.** This
|
|
150
|
-
repo previously established that 8090's public site yields no verifiable product
|
|
151
|
-
evidence, and I was asked to confirm that in one fetch and move on. It does not
|
|
152
|
-
confirm [sourced: https://www.8090.ai/software-factory]. Present today: a free
|
|
153
|
-
trial at `factory.8090.ai` ("Get Started for Free"), public documentation at
|
|
154
|
-
`docs.8090.ai`, a demo video playlist, a pricing page, and a published CMS
|
|
155
|
-
claims-modernization case study. Still absent: benchmark numbers, a changelog,
|
|
156
|
-
and any version history.
|
|
157
|
-
|
|
158
|
-
So the accurate 2026-09-14 statement is **not** "no public product surface". It
|
|
159
|
-
is: 8090 has a self-serve on-ramp and docs, but publishes **no versioned release
|
|
160
|
-
history and no benchmarks**, which is what actually blocks capability comparison.
|
|
161
|
-
Recording this as a correction rather than a confirmation because the prior
|
|
162
|
-
finding's premise has changed, and re-asserting a stale finding as verified is
|
|
163
|
-
the same failure shape as an uncited claim.
|
|
164
|
-
|
|
165
|
-
### Correction to a prior repo finding
|
|
166
|
-
|
|
167
|
-
The repo previously established that factory.ai, 8090 and Devin have **no**
|
|
168
|
-
G2/Capterra/ProductHunt listings. **That is now outdated for factory.ai.** A G2
|
|
169
|
-
product page exists at https://www.g2.com/products/factory-ai/reviews. I could
|
|
170
|
-
not fetch it (403), and the secondary report of "1 G2 review, 3 Product Hunt
|
|
171
|
-
reviews, Capterra 4.2"
|
|
172
|
-
[sourced: https://theaiagentindex.com/agents/factory-ai] is second-hand, so treat
|
|
173
|
-
the counts as unconfirmed. The correct statement is: **a listing now exists, with
|
|
174
|
-
too few reviews for aggregate sentiment comparison.** For 8090 I found listings
|
|
175
|
-
only on SourceForge/Slashdot aggregators, no G2/Capterra/ProductHunt.
|
|
176
|
-
|
|
177
|
-
---
|
|
178
|
-
|
|
179
|
-
## 3. Ranked build list
|
|
180
|
-
|
|
181
|
-
Ranked by (user value delivered) / (effort). Each item states whether it closes a
|
|
182
|
-
gap or extends a lead, and its NEXT-10 status.
|
|
183
|
-
|
|
184
|
-
### 1. Put an information barrier between checklist authoring and implementation
|
|
185
|
-
|
|
186
|
-
**Closes a gap. Extends an existing lead once closed. Not in NEXT-10.**
|
|
187
|
-
|
|
188
|
-
**ALREADY EXISTS: PARTIALLY - and the existing half is the valuable half.**
|
|
189
|
-
I nearly filed this as "build a validator role", which would have repeated the
|
|
190
|
-
8-of-11 mistake. The check refuted it:
|
|
191
|
-
|
|
192
|
-
- The checklist is derived from the PRD **before** implementation
|
|
193
|
-
`[measured]` `autonomy/run.sh:20483` instructs the agent to create
|
|
194
|
-
`.loki/checklist/checklist.json` from the PRD on the first iteration, with
|
|
195
|
-
per-item verification checks (`file_exists`, `file_contains`, `tests_pass`,
|
|
196
|
-
`grep_codebase`, `command`). `checklist_init` is called at
|
|
197
|
-
`autonomy/run.sh:22349`; the module is `autonomy/prd-checklist.sh`.
|
|
198
|
-
- It is **independently re-verified after** a completion claim `[measured]`
|
|
199
|
-
`checklist_verify` at `autonomy/run.sh:23973`, and
|
|
200
|
-
`council_reverify_checklist` at `:24858` and `:25604`, executed by the
|
|
201
|
-
deterministic `autonomy/checklist-verify.py`.
|
|
202
|
-
|
|
203
|
-
So Loki already has a pre-authored, deterministically-rechecked standard. What it
|
|
204
|
-
does **not** have is Factory's wall: the **same agent** writes the checklist and
|
|
205
|
-
then implements against it. That is precisely the failure mode
|
|
206
|
-
`arXiv:2606.28430` names, and it is independent of Factory's marketing.
|
|
207
|
-
|
|
208
|
-
**Do:** have the checklist authored by a dispatch that does not see, and is not
|
|
209
|
-
continued by, the implementing context. The verification machinery is already
|
|
210
|
-
built, so this is a dispatch-boundary change, not a new subsystem. Highest
|
|
211
|
-
value-over-effort item on this list because the expensive half already exists.
|
|
212
|
-
|
|
213
|
-
### 2. Tamper-evidence: nothing to build, and a stale internal note to retract
|
|
214
|
-
|
|
215
|
-
**Extends the lead. Not in NEXT-10. NO BUILD REQUIRED.**
|
|
216
|
-
|
|
217
|
-
**ALREADY EXISTS: YES, FULLY.** I nearly ranked "wire `writeWitness`" as the
|
|
218
|
-
single highest-value item on this list, on the strength of an internal memory
|
|
219
|
-
(`project-audit-chain-not-tamper-proof`) stating it existed with **zero
|
|
220
|
-
production callers**. Checking the source refuted that memory outright. It is
|
|
221
|
-
outdated and should be retracted:
|
|
222
|
-
|
|
223
|
-
- `writeWitness` is defined at `src/audit/crosslink.js:234` and **has production
|
|
224
|
-
callers** `[measured]`: `src/audit/subscriber.js:142` (periodic) and `:155`
|
|
225
|
-
(session end), both via `writeWitnessSafely`, plus the wrapper export at
|
|
226
|
-
`src/audit/index.js:226,278`.
|
|
227
|
-
- The reconciliation half that the memory said was missing now exists:
|
|
228
|
-
`reconcileWitnessedPrefix` at `src/audit/crosslink.js:560` `[measured]`.
|
|
229
|
-
- Most importantly it is **wired into the verdict**, not merely present:
|
|
230
|
-
`crosslink.js:392` calls it and `:398-399` ANDs `witnessedPrefix.valid` into
|
|
231
|
-
the overall `valid`. The comparison is prefix-based, not tip-equality, so
|
|
232
|
-
legitimate chain growth does not false-positive, and the three states
|
|
233
|
-
(`checked` / `no_records` / `unreadable`) are explicitly never collapsed, with
|
|
234
|
-
`no_records` documented as "NOT a pass."
|
|
235
|
-
|
|
236
|
-
So the honest competitive position is stronger than the memory implied, and it
|
|
237
|
-
needs no work. Against Docket: `autonomy/receipt_jwt.py` exists `[measured]` and
|
|
238
|
-
`loki proof verify --jwks <url|file>` checks an Ed25519 attestation against a
|
|
239
|
-
published key set including **from a local `jwks.json` with no network**
|
|
240
|
-
`[measured: autonomy/loki:900-910]`, so a third party can check WHO produced a
|
|
241
|
-
receipt with no API token and no key import. Docket has neither signatures nor
|
|
242
|
-
offline verification, and concedes an operator can erase its log and backup.
|
|
243
|
-
|
|
244
|
-
**Do: nothing in code.** Update the memory file so a future session does not
|
|
245
|
-
re-propose this. The lesson generalises and is why this section survives at rank
|
|
246
|
-
2 despite requiring no build: a memory is evidence of what was true when written,
|
|
247
|
-
never of what is true now.
|
|
248
|
-
|
|
249
|
-
### 3. Persist a per-run human-intervention counter
|
|
250
|
-
|
|
251
|
-
**Closes a gap against Agent Effectiveness. Cheap. Not in NEXT-10.**
|
|
252
|
-
|
|
253
|
-
**REFUTED 2026-09-14 by the integrator. ALREADY EXISTS: FULLY.** This item was
|
|
254
|
-
filed on the belief that the reader exists and the writer does not. Both halves
|
|
255
|
-
of the chain ship:
|
|
256
|
-
|
|
257
|
-
- **Writer:** `handle_pause` in `autonomy/run.sh` increments
|
|
258
|
-
`.loki/state/interventions.json` (the counter block at `autonomy/run.sh:25712`).
|
|
259
|
-
Counted there deliberately: every pause path funnels through `handle_pause`,
|
|
260
|
-
and the `_PAUSE_IN_PROGRESS` guard makes one blocking pause count once.
|
|
261
|
-
- **Proof:** `autonomy/lib/proof-generator.py:1147-1151` reads that file into the
|
|
262
|
-
journey; `:1544-1545` mirrors it to top-level `proof["interventions"]`.
|
|
263
|
-
- **Reader:** `_interventions_value` in `autonomy/lib/trust_trajectory.py`.
|
|
264
|
-
|
|
265
|
-
Verified empirically: replaying the writer's logic yields count 1 then 2 across
|
|
266
|
-
two pauses. **No build required.**
|
|
267
|
-
|
|
268
|
-
The trap that made it look missing: `_AXIS_HIGHER_IS_BETTER = {"interventions":
|
|
269
|
-
False, ...}` reads like a disable flag. It is POLARITY (lower-is-better), and
|
|
270
|
-
`iterations` is `False` for the same reason. There are only three `_AXIS_*`
|
|
271
|
-
dicts and none gates availability. The axis honestly reports `available: false`
|
|
272
|
-
until a proof carries the count, which is by design, not a gap.
|
|
273
|
-
|
|
274
|
-
The stale docstring that seeded this belief was corrected in `a9a02681`.
|
|
275
|
-
|
|
276
|
-
Original (incorrect) reasoning retained below for the record:
|
|
277
|
-
|
|
278
|
-
**~~ALREADY EXISTS: THE READER DOES, THE WRITER DOES NOT.~~** This is the
|
|
279
|
-
reader/writer key contract trap this repo has hit before.
|
|
280
|
-
`autonomy/lib/trust_trajectory.py` already declares `interventions` as a tracked
|
|
281
|
-
metric with a lower-is-better direction `[measured: lines 13, 46, 54, 61]`, and
|
|
282
|
-
`_interventions_value` reads it at line 145. But its own docstring says
|
|
283
|
-
`[measured]`: *"There is no per-run intervention counter persisted today."* It is
|
|
284
|
-
gated off (`"interventions": False` at line 46).
|
|
285
|
-
|
|
286
|
-
Loki already computes things Factory's Private-Preview product does not, notably
|
|
287
|
-
**cost-per-VERIFIED-task** `[measured: autonomy/lib/trust_metrics.py:397]`, whose
|
|
288
|
-
denominator is a verified outcome rather than a shipped artifact. Factory's
|
|
289
|
-
headline metric is "how much independent work Droids do between human
|
|
290
|
-
interactions" - which is exactly the field Loki declares and never fills.
|
|
291
|
-
|
|
292
|
-
**Do:** persist the counter at the existing `check_human_intervention` site and
|
|
293
|
-
flip the flag. Small diff, and it completes a metric surface that is otherwise
|
|
294
|
-
stronger than the competitor's.
|
|
295
|
-
|
|
296
|
-
### 4. Run healing mode against Legacy-Bench
|
|
297
|
-
|
|
298
|
-
**Extends the lead, on the one axis with the most headroom. Not in NEXT-10.**
|
|
299
|
-
|
|
300
|
-
**ALREADY EXISTS: NO.** `[measured]` `benchmarks/` contains SWE-bench,
|
|
301
|
-
swebench-pro-pilot, HumanEval and internal A/B harnesses. Zero Legacy-Bench
|
|
302
|
-
coverage; the only `legacy|COBOL` hits are unrelated (`speed-benchmark.sh`,
|
|
303
|
-
equivalence reports, SWE-bench patch files).
|
|
304
|
-
|
|
305
|
-
This is the best-matched external scoreboard in existence for this repo's
|
|
306
|
-
brownfield wedge (`project-brownfield-wedge-evidence`: proof-of-function on
|
|
307
|
-
legacy migration is unclaimed by every vendor). It is Apache-2.0, runnable via
|
|
308
|
-
the Harbor harness, and Factory explicitly welcomes submissions. Frontier agents
|
|
309
|
-
score 16.9-42.5% where they score >70% elsewhere, so the headroom is real and the
|
|
310
|
-
ceiling is not yet claimed by anyone credible.
|
|
311
|
-
|
|
312
|
-
**Caveat that must not be skipped:** only 10 of the tasks are public and the full
|
|
313
|
-
set requires contacting Factory, who also own the benchmark and currently top it.
|
|
314
|
-
A self-run number on 10 public tasks is a pilot, not a leaderboard claim, and must
|
|
315
|
-
be published as such.
|
|
316
|
-
|
|
317
|
-
### 4b. Surface the receipt at the end of the run (added after founder steer)
|
|
318
|
-
|
|
319
|
-
**Closes a delivery gap, not a capability gap. Very cheap. Not in NEXT-10.**
|
|
320
|
-
|
|
321
|
-
The founder's framing reorders this list: rank by whether an item reduces human
|
|
322
|
-
round-trips or proves completion, not by feature parity. Under that lens the
|
|
323
|
-
sharpest question is not "do we have proof" but "does the user GET it at the end
|
|
324
|
-
of one run, without asking".
|
|
325
|
-
|
|
326
|
-
**ALREADY EXISTS: GENERATION YES, SURFACING BARELY.** `[measured]` The receipt is
|
|
327
|
-
produced automatically and is opt-OUT, not opt-in: `LOKI_PROOF` defaults to `1`
|
|
328
|
-
at `autonomy/run.sh:26810`, `:26894` and `:26979` (all `${LOKI_PROOF:-1}`), and
|
|
329
|
-
`generate_proof_of_run` is called on the terminal path "fire-and-forget on both
|
|
330
|
-
success and failure runs" (`:26893`). It is regenerated idempotently after
|
|
331
|
-
HANDOFF.md and commit writers finish so `proof.tree_sha256` describes the exact
|
|
332
|
-
returned tree (`:26975-26981`).
|
|
333
|
-
|
|
334
|
-
So we already beat the bar the steer set. What is thin is the last inch: the
|
|
335
|
-
only place the artifact's location reaches the user is a single echo,
|
|
336
|
-
`autonomy/run.sh:8115` - `(or open $proofs_dir/$latest/index.html)`. A user who
|
|
337
|
-
does not already know `loki proof` exists can finish a successful run without
|
|
338
|
-
ever learning a checkable receipt was written for them.
|
|
339
|
-
|
|
340
|
-
**Do:** print the receipt path plus the one-line verdict and the
|
|
341
|
-
`loki proof verify <id>` command in the end-of-run summary. This is the
|
|
342
|
-
highest value-per-line-of-code item on the list: the expensive machinery is
|
|
343
|
-
built, defaulted on, and currently under-announced.
|
|
344
|
-
|
|
345
|
-
### 5. Publish the verification-bottleneck position while the market names it
|
|
346
|
-
|
|
347
|
-
**Extends the lead. Zero engineering effort. Not in NEXT-10.**
|
|
348
|
-
|
|
349
|
-
**ALREADY EXISTS: THE CAPABILITY YES, THE POSITIONING ARTIFACT NO.**
|
|
350
|
-
`loki verify --fast` runs exogenous checks only, no model call and no network, so
|
|
351
|
-
every verdict is reproducible by anyone at the same commit
|
|
352
|
-
`[measured: autonomy/loki:18131-18145]`, at a documented 19ms diff-scoped against
|
|
353
|
-
an 11,040ms baseline. `loki proof verify` re-checks a receipt for tamper and
|
|
354
|
-
drift with `--human` prose output `[measured: autonomy/loki:36195-36210]`.
|
|
355
|
-
|
|
356
|
-
Independent HN consensus has now converged on "verification is the bottleneck",
|
|
357
|
-
and the nearest competing artifact (Docket) is a 0-star beta with no signatures.
|
|
358
|
-
This is the cheapest item here: the product is built, the market just started
|
|
359
|
-
using our vocabulary.
|
|
360
|
-
|
|
361
|
-
### Models are not products: the cost-per-task question
|
|
362
|
-
|
|
363
|
-
The founder's comparison set mixes two categories and they must not be blurred.
|
|
364
|
-
**factory.ai and 8090 are PRODUCTS** and are ranked above. **DeepSeek, Muse,
|
|
365
|
-
Claude Fable and GPT-6 Astra are MODELS.** For a provider-agnostic harness the
|
|
366
|
-
useful question about a model is not feature parity but: can we route to it, and
|
|
367
|
-
at what cost per completed task.
|
|
368
|
-
|
|
369
|
-
**ALREADY EXISTS: TIERED ROUTING YES, CHEAP-FRONTIER ROUTE NO.** `[measured]`
|
|
370
|
-
`providers/models.sh:19-24` exposes generic `small|medium|high` tiers and
|
|
371
|
-
explicitly forbids callers naming a vendor model, with `medium` the default.
|
|
372
|
-
That is the right abstraction and it is already built. But the only MiniMax
|
|
373
|
-
reference in the entire repo is a **comment**, not a route:
|
|
374
|
-
`providers/opencode.sh:25` records "MiniMax M2.5 resolved 75.8 at $36.64 total
|
|
375
|
-
vs Claude Opus 4.6 at 75.6". That is a measured near-parity result at roughly
|
|
376
|
-
7.5x lower cost, sitting in a comment.
|
|
377
|
-
|
|
378
|
-
Two competitive signals make this timely. Factory **removed MiniMax M2.5 from
|
|
379
|
-
model selection** on Aug 11 [sourced: https://docs.factory.ai/changelog/release-notes],
|
|
380
|
-
and DeepSeek V4 ships open weights under MIT (Aug 13 V4-Pro GA, secondary
|
|
381
|
-
source). A competitor retiring the cheap near-parity model is the moment a
|
|
382
|
-
provider-agnostic harness can differentiate on cost per completed task.
|
|
383
|
-
|
|
384
|
-
**Do:** promote the opencode cheap-frontier result from a comment to a selectable
|
|
385
|
-
tier with a published cost-per-completed-task number. Low effort, directly serves
|
|
386
|
-
the "2-3x value" ask, and it is the one item here where being provider-agnostic
|
|
387
|
-
is a structural advantage rather than a parity feature.
|
|
388
|
-
|
|
389
|
-
**Honest limit:** the $36.64 figure is a single internal benchmark run recorded
|
|
390
|
-
in a comment. It must be re-measured before it is published anywhere external.
|
|
391
|
-
|
|
392
|
-
### Ranked separately: apply to the Claude Marketplace partner waitlist
|
|
393
|
-
|
|
394
|
-
**Business decision, not an engineering item** - listed apart so it does not
|
|
395
|
-
distort the value-over-effort ordering above.
|
|
396
|
-
|
|
397
|
-
Factory listed on Sep 9 [sourced: https://factory.ai/news/claude-marketplace] and
|
|
398
|
-
the marketplace lets enterprises spend existing Anthropic commitments, which
|
|
399
|
-
removes a procurement cycle. Loki already ships the required artifacts
|
|
400
|
-
`[measured]`: `.claude-plugin/marketplace.json` and
|
|
401
|
-
`plugins/loki-mode/.claude-plugin/plugin.json` at 9.50.1. Entry is via a partner
|
|
402
|
-
waitlist [sourced: https://claude.com/platform/marketplace], so this is a founder
|
|
403
|
-
call, not a build.
|
|
404
|
-
|
|
405
|
-
### Deliberately NOT ranked (open NEXT-10 items, status unchanged)
|
|
406
|
-
|
|
407
|
-
Re-listed only so this document cannot be read as re-proposing them. See
|
|
408
|
-
`docs/COMPETITIVE-NEXT-10.md` for the evidence:
|
|
409
|
-
|
|
410
|
-
- **NEXT-10 item 5** - `loki init` writes dead config keys. Open.
|
|
411
|
-
- **NEXT-10 item 6** - `llm_review` verdict influence deferred. Open by design.
|
|
412
|
-
- **NEXT-10 item 8** - flaky `test-review-assurance-tail.sh`. Open; deliberately
|
|
413
|
-
not patched.
|
|
414
|
-
- **NEXT-10 item 10** - hard command blocklist. Designed, not built.
|
|
415
|
-
|
|
416
|
-
---
|
|
417
|
-
|
|
418
|
-
## 4. UNVERIFIED
|
|
419
|
-
|
|
420
|
-
Excluded from the ranked list. Listed so the absence is a finding.
|
|
421
|
-
|
|
422
|
-
- **Factory, "Why model routing must be in the harness" (24 Aug 2026).** The news
|
|
423
|
-
index lists this title and date, but three URL attempts 404'd and two searches
|
|
424
|
-
failed to surface the post. Worse, three **conflicting** cost figures circulate
|
|
425
|
-
for Factory routing: "20-25%" (Factory Router, Jun 1, primary source), "58%",
|
|
426
|
-
and "cost per successful run is 80.5% of Opus on Terminal-Bench 2 and 78.0% on
|
|
427
|
-
Legacy-Bench" (both secondary). I did not pick one. The post may exist; I could
|
|
428
|
-
not read it, and no routing figure other than the Jun 1 "20-25%" should be
|
|
429
|
-
quoted.
|
|
430
|
-
- **Claude Fable 5.1 exact release date.** Named in a secondary roundup only; no
|
|
431
|
-
primary Anthropic announcement fetched.
|
|
432
|
-
- **Meta Muse Spark 1.3.** Secondary roundup only. The founder named "Muse"; I
|
|
433
|
-
could not confirm whether this is the same product.
|
|
434
|
-
- **DeepSeek V4 SWE-bench figures.** Several secondary blogs give numbers. No
|
|
435
|
-
primary DeepSeek source fetched, so none are reproduced here.
|
|
436
|
-
- **G2/Capterra/ProductHunt review counts for factory.ai.** g2.com returned 403.
|
|
437
|
-
The listing exists; the counts are second-hand.
|
|
438
|
-
- **Factory's ProgramBench table.** Vendor-claimed, self-run. ProgramBench is an
|
|
439
|
-
academic benchmark (arXiv:2605.03546) but I found no independent leaderboard
|
|
440
|
-
entry confirming Factory's figures (benchmarklist.com returned 403).
|
|
441
|
-
- **"Claude Mythos Preview tops SWE-bench Verified at 93.9%"** surfaced in
|
|
442
|
-
search. Not fetched from a primary source, outside the window, and this repo
|
|
443
|
-
has been burned by exactly this class of number
|
|
444
|
-
(`project-swebench-9967-was-string-counting`). Recorded, not used.
|
|
@@ -1,145 +0,0 @@
|
|
|
1
|
-
# Founder Decisions: trust semantics (2026-08-01)
|
|
2
|
-
|
|
3
|
-
Two questions this session surfaced that I deliberately did not answer, because
|
|
4
|
-
answering them changes what the word "Verified" means on a receipt. Both are
|
|
5
|
-
recorded with the measurement that raised them, so the decision can be made on
|
|
6
|
-
evidence rather than on my summary of it.
|
|
7
|
-
|
|
8
|
-
Everything else from this session shipped. These are the only open items.
|
|
9
|
-
|
|
10
|
-
## Why these are yours and not mine
|
|
11
|
-
|
|
12
|
-
`autonomy/lib/proof-generator.py` already records the rule, in the note on the
|
|
13
|
-
functional-verification fact:
|
|
14
|
-
|
|
15
|
-
> Making functional-satisfaction gate the green headline is a trust-semantics
|
|
16
|
-
> product decision (council + founder), the second half of FV-2. Recording it
|
|
17
|
-
> first lets the signal be seen and validated safely.
|
|
18
|
-
|
|
19
|
-
Record first, gate later. Both items below follow that pattern: the fact is now
|
|
20
|
-
recorded and visible on every surface, and the verdict is unchanged pending your
|
|
21
|
-
call.
|
|
22
|
-
|
|
23
|
-
---
|
|
24
|
-
|
|
25
|
-
## Decision 1: should a disabled trust gate demote "Verified"?
|
|
26
|
-
|
|
27
|
-
**Status today:** the gate is recorded and shown everywhere. The headline is
|
|
28
|
-
unchanged.
|
|
29
|
-
|
|
30
|
-
A run can disable any phase by environment variable. When code review or
|
|
31
|
-
security is switched off, the proof now records it (`quality_gates.disabled_phases`),
|
|
32
|
-
the honesty ledger carries a per-gate entry, both receipts name it, and the
|
|
33
|
-
plain-English owner page refuses to show a green "ready" badge.
|
|
34
|
-
|
|
35
|
-
What it does **not** do is change `honesty.headline`. A receipt with code review
|
|
36
|
-
disabled can still read `VERIFIED`.
|
|
37
|
-
|
|
38
|
-
**The case for demoting it:** "Verified" on a receipt with two correctness gates
|
|
39
|
-
switched off is defensible only if it means "verified as far as it was checked".
|
|
40
|
-
A reader who has not read the gaps list will not make that distinction.
|
|
41
|
-
|
|
42
|
-
**The case against:** the headline is re-derived by `proof-verify` to detect
|
|
43
|
-
tampering. Changing what feeds it changes what an honest proof looks like, and
|
|
44
|
-
older signed receipts would re-derive differently. There is also a real risk of
|
|
45
|
-
teaching users to ignore the headline if it moves for reasons they consider
|
|
46
|
-
routine.
|
|
47
|
-
|
|
48
|
-
**If you decide to demote:** the change is in `_compute_headline`, and the
|
|
49
|
-
disabled-gate entries must then be included in the verifier's re-derivation
|
|
50
|
-
rather than filtered out (`_recorded_degraded_raw`). Both halves must move
|
|
51
|
-
together or every honest receipt is reported as forged, which is exactly the bug
|
|
52
|
-
v8.19.2 fixed.
|
|
53
|
-
|
|
54
|
-
---
|
|
55
|
-
|
|
56
|
-
## Decision 2: is an unrun security scan a gap?
|
|
57
|
-
|
|
58
|
-
**Status today:** deliberately not a gap. There is a test asserting it.
|
|
59
|
-
|
|
60
|
-
Measured on a real generated proof, with a clean build and passing tests:
|
|
61
|
-
|
|
62
|
-
```
|
|
63
|
-
headline : VERIFIED
|
|
64
|
-
gaps listed : []
|
|
65
|
-
security fact : {'ran': False, 'status': 'not_run', 'high_active': 0}
|
|
66
|
-
```
|
|
67
|
-
|
|
68
|
-
So a receipt reads `VERIFIED` with an empty gaps list while the security scan
|
|
69
|
-
never ran at all.
|
|
70
|
-
|
|
71
|
-
Tests and build are both flagged as gaps when they do not run. Security is
|
|
72
|
-
flagged only when it **ran and found** an active HIGH finding. The asymmetry is
|
|
73
|
-
intentional: `tests/test_proof_generator.py::test_no_security_file_is_not_a_gap`
|
|
74
|
-
states it outright, "Absence of a scan is not a security gap (the gate did not
|
|
75
|
-
run)".
|
|
76
|
-
|
|
77
|
-
I prototyped changing it and reverted, because overriding a documented decision
|
|
78
|
-
with a test guarding it is not a call I should make silently.
|
|
79
|
-
|
|
80
|
-
**The case for treating it as a gap:** the honesty ledger exists so "a reader
|
|
81
|
-
sees exactly what was NOT verified rather than inferring it from silence"
|
|
82
|
-
(`_compute_degraded`'s own docstring). An unrun security scan is precisely that,
|
|
83
|
-
and it is the one omission a reader is least likely to notice on their own.
|
|
84
|
-
|
|
85
|
-
**The case against:** most projects have no security scanner configured, so
|
|
86
|
-
every receipt would carry a security gap that means nothing about that
|
|
87
|
-
project's quality. A gap list that is never empty is a gap list nobody reads,
|
|
88
|
-
and that costs more than the omission.
|
|
89
|
-
|
|
90
|
-
**A middle option:** treat it as a gap only when the project has a scanner
|
|
91
|
-
configured and it did not run. That distinguishes "not applicable" from
|
|
92
|
-
"skipped", which is the distinction the current binary answer cannot express.
|
|
93
|
-
|
|
94
|
-
---
|
|
95
|
-
|
|
96
|
-
## Context found after writing the above: it is three tiers, not two
|
|
97
|
-
|
|
98
|
-
Sweeping every fact the proof records showed the exclusion is a deliberate,
|
|
99
|
-
consistent pattern rather than an oversight in one place. Three facts are
|
|
100
|
-
recorded and never enter the honesty ledger on a clean run:
|
|
101
|
-
|
|
102
|
-
| Fact | In the ledger? | Documented as record-half? |
|
|
103
|
-
|---|---|---|
|
|
104
|
-
| `functional` | never | yes, explicitly (FV-2) |
|
|
105
|
-
| `healthcheck` | never | yes, explicitly |
|
|
106
|
-
| `security` | only on an active HIGH finding | no note, but behaves the same when unrun |
|
|
107
|
-
|
|
108
|
-
`functional` and `healthcheck` both carry a note in their collector saying they
|
|
109
|
-
are "NOT read by `_compute_headline` / `_compute_degraded`" and that gating on
|
|
110
|
-
them is founder-gated. `security` carries no such note, yet an unrun scan is
|
|
111
|
-
excluded exactly as they are.
|
|
112
|
-
|
|
113
|
-
That changes decision 2 from "is this an oversight?" to "should the third tier
|
|
114
|
-
join the first two, or move the other way?". It also means answering it in
|
|
115
|
-
isolation would leave the receipt inconsistent: if an unrun security scan
|
|
116
|
-
becomes a gap, a reader will reasonably ask why an unrun healthcheck is not.
|
|
117
|
-
|
|
118
|
-
The cheapest correct step, if you want one, is to give `security` the same
|
|
119
|
-
explicit record-half note the other two have, so the current behaviour is
|
|
120
|
-
documented where a reader will find it rather than inferred from its absence.
|
|
121
|
-
|
|
122
|
-
---
|
|
123
|
-
|
|
124
|
-
## What I would recommend, briefly
|
|
125
|
-
|
|
126
|
-
Revised after the three-tier finding above, which I had not made when I first
|
|
127
|
-
wrote this section.
|
|
128
|
-
|
|
129
|
-
Decision 2 should not be taken in isolation. Answering it for security alone
|
|
130
|
-
makes the receipt internally inconsistent, because a reader who sees an unrun
|
|
131
|
-
security scan flagged will reasonably ask why an unrun healthcheck is not. The
|
|
132
|
-
real question is what the whole record-half tier should do, and that is one
|
|
133
|
-
decision, not three.
|
|
134
|
-
|
|
135
|
-
Decision 1 is the larger change and depends on how you want "Verified" to read
|
|
136
|
-
to someone who did not run the build. It is worth deciding deliberately rather
|
|
137
|
-
than inheriting, and it is independent of decision 2.
|
|
138
|
-
|
|
139
|
-
Neither is blocking. Both facts are already visible to anyone who reads the
|
|
140
|
-
receipt.
|
|
141
|
-
|
|
142
|
-
The one change I made without asking is documentation only: `security` now
|
|
143
|
-
carries the same explicit record-half note that `functional` and `healthcheck`
|
|
144
|
-
have. That records the existing behaviour where a reader will find it. It
|
|
145
|
-
changes no verdict and decides nothing.
|