loki-mode 11.2.2 → 11.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -650
- package/SKILL.md +4 -4
- package/VERSION +1 -1
- package/autonomy/completion-council.sh +2 -2
- package/autonomy/context-tracker.py +4 -4
- package/autonomy/council-v2.sh +1 -1
- package/autonomy/grill.sh +1 -1
- package/autonomy/lib/done-recognition.sh +1 -1
- package/autonomy/lib/prd-enrich.sh +1 -1
- package/autonomy/lib/skill-link-heal.sh +78 -1
- package/autonomy/lib/voter-agents.sh +1 -1
- package/autonomy/loki +99 -75
- package/autonomy/run.sh +30 -18
- package/autonomy/sandbox.sh +1 -1
- package/autonomy/tui.sh +6 -5
- package/autonomy/verify.sh +1 -1
- package/bin/loki +166 -59
- package/completions/_loki +2 -0
- package/completions/loki.bash +2 -2
- package/dashboard/__init__.py +1 -1
- package/dashboard/server.py +25 -31
- package/docs/ARCHITECTURE.md +337 -0
- package/docs/BRANDING.md +124 -0
- package/docs/CLI-REFERENCE.md +7 -1
- package/docs/COMPONENTS.md +447 -0
- package/docs/DEPLOYMENT.md +9 -9
- package/docs/DOCKER.md +469 -0
- package/docs/INSTALLATION.md +67 -1045
- package/docs/KNOWN-ISSUES.md +11 -35
- package/docs/TESTING.md +161 -0
- package/docs/air-gapped.md +1 -2
- package/docs/alternative-installations.md +14 -24
- package/docs/architecture/DECISIONS.md +333 -0
- package/docs/audit-logging.md +7 -7
- package/docs/authentication.md +3 -3
- package/docs/authorization.md +1 -1
- package/docs/certification/04-production-deployment/lab.md +6 -6
- package/docs/certification/04-production-deployment/lesson.md +4 -4
- package/docs/certification/README.md +1 -1
- package/docs/control-plane-container.md +2 -2
- package/docs/dev/real-run.md +34 -0
- package/docs/dev/release-checklist.md +2 -2
- package/docs/exit-codes.md +3 -3
- package/docs/git-workflow.md +3 -3
- package/docs/metrics.md +37 -55
- package/docs/network-security.md +1 -1
- package/docs/siem-integration.md +5 -5
- package/docs/v10/CLI-MODERN.md +4 -3
- package/docs/v10/DECISIONS.md +44 -0
- package/docs/v10/DEPS.md +22 -22
- package/docs/v10/ENGINE.md +1 -1
- package/docs/v10/FAILURE-CLASSES.md +129 -1
- package/docs/v10/FOUNDER-QUEUE.md +1 -0
- package/docs/v10/GUARDS.md +13 -0
- package/docs/v10/LEGACY-REMOVAL.md +4 -6
- package/docs/v10/METRICS.md +2 -2
- package/docs/v10/MODERNIZE.md +1 -1
- package/docs/v10/OPERATING-MODEL.md +2 -1
- package/docs/v10/PROGRESS.md +29 -0
- package/docs/v11/COST-HALF.md +352 -0
- package/docs/v11/MARKET-SLICES.md +315 -0
- package/docs/v11/REGISTRIES.md +730 -0
- package/loki-ts/data/model-pricing.json +52 -24
- package/loki-ts/dist/loki.js +1036 -957
- package/mcp/__init__.py +1 -1
- package/package.json +1 -1
- package/packages/control-plane/dist/server.js +2467 -736
- package/plugins/loki-mode/.claude-plugin/plugin.json +1 -1
- package/src/protocols/mcp-client.js +6 -2
- package/src/protocols/mcp-server.js +13 -3
- package/tools/audit-docs.py +1 -1
- package/web-app/dist/assets/{AdminPage-C-iMLNxe.js → AdminPage-CMkHNJbF.js} +1 -1
- package/web-app/dist/assets/{Avatar-C7VUvwsO.js → Avatar-BJyjV7Zi.js} +1 -1
- package/web-app/dist/assets/{Badge-tGvQEhnw.js → Badge-D572l0uq.js} +1 -1
- package/web-app/dist/assets/{Button-_iHlH_yL.js → Button-T4BnOZgl.js} +1 -1
- package/web-app/dist/assets/{CockpitPage-J_nNjbcR.js → CockpitPage-Cj_GFsNn.js} +1 -1
- package/web-app/dist/assets/{ComparePage-B05ui6Xt.js → ComparePage-A8m4ZXak.js} +1 -1
- package/web-app/dist/assets/{ErrorBoundary-Bxex6DLn.js → ErrorBoundary-hugCg5yg.js} +1 -1
- package/web-app/dist/assets/{EvidenceReceiptPanel-BsgWfdD1.js → EvidenceReceiptPanel-CuJKDC2G.js} +1 -1
- package/web-app/dist/assets/{GitHubIssuesPanel-DZBhZ8_X.js → GitHubIssuesPanel-BYGKjarC.js} +1 -1
- package/web-app/dist/assets/{GitHubPRsPanel-BaxlAl6R.js → GitHubPRsPanel-c7sFIJlf.js} +1 -1
- package/web-app/dist/assets/{HomePage-mVJvzqbF.js → HomePage-A1MIkG2P.js} +2 -2
- package/web-app/dist/assets/{LoginPage-B0zd4A_G.js → LoginPage-8KQMRhtR.js} +1 -1
- package/web-app/dist/assets/{MagicPage-DXLQ-y_R.js → MagicPage-C05QW2Lx.js} +1 -1
- package/web-app/dist/assets/{MetricsPage-CLS34Xvr.js → MetricsPage-Oukv5hme.js} +1 -1
- package/web-app/dist/assets/{NotFoundPage-BXG8PbNV.js → NotFoundPage-DKIwsawz.js} +1 -1
- package/web-app/dist/assets/{ProjectPage-D3kFFC0F.js → ProjectPage-DwFRPLCG.js} +3 -3
- package/web-app/dist/assets/{ProjectsPage-2bFn-gnC.js → ProjectsPage-CgY4YYDX.js} +1 -1
- package/web-app/dist/assets/{SettingsPage-CBM0ziWE.js → SettingsPage-BpfcSjUA.js} +1 -1
- package/web-app/dist/assets/{ShowcasePage-BTaXhjI3.js → ShowcasePage-BqBakC4K.js} +1 -1
- package/web-app/dist/assets/{SystemSettingsPage-BrC2hyAr.js → SystemSettingsPage-CZLu5VB0.js} +1 -1
- package/web-app/dist/assets/{TeamsPage-oqxdbVbY.js → TeamsPage-D55S1jKz.js} +1 -1
- package/web-app/dist/assets/{TemplatesPage-CEbUIY21.js → TemplatesPage-D3SydvxN.js} +1 -1
- package/web-app/dist/assets/{TerminalOutput-aIoWLbJA.js → TerminalOutput-hNg8G1-e.js} +1 -1
- package/web-app/dist/assets/{activity-CPyi45gt.js → activity-D2C0VagO.js} +1 -1
- package/web-app/dist/assets/{bell-BhaNBaJp.js → bell-CMvYi36w.js} +1 -1
- package/web-app/dist/assets/{bot-Da8REaEP.js → bot-BvtQ6lmo.js} +1 -1
- package/web-app/dist/assets/{check-DPj99ArM.js → check-B13xkXed.js} +1 -1
- package/web-app/dist/assets/{chevron-left-CkxonN0X.js → chevron-left-emu6r49E.js} +1 -1
- package/web-app/dist/assets/{circle-alert-C0JkB1y5.js → circle-alert-CExlwjuy.js} +1 -1
- package/web-app/dist/assets/{clock-COzFIiWi.js → clock-D7Uas3oL.js} +1 -1
- package/web-app/dist/assets/{cloud-bjkac4kn.js → cloud-CkB4S86C.js} +1 -1
- package/web-app/dist/assets/{code-xml-hlKlvXUJ.js → code-xml-SL9IIWIn.js} +1 -1
- package/web-app/dist/assets/{database-D555S5Km.js → database-BQFkd7t1.js} +1 -1
- package/web-app/dist/assets/{dollar-sign-CVcApzcD.js → dollar-sign-qHmnNYsU.js} +1 -1
- package/web-app/dist/assets/{file-code-corner-UORaMIcO.js → file-code-corner-C2YUDP0K.js} +1 -1
- package/web-app/dist/assets/{file-plus-ALJtDoI2.js → file-plus-C_ooIkDe.js} +1 -1
- package/web-app/dist/assets/{globe-CPlrM_IC.js → globe-CQ-GL485.js} +1 -1
- package/web-app/dist/assets/{hammer-qaGZHpja.js → hammer-U05L_D7V.js} +1 -1
- package/web-app/dist/assets/{index-BcXljIAO.js → index-CooHd7I-.js} +3 -3
- package/web-app/dist/assets/{layers-BV8R9VUK.js → layers-Buo17rjW.js} +1 -1
- package/web-app/dist/assets/{loader-circle-6TlDS9Tu.js → loader-circle-15vK3oV1.js} +1 -1
- package/web-app/dist/assets/{lock-B9t6oWI0.js → lock-BeXLebuJ.js} +1 -1
- package/web-app/dist/assets/{package-2gEUAtZJ.js → package-MVh_6-6R.js} +1 -1
- package/web-app/dist/assets/{plus-DXocxPjU.js → plus-Dh_nd9vS.js} +1 -1
- package/web-app/dist/assets/{refresh-cw-D3dWHBd6.js → refresh-cw-C9cBJEAT.js} +1 -1
- package/web-app/dist/assets/{rotate-ccw-CvuWqaSB.js → rotate-ccw-D3PAtzST.js} +1 -1
- package/web-app/dist/assets/{scroll-text-ktmD_xJY.js → scroll-text-DZCoXQjo.js} +1 -1
- package/web-app/dist/assets/{server-E7pZtj5A.js → server-OCOLI5J_.js} +1 -1
- package/web-app/dist/assets/{shield-alert-968j9z0V.js → shield-alert-oVhmf1_h.js} +1 -1
- package/web-app/dist/assets/{thumbs-up-Tz_8rpU_.js → thumbs-up-B-kbX4vN.js} +1 -1
- package/web-app/dist/assets/{trash-2-Drh9IIj8.js → trash-2-EL6moveW.js} +1 -1
- package/web-app/dist/assets/{trending-down-DCQXtVSL.js → trending-down-Cl3kjpsx.js} +1 -1
- package/web-app/dist/assets/{trending-up-BIiwQbWV.js → trending-up-DST7I6M2.js} +1 -1
- package/web-app/dist/assets/{upload-sCbPkfmy.js → upload-CUOT780G.js} +1 -1
- package/web-app/dist/assets/{usePolling-BMIg6ZVT.js → usePolling-Djmt90eb.js} +1 -1
- package/web-app/dist/assets/{user-kGzXC3V2.js → user-BRBohA6J.js} +1 -1
- package/web-app/dist/index.html +1 -1
- package/docs/ARCHITECTURE-OVERVIEW.md +0 -209
- package/docs/AUTONOMI-ECOSYSTEM.md +0 -107
- package/docs/COCKPIT-SPEC.md +0 -72
- package/docs/COCKPIT-UX-MANDATE.md +0 -53
- package/docs/COMPETITIVE-ANALYSIS.md +0 -303
- package/docs/COMPETITIVE-NEXT-10.md +0 -316
- package/docs/COMPETITIVE-RALPH-ZEROSHOT-2026-07-08.md +0 -81
- package/docs/DASHBOARD-9.12-EVIDENCE.md +0 -97
- package/docs/DASHBOARD-ARCHITECTURE.md +0 -425
- package/docs/DASHBOARD_V2_CHECKLIST.md +0 -243
- package/docs/DEMOS.md +0 -63
- package/docs/ENTERPRISE-IDENTITY-ROADMAP.md +0 -206
- package/docs/EVIDENCE-RECEIPT-BUILD-APPLICABILITY.md +0 -79
- package/docs/FEAT-PRDREUSE-DOCKER-PLAN.md +0 -144
- package/docs/GATE-FAILURE-TRIAGE.md +0 -254
- package/docs/HANDOFF-2026-08-03.md +0 -439
- package/docs/LOOP-CANDIDATE-PROPOSAL-v1.md +0 -167
- package/docs/LOOP-HARNESS-AUDIT.md +0 -684
- package/docs/MERGE-DEDUP-MAP.md +0 -420
- package/docs/MERGE-ROUTE-MAP.md +0 -139
- package/docs/MODEL-REGISTRY-PLAN.md +0 -155
- package/docs/OUTCOME-FRONTIER.md +0 -536
- package/docs/P2-SPEC-ROBUSTNESS-PLAN.md +0 -192
- package/docs/PLANS-INDEX.md +0 -32
- package/docs/PRD-REUSE-DONE-RECOGNITION-PLAN.md +0 -536
- package/docs/PROMPT-ABLATION-RESULT.md +0 -97
- package/docs/R4-TRUST-TRAJECTORY-DESIGN.md +0 -127
- package/docs/R5-AUTO-WIKI-DESIGN.md +0 -137
- package/docs/R6-ROLLBACK-CHECKPOINT-PLAN.md +0 -107
- package/docs/R7-ZERO-CONFIG-FIRST-RUN-PLAN.md +0 -137
- package/docs/R8-SHAREABLE-TEAM-ASSETS-PLAN.md +0 -129
- package/docs/R9-OPEN-CORE-HOOKS-PLAN.md +0 -113
- package/docs/RARV-C-100X-PLAN.md +0 -286
- package/docs/RARV-C-LOOP-EFFICIENCY-PLAN.md +0 -470
- package/docs/SHAREABLE-PROOF-PLAN.md +0 -194
- package/docs/SONNET5-DEFAULT-PLAN.md +0 -157
- package/docs/SPEED-AND-FIRST-PASS-PLAN.md +0 -281
- package/docs/STALE-STATE-AUDIT.md +0 -174
- package/docs/SYNERGY-TASKS.md +0 -143
- package/docs/TASKLIST-ACCURACY-PLAN.md +0 -355
- package/docs/TOP-100-BACKLOG.md +0 -159
- package/docs/V8-ACCEPTANCE-TRIAGE-2026-07-24.md +0 -114
- package/docs/V8-COMPLEXITY-AUDIT-2026-07-24.md +0 -45
- package/docs/V8-MAJOR-RELEASE-PLAN.md +0 -137
- package/docs/V8-OVERNIGHT-PLAN-2026-07-25.md +0 -82
- package/docs/V8-SDK-RESEARCH-RAW.md +0 -129
- package/docs/WELCOME-OPENER-PLAN.md +0 -106
- package/docs/adoption-baseline-2026-07-31.md +0 -84
- package/docs/architecture/bmad-integration-epic.md +0 -271
- package/docs/architecture/bmad-integration-review.md +0 -86
- package/docs/architecture/bmad-integration-validation.md +0 -249
- package/docs/architecture/bmad-loki-voice-agent-council-analysis.md +0 -61
- package/docs/auto-claude-comparison.md +0 -250
- package/docs/bug-fixes/agent-01-cli-fixes.md +0 -101
- package/docs/bug-fixes/agent-02-purplelab-fixes.md +0 -88
- package/docs/bug-fixes/agent-03-dashboard-fixes.md +0 -119
- package/docs/bug-fixes/agent-04-memory-fixes.md +0 -105
- package/docs/bug-fixes/agent-05-provider-fixes.md +0 -86
- package/docs/bug-fixes/agent-06-integration-fixes.md +0 -101
- package/docs/bug-fixes/agent-07-dash-run-fixes.md +0 -101
- package/docs/bug-fixes/agent-08-docker-fixes.md +0 -164
- package/docs/bug-fixes/agent-09-e2e-build-fixes.md +0 -69
- package/docs/bug-fixes/agent-10-e2e-fullstack-fixes.md +0 -102
- package/docs/bug-fixes/agent-11-e2e-session-fixes.md +0 -70
- package/docs/bug-fixes/agent-12-scenario-fixes.md +0 -120
- package/docs/bug-fixes/agent-13-enterprise-fixes.md +0 -143
- package/docs/bug-fixes/agent-14-uat-newuser-fixes.md +0 -88
- package/docs/bug-fixes/agent-15-uat-poweruser-fixes.md +0 -132
- package/docs/bug-fixes/agent-19-code-review.md +0 -316
- package/docs/bug-fixes/agent-20-architecture-review.md +0 -331
- package/docs/competitive/bolt-new-analysis.md +0 -579
- package/docs/competitive/emergence-others-analysis.md +0 -605
- package/docs/competitive/replit-lovable-analysis.md +0 -622
- package/docs/dashboard-guide.md +0 -487
- package/docs/enterprise/migration.md +0 -265
- package/docs/history/one-run-audit.md +0 -304
- package/docs/plans/FORGE-AUTONOMOUS-QUEUE.md +0 -312
- package/docs/plans/ULTRAPLAN-FORGE-BAAS.md +0 -612
- package/docs/plans/UT2-6-LSP-DIAGNOSTIC-BROADCAST.md +0 -394
- package/docs/prd-purple-lab-ide.md +0 -152
- package/docs/prd-purple-lab-platform-v2.md +0 -223
- package/docs/prd-purple-lab-platform.md +0 -319
- package/docs/retrospectives/v7.5.15-fleet-postmortem.md +0 -325
- package/docs/retrospectives/v7.5.15-honesty-audit.md +0 -136
- package/docs/retrospectives/v7.5.15-llm-failure-modes.md +0 -49
- package/docs/show-hn-post.md +0 -46
- package/docs/test-scenarios/enterprise-scenarios.md +0 -732
- package/docs/thick2thin.md +0 -173
|
@@ -1,303 +0,0 @@
|
|
|
1
|
-
# Loki Mode Competitive Analysis
|
|
2
|
-
|
|
3
|
-
*Last Updated: 2026-02-06*
|
|
4
|
-
|
|
5
|
-
## Executive Summary
|
|
6
|
-
|
|
7
|
-
Loki Mode has **unique differentiation** in autonomous SDLC orchestration, quality assurance (Completion Council), and memory systems. The most direct competitor is **get-shit-done** (11.9K stars, 21K npm/wk) which solves context management with simpler architecture. Loki Mode is technically deeper but faces significant adoption gaps.
|
|
8
|
-
|
|
9
|
-
---
|
|
10
|
-
|
|
11
|
-
## Direct Competitor: get-shit-done (GSD)
|
|
12
|
-
|
|
13
|
-
**Repo:** [github.com/glittercowboy/get-shit-done](https://github.com/glittercowboy/get-shit-done) | **Stars:** 11,903 | **npm:** 21.4K/wk
|
|
14
|
-
|
|
15
|
-
GSD is the closest competitor -- a context engineering system that spawns fresh subagent contexts per task. It installs as slash commands into Claude Code, OpenCode, or Gemini CLI.
|
|
16
|
-
|
|
17
|
-
| Dimension | Loki Mode | GSD | Winner |
|
|
18
|
-
|-----------|-----------|-----|--------|
|
|
19
|
-
| Feature depth | Deep (memory, council, dashboard, events, agent roles) | Shallow (6 commands, 11 agents, markdown state) | Loki |
|
|
20
|
-
| Adoption | 594 stars, 6K/wk npm | 11,903 stars, 21K/wk npm | GSD (20x) |
|
|
21
|
-
| Simplicity | Complex (5.4K-line run.sh, 12 Python modules) | Simple (markdown agents + slash commands) | GSD |
|
|
22
|
-
| Full autonomy | Walk away, come back to deployed product | Human checkpoints at discuss/verify/milestone | Loki |
|
|
23
|
-
| Quality gates | 8-gate + Completion Council + anti-sycophancy | User verification only | Loki |
|
|
24
|
-
| Memory system | Episodic/semantic/procedural + vector search | None | Loki |
|
|
25
|
-
| Context management | Standard | Fresh subagent contexts per task (core innovation) | GSD |
|
|
26
|
-
| Time to value | Learn architecture, understand CLI flags | `npx get-shit-done-cc` and go | GSD |
|
|
27
|
-
| Multi-provider | Orchestrates across Claude/Cline/Codex/Aider/opencode with degraded mode | Installs separately to each | Loki |
|
|
28
|
-
| Real-world proof | No public case studies | Testimonials, 1.1K forks, 75 PRs | GSD |
|
|
29
|
-
|
|
30
|
-
**Key takeaway:** GSD proved the market wants sharp blades, not Swiss Army knives. Loki Mode's Completion Council and quality gates are genuinely unique IP that no competitor has.
|
|
31
|
-
|
|
32
|
-
---
|
|
33
|
-
|
|
34
|
-
## Factual Comparison Table
|
|
35
|
-
|
|
36
|
-
| Feature | Loki Mode | Claude-Flow | MetaGPT | CrewAI | Cursor Agent | Devin |
|
|
37
|
-
|---------|-----------|-------------|---------|--------|--------------|-------|
|
|
38
|
-
| **GitHub Stars** | 594 | 13,700 | 62,400 | 25,000+ | N/A (Commercial) | N/A (Commercial) |
|
|
39
|
-
| **Parallel Execution** | Yes (multi-agent) | Yes (swarms) | Sequential | Yes (crews) | Yes (8 worktrees) | Yes (fleet) |
|
|
40
|
-
| **Published Benchmarks** | None | None | 85.9-87.7% HumanEval | None | ~250 tok/s | 15% complex tasks |
|
|
41
|
-
| **SWE-bench Score** | Not measured (patch generation harness exists; official evaluator not run, so no resolve rate exists) | Unknown | Unknown | Unknown | Unknown | 15% complex |
|
|
42
|
-
| **Full SDLC** | Yes (8 phases) | Yes | Partial | Partial | No | Partial |
|
|
43
|
-
| **Business Ops** | **Yes** | No | No | No | No | No |
|
|
44
|
-
| **Enterprise Security** | `--dangerously-skip-permissions` | MCP sandboxed | Sandboxed | Audit logs, RBAC | Staged autonomy | Sandboxed |
|
|
45
|
-
| **Cross-Project Learning** | No | AgentDB | No | No | No | Limited |
|
|
46
|
-
| **Observability** | Dashboard + STATUS.txt | Real-time tracing | Logs | Full tracing | Built-in | Full |
|
|
47
|
-
| **Pricing** | Free (source-available) | Free (OSS) | Free (OSS) | $25+/mo | $20-400/mo | $20-500/mo |
|
|
48
|
-
| **Production Ready** | Experimental | Production | Production | Production | Production | Production |
|
|
49
|
-
| **Resource Monitoring** | Yes (v2.18.5) | Unknown | No | No | No | No |
|
|
50
|
-
| **State Recovery** | Yes (checkpoints) | Yes (AgentDB) | Limited | Yes | Git worktrees | Yes |
|
|
51
|
-
| **Self-Verification** | Yes (RARV) | Unknown | Yes (SOP) | No | YOLO mode | Yes |
|
|
52
|
-
|
|
53
|
-
---
|
|
54
|
-
|
|
55
|
-
## Detailed Competitor Analysis
|
|
56
|
-
|
|
57
|
-
### Claude-Flow (10.7K Stars)
|
|
58
|
-
**Repository:** [ruvnet/claude-flow](https://github.com/ruvnet/claude-flow)
|
|
59
|
-
|
|
60
|
-
**Strengths:**
|
|
61
|
-
- 64+ agent system with hive-mind coordination
|
|
62
|
-
- AgentDB v1.3.9 with 96x-164x faster vector search
|
|
63
|
-
- 25 Claude Skills with natural language activation
|
|
64
|
-
- 100 MCP Tools for swarm orchestration
|
|
65
|
-
- Built on official Claude Agent SDK (v2.5.0)
|
|
66
|
-
- 50-100x speedup from in-process MCP + 10-20x from parallel spawning
|
|
67
|
-
- Enterprise features: compliance, scalability, Agile support
|
|
68
|
-
|
|
69
|
-
**Weaknesses:**
|
|
70
|
-
- No business operations automation
|
|
71
|
-
- Complex setup compared to single-skill approach
|
|
72
|
-
- Heavy infrastructure requirements
|
|
73
|
-
|
|
74
|
-
**What Loki Mode Can Learn:**
|
|
75
|
-
- AgentDB-style persistent memory across projects
|
|
76
|
-
- MCP protocol integration for tool orchestration
|
|
77
|
-
- Enterprise CLAUDE.MD templates (Agile, Enterprise, Compliance)
|
|
78
|
-
|
|
79
|
-
---
|
|
80
|
-
|
|
81
|
-
### MetaGPT (62.4K Stars)
|
|
82
|
-
**Repository:** [FoundationAgents/MetaGPT](https://github.com/FoundationAgents/MetaGPT)
|
|
83
|
-
**Paper:** ICLR 2024 Oral (Top 1.8%)
|
|
84
|
-
|
|
85
|
-
**Strengths:**
|
|
86
|
-
- 85.9-87.7% Pass@1 on HumanEval
|
|
87
|
-
- High task completion rate in evaluations (100% reported by MetaGPT authors; not independently verified)
|
|
88
|
-
- Standard Operating Procedures (SOPs) reduce hallucinations
|
|
89
|
-
- Assembly line paradigm with role specialization
|
|
90
|
-
- Low cost: ~$1.09 per project completion
|
|
91
|
-
- Academic validation and peer review
|
|
92
|
-
|
|
93
|
-
**Weaknesses:**
|
|
94
|
-
- Sequential execution (not massively parallel)
|
|
95
|
-
- Python-focused benchmarks
|
|
96
|
-
- No real-time monitoring/dashboard
|
|
97
|
-
- No business operations
|
|
98
|
-
|
|
99
|
-
**What Loki Mode Can Learn:**
|
|
100
|
-
- SOP encoding into prompts (reduces cascading errors)
|
|
101
|
-
- Benchmark methodology for HumanEval/SWE-bench
|
|
102
|
-
- Token cost tracking per task
|
|
103
|
-
|
|
104
|
-
---
|
|
105
|
-
|
|
106
|
-
### CrewAI (25K+ Stars, $18M Raised)
|
|
107
|
-
**Repository:** [crewAIInc/crewAI](https://github.com/crewAIInc/crewAI)
|
|
108
|
-
|
|
109
|
-
**Strengths:**
|
|
110
|
-
- 5.76x faster than LangGraph
|
|
111
|
-
- 1.4 billion agentic automations orchestrated
|
|
112
|
-
- 100,000+ certified developers
|
|
113
|
-
- Enterprise customers: PwC, IBM, Capgemini, NVIDIA
|
|
114
|
-
- Full observability with tracing
|
|
115
|
-
- On-premise deployment options
|
|
116
|
-
- Audit logs and access controls
|
|
117
|
-
|
|
118
|
-
**Weaknesses:**
|
|
119
|
-
- Not Claude-specific (model agnostic)
|
|
120
|
-
- Scaling requires careful resource management
|
|
121
|
-
- Enterprise features require paid tier
|
|
122
|
-
|
|
123
|
-
**What Loki Mode Can Learn:**
|
|
124
|
-
- Flows architecture for production deployments
|
|
125
|
-
- Tracing and observability patterns
|
|
126
|
-
- Enterprise security features (audit logs, RBAC)
|
|
127
|
-
|
|
128
|
-
---
|
|
129
|
-
|
|
130
|
-
### Cursor Agent Mode (Commercial, $29B Valuation)
|
|
131
|
-
**Website:** [cursor.com](https://cursor.com)
|
|
132
|
-
|
|
133
|
-
**Strengths:**
|
|
134
|
-
- Up to 8 parallel agents via git worktrees
|
|
135
|
-
- Composer model: ~250 tokens/second
|
|
136
|
-
- YOLO mode for auto-applying changes
|
|
137
|
-
- `.cursor/rules` for agent constraints
|
|
138
|
-
- Staged autonomy with plan approval
|
|
139
|
-
- Massive enterprise adoption
|
|
140
|
-
|
|
141
|
-
**Weaknesses:**
|
|
142
|
-
- Commercial product ($20-400/month)
|
|
143
|
-
- IDE-locked (VS Code fork)
|
|
144
|
-
- No full SDLC (code editing focus)
|
|
145
|
-
- No business operations
|
|
146
|
-
|
|
147
|
-
**What Loki Mode Can Learn:**
|
|
148
|
-
- `.cursor/rules` equivalent for agent constraints
|
|
149
|
-
- Staged autonomy patterns
|
|
150
|
-
- Git worktree isolation for parallel work
|
|
151
|
-
|
|
152
|
-
---
|
|
153
|
-
|
|
154
|
-
### Devin AI (Commercial, $10.2B Valuation)
|
|
155
|
-
**Website:** [cognition.ai](https://cognition.ai)
|
|
156
|
-
|
|
157
|
-
**Strengths:**
|
|
158
|
-
- 25% of Cognition's own PRs generated by Devin
|
|
159
|
-
- 4x faster, 2x more efficient than previous year
|
|
160
|
-
- 67% PR merge rate (up from 34%)
|
|
161
|
-
- Enterprise adoption: Goldman Sachs pilot
|
|
162
|
-
- Excellent at migrations (SAS->PySpark, COBOL, Angular->React)
|
|
163
|
-
|
|
164
|
-
**Weaknesses:**
|
|
165
|
-
- Only 15% success rate on complex autonomous tasks
|
|
166
|
-
- Gets stuck on ambiguous requirements
|
|
167
|
-
- Requires clear upfront specifications
|
|
168
|
-
- $20-500/month pricing
|
|
169
|
-
|
|
170
|
-
**What Loki Mode Can Learn:**
|
|
171
|
-
- Fleet parallelization for repetitive tasks
|
|
172
|
-
- Migration-specific agent capabilities
|
|
173
|
-
- PR merge tracking as success metric
|
|
174
|
-
|
|
175
|
-
---
|
|
176
|
-
|
|
177
|
-
## Benchmark Results
|
|
178
|
-
|
|
179
|
-
No benchmark result is published. A self-reported HumanEval run from
|
|
180
|
-
2026-01-05 is withdrawn as a headline figure.
|
|
181
|
-
|
|
182
|
-
### SWE-bench Lite: Honest Status
|
|
183
|
-
|
|
184
|
-
**Loki Mode has NO SWE-bench score.** What exists: a harness that GENERATED candidate
|
|
185
|
-
patches for 299 of 300 SWE-bench Lite problems (Claude Opus 4.5; single-agent run
|
|
186
|
-
6.17h, 4-agent RARV pipeline 3.5h). Patch GENERATION only means a diff was produced.
|
|
187
|
-
It does NOT mean the patch fixes the issue. The official SWE-bench evaluator (apply
|
|
188
|
-
patch, run the repo's test suite) was never run, so there is no resolve rate, and
|
|
189
|
-
generation rates must not be compared against other tools' task-resolution rates
|
|
190
|
-
(e.g. Devin's independently-tested task success). For context, frontier model resolve
|
|
191
|
-
rates on SWE-bench Verified are in the ~85-90 percent range; any higher number from
|
|
192
|
-
any tool should be treated with suspicion, including ours.
|
|
193
|
-
|
|
194
|
-
**What a fair claim looks like:** "A reproducible patch-generation harness exists;
|
|
195
|
-
resolve rate is not yet measured." Nothing stronger.
|
|
196
|
-
|
|
197
|
-
---
|
|
198
|
-
|
|
199
|
-
## Critical Gaps to Address
|
|
200
|
-
|
|
201
|
-
### Priority 1: Benchmarks (PARTIAL)
|
|
202
|
-
- **NOT real:** SWE-bench. Patch generation ran (299/300 diffs produced); the official
|
|
203
|
-
evaluator never ran, so there is NO resolve rate and no comparable score.
|
|
204
|
-
- **Next:** run a contamination-resistant evaluator end-to-end (or publish nothing for SWE-bench)
|
|
205
|
-
|
|
206
|
-
### Priority 2: Security Model (Critical for Enterprise)
|
|
207
|
-
- **Gap:** Relies on `--dangerously-skip-permissions`
|
|
208
|
-
- **Impact:** Enterprise adoption blocked
|
|
209
|
-
- **Solution:** Implement sandbox mode, staged autonomy, audit logs
|
|
210
|
-
|
|
211
|
-
### Priority 3: Cross-Project Learning (IMPLEMENTED v5.15.0)
|
|
212
|
-
- **Gap:** ~~Each project starts fresh; no accumulated knowledge~~ RESOLVED
|
|
213
|
-
- **Result:** 3-tier memory system (episodic/semantic/procedural) with vector search
|
|
214
|
-
- **Next:** Improve consolidation pipeline, add cross-project retrieval benchmarks
|
|
215
|
-
|
|
216
|
-
### Priority 4: Observability (IMPLEMENTED v5.23.0)
|
|
217
|
-
- **Gap:** ~~Basic dashboard, no tracing~~ PARTIALLY RESOLVED
|
|
218
|
-
- **Result:** Dark-themed web dashboard with real-time monitoring, task board, Completion Council views
|
|
219
|
-
- **Next:** OpenTelemetry tracing, agent lineage visualization
|
|
220
|
-
|
|
221
|
-
### Priority 5: Community/Documentation (IN PROGRESS)
|
|
222
|
-
- **Gap:** 594 stars vs. 11K-60K for competitors
|
|
223
|
-
- **Impact:** Limited trust and contribution
|
|
224
|
-
- **Done:** CONTRIBUTING.md, issue templates, PR template, CODE_OF_CONDUCT, dev setup docs
|
|
225
|
-
- **Next:** Video tutorials, case studies, example PRDs, reduce onboarding friction
|
|
226
|
-
|
|
227
|
-
---
|
|
228
|
-
|
|
229
|
-
## Loki Mode's Unique Advantages
|
|
230
|
-
|
|
231
|
-
### 1. Business Operations Automation (No Competitor Has This)
|
|
232
|
-
- Marketing agents (campaigns, content, SEO)
|
|
233
|
-
- Sales agents (outreach, CRM, pipeline)
|
|
234
|
-
- Finance agents (budgets, forecasts, reporting)
|
|
235
|
-
- Legal agents (contracts, compliance, IP)
|
|
236
|
-
- HR agents (hiring, onboarding, culture)
|
|
237
|
-
- Investor relations agents (pitch decks, updates)
|
|
238
|
-
- Partnership agents (integrations, BD)
|
|
239
|
-
|
|
240
|
-
### 2. Full Startup Simulation
|
|
241
|
-
- PRD -> Research -> Architecture -> Development -> QA -> Deploy -> Marketing -> Revenue
|
|
242
|
-
- Complete lifecycle, not just coding
|
|
243
|
-
|
|
244
|
-
### 3. RARV Self-Verification Loop
|
|
245
|
-
- Reason-Act-Reflect-Verify cycle
|
|
246
|
-
- 2-3x quality improvement through self-correction
|
|
247
|
-
- Mistakes & Learnings tracking
|
|
248
|
-
|
|
249
|
-
### 4. Resource Monitoring (v2.18.5)
|
|
250
|
-
- Prevents system overload from too many agents
|
|
251
|
-
- Self-throttling based on CPU/memory
|
|
252
|
-
- No competitor has this built-in
|
|
253
|
-
|
|
254
|
-
---
|
|
255
|
-
|
|
256
|
-
## Improvement Roadmap
|
|
257
|
-
|
|
258
|
-
### Phase 1: Credibility (Week 1-2)
|
|
259
|
-
1. Run HumanEval benchmark, publish results
|
|
260
|
-
2. Run SWE-bench Lite, publish results
|
|
261
|
-
3. Add benchmark badge to README
|
|
262
|
-
4. Create benchmark runner script
|
|
263
|
-
|
|
264
|
-
### Phase 2: Security (Week 2-3)
|
|
265
|
-
1. Implement sandbox mode (containerized execution)
|
|
266
|
-
2. Add staged autonomy (plan approval before execution)
|
|
267
|
-
3. Implement audit logging
|
|
268
|
-
4. Create reduced-permissions mode
|
|
269
|
-
|
|
270
|
-
### Phase 3: Learning System (Week 3-4)
|
|
271
|
-
1. Implement `.loki/learnings/` knowledge base
|
|
272
|
-
2. Cross-project pattern extraction
|
|
273
|
-
3. Mistake avoidance database
|
|
274
|
-
4. Success pattern library
|
|
275
|
-
|
|
276
|
-
### Phase 4: Observability (Week 4-5)
|
|
277
|
-
1. OpenTelemetry integration
|
|
278
|
-
2. Agent lineage visualization
|
|
279
|
-
3. Token cost tracking
|
|
280
|
-
4. Performance metrics dashboard
|
|
281
|
-
|
|
282
|
-
### Phase 5: Community (Ongoing)
|
|
283
|
-
1. Video tutorials
|
|
284
|
-
2. More example PRDs
|
|
285
|
-
3. Case study documentation
|
|
286
|
-
4. Integration guides (Vibe Kanban, etc.)
|
|
287
|
-
|
|
288
|
-
---
|
|
289
|
-
|
|
290
|
-
## Sources
|
|
291
|
-
|
|
292
|
-
- [Claude-Flow GitHub](https://github.com/ruvnet/claude-flow)
|
|
293
|
-
- [MetaGPT GitHub](https://github.com/FoundationAgents/MetaGPT)
|
|
294
|
-
- [MetaGPT Paper (ICLR 2024)](https://openreview.net/forum?id=VtmBAGCN7o)
|
|
295
|
-
- [CrewAI GitHub](https://github.com/crewAIInc/crewAI)
|
|
296
|
-
- [CrewAI Framework 2025 Review](https://latenode.com/blog/ai-frameworks-technical-infrastructure/crewai-framework/crewai-framework-2025-complete-review-of-the-open-source-multi-agent-ai-platform)
|
|
297
|
-
- [Cursor AI Review 2025](https://skywork.ai/blog/cursor-ai-review-2025-agent-refactors-privacy/)
|
|
298
|
-
- [Cursor 2.0 Features](https://cursor.com/changelog/2-0)
|
|
299
|
-
- [Devin 2025 Performance Review](https://cognition.ai/blog/devin-annual-performance-review-2025)
|
|
300
|
-
- [Devin AI Real Tests](https://trickle.so/blog/devin-ai-review)
|
|
301
|
-
- [SWE-bench Verified Leaderboard](https://llm-stats.com/benchmarks/swe-bench-verified)
|
|
302
|
-
- [SWE-bench Official](https://www.swebench.com/)
|
|
303
|
-
- [Claude Code Best Practices](https://www.anthropic.com/engineering/claude-code-best-practices)
|
|
@@ -1,316 +0,0 @@
|
|
|
1
|
-
# The next items vs Factory.ai and 8090
|
|
2
|
-
|
|
3
|
-
Written 2026-09-09. Every item below cites a file:line, a command output, or a
|
|
4
|
-
fetched URL. Items that research proposed but that turned out to be **already
|
|
5
|
-
shipped** are listed at the bottom under "Not items" with the evidence, because
|
|
6
|
-
a plan that re-builds working code is worse than a shorter plan.
|
|
7
|
-
|
|
8
|
-
There are **eight** real items, not ten. Three of the research's candidates were
|
|
9
|
-
already implemented, and two more are architecturally unavailable to Loki as
|
|
10
|
-
designed. Padding to ten would mean inventing work.
|
|
11
|
-
|
|
12
|
-
Items 1, 2 and 7 shipped in v9.27.0-v9.27.3. Items 3, 4 and 9 shipped in
|
|
13
|
-
v9.28.0. Item 8 (a flaky trust suite) and item 10 (a hard command blocklist,
|
|
14
|
-
designed but not built) are OPEN with their reasons stated, rather than quietly
|
|
15
|
-
closed by loosening an assertion or shipping a claim we cannot keep.
|
|
16
|
-
|
|
17
|
-
---
|
|
18
|
-
|
|
19
|
-
## What the market actually rewards
|
|
20
|
-
|
|
21
|
-
Both competitors lead with **governance, not speed**. Factory.ai ($150M Series C,
|
|
22
|
-
Apr 2026) sells fleets of droids under "engineers governing how much autonomy
|
|
23
|
-
each workflow receives". 8090 ($135M Series A, EY partnership) sells a "governed
|
|
24
|
-
multiplayer platform under human-led oversight".
|
|
25
|
-
|
|
26
|
-
Loki's wedge is the same shape and sharper: **the only agent that hands you a
|
|
27
|
-
receipt you can check yourself.** The items below are ranked by how much each
|
|
28
|
-
one strengthens a claim a buyer can verify without trusting us.
|
|
29
|
-
|
|
30
|
-
---
|
|
31
|
-
|
|
32
|
-
## 1. Model provenance on the receipt -- SHIPPED v9.27.0
|
|
33
|
-
|
|
34
|
-
**Corrected.** An earlier draft of this item claimed the exogenous/advisory gate
|
|
35
|
-
split, files-changed and cost "do not reach the top of the receipt". That was
|
|
36
|
-
wrong, and reading the renderers disproved it: `proof-template.html:1165-1192`
|
|
37
|
-
already renders the split with the strongest honesty framing in the repo
|
|
38
|
-
("a model that is confidently wrong scores itself green here"), plus
|
|
39
|
-
disabled-gate disclosure, cost, tokens, wall clock and files changed.
|
|
40
|
-
|
|
41
|
-
Five of the six fields were already shipped. **One was genuinely missing:**
|
|
42
|
-
per-iteration model attribution. `autonomy/lib/decision_record.py` writes an
|
|
43
|
-
append-only trail to `.loki/decisions/decisions.jsonl` once per dispatch, and
|
|
44
|
-
already computes the audit fact worth showing -- `model_changed`, meaning more
|
|
45
|
-
than one model id served one project. The proof generator read that file
|
|
46
|
-
**zero times**.
|
|
47
|
-
|
|
48
|
-
That is the question a regulated buyer actually asks, and the one Factory users
|
|
49
|
-
complain about: did the deciding component change mid-run without anyone saying
|
|
50
|
-
so? The run-level "Model" row cannot answer it.
|
|
51
|
-
|
|
52
|
-
**Shipped in v9.27.0** in both renderers, with three states never collapsed --
|
|
53
|
-
`measured`, `no_records`, `unreadable`. An absent trail is reported as absent
|
|
54
|
-
rather than rendered as silence that reads like "no swap happened", corrupt
|
|
55
|
-
trail lines are counted rather than dropped, and `affects_verdict` is `false`:
|
|
56
|
-
a mid-flight change is a disclosed fact (tier clamp, operator override,
|
|
57
|
-
failover all cause it legitimately), never a fault and never a verdict input.
|
|
58
|
-
|
|
59
|
-
## 2. Make the machine contract discoverable
|
|
60
|
-
|
|
61
|
-
**Status: the contract exists and is documented; it is invisible where a user
|
|
62
|
-
would look.**
|
|
63
|
-
|
|
64
|
-
`docs/exit-codes.md` documents a genuinely good tiered contract: with
|
|
65
|
-
`LOKI_DURABLE_STATE=1`, `loki start` distinguishes "failed the quality gate"
|
|
66
|
-
from "crashed" -- written for a Kubernetes Job or an ECS task. `loki verify` has
|
|
67
|
-
its own documented contract (`docs/exit-codes.md:53`).
|
|
68
|
-
|
|
69
|
-
But `loki start --help` mentions `LOKI_DURABLE_STATE` **zero times** (measured).
|
|
70
|
-
A CI author reads `--help`, sees "0 on success, nonzero on failure", and builds
|
|
71
|
-
the coarse gate. Factory's `droid exec` advertises its exit codes in its own help
|
|
72
|
-
output; ours are a doc you have to already know exists.
|
|
73
|
-
|
|
74
|
-
**Shipped in v9.27.1** for `loki start --help`: the two-tier contract, code 20,
|
|
75
|
-
its retry semantics, and a pointer to `docs/exit-codes.md`. A drift assertion
|
|
76
|
-
fails if the code stated in the help stops matching the code in the doc, since
|
|
77
|
-
two documents disagreeing about a value a Kubernetes Job is configured on is
|
|
78
|
-
worse than one.
|
|
79
|
-
|
|
80
|
-
`loki verify --help` needed no change -- it already carried an `EXIT CODES`
|
|
81
|
-
section. Adding a second one (which I briefly did) would have created exactly
|
|
82
|
-
the duplicated-and-drifting help this item exists to prevent; a test now asserts
|
|
83
|
-
there is exactly one.
|
|
84
|
-
|
|
85
|
-
**Note:** the research framed this as "Loki has no headless one-shot contract".
|
|
86
|
-
That framing was wrong -- the contract exists. The defect is discoverability,
|
|
87
|
-
which is a much cheaper fix.
|
|
88
|
-
|
|
89
|
-
---
|
|
90
|
-
|
|
91
|
-
## 3. Stop latency -- SHIPPED v9.28.0, and the premise was inverted
|
|
92
|
-
|
|
93
|
-
**Status: published in `docs/stop-latency.md`.**
|
|
94
|
-
|
|
95
|
-
The premise of this item was backwards. `loki stop` is NOT slow: it kills the
|
|
96
|
-
whole process group with SIGTERM, a 1 second grace, then SIGKILL
|
|
97
|
-
(`autonomy/loki`, `_stop_group_by_pgid_files`), and that bound does not depend
|
|
98
|
-
on what the run was doing.
|
|
99
|
-
|
|
100
|
-
What was slow is the mechanism the product itself recommended.
|
|
101
|
-
`autonomy/run.sh:161` told users `touch .loki/STOP - stops immediately`. That
|
|
102
|
-
was false: the STOP file is read only at the top of an iteration
|
|
103
|
-
(`check_human_intervention` at `:25149`, called from the single site `:22301`),
|
|
104
|
-
so a STOP written mid-dispatch waits for the provider call to return, bounded by
|
|
105
|
-
`LOKI_PROVIDER_CALL_TIMEOUT` (default 7200s). The docs pointed at the two-hour
|
|
106
|
-
path and called it immediate.
|
|
107
|
-
|
|
108
|
-
Shipped: the false "stops immediately" claim is corrected in place,
|
|
109
|
-
`docs/stop-latency.md` publishes both numbers with their derivations, and the
|
|
110
|
-
Bun route's provider call now honors `LOKI_PROVIDER_CALL_TIMEOUT` -- it
|
|
111
|
-
previously passed no timeout at all, leaving that route's STOP path with **no
|
|
112
|
-
upper bound**.
|
|
113
|
-
|
|
114
|
-
## 4. Close the config-diagnostic gap for the remaining format
|
|
115
|
-
|
|
116
|
-
**Status: shipped for JSON, `.env`, and YAML. Narrow residual gap only.**
|
|
117
|
-
|
|
118
|
-
`loki config validate` reports unknown keys in JSON and `.env` (v9.26.1) and in
|
|
119
|
-
YAML via pyyaml with a `yq` fallback (v9.26.2). `yq` is preinstalled on the
|
|
120
|
-
GitHub ubuntu-24.04 runner, and the fallback was verified against a stand-in
|
|
121
|
-
honouring both invocation shapes the real `yq` is called with, so CI and any
|
|
122
|
-
Linux host with either parser get full detection.
|
|
123
|
-
|
|
124
|
-
**CORRECTED, and closed in v9.28.0.** An earlier revision of this item called
|
|
125
|
-
the no-parser path "quiet degradation, which is correct". That was wrong, and
|
|
126
|
-
reproducing it settled the matter: with no parser the command printed
|
|
127
|
-
`config validate: OK` and exit 0 for a YAML file full of bogus keys. Silence in
|
|
128
|
-
the helper is fine; the CALLER turning that silence into an affirmative `OK` was
|
|
129
|
-
an assertion of validity nobody had checked, which is the exact false green this
|
|
130
|
-
project exists to prevent.
|
|
131
|
-
|
|
132
|
-
The verdict is now `INCOMPLETE` with a stderr line naming the skipped check, and
|
|
133
|
-
the helper reports "could not check" as a distinct status rather than an empty
|
|
134
|
-
result indistinguishable from "checked, nothing found". Guarded by
|
|
135
|
-
`tests/test-config-unknown-keys.sh` case 5d.
|
|
136
|
-
|
|
137
|
-
---
|
|
138
|
-
|
|
139
|
-
## 5. `loki init` writes a config nothing reads
|
|
140
|
-
|
|
141
|
-
**Status: confirmed defect, low blast radius.**
|
|
142
|
-
|
|
143
|
-
`loki init` writes `.loki/loki.config.json` with six behavioral-looking keys --
|
|
144
|
-
`provider`, `complexity`, `quality_gates`, `parallel_mode`, `dashboard` -- and
|
|
145
|
-
labels it "project configuration" (`autonomy/loki:16598`). **Nothing reads any
|
|
146
|
-
of them** (measured: 0 read sites for all five; the only `template` match is a
|
|
147
|
-
comment). A user who sets `"quality_gates": false` is silently ignored.
|
|
148
|
-
|
|
149
|
-
Note this is a *different file* from `.loki/config.json`, which is live and is
|
|
150
|
-
read for `memory.disabled` and `otel_endpoint`.
|
|
151
|
-
|
|
152
|
-
**Mitigated already:** as of v9.26.1, `loki config validate` on that file
|
|
153
|
-
reports each dead key by name. The gap is now diagnosable rather than silent.
|
|
154
|
-
|
|
155
|
-
**Do:** the honest minimum is to stop describing it as "project configuration".
|
|
156
|
-
Removing the keys outright flips `tests/test-init-command.sh:141`, which asserts
|
|
157
|
-
`'provider' in d` -- so that is a deliberate contract change, not a cleanup.
|
|
158
|
-
|
|
159
|
-
---
|
|
160
|
-
|
|
161
|
-
## 6. Verdict influence for the LLM review stage
|
|
162
|
-
|
|
163
|
-
**Status: stage ships in v9.26.0/9.26.1; verdict influence deliberately deferred.**
|
|
164
|
-
|
|
165
|
-
`llm_review` now runs by default and is recorded, but `affects_verdict` is
|
|
166
|
-
`false`. That was the right call for the release -- flipping it would silently
|
|
167
|
-
break anyone gating CI on exit 0.
|
|
168
|
-
|
|
169
|
-
**Do:** measure the reviewer on real diffs, then promote verdict influence behind
|
|
170
|
-
an explicit flag (`--llm-blocks`) before considering it as a default. The
|
|
171
|
-
sequencing matters more than the speed: a reviewer that returns CONCERNS where
|
|
172
|
-
deterministic-only returned VERIFIED is a breaking change to a published exit
|
|
173
|
-
contract.
|
|
174
|
-
|
|
175
|
-
---
|
|
176
|
-
|
|
177
|
-
## 7. Make the deferred-suite gap structural
|
|
178
|
-
|
|
179
|
-
**Status: process defect, cost two broken releases this cycle.**
|
|
180
|
-
|
|
181
|
-
v9.25.0 and v9.25.1 both failed to publish because a check that guards the
|
|
182
|
-
shipped artifact was **deferred by the fast tier** -- the only tier that runs
|
|
183
|
-
before every push. v9.26.0 failed the same way on repo-wide ShellCheck.
|
|
184
|
-
|
|
185
|
-
CLAUDE.md already states the rule ("a check that guards the shipped artifact must
|
|
186
|
-
run in the FAST tier"). The rule is not enforced.
|
|
187
|
-
|
|
188
|
-
**Shipped in v9.26.3** as `scripts/guard-changed.sh`: runs the suites that
|
|
189
|
-
reference the files in your diff, plus ShellCheck on the changed shell files.
|
|
190
|
-
Measured 8s for a one-file change, ~135s worst case, against 26m50s for the FULL
|
|
191
|
-
tier. Verified it would have blocked the v9.26.0 push locally.
|
|
192
|
-
|
|
193
|
-
**Remaining:** it is necessary, not sufficient. It cannot catch a failure that
|
|
194
|
-
depends on CI differing from your machine -- which is exactly how v9.26.2 failed,
|
|
195
|
-
on a suite this script selects and runs. The follow-on work is making
|
|
196
|
-
environment-conditional assertions name their condition (`command -v yq`) rather
|
|
197
|
-
than assume the author's host. That is a review habit, not a script.
|
|
198
|
-
|
|
199
|
-
---
|
|
200
|
-
|
|
201
|
-
## 8. Fix the flaky assurance-tail suite (OPEN, not yet fixed)
|
|
202
|
-
|
|
203
|
-
**Status: mechanism identified, not reproduced locally, deliberately not patched.**
|
|
204
|
-
|
|
205
|
-
`tests/test-review-assurance-tail.sh` has failed CI twice on unrelated commits
|
|
206
|
-
(v9.26.3 and v9.27.1), each costing a release cycle. Three different assertions
|
|
207
|
-
failed across the two incidents:
|
|
208
|
-
|
|
209
|
-
- `semantic shard FAIL: calls=3 (expected 4)`
|
|
210
|
-
- `valid structured requirements coverage did not pass`
|
|
211
|
-
- `requirements-forged-pass-error escaped parent-bound result publication`
|
|
212
|
-
|
|
213
|
-
They share a mechanism: the suite drives **real background subshells** (`sleep 30`,
|
|
214
|
-
`&`) and asserts **exact provider-call counts** (`= "1"`, `= "4"`) while also
|
|
215
|
-
testing that a FAIL *cancels* sibling lineages. On a 4-way-sharded runner the
|
|
216
|
-
cancellation can land before the last dispatch is logged, so the count races.
|
|
217
|
-
|
|
218
|
-
The suite already anticipates contention: `REVIEW_TIMEOUT_SCALE` is 4x when
|
|
219
|
-
`LOKI_TEST_SHARD` is set, and CI does set it (`.github/workflows/test.yml:149`).
|
|
220
|
-
So the timeout budget is **not** the binding constraint -- the exact-equality
|
|
221
|
-
call counts are.
|
|
222
|
-
|
|
223
|
-
**Why it is not patched here.** Both incidents were settled by a rerun on the
|
|
224
|
-
identical SHA (green both times), which proves flake and diff-innocence without
|
|
225
|
-
touching a fail-closed trust suite. It did not reproduce locally across six runs
|
|
226
|
-
including shard-scaled and CPU-loaded ones. Relaxing an exact count on a theory
|
|
227
|
-
would weaken a gate that deliberately distinguishes a LOST shard from a
|
|
228
|
-
CANCELLED one.
|
|
229
|
-
|
|
230
|
-
**Do:** make the timing deterministic rather than the assertion looser -- have
|
|
231
|
-
each dispatch log its intent *before* the call rather than after, so the count is
|
|
232
|
-
stable regardless of when cancellation lands. That changes the contract the suite
|
|
233
|
-
guards, so it needs its own cycle with the expected counts re-derived.
|
|
234
|
-
|
|
235
|
-
---
|
|
236
|
-
|
|
237
|
-
## 9. Enforcement claims in buyer-facing docs -- SHIPPED v9.28.0
|
|
238
|
-
|
|
239
|
-
The source was scrupulously honest and the docs were not.
|
|
240
|
-
`autonomy/run.sh:515` states `LOKI_ALLOWED_PATHS` "Does NOT restrict
|
|
241
|
-
provider-driven agent writes"; `check_command_allowed` is "intentionally NOT
|
|
242
|
-
called" with zero callers. Meanwhile `wiki/Enterprise-Features.md` listed both
|
|
243
|
-
as production security controls and `docs/certification/answer-key.md` marked
|
|
244
|
-
"restricts which directories agents can modify" as the **correct exam answer**.
|
|
245
|
-
Nine buyer-facing files, zero caveats.
|
|
246
|
-
|
|
247
|
-
All nine now carry a `SANDBOX-SCOPED` note saying what is enforced (2 mount-time
|
|
248
|
-
call sites in `autonomy/sandbox.sh`, and one for operator-typed
|
|
249
|
-
`loki sandbox run` argv), what is not, and that both enforce nothing unless
|
|
250
|
-
`LOKI_SANDBOX_MODE=true`. `tests/test-enforcement-doc-honesty.sh` guards it with
|
|
251
|
-
marker-presence rather than phrase-absence, because a grep for "restrict" near
|
|
252
|
-
the variable name fires on the caveat itself.
|
|
253
|
-
|
|
254
|
-
Two errors in my own brief were caught while planning: it is 2 path-enforcement
|
|
255
|
-
call sites, not 4 (I had counted the definition and a comment), and a
|
|
256
|
-
pre-existing answer-key letter mismatch (key said A, quiz option was B) was
|
|
257
|
-
fixed at the same time.
|
|
258
|
-
|
|
259
|
-
---
|
|
260
|
-
|
|
261
|
-
## Not items (research proposed these; they are already shipped)
|
|
262
|
-
|
|
263
|
-
- **Signed receipts default-off is a moat gated behind an env var.** The receipt
|
|
264
|
-
already states signature status explicitly in both directions
|
|
265
|
-
(`proof-generator.py:1808-1820`): SIGNED, or UNSIGNED with the honest line that
|
|
266
|
-
the integrity hash "does NOT prove who produced them, so this receipt trusts
|
|
267
|
-
its generator." That is the shippable version. Signing is now default-on through an
|
|
268
|
-
auto-generated local Ed25519 key (A-120); the old gpg switch was deleted.
|
|
269
|
-
Key distribution remains a founder decision.
|
|
270
|
-
- **OTEL to a customer-owned collector.** Already implemented -- `otel_endpoint`
|
|
271
|
-
is persisted and read (`autonomy/loki:26755`, `:26828-26837`).
|
|
272
|
-
- **A headless exec contract with documented exit codes.** Already exists via
|
|
273
|
-
`LOKI_DURABLE_STATE=1`; see item 2, which is the real (smaller) gap.
|
|
274
|
-
|
|
275
|
-
## 10. Hard command blocklist (OPEN, designed, not built)
|
|
276
|
-
|
|
277
|
-
**CORRECTION.** This section previously listed a hard blocklist as
|
|
278
|
-
"architecturally unavailable". That was too pessimistic. It is unavailable in
|
|
279
|
-
`run.sh`, which never observes agent commands -- but a design review found a
|
|
280
|
-
boundary where it IS enforceable, and the honest scope is narrower than Factory
|
|
281
|
-
states for their own.
|
|
282
|
-
|
|
283
|
-
What the review established:
|
|
284
|
-
|
|
285
|
-
- **Seccomp cannot do it.** `autonomy/seccomp-sandbox.json` allows `execve`
|
|
286
|
-
unconditionally, and seccomp-bpf filters on register values: `execve`'s
|
|
287
|
-
pathname is a userspace pointer a filter cannot dereference. Any proposal to
|
|
288
|
-
"add blocked commands to seccomp" is not implementable.
|
|
289
|
-
- **A PATH shim cannot do it.** Defeated by an absolute path, by `env -i`, or by
|
|
290
|
-
resetting PATH. That is precisely the class Factory claims to defeat.
|
|
291
|
-
- **A read-only bind-mount over the resolved binary inode CAN.** The mount is
|
|
292
|
-
decided on the host before the container exists, and cannot be undone from
|
|
293
|
-
inside: `--cap-drop=ALL` (no `CAP_SYS_ADMIN`), `no-new-privileges`, non-root
|
|
294
|
-
user, and the seccomp profile denies `umount`. Because it targets an inode, no
|
|
295
|
-
string is matched, so `bash -c`, absolute paths, quoting and command
|
|
296
|
-
substitution all converge on the same blocked inode. That is a stronger claim
|
|
297
|
-
than argv parsing, not a weaker one.
|
|
298
|
-
|
|
299
|
-
**The scoping finding that makes this honest.** `loki sandbox` has three modes,
|
|
300
|
-
and only `docker` supports it. `docker sandbox create` (Docker Desktop microVM
|
|
301
|
-
mode) takes no mount flags, and worktree mode has no container at all -- and
|
|
302
|
-
auto-detect prefers Docker Desktop first. So the guarantee must be paired with
|
|
303
|
-
**fail-closed refusal to start** in every unsupported mode, or it becomes the
|
|
304
|
-
same overstatement this release just spent its time removing.
|
|
305
|
-
|
|
306
|
-
Also worth stating plainly, which Factory's docs elide: blocking a program does
|
|
307
|
-
not remove the capability. With the default bridge network an agent can fetch a
|
|
308
|
-
replacement binary. `LOKI_SANDBOX_NETWORK=none` removes the capability.
|
|
309
|
-
|
|
310
|
-
**Do:** implement as `LOKI_BLOCKLIST_COMMANDS` (a new key -- the existing
|
|
311
|
-
`LOKI_BLOCKED_COMMANDS` is an advisory substring filter with different
|
|
312
|
-
semantics), with the adversarial test matrix and a SKIP-not-PASS rule when
|
|
313
|
-
Docker is unavailable. Design is complete; the build is a separate cycle.
|
|
314
|
-
|
|
315
|
-
- **Per-command risk tiers** remain out of reach for the same original reason:
|
|
316
|
-
you cannot classify a command you never observe.
|