loki-mode 7.129.5 → 8.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +81 -5
- package/SKILL.md +66 -5
- package/VERSION +1 -1
- package/autonomy/app-runner.sh +37 -0
- package/autonomy/completion-council.sh +862 -28
- package/autonomy/council-v2.sh +39 -0
- package/autonomy/crash.sh +3 -2
- package/autonomy/grill.sh +42 -0
- package/autonomy/hooks/validate-bash.sh +301 -4
- package/autonomy/lib/cockpit-render.sh +8 -2
- package/autonomy/lib/cr-rematerialize.py +101 -4
- package/autonomy/lib/deadline.py +957 -0
- package/autonomy/lib/dependency-setup.sh +205 -0
- package/autonomy/lib/done-recognition.sh +45 -0
- package/autonomy/lib/efficiency_cost.py +13 -6
- package/autonomy/lib/no_mock_scan.py +793 -0
- package/autonomy/lib/prd-enrich.sh +42 -0
- package/autonomy/lib/proof-analytics-props.py +69 -0
- package/autonomy/lib/proof-generator.py +282 -95
- package/autonomy/lib/proof-template.html +15 -12
- package/autonomy/lib/proof-verify.py +151 -102
- package/autonomy/lib/requirements_contract.py +848 -0
- package/autonomy/lib/sdk-mode.sh +103 -0
- package/autonomy/lib/secret-scan.sh +139 -0
- package/autonomy/lib/spec-expand.sh +208 -0
- package/autonomy/lib/tree_digest.py +266 -0
- package/autonomy/lib/voter-agents.sh +58 -8
- package/autonomy/lib/workspace_diff.py +124 -0
- package/autonomy/loki +189 -17
- package/autonomy/playwright-verify.sh +501 -0
- package/autonomy/prd-checklist.sh +17 -8
- package/autonomy/provider-offer.sh +111 -0
- package/autonomy/run.sh +4417 -676
- package/autonomy/sandbox.sh +42 -0
- package/autonomy/spec-interrogation.sh +148 -5
- package/autonomy/spec.sh +118 -1
- package/autonomy/telemetry.sh +133 -7
- package/autonomy/verify.sh +107 -0
- package/bin/loki +110 -3
- package/completions/_loki +10 -0
- package/completions/loki.bash +1 -1
- package/dashboard/__init__.py +1 -1
- package/dashboard/audit.py +96 -7
- package/dashboard/build_supervisor.py +1619 -0
- package/dashboard/control.py +8 -0
- package/dashboard/prompt_optimizer.py +7 -0
- package/dashboard/server.py +577 -22
- package/dashboard/static/trust.html +1 -1
- package/docs/ARCHITECTURE-OVERVIEW.md +207 -0
- package/docs/AUTONOMI-ECOSYSTEM.md +107 -0
- package/docs/INSTALLATION.md +19 -2
- package/docs/MODEL-EQUIVALENCE-HARNESS-PLAN.md +70 -0
- package/docs/PLANS-INDEX.md +33 -0
- package/docs/PRIVACY.md +29 -1
- package/docs/SIGNED-RECEIPTS.md +102 -0
- package/docs/SONNET5-DEFAULT-PLAN.md +1 -1
- package/docs/V8-ACCEPTANCE-TRIAGE-2026-07-24.md +114 -0
- package/docs/V8-AGENT-SDK-PLAN.md +692 -0
- package/docs/V8-COMPLEXITY-AUDIT-2026-07-24.md +45 -0
- package/docs/V8-MAJOR-RELEASE-PLAN.md +137 -0
- package/docs/V8-OVERNIGHT-PLAN-2026-07-25.md +82 -0
- package/docs/V8-RUNTIME-TRUTH-2026-07-25.md +232 -0
- package/docs/V8-SDK-RESEARCH-RAW.md +129 -0
- package/docs/alternative-installations.md +4 -4
- package/loki-ts/dist/loki.js +674 -285
- package/loki-ts/package.json +2 -0
- package/mcp/__init__.py +1 -1
- package/package.json +7 -6
- package/plugins/loki-mode/.claude-plugin/plugin.json +1 -1
- package/providers/claude.sh +75 -0
- package/providers/codex.sh +85 -13
- package/references/sdk-mode.md +106 -0
- package/skills/model-selection.md +1 -1
- package/skills/quality-gates.md +22 -0
|
@@ -0,0 +1,692 @@
|
|
|
1
|
+
# V8: SDK-Based Runtime Migration Plan
|
|
2
|
+
|
|
3
|
+
Status: IMPLEMENTED (shipped on feature/v8-agent-sdk as v8.0.0 + v8.1; kept as design history)
|
|
4
|
+
Target branch: `feature/v8-agent-sdk`
|
|
5
|
+
Author: architecture spike (research-grounded, live SDK docs verified)
|
|
6
|
+
Scope: replace the bash `claude -p` wrapper with the Anthropic TypeScript SDKs. Codex/Cline/Aider stay on bash. This is a multi-week arc, not one workflow.
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## 0. The one fact that reframes everything
|
|
11
|
+
|
|
12
|
+
Loki already has a full TypeScript port of the autonomous runner. It is not hypothetical and it is not a stub.
|
|
13
|
+
|
|
14
|
+
`loki-ts/src/runner/` contains 12,300 lines across 22 modules that mirror the bash runner byte-for-byte: `autonomous.ts` (983), `build_prompt.ts` (1582), `council.ts` (806), `quality_gates.ts` (2805), `providers.ts` (562), `rarv.ts`, `budget.ts`, `checkpoint.ts`, `completion.ts`, `state.ts`, etc. Many of these are explicitly "parity-locked" with their bash siblings (grep `run.sh` for "Parity-locked with ... loki-ts/src/runner").
|
|
15
|
+
|
|
16
|
+
The catch, verified at `autonomy/run.sh:17271`:
|
|
17
|
+
|
|
18
|
+
> "The claude provider in loki-ts/src/runner/providers.ts is implemented but is NOT reached for `start` (start is not ported to the Bun router; the shim falls through to bash), so its flag set has zero live impact today."
|
|
19
|
+
|
|
20
|
+
And the transport, `loki-ts/src/runner/providers.ts:305`: `const r = await shellRun(argv, ...)` - the TS runner shells out to the `claude` binary exactly like bash does.
|
|
21
|
+
|
|
22
|
+
So the v8 work is NOT "port 35k lines of bash to TypeScript." That port largely exists. The v8 work is two much smaller things:
|
|
23
|
+
|
|
24
|
+
1. **Finish wiring the existing TS runner into `loki start`** (the Bun router in `bin/loki` already routes ~8 commands to Bun and falls through to bash for the rest, including `start`).
|
|
25
|
+
2. **Swap the invocation transport inside that TS runner** from `shellRun(["claude", ...])` to the SDK: `query()` for the agentic loop, `messages.create()` for the one-shot judges.
|
|
26
|
+
|
|
27
|
+
This changes the shape, risk, and phasing of the whole migration. It is a transport swap inside an existing, parity-locked TS codebase, done incrementally behind the `LOKI_LEGACY_BASH` rollback flag that already exists.
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## 1. Target architecture
|
|
32
|
+
|
|
33
|
+
### 1.1 Two SDKs, two layers (the split is real and load-bearing)
|
|
34
|
+
|
|
35
|
+
| Layer | What runs there | SDK | Why |
|
|
36
|
+
|---|---|---|---|
|
|
37
|
+
| **One-shot judges / graders** - code review (B1), completion council members (B3/B4), council-v2 (B5/B6), done-recognition (B7), PRD enrich (C1), grill (C2), USAGE regen (C3), doc-gen (C4), merge-conflict resolve (B8) | `@anthropic-ai/sdk` (raw Client SDK, `messages.create` / `messages.parse` / `messages.stream`) | Pure HTTPS to `api.anthropic.com`. **Zero binary.** One prompt in, one (optionally schema-constrained) JSON answer out. No tool loop, no filesystem, no subprocess. This is the honest `--bare` path. |
|
|
38
|
+
| **The autonomous RARV dev loop** (A1) and the migration/heal agentic execs (C5/C6/C7) | `@anthropic-ai/claude-agent-sdk` (`query()`) | Needs the built-in Read/Write/Edit/Bash/Glob/Grep tool loop, MCP, hooks, subagents, sessions. Reimplementing all of that on the raw SDK is exactly the wheel the Agent SDK exists to avoid. |
|
|
39
|
+
|
|
40
|
+
**This split is not gold-plating - it is forced by what each site needs.** A judge that returns one JSON verdict must not carry a filesystem-agent harness. A dev iteration that writes files agentically must not be hand-rolled on `messages.create`. The `providers/` abstraction already models a per-provider invoke; we add an SDK-backed claude invoker with two code paths keyed on a flag the runner already threads: `call.mainLoop` (agentic → Agent SDK) vs one-shot (raw SDK).
|
|
41
|
+
|
|
42
|
+
### 1.2 Who owns it: the existing loki-ts runner (do NOT start a new module)
|
|
43
|
+
|
|
44
|
+
The runner lives in `loki-ts/src/runner/`. It is parity-locked and mostly written. We do not create a parallel module and we do not move logic out of it. We:
|
|
45
|
+
|
|
46
|
+
- Add `loki-ts/src/runner/sdk_invoker.ts` - the SDK-backed replacement for the `shellRun(["claude", ...])` call inside `providers.ts` `claudeProvider()`. Same `ProviderInvoker` contract (`invoke(call): {exitCode, capturedOutputPath}`), so the runner above it does not change.
|
|
47
|
+
- Add `loki-ts/src/runner/sdk_stream_parser.ts` - consumes typed `SDKMessage` objects and writes the same `.loki/state/agents.json`, `.loki/events.jsonl` hook events, and `.loki/metrics/result-cost-<iter>.json` that the ~350-line embedded Python stream-json parser writes today (`run.sh:17371+`). This is the one genuinely new piece of logic, and it is a translation of an existing parser, not a new design.
|
|
48
|
+
- Finish the `bin/loki` router so `start` (and `heal`, `migrate`) route to Bun when the SDK path is enabled, and fall through to bash otherwise.
|
|
49
|
+
|
|
50
|
+
### 1.3 Honest correction to the founder premise: "no binary" is only true of the raw SDK
|
|
51
|
+
|
|
52
|
+
Verified from the unpacked `@anthropic-ai/claude-agent-sdk@0.3.207` tarball and live docs:
|
|
53
|
+
|
|
54
|
+
- **`@anthropic-ai/claude-agent-sdk` bundles and spawns a native Claude Code binary** (8 platform-specific `optionalDependencies`; `sdk.mjs` calls `spawn()`; `Options.pathToClaudeCodeExecutable` exists). It is Claude Code as a library, not a pure HTTP client. It removes the *separately-installed, interactively-authed, independently-versioned* CLI - but a native executable still runs as a child process.
|
|
55
|
+
- **`@anthropic-ai/sdk` (raw Client SDK, v0.111.0) is pure HTTPS. No binary.**
|
|
56
|
+
|
|
57
|
+
So the accurate strategic pitch (Section 3) is: **the judge layer (raw SDK) is genuinely binary-free; the dev-loop layer (Agent SDK) replaces the unmanaged PATH CLI with a bundled, version-pinned, API-key-authed one.** Both deliver "no `claude auth`, no CLI install step, no runtime flag drift." Only the raw-SDK layer delivers "literally zero binary." State it that way to the founder - a reviewer will catch "no binary at all" and it is not true of the Agent SDK.
|
|
58
|
+
|
|
59
|
+
### 1.4 Model IDs (verified against live model catalog, replacing the placeholders in the codebase)
|
|
60
|
+
|
|
61
|
+
The codebase memory lists placeholder model names. The live, correct IDs to wire in:
|
|
62
|
+
|
|
63
|
+
| RARV tier | Model ID | Used for |
|
|
64
|
+
|---|---|---|
|
|
65
|
+
| planning / architecture | `claude-opus-4-8` | Opus tier - planning, devil's advocate, requirements-verifier |
|
|
66
|
+
| development / execution | `claude-sonnet-5` | Sonnet tier - the main dev loop default (`LOKI_SESSION_MODEL`), test-auditor |
|
|
67
|
+
| fast / unit / simple judges | `claude-haiku-4-5` | Haiku tier - council members, convergence-voter, USAGE regen |
|
|
68
|
+
|
|
69
|
+
Effort maps directly: `output_config.effort` / Agent SDK `effort` supports `low|medium|high|xhigh|max`. `xhigh` is the recommended default for coding/agentic work on Sonnet 5 / Opus 4.8.
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## 2. Feature-preservation matrix ("lose nothing")
|
|
74
|
+
|
|
75
|
+
Every capability the bash route uses today, and how v8 preserves or enhances it. Nothing is silently dropped. Grounded in `autonomy/run.sh:17195-17369`, `lib/claude-flags.sh`, and the two SDKs' verified APIs.
|
|
76
|
+
|
|
77
|
+
| # | Claude Code capability today | Bash site | v8 preservation | E = enhancement |
|
|
78
|
+
|---|---|---|---|---|
|
|
79
|
+
| 1 | Agentic tool-use loop (RARV dev iteration) | `run.sh:17365` `-p` + stream-json | Agent SDK `query()` runs the loop; consume `SDKMessage` stream | E: typed messages replace ~350-line stdout parser |
|
|
80
|
+
| 2 | Built-in tools Read/Write/Edit/Bash/Glob/Grep/WebSearch/WebFetch | implicit in `claude -p` | Agent SDK ships them; gate via `allowedTools` | E: adds `Monitor` (watch background script), not available today |
|
|
81
|
+
| 3 | `--dangerously-skip-permissions` | `run.sh:17195` | `permissionMode: 'bypassPermissions'` | - |
|
|
82
|
+
| 4 | `--allowedTools` least-privilege allowlist | `run.sh:11003` (review) | `allowedTools: [...]` | note: SDK `allowedTools` = auto-approve, not restrict; use `disallowedTools` to actually block |
|
|
83
|
+
| 5 | `--disallowedTools` reviewer denylist | council/grill | `disallowedTools: ["Bash(rm *)", ...]` | E: scoped patterns block even under bypass - stronger than today |
|
|
84
|
+
| 6 | Subagents `--agents <json>` | `voter-agents.sh:284` | `agents: Record<string, AgentDefinition>` + `Agent` in allowedTools | E: typed AgentDefinition; delete the `VOTE:` regex fallback (structured output guaranteed) |
|
|
85
|
+
| 7 | Hooks (SessionStart/PreToolUse/Stop) | `.claude/settings.json`, migration-hooks | Agent SDK `hooks: {...}` in-process callbacks | E: healing hooks run in-process with structured input, not shelled scripts |
|
|
86
|
+
| 8 | MCP `--mcp-config` / `--strict-mcp-config` | `run.sh:4166` | Agent SDK `mcpServers: {...}` + `strictMcpConfig`; raw SDK `mcp_servers` (beta `mcp-client-2025-11-20`) | Loki's own `mcp/server.py` (34 tools) plugs in unchanged |
|
|
87
|
+
| 9 | `--append-system-prompt` (autonomy override) | `run.sh:17200` | Agent SDK `systemPrompt: {preset:'claude_code', append}`; raw SDK `system:"..."` | - |
|
|
88
|
+
| 10 | `--setting-sources user,project,local` | `run.sh:17210` | Agent SDK `settingSources: ['user','project','local']` | 1:1 |
|
|
89
|
+
| 11 | CLAUDE.md auto-discovery | implicit | Agent SDK loads it when `settingSources` includes source | E: native; the bash route fights CLAUDE.md via `--append-system-prompt` |
|
|
90
|
+
| 12 | Prompt caching | implicit | Agent SDK: managed by harness. Raw SDK: manual `cache_control` breakpoints | E: the inert `[CACHE_BREAKPOINT]` marker in `build_prompt` can finally set real `cache_control` on the stable prefix |
|
|
91
|
+
| 13 | `--json-schema` structured output | `done-recognition.sh:66`, `council-v2.sh:312`, `voter-agents.sh` | raw SDK `output_config:{format:{type:'json_schema',schema}}` / `messages.parse()`; Agent SDK `outputFormat:{type:'json_schema',schema}` | E: `messages.parse()` validates + types the result; deletes `cr-rematerialize.py` re-materialization step |
|
|
92
|
+
| 14 | `--effort` per RARV tier | `run.sh:17307` | raw SDK `output_config.effort`; Agent SDK `effort` | 1:1 (`low..max`) |
|
|
93
|
+
| 15 | **`--max-budget-usd` per-call backstop** | `run.sh:17315` | **NO per-call USD cap primitive in either SDK.** Keep Loki's own cumulative `check_budget_limit` PAUSE gate (already ported: `loki-ts/src/runner/budget.ts`). Nearest SDK primitive: `output_config.task_budget` (TOKENS, beta `task-budgets-2026-03-13`) or Agent SDK `maxBudgetUsd` field | **REAL GAP - see Risks §8.** Agent SDK exposes `maxBudgetUsd` per docs; raw SDK does not. Convert USD estimate to token `task_budget` on the judge path, keep the deterministic budget gate on both. |
|
|
94
|
+
| 16 | `--fallback-model` | `run.sh:17323` | Agent SDK `fallbackModel` field; raw SDK: catch overload error + retry with fallback model, or server-side `fallbacks` (Fable-5-only beta - not applicable to Opus/Sonnet tiers) | verify Agent SDK `fallbackModel` before relying (Risks §8) |
|
|
95
|
+
| 17 | Session resume `--resume`/`--fork-session`/`--session-id` | `run.sh:17239` | Agent SDK `resume`, `forkSession`, `sessionId`, `continue`, `persistSession`, `resumeSessionAt` | E: `resumeSessionAt` (resume at a message UUID) finer than CLI; `listSessions()`/`getSessionMessages()` replace `~/.claude` JSONL filename-scraping for dashboard correlation |
|
|
96
|
+
| 18 | Streaming `stream-json --verbose` + `--include-partial-messages` | `run.sh:17366` | Agent SDK `query()` async iterator; `includePartialMessages`; `includeHookEvents` | E: dashboard stream parser replaced by typed message objects |
|
|
97
|
+
| 19 | `--bare` cheap-subcall mode | `claude-flags.sh:147` | raw SDK IS the bare path - one HTTP call, no discovery. The `--bare` OAuth-vs-keychain gymnastics (`claude-flags.sh:157-185`) vanish | E: always `ANTHROPIC_API_KEY`, no OAuth branch |
|
|
98
|
+
| 20 | Model selection `--model` (tiers) | `run.sh:17195` | Agent SDK `model:'opus'/'sonnet'/'haiku'` or full ID; raw SDK `model:"claude-opus-4-8"` | 1:1 |
|
|
99
|
+
| 21 | `claude auth status` preflight | `run.sh:2169`,`2382` | replaced by "is `ANTHROPIC_API_KEY` set?" (or Bedrock/Vertex/Foundry env). Third-party products may NOT use claude.ai login | E: auth-preflight branch (`run.sh:2355-2456`) collapses to a key check |
|
|
100
|
+
| D1 | `claude ultrareview` (native cloud review) | `loki:18013` | **NO SDK equivalent.** Keep the binary for this command OR reimplement as an Agent SDK multi-agent workflow | BLOCKER - see §2.1 |
|
|
101
|
+
| D2 | `claude -p "ultracode: ..."` (Dynamic Workflows) | `loki:18137` | **NO SDK equivalent.** Same options as D1 | BLOCKER |
|
|
102
|
+
| D3 | `claude remote-control` (cockpit) | `loki:23619` | **NO SDK equivalent** (long-lived interactive exec). Keep the binary | BLOCKER |
|
|
103
|
+
| D4 | quick-start `claude --dangerously-skip-permissions` (literal launcher) | `loki:13274` | printed entrypoint for humans; keep as-is or point at `claude` if installed | cosmetic |
|
|
104
|
+
|
|
105
|
+
### 2.1 The Cluster-D landmine (the real "lose no features" risk)
|
|
106
|
+
|
|
107
|
+
`ultrareview`, `ultracode`/Dynamic Workflows, and `remote-control` are native Claude Code *CLI subcommands* with no `@anthropic-ai/sdk` or `@anthropic-ai/claude-agent-sdk` equivalent. Two honest options, decided before claiming zero-binary deployment:
|
|
108
|
+
|
|
109
|
+
- **(a) Carve-out (recommended for v8):** keep the `claude` binary available for exactly these three commands; everything else goes SDK. The SaaS/enterprise win (Section 3) still lands for the 99% hot path (RARV loop + judges); these three are power-user/interactive commands rarely run in a headless container.
|
|
110
|
+
- **(b) Reimplement** each as an Agent SDK multi-agent workflow (`ultrareview` → a review subagent fan-out; `ultracode` → a planned Agent SDK workflow). Large, deferrable, out of the v8 critical path.
|
|
111
|
+
|
|
112
|
+
The plan proceeds with (a). D1-D4 are explicitly OUT of the v8 phases below and tracked as a separate arc. This must be stated to the founder: "zero binary" is true of the hot path, not of these three commands, unless we fund option (b).
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## 3. Enterprise / SaaS deployment section (the headline win, made concrete)
|
|
117
|
+
|
|
118
|
+
The strategic win is real, with the §1.3 correction applied. Three concrete deletions and three concrete gains.
|
|
119
|
+
|
|
120
|
+
### 3.1 What gets deleted from the deployment surface
|
|
121
|
+
|
|
122
|
+
1. **No CLI install step.** Today a container needs the `claude` binary installed and on PATH (`curl | sh`, PATH wiring in the Dockerfile). With the raw SDK: `npm install @anthropic-ai/sdk` - pure JS, no native dep. With the Agent SDK: `npm install @anthropic-ai/claude-agent-sdk` - the platform binary ships as a pinned optional dependency inside the package; still no separate install step, no PATH wiring.
|
|
123
|
+
2. **No interactive login.** The bash route runs `claude auth status` preflight and branches on OAuth vs keychain (`run.sh:2355-2456`, `claude-flags.sh:157-185`). Both SDKs authenticate from `ANTHROPIC_API_KEY` (or Bedrock `CLAUDE_CODE_USE_BEDROCK=1` + `ANTHROPIC_AWS_WORKSPACE_ID`, Vertex `CLAUDE_CODE_USE_VERTEX=1`, Foundry `CLAUDE_CODE_USE_FOUNDRY=1`). No `claude login`, no `~/.claude/.credentials.json` dance. Third-party products may NOT use claude.ai subscription login via the SDK - API-key/cloud-provider only. This deletes the entire auth-preflight branch.
|
|
124
|
+
3. **No runtime CLI-version drift.** Today Loki greps `claude --help` at runtime to feature-detect `--json-schema`/`--agents`/`--effort`/`--fallback-model` (`claude-flags.sh:120-136`, `loki_claude_flag_supported` gating ~10 flags) precisely *because* the external CLI drifts. The SDK either has the typed option or it does not - no `--help` grep. The whole capability-probe layer (`lib/claude-flags.sh`) plus ~12 `command -v claude` presence probes and the `pkill claude` cleanup disappear.
|
|
125
|
+
|
|
126
|
+
### 3.2 The three concrete gains for Autonomi SaaS
|
|
127
|
+
|
|
128
|
+
1. **Multi-tenant by env injection.** Per-tenant `ANTHROPIC_API_KEY` (or per-tenant Bedrock workspace via `ANTHROPIC_AWS_WORKSPACE_ID`) injected as env at container spawn. One image, no per-container CLI provisioning, no per-tenant `claude login`.
|
|
129
|
+
2. **Deterministic packaging / reproducible builds.** `loki@X ⇒ @anthropic-ai/sdk@Y` (judges) and `⇒ @anthropic-ai/claude-agent-sdk@Z ⇒ pinned binary@Z` (loop). No "works-on-my-CLI-version." The Agent SDK pins its own binary to its package version in `package.json`.
|
|
130
|
+
3. **Richer structured telemetry, no stdout scraping.** Result messages carry `total_cost_usd`, `usage` (incl. cache read/creation), `modelUsage`, `num_turns`, `duration_ms`, `permission_denials`. The dashboard reads a typed object instead of the embedded Python stream-json parser (`run.sh:17371+`).
|
|
131
|
+
|
|
132
|
+
### 3.3 The honest asterisk (must appear in the founder deck)
|
|
133
|
+
|
|
134
|
+
The Agent SDK layer still spawns a bundled native binary per container. The accurate claim is **"no unmanaged, separately-installed, interactively-authed, independently-versioned CLI"** - not "no binary at all." The judge layer (raw SDK) IS literally binary-free. If the founder needs literally-zero-binary for the whole runtime, the fork is: run the RARV loop on the raw SDK too and re-own the tool loop (which `run_autonomous` + the 8 gates + council mostly already own) - larger, and evaluated as a future arc, not v8.
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## 4. Phases (ordered by value / risk / dependency)
|
|
139
|
+
|
|
140
|
+
Each phase is a discrete, agent-sized, independently-shippable unit with parity + rollback. The `LOKI_LEGACY_BASH=1` flag and the per-command `bin/loki` router already exist, so every phase ships behind a flag with bash as the live fallback until SDK-proven.
|
|
141
|
+
|
|
142
|
+
Guiding principle: **judges before the loop.** The one-shot judges (raw SDK) are low-risk, high-parity-testability (deterministic verdict comparison), and prove the SDK bridge end-to-end without touching the RARV hot path. Do them first. The Agent SDK loop is last and hardest.
|
|
143
|
+
|
|
144
|
+
### Phase 0 - Spikes (no production code; de-risk the unknowns)
|
|
145
|
+
|
|
146
|
+
- Spike A: `bun add @anthropic-ai/claude-agent-sdk`, run a trivial `query()` under Bun 1.3.13. Confirm Bun resolves the platform-gated optionalDependencies and spawns the bundled binary. If it fails, the loop runs under Node while judges stay on Bun. **This gates Phase 4's runtime choice.**
|
|
147
|
+
- Spike B: confirm `@anthropic-ai/sdk` `messages.parse()` + `output_config.format` produces the exact JSON shape `loki-ts/data/done-recognition-schema.json` expects.
|
|
148
|
+
- Spike C: confirm Agent SDK `maxBudgetUsd` and `fallbackModel` fields exist in `sdk.d.ts@0.3.207` (research flagged `fallbackModel` present, `maxBudgetUsd` present; `--max-budget-usd` has no raw-SDK analog).
|
|
149
|
+
- Verification: a throwaway script per spike; no `.loki/` writes; results recorded in the phase's PR description.
|
|
150
|
+
- Rollback: n/a (no production code).
|
|
151
|
+
|
|
152
|
+
### Phase 1 - done-recognition on the raw SDK (the bridge proof)
|
|
153
|
+
|
|
154
|
+
**Smallest real SDK adoption that proves the bridge end-to-end.** done-recognition (B7) is CLAUDE-ONLY (no provider sibling to keep in sync), one prompt → one schema-constrained JSON answer, already has an inline schema (`loki-ts/data/done-recognition-schema.json`), and fails inconclusive-safe. Perfect first target.
|
|
155
|
+
|
|
156
|
+
- Files touched: `loki-ts/src/runner/sdk_invoker.ts` (new, judge path only), `autonomy/lib/done-recognition.sh` (add an SDK branch gated behind `LOKI_SDK_DONE_RECOG=1`, keeping the `claude`/deterministic fallback), `loki-ts/data/done-recognition-schema.json` (reused as-is).
|
|
157
|
+
- Stays on bash (fallback): the existing `claude -p --json-schema` path and the deterministic fallback, both live when the flag is off.
|
|
158
|
+
- Moves to SDK: the LLM call becomes `@anthropic-ai/sdk` `messages.parse()` with `output_config.format` + the existing schema, `model: claude-haiku-4-5`, `effort: low`.
|
|
159
|
+
- Parity proof: run both routes (`LOKI_SDK_DONE_RECOG=0` vs `=1`) over a fixed corpus of `.loki` states; the parsed `requirements` verdict object must match. Wire into `local-ci.sh` bun-parity matrix.
|
|
160
|
+
- Rollback: unset `LOKI_SDK_DONE_RECOG`. One env var. Zero blast radius (single CLAUDE-ONLY helper).
|
|
161
|
+
|
|
162
|
+
### Phase 2 - the rest of the one-shot judges on the raw SDK
|
|
163
|
+
|
|
164
|
+
Extend `sdk_invoker.ts`'s judge path to the remaining single-shot sites, one PR per cluster, each behind its own flag with the bash arm intact:
|
|
165
|
+
|
|
166
|
+
- 2a: council-v2 reviewers (B5/B6) - `council-v2.sh`, schema `loki-ts/data/council-v2-schema.json`. Delete the sed-carving text fallback (structured output guaranteed).
|
|
167
|
+
- 2b: completion-council members + devil's advocate (B3/B4) - `completion-council.sh`. Replace `VOTE:` regex parsing with `messages.parse()`.
|
|
168
|
+
- 2c: code-review 3-reviewer (B1) - `run.sh:11035`, schema `loki-ts/data/code-review-schema.json`. Deletes the `cr-rematerialize.py` re-materialization to legacy `VERDICT:` text.
|
|
169
|
+
- 2d: aux helpers - PRD enrich (C1), grill (C2), USAGE regen (C3), doc-gen (C4), merge-conflict resolve (B8). All captured-text, no schema.
|
|
170
|
+
- Files: the six `.sh`/`run.sh` sites above + `sdk_invoker.ts`. Codex/Cline/Aider `case` arms untouched.
|
|
171
|
+
- Parity proof: verdict/text comparison per cluster over a fixed corpus; council decisions must be identical.
|
|
172
|
+
- Rollback: per-cluster env flag; bash arm live throughout.
|
|
173
|
+
|
|
174
|
+
### Phase 3 - the council as a single-dispatch Agent SDK `agents` call (the cleanest enhancement)
|
|
175
|
+
|
|
176
|
+
`voter-agents.sh:284` already fans out to N named reviewers in one `claude --agents <json>` call. This maps 1:1 to Agent SDK `agents: Record<string, AgentDefinition>`.
|
|
177
|
+
|
|
178
|
+
- Files: `loki-ts/src/runner/council.ts` (already ported), `autonomy/lib/voter-agents.sh` (SDK branch behind flag), schema `loki-ts/data/finding-schema.json`.
|
|
179
|
+
- Moves to SDK: the Python-generated `agents_json` becomes a typed `Record<string, AgentDefinition>`; `--json-schema` becomes `outputFormat`. KEEP the decision engine (effective-threshold floor, exact-quorum gate, devil's-advocate override, transcript writing) - none of that is an SDK concept.
|
|
180
|
+
- Parity proof: `.loki/council/votes/round-<iter>.json` must match across routes over a corpus.
|
|
181
|
+
- Rollback: flag; bash `--agents` arm and the heuristic council fallback both stay.
|
|
182
|
+
- Note: this is the first use of the Agent SDK (bundled binary). If Spike A found Bun can't spawn it, this phase and Phase 4 run under Node.
|
|
183
|
+
|
|
184
|
+
### Phase 4 - the RARV main loop on the Agent SDK (the hot path, highest risk, done last)
|
|
185
|
+
|
|
186
|
+
The single hardest site (A1). Replace `shellRun(["claude", ...stream-json...])` in `providers.ts` `claudeProvider()` with `query()`.
|
|
187
|
+
|
|
188
|
+
- Files: `loki-ts/src/runner/providers.ts` (claude mainLoop path → `sdk_invoker.ts` agentic path), `loki-ts/src/runner/sdk_stream_parser.ts` (new - translates the embedded Python stream-json parser: writes `.loki/state/agents.json`, `.loki/events.jsonl` hook events, `.loki/metrics/result-cost-<iter>.json`), `bin/loki` (route `start` to Bun when `LOKI_SDK_LOOP=1`).
|
|
189
|
+
- KEEP unchanged: `run_autonomous` outer loop, `build_prompt()` (parity-locked; consumed as `query({prompt})`), all 8 quality gates + evidence/checklist/heldout/assumption gates, the completion council, `.loki/` state machine, the stateless-per-iteration session design. **RARV-C is NOT replaced by any SDK primitive** (verified: no `outcome`/iterate-until-done primitive in the Agent SDK; `maxTurns`/`maxBudgetUsd`/`taskBudget` bound ONE call, not the grader loop).
|
|
190
|
+
- Flag mapping: `--model`→`model`, `--effort`→`effort`, `--append-system-prompt`→`systemPrompt:{preset:'claude_code',append}`, `--setting-sources`→`settingSources`, `--include-partial-messages`→`includePartialMessages`, `--dangerously-skip-permissions`→`permissionMode:'bypassPermissions'`, `--session-id`/`--resume`/`--fork-session`→`sessionId`/`resume`/`forkSession`. `--max-budget-usd`→ keep the deterministic budget gate + `maxBudgetUsd` if verified (Spike C). `--fallback-model`→`fallbackModel` if verified, else catch-and-retry.
|
|
191
|
+
- Enhancement to fold in: split the inert `[CACHE_BREAKPOINT]` in `build_prompt` and set real `cache_control` on the stable prefix (the migration is the moment this becomes possible).
|
|
192
|
+
- Parity proof: run a fixed PRD corpus through both routes (`LOKI_SDK_LOOP=0` vs `=1`); compare per-iteration cost/usage, gate outcomes, and completion decision. This is the phase that needs the `sdlc-fleet` council (3 Opus/Sonnet reviewers, unanimous APPROVE) per `CLAUDE.md`.
|
|
193
|
+
- Rollback: `LOKI_SDK_LOOP=0` or `LOKI_LEGACY_BASH=1` → `start` falls through to bash. Bash route stays the canonical route until this is SDK-proven across the discriminator corpus.
|
|
194
|
+
|
|
195
|
+
### Phase 5 - agentic exec sites + probe/cleanup deletion
|
|
196
|
+
|
|
197
|
+
- Migration/heal execs (C5/C6/C7) → Agent SDK `query()` (same agentic path as Phase 4), behind flags, bash arms intact.
|
|
198
|
+
- Delete the now-dead capability layer: `lib/claude-flags.sh` `loki_claude_flag_supported`, the `claude --help` cache, ~12 `command -v claude` probes, `claude --version` calls, `pkill claude` - but ONLY on the SDK route; the bash route still needs them until every phase is proven and the bash route is retired (a later decision, not v8).
|
|
199
|
+
- Rollback: flags per site.
|
|
200
|
+
|
|
201
|
+
### Phase 6 (later arc, NOT v8) - Cluster D + bash-route retirement
|
|
202
|
+
|
|
203
|
+
`ultrareview`/`ultracode`/`remote-control` reimplementation (or documented binary carve-out), and the eventual removal of the bash route once all SDK phases are proven in production. Explicitly out of v8 scope.
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
207
|
+
## 5. Per-phase: bash vs SDK vs parity vs rollback (summary)
|
|
208
|
+
|
|
209
|
+
| Phase | Stays on bash (fallback) | Moves to SDK | Parity proof | Rollback flag |
|
|
210
|
+
|---|---|---|---|---|
|
|
211
|
+
| 1 | done-recog `claude`+deterministic | raw SDK `messages.parse` | verdict object match over corpus | `LOKI_SDK_DONE_RECOG` |
|
|
212
|
+
| 2 | all six judge `case` arms (+ codex/cline/aider) | raw SDK judges | per-cluster verdict/text match | per-cluster flag |
|
|
213
|
+
| 3 | `--agents` + heuristic council | Agent SDK `agents` | votes/round-N.json match | council SDK flag |
|
|
214
|
+
| 4 | full bash RARV route | Agent SDK `query()` loop | cost/gate/completion match over PRD corpus | `LOKI_SDK_LOOP` / `LOKI_LEGACY_BASH` |
|
|
215
|
+
| 5 | migration/heal `case` arms | Agent SDK `query()` | display/output equivalence | per-site flag |
|
|
216
|
+
|
|
217
|
+
In every phase the codex/cline/aider `case` arms are untouched, and bash is the live route until the SDK route passes parity in `local-ci.sh`.
|
|
218
|
+
|
|
219
|
+
---
|
|
220
|
+
|
|
221
|
+
## 6. RARV-C / council / memory → SDK primitives (honest mapping)
|
|
222
|
+
|
|
223
|
+
| Component | Verdict | Grounding |
|
|
224
|
+
|---|---|---|
|
|
225
|
+
| RARV-C outer loop (`run_autonomous`) | **KEEP** - no SDK primitive replaces it | No `outcome`/iterate-until-grader-says-done in the Agent SDK (verified live). Managed Agents' Outcomes (`user.define_outcome` + rubric, hosted REST) is the nearest analog but is a **different product** - hosting completion on Anthropic's grader would cede Loki's council. Do NOT wire it. |
|
|
226
|
+
| The 8 quality gates + evidence/checklist/heldout/assumption gates | **KEEP** - deterministic graders, zero SDK equivalent | Loki's trust moat. Already ported: `quality_gates.ts`. |
|
|
227
|
+
| `build_prompt()` | **KEEP verbatim** - SDK consumes its string | Parity-locked with `build_prompt.ts`. `[CACHE_BREAKPOINT]` becomes a real `cache_control` split (E). |
|
|
228
|
+
| Completion council dispatch | **WRAP** - `--agents`→`agents`, `--json-schema`→`outputFormat` | Cleanest SDK fit. Delete `VOTE:` regex fallback. KEEP threshold-floor/quorum/devil's-advocate. |
|
|
229
|
+
| `memory/` package (15 modules) | **KEEP** - richer than any SDK memory feature | SDK `AgentDefinition.memory` is a source selector, not task-aware top-k retrieval / episodic→semantic consolidation / anti-pattern retrieval / cross-project RAG. Retrieved text still flows into `prompt` (or `systemPrompt.append` for better prefix-cache). **Preserve `rag_injector.py` sanitization** - the SDK does not sanitize stored memory (prompt-injection property). |
|
|
230
|
+
| `memory/managed_memory/` | **KEEP** (already on `anthropic` SDK, default-OFF) | The one existing SDK touchpoint; gated on `LOKI_MANAGED_MEMORY=true`. Out of the critical path. |
|
|
231
|
+
| Session UUID + resume/fork | **WRAP** - field-for-field SDK options | KEEP stateless-per-iteration design. `resumeSessionAt`/`listSessions` are E wins. |
|
|
232
|
+
|
|
233
|
+
The founder's "SDK outcome may map onto RARV-C" hypothesis is **false per live docs**. The SDK gives a bounded tool-use loop; the iterate-until-council-says-done loop stays Loki's. This is a WRAP-the-invocation migration, not a re-architecture.
|
|
234
|
+
|
|
235
|
+
---
|
|
236
|
+
|
|
237
|
+
## 7. Provider-agnosticism (where Anthropic-only forces a claude-gated path)
|
|
238
|
+
|
|
239
|
+
Both SDKs are Anthropic-specific. 18 of 25 model-work sites already branch to codex/cline/aider in the same `case`. The v8 SDK port replaces ONLY the `claude)` arm; the sibling arms keep their bash invocation:
|
|
240
|
+
|
|
241
|
+
- Codex: `codex exec --sandbox workspace-write` (`CODEX_MODEL_REASONING_EFFORT` env). Ported model in `providers.ts` codex arm.
|
|
242
|
+
- Cline: `invoke_cline` / `cline -y`.
|
|
243
|
+
- Aider: `aider --message ... --yes-always`.
|
|
244
|
+
|
|
245
|
+
The provider loader gates the SDK path on `LOKI_PROVIDER=claude`. The 7 CLAUDE-ONLY sites (B7, C1, C3, D1-D4) have no cross-provider story today and gain none from v8. An OpenAI-compatible layer (Codex/Cline via a shared HTTP client) is a **LATER arc, explicitly out of scope here**. The bash route staying alive per-piece IS the multi-provider + rollback fallback until each SDK piece is proven.
|
|
246
|
+
|
|
247
|
+
---
|
|
248
|
+
|
|
249
|
+
## 8. Risks + unknowns (spikes needed)
|
|
250
|
+
|
|
251
|
+
All UNVERIFIED items from the research, plus what the phases surface:
|
|
252
|
+
|
|
253
|
+
1. **Bun + Agent SDK bundled binary (Spike A, gates Phase 3/4).** Whether Bun 1.3.13 resolves the platform-gated `optionalDependencies` and spawns the bundled `claude` binary cleanly is unverified. If not: loop under Node 22+, judges under Bun (raw SDK is pure JS, Bun-safe). Test before choosing the loop runtime.
|
|
254
|
+
2. **`--max-budget-usd` has NO raw-SDK equivalent (§2 #15).** Confirmed absent from `messages.create`. Agent SDK exposes `maxBudgetUsd` per research; verify in `sdk.d.ts@0.3.207` (Spike C). Regardless, keep Loki's deterministic cumulative budget gate (`budget.ts`) on both paths - do not delegate budget enforcement to the SDK.
|
|
255
|
+
3. **`--fallback-model` (§2 #16).** Agent SDK `fallbackModel` field reported present but not code-verified; raw SDK has no per-call fallback field for Opus/Sonnet (server-side `fallbacks` is Fable-5-only beta). Implement catch-and-retry on the judge path as the safe default; use `fallbackModel` on the loop path only after Spike C confirms it.
|
|
256
|
+
4. **UNVERIFIED SDK type shapes.** `SDKAssistantMessage`/`SDKPartialAssistantMessage` content-block shapes, `ThinkingConfig` union, `CanUseTool` options - the fields exist but inner shapes must be re-read from `sdk.d.ts@0.3.207` before writing the stream parser (Phase 4). The verified `.d.ts` is at `scratchpad/package/sdk.d.ts`.
|
|
257
|
+
5. **Only one valid `SdkBeta`** (`context-1m-2025-08-07`). Other API betas (compaction, fast-mode) are not first-class Agent SDK options. If a phase needs one, it may require the raw SDK path instead.
|
|
258
|
+
6. **Bedrock/Vertex/Foundry env pass-through** through the Agent SDK `env` option is documented at overview level, not shown as a `query()` example - smoke-test before promising enterprise cloud auth in the SaaS pitch.
|
|
259
|
+
7. **Cluster D (§2.1)** - `ultrareview`/`ultracode`/`remote-control` have no SDK equivalent. v8 keeps the binary carve-out; "zero binary" is hot-path-only until a later arc reimplements them.
|
|
260
|
+
8. **Managed Agents temptation.** It IS the hosted grader/outcome loop the founder hypothesized, but adopting it cedes Loki's council and re-platforms completion onto Anthropic's grader. Deliberately NOT wired in v8. Flag if the founder wants to reconsider - it is a strategy decision, not a migration step.
|
|
261
|
+
9. **Model-name drift.** The codebase memory carries placeholder model names; wire the verified live IDs (`claude-opus-4-8` / `claude-sonnet-5` / `claude-haiku-4-5`, §1.4). Re-verify against the live model catalog at implementation time (IDs evolve).
|
|
262
|
+
|
|
263
|
+
---
|
|
264
|
+
|
|
265
|
+
## 9. The first implementable work item (concrete enough to start)
|
|
266
|
+
|
|
267
|
+
**Phase 1: migrate done-recognition to the raw SDK behind `LOKI_SDK_DONE_RECOG=1`.**
|
|
268
|
+
|
|
269
|
+
Preconditions: run Spike A/B/C first (they are cheap throwaway scripts and de-risk the whole arc). Then:
|
|
270
|
+
|
|
271
|
+
1. `cd loki-ts && bun add @anthropic-ai/sdk` (pure JS, Bun-safe - no binary).
|
|
272
|
+
2. Create `loki-ts/src/runner/sdk_invoker.ts` exporting a judge function:
|
|
273
|
+
- Input: `{ prompt: string, schemaPath: string, model: string, effort: string, timeoutMs: number }`.
|
|
274
|
+
- Body: `const client = new Anthropic()` (reads `ANTHROPIC_API_KEY`); `client.messages.create({ model, max_tokens: 16000, output_config: { format: { type: 'json_schema', schema } }, messages: [{ role: 'user', content: prompt }] })`; parse the single text block as JSON (guaranteed valid by `output_config.format`). Model `claude-haiku-4-5`, effort `low`.
|
|
275
|
+
- Return the parsed object; on any error, throw so the caller falls to the deterministic path (inconclusive-safe, matching current behavior).
|
|
276
|
+
- Load the schema from `loki-ts/data/done-recognition-schema.json` (already exists; do not invent a schema).
|
|
277
|
+
3. In `autonomy/lib/done-recognition.sh`, add a branch at the top of the invoke helper: if `LOKI_SDK_DONE_RECOG=1` and `ANTHROPIC_API_KEY` is set, shell into `bun loki-ts/... internal done-recog-sdk "$prompt"` (a thin `internal` subcommand wrapping the judge function); otherwise keep the existing `claude -p --json-schema` → deterministic fallback chain exactly as-is.
|
|
278
|
+
4. Parity harness: a script that runs the same fixed set of `.loki` states through both `LOKI_SDK_DONE_RECOG=0` and `=1`, asserts the parsed `requirements` verdict object is equal, and cleans up. Add it to `scripts/local-ci.sh`'s bun-parity matrix.
|
|
279
|
+
5. Do NOT bump version, do NOT commit, do NOT touch any other site. One helper, one flag, one parity test.
|
|
280
|
+
|
|
281
|
+
This proves the entire bridge (SDK auth from env, structured output via `output_config.format`, schema reuse, bash-fallback rollback, parity testing) on the single lowest-risk CLAUDE-ONLY site before anything touches the RARV hot path.
|
|
282
|
+
|
|
283
|
+
---
|
|
284
|
+
|
|
285
|
+
## Appendix: verified sources
|
|
286
|
+
|
|
287
|
+
- Agent SDK types: unpacked `@anthropic-ai/claude-agent-sdk@0.3.207` `sdk.d.ts` (6923 lines) at `scratchpad/package/sdk.d.ts`.
|
|
288
|
+
- Raw SDK: `@anthropic-ai/sdk@0.111.0`; `messages.parse` / `output_config.format` / `task_budget` / model IDs from the live claude-api skill (authoritative for API shapes).
|
|
289
|
+
- Installed `claude` CLI: v2.1.207.
|
|
290
|
+
- Existing TS runner: `loki-ts/src/runner/` (12,300 lines, parity-locked, NOT reached for `start` per `run.sh:17271`).
|
|
291
|
+
- Bash invocation surface: `autonomy/run.sh:17195-17369`, `lib/claude-flags.sh`, `lib/done-recognition.sh`, `lib/prd-enrich.sh`, `completion-council.sh`, `council-v2.sh`, `grill.sh`, `lib/voter-agents.sh`, `providers/claude.sh`.
|
|
292
|
+
- Existing rollback flag + router: `bin/loki` (`LOKI_LEGACY_BASH`, per-command Bun routing).
|
|
293
|
+
- Model IDs: live catalog - `claude-opus-4-8`, `claude-sonnet-5`, `claude-haiku-4-5`.
|
|
294
|
+
|
|
295
|
+
---
|
|
296
|
+
|
|
297
|
+
# Part B: Build-Ready Implementation Specs (per-phase, codeable)
|
|
298
|
+
|
|
299
|
+
Status: BUILD-READY. Written against verified source on this branch (`feature/v8-agent-sdk`).
|
|
300
|
+
This part supersedes the phase sketches in Section 4 where they conflict; Section 4 stays as the
|
|
301
|
+
strategic map. All line numbers re-verified against the working tree; re-grep before editing (they drift).
|
|
302
|
+
|
|
303
|
+
## B.0 Ground-truth delta (what already exists on this branch vs the Section-4 sketch)
|
|
304
|
+
|
|
305
|
+
The Section-4 phasing assumed nothing was wired. Verified on-branch, several pieces are ALREADY BUILT.
|
|
306
|
+
The specs below say "complete" or "extend", not "create from scratch", where that is the true state.
|
|
307
|
+
|
|
308
|
+
| Piece | Status on branch | Anchor |
|
|
309
|
+
|---|---|---|
|
|
310
|
+
| `judgeJson()` raw-SDK judge | DONE (+ unit test) | `loki-ts/src/runner/sdk_invoker.ts`; `loki-ts/tests/runner/sdk_invoker.test.ts` |
|
|
311
|
+
| `loki internal sdk-judge` bridge | DONE (fail-closed, exit 0/1/2/3) | `loki-ts/src/commands/internal_sdk_judge.ts`; dispatch `loki-ts/src/cli.ts:264` |
|
|
312
|
+
| `internal` routes to Bun | DONE | `bin/loki:263` |
|
|
313
|
+
| done-recognition SDK branch (`LOKI_SDK_DONE_RECOG=1`) | DONE (runs BEFORE `command -v claude` guard) | `autonomy/lib/done-recognition.sh:54-85` |
|
|
314
|
+
| council-v2 SDK branch (`LOKI_SDK_COUNCIL_V2=1`) | DONE | `autonomy/council-v2.sh:284-305` |
|
|
315
|
+
| done-recognition schema | DONE | `loki-ts/data/done-recognition-schema.json` |
|
|
316
|
+
| council-v2 / code-review schemas | DONE | `loki-ts/data/{council-v2,code-review}-schema.json` |
|
|
317
|
+
| `cr-rematerialize.py` (JSON envelope -> legacy VERDICT/FINDINGS text) | DONE | `autonomy/lib/cr-rematerialize.py` |
|
|
318
|
+
| structural bridge test (no billable call) | DONE | `tests/test-sdk-done-recog-bridge.sh` (85 lines) |
|
|
319
|
+
| **verdict-EQUALITY parity over a real corpus** | **MISSING** | -- Phase 1 gap |
|
|
320
|
+
| **`cache_control` on the judge prefix** | **MISSING** (`judgeJson` sets none; `sdk_invoker.ts` has zero `cache`) | -- Phase 1 enhancement |
|
|
321
|
+
| code-review site SDK branch (Phase 2c) | MISSING | `autonomy/run.sh:11027-11048` |
|
|
322
|
+
| completion-council members SDK branch (Phase 2b) | MISSING | `autonomy/completion-council.sh` |
|
|
323
|
+
| council Agent-SDK dispatch (Phase 3) | MISSING | `autonomy/lib/voter-agents.sh` |
|
|
324
|
+
| RARV loop on Agent SDK (Phase 4) | MISSING | `loki-ts/src/runner/providers.ts` |
|
|
325
|
+
|
|
326
|
+
**Honest reframing of the deliverable:** Phase 1's *code path* is already end-to-end. Phase 1 is NOT
|
|
327
|
+
"done" because it has (a) no verdict-equality proof against real model output, and (b) does not yet
|
|
328
|
+
realize the cost thesis (no `cache_control`). B.1 closes exactly those two gaps. This is the correct,
|
|
329
|
+
non-fabricated status.
|
|
330
|
+
|
|
331
|
+
## B.1 Phase 1 COMPLETE -- prove + optimize the done-recognition bridge
|
|
332
|
+
|
|
333
|
+
The branch runs the SDK judge; two things make it *complete and testable*.
|
|
334
|
+
|
|
335
|
+
### B.1.1 (gap) Verdict-equality parity test over a corpus of `.loki` states
|
|
336
|
+
|
|
337
|
+
The existing `tests/test-sdk-done-recog-bridge.sh` is STRUCTURAL only (asserts wiring, fail-closed,
|
|
338
|
+
branch ordering; no model call). It does not prove the SDK route and the claude route reach the SAME
|
|
339
|
+
verdict. That proof is the whole point of "parity + rollback per phase".
|
|
340
|
+
|
|
341
|
+
**Design (billable, opt-in, cheap).** The judge is a pure `prompt -> schema-JSON` function. Parity does
|
|
342
|
+
NOT require running the full `reuse_done_recognition_gate` twice; it requires: for each corpus prompt,
|
|
343
|
+
call `loki internal sdk-judge` (SDK route) and `claude -p --json-schema` (claude route), then compare the
|
|
344
|
+
verdict the SAME downstream parser derives from each. Compare the DERIVED verdict, not raw JSON bytes
|
|
345
|
+
(two valid judgments can differ in prose but must not differ in verdict/coverage).
|
|
346
|
+
|
|
347
|
+
New file `tests/corpus/done-recog/*.json` (3-5 fixtures): each is `{ "prompt": "<full DR prompt>",
|
|
348
|
+
"expect": "done|incomplete|inconclusive" }`. Build them by capturing real prompts from
|
|
349
|
+
`reuse_done_recognition_gate` on 3-5 tiny throwaway projects (one already-satisfied, one half-built,
|
|
350
|
+
one unbuildable) via a `LOKI_DONE_RECOG_DUMP_PROMPT=<path>` hook added at `done-recognition.sh:335`
|
|
351
|
+
(one line: `[ -n "${LOKI_DONE_RECOG_DUMP_PROMPT:-}" ] && printf '%s' "$prompt" > "$LOKI_DONE_RECOG_DUMP_PROMPT"`).
|
|
352
|
+
|
|
353
|
+
New file `tests/test-sdk-done-recog-parity.sh` (billable; gated on `RUN_BILLABLE=1` + `ANTHROPIC_API_KEY`,
|
|
354
|
+
SKIP otherwise so `local-ci.sh` stays free):
|
|
355
|
+
```
|
|
356
|
+
for f in tests/corpus/done-recog/*.json; do
|
|
357
|
+
prompt="$(jq -r .prompt "$f")"; expect="$(jq -r .expect "$f")"
|
|
358
|
+
printf '%s' "$prompt" > "$pf"
|
|
359
|
+
sdk_json="$("$LOKI" internal sdk-judge --prompt-file "$pf" --schema-file "$SCHEMA" \
|
|
360
|
+
--model claude-haiku-4-5 --effort low)" # SDK route
|
|
361
|
+
cli_env="$(CAVEMAN_DEFAULT_MODE=off claude --dangerously-skip-permissions -p "$prompt" \
|
|
362
|
+
--json-schema "$(cat "$SCHEMA")" --output-format json)" # claude route
|
|
363
|
+
cli_json="$(printf '%s' "$cli_env" | jq -c '.structured_output // (.result|fromjson)')"
|
|
364
|
+
# derive the verdict from EACH via the SAME parser done-recognition.sh uses (DR_PARSE_EOF).
|
|
365
|
+
sdk_v="$(printf '%s' "$sdk_json" | LOKI_DR_TESTS=... python3 dr_parse.py)"
|
|
366
|
+
cli_v="$(printf '%s' "$cli_json" | LOKI_DR_TESTS=... python3 dr_parse.py)"
|
|
367
|
+
[ "$sdk_v" = "$cli_v" ] && [ "$sdk_v" = "$expect" ] && ok || bad
|
|
368
|
+
done
|
|
369
|
+
```
|
|
370
|
+
Refactor the inline `DR_PARSE_EOF` python heredoc (`done-recognition.sh:356-645`) into a standalone
|
|
371
|
+
`autonomy/lib/dr-parse.py` invoked by BOTH the shell gate and this test (one parser, no divergence --
|
|
372
|
+
same discipline `cr-rematerialize.py` already follows). This refactor is itself parity-preserving:
|
|
373
|
+
the gate's behavior is unchanged, the heredoc body just moves to a file it now execs.
|
|
374
|
+
|
|
375
|
+
Wire into `scripts/local-ci.sh`: run structural test always; run parity test only when `RUN_BILLABLE=1`.
|
|
376
|
+
Parity acceptance: SDK-derived verdict == claude-derived verdict on 100% of the corpus (verdict axis;
|
|
377
|
+
NOT prose). A single divergence blocks the phase.
|
|
378
|
+
|
|
379
|
+
### B.1.2 (enhancement) realize the cost thesis: `cache_control` on the judge prefix
|
|
380
|
+
|
|
381
|
+
VERIFIED: `judgeJson()` sets NO `cache_control` (`sdk_invoker.ts` has zero `cache` tokens), and the
|
|
382
|
+
`[CACHE_BREAKPOINT]` marker (`build_prompt.ts:1362,1425`; `run.sh:15280,15327`) is inert documentation.
|
|
383
|
+
The raw SDK is where a real breakpoint first becomes settable. The done-recognition prompt is
|
|
384
|
+
prefix-stable across the requirements it judges (system rubric + PRD + prior-claims header are constant;
|
|
385
|
+
only the tail varies), so a 1h ephemeral breakpoint on the stable head is a genuine, cited cost win
|
|
386
|
+
(cache-read 0.1x, 1h-write 2.0x, break-even 3 requests -- verified via the claude-api catalog).
|
|
387
|
+
|
|
388
|
+
Extend `JudgeParams` + `judgeJson()` (`sdk_invoker.ts`) with an OPTIONAL `system` broken into a cached
|
|
389
|
+
prefix. Two safe ways, both GA (no beta header -- verified: `output_config.format`/`effort`/`cache_control`
|
|
390
|
+
all GA on Opus 4.8 / Sonnet 5 / Haiku 4.5):
|
|
391
|
+
```ts
|
|
392
|
+
// Option A (precise): put the stable rubric as a system TextBlockParam with a 1h breakpoint.
|
|
393
|
+
system: systemPrefix
|
|
394
|
+
? [{ type: "text", text: systemPrefix, cache_control: { type: "ephemeral", ttl: "1h" } }]
|
|
395
|
+
: undefined,
|
|
396
|
+
// Option B (simplest): top-level cache_control auto-marks the last cacheable block.
|
|
397
|
+
```
|
|
398
|
+
Use Option A: the judge's stable rubric is a natural, explicit prefix. Gate min-prefix: the Haiku/Opus
|
|
399
|
+
minimum cacheable prefix is 4096 tokens (catalog-verified) -- the ~8K rubric+PRD prefix clears it;
|
|
400
|
+
`sdk-judge` should pass the rubric via a new `--system-file` flag so the bash caller (which already holds
|
|
401
|
+
the rubric text) supplies it. Surface `usage.cache_read_input_tokens` on stderr behind
|
|
402
|
+
`LOKI_SDK_JUDGE_DEBUG=1` so the parity test can ASSERT cache hits on request 2+ (the observable proof the
|
|
403
|
+
optimization fired -- if it's 0 across identical-prefix calls, a silent invalidator is present).
|
|
404
|
+
|
|
405
|
+
**Parity guard for caching:** `cache_control` MUST NOT change the verdict (it only changes token
|
|
406
|
+
accounting). The B.1.1 corpus test already proves verdict-equality; run it once with caching on and once
|
|
407
|
+
off (`LOKI_SDK_JUDGE_CACHE=0`) and assert the verdict axis is identical across both -- caching is a
|
|
408
|
+
transparent cost lever, never a correctness lever.
|
|
409
|
+
|
|
410
|
+
### B.1.3 rollback
|
|
411
|
+
`unset LOKI_SDK_DONE_RECOG` (already the escape hatch). One env var, single CLAUDE-ONLY helper, zero
|
|
412
|
+
blast radius. `LOKI_SDK_JUDGE_CACHE=0` independently disables just the cache breakpoint.
|
|
413
|
+
|
|
414
|
+
---
|
|
415
|
+
|
|
416
|
+
## B.2 Phase 2 -- remaining one-shot judges on `judgeJson`, one cluster per flag
|
|
417
|
+
|
|
418
|
+
Each cluster: the SITE, its SCHEMA, the regex/sed PARSER it deletes, the FLAG, the PARITY proof.
|
|
419
|
+
All reuse the SAME `loki internal sdk-judge` bridge (already built) -- Phase 2 adds NO new TS, only new
|
|
420
|
+
`.sh` branches (mirroring the done-recognition/council-v2 pattern verbatim) plus, for code-review, reuse
|
|
421
|
+
of the existing `cr-rematerialize.py`.
|
|
422
|
+
|
|
423
|
+
### 2a. council-v2 reviewers -- ALREADY WIRED, needs its parity test
|
|
424
|
+
- Site: `autonomy/council-v2.sh:284-305` (branch DONE, `LOKI_SDK_COUNCIL_V2=1`).
|
|
425
|
+
- Schema: `loki-ts/data/council-v2-schema.json` (`verdict` enum APPROVE|REJECT + `issues[]`).
|
|
426
|
+
- Parser deleted-on-success: the sed-carve at `council-v2.sh:350+` (the `structured_output // fromjson`
|
|
427
|
+
python slice). On SDK success the branch `echo "$_c2_sdk_out" > "$output_file"; return 0` bypasses it.
|
|
428
|
+
- Gap: same as B.1.1 -- add `tests/corpus/council-v2/*.json` + a billable parity test asserting the SDK
|
|
429
|
+
verdict token equals the claude verdict token over the corpus. Flag already exists.
|
|
430
|
+
- Rollback: `unset LOKI_SDK_COUNCIL_V2`.
|
|
431
|
+
|
|
432
|
+
### 2b. completion-council members + devil's advocate (VOTE: regex -> schema)
|
|
433
|
+
- Site: `autonomy/completion-council.sh` -- the member/contrarian invokers whose output feeds
|
|
434
|
+
`_council_parse_vote` (the word-bounded `VOTE:APPROVE|REJECT|CANNOT_VALIDATE` regex, `:1002-1004`;
|
|
435
|
+
contrarian at `council_devils_advocate`, consumed `:814-816`).
|
|
436
|
+
- Schema (NEW): `loki-ts/data/completion-vote-schema.json` --
|
|
437
|
+
`{ verdict: enum["APPROVE","REJECT","CANNOT_VALIDATE"], reasoning: string }`. Model MUST emit the same
|
|
438
|
+
three tokens the regex accepts, so the DERIVED vote is identical.
|
|
439
|
+
- Parser deleted-on-success: nothing is *deleted* (the regex stays as fail-closed fallback); on SDK
|
|
440
|
+
success the branch writes a `VOTE:<verdict>` line (re-materialized from JSON, one `printf`) so
|
|
441
|
+
`_council_parse_vote` stays byte-identical -- the cr-rematerialize discipline applied to a 1-token
|
|
442
|
+
verdict. This is the safest possible adoption: the downstream parser never changes.
|
|
443
|
+
- Flag: `LOKI_SDK_COMPLETION_COUNCIL=1`. Branch runs BEFORE the `command -v claude` guard (binary-free),
|
|
444
|
+
identical shape to `done-recognition.sh:63-85`.
|
|
445
|
+
- Parity: corpus of captured member prompts; assert SDK-derived vote == claude-derived vote == expected,
|
|
446
|
+
AND that a hedged/empty judgment still fails closed to REJECT on BOTH routes (the anti-sycophancy
|
|
447
|
+
invariant at `:824` must not regress).
|
|
448
|
+
- Rollback: `unset LOKI_SDK_COMPLETION_COUNCIL`.
|
|
449
|
+
|
|
450
|
+
### 2c. code-review 3-reviewer (the highest-value delete: reuse cr-rematerialize)
|
|
451
|
+
- Site: `autonomy/run.sh:11027-11048` (the `--json-schema` claude branch + text fallback).
|
|
452
|
+
- Schema: `loki-ts/data/code-review-schema.json` (`verdict` PASS|FAIL + `findings[].severity`).
|
|
453
|
+
- Parser reused, NOT deleted: `autonomy/lib/cr-rematerialize.py` already converts a JSON envelope to
|
|
454
|
+
legacy `VERDICT: X\nFINDINGS:\n- [sev] desc` text AND enforces the cross-field T1 safety rule
|
|
455
|
+
(any Critical/High => force FAIL). The SDK branch feeds `judgeJson`'s object into the SAME
|
|
456
|
+
rematerializer, so `_classify_verdict` / `_severity_is_blocking` / `_count_nonblocking_findings` /
|
|
457
|
+
mergeability / DA arm / aggregate.json stay byte-identical. The ONE change vs 2a/2b: `sdk-judge` prints
|
|
458
|
+
the BARE payload object (top-level `verdict`+`findings`), but `cr-rematerialize.py` reads an ENVELOPE --
|
|
459
|
+
VERIFIED at `cr-rematerialize.py:49-58` it looks only at `env["structured_output"]` then `env["result"]`
|
|
460
|
+
and raises `ValueError(5)` on a bare payload. So a raw `judgeJson` object would fail-close (safe, but
|
|
461
|
+
the SDK path never engages). Fix WITHOUT touching the rematerializer: wrap the payload as an envelope in
|
|
462
|
+
bash before piping it, so the adapter stays byte-identical:
|
|
463
|
+
```
|
|
464
|
+
if [ "${LOKI_SDK_CODE_REVIEW:-0}" = "1" ]; then
|
|
465
|
+
_cr_sdk_out="$("$LOKI" internal sdk-judge --prompt-file "$pf" \
|
|
466
|
+
--schema-file "$_cr_schema" --model "${LOKI_SDK_JUDGE_MODEL:-claude-haiku-4-5}" --effort low)"
|
|
467
|
+
if [ -n "$_cr_sdk_out" ]; then
|
|
468
|
+
# wrap the bare payload as {structured_output: <payload>, stop_reason: "tool_use"}
|
|
469
|
+
# so cr-rematerialize.py (:49) reads it unchanged; jq -n keeps it valid on any input.
|
|
470
|
+
_cr_env="$(printf '%s' "$_cr_sdk_out" | jq -c '{structured_output: ., stop_reason: "tool_use"}' 2>/dev/null)"
|
|
471
|
+
if [ -n "$_cr_env" ] \
|
|
472
|
+
&& _LOKI_CR_JSON="$_cr_env" _LOKI_CR_OUT="$review_output" python3 "$_cr_remat"; then
|
|
473
|
+
return 0
|
|
474
|
+
fi
|
|
475
|
+
fi
|
|
476
|
+
# fall through to the existing --json-schema claude branch (fail-closed)
|
|
477
|
+
fi
|
|
478
|
+
```
|
|
479
|
+
runs BEFORE the claude branch; on any miss (empty out, malformed, non-zero rematerialize) falls through
|
|
480
|
+
unchanged. NEVER a PASS on a miss (the rematerializer is fail-closed by contract: exit 5 on no payload,
|
|
481
|
+
exit 6 on missing verdict, and force-FAIL on any Critical/High finding). Alternative (if a new dep-free
|
|
482
|
+
seam is preferred over `jq`): `sdk-judge` could grow an `--envelope` flag that prints
|
|
483
|
+
`{"structured_output": <obj>, "stop_reason": "tool_use"}` itself -- but the bash wrap above needs no TS
|
|
484
|
+
change and is the lazier correct fix.
|
|
485
|
+
- Flag: `LOKI_SDK_CODE_REVIEW=1`.
|
|
486
|
+
- Parity: corpus of captured review prompts + diffs; assert the re-materialized legacy text (hence the
|
|
487
|
+
classified verdict, blocking severity, and mergeability score) is IDENTICAL across SDK and claude
|
|
488
|
+
routes. This is the strongest parity signal in Phase 2 because both routes converge on the exact same
|
|
489
|
+
`cr-rematerialize.py` output.
|
|
490
|
+
- Rollback: `unset LOKI_SDK_CODE_REVIEW`.
|
|
491
|
+
|
|
492
|
+
### 2d. aux helpers (captured-text, no schema): PRD enrich (C1), grill (C2), USAGE regen (C3), doc-gen (C4)
|
|
493
|
+
- Sites: `autonomy/lib/prd-enrich.sh:_loki_prd_enrich_invoke:43`, `autonomy/grill.sh`, USAGE/doc-gen.
|
|
494
|
+
- These return FREE TEXT, not a gating verdict, so they do NOT go through `judgeJson` (schema path).
|
|
495
|
+
They need a sibling `judgeText()` in `sdk_invoker.ts`: same client, `messages.create` with NO
|
|
496
|
+
`output_config.format`, return the concatenated text blocks (or null). Add `loki internal sdk-text`
|
|
497
|
+
wrapping it. Lower priority: no verdict to protect, failure is cosmetic (a worse PRD, not a wrong gate).
|
|
498
|
+
- Flag: `LOKI_SDK_AUX=1` (one flag for the whole low-risk cluster).
|
|
499
|
+
- Parity: text sites have no deterministic verdict; "parity" = a smoke assertion that both routes return
|
|
500
|
+
non-empty text of comparable length + that fail-closed still yields the deterministic fallback. Do NOT
|
|
501
|
+
claim byte-parity on free text.
|
|
502
|
+
- Rollback: `unset LOKI_SDK_AUX`.
|
|
503
|
+
|
|
504
|
+
### Phase 2 provider-agnosticism (unchanged, load-bearing)
|
|
505
|
+
Every branch above is inside the `claude)` / `PROVIDER_NAME=claude` arm ONLY. The codex/cline/aider
|
|
506
|
+
`case` arms are UNTOUCHED. The SDK path is additionally gated by `judgeJson` returning null when
|
|
507
|
+
`ANTHROPIC_API_KEY` is absent (fail-closed to the claude arm). Bash stays the live route until each
|
|
508
|
+
cluster's parity test passes in `local-ci.sh` under `RUN_BILLABLE=1`.
|
|
509
|
+
|
|
510
|
+
---
|
|
511
|
+
|
|
512
|
+
## B.3 Phase 3 -- the council: Batch API on the raw SDK, NOT the Agent SDK. DECISION + why.
|
|
513
|
+
|
|
514
|
+
The Section-4 sketch offered "Agent SDK `agents` dispatch OR Batch API -- decide". **Decision: neither the
|
|
515
|
+
Agent SDK nor the current sync path; use the raw-SDK parallel-`create` fan-out for the ON-critical-path
|
|
516
|
+
council, and reserve the Batch API for the OFF-critical-path judge cohort. Reasoning grounded in the
|
|
517
|
+
verified cost/latency facts:**
|
|
518
|
+
|
|
519
|
+
1. The council reviewers are pure one-shot JUDGES (each = one prompt -> one schema verdict). They do NOT
|
|
520
|
+
need the Agent SDK's tool loop / filesystem / subagents. Routing them through the Agent SDK would spawn
|
|
521
|
+
the ~240MB binary per council -- strictly worse on both cost (binary) and the deploy thesis (not
|
|
522
|
+
binary-free). The whole point of the two-SDK split is that judges stay on the raw HTTPS SDK.
|
|
523
|
+
2. **Batch API is a COST lever, not a latency lever** (verified: 50% off via `service_tier:"batch"`, but
|
|
524
|
+
async with up to 24h SLA, results keyed by `custom_id`). The completion council is ON the critical path
|
|
525
|
+
(the run blocks on its verdict), so batch's 24h SLA is disqualifying THERE. For N independent reviewers
|
|
526
|
+
on the critical path, N parallel `judgeJson` promises against one in-process client (`Promise.all`) is
|
|
527
|
+
the right primitive: one process, N concurrent HTTPS calls, wall-clock = max(reviewer), no binary.
|
|
528
|
+
3. **Where Batch DOES win:** the OFF-critical-path judge cohort -- done-recognition sweeps run ahead of
|
|
529
|
+
time, offline re-scoring, doc-coverage / magic-debate gates that are advisory and not blocking. Those
|
|
530
|
+
can be submitted as one `messages.batches.create({requests:[...]})` at `service_tier:"batch"` for -50%,
|
|
531
|
+
because nothing blocks on them synchronously. This is a SEPARATE, later, opt-in lever
|
|
532
|
+
(`LOKI_SDK_BATCH_ADVISORY=1`), explicitly NOT part of the blocking council.
|
|
533
|
+
|
|
534
|
+
**Phase 3 build (parallel raw-SDK council, keep Loki's decision engine):**
|
|
535
|
+
- New `judgeCouncil(reviewers: JudgeParams[]): Promise<(Record|null)[]>` in `sdk_invoker.ts`:
|
|
536
|
+
`Promise.all(reviewers.map(judgeJson))`. Each reviewer carries its OWN system prefix (its lens) with the
|
|
537
|
+
SHARED evidence+PRD prefix carrying the 1h `cache_control` breakpoint -- so reviewer 2..N read the cache
|
|
538
|
+
reviewer 1 wrote (cross-reviewer cache reuse within one run; the exact cost mechanism from Axis 1, now
|
|
539
|
+
real). Fire reviewer 1, await its first response, then fire 2..N so they hit the warm cache (the catalog
|
|
540
|
+
caveat: a cache entry is only readable after the first response BEGINS -- N truly-parallel calls each
|
|
541
|
+
pay full price; serialize the first, then fan out).
|
|
542
|
+
- New `loki internal sdk-council --reviewers-file <json>` wrapping it; `autonomy/lib/voter-agents.sh` gets
|
|
543
|
+
an SDK branch behind `LOKI_SDK_COUNCIL=1` that builds the reviewers JSON (same content it builds for
|
|
544
|
+
`claude --agents` today) and writes each verdict to the same `.loki/council/votes/round-<iter>.json`
|
|
545
|
+
the bash path writes.
|
|
546
|
+
- **KEEP unchanged:** the decision engine -- effective-threshold floor, exact-quorum gate,
|
|
547
|
+
devil's-advocate override, transcript writing. None of that is an SDK concept; the SDK only supplies the
|
|
548
|
+
N verdicts. This is the same "WRAP the invocation, keep RARV-C" discipline as everywhere else.
|
|
549
|
+
- Parity: `.loki/council/votes/round-<iter>.json` must match across routes over a corpus; the final
|
|
550
|
+
council DECISION (approve/block + quorum) must be identical. Rollback: `unset LOKI_SDK_COUNCIL`.
|
|
551
|
+
- Cache-proof: assert reviewer 2..N show `cache_read_input_tokens > 0` (the Axis-1 mechanism, now
|
|
552
|
+
observable). If they don't, the shared-prefix breakpoint has a silent invalidator (per-reviewer bytes
|
|
553
|
+
leaking into the shared head) -- fix by moving the reviewer lens strictly AFTER the breakpoint.
|
|
554
|
+
|
|
555
|
+
Note: Phase 3 uses the raw SDK, so it does NOT depend on Spike A (Bun-can-spawn-the-Agent-binary). It
|
|
556
|
+
ships independently of the Agent SDK. The `--agents` bash path and heuristic council fallback both stay.
|
|
557
|
+
|
|
558
|
+
---
|
|
559
|
+
|
|
560
|
+
## B.4 Phase 4 (SPEC ONLY -- highest risk) -- RARV loop on the Agent SDK; what the stream parser MUST emit
|
|
561
|
+
|
|
562
|
+
This replaces `shellRun(["claude", ...stream-json...])` in `providers.ts` `claudeProvider()` with the
|
|
563
|
+
Agent SDK `query()`. It is the ONLY site that needs `@anthropic-ai/claude-agent-sdk` (must be
|
|
564
|
+
`bun add`-ed first -- NOT yet installed; only `@anthropic-ai/sdk` is in `loki-ts/package.json`). Spec only:
|
|
565
|
+
no code this arc.
|
|
566
|
+
|
|
567
|
+
### B.4.1 the invocation
|
|
568
|
+
```ts
|
|
569
|
+
import { query } from "@anthropic-ai/claude-agent-sdk"; // 0.3.208, verified types
|
|
570
|
+
const q = query({
|
|
571
|
+
prompt: builtPrompt, // from build_prompt() VERBATIM (parity-locked)
|
|
572
|
+
options: {
|
|
573
|
+
model, effort, // <- --model / --effort tier
|
|
574
|
+
permissionMode: "bypassPermissions", // <- --dangerously-skip-permissions
|
|
575
|
+
allowDangerouslySkipPermissions: true, // REQUIRED alongside bypassPermissions
|
|
576
|
+
systemPrompt: { type: "preset", preset: "claude_code", append: autonomyAppend }, // <- --append-system-prompt
|
|
577
|
+
settingSources: ["project"], // <- load CLAUDE.md (omit=all, []=isolate)
|
|
578
|
+
includePartialMessages: true, // <- --include-partial-messages (dashboard stream)
|
|
579
|
+
maxTurns, maxBudgetUsd, // budget guards (see B.4.3)
|
|
580
|
+
sessionId, resume, forkSession, // <- --session-id / --resume / --fork-session
|
|
581
|
+
mcpServers, // <- Loki's mcp/server.py (34 tools) unchanged
|
|
582
|
+
executable: "bun", // Bun auto-selected (isBun()); pin explicitly
|
|
583
|
+
},
|
|
584
|
+
});
|
|
585
|
+
for await (const m of q) { sdkStreamParser.consume(m); } // typed SDKMessage union
|
|
586
|
+
```
|
|
587
|
+
|
|
588
|
+
### B.4.2 what `sdk_stream_parser.ts` MUST emit (the one genuinely new piece)
|
|
589
|
+
It translates the typed `SDKMessage` stream into the EXACT `.loki/` artifacts the embedded Python
|
|
590
|
+
stream-json parser writes today (`run.sh:17371+`), so every downstream consumer is byte-compatible:
|
|
591
|
+
|
|
592
|
+
| SDKMessage variant (verified 0.3.208 `sdk.d.ts`) | Emit to `.loki/` | Notes |
|
|
593
|
+
|---|---|---|
|
|
594
|
+
| `type:"system", subtype:"init"` | `.loki/state/agents.json` seed (session_id, model, tools, mcp_servers, permissionMode) | first frame |
|
|
595
|
+
| `type:"assistant"` (`message`: full Beta message w/ content blocks + usage) | append content to the iteration transcript; accumulate `usage` | `subagent_type`/`task_description` when present |
|
|
596
|
+
| `type:"stream_event"` (`SDKPartialAssistantMessage`, only when `includePartialMessages`) | dashboard live stream (`.loki/events.jsonl`) | best-effort; ignore if absent |
|
|
597
|
+
| `type:"result"` (`SDKResultMessage`) | **`.loki/metrics/result-cost-<iter>.json`** <- `total_cost_usd`, `usage`, `modelUsage`, `num_turns`, `duration_ms` | THE accounting frame; also the completion signal |
|
|
598
|
+
| hook events (`SubagentStart/Stop`, `PreCompact/PostCompact`, `PreToolUse/PostToolUse`) | `.loki/events.jsonl` hook records | map to the bash hook-event shape |
|
|
599
|
+
| any other `type`/`subtype` (task_progress, rate_limit_event, plugin_install, prompt_suggestion, ...) | IGNORE | parser MUST tolerate unknown variants -- do not throw |
|
|
600
|
+
|
|
601
|
+
Load-bearing rules for the parser:
|
|
602
|
+
- Key completion on `type==="result"`; key content on `type==="assistant"`. Everything else is telemetry
|
|
603
|
+
or ignorable.
|
|
604
|
+
- `SDKResultMessage.subtype` is the STOP signal your loop reads: `success` | `error_during_execution` |
|
|
605
|
+
`error_max_turns` | `error_max_budget_usd` | `error_max_structured_output_retries`. Map each to Loki's
|
|
606
|
+
existing iteration outcome (max_budget/max_turns -> the same PAUSE/stop the bash route reaches).
|
|
607
|
+
- **RARV-C is NOT replaced** (verified: no outcome/iterate-until-done primitive in the Agent SDK).
|
|
608
|
+
`query()` runs ONE agentic iteration (one tool-use loop, bounded by maxTurns/maxBudgetUsd). Loki's
|
|
609
|
+
`run_autonomous` outer loop, the 8 gates, the completion council, and the `.loki/` state machine ALL
|
|
610
|
+
stay. `maxTurns`/`maxBudgetUsd`/`error_max_*` bound one call, never the grader loop.
|
|
611
|
+
- `total_cost_usd` on the result frame REPLACES the price-table computation the raw-SDK judges need
|
|
612
|
+
(the Agent SDK returns dollars; the raw SDK returns tokens only -- note this asymmetry in accounting).
|
|
613
|
+
|
|
614
|
+
### B.4.3 budget: NO per-call USD cap on the raw SDK; keep the deterministic gate on BOTH
|
|
615
|
+
- Agent SDK exposes `maxBudgetUsd` (verified) -> wire `--max-budget-usd` to it.
|
|
616
|
+
- The raw-SDK judges have NO USD cap primitive (verified). KEEP Loki's cumulative `check_budget_limit`
|
|
617
|
+
PAUSE gate (`budget.ts`, already ported) on both routes -- do NOT delegate budget enforcement to the
|
|
618
|
+
SDK. `taskBudget:{total}` (@alpha, token budget + `task-budgets-2026-03-13` beta) is optional and NOT
|
|
619
|
+
required for v8.
|
|
620
|
+
|
|
621
|
+
### B.4.4 the cache enhancement finally lands
|
|
622
|
+
Split the inert `[CACHE_BREAKPOINT]` (`build_prompt.ts:1362,1425`) into a real `cache_control` on the
|
|
623
|
+
stable RARV/SDLC/constitution prefix (~6K tokens, catalog-verified >= the 4096 Opus min). This is the
|
|
624
|
+
Axis-1 dev-loop line item; it becomes settable ONLY at this migration. Guard: verdict/behavior must be
|
|
625
|
+
identical with the breakpoint on vs off (transparent cost lever).
|
|
626
|
+
|
|
627
|
+
### B.4.5 parity + rollback (the sdlc-fleet phase)
|
|
628
|
+
- Parity: run a fixed PRD corpus through both routes (`LOKI_SDK_LOOP=0` vs `=1`); compare per-iteration
|
|
629
|
+
cost/usage, all 8 gate outcomes, and the completion decision. Needs the full sdlc-fleet council
|
|
630
|
+
(3 reviewers, unanimous APPROVE) per `CLAUDE.md`.
|
|
631
|
+
- Rollback: `LOKI_SDK_LOOP=0` or `LOKI_LEGACY_BASH=1` -> `start` falls through to bash. Bash stays the
|
|
632
|
+
canonical route until SDK-proven across the discriminator corpus.
|
|
633
|
+
- Depends on Spike A (does Bun 1.3.14 spawn the 0.3.208 bundled binary cleanly?). If NO: loop runs under
|
|
634
|
+
Node, judges stay on Bun (raw SDK is pure JS). This gates the runtime choice, not the design.
|
|
635
|
+
|
|
636
|
+
---
|
|
637
|
+
|
|
638
|
+
## B.5 Cross-phase build order + the single reusable seam
|
|
639
|
+
|
|
640
|
+
Build order (value / risk / dependency): **B.1 (complete Phase 1) -> 2c (code-review, highest-value delete,
|
|
641
|
+
reuses cr-rematerialize) -> 2b (completion council) -> 2a parity (branch already exists) -> 2d aux ->
|
|
642
|
+
B.3 (parallel council) -> B.4 (RARV loop, last).**
|
|
643
|
+
|
|
644
|
+
Every judge phase reuses ONE seam already on the branch: `loki internal sdk-judge` -> `judgeJson()`.
|
|
645
|
+
Phase 2 adds only `.sh` branches + (2b) one schema + (2d) a `judgeText` sibling. No phase reinvents the
|
|
646
|
+
bridge. The `cache_control` extension (B.1.2) is written once in `judgeJson` and every later judge phase
|
|
647
|
+
inherits it. This is why "judges before the loop" is not just risk-ordering -- it amortizes one TS change
|
|
648
|
+
(the cached judge) across ~100 call sites before the hard Agent-SDK work begins.
|
|
649
|
+
|
|
650
|
+
|
|
651
|
+
## SPIKE RESULTS (2026-07-13, gating Phase 4 / EPIC D+G) - all PASS, decision = GO
|
|
652
|
+
|
|
653
|
+
Verified live with `@anthropic-ai/claude-agent-sdk@0.3.208` (LATEST) under Bun 1.3.13,
|
|
654
|
+
in an isolated scratch install (no production code touched):
|
|
655
|
+
|
|
656
|
+
- **Spike A (Bun can spawn the Agent SDK): PASS.** `bun add` resolved the platform-gated
|
|
657
|
+
optionalDependencies and installed `@anthropic-ai/claude-agent-sdk-darwin-arm64`, which
|
|
658
|
+
SHIPS the `claude` binary at `node_modules/.../claude-agent-sdk-darwin-arm64/claude`
|
|
659
|
+
(bundled as an npm dep - no separate CLI install, no interactive login, no version drift;
|
|
660
|
+
the enterprise/SaaS win holds). `bun run` imports `query()` as a function; full export
|
|
661
|
+
surface present: query, tool, createSdkMcpServer, forkSession, getSessionMessages,
|
|
662
|
+
resolveSettings, HOOK_EVENTS, DirectConnectTransport, InMemorySessionStore, etc.
|
|
663
|
+
- **Spike B (raw-SDK structured output): ALREADY PROVEN** by the shipped judge sites
|
|
664
|
+
(Epics A+B, council-approved). `messages.create` + `output_config.format` works end to end.
|
|
665
|
+
- **Spike C (Agent SDK option fields exist): PASS.** All Phase-4 flag-mapping fields present
|
|
666
|
+
in sdk.d.ts@0.3.208: maxBudgetUsd (L1644 - the `--max-budget-usd` analog the plan flagged
|
|
667
|
+
uncertain; CONFIRMED), fallbackModel (L1436), effort, model, systemPrompt, settingSources,
|
|
668
|
+
includePartialMessages (L1592), permissionMode, sessionId, resume, forkSession, maxTurns.
|
|
669
|
+
No capability loss vs the bash `claude ... stream-json` invocation.
|
|
670
|
+
|
|
671
|
+
**API shape confirmed for the build:**
|
|
672
|
+
- `query({ prompt, options }): Query extends AsyncGenerator<SDKMessage>` (L2231/L2528).
|
|
673
|
+
- Message types to consume in `sdk_stream_parser.ts`: SDKAssistantMessage (L2787),
|
|
674
|
+
SDKPartialAssistantMessage (L399), SDKResultMessage (L407), SDKUserMessage,
|
|
675
|
+
SDKAssistantMessageError (L2823: authentication_failed | rate_limit | overloaded | ...).
|
|
676
|
+
This is a TYPED async iterator - far cleaner than parsing raw stream-json TEXT through the
|
|
677
|
+
~350-line embedded Python parser. The parser's job becomes: iterate typed events -> write the
|
|
678
|
+
same `.loki/state/agents.json`, `.loki/events.jsonl` hook events, `.loki/metrics/result-cost-<iter>.json`.
|
|
679
|
+
|
|
680
|
+
**Build anchors (both routes):**
|
|
681
|
+
- bash main loop: `autonomy/run.sh:17465-17475` (`claude "${_loki_claude_argv[@]}" -p "$prompt"
|
|
682
|
+
--output-format stream-json --verbose | tee | python3 parser`). argv build ~17298-17430.
|
|
683
|
+
- TS main loop: `loki-ts/src/runner/providers.ts:274-285` (claudeProvider argv, `call.mainLoop`
|
|
684
|
+
flag; transport `shellRun(argv)` ~L305). RARV outer loop: `loki-ts/src/runner/autonomous.ts`.
|
|
685
|
+
- Phase 4 = replace the mainLoop `shellRun(["claude", ...stream-json...])` with `query()` +
|
|
686
|
+
`sdk_stream_parser.ts`, gated `LOKI_SDK_LOOP=1`, bash arm intact. KEEP build_prompt (parity-lock),
|
|
687
|
+
all 8 gates, the council, `.loki/` state machine. RARV-C is NOT an SDK primitive - the grader
|
|
688
|
+
loop stays. Route `bin/loki start` to Bun when `LOKI_SDK_LOOP=1`.
|
|
689
|
+
|
|
690
|
+
DECISION: GO. Phase 4 is feasible under Bun with zero capability loss. It remains the highest-risk
|
|
691
|
+
site (agentic tool loop + streaming) and stays LAST, done behind `LOKI_SDK_LOOP=1` with the bash
|
|
692
|
+
`claude` main loop live as the default until E2E-proven on real apps (simple + full-stack).
|