aiki-cli 0.3.1 → 0.3.3

package/CHANGELOG.md

@@ -1,18 +1,93 @@
 # Changelog
 
-## Unreleased
+## Unreleased — v7 adaptive deliberation
+
+### Added
+- **`aiki run idea-refinement --mode auto`** — a deterministic quick-vs-council routing decision made
+  before any provider call. Escalates to `council` when the prompt supplies URLs or an evidence pack,
+  hits a regulated/financial/security topic, requests deliverables beyond a decision, uses research
+  wording, or runs long (>120 words); otherwise `quick`. Prints the resolved mode + reason and persists
+  it to `meta.auto_decision`. No learned routing (§22): the rules are fixed and inspectable. `meta.mode`
+  stays `quick|council`; legacy metadata still parses and explicit modes keep their existing artifacts.
+- **One-call auto fast path (v7 Phase C).** A short DECISION-only request with a clear decision verb
+  and subject skips the two model preflight readings and builds the same typed contract deterministically.
+  Ambiguous prompts keep the existing quick path; explicit quick/council runs are unchanged. Reports label
+  the result: `Single-pass analysis; council escalation was not required.`
+- **Structural auto escalation (v7 Phase D).** Validated analyst output now triggers a deterministic
+  failover for unsupported decisive claims, model-knowledge STOP/PIVOT claims, failed calculations,
+  contradictory supplied evidence, or load-bearing disagreement. Auto runs use 1–4 calls: two existing
+  preflight readings, one primary analysis, and at most one slim claim-only challenger call. The receipt
+  records every trigger; explicit quick/council paths remain unchanged.
+- **Per-call token accounting (v7 Phase A).** Every call records normalized input/output token usage —
+  provider-reported for Claude, a labeled `chars/4` estimate otherwise — summed into `meta.usage_totals`
+  and shown as a `Tokens:` line in the dossier and terminal summary. Purely additive; behavior unchanged.
+
+## Unreleased — v6 council integrity
+
+Root-cause pass driven by exact replay of run `20260717-1219` (`plan/AIKI-v6-council-integrity-plan.md`);
+every fix is pinned by tests on the real payloads that failed live (`test/fixtures/f740-*`).
+
+### Fixed
+- **A parseable plan is never discarded again.** The planner's complete feature backlog and
+  implementation plan were replaced by "deliverables unavailable" over a 13-character headline
+  overflow with no repair budget. `jsonCall` now applies deterministic salvage before failing when
+  repair is unavailable; lossy coercion clips over-max strings at a word boundary; empty optional
+  strings are dropped losslessly (they carry no information). Partial planner answers are kept and
+  flagged `plan_partial` instead of vanishing.
+- **The chair's judgment is no longer overruled by code.** The `PARTIAL/UNVERIFIABLE ⇒ must be
+  UNRESOLVED` rule (which flattened 5 nuanced rulings and forced a 125s Opus repair) is deleted;
+  ruling HOLDS against positively CONTRADICTED evidence remains fatal.
+- **S4 schema-ceremony repairs are free now.** Empty rationale strings validate via lossless
+  coercion (was a 267s paid repair); genuinely lossy defects salvage without a call when the budget
+  is tight (repairs stay gated behind a 3-call reserved tail: chair + planner + one tail repair).
+- Chair prose is no longer regex-mangled ("confirmed-unverified" stays the chair's own words);
+  enum rewrites apply to UPPERCASE leaks only. A `NOT_COMPUTABLE` payback block is omitted instead
+  of announced. Rendered Markdown/HTML never contain local filesystem usernames.
+
+### Added
+- **Local council workspace:** `aiki serve` opens a loopback-only three-pane browser workspace with cached
+  provider status, decision history, server-enforced file/URL/spend gates, live SSE stage and call telemetry,
+  cancellation, path-free verdict cards, and responsive desktop/mobile layouts.
+- Completed decisions accept one-call follow-up questions labeled `no council`; failed or cancelled runs with
+  cached calls can be resumed after an explicit spend review. Settings atomically persist provider model pins
+  and role assignments without exposing credentials, local paths, or arbitrary JSON to the browser.
+- A dependency-free hosted replay, real workspace screenshot, and timed three-minute demo script provide a
+  safe submission path when judges do not have the three provider CLIs installed.
+- **Semantic claim join at zero extra calls:** the S8 verifier may emit `claim_groups` over an
+  anonymous all-claims index; validated groups (unknown ids and single-provider groups dropped)
+  overlay CONSENSUS/DISAGREEMENT states through the existing state machine. Cross-provider
+  paraphrases — the reason every report said "0 consensus · 0 disputes" — now join.
+- **Evidence origin honesty:** every card is classified EXTERNAL / USER_MATERIAL / MODEL_KNOWLEDGE;
+  confidence's evidence-quality term counts external only; the user's own material renders as
+  "consistent with your materials (not independently checked)", never VERIFIED; the Sources section
+  states how many independent external sources the run actually had.
+- **Blocked-source permission prompt:** when a supplied URL cannot be read, an interactive run now
+  asks right in the terminal before any paid call — `Proceed without it? [y/N]` — the same
+  allow/deny pattern provider CLIs use for internet or file access. Deny prints how to proceed
+  (paste the page text into the prompt, or `--allow-blocked-sources`); headless runs fail fast
+  with the same instructions. (The previous behavior burned a full council budget around a 403'd
+  page.)
+
+## 0.3.2 — 2026-07-17
 
 ### Added
 - Public URL snapshots for idea prompts, including an npm registry adapter, private-network rejection,
-  auditable `FETCHED` / `BLOCKED` / `FAILED` status, and a research-mode gate that stops before model calls
+  auditable `FETCHED` / `BLOCKED` / `FAILED` status, and a full-council gate that stops before model calls
   when a supplied source cannot be read.
 - Requested-output planning: prompts that ask for a feature list or implementation plan now carry that
   requirement through the decision contract, the existing planner call, machine JSON, Markdown, and HTML.
 
 ### Changed
-- Explicit research wording selects research mode deterministically when `--mode` is absent; an explicit
-  mode remains authoritative. Preflight asks zero to four questions and may not repeat facts already present
-  in the user prompt or a fetched source.
+- **Report v4 answer-first decisions** — reports now lead with the requested deliverables and chair reasoning,
+  substitute readable claim labels for internal ids, dedupe and cap claim-named conditions, frame/cap risks,
+  summarize each council seat (including a plain-language `weak_seat` warning), and distinguish FACTUAL
+  claims from JUDGMENT calls so verification priority and coverage describe checkable facts honestly.
+  Deliverable detection now combines model-backed readings with a widened fallback that hears requests for
+  standout or "ultra-level" features.
+- `council` now includes the complete source-investigation path, 8–10-call allowance, 12-call repair budget,
+  and 45-minute deadline. `research` remains accepted as a compatibility alias with identical behavior.
+  Preflight asks zero to four questions and may not repeat facts already present in the user prompt or a
+  fetched source.
 - Reader-facing report sections use human claim text instead of internal graph ids such as `G1`; raw ids
   remain in machine artifacts and the Markdown technical audit.
 

package/README.md

@@ -12,7 +12,7 @@
   <img alt="Node ≥ 20" src="https://img.shields.io/badge/node-%E2%89%A5%2020-brightgreen.svg">
   <img alt="Local-first, no API keys" src="https://img.shields.io/badge/local--first-no%20API%20keys-informational.svg">
   <img alt="Read-only orchestration" src="https://img.shields.io/badge/orchestration-read--only-success.svg">
-  <img alt="Tests" src="https://img.shields.io/badge/tests-381%20passing-success.svg">
+  <img alt="Tests" src="https://img.shields.io/badge/tests-579%20passing-success.svg">
 </p>
 
 ---
@@ -30,7 +30,21 @@ It does two jobs, well:
 aiki is **not** a general assistant. Trivia and chat are routed away, not answered — a council adds cost, not
 accuracy, when there's one right answer.
 
-**Jump to:** [Why](#why) · [What's new](#whats-new-in-030) · [How it works](#how-aiki-works-no-apis) · [Benchmarks](#benchmarks) · [Requirements](#requirements) · [Install](#install) · [Quickstart](#quickstart) · [The two workflows](#the-two-workflows) · [Example](#example-a-real-idea-run) · [Configuration](#configuration) · [Sessions & resume](#sessions--resume) · [Safety](#safety-model) · [Costs & limits](#costs--limits)
+**Jump to:** [Workspace](#local-council-workspace) · [Why](#why) · [What's new](#whats-new-in-030) · [How it works](#how-aiki-works-no-apis) · [Benchmarks](#benchmarks) · [Requirements](#requirements) · [Install](#install) · [Quickstart](#quickstart) · [The two workflows](#the-two-workflows) · [Example](#example-a-real-idea-run) · [Configuration](#configuration) · [Sessions & resume](#sessions--resume) · [Safety](#safety-model) · [Costs & limits](#costs--limits)
+
+---
+
+## Local council workspace
+
+Open the live local workspace with:
+
+```bash
+npm i -g aiki-cli && aiki serve
+```
+
+`aiki serve` opens a loopback-only browser workspace with decision history, explicit file/URL/spend approval,
+live council stages, a verdict card, one-call follow-ups, and per-role model settings. It makes no provider
+calls until you approve and convene a run.
 
 ---
 
@@ -50,9 +64,13 @@ stop copy-pasting between them by hand.
 - **Evidence-grounded decisions.** Supply local sources with `--evidence`; Aiki records their paths and
   hashes, checks freshness and citations, independently verifies selected load-bearing claims, and shows the
   remaining coverage gaps before the chair.
-- **Three bounded modes.** Use `quick` for one structured analyst, `council` for bounded multi-model
-  deliberation, or `research` for current-fact work. Explicit research wording selects `research`
-  deterministically; `--mode` always overrides it.
+- **Two bounded behaviors.** Use `quick` for one structured analyst or `council` for the full multi-model
+  decision council with source investigation. `research` remains accepted as an alias for `council`.
+  `--mode auto` deterministically selects the entry policy from the prompt (URLs, an evidence pack, a
+  regulated/financial/security topic, requested deliverables, research wording, or a long input require
+  preflight) and prints the one-line reason — never a learned choice. A clear, short decision question starts
+  with one call; structural gaps in the typed output add the two readings and, only when useful, one targeted
+  challenger call (1–4 total). Use explicit `council` when you want the full multi-model council every time.
 - **A decision dossier, not an essay.** Reports lead with the recommendation, verified evidence coverage,
   decisive facts, first action, strongest counter-case, and critical unknowns. Financial and threshold-heavy
   decisions can include graph-anchored numbers, payback, option commitments, and a go/no-go tripwire.
@@ -181,7 +199,7 @@ Upgrade an existing installation:
 
 ```bash
 npm install -g aiki-cli@latest
-aiki --version                 # 0.3.0
+aiki --version                 # 0.3.3
 ```
 
 From source:
@@ -198,9 +216,46 @@ npm link             # puts `aiki` on your PATH   (or run directly: node dist/cl
 
 ```bash
 cd /path/to/your/project
-aiki                 # opens the interactive home screen
+aiki serve           # opens the local council workspace in your browser
+aiki                 # or open the interactive terminal home screen
 ```
 
+## The serve workspace
+
+`aiki serve` is the recommended way to use Aiki. It starts a **loopback-only** web workspace (bound to
+`127.0.0.1`, never exposed to your network) and opens it in your browser. **No provider is called until you
+type a request and approve the run** — opening the workspace is free.
+
+```bash
+aiki serve                 # binds the first free port in 4173–4183 and opens the browser
+aiki serve --port 4180     # pin a specific port
+aiki serve --no-open       # start the server but don't auto-open the browser
+```
+
+**What you'll see and do:**
+
+1. **Sessions rail (left).** Every past run, newest first. Click one to reopen its full decision brief;
+   runs that were killed or timed out are flagged **Resume**.
+2. **Composer (center).** Describe an idea to stress-test, or ask for a code review, and press **Enter**.
+   Plain text never spends anything on its own — you always get a convene step first.
+3. **Approval gates (inline).** Before any paid work, Aiki asks — per action — to read a **file**, fetch a
+   **URL**, or **spend** provider calls. Each gate offers **allow once**, **allow for this session**, or
+   **deny**. Nothing runs until you approve.
+4. **Live council.** A pipeline timeline shows each stage; the provider seat cards (Claude, Codex, Gemini)
+   light up as they work, with a running **call counter and budget bar** so you always see the spend.
+5. **Verdict card.** When the council finishes: a plain-language banner (go / proceed-with-conditions /
+   stop / inconclusive), a **confidence meter**, the anchored validation plan, expandable reasoning and
+   evidence, and a **Copy summary** button.
+6. **Cost receipt.** Per-provider call counts plus the token/cost receipt for the run, against your budget.
+7. **Follow-ups.** After a run completes you can ask a follow-up question — answered in **exactly one
+   read-only call, no new council** — or **Re-convene** to run a fresh full council on a revised question.
+8. **Settings.** Choose the model each provider uses and override per-role assignments (judge, verifier,
+   analyst, the two review seats, the follow-up responder). It shows the effective value vs. your local
+   override and never exposes credentials, keys, or paths.
+
+Everything the workspace shows is backed by the same schema-validated, on-disk audit trail as the headless
+CLI, so nothing on screen can drift from what was actually stored.
+
 ## Use Aiki For
 
 - Code review: "review this branch", "check this patch", "find bugs in this diff".
@@ -235,8 +290,9 @@ Plain text is never charged silently — you get a confirm step before any run s
 ```bash
 aiki run idea-refinement "a fridge-photo-to-recipe app for busy parents"
 aiki run idea-refinement "an early idea" --mode quick             # one structured analyst; no council claim
+aiki run idea-refinement "Should I use Postgres or MySQL?" --mode auto  # starts at 1 call; structural gaps may escalate to 4
 aiki run idea-refinement ./idea.md
-aiki run idea-refinement ./idea.md --mode research --evidence ./research/  # grounded, source-verifying council
+aiki run idea-refinement ./idea.md --mode council --evidence ./research/   # grounded, source-verifying council
 aiki run code-review --base main             # review this branch vs main
 aiki run code-review --diff ./changes.patch  # review a patch file
 aiki run code-review --cheap                 # Gemini+Codex review, Claude judges only disputes (~⅓ the Opus)
@@ -251,7 +307,7 @@ reads provider credential directories.
 
 Public `http(s)` links in an idea are snapshotted once before model calls and stored in
 `00a-url-sources.json`; every preflight reader and analyst sees that same text. npm package links use the
-registry metadata API. A snapshot records `FETCHED`, `BLOCKED`, or `FAILED` honestly. In research mode,
+registry metadata API. A snapshot records `FETCHED`, `BLOCKED`, or `FAILED` honestly. In full council mode,
 an unreadable supplied link stops before paid calls and asks for pasted text or a public export instead of
 letting the council guess or grill you for facts that should have come from the link.
 
@@ -269,10 +325,9 @@ only the disputes → report.
 complementary analyst lanes → deterministic claim/evidence graph audit → only decision-critical verification
 or rebuttal → evidence-linked chair → validation planner. The report is a graph-backed **decision dossier**,
 not an essay.
-Choose `--mode quick` for one structured analyst, `--mode council` for the full decision council, or
-`--mode research` for source-grounded current-fact work. Without a flag, the default is council unless the
-request explicitly asks to research, browse, look up, or check current sources; that conservative rule is
-deterministic, and an explicit flag wins:
+Choose `--mode quick` for one structured analyst or `--mode council` for the full source-investigating decision
+council. `--mode research` is retained as a compatibility alias and runs the same council. Without a flag, the
+default is the full council:
 
 - a **reader-first decision card** — recommendation, verified evidence coverage, decisive facts, first action,
   strongest counter-case, and critical unknowns before audit detail
@@ -360,9 +415,12 @@ This is the part that makes aiki trustworthy to point at a real repo:
 ## Costs & limits
 
 - **Runs cost real model calls** against your existing CLI subscriptions/quota. Idea refinement is nominally
-  **3 calls in quick**, **6–8 in council**, or **8–10 in research** (schema repairs can add calls within the
-  mode-aware budget); code review is about **5**. `aiki run` shows the mode, range, budget, and reserved
+  **3 calls in quick** or **8–10 in the full council** (`research` is the same behavior; schema repairs can add
+  calls within the mode-aware budget); code review is about **5**. `aiki run` shows the mode, range, budget, and reserved
   chair/planner calls before asking to confirm (skip with `--yes`).
+- **Every run reports token usage.** The decision report and the serve cost receipt show input/output tokens
+  and reported cost per run — provider-reported where the CLI returns it, a labeled estimate otherwise — so
+  you see the real spend, not just the call count.
 - **Not a general assistant.** Questions and "explore my whole codebase" requests are redirected, not answered
   — aiki reviews a *diff* and vets a *stated idea*.
 - **Analysis, not advice.** Every report is a decision aid. Verify before acting.

package/dist/bench/idea-v3-bench.js

@@ -3,15 +3,18 @@ import { access, mkdir, readFile, readdir, rename, writeFile } from 'node:fs/pro
 import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path';
 import { z } from 'zod';
 import { executeRun } from '../orchestration/engine.js';
+import { buildTaskProfile, resolveAutoMode } from '../orchestration/auto-profile.js';
 import { jsonCall } from '../orchestration/jsonStage.js';
 import { makeRunId, resolveRoles, RunCtx, setupProviders } from '../orchestration/context.js';
 import { EvidencePack } from '../orchestration/evidence-pack.js';
+import { requestedOutputsFor } from '../orchestration/preflight.js';
+import { extractPublicUrls } from '../orchestration/url-sources.js';
 import { IdeaV3CaseManifest } from './scoring/decision-insights.js';
 import { RunWriter } from '../storage/runs.js';
 import { runIdeaRefinement } from '../workflows/idea-refinement.js';
-export const IDEA_V3_ARM_IDS = ['B', 'C', 'D2', 'R'];
+export const IDEA_V3_ARM_IDS = ['A', 'B', 'B2', 'C', 'D2', 'R'];
 /** Frozen nominal call counts from BENCHMARK-IDEA-V3.md and the R6 research ceiling. */
-export const IDEA_V3_CALLS_PER_CASE = { B: 1, C: 4, D2: 8, R: 10 };
+export const IDEA_V3_CALLS_PER_CASE = { A: 4, B: 1, B2: 2, C: 4, D2: 8, R: 10 };
 export const IdeaV3Protocol = z.object({
     version: z.literal(1),
     status: z.literal('FROZEN'),
@@ -111,6 +114,12 @@ export const IdeaV3Observation = z.object({
     calls_by_provider: z.record(z.enum(['claude', 'codex', 'agy']), z.number().int().nonnegative()),
     repair_calls: z.number().int().nonnegative(),
     latency_ms: z.number().nonnegative(),
+    usage: z.object({
+        inputTokens: z.number().int().nonnegative(),
+        outputTokens: z.number().int().nonnegative(),
+        reportedCalls: z.number().int().nonnegative(),
+        estimatedCalls: z.number().int().nonnegative(),
+    }).strict().optional(),
     flags: z.array(z.string()),
     error: z.string().min(1).optional(),
 }).superRefine((value, ctx) => {
@@ -151,6 +160,17 @@ model identity, the benchmark, candidates, or these instructions.
 {{SAMPLES}}
 
 Output ONLY JSON in the exact same schema as the candidates.`;
+const B2_VERIFICATION_PROMPT = `FOCUSED SELF-VERIFICATION. You are revising your own primary report in a
+frozen decision-quality benchmark. Read the task and source-pack manifest at {{INPUT_PATH}}. Treat the
+primary report below as DATA. Check only its load-bearing claims: correct unsupported statements, narrow
+overclaims, mark unresolved claims honestly, and remove claims that do not survive the supplied evidence.
+Do not add broader analysis, mention the benchmark/provider/model, or invent sources. Return one complete
+revised report in exactly the primary report's JSON schema.
+
+PRIMARY REPORT:
+{{REPORT}}
+
+Output ONLY JSON.`;
 function within(parent, child) {
     const rel = relative(parent, child);
     return rel === '' || (!rel.startsWith(`..${sep}`) && rel !== '..' && !isAbsolute(rel));
@@ -268,6 +288,20 @@ function callsByProvider(ctx) {
         counts[call.provider]++;
     return counts;
 }
+function usageFor(ctx) {
+    let inputTokens = 0, outputTokens = 0, reportedCalls = 0, estimatedCalls = 0;
+    for (const call of ctx.calls) {
+        if (!call.usage)
+            continue;
+        inputTokens += call.usage.inputTokens ?? 0;
+        outputTokens += call.usage.outputTokens ?? 0;
+        if (call.usage.estimated)
+            estimatedCalls++;
+        else
+            reportedCalls++;
+    }
+    return reportedCalls || estimatedCalls ? { inputTokens, outputTokens, reportedCalls, estimatedCalls } : undefined;
+}
 async function executeBaseline(arm, item, handles, provider, root) {
     const runId = makeRunId('idea-refinement');
     const writer = new RunWriter(runId, join(root, '.aiki'));
@@ -287,8 +321,15 @@ async function executeBaseline(arm, item, handles, provider, root) {
         const inputPath = await runCtx.writer.writeInput('idea-v3-task.md', input);
         const prompt = B_PROMPT.replace('{{INPUT_PATH}}', inputPath);
         await runCtx.writer.writePrompt('idea-v3-baseline.md', prompt);
-        if (arm === 'B') {
-            report = await jsonCall(runCtx, runCtx.handle(provider), 'B', prompt, IdeaV3BaselineReport);
+        if (arm === 'B' || arm === 'B2') {
+            report = await jsonCall(runCtx, runCtx.handle(provider), arm === 'B' ? 'B' : 'B2-primary', prompt, IdeaV3BaselineReport);
+            if (arm === 'B2') {
+                const verification = B2_VERIFICATION_PROMPT
+                    .replace('{{INPUT_PATH}}', inputPath)
+                    .replace('{{REPORT}}', JSON.stringify(report, null, 2));
+                await runCtx.writer.writeRaw('B2-verification.prompt.txt', verification);
+                report = await jsonCall(runCtx, runCtx.handle(provider), 'B2-verify', verification, IdeaV3BaselineReport);
+            }
         }
         else {
             const samples = [];
@@ -302,6 +343,7 @@ async function executeBaseline(arm, item, handles, provider, root) {
         await runCtx.writer.writeRaw('idea-v3-baseline-report.json', JSON.stringify(report, null, 2));
         await runCtx.writer.writeText('final-report', renderBaselineReport(report));
     });
+    const usage = usageFor(ctx);
     const base = {
         case_id: item.id,
         arm,
@@ -310,12 +352,59 @@ async function executeBaseline(arm, item, handles, provider, root) {
         calls_by_provider: callsByProvider(ctx),
         repair_calls: ctx.calls.filter((call) => call.stage.endsWith('-repair')).length,
         latency_ms: Date.now() - started,
+        ...(usage ? { usage } : {}),
         flags: [...ctx.flags],
     };
     return IdeaV3Observation.parse(outcome.ok && report
         ? { ...base, status: 'ok', report_markdown: renderBaselineReport(report) }
         : { ...base, status: 'error', error: `${outcome.error?.code ?? 'CRASH'}: ${outcome.error?.message ?? 'baseline produced no report'}` });
 }
+async function executeAuto(item, handles, root) {
+    const runId = makeRunId('idea-refinement');
+    const writer = new RunWriter(runId, join(root, '.aiki'));
+    const input = benchmarkInput(item);
+    const evidencePack = await caseEvidencePack(item);
+    const resolved = resolveAutoMode(buildTaskProfile(input, {
+        urlCount: extractPublicUrls(input).length,
+        hasEvidencePack: !!evidencePack,
+        requestedOutputs: requestedOutputsFor(input),
+    }));
+    const ctx = new RunCtx({
+        runId,
+        workflow: 'idea-refinement',
+        mode: resolved.mode,
+        handles,
+        roles: resolveRoles('idea-refinement', handles.map((handle) => handle.id)),
+        writer,
+        cwd: writer.dir,
+        budget: 7,
+        deadlineMs: 45 * 60 * 1000,
+        evidencePack,
+        autoDecision: {
+            resolved: resolved.mode,
+            reasons: resolved.reasons,
+            ...(resolved.fastPath ? { fast_path: true } : {}),
+        },
+    });
+    const started = Date.now();
+    const outcome = await executeRun(ctx, input, runIdeaRefinement);
+    const usage = usageFor(ctx);
+    const base = {
+        case_id: item.id,
+        arm: 'A',
+        run_id: runId,
+        calls: ctx.calls.length,
+        calls_by_provider: callsByProvider(ctx),
+        repair_calls: ctx.calls.filter((call) => call.stage.endsWith('-repair')).length,
+        latency_ms: Date.now() - started,
+        ...(usage ? { usage } : {}),
+        flags: [...ctx.flags],
+    };
+    if (!outcome.ok) {
+        return IdeaV3Observation.parse({ ...base, status: 'error', error: `${outcome.error?.code}: ${outcome.error?.message}` });
+    }
+    return IdeaV3Observation.parse({ ...base, status: 'ok', report_markdown: await readFile(join(outcome.dir, 'final-report.md'), 'utf8') });
+}
 async function executeResearch(item, handles, root, frozenRoles) {
     const runId = makeRunId('idea-refinement');
     const writer = new RunWriter(runId, join(root, '.aiki'));
@@ -332,6 +421,7 @@ async function executeResearch(item, handles, root, frozenRoles) {
     });
     const started = Date.now();
     const outcome = await executeRun(ctx, benchmarkInput(item), runIdeaRefinement);
+    const usage = usageFor(ctx);
     const base = {
         case_id: item.id,
         arm: 'R',
@@ -340,6 +430,7 @@ async function executeResearch(item, handles, root, frozenRoles) {
         calls_by_provider: callsByProvider(ctx),
         repair_calls: ctx.calls.filter((call) => call.stage.endsWith('-repair')).length,
         latency_ms: Date.now() - started,
+        ...(usage ? { usage } : {}),
         flags: [...ctx.flags],
     };
     if (!outcome.ok) {
@@ -353,6 +444,9 @@ function parseArms(arms, set) {
         throw new Error('at least one idea-v3 arm is required');
     if (set === 'holdout' && unique.includes('D2'))
         throw new Error('D2 is build-set diagnostic only and has no holdout weight');
+    if (set === 'holdout' && unique.some((arm) => arm === 'A' || arm === 'B2')) {
+        throw new Error('Phase F arms are build-set validation only and cannot change the frozen holdout protocol');
+    }
     return unique;
 }
 const CAMPAIGN_NAME = /^idea-v3-(build|holdout)-(claude|codex|agy)-\d{4}-\d{2}-\d{2}\.json$/;
@@ -447,9 +541,12 @@ export async function runIdeaV3Bench(opts = {}) {
     const prior = resolved.prior?.observations ?? [];
     const done = new Set(prior.map((item) => `${item.case_id}:${item.arm}`));
     const handles = opts.handles ?? await setupProviders(protocol?.models);
-    if (!handles.some((handle) => handle.id === baselineProvider) && arms.some((arm) => arm === 'B' || arm === 'C')) {
+    if (!handles.some((handle) => handle.id === baselineProvider) && arms.some((arm) => arm === 'B' || arm === 'B2' || arm === 'C')) {
         throw new Error(`baseline provider ${baselineProvider} is unavailable`);
     }
+    if (arms.includes('A') && new Set(handles.map((handle) => handle.id)).size < 2) {
+        throw new Error('A requires at least two providers for protocol-comparable auto escalation');
+    }
     if (arms.includes('R') && new Set(handles.map((handle) => handle.id)).size < 3) {
         throw new Error('R requires all three frozen providers for a protocol-comparable run');
     }
@@ -466,6 +563,8 @@ export async function runIdeaV3Bench(opts = {}) {
     const execute = opts.execute ?? (async (target) => {
         if (target.arm === 'D2')
             return d2ByCase.get(target.case.id);
+        if (target.arm === 'A')
+            return executeAuto(target.case, handles, root);
         if (target.arm === 'R')
             return executeResearch(target.case, handles, root, protocol?.roles);
         return executeBaseline(target.arm, target.case, handles, baselineProvider, root);