openmerit 0.1.7-preview.1 → 0.1.7-preview.3

package/CHANGELOG.md

@@ -1,5 +1,62 @@
 # Changelog
 
+## 0.1.7-preview.3 — 2026-10-16
+
+- Update packaged Scout API/MCP and agent-evaluation guides for the sole `/scout/` service and current request, inventory, data-handling and reliability contracts.
+- Keep the hosted hourly research and matching service separate from npm installation. Hosted publication recovery, per-source freshness, bounded Decisions retries and reduced overlapping pool reads were deployed independently.
+- Exclude internal website and deployment maintenance notes from the npm artifact. Keep existing commands, exports, legal metadata and stable `latest` behavior unchanged.
+
+## Scout semantic benchmark interpretation — hosted beta, 2026-10-15
+
+- Replace benchmark capability/name rules and family-name regexes with hourly Decisions interpretation of source definitions and methodology.
+- Cache unchanged interpretations; leave unsupported or failed scopes unclassified and preserve evidence provenance.
+- Keep prompt matching local to prepared benchmark metadata, with unchanged UI, prompt/specialist calls and hourly schedule. All 24 semantic benchmark cases, 107 local/live HTTP/MCP matching cases and 546 automated checks passed. npm publication remains deferred.
+- Store compressed analytics payloads as gzip bytes to avoid base64 overhead on larger evidence records; retain legacy decoding and complete-record size limits.
+
+## Scout semantic specialist matching — hosted beta, 2026-10-15
+
+- Replace Hugging Face domain/language keyword dictionaries and partial label overlap with Decisions compatibility checks.
+- Prepare and cache specialist profiles during the hourly refresh; batch only relevant compatibility checks when a prompt arrives.
+- Require all applicable domain, language and fixed-label checks to pass. Preserve unknown metrics and ordinary results if specialist checking fails.
+- Keep the UI unchanged. All 107 cases passed locally and through live HTTP/MCP; 526 automated tests passed. npm remains deferred.
+
+## Scout semantic formats and SQL interpretation — hosted beta, 2026-10-15
+
+- Remove prompt keyword scans from strict-format and structured-data interpretation; use the existing Decisions answers, including described record layouts and non-English requests.
+- Recognize read-only SQL query writing and repair without requiring schema details in the task description. Keep query source distinct from executing it, its result format, prose explanation and data modification.
+- Keep uncertain format answers visible and exclude them from narrow calibration matching. Preserve response validation, general consistency checks, confidence thresholds, one interpretation request, hourly collection and the UI.
+- All 91 semantic cases passed locally and through live HTTP/MCP, including the previously failing SQL request and four fresh SQL holdouts. All 511 automated tests passed. npm publication remains deferred.
+
+## Scout category and content interpretation — hosted beta, 2026-10-15
+
+- Recognize conversationally supplied categories without requiring a formal label list; keep subtype and individual-identity discovery open-ended.
+- Require document understanding for requested written/table content, including mixed outputs, without inferring it merely from a photographed object.
+- Update two existing semantic questions with unchanged confidence thresholds, UI, hourly refresh and request count. Local and live HTTP/MCP acceptance passed 43 cases; an opt-in semantic audit preserves those cases for future interpretation changes. npm publication remains deferred.
+
+## Scout task interpretation and ambiguous matching — hosted beta, 2026-10-15
+
+- Frame every task description as quoted data for requirement analysis, preserving supplied-label and negation semantics without prompt-specific exceptions or lower confidence thresholds.
+- Keep decision-plus-prose or extraction requests on generative endpoints, reject narrow calibration for uncertain operations/outputs, and ignore weak document-layout guesses for ordinary photos.
+- All 25 routing cases passed locally and through live HTTP/MCP, including the prior binary-label failure. UI, hourly refresh and the single interpretation request are unchanged. npm publication remains deferred.
+
+## Scout identification matching — hosted beta, 2026-10-15
+
+- Distinguish open identification from supplied-label classification within the existing interpretation call. Open or uncertain identification uses evidence-ranked generative endpoints; fixed-choice APIs and label-dependent classifiers require supplied choices.
+- Require relevant visual benchmark evidence for visual requests, preserve classification label scope in calibration matching, and use ordinary output-length estimates for generated names. UI and hourly refresh are unchanged; no npm release is included.
+
+## Scout hosted beta — 2026-10-15
+
+- Apply shared modality, output-format, tool and operation requirements across Scout recommendation paths. Catalogue refresh now retains explicit text-output contracts and excludes mixed media output from general text candidates.
+- Preserve unknown and partial token prices as `null`; keep genuine advertised zero rates distinct.
+- Suppress separate specialist discovery for endpoint-scoped requests, including empty inventories.
+- Honor latency priority when choosing a model's endpoint, using only timing tied to that route. UI, hourly scheduling and interpretation calls are unchanged. These four service corrections were deployed separately from npm.
+
+## 0.1.7-preview.2 — 2026-10-15
+
+- Ask for a $10, custom or no-fixed-cap evaluation budget after model scope; save the explicit choice, handle reservations internally, and preserve cumulative usage across reruns and budget changes. Add `eval plan --budget <USD|unlimited>` for coding agents and noninteractive setup.
+- Separate native budget and environment blocks from task scores in agent evaluations; probe the Codex Linux sandbox before making model requests.
+- Recover partial Claude usage, retain partial cost estimates, keep missing native pricing unknown, and preserve budget reservations independently of estimated charges. Report version 2 separates these accounting rules from earlier comparisons.
+
 ## 0.1.7-preview.1 — 2026-10-15
 
 - Add `openmerit eval init/doctor/plan/run/watch/report`: Scout shortlists, explicit model scope, Claude Code/Codex/OpenCode adapters, fresh task attempts, local graders, balanced adaptive 3–10 repetitions, revision comparisons and cumulative budget accounting.

package/README.md

@@ -22,7 +22,7 @@ Explore the [hosted sample](https://dash.openmerit.ai/) or [before/after dashboa
 
 ## Compare models with Pi
 
-The **npm preview `0.1.7-preview.1`** adds `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose existing connections or Scout's full index, freeze a shortlist, and compare 3–10 fresh attempts after source changes. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@0.1.7-preview.1/docs/agent-evaluations.md) for setup, graders, budgets and availability. This is separate from the Pi workflow below.
+The **npm preview `0.1.7-preview.3`** provides `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose model scope and a $10, custom or no-fixed-cap budget, freeze Scout's shortlist, and compare 3–10 fresh attempts after source changes. Budget and sandbox blocks stay separate from task scores; unknown costs stay unknown. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@0.1.7-preview.3/docs/agent-evaluations.md) for setup, graders, budgets and availability. Scout's hosted API and MCP service update independently of the npm package. This is separate from the Pi workflow below.
 
 With Pi 0.87 and a model provider configured:
 

package/dist/eval/src/adapters.mjs

@@ -2,18 +2,28 @@ import { execute, agentEnvironment } from './process.mjs';
 
 const number = value => typeof value === 'number' && Number.isFinite(value) && value >= 0 ? value : null;
 const sum = values => values.length && values.every(v => v !== null) ? values.reduce((a, b) => a + b, 0) : null;
+const maximum = values => values.some(v => v !== null) ? Math.max(...values.filter(v => v !== null)) : null;
 export const agents = ['claude', 'codex', 'opencode'];
 
 export function connectionFailure(text) {
-  if (/401|unauthorized|authentication[_ -]?(?:error|required)|invalid api key|not logged in|please log in/i.test(text)) return 'authentication';
+  if (/budget_exhausted|error_max_budget_usd|(?:maximum|exceeded|exhausted|insufficient).{0,25}(?:budget|credits)|(?:budget|credits).{0,25}(?:exceeded|exhausted)/i.test(text)) return 'budget_exhausted';
+  if (/(?:bwrap|bubblewrap|sandbox|landlock|seccomp|unshare|user namespace|uid_map).{0,160}(?:failed|not permitted|unavailable|denied)|failed to (?:start|initialize|create).{0,40}sandbox/i.test(text)) return 'sandbox_unavailable';
+  if (/\b(?:HTTP\s*)?401\b.{0,30}(?:unauthorized|authentication)|unauthorized|authentication[_ -]?(?:error|required)|invalid api key|not logged in|please log in/i.test(text)) return 'authentication';
   if (/model.{0,100}(?:not supported|not found|not available|does not exist)|(?:unknown|invalid|unsupported) model|ProviderModelNotFoundError/i.test(text)) return 'model_unavailable';
   if (/rejected permission|permission.{0,30}(?:denied|rejected)|not allowed to use.{0,30}tool/i.test(text)) return 'permission_denied';
   return null;
 }
 
-export function invocation(connection, reservationUsd) {
+function toolFailure(text) {
+  const reason = connectionFailure(text);
+  // A task's own HTTP/auth/budget errors are task outcomes, not evidence that
+  // the coding agent's provider connection failed.
+  return ['sandbox_unavailable', 'permission_denied'].includes(reason) ? reason : null;
+}
+
+export function invocation(connection, reservationUsd, unlimitedBudget = false) {
   const model = connection.model;
-  if (connection.agent === 'claude') return ['-p', '--output-format', 'json', '--model', model, '--no-session-persistence', '--permission-mode', 'dontAsk', '--allowedTools', 'Read,Glob,Grep,Bash,Edit,Write', '--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}', '--setting-sources', '', '--max-budget-usd', String(reservationUsd)];
+  if (connection.agent === 'claude') return ['-p', '--output-format', 'json', '--model', model, '--no-session-persistence', '--permission-mode', 'dontAsk', '--allowedTools', 'Read,Glob,Grep,Bash,Edit,Write', '--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}', '--setting-sources', '', ...(unlimitedBudget ? [] : ['--max-budget-usd', String(reservationUsd)])];
   if (connection.agent === 'codex') return ['exec', '--json', '--model', model, '--sandbox', 'workspace-write', '--skip-git-repo-check', '--ephemeral', '-'];
   if (connection.agent === 'opencode') return ['run', '--format', 'json', '--model', model, '--auto'];
   throw new Error('Unknown agent. Use claude, codex or opencode.');
@@ -23,7 +33,11 @@ export function parseOutput(agent, stdout) {
   let events;
   try { const parsed = JSON.parse(stdout); events = Array.isArray(parsed) ? parsed : [parsed]; }
   catch { events = stdout.split('\n').flatMap(line => { try { return [JSON.parse(line)]; } catch { return []; } }); }
-  let output = '', usage = null, costUsd = null, completed = false, failed = false;
+  let output = '', usage = null, costUsd = null, partialCostUsd = null, completed = false, failed = false, failureReason = null;
+  // Only native failure fields are evidence of a blocked execution. Task text,
+  // successful tool output and model commentary can legitimately discuss errors.
+  const errors = events.filter(e => ['error', 'turn.failed'].includes(e.type));
+  for (const event of errors) failureReason ??= connectionFailure(JSON.stringify(event));
   const observedModels = new Set();
   if (agent === 'claude') {
     const result = events.findLast(e => e.type === 'result');
@@ -32,8 +46,16 @@ export function parseOutput(agent, stdout) {
       failed = !completed; output = result.result ?? '';
       const u = result.usage;
       if (u) usage = { input: sum([number(u.input_tokens), number(u.cache_read_input_tokens ?? 0), number(u.cache_creation_input_tokens ?? 0)]), output: number(u.output_tokens), cached: number(u.cache_read_input_tokens ?? 0) };
-      costUsd = number(result.total_cost_usd);
+      const models = Object.values(result.modelUsage ?? {});
+      if (models.length) {
+        const modelUsage = { input: sum(models.map(m => sum([number(m.inputTokens), number(m.cacheReadInputTokens ?? 0), number(m.cacheCreationInputTokens ?? 0)]))), output: sum(models.map(m => number(m.outputTokens))), cached: sum(models.map(m => number(m.cacheReadInputTokens ?? 0))) };
+        // A budget stop can emit zero aggregate usage but nonzero per-model usage.
+        usage = Object.fromEntries(Object.keys(modelUsage).map(k => [k, maximum([usage?.[k] ?? null, modelUsage[k]])]));
+      }
+      costUsd = maximum([number(result.total_cost_usd), sum(models.map(m => number(m.costUSD)))]);
       for (const model of Object.keys(result.modelUsage ?? {})) observedModels.add(model);
+      if (failed) failureReason ??= connectionFailure(JSON.stringify({ subtype: result.subtype, terminal_reason: result.terminal_reason, errors: result.errors }));
+      if (result.permission_denials?.length) failureReason ??= 'permission_denied';
     }
   } else if (agent === 'codex') {
     const turns = events.filter(e => e.type === 'turn.completed');
@@ -41,6 +63,9 @@ export function parseOutput(agent, stdout) {
     failed = events.some(e => ['turn.failed', 'error'].includes(e.type));
     output = events.findLast(e => e.type === 'item.completed' && e.item?.type === 'agent_message')?.item.text ?? '';
     if (turns.length) usage = { input: sum(turns.map(e => number(e.usage?.input_tokens))), output: sum(turns.map(e => number(e.usage?.output_tokens))), cached: sum(turns.map(e => number(e.usage?.cached_input_tokens))) };
+    for (const event of events.filter(e => e.type === 'item.completed' && e.item?.type === 'command_execution' && (e.item.exit_code > 0 || e.item.status === 'failed'))) {
+      failureReason ??= toolFailure(event.item.aggregated_output ?? '');
+    }
   } else if (agent === 'opencode') {
     const steps = events.filter(e => e.type === 'step_finish');
     const finalStep = steps.findLast(e => ['stop', 'end_turn'].includes(e.part?.reason));
@@ -50,24 +75,38 @@ export function parseOutput(agent, stdout) {
     output = finalStep?.part?.messageID ? texts.filter(e => e.part?.messageID === finalStep.part.messageID).map(e => e.part?.text ?? '').join('\n') : texts.at(-1)?.part?.text ?? '';
     if (steps.length) {
       usage = { input: sum(steps.map(e => sum([number(e.part?.tokens?.input), number(e.part?.tokens?.cache?.read ?? 0), number(e.part?.tokens?.cache?.write ?? 0)]))), output: sum(steps.map(e => sum([number(e.part?.tokens?.output), number(e.part?.tokens?.reasoning ?? 0)]))), cached: sum(steps.map(e => number(e.part?.tokens?.cache?.read ?? 0))) };
-      costUsd = sum(steps.map(e => number(e.part?.cost)));
+      const costs = steps.map(e => number(e.part?.cost));
+      costUsd = costs.every(c => c !== null && c > 0) ? sum(costs) : null;
+      if (costUsd === null) partialCostUsd = sum(costs.filter(c => c !== null && c > 0));
     }
+    for (const event of events.filter(e => e.type === 'tool_use' && e.part?.state?.status === 'error')) failureReason ??= toolFailure(event.part.state.error ?? '');
   }
-  return { output, usage, costUsd, completed: completed && !failed, observedModels: [...observedModels] };
+  // Native zero prices can mean missing pricing metadata, not free inference.
+  return { output, usage, costUsd: costUsd > 0 ? costUsd : null, partialCostUsd, completed: completed && !failed, failureReason, observedModels: [...observedModels] };
 }
 
-export async function preflight(connection) {
+export async function preflight(connection, { cwd, executeCommand = execute, platform = process.platform } = {}) {
   const command = connection.command ?? connection.agent;
-  const version = await execute(command, ['--version'], { timeoutMs: 10_000, env: agentEnvironment(connection.env) });
-  const help = await execute(command, connection.agent === 'codex' ? ['exec', '--help'] : connection.agent === 'opencode' ? ['run', '--help'] : ['--help'], { timeoutMs: 10_000, env: agentEnvironment(connection.env) });
+  const options = { cwd, timeoutMs: 10_000, env: agentEnvironment(connection.env) };
+  const version = await executeCommand(command, ['--version'], options);
+  const help = await executeCommand(command, connection.agent === 'codex' ? ['exec', '--help'] : connection.agent === 'opencode' ? ['run', '--help'] : ['--help'], options);
   const flags = connection.agent === 'codex' ? ['--json', '--ephemeral', '--sandbox'] : connection.agent === 'claude' ? ['--output-format', '--no-session-persistence', '--max-budget-usd'] : ['--format', '--model', '--auto'];
   if (version.code !== 0 || help.code !== 0 || !flags.every(f => (help.stdout + help.stderr).includes(f))) throw new Error(`${connection.agent} lacks the required non-interactive JSON interface. Update it or set this connection's command to a compatible executable.`);
+  if (connection.agent === 'codex' && platform === 'linux') {
+    const sandboxHelp = await executeCommand(command, ['sandbox', '--help'], options);
+    const legacy = /^\s+linux\s/m.test(sandboxHelp.stdout);
+    const probe = await executeCommand(command, ['-c', 'sandbox_mode="workspace-write"', 'sandbox', ...(legacy ? ['linux'] : []), '--', process.execPath, '-e', 'process.stdout.write("openmerit-sandbox-ok")'], options);
+    if (probe.code !== 0 || probe.reason || probe.stdout !== 'openmerit-sandbox-ok') {
+      throw Object.assign(new Error('Codex sandbox could not run a local probe. Fix host sandbox support before evaluating; no model request was made.'), { failureReason: 'sandbox_unavailable' });
+    }
+  }
   return version.stdout.trim().slice(0, 160);
 }
 
 export async function runAgent(connection, prompt, options) {
-  const result = await execute(connection.command ?? connection.agent, invocation(connection, options.reservationUsd), { ...options, input: prompt, env: agentEnvironment(connection.env) });
+  const result = await execute(connection.command ?? connection.agent, invocation(connection, options.reservationUsd, options.unlimitedBudget), { ...options, input: prompt, env: agentEnvironment(connection.env) });
   const parsed = parseOutput(connection.agent, result.stdout);
-  const status = result.reason ?? (result.code !== 0 || !parsed.completed ? 'agent_error' : 'ok');
-  return { ...parsed, costUsd: status === 'ok' ? parsed.costUsd : null, partialCostUsd: status === 'ok' ? null : parsed.costUsd, seconds: result.seconds, status, failureReason: status === 'agent_error' ? connectionFailure(result.stdout + result.stderr) : null };
+  const failureReason = parsed.failureReason ?? (result.code !== 0 ? connectionFailure(result.stderr) : null) ?? (['spawn_failed', 'cancelled', 'output_limit'].includes(result.reason) ? result.reason : null);
+  const status = failureReason === 'budget_exhausted' ? 'budget_exhausted' : ['sandbox_unavailable', 'permission_denied', 'spawn_failed'].includes(failureReason) ? 'environment_error' : result.reason ?? (failureReason || result.code !== 0 || !parsed.completed ? 'agent_error' : 'ok');
+  return { ...parsed, costUsd: status === 'ok' ? parsed.costUsd : null, partialCostUsd: status === 'ok' ? parsed.partialCostUsd : maximum([parsed.costUsd, parsed.partialCostUsd]), costBasis: parsed.costUsd === null && parsed.partialCostUsd === null ? 'unknown' : 'agent_estimate', seconds: result.seconds, status, failureReason };
 }

package/dist/eval/src/cli.mjs

@@ -11,8 +11,8 @@ export const help = `OpenMerit · Agent evaluations (source preview)
 
   openmerit eval init                 Add a starter suite and project agent instructions
   openmerit eval doctor               Check installed Claude Code, Codex and OpenCode CLIs
-  openmerit eval plan [--scope existing|all]
-                                     Ask scope, then freeze Scout's shortlist per task
+  openmerit eval plan [--scope existing|all] [--budget <USD|unlimited>]
+                                     Choose scope and budget, then freeze Scout's shortlist
   openmerit eval run [--json]         Evaluate the frozen shortlist in fresh workspaces
   openmerit eval watch [--json]       Repeat after selected CLI/docs/fixture files change
   openmerit eval report [--json]      Read the latest result without running models
@@ -21,6 +21,9 @@ export const help = `OpenMerit · Agent evaluations (source preview)
 
 Configure .openmerit/eval.json: explicit agent connections, task fixtures, local
 graders, CLI variants, 3–10 repeats, and a cumulative budget. Init starts at $0.
+Plan asks for $10, a custom amount, or no fixed cap; nothing is preselected.
+Saved choices and prior budget usage persist across reruns. --budget changes
+the total allowance, not the amount remaining. Reservations are automatic.
 Scout receives task descriptions and, for existing scope, model IDs only.
 Run/watch execute local agents and may incur charges. No background daemon.
 `;
@@ -34,9 +37,10 @@ When building or changing this project's CLI, tools or harness documentation, us
 
 - Read .openmerit/eval.json and run \`openmerit eval doctor\`.
 - Ask once: "Models available through this project's existing connections, or all models Scout indexes?" Use \`eval plan --scope existing\` or \`--scope all\`. Reuse the saved choice. Never infer endpoint access from a model name or expose credentials.
+- Then ask once: "What budget should OpenMerit use for this project's evaluations? 1. $10; 2. Custom amount (how much?); 3. No fixed cap." Pass the explicit answer as \`eval plan --budget 10\`, \`--budget <amount>\` or \`--budget unlimited\`, together with scope on first setup. Never choose a default or infer unlimited spending. Reuse an existing positive or unlimited budget unless the user changes it. The allowance is cumulative across reruns, not reset by edits or replanning. Reservations are internal; do not ask the user to calculate them. No fixed cap still has bounded tasks, 3–10 repetitions, timeouts and user-controlled watch; subsequent watched edits can incur further charges until stopped.
 - Declare real connections with Scout model IDs, agent-native model IDs, and environment variable NAMES. Existing scope uses this project-declared inventory. All scope may require new authenticated connections before execution.
 - Replace starter cases with representative tasks, fresh fixtures and deterministic standalone Node graders. A grader reads {workspace, output} JSON on stdin and returns {quality: 0..1}. Keep answer keys outside agent workspaces. Include a bare variant and a CLI/docs variant where useful.
-- Confirm the suite, variants and cumulative budget with the user before enabling paid runs. Then plan once, run a baseline, make the requested CLI changes, and run again. Use \`eval watch\` only when the user wants continuous local reruns; it exits with the terminal and retains budget usage across restarts.
+- Confirm the suite and variants before enabling paid runs, using the budget already selected above. Do not ask again for an unchanged authorized budget. Then plan once, run a baseline, make the requested CLI changes, and run again. Use \`eval watch\` only when the user wants continuous local reruns; it exits with the terminal and retains budget usage across restarts.
 - Report quality, failures, tokens, reported costs and elapsed time per task/model/variant. A shortlist is a prior, not measured success. Do not claim missing costs are zero or that 3–10 repeats prove an exact ordering. Keep fresh holdouts for final validation.
 - If a compatible watcher is already active, let it run; never start overlapping loops. Do not overwrite the frozen plan merely because the CLI changed.
 <!-- openmerit-eval:end -->`;
@@ -61,16 +65,57 @@ export async function initialize(root) {
   if (!ignore.split('\n').includes('/.openmerit/eval/')) await writeFile(join(root, '.gitignore'), `${ignore}${ignore.endsWith('\n') || !ignore ? '' : '\n'}/.openmerit/eval/\n`);
 }
 
+export function parseBudget(value) {
+  const input = value.trim();
+  if (input === 'unlimited') return input;
+  const amount = input.replace(/^\$/, '');
+  if (!/^(?:\d+(?:\.\d{1,2})?|\.\d{1,2})$/.test(amount) || Number(amount) <= 0 || Number(amount) > Number.MAX_SAFE_INTEGER / 100) throw new Error('Enter a positive USD amount with at most two decimal places, or "unlimited" for no fixed cap.');
+  return Number(amount);
+}
+
+// ask is supplied only for an interactive terminal. Coding agents pass explicit
+// flags after asking the user; EOF, blank input and missing flags never consent.
+export async function choosePlanOptions(config, { scope, budget, ask, tell = () => {} } = {}) {
+  const next = structuredClone(config);
+  if (scope) next.scope = scope;
+  while (!['existing', 'all'].includes(next.scope)) {
+    if (!ask) throw new Error('Choose a scope with eval plan --scope existing or --scope all, then choose --budget <USD|unlimited>.');
+    const answer = (await ask('Use existing project connections or all models Scout indexes? [existing/all] ')).trim();
+    if (['existing', 'all'].includes(answer)) next.scope = answer;
+    else tell('Choose existing or all.');
+  }
+  if (budget !== undefined) next.budget = { maxUsd: parseBudget(budget), reservationUsd: 'auto' };
+  while (next.budget.maxUsd === 0) {
+    if (!ask) throw new Error('Choose an evaluation budget after model scope: --budget 10, --budget <USD>, or --budget unlimited. No amount is selected automatically.');
+    const choice = (await ask('What budget should OpenMerit use for this project’s evaluations?\n  1. $10\n  2. Custom amount\n  3. No fixed cap (watched changes can incur further charges until stopped)\nChoose [1/2/3]: ')).trim();
+    if (choice === '1') next.budget = { maxUsd: 10, reservationUsd: 'auto' };
+    else if (choice === '3') next.budget = { maxUsd: 'unlimited', reservationUsd: 'auto' };
+    else if (choice === '2') {
+      while (next.budget.maxUsd === 0) {
+        const amount = await ask('How much in USD? ');
+        try {
+          const parsed = parseBudget(amount);
+          if (parsed === 'unlimited') throw new Error('Enter a custom USD amount.');
+          next.budget = { maxUsd: parsed, reservationUsd: 'auto' };
+        } catch { tell('Enter a positive USD amount with at most two decimal places.'); }
+      }
+    } else tell('Choose 1, 2 or 3. Nothing is selected automatically.');
+  }
+  return next;
+}
+
 export async function main(args) {
   if (!args.length || args.includes('--help') || args.includes('-h')) { console.log(help); return; }
-  const action = args.shift(); let root = process.cwd(), scope, json = false;
+  const action = args.shift(); let root = process.cwd(), scope, budget, json = false;
   for (let i = 0; i < args.length; i++) {
     if (args[i] === '--path' && args[i + 1]) root = resolve(args[++i]);
     else if (args[i] === '--scope' && ['existing', 'all'].includes(args[i + 1])) scope = args[++i];
+    else if (args[i] === '--budget' && args[i + 1] && budget === undefined) { budget = args[++i]; parseBudget(budget); }
     else if (args[i] === '--json') json = true;
     else throw new Error(`Unknown eval argument: ${args[i]}`);
   }
   if (scope && action !== 'plan') throw new Error('--scope belongs to eval plan.');
+  if (budget !== undefined && action !== 'plan') throw new Error('--budget belongs to eval plan.');
   if (action === 'init') { await initialize(root); console.log('Created .openmerit/eval.json, starter fixtures, and AGENTS.md / CLAUDE.md instructions. Declare connections, replace the starter task, and set an approved budget before running.'); return; }
   if (action === 'doctor') {
     const results = [];
@@ -84,14 +129,17 @@ export async function main(args) {
   process.once('SIGINT', stop); process.once('SIGTERM', stop);
   try {
     if (action === 'plan') {
-      const config = await loadConfig(root);
-      if (scope) config.scope = scope;
-      if (!config.scope && process.stdin.isTTY && !json) {
-        const rl = createInterface({ input: process.stdin, output: process.stdout });
-        try { const answer = await rl.question('Use existing project connections or all models Scout indexes? [existing/all] ', { signal: controller.signal }); if (['existing', 'all'].includes(answer.trim())) config.scope = answer.trim(); } finally { rl.close(); }
-      }
+      const saved = await loadConfig(root);
+      const rl = process.stdin.isTTY && !json ? createInterface({ input: process.stdin, output: process.stdout }) : null;
+      const inputClosed = new AbortController();
+      rl?.once('close', () => inputClosed.abort());
+      let config;
+      try { config = await choosePlanOptions(saved, { scope, budget, ask: rl ? question => rl.question(question, { signal: AbortSignal.any([controller.signal, inputClosed.signal]) }) : undefined, tell: message => console.log(message) }); }
+      finally { rl?.close(); }
+      // Preserve explicit choices even if Scout is temporarily unavailable.
+      await saveJSON(join(root, configPath), config);
       const plan = await makePlan(config);
-      await saveJSON(join(root, configPath), config); await savePlan(root, plan);
+      await savePlan(root, plan);
       console.log(JSON.stringify(plan, null, 2));
       return;
     }

package/dist/eval/src/config.mjs

@@ -43,7 +43,7 @@ export function validateConfig(config) {
     v.files.forEach(safePath);
   }
   requireThat(Number.isInteger(config.repeats?.min) && Number.isInteger(config.repeats?.max) && config.repeats.min >= 3 && config.repeats.max <= 10 && config.repeats.min <= config.repeats.max, 'repeats must be between 3 and 10.');
-  requireThat(Number.isFinite(config.budget?.maxUsd) && config.budget.maxUsd >= 0 && Number.isFinite(config.budget.reservationUsd) && config.budget.reservationUsd > 0, 'set nonnegative maxUsd and positive reservationUsd.');
+  requireThat((config.budget?.maxUsd === 'unlimited' || Number.isFinite(config.budget?.maxUsd) && config.budget.maxUsd >= 0) && (config.budget.reservationUsd === 'auto' || Number.isFinite(config.budget.reservationUsd) && config.budget.reservationUsd > 0), 'set nonnegative maxUsd or explicit "unlimited", and reservationUsd to "auto" or a positive amount.');
   requireThat(Number.isInteger(config.timeoutSeconds) && config.timeoutSeconds >= 1 && config.timeoutSeconds <= 1800, 'timeoutSeconds must be 1–1800.');
   if (config.scoutURL !== undefined) {
     const url = new URL(config.scoutURL);
@@ -53,6 +53,12 @@ export function validateConfig(config) {
 }
 export async function loadConfig(root) { return validateConfig(await readJSON(join(root, configPath))); }
 
+export function reservationFor(config, plan) {
+  if (config.budget.reservationUsd !== 'auto') return config.budget.reservationUsd;
+  const attempts = plan.tasks.reduce((total, task) => total + task.candidates.reduce((n, model) => n + model.connections.length, 0), 0) * config.variants.length * config.repeats.min;
+  return config.budget.maxUsd === 'unlimited' || config.budget.maxUsd === 0 ? .25 : Math.min(.25, config.budget.maxUsd / Math.max(1, attempts));
+}
+
 // Snapshot exact bytes before executing; edits during a run belong to the next revision.
 export async function filesAt(root, paths) {
   const files = Object.create(null); let bytes = 0; const canonicalRoot = await realpath(root);
@@ -98,5 +104,5 @@ export const starter = {
   version: 1, scope: null, connections: [],
   tasks: [{ id: 'json', description: 'Extract a customer name and item count from text into a strict JSON object.', prompt: 'Read request.txt. Return only JSON with keys name (string) and count (number).', fixture: 'eval/fixtures/json', grader: 'eval/grade.mjs' }],
   variants: [{ id: 'bare', files: [], instructions: '' }, { id: 'cli', files: ['eval/tool.mjs'], instructions: 'You can use node eval/tool.mjs request.txt to extract the fields.' }],
-  repeats: { min: 3, max: 10 }, budget: { maxUsd: 0, reservationUsd: 0.25 }, timeoutSeconds: 180,
+  repeats: { min: 3, max: 10 }, budget: { maxUsd: 0, reservationUsd: 'auto' }, timeoutSeconds: 180,
 };

package/dist/eval/src/report.mjs

@@ -9,10 +9,13 @@ export function summarize(rows) {
   }
   return [...groups].map(([key, runs]) => ({
     key, task: runs[0].task, connection: runs[0].connection, variant: runs[0].variant, runs: runs.length,
-    quality: completeMean(runs.map(r => r.quality)), failures: runs.filter(r => r.status !== 'ok').length,
+    quality: completeMean(runs.map(r => r.quality)), failures: runs.filter(r => r.status !== 'ok' && r.quality !== null).length,
+    blockedAttempts: runs.filter(r => r.failureReason).length, scoredRuns: runs.filter(r => r.quality !== null).length,
+    failedTaskChecks: runs.filter(r => r.status === 'ok' && r.quality === 0).length,
     qualityRange: runs.every(r => r.quality !== null) ? [Math.min(...runs.map(r => r.quality)), Math.max(...runs.map(r => r.quality))] : null,
     inputTokens: completeMean(runs.map(r => r.usage?.input ?? null)), outputTokens: completeMean(runs.map(r => r.usage?.output ?? null)),
-    reportedCostUsd: runs.filter(r => r.costUsd !== null).reduce((s, r) => s + r.costUsd, 0), unknownCosts: runs.filter(r => r.costUsd === null).length,
+    reportedCostUsd: runs.some(r => r.costUsd !== null) ? runs.filter(r => r.costUsd !== null).reduce((s, r) => s + r.costUsd, 0) : null, unknownCosts: runs.filter(r => r.costUsd === null).length,
+    partialReportedCostUsd: runs.reduce((s, r) => s + (r.partialCostUsd ?? 0), 0),
     meanCostUsd: completeMean(runs.map(r => r.costUsd)), medianSeconds: median(runs.map(r => r.seconds)),
   }));
 }
@@ -39,13 +42,18 @@ export function compareReports(current, previous) {
 }
 const fmt = (v, digits = 2) => v === null || v === undefined ? '?' : v.toFixed(digits);
 export function formatReport(report) {
-  const lines = [`OpenMerit eval · ${report.status} · ${report.revision.slice(0, 12)}`, '', 'Task / connection / variant | runs | quality | input / output tokens | reported $/run | seconds (median)'];
-  for (const row of report.summary) lines.push(`${row.key} | ${row.runs} | ${fmt(row.quality === null ? null : row.quality * 100, 1)}% | ${fmt(row.inputTokens, 0)} / ${fmt(row.outputTokens, 0)} | ${fmt(row.meanCostUsd, 5)} | ${fmt(row.medianSeconds)}`);
+  const lines = [`OpenMerit eval · ${report.status} · ${report.revision.slice(0, 12)}`, '', 'Task / connection / variant | scored / attempts | blocked | quality | input / output tokens | estimated $/run | seconds (median)'];
+  for (const row of report.summary) {
+    lines.push(`${row.key} | ${row.scoredRuns ?? row.runs} / ${row.runs} | ${row.blockedAttempts ?? 0} | ${row.quality === null ? '?' : fmt(row.quality * 100, 1) + '%'} | ${fmt(row.inputTokens, 0)} / ${fmt(row.outputTokens, 0)} | ${fmt(row.meanCostUsd, 5)} | ${fmt(row.medianSeconds)}`);
+    if (row.partialReportedCostUsd > 0) lines.push(`  Partial cost estimate: $${fmt(row.partialReportedCostUsd, 5)}; full attempt cost unknown.`);
+  }
   if (report.comparison) {
     lines.push('', 'Change from the previous compatible revision (negative tokens/cost/time is lower):');
     for (const row of report.comparison) lines.push(`${row.key}: quality ${fmt(row.qualityDelta === null ? null : row.qualityDelta * 100, 1)} pp; tokens ${fmt(row.inputTokensDelta, 0)} / ${fmt(row.outputTokensDelta, 0)}; $ ${fmt(row.meanCostUsdDelta, 5)}; seconds ${fmt(row.medianSecondsDelta)}`);
   } else lines.push('', 'No complete, compatible previous revision for comparison.');
   lines.push('', '3–10 repetitions measure variability, not statistical certainty. ? means unavailable.', 'Costs are agent-reported estimates, not reconciled bills. Compare agents as different harnesses.');
+  if (report.budgetLimitUsd !== undefined) lines.push(report.budgetLimitUsd === 'unlimited' ? 'Budget: no fixed cap. Task/repetition limits and timeouts still apply; watched changes can incur further charges until stopped.' : `Budget: $${fmt(report.budgetLimitUsd)} cumulative across reruns.`);
+  lines.push(`Reserved allowance used: $${fmt(report.budgetUsedUsd, 5)} (not billed spend). Blocked attempts are unscored.`);
   if (report.reason) lines.push(`Stopped: ${report.reason}`);
   return lines.join('\n');
 }

package/dist/eval/src/runner.mjs

@@ -2,7 +2,7 @@ import { mkdir, mkdtemp, rm, readFile, writeFile } from 'node:fs/promises';
 import { join } from 'node:path';
 import { tmpdir } from 'node:os';
 import { randomUUID } from 'node:crypto';
-import { loadConfig, snapshot, materialize, readJSON, saveJSON, statePath, digest } from './config.mjs';
+import { loadConfig, snapshot, materialize, readJSON, saveJSON, statePath, digest, reservationFor } from './config.mjs';
 import { validatePlan } from './scout.mjs';
 import { preflight, runAgent, invocation } from './adapters.mjs';
 import { execute, agentEnvironment } from './process.mjs';
@@ -17,7 +17,9 @@ export async function acquireLock(root) {
 }
 export function budgetUsed(ledger) {
   if (ledger?.version !== 1 || !Array.isArray(ledger.entries) || ledger.entries.some(e => !Number.isFinite(e.reservationUsd) || e.reservationUsd <= 0 || e.costUsd !== null && (!Number.isFinite(e.costUsd) || e.costUsd < 0))) throw new Error('Invalid budget ledger. Restore it before running; never discard unknown charges.');
-  return ledger.entries.reduce((sum, e) => sum + (e.costUsd ?? e.reservationUsd), 0);
+  // Agent prices are estimates, not provider settlement. Never release a
+  // reservation on that evidence, including entries written by older previews.
+  return ledger.entries.reduce((sum, e) => sum + Math.max(e.costUsd ?? 0, e.reservationUsd), 0);
 }
 
 export async function evaluate(root, { signal, onProgress = () => {}, agentRunner = runAgent, checkAgent = preflight } = {}) {
@@ -28,17 +30,22 @@ export async function evaluate(root, { signal, onProgress = () => {}, agentRunne
   const versions = {};
   const ledgerPath = join(root, statePath, 'budget.json');
   const ledger = await optionalJSON(ledgerPath) ?? { version: 1, entries: [] };
-  if (budgetUsed(ledger) + config.budget.reservationUsd > config.budget.maxUsd + 1e-10) throw new Error('Evaluation budget is exhausted or disabled. Set budget.maxUsd in .openmerit/eval.json; prior reservations still count.');
-  for (const connection of config.connections.filter(c => selected.has(c.id))) versions[connection.id] = await checkAgent(connection);
+  const reservationUsd = reservationFor(config, plan), unlimitedBudget = config.budget.maxUsd === 'unlimited';
+  const exhausted = () => { const used = budgetUsed(ledger); return !unlimitedBudget && used + reservationUsd > config.budget.maxUsd + 1e-10; };
+  if (exhausted()) throw new Error('Evaluation budget is exhausted or disabled. Choose a budget with eval plan --budget <USD|unlimited>; prior reservations still count.');
   const runnerVersion = (await readJSON(new URL('../../../package.json', import.meta.url))).version;
-  const commands = config.connections.filter(c => selected.has(c.id)).map(c => ({ id: c.id, args: invocation(c, config.budget.reservationUsd) }));
-  const report = { version: 1, runnerVersion, id: randomUUID(), startedAt: new Date().toISOString(), revision: snap.revision, compatibility: digest({ snapshot: snap.compatibility, versions, runnerVersion, commands }), planHash: digest(plan.tasks), agentVersions: versions, status: 'running', rows: [], summary: [] };
+  const commands = config.connections.filter(c => selected.has(c.id)).map(c => ({ id: c.id, args: invocation(c, reservationUsd, unlimitedBudget) }));
+  const report = { version: 2, runnerVersion, budgetLimitUsd: config.budget.maxUsd, reservationUsd, id: randomUUID(), startedAt: new Date().toISOString(), revision: snap.revision, compatibility: null, planHash: digest(plan.tasks), agentVersions: versions, status: 'running', rows: [], summary: [] };
   const reportPath = join(root, statePath, 'runs', `${report.startedAt.replace(/[:.]/g, '-')}-${report.id}.json`);
   const previous = await optionalJSON(join(root, statePath, 'latest.json'));
   const temp = await mkdtemp(join(tmpdir(), 'openmerit-eval-'));
   const checkpoint = async () => { report.summary = summarize(report.rows); await saveJSON(reportPath, report); };
   try {
     await checkpoint();
+    for (const connection of config.connections.filter(c => selected.has(c.id))) {
+      try { versions[connection.id] = await checkAgent(connection, { cwd: temp }); }
+      catch (error) { error.connection = connection.id; throw error; }
+    }
     // Graders are copied outside all agent workspaces, before any trial runs.
     for (const task of snap.tasks) await materialize(task.graderFiles, join(temp, 'graders', task.id));
     outer: for (const task of snap.tasks) {
@@ -50,15 +57,16 @@ export async function evaluate(root, { signal, onProgress = () => {}, agentRunne
         if (round % 2) ordered.reverse();
         for (const { connection, variant, candidate } of ordered) {
           if (signal?.aborted) { report.status = 'cancelled'; report.reason = 'Interrupted'; break outer; }
-          if (budgetUsed(ledger) + config.budget.reservationUsd > config.budget.maxUsd + 1e-10) { report.status = 'budget_stopped'; report.reason = 'Cumulative budget cannot admit another reserved attempt.'; break outer; }
-          const entry = { id: randomUUID(), run: report.id, reservationUsd: config.budget.reservationUsd, costUsd: null, settled: false };
+          if (exhausted()) { report.status = 'budget_stopped'; report.reason = 'Cumulative budget cannot admit another reserved attempt.'; break outer; }
+          const entry = { id: randomUUID(), run: report.id, reservationUsd, costUsd: null, settled: false };
           ledger.entries.push(entry); await saveJSON(ledgerPath, ledger);
           const workspace = join(temp, entry.id); await mkdir(workspace);
           await materialize(task.fixtureFiles, workspace, task.fixture);
           await materialize(variant.sourceFiles, workspace);
           const prompt = `${task.prompt}\n\n${variant.instructions}\n\nThis is an isolated evaluation attempt. Do not run OpenMerit evaluation commands or inspect paths outside this workspace.`;
-          const result = await agentRunner(connection, prompt, { cwd: workspace, timeoutMs: config.timeoutSeconds * 1000, signal, reservationUsd: config.budget.reservationUsd });
-          entry.costUsd = result.costUsd; entry.reservationUsd = Math.max(entry.reservationUsd, result.partialCostUsd ?? 0); entry.settled = true; await saveJSON(ledgerPath, ledger);
+          const result = await agentRunner(connection, prompt, { cwd: workspace, timeoutMs: config.timeoutSeconds * 1000, signal, reservationUsd, unlimitedBudget });
+          entry.costUsd = result.costUsd; entry.costBasis = result.costBasis ?? (result.costUsd === null ? 'unknown' : 'agent_estimate');
+          entry.partialCostUsd = result.partialCostUsd ?? null; entry.reservationUsd = Math.max(entry.reservationUsd, result.partialCostUsd ?? 0); entry.settled = true; await saveJSON(ledgerPath, ledger);
           let quality = result.status === 'ok' || result.failureReason ? null : 0, status = result.status;
           if (result.status === 'ok') {
             const grader = await execute(process.execPath, [join(temp, 'graders', task.id, task.grader)], { cwd: temp, env: agentEnvironment(), input: JSON.stringify({ workspace, output: result.output }), timeoutMs: 30_000, signal });
@@ -68,10 +76,14 @@ export async function evaluate(root, { signal, onProgress = () => {}, agentRunne
               quality = value.quality;
             } catch { status = 'grader_error'; }
           }
-          const row = { task: task.id, connection: connection.id, agent: connection.agent, requestedModel: connection.model, observedModels: result.observedModels, scoutId: candidate.scoutId, scoutRank: candidate.rank, variant: variant.id, round: round + 1, quality, status, failureReason: result.failureReason ?? null, usage: result.usage, costUsd: result.costUsd, seconds: result.seconds };
+          const row = { task: task.id, connection: connection.id, agent: connection.agent, requestedModel: connection.model, observedModels: result.observedModels, scoutId: candidate.scoutId, scoutRank: candidate.rank, variant: variant.id, round: round + 1, quality, status, failureReason: result.failureReason ?? null, usage: result.usage, costUsd: result.costUsd, partialCostUsd: result.partialCostUsd ?? null, costBasis: entry.costBasis, seconds: result.seconds };
           report.rows.push(row); await checkpoint(); await rm(workspace, { recursive: true, force: true });
           onProgress(row);
-          if (result.failureReason) { report.status = 'connection_blocked'; report.reason = `${connection.id}: ${result.failureReason}. Fix the native agent connection before retrying.`; break outer; }
+          if (result.failureReason) {
+            report.status = result.failureReason === 'budget_exhausted' ? 'budget_stopped' : result.failureReason === 'cancelled' ? 'cancelled' : ['sandbox_unavailable', 'permission_denied', 'spawn_failed'].includes(result.failureReason) ? 'environment_blocked' : 'connection_blocked';
+            report.reason = `${connection.id}: ${result.failureReason}. This attempt is unscored; resolve the execution block before retrying.`;
+            break outer;
+          }
           if (status === 'grader_error') { report.status = 'invalid'; report.reason = `Grader failed for ${task.id}. Fix it before collecting more trials.`; break outer; }
         }
         if (round + 1 >= config.repeats.min && !isVolatile(report.rows.filter(r => r.task === task.id))) break;
@@ -79,11 +91,16 @@ export async function evaluate(root, { signal, onProgress = () => {}, agentRunne
     }
     if (report.status === 'running') report.status = 'complete';
   } catch (error) {
-    report.status = 'error'; report.reason = 'Execution failed; unsettled reservations remain charged. Check configuration and agent authentication.';
-    throw error;
+    if (error.failureReason) {
+      report.status = 'environment_blocked'; report.reason = `${error.connection}: ${error.failureReason}. ${error.message}`;
+      report.blockers = [{ connection: error.connection, failureReason: error.failureReason }];
+    } else {
+      report.status = 'error'; report.reason = 'Execution failed; unsettled reservations remain charged. Check configuration and agent authentication.';
+      throw error;
+    }
   } finally {
     report.finishedAt = new Date().toISOString(); report.budgetUsedUsd = budgetUsed(ledger);
-    report.compatibility = digest({ base: report.compatibility, observedModels: Object.fromEntries([...selected].sort().map(id => [id, [...new Set(report.rows.filter(r => r.connection === id).flatMap(r => r.observedModels ?? []))].sort()])) });
+    report.compatibility = digest({ accountingVersion: report.version, snapshot: snap.compatibility, versions, runnerVersion, commands, observedModels: Object.fromEntries([...selected].sort().map(id => [id, [...new Set(report.rows.filter(r => r.connection === id).flatMap(r => r.observedModels ?? []))].sort()])) });
     report.summary = summarize(report.rows); report.comparison = compareReports(report, previous);
     await saveJSON(reportPath, report); await saveJSON(join(root, statePath, 'latest.json'), report);
     await rm(temp, { recursive: true, force: true });