openmerit 0.1.6 → 0.1.7-preview.2

package/CHANGELOG.md

@@ -1,10 +1,16 @@
 # Changelog
 
-## 0.1.6 — 2026-10-11
+## 0.1.7-preview.2 — 2026-10-15
 
-- Promote the code from `0.1.6-preview.5` to the default npm release. Install with `npm install -g openmerit`, then run `openmerit dash` from an application directory.
-- Update the packaged README and terminal guides to present current capabilities and limits. The terminal is a starting point for OpenMerit; this release introduces no new routing or evaluation behavior.
-- Retain the existing coordinator, protocol, Pi adapter, package exports, and Node.js requirement. Earlier npm versions remain available.
+- Ask for a $10, custom or no-fixed-cap evaluation budget after model scope; save the explicit choice, handle reservations internally, and preserve cumulative usage across reruns and budget changes. Add `eval plan --budget <USD|unlimited>` for coding agents and noninteractive setup.
+- Separate native budget and environment blocks from task scores in agent evaluations; probe the Codex Linux sandbox before making model requests.
+- Recover partial Claude usage, retain partial cost estimates, keep missing native pricing unknown, and preserve budget reservations independently of estimated charges. Report version 2 separates these accounting rules from earlier comparisons.
+
+## 0.1.7-preview.1 — 2026-10-15
+
+- Add `openmerit eval init/doctor/plan/run/watch/report`: Scout shortlists, explicit model scope, Claude Code/Codex/OpenCode adapters, fresh task attempts, local graders, balanced adaptive 3–10 repetitions, revision comparisons and cumulative budget accounting.
+- Bundle project agent instructions without install-time hooks. Missing usage/cost remains unknown and partial comparisons are disclosed. Terminal/Pi behavior is unchanged.
+- Use the hosted Scout HTTP/MCP `modelIds` filter to select eligible candidates before ranking. This API change was deployed separately on 15 October 2026.
 
 ## 0.1.6-preview.5 — 2026-10-10
 

package/README.md

@@ -1,76 +1,97 @@
 # OpenMerit
 
-OpenMerit is being built to help applications make better inference decisions using evidence about their models, workloads, quality, cost, and latency.
+OpenMerit provides a local Inference Terminal for understanding an application's models, services, recorded activity, and compute, plus a Pi integration for evidence-driven model comparisons.
 
-Version **0.1.6** starts with visibility: the Inference Terminal gives you a local view of the inference your application declares and records. It is a starting point for OpenMerit, not the full product vision. This release brings the existing npm preview into the default installation.
+The terminal reads existing evidence. The Pi integration asks a coding harness to build evaluations, measure candidates, and carry out authorized changes. OpenMerit coordinates that workflow and validates its structured results; the terminal does not route requests, run evaluations, or swap models.
 
-## Install and run
+## Open the terminal
 
-Requires **Node.js 22.19 or newer**. In your application's directory:
+Requires Node.js 22.19 or newer and npm:
 
 ```sh
 npm install -g openmerit
+cd /path/to/your/application
 openmerit dash
 ```
 
-The terminal opens in your browser while collection starts. It runs locally, bound to `127.0.0.1`, and prints its URL. Keep the command running while using it; press Ctrl+C to stop. Pi and model-provider credentials are not required for the terminal.
+Stable `openmerit@0.1.6` includes the terminal. No Pi installation or provider API key is needed. It opens a browser connected to a read-only loopback service. Supported source/configuration declarations establish inventory; existing application logs or exports supply request history. Prompts, responses, and credentials are not retained. Unsupported or missing evidence remains unknown.
 
-You can also point it at a project explicitly:
+The source interface and hosted dashboard follow the device's light/dark preference without a theme control. This appearance change is not included in npm 0.1.6.
+
+Explore the [hosted sample](https://dash.openmerit.ai/) or [before/after dashboard demo](https://dash.openmerit.ai/demo). Both use fictional data and make no inference calls. See [the terminal guide](docs/inference-terminal.md) for configuration, discovery, diagnostics, history limits, and agent access.
+
+## Compare models with Pi
+
+The **npm preview `0.1.7-preview.2`** provides `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose model scope and a $10, custom or no-fixed-cap budget, freeze Scout's shortlist, and compare 3–10 fresh attempts after source changes. Budget and sandbox blocks stay separate from task scores; unknown costs stay unknown. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@0.1.7-preview.2/docs/agent-evaluations.md) for setup, graders, budgets and availability. This is separate from the Pi workflow below.
+
+With Pi 0.87 and a model provider configured:
 
 ```sh
-openmerit dash /path/to/your/project
+pi install npm:openmerit
+pi list
 ```
 
-To update from an earlier release or `openmerit@preview`, run `npm install -g openmerit`, stop the old terminal process, and launch it again. To pin this release, use `npm install -g openmerit@0.1.6`. Remove it with `npm uninstall -g openmerit`.
+Open the product repository in interactive Pi. An unconfigured project is checked for an application LLM call at session start and after settled build turns. Pi confirms the application target and incumbent model, proposes task-specific metrics and sampling, explains estimated evaluation cost, and requests confirmation of the plan, budget, and permissions. `/openmerit setup` can recover or rerun setup. Pi's own model is never the application target.
 
-## What you can see today
+Core checks stored baseline windows against the confirmed requirements. When ready, it asks Pi to discover candidates, run controlled comparisons, and calculate a frontier. Core validates the returned evidence contracts and Pareto conformance. The harness must execute the work and provide honest evidence.
 
-- **Models and services:** application declarations from supported code and configuration, recorded provider connections, and supported project-linked runtime inventory.
-- **Inference activity:** requests, tokens, latency, costs, and agent/task identifiers where supported application logs or explicit exports provide them.
-- **Code and configuration evidence:** file-and-line references, indexing progress, unresolved references, and source coverage.
-- **Compute and deployments:** explicitly linked hosts, model residency, and supplied hardware or capacity measurements, with their freshness and scope shown.
+Model changes normally require approval. Explicit automatic-swap opt-in can allow changes once the configured verified-swap threshold is reached, including a threshold of zero. Post-swap evaluation and conditional rollback depend on the harness performing and reporting them.
 
-The terminal keeps declarations, runtime inventory, and recorded usage separate. Missing evidence stays unknown. Code alone cannot reconstruct request history, usage, costs, or running agents. Discovery supports specific source formats and syntax; it is not complete coverage of every application or provider.
+Read [getting started](docs/getting-started.md) and [your first supervised trial](docs/first-trial.md).
 
-## Local and read-only
+## Implemented orchestration
 
-The terminal reads supported project files, logs, exports, and authorized loopback runtime inventory endpoints. It does not execute your application code, call models, route inference, run evaluations, or change your model configuration. It does not retain prompts, responses, or credential values in its snapshots. Closing it leaves your workloads running.
+- Task profiles with confirmed metrics, sampling plans, constraints, budgets, and permissions.
+- Preliminary model research, incumbent identity, durable candidate handoffs, and controlled challenger experiments.
+- Deterministic baseline checks and uncertainty-aware frontier validation.
+- Versioned result schemas for all ten intents and a reusable harness adapter boundary.
+- Challenger-spend accounting, bounded intent admission, approval, verification, and rollback requests.
+- Project-local state, redacted audit events, evidence manifests, and optional event exporters.
+- Task-count, elapsed-time, catalogue-change, regression, and post-swap signals.
+- Durable pause and authorized macOS user LaunchAgents; other platforms require an external scheduler.
 
-Repository indexing runs in the background and reuses unchanged metadata in memory. Request history and runtime readers have documented limits. The local API can be reached by other processes or users on the same machine, so treat its model names, endpoint identifiers, and file paths as project data.
+These implementations have automated coverage. They do not establish application-quality improvements or reliable unattended model changes. See [current limitations](docs/roadmap.md).
 
-## Useful commands
+## Pi commands
 
-```sh
-openmerit dash --no-open       # Print the local URL without opening a browser
-openmerit dash --json          # Print one collected snapshot, then exit
-openmerit dash --diagnostics   # Print repository-index coverage counts only
-openmerit dash --demo          # Explore fictional data without local collection
-openmerit dash --help
-```
+- `/openmerit` or `/openmerit status` — inspect readiness, active work, and lifecycle state.
+- `/openmerit setup` — request or recover setup.
+- `/openmerit assess` — check stored baseline windows; discover candidates only when ready.
+- `/openmerit frontier` — request a harness-calculated comparison for validation.
+- `/openmerit approve` — approve the current verified proposal.
+- `/openmerit pause` / `/openmerit resume` — persist or clear the automatic-work pause and reconcile authorized scheduling; active work is not cancelled.
+- `/openmerit cancel` — record abandonment of an interrupted intent; it does not terminate a provider request.
+- `/openmerit logs` — locate local state, audit, and evidence files.
+- `/openmerit doctor` — inspect adapter and scheduler diagnostics.
 
-See the [Inference Terminal guide](docs/inference-terminal.md) for supported sources, setup, options, data handling, and limitations. You can also [explore the hosted fictional sample](https://dash.openmerit.site/) without installing anything; it does not inspect your environment.
+## Documentation
 
-## Optional Pi integration
+The [documentation overview](docs/index.md) links all current guides. The website at [openmerit.ai/docs](https://openmerit.ai/docs/) is deployed separately from the package and this checkout.
 
-The package also retains the Pi extension, reusable coordinator, and protocol. These are separate from the terminal. Pi can gather application evidence and carry out user-authorized model comparisons and changes; OpenMerit coordinates typed outcomes and validates the returned structure and frontier evidence.
+- [Inference Terminal](docs/inference-terminal.md)
+- [Metrics and evidence](docs/metrics-and-evidence.md)
+- [Lifecycle](docs/lifecycle.md), [automation](docs/automation.md), and [budgets](docs/budgets.md)
+- [Commands](docs/commands.md), [operations](docs/operations.md), and [troubleshooting](docs/troubleshooting.md)
+- [Security and data handling](docs/security.md)
+- [Architecture](docs/architecture.md), [Pi adapter](docs/pi-extension.md), and [adapter development](docs/adapter-guide.md)
+- [Pareto specification](docs/pareto-spec.md) and [testing](docs/testing.md)
 
-For Pi 0.87 with a supported model provider configured:
+## Development
 
 ```sh
-pi install npm:openmerit
+npm ci
+npm run verify
+npm run dash -- /path/to/application
 ```
 
-Then use `/openmerit doctor` inside Pi. See [Pi setup](docs/getting-started.md), [commands](docs/commands.md), [budgets and permissions](docs/budgets.md), and [current limitations](docs/roadmap.md). The terminal itself does not perform this evaluation or routing work.
+For Pi extension development:
 
-The existing `openmerit/core`, `openmerit/protocol`, and `openmerit/pi` exports remain available for integrations.
+```sh
+./node_modules/.bin/pi --no-extensions -e ./packages/pi/src/index.ts
+```
 
-## Documentation
+`--no-extensions` avoids registering an installed npm copy alongside the source extension. Credentials belong to the harness or its environment and must never be committed. Jev, if configured, is invoked by the harness; OpenMerit provides no Jev client.
 
-- [Inference Terminal](docs/inference-terminal.md)
-- [Security and data handling](docs/security.md)
-- [Project files and logs](docs/operations.md)
-- [Architecture and integrations](docs/architecture.md)
-- [Testing and validation](docs/testing.md)
-- [Changelog](CHANGELOG.md)
+`packages/terminal` contains the CLI, collectors, hosted sample, and browser interface. `packages/protocol`, `packages/core`, and `packages/pi` provide the protocol, coordinator, and Pi adapter. The package exports `openmerit/core`, `openmerit/protocol`, and `openmerit/pi`.
 
-The bundled Inference Terminal guide follows this package version. The separately deployed [documentation website](https://openmerit.site/docs/) may reflect a different release.
+Edit documentation in `docs/`, then run `npm run check:docs`. The website is generated from those Markdown files. See [documentation maintenance](docs/site-documentation.md) and the [production-source reconciliation note](docs/site-domain-migration.md) before deployment; a local build does not publish changes.

package/dist/eval/src/adapters.mjs

@@ -0,0 +1,112 @@
+import { execute, agentEnvironment } from './process.mjs';
+
+const number = value => typeof value === 'number' && Number.isFinite(value) && value >= 0 ? value : null;
+const sum = values => values.length && values.every(v => v !== null) ? values.reduce((a, b) => a + b, 0) : null;
+const maximum = values => values.some(v => v !== null) ? Math.max(...values.filter(v => v !== null)) : null;
+export const agents = ['claude', 'codex', 'opencode'];
+
+export function connectionFailure(text) {
+  if (/budget_exhausted|error_max_budget_usd|(?:maximum|exceeded|exhausted|insufficient).{0,25}(?:budget|credits)|(?:budget|credits).{0,25}(?:exceeded|exhausted)/i.test(text)) return 'budget_exhausted';
+  if (/(?:bwrap|bubblewrap|sandbox|landlock|seccomp|unshare|user namespace|uid_map).{0,160}(?:failed|not permitted|unavailable|denied)|failed to (?:start|initialize|create).{0,40}sandbox/i.test(text)) return 'sandbox_unavailable';
+  if (/\b(?:HTTP\s*)?401\b.{0,30}(?:unauthorized|authentication)|unauthorized|authentication[_ -]?(?:error|required)|invalid api key|not logged in|please log in/i.test(text)) return 'authentication';
+  if (/model.{0,100}(?:not supported|not found|not available|does not exist)|(?:unknown|invalid|unsupported) model|ProviderModelNotFoundError/i.test(text)) return 'model_unavailable';
+  if (/rejected permission|permission.{0,30}(?:denied|rejected)|not allowed to use.{0,30}tool/i.test(text)) return 'permission_denied';
+  return null;
+}
+
+function toolFailure(text) {
+  const reason = connectionFailure(text);
+  // A task's own HTTP/auth/budget errors are task outcomes, not evidence that
+  // the coding agent's provider connection failed.
+  return ['sandbox_unavailable', 'permission_denied'].includes(reason) ? reason : null;
+}
+
+export function invocation(connection, reservationUsd, unlimitedBudget = false) {
+  const model = connection.model;
+  if (connection.agent === 'claude') return ['-p', '--output-format', 'json', '--model', model, '--no-session-persistence', '--permission-mode', 'dontAsk', '--allowedTools', 'Read,Glob,Grep,Bash,Edit,Write', '--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}', '--setting-sources', '', ...(unlimitedBudget ? [] : ['--max-budget-usd', String(reservationUsd)])];
+  if (connection.agent === 'codex') return ['exec', '--json', '--model', model, '--sandbox', 'workspace-write', '--skip-git-repo-check', '--ephemeral', '-'];
+  if (connection.agent === 'opencode') return ['run', '--format', 'json', '--model', model, '--auto'];
+  throw new Error('Unknown agent. Use claude, codex or opencode.');
+}
+
+export function parseOutput(agent, stdout) {
+  let events;
+  try { const parsed = JSON.parse(stdout); events = Array.isArray(parsed) ? parsed : [parsed]; }
+  catch { events = stdout.split('\n').flatMap(line => { try { return [JSON.parse(line)]; } catch { return []; } }); }
+  let output = '', usage = null, costUsd = null, partialCostUsd = null, completed = false, failed = false, failureReason = null;
+  // Only native failure fields are evidence of a blocked execution. Task text,
+  // successful tool output and model commentary can legitimately discuss errors.
+  const errors = events.filter(e => ['error', 'turn.failed'].includes(e.type));
+  for (const event of errors) failureReason ??= connectionFailure(JSON.stringify(event));
+  const observedModels = new Set();
+  if (agent === 'claude') {
+    const result = events.findLast(e => e.type === 'result');
+    if (result) {
+      completed = result.subtype === 'success' && !result.is_error;
+      failed = !completed; output = result.result ?? '';
+      const u = result.usage;
+      if (u) usage = { input: sum([number(u.input_tokens), number(u.cache_read_input_tokens ?? 0), number(u.cache_creation_input_tokens ?? 0)]), output: number(u.output_tokens), cached: number(u.cache_read_input_tokens ?? 0) };
+      const models = Object.values(result.modelUsage ?? {});
+      if (models.length) {
+        const modelUsage = { input: sum(models.map(m => sum([number(m.inputTokens), number(m.cacheReadInputTokens ?? 0), number(m.cacheCreationInputTokens ?? 0)]))), output: sum(models.map(m => number(m.outputTokens))), cached: sum(models.map(m => number(m.cacheReadInputTokens ?? 0))) };
+        // A budget stop can emit zero aggregate usage but nonzero per-model usage.
+        usage = Object.fromEntries(Object.keys(modelUsage).map(k => [k, maximum([usage?.[k] ?? null, modelUsage[k]])]));
+      }
+      costUsd = maximum([number(result.total_cost_usd), sum(models.map(m => number(m.costUSD)))]);
+      for (const model of Object.keys(result.modelUsage ?? {})) observedModels.add(model);
+      if (failed) failureReason ??= connectionFailure(JSON.stringify({ subtype: result.subtype, terminal_reason: result.terminal_reason, errors: result.errors }));
+      if (result.permission_denials?.length) failureReason ??= 'permission_denied';
+    }
+  } else if (agent === 'codex') {
+    const turns = events.filter(e => e.type === 'turn.completed');
+    completed = turns.length > 0;
+    failed = events.some(e => ['turn.failed', 'error'].includes(e.type));
+    output = events.findLast(e => e.type === 'item.completed' && e.item?.type === 'agent_message')?.item.text ?? '';
+    if (turns.length) usage = { input: sum(turns.map(e => number(e.usage?.input_tokens))), output: sum(turns.map(e => number(e.usage?.output_tokens))), cached: sum(turns.map(e => number(e.usage?.cached_input_tokens))) };
+    for (const event of events.filter(e => e.type === 'item.completed' && e.item?.type === 'command_execution' && (e.item.exit_code > 0 || e.item.status === 'failed'))) {
+      failureReason ??= toolFailure(event.item.aggregated_output ?? '');
+    }
+  } else if (agent === 'opencode') {
+    const steps = events.filter(e => e.type === 'step_finish');
+    const finalStep = steps.findLast(e => ['stop', 'end_turn'].includes(e.part?.reason));
+    completed = !!finalStep;
+    failed = events.some(e => e.type === 'error');
+    const texts = events.filter(e => e.type === 'text');
+    output = finalStep?.part?.messageID ? texts.filter(e => e.part?.messageID === finalStep.part.messageID).map(e => e.part?.text ?? '').join('\n') : texts.at(-1)?.part?.text ?? '';
+    if (steps.length) {
+      usage = { input: sum(steps.map(e => sum([number(e.part?.tokens?.input), number(e.part?.tokens?.cache?.read ?? 0), number(e.part?.tokens?.cache?.write ?? 0)]))), output: sum(steps.map(e => sum([number(e.part?.tokens?.output), number(e.part?.tokens?.reasoning ?? 0)]))), cached: sum(steps.map(e => number(e.part?.tokens?.cache?.read ?? 0))) };
+      const costs = steps.map(e => number(e.part?.cost));
+      costUsd = costs.every(c => c !== null && c > 0) ? sum(costs) : null;
+      if (costUsd === null) partialCostUsd = sum(costs.filter(c => c !== null && c > 0));
+    }
+    for (const event of events.filter(e => e.type === 'tool_use' && e.part?.state?.status === 'error')) failureReason ??= toolFailure(event.part.state.error ?? '');
+  }
+  // Native zero prices can mean missing pricing metadata, not free inference.
+  return { output, usage, costUsd: costUsd > 0 ? costUsd : null, partialCostUsd, completed: completed && !failed, failureReason, observedModels: [...observedModels] };
+}
+
+export async function preflight(connection, { cwd, executeCommand = execute, platform = process.platform } = {}) {
+  const command = connection.command ?? connection.agent;
+  const options = { cwd, timeoutMs: 10_000, env: agentEnvironment(connection.env) };
+  const version = await executeCommand(command, ['--version'], options);
+  const help = await executeCommand(command, connection.agent === 'codex' ? ['exec', '--help'] : connection.agent === 'opencode' ? ['run', '--help'] : ['--help'], options);
+  const flags = connection.agent === 'codex' ? ['--json', '--ephemeral', '--sandbox'] : connection.agent === 'claude' ? ['--output-format', '--no-session-persistence', '--max-budget-usd'] : ['--format', '--model', '--auto'];
+  if (version.code !== 0 || help.code !== 0 || !flags.every(f => (help.stdout + help.stderr).includes(f))) throw new Error(`${connection.agent} lacks the required non-interactive JSON interface. Update it or set this connection's command to a compatible executable.`);
+  if (connection.agent === 'codex' && platform === 'linux') {
+    const sandboxHelp = await executeCommand(command, ['sandbox', '--help'], options);
+    const legacy = /^\s+linux\s/m.test(sandboxHelp.stdout);
+    const probe = await executeCommand(command, ['-c', 'sandbox_mode="workspace-write"', 'sandbox', ...(legacy ? ['linux'] : []), '--', process.execPath, '-e', 'process.stdout.write("openmerit-sandbox-ok")'], options);
+    if (probe.code !== 0 || probe.reason || probe.stdout !== 'openmerit-sandbox-ok') {
+      throw Object.assign(new Error('Codex sandbox could not run a local probe. Fix host sandbox support before evaluating; no model request was made.'), { failureReason: 'sandbox_unavailable' });
+    }
+  }
+  return version.stdout.trim().slice(0, 160);
+}
+
+export async function runAgent(connection, prompt, options) {
+  const result = await execute(connection.command ?? connection.agent, invocation(connection, options.reservationUsd, options.unlimitedBudget), { ...options, input: prompt, env: agentEnvironment(connection.env) });
+  const parsed = parseOutput(connection.agent, result.stdout);
+  const failureReason = parsed.failureReason ?? (result.code !== 0 ? connectionFailure(result.stderr) : null) ?? (['spawn_failed', 'cancelled', 'output_limit'].includes(result.reason) ? result.reason : null);
+  const status = failureReason === 'budget_exhausted' ? 'budget_exhausted' : ['sandbox_unavailable', 'permission_denied', 'spawn_failed'].includes(failureReason) ? 'environment_error' : result.reason ?? (failureReason || result.code !== 0 || !parsed.completed ? 'agent_error' : 'ok');
+  return { ...parsed, costUsd: status === 'ok' ? parsed.costUsd : null, partialCostUsd: status === 'ok' ? parsed.partialCostUsd : maximum([parsed.costUsd, parsed.partialCostUsd]), costBasis: parsed.costUsd === null && parsed.partialCostUsd === null ? 'unknown' : 'agent_estimate', seconds: result.seconds, status, failureReason };
+}

package/dist/eval/src/cli.mjs

@@ -0,0 +1,151 @@
+import { mkdir, readFile, writeFile } from 'node:fs/promises';
+import { join, resolve } from 'node:path';
+import { createInterface } from 'node:readline/promises';
+import { agents, preflight } from './adapters.mjs';
+import { configPath, statePath, starter, loadConfig, readJSON, saveJSON } from './config.mjs';
+import { makePlan, savePlan } from './scout.mjs';
+import { acquireLock, evaluate, watch } from './runner.mjs';
+import { formatReport } from './report.mjs';
+
+export const help = `OpenMerit · Agent evaluations (source preview)
+
+  openmerit eval init                 Add a starter suite and project agent instructions
+  openmerit eval doctor               Check installed Claude Code, Codex and OpenCode CLIs
+  openmerit eval plan [--scope existing|all] [--budget <USD|unlimited>]
+                                     Choose scope and budget, then freeze Scout's shortlist
+  openmerit eval run [--json]         Evaluate the frozen shortlist in fresh workspaces
+  openmerit eval watch [--json]       Repeat after selected CLI/docs/fixture files change
+  openmerit eval report [--json]      Read the latest result without running models
+
+  --path <directory>                 Project directory (default: current directory)
+
+Configure .openmerit/eval.json: explicit agent connections, task fixtures, local
+graders, CLI variants, 3–10 repeats, and a cumulative budget. Init starts at $0.
+Plan asks for $10, a custom amount, or no fixed cap; nothing is preselected.
+Saved choices and prior budget usage persist across reruns. --budget changes
+the total allowance, not the amount remaining. Reservations are automatic.
+Scout receives task descriptions and, for existing scope, model IDs only.
+Run/watch execute local agents and may incur charges. No background daemon.
+`;
+
+const instructions = `<!-- openmerit-eval:start -->
+## OpenMerit agent evaluations
+
+When building or changing this project's CLI, tools or harness documentation, use
+\`openmerit eval\` to compare task performance. Skip this workflow when
+\`OPENMERIT_EVAL=1\` to avoid recursive evaluations.
+
+- Read .openmerit/eval.json and run \`openmerit eval doctor\`.
+- Ask once: "Models available through this project's existing connections, or all models Scout indexes?" Use \`eval plan --scope existing\` or \`--scope all\`. Reuse the saved choice. Never infer endpoint access from a model name or expose credentials.
+- Then ask once: "What budget should OpenMerit use for this project's evaluations? 1. $10; 2. Custom amount (how much?); 3. No fixed cap." Pass the explicit answer as \`eval plan --budget 10\`, \`--budget <amount>\` or \`--budget unlimited\`, together with scope on first setup. Never choose a default or infer unlimited spending. Reuse an existing positive or unlimited budget unless the user changes it. The allowance is cumulative across reruns, not reset by edits or replanning. Reservations are internal; do not ask the user to calculate them. No fixed cap still has bounded tasks, 3–10 repetitions, timeouts and user-controlled watch; subsequent watched edits can incur further charges until stopped.
+- Declare real connections with Scout model IDs, agent-native model IDs, and environment variable NAMES. Existing scope uses this project-declared inventory. All scope may require new authenticated connections before execution.
+- Replace starter cases with representative tasks, fresh fixtures and deterministic standalone Node graders. A grader reads {workspace, output} JSON on stdin and returns {quality: 0..1}. Keep answer keys outside agent workspaces. Include a bare variant and a CLI/docs variant where useful.
+- Confirm the suite and variants before enabling paid runs, using the budget already selected above. Do not ask again for an unchanged authorized budget. Then plan once, run a baseline, make the requested CLI changes, and run again. Use \`eval watch\` only when the user wants continuous local reruns; it exits with the terminal and retains budget usage across restarts.
+- Report quality, failures, tokens, reported costs and elapsed time per task/model/variant. A shortlist is a prior, not measured success. Do not claim missing costs are zero or that 3–10 repeats prove an exact ordering. Keep fresh holdouts for final validation.
+- If a compatible watcher is already active, let it run; never start overlapping loops. Do not overwrite the frozen plan merely because the CLI changed.
+<!-- openmerit-eval:end -->`;
+
+export async function initialize(root) {
+  // Refuse partial overwrite of an existing suite or someone's fixtures.
+  const files = {
+    [configPath]: JSON.stringify(starter, null, 2) + '\n',
+    'eval/fixtures/json/request.txt': 'Customer: Mira Chen\nItems: 7\n',
+    'eval/tool.mjs': `import {readFileSync} from 'node:fs';\nconst s=readFileSync(process.argv[2],'utf8');\nconsole.log(JSON.stringify({name:s.match(/Customer: (.*)/)?.[1],count:Number(s.match(/Items: (\\d+)/)?.[1])}));\n`,
+    'eval/grade.mjs': `let input=''; for await (const chunk of process.stdin) input+=chunk;\nconst {output}=JSON.parse(input); let quality=0;\ntry {const v=JSON.parse(output); quality=(Number(v.name==='Mira Chen')+Number(v.count===7)+Number(Object.keys(v).sort().join(',')==='count,name'))/3;} catch {}\nconsole.log(JSON.stringify({quality}));\n`,
+  };
+  for (const file of Object.keys(files)) {
+    try { await readFile(join(root, file)); throw new Error(`Already exists: ${file}. Init does not overwrite a suite.`); } catch (error) { if (error.code !== 'ENOENT') throw error; }
+  }
+  for (const [file, body] of Object.entries(files)) { await mkdir(join(root, file, '..'), { recursive: true }); await writeFile(join(root, file), body, { flag: 'wx' }); }
+  for (const name of ['AGENTS.md', 'CLAUDE.md']) {
+    let body = ''; try { body = await readFile(join(root, name), 'utf8'); } catch (e) { if (e.code !== 'ENOENT') throw e; }
+    if (!body.includes('<!-- openmerit-eval:start -->')) await writeFile(join(root, name), `${body}${body ? '\n\n' : ''}${instructions}\n`);
+  }
+  let ignore = ''; try { ignore = await readFile(join(root, '.gitignore'), 'utf8'); } catch (e) { if (e.code !== 'ENOENT') throw e; }
+  if (!ignore.split('\n').includes('/.openmerit/eval/')) await writeFile(join(root, '.gitignore'), `${ignore}${ignore.endsWith('\n') || !ignore ? '' : '\n'}/.openmerit/eval/\n`);
+}
+
+export function parseBudget(value) {
+  const input = value.trim();
+  if (input === 'unlimited') return input;
+  const amount = input.replace(/^\$/, '');
+  if (!/^(?:\d+(?:\.\d{1,2})?|\.\d{1,2})$/.test(amount) || Number(amount) <= 0 || Number(amount) > Number.MAX_SAFE_INTEGER / 100) throw new Error('Enter a positive USD amount with at most two decimal places, or "unlimited" for no fixed cap.');
+  return Number(amount);
+}
+
+// ask is supplied only for an interactive terminal. Coding agents pass explicit
+// flags after asking the user; EOF, blank input and missing flags never consent.
+export async function choosePlanOptions(config, { scope, budget, ask, tell = () => {} } = {}) {
+  const next = structuredClone(config);
+  if (scope) next.scope = scope;
+  while (!['existing', 'all'].includes(next.scope)) {
+    if (!ask) throw new Error('Choose a scope with eval plan --scope existing or --scope all, then choose --budget <USD|unlimited>.');
+    const answer = (await ask('Use existing project connections or all models Scout indexes? [existing/all] ')).trim();
+    if (['existing', 'all'].includes(answer)) next.scope = answer;
+    else tell('Choose existing or all.');
+  }
+  if (budget !== undefined) next.budget = { maxUsd: parseBudget(budget), reservationUsd: 'auto' };
+  while (next.budget.maxUsd === 0) {
+    if (!ask) throw new Error('Choose an evaluation budget after model scope: --budget 10, --budget <USD>, or --budget unlimited. No amount is selected automatically.');
+    const choice = (await ask('What budget should OpenMerit use for this project’s evaluations?\n  1. $10\n  2. Custom amount\n  3. No fixed cap (watched changes can incur further charges until stopped)\nChoose [1/2/3]: ')).trim();
+    if (choice === '1') next.budget = { maxUsd: 10, reservationUsd: 'auto' };
+    else if (choice === '3') next.budget = { maxUsd: 'unlimited', reservationUsd: 'auto' };
+    else if (choice === '2') {
+      while (next.budget.maxUsd === 0) {
+        const amount = await ask('How much in USD? ');
+        try {
+          const parsed = parseBudget(amount);
+          if (parsed === 'unlimited') throw new Error('Enter a custom USD amount.');
+          next.budget = { maxUsd: parsed, reservationUsd: 'auto' };
+        } catch { tell('Enter a positive USD amount with at most two decimal places.'); }
+      }
+    } else tell('Choose 1, 2 or 3. Nothing is selected automatically.');
+  }
+  return next;
+}
+
+export async function main(args) {
+  if (!args.length || args.includes('--help') || args.includes('-h')) { console.log(help); return; }
+  const action = args.shift(); let root = process.cwd(), scope, budget, json = false;
+  for (let i = 0; i < args.length; i++) {
+    if (args[i] === '--path' && args[i + 1]) root = resolve(args[++i]);
+    else if (args[i] === '--scope' && ['existing', 'all'].includes(args[i + 1])) scope = args[++i];
+    else if (args[i] === '--budget' && args[i + 1] && budget === undefined) { budget = args[++i]; parseBudget(budget); }
+    else if (args[i] === '--json') json = true;
+    else throw new Error(`Unknown eval argument: ${args[i]}`);
+  }
+  if (scope && action !== 'plan') throw new Error('--scope belongs to eval plan.');
+  if (budget !== undefined && action !== 'plan') throw new Error('--budget belongs to eval plan.');
+  if (action === 'init') { await initialize(root); console.log('Created .openmerit/eval.json, starter fixtures, and AGENTS.md / CLAUDE.md instructions. Declare connections, replace the starter task, and set an approved budget before running.'); return; }
+  if (action === 'doctor') {
+    const results = [];
+    for (const agent of agents) { try { results.push({ agent, version: await preflight({ agent }), ready: true }); } catch (error) { results.push({ agent, ready: false, reason: error.message }); } }
+    console.log(JSON.stringify(results, null, 2)); return;
+  }
+  if (action === 'report') { const report = await readJSON(join(root, statePath, 'latest.json')); console.log(json ? JSON.stringify(report, null, 2) : formatReport(report)); return; }
+  if (!['plan', 'run', 'watch'].includes(action)) throw new Error('Unknown eval command. Run openmerit eval --help.');
+  const unlock = await acquireLock(root);
+  const controller = new AbortController(), stop = () => controller.abort();
+  process.once('SIGINT', stop); process.once('SIGTERM', stop);
+  try {
+    if (action === 'plan') {
+      const saved = await loadConfig(root);
+      const rl = process.stdin.isTTY && !json ? createInterface({ input: process.stdin, output: process.stdout }) : null;
+      const inputClosed = new AbortController();
+      rl?.once('close', () => inputClosed.abort());
+      let config;
+      try { config = await choosePlanOptions(saved, { scope, budget, ask: rl ? question => rl.question(question, { signal: AbortSignal.any([controller.signal, inputClosed.signal]) }) : undefined, tell: message => console.log(message) }); }
+      finally { rl?.close(); }
+      // Preserve explicit choices even if Scout is temporarily unavailable.
+      await saveJSON(join(root, configPath), config);
+      const plan = await makePlan(config);
+      await savePlan(root, plan);
+      console.log(JSON.stringify(plan, null, 2));
+      return;
+    }
+    const onReport = report => { console.log(json ? JSON.stringify(report) : formatReport(report)); if (report.status !== 'complete') process.exitCode = 1; };
+    const onProgress = json ? () => {} : row => process.stderr.write(`${row.task} · ${row.connection} · ${row.variant} · ${row.round}: ${row.status}\n`);
+    if (action === 'run') onReport(await evaluate(root, { signal: controller.signal, onProgress }));
+    else { if (!json) console.log('Watching selected files. Ctrl+C stops; budget usage persists.'); await watch(root, { signal: controller.signal, onReport, onProgress }); }
+  } finally { process.removeListener('SIGINT', stop); process.removeListener('SIGTERM', stop); await unlock(); }
+}

package/dist/eval/src/config.mjs

@@ -0,0 +1,108 @@
+import { readFile, writeFile, mkdir, rename, readdir, lstat, realpath } from 'node:fs/promises';
+import { resolve, relative, join, dirname, isAbsolute } from 'node:path';
+import { createHash } from 'node:crypto';
+import { agents } from './adapters.mjs';
+
+export const configPath = '.openmerit/eval.json';
+export const statePath = '.openmerit/eval';
+export const digest = value => createHash('sha256').update(JSON.stringify(value)).digest('hex');
+export async function readJSON(path) { return JSON.parse(await readFile(path, 'utf8')); }
+export async function saveJSON(path, data) {
+  await mkdir(dirname(path), { recursive: true });
+  const temp = `${path}.${process.pid}.tmp`;
+  await writeFile(temp, JSON.stringify(data, null, 2) + '\n', { mode: 0o600 }); await rename(temp, path);
+}
+const text = v => typeof v === 'string' && v.trim().length > 0;
+const id = v => text(v) && /^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,79}$/.test(v);
+function requireThat(ok, message) { if (!ok) throw new Error(`Eval config: ${message}`); }
+export function safePath(path) {
+  requireThat(text(path) && !isAbsolute(path) && !path.split(/[\\/]/).some(p => ['..', '.', '.git', '.openmerit', 'node_modules'].includes(p) || p.startsWith('.env')), `unsafe file path ${path}. Use explicit project-relative source/fixture paths, without secrets or symlinks.`);
+  return path;
+}
+export function validateConfig(config) {
+  requireThat(config.version === 1, 'version must be 1.');
+  requireThat([null, 'existing', 'all'].includes(config.scope), 'scope must be null, existing or all.');
+  requireThat(Array.isArray(config.connections) && config.connections.length <= 100, 'connections must be a list of up to 100 routes.');
+  requireThat(new Set(config.connections.map(c => c.id)).size === config.connections.length, 'connection IDs must be unique.');
+  for (const c of config.connections) {
+    requireThat(id(c.id) && agents.includes(c.agent) && text(c.scoutId) && text(c.model), 'each connection needs id, scoutId, agent and native model.');
+    requireThat(!c.model.startsWith('-') && c.model.length <= 200, 'invalid native model.');
+    requireThat(c.env === undefined || Array.isArray(c.env) && c.env.every(k => /^[A-Z][A-Z0-9_]*$/.test(k)), 'env contains variable names, never credential values.');
+    requireThat(c.command === undefined || text(c.command), 'command must be an executable path.');
+  }
+  requireThat(Array.isArray(config.tasks) && config.tasks.length > 0 && config.tasks.length <= 30, 'provide 1–30 tasks.');
+  requireThat(new Set(config.tasks.map(t => t.id)).size === config.tasks.length, 'task IDs must be unique.');
+  for (const t of config.tasks) {
+    requireThat(id(t.id) && text(t.description) && t.description.length <= 4000 && text(t.prompt), 'each task needs id, description for Scout and a local execution prompt.');
+    safePath(t.fixture); safePath(t.grader);
+  }
+  requireThat(Array.isArray(config.variants) && config.variants.length > 0 && config.variants.length <= 5, 'provide 1–5 variants.');
+  requireThat(new Set(config.variants.map(v => v.id)).size === config.variants.length, 'variant IDs must be unique.');
+  for (const v of config.variants) {
+    requireThat(id(v.id) && Array.isArray(v.files) && typeof v.instructions === 'string', 'each variant needs id, files and instructions.');
+    v.files.forEach(safePath);
+  }
+  requireThat(Number.isInteger(config.repeats?.min) && Number.isInteger(config.repeats?.max) && config.repeats.min >= 3 && config.repeats.max <= 10 && config.repeats.min <= config.repeats.max, 'repeats must be between 3 and 10.');
+  requireThat((config.budget?.maxUsd === 'unlimited' || Number.isFinite(config.budget?.maxUsd) && config.budget.maxUsd >= 0) && (config.budget.reservationUsd === 'auto' || Number.isFinite(config.budget.reservationUsd) && config.budget.reservationUsd > 0), 'set nonnegative maxUsd or explicit "unlimited", and reservationUsd to "auto" or a positive amount.');
+  requireThat(Number.isInteger(config.timeoutSeconds) && config.timeoutSeconds >= 1 && config.timeoutSeconds <= 1800, 'timeoutSeconds must be 1–1800.');
+  if (config.scoutURL !== undefined) {
+    const url = new URL(config.scoutURL);
+    requireThat(!url.username && !url.password && (url.protocol === 'https:' || url.protocol === 'http:' && ['127.0.0.1', 'localhost', '[::1]'].includes(url.hostname)), 'scoutURL requires HTTPS, or HTTP on loopback for development.');
+  }
+  return config;
+}
+export async function loadConfig(root) { return validateConfig(await readJSON(join(root, configPath))); }
+
+export function reservationFor(config, plan) {
+  if (config.budget.reservationUsd !== 'auto') return config.budget.reservationUsd;
+  const attempts = plan.tasks.reduce((total, task) => total + task.candidates.reduce((n, model) => n + model.connections.length, 0), 0) * config.variants.length * config.repeats.min;
+  return config.budget.maxUsd === 'unlimited' || config.budget.maxUsd === 0 ? .25 : Math.min(.25, config.budget.maxUsd / Math.max(1, attempts));
+}
+
+// Snapshot exact bytes before executing; edits during a run belong to the next revision.
+export async function filesAt(root, paths) {
+  const files = Object.create(null); let bytes = 0; const canonicalRoot = await realpath(root);
+  async function visit(path) {
+    safePath(path);
+    const full = resolve(root, path), info = await lstat(full);
+    if (info.isSymbolicLink() || relative(canonicalRoot, await realpath(full)).startsWith('..')) throw new Error(`Eval refuses symlink/outside path: ${path}`);
+    if (info.isDirectory()) {
+      for (const name of (await readdir(full)).sort()) await visit(join(path, name));
+    } else if (info.isFile()) {
+      bytes += info.size;
+      if (bytes > 25_000_000) throw new Error('Eval fixture/source snapshot exceeds 25 MB. Select smaller paths.');
+      files[path] = { data: (await readFile(full)).toString('base64'), executable: !!(info.mode & 0o111) };
+    } else throw new Error(`Eval requires regular files: ${path}`);
+  }
+  for (const path of paths) await visit(path);
+  return files;
+}
+export async function snapshot(root, config) {
+  const tasks = [];
+  for (const task of config.tasks) {
+    if (!(await lstat(join(root, task.fixture))).isDirectory() || !(await lstat(join(root, task.grader))).isFile()) throw new Error(`${task.id}: fixture must be a directory and grader a standalone file.`);
+    tasks.push({ ...task, fixtureFiles: await filesAt(root, [task.fixture]), graderFiles: await filesAt(root, [task.grader]) });
+  }
+  const variants = [];
+  for (const variant of config.variants) variants.push({ ...variant, sourceFiles: await filesAt(root, variant.files) });
+  for (const task of tasks) for (const variant of variants) {
+    const fixtureNames = new Set(Object.keys(task.fixtureFiles).map(path => relative(task.fixture, path)));
+    if (Object.keys(variant.sourceFiles).some(path => fixtureNames.has(path))) throw new Error(`${variant.id} overwrites a fixture file for ${task.id}. Keep CLI files and task inputs separate.`);
+  }
+  const compatibility = digest({ tasks, connections: config.connections, repeats: config.repeats, variantIds: variants.map(v => v.id), timeoutSeconds: config.timeoutSeconds });
+  return { tasks, variants, compatibility, revision: digest({ tasks, variants, config }) };
+}
+export async function materialize(files, dest, strip = '') {
+  for (const [path, file] of Object.entries(files)) {
+    const target = join(dest, strip ? relative(strip, path) : path);
+    await mkdir(dirname(target), { recursive: true });
+    await writeFile(target, Buffer.from(file.data, 'base64'), { mode: file.executable ? 0o700 : 0o600 });
+  }
+}
+
+export const starter = {
+  version: 1, scope: null, connections: [],
+  tasks: [{ id: 'json', description: 'Extract a customer name and item count from text into a strict JSON object.', prompt: 'Read request.txt. Return only JSON with keys name (string) and count (number).', fixture: 'eval/fixtures/json', grader: 'eval/grade.mjs' }],
+  variants: [{ id: 'bare', files: [], instructions: '' }, { id: 'cli', files: ['eval/tool.mjs'], instructions: 'You can use node eval/tool.mjs request.txt to extract the fields.' }],
+  repeats: { min: 3, max: 10 }, budget: { maxUsd: 0, reservationUsd: 'auto' }, timeoutSeconds: 180,
+};

package/dist/eval/src/process.mjs

@@ -0,0 +1,44 @@
+import { spawn } from 'node:child_process';
+
+// No shell interpolation. Bound both time and output, and stop descendants too.
+export function execute(command, args, { cwd, env = process.env, input = '', timeoutMs = 180_000, signal } = {}) {
+  return new Promise(resolve => {
+    const start = performance.now();
+    let stdout = '', stderr = '', bytes = 0, reason = null, escalation;
+    const child = spawn(command, args, { cwd, env, detached: process.platform !== 'win32', stdio: ['pipe', 'pipe', 'pipe'] });
+    const kill = sig => {
+      try {
+        if (process.platform === 'win32') { const killer = spawn('taskkill', ['/pid', String(child.pid), '/T', '/F'], { stdio: 'ignore' }); killer.on('error', () => child.kill(sig)); }
+        else process.kill(-child.pid, sig);
+      } catch {}
+    };
+    const stop = why => { if (reason) return; reason = why; kill('SIGTERM'); escalation = setTimeout(() => kill('SIGKILL'), 500); };
+    const abort = () => stop('cancelled');
+    signal?.addEventListener('abort', abort, { once: true });
+    if (signal?.aborted) abort();
+    const timer = setTimeout(() => stop('timeout'), timeoutMs);
+    const collect = (chunk, which) => {
+      bytes += chunk.length;
+      if (bytes > 4_000_000) return stop('output_limit');
+      if (which === 'out') stdout += chunk; else stderr += chunk;
+    };
+    child.stdout.on('data', chunk => collect(chunk, 'out'));
+    child.stderr.on('data', chunk => collect(chunk, 'err'));
+    child.on('error', () => { reason = 'spawn_failed'; });
+    child.stdin.on('error', () => {});
+    child.stdin.end(input);
+    child.on('close', code => {
+      clearTimeout(timer); clearTimeout(escalation); signal?.removeEventListener('abort', abort);
+      resolve({ code, reason, stdout, stderr, seconds: (performance.now() - start) / 1000 });
+    });
+  });
+}
+
+export function agentEnvironment(names = []) {
+  const result = {};
+  for (const key of ['PATH', 'HOME', 'USERPROFILE', 'SystemRoot', 'TMPDIR', 'TMP', 'TEMP', 'LANG', 'TERM', ...names]) {
+    if (process.env[key] !== undefined) result[key] = process.env[key];
+  }
+  result.OPENMERIT_EVAL = '1';
+  return result;
+}