openmerit 0.1.6-preview.5 → 0.1.7-preview.1

package/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 0.1.7-preview.1 — 2026-10-15
+
+- Add `openmerit eval init/doctor/plan/run/watch/report`: Scout shortlists, explicit model scope, Claude Code/Codex/OpenCode adapters, fresh task attempts, local graders, balanced adaptive 3–10 repetitions, revision comparisons and cumulative budget accounting.
+- Bundle project agent instructions without install-time hooks. Missing usage/cost remains unknown and partial comparisons are disclosed. Terminal/Pi behavior is unchanged.
+- Use the hosted Scout HTTP/MCP `modelIds` filter to select eligible candidates before ranking. This API change was deployed separately on 15 October 2026.
+
 ## 0.1.6-preview.5 — 2026-10-10
 
 - Open the terminal before initial collection, then index repository declarations in a background worker with visible discovery, indexing, and linking progress. Replace repository-wide scan caps with continuing work batches.

package/README.md

@@ -1,137 +1,97 @@
 # OpenMerit
 
-OpenMerit is a harness-neutral orchestration layer that helps a coding harness continuously find and maintain the Pareto frontier of models for a real product task.
+OpenMerit provides a local Inference Terminal for understanding an application's models, services, recorded activity, and compute, plus a Pi integration for evidence-driven model comparisons.
 
-OpenMerit does not perform the intelligent work. It nudges a connected coding harness, such as Pi, with typed outcome requests. The harness understands the project, establishes evaluations and observability, gathers evidence, discovers and tests candidates, calculates the Pareto frontier, and performs an authorized model change. OpenMerit validates the returned contracts, preserves evidence and preferences, and advances the approved lifecycle.
+The terminal reads existing evidence. The Pi integration asks a coding harness to build evaluations, measure candidates, and carry out authorized changes. OpenMerit coordinates that workflow and validates its structured results; the terminal does not route requests, run evaluations, or swap models.
 
-The npm package `openmerit` contains the Pi extension, the reusable coordinator,
-and the protocol in one install. The Pi adapter is the first integration; other
-harnesses can use the same core and protocol exports.
+## Open the terminal
 
-## Why OpenMerit
+Requires Node.js 22.19 or newer and npm:
 
-The cheapest model, fastest model, and highest-quality model are often different. Public benchmarks are useful before product evidence exists, but they cannot establish which model is best for a particular workload. OpenMerit connects the initial choice to production evidence and repeated reassessment.
-
-It distinguishes:
-
-- one-run observations from rate, percentile, consistency, and reliability claims;
-- missing evidence from poor performance;
-- feasible candidates from dominated, unresolved, or ineligible candidates;
-- an automatic nudge from an intelligent action performed by the harness;
-- a recommendation from an authorized, verified model swap.
-
-## Current capabilities
-
-- Pi 0.87 extension with a single `/openmerit` command surface.
-- Harness-inferred, user-confirmed task profiles with explicit metrics, sampling designs, constraints, and evaluation budgets.
-- The complete initial quality, reliability, cost, latency, efficiency, and human-intervention metric catalogue.
-- Setup-time preliminary model research in the source checkout, followed by baseline evidence collection, refreshed candidate discovery, and controlled challenger trials. The published `0.1.5` package still begins candidate discovery after the baseline.
-- A normative, uncertainty-aware Pareto definition owned by OpenMerit.
-- Harness-calculated frontier results independently checked by OpenMerit's conformance verifier.
-- Versioned result schemas for all ten intents, reusable across harness adapters and exposed dynamically by Pi.
-- Configurable swap approval, post-swap verification requests, and policy-based rollback requests. Automatic swaps can begin immediately if explicitly enabled with a zero verified-swap threshold.
-- Project-local snapshots, redacted append-only audit events, evidence manifests, and optional best-effort event exporters under `.openmerit/`.
-- Model-catalog change detection that can nudge the harness to reassess a mature baseline.
-- User-confirmed task-count, elapsed-time, regression, catalogue-change, and post-swap check policies.
-- Idempotent harness signals, durable pending work, native wakeup planning, and explicit scheduler capability gaps.
-- In the unreleased source checkout, an opt-in macOS Pi LaunchAgent for time-based checks and a durable pause; published `0.1.5` still requires an external scheduler and has a partial, session-local pause.
+```sh
+npm install -g openmerit
+cd /path/to/your/application
+openmerit dash
+```
 
-## Responsibility boundary
+Stable `openmerit@0.1.6` includes the terminal. No Pi installation or provider API key is needed. It opens a browser connected to a read-only loopback service. Supported source/configuration declarations establish inventory; existing application logs or exports supply request history. Prompts, responses, and credentials are not retained. Unsupported or missing evidence remains unknown.
 
-| OpenMerit | Coding harness |
-| --- | --- |
-| Defines typed outcomes and Pareto conformance rules | Understands the product and user task |
-| Tracks lifecycle, policy, budget, and evidence references | Builds evals and observability |
-| Determines which approved nudge is due | Collects and calculates metrics |
-| Verifies returned structure and frontier conformance | Discovers candidates and calculates the frontier |
-| Selects intents according to confirmation and rollback policy | Applies and verifies an authorized model change |
+The source interface and hosted dashboard follow the device's light/dark preference without a theme control. This appearance change is not included in npm 0.1.6.
 
-Jev System One, when configured, belongs to the coding harness. Pi may use it aggressively for bounded ranking, triage, and fast judgment. OpenMerit contains no Jev client and accepts no Jev credential.
+Explore the [hosted sample](https://dash.openmerit.ai/) or [before/after dashboard demo](https://dash.openmerit.ai/demo). Both use fictional data and make no inference calls. See [the terminal guide](docs/inference-terminal.md) for configuration, discovery, diagnostics, history limits, and agent access.
 
-## Install with Pi
+## Compare models with Pi
 
-Requirements:
+The **npm preview `0.1.7-preview.1`** adds `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose existing connections or Scout's full index, freeze a shortlist, and compare 3–10 fresh attempts after source changes. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@0.1.7-preview.1/docs/agent-evaluations.md) for setup, graders, budgets and availability. This is separate from the Pi workflow below.
 
-- Node.js 22.19 or newer
-- Pi 0.87 with a supported model provider configured
+With Pi 0.87 and a model provider configured:
 
 ```sh
 pi install npm:openmerit
 pi list
 ```
 
-When Pi enters an unconfigured project with an interactive UI, the adapter first looks for an actual application LLM call pattern. If the project starts empty, the unreleased source checkout scans again after each settled build turn and automatically begins setup when Pi creates the application call. `/openmerit setup` remains a recovery command rather than a required demo step. Pi confirms the application target and exact incumbent application model, infers task-relevant metrics and their repeated-input or representative-case sampling designs, explains the proposed collection plan and estimated cost, and asks the user to confirm or edit it. It then establishes task-specific eval and observability artifacts, verifies them, and reports evidence to OpenMerit. Pi's own model is never the application target.
+Open the product repository in interactive Pi. An unconfigured project is checked for an application LLM call at session start and after settled build turns. Pi confirms the application target and incumbent model, proposes task-specific metrics and sampling, explains estimated evaluation cost, and requests confirmation of the plan, budget, and permissions. `/openmerit setup` can recover or rerun setup. Pi's own model is never the application target.
 
-Run `/openmerit doctor` inside Pi to check the installation. OpenMerit keeps
-project evidence under `.openmerit/` in the product repository. Version 0.1.5
-replaces the 0.1.4 background watcher and standalone CLI with this
-harness-driven lifecycle; it does not migrate the earlier home-directory state.
+Core checks stored baseline windows against the confirmed requirements. When ready, it asks Pi to discover candidates, run controlled comparisons, and calculate a frontier. Core validates the returned evidence contracts and Pareto conformance. The harness must execute the work and provide honest evidence.
 
-## Commands
+Model changes normally require approval. Explicit automatic-swap opt-in can allow changes once the configured verified-swap threshold is reached, including a threshold of zero. Post-swap evaluation and conditional rollback depend on the harness performing and reporting them.
 
-- `/openmerit` or `/openmerit status` — show evidence readiness and lifecycle stage.
-- `/openmerit setup` — explicitly rerun or recover automatic setup.
-- `/openmerit assess` — check stored application metric windows against the baseline thresholds now (in the unreleased source checkout); start candidate discovery only when ready. Published `0.1.5` still asks Pi to assess.
-- `/openmerit logs` — show the canonical audit, exporter-error, evidence, and state paths.
-- `/openmerit frontier` — ask the harness to calculate a frontier now; OpenMerit verifies it.
-- `/openmerit approve` — approve a verified proposal during supervised graduation.
-- `/openmerit pause` and `/openmerit resume` — control some proactive nudges in the current Pi session; task-triggered checks can still run in 0.1.5. See [pause limitations](docs/commands.md#pause-and-resume).
-- `/openmerit doctor` — show sanitized adapter diagnostics.
+Read [getting started](docs/getting-started.md) and [your first supervised trial](docs/first-trial.md).
 
-## Inference Terminal (test preview)
+## Implemented orchestration
 
-A local, dark-only overview of application inference inventory, connections, activity, and compute. Coding-harness usage is outside its scope. It reads existing metadata passively and makes no model calls.
+- Task profiles with confirmed metrics, sampling plans, constraints, budgets, and permissions.
+- Preliminary model research, incumbent identity, durable candidate handoffs, and controlled challenger experiments.
+- Deterministic baseline checks and uncertainty-aware frontier validation.
+- Versioned result schemas for all ten intents and a reusable harness adapter boundary.
+- Challenger-spend accounting, bounded intent admission, approval, verification, and rollback requests.
+- Project-local state, redacted audit events, evidence manifests, and optional event exporters.
+- Task-count, elapsed-time, catalogue-change, regression, and post-swap signals.
+- Durable pause and authorized macOS user LaunchAgents; other platforms require an external scheduler.
 
-With Node.js 22.19+ installed, run this from your application's directory:
+These implementations have automated coverage. They do not establish application-quality improvements or reliable unattended model changes. See [current limitations](docs/roadmap.md).
 
-```sh
-npm install -g openmerit@preview
-openmerit dash
-```
+## Pi commands
 
-The preview uses the existing `openmerit` npm package, including its coordinator, protocol, Pi adapter, and compiled terminal assets. It opens its localhost URL in your default browser. The stable npm package `openmerit@0.1.5` does not include this command. See the [Inference Terminal guide](docs/inference-terminal.md) for supported sources, updating or removing the preview, source builds, and read-only agent access.
+- `/openmerit` or `/openmerit status` — inspect readiness, active work, and lifecycle state.
+- `/openmerit setup` — request or recover setup.
+- `/openmerit assess` — check stored baseline windows; discover candidates only when ready.
+- `/openmerit frontier` — request a harness-calculated comparison for validation.
+- `/openmerit approve` — approve the current verified proposal.
+- `/openmerit pause` / `/openmerit resume` — persist or clear the automatic-work pause and reconcile authorized scheduling; active work is not cancelled.
+- `/openmerit cancel` — record abandonment of an interrupted intent; it does not terminate a provider request.
+- `/openmerit logs` — locate local state, audit, and evidence files.
+- `/openmerit doctor` — inspect adapter and scheduler diagnostics.
 
 ## Documentation
 
-Read the searchable [OpenMerit documentation](https://openmerit.site/docs/). The Markdown sources below also ship with the package.
+The [documentation overview](docs/index.md) links all current guides. The website at [openmerit.ai/docs](https://openmerit.ai/docs/) is deployed separately from the package and this checkout.
 
-- [Getting started](docs/getting-started.md)
-- [Your first trial](docs/first-trial.md)
-- [Command reference](docs/commands.md)
-- [Budgets and permissions](docs/budgets.md)
-- [Troubleshooting](docs/troubleshooting.md)
-- [Architecture and responsibility boundary](docs/architecture.md)
+- [Inference Terminal](docs/inference-terminal.md)
 - [Metrics and evidence](docs/metrics-and-evidence.md)
-- [Improvement lifecycle](docs/lifecycle.md)
-- [Harness-neutral automation](docs/automation.md)
-- [Pi extension guide](docs/pi-extension.md)
-- [Building another harness adapter](docs/adapter-guide.md)
-- [Pareto conformance specification](docs/pareto-spec.md)
-- [Project state and operations](docs/operations.md)
-- [Security](docs/security.md)
-- [Testing](docs/testing.md)
-- [Current limitations and roadmap](docs/roadmap.md)
-
-### Maintaining documentation
+- [Lifecycle](docs/lifecycle.md), [automation](docs/automation.md), and [budgets](docs/budgets.md)
+- [Commands](docs/commands.md), [operations](docs/operations.md), and [troubleshooting](docs/troubleshooting.md)
+- [Security and data handling](docs/security.md)
+- [Architecture](docs/architecture.md), [Pi adapter](docs/pi-extension.md), and [adapter development](docs/adapter-guide.md)
+- [Pareto specification](docs/pareto-spec.md) and [testing](docs/testing.md)
 
-Edit the Markdown in `docs/` alongside product changes. The public documentation is generated from these same files; there is no separate article copy to update in `site/`. Run `npm run check:docs` to rebuild and validate it. The normal Cloudflare deployment rebuilds and publishes the documentation. See [the documentation workflow](docs/site-documentation.md) for adding guides and previewing changes.
-
-## Repository layout
-
-- `packages/protocol` — harness-neutral types for intents, results, metrics, policies, and frontier evidence.
-- `packages/core` — adapter SDK, durable coordinator, pluggable persistence, policy, and frontier conformance verification.
-- `packages/pi` — Pi extension that turns OpenMerit intents into harness work.
-- `docs` — user, operator, architecture, and normative documentation.
-
-The workspace modules are private development boundaries. The published
-package exposes `openmerit/core`, `openmerit/protocol`, and `openmerit/pi`.
-
-## Development from source
+## Development
 
 ```sh
 npm ci
 npm run verify
+npm run dash -- /path/to/application
+```
+
+For Pi extension development:
+
+```sh
 ./node_modules/.bin/pi --no-extensions -e ./packages/pi/src/index.ts
 ```
 
-`--no-extensions` prevents an installed npm copy of OpenMerit from registering the same tools as the source extension. Credentials must never be committed. The repository intentionally contains no provider or Jev API key.
+`--no-extensions` avoids registering an installed npm copy alongside the source extension. Credentials belong to the harness or its environment and must never be committed. Jev, if configured, is invoked by the harness; OpenMerit provides no Jev client.
+
+`packages/terminal` contains the CLI, collectors, hosted sample, and browser interface. `packages/protocol`, `packages/core`, and `packages/pi` provide the protocol, coordinator, and Pi adapter. The package exports `openmerit/core`, `openmerit/protocol`, and `openmerit/pi`.
+
+Edit documentation in `docs/`, then run `npm run check:docs`. The website is generated from those Markdown files. See [documentation maintenance](docs/site-documentation.md) and the [production-source reconciliation note](docs/site-domain-migration.md) before deployment; a local build does not publish changes.

package/dist/eval/src/adapters.mjs

@@ -0,0 +1,73 @@
+import { execute, agentEnvironment } from './process.mjs';
+
+const number = value => typeof value === 'number' && Number.isFinite(value) && value >= 0 ? value : null;
+const sum = values => values.length && values.every(v => v !== null) ? values.reduce((a, b) => a + b, 0) : null;
+export const agents = ['claude', 'codex', 'opencode'];
+
+export function connectionFailure(text) {
+  if (/401|unauthorized|authentication[_ -]?(?:error|required)|invalid api key|not logged in|please log in/i.test(text)) return 'authentication';
+  if (/model.{0,100}(?:not supported|not found|not available|does not exist)|(?:unknown|invalid|unsupported) model|ProviderModelNotFoundError/i.test(text)) return 'model_unavailable';
+  if (/rejected permission|permission.{0,30}(?:denied|rejected)|not allowed to use.{0,30}tool/i.test(text)) return 'permission_denied';
+  return null;
+}
+
+export function invocation(connection, reservationUsd) {
+  const model = connection.model;
+  if (connection.agent === 'claude') return ['-p', '--output-format', 'json', '--model', model, '--no-session-persistence', '--permission-mode', 'dontAsk', '--allowedTools', 'Read,Glob,Grep,Bash,Edit,Write', '--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}', '--setting-sources', '', '--max-budget-usd', String(reservationUsd)];
+  if (connection.agent === 'codex') return ['exec', '--json', '--model', model, '--sandbox', 'workspace-write', '--skip-git-repo-check', '--ephemeral', '-'];
+  if (connection.agent === 'opencode') return ['run', '--format', 'json', '--model', model, '--auto'];
+  throw new Error('Unknown agent. Use claude, codex or opencode.');
+}
+
+export function parseOutput(agent, stdout) {
+  let events;
+  try { const parsed = JSON.parse(stdout); events = Array.isArray(parsed) ? parsed : [parsed]; }
+  catch { events = stdout.split('\n').flatMap(line => { try { return [JSON.parse(line)]; } catch { return []; } }); }
+  let output = '', usage = null, costUsd = null, completed = false, failed = false;
+  const observedModels = new Set();
+  if (agent === 'claude') {
+    const result = events.findLast(e => e.type === 'result');
+    if (result) {
+      completed = result.subtype === 'success' && !result.is_error;
+      failed = !completed; output = result.result ?? '';
+      const u = result.usage;
+      if (u) usage = { input: sum([number(u.input_tokens), number(u.cache_read_input_tokens ?? 0), number(u.cache_creation_input_tokens ?? 0)]), output: number(u.output_tokens), cached: number(u.cache_read_input_tokens ?? 0) };
+      costUsd = number(result.total_cost_usd);
+      for (const model of Object.keys(result.modelUsage ?? {})) observedModels.add(model);
+    }
+  } else if (agent === 'codex') {
+    const turns = events.filter(e => e.type === 'turn.completed');
+    completed = turns.length > 0;
+    failed = events.some(e => ['turn.failed', 'error'].includes(e.type));
+    output = events.findLast(e => e.type === 'item.completed' && e.item?.type === 'agent_message')?.item.text ?? '';
+    if (turns.length) usage = { input: sum(turns.map(e => number(e.usage?.input_tokens))), output: sum(turns.map(e => number(e.usage?.output_tokens))), cached: sum(turns.map(e => number(e.usage?.cached_input_tokens))) };
+  } else if (agent === 'opencode') {
+    const steps = events.filter(e => e.type === 'step_finish');
+    const finalStep = steps.findLast(e => ['stop', 'end_turn'].includes(e.part?.reason));
+    completed = !!finalStep;
+    failed = events.some(e => e.type === 'error');
+    const texts = events.filter(e => e.type === 'text');
+    output = finalStep?.part?.messageID ? texts.filter(e => e.part?.messageID === finalStep.part.messageID).map(e => e.part?.text ?? '').join('\n') : texts.at(-1)?.part?.text ?? '';
+    if (steps.length) {
+      usage = { input: sum(steps.map(e => sum([number(e.part?.tokens?.input), number(e.part?.tokens?.cache?.read ?? 0), number(e.part?.tokens?.cache?.write ?? 0)]))), output: sum(steps.map(e => sum([number(e.part?.tokens?.output), number(e.part?.tokens?.reasoning ?? 0)]))), cached: sum(steps.map(e => number(e.part?.tokens?.cache?.read ?? 0))) };
+      costUsd = sum(steps.map(e => number(e.part?.cost)));
+    }
+  }
+  return { output, usage, costUsd, completed: completed && !failed, observedModels: [...observedModels] };
+}
+
+export async function preflight(connection) {
+  const command = connection.command ?? connection.agent;
+  const version = await execute(command, ['--version'], { timeoutMs: 10_000, env: agentEnvironment(connection.env) });
+  const help = await execute(command, connection.agent === 'codex' ? ['exec', '--help'] : connection.agent === 'opencode' ? ['run', '--help'] : ['--help'], { timeoutMs: 10_000, env: agentEnvironment(connection.env) });
+  const flags = connection.agent === 'codex' ? ['--json', '--ephemeral', '--sandbox'] : connection.agent === 'claude' ? ['--output-format', '--no-session-persistence', '--max-budget-usd'] : ['--format', '--model', '--auto'];
+  if (version.code !== 0 || help.code !== 0 || !flags.every(f => (help.stdout + help.stderr).includes(f))) throw new Error(`${connection.agent} lacks the required non-interactive JSON interface. Update it or set this connection's command to a compatible executable.`);
+  return version.stdout.trim().slice(0, 160);
+}
+
+export async function runAgent(connection, prompt, options) {
+  const result = await execute(connection.command ?? connection.agent, invocation(connection, options.reservationUsd), { ...options, input: prompt, env: agentEnvironment(connection.env) });
+  const parsed = parseOutput(connection.agent, result.stdout);
+  const status = result.reason ?? (result.code !== 0 || !parsed.completed ? 'agent_error' : 'ok');
+  return { ...parsed, costUsd: status === 'ok' ? parsed.costUsd : null, partialCostUsd: status === 'ok' ? null : parsed.costUsd, seconds: result.seconds, status, failureReason: status === 'agent_error' ? connectionFailure(result.stdout + result.stderr) : null };
+}

package/dist/eval/src/cli.mjs

@@ -0,0 +1,103 @@
+import { mkdir, readFile, writeFile } from 'node:fs/promises';
+import { join, resolve } from 'node:path';
+import { createInterface } from 'node:readline/promises';
+import { agents, preflight } from './adapters.mjs';
+import { configPath, statePath, starter, loadConfig, readJSON, saveJSON } from './config.mjs';
+import { makePlan, savePlan } from './scout.mjs';
+import { acquireLock, evaluate, watch } from './runner.mjs';
+import { formatReport } from './report.mjs';
+
+export const help = `OpenMerit · Agent evaluations (source preview)
+
+  openmerit eval init                 Add a starter suite and project agent instructions
+  openmerit eval doctor               Check installed Claude Code, Codex and OpenCode CLIs
+  openmerit eval plan [--scope existing|all]
+                                     Ask scope, then freeze Scout's shortlist per task
+  openmerit eval run [--json]         Evaluate the frozen shortlist in fresh workspaces
+  openmerit eval watch [--json]       Repeat after selected CLI/docs/fixture files change
+  openmerit eval report [--json]      Read the latest result without running models
+
+  --path <directory>                 Project directory (default: current directory)
+
+Configure .openmerit/eval.json: explicit agent connections, task fixtures, local
+graders, CLI variants, 3–10 repeats, and a cumulative budget. Init starts at $0.
+Scout receives task descriptions and, for existing scope, model IDs only.
+Run/watch execute local agents and may incur charges. No background daemon.
+`;
+
+const instructions = `<!-- openmerit-eval:start -->
+## OpenMerit agent evaluations
+
+When building or changing this project's CLI, tools or harness documentation, use
+\`openmerit eval\` to compare task performance. Skip this workflow when
+\`OPENMERIT_EVAL=1\` to avoid recursive evaluations.
+
+- Read .openmerit/eval.json and run \`openmerit eval doctor\`.
+- Ask once: "Models available through this project's existing connections, or all models Scout indexes?" Use \`eval plan --scope existing\` or \`--scope all\`. Reuse the saved choice. Never infer endpoint access from a model name or expose credentials.
+- Declare real connections with Scout model IDs, agent-native model IDs, and environment variable NAMES. Existing scope uses this project-declared inventory. All scope may require new authenticated connections before execution.
+- Replace starter cases with representative tasks, fresh fixtures and deterministic standalone Node graders. A grader reads {workspace, output} JSON on stdin and returns {quality: 0..1}. Keep answer keys outside agent workspaces. Include a bare variant and a CLI/docs variant where useful.
+- Confirm the suite, variants and cumulative budget with the user before enabling paid runs. Then plan once, run a baseline, make the requested CLI changes, and run again. Use \`eval watch\` only when the user wants continuous local reruns; it exits with the terminal and retains budget usage across restarts.
+- Report quality, failures, tokens, reported costs and elapsed time per task/model/variant. A shortlist is a prior, not measured success. Do not claim missing costs are zero or that 3–10 repeats prove an exact ordering. Keep fresh holdouts for final validation.
+- If a compatible watcher is already active, let it run; never start overlapping loops. Do not overwrite the frozen plan merely because the CLI changed.
+<!-- openmerit-eval:end -->`;
+
+export async function initialize(root) {
+  // Refuse partial overwrite of an existing suite or someone's fixtures.
+  const files = {
+    [configPath]: JSON.stringify(starter, null, 2) + '\n',
+    'eval/fixtures/json/request.txt': 'Customer: Mira Chen\nItems: 7\n',
+    'eval/tool.mjs': `import {readFileSync} from 'node:fs';\nconst s=readFileSync(process.argv[2],'utf8');\nconsole.log(JSON.stringify({name:s.match(/Customer: (.*)/)?.[1],count:Number(s.match(/Items: (\\d+)/)?.[1])}));\n`,
+    'eval/grade.mjs': `let input=''; for await (const chunk of process.stdin) input+=chunk;\nconst {output}=JSON.parse(input); let quality=0;\ntry {const v=JSON.parse(output); quality=(Number(v.name==='Mira Chen')+Number(v.count===7)+Number(Object.keys(v).sort().join(',')==='count,name'))/3;} catch {}\nconsole.log(JSON.stringify({quality}));\n`,
+  };
+  for (const file of Object.keys(files)) {
+    try { await readFile(join(root, file)); throw new Error(`Already exists: ${file}. Init does not overwrite a suite.`); } catch (error) { if (error.code !== 'ENOENT') throw error; }
+  }
+  for (const [file, body] of Object.entries(files)) { await mkdir(join(root, file, '..'), { recursive: true }); await writeFile(join(root, file), body, { flag: 'wx' }); }
+  for (const name of ['AGENTS.md', 'CLAUDE.md']) {
+    let body = ''; try { body = await readFile(join(root, name), 'utf8'); } catch (e) { if (e.code !== 'ENOENT') throw e; }
+    if (!body.includes('<!-- openmerit-eval:start -->')) await writeFile(join(root, name), `${body}${body ? '\n\n' : ''}${instructions}\n`);
+  }
+  let ignore = ''; try { ignore = await readFile(join(root, '.gitignore'), 'utf8'); } catch (e) { if (e.code !== 'ENOENT') throw e; }
+  if (!ignore.split('\n').includes('/.openmerit/eval/')) await writeFile(join(root, '.gitignore'), `${ignore}${ignore.endsWith('\n') || !ignore ? '' : '\n'}/.openmerit/eval/\n`);
+}
+
+export async function main(args) {
+  if (!args.length || args.includes('--help') || args.includes('-h')) { console.log(help); return; }
+  const action = args.shift(); let root = process.cwd(), scope, json = false;
+  for (let i = 0; i < args.length; i++) {
+    if (args[i] === '--path' && args[i + 1]) root = resolve(args[++i]);
+    else if (args[i] === '--scope' && ['existing', 'all'].includes(args[i + 1])) scope = args[++i];
+    else if (args[i] === '--json') json = true;
+    else throw new Error(`Unknown eval argument: ${args[i]}`);
+  }
+  if (scope && action !== 'plan') throw new Error('--scope belongs to eval plan.');
+  if (action === 'init') { await initialize(root); console.log('Created .openmerit/eval.json, starter fixtures, and AGENTS.md / CLAUDE.md instructions. Declare connections, replace the starter task, and set an approved budget before running.'); return; }
+  if (action === 'doctor') {
+    const results = [];
+    for (const agent of agents) { try { results.push({ agent, version: await preflight({ agent }), ready: true }); } catch (error) { results.push({ agent, ready: false, reason: error.message }); } }
+    console.log(JSON.stringify(results, null, 2)); return;
+  }
+  if (action === 'report') { const report = await readJSON(join(root, statePath, 'latest.json')); console.log(json ? JSON.stringify(report, null, 2) : formatReport(report)); return; }
+  if (!['plan', 'run', 'watch'].includes(action)) throw new Error('Unknown eval command. Run openmerit eval --help.');
+  const unlock = await acquireLock(root);
+  const controller = new AbortController(), stop = () => controller.abort();
+  process.once('SIGINT', stop); process.once('SIGTERM', stop);
+  try {
+    if (action === 'plan') {
+      const config = await loadConfig(root);
+      if (scope) config.scope = scope;
+      if (!config.scope && process.stdin.isTTY && !json) {
+        const rl = createInterface({ input: process.stdin, output: process.stdout });
+        try { const answer = await rl.question('Use existing project connections or all models Scout indexes? [existing/all] ', { signal: controller.signal }); if (['existing', 'all'].includes(answer.trim())) config.scope = answer.trim(); } finally { rl.close(); }
+      }
+      const plan = await makePlan(config);
+      await saveJSON(join(root, configPath), config); await savePlan(root, plan);
+      console.log(JSON.stringify(plan, null, 2));
+      return;
+    }
+    const onReport = report => { console.log(json ? JSON.stringify(report) : formatReport(report)); if (report.status !== 'complete') process.exitCode = 1; };
+    const onProgress = json ? () => {} : row => process.stderr.write(`${row.task} · ${row.connection} · ${row.variant} · ${row.round}: ${row.status}\n`);
+    if (action === 'run') onReport(await evaluate(root, { signal: controller.signal, onProgress }));
+    else { if (!json) console.log('Watching selected files. Ctrl+C stops; budget usage persists.'); await watch(root, { signal: controller.signal, onReport, onProgress }); }
+  } finally { process.removeListener('SIGINT', stop); process.removeListener('SIGTERM', stop); await unlock(); }
+}

package/dist/eval/src/config.mjs

@@ -0,0 +1,102 @@
+import { readFile, writeFile, mkdir, rename, readdir, lstat, realpath } from 'node:fs/promises';
+import { resolve, relative, join, dirname, isAbsolute } from 'node:path';
+import { createHash } from 'node:crypto';
+import { agents } from './adapters.mjs';
+
+export const configPath = '.openmerit/eval.json';
+export const statePath = '.openmerit/eval';
+export const digest = value => createHash('sha256').update(JSON.stringify(value)).digest('hex');
+export async function readJSON(path) { return JSON.parse(await readFile(path, 'utf8')); }
+export async function saveJSON(path, data) {
+  await mkdir(dirname(path), { recursive: true });
+  const temp = `${path}.${process.pid}.tmp`;
+  await writeFile(temp, JSON.stringify(data, null, 2) + '\n', { mode: 0o600 }); await rename(temp, path);
+}
+const text = v => typeof v === 'string' && v.trim().length > 0;
+const id = v => text(v) && /^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,79}$/.test(v);
+function requireThat(ok, message) { if (!ok) throw new Error(`Eval config: ${message}`); }
+export function safePath(path) {
+  requireThat(text(path) && !isAbsolute(path) && !path.split(/[\\/]/).some(p => ['..', '.', '.git', '.openmerit', 'node_modules'].includes(p) || p.startsWith('.env')), `unsafe file path ${path}. Use explicit project-relative source/fixture paths, without secrets or symlinks.`);
+  return path;
+}
+export function validateConfig(config) {
+  requireThat(config.version === 1, 'version must be 1.');
+  requireThat([null, 'existing', 'all'].includes(config.scope), 'scope must be null, existing or all.');
+  requireThat(Array.isArray(config.connections) && config.connections.length <= 100, 'connections must be a list of up to 100 routes.');
+  requireThat(new Set(config.connections.map(c => c.id)).size === config.connections.length, 'connection IDs must be unique.');
+  for (const c of config.connections) {
+    requireThat(id(c.id) && agents.includes(c.agent) && text(c.scoutId) && text(c.model), 'each connection needs id, scoutId, agent and native model.');
+    requireThat(!c.model.startsWith('-') && c.model.length <= 200, 'invalid native model.');
+    requireThat(c.env === undefined || Array.isArray(c.env) && c.env.every(k => /^[A-Z][A-Z0-9_]*$/.test(k)), 'env contains variable names, never credential values.');
+    requireThat(c.command === undefined || text(c.command), 'command must be an executable path.');
+  }
+  requireThat(Array.isArray(config.tasks) && config.tasks.length > 0 && config.tasks.length <= 30, 'provide 1–30 tasks.');
+  requireThat(new Set(config.tasks.map(t => t.id)).size === config.tasks.length, 'task IDs must be unique.');
+  for (const t of config.tasks) {
+    requireThat(id(t.id) && text(t.description) && t.description.length <= 4000 && text(t.prompt), 'each task needs id, description for Scout and a local execution prompt.');
+    safePath(t.fixture); safePath(t.grader);
+  }
+  requireThat(Array.isArray(config.variants) && config.variants.length > 0 && config.variants.length <= 5, 'provide 1–5 variants.');
+  requireThat(new Set(config.variants.map(v => v.id)).size === config.variants.length, 'variant IDs must be unique.');
+  for (const v of config.variants) {
+    requireThat(id(v.id) && Array.isArray(v.files) && typeof v.instructions === 'string', 'each variant needs id, files and instructions.');
+    v.files.forEach(safePath);
+  }
+  requireThat(Number.isInteger(config.repeats?.min) && Number.isInteger(config.repeats?.max) && config.repeats.min >= 3 && config.repeats.max <= 10 && config.repeats.min <= config.repeats.max, 'repeats must be between 3 and 10.');
+  requireThat(Number.isFinite(config.budget?.maxUsd) && config.budget.maxUsd >= 0 && Number.isFinite(config.budget.reservationUsd) && config.budget.reservationUsd > 0, 'set nonnegative maxUsd and positive reservationUsd.');
+  requireThat(Number.isInteger(config.timeoutSeconds) && config.timeoutSeconds >= 1 && config.timeoutSeconds <= 1800, 'timeoutSeconds must be 1–1800.');
+  if (config.scoutURL !== undefined) {
+    const url = new URL(config.scoutURL);
+    requireThat(!url.username && !url.password && (url.protocol === 'https:' || url.protocol === 'http:' && ['127.0.0.1', 'localhost', '[::1]'].includes(url.hostname)), 'scoutURL requires HTTPS, or HTTP on loopback for development.');
+  }
+  return config;
+}
+export async function loadConfig(root) { return validateConfig(await readJSON(join(root, configPath))); }
+
+// Snapshot exact bytes before executing; edits during a run belong to the next revision.
+export async function filesAt(root, paths) {
+  const files = Object.create(null); let bytes = 0; const canonicalRoot = await realpath(root);
+  async function visit(path) {
+    safePath(path);
+    const full = resolve(root, path), info = await lstat(full);
+    if (info.isSymbolicLink() || relative(canonicalRoot, await realpath(full)).startsWith('..')) throw new Error(`Eval refuses symlink/outside path: ${path}`);
+    if (info.isDirectory()) {
+      for (const name of (await readdir(full)).sort()) await visit(join(path, name));
+    } else if (info.isFile()) {
+      bytes += info.size;
+      if (bytes > 25_000_000) throw new Error('Eval fixture/source snapshot exceeds 25 MB. Select smaller paths.');
+      files[path] = { data: (await readFile(full)).toString('base64'), executable: !!(info.mode & 0o111) };
+    } else throw new Error(`Eval requires regular files: ${path}`);
+  }
+  for (const path of paths) await visit(path);
+  return files;
+}
+export async function snapshot(root, config) {
+  const tasks = [];
+  for (const task of config.tasks) {
+    if (!(await lstat(join(root, task.fixture))).isDirectory() || !(await lstat(join(root, task.grader))).isFile()) throw new Error(`${task.id}: fixture must be a directory and grader a standalone file.`);
+    tasks.push({ ...task, fixtureFiles: await filesAt(root, [task.fixture]), graderFiles: await filesAt(root, [task.grader]) });
+  }
+  const variants = [];
+  for (const variant of config.variants) variants.push({ ...variant, sourceFiles: await filesAt(root, variant.files) });
+  for (const task of tasks) for (const variant of variants) {
+    const fixtureNames = new Set(Object.keys(task.fixtureFiles).map(path => relative(task.fixture, path)));
+    if (Object.keys(variant.sourceFiles).some(path => fixtureNames.has(path))) throw new Error(`${variant.id} overwrites a fixture file for ${task.id}. Keep CLI files and task inputs separate.`);
+  }
+  const compatibility = digest({ tasks, connections: config.connections, repeats: config.repeats, variantIds: variants.map(v => v.id), timeoutSeconds: config.timeoutSeconds });
+  return { tasks, variants, compatibility, revision: digest({ tasks, variants, config }) };
+}
+export async function materialize(files, dest, strip = '') {
+  for (const [path, file] of Object.entries(files)) {
+    const target = join(dest, strip ? relative(strip, path) : path);
+    await mkdir(dirname(target), { recursive: true });
+    await writeFile(target, Buffer.from(file.data, 'base64'), { mode: file.executable ? 0o700 : 0o600 });
+  }
+}
+
+export const starter = {
+  version: 1, scope: null, connections: [],
+  tasks: [{ id: 'json', description: 'Extract a customer name and item count from text into a strict JSON object.', prompt: 'Read request.txt. Return only JSON with keys name (string) and count (number).', fixture: 'eval/fixtures/json', grader: 'eval/grade.mjs' }],
+  variants: [{ id: 'bare', files: [], instructions: '' }, { id: 'cli', files: ['eval/tool.mjs'], instructions: 'You can use node eval/tool.mjs request.txt to extract the fields.' }],
+  repeats: { min: 3, max: 10 }, budget: { maxUsd: 0, reservationUsd: 0.25 }, timeoutSeconds: 180,
+};

package/dist/eval/src/process.mjs

@@ -0,0 +1,44 @@
+import { spawn } from 'node:child_process';
+
+// No shell interpolation. Bound both time and output, and stop descendants too.
+export function execute(command, args, { cwd, env = process.env, input = '', timeoutMs = 180_000, signal } = {}) {
+  return new Promise(resolve => {
+    const start = performance.now();
+    let stdout = '', stderr = '', bytes = 0, reason = null, escalation;
+    const child = spawn(command, args, { cwd, env, detached: process.platform !== 'win32', stdio: ['pipe', 'pipe', 'pipe'] });
+    const kill = sig => {
+      try {
+        if (process.platform === 'win32') { const killer = spawn('taskkill', ['/pid', String(child.pid), '/T', '/F'], { stdio: 'ignore' }); killer.on('error', () => child.kill(sig)); }
+        else process.kill(-child.pid, sig);
+      } catch {}
+    };
+    const stop = why => { if (reason) return; reason = why; kill('SIGTERM'); escalation = setTimeout(() => kill('SIGKILL'), 500); };
+    const abort = () => stop('cancelled');
+    signal?.addEventListener('abort', abort, { once: true });
+    if (signal?.aborted) abort();
+    const timer = setTimeout(() => stop('timeout'), timeoutMs);
+    const collect = (chunk, which) => {
+      bytes += chunk.length;
+      if (bytes > 4_000_000) return stop('output_limit');
+      if (which === 'out') stdout += chunk; else stderr += chunk;
+    };
+    child.stdout.on('data', chunk => collect(chunk, 'out'));
+    child.stderr.on('data', chunk => collect(chunk, 'err'));
+    child.on('error', () => { reason = 'spawn_failed'; });
+    child.stdin.on('error', () => {});
+    child.stdin.end(input);
+    child.on('close', code => {
+      clearTimeout(timer); clearTimeout(escalation); signal?.removeEventListener('abort', abort);
+      resolve({ code, reason, stdout, stderr, seconds: (performance.now() - start) / 1000 });
+    });
+  });
+}
+
+export function agentEnvironment(names = []) {
+  const result = {};
+  for (const key of ['PATH', 'HOME', 'USERPROFILE', 'SystemRoot', 'TMPDIR', 'TMP', 'TEMP', 'LANG', 'TERM', ...names]) {
+    if (process.env[key] !== undefined) result[key] = process.env[key];
+  }
+  result.OPENMERIT_EVAL = '1';
+  return result;
+}

package/dist/eval/src/report.mjs

@@ -0,0 +1,51 @@
+const mean = values => values.length ? values.reduce((a, b) => a + b, 0) / values.length : null;
+const median = values => { const sorted = [...values].sort((a, b) => a - b); return sorted.length ? (sorted[Math.floor((sorted.length - 1) / 2)] + sorted[Math.floor(sorted.length / 2)]) / 2 : null; };
+const completeMean = values => values.every(v => v !== null) ? mean(values) : null;
+export function summarize(rows) {
+  const groups = new Map();
+  for (const row of rows) {
+    const key = [row.task, row.connection, row.variant].join('/');
+    if (!groups.has(key)) groups.set(key, []); groups.get(key).push(row);
+  }
+  return [...groups].map(([key, runs]) => ({
+    key, task: runs[0].task, connection: runs[0].connection, variant: runs[0].variant, runs: runs.length,
+    quality: completeMean(runs.map(r => r.quality)), failures: runs.filter(r => r.status !== 'ok').length,
+    qualityRange: runs.every(r => r.quality !== null) ? [Math.min(...runs.map(r => r.quality)), Math.max(...runs.map(r => r.quality))] : null,
+    inputTokens: completeMean(runs.map(r => r.usage?.input ?? null)), outputTokens: completeMean(runs.map(r => r.usage?.output ?? null)),
+    reportedCostUsd: runs.filter(r => r.costUsd !== null).reduce((s, r) => s + r.costUsd, 0), unknownCosts: runs.filter(r => r.costUsd === null).length,
+    meanCostUsd: completeMean(runs.map(r => r.costUsd)), medianSeconds: median(runs.map(r => r.seconds)),
+  }));
+}
+export function isVolatile(rows) {
+  const spread = values => {
+    if (values.length < 3 || values.some(v => v === null)) return false;
+    const avg = mean(values);
+    return avg > 0 && Math.sqrt(mean(values.map(v => (v - avg) ** 2))) / avg > .25;
+  };
+  for (const group of summarize(rows)) {
+    const runs = rows.filter(r => `${r.task}/${r.connection}/${r.variant}` === group.key);
+    if (group.qualityRange && group.qualityRange[1] - group.qualityRange[0] > .05 || runs.some(r => r.status !== 'ok') || spread(runs.map(r => r.seconds)) || spread(runs.map(r => r.costUsd)) || spread(runs.map(r => r.usage?.input ?? null))) return true;
+  }
+  return false;
+}
+export function compareReports(current, previous) {
+  if (!previous || previous.compatibility !== current.compatibility || previous.planHash !== current.planHash || previous.status !== 'complete' || current.status !== 'complete') return null;
+  return current.summary.map(now => {
+    const before = previous.summary.find(row => row.key === now.key);
+    if (!before) return null;
+    const delta = key => now[key] === null || before[key] === null ? null : now[key] - before[key];
+    return { key: now.key, previousRuns: before.runs, runs: now.runs, qualityDelta: delta('quality'), inputTokensDelta: delta('inputTokens'), outputTokensDelta: delta('outputTokens'), meanCostUsdDelta: delta('meanCostUsd'), medianSecondsDelta: delta('medianSeconds') };
+  }).filter(Boolean);
+}
+const fmt = (v, digits = 2) => v === null || v === undefined ? '?' : v.toFixed(digits);
+export function formatReport(report) {
+  const lines = [`OpenMerit eval · ${report.status} · ${report.revision.slice(0, 12)}`, '', 'Task / connection / variant | runs | quality | input / output tokens | reported $/run | seconds (median)'];
+  for (const row of report.summary) lines.push(`${row.key} | ${row.runs} | ${fmt(row.quality === null ? null : row.quality * 100, 1)}% | ${fmt(row.inputTokens, 0)} / ${fmt(row.outputTokens, 0)} | ${fmt(row.meanCostUsd, 5)} | ${fmt(row.medianSeconds)}`);
+  if (report.comparison) {
+    lines.push('', 'Change from the previous compatible revision (negative tokens/cost/time is lower):');
+    for (const row of report.comparison) lines.push(`${row.key}: quality ${fmt(row.qualityDelta === null ? null : row.qualityDelta * 100, 1)} pp; tokens ${fmt(row.inputTokensDelta, 0)} / ${fmt(row.outputTokensDelta, 0)}; $ ${fmt(row.meanCostUsdDelta, 5)}; seconds ${fmt(row.medianSecondsDelta)}`);
+  } else lines.push('', 'No complete, compatible previous revision for comparison.');
+  lines.push('', '3–10 repetitions measure variability, not statistical certainty. ? means unavailable.', 'Costs are agent-reported estimates, not reconciled bills. Compare agents as different harnesses.');
+  if (report.reason) lines.push(`Stopped: ${report.reason}`);
+  return lines.join('\n');
+}