openmerit 0.2.0 → 0.3.0

package/docs/roadmap.md

@@ -2,7 +2,7 @@
 
 ## Current status
 
-The Inference Terminal provides a read-only view of existing application evidence. The Pi **1.5.0** adapter in OpenMerit **0.1.12** provides only native Scout MCP and shared guidance, not evaluation coordination. The standalone evaluation CLI remains separate. Neither a populated dashboard nor passing implementation tests establishes that an application has better models, trustworthy evaluations, or reliable unattended automation.
+The Pi **1.5.0** adapter in OpenMerit **0.1.12** provides only native Scout MCP and shared guidance, not evaluation coordination. OpenMerit **0.3.0** retires the Inference Terminal and the standalone evaluation CLI; unreleased Scout source retires the legacy recommendation API. Passing implementation tests do not establish that an application has better models, trustworthy evaluations, or reliable unattended automation.
 
 ## Known limitations
 
@@ -13,4 +13,4 @@ The Inference Terminal provides a read-only view of existing application evidenc
 - The Pi 0.87 evaluation adapter is removed in source. Pi 1.5.0 setup needs explicit browser approval, a new session and an explicit installer rerun for updates. It has no lifecycle commands, scheduler, LaunchAgent or auto-update hooks. The host owns tools, model, permissions and sessions; SDK hosts must load Pi's native MCP extension explicitly.
 - Scout rule/skill availability does not guarantee invocation or measured application quality. Missing project policy requires asking; unavailable or insufficient Scout evidence requires pausing the model choice.
 - Existing legacy Pi 0.87 background jobs require separate cleanup during migration.
-- Agent-evaluation budget admission depends on reported costs. It cannot independently meter provider billing or stop an in-flight request. Use provider spending controls as well.
+- Scout guidance is not execution or evaluation. Use representative application checks and provider spending controls independently.

package/docs/scout.md

@@ -1,8 +1,12 @@
 # Scout by OpenMerit
 
-Scout is OpenMerit's model search agent. Describe your task and get model recommendations. Scout returns up to five model-and-endpoint candidates per task. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page and JSON API use the same index and ranking.
+Scout is OpenMerit's model search agent. Describe your task and get provisional model guidance through the conversational page or MCP. Public catalogue evidence is not proof of application quality.
 
-Scout is a separate hosted beta product. Npm 0.1.7 introduced optional Claude Code setup for it. OpenMerit 0.1.12 adds native Scout access for Pi 1.5.0, replacing the old Pi evaluation adapter without changing the standalone evaluation CLI or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
+**Unreleased retirement, November 2026:** this source removes `/scout/api/recommend`, its alpha/versioned aliases and the complete standalone `openmerit eval` CLI, including `plan`. Retired paths return 404 before authentication, redirects or asset fallback. The agent, canonical `/scout/mcp`, coding-agent setup, catalogue publisher, status and account/privacy capabilities remain. No remote data or existing local evaluation files are deleted. Package publication and Worker deployment require separate approval; neither is implied by this source change.
+
+Dated receipts and the old search/ranking/JSON protocol descriptions below are historical, not supported interfaces in this source. There is no replacement frozen-plan API. See [evaluation retirement](agent-evaluations.md) and [commands](commands.md) for migration boundaries.
+
+Scout is a separate hosted beta product. Npm 0.1.7 introduced optional Claude Code setup for it. OpenMerit 0.1.12 added native Scout access for Pi 1.5.0, replacing the old Pi evaluation adapter. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
 
 **Website rebuild — published 24 October 2026:** the rebuild preserves the existing landing page at `/` and Scout at `/scout/`, and takes all hosted `/docs/` pages offline, including this guide. `/connect/`, the JSON API/MCP and the standalone `/scout/data-policy/` remain available. The guide remains maintained in Markdown; this does not remove the local CLI or its documentation from the package. See the [cutover notes](site-domain-migration.md) for publication and verification details.
 

package/docs/security.md

@@ -2,19 +2,19 @@
 
 **Pi adapter — npm 0.1.12 and later:** Pi 1.5.0 uses native MCP plus shared Scout rule/skill sections, not the former evaluation adapter. It never initiates login, changes the host model/tools/session, or provisions background jobs. Browser-approved credentials are independent per client/scope/project, stored privately outside the repository, and passed as MCP HTTP headers, never instructions. Pi's installed connection JSON contains only the private credential path. See [Pi paths and targeted removal](pi-extension.md). The standalone evaluation CLI remains separate.
 
-The [preview agent-evaluation runner](agent-evaluations.md) is a separate execution path: it runs trusted local agents/graders, passes only basic OS variables and explicitly named environment variables, and preserves agent-owned authentication. It sends task descriptions and optional model IDs to Scout. Local metric reports and budget reservations persist; temporary task outputs are removed. Fresh directories are not security sandboxes. Its budget admission controls do not enforce provider billing limits.
+**Retirement:** npm 0.3.0 removes the standalone [agent-evaluation runner](agent-evaluations.md), including planning; unreleased Scout source removes `/scout/api/recommend`. Existing local reports, budgets, configuration and host-agent instructions are not deleted; review them separately. No provider billing control or remote data deletion is implied.
 
-The [budget selector](agent-evaluations.md#budget-selection) in `0.1.7-preview.2` requires an explicit $10, custom or no-fixed-cap choice after model scope. No fixed cap removes monetary admission limits, including Claude's per-attempt dollar flag, while retaining tasks, repetition bounds, timeouts, native permissions and accounting. Saved choices apply across reruns; a foreground watcher can incur further charges on each change until stopped. Provider limits remain independent.
+The Scout agent, MCP, publisher and account analytics/deletion capabilities remain. Historical analytics and stored catalogue/research data are not erased by endpoint retirement. Necessary provider processing for the agent and publisher continues under the [Scout data policy](scout-data-policy.md).
 
 ## Credential ownership
 
 OpenMerit accepts no model-provider or Jev credential. Credentials belong to the coding harness, its environment, or an external secret store.
 
-Never place secrets in evaluation configuration, task fixtures, Scout project policy, terminal configuration, activity exports, traces, or committed environment files.
+Never place secrets in evaluation configuration, task fixtures, Scout project policy, traces, or committed environment files.
 
 ## Local state
 
-`.openmerit/` may hold evaluation configuration, task fixtures, reports and budget usage, Scout project policy, and terminal configuration or activity exports. It may reveal task names, model choices, budgets or endpoint identifiers. Treat it as sensitive project metadata even though it must not contain credentials.
+`.openmerit/` may hold evaluation configuration, task fixtures, reports and budget usage, and Scout project policy, plus files left by the retired Inference Terminal (`terminal.json`, `inference.jsonl`), which upgrades do not delete. It may reveal task names, model choices, budgets or endpoint identifiers. Treat it as sensitive project metadata even though it must not contain credentials.
 
 With native Scout, Pi manages its own session/tool records and provider context; the adapter adds no separate evaluation-history store. Task descriptions sent to hosted Scout follow its data policy. Local OpenMerit storage does not imply that all processing stays on your machine.
 
@@ -24,19 +24,6 @@ OpenMerit does not establish the truth of imported or external evidence. Use aut
 
 Do not include credentials, private traces, or proprietary evaluation data in public issues.
 
-## Inference Terminal
-
-The terminal reads supported project metadata, request logs, explicit exports, and authorized loopback runtime inventory/metrics endpoints. It calls no inference endpoint and exposes no mutation API. A local syntax index reads supported Python, JavaScript/TypeScript, JSON, YAML, TOML, and project-local dotenv declarations. It does not execute or import project code, source shell files, or resolve arbitrary expressions. Known credential-variable presence can identify configuration; credential values are discarded and never used for provider calls. Endpoint addresses are sanitized before retention. Static URLs found by the syntax index never authorize network probes.
-
-Loopback runtime probes require supported project endpoint declarations or explicit terminal configuration. No standard ports, remote provider accounts, credential stores, running processes, or general home directories are scanned. Whole-machine readings are context, not per-request measurements; the compute view requires explicit links or supplied host evidence.
-
-Coding-harness histories are outside the terminal's application scope and are not scanned or imported. Application request logs and explicit exports retain only allowlisted metadata, without prompts or responses. Sources must belong to the selected application; the terminal cannot infer the purpose of an unlabeled API response. Records explicitly marked `scope: "development"` are rejected.
-
-The local web service binds to `127.0.0.1` and rejects unexpected Host, Origin, and cross-site browser requests. It only serves bundled assets and specific read-only JSON routes. It is not authenticated against other local processes or machine users. Model names, endpoint identifiers, and relative source paths can still be sensitive. Names and identifiers in imported metadata must be kept free of secrets.
-
-Repository metadata and usage caches live in process memory and disappear when the terminal exits. The terminal creates no persistent index or files in the application. Per-file/parser protections, request-log read limits, runtime inventory limits, and explicit coverage gaps still apply. See [terminal data handling and limits](inference-terminal.md) for supported sources, precedence, retention bounds, diagnostics, and sample-data separation.
-
-
 ## Hosted Scout authentication
 
 Hosted Scout uses Clerk Google/GitHub sign-in. The page and status are public; JSON recommendations require a valid Clerk session bearer token, not just a cookie. MCP methods require a device key from the setup below. Missing/invalid credentials return HTTP 401, and missing authentication configuration fails closed with HTTP 503. Clerk remains the identity system; the analytics integration below can synchronize selected profile fields to PostHog. Raw account IDs are used transiently for the per-user edge limiter and are not joined to historical Cloudflare prompt records. A prompt draft is temporarily stored in the current browser tab during sign-in. No Clerk Billing or automatic agent OAuth is included. See [Scout account sign-in](scout.md#account-sign-in-published-21-october-2026) for token lifetime, rate limits, data handling and the remaining real-login validation limits.
@@ -79,7 +66,7 @@ Historical records are accessible only through authenticated Cloudflare account
 
 ### Optional product analytics
 
-Scout has a visible privacy disclosure and controls. Public homepage/docs navigation events remain source-only; npm 0.1.7 includes opt-in local terminal/map events and aggregate counts. The terminal relay cannot mutate project or inference state, retains loopback Host/Origin guards, and forwards only bounded allowlisted telemetry. Terminal and CLI use a process-only random ID; website events use a separate tab ID. Neither is implicitly linked to Clerk. Application names, raw inputs/outputs, local identifiers and source files remain local. Sanitized error classes/categories (and Scout application file/line frames) exclude raw messages, credentials, source context, arbitrary URLs and filesystem paths. No replay is enabled. All expanded account-linked events pass the same preference/deletion gate as profile synchronization; ordinary event delivery remains best effort. New search details link to identified accounts when profile sync is active, unlike the legacy Cloudflare archive. No external retention setting is changed.
+Scout has a visible privacy disclosure and controls. Public homepage/docs navigation events remain source-only; npm 0.1.7 includes opt-in local terminal/map events and aggregate counts, which npm 0.3.0 removes with the [Inference Terminal](inference-terminal.md). The terminal relay cannot mutate project or inference state, retains loopback Host/Origin guards, and forwards only bounded allowlisted telemetry. Terminal and CLI use a process-only random ID; website events use a separate tab ID. Neither is implicitly linked to Clerk. Application names, raw inputs/outputs, local identifiers and source files remain local. Sanitized error classes/categories (and Scout application file/line frames) exclude raw messages, credentials, source context, arbitrary URLs and filesystem paths. No replay is enabled. All expanded account-linked events pass the same preference/deletion gate as profile synchronization; ordinary event delivery remains best effort. New search details link to identified accounts when profile sync is active, unlike the legacy Cloudflare archive. No external retention setting is changed.
 
 The [PostHog integration](scout.md#product-analytics) uses the EU ingestion endpoint; unconfigured deployments remain off. Verified Clerk user IDs become namespaced SHA-256 pseudonyms for cross-client account retention. Profile sync can associate those identities with names and verified email addresses, so they are identifiable personal data. No identity is added to the historical Cloudflare prompt archive. The browser retains an anonymous ID and hashed account/session transition markers in tab storage, rotating the anonymous ID on account changes. There is no autocapture, replay or visitor-IP forwarding. PostHog has its own retention/access policies, not Cloudflare's three-month policy.
 

package/docs/testing.md

@@ -1,5 +1,7 @@
 # Testing and validation
 
+**Cleanup:** npm 0.3.0 retires the standalone evaluation CLI and the Inference Terminal; unreleased source also retires the legacy JSON recommendation pipeline and unpublished docs website shell. Dated evaluation/API acceptance records below describe historical releases, not current supported commands. Current verification retains agent, MCP, publisher, status, policy and protected-memo coverage; it does not publish or deploy anything.
+
 **Pi adapter — npm 0.1.12 and later:** the Pi 1.5.0 adapter uses native Scout MCP and shared guidance only. Dated acceptance records below describe the packages tested at the time; some mention package exports that have since been removed. Standalone evaluation tests remain separate.
 
 ## Stable 0.1.7 acceptance — 24 October 2026
@@ -58,25 +60,11 @@ npm run verify
 npm pack --dry-run
 ```
 
-`verify` runs workspace tests, syntax checks, strict TypeScript checks and the TypeScript migration ratchet, the package build, demo tests, generated-documentation checks, and whitespace validation. It does not publish a package or deploy the website.
-
-## Inference Terminal coverage
-
-```sh
-npm test --workspace=@openmerit/terminal
-```
-
-Tests cover metadata projection and redaction, source containment, recursive syntax indexing, phased worker progress, incremental reuse, coverage limits, endpoint classification, authorized loopback probes, supported request-log formats, UTF-8 boundaries, rotation/truncation, source reconciliation, historical ranges, unknown versus zero measurements, and the read-only API.
-
-Inventory and UI coverage includes model/service identity, explicit deployment and host links, shared capacity, stale/missing hardware, path classification, map completeness, search and filtering, and preservation of unknown costs. Runtime fixtures establish which endpoints collectors call; they do not prove compatibility with every external runtime version.
-
-The local fictional sample (`--demo` or the sample selector) has 40 agents, 140 tasks, 21 models and 9,600 requests over 24 hours. Sample tests exercise interface scale and arithmetic, not measured throughput, application quality, or hardware performance.
-
-Browser review covers the six views, inspectors, search, activity ranges, map and chart interaction, keyboard navigation, reduced motion, and desktop/narrow layouts. Browser checks are distinct from the committed unit suite.
+`verify` runs workspace tests, syntax checks, strict TypeScript checks and the TypeScript migration ratchet, the package build, memo tests, policy/memo build and Markdown-guide checks, Scout tests/build, and whitespace validation. It does not publish a package or deploy the website. `npm run check:worker-types` separately checks generated Worker types; CI also performs packed-install, CLI help and Pi load smoke checks.
 
 ## Pi load and package smoke tests
 
-CI packs the package, installs it in a temporary directory, imports `openmerit/pi`, smoke-tests the installed terminal command, and loads the installed extension in Pi with `--list-models`. These checks establish packaging and loading, not successful provider authentication or Scout recommendations.
+CI packs the package, installs it in a temporary directory, imports `openmerit/pi`, checks the installed `openmerit --help`, and loads the installed extension in Pi with `--list-models`. These checks establish packaging and loading, not successful provider authentication or Scout recommendations.
 
 ## What evidence establishes
 

package/docs/troubleshooting.md

@@ -39,9 +39,9 @@ They were removed, not renamed. The Pi adapter now supplies only native Scout MC
 
 The rule and skill are model guidance, not a guaranteed tool gate. Confirm they loaded in a new session and that the host permits the MCP tools. When project model policy is missing, ask the user; when Scout is unavailable or lacks suitable evidence, pause the model choice rather than invent a result. Host model/tool/session restrictions remain authoritative. Scout candidates are not measurements of your application's quality.
 
-## Evaluation or terminal problems
+## `openmerit dash` or `openmerit eval` is an unknown command
 
-The standalone products are separate from Pi setup. Use [agent-evaluation diagnostics](agent-evaluations.md) for tasks, graders, budgets and execution blocks, or [terminal diagnostics](inference-terminal.md) for missing inventory and request evidence. A populated dashboard is not proof of successful evaluations. Never mark missing or rejected evidence successful merely to advance a workflow.
+Both are retired in npm 0.3.0; see the [Inference Terminal](inference-terminal.md) and [evaluation CLI](agent-evaluations.md) retirement notes. Existing local files are not deleted. Never mark missing or rejected evidence successful merely to advance a workflow.
 
 ## Report a reproducible problem
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {
@@ -16,7 +16,7 @@
   "files": [
     "dist/pi/",
     "dist/terminal/",
-    "dist/eval/",
+    "!dist/terminal/src/*.ts",
     "docs/",
     "CHANGELOG.md",
     "LICENSE",
@@ -55,6 +55,7 @@
   "scripts": {
     "check": "npm run check --workspaces --if-present",
     "test": "npm run test --workspaces --if-present",
+    "pretypecheck": "npm run build:site",
     "typecheck": "tsc -p tsconfig.json && tsc -p tsconfig.scout.json && tsc -p scout/agent/tsconfig.json",
     "check:typescript": "node scripts/check-typescript.ts",
     "types:workers": "node scripts/worker-types.ts",
@@ -62,26 +63,17 @@
     "deploy": "node scripts/deploy.ts",
     "deploy:status": "node scripts/deploy.ts status",
     "build": "node scripts/build.mjs",
-    "predash": "node scripts/build-terminal.mjs",
-    "dash": "node packages/terminal/src/cli.mjs dash",
-    "build:docs": "node scripts/build-docs.mjs",
-    "check:docs": "npm run build:docs && node --test scripts/docs.test.mjs",
+    "build:site": "node scripts/build-site.ts",
+    "check:docs": "npm run build:site && node --test scripts/docs.test.ts",
     "memo": "node demo/memo/preview.ts",
-    "check:demo": "npm ci --ignore-scripts --prefix demo && npm run build:docs && tsc -p tsconfig.memo.json && node --test demo/test/*.test.mjs demo/test/*.test.ts",
+    "check:demo": "npm ci --ignore-scripts --prefix demo && npm run build:site && tsc -p tsconfig.memo.json && node --test demo/test/*.test.mjs demo/test/*.test.ts",
     "prepare": "npm run build",
     "verify": "npm test && npm run check && npm run typecheck && npm run check:typescript && npm run build && npm run check:demo && npm run check:scout && npm run check:docs && npm run build:scout && git diff --check",
     "prepublishOnly": "npm run verify",
-    "scout": "node scout/dev.mjs",
     "scout:refresh": "node scout/refresh.mjs",
     "check:scout": "node --test scout/test/*.test.mjs scout/test/*.test.ts && node --check scout/worker.mjs",
     "build:scout": "node scripts/build-scout.mjs"
   },
-  "dependencies": {
-    "@iarna/toml": "2.2.5",
-    "@lezer/javascript": "1.5.6",
-    "@lezer/python": "1.1.19",
-    "yaml": "2.9.1"
-  },
   "peerDependencies": {
     "@earendil-works/pi-coding-agent": "1.5.0"
   },
@@ -91,26 +83,21 @@
     }
   },
   "devDependencies": {
-    "@earendil-works/pi-coding-agent": "1.5.0",
     "@clerk/backend": "3.26.1",
     "@clerk/shared": "4.41.0",
+    "@earendil-works/pi-coding-agent": "1.5.0",
+    "@lezer/javascript": "1.5.6",
     "@resvg/resvg-js": "2.6.2",
-    "@types/markdown-it": "14.1.2",
     "@types/node": "^22.18.6",
     "esbuild": "0.28.2",
     "fflate": "0.8.2",
     "hyparquet": "1.31.5",
-    "liveline": "0.0.7",
     "markdown-it": "15.0.2",
     "miniflare": "5.20260921.0-alpha",
-    "minisearch": "7.2.0",
-    "morphdom": "2.7.8",
-    "react": "19.3.0",
-    "react-dom": "19.3.0",
-    "torph": "0.1.3",
     "typebox": "1.3.27",
     "typescript": "^5.9.3",
     "web-vitals": "6.2.3",
-    "wrangler": "4.136.3"
+    "wrangler": "4.136.3",
+    "yaml": "2.9.1"
   }
 }

package/dist/eval/src/adapters.mjs

@@ -1,112 +0,0 @@
-import { execute, agentEnvironment } from './process.mjs';
-
-const number = value => typeof value === 'number' && Number.isFinite(value) && value >= 0 ? value : null;
-const sum = values => values.length && values.every(v => v !== null) ? values.reduce((a, b) => a + b, 0) : null;
-const maximum = values => values.some(v => v !== null) ? Math.max(...values.filter(v => v !== null)) : null;
-export const agents = ['claude', 'codex', 'opencode'];
-
-export function connectionFailure(text) {
-  if (/budget_exhausted|error_max_budget_usd|(?:maximum|exceeded|exhausted|insufficient).{0,25}(?:budget|credits)|(?:budget|credits).{0,25}(?:exceeded|exhausted)/i.test(text)) return 'budget_exhausted';
-  if (/(?:bwrap|bubblewrap|sandbox|landlock|seccomp|unshare|user namespace|uid_map).{0,160}(?:failed|not permitted|unavailable|denied)|failed to (?:start|initialize|create).{0,40}sandbox/i.test(text)) return 'sandbox_unavailable';
-  if (/\b(?:HTTP\s*)?401\b.{0,30}(?:unauthorized|authentication)|unauthorized|authentication[_ -]?(?:error|required)|invalid api key|not logged in|please log in/i.test(text)) return 'authentication';
-  if (/model.{0,100}(?:not supported|not found|not available|does not exist)|(?:unknown|invalid|unsupported) model|ProviderModelNotFoundError/i.test(text)) return 'model_unavailable';
-  if (/rejected permission|permission.{0,30}(?:denied|rejected)|not allowed to use.{0,30}tool/i.test(text)) return 'permission_denied';
-  return null;
-}
-
-function toolFailure(text) {
-  const reason = connectionFailure(text);
-  // A task's own HTTP/auth/budget errors are task outcomes, not evidence that
-  // the coding agent's provider connection failed.
-  return ['sandbox_unavailable', 'permission_denied'].includes(reason) ? reason : null;
-}
-
-export function invocation(connection, reservationUsd, unlimitedBudget = false) {
-  const model = connection.model;
-  if (connection.agent === 'claude') return ['-p', '--output-format', 'json', '--model', model, '--no-session-persistence', '--permission-mode', 'dontAsk', '--allowedTools', 'Read,Glob,Grep,Bash,Edit,Write', '--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}', '--setting-sources', '', ...(unlimitedBudget ? [] : ['--max-budget-usd', String(reservationUsd)])];
-  if (connection.agent === 'codex') return ['exec', '--json', '--model', model, '--sandbox', 'workspace-write', '--skip-git-repo-check', '--ephemeral', '-'];
-  if (connection.agent === 'opencode') return ['run', '--format', 'json', '--model', model, '--auto'];
-  throw new Error('Unknown agent. Use claude, codex or opencode.');
-}
-
-export function parseOutput(agent, stdout) {
-  let events;
-  try { const parsed = JSON.parse(stdout); events = Array.isArray(parsed) ? parsed : [parsed]; }
-  catch { events = stdout.split('\n').flatMap(line => { try { return [JSON.parse(line)]; } catch { return []; } }); }
-  let output = '', usage = null, costUsd = null, partialCostUsd = null, completed = false, failed = false, failureReason = null;
-  // Only native failure fields are evidence of a blocked execution. Task text,
-  // successful tool output and model commentary can legitimately discuss errors.
-  const errors = events.filter(e => ['error', 'turn.failed'].includes(e.type));
-  for (const event of errors) failureReason ??= connectionFailure(JSON.stringify(event));
-  const observedModels = new Set();
-  if (agent === 'claude') {
-    const result = events.findLast(e => e.type === 'result');
-    if (result) {
-      completed = result.subtype === 'success' && !result.is_error;
-      failed = !completed; output = result.result ?? '';
-      const u = result.usage;
-      if (u) usage = { input: sum([number(u.input_tokens), number(u.cache_read_input_tokens ?? 0), number(u.cache_creation_input_tokens ?? 0)]), output: number(u.output_tokens), cached: number(u.cache_read_input_tokens ?? 0) };
-      const models = Object.values(result.modelUsage ?? {});
-      if (models.length) {
-        const modelUsage = { input: sum(models.map(m => sum([number(m.inputTokens), number(m.cacheReadInputTokens ?? 0), number(m.cacheCreationInputTokens ?? 0)]))), output: sum(models.map(m => number(m.outputTokens))), cached: sum(models.map(m => number(m.cacheReadInputTokens ?? 0))) };
-        // A budget stop can emit zero aggregate usage but nonzero per-model usage.
-        usage = Object.fromEntries(Object.keys(modelUsage).map(k => [k, maximum([usage?.[k] ?? null, modelUsage[k]])]));
-      }
-      costUsd = maximum([number(result.total_cost_usd), sum(models.map(m => number(m.costUSD)))]);
-      for (const model of Object.keys(result.modelUsage ?? {})) observedModels.add(model);
-      if (failed) failureReason ??= connectionFailure(JSON.stringify({ subtype: result.subtype, terminal_reason: result.terminal_reason, errors: result.errors }));
-      if (result.permission_denials?.length) failureReason ??= 'permission_denied';
-    }
-  } else if (agent === 'codex') {
-    const turns = events.filter(e => e.type === 'turn.completed');
-    completed = turns.length > 0;
-    failed = events.some(e => ['turn.failed', 'error'].includes(e.type));
-    output = events.findLast(e => e.type === 'item.completed' && e.item?.type === 'agent_message')?.item.text ?? '';
-    if (turns.length) usage = { input: sum(turns.map(e => number(e.usage?.input_tokens))), output: sum(turns.map(e => number(e.usage?.output_tokens))), cached: sum(turns.map(e => number(e.usage?.cached_input_tokens))) };
-    for (const event of events.filter(e => e.type === 'item.completed' && e.item?.type === 'command_execution' && (e.item.exit_code > 0 || e.item.status === 'failed'))) {
-      failureReason ??= toolFailure(event.item.aggregated_output ?? '');
-    }
-  } else if (agent === 'opencode') {
-    const steps = events.filter(e => e.type === 'step_finish');
-    const finalStep = steps.findLast(e => ['stop', 'end_turn'].includes(e.part?.reason));
-    completed = !!finalStep;
-    failed = events.some(e => e.type === 'error');
-    const texts = events.filter(e => e.type === 'text');
-    output = finalStep?.part?.messageID ? texts.filter(e => e.part?.messageID === finalStep.part.messageID).map(e => e.part?.text ?? '').join('\n') : texts.at(-1)?.part?.text ?? '';
-    if (steps.length) {
-      usage = { input: sum(steps.map(e => sum([number(e.part?.tokens?.input), number(e.part?.tokens?.cache?.read ?? 0), number(e.part?.tokens?.cache?.write ?? 0)]))), output: sum(steps.map(e => sum([number(e.part?.tokens?.output), number(e.part?.tokens?.reasoning ?? 0)]))), cached: sum(steps.map(e => number(e.part?.tokens?.cache?.read ?? 0))) };
-      const costs = steps.map(e => number(e.part?.cost));
-      costUsd = costs.every(c => c !== null && c > 0) ? sum(costs) : null;
-      if (costUsd === null) partialCostUsd = sum(costs.filter(c => c !== null && c > 0));
-    }
-    for (const event of events.filter(e => e.type === 'tool_use' && e.part?.state?.status === 'error')) failureReason ??= toolFailure(event.part.state.error ?? '');
-  }
-  // Native zero prices can mean missing pricing metadata, not free inference.
-  return { output, usage, costUsd: costUsd > 0 ? costUsd : null, partialCostUsd, completed: completed && !failed, failureReason, observedModels: [...observedModels] };
-}
-
-export async function preflight(connection, { cwd, executeCommand = execute, platform = process.platform } = {}) {
-  const command = connection.command ?? connection.agent;
-  const options = { cwd, timeoutMs: 10_000, env: agentEnvironment(connection.env) };
-  const version = await executeCommand(command, ['--version'], options);
-  const help = await executeCommand(command, connection.agent === 'codex' ? ['exec', '--help'] : connection.agent === 'opencode' ? ['run', '--help'] : ['--help'], options);
-  const flags = connection.agent === 'codex' ? ['--json', '--ephemeral', '--sandbox'] : connection.agent === 'claude' ? ['--output-format', '--no-session-persistence', '--max-budget-usd'] : ['--format', '--model', '--auto'];
-  if (version.code !== 0 || help.code !== 0 || !flags.every(f => (help.stdout + help.stderr).includes(f))) throw new Error(`${connection.agent} lacks the required non-interactive JSON interface. Update it or set this connection's command to a compatible executable.`);
-  if (connection.agent === 'codex' && platform === 'linux') {
-    const sandboxHelp = await executeCommand(command, ['sandbox', '--help'], options);
-    const legacy = /^\s+linux\s/m.test(sandboxHelp.stdout);
-    const probe = await executeCommand(command, ['-c', 'sandbox_mode="workspace-write"', 'sandbox', ...(legacy ? ['linux'] : []), '--', process.execPath, '-e', 'process.stdout.write("openmerit-sandbox-ok")'], options);
-    if (probe.code !== 0 || probe.reason || probe.stdout !== 'openmerit-sandbox-ok') {
-      throw Object.assign(new Error('Codex sandbox could not run a local probe. Fix host sandbox support before evaluating; no model request was made.'), { failureReason: 'sandbox_unavailable' });
-    }
-  }
-  return version.stdout.trim().slice(0, 160);
-}
-
-export async function runAgent(connection, prompt, options) {
-  const result = await execute(connection.command ?? connection.agent, invocation(connection, options.reservationUsd, options.unlimitedBudget), { ...options, input: prompt, env: agentEnvironment(connection.env) });
-  const parsed = parseOutput(connection.agent, result.stdout);
-  const failureReason = parsed.failureReason ?? (result.code !== 0 ? connectionFailure(result.stderr) : null) ?? (['spawn_failed', 'cancelled', 'output_limit'].includes(result.reason) ? result.reason : null);
-  const status = failureReason === 'budget_exhausted' ? 'budget_exhausted' : ['sandbox_unavailable', 'permission_denied', 'spawn_failed'].includes(failureReason) ? 'environment_error' : result.reason ?? (failureReason || result.code !== 0 || !parsed.completed ? 'agent_error' : 'ok');
-  return { ...parsed, costUsd: status === 'ok' ? parsed.costUsd : null, partialCostUsd: status === 'ok' ? parsed.partialCostUsd : maximum([parsed.costUsd, parsed.partialCostUsd]), costBasis: parsed.costUsd === null && parsed.partialCostUsd === null ? 'unknown' : 'agent_estimate', seconds: result.seconds, status, failureReason };
-}

package/dist/eval/src/cli.mjs

@@ -1,151 +0,0 @@
-import { mkdir, readFile, writeFile } from 'node:fs/promises';
-import { join, resolve } from 'node:path';
-import { createInterface } from 'node:readline/promises';
-import { agents, preflight } from './adapters.mjs';
-import { configPath, statePath, starter, loadConfig, readJSON, saveJSON } from './config.mjs';
-import { makePlan, savePlan } from './scout.mjs';
-import { acquireLock, evaluate, watch } from './runner.mjs';
-import { formatReport } from './report.mjs';
-
-export const help = `OpenMerit · Agent evaluations (source preview)
-
-  openmerit eval init                 Add a starter suite and project agent instructions
-  openmerit eval doctor               Check installed Claude Code, Codex and OpenCode CLIs
-  openmerit eval plan [--scope existing|all] [--budget <USD|unlimited>]
-                                     Choose scope and budget, then freeze Scout's shortlist
-  openmerit eval run [--json]         Evaluate the frozen shortlist in fresh workspaces
-  openmerit eval watch [--json]       Repeat after selected CLI/docs/fixture files change
-  openmerit eval report [--json]      Read the latest result without running models
-
-  --path <directory>                 Project directory (default: current directory)
-
-Configure .openmerit/eval.json: explicit agent connections, task fixtures, local
-graders, CLI variants, 3–10 repeats, and a cumulative budget. Init starts at $0.
-Plan asks for $10, a custom amount, or no fixed cap; nothing is preselected.
-Saved choices and prior budget usage persist across reruns. --budget changes
-the total allowance, not the amount remaining. Reservations are automatic.
-Scout receives task descriptions and, for existing scope, model IDs only.
-Run/watch execute local agents and may incur charges. No background daemon.
-`;
-
-const instructions = `<!-- openmerit-eval:start -->
-## OpenMerit agent evaluations
-
-When building or changing this project's CLI, tools or harness documentation, use
-\`openmerit eval\` to compare task performance. Skip this workflow when
-\`OPENMERIT_EVAL=1\` to avoid recursive evaluations.
-
-- Read .openmerit/eval.json and run \`openmerit eval doctor\`.
-- Ask once: "Models available through this project's existing connections, or all models Scout indexes?" Use \`eval plan --scope existing\` or \`--scope all\`. Reuse the saved choice. Never infer endpoint access from a model name or expose credentials.
-- Then ask once: "What budget should OpenMerit use for this project's evaluations? 1. $10; 2. Custom amount (how much?); 3. No fixed cap." Pass the explicit answer as \`eval plan --budget 10\`, \`--budget <amount>\` or \`--budget unlimited\`, together with scope on first setup. Never choose a default or infer unlimited spending. Reuse an existing positive or unlimited budget unless the user changes it. The allowance is cumulative across reruns, not reset by edits or replanning. Reservations are internal; do not ask the user to calculate them. No fixed cap still has bounded tasks, 3–10 repetitions, timeouts and user-controlled watch; subsequent watched edits can incur further charges until stopped.
-- Declare real connections with Scout model IDs, agent-native model IDs, and environment variable NAMES. Existing scope uses this project-declared inventory. All scope may require new authenticated connections before execution.
-- Replace starter cases with representative tasks, fresh fixtures and deterministic standalone Node graders. A grader reads {workspace, output} JSON on stdin and returns {quality: 0..1}. Keep answer keys outside agent workspaces. Include a bare variant and a CLI/docs variant where useful.
-- Confirm the suite and variants before enabling paid runs, using the budget already selected above. Do not ask again for an unchanged authorized budget. Then plan once, run a baseline, make the requested CLI changes, and run again. Use \`eval watch\` only when the user wants continuous local reruns; it exits with the terminal and retains budget usage across restarts.
-- Report quality, failures, tokens, reported costs and elapsed time per task/model/variant. A shortlist is a prior, not measured success. Do not claim missing costs are zero or that 3–10 repeats prove an exact ordering. Keep fresh holdouts for final validation.
-- If a compatible watcher is already active, let it run; never start overlapping loops. Do not overwrite the frozen plan merely because the CLI changed.
-<!-- openmerit-eval:end -->`;
-
-export async function initialize(root) {
-  // Refuse partial overwrite of an existing suite or someone's fixtures.
-  const files = {
-    [configPath]: JSON.stringify(starter, null, 2) + '\n',
-    'eval/fixtures/json/request.txt': 'Customer: Mira Chen\nItems: 7\n',
-    'eval/tool.mjs': `import {readFileSync} from 'node:fs';\nconst s=readFileSync(process.argv[2],'utf8');\nconsole.log(JSON.stringify({name:s.match(/Customer: (.*)/)?.[1],count:Number(s.match(/Items: (\\d+)/)?.[1])}));\n`,
-    'eval/grade.mjs': `let input=''; for await (const chunk of process.stdin) input+=chunk;\nconst {output}=JSON.parse(input); let quality=0;\ntry {const v=JSON.parse(output); quality=(Number(v.name==='Mira Chen')+Number(v.count===7)+Number(Object.keys(v).sort().join(',')==='count,name'))/3;} catch {}\nconsole.log(JSON.stringify({quality}));\n`,
-  };
-  for (const file of Object.keys(files)) {
-    try { await readFile(join(root, file)); throw new Error(`Already exists: ${file}. Init does not overwrite a suite.`); } catch (error) { if (error.code !== 'ENOENT') throw error; }
-  }
-  for (const [file, body] of Object.entries(files)) { await mkdir(join(root, file, '..'), { recursive: true }); await writeFile(join(root, file), body, { flag: 'wx' }); }
-  for (const name of ['AGENTS.md', 'CLAUDE.md']) {
-    let body = ''; try { body = await readFile(join(root, name), 'utf8'); } catch (e) { if (e.code !== 'ENOENT') throw e; }
-    if (!body.includes('<!-- openmerit-eval:start -->')) await writeFile(join(root, name), `${body}${body ? '\n\n' : ''}${instructions}\n`);
-  }
-  let ignore = ''; try { ignore = await readFile(join(root, '.gitignore'), 'utf8'); } catch (e) { if (e.code !== 'ENOENT') throw e; }
-  if (!ignore.split('\n').includes('/.openmerit/eval/')) await writeFile(join(root, '.gitignore'), `${ignore}${ignore.endsWith('\n') || !ignore ? '' : '\n'}/.openmerit/eval/\n`);
-}
-
-export function parseBudget(value) {
-  const input = value.trim();
-  if (input === 'unlimited') return input;
-  const amount = input.replace(/^\$/, '');
-  if (!/^(?:\d+(?:\.\d{1,2})?|\.\d{1,2})$/.test(amount) || Number(amount) <= 0 || Number(amount) > Number.MAX_SAFE_INTEGER / 100) throw new Error('Enter a positive USD amount with at most two decimal places, or "unlimited" for no fixed cap.');
-  return Number(amount);
-}
-
-// ask is supplied only for an interactive terminal. Coding agents pass explicit
-// flags after asking the user; EOF, blank input and missing flags never consent.
-export async function choosePlanOptions(config, { scope, budget, ask, tell = () => {} } = {}) {
-  const next = structuredClone(config);
-  if (scope) next.scope = scope;
-  while (!['existing', 'all'].includes(next.scope)) {
-    if (!ask) throw new Error('Choose a scope with eval plan --scope existing or --scope all, then choose --budget <USD|unlimited>.');
-    const answer = (await ask('Use existing project connections or all models Scout indexes? [existing/all] ')).trim();
-    if (['existing', 'all'].includes(answer)) next.scope = answer;
-    else tell('Choose existing or all.');
-  }
-  if (budget !== undefined) next.budget = { maxUsd: parseBudget(budget), reservationUsd: 'auto' };
-  while (next.budget.maxUsd === 0) {
-    if (!ask) throw new Error('Choose an evaluation budget after model scope: --budget 10, --budget <USD>, or --budget unlimited. No amount is selected automatically.');
-    const choice = (await ask('What budget should OpenMerit use for this project’s evaluations?\n  1. $10\n  2. Custom amount\n  3. No fixed cap (watched changes can incur further charges until stopped)\nChoose [1/2/3]: ')).trim();
-    if (choice === '1') next.budget = { maxUsd: 10, reservationUsd: 'auto' };
-    else if (choice === '3') next.budget = { maxUsd: 'unlimited', reservationUsd: 'auto' };
-    else if (choice === '2') {
-      while (next.budget.maxUsd === 0) {
-        const amount = await ask('How much in USD? ');
-        try {
-          const parsed = parseBudget(amount);
-          if (parsed === 'unlimited') throw new Error('Enter a custom USD amount.');
-          next.budget = { maxUsd: parsed, reservationUsd: 'auto' };
-        } catch { tell('Enter a positive USD amount with at most two decimal places.'); }
-      }
-    } else tell('Choose 1, 2 or 3. Nothing is selected automatically.');
-  }
-  return next;
-}
-
-export async function main(args) {
-  if (!args.length || args.includes('--help') || args.includes('-h')) { console.log(help); return; }
-  const action = args.shift(); let root = process.cwd(), scope, budget, json = false;
-  for (let i = 0; i < args.length; i++) {
-    if (args[i] === '--path' && args[i + 1]) root = resolve(args[++i]);
-    else if (args[i] === '--scope' && ['existing', 'all'].includes(args[i + 1])) scope = args[++i];
-    else if (args[i] === '--budget' && args[i + 1] && budget === undefined) { budget = args[++i]; parseBudget(budget); }
-    else if (args[i] === '--json') json = true;
-    else throw new Error(`Unknown eval argument: ${args[i]}`);
-  }
-  if (scope && action !== 'plan') throw new Error('--scope belongs to eval plan.');
-  if (budget !== undefined && action !== 'plan') throw new Error('--budget belongs to eval plan.');
-  if (action === 'init') { await initialize(root); console.log('Created .openmerit/eval.json, starter fixtures, and AGENTS.md / CLAUDE.md instructions. Declare connections, replace the starter task, and set an approved budget before running.'); return; }
-  if (action === 'doctor') {
-    const results = [];
-    for (const agent of agents) { try { results.push({ agent, version: await preflight({ agent }), ready: true }); } catch (error) { results.push({ agent, ready: false, reason: error.message }); } }
-    console.log(JSON.stringify(results, null, 2)); return;
-  }
-  if (action === 'report') { const report = await readJSON(join(root, statePath, 'latest.json')); console.log(json ? JSON.stringify(report, null, 2) : formatReport(report)); return; }
-  if (!['plan', 'run', 'watch'].includes(action)) throw new Error('Unknown eval command. Run openmerit eval --help.');
-  const unlock = await acquireLock(root);
-  const controller = new AbortController(), stop = () => controller.abort();
-  process.once('SIGINT', stop); process.once('SIGTERM', stop);
-  try {
-    if (action === 'plan') {
-      const saved = await loadConfig(root);
-      const rl = process.stdin.isTTY && !json ? createInterface({ input: process.stdin, output: process.stdout }) : null;
-      const inputClosed = new AbortController();
-      rl?.once('close', () => inputClosed.abort());
-      let config;
-      try { config = await choosePlanOptions(saved, { scope, budget, ask: rl ? question => rl.question(question, { signal: AbortSignal.any([controller.signal, inputClosed.signal]) }) : undefined, tell: message => console.log(message) }); }
-      finally { rl?.close(); }
-      // Preserve explicit choices even if Scout is temporarily unavailable.
-      await saveJSON(join(root, configPath), config);
-      const plan = await makePlan(config);
-      await savePlan(root, plan);
-      console.log(JSON.stringify(plan, null, 2));
-      return;
-    }
-    const onReport = report => { console.log(json ? JSON.stringify(report) : formatReport(report)); if (report.status !== 'complete') process.exitCode = 1; };
-    const onProgress = json ? () => {} : row => process.stderr.write(`${row.task} · ${row.connection} · ${row.variant} · ${row.round}: ${row.status}\n`);
-    if (action === 'run') onReport(await evaluate(root, { signal: controller.signal, onProgress }));
-    else { if (!json) console.log('Watching selected files. Ctrl+C stops; budget usage persists.'); await watch(root, { signal: controller.signal, onReport, onProgress }); }
-  } finally { process.removeListener('SIGINT', stop); process.removeListener('SIGTERM', stop); await unlock(); }
-}