openmerit 0.2.0 → 0.3.1

package/CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## 0.3.1 — 2026-11-05
+
+- `openmerit scout --conversation <id> --results <file.json>` reports aggregate results you measured with your own evals. Scout then ranks them above public benchmarks for your recommendations only. Terminal output shows each task's evaluation plan, its agent and calls per run, and marks quality that rests on measured results.
+- The installed Scout instructions for Claude Code and Pi tell the agent to measure shortlisted candidates as extra runs of the project's existing evals, never a second eval, and to report aggregates only. This needs the matching hosted Scout release, whose `scout_models` tool returns `tasks[].evaluation` and accepts `results`.
+
+## 0.3.0 — 2026-11-04
+
+- **Breaking:** retire the standalone `openmerit eval` CLI, including `init`, `doctor`, `plan`, `run`, `watch` and `report`. Existing local evaluation files are not deleted. The hosted `/scout/api/recommend` retirement ships separately with the Scout serving Worker.
+- **Breaking:** retire the local Inference Terminal. `openmerit dash` and all of its options, including `--demo`, `--json` and `--diagnostics`, are removed, along with repository indexing, request-log and activity-export import, runtime and hardware discovery, the browser interface and its opt-in terminal analytics events. Existing `.openmerit/terminal.json`, `.openmerit/inference.jsonl` and application logs are not deleted. Scout setup, `openmerit scout` and `openmerit update` are unchanged.
+- Drop the terminal-only runtime dependencies (`@iarna/toml`, `@lezer/python`) and browser bundle dependencies.
+
 ## 0.2.0 — 2026-11-04
 
 - Add `openmerit scout`, a conversation with Scout from the terminal: clarifying questions answered with `--brief` or `--skip`, then up to five models per task, cheapest first among models that reach Scout's quality bar. When you ask for the fastest or newest model, the output names that order and shows each model's time per task or release date. It uses this device's Claude Code or Pi key and limits new conversations to the endpoint IDs in `.openmerit/scout.json`.

package/README.md

@@ -1,24 +1,8 @@
 # OpenMerit
 
-OpenMerit provides a local Inference Terminal for understanding an application's models, services, recorded activity, and compute, plus Scout guidance for application-model choices.
+OpenMerit provides Scout guidance for application-model choices: in the browser, from the `openmerit scout` command, and through MCP for Claude Code and Pi.
 
-The terminal reads existing evidence; Scout recommends candidates to evaluate. Neither is a request router or proof of application-quality improvements. A separate standalone evaluation CLI supports explicit evaluation workflows.
-
-## Open the terminal
-
-Requires Node.js 22.19 or newer and npm:
-
-```sh
-npm install -g openmerit
-cd /path/to/your/application
-openmerit dash
-```
-
-No Pi installation or provider API key is needed. The terminal opens a browser connected to a read-only loopback service. Supported source/configuration declarations establish inventory; existing application logs or exports supply request history. Prompts, responses, and credentials are not retained by terminal collection. Unsupported or missing evidence remains unknown.
-
-The terminal follows the device's light/dark preference without a theme control. CLI and terminal product analytics are off unless explicitly enabled; see [the analytics contract](docs/inference-terminal.md#optional-cli-product-analytics).
-
-Run `openmerit dash --demo` to explore a fictional sample workspace. See [the terminal guide](docs/inference-terminal.md) for configuration, discovery, diagnostics, history limits, and agent access.
+Scout recommends candidates to evaluate. It is not a request router or proof of application-quality improvements.
 
 ## Connect Scout to Claude Code
 
@@ -38,17 +22,21 @@ There are no `/openmerit` lifecycle commands, evaluation setup, scheduler, Launc
 
 Package publication and website deployment are separate releases. Pi setup requires OpenMerit 0.1.12 or newer; check the resolved package version when using `@latest`.
 
-## Evaluate application changes
+## Evaluation CLI retirement — npm 0.3.0
+
+OpenMerit 0.3.0 removes the standalone `openmerit eval` commands, including planning. Existing local evaluation files are not deleted. The old Scout JSON recommendation API went offline with the Scout serving deploy of 5 November 2026. Scout's conversational agent, MCP, coding-agent setup and catalogue publication remain available; see [retirement details](docs/agent-evaluations.md).
+
+## Inference Terminal retirement — npm 0.3.0
 
-The standalone `openmerit eval` CLI is unchanged by the Pi replacement. Evaluate Claude Code, Codex and OpenCode with explicit tasks, model scope, budget and graders. Freeze Scout's shortlist and compare fresh repeated attempts after source changes. Budget and sandbox blocks stay separate from task scores; unknown costs stay unknown. See [agent evaluations](docs/agent-evaluations.md) for setup, graders, budgets and availability.
+OpenMerit 0.3.0 also removes the local Inference Terminal: `openmerit dash`, its `--demo` sample, collectors, repository index and browser interface. Existing `.openmerit/terminal.json`, `.openmerit/inference.jsonl` and application logs are not deleted. Older installed packages keep the command until they are upgraded. See [retirement details](docs/inference-terminal.md).
 
 ## Documentation
 
-The [documentation overview](docs/index.md) links the current guides. The website at [openmerit.ai/docs](https://openmerit.ai/docs/) is deployed separately from the package and this checkout.
+The [documentation overview](docs/index.md) links the maintained Markdown guides. The unpublished `/docs/` website shell is retired; the live Scout data policy and protected memo have separate builds.
 
 - [Getting started](docs/getting-started.md) and [troubleshooting](docs/troubleshooting.md)
 - [Scout](docs/scout.md) and [Pi adapter](docs/pi-extension.md)
-- [Inference Terminal](docs/inference-terminal.md) and [agent evaluations](docs/agent-evaluations.md)
+- Retired: [Inference Terminal](docs/inference-terminal.md) and [agent evaluations](docs/agent-evaluations.md)
 - [Security and data handling](docs/security.md)
 - [Testing](docs/testing.md) and [current limitations](docs/roadmap.md)
 
@@ -57,9 +45,8 @@ The [documentation overview](docs/index.md) links the current guides. The websit
 ```sh
 npm ci
 npm run verify
-npm run dash -- /path/to/application
 ```
 
-`packages/terminal` contains the CLI, collectors and browser interface. `packages/eval` contains the standalone `openmerit eval` runner, and `packages/pi` supplies only the native Scout adapter. The package exports `openmerit/pi` and the `openmerit` command. See the [Pi package README](packages/pi/README.md) for bounded adapter tests. Provider credentials belong to the harness or its environment and must never be committed.
+`packages/terminal` contains the `openmerit` CLI: Scout setup, updates and the `scout` command; `packages/pi` supplies only the native Scout adapter. The package exports `openmerit/pi` and the `openmerit` command. See the [Pi package README](packages/pi/README.md) for bounded adapter tests. Provider credentials belong to the harness or its environment and must never be committed.
 
-Edit documentation in `docs/`, then run `npm run check:docs`. The website is generated from those Markdown files. See [documentation maintenance](docs/site-documentation.md) and the [production-source reconciliation note](docs/site-domain-migration.md) before deployment; a local build does not publish changes.
+Edit documentation in `docs/`, then run `npm run check:docs`. `npm run build:site` builds only the Scout data policy and protected memo, not a docs website. A local build does not publish changes; follow [deployment maintenance](docs/site-deployment.md) before any approved release.

package/dist/pi/scout-extension.mjs

@@ -117,6 +117,26 @@ Compare quality evidence, reference cost/time and watch-outs. Prefer the cheapes
 usable option the evidence supports, without claiming benchmarks prove
 application accuracy. Use the project's actual SDK and protocol.
 
+## Measuring candidates
+
+Each task also has an \`evaluation\`: the checks (per model call, agent run or
+end-to-end outcome), labelled examples per candidate and when to stop a losing
+candidate. Offer to run it before the user commits to a model. Use the evals the
+project already has, such as datasets and experiments in its eval or tracing
+platform, or its own eval scripts and fixtures: add the shortlisted candidates as
+extra runs of those evals. Do not build a second eval or copy their datasets.
+If the project has no eval for the task, propose a minimal one built from its
+existing traces and the decisions people already make, and create it only where
+and how the user agrees. Ask before spending on model calls or writing to an
+eval platform.
+
+Report what you measured with \`scout_models\`: the same \`conversationId\` and
+\`results\`, one aggregate per model and metric (\`level\`, \`task\` title, \`scope\`,
+\`model\` endpoint or model ID, \`metric\`, \`value\`, \`higherIsBetter\`,
+\`samples\`, \`source\`). Send aggregates only, never inputs, outputs or records.
+Scout ranks them above public benchmarks for this user only; models whose
+\`quality.measured\` is true rest on them.
+
 ## Boundaries
 
 Ask before adding providers or credentials, spending, deploying or external

package/dist/terminal/scout-update.json

@@ -1 +1 @@
-{"format":1,"contract":"scout-recommend-v1","version":"0.2.0","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}
+{"format":1,"contract":"scout-recommend-v1","version":"0.3.1","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Measuring candidates\n\nEach task also has an `evaluation`: the checks (per model call, agent run or\nend-to-end outcome), labelled examples per candidate and when to stop a losing\ncandidate. Offer to run it before the user commits to a model. Use the evals the\nproject already has, such as datasets and experiments in its eval or tracing\nplatform, or its own eval scripts and fixtures: add the shortlisted candidates as\nextra runs of those evals. Do not build a second eval or copy their datasets.\nIf the project has no eval for the task, propose a minimal one built from its\nexisting traces and the decisions people already make, and create it only where\nand how the user agrees. Ask before spending on model calls or writing to an\neval platform.\n\nReport what you measured with `scout_models`: the same `conversationId` and\n`results`, one aggregate per model and metric (`level`, `task` title, `scope`,\n`model` endpoint or model ID, `metric`, `value`, `higherIsBetter`,\n`samples`, `source`). Send aggregates only, never inputs, outputs or records.\nScout ranks them above public benchmarks for this user only; models whose\n`quality.measured` is true rest on them.\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}

package/dist/terminal/src/browser.mjs

@@ -1,9 +1,7 @@
 import { spawn } from 'node:child_process';
 
-// By default, only open the loopback address returned by our own server.
-// Callers can pass a stricter or different check. Never use a shell.
-const terminalURL = url => { const target = new URL(url); return target.protocol === 'http:' && target.hostname === '127.0.0.1' && /^http:\/\/127\.0\.0\.1:[0-9]+$/.test(url) && !target.username && !target.password && target.pathname === '/' && !target.search && !target.hash; };
-export function browserCommand(url, platform = process.platform, env = process.env, allowed = terminalURL) {
+// Callers decide which URLs may open. Never use a shell.
+export function browserCommand(url, platform = process.platform, env = process.env, allowed) {
   if (!allowed(url)) throw new Error('Refusing to open an unexpected URL.');
   if (env.SSH_CONNECTION || env.SSH_TTY) return null;
   if (platform === 'darwin') return ['open', [url]];

package/dist/terminal/src/cli.mjs

@@ -1,77 +1,25 @@
 #!/usr/bin/env node
-import { resolve } from 'node:path';
-import { stat } from 'node:fs/promises';
-import { startTerminal } from './server.mjs';
-import { createCollector, loadConfig } from './collect.mjs';
-import { sampleSnapshot } from './sample.mjs';
-import { openBrowser } from './browser.mjs';
 import { commandTelemetry } from './telemetry.mjs';
 
-const help = `OpenMerit · Inference Terminal
+const help = `OpenMerit
 
-  openmerit dash [path] [options]
   openmerit scout --help             Ask Scout which models fit your tasks
   openmerit install --help           Connect OpenMerit to your coding agent
   openmerit update --help            Update installed Scout instructions
-  openmerit eval --help              Evaluate CLI changes across coding agents
-
-  --port <number>      Require a specific local port (default: 4310 or a free port)
-  --interval <seconds> Collection interval, 10–30 (default: 15)
-  --no-discovery       Skip project/log discovery and extra runtime discovery
-  --no-open            Print the URL without opening a browser
-  --json               Print one snapshot for an agent, then exit
-  --diagnostics        Print shareable index coverage counts, then exit
-  --demo               Explore a clearly labeled fictional environment
-  --help               Show this help
-
-Opens your default browser after the local terminal is ready.
-Reads application metadata and local runtime inventory. Makes no inference calls.
-Coding-harness usage is outside this terminal's scope.
+  openmerit --help                   Show this help
 `;
 
 async function main() {
   const args = process.argv.slice(2);
   if (['install', 'status', 'remove', 'login', 'logout', 'project', 'doctor', 'update'].includes(args[0])) { const { main } = await import('./install.mjs'); process.exitCode = await main(args); return; }
   if (args[0] === 'scout') { const { main } = await import('./scout.js'); process.exitCode = await main(args.slice(1)); return; }
-  if (args[0] === 'eval') { const { main } = await import('../../eval/src/cli.mjs'); await main(args.slice(1)); return; }
   if (!args.length || args.includes('--help') || args.includes('-h')) { console.log(help); return; }
-  if (args.shift() !== 'dash') throw new Error('Unknown command. Run openmerit --help.');
-  let root = process.cwd(), port = 4310, interval = 15, demo = false, json = false, pathSeen = false, discovery = true, noOpen = false, explicitPort = false, diagnostics = false;
-  for (let i = 0; i < args.length; i++) {
-    const arg = args[i];
-    if (arg === '--port' || arg === '--interval') {
-      const value = args[++i];
-      if (!/^\d+$/.test(value ?? '')) throw new Error(`${arg} requires an integer.`);
-      if (arg === '--port') { port = Number(value); explicitPort = true; } else interval = Number(value);
-    } else if (arg === '--no-open') noOpen = true;
-    else if (arg === '--no-discovery') discovery = false;
-    else if (arg === '--access' || arg === '--revoke-access') throw new Error('Coding-harness history is no longer collected. Saved history permissions are ignored; these access options have been removed.');
-    else if (arg === '--demo') demo = true;
-    else if (arg === '--json') json = true;
-    else if (arg === '--diagnostics') diagnostics = true;
-    else if (!arg.startsWith('-') && !pathSeen) { root = resolve(arg); pathSeen = true; }
-    else throw new Error(`Unknown argument: ${arg}`);
-  }
-  if (port < 1 || port > 65535) throw new Error('Port must be between 1 and 65535.');
-  if (interval < 10 || interval > 30) throw new Error('Collection interval must be between 10 and 30 seconds.');
-  if (!(await stat(root)).isDirectory()) throw new Error('Choose a project directory.');
-  const config = demo ? null : await loadConfig(root);
-  if (config && !discovery) config.discovery = false;
-  if (diagnostics) { const {indexRepository,indexDiagnostics}=await import('./repository-index.mjs'); const index = !demo && config.discovery ? await indexRepository(root) : null; console.log(JSON.stringify(indexDiagnostics(index), null, 2)); return; }
-  if (json) { console.log(JSON.stringify(demo ? sampleSnapshot() : await createCollector(root, config, interval)(), null, 2)); return; }
-  const terminal = await startTerminal({ root, port, interval, demo, config, fallbackPort: !explicitPort });
-  console.log(`\n  OpenMerit · Inference Terminal\n\n  ${terminal.url}\n\n  ${demo ? 'Sample environment · fictional data' : 'Current project · passive collection'}\n  Updates every ${interval}s. Press Ctrl+C to close.\n`);
-  const close = () => { void terminal.close().then(() => process.exit(0)); };
-  process.once('SIGINT', close); process.once('SIGTERM', close);
-  if (!demo && config.discovery) {
-    console.log('  Automatic discovery: repository code/configuration, saved request logs, and project-linked runtimes.');
-  }
-  if (!noOpen && !await openBrowser(terminal.url)) console.log('  Open the local URL above in your browser.');
+  throw new Error('Unknown command. Run openmerit --help.');
 }
 const completeTelemetry = commandTelemetry(process.argv.slice(2));
 const setupCodes = { 10: 'approval_pending', 11: 'access_denied', 20: 'install_conflict', 30: 'openmerit_unreachable' };
 main().then(() => completeTelemetry(process.exitCode ? 'error' : 'ok', undefined, setupCodes[process.exitCode]), async error => {
-  console.error(`\n  ${error.code === 'EADDRINUSE' ? 'Port is in use. Choose another with --port.' : error.message}\n`);
+  console.error(`\n  ${error.message}\n`);
   process.exitCode = 1;
   await completeTelemetry('error', error);
 });

package/dist/terminal/src/install.mjs

@@ -15,13 +15,13 @@ export { rule, skill } from './scout-instructions.js';
 // One-command coding-agent setup: device sign-in in the browser, then the MCP
 // server, startup rule and Scout skill. Built to be run by a person or by a coding agent:
 // no interactive prompts, bounded waits, resumable logins and explicit exit codes.
-export const exitCodes = { ok: 0, error: 1, pending: 10, denied: 11, conflict: 20, unreachable: 30 };
+const exitCodes = { ok: 0, error: 1, pending: 10, denied: 11, conflict: 20, unreachable: 30 };
 const exec = promisify(execFile);
 const hash = value => createHash('sha256').update(value).digest('hex');
 const policyPath = '.openmerit/scout.json';
 const clients = { 'claude-code': 'Claude Code', pi: 'Pi' };
 
-export const help = `OpenMerit · coding-agent setup
+const help = `OpenMerit · coding-agent setup
 
   openmerit install [--client claude-code|pi] [--scope user|project] [--setup-id UUID] [--auto-update on|off] [--plan] [--json] [--no-open]
   openmerit status  [--client claude-code|pi] [--scope user|project] [--check] [--json]

package/dist/terminal/src/scout-instructions.js

@@ -83,6 +83,26 @@ Compare quality evidence, reference cost/time and watch-outs. Prefer the cheapes
 usable option the evidence supports, without claiming benchmarks prove
 application accuracy. Use the project's actual SDK and protocol.
 
+## Measuring candidates
+
+Each task also has an \`evaluation\`: the checks (per model call, agent run or
+end-to-end outcome), labelled examples per candidate and when to stop a losing
+candidate. Offer to run it before the user commits to a model. Use the evals the
+project already has, such as datasets and experiments in its eval or tracing
+platform, or its own eval scripts and fixtures: add the shortlisted candidates as
+extra runs of those evals. Do not build a second eval or copy their datasets.
+If the project has no eval for the task, propose a minimal one built from its
+existing traces and the decisions people already make, and create it only where
+and how the user agrees. Ask before spending on model calls or writing to an
+eval platform.
+
+Report what you measured with \`scout_models\`: the same \`conversationId\` and
+\`results\`, one aggregate per model and metric (\`level\`, \`task\` title, \`scope\`,
+\`model\` endpoint or model ID, \`metric\`, \`value\`, \`higherIsBetter\`,
+\`samples\`, \`source\`). Send aggregates only, never inputs, outputs or records.
+Scout ranks them above public benchmarks for this user only; models whose
+\`quality.measured\` is true rest on them.
+
 ## Boundaries
 
 Ask before adding providers or credentials, spending, deploying or external

package/dist/terminal/src/scout-pi.js

@@ -26,7 +26,7 @@ export function piPaths(root, scope) {
     const base = scope === 'user' ? agentDir : join(root, '.pi');
     return { skill: join(base, 'skills/openmerit-scout/SKILL.md'), rule: join(base, 'extensions/openmerit-scout.ts'),
         connection: join(base, 'extensions/openmerit-scout.json'), record: join(base, 'openmerit-install.json'),
-        legacyRule: join(base, 'rules/openmerit.md'), mcpFile: join(base, 'mcp.json'), mcpScope: scope, agentDir };
+        mcpFile: join(base, 'mcp.json'), mcpScope: scope, agentDir };
 }
 export async function inspectPi(root, scope) {
     const paths = piPaths(root, scope);

package/dist/terminal/src/scout.js

@@ -3,20 +3,25 @@
 // the same scout_models MCP tool with this device's key, so every client gets the
 // same clarifying questions and per-task shortlists.
 import { readFile } from 'node:fs/promises';
-import { join } from 'node:path';
+import { join, resolve } from 'node:path';
 import { credentialsPath, readCredentials } from "./scout-credentials.js";
-export const policyFile = '.openmerit/scout.json';
+const policyFile = '.openmerit/scout.json';
 export const components = ['input', 'work', 'output', 'criteria', 'constraints'];
-export const help = `OpenMerit · Scout
+const help = `OpenMerit · Scout
 
   openmerit scout "<the work models need to do>" [--all-endpoints] [--json]
   openmerit scout --conversation <id> [--brief <component>=<text> ...] [--skip] ["<follow-up>"] [--json]
+  openmerit scout --conversation <id> --results <file.json> [--json]
 
 Asks Scout, the agent behind openmerit.ai/scout. Scout may first return a task
 brief with clarifying questions: answer them with --brief (components: input,
 work, output, criteria, constraints) in the same conversation, or --skip to let
 Scout assume. It then splits the work into tasks and recommends up to five
-models for each. A turn usually takes one to three minutes.
+models for each, with an evaluation: how to measure the candidates with your
+own evals. Report what you measured with --results: a JSON list of aggregate
+results ({level, task?, scope, model, metric, value, higherIsBetter, samples,
+source}), never inputs or outputs. Scout ranks them above public benchmarks, for
+your recommendations only. A turn usually takes one to three minutes.
 
 A new conversation is limited to the endpoint IDs in ${policyFile}
 (set with openmerit project). --all-endpoints searches the whole catalogue and is
@@ -46,7 +51,7 @@ function deviceKey(root) {
     return null;
 }
 // One turn through the scout_models MCP tool. Errors are written for the person or agent running it.
-export async function askScout(root, request, signal) {
+async function askScout(root, request, signal) {
     const credentials = deviceKey(root);
     if (!credentials)
         throw new Error('This device is not signed in to OpenMerit. Run: npx -y openmerit@latest install');
@@ -114,7 +119,7 @@ export function renderScout(result) {
         const tasks = result.tasks ?? [];
         tasks.forEach((task, i) => {
             if (tasks.length > 1)
-                lines.push(`${i + 1}. ${task.title}`, `   ${task.work}`);
+                lines.push(`${i + 1}. ${task.title}${task.agent ? ` (${task.agent}${task.callsPerRun && task.callsPerRun > 1 ? `, ${task.callsPerRun} calls per run` : ''})` : ''}`, `   ${task.work}`);
             // As on the page, the default cheapest-first order is unlabelled; another priority is named with the value it sorts by.
             const priority = task.priority ?? 'cost';
             if (priority !== 'cost')
@@ -123,14 +128,20 @@ export function renderScout(result) {
                 const price = m.cost ? `${money(m.cost.inputPerMillion)} in / ${money(m.cost.outputPerMillion)} out per 1M via ${m.cost.route} (${m.cost.endpointId})` : 'no priced route';
                 const metric = priority === 'latency' ? ` · ${m.latency ? `${m.latency.seconds.toFixed(1)}s per task` : 'time unknown'}`
                     : priority === 'recency' ? ` · ${m.recency ? `released ${m.recency.released}` : 'release date unknown'}` : '';
-                lines.push(`   ${m.rank}. ${m.name}${metric} · quality ${m.quality.score}/100 · ${price}`, `      ${m.rationale}`);
+                lines.push(`   ${m.rank}. ${m.name}${metric} · quality ${m.quality.score}/100${m.quality.measured ? ' (measured)' : ''} · ${price}`, `      ${m.rationale}`);
             }
-            lines.push(`   ${task.summary}`, '');
+            lines.push(`   ${task.summary}`);
+            if (task.evaluation)
+                lines.push(`   Measure with your evals (${task.evaluation.samples} labelled examples per candidate; ${task.evaluation.stopEarly}):`, ...task.evaluation.checks.map(c => `     - ${c.level}: ${c.metric}. ${c.how}`));
+            lines.push('');
         });
         if (result.assumptions?.length)
             lines.push('Assumptions:', ...result.assumptions.map(a => `  - ${a}`), '');
         lines.push(`Refine: openmerit scout --conversation ${result.conversationId} "<follow-up>"`);
-        lines.push('Quality is Scout\'s estimate from public benchmarks, not measured accuracy for your application.');
+        lines.push(`Report measured results: openmerit scout --conversation ${result.conversationId} --results <file.json>`);
+        lines.push(tasks.some(t => t.models.some(m => m.quality.measured))
+            ? 'Quality marked (measured) rests on the results you reported; the rest is Scout\'s estimate from public benchmarks.'
+            : 'Quality is Scout\'s estimate from public benchmarks, not measured accuracy for your application.');
     }
     else if (result.status === 'working')
         lines.push(`Scout is still working. Read the outcome with: openmerit scout --conversation ${result.conversationId}`);
@@ -164,6 +175,19 @@ export async function main(args, { root = process.cwd(), print = console.log } =
             request.allEndpoints = true;
         else if (arg === '--json')
             json = true;
+        else if (arg === '--results') {
+            const file = value();
+            let results;
+            try {
+                results = JSON.parse(await readFile(resolve(root, file), 'utf8'));
+            }
+            catch {
+                throw new Error(`Could not read results from ${file}: it must be a JSON list.`);
+            }
+            if (!Array.isArray(results))
+                throw new Error(`${file} must hold a JSON list of results.`);
+            request.results = results;
+        }
         else if (arg.startsWith('-'))
             throw new Error(`Unknown option for scout: ${arg}`);
         else
@@ -175,6 +199,8 @@ export async function main(args, { root = process.cwd(), print = console.log } =
         print(help);
         return 1;
     }
+    if (request.results && !request.conversationId)
+        throw new Error('--results continues a conversation: pass --conversation <id> from the recommendation you measured.');
     const result = await scoutTurn(root, request);
     print(json ? JSON.stringify(result, null, 2) : renderScout(result));
     return result.status === 'error' ? 1 : 0;

package/dist/terminal/src/telemetry.mjs

@@ -5,12 +5,8 @@ export let clientVersion;
 for (const path of ['../../../package.json', '../../../../package.json']) {
   try { const pkg = JSON.parse(readFileSync(new URL(path, import.meta.url), 'utf8')); if (pkg.name === 'openmerit') { clientVersion = pkg.version; break; } } catch {}
 }
-export const frameFiles = Object.freeze(['cli.mjs', 'server.mjs', 'collect.mjs', 'repository-index.mjs', 'install.mjs', 'scout.mjs', 'config.mjs', 'runner.mjs', 'adapters.mjs', 'process.mjs', 'report.mjs', 'app.js']);
-const applicationURLs = frameFiles.flatMap(file => ['terminal/src/', 'eval/src/'].map(dir => new URL(`../../${dir}${file}`, import.meta.url).href));
-const diagnostics = new WeakMap();
-export function scoutDiagnostic(error, trace, code) {
-  if (error && typeof error === 'object' && ['scout_network_error', 'scout_http_error', 'scout_response_error', 'scout_timeout'].includes(code)) diagnostics.set(error, { operation: 'scout_recommend', code, ...(typeof trace === 'string' && /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(trace) ? { trace_id: trace } : {}) });
-}
+const frameFiles = Object.freeze(['cli.mjs', 'install.mjs']);
+const applicationURLs = frameFiles.map(file => new URL(`../../terminal/src/${file}`, import.meta.url).href);
 export function safeFrames(error) {
   try {
     const stack = error?.stack;
@@ -24,7 +20,7 @@ export function safeFrames(error) {
   } catch { return []; }
 }
 export function errorProperties(error) {
-  try { return { error_type: errorType(error), frames: safeFrames(error), ...(diagnostics.get(error) ?? {}) }; }
+  try { return { error_type: errorType(error), frames: safeFrames(error) }; }
   catch { return { error_type: 'Error', frames: [] }; }
 }
 const environment = () => ({ ...(clientVersion ? { client_version: clientVersion } : {}), platform: ['darwin', 'linux', 'win32'].includes(process.platform) ? process.platform : 'unknown', runtime_major: Number(process.versions.node.split('.')[0]) });
@@ -99,65 +95,17 @@ export async function sendInstallationEvent(input) {
   } catch { /* Never affect setup, even if the transport ignores abort. */ }
   finally { clearTimeout(timer); }
 }
-export const errorType = error => ['Error', 'TypeError', 'RangeError', 'SyntaxError', 'ReferenceError', 'URIError', 'EvalError', 'AggregateError', 'AbortError'].includes(error?.name) ? error.name : 'Error';
-export function terminalProperties(event, input) {
-  if (!input || typeof input !== 'object' || Array.isArray(input)) return null;
-  const out = {};
-  const select = (key, values) => { if (!values.includes(input[key])) return false; out[key] = input[key]; return true; };
-  if (event === 'terminal_error') {
-    if (!select('code', ['collection_error', 'snapshot_error', 'render_error', 'uncaught_error']) || !select('error_type', ['Error', 'TypeError', 'RangeError', 'SyntaxError', 'ReferenceError', 'URIError', 'EvalError', 'AggregateError', 'AbortError'])) return null;
-    if (Array.isArray(input.frames)) out.frames = input.frames.slice(0, 8).filter(f => f && frameFiles.includes(f.filename) && Number.isInteger(f.lineno) && f.lineno > 0 && f.lineno <= 1000000 && Number.isInteger(f.colno) && f.colno > 0 && f.colno <= 1000000).map(f => ({ filename: f.filename, lineno: f.lineno, colno: f.colno }));
-    return out;
-  }
-  if (!select('mode', ['sample', 'live'])) return null;
-  if (event === 'terminal_visited') return out;
-  if (event === 'terminal_page_viewed') return select('page', ['overview', 'models', 'services', 'activity', 'compute', 'sources', 'connections']) ? out : null;
-  if (event === 'terminal_interaction') return select('action', ['inspect', 'filter', 'search', 'map_expand', 'mode', 'command']) && select('kind', ['model', 'agent', 'task', 'service', 'source', 'host', 'deployment', 'pool', 'record', 'path', 'other']) ? out : null;
-  if (event !== 'terminal_snapshot') return null;
-  for (const key of ['model_count', 'record_count', 'source_count', 'host_count', 'deployment_count', 'agent_count']) {
-    if (!Number.isInteger(input[key]) || input[key] < 0 || input[key] > 1000000000) return null;
-    out[key] = input[key];
-  }
-  return out;
-}
-// Shared, process-wide budget; no queue and at most four in-flight requests.
-let windowStart = 0, count = 0, active = 0;
-export function sendTerminalEvent(event, input) {
-  if (!telemetryEnabled()) return;
-  const properties = terminalProperties(event, input);
-  if (!properties) return;
-  if (Date.now() - windowStart >= 60000) { windowStart = Date.now(); count = 0; }
-  if (count >= 60 || active >= 4) return;
-  count++; active++;
-  void sendAnalytics(event, properties).finally(() => active--);
-}
-async function sendAnalytics(event, properties) {
-  const controller = new AbortController();
-  const body = JSON.stringify({ event, event_id: randomUUID(), anonymousId, properties: { ...properties, ...environment() } });
-  let timer;
-  try {
-    await Promise.race([
-      Promise.resolve().then(() => fetch('https://openmerit.ai/scout/api/analytics', {
-        method: 'POST', headers: { 'Content-Type': 'application/json', 'X-Scout-Client': 'cli', 'X-Scout-Analytics': 'on', ...(process.env.OPENMERIT_ANALYTICS_ACCOUNT === 'off' ? { 'X-Scout-Analytics-Account': 'off' } : {}) },
-        body, signal: controller.signal, redirect: 'error',
-      })).then(response => response.body?.cancel()),
-      new Promise(resolve => { timer = setTimeout(() => { controller.abort(); resolve(); }, 750); timer.unref(); }),
-    ]);
-  } catch { /* Never affect product behavior. */ }
-  finally { clearTimeout(timer); }
-}
-
-// Consent is independent of terminal discovery. Command identity is ephemeral;
-// the installer separately manages the owned installation's correlation IDs.
+const errorType = error => ['Error', 'TypeError', 'RangeError', 'SyntaxError', 'ReferenceError', 'URIError', 'EvalError', 'AggregateError', 'AbortError'].includes(error?.name) ? error.name : 'Error';
+// Command identity is ephemeral; the installer separately manages the owned
+// installation's correlation IDs.
 export function commandTelemetry(args) {
   const setup = ['install', 'status', 'remove', 'login', 'logout', 'project', 'doctor'];
   const enabled = () => telemetryEnabled(setup.includes(args[0]));
   if (!enabled()) return async () => {};
   const started = performance.now();
   const command_id = randomUUID();
-  const actions = ['init', 'doctor', 'plan', 'run', 'watch', 'report'];
-  const command = !args.length || args.includes('--help') || args.includes('-h') || args[0] === 'eval' && args.length === 1
-    ? 'help' : args[0] === 'dash' ? 'dash' : setup.includes(args[0]) ? args[0] : args[0] === 'eval' && actions.includes(args[1]) ? `eval_${args[1]}` : 'unknown';
+  const command = !args.length || args.includes('--help') || args.includes('-h')
+    ? 'help' : setup.includes(args[0]) ? args[0] : 'unknown';
   async function send(event, properties) {
     if (!enabled()) return;
     const controller = new AbortController();