openmerit 0.3.0 → 0.3.1

package/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 0.3.1 — 2026-11-05
+
+- `openmerit scout --conversation <id> --results <file.json>` reports aggregate results you measured with your own evals. Scout then ranks them above public benchmarks for your recommendations only. Terminal output shows each task's evaluation plan, its agent and calls per run, and marks quality that rests on measured results.
+- The installed Scout instructions for Claude Code and Pi tell the agent to measure shortlisted candidates as extra runs of the project's existing evals, never a second eval, and to report aggregates only. This needs the matching hosted Scout release, whose `scout_models` tool returns `tasks[].evaluation` and accepts `results`.
+
 ## 0.3.0 — 2026-11-04
 
 - **Breaking:** retire the standalone `openmerit eval` CLI, including `init`, `doctor`, `plan`, `run`, `watch` and `report`. Existing local evaluation files are not deleted. The hosted `/scout/api/recommend` retirement ships separately with the Scout serving Worker.

package/README.md

@@ -24,7 +24,7 @@ Package publication and website deployment are separate releases. Pi setup requi
 
 ## Evaluation CLI retirement — npm 0.3.0
 
-OpenMerit 0.3.0 removes the standalone `openmerit eval` commands, including planning. Existing local evaluation files are not deleted. The old Scout JSON recommendation API is retired in source and goes offline with a separate Scout deployment. Scout's conversational agent, MCP, coding-agent setup and catalogue publication remain available; see [retirement details](docs/agent-evaluations.md).
+OpenMerit 0.3.0 removes the standalone `openmerit eval` commands, including planning. Existing local evaluation files are not deleted. The old Scout JSON recommendation API went offline with the Scout serving deploy of 5 November 2026. Scout's conversational agent, MCP, coding-agent setup and catalogue publication remain available; see [retirement details](docs/agent-evaluations.md).
 
 ## Inference Terminal retirement — npm 0.3.0
 

package/dist/pi/scout-extension.mjs

@@ -117,6 +117,26 @@ Compare quality evidence, reference cost/time and watch-outs. Prefer the cheapes
 usable option the evidence supports, without claiming benchmarks prove
 application accuracy. Use the project's actual SDK and protocol.
 
+## Measuring candidates
+
+Each task also has an \`evaluation\`: the checks (per model call, agent run or
+end-to-end outcome), labelled examples per candidate and when to stop a losing
+candidate. Offer to run it before the user commits to a model. Use the evals the
+project already has, such as datasets and experiments in its eval or tracing
+platform, or its own eval scripts and fixtures: add the shortlisted candidates as
+extra runs of those evals. Do not build a second eval or copy their datasets.
+If the project has no eval for the task, propose a minimal one built from its
+existing traces and the decisions people already make, and create it only where
+and how the user agrees. Ask before spending on model calls or writing to an
+eval platform.
+
+Report what you measured with \`scout_models\`: the same \`conversationId\` and
+\`results\`, one aggregate per model and metric (\`level\`, \`task\` title, \`scope\`,
+\`model\` endpoint or model ID, \`metric\`, \`value\`, \`higherIsBetter\`,
+\`samples\`, \`source\`). Send aggregates only, never inputs, outputs or records.
+Scout ranks them above public benchmarks for this user only; models whose
+\`quality.measured\` is true rest on them.
+
 ## Boundaries
 
 Ask before adding providers or credentials, spending, deploying or external

package/dist/terminal/scout-update.json

@@ -1 +1 @@
-{"format":1,"contract":"scout-recommend-v1","version":"0.3.0","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}
+{"format":1,"contract":"scout-recommend-v1","version":"0.3.1","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Measuring candidates\n\nEach task also has an `evaluation`: the checks (per model call, agent run or\nend-to-end outcome), labelled examples per candidate and when to stop a losing\ncandidate. Offer to run it before the user commits to a model. Use the evals the\nproject already has, such as datasets and experiments in its eval or tracing\nplatform, or its own eval scripts and fixtures: add the shortlisted candidates as\nextra runs of those evals. Do not build a second eval or copy their datasets.\nIf the project has no eval for the task, propose a minimal one built from its\nexisting traces and the decisions people already make, and create it only where\nand how the user agrees. Ask before spending on model calls or writing to an\neval platform.\n\nReport what you measured with `scout_models`: the same `conversationId` and\n`results`, one aggregate per model and metric (`level`, `task` title, `scope`,\n`model` endpoint or model ID, `metric`, `value`, `higherIsBetter`,\n`samples`, `source`). Send aggregates only, never inputs, outputs or records.\nScout ranks them above public benchmarks for this user only; models whose\n`quality.measured` is true rest on them.\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}

package/dist/terminal/src/scout-instructions.js

@@ -83,6 +83,26 @@ Compare quality evidence, reference cost/time and watch-outs. Prefer the cheapes
 usable option the evidence supports, without claiming benchmarks prove
 application accuracy. Use the project's actual SDK and protocol.
 
+## Measuring candidates
+
+Each task also has an \`evaluation\`: the checks (per model call, agent run or
+end-to-end outcome), labelled examples per candidate and when to stop a losing
+candidate. Offer to run it before the user commits to a model. Use the evals the
+project already has, such as datasets and experiments in its eval or tracing
+platform, or its own eval scripts and fixtures: add the shortlisted candidates as
+extra runs of those evals. Do not build a second eval or copy their datasets.
+If the project has no eval for the task, propose a minimal one built from its
+existing traces and the decisions people already make, and create it only where
+and how the user agrees. Ask before spending on model calls or writing to an
+eval platform.
+
+Report what you measured with \`scout_models\`: the same \`conversationId\` and
+\`results\`, one aggregate per model and metric (\`level\`, \`task\` title, \`scope\`,
+\`model\` endpoint or model ID, \`metric\`, \`value\`, \`higherIsBetter\`,
+\`samples\`, \`source\`). Send aggregates only, never inputs, outputs or records.
+Scout ranks them above public benchmarks for this user only; models whose
+\`quality.measured\` is true rest on them.
+
 ## Boundaries
 
 Ask before adding providers or credentials, spending, deploying or external

package/dist/terminal/src/scout.js

@@ -3,7 +3,7 @@
 // the same scout_models MCP tool with this device's key, so every client gets the
 // same clarifying questions and per-task shortlists.
 import { readFile } from 'node:fs/promises';
-import { join } from 'node:path';
+import { join, resolve } from 'node:path';
 import { credentialsPath, readCredentials } from "./scout-credentials.js";
 const policyFile = '.openmerit/scout.json';
 export const components = ['input', 'work', 'output', 'criteria', 'constraints'];
@@ -11,12 +11,17 @@ const help = `OpenMerit · Scout
 
   openmerit scout "<the work models need to do>" [--all-endpoints] [--json]
   openmerit scout --conversation <id> [--brief <component>=<text> ...] [--skip] ["<follow-up>"] [--json]
+  openmerit scout --conversation <id> --results <file.json> [--json]
 
 Asks Scout, the agent behind openmerit.ai/scout. Scout may first return a task
 brief with clarifying questions: answer them with --brief (components: input,
 work, output, criteria, constraints) in the same conversation, or --skip to let
 Scout assume. It then splits the work into tasks and recommends up to five
-models for each. A turn usually takes one to three minutes.
+models for each, with an evaluation: how to measure the candidates with your
+own evals. Report what you measured with --results: a JSON list of aggregate
+results ({level, task?, scope, model, metric, value, higherIsBetter, samples,
+source}), never inputs or outputs. Scout ranks them above public benchmarks, for
+your recommendations only. A turn usually takes one to three minutes.
 
 A new conversation is limited to the endpoint IDs in ${policyFile}
 (set with openmerit project). --all-endpoints searches the whole catalogue and is
@@ -114,7 +119,7 @@ export function renderScout(result) {
         const tasks = result.tasks ?? [];
         tasks.forEach((task, i) => {
             if (tasks.length > 1)
-                lines.push(`${i + 1}. ${task.title}`, `   ${task.work}`);
+                lines.push(`${i + 1}. ${task.title}${task.agent ? ` (${task.agent}${task.callsPerRun && task.callsPerRun > 1 ? `, ${task.callsPerRun} calls per run` : ''})` : ''}`, `   ${task.work}`);
             // As on the page, the default cheapest-first order is unlabelled; another priority is named with the value it sorts by.
             const priority = task.priority ?? 'cost';
             if (priority !== 'cost')
@@ -123,14 +128,20 @@ export function renderScout(result) {
                 const price = m.cost ? `${money(m.cost.inputPerMillion)} in / ${money(m.cost.outputPerMillion)} out per 1M via ${m.cost.route} (${m.cost.endpointId})` : 'no priced route';
                 const metric = priority === 'latency' ? ` · ${m.latency ? `${m.latency.seconds.toFixed(1)}s per task` : 'time unknown'}`
                     : priority === 'recency' ? ` · ${m.recency ? `released ${m.recency.released}` : 'release date unknown'}` : '';
-                lines.push(`   ${m.rank}. ${m.name}${metric} · quality ${m.quality.score}/100 · ${price}`, `      ${m.rationale}`);
+                lines.push(`   ${m.rank}. ${m.name}${metric} · quality ${m.quality.score}/100${m.quality.measured ? ' (measured)' : ''} · ${price}`, `      ${m.rationale}`);
             }
-            lines.push(`   ${task.summary}`, '');
+            lines.push(`   ${task.summary}`);
+            if (task.evaluation)
+                lines.push(`   Measure with your evals (${task.evaluation.samples} labelled examples per candidate; ${task.evaluation.stopEarly}):`, ...task.evaluation.checks.map(c => `     - ${c.level}: ${c.metric}. ${c.how}`));
+            lines.push('');
         });
         if (result.assumptions?.length)
             lines.push('Assumptions:', ...result.assumptions.map(a => `  - ${a}`), '');
         lines.push(`Refine: openmerit scout --conversation ${result.conversationId} "<follow-up>"`);
-        lines.push('Quality is Scout\'s estimate from public benchmarks, not measured accuracy for your application.');
+        lines.push(`Report measured results: openmerit scout --conversation ${result.conversationId} --results <file.json>`);
+        lines.push(tasks.some(t => t.models.some(m => m.quality.measured))
+            ? 'Quality marked (measured) rests on the results you reported; the rest is Scout\'s estimate from public benchmarks.'
+            : 'Quality is Scout\'s estimate from public benchmarks, not measured accuracy for your application.');
     }
     else if (result.status === 'working')
         lines.push(`Scout is still working. Read the outcome with: openmerit scout --conversation ${result.conversationId}`);
@@ -164,6 +175,19 @@ export async function main(args, { root = process.cwd(), print = console.log } =
             request.allEndpoints = true;
         else if (arg === '--json')
             json = true;
+        else if (arg === '--results') {
+            const file = value();
+            let results;
+            try {
+                results = JSON.parse(await readFile(resolve(root, file), 'utf8'));
+            }
+            catch {
+                throw new Error(`Could not read results from ${file}: it must be a JSON list.`);
+            }
+            if (!Array.isArray(results))
+                throw new Error(`${file} must hold a JSON list of results.`);
+            request.results = results;
+        }
         else if (arg.startsWith('-'))
             throw new Error(`Unknown option for scout: ${arg}`);
         else
@@ -175,6 +199,8 @@ export async function main(args, { root = process.cwd(), print = console.log } =
         print(help);
         return 1;
     }
+    if (request.results && !request.conversationId)
+        throw new Error('--results continues a conversation: pass --conversation <id> from the recommendation you measured.');
     const result = await scoutTurn(root, request);
     print(json ? JSON.stringify(result, null, 2) : renderScout(result));
     return result.status === 'error' ? 1 : 0;

package/docs/agent-evaluations.md

@@ -1,6 +1,6 @@
 # Agent-evaluation CLI retirement
 
-**npm 0.3.0, 4 November 2026.** The standalone `openmerit eval` CLI is retired in full, including `init`, `doctor`, `plan`, `run`, `watch` and `report`. Older installed packages keep it until they are upgraded. The old Scout JSON recommendation API is retired in unreleased source and goes offline only with a separate Scout deployment.
+**npm 0.3.0, 4 November 2026.** The standalone `openmerit eval` CLI is retired in full, including `init`, `doctor`, `plan`, `run`, `watch` and `report`. Older installed packages keep it until they are upgraded. The old Scout JSON recommendation API went offline with the Scout serving deploy of 5 November 2026.
 
 After an approved Scout release, `/scout/api/recommend` and its legacy alpha/versioned paths return 404, regardless of authentication. There is no replacement frozen-plan protocol. Use the [Scout conversational agent or MCP](scout.md) for provisional model guidance and your own representative task checks for evaluation. The Inference Terminal is [retired as well](inference-terminal.md); the Pi adapter remains a native Scout MCP integration, not an evaluation runner.
 

package/docs/commands.md

@@ -4,6 +4,6 @@
 
 The Pi 0.87 `/openmerit` slash commands were removed with that adapter and have no replacement.
 
-**Retired in npm 0.3.0:** `openmerit dash` ([details](inference-terminal.md)) and all `openmerit eval` commands, including `plan`. Existing local files are untouched. Unreleased Scout source also retires `/scout/api/recommend`; use Scout's conversational agent or MCP. Hosted Workers change only through separate approved deployments. See [retirement details](agent-evaluations.md).
+**Retired in npm 0.3.0:** `openmerit dash` ([details](inference-terminal.md)) and all `openmerit eval` commands, including `plan`. Existing local files are untouched. The Scout serving deploy of 5 November 2026 retired `/scout/api/recommend`; use Scout's conversational agent or MCP. Hosted Workers change only through separate approved deployments. See [retirement details](agent-evaluations.md).
 
 The [Scout setup commands](scout.md#commands) remain available, beginning with `npx -y openmerit@latest install --client claude-code`.

package/docs/getting-started.md

@@ -34,4 +34,4 @@ See [the Pi extension guide](pi-extension.md) for installed paths, migration fro
 
 ## Evaluate application changes
 
-The standalone evaluation CLI is retired in npm 0.3.0, and the legacy JSON recommendation API in unreleased Scout source. Scout guidance is not measured application quality; use your own task checks and provider budget controls. See [retirement details](agent-evaluations.md).
+The standalone evaluation CLI is retired in npm 0.3.0, and the legacy JSON recommendation API went offline with the Scout serving deploy of 5 November 2026. Scout guidance is not measured application quality; use your own task checks and provider budget controls. See [retirement details](agent-evaluations.md).

package/docs/index.md

@@ -1,6 +1,6 @@
 # OpenMerit documentation
 
-These maintained Markdown guides distinguish current source work from published behavior. OpenMerit **0.1.12** added native Scout support for Pi **1.5.0**, replacing the old Pi 0.87 evaluation extension. OpenMerit **0.3.0** retires the Inference Terminal (`openmerit dash`) and standalone evaluation CLI. Unreleased source also retires the legacy recommendation API and the unpublished docs website shell. The policy, protected memo, Scout service and npm package are deployed independently; dated receipts describe the versions actually tested.
+These maintained Markdown guides distinguish current source work from published behavior. OpenMerit **0.1.12** added native Scout support for Pi **1.5.0**, replacing the old Pi 0.87 evaluation extension. OpenMerit **0.3.0** retires the Inference Terminal (`openmerit dash`) and standalone evaluation CLI. The Scout serving deploy of 5 November 2026 retired the legacy recommendation API; unreleased source also retires the unpublished docs website shell. The policy, protected memo, Scout service and npm package are deployed independently; dated receipts describe the versions actually tested.
 
 ## Discover model candidates
 

package/docs/roadmap.md

@@ -2,7 +2,7 @@
 
 ## Current status
 
-The Pi **1.5.0** adapter in OpenMerit **0.1.12** provides only native Scout MCP and shared guidance, not evaluation coordination. OpenMerit **0.3.0** retires the Inference Terminal and the standalone evaluation CLI; unreleased Scout source retires the legacy recommendation API. Passing implementation tests do not establish that an application has better models, trustworthy evaluations, or reliable unattended automation.
+The Pi **1.5.0** adapter in OpenMerit **0.1.12** provides only native Scout MCP and shared guidance, not evaluation coordination. OpenMerit **0.3.0** retires the Inference Terminal and the standalone evaluation CLI; the Scout serving deploy of 5 November 2026 retired the legacy recommendation API. Passing implementation tests do not establish that an application has better models, trustworthy evaluations, or reliable unattended automation.
 
 ## Known limitations
 

package/docs/scout-data-policy.md

@@ -1,6 +1,6 @@
 # Scout data policy
 
-Updated 29 October 2026.
+Updated 5 November 2026.
 
 OpenMerit operates Scout. This policy covers its hosted website and connected clients.
 
@@ -9,6 +9,7 @@ OpenMerit operates Scout. This policy covers its hosted website and connected cl
 ## Data we use
 
 - **Service data:** Your submitted task and related information are processed by an AI service provider, even with analytics off. We also retain account, authorization and device details needed to provide access.
+- **Measured results:** Aggregate evaluation results you report on a Scout conversation, such as a model's score on your own eval, are stored with that conversation and used only for its recommendations. Send aggregates only, never the inputs, outputs or records they were measured on.
 - **Optional analytics:** When enabled, we collect usage, recommendations, feedback, setup activity, device/browser categories, performance and error information to improve Scout and fix problems. Account analytics may include your name, verified email and sign-in provider, and may link activity before and after sign-in.
 - **Shared content:** When prompt sharing is enabled, analytics also includes task text and analysis. We use this for product improvement, not foundation model training. There is no session recording.
 

package/docs/scout.md

@@ -2,7 +2,7 @@
 
 Scout is OpenMerit's model search agent. Describe your task and get provisional model guidance through the conversational page or MCP. Public catalogue evidence is not proof of application quality.
 
-**Unreleased retirement, November 2026:** this source removes `/scout/api/recommend`, its alpha/versioned aliases and the complete standalone `openmerit eval` CLI, including `plan`. Retired paths return 404 before authentication, redirects or asset fallback. The agent, canonical `/scout/mcp`, coding-agent setup, catalogue publisher, status and account/privacy capabilities remain. No remote data or existing local evaluation files are deleted. Package publication and Worker deployment require separate approval; neither is implied by this source change.
+**Retirement, November 2026:** npm 0.3.0 removed the complete standalone `openmerit eval` CLI, including `plan`. The Scout serving deploy of 5 November 2026 removed `/scout/api/recommend` and its alpha/versioned aliases; retired paths return 404 before authentication, redirects or asset fallback. The agent, canonical `/scout/mcp`, coding-agent setup, catalogue publisher, status and account/privacy capabilities remain. No remote data or existing local evaluation files are deleted.
 
 Dated receipts and the old search/ranking/JSON protocol descriptions below are historical, not supported interfaces in this source. There is no replacement frozen-plan API. See [evaluation retirement](agent-evaluations.md) and [commands](commands.md) for migration boundaries.
 
@@ -18,6 +18,21 @@ Scout is a separate hosted beta product. Npm 0.1.7 introduced optional Claude Co
 
 **Product wording — unreleased source:** the homepage, Scout page, onboarding and search/social descriptions call Scout “OpenMerit's model search agent,” with “model search” as the shorthand and “Find the right AI models for your task” as its promise. “Model recommendations” describes its output; the web and coding agents are access channels, not separate product categories. This copy is not yet deployed. Dated receipts below retain the wording and behavior of their original releases.
 
+## Evaluation plans and measured results — deployed 5 November 2026, npm 0.3.1
+
+The Scout serving deploy of 5 November 2026 (version `ea371c78-6ccf-4931-a0b8-aa28a6599aaa`) serves this behavior to every client. A live MCP check returned an evaluation plan and agent grouping, and reported results moved a measured model below the quality bar. npm 0.3.1 adds `openmerit scout --results` and the installed instructions for measuring; older installations reach the new `scout_models` fields through MCP but their instructions do not mention measuring until the installer is rerun.
+
+OpenMerit does not run evaluations or host an eval platform. Scout tells you what to measure, your own eval system measures it, and Scout uses the aggregate results:
+
+- **Agents and calls.** A task is still one model call. When the work is an agent that calls models in a loop, a task can name its `agent` and `callsPerRun`; each model's `cost` then also has `usdPer1kAgentRuns`, the same route's cost of 1,000 runs of that agent.
+- **Evaluation plan.** Every task in a recommendation has an `evaluation`: `checks` at the `task` (one model call), `agent` (one agent run) or `environment` (end-to-end outcome) level, labelled `samples` per candidate, and a `stopEarly` rule for a candidate that is clearly losing. The installed instructions tell the coding agent to add the candidates as extra runs of evals the project already has, such as datasets and experiments in its eval or tracing platform, or its own eval scripts, and not to build a second eval. If none exists, it proposes a minimal one and creates it only where and how you agree.
+- **Measured results.** Continue the conversation with `results` (`scout_models`, `POST /scout/api/agent`) or `openmerit scout --conversation <id> --results <file.json>`. Each result is one aggregate: `level`, optional `task` title, `scope`, `model` (catalogue model or endpoint ID), `metric`, `value`, `higherIsBetter`, `samples` and `source`, at most 50 per call and 200 per conversation. Inputs, outputs and records are not accepted. Scout recommends again with measured results ranked above public benchmarks: a candidate you measured on a task must cite that result, cited results appear as evidence named "Your eval", ranked among your results on the same metric, and `quality.measured` is true for models whose estimate rests on them.
+- **Scope.** Results are stored with the conversation for your account and inform only your recommendations. They are not shared with other accounts or used to change Scout's catalogue.
+
+The quality bar now considers what an error costs once your environment's controls, such as a person approving every payment, have acted. Cost remains the default ranking among models that reach the bar.
+
+Known limitations: Scout does not connect to eval platforms yet, so results arrive only through the coding agent or the CLI; the web page neither shows the evaluation plan nor accepts results; and Scout cannot verify a reported result.
+
 ## Five-step introduction modal — deployed, 28 October 2026
 
 **Deployment receipt:** the owner approved commit, push and deployment. [PR #28](https://github.com/laz-aslam/openmerit/pull/28) merged as [a70103a](https://github.com/laz-aslam/openmerit/commit/a70103a1c8eb8a701ee262106ae6e6c026cf04b4). All PR and merged-commit CI checks passed. The guarded Scout-serving deploy moved `openmerit-scout-alpha` from `2cf2c2d3-8aef-40f0-9c44-800bcab247bf` to `17ab0166-9a5c-43e9-932b-c13bee040db7` at 20:45 UTC. Live Scout HTML, chat/auth/onboarding JavaScript, styles and Claude screenshot matched the release build. `/` and `/connect/` stayed byte-identical; other Workers were not redeployed. The catalogue remained healthy on snapshot `588855a0bf28212fad48a9513dbf3d5b`.

package/docs/security.md

@@ -2,7 +2,7 @@
 
 **Pi adapter — npm 0.1.12 and later:** Pi 1.5.0 uses native MCP plus shared Scout rule/skill sections, not the former evaluation adapter. It never initiates login, changes the host model/tools/session, or provisions background jobs. Browser-approved credentials are independent per client/scope/project, stored privately outside the repository, and passed as MCP HTTP headers, never instructions. Pi's installed connection JSON contains only the private credential path. See [Pi paths and targeted removal](pi-extension.md). The standalone evaluation CLI remains separate.
 
-**Retirement:** npm 0.3.0 removes the standalone [agent-evaluation runner](agent-evaluations.md), including planning; unreleased Scout source removes `/scout/api/recommend`. Existing local reports, budgets, configuration and host-agent instructions are not deleted; review them separately. No provider billing control or remote data deletion is implied.
+**Retirement:** npm 0.3.0 removes the standalone [agent-evaluation runner](agent-evaluations.md), including planning; the Scout serving deploy of 5 November 2026 removed `/scout/api/recommend`. Existing local reports, budgets, configuration and host-agent instructions are not deleted; review them separately. No provider billing control or remote data deletion is implied.
 
 The Scout agent, MCP, publisher and account analytics/deletion capabilities remain. Historical analytics and stored catalogue/research data are not erased by endpoint retirement. Necessary provider processing for the agent and publisher continues under the [Scout data policy](scout-data-policy.md).
 

package/docs/testing.md

@@ -1,6 +1,6 @@
 # Testing and validation
 
-**Cleanup:** npm 0.3.0 retires the standalone evaluation CLI and the Inference Terminal; unreleased source also retires the legacy JSON recommendation pipeline and unpublished docs website shell. Dated evaluation/API acceptance records below describe historical releases, not current supported commands. Current verification retains agent, MCP, publisher, status, policy and protected-memo coverage; it does not publish or deploy anything.
+**Cleanup:** npm 0.3.0 retires the standalone evaluation CLI and the Inference Terminal; the Scout serving deploy of 5 November 2026 retired the legacy JSON recommendation pipeline, and unreleased source retires the unpublished docs website shell. Dated evaluation/API acceptance records below describe historical releases, not current supported commands. Current verification retains agent, MCP, publisher, status, policy and protected-memo coverage; it does not publish or deploy anything.
 
 **Pi adapter — npm 0.1.12 and later:** the Pi 1.5.0 adapter uses native Scout MCP and shared guidance only. Dated acceptance records below describe the packages tested at the time; some mention package exports that have since been removed. Standalone evaluation tests remain separate.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.3.0",
+  "version": "0.3.1",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {