openmerit 0.3.2 → 0.3.3

package/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 0.3.3 — 2026-11-05
+
+- Fix `openmerit install --client claude-code` exiting with `ENOENT … dist/terminal/src/updates.ts` (0.3.1 and 0.3.2). Setting up automatic Scout instruction updates copied the updater from its TypeScript source, which the package does not ship; it now copies the module that is running. The MCP server, skill and rule were already written, but automatic updates stayed off and the installer reported failure.
+- `openmerit scout --help` and the installed Scout instructions describe the new optional `kind` field for reported results (`quality`, `usd` or `seconds`), so measured cost replaces Scout's estimate, and tell the agent to report each agent's own checks at the agent level because end-to-end results also measure the agents it did not swap. Terminal output shows measured cost per 1,000 agent runs or tasks. The matching Scout serving release is deployed.
+
 ## 0.3.2 — 2026-11-05
 
 - Fix Claude Code project-scope installation for a project inside a larger git repository, such as a monorepo package. Claude Code keeps its private MCP entry under the repository root, so the installer reported "Claude Code did not save the openmerit MCP server", `status` reported it missing and a rerun clashed with it. The installer now reads the entry where Claude Code keeps it and says that the MCP server applies to the whole repository; the skill and rule stay in the project.

package/dist/pi/scout-extension.mjs

@@ -133,7 +133,11 @@ eval platform.
 Report what you measured with \`scout_models\`: the same \`conversationId\` and
 \`results\`, one aggregate per model and metric (\`level\`, \`task\` title, \`scope\`,
 \`model\` endpoint or model ID, \`metric\`, \`value\`, \`higherIsBetter\`,
-\`samples\`, \`source\`). Send aggregates only, never inputs, outputs or records.
+\`samples\`, \`source\`). Set \`kind\` to \`usd\` or \`seconds\` for measured cost
+or time (per call at the task level, per run at the agent level), so Scout uses
+your cost instead of its estimate. Report each agent's own checks at the agent
+level: an end-to-end result also measures the agents you did not swap. Send
+aggregates only, never inputs, outputs or records.
 Scout ranks them above public benchmarks for this user only; models whose
 \`quality.measured\` is true rest on them.
 

package/dist/terminal/scout-update.json

@@ -1 +1 @@
-{"format":1,"contract":"scout-recommend-v1","version":"0.3.2","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Measuring candidates\n\nEach task also has an `evaluation`: the checks (per model call, agent run or\nend-to-end outcome), labelled examples per candidate and when to stop a losing\ncandidate. Offer to run it before the user commits to a model. Use the evals the\nproject already has, such as datasets and experiments in its eval or tracing\nplatform, or its own eval scripts and fixtures: add the shortlisted candidates as\nextra runs of those evals. Do not build a second eval or copy their datasets.\nIf the project has no eval for the task, propose a minimal one built from its\nexisting traces and the decisions people already make, and create it only where\nand how the user agrees. Ask before spending on model calls or writing to an\neval platform.\n\nReport what you measured with `scout_models`: the same `conversationId` and\n`results`, one aggregate per model and metric (`level`, `task` title, `scope`,\n`model` endpoint or model ID, `metric`, `value`, `higherIsBetter`,\n`samples`, `source`). Send aggregates only, never inputs, outputs or records.\nScout ranks them above public benchmarks for this user only; models whose\n`quality.measured` is true rest on them.\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. When an install or login waits for approval (exit code\n10), show the user the approval link and its UTC expiry in your reply, not only in\ncommand output, then rerun the same command once they approve. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}
+{"format":1,"contract":"scout-recommend-v1","version":"0.3.3","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Measuring candidates\n\nEach task also has an `evaluation`: the checks (per model call, agent run or\nend-to-end outcome), labelled examples per candidate and when to stop a losing\ncandidate. Offer to run it before the user commits to a model. Use the evals the\nproject already has, such as datasets and experiments in its eval or tracing\nplatform, or its own eval scripts and fixtures: add the shortlisted candidates as\nextra runs of those evals. Do not build a second eval or copy their datasets.\nIf the project has no eval for the task, propose a minimal one built from its\nexisting traces and the decisions people already make, and create it only where\nand how the user agrees. Ask before spending on model calls or writing to an\neval platform.\n\nReport what you measured with `scout_models`: the same `conversationId` and\n`results`, one aggregate per model and metric (`level`, `task` title, `scope`,\n`model` endpoint or model ID, `metric`, `value`, `higherIsBetter`,\n`samples`, `source`). Set `kind` to `usd` or `seconds` for measured cost\nor time (per call at the task level, per run at the agent level), so Scout uses\nyour cost instead of its estimate. Report each agent's own checks at the agent\nlevel: an end-to-end result also measures the agents you did not swap. Send\naggregates only, never inputs, outputs or records.\nScout ranks them above public benchmarks for this user only; models whose\n`quality.measured` is true rest on them.\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. When an install or login waits for approval (exit code\n10), show the user the approval link and its UTC expiry in your reply, not only in\ncommand output, then rerun the same command once they approve. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}

package/dist/terminal/src/scout-instructions.js

@@ -99,7 +99,11 @@ eval platform.
 Report what you measured with \`scout_models\`: the same \`conversationId\` and
 \`results\`, one aggregate per model and metric (\`level\`, \`task\` title, \`scope\`,
 \`model\` endpoint or model ID, \`metric\`, \`value\`, \`higherIsBetter\`,
-\`samples\`, \`source\`). Send aggregates only, never inputs, outputs or records.
+\`samples\`, \`source\`). Set \`kind\` to \`usd\` or \`seconds\` for measured cost
+or time (per call at the task level, per run at the agent level), so Scout uses
+your cost instead of its estimate. Report each agent's own checks at the agent
+level: an end-to-end result also measures the agents you did not swap. Send
+aggregates only, never inputs, outputs or records.
 Scout ranks them above public benchmarks for this user only; models whose
 \`quality.measured\` is true rest on them.
 

package/dist/terminal/src/scout.js

@@ -19,9 +19,10 @@ work, output, criteria, constraints) in the same conversation, or --skip to let
 Scout assume. It then splits the work into tasks and recommends up to five
 models for each, with an evaluation: how to measure the candidates with your
 own evals. Report what you measured with --results: a JSON list of aggregate
-results ({level, task?, scope, model, metric, value, higherIsBetter, samples,
-source}), never inputs or outputs. Scout ranks them above public benchmarks, for
-your recommendations only. A turn usually takes one to three minutes.
+results ({level, kind?, task?, scope, model, metric, value, higherIsBetter,
+samples, source}), never inputs or outputs. kind is quality (the default), usd
+or seconds; measured usd replaces Scout's cost estimate. Scout ranks them above
+public benchmarks, for your recommendations only. A turn usually takes one to three minutes.
 
 A new conversation is limited to the endpoint IDs in ${policyFile}
 (set with openmerit project). --all-endpoints searches the whole catalogue and is
@@ -125,7 +126,8 @@ export function renderScout(result) {
             if (priority !== 'cost')
                 lines.push(`   ${rankings[priority]}`);
             for (const m of task.models) {
-                const price = m.cost ? `${money(m.cost.inputPerMillion)} in / ${money(m.cost.outputPerMillion)} out per 1M via ${m.cost.route} (${m.cost.endpointId})` : 'no priced route';
+                const measuredCost = m.cost?.measured ? `measured ${money(m.cost.measured.usdPer1kAgentRuns ?? m.cost.measured.usdPer1kTasks)} per 1k ${m.cost.measured.usdPer1kAgentRuns === undefined ? 'tasks' : 'agent runs'}, ` : '';
+                const price = m.cost ? `${measuredCost}${money(m.cost.inputPerMillion)} in / ${money(m.cost.outputPerMillion)} out per 1M via ${m.cost.route} (${m.cost.endpointId})` : 'no priced route';
                 const metric = priority === 'latency' ? ` · ${m.latency ? `${m.latency.seconds.toFixed(1)}s per task` : 'time unknown'}`
                     : priority === 'recency' ? ` · ${m.recency ? `released ${m.recency.released}` : 'release date unknown'}` : '';
                 lines.push(`   ${m.rank}. ${m.name}${metric} · quality ${m.quality.score}/100${m.quality.measured ? ' (measured)' : ''} · ${price}`, `      ${m.rationale}`);

package/dist/terminal/src/updates.js

@@ -166,7 +166,8 @@ export async function configureUpdates(base, scope, record, version, enabled) {
     const state = await inspectUpdates(base, scope, record);
     if (state.conflicts.length)
         throw new Error(state.conflicts.join('\n'));
-    const source = await readFile(new URL('./updates.ts', import.meta.url), 'utf8');
+    // This module is the runner: updates.ts in source, the compiled updates.js in the npm package.
+    const source = await readFile(new URL(import.meta.url), 'utf8');
     const hook = { hooks: [{ type: 'command', command: `${quote(process.execPath)} ${quote(state.paths.runner)} --base ${quote(base)} --scope ${scope} --hook`, async: true, timeout: 150 }] };
     const hooks = { ...state.hooks };
     for (const event of ['SessionStart', 'Stop']) {

package/docs/scout.md

@@ -27,6 +27,11 @@ OpenMerit does not run evaluations or host an eval platform. Scout tells you wha
 - **Agents and calls.** A task is still one model call. When the work is an agent that calls models in a loop, a task can name its `agent` and `callsPerRun`; each model's `cost` then also has `usdPer1kAgentRuns`, the same route's cost of 1,000 runs of that agent.
 - **Evaluation plan.** Every task in a recommendation has an `evaluation`: `checks` at the `task` (one model call), `agent` (one agent run) or `environment` (end-to-end outcome) level, labelled `samples` per candidate, and a `stopEarly` rule for a candidate that is clearly losing. The installed instructions tell the coding agent to add the candidates as extra runs of evals the project already has, such as datasets and experiments in its eval or tracing platform, or its own eval scripts, and not to build a second eval. If none exists, it proposes a minimal one and creates it only where and how you agree.
 - **Measured results.** Continue the conversation with `results` (`scout_models`, `POST /scout/api/agent`) or `openmerit scout --conversation <id> --results <file.json>`. Each result is one aggregate: `level`, optional `task` title, `scope`, `model` (catalogue model or endpoint ID), `metric`, `value`, `higherIsBetter`, `samples` and `source`, at most 50 per call and 200 per conversation. Inputs, outputs and records are not accepted. Scout recommends again with measured results ranked above public benchmarks: a candidate you measured on a task must cite that result, cited results appear as evidence named "Your eval", ranked among your results on the same metric, and `quality.measured` is true for models whose estimate rests on them. Since the Scout serving deploy of 5 November 2026 (version `8782d345-a63c-4560-bb08-1a6a95e8c0ae`), models whose estimate rests on your results also rank ahead of unmeasured ones, in the conversation's priority order within each group (cheapest first by default), so a model estimated only from benchmarks no longer leads a shortlist you measured. Before it, measured results could only move a model below the bar.
+
+**Since the Scout serving deploy of 5 November 2026 (version `ecd0bd6b-83a9-4e8d-9fd8-d5ca6de88271`).** npm 0.3.3 documents the field in `openmerit scout --help` and the installed instructions and shows measured cost in terminal output; `--results` in 0.3.2 already forwards it.
+
+- **Measured cost.** A result can set `kind` to `usd` or `seconds` (default `quality`): cost or time per model call at the task level, per agent run at the agent level, per outcome at the environment level. Cost and time results are never quality evidence. A model's agent- or task-level `usd` result on a task appears as `cost.measured` (`usdPer1kTasks`, `usdPer1kAgentRuns`, and the `result` it came from), and when every model in a group (measured or unmeasured) has one, the group is ranked by measured cost instead of Scout's catalogue estimate, which can be far off for agent runs.
+- **Environment results and attribution.** An environment-level result also measures every other agent and control. Scout no longer drops a candidate below the bar when its own task- and agent-level quality results on that task match a candidate that reaches it; an environment miss alone does not show the swapped model caused it.
 - **Scope.** Results are stored with the conversation for your account and inform only your recommendations. They are not shared with other accounts or used to change Scout's catalogue.
 
 The quality bar now considers what an error costs once your environment's controls, such as a person approving every payment, have acted. Cost remains the default ranking among models that reach the bar.

package/docs/troubleshooting.md

@@ -11,7 +11,7 @@ npx -y openmerit@latest status --client pi --scope user --check
 npx -y openmerit@latest doctor --client pi --scope user
 ```
 
-Use `--scope project` from the original project root for project setup. In a project inside a larger git repository, Claude Code project setup up to npm 0.3.1 ends with "Claude Code did not save the openmerit MCP server" although Claude Code saved it under the repository root; Scout still works in that repository, and npm 0.3.2 reads the entry correctly; rerun the install with it. Check Pi and Node.js versions against [setup](getting-started.md), complete browser device approval, then start a **new session**. Installing the npm package or reconnecting an old session does not load new extension instructions. For SDK embedding, load Pi's `createMcpExtension()` explicitly alongside `createScoutPiExtension`; the Scout factory alone is not an MCP runtime.
+Use `--scope project` from the original project root for project setup. In a project inside a larger git repository, Claude Code project setup up to npm 0.3.1 ends with "Claude Code did not save the openmerit MCP server" although Claude Code saved it under the repository root; Scout still works in that repository, and npm 0.3.2 reads the entry correctly; rerun the install with it. npm 0.3.1 and 0.3.2 also end every Claude Code install with `ENOENT … dist/terminal/src/updates.ts`: the MCP server, skill and rule are installed, but automatic instruction updates stay off. Rerun the install with npm 0.3.3 or later. Check Pi and Node.js versions against [setup](getting-started.md), complete browser device approval, then start a **new session**. Installing the npm package or reconnecting an old session does not load new extension instructions. For SDK embedding, load Pi's `createMcpExtension()` explicitly alongside `createScoutPiExtension`; the Scout factory alone is not an MCP runtime.
 
 ## The connection configuration is invalid
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.3.2",
+  "version": "0.3.3",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {