openmerit 0.3.6 → 0.3.7

package/CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.3.7 — 2026-11-06 (release preparation)
+
+- The installed Scout skill distinguishes no eval setup, insufficient evals and suitable existing evals. Langfuse is the default customer-owned platform suggestion, not a requirement; local/custom evals can report aggregates. The customer's coding agent implements evals, while OpenMerit remains Viewer-only. Instructions require a local Markdown decision record before proposing explicitly approved cleanup of newly created comparison runs; existing records and reusable fixtures/definitions stay intact. This adds guidance, not an eval runner, automatic deletion or another platform connector. Fresh-agent acceptance remains a release gate.
+
 ## 0.3.6 — 2026-11-06
 
 - Distinguish a connected Langfuse key from incomplete score-import setup. Connection status exposes only score-mapping and model-alias counts, and the web UI explains when either is missing. The CLI preserves those counts when the server provides them. Requires the corresponding Scout deployment and npm patch; does not change credentials, mappings or experiment attribution.

package/dist/pi/scout-extension.mjs

@@ -121,14 +121,27 @@ application accuracy. Use the project's actual SDK and protocol.
 
 Each task also has an \`evaluation\`: the checks (per model call, agent run or
 end-to-end outcome), labelled examples per candidate and when to stop a losing
-candidate. Offer to run it before the user commits to a model. Use the evals the
-project already has, such as datasets and experiments in its eval or tracing
-platform, or its own eval scripts and fixtures: add the shortlisted candidates as
-extra runs of those evals. Do not build a second eval or copy their datasets.
-If the project has no eval for the task, propose a minimal one built from its
-existing traces and the decisions people already make, and create it only where
-and how the user agrees. Ask before spending on model calls or writing to an
-eval platform.
+candidate. Offer to run it before the user commits to a model. First assess the
+project's readiness for this task:
+
+- **No eval setup:** propose a customer-owned setup. Langfuse is the default
+  suggestion for its supported aggregate connector, not a requirement: existing
+  local/custom eval scripts can report aggregates without a hosted platform.
+  Phoenix/Arize and Braintrust are alternatives, not supported Scout connectors.
+  Ask which system the user wants before creating accounts or resources.
+- **Setup exists but evidence is insufficient:** keep that system. Write a
+  task-specific eval plan in the project's local Markdown decision record for
+  the user or their coding agent to implement. Name exact candidates/endpoints,
+  quality floor and task/agent/outcome checks, labelled boundary cases and
+  controls, sample size, early-stop rule, spend cap, aggregate mappings and
+  temporary run retention. Explain what the current evidence cannot decide.
+- **Suitable evals exist:** reuse their fixtures and definitions; add only the
+  approved candidate runs. Do not build a second eval or copy their datasets.
+
+OpenMerit does not host or run these evals. The customer or their coding agent
+implements and maintains them with the customer's tools; Scout keeps Viewer-only
+access. Ask before model spend or any platform write. Without measured evidence,
+Scout may suggest candidates, but do not claim validated model quality.
 
 Report what you measured with \`scout_models\`: the same \`conversationId\` and
 \`results\`, one aggregate per model and metric (\`level\`, \`task\` title, \`scope\`,
@@ -146,6 +159,25 @@ fixed. Use each agent's own checks to attribute failures. If all candidates scor
 100%, the eval cannot separate their quality: add harder labelled cases before
 claiming equivalent quality. Cost may break the observed tie, not prove equality.
 
+## Preserve evidence before cleanup
+
+Before removing temporary comparison runs, save a local Markdown decision record
+with the package/service versions, date, task and policy, exact candidates,
+unchanged controls, metric definitions/directions, sample counts, aggregate
+quality/cost/time, failures and exclusions, chosen model, limitations and next
+review. Record regret against the cheapest measured candidate meeting the same
+quality floor; write "not measurable" when comparable evidence is missing.
+Exclude secrets, private record IDs, prompts, outputs and trace bodies.
+
+Temporary comparison traces/records are not permanent: after the record is saved
+and checked, propose the exact newly created comparison runs to remove. Obtain
+explicit scoped approval, including any cascading deletion, before the customer
+or their coding agent deletes them with customer-controlled write access. Scout's
+Viewer key cannot do this. Never delete pre-existing customer records, reusable
+fixtures, dataset definitions or eval definitions. Verify the approved cleanup
+and retained controls, then record the outcome locally. If not approved, leave
+records unchanged and state cleanup is pending; do not imply it happened.
+
 If the user has Langfuse, use \`openmerit connect langfuse --help\`: the web UI is
 optional. Headless setup requires a separate connection-manager approval with
 \`openmerit connect langfuse --authorize --no-open\`. Relay its approval link and

package/dist/terminal/scout-update.json

@@ -1 +1 @@
-{"format":1,"contract":"scout-recommend-v2","version":"0.3.6","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Measuring candidates\n\nEach task also has an `evaluation`: the checks (per model call, agent run or\nend-to-end outcome), labelled examples per candidate and when to stop a losing\ncandidate. Offer to run it before the user commits to a model. Use the evals the\nproject already has, such as datasets and experiments in its eval or tracing\nplatform, or its own eval scripts and fixtures: add the shortlisted candidates as\nextra runs of those evals. Do not build a second eval or copy their datasets.\nIf the project has no eval for the task, propose a minimal one built from its\nexisting traces and the decisions people already make, and create it only where\nand how the user agrees. Ask before spending on model calls or writing to an\neval platform.\n\nReport what you measured with `scout_models`: the same `conversationId` and\n`results`, one aggregate per model and metric (`level`, `task` title, `scope`,\n`model` endpoint or model ID, `metric`, `value`, `higherIsBetter`,\n`samples`, `source`). Set `kind` to `usd` or `seconds` for measured cost\nor time (per call at the task level, per run at the agent level), so Scout uses\nyour cost instead of its estimate. Report each agent's own checks at the agent\nlevel: an end-to-end result also measures the agents you did not swap. Send\naggregates only, never inputs, outputs or records.\nScout ranks them above public benchmarks for this user only; models whose\n`quality.measured` is true rest on them.\n\nSwap one agent at a time, holding the other agents, prompts, fixtures and controls\nfixed. Use each agent's own checks to attribute failures. If all candidates score\n100%, the eval cannot separate their quality: add harder labelled cases before\nclaiming equivalent quality. Cost may break the observed tie, not prove equality.\n\nIf the user has Langfuse, use `openmerit connect langfuse --help`: the web UI is\noptional. Headless setup requires a separate connection-manager approval with\n`openmerit connect langfuse --authorize --no-open`. Relay its approval link and\nexpiry; after approval, rerun that command. Existing recommendation keys cannot\nmanage connections. The user creates a separate Viewer key; receive it through\nprotected stdin or a secret-manager-injected environment, never chat, tool\narguments, shell literals, traced commands or tracked files. The CLI accepts\nnon-secret settings via `--config mappings.json` and reads LANGFUSE_PUBLIC_KEY\nand LANGFUSE_SECRET_KEY from its environment, or reads full settings via --stdin.\nA separate trusted MCP connection with the management key exposes\n`scout_connection` (status/configure/sync/disconnect/clear). For configure, keys\nmust come from client-controlled HTTP headers outside model context; if the\nclient cannot do this securely, use protected CLI input or the web UI instead.\nAfter they confirm connection and score/model mappings, call `scout_models` with\n`sync: \"langfuse\"` to refresh aggregate measurements before recommending. The CLI\nequivalent is `openmerit scout --conversation <id> --sync langfuse`. Never read\ntrace bodies to work around missing mappings or untagged runs; use the aggregate\nreport path instead. Results are stored per user, used in new conversations and\nexpire after 90 days. Disconnect deletes the Viewer key and imported aggregates;\nDelete all evaluation data also deletes manually reported aggregates. Obtain\nconsent before deleting: headless CLI requires --yes, MCP requires confirm: true.\nDeleting the connection does not revoke the separate management device key.\nPast conversation text is not erased. Never put record IDs, emails or secrets in\n`scope` or `source`; use labels such as \"coding agent\" and \"held-out eval\".\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. When an install or login waits for approval (exit code\n10), show the user the approval link and its UTC expiry in your reply, not only in\ncommand output, then rerun the same command once they approve. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}
+{"format":1,"contract":"scout-recommend-v2","version":"0.3.7","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Measuring candidates\n\nEach task also has an `evaluation`: the checks (per model call, agent run or\nend-to-end outcome), labelled examples per candidate and when to stop a losing\ncandidate. Offer to run it before the user commits to a model. First assess the\nproject's readiness for this task:\n\n- **No eval setup:** propose a customer-owned setup. Langfuse is the default\n  suggestion for its supported aggregate connector, not a requirement: existing\n  local/custom eval scripts can report aggregates without a hosted platform.\n  Phoenix/Arize and Braintrust are alternatives, not supported Scout connectors.\n  Ask which system the user wants before creating accounts or resources.\n- **Setup exists but evidence is insufficient:** keep that system. Write a\n  task-specific eval plan in the project's local Markdown decision record for\n  the user or their coding agent to implement. Name exact candidates/endpoints,\n  quality floor and task/agent/outcome checks, labelled boundary cases and\n  controls, sample size, early-stop rule, spend cap, aggregate mappings and\n  temporary run retention. Explain what the current evidence cannot decide.\n- **Suitable evals exist:** reuse their fixtures and definitions; add only the\n  approved candidate runs. Do not build a second eval or copy their datasets.\n\nOpenMerit does not host or run these evals. The customer or their coding agent\nimplements and maintains them with the customer's tools; Scout keeps Viewer-only\naccess. Ask before model spend or any platform write. Without measured evidence,\nScout may suggest candidates, but do not claim validated model quality.\n\nReport what you measured with `scout_models`: the same `conversationId` and\n`results`, one aggregate per model and metric (`level`, `task` title, `scope`,\n`model` endpoint or model ID, `metric`, `value`, `higherIsBetter`,\n`samples`, `source`). Set `kind` to `usd` or `seconds` for measured cost\nor time (per call at the task level, per run at the agent level), so Scout uses\nyour cost instead of its estimate. Report each agent's own checks at the agent\nlevel: an end-to-end result also measures the agents you did not swap. Send\naggregates only, never inputs, outputs or records.\nScout ranks them above public benchmarks for this user only; models whose\n`quality.measured` is true rest on them.\n\nSwap one agent at a time, holding the other agents, prompts, fixtures and controls\nfixed. Use each agent's own checks to attribute failures. If all candidates score\n100%, the eval cannot separate their quality: add harder labelled cases before\nclaiming equivalent quality. Cost may break the observed tie, not prove equality.\n\n## Preserve evidence before cleanup\n\nBefore removing temporary comparison runs, save a local Markdown decision record\nwith the package/service versions, date, task and policy, exact candidates,\nunchanged controls, metric definitions/directions, sample counts, aggregate\nquality/cost/time, failures and exclusions, chosen model, limitations and next\nreview. Record regret against the cheapest measured candidate meeting the same\nquality floor; write \"not measurable\" when comparable evidence is missing.\nExclude secrets, private record IDs, prompts, outputs and trace bodies.\n\nTemporary comparison traces/records are not permanent: after the record is saved\nand checked, propose the exact newly created comparison runs to remove. Obtain\nexplicit scoped approval, including any cascading deletion, before the customer\nor their coding agent deletes them with customer-controlled write access. Scout's\nViewer key cannot do this. Never delete pre-existing customer records, reusable\nfixtures, dataset definitions or eval definitions. Verify the approved cleanup\nand retained controls, then record the outcome locally. If not approved, leave\nrecords unchanged and state cleanup is pending; do not imply it happened.\n\nIf the user has Langfuse, use `openmerit connect langfuse --help`: the web UI is\noptional. Headless setup requires a separate connection-manager approval with\n`openmerit connect langfuse --authorize --no-open`. Relay its approval link and\nexpiry; after approval, rerun that command. Existing recommendation keys cannot\nmanage connections. The user creates a separate Viewer key; receive it through\nprotected stdin or a secret-manager-injected environment, never chat, tool\narguments, shell literals, traced commands or tracked files. The CLI accepts\nnon-secret settings via `--config mappings.json` and reads LANGFUSE_PUBLIC_KEY\nand LANGFUSE_SECRET_KEY from its environment, or reads full settings via --stdin.\nA separate trusted MCP connection with the management key exposes\n`scout_connection` (status/configure/sync/disconnect/clear). For configure, keys\nmust come from client-controlled HTTP headers outside model context; if the\nclient cannot do this securely, use protected CLI input or the web UI instead.\nAfter they confirm connection and score/model mappings, call `scout_models` with\n`sync: \"langfuse\"` to refresh aggregate measurements before recommending. The CLI\nequivalent is `openmerit scout --conversation <id> --sync langfuse`. Never read\ntrace bodies to work around missing mappings or untagged runs; use the aggregate\nreport path instead. Results are stored per user, used in new conversations and\nexpire after 90 days. Disconnect deletes the Viewer key and imported aggregates;\nDelete all evaluation data also deletes manually reported aggregates. Obtain\nconsent before deleting: headless CLI requires --yes, MCP requires confirm: true.\nDeleting the connection does not revoke the separate management device key.\nPast conversation text is not erased. Never put record IDs, emails or secrets in\n`scope` or `source`; use labels such as \"coding agent\" and \"held-out eval\".\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. When an install or login waits for approval (exit code\n10), show the user the approval link and its UTC expiry in your reply, not only in\ncommand output, then rerun the same command once they approve. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}

package/dist/terminal/src/scout-instructions.js

@@ -87,14 +87,27 @@ application accuracy. Use the project's actual SDK and protocol.
 
 Each task also has an \`evaluation\`: the checks (per model call, agent run or
 end-to-end outcome), labelled examples per candidate and when to stop a losing
-candidate. Offer to run it before the user commits to a model. Use the evals the
-project already has, such as datasets and experiments in its eval or tracing
-platform, or its own eval scripts and fixtures: add the shortlisted candidates as
-extra runs of those evals. Do not build a second eval or copy their datasets.
-If the project has no eval for the task, propose a minimal one built from its
-existing traces and the decisions people already make, and create it only where
-and how the user agrees. Ask before spending on model calls or writing to an
-eval platform.
+candidate. Offer to run it before the user commits to a model. First assess the
+project's readiness for this task:
+
+- **No eval setup:** propose a customer-owned setup. Langfuse is the default
+  suggestion for its supported aggregate connector, not a requirement: existing
+  local/custom eval scripts can report aggregates without a hosted platform.
+  Phoenix/Arize and Braintrust are alternatives, not supported Scout connectors.
+  Ask which system the user wants before creating accounts or resources.
+- **Setup exists but evidence is insufficient:** keep that system. Write a
+  task-specific eval plan in the project's local Markdown decision record for
+  the user or their coding agent to implement. Name exact candidates/endpoints,
+  quality floor and task/agent/outcome checks, labelled boundary cases and
+  controls, sample size, early-stop rule, spend cap, aggregate mappings and
+  temporary run retention. Explain what the current evidence cannot decide.
+- **Suitable evals exist:** reuse their fixtures and definitions; add only the
+  approved candidate runs. Do not build a second eval or copy their datasets.
+
+OpenMerit does not host or run these evals. The customer or their coding agent
+implements and maintains them with the customer's tools; Scout keeps Viewer-only
+access. Ask before model spend or any platform write. Without measured evidence,
+Scout may suggest candidates, but do not claim validated model quality.
 
 Report what you measured with \`scout_models\`: the same \`conversationId\` and
 \`results\`, one aggregate per model and metric (\`level\`, \`task\` title, \`scope\`,
@@ -112,6 +125,25 @@ fixed. Use each agent's own checks to attribute failures. If all candidates scor
 100%, the eval cannot separate their quality: add harder labelled cases before
 claiming equivalent quality. Cost may break the observed tie, not prove equality.
 
+## Preserve evidence before cleanup
+
+Before removing temporary comparison runs, save a local Markdown decision record
+with the package/service versions, date, task and policy, exact candidates,
+unchanged controls, metric definitions/directions, sample counts, aggregate
+quality/cost/time, failures and exclusions, chosen model, limitations and next
+review. Record regret against the cheapest measured candidate meeting the same
+quality floor; write "not measurable" when comparable evidence is missing.
+Exclude secrets, private record IDs, prompts, outputs and trace bodies.
+
+Temporary comparison traces/records are not permanent: after the record is saved
+and checked, propose the exact newly created comparison runs to remove. Obtain
+explicit scoped approval, including any cascading deletion, before the customer
+or their coding agent deletes them with customer-controlled write access. Scout's
+Viewer key cannot do this. Never delete pre-existing customer records, reusable
+fixtures, dataset definitions or eval definitions. Verify the approved cleanup
+and retained controls, then record the outcome locally. If not approved, leave
+records unchanged and state cleanup is pending; do not imply it happened.
+
 If the user has Langfuse, use \`openmerit connect langfuse --help\`: the web UI is
 optional. Headless setup requires a separate connection-manager approval with
 \`openmerit connect langfuse --authorize --no-open\`. Relay its approval link and

package/docs/scout-close-loop.md

@@ -28,6 +28,32 @@ Each recommended model names an exact endpoint and provider model ID.
 
 ## Measure with your own evals
 
+**Eval-readiness and evidence guidance — npm 0.3.7 release preparation:** this
+package adds the readiness and cleanup workflow below to the installed Scout
+skill. Existing npm 0.3.6 instructions cover measuring and aggregate reporting,
+but do not yet prescribe all three readiness paths or an evidence-first cleanup
+record. Registry publication is separate from these source changes. This is
+guidance for the customer's coding agent, not an automated eval runner, platform
+provisioner or deletion service.
+
+### Choose the readiness path
+
+- **No eval setup.** Scout can still suggest candidates from public evidence;
+  that is not measured application quality. Propose a minimal customer-owned
+  eval setup and obtain approval before creating it. Langfuse is the default
+  suggestion because Scout has its aggregate connector. A hosted platform is
+  optional: customer-owned scripts and fixtures can report aggregates directly.
+  Phoenix/Arize and Braintrust are alternatives, not supported Scout connectors.
+- **A setup exists, but the evals are missing or too weak for model selection.**
+  Keep the existing system. Have the user or their coding agent write a local
+  task-specific Markdown eval plan with exact candidates/endpoints, quality floor,
+  task/agent/outcome checks, labelled boundary cases and controls, sample size,
+  early stopping, spend cap, aggregate mappings and temporary-run retention.
+  Review it before implementing it with the customer's tools. OpenMerit's access
+  remains Viewer-only; it does not install or maintain those evals.
+- **Suitable evals already exist.** Reuse their fixtures and definitions. Add
+  approved candidate runs rather than creating another dataset or platform.
+
 Add the shortlist as runs of your existing evals, not a second platform. Obtain
 approval before model spend or platform writes. For agents, change **one agent at
 a time** while holding the other models, prompts, tools, fixtures and controls
@@ -38,6 +64,34 @@ If every candidate scores 100%, the eval does not distinguish their quality.
 Add harder labelled boundary cases, rare failures and adversarial examples;
 cheaper on this test is not proof of equal quality in production.
 
+### Preserve evidence, then request cleanup
+
+Save a local Markdown decision record before removing temporary comparison data.
+Include the date, package and service versions, task and policy, exact candidates,
+unchanged controls, metric definitions and direction, sample counts, aggregate
+quality/cost/time, failures and exclusions, decision, limitations and next review.
+Exclude keys, private record identifiers, prompts, outputs and trace bodies.
+
+State cost regret in a consistent unit (for example USD per agent run): the
+chosen model's measured cost minus the cheapest measured candidate meeting the
+same quality floor on comparable evals. If there is no measured comparison,
+write **not measurable**, not zero. One success or all candidates scoring 100%
+on a tiny eval is not proof of equal production quality.
+
+Comparison records are temporary, but deletion is **not automatic**. After
+checking the saved evidence, propose the exact newly created comparison runs and
+any cascading deletions, then obtain explicit scoped approval. The customer or
+their coding agent performs cleanup using customer-controlled write access;
+Scout's Viewer key cannot delete Langfuse data. Never delete pre-existing customer
+records, reusable fixtures, dataset definitions or eval definitions. Verify that
+only the approved records disappeared and retained controls still exist. Record
+cleanup as pending when permission or verification is missing.
+
+Deleting platform runs, disconnecting Scout and deleting Scout's stored aggregates
+are separate operations. Disconnect does not erase past conversations or customer
+platform records. Keep enough permitted aggregate evidence locally to explain a
+later decision, without promising Scout can refresh a deleted source experiment.
+
 ## Report aggregates
 
 Report up to 50 aggregates per call, one per model, metric and scope. Use public
@@ -236,8 +290,15 @@ experiment headers/metadata and aggregate Metrics v2 reads are requested. A
 deliberately invalid dataset-create request checks read-only permission without
 ever sending a valid write payload. Only an authenticated read followed by HTTP
 403 on that probe is accepted; other responses fail closed. Live Viewer-key
-connection acceptance was verified on 6 November 2026; fresh mapped aggregate
-import and reranking remain separate release checks.
+connection acceptance, one fresh mapped aggregate import and measured reranking
+were verified on 6 November 2026 with npm 0.3.6 and the matching deployments.
+This one-sample plumbing check is not comparative quality acceptance. Two new
+conversations received the aggregate but returned prose before the 6 November
+structured-completion deploy. After it, three fresh published-CLI conversations
+returned five structured tasks each, reused the measured coding result without
+resending it and retained the one-sample caveat. This focused check does not
+establish fresh-coding-agent usability; full fresh-agent and two-release
+acceptance remain open.
 
 Click **Sync aggregates**, or request a fresh recommendation with:
 

package/docs/scout.md

@@ -20,6 +20,32 @@ Scout is a separate hosted beta product. Npm 0.1.7 introduced optional Claude Co
 
 ## Evaluation plans and measured results — deployed 5 November 2026, npm 0.3.1
 
+**Rank-aligned summaries — deployed 6 November 2026:** after filtering and ranking,
+Scout builds each nonempty task summary from the first model's name, rationale
+and caveat. A summary written before sorting could previously name a different
+leader, particularly after measured evidence or measured cost changed the order.
+Ranking and quality scores are unchanged. One measured success still does not
+establish superiority over untested candidates. Scout serving version
+`86d43d6c-e60f-4989-be50-9461d48f37cf` passed a live published-CLI check: all five
+task summaries matched their final first model, including measured coding review.
+Past saved answers do not change until the conversation is recomputed. Fresh
+conversations returned prose rather than validated shortlists in two acceptance
+runs before the structured-completion deployment below. A prose fallback is not
+a successful structured recommendation.
+
+**Structured completion recovery — deployed 6 November 2026:** when the model ends a
+turn with prose before an answer tool accepts a clarification or all planned
+shortlists, Scout continues the same durable run at most twice. Recovery keeps
+accepted task shortlists and names the remaining tasks; it does not bypass
+catalogue, evidence, scope or quality validation. The attempt budget survives
+object eviction and resets only for a new user turn. An already accepted answer
+adds no model call. If recovery is exhausted, the existing prose/error fallback
+remains explicitly unstructured; provider failures do not start completion
+recovery. PR #83 passed all PR/main CI checks and deployed Scout serving version
+`fe34446b-528c-4b90-b0a0-60946caa0d78` at 09:38 UTC. Landing, Scout, connection,
+status and version endpoints returned HTTP 200. Fresh-conversation acceptance is
+separate from deployment health and is required before this release gate closes.
+
 The Scout serving deploy of 5 November 2026 (version `ea371c78-6ccf-4931-a0b8-aa28a6599aaa`) serves this behavior to every client. A live MCP check returned an evaluation plan and agent grouping, and reported results moved a measured model below the quality bar. npm 0.3.1 adds `openmerit scout --results` and the installed instructions for measuring; older installations reach the new `scout_models` fields through MCP but their instructions do not mention measuring until the installer is rerun.
 
 OpenMerit does not run evaluations or host an eval platform. Scout tells you what to measure, your own eval system measures it, and Scout uses the aggregate results:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.3.6",
+  "version": "0.3.7",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {