openmerit 0.1.7-preview.4 → 0.1.7-preview.5

package/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 0.1.7-preview.5 — 2026-10-17
+
+- Update packaged Scout documentation for the independently deployed matching repair: shared evidence targeting, retrieval coverage and bounded semantic consistency checks.
+- Keep earlier failed acceptance and task-execution results visible; matching checks do not establish task accuracy or globally optimal recommendations.
+- Preserve the package runtime, CLI commands, exports and legal metadata. Target the `preview` channel; stable `latest` remains unchanged.
+
 ## 0.1.7-preview.4 — 2026-10-17
 
 - Update the packaged Scout guide for the independently deployed task-relevance beta, including metadata filters, existing ambiguity responses, measured-evidence preference and bounded matching.

package/README.md

@@ -22,7 +22,7 @@ Explore the [hosted sample](https://dash.openmerit.ai/) or [before/after dashboa
 
 ## Compare models with Pi
 
-The **npm preview `0.1.7-preview.4`** provides `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose model scope and a $10, custom or no-fixed-cap budget, freeze Scout's shortlist, and compare 3–10 fresh attempts after source changes. Budget and sandbox blocks stay separate from task scores; unknown costs stay unknown. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@0.1.7-preview.4/docs/agent-evaluations.md) for setup, graders, budgets and availability. Scout's hosted API and MCP service update independently of the npm package. This is separate from the Pi workflow below.
+The **npm preview** provides `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose model scope and a $10, custom or no-fixed-cap budget, freeze Scout's shortlist, and compare 3–10 fresh attempts after source changes. Budget and sandbox blocks stay separate from task scores; unknown costs stay unknown. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@preview/docs/agent-evaluations.md) for setup, graders, budgets and availability. Scout's hosted API and MCP service update independently of the npm package. This is separate from the Pi workflow below.
 
 With Pi 0.87 and a model provider configured:
 

package/docs/commands.md

@@ -2,7 +2,7 @@
 
 The standalone `openmerit dash` command runs in your shell. The `/openmerit` slash commands run inside Pi with the extension loaded.
 
-The npm `0.1.7-preview.4` preview (`openmerit@preview`) provides `openmerit eval init`, `doctor`, `plan`, `run`, `watch` and `report`. See [agent evaluations](agent-evaluations.md) for scope selection, repeated task checks and budgets. These are shell commands, separate from Pi commands.
+The npm preview (`openmerit@preview`) provides `openmerit eval init`, `doctor`, `plan`, `run`, `watch` and `report`. See [agent evaluations](agent-evaluations.md) for scope selection, repeated task checks and budgets. These are shell commands, separate from Pi commands.
 
 Since `0.1.7-preview.2`, `openmerit eval plan [--scope existing|all] [--budget <USD|unlimited>]` asks for a budget after scope. Choose $10, a custom amount, or no fixed cap; saved choices and cumulative usage persist. See [budget selection](agent-evaluations.md#budget-selection).
 

package/docs/scout.md

@@ -34,7 +34,7 @@ Acceptance retained the earlier provider failures rather than counting them as s
 
 These corrections are deployed as a limited hosted beta. Hourly evidence documents retain verified research descriptions and directly selected methodology alongside feed metadata; embedding caches change when that evidence changes. The already verified descriptions are indexed directly, without a second model-generated summary; original citation excerpts remain in the source records rather than the semantic search text. Structured extraction separates the downstream work from model restrictions before Decisions interprets its capabilities and output contract. Name resolution can run alongside Decisions. This avoids interpreting model selection itself as a downstream capability; it does add a sequential interpretation dependency within the existing six-second budget.
 
-Query vectors describe the downstream work, excluding model-selection restrictions. They retrieve methodology candidates, with prepared memberships as a fallback when the query has no matches and a backstop for required capabilities missing from the retrieved set. The query order and hourly model/benchmark availability map propose up to sixteen methodology records. Decisions checks each record independently against the actual workload, without mixing unrelated methodologies in one input. Both general relevance and a deliberately scored required ability must be sufficiently supported. A broader ability can be a component proxy without testing the exact domain or answer format; incidental prerequisites cannot qualify. Ambiguous labels, broad capability overlap and unavailable checks cannot grant quality evidence. Fine-grained scope is checked against the full workload and methodology rather than asking every discovered capability again on each search. Core task/format interpretation stays bounded as the hourly directory grows.
+The initial task-relevance beta used query vectors to retrieve up to sixteen methodologies, then required both a general-relevance predicate and a scored-ability predicate for each. Although intended to admit component proxies, these overlapping questions could disagree and reject a benchmark because its domain differed from the requested task. The matching repair below replaces that assessment path; earlier acceptance results remain historical records.
 
 Classification and judgment searches consider applicable benchmark evidence before an unmeasured native endpoint contract. Without comparable candidates they retain an explicitly unmeasured task-fit fallback; contract compatibility alone does not demonstrate accuracy. Spatial pipelines keep their separate compatibility checks. Ambiguous output representations can use the existing clarification response, with model restrictions preserved. No task-name, object-name or model-specific exception is added.
 
@@ -52,6 +52,20 @@ One model confused two bird species; two invoice candidates added an extra zero
 
 Publication retained the latest completed hourly research and a healthy refreshed feed set. The final rollout check passed 9/11 cases on its first attempt: bird matching reached the deadline and explicit SVG generation returned no shortlist. Both passed one focused recheck without a matcher change. After deployment, the public HTTP suite again passed 9/11: the explicit SVG request returned no shortlist, and a Qwen-family code-authoring request admitted tool-calling evidence. Creator/family/context/date eligibility checks and impossible-context no-match passed. The live browser bird prompt displayed five recommendations without overflow. These are retained semantic and latency limitations of the beta, not a clean first-pass acceptance or proof of best-five performance.
 
+## Matching reliability repair — 17 October 2026
+
+The task Decisions batch now distinguishes the primary quality target from endpoint requirements such as image input or serialization. Retrieval and evidence assessment use that same target. Verified hourly memberships backstop the vector results even when some primary evidence was already retrieved, so an incomplete vector result cannot suppress other prepared evidence that fits the selected inventory. The existing sixteen-methodology request bound remains; this is not exhaustive methodology retrieval.
+
+If the primary target contradicts the required abilities, Scout makes one bounded Decisions reconciliation using the quoted task and previous answers. It has at most 750 milliseconds within the existing matching deadline. An unresolved contradiction returns unavailable instead of being presented as a verified no-match. This is a general consistency check, without task-name or format keywords.
+
+One Decisions batch identifies each proposed benchmark's deliberately scored ability from its own methodology and the registry's evaluation definitions. It does not require a general component benchmark to reproduce the task's exact domain, labels or output artifact. Only a recognized ability with sufficient confidence contributes proxy quality; incidental operations, unknown scope, rejected answers and unavailable checks do not. A primary-target proxy does not establish the quality of every component in a combined task or exact task accuracy. Endpoint modality, format and tool restrictions still apply independently. No model, provider or task-object exceptions are used.
+
+Exact metadata selection and query embedding start after extraction and canonical-name resolution, alongside task Decisions. Model evidence loads alongside the benchmark assessment; unaccepted observations are removed before ranking. These changes reuse the existing hourly catalogue, D1 projection and vectors, add no background service and retain the six-second matching deadline. The UI and API/MCP request inputs are unchanged. No new npm publication is included.
+
+Diagnostic stage timings overlap and must not be added together. `interpretationMs` includes the concurrently prepared eligibility/embedding result needed to complete interpretation; individual Decisions, extraction and database timings remain available in `runtime`. Cache hits retain the original evaluation time and report zero new interpretation time as before.
+
+Before deployment, full repository verification passed, including 336 Scout tests. Eight filter/ambiguity checks passed against the prepared metadata oracle: provider, family, context, date, impossible context, an incidental provider mention, an ambiguous diagram and explicit SVG source generation. The isolated Ubuntu VM run passed 203/203 assertions against staging. All seven workloads passed through both HTTP and the official MCP SDK; installed CLI email and SQL plans contained only the declared connections. An earlier staged run exposed the remaining email retrieval gap. A subsequent test setup reused an initialized directory and correctly failed `eval init`; the final isolated run used a fresh directory. Those attempts remain recorded separately. Final production acceptance also passed 203/203 VM assertions and all eight filter/ambiguity cases after the consistency correction. The HTTP workload median was 3.665 seconds versus the earlier 4.924-second sample; no final request timed out. These checks establish tested matching and integration contracts, not exact task accuracy or a latency guarantee. The [testing guide](testing.md#hosted-matching-repair-17-october-2026) retains the earlier failed rollout and separate final results.
+
 ## Latency improvements — hosted beta, 17 October 2026
 
 The hosted implementation keeps the existing task and format Decisions, exact metadata comparisons and numerical selection. Numeric and date vocabularies are omitted from interpretation input because their bounds are arbitrary; small canonical vocabularies remain in the first call. Larger vocabularies are deferred until an extracted restriction references them; a model-assisted resolution checks against the complete published vocabulary without changing unrelated conditions. Unknown mandatory names remain unresolved and cannot widen the result set. Stable reference data has an explicit provider prompt-cache boundary separate from the prompt and UTC reference time. Cache availability is measured rather than guaranteed.
@@ -347,6 +361,10 @@ Open `http://127.0.0.1:45833/scout/`. This lightweight local preview uses the sh
 
 Publication records a durable completion receipt after updating the active pointer, pending-research queue and health status. Interrupted publication resumes unfinished work; a lost success reply can be acknowledged again after the lease is released. Replaying a completed publication cannot roll back a newer catalogue or clear a later refresh's lease. Completed research receipts leave the queue only when publication commits; unfinished research retains its original paid job for a later hourly run.
 
+Research rediscovery updates an existing record with the same stable primary key, even when semantic interpretation does not link it to another catalogue record. Different source/version identities remain separate unless the evidence supports a semantic link. Duplicate primary keys fail publication validation; searches continue against the last committed catalogue while a later refresh reuses durable research receipts.
+
+The 17 October post-deployment recovery reused all 24 saved research jobs and published successfully after this identity correction, with a matching completion receipt and empty backlog. Memory-limit and vector-index readiness retries occurred during that run. Its fresh npm VM acceptance also exposed unavailable books/posters and invoice HTTP responses, empty SQL and safety-rewrite shortlists, and empty installed-CLI plans. These remain beta limitations; the detailed results are in the [release validation record](testing.md#preview4-release-checks-17-october-2026).
+
 Prepared capability pools can overlap. Searches use their prepared ID lists to choose a smaller covering set of pools, preserving the selected candidates and complete evidence while avoiding repeated downloads of the same model records. Explicit model and endpoint inventories retain their existing scope checks. These metadata reads do not fetch source pages or redo hourly research.
 
 `demo/domain-router.mjs` routes Scout paths to this service while retaining the existing website, protected memo, authentication, and terminal services. There is no Scout admin/report route. The public guide is generated and published with the main documentation. Do not deploy the older root `wrangler.jsonc` to publish Scout. `npm run build:scout` packages only Scout, public documentation, and its shared assets; private memo and demo applications are excluded.

package/docs/testing.md

@@ -1,10 +1,28 @@
 # Testing and validation
 
+## Preview.5 release preparation — 17 October 2026
+
+The `0.1.7-preview.5` candidate packages the current Scout guides after the independently deployed matching repair below. The intended channel is `preview`, with stable `latest` retained at `0.1.6`. Full release verification, comparison with the published preview.4 artifact, registry publication and a fresh-install VM HTTP/MCP/CLI acceptance run are release gates; preparation alone does not establish that those gates passed. Scout remains a separately deployed service, and the earlier bird/invoice execution limitations remain in scope.
+
+## Hosted matching repair — 17 October 2026
+
+The repair passed full repository verification, including 336 Scout tests and documentation checks, plus eight live-provider staging filter/ambiguity cases checked against the catalogue metadata oracle. The isolated Ubuntu ARM64 VM staging run passed 203/203 assertions using the installed npm preview.4 package, HTTP API, official MCP SDK and CLI planning. Both declared-connection CLI workloads had candidates: email returned one, SQL returned two. All original seven workloads passed through both interfaces. Package hashes/exports, inventories and request/protocol errors passed; no recommended models were executed. The package version and npm tags are unchanged.
+
+Earlier staged attempts are retained: the first run exposed an email retrieval gap after SQL was fixed; another reused the CLI project directory and failed the expected fresh-directory `eval init` assertion. The final run corrected test isolation without changing that assertion. This record does not replace the failed preview.4 post-publication acceptance below. Production rollout results are recorded after deployment. These are matching and integration checks, not proof of task accuracy or a production latency percentile.
+
+The first production rollout passed its 203 VM checks, but the separate filter sweep passed 6/8: impossible context returned unavailable once and explicit SVG produced contradictory task fields. A generic bounded Decisions consistency correction was then added. Full verification passed with 336 Scout tests; the revised staging run passed all 203 VM checks and all eight filter cases. The SVG case exercised the reconciliation path. These phases are recorded separately rather than treating the earlier failures as successful attempts.
+
+The final production deployment passed **203/203 VM assertions and 8/8 filter/ambiguity cases**. All seven HTTP/MCP workloads, both CLI plans, package hashes/exports, exact inventories and request/protocol errors passed. The seven HTTP workload samples had a median wall time of **3.665 seconds**, compared with **4.924 seconds** in the earlier published-preview failure run. Maximum HTTP time was 4.268 seconds; the slowest HTTP/MCP request was 4.840 seconds. Several exact repeated MCP requests used the isolate-local cache; the books/posters MCP request did fresh work, so cache availability is not assumed. No task timed out in this final run. These are small end-to-end observations under different load and cache conditions, not a production percentile or guaranteed response time. The hourly schedule and all 127 UI source files are unchanged; five checked live scripts/styles match source bytes. npm remains preview.4 and no new package was published.
+
 ## Preview.4 release checks — 17 October 2026
 
-The `0.1.7-preview.4` candidate passed the full repository verification, including 329 Scout tests and documentation checks. Its packaged runtime is byte-for-byte unchanged from preview.3; version metadata and public guides carry this release. Scout deploys independently and its UI files are unchanged. Broader evidence coverage, task calibration and complete quality/cost/latency validation remain follow-up work.
+The published `0.1.7-preview.4` package passed the full repository verification, including 329 Scout tests and documentation checks. Its packaged runtime is byte-for-byte unchanged from preview.3; version metadata and public guides carry this release. Registry integrity matches the reviewed tarball, the `preview` tag resolves to this version, and stable `latest` remains `0.1.6`. Scout deploys independently and its UI files are unchanged. Broader evidence coverage, task calibration and complete quality/cost/latency validation remain follow-up work.
+
+The hosted beta's public post-deployment suite passed 9/11 cases. Explicit SVG generation returned no shortlist; a Qwen-family code-authoring request admitted tool-calling evidence. Restriction checks passed for creator, family, context and dates, including known matches and impossible-context no-match. The actual bird prompt also displayed five recommendations in a browser. Earlier candidate deadline and interpretation failures are retained in the [Scout validation record](scout.md#task-relevance-corrections-hosted-beta-17-october-2026). These checks do not establish universal semantic reliability or improved task accuracy.
+
+Fresh registry installation in the existing Ubuntu ARM64 Lima VM verified all 101 packaged file hashes, registry integrity and public core/protocol/Pi exports. Seven workloads were sent through both HTTP and the official MCP SDK. Six of those fourteen requests failed their functional expectations: books/posters and invoice HTTP requests returned unavailable responses, while SQL generation and safety classification with rewriting returned empty shortlists through both interfaces. The matching MCP books/posters and invoice requests succeeded; bird identification, label-only classification and unsupported audio behaved as expected. The first harness stopped when it assumed SQL had returned an endpoint. A separate continuation tested exact endpoint scope using the successful bird result, request/protocol errors and installed CLI planning, preserving all original failures. Together the runs recorded 187 passing checks and nine failed assertions, including duplicate agreement failures caused by unavailable responses and an installed-CLI candidate check. Both CLI email and SQL plans were empty. Planning created no execution or spend state, and the existing global VM installation was retained. This is not a clean acceptance pass. The seven first HTTP workload requests had a median wall time of 4.924 seconds and a maximum of 6.229 seconds, including unavailable responses; repeated cached requests were faster.
 
-The hosted beta's public post-deployment suite passed 9/11 cases. Explicit SVG generation returned no shortlist; a Qwen-family code-authoring request admitted tool-calling evidence. Restriction checks passed for creator, family, context and dates, including known matches and impossible-context no-match. The actual bird prompt also displayed five recommendations in a browser. Earlier candidate deadline and interpretation failures are retained in the [Scout validation record](scout.md#task-relevance-corrections-hosted-beta-17-october-2026). These checks do not establish universal semantic reliability or improved task accuracy. Registry installation and post-publication VM results are recorded separately after publication.
+Post-deployment refresh validation exposed duplicate stable research identities. The hosted integration now updates those exact identities without introducing model or benchmark exceptions. Full verification then passed 331 Scout tests. The normal recovery refresh reused all 24 saved research jobs, completed all 24 and published its catalogue with a matching durable receipt and empty backlog. Its interpretation and staging steps needed memory-limit and vector-readiness retries; successful publication does not establish sufficient memory headroom at larger scale. Live status reported neither stale nor degraded data after recovery. This repair deploys independently of the already published npm runtime.
 
 ## Published preview.3 acceptance — 16 October 2026
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.1.7-preview.4",
+  "version": "0.1.7-preview.5",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {