openmerit 0.1.7-preview.2 → 0.1.7-preview.4

package/CHANGELOG.md

@@ -1,5 +1,62 @@
 # Changelog
 
+## 0.1.7-preview.4 — 2026-10-17
+
+- Update the packaged Scout guide for the independently deployed task-relevance beta, including metadata filters, existing ambiguity responses, measured-evidence preference and bounded matching.
+- Document retained execution-test failures, incomplete coverage and provisional Pareto comparisons; no improvement in exact task accuracy or globally optimal membership is claimed.
+- Preserve the package runtime, CLI commands, exports and legal metadata. Publish to `preview`; stable `latest` remains unchanged.
+
+## 0.1.7-preview.3 — 2026-10-16
+
+- Update packaged Scout API/MCP and agent-evaluation guides for the sole `/scout/` service and current request, inventory, data-handling and reliability contracts.
+- Keep the hosted hourly research and matching service separate from npm installation. Hosted publication recovery, per-source freshness, bounded Decisions retries and reduced overlapping pool reads were deployed independently.
+- Exclude internal website and deployment maintenance notes from the npm artifact. Keep existing commands, exports, legal metadata and stable `latest` behavior unchanged.
+
+## Scout semantic benchmark interpretation — hosted beta, 2026-10-15
+
+- Replace benchmark capability/name rules and family-name regexes with hourly Decisions interpretation of source definitions and methodology.
+- Cache unchanged interpretations; leave unsupported or failed scopes unclassified and preserve evidence provenance.
+- Keep prompt matching local to prepared benchmark metadata, with unchanged UI, prompt/specialist calls and hourly schedule. All 24 semantic benchmark cases, 107 local/live HTTP/MCP matching cases and 546 automated checks passed. npm publication remains deferred.
+- Store compressed analytics payloads as gzip bytes to avoid base64 overhead on larger evidence records; retain legacy decoding and complete-record size limits.
+
+## Scout semantic specialist matching — hosted beta, 2026-10-15
+
+- Replace Hugging Face domain/language keyword dictionaries and partial label overlap with Decisions compatibility checks.
+- Prepare and cache specialist profiles during the hourly refresh; batch only relevant compatibility checks when a prompt arrives.
+- Require all applicable domain, language and fixed-label checks to pass. Preserve unknown metrics and ordinary results if specialist checking fails.
+- Keep the UI unchanged. All 107 cases passed locally and through live HTTP/MCP; 526 automated tests passed. npm remains deferred.
+
+## Scout semantic formats and SQL interpretation — hosted beta, 2026-10-15
+
+- Remove prompt keyword scans from strict-format and structured-data interpretation; use the existing Decisions answers, including described record layouts and non-English requests.
+- Recognize read-only SQL query writing and repair without requiring schema details in the task description. Keep query source distinct from executing it, its result format, prose explanation and data modification.
+- Keep uncertain format answers visible and exclude them from narrow calibration matching. Preserve response validation, general consistency checks, confidence thresholds, one interpretation request, hourly collection and the UI.
+- All 91 semantic cases passed locally and through live HTTP/MCP, including the previously failing SQL request and four fresh SQL holdouts. All 511 automated tests passed. npm publication remains deferred.
+
+## Scout category and content interpretation — hosted beta, 2026-10-15
+
+- Recognize conversationally supplied categories without requiring a formal label list; keep subtype and individual-identity discovery open-ended.
+- Require document understanding for requested written/table content, including mixed outputs, without inferring it merely from a photographed object.
+- Update two existing semantic questions with unchanged confidence thresholds, UI, hourly refresh and request count. Local and live HTTP/MCP acceptance passed 43 cases; an opt-in semantic audit preserves those cases for future interpretation changes. npm publication remains deferred.
+
+## Scout task interpretation and ambiguous matching — hosted beta, 2026-10-15
+
+- Frame every task description as quoted data for requirement analysis, preserving supplied-label and negation semantics without prompt-specific exceptions or lower confidence thresholds.
+- Keep decision-plus-prose or extraction requests on generative endpoints, reject narrow calibration for uncertain operations/outputs, and ignore weak document-layout guesses for ordinary photos.
+- All 25 routing cases passed locally and through live HTTP/MCP, including the prior binary-label failure. UI, hourly refresh and the single interpretation request are unchanged. npm publication remains deferred.
+
+## Scout identification matching — hosted beta, 2026-10-15
+
+- Distinguish open identification from supplied-label classification within the existing interpretation call. Open or uncertain identification uses evidence-ranked generative endpoints; fixed-choice APIs and label-dependent classifiers require supplied choices.
+- Require relevant visual benchmark evidence for visual requests, preserve classification label scope in calibration matching, and use ordinary output-length estimates for generated names. UI and hourly refresh are unchanged; no npm release is included.
+
+## Scout hosted beta — 2026-10-15
+
+- Apply shared modality, output-format, tool and operation requirements across Scout recommendation paths. Catalogue refresh now retains explicit text-output contracts and excludes mixed media output from general text candidates.
+- Preserve unknown and partial token prices as `null`; keep genuine advertised zero rates distinct.
+- Suppress separate specialist discovery for endpoint-scoped requests, including empty inventories.
+- Honor latency priority when choosing a model's endpoint, using only timing tied to that route. UI, hourly scheduling and interpretation calls are unchanged. These four service corrections were deployed separately from npm.
+
 ## 0.1.7-preview.2 — 2026-10-15
 
 - Ask for a $10, custom or no-fixed-cap evaluation budget after model scope; save the explicit choice, handle reservations internally, and preserve cumulative usage across reruns and budget changes. Add `eval plan --budget <USD|unlimited>` for coding agents and noninteractive setup.

package/README.md

@@ -22,7 +22,7 @@ Explore the [hosted sample](https://dash.openmerit.ai/) or [before/after dashboa
 
 ## Compare models with Pi
 
-The **npm preview `0.1.7-preview.2`** provides `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose model scope and a $10, custom or no-fixed-cap budget, freeze Scout's shortlist, and compare 3–10 fresh attempts after source changes. Budget and sandbox blocks stay separate from task scores; unknown costs stay unknown. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@0.1.7-preview.2/docs/agent-evaluations.md) for setup, graders, budgets and availability. This is separate from the Pi workflow below.
+The **npm preview `0.1.7-preview.4`** provides `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose model scope and a $10, custom or no-fixed-cap budget, freeze Scout's shortlist, and compare 3–10 fresh attempts after source changes. Budget and sandbox blocks stay separate from task scores; unknown costs stay unknown. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@0.1.7-preview.4/docs/agent-evaluations.md) for setup, graders, budgets and availability. Scout's hosted API and MCP service update independently of the npm package. This is separate from the Pi workflow below.
 
 With Pi 0.87 and a model provider configured:
 

package/docs/agent-evaluations.md

@@ -1,6 +1,6 @@
 # Evaluate CLI changes across coding agents
 
-**npm preview `0.1.7-preview.2`.** Install from the `preview` tag; stable `openmerit@0.1.6` does not include these commands. This evaluates the coding agent with your CLI and documentation. The [Pi workflow](getting-started.md) still evaluates application inference; the [Inference Terminal](inference-terminal.md) remains read-only.
+**npm preview `0.1.7-preview.3`.** Install from the `preview` tag; stable `openmerit@0.1.6` does not include these commands. This evaluates the coding agent with your CLI and documentation. The [Pi workflow](getting-started.md) still evaluates application inference; the [Inference Terminal](inference-terminal.md) remains read-only. Scout's hosted API and MCP service deploy independently; npm installation does not start or bundle the hourly research service.
 
 Compared with `0.1.7-preview.1`, this version adds [corrected cost accounting and blocked-execution handling](#accounting-and-blocked-executions), plus [budget selection after model scope](#budget-selection). It does not claim reconciled provider billing or a guaranteed exact model order.
 

package/docs/commands.md

@@ -2,7 +2,7 @@
 
 The standalone `openmerit dash` command runs in your shell. The `/openmerit` slash commands run inside Pi with the extension loaded.
 
-The npm `0.1.7-preview.2` preview (`openmerit@preview`) provides `openmerit eval init`, `doctor`, `plan`, `run`, `watch` and `report`. See [agent evaluations](agent-evaluations.md) for scope selection, repeated task checks and budgets. These are shell commands, separate from Pi commands.
+The npm `0.1.7-preview.4` preview (`openmerit@preview`) provides `openmerit eval init`, `doctor`, `plan`, `run`, `watch` and `report`. See [agent evaluations](agent-evaluations.md) for scope selection, repeated task checks and budgets. These are shell commands, separate from Pi commands.
 
 Since `0.1.7-preview.2`, `openmerit eval plan [--scope existing|all] [--budget <USD|unlimited>]` asks for a budget after scope. Choose $10, a custom amount, or no fixed cap; saved choices and cumulative usage persist. See [budget selection](agent-evaluations.md#budget-selection).