openmerit 0.1.7-preview.2 → 0.1.7-preview.3

package/CHANGELOG.md

@@ -1,5 +1,56 @@
 # Changelog
 
+## 0.1.7-preview.3 — 2026-10-16
+
+- Update packaged Scout API/MCP and agent-evaluation guides for the sole `/scout/` service and current request, inventory, data-handling and reliability contracts.
+- Keep the hosted hourly research and matching service separate from npm installation. Hosted publication recovery, per-source freshness, bounded Decisions retries and reduced overlapping pool reads were deployed independently.
+- Exclude internal website and deployment maintenance notes from the npm artifact. Keep existing commands, exports, legal metadata and stable `latest` behavior unchanged.
+
+## Scout semantic benchmark interpretation — hosted beta, 2026-10-15
+
+- Replace benchmark capability/name rules and family-name regexes with hourly Decisions interpretation of source definitions and methodology.
+- Cache unchanged interpretations; leave unsupported or failed scopes unclassified and preserve evidence provenance.
+- Keep prompt matching local to prepared benchmark metadata, with unchanged UI, prompt/specialist calls and hourly schedule. All 24 semantic benchmark cases, 107 local/live HTTP/MCP matching cases and 546 automated checks passed. npm publication remains deferred.
+- Store compressed analytics payloads as gzip bytes to avoid base64 overhead on larger evidence records; retain legacy decoding and complete-record size limits.
+
+## Scout semantic specialist matching — hosted beta, 2026-10-15
+
+- Replace Hugging Face domain/language keyword dictionaries and partial label overlap with Decisions compatibility checks.
+- Prepare and cache specialist profiles during the hourly refresh; batch only relevant compatibility checks when a prompt arrives.
+- Require all applicable domain, language and fixed-label checks to pass. Preserve unknown metrics and ordinary results if specialist checking fails.
+- Keep the UI unchanged. All 107 cases passed locally and through live HTTP/MCP; 526 automated tests passed. npm remains deferred.
+
+## Scout semantic formats and SQL interpretation — hosted beta, 2026-10-15
+
+- Remove prompt keyword scans from strict-format and structured-data interpretation; use the existing Decisions answers, including described record layouts and non-English requests.
+- Recognize read-only SQL query writing and repair without requiring schema details in the task description. Keep query source distinct from executing it, its result format, prose explanation and data modification.
+- Keep uncertain format answers visible and exclude them from narrow calibration matching. Preserve response validation, general consistency checks, confidence thresholds, one interpretation request, hourly collection and the UI.
+- All 91 semantic cases passed locally and through live HTTP/MCP, including the previously failing SQL request and four fresh SQL holdouts. All 511 automated tests passed. npm publication remains deferred.
+
+## Scout category and content interpretation — hosted beta, 2026-10-15
+
+- Recognize conversationally supplied categories without requiring a formal label list; keep subtype and individual-identity discovery open-ended.
+- Require document understanding for requested written/table content, including mixed outputs, without inferring it merely from a photographed object.
+- Update two existing semantic questions with unchanged confidence thresholds, UI, hourly refresh and request count. Local and live HTTP/MCP acceptance passed 43 cases; an opt-in semantic audit preserves those cases for future interpretation changes. npm publication remains deferred.
+
+## Scout task interpretation and ambiguous matching — hosted beta, 2026-10-15
+
+- Frame every task description as quoted data for requirement analysis, preserving supplied-label and negation semantics without prompt-specific exceptions or lower confidence thresholds.
+- Keep decision-plus-prose or extraction requests on generative endpoints, reject narrow calibration for uncertain operations/outputs, and ignore weak document-layout guesses for ordinary photos.
+- All 25 routing cases passed locally and through live HTTP/MCP, including the prior binary-label failure. UI, hourly refresh and the single interpretation request are unchanged. npm publication remains deferred.
+
+## Scout identification matching — hosted beta, 2026-10-15
+
+- Distinguish open identification from supplied-label classification within the existing interpretation call. Open or uncertain identification uses evidence-ranked generative endpoints; fixed-choice APIs and label-dependent classifiers require supplied choices.
+- Require relevant visual benchmark evidence for visual requests, preserve classification label scope in calibration matching, and use ordinary output-length estimates for generated names. UI and hourly refresh are unchanged; no npm release is included.
+
+## Scout hosted beta — 2026-10-15
+
+- Apply shared modality, output-format, tool and operation requirements across Scout recommendation paths. Catalogue refresh now retains explicit text-output contracts and excludes mixed media output from general text candidates.
+- Preserve unknown and partial token prices as `null`; keep genuine advertised zero rates distinct.
+- Suppress separate specialist discovery for endpoint-scoped requests, including empty inventories.
+- Honor latency priority when choosing a model's endpoint, using only timing tied to that route. UI, hourly scheduling and interpretation calls are unchanged. These four service corrections were deployed separately from npm.
+
 ## 0.1.7-preview.2 — 2026-10-15
 
 - Ask for a $10, custom or no-fixed-cap evaluation budget after model scope; save the explicit choice, handle reservations internally, and preserve cumulative usage across reruns and budget changes. Add `eval plan --budget <USD|unlimited>` for coding agents and noninteractive setup.

package/README.md

@@ -22,7 +22,7 @@ Explore the [hosted sample](https://dash.openmerit.ai/) or [before/after dashboa
 
 ## Compare models with Pi
 
-The **npm preview `0.1.7-preview.2`** provides `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose model scope and a $10, custom or no-fixed-cap budget, freeze Scout's shortlist, and compare 3–10 fresh attempts after source changes. Budget and sandbox blocks stay separate from task scores; unknown costs stay unknown. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@0.1.7-preview.2/docs/agent-evaluations.md) for setup, graders, budgets and availability. This is separate from the Pi workflow below.
+The **npm preview `0.1.7-preview.3`** provides `openmerit eval` for evaluating Claude Code, Codex and OpenCode with your CLI. Choose model scope and a $10, custom or no-fixed-cap budget, freeze Scout's shortlist, and compare 3–10 fresh attempts after source changes. Budget and sandbox blocks stay separate from task scores; unknown costs stay unknown. Install with `npm install -g openmerit@preview`. See [agent evaluations](https://unpkg.com/openmerit@0.1.7-preview.3/docs/agent-evaluations.md) for setup, graders, budgets and availability. Scout's hosted API and MCP service update independently of the npm package. This is separate from the Pi workflow below.
 
 With Pi 0.87 and a model provider configured:
 

package/docs/agent-evaluations.md

@@ -1,6 +1,6 @@
 # Evaluate CLI changes across coding agents
 
-**npm preview `0.1.7-preview.2`.** Install from the `preview` tag; stable `openmerit@0.1.6` does not include these commands. This evaluates the coding agent with your CLI and documentation. The [Pi workflow](getting-started.md) still evaluates application inference; the [Inference Terminal](inference-terminal.md) remains read-only.
+**npm preview `0.1.7-preview.3`.** Install from the `preview` tag; stable `openmerit@0.1.6` does not include these commands. This evaluates the coding agent with your CLI and documentation. The [Pi workflow](getting-started.md) still evaluates application inference; the [Inference Terminal](inference-terminal.md) remains read-only. Scout's hosted API and MCP service deploy independently; npm installation does not start or bundle the hourly research service.
 
 Compared with `0.1.7-preview.1`, this version adds [corrected cost accounting and blocked-execution handling](#accounting-and-blocked-executions), plus [budget selection after model scope](#budget-selection). It does not claim reconciled provider billing or a guaranteed exact model order.
 

package/docs/commands.md

@@ -2,7 +2,7 @@
 
 The standalone `openmerit dash` command runs in your shell. The `/openmerit` slash commands run inside Pi with the extension loaded.
 
-The npm `0.1.7-preview.2` preview (`openmerit@preview`) provides `openmerit eval init`, `doctor`, `plan`, `run`, `watch` and `report`. See [agent evaluations](agent-evaluations.md) for scope selection, repeated task checks and budgets. These are shell commands, separate from Pi commands.
+The npm `0.1.7-preview.3` preview (`openmerit@preview`) provides `openmerit eval init`, `doctor`, `plan`, `run`, `watch` and `report`. See [agent evaluations](agent-evaluations.md) for scope selection, repeated task checks and budgets. These are shell commands, separate from Pi commands.
 
 Since `0.1.7-preview.2`, `openmerit eval plan [--scope existing|all] [--budget <USD|unlimited>]` asks for a budget after scope. Choose $10, a custom amount, or no fixed cap; saved choices and cumulative usage persist. See [budget selection](agent-evaluations.md#budget-selection).
 

package/docs/scout.md

@@ -1,18 +1,103 @@
 # Scout by OpenMerit
 
-Describe what you need a model to do. Scout returns up to five available models using relevant public benchmark results, task-shaped reference cost, and completion time. The `/scout/` page, JSON API, and MCP tool all use the same index and ranking.
+Describe what you need a model to do. Scout returns up to five model-and-endpoint candidates. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page, JSON API, and MCP tool all use the same index and ranking.
 
 Scout is a separate beta product. It does not change the OpenMerit npm package, Pi coordinator, or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
 
+## Default architecture — hosted beta, 16 October 2026
+
+Scout is served only at `/scout/`. It retains the same prompt and results interface, with one maintained implementation and one hourly workflow. The dated sections below record earlier changes; the current architecture and contract below supersede conflicting historical implementation details. No npm release is included.
+
+The hourly workflow refreshes existing structured feeds and uses Parallel to research public model, provider and benchmark sources. Decisions interprets source-backed methodology, capability definitions, exact identities, deployment contracts and specialist metadata. Each research record requires supporting source excerpts. Scores and prices require their own supported facts; semantic similarity does not establish accuracy or permit scores from different versions to be combined.
+
+Scout maintains separate model, endpoint, benchmark and observation records. It prepares candidate indexes and specialist profiles during refresh, then publishes an immutable catalogue version. Failed research or interpretation retains previously validated records. An incomplete publication cannot replace the active catalogue. Searches continue using the last published version during refresh.
+
+A search uses Decisions to interpret the task and may make a bounded follow-up for weak answers or relevant specialist compatibility. It retrieves a bounded candidate pool from the prepared index, applies endpoint and inventory requirements, and compares applicable evidence locally. There is no request-time web research, benchmark interpretation, model execution or embedding search. Responses retain the existing JSON shape and add `snapshotVersion` and retrieval metadata. The canonical endpoints remain `/scout/api/recommend`, `/scout/api/status` and `/scout/mcp`. Versioned preview pages and API/MCP aliases have been removed.
+
+The hourly trigger does not promise that every public model or source is researched every hour. Known feeds are refreshed each cycle; discovery and enrichment are incremental, bounded and cached. The initial research policy schedules up to 24 jobs per hourly run, including discovery; unchanged subjects are rechecked after seven days. The policy can be changed independently of model names. Published source dates remain visible, and failed or delayed updates do not become fresh measurements.
+
+The initial index retrieves up to 128 benchmark candidates plus the small existing calibration pool. Final ordering still uses comparable observations; the retrieval pool is approximate and can omit a globally stronger candidate. Relevant specialist checks are bounded to 64 prepared profiles. Existing explicit inventories remain restrictive. Unknown quality, prices and times remain unknown. Decisions checks and citations reduce unsupported claims but are not independent reproduction of benchmark results. Request latency includes external Decisions and storage calls and is not guaranteed to be instantaneous.
+
+Parallel receives public catalogue research tasks, not visitor prompts. Decisions receives visitor task descriptions and relevant public metadata. Scout stores public research and versioned catalogues in private Cloudflare R2. Search analytics retain the same prompt-and-evidence policy described below. Provider credentials are server-side secrets. No search launches an hourly research job.
+
+Decisions also identifies the response serialization separately from general structured-output capability. JSON-only calibration cannot supply measured performance for other formats or an uncertain serialization. Field extraction is identified from the required record, independently of the wording used to request it. Uncertain capability guesses contribute no ranking weight, while independently established task contracts still imply their required capabilities. Low-confidence operation guesses cannot add narrow requirements; ambiguous visual separation does not grant a pixel-mask contract merely from its downstream purpose. Numerical comparison and inventory validation remain code. Ranked-pairs cycle checks use incremental reachability, preserving the comparison policy while reducing request-time CPU work.
+
+## Matching corrections — hosted beta, 15 October 2026
+
+The hosted beta includes the following corrections. This update does not publish a new npm package:
+
+- Benchmark, measured and task-fit candidates share the same endpoint requirements for image input, structured output and tools. Reviewed typed endpoints must support the requested operation. General catalogue entries require declared text-only output; text accompanying audio, image or video generation does not establish a general text contract. Mixed-output endpoints need a separately reviewed operation contract. Image inputs with text outputs remain supported. These are metadata eligibility checks, not task-accuracy guarantees.
+- Unknown input/output prices and the compatibility blended price remain `null`, including when only one rate is known. Actual advertised zero rates remain zero. The existing page renders unknown prices with a dash.
+- Either `modelIds` or `endpointIds` suppresses separate specialist discovery, including empty and unknown-only endpoint inventories. Explicitly selected compatible specialist endpoints can still enter a task-fit shortlist.
+- When selecting among routes for the same model, an explicit latency priority compares known endpoint-bound completion times first, then cost. Without comparable timing, known timing precedes unknown timing; if neither route has timing, known cost and stable endpoint IDs decide. Other priorities retain cost-first route selection. Task-fit candidates retain their contract-fit and deployment tiers. Timing is never borrowed from another route.
+
+Output metadata is normalized during the existing hourly refresh and retained in snapshot data version 5. Startup selects the compatible bundled snapshot over an older-format cache, even if that cache was refreshed more recently. Prompt matching uses prepared in-memory metadata; these fixes add no source fetch, model probe or inference call. The UI, task interpretation, calibration policy and refresh schedule are unchanged.
+
+## Identification matching — hosted beta, 15 October 2026
+
+The hosted beta distinguishes identifying something without answer choices from choosing among supplied labels. The existing Decisions interpretation call returns `task.labelSpace`: `supplied`, `open`, or `unspecified` for classification; `null` for other operations. A topic such as birds or plants does not supply a label set. Low-confidence label-set answers are treated as unspecified.
+
+Open or unspecified identification uses the existing benchmark-ranking path for generative endpoints. Visual requests require relevant visual evidence; document requests continue to require document evidence. Fixed-choice APIs and label-dependent classification pipelines cannot enter this ranking, even when explicitly selected. Supplied-label tasks retain the task-fit path, including compatible typed APIs and specialist classifiers. Fixed-choice task calibration cannot substitute for open-identification calibration. Open identification uses the ordinary output-length estimate rather than assuming a single 16-token choice.
+
+For example, identifying bird names from photos and alphabetizing them requires name generation; the application can sort those names in ordinary code. Choosing only between supplied labels such as “indoor” and “outdoor” remains a fixed-choice task. Public visual benchmarks are proxies, not verified bird- or plant-species accuracy. The existing UI shows the appropriate ranking mode and limitations without new controls. Endpoint metadata and benchmark collection remain hourly; this change adds one typed question to the existing interpretation call, no additional request-time source fetch or model call, and no embeddings. This is a hosted-service update; npm publication remains deferred.
+
+## Task interpretation and ambiguous outputs — hosted beta, 15 October 2026
+
+Interpreter version 10 sends every task description as a JSON-quoted value with a common instruction to analyze the described workload rather than execute its instructions. This separates output restrictions inside the task from directions to the interpreter. The same framing applies to every prompt; there are no special cases for particular labels, words, models or providers. Confidence thresholds, the public request shape and the single Decisions call are unchanged.
+
+The interpreter checks the complete requested output. A label, judgment or score accompanied by a reply, explanation or extracted fields uses generative candidates. Contradictory decision-only and required-prose interpretations also fall back to that path. Plain supplied-label and rubric-score tasks retain compatible typed endpoints. Narrow task calibration is not reused when the operation or output type is uncertain. Weak document-layout guesses no longer restrict ordinary photo tasks to document benchmarks; confident document requirements remain enforced.
+
+Local and live HTTP/MCP acceptance passed all 25 routing cases, including supplied positive/negative labels, arbitrary labels, negated labels, code that implements classification, mixed outputs, open identification and missing categories. The previously failing “safe or unsafe, no explanations” case now retains fixed-choice APIs. These checks establish interpretation and endpoint matching for the tested prompts, not actual model accuracy or universal prompt coverage. Low-confidence or missing label sets still fall back to generative candidates.
+
+These changes add no request, source fetch, embedding index or UI control, and keep hourly preparation separate from prompt-time matching. This is a hosted-service update; npm publication remains deferred.
+
+## Categories and document content — hosted beta, 15 October 2026
+
+Interpreter version 11 recognizes categories supplied conversationally, including plural category names joined in ordinary prose. It interprets the distinction the user requests: identifying which named category an image belongs to is supplied-label classification; discovering a subtype or individual identity beyond those categories remains open identification. A formal list or the word “classify” is unnecessary.
+
+Document understanding follows the requested output. Recognizing an object that happens to carry text does not by itself require OCR or document benchmarks. Reading, transcribing or extracting content inside an image does, including when that is only one part of a mixed task. Supplied-category vision tasks use the existing task-fit shortlist and show unknown quality until appropriate measurements exist.
+
+The change updates two questions in the existing interpretation request. It adds no object-name rules, model preferences, services, dependencies or inference calls. Confidence thresholds, hourly preparation and the UI are unchanged. Local and live HTTP/MCP semantic acceptance passed 43 cases, including classification/reading contrasts and the prior 25-case suite. These are routing checks, not executions of the recommended models or a guarantee for every possible prompt. This is a hosted-service update; npm publication remains deferred.
+
+## Semantic formats and SQL interpretation — hosted beta, 15 October 2026
+
+Interpreter version 13 includes removal of the English keyword scans for strict output formats and structured-data capability. The existing Decisions questions now supply those judgments directly: a prescribed record or delimited layout for software to parse can require structured output even without a named format. The capability question distinguishes extraction and categorization from merely organizing prose or reading already formatted input. The same 21 questions run in one request; no UI control, source fetch, new dependency or additional inference call is added. Hourly catalogue preparation is unchanged.
+
+The response decoder consumes typed answers without rereading the prompt. It retains response validation and general capability-consistency rules. The existing 0.4 confidence threshold now also guards structured-data capability and strict-format requirements. A low-confidence format answer in either direction remains marked uncertain, does not impose a hard format requirement, and cannot match narrow calibration evidence. This threshold is a heuristic, not a calibrated probability of correctness. Semantic interpretation can still misread a request, including unfamiliar formats or languages; catalogue format support is not proof that a model will satisfy an exact output schema.
+
+The operation question recognizes writing, revising or debugging a read-only SQL query even when concrete schema details are omitted from the task description. Those details can be supplied when the model performs the task. Expressions that produce records or serialized values remain query-writing work; they do not make the model's answer a serialized data result. Executing a query, modifying stored data, explaining a query and generating another program remain distinct operations. This changes the general Decisions definition, with no query-text matching, function-name exceptions, dialect lookup or extra question.
+
+This hosted update combines the format changes with SQL interpretation. Real Decisions interpretation passed all 91 cases locally and through live HTTP/MCP: the prior 75 cases, including the previously failing SQL request, twelve additional SQL boundaries and four fresh holdouts. Tests include query source versus serialized output, omitted/runtime schema details, recursive queries, execution loops, data modifications and multilingual requests. npm publication remains deferred. Semantic checks establish tested interpretation and endpoint matching, not query execution, exact-schema compliance or universal prompt accuracy.
+
+## Semantic specialist matching — hosted beta, 15 October 2026
+
+Interpreter version 14 and snapshot data version 6 replace Hugging Face prompt keyword dictionaries, language-name lookup and partial label overlap with Decisions compatibility checks. The hourly refresh prepares each repository's specialist operation and subject scope from public metadata, caches that analysis by revision and relevant metadata, and groups equivalent profiles. Changed repositories are analyzed in batches of up to twenty; unchanged profiles are reused. Preparation uses the server's Decisions API key and incurs provider usage. If preparation fails, Scout retains the previous complete specialist snapshot and its original timestamp while other sources can update.
+
+A search still starts with the existing 21-question task interpretation. When eligible Hub profiles are relevant, a further batched Decisions request checks their subject, explicit language requirements and complete fixed-label coverage. Shared checks are evaluated once; all applicable checks must pass with the existing 0.4 confidence threshold. Requests contain at most 180 compatibility checks plus one language-requirement question; larger sets are split into bounded batches. Open identification, unsupported work, tool/memory tasks and inventories without eligible Hub routes skip this work. There are no repository fetches, model executions or per-repository calls during matching. A failed compatibility request omits Hub candidates for that search and leaves ordinary eligible results available with a warning. There is no keyword fallback.
+
+Compatibility is eligibility, not predicted accuracy. Scout still enforces endpoint contracts, preserves unknown quality/cost/time, and orders discovery components by metadata completeness and diversity. Metadata and semantic interpretation can be wrong, and the existing bounded catalogue can omit suitable specialists. The UI and hourly schedule are unchanged. This revision is deployed on the hosted beta; npm publication remains deferred. All 107 semantic cases passed locally and through live HTTP/MCP: the prior 91 regression cases plus twelve specialist cases and four fresh holdouts. They check domain paraphrases, background context, negation, language requirements, multilingual instructions, full versus partial fixed-label coverage and open identification. These are interpretation and matching checks, not executions of recommended models.
+
+## Semantic benchmark interpretation — hosted beta, 15 October 2026
+
+Snapshot data version 7 prepares benchmark capabilities and correlated benchmark families with Decisions during the hourly refresh. It replaces benchmark-name regexes, manually assigned capability tags and the automatic reasoning fallback. Source adapters retain metric identifiers, score units and parsing contracts. Epoch descriptions and methodology are read from its published catalogue; Artificial Analysis and BFCL methodology are read from their public pages. Unavailable definitions remain explicit limitations, and names alone do not establish an opaque benchmark's capabilities.
+
+Each changed benchmark receives an isolated interpretation request so another benchmark's description cannot supply its capabilities. For shared publisher methodology, a preceding Decisions request selects the directly applicable definition for that benchmark and evaluated subset; unrelated sections are omitted from capability interpretation. Publisher-declared suite identities are retained as source facts; other family relationships are interpreted by Decisions. Capability choices and family references are validated, and positive choices below the existing 0.4 confidence threshold are not used. Family links group related evidence; numerical comparisons still require the exact same benchmark series. Unknown scopes receive no capability tags and cannot contribute capability-ranking evidence. This is semantic metadata interpretation, not a measurement of model quality or a guarantee that the classifications are correct.
+
+Interpretations are cached by definition, relevant publisher methodology, family inventory and interpreter version. Score, model, price and collection-time changes alone do not invalidate that cache. Family inventory changes trigger rechecking its references. If an interpretation fails, unchanged cached records remain usable, affected new or changed series stay unclassified, and source failure status is exposed until a later refresh succeeds. A methodology download failure retains the prior methodology when available and marks its source degraded. There is no keyword fallback.
+
+Searches read the prepared tags and family IDs without fetching methodology or making benchmark interpretation calls. The existing prompt interpretation and optional specialist compatibility calls remain. `benchmarkInterpretation` in status/recommendation responses reports prepared, classified, unclassified and unavailable series counts; each returned benchmark observation includes its capability tags, family and interpretation version/time. The UI and hourly schedule are unchanged. npm publication remains deferred.
+
+The initial deployed snapshot processed all 146 benchmark series: 136 have supported capability tags and ten remain unclassified. Individual weak capability judgments are also omitted, even when other capabilities for that series are supported. A repeat preparation with unchanged metadata made zero Decisions calls. All 24 semantic benchmark cases and 107 task-interpretation/matching cases passed both locally and through live HTTP/MCP, alongside 546 automated checks. These checks cover interpretation, caching, source failure and matching; they do not execute recommended models or establish universal semantic accuracy.
+
 ## Use Scout
 
-Open [Scout](https://openmerit.ai/scout/), type a task, and select the arrow button (**Find models**) or press Enter. For example: “I want to upload an invoice and get the invoice number, date, vendor, subtotal, tax, and total as JSON.” The task field grows and shrinks as you type or paste. Very long prompts scroll within the field once it reaches its height limit. Shift + Enter adds a line. When the field is empty and unfocused, its placeholder cycles through detailed example tasks every ten seconds with a brief fade. Rotation pauses while you type, while the field is focused, or while the page is hidden. With reduced motion enabled, the example stays still. Placeholders never fill the input or send a request. The heading shows live counts of indexed models, model providers, benchmark series and source feeds. Provider aliases are counted once; these are model creators, not every hosting route. The evaluation guide remains available in the fixed bottom footer, with space reserved beneath the page content.
+Open [Scout](https://openmerit.ai/scout/), type a task, and select the arrow button (**Find models**) or press Enter. For example: “I want to upload an invoice and get the invoice number, date, vendor, subtotal, tax, and total as JSON.” The task field grows and shrinks as you type or paste. Very long prompts scroll within the field once it reaches its height limit. Shift + Enter adds a line. When the field is empty and unfocused, its placeholder cycles through detailed example tasks every ten seconds with a brief fade. Rotation pauses while you type, while the field is focused, or while the page is hidden. With reduced motion enabled, the example stays still. Placeholders never fill the input or send a request. The heading shows live counts of API catalogue models, Hugging Face repositories, benchmark series and sources. API identities and Hub repositories are separate counts and can overlap; neither count promises a runnable endpoint for every entry. The evaluation guide remains available in the fixed bottom footer, with space reserved beneath the page content.
 
-The comparison table keeps recommendations in Scout rank order. When a matching measured task profile is available, it shows calibration accuracy, mean reported cost per 1,000 similar tasks, and median full request time. Otherwise it shows relative benchmark quality and reference cost/time estimates. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; missing marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **API & MCP**, **Capabilities**, and **Sources** expand beneath the search and results. Collection times and refresh failures are shown inside **Sources**.
+The comparison table keeps recommendations in Scout rank order. When a matching measured task profile is available, it shows calibration accuracy, mean reported cost per 1,000 similar tasks, and median full request time. Benchmark mode shows relative benchmark quality and reference cost/time estimates. **Candidates to evaluate** denotes task-fit mode: quality is shown as unknown, alongside known reference costs and deployment requirements. When image separation is ambiguous, Scout first offers labels, bounding boxes or pixel masks/cutouts as output choices. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; Hugging Face results resolve the repository publisher, using a known lab mark or its bundled organization avatar; otherwise a Hugging Face mark identifies the repository platform. Missing or failed marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **API & MCP**, **Capabilities**, and **Sources** expand beneath the search and results. **Sources** uses a compact list; expand a source for its coverage, methods, limits, collection times and refresh failures.
 
 The capability index lists each area with its benchmark-series and model counts. After a search, task matches appear first with a check mark. Counts describe index coverage, not the number of models eligible for your task. The source panel describes what each feed measures, its coverage, source dates, archived results, and limitations. Collection time is separate from source dates, which can be publication dates rather than measurement dates. Scout task calibration appears separately with its tested pool, scope and seven-day expiry.
 
-**Provider page** in each model's evidence opens its provider's official pricing, model documentation, or website. Model links never fall back to a router. Token prices shown in Scout are still quoted via OpenRouter and may differ from direct provider rates. Benchmark links lead to the original evaluation sources.
+**Provider page** in each model's evidence opens its provider's official pricing, model documentation, or website. Model links never fall back to a router. Each quote identifies its source: a router route, independent provider metadata, or a reviewed provider contract. Routes have separate prices. Self-hosted infrastructure cost and incomplete quotes remain unknown. Hugging Face deployment links open the publisher’s model repository and pinned card. Benchmark links lead to the original evaluation sources.
 
 ## Connect a coding agent
 
@@ -44,19 +129,21 @@ curl https://openmerit.ai/scout/api/recommend \
 
 `POST /scout/api/recommend` requires `prompt`: a non-empty string of at most 4,000 characters. The JSON body is limited to 18,000 bytes. Unknown properties are rejected.
 
-HTTP and MCP additionally accept optional `modelIds`, a unique list of up to 100 catalogue IDs. This filters eligible candidates before ranking; an empty or unknown-only list returns no models. The response echoes `selection.modelIds` (`null` for all). Scoped requests omit specialist discovery because it is outside the declared inventory. No endpoint URL or credential is accepted. The [agent evaluation runner](agent-evaluations.md) uses this for existing-project scope; the hosted API supports it as of 15 October 2026. The npm evaluation commands are available in `openmerit@0.1.7-preview.1` under the `preview` tag, separate from stable `0.1.6`.
+HTTP and MCP additionally accept optional `modelIds`, a unique list of up to 100 legacy OpenRouter catalogue IDs. This preserves the existing-project inventory contract; a model ID alone does not grant a direct-provider endpoint. To select specific routes instead, use `endpointIds`, a unique list of up to 100 IDs returned in `recommendations[].endpoint.id`. Both filters intersect if supplied. This filters eligible candidates before ranking; an empty or unknown-only list returns no models. The response echoes `selection.modelIds` and `selection.endpointIds` (`null` for unscoped). Scoped requests omit specialist discovery because it is outside the declared inventory. No endpoint URL or credential is accepted. The [agent evaluation runner](agent-evaluations.md) uses this for existing-project scope; the hosted API supports it as of 15 October 2026. The npm evaluation commands are available in `openmerit@0.1.7-preview.3` under the `preview` tag, separate from stable `0.1.6`.
+
+Recommendation `id` preserves legacy router IDs; a new protocol receives a distinct catalogue ID. `modelId` is the canonical model identity, while `endpoint.modelId` is the identifier to send to that provider. Clients must use the returned protocol and deployment requirements rather than treating every option as a chat-completions model. Existing npm clients using legacy `modelIds` remain scoped to their router inventory.
 
 The response contains:
 
-- `recommendations`: up to five distinct model IDs with rank, provider, reason, quality signal, evidence coverage, input/output pricing, reference timing when available, supported inputs, and source-linked evidence. `reference` gives cost in USD and completion seconds (`null` when unavailable). Benchmark mode includes assumed input/output tokens; measured mode sets those token counts to `null` and supplies `measured` with observed field accuracy, the reliability-capped `effectiveQuality`, cross-task `availability`, valid outputs, sample size, charge coverage, provider routes and evidence provenance. `comparison` gives the number of comparable candidates and average shared-capability coverage. The compatibility `pricing.blendedPerMillion` field retains its old 3:1 calculation but is no longer scored.
-- `task`: interpreted capabilities and weights; image-input, structured-output, tool-calling and persistent-memory requirements; `requirements.tools` (`yes`, `no`, `unspecified`); `requirements.memory` (`none`, `session`, `persistent`, `unspecified`); consequential `uncertainty`; `workload.input` and `workload.output` (`short`, `medium`, `long`), `workload.priority` (`balanced`, `quality`, `cost`, `latency`), `kind` (`extract`, `classify`, `dispatch`, `code`, `act`, `remember`, `sql`, `calculate`, `retrieve`, `other`), and `interpreter: "openai-decisions"`. `keywords` is retained as an empty compatibility field. A general-purpose fallback means no specific capability was identified, not that the classifier was unavailable.
-- `specialists`: a separate discovery object with up to three Hub repositories, metadata-based task fit, declared licenses, pinned model-card revisions, live hosting metadata when supplied, and original reported evaluation records. `mode` is `discovery`; quality, cost and latency are `null`. These candidates are not additional ranked places. Unsupported task types can omit this field.
+- `recommendations`: up to five distinct model IDs with rank, provider, `endpoint` (provider model ID, protocol, configuration, deployment, provenance and limits), reason, quality signal, evidence coverage, input/output pricing, reference timing when available, supported inputs, and source-linked evidence. `reference` gives cost in USD and completion seconds (`null` when unavailable). Benchmark mode includes assumed input/output tokens; measured mode sets those token counts to `null` and supplies `measured` with observed field accuracy, the reliability-capped `effectiveQuality`, cross-task `availability`, valid outputs, sample size, charge coverage, provider routes and evidence provenance. `comparison` gives the number of comparable candidates and average shared-capability coverage. The compatibility `pricing.blendedPerMillion` field retains its old 3:1 calculation but is no longer scored.
+- `task`: interpreted capabilities and weights; image-input, structured-output, tool-calling and persistent-memory requirements; `requirements.tools` (`yes`, `no`, `unspecified`); `requirements.memory` (`none`, `session`, `persistent`, `unspecified`); consequential `uncertainty`; `workload.input` and `workload.output` (`short`, `medium`, `long`), `workload.priority` (`balanced`, `quality`, `cost`, `latency`), `kind` (`extract`, `classify`, `dispatch`, `code`, `act`, `remember`, `sql`, `calculate`, `retrieve`, `judge`, `detect`, `segment`, `visual_unspecified`, `other`), `outputContract` (labels, decision, boxes, masks, text, or unspecified), `outputFormat` (`json`, `other`, `none`, or `unspecified`; only specifically established formats can match format-specific calibration), `outputCardinality`, optional `clarification`, and `interpreter: "openai-decisions"`. `keywords` is retained as an empty compatibility field. A general-purpose fallback means no specific capability was identified, not that the classifier was unavailable.
+- `specialists`: a separate discovery object with up to three Hub repositories, metadata-based task fit, declared licenses, pinned model-card revisions, live hosting metadata when supplied, and original reported evaluation records. `mode` is `discovery`; quality, cost and latency are `null`. These candidates are not additional ranked places. Compatible vision deployments may instead enter a task-fit shortlist; they are not counted twice. Unsupported task types can omit this field.
 - `specialistRepositories`: bounded Hub catalogue count, separate from the evidence-backed `models` count.
 - `calibration`: released task profiles, scopes, sample sizes, measurement dates and expiry status.
-- `checkedAt`, `stale`, `degraded`, `sources`: index and source health, feed purpose, limitations and coverage. Each source has its own last successful check. A failed source is marked degraded; any source unchecked for more than two hours makes the index stale.
-- `benchmarks`, `benchmarksWithResults`, `models`, `providers`, `availableModels`, `eligibleModels`: indexed series, series with matched results, evidence-backed models, distinct indexed model providers (including alias consolidation), catalogue models, and eligible candidates for this task.
+- `checkedAt`, `stale`, `degraded`, `sources`: index and source health, feed purpose, limitations and coverage. Each source has its own last successful check. A failed source is marked degraded; any hourly source unchecked for more than two hours makes the index stale. Versioned endpoint contracts disclose their manual review date separately; a scheduled metadata refresh does not renew those price claims.
+- `benchmarks`, `benchmarksWithResults`, `models`, `providers`, `availableModels`, `eligibleModels`: indexed series, series with matched results, evidence-backed models, distinct indexed model providers (including alias consolidation), catalogue models, and eligible candidates for this task. `endpoints` counts hosted route listings, including reviewed typed contracts; `independentModels` counts catalogue identities with metadata independent of OpenRouter. Neither number implies credentials or verified account access.
 - `wordCloud`: capability vocabulary with benchmark and model coverage counts.
-- `ranking`: algorithm version, `mode` (`measured` or `benchmarks`), workload and tradeoff weights. A measured `profile` gives the task type, sample size, eligible tested pool, protocol hash, collection and expiry times, and public evidence URL. `benchmarkEligibleModels` reports the separate public-benchmark pool. Rank is the final order; the aggregate `score` is a supporting signal and can be non-monotonic because stronger direct comparisons take precedence.
+- `ranking`: algorithm version, `mode` (`measured`, `benchmarks`, `task-fit`, or `clarification`), workload and tradeoff weights. A measured `profile` gives the task type, sample size, eligible tested pool, protocol hash, collection and expiry times, and public evidence URL. `benchmarkEligibleModels` reports the separate public-benchmark pool. Rank is the final order; the aggregate `score` is a supporting signal and can be non-monotonic because stronger direct comparisons take precedence.
 - `warnings`, `methodology`, `interpretationMs`, `elapsedMs`: limitations, ranking method, Decisions request time, and measured server ranking time when the runtime provides a sufficiently precise clock (`null` otherwise). Cloudflare freezes its clock during synchronous work, so the hosted page reports measured HTTP round-trip time instead of claiming zero-millisecond computation.
 
 `pricing.url` is the official provider destination, not the source of the quoted rates. It is `null` for a provider without a reviewed destination; the page then omits the link. The same rule applies to MCP responses. `pricing.basis` identifies the price source and calculation assumptions.
@@ -67,7 +154,7 @@ Scout recommends models; it does not process an uploaded invoice, execute a task
 
 ## Real sources and their limits
 
-The hourly updater fetches seven public sources (six existing feeds and eight bounded Hugging Face queries) and prepares the index before requests arrive. Queries make one Decisions call for intent classification; they do not browse benchmark sites or ask a model to invent rankings.
+The hourly updater fetches eight independent public sources, including Models.dev and fourteen bounded Hugging Face pipeline queries. Two additional versioned provider-contract sources have explicit manual review dates. It prepares the index before requests arrive. Queries use batched Decisions questions for intent classification, a bounded verification pass for weak answers, and relevant specialist compatibility checks; they do not browse benchmark sites or ask a model to invent rankings.
 
 | Feed | Used for |
 | --- | --- |
@@ -77,9 +164,12 @@ The hourly updater fetches seven public sources (six existing feeds and eight bo
 | [OpenCompass / VLMEvalKit](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) | Visual understanding and OCR benchmark results. Its feed currently identifies itself as 17 September 2025. |
 | [Berkeley Function Calling Leaderboard (BFCL v4)](https://gorilla.cs.berkeley.edu/leaderboard) | Native function-calling single-turn, live, and multi-turn scores; separate memory scores for key-value, vector, and recursive-summarization stores. |
 | [OpenRouter](https://openrouter.ai/api/v1/models) | Available model IDs, input modalities, declared tools and structured-output support, context limits, and advertised token prices. |
+| [Models.dev](https://models.dev) | Independent native model IDs, declared capabilities and prices via ten direct-provider adapters: OpenAI, Anthropic, Google, DeepSeek, Mistral, xAI, Cohere, Moonshot, MiniMax and Z.AI. Not a performance benchmark or account-availability check. |
+| [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions) | Reviewed typed-choice, predicate and score endpoint contract, inputs and endpoint-specific rates. No general chat timing is inherited. |
+| [TypeSafe Jev](https://typesafe.ai/blog/jev-1-14) | Reviewed System One contract. Image and single-box output are experimental; headers, version and incomplete total pricing are disclosed. Jev is a candidate for compatible tasks, not Scout’s ranking engine. |
 | [Hugging Face](https://huggingface.co/models) | Public repository metadata, model-card results and structured evaluation records for specialist discovery. Separate from performance ranking. |
 
-Over 100 benchmark **series** are indexed, including source-specific implementations and versions. This is not 100 independent publishers, and coverage is not uniformly current. Some definitions have no results matched to an available model. Both counts are exposed. Missing metrics and ambiguous or unmatched model versions are omitted. Ranked availability is limited to OpenRouter's text-output catalogue, excluding batch/free variants and known routers. Specialist discovery also indexes repositories outside that catalogue.
+Over 100 benchmark **series** are indexed, including source-specific implementations and versions. This is not 100 independent publishers, and coverage is not uniformly current. Some definitions have no results matched to an available model. Both counts are exposed. Missing metrics and ambiguous or unmatched model versions are omitted. Model discovery merges OpenRouter with independent native-provider records before matching public benchmark results. An OpenRouter outage does not prevent an independent catalogue from bootstrapping. Distinct endpoints retain prices and parameters; duplicate hosting entries do not create additional model votes or consume multiple places in the five. Endpoint listings are metadata, not verified access in the caller’s account.
 
 Collection time is not measurement time. A successful hourly check does not make an old result new. Some feeds omit measurement dates; those remain unknown. Model release dates are never substituted. OmniDocBench's repository version and leaderboard version differ; Scout uses the version actually printed on the score table. Document parsing is a proxy for invoice extraction, not an evaluation of your exact six fields. New models without comparable evidence may be absent from a task's shortlist.
 
@@ -89,29 +179,31 @@ Epoch AI attribution: “Capabilities & benchmarking,” published at epoch.ai,
 
 Some specialist models have useful evaluations outside general LLM leaderboards. Scout reads both legacy model-card `model-index` results and the Hub’s [structured evaluation records](https://huggingface.co/docs/hub/eval-results). It retains the original dataset, task or metric, units, configuration, split, reported date, setup notes and source link when supplied. Scores are not averaged across unrelated datasets. A repository revision pins the metadata collected; it does not establish which checkpoint produced every reported score. Reported dates can also default to a commit date. Scout does not validate verification tokens or independently reproduce these scores.
 
-The current hourly discovery sample takes up to 30 repositories sorted by downloads for each of eight pipelines: document question answering, token classification, text classification, sentence similarity, text-to-text generation, text generation, image-to-text, and image-text-to-text. The two image queries are filtered to OCR. Duplicates, private or disabled repositories, adapters and quantization packages are excluded; fine-tunes remain separate. Downloads determine sampling coverage only, never quality. This is a bounded sample, not the whole Hub, and can miss low-download specialists. The initial snapshot contains 175 repositories, 49 with structured reported results; live counts change as sources refresh.
+The hourly discovery sample takes up to 30 repositories sorted by downloads for each of fourteen pipelines: document question answering, token classification, text classification, sentence similarity, text-to-text generation, text generation, image-to-text, image-text-to-text, image classification, zero-shot image classification, object detection, zero-shot object detection, image segmentation and mask generation. The two image-to-text queries remain OCR-filtered. Bounded pinned configs add label vocabularies for up to six repositories in each fixed-label vision pipeline. Private/disabled repositories, adapters and quantization packages are excluded; fine-tunes remain separate. Downloads select coverage only, never quality. This bounded sample can miss capable low-download specialists.
 
-When metadata matches the task, **Specialist models** expands beneath the five recommendations. Up to three matches cover different components, with task terms, metadata completeness and component diversity determining discovery order. Document OCR requires document-related metadata; incidental words such as “object” in “JSON object” do not select object detectors. Explicit supported-language declarations and known domain tags constrain matching, but metadata can still be incomplete or wrong. Tool execution and persistent-memory tasks do not receive unvalidated specialist matches. Entity taggers, OCR and embedding models are labelled as components requiring integration, not interchangeable chat APIs. Repeated repository names are de-emphasized for display diversity without merging their identities.
+For document, code and retrieval components, **Specialist models** expands beneath the five recommendations when compatible prepared profiles match. For classification, detection and segmentation, compatible native pipelines may enter the five **Candidates to evaluate** instead. Fixed-label pipelines need complete semantic coverage of the requested categories; one overlapping label is insufficient. Zero-shot pipelines accept supplied labels; masks can require points or boxes. Pipeline metadata is not an accuracy guarantee. Up to three components are ordered by metadata completeness and component diversity after compatibility checks. Domain and language requirements come from Decisions, with no prompt keyword dictionary. Tool execution and persistent-memory tasks do not receive unvalidated specialist matches. Entity taggers, OCR and embedding models are components requiring integration, not interchangeable chat APIs. Repeated repository names are de-emphasized for display diversity without merging their identities. See the release status in [semantic specialist matching](#semantic-specialist-matching-hosted-beta-15-october-2026).
 
 Cards show declared licenses, access gates, language, library, parameter count when known, and live hosting routes listed by the Hub. Missing hosting metadata does not prove that no provider exists. Size is not a hardware-fit estimate. Public weights do not establish an open-source license or permission for a particular use; check the pinned model card and actual license. Scout neither downloads weights nor executes repository code, and it does not send task descriptions to Hugging Face.
 
-Missing evaluations mean **unknown**, not poor performance. Discovery models do not enter the five through download counts or self-reported scores. Promotion is not automated: it requires a runnable model or endpoint and comparable task checks, full workload cost and timing under a recorded deployment configuration. Self-hosted cost depends on hardware and utilization; a zero token price would be misleading. The same quality/cost/speed comparison can then be used without a provider preference. Model-specific exceptions and inferred free inference are not used.
+Missing evaluations mean **unknown**, not poor performance. Task-fit candidates do not receive quality scores through download counts, self-reported results, or general reasoning benchmarks. A performance ranking requires comparable task checks, full workload cost and timing under a recorded deployment configuration. Self-hosted cost depends on hardware and utilization; a zero token price would be misleading. The same quality/cost/speed comparison can then be used without a provider preference. Model-specific exceptions and inferred free inference are not used.
 
 ## Ranking
 
-One server-side call to [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions), using `gpt-6-luna`, interprets the work the selected model must perform. It distinguishes executing tools from writing code about them, incidental words such as “tool” or “agent,” supplied plain text versus visual inputs, and session state versus persistent memory. Eighteen typed choices become capability weights, requirements, reference input/output sizes, a tradeoff priority, and a narrow task operation. One function dispatch is distinct from a dependent action loop. Typed code, action, and external-memory operations keep the independent capability answers coherent: an action or memory-tool operation cannot silently become a reasoning-only task. SQL generation counts as coding. Required extraction is not downweighted merely because reasoning is marked useful. Long-context requirements imply a long reference input. Visual input uses at least the medium category, even when the task description is short. Decisions sees the task, not candidate names or rankings; the shortlist still comes from collected scores.
+The initial server-side call to [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions), using `gpt-6-luna`, interprets the work the selected model must perform. It distinguishes executing tools from writing code about them, incidental words such as “tool” or “agent,” supplied plain text versus visual inputs, and session state versus persistent memory. Typed choices from the current capability registry and application contract become capability weights, requirements, reference input/output sizes, a tradeoff priority, a narrow task operation and the required output serialization. One function dispatch is distinct from a dependent action loop. Typed code, action, and external-memory operations keep the independent capability answers coherent: an action or memory-tool operation cannot silently become a reasoning-only task. SQL generation counts as coding. Required extraction is not downweighted merely because reasoning is marked useful. Long-context requirements imply a long reference input. Visual input uses at least the medium category, even when the task description is short. Initial task interpretation sees the task without candidate names or rankings. The separate specialist compatibility stage sees prepared public metadata restrictions; it does not assign quality scores. The shortlist uses collected endpoint contracts and evidence.
 
-Strong JSON/schema filtering also needs explicit format wording in the prompt. An uncertain format answer cannot turn an ordinary story or comparison into a JSON task. Low-confidence consequential classifications and unspecified retention/tool interfaces are exposed as uncertainty. This is still a probabilistic classifier: paraphrases can differ, ambiguous requests can be misread, and English format-grounding rules limit multilingual handling. Scout has no clarification dialogue or user-defined constraints yet.
+Strong JSON/schema filtering follows the semantic format answer and its confidence check, without rereading the prompt for keywords. Uncertain format judgments remain disclosed and cannot inherit narrow calibration evidence. Low-confidence consequential classifications and unspecified retention/tool interfaces are exposed as uncertainty. This is still a probabilistic classifier: paraphrases can differ, ambiguous requests can be misread, and unfamiliar languages or formats can be misinterpreted. Ambiguous visual separation returns `task.clarification`; HTTP/MCP clients should ask the output question, and the page offers three explicit choices. Model and endpoint inventories are the only supported selection constraints.
 
-Image tasks require an image-capable model. JSON/schema tasks require declared `response_format` or `structured_outputs` support. Visual document tasks require document-specific evidence. Required tool use needs declared `tools` support and relevant tool-calling evidence. A matching measured task profile can supply this evidence; the benchmark fallback requires published results for the capability. Persistent memory needs declared tools support and dedicated memory evidence; generic reasoning or a large context window cannot substitute. Session-only state does not trigger this filter. Recognized image/video/audio generation and transcription tasks return no recommendations because those outputs are outside the current index.
+Image tasks require an image-capable model and relevant visual evidence in benchmark mode. JSON/schema tasks require declared `response_format` or `structured_outputs` support. Visual document tasks require document-specific evidence. Required tool use needs declared `tools` support and relevant tool-calling evidence. A matching measured task profile can supply this evidence; the benchmark fallback requires published results for the capability. Persistent memory needs declared tools support and dedicated memory evidence; generic reasoning or a large context window cannot substitute. Session-only state does not trigger this filter. Recognized image/video/audio generation and transcription tasks return no recommendations because those outputs are outside the current index.
 
 BFCL contributes only native function-calling (`FC`) configurations, preserving their published labels. Its six correlated series count as one benchmark family, not six independent sources. Missing scores stay unknown and zero remains a score. Model versions are matched conservatively; ambiguous versions are omitted. BFCL does not supply per-model measurement dates, so these remain unknown.
 
 The [BFCL memory evaluations](https://gorilla.cs.berkeley.edu/blogs/16_bfcl_v4_memory.html) test using external memory stores across sessions. They do **not** establish built-in model memory, automatic retention in your app, or performance on your storage system. The application still implements storage and tool execution. Without a matching measured profile, models without matched memory results are excluded when persistent memory is required, including newer models with other strong scores.
 
-Ranking version 4 first checks for a matching, recent **measured task profile**. The static evidence file contains only calibration aggregates: mean per-case task-check accuracy, mean reported charge, median full task time, valid-output counts, provider routes, and provenance. Malformed schemas and request failures receive zero accuracy; failure time remains in the latency median. Unknown charges are excluded from the cost mean and the known-charge count is disclosed. A model with no known charges earns no cost advantage. For multi-call episodes, reported cost includes every call and elapsed time covers the whole episode, including reasoning, tool round trips, and all memory sessions. Request reliability is shared across calibration tasks: after two or more failed requests, the observed HTTP success rate caps the model’s task-quality signal. A clean task slice cannot erase repeated route failures elsewhere. One isolated failure remains in its own task score rather than penalizing every other task. This is route evidence from the evaluation period, not a promise about current provider availability.
+Ranking version 6 routes supplied-label classification, yes/no or rubric judgments, boxes and masks through endpoint contracts first. Open or unspecified identification uses the evidence-based path for generative endpoints. Native contracts, deployment readiness (hosted before self-hosted within a fit tier), known reference cost, then known speed determine a **task-fit** shortlist. Unknown quality stays `null`; it is not presented as a proven quality/cost/speed optimum. Fixed typed endpoints cannot write arbitrary code or extract arbitrary JSON fields. Single-box endpoints cannot silently satisfy all-instance detection. Supplied-label choices use a 16-token output reference; Decisions has no output-token charge. Self-hosted cost is not inferred from parameter count, downloads, or a zero token rate.
 
-Profiles cover small text-field extraction, rule-based classification, single-function dispatch, read-only SQL, short numerical problems, long-text record retrieval, visual-field extraction, dependent tool actions, and external memory across sessions. The matcher also supports code-function operations, but their proposed measured profile and an additional medium-input extraction profile were withheld after fresh validation regressions; those tasks retain their prior evidence path. Matching requires the same typed operation, image/tool/schema requirements, and input/output size category. A new required capability outside the profile prevents a match. Retention scope (none, session, persistent) and whether execution uses an action loop must also match; tool dispatch, dependent actions, and cross-session memory are separate operations. At least five currently eligible tested models and six calibration cases per model are required. Availability and declared capabilities still come from the refreshed catalogue. Evidence expires after seven days; expired, undersampled and unmatched profiles fall back to public benchmarks.
+Other tasks first check for a matching, recent **measured task profile**. The static evidence file contains only calibration aggregates: mean per-case task-check accuracy, mean reported charge, median full task time, valid-output counts, provider routes, and provenance. Malformed schemas and request failures receive zero accuracy; failure time remains in the latency median. Unknown charges are excluded from the cost mean and the known-charge count is disclosed. A model with no known charges earns no cost advantage. For multi-call episodes, reported cost includes every call and elapsed time covers the whole episode, including reasoning, tool round trips, and all memory sessions. Request reliability is shared across calibration tasks: after two or more failed requests, the observed HTTP success rate caps the model’s task-quality signal. A clean task slice cannot erase repeated route failures elsewhere. One isolated failure remains in its own task score rather than penalizing every other task. These observations apply only to their tested OpenRouter route, never to a new direct-provider endpoint. This is route evidence from the evaluation period, not a promise about current provider availability.
+
+Profiles cover small text-field extraction, rule-based classification, single-function dispatch, read-only SQL, short numerical problems, long-text record retrieval, visual-field extraction, dependent tool actions, and external memory across sessions. The matcher also supports code-function operations, but their proposed measured profile and an additional medium-input extraction profile were withheld after fresh validation regressions; those tasks retain their prior evidence path. Matching requires the same typed operation, image/tool/schema requirements, and input/output size category. Classification profiles must also explicitly match the supplied/open label scope. A new required capability outside the profile prevents a match. Retention scope (none, session, persistent) and whether execution uses an action loop must also match; tool dispatch, dependent actions, and cross-session memory are separate operations. At least five currently eligible tested models and six calibration cases per model are required. Availability and declared capabilities still come from the refreshed catalogue. Evidence expires after seven days; expired, undersampled and unmatched profiles fall back to public benchmarks.
 
 When several profiles match, the most recently measured complete profile is used. Measured mode ranks **within its disclosed tested pool**, not every catalogue model. It applies the same pairwise policy to actual observed accuracy, charges and time. Memory quality uses correct updates, retrieval, and final store state across fresh conversations. In offline candidate evaluations, code uses executable hidden checks and tools use permitted actions and their prerequisites. If the task does not require strict JSON, evaluation-only JSON serialization does not erase correct actions or facts. Tool profiles score observable actions, prerequisites, and a nonempty final reply; they do not grade the precision of its prose. Memory facts accept a JSON object surrounded by prose or a fence. Raw JSON-valid counts remain separate. A strict-JSON task cannot match these non-strict profiles. A ten-percentage-point observed accuracy gap gives full pairwise quality preference. These are small development samples, not confidence intervals or proof of accuracy on an arbitrary prompt. Text cases are authored; the visual calibration set contains eight public Portuguese invoices. Classification, arithmetic and simple dispatch can saturate at perfect accuracy, leaving cost and variable latency to determine close positions. Public calibration aggregates and their protocol hash are available through each model’s **Measured task evidence** link. Raw answer keys and held-out test outcomes are not runtime inputs. The source images are attributed to [Cruz and Castelli's invoice dataset](https://zenodo.org/records/6371710), under that dataset's CC BY 4.0 terms.
 
@@ -122,15 +214,23 @@ The strongest supported pairwise preferences are applied first, skipping any tha
 
 Reference input sizes are **1,024 / 4,096 / 32,768 tokens**, and visible output sizes are **128 / 512 / 2,048 tokens**, for short / medium / long. Cost is input tokens × advertised input rate + output tokens × advertised output rate. This excludes hidden reasoning, cache effects, image-specific tokenization and other modality charges. An image task's input size is an illustrative token workload, not a prediction of that image's bill. Separate rates and the assumptions are returned so clients can substitute their own measured usage.
 
-Reference completion time is **time to the first answer token + visible output tokens / throughput**. When the source publishes multiple configurations, Scout calculates completion time for each paired measurement and takes the median. It does not mix one configuration's delay with another's throughput, or confuse early streamed reasoning with the first answer. Missing first-answer timing for a reasoning configuration stays unknown. A legacy snapshot with only first-token timing supplies no completion estimate; deployment replaces an older data format with the bundled compatible snapshot even if the legacy refresh timestamp is newer. These measurements are from Artificial Analysis, not the caller's OpenRouter route; provider and reasoning settings may differ. They are not guaranteed task latency.
+Reference completion time, when tied to the selected endpoint, is **time to the first answer token + visible output tokens / throughput**. When the source publishes multiple configurations, Scout calculates completion time for each paired measurement and takes the median. It does not mix one configuration's delay with another's throughput, or confuse early streamed reasoning with the first answer. Missing first-answer timing for a reasoning configuration stays unknown. A legacy snapshot with only first-token timing supplies no completion estimate; deployment replaces an older data format with the bundled compatible snapshot even if the legacy refresh timestamp is newer. Artificial Analysis measurements remain source evidence; they are not transferred to a different provider route. Since ranking version 5, reference latency stays unknown unless a timing record identifies the selected endpoint. Measured task profiles retain timing only for their tested route. Listed provider metadata is not an endpoint latency probe.
 
 Matching is deliberately narrow; code-function tests do not establish repository-level engineering skill, and small tool loops or key-value memory tasks do not establish long-horizon agent reliability. A profile label and quality basis are returned with every measured shortlist. No model-name bonuses, prompt-example rules, or task answer keys enter the ranking.
 
 The goal is an ordered five-model shortlist, not just a first-place pick. Regression checks cover all five positions, all ten within-five pairs, cutoff membership, provider-name invariance, unknown cost/time, capability and modality boundaries, expiry, and rejection of held-out observations during evidence building. Paid evaluation uses separate calibration and held-out cases across multiple task types. Held-out measurements compare both membership and ordering within the tested candidate pool; they do not establish a global optimum or a perfect ordering for every prompt. Test the five candidates on your own examples before choosing a production model.
-If fewer than five eligible models have evidence, Scout returns fewer and explains why. There are no accounts, saved prompt history, vector database, user selection controls, or language-model reranker.
+If fewer than five eligible models have evidence, Scout returns fewer and explains why. There are no visitor accounts, visitor-facing prompt history, vector database, user selection controls, or language-model reranker. Private team search records are described below.
 
 ## Current validation limits
 
+The identification correction was checked on the original bird-name prompt, plant-species identification, supplied indoor/outdoor choices and supplied ticket categories with strict JSON, using real prompt interpretation. Regression checks also cover missing/uncertain label sets, explicit endpoint scopes, specialist discovery, visual evidence and classification calibration boundaries. These checks verify interpretation and eligibility, not the accuracy of model-generated bird or plant names.
+
+The earlier 15 October endpoint-catalogue revision passed **20/20 fresh task-interpretation cases** with expectations frozen before dispatch. They covered image/text labels, predicates, rubric scores, single/all-instance boxes, masks, ambiguous separation, visual/text extraction, code, SQL, tool loops, memory, prose and unsupported generation. A prior 18-case development check exposed two routing errors; those cases were retired before the final set. Unit checks cover output-contract eligibility, exact endpoint scoping, name invariance, duplicate endpoints and unknown prices. This validates tested routing boundaries, not best-five task performance.
+
+A separate execution check used the same twelve 32×32 images from the public [CIFAR-10 test split](https://huggingface.co/datasets/uoft-cs/cifar10), rows 80–91, with ten category labels. Decisions `gpt-6-luna` returned valid choices for 12/12 and matched 5/12 labels, with 0.344s median request time and $0.0002148 total estimated input-token charges at the published rate. This is an estimate, not reconciled billing. A local CPU run of `Xenova/clip-vit-base-patch32`, ONNX q8 through Transformers.js, matched 7/12, with 0.022s median warm inference and about 8.6s initial loading. Local hardware and operating costs were not measured and are not zero. The deployment configuration and cold-start difference matter; these timings are not a universal speed comparison. Public test images may have appeared in training.
+
+These small, low-resolution results are **not ingested into production ranking** and do not establish the correct order of five endpoints. They also do not validate the original full-precision CLIP weights or other checkpoints. Jev is indexed from its published task/endpoint contract and was not executed without its separate credential. Task-fit mode keeps quality unknown instead of promoting cheap or specialized options through an invented accuracy score.
+
 The latest 15 October coding/extraction follow-up made **132 requests**: six new calibration cases and six disjoint new holdout cases per category, with six coding candidates and five extraction candidates. These are authored application scenarios, not private production records. Predictions, scorer and evidence were frozen before holdout dispatch. The v4 quality/cost/speed policy was unchanged. Candidate evidence used only the new calibration split; no model-specific rule or held-out result entered it.
 
 | Latest proposal | Top-five membership, before → candidate | Correct ordered pairs, before → candidate | Release |
@@ -170,11 +270,19 @@ The first test exposed cross-task request failures; those test outcomes were not
 
 ## Refresh and data handling
 
-The task description is sent to Scout and then to OpenAI Decisions for interpretation. Scout's application code processes it in memory without saving or logging it. Only the public index and published calibration aggregates are persisted; prompts are not sent to benchmark sites or recommended-model providers. [OpenAI's API data controls and retention policies](https://developers.openai.com/api/docs/guides/your-data) apply to that provider call. Scout does not claim zero retention by OpenAI. Hosting infrastructure can have separate request-metadata policies. Clipboard actions occur only when selected. Describe the task rather than pasting actual invoices, credentials or private customer records.
+The task description is sent to Scout and then to OpenAI Decisions for interpretation and, when relevant, specialist compatibility checks. Hourly preparation also sends public repository metadata and benchmark definitions/methodology to Decisions; no user prompts are involved in that refresh. Scout stores the submitted prompt alongside its recommendations in private Cloudflare Analytics Engine records. Runtime request logging remains disabled. The public index, published calibration aggregates and the limited search analytics described below are persisted; prompts are not sent to benchmark sites or recommended-model providers. [OpenAI's API data controls and retention policies](https://developers.openai.com/api/docs/guides/your-data) apply to those provider calls. Scout does not claim zero retention by OpenAI. Hosting infrastructure can have separate request-metadata policies. Clipboard actions occur only when selected. Describe the task rather than pasting actual invoices, credentials or private customer records.
 
-Each query has one Decisions request with a four-second timeout and no automatic retry. Shared in-memory admission limits allow at most eight active requests and 120 starts per minute for the Scout service. Excess requests fail promptly instead of queuing. These counters reset with the process and are not an account-level billing cap. Provider-side limits can also make interpretation unavailable. The API key is held server-side; callers never receive it.
+Scout records one server-side Analytics Engine event for each valid recommendation attempt on the website/JSON API or MCP. It keeps the original submitted prompt (up to 4,000 characters, including whitespace), fixed task categories, capability/input/output requirements, priority and workload buckets together with the ordered model/endpoint shortlist (up to five), its quality/cost/time values and their calibration/reference/unknown basis, attached source evidence, ranking and interpreter versions, index date, scope category, freshness, outcome and server duration. Clarifications, unsupported tasks, empty results, admission limits and interpretation errors are distinct outcomes. A failure before interpretation has an unknown task category. Invalid requests rejected before recommendation, status checks and MCP discovery are not search events. Website and other JSON API calls share the `api` channel; MCP uses `mcp`.
 
-Sources refresh independently. A failed download, changed parser format, or unexpected loss of coverage preserves that source's previous evidence and timestamp while healthy sources and prices continue updating. Fresh catalogue removals still remove unavailable models. All eight Hub queries must succeed for its discovery snapshot to replace the previous one. A complete outage preserves the previous index. Failure and stale status are exposed in both interfaces.
+Apart from the submitted prompt and allowlisted public source evidence, analytics excludes free-text task summaries, prompt hashes, requested inventory IDs, endpoint API addresses, error messages, IP addresses, user agents, cookies, visitor IDs and session IDs. Prompt text is not redacted: personal information or secrets typed into it would be retained, so these records are not guaranteed to be anonymous. No visitor identity is collected separately or attached to a search. It adds no browser tracker or inference call. Analytics writes are best effort and cannot interrupt recommendations. Cloudflare [retains Analytics Engine events for three months](https://developers.cloudflare.com/analytics/analytics-engine/limits/) and may sample them; this is usage evidence, not an exhaustive audit log or evidence of task success. Hosting infrastructure and the existing Decisions provider call have their own data policies.
+
+Search records are private Cloudflare data, queried on demand through authenticated account access. Scout has no report generator, analytics dashboard, analytics sign-in, or public history/query endpoint. Each new record keeps the submitted prompt, the ordered five (or fewer), and a snapshot of their supporting evidence together: benchmark scores, source links/dates and the prepared capability/family interpretation version/time, applicable calibration measurements and protocol, advertised pricing, speed sources, and endpoint/model-card task-fit evidence, including the prepared semantic profile and compatibility decision for a selected Hub deployment. Capturing source facts at recommendation time avoids relying only on links whose contents can change. Unknown measurements remain unknown, and reported model-card results remain unverified claims.
+
+Earlier records are preserved as collected: version 1 has no prompt, and versions 1–2 have no attached evidence snapshot. Missing historical fields cannot be reconstructed. New records use lossless compression when needed to fit Cloudflare's event-size limit. Writes remain best effort; a record too large to fit is not silently truncated or presented as complete.
+
+Each query starts with batched Decisions questions and can make one verification pass for weak answers, followed by relevant specialist compatibility batches. Task interpretation and specialist comparison share a four-second request budget; catalogue reads and ranking add their own time. Optional verification has at most 750 milliseconds and leaves weak answers uncertain if unavailable. Independent specialist batches run in parallel, with at most 100 questions per call. Transient connection, timeout, rate-limit and server failures can receive one retry within the remaining deadline; valid `Retry-After` delays are honored. Authentication, exhausted-credit and quota failures are not retried. Failed interpretation returns an explicit unavailable response; failed specialist checks omit unverified specialist candidates. No keyword fallback is used. These bounds reduce avoidable waits and failures; they do not guarantee a successful search or a particular latency. In-memory admission allows 32 active requests and 600 starts per minute per Worker isolate; a separate edge limiter allows 60 POST requests per minute per IP. Excess requests fail promptly instead of queuing. These are operational limits, not an account-level billing cap or a throughput guarantee. Credentials remain server-side.
+
+Hourly sources refresh independently; reviewed typed-endpoint contracts retain their manual review dates. Freshness follows each source's prepared policy: public feeds have a two-hour freshness allowance, while incremental research uses the configured seven-day recheck interval. A source does not become stale merely because it follows a longer research interval. Missing or invalid check dates remain overdue, and actual collection or preparation failures remain degraded. A failed download, changed parser format, or unexpected loss of coverage preserves that source's previous evidence and timestamp while healthy sources and prices continue updating. Fresh catalogue removals still remove unavailable models. All fourteen Hub queries and required semantic preparation must succeed for its discovery snapshot to replace the previous one. A complete outage preserves the previous index. Failure and stale status are exposed in both interfaces.
 
 Source content is parsed as data, never executed. Downloads, decompression, and request bodies are size-bounded; source downloads time out. Links are restricted to HTTP(S). The publisher's OpenVLM data URL uses HTTP and lacks transport authentication; the remaining feeds use HTTPS. Scout does not independently authenticate benchmark submissions.
 
@@ -187,15 +295,19 @@ npm ci
 npm run scout
 ```
 
-Open `http://127.0.0.1:45833/scout/`. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.
+Open `http://127.0.0.1:45833/scout/`. This lightweight local preview uses the shared interpreter and matcher with bundled/source-feed data; it does not run the hosted Parallel research workflow or R2 publication system. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.
 
-`npm run scout:refresh` fetches the seven public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Refreshing public sources requires no provider key or inference budget. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid hourly evaluation job is installed. Recommendation queries require the server key and incur Decisions API usage.
+`npm run scout:refresh` fetches the eight hourly public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Source downloads need no provider inference, but preparing new or changed specialist profiles and benchmark interpretations requires `OPENAI_API_KEY` and incurs Decisions usage. Unchanged metadata reuses cached analysis. Without a usable key, required specialist preparation retains the previous specialist snapshot; affected new or changed benchmark definitions remain unclassified. Both failures are reported on the affected source. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid hourly evaluation job is installed. Recommendation queries require the server key and incur Decisions API usage.
 
 ## Hosting and verification
 
-`wrangler.scout.jsonc` defines a separate Scout Worker, one SQLite Durable Object holding the index, and an hourly cron at minute 7 UTC. Configure `OPENAI_API_KEY` as a secret on this Scout Worker before deployment. Searches interpret the task once and use the prepared in-memory index. The bundled snapshot allows immediate startup; subsequent successful updates persist across restarts. Refreshes are serialized.
+`wrangler.scout.jsonc` defines the Scout Worker, a private R2 catalogue, an hourly Workflow and a SQLite Durable Object coordinating publication. The cron runs at minute 7 UTC. Configure `OPENAI_API_KEY` and `PARALLEL_API_KEY` as Worker secrets. Refreshes build immutable catalogue versions and prepared candidate pools; searches read those pools directly and never pass through the publication coordinator. A running refresh does not block searches against the active catalogue. The first hosted publication must be validated before serving searches; a missing catalogue fails explicitly. The Worker is `openmerit-scout`, the private catalogue bucket is `openmerit-scout-catalogue`, and the Workflow is `openmerit-scout-refresh`. Both API keys are stored as Cloudflare Worker secrets and read from server-side bindings; they are not shipped to the browser. Separate workers.dev and preview URLs are disabled.
+
+Publication records a durable completion receipt after updating the active pointer, pending-research queue and health status. Interrupted publication resumes unfinished work; a lost success reply can be acknowledged again after the lease is released. Replaying a completed publication cannot roll back a newer catalogue or clear a later refresh's lease. Completed research receipts leave the queue only when publication commits; unfinished research retains its original paid job for a later hourly run.
+
+Prepared capability pools can overlap. Searches use their prepared ID lists to choose a smaller covering set of pools, preserving the selected candidates and complete evidence while avoiding repeated downloads of the same model records. Explicit model and endpoint inventories retain their existing scope checks. These metadata reads do not fetch source pages or redo hourly research.
 
-`demo/domain-router.mjs` routes Scout paths to this service while retaining the existing website, protected memo, authentication, and dashboard services. The public guide is generated and published with the main documentation. Do not deploy the older root `wrangler.jsonc` to publish Scout. `npm run build:scout` packages only Scout, public documentation, and its shared assets; private memo and demo applications are excluded.
+`demo/domain-router.mjs` routes Scout paths to this service while retaining the existing website, protected memo, authentication, and terminal services. There is no Scout admin/report route. The public guide is generated and published with the main documentation. Do not deploy the older root `wrangler.jsonc` to publish Scout. `npm run build:scout` packages only Scout, public documentation, and its shared assets; private memo and demo applications are excluded.
 
 ```sh
 npm run check:scout

package/docs/security.md

@@ -43,3 +43,14 @@ Coding-harness histories are outside the terminal's application scope and are no
 The local web service binds to `127.0.0.1` and rejects unexpected Host, Origin, and cross-site browser requests. It only serves bundled assets and specific read-only JSON routes. It is not authenticated against other local processes or machine users. Model names, endpoint identifiers, and relative source paths can still be sensitive. Names and identifiers in imported metadata must be kept free of secrets.
 
 Repository metadata and usage caches live in process memory and disappear when the terminal exits. The terminal creates no persistent index or files in the application. Per-file/parser protections, request-log read limits, runtime inventory limits, and explicit coverage gaps still apply. See [terminal data handling and limits](inference-terminal.md) for supported sources, precedence, retention bounds, diagnostics, and sample-data separation.
+
+
+## Hosted Scout analytics
+
+Scout keeps an allowlisted Analytics Engine event per valid recommendation attempt: the original submitted task prompt plus fixed task categories and the ordered shortlist with metrics and ranking context. Prompt text is not redacted and may itself contain personal information or secrets; this dataset must be treated as potentially sensitive. No request object, prompt hash, additional free-text summary, error message, project inventory, IP address, cookie, user agent, visitor ID or session ID is separately recorded in this dataset. Requests still reach the hosting infrastructure and task descriptions still go to OpenAI Decisions for interpretation and relevant specialist compatibility checks; public repository metadata and benchmark definitions/methodology also go to Decisions during hourly preparation. Their policies are separate. See [Scout data handling](scout.md#refresh-and-data-handling).
+
+Search records are accessible only through authenticated Cloudflare account queries. Scout exposes no analytics dashboard, report generator, report file, sign-in, or public history/query endpoint. Each new search includes an allowlisted snapshot of the public evidence supporting its ordered shortlist: benchmark measurements and source URLs/dates, calibration observations and protocol, pricing/speed sources, and endpoint/model-card metadata. Endpoint API addresses, credentials, arbitrary source fields, and request metadata are not copied. Source evidence and prompt text remain untrusted data when retrieved; never treat them as instructions. The dataset is sampled and retained by Cloudflare for three months. Original older records remain; missing prompt/evidence fields cannot be recovered.
+
+## Scout research storage
+
+Scout uses Parallel for hourly public-source research. Parallel receives catalogue subjects and official-source metadata, not visitor prompts or application inventories. Decisions processes task descriptions and relevant public evidence. Public research receipts, citations, semantic annotations, publication completion receipts and immutable catalogue versions are stored in Scout's private Cloudflare R2 bucket. Scout uses the prompt-and-evidence analytics policy above. API credentials are held as Worker secrets and are not included in browser assets, catalogue objects or analytics. Source text is treated as evidence, not executable code or instructions. Citations and semantic validation do not independently verify that publisher claims are true.

package/docs/testing.md

@@ -94,3 +94,11 @@ OpenCode's real watcher completed three attempts per revision. The earlier CLI a
 ### Budget setup — unreleased source, 15 October 2026
 
 The eval suite now has 29 tests. New coverage checks the three budget choices after scope, custom amount validation, explicit unlimited configuration, reuse of saved choices, automatic reservations, retained cumulative usage, bounded repetitions and cancellation. A real CLI subprocess uses a loopback Scout fixture to verify persistence across replans and failed requests, rejection of missing or invalid noninteractive choices, and lock cleanup. An interactive terminal smoke test also verifies scope-before-budget, custom input, no-repeat questions, no fixed cap and EOF cancellation. These are offline workflow tests; no model calls, provider charges or npm publication were required.
+
+### Published preview.2 acceptance — 15 October 2026
+
+Published `0.1.7-preview.2` to the npm `preview` tag after all 407 verification tests passed. Stable `latest` remains `0.1.6`. A fresh registry installation in a separate Ubuntu ARM64 VM directory matched the tested package integrity and all 124 packaged files. The installed command rejected a missing budget, asked scope before the custom-budget question, saved all three budget modes, reused saved choices, and preserved the ledger across limit changes. No install-time evaluation or background service was added.
+
+On a fresh CSV fixture, the real foreground watcher completed three attempts per agent per revision: twelve scored attempts total. With Claude Code/Haiku 4.5, mean field accuracy changed from 66.7% to 100%, mean input/output tokens from 87,826/666 to 87,781/653, native estimated cost from $0.09333 to $0.06534 per attempt, and median elapsed time from 13.87 to 13.42 seconds. With OpenCode/Gemini 2.5 Flash Lite, accuracy changed from 44.4% to 100%, tokens from 20,093/306 to 22,726/511, estimated cost from $0.001501 to $0.001896, and median time from 4.97 to 5.72 seconds. One baseline OpenCode output correctly remained a zero-score task failure. Both revised tools passed all checks; better quality did not imply lower cost or faster execution on every route. These are native estimates and one integration fixture, not reconciled per-attempt bills or a general model ranking.
+
+A deliberate native Claude budget stop retained 21,516 input tokens and a $0.0286866 partial estimate, with unknown quality and no revision comparison. Codex's Linux sandbox probe stopped before inference with zero attempts and no reserved spend. The initial sandbox fixture had an empty route list, which planning correctly rejected; declaring the intended Codex route allowed the separate sandbox check without repeating paid trials. Across the acceptance run, provider-reported account usage increased by approximately $0.17, below the $1 test allowance. All three projects released their locks, the watcher exited, and a scan found no temporary credential in 57 test files. Reports and registry verification are retained in the task's `openmerit-preview2-release` directory. The website was not deployed as part of this npm release.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.1.7-preview.2",
+  "version": "0.1.7-preview.3",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {