openmerit 0.1.7-preview.2 → 0.1.7-preview.4

package/docs/scout.md

@@ -1,18 +1,151 @@
 # Scout by OpenMerit
 
-Describe what you need a model to do. Scout returns up to five available models using relevant public benchmark results, task-shaped reference cost, and completion time. The `/scout/` page, JSON API, and MCP tool all use the same index and ranking.
+Describe what you need a model to do. Scout returns up to five model-and-endpoint candidates. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page, JSON API, and MCP tool all use the same index and ranking.
 
 Scout is a separate beta product. It does not change the OpenMerit npm package, Pi coordinator, or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
 
+## Evidence architecture — hosted beta, 16 October 2026
+
+Scout separates semantic interpretation from exact metadata checks and numerical selection. The hosted service uses this evidence architecture after publication of its validated serving generation. The prompt and results interface is unchanged; no npm release is included.
+
+| Stage | Hosted implementation |
+| --- | --- |
+| Hourly discovery | Structured feeds and incremental Parallel research; resumable Hub discovery archives public listings without inventing usable endpoints or scores. |
+| Hourly evidence | Source-backed identities, deployment facts, benchmark methodology and specialist profiles; cached summaries fall back to the original source when support is uncertain. |
+| Hourly publication | Private R2 records, an indexed D1 fact/observation projection and versioned Vectorize benchmark embeddings. The active pointer changes only after the complete generation is ready. |
+| Task interpretation | Structured Outputs extracts metadata restrictions, alternatives, preferences and the downstream work. Decisions interprets that work for capabilities, operations and output serialization. No prompt keywords or provider/family exception lists are used. |
+| Eligibility | Generic typed equality, membership and range comparisons against documented facts, intersected with explicit inventories. Missing or incorrectly typed facts cannot establish a restriction. |
+| Evidence retrieval | A query embedding retrieves benchmark methodology, supplemented by hourly capability memberships. Semantic similarity supplies evidence candidates, never a model's score. |
+| Selection | All eligible verified catalogue models are considered before selecting up to five distinct models. Numerical Pareto comparisons apply only within comparable evaluation scopes with known quality, cost and latency. |
+
+Mandatory restrictions use alternatives of complete condition groups. Soft preferences affect ordering, not eligibility. Relative dates use a supplied current UTC reference; arbitrary numeric bounds are retained, including impossible requirements. An unsupported mandatory condition cannot silently disappear. A no-match result describes the available verified catalogue and inventory, not every model in existence.
+
+Quality remains a relative benchmark signal, not predicted accuracy on an arbitrary task. Correlated benchmark families are grouped. Incomplete or incomparable objectives remain provisional and cannot establish a global three-objective frontier. Narrow typed tasks can still use a task-fit shortlist with unknown quality. Comparable direct calibration observations supplement quality evidence without excluding other eligible models. Calibration episode charges and timing do not replace the common token-workload reference costs and endpoint-bound timing; unknown values remain unknown.
+
+The UI and HTTP/MCP inputs remain unchanged. This path has no request-time web research, catalogue refresh, evidence preparation or recommended-model execution. Interpretation, query embedding and bounded Decisions checks are external calls; latency is measured end to end rather than assumed instantaneous. The matching budget is six seconds. A provider failure returns an unavailable response instead of silently relaxing restrictions.
+
+OpenAI receives the submitted task for structured interpretation, Decisions and query embeddings. Query vectors are used transiently; only public benchmark embeddings enter Vectorize. Public model/endpoint facts and observations enter private D1, with source records in private R2. Parallel receives public research subjects, not visitor prompts. Existing private prompt-and-evidence analytics and three-month sampled retention remain unchanged. Credentials remain server-side Cloudflare secrets.
+
+The canonical paths remain `/scout/`, `/scout/api/recommend`, `/scout/api/status` and `/scout/mcp`. There are no versioned preview pages or API aliases. The dated sections below retain historical implementation and validation records; this architecture and the current Ranking section supersede conflicting earlier policies.
+
+Acceptance retained the earlier provider failures rather than counting them as successes. The final remote filter/control run initially passed 18 of 27 cases; nine reached the six-second deadline. Each of those nine passed one focused recheck without changing the matcher. Ten additional ambiguous-output cases passed on the revised interpreter. These checks establish interpretation, eligibility and API/MCP behavior for the tested prompts, not universal reliability or actual recommended-model accuracy. The isolated real-feed refresh completed R2/D1/Vectorize publication with new Parallel purchases disabled; the normal production workflow was then exercised separately. The production refresh published successfully after a staging-memory correction. Public HTTP/MCP checks initially passed six of eight cases; the provider and invoice timeouts both passed one focused recheck. Successful live observations were approximately 2.4–6.4 seconds. Six research jobs remain queued for later enrichment. These are separate attempts, not an eight-case clean first pass.
+
+## Task relevance corrections — hosted beta, 17 October 2026
+
+These corrections are deployed as a limited hosted beta. Hourly evidence documents retain verified research descriptions and directly selected methodology alongside feed metadata; embedding caches change when that evidence changes. The already verified descriptions are indexed directly, without a second model-generated summary; original citation excerpts remain in the source records rather than the semantic search text. Structured extraction separates the downstream work from model restrictions before Decisions interprets its capabilities and output contract. Name resolution can run alongside Decisions. This avoids interpreting model selection itself as a downstream capability; it does add a sequential interpretation dependency within the existing six-second budget.
+
+Query vectors describe the downstream work, excluding model-selection restrictions. They retrieve methodology candidates, with prepared memberships as a fallback when the query has no matches and a backstop for required capabilities missing from the retrieved set. The query order and hourly model/benchmark availability map propose up to sixteen methodology records. Decisions checks each record independently against the actual workload, without mixing unrelated methodologies in one input. Both general relevance and a deliberately scored required ability must be sufficiently supported. A broader ability can be a component proxy without testing the exact domain or answer format; incidental prerequisites cannot qualify. Ambiguous labels, broad capability overlap and unavailable checks cannot grant quality evidence. Fine-grained scope is checked against the full workload and methodology rather than asking every discovered capability again on each search. Core task/format interpretation stays bounded as the hourly directory grows.
+
+Classification and judgment searches consider applicable benchmark evidence before an unmeasured native endpoint contract. Without comparable candidates they retain an explicitly unmeasured task-fit fallback; contract compatibility alone does not demonstrate accuracy. Spatial pipelines keep their separate compatibility checks. Ambiguous output representations can use the existing clarification response, with model restrictions preserved. No task-name, object-name or model-specific exception is added.
+
+The Artificial Analysis parser locates serialized records by their data shape rather than a component-property name and never executes downloaded scripts. Omitted historical observations are retained only for models still listed and unchanged benchmark definitions, keeping their original provenance and historical status. Missing endpoint-specific timing and incomparable quality scopes remain unknown or provisional; restoring a feed does not authorize copying model-level timing to an unverified route. Matching retains its six-second deadline, and the hourly schedule and UI are unchanged. Default selection orders supported frontier points by quality and known cost/time, without reserving slots for the cheapest or fastest point. Explicit cost and latency priorities and metadata preferences still apply.
+
+The initial release was withheld after real-task validation; the owner subsequently approved a limited beta release with the evidence and accuracy limitations retained. Earlier development attempts exposed source-relevance, retrieval and interpretation errors; a passing interpretation/filter run alone did not authorize deployment. A fresh execution set made **38 OpenRouter requests**, with **$0.223226** in reported charges and no unknown-charge calls. Five shortlisted models per task received two public examples each; bird and invoice tasks also included two comparison models from the earlier shortlist. The source images, answer keys, scorer and selected pool were frozen before dispatch. These executions used recorded OpenRouter routes, which do not establish latency or cost on a different recommended host.
+
+| Fresh task check | Shortlisted models | Comparison models | Release gate |
+| --- | --- | --- | --- |
+| Bird species names in alphabetical order | 19/20 required items correct | 8/8 correct | Lower sampled accuracy |
+| Book versus poster labels | 20/20 labels correct | No separate comparison pool | Passed |
+| Invoice number and total, strict JSON | 18/20 fields correct | 8/8 correct | Lower sampled accuracy |
+
+One model confused two bird species; two invoice candidates added an extra zero to a printed total. All outputs parsed successfully, so schema compliance did not establish accuracy. These failures remain recorded; no model-specific exception, held-out score or answer key was added to ranking. The final downstream-interpretation correction passed 11/11 matching cases, including exact provider/family/context/date filters, known matches, no-match and ambiguous-output cases. Observed client durations were approximately 2.2–5.9 seconds, with no percentile guarantee. That correction changed some bird/category shortlist membership, so the frozen execution results do not certify the final pool; the invoice pool remained the same and failed the original accuracy-only gate. That gate compared five shortlisted models with two comparison models rather than evaluating quality, cost and latency together. In this sample the comparison models had approximately 4.1 times the mean request charge for birds and 3.4 times for invoices. Lower mean accuracy alone does not establish a worse Pareto shortlist. The beta makes no claim of improved task accuracy or globally optimal five-model membership. Broader coverage, comparable deployment measurements and independent validation of all three objectives remain follow-up work. The held-out outcomes remain excluded from ranking.
+
+Publication retained the latest completed hourly research and a healthy refreshed feed set. The final rollout check passed 9/11 cases on its first attempt: bird matching reached the deadline and explicit SVG generation returned no shortlist. Both passed one focused recheck without a matcher change. After deployment, the public HTTP suite again passed 9/11: the explicit SVG request returned no shortlist, and a Qwen-family code-authoring request admitted tool-calling evidence. Creator/family/context/date eligibility checks and impossible-context no-match passed. The live browser bird prompt displayed five recommendations without overflow. These are retained semantic and latency limitations of the beta, not a clean first-pass acceptance or proof of best-five performance.
+
+## Latency improvements — hosted beta, 17 October 2026
+
+The hosted implementation keeps the existing task and format Decisions, exact metadata comparisons and numerical selection. Numeric and date vocabularies are omitted from interpretation input because their bounds are arbitrary; small canonical vocabularies remain in the first call. Larger vocabularies are deferred until an extracted restriction references them; a model-assisted resolution checks against the complete published vocabulary without changing unrelated conditions. Unknown mandatory names remain unresolved and cannot widen the result set. Stable reference data has an explicit provider prompt-cache boundary separate from the prompt and UTC reference time. Cache availability is measured rather than guaranteed.
+
+Hourly staging prepares content-addressed specialist packs alongside profile indexes. Searches select the same scoped profiles and use those packs instead of many repository reads. A version-checked derived pack index can also be prepared for an existing immutable catalogue without repeating research. Existing generations remain readable through the original shard path when no matching pack index is available. Source research and the hourly trigger are unchanged.
+
+Successful searches may be reused within a Worker isolate for up to 30 seconds, bounded to 32 entries and approximately 4 MiB. Exact prompt text, model/endpoint inventories, publication, interpretation version and UTC calendar date identify a cache entry. Cache hits preserve the original `evaluatedAt` time; date conditions remain evaluated as of that time for at most 30 seconds, and results are not reused across UTC midnight. Identical concurrent requests share computation. Errors and results missing a specialist or applicability check are not cached. This temporary memory cache adds no external prompt storage and is not semantic similarity caching.
+
+Private search records now include fixed stage durations, pending stages at a deadline, cache status and allowlisted provider status/error codes. They do not store raw errors, stack traces, headers or credentials. API results include `runtime` timings/cache status; cache hits report their current serving duration and zero new interpretation time. The page remains unchanged. A six-second matching deadline remains a reliability boundary; these changes do not establish a latency guarantee. The full repository release checks passed, including 318 Scout tests. An initial candidate timed out on books/posters; that failure was retained, and the interpretation was then reduced further. The revised semantic suite passed 17 of 17 cases, including cold repetitions and exact-cache reuse, and four final cold checks passed after the independent database reads were parallelized. These small samples do not establish a production percentile or a latency guarantee. Post-deployment HTTP/MCP checks passed five of five on their first attempts: supplied-label vision, creator restriction, impossible context, JSON invoice extraction through MCP and an exact repeat. Observed end-to-end durations were approximately 2.0–4.3 seconds for new searches and 146 ms for the repeated prompt served from the same isolate cache. The 127 UI source files and five live scripts/styles were unchanged. These are individual observations, not percentile guarantees.
+
+## Default architecture — hosted beta, 16 October 2026
+
+Scout maintains separate model, endpoint, benchmark and observation records. Structured feeds, resumable discovery and bounded Parallel research prepare source-backed public metadata each hour. Unreviewed discovery listings remain outside the executable catalogue. New or changed methodology, metadata and embeddings use source-addressed caches. An incomplete generation cannot replace the active catalogue; searches continue against the last committed version.
+
+A search runs Decisions task interpretation, Structured Outputs metadata extraction and a transient query embedding. It applies exact restrictions in D1, retrieves prepared evidence and relevant specialist profiles, and compares verified eligible configurations locally. There is no request-time web research, source preparation or model execution. The model pool has no pre-frontier top-K cap; evidence retrieval and specialist checks remain bounded. Missing objectives, incomplete benchmark coverage and approximate evidence retrieval prevent a claim of a globally verified optimum.
+
+The trigger remains hourly at `7 * * * *` UTC, with up to 24 Parallel jobs, refresh concurrency six and seven-day research rechecks. Discovery and enrichment are incremental; the trigger does not promise complete world coverage or a revisit to every source each hour. Relevant specialist checks remain bounded to 64 prepared profiles. Matching has a six-second budget and includes external provider/storage calls. Successful focused remote checks were approximately 2.4–5.4 seconds; earlier timeouts are disclosed above. Instantaneous results are not guaranteed.
+
+## Matching corrections — hosted beta, 15 October 2026
+
+The hosted beta includes the following corrections. This update does not publish a new npm package:
+
+- Benchmark, measured and task-fit candidates share the same endpoint requirements for image input, structured output and tools. Reviewed typed endpoints must support the requested operation. General catalogue entries require declared text-only output; text accompanying audio, image or video generation does not establish a general text contract. Mixed-output endpoints need a separately reviewed operation contract. Image inputs with text outputs remain supported. These are metadata eligibility checks, not task-accuracy guarantees.
+- Unknown input/output prices and the compatibility blended price remain `null`, including when only one rate is known. Actual advertised zero rates remain zero. The existing page renders unknown prices with a dash.
+- Either `modelIds` or `endpointIds` suppresses separate specialist discovery, including empty and unknown-only endpoint inventories. Explicitly selected compatible specialist endpoints can still enter a task-fit shortlist.
+- When selecting among routes for the same model, an explicit latency priority compares known endpoint-bound completion times first, then cost. Without comparable timing, known timing precedes unknown timing; if neither route has timing, known cost and stable endpoint IDs decide. Other priorities retain cost-first route selection. Task-fit candidates retain their contract-fit and deployment tiers. Timing is never borrowed from another route.
+
+Output metadata is normalized during the existing hourly refresh and retained in snapshot data version 5. Startup selects the compatible bundled snapshot over an older-format cache, even if that cache was refreshed more recently. Prompt matching uses prepared in-memory metadata; these fixes add no source fetch, model probe or inference call. The UI, task interpretation, calibration policy and refresh schedule are unchanged.
+
+## Identification matching — hosted beta, 15 October 2026
+
+The hosted beta distinguishes identifying something without answer choices from choosing among supplied labels. The existing Decisions interpretation call returns `task.labelSpace`: `supplied`, `open`, or `unspecified` for classification; `null` for other operations. A topic such as birds or plants does not supply a label set. Low-confidence label-set answers are treated as unspecified.
+
+Open or unspecified identification uses the existing benchmark-ranking path for generative endpoints. Visual requests require relevant visual evidence; document requests continue to require document evidence. Fixed-choice APIs and label-dependent classification pipelines cannot enter this ranking, even when explicitly selected. Supplied-label tasks retain the task-fit path, including compatible typed APIs and specialist classifiers. Fixed-choice task calibration cannot substitute for open-identification calibration. Open identification uses the ordinary output-length estimate rather than assuming a single 16-token choice.
+
+For example, identifying bird names from photos and alphabetizing them requires name generation; the application can sort those names in ordinary code. Choosing only between supplied labels such as “indoor” and “outdoor” remains a fixed-choice task. Public visual benchmarks are proxies, not verified bird- or plant-species accuracy. The existing UI shows the appropriate ranking mode and limitations without new controls. Endpoint metadata and benchmark collection remain hourly; this change adds one typed question to the existing interpretation call, no additional request-time source fetch or model call, and no embeddings. This is a hosted-service update; npm publication remains deferred.
+
+## Task interpretation and ambiguous outputs — hosted beta, 15 October 2026
+
+Interpreter version 10 sends every task description as a JSON-quoted value with a common instruction to analyze the described workload rather than execute its instructions. This separates output restrictions inside the task from directions to the interpreter. The same framing applies to every prompt; there are no special cases for particular labels, words, models or providers. Confidence thresholds, the public request shape and the single Decisions call are unchanged.
+
+The interpreter checks the complete requested output. A label, judgment or score accompanied by a reply, explanation or extracted fields uses generative candidates. Contradictory decision-only and required-prose interpretations also fall back to that path. Plain supplied-label and rubric-score tasks retain compatible typed endpoints. Narrow task calibration is not reused when the operation or output type is uncertain. Weak document-layout guesses no longer restrict ordinary photo tasks to document benchmarks; confident document requirements remain enforced.
+
+Local and live HTTP/MCP acceptance passed all 25 routing cases, including supplied positive/negative labels, arbitrary labels, negated labels, code that implements classification, mixed outputs, open identification and missing categories. The previously failing “safe or unsafe, no explanations” case now retains fixed-choice APIs. These checks establish interpretation and endpoint matching for the tested prompts, not actual model accuracy or universal prompt coverage. Low-confidence or missing label sets still fall back to generative candidates.
+
+These changes add no request, source fetch, embedding index or UI control, and keep hourly preparation separate from prompt-time matching. This is a hosted-service update; npm publication remains deferred.
+
+## Categories and document content — hosted beta, 15 October 2026
+
+Interpreter version 11 recognizes categories supplied conversationally, including plural category names joined in ordinary prose. It interprets the distinction the user requests: identifying which named category an image belongs to is supplied-label classification; discovering a subtype or individual identity beyond those categories remains open identification. A formal list or the word “classify” is unnecessary.
+
+Document understanding follows the requested output. Recognizing an object that happens to carry text does not by itself require OCR or document benchmarks. Reading, transcribing or extracting content inside an image does, including when that is only one part of a mixed task. Supplied-category vision tasks use the existing task-fit shortlist and show unknown quality until appropriate measurements exist.
+
+The change updates two questions in the existing interpretation request. It adds no object-name rules, model preferences, services, dependencies or inference calls. Confidence thresholds, hourly preparation and the UI are unchanged. Local and live HTTP/MCP semantic acceptance passed 43 cases, including classification/reading contrasts and the prior 25-case suite. These are routing checks, not executions of the recommended models or a guarantee for every possible prompt. This is a hosted-service update; npm publication remains deferred.
+
+## Semantic formats and SQL interpretation — hosted beta, 15 October 2026
+
+Interpreter version 13 includes removal of the English keyword scans for strict output formats and structured-data capability. The existing Decisions questions now supply those judgments directly: a prescribed record or delimited layout for software to parse can require structured output even without a named format. The capability question distinguishes extraction and categorization from merely organizing prose or reading already formatted input. The same 21 questions run in one request; no UI control, source fetch, new dependency or additional inference call is added. Hourly catalogue preparation is unchanged.
+
+The response decoder consumes typed answers without rereading the prompt. It retains response validation and general capability-consistency rules. The existing 0.4 confidence threshold now also guards structured-data capability and strict-format requirements. A low-confidence format answer in either direction remains marked uncertain, does not impose a hard format requirement, and cannot match narrow calibration evidence. This threshold is a heuristic, not a calibrated probability of correctness. Semantic interpretation can still misread a request, including unfamiliar formats or languages; catalogue format support is not proof that a model will satisfy an exact output schema.
+
+The operation question recognizes writing, revising or debugging a read-only SQL query even when concrete schema details are omitted from the task description. Those details can be supplied when the model performs the task. Expressions that produce records or serialized values remain query-writing work; they do not make the model's answer a serialized data result. Executing a query, modifying stored data, explaining a query and generating another program remain distinct operations. This changes the general Decisions definition, with no query-text matching, function-name exceptions, dialect lookup or extra question.
+
+This hosted update combines the format changes with SQL interpretation. Real Decisions interpretation passed all 91 cases locally and through live HTTP/MCP: the prior 75 cases, including the previously failing SQL request, twelve additional SQL boundaries and four fresh holdouts. Tests include query source versus serialized output, omitted/runtime schema details, recursive queries, execution loops, data modifications and multilingual requests. npm publication remains deferred. Semantic checks establish tested interpretation and endpoint matching, not query execution, exact-schema compliance or universal prompt accuracy.
+
+## Semantic specialist matching — hosted beta, 15 October 2026
+
+Interpreter version 14 and snapshot data version 6 replace Hugging Face prompt keyword dictionaries, language-name lookup and partial label overlap with Decisions compatibility checks. The hourly refresh prepares each repository's specialist operation and subject scope from public metadata, caches that analysis by revision and relevant metadata, and groups equivalent profiles. Changed repositories are analyzed in batches of up to twenty; unchanged profiles are reused. Preparation uses the server's Decisions API key and incurs provider usage. If preparation fails, Scout retains the previous complete specialist snapshot and its original timestamp while other sources can update.
+
+A search still starts with the existing 21-question task interpretation. When eligible Hub profiles are relevant, a further batched Decisions request checks their subject, explicit language requirements and complete fixed-label coverage. Shared checks are evaluated once; all applicable checks must pass with the existing 0.4 confidence threshold. Requests contain at most 180 compatibility checks plus one language-requirement question; larger sets are split into bounded batches. Open identification, unsupported work, tool/memory tasks and inventories without eligible Hub routes skip this work. There are no repository fetches, model executions or per-repository calls during matching. A failed compatibility request omits Hub candidates for that search and leaves ordinary eligible results available with a warning. There is no keyword fallback.
+
+Compatibility is eligibility, not predicted accuracy. Scout still enforces endpoint contracts, preserves unknown quality/cost/time, and orders discovery components by metadata completeness and diversity. Metadata and semantic interpretation can be wrong, and the existing bounded catalogue can omit suitable specialists. The UI and hourly schedule are unchanged. This revision is deployed on the hosted beta; npm publication remains deferred. All 107 semantic cases passed locally and through live HTTP/MCP: the prior 91 regression cases plus twelve specialist cases and four fresh holdouts. They check domain paraphrases, background context, negation, language requirements, multilingual instructions, full versus partial fixed-label coverage and open identification. These are interpretation and matching checks, not executions of recommended models.
+
+## Semantic benchmark interpretation — hosted beta, 15 October 2026
+
+Snapshot data version 7 prepares benchmark capabilities and correlated benchmark families with Decisions during the hourly refresh. It replaces benchmark-name regexes, manually assigned capability tags and the automatic reasoning fallback. Source adapters retain metric identifiers, score units and parsing contracts. Epoch descriptions and methodology are read from its published catalogue; Artificial Analysis and BFCL methodology are read from their public pages. Unavailable definitions remain explicit limitations, and names alone do not establish an opaque benchmark's capabilities.
+
+Each changed benchmark receives an isolated interpretation request so another benchmark's description cannot supply its capabilities. For shared publisher methodology, a preceding Decisions request selects the directly applicable definition for that benchmark and evaluated subset; unrelated sections are omitted from capability interpretation. Publisher-declared suite identities are retained as source facts; other family relationships are interpreted by Decisions. Capability choices and family references are validated, and positive choices below the existing 0.4 confidence threshold are not used. Family links group related evidence; numerical comparisons still require the exact same benchmark series. Unknown scopes receive no capability tags and cannot contribute capability-ranking evidence. This is semantic metadata interpretation, not a measurement of model quality or a guarantee that the classifications are correct.
+
+Interpretations are cached by definition, relevant publisher methodology, family inventory and interpreter version. Score, model, price and collection-time changes alone do not invalidate that cache. Family inventory changes trigger rechecking its references. If an interpretation fails, unchanged cached records remain usable, affected new or changed series stay unclassified, and source failure status is exposed until a later refresh succeeds. A methodology download failure retains the prior methodology when available and marks its source degraded. There is no keyword fallback.
+
+Searches read the prepared tags and family IDs without fetching methodology or making benchmark interpretation calls. The existing prompt interpretation and optional specialist compatibility calls remain. `benchmarkInterpretation` in status/recommendation responses reports prepared, classified, unclassified and unavailable series counts; each returned benchmark observation includes its capability tags, family and interpretation version/time. The UI and hourly schedule are unchanged. npm publication remains deferred.
+
+The initial deployed snapshot processed all 146 benchmark series: 136 have supported capability tags and ten remain unclassified. Individual weak capability judgments are also omitted, even when other capabilities for that series are supported. A repeat preparation with unchanged metadata made zero Decisions calls. All 24 semantic benchmark cases and 107 task-interpretation/matching cases passed both locally and through live HTTP/MCP, alongside 546 automated checks. These checks cover interpretation, caching, source failure and matching; they do not execute recommended models or establish universal semantic accuracy.
+
 ## Use Scout
 
-Open [Scout](https://openmerit.ai/scout/), type a task, and select the arrow button (**Find models**) or press Enter. For example: “I want to upload an invoice and get the invoice number, date, vendor, subtotal, tax, and total as JSON.” The task field grows and shrinks as you type or paste. Very long prompts scroll within the field once it reaches its height limit. Shift + Enter adds a line. When the field is empty and unfocused, its placeholder cycles through detailed example tasks every ten seconds with a brief fade. Rotation pauses while you type, while the field is focused, or while the page is hidden. With reduced motion enabled, the example stays still. Placeholders never fill the input or send a request. The heading shows live counts of indexed models, model providers, benchmark series and source feeds. Provider aliases are counted once; these are model creators, not every hosting route. The evaluation guide remains available in the fixed bottom footer, with space reserved beneath the page content.
+Open [Scout](https://openmerit.ai/scout/), type a task, and select the arrow button (**Find models**) or press Enter. For example: “I want to upload an invoice and get the invoice number, date, vendor, subtotal, tax, and total as JSON.” The task field grows and shrinks as you type or paste. Very long prompts scroll within the field once it reaches its height limit. Shift + Enter adds a line. When the field is empty and unfocused, its placeholder cycles through detailed example tasks every ten seconds with a brief fade. Rotation pauses while you type, while the field is focused, or while the page is hidden. With reduced motion enabled, the example stays still. Placeholders never fill the input or send a request. The heading shows live counts of API catalogue models, Hugging Face repositories, benchmark series and sources. API identities and Hub repositories are separate counts and can overlap; neither count promises a runnable endpoint for every entry. The evaluation guide remains available in the fixed bottom footer, with space reserved beneath the page content.
 
-The comparison table keeps recommendations in Scout rank order. When a matching measured task profile is available, it shows calibration accuracy, mean reported cost per 1,000 similar tasks, and median full request time. Otherwise it shows relative benchmark quality and reference cost/time estimates. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; missing marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **API & MCP**, **Capabilities**, and **Sources** expand beneath the search and results. Collection times and refresh failures are shown inside **Sources**.
+The comparison table keeps recommendations in Scout rank order. Benchmark mode shows a relative quality signal and reference cost/time estimates. Applicable calibration contributes quality evidence within its recorded scope; its episode charges and timing do not replace the common reference workload. **Candidates to evaluate** denotes task-fit mode: quality is shown as unknown, alongside known reference costs and deployment requirements. When image separation is ambiguous, Scout first offers labels, bounding boxes or pixel masks/cutouts as output choices. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; Hugging Face results resolve the repository publisher, using a known lab mark or its bundled organization avatar; otherwise a Hugging Face mark identifies the repository platform. Missing or failed marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **API & MCP**, **Capabilities**, and **Sources** expand beneath the search and results. **Sources** uses a compact list; expand a source for its coverage, methods, limits, collection times and refresh failures.
 
 The capability index lists each area with its benchmark-series and model counts. After a search, task matches appear first with a check mark. Counts describe index coverage, not the number of models eligible for your task. The source panel describes what each feed measures, its coverage, source dates, archived results, and limitations. Collection time is separate from source dates, which can be publication dates rather than measurement dates. Scout task calibration appears separately with its tested pool, scope and seven-day expiry.
 
-**Provider page** in each model's evidence opens its provider's official pricing, model documentation, or website. Model links never fall back to a router. Token prices shown in Scout are still quoted via OpenRouter and may differ from direct provider rates. Benchmark links lead to the original evaluation sources.
+**Provider page** in each model's evidence opens its provider's official pricing, model documentation, or website. Model links never fall back to a router. Each quote identifies its source: a router route, independent provider metadata, or a reviewed provider contract. Routes have separate prices. Self-hosted infrastructure cost and incomplete quotes remain unknown. Hugging Face deployment links open the publisher’s model repository and pinned card. Benchmark links lead to the original evaluation sources.
 
 ## Connect a coding agent
 
@@ -28,7 +161,7 @@ Scout exposes one read-only tool, `scout_models`, with a required prompt:
 {"prompt":"Upload an invoice and extract six fields as JSON"}
 ```
 
-No API key is required from the caller. Scout sends the task description to OpenAI Decisions to interpret requirements, then ranks its collected evidence. It does not execute the recommended models. The MCP endpoint supports protocol versions `2025-03-26`, `2025-06-18`, and `2025-11-25`, including initialization, tool discovery, and tool calls. It uses JSON responses without sessions or a long-lived SSE stream. Browser requests must have the same origin; ordinary agent clients omit the Origin header. An unsupported protocol version is rejected. Agent configuration is client-specific; connecting an endpoint does not automatically make an agent use it.
+No API key is required from the caller. Scout sends the task description to OpenAI for Decisions interpretation, structured metadata extraction and a transient query embedding, then compares its collected evidence. It does not execute the recommended models. The MCP endpoint supports protocol versions `2025-03-26`, `2025-06-18`, and `2025-11-25`, including initialization, tool discovery, and tool calls. It uses JSON responses without sessions or a long-lived SSE stream. Browser requests must have the same origin; ordinary agent clients omit the Origin header. An unsupported protocol version is rejected. Agent configuration is client-specific; connecting an endpoint does not automatically make an agent use it.
 
 Tell the agent: “Before choosing an inference model for this feature, call Scout with the task description. Compare the five candidates, read their evidence and warnings, and evaluate the shortlist on our own examples.”
 
@@ -44,19 +177,21 @@ curl https://openmerit.ai/scout/api/recommend \
 
 `POST /scout/api/recommend` requires `prompt`: a non-empty string of at most 4,000 characters. The JSON body is limited to 18,000 bytes. Unknown properties are rejected.
 
-HTTP and MCP additionally accept optional `modelIds`, a unique list of up to 100 catalogue IDs. This filters eligible candidates before ranking; an empty or unknown-only list returns no models. The response echoes `selection.modelIds` (`null` for all). Scoped requests omit specialist discovery because it is outside the declared inventory. No endpoint URL or credential is accepted. The [agent evaluation runner](agent-evaluations.md) uses this for existing-project scope; the hosted API supports it as of 15 October 2026. The npm evaluation commands are available in `openmerit@0.1.7-preview.1` under the `preview` tag, separate from stable `0.1.6`.
+HTTP and MCP additionally accept optional `modelIds`, a unique list of up to 100 legacy OpenRouter catalogue IDs. This preserves the existing-project inventory contract; a model ID alone does not grant a direct-provider endpoint. To select specific routes instead, use `endpointIds`, a unique list of up to 100 IDs returned in `recommendations[].endpoint.id`. Both filters intersect if supplied. This filters eligible candidates before ranking; an empty or unknown-only list returns no models. The response echoes `selection.modelIds` and `selection.endpointIds` (`null` for unscoped). Scoped requests omit specialist discovery because it is outside the declared inventory. No endpoint URL or credential is accepted. The [agent evaluation runner](agent-evaluations.md) uses this for existing-project scope; the hosted API supports it as of 15 October 2026. The npm evaluation commands are available in `openmerit@0.1.7-preview.3` under the `preview` tag, separate from stable `0.1.6`.
+
+Recommendation `id` preserves legacy router IDs; a new protocol receives a distinct catalogue ID. `modelId` is the canonical model identity, while `endpoint.modelId` is the identifier to send to that provider. Clients must use the returned protocol and deployment requirements rather than treating every option as a chat-completions model. Existing npm clients using legacy `modelIds` remain scoped to their router inventory.
 
 The response contains:
 
-- `recommendations`: up to five distinct model IDs with rank, provider, reason, quality signal, evidence coverage, input/output pricing, reference timing when available, supported inputs, and source-linked evidence. `reference` gives cost in USD and completion seconds (`null` when unavailable). Benchmark mode includes assumed input/output tokens; measured mode sets those token counts to `null` and supplies `measured` with observed field accuracy, the reliability-capped `effectiveQuality`, cross-task `availability`, valid outputs, sample size, charge coverage, provider routes and evidence provenance. `comparison` gives the number of comparable candidates and average shared-capability coverage. The compatibility `pricing.blendedPerMillion` field retains its old 3:1 calculation but is no longer scored.
-- `task`: interpreted capabilities and weights; image-input, structured-output, tool-calling and persistent-memory requirements; `requirements.tools` (`yes`, `no`, `unspecified`); `requirements.memory` (`none`, `session`, `persistent`, `unspecified`); consequential `uncertainty`; `workload.input` and `workload.output` (`short`, `medium`, `long`), `workload.priority` (`balanced`, `quality`, `cost`, `latency`), `kind` (`extract`, `classify`, `dispatch`, `code`, `act`, `remember`, `sql`, `calculate`, `retrieve`, `other`), and `interpreter: "openai-decisions"`. `keywords` is retained as an empty compatibility field. A general-purpose fallback means no specific capability was identified, not that the classifier was unavailable.
-- `specialists`: a separate discovery object with up to three Hub repositories, metadata-based task fit, declared licenses, pinned model-card revisions, live hosting metadata when supplied, and original reported evaluation records. `mode` is `discovery`; quality, cost and latency are `null`. These candidates are not additional ranked places. Unsupported task types can omit this field.
+- `recommendations`: up to five distinct model IDs with rank, provider, `endpoint` (provider model ID, protocol, configuration, deployment, provenance and limits), reason, quality signal, evidence coverage, input/output pricing, reference timing when available, supported inputs, and source-linked evidence. `reference` gives cost in USD and completion seconds (`null` when unavailable). Benchmark mode includes assumed input/output tokens; measured mode sets those token counts to `null` and supplies `measured` with observed field accuracy, the reliability-capped `effectiveQuality`, cross-task `availability`, valid outputs, sample size, charge coverage, provider routes and evidence provenance. `comparison` gives the number of comparable candidates and average shared-capability coverage. The compatibility `pricing.blendedPerMillion` field retains its old 3:1 calculation but is no longer scored.
+- `task`: interpreted capabilities and weights; image-input, structured-output, tool-calling and persistent-memory requirements; `requirements.tools` (`yes`, `no`, `unspecified`); `requirements.memory` (`none`, `session`, `persistent`, `unspecified`); consequential `uncertainty`; `workload.input` and `workload.output` (`short`, `medium`, `long`), `workload.priority` (`balanced`, `quality`, `cost`, `latency`), `kind` (`extract`, `classify`, `dispatch`, `code`, `act`, `remember`, `sql`, `calculate`, `retrieve`, `judge`, `detect`, `segment`, `visual_unspecified`, `other`), `outputContract` (labels, decision, boxes, masks, text, or unspecified), `outputFormat` (`json`, `other`, `none`, or `unspecified`; only specifically established formats can match format-specific calibration), `outputCardinality`, optional `clarification`, and `interpreter: "openai-hybrid"`. `keywords` is retained as an empty compatibility field. A general-purpose fallback means no specific capability was identified, not that the classifier was unavailable.
+- `specialists`: a separate discovery object with up to three Hub repositories, metadata-based task fit, declared licenses, pinned model-card revisions, live hosting metadata when supplied, and original reported evaluation records. `mode` is `discovery`; quality, cost and latency are `null`. These candidates are not additional ranked places. Compatible vision deployments may instead enter a task-fit shortlist; they are not counted twice. Unsupported task types can omit this field.
 - `specialistRepositories`: bounded Hub catalogue count, separate from the evidence-backed `models` count.
 - `calibration`: released task profiles, scopes, sample sizes, measurement dates and expiry status.
-- `checkedAt`, `stale`, `degraded`, `sources`: index and source health, feed purpose, limitations and coverage. Each source has its own last successful check. A failed source is marked degraded; any source unchecked for more than two hours makes the index stale.
-- `benchmarks`, `benchmarksWithResults`, `models`, `providers`, `availableModels`, `eligibleModels`: indexed series, series with matched results, evidence-backed models, distinct indexed model providers (including alias consolidation), catalogue models, and eligible candidates for this task.
+- `checkedAt`, `stale`, `degraded`, `sources`: index and source health, feed purpose, limitations and coverage. Each source has its own last successful check. A failed source is marked degraded; any hourly source unchecked for more than two hours makes the index stale. Versioned endpoint contracts disclose their manual review date separately; a scheduled metadata refresh does not renew those price claims.
+- `benchmarks`, `benchmarksWithResults`, `models`, `providers`, `availableModels`, `eligibleModels`: indexed series, series with matched results, evidence-backed models, distinct indexed model providers (including alias consolidation), catalogue models, and eligible candidates for this task. `endpoints` counts hosted route listings, including reviewed typed contracts; `independentModels` counts catalogue identities with metadata independent of OpenRouter. Neither number implies credentials or verified account access.
 - `wordCloud`: capability vocabulary with benchmark and model coverage counts.
-- `ranking`: algorithm version, `mode` (`measured` or `benchmarks`), workload and tradeoff weights. A measured `profile` gives the task type, sample size, eligible tested pool, protocol hash, collection and expiry times, and public evidence URL. `benchmarkEligibleModels` reports the separate public-benchmark pool. Rank is the final order; the aggregate `score` is a supporting signal and can be non-monotonic because stronger direct comparisons take precedence.
+- `ranking`: algorithm version, `mode` (`measured`, `benchmarks`, `task-fit`, or `clarification`), workload and tradeoff weights. A measured `profile` gives the task type, sample size, eligible tested pool, protocol hash, collection and expiry times, and public evidence URL. `benchmarkEligibleModels` reports the separate public-benchmark pool. Rank is the final shortlist order. `ranking.frontier` reports compared configurations, selected distinct models and whether the objective scope is comparable or partial. The compatibility `score` can be `null`; there is no weighted quality/cost/time composite in frontier selection.
 - `warnings`, `methodology`, `interpretationMs`, `elapsedMs`: limitations, ranking method, Decisions request time, and measured server ranking time when the runtime provides a sufficiently precise clock (`null` otherwise). Cloudflare freezes its clock during synchronous work, so the hosted page reports measured HTTP round-trip time instead of claiming zero-millisecond computation.
 
 `pricing.url` is the official provider destination, not the source of the quoted rates. It is `null` for a provider without a reviewed destination; the page then omits the link. The same rule applies to MCP responses. `pricing.basis` identifies the price source and calculation assumptions.
@@ -67,7 +202,7 @@ Scout recommends models; it does not process an uploaded invoice, execute a task
 
 ## Real sources and their limits
 
-The hourly updater fetches seven public sources (six existing feeds and eight bounded Hugging Face queries) and prepares the index before requests arrive. Queries make one Decisions call for intent classification; they do not browse benchmark sites or ask a model to invent rankings.
+The hourly updater fetches eight independent public sources, including Models.dev and fourteen bounded Hugging Face pipeline queries. Two additional versioned provider-contract sources have explicit manual review dates. It prepares the index before requests arrive. Queries use batched Decisions questions for intent classification, a bounded verification pass for weak answers, and relevant specialist compatibility checks; they do not browse benchmark sites or ask a model to invent rankings.
 
 | Feed | Used for |
 | --- | --- |
@@ -77,9 +212,12 @@ The hourly updater fetches seven public sources (six existing feeds and eight bo
 | [OpenCompass / VLMEvalKit](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) | Visual understanding and OCR benchmark results. Its feed currently identifies itself as 17 September 2025. |
 | [Berkeley Function Calling Leaderboard (BFCL v4)](https://gorilla.cs.berkeley.edu/leaderboard) | Native function-calling single-turn, live, and multi-turn scores; separate memory scores for key-value, vector, and recursive-summarization stores. |
 | [OpenRouter](https://openrouter.ai/api/v1/models) | Available model IDs, input modalities, declared tools and structured-output support, context limits, and advertised token prices. |
+| [Models.dev](https://models.dev) | Independent native model IDs, declared capabilities and prices via ten direct-provider adapters: OpenAI, Anthropic, Google, DeepSeek, Mistral, xAI, Cohere, Moonshot, MiniMax and Z.AI. Not a performance benchmark or account-availability check. |
+| [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions) | Reviewed typed-choice, predicate and score endpoint contract, inputs and endpoint-specific rates. No general chat timing is inherited. |
+| [TypeSafe Jev](https://typesafe.ai/blog/jev-1-14) | Reviewed System One contract. Image and single-box output are experimental; headers, version and incomplete total pricing are disclosed. Jev is a candidate for compatible tasks, not Scout’s ranking engine. |
 | [Hugging Face](https://huggingface.co/models) | Public repository metadata, model-card results and structured evaluation records for specialist discovery. Separate from performance ranking. |
 
-Over 100 benchmark **series** are indexed, including source-specific implementations and versions. This is not 100 independent publishers, and coverage is not uniformly current. Some definitions have no results matched to an available model. Both counts are exposed. Missing metrics and ambiguous or unmatched model versions are omitted. Ranked availability is limited to OpenRouter's text-output catalogue, excluding batch/free variants and known routers. Specialist discovery also indexes repositories outside that catalogue.
+Over 100 benchmark **series** are indexed, including source-specific implementations and versions. This is not 100 independent publishers, and coverage is not uniformly current. Some definitions have no results matched to an available model. Both counts are exposed. Missing metrics and ambiguous or unmatched model versions are omitted. Model discovery merges OpenRouter with independent native-provider records before matching public benchmark results. An OpenRouter outage does not prevent an independent catalogue from bootstrapping. Distinct endpoints retain prices and parameters; duplicate hosting entries do not create additional model votes or consume multiple places in the five. Endpoint listings are metadata, not verified access in the caller’s account.
 
 Collection time is not measurement time. A successful hourly check does not make an old result new. Some feeds omit measurement dates; those remain unknown. Model release dates are never substituted. OmniDocBench's repository version and leaderboard version differ; Scout uses the version actually printed on the score table. Document parsing is a proxy for invoice extraction, not an evaluation of your exact six fields. New models without comparable evidence may be absent from a task's shortlist.
 
@@ -89,47 +227,51 @@ Epoch AI attribution: “Capabilities & benchmarking,” published at epoch.ai,
 
 Some specialist models have useful evaluations outside general LLM leaderboards. Scout reads both legacy model-card `model-index` results and the Hub’s [structured evaluation records](https://huggingface.co/docs/hub/eval-results). It retains the original dataset, task or metric, units, configuration, split, reported date, setup notes and source link when supplied. Scores are not averaged across unrelated datasets. A repository revision pins the metadata collected; it does not establish which checkpoint produced every reported score. Reported dates can also default to a commit date. Scout does not validate verification tokens or independently reproduce these scores.
 
-The current hourly discovery sample takes up to 30 repositories sorted by downloads for each of eight pipelines: document question answering, token classification, text classification, sentence similarity, text-to-text generation, text generation, image-to-text, and image-text-to-text. The two image queries are filtered to OCR. Duplicates, private or disabled repositories, adapters and quantization packages are excluded; fine-tunes remain separate. Downloads determine sampling coverage only, never quality. This is a bounded sample, not the whole Hub, and can miss low-download specialists. The initial snapshot contains 175 repositories, 49 with structured reported results; live counts change as sources refresh.
+The hourly discovery sample takes up to 30 repositories sorted by downloads for each of fourteen pipelines: document question answering, token classification, text classification, sentence similarity, text-to-text generation, text generation, image-to-text, image-text-to-text, image classification, zero-shot image classification, object detection, zero-shot object detection, image segmentation and mask generation. The two image-to-text queries remain OCR-filtered. Bounded pinned configs add label vocabularies for up to six repositories in each fixed-label vision pipeline. Private/disabled repositories, adapters and quantization packages are excluded; fine-tunes remain separate. Downloads select coverage only, never quality. This bounded sample can miss capable low-download specialists.
 
-When metadata matches the task, **Specialist models** expands beneath the five recommendations. Up to three matches cover different components, with task terms, metadata completeness and component diversity determining discovery order. Document OCR requires document-related metadata; incidental words such as “object” in “JSON object” do not select object detectors. Explicit supported-language declarations and known domain tags constrain matching, but metadata can still be incomplete or wrong. Tool execution and persistent-memory tasks do not receive unvalidated specialist matches. Entity taggers, OCR and embedding models are labelled as components requiring integration, not interchangeable chat APIs. Repeated repository names are de-emphasized for display diversity without merging their identities.
+For document, code and retrieval components, **Specialist models** expands beneath the five recommendations when compatible prepared profiles match. For classification, detection and segmentation, compatible native pipelines may enter the five **Candidates to evaluate** instead. Fixed-label pipelines need complete semantic coverage of the requested categories; one overlapping label is insufficient. Zero-shot pipelines accept supplied labels; masks can require points or boxes. Pipeline metadata is not an accuracy guarantee. Up to three components are ordered by metadata completeness and component diversity after compatibility checks. Domain and language requirements come from Decisions, with no prompt keyword dictionary. Tool execution and persistent-memory tasks do not receive unvalidated specialist matches. Entity taggers, OCR and embedding models are components requiring integration, not interchangeable chat APIs. Repeated repository names are de-emphasized for display diversity without merging their identities. See the release status in [semantic specialist matching](#semantic-specialist-matching-hosted-beta-15-october-2026).
 
 Cards show declared licenses, access gates, language, library, parameter count when known, and live hosting routes listed by the Hub. Missing hosting metadata does not prove that no provider exists. Size is not a hardware-fit estimate. Public weights do not establish an open-source license or permission for a particular use; check the pinned model card and actual license. Scout neither downloads weights nor executes repository code, and it does not send task descriptions to Hugging Face.
 
-Missing evaluations mean **unknown**, not poor performance. Discovery models do not enter the five through download counts or self-reported scores. Promotion is not automated: it requires a runnable model or endpoint and comparable task checks, full workload cost and timing under a recorded deployment configuration. Self-hosted cost depends on hardware and utilization; a zero token price would be misleading. The same quality/cost/speed comparison can then be used without a provider preference. Model-specific exceptions and inferred free inference are not used.
+Missing evaluations mean **unknown**, not poor performance. Task-fit candidates do not receive quality scores through download counts, self-reported results, or general reasoning benchmarks. A performance ranking requires comparable task checks, full workload cost and timing under a recorded deployment configuration. Self-hosted cost depends on hardware and utilization; a zero token price would be misleading. The same quality/cost/speed comparison can then be used without a provider preference. Model-specific exceptions and inferred free inference are not used.
 
 ## Ranking
 
-One server-side call to [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions), using `gpt-6-luna`, interprets the work the selected model must perform. It distinguishes executing tools from writing code about them, incidental words such as “tool” or “agent,” supplied plain text versus visual inputs, and session state versus persistent memory. Eighteen typed choices become capability weights, requirements, reference input/output sizes, a tradeoff priority, and a narrow task operation. One function dispatch is distinct from a dependent action loop. Typed code, action, and external-memory operations keep the independent capability answers coherent: an action or memory-tool operation cannot silently become a reasoning-only task. SQL generation counts as coding. Required extraction is not downweighted merely because reasoning is marked useful. Long-context requirements imply a long reference input. Visual input uses at least the medium category, even when the task description is short. Decisions sees the task, not candidate names or rankings; the shortlist still comes from collected scores.
+[OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions), using `gpt-6-luna`, interprets the work, capability requirements, operation, answer space, serialization and tradeoff priority. It distinguishes writing source from executing it, supplied categories from open identification, text-bearing objects from reading their contents, and session state from persistent memory. Typed answers are validated and checked for consistency. Uncertain answers remain disclosed; they cannot authorize narrow calibration evidence. No prompt keyword rules or model-name bonuses enter selection.
 
-Strong JSON/schema filtering also needs explicit format wording in the prompt. An uncertain format answer cannot turn an ordinary story or comparison into a JSON task. Low-confidence consequential classifications and unspecified retention/tool interfaces are exposed as uncertainty. This is still a probabilistic classifier: paraphrases can differ, ambiguous requests can be misread, and English format-grounding rules limit multilingual handling. Scout has no clarification dialogue or user-defined constraints yet.
+A Structured Outputs call first extracts downstream work, explicit model/endpoint restrictions and preferences using the published metadata definitions and canonical vocabulary. Mandatory conditions use alternatives of complete AND groups; exclusions and arbitrary numeric/date bounds are preserved. Relative dates use a supplied current UTC reference. Generic typed comparisons enforce those conditions and intersect optional `modelIds`/`endpointIds` inventories. Missing facts cannot satisfy a restriction, including an exclusion. Unexpressible mandatory restrictions produce no verified matches rather than silently broadening the search. Names mentioned as task data, clients or comparison subjects are not automatic filters. Interpretation is probabilistic and can still misread a prompt.
 
-Image tasks require an image-capable model. JSON/schema tasks require declared `response_format` or `structured_outputs` support. Visual document tasks require document-specific evidence. Required tool use needs declared `tools` support and relevant tool-calling evidence. A matching measured task profile can supply this evidence; the benchmark fallback requires published results for the capability. Persistent memory needs declared tools support and dedicated memory evidence; generic reasoning or a large context window cannot substitute. Session-only state does not trigger this filter. Recognized image/video/audio generation and transcription tasks return no recommendations because those outputs are outside the current index.
+Image tasks require declared image input and relevant visual evidence in performance ranking. Document reading needs document evidence. JSON/schema tasks need declared `response_format` or `structured_outputs`; required tool execution needs tools support and relevant tool evidence. Persistent memory requires external-memory evidence rather than a large context window. [BFCL memory evaluations](https://gorilla.cs.berkeley.edu/blogs/16_bfcl_v4_memory.html) do not establish built-in memory or performance on the visitor's storage system. Recognized image/video/audio generation and transcription remain outside the supported output index. Ambiguous spatial output produces the existing clarification choices.
 
-BFCL contributes only native function-calling (`FC`) configurations, preserving their published labels. Its six correlated series count as one benchmark family, not six independent sources. Missing scores stay unknown and zero remains a score. Model versions are matched conservatively; ambiguous versions are omitted. BFCL does not supply per-model measurement dates, so these remain unknown.
+Classification and judgment tasks first use applicable benchmark evidence. With no supported benchmark candidates, supplied-label classification and judgments retain the explicitly labelled **task-fit** fallback. Boxes and masks use their compatible task-fit path. Documented operation/output fit, deployment readiness and known reference costs/times determine unmeasured shortlists; quality stays unknown. Open or uncertain identification instead needs name-generating endpoints and relevant evidence; fixed-choice APIs cannot satisfy it. Specialist compatibility uses hourly-prepared public profiles. Unknown compatibility omits a repository with a warning, rather than assuming task accuracy.
 
-The [BFCL memory evaluations](https://gorilla.cs.berkeley.edu/blogs/16_bfcl_v4_memory.html) test using external memory stores across sessions. They do **not** establish built-in model memory, automatic retention in your app, or performance on your storage system. The application still implements storage and tool execution. Without a matching measured profile, models without matched memory results are excluded when persistent memory is required, including newer models with other strong scores.
+For performance ranking, a query embedding retrieves benchmark methodologies, supplemented by their hourly capability memberships. Bounded independent Decisions predicates verify relevance and a deliberately scored required ability before a methodology contributes proxy evidence. Similarity never supplies a quality score. Rejected, unknown and unavailable evidence is withheld. If some checks fail, verified evidence can still be used with an explicit warning; if no applicability can be verified because checks are unavailable, the search returns unavailable rather than reporting a false no-match.
 
-Ranking version 4 first checks for a matching, recent **measured task profile**. The static evidence file contains only calibration aggregates: mean per-case task-check accuracy, mean reported charge, median full task time, valid-output counts, provider routes, and provenance. Malformed schemas and request failures receive zero accuracy; failure time remains in the latency median. Unknown charges are excluded from the cost mean and the known-charge count is disclosed. A model with no known charges earns no cost advantage. For multi-call episodes, reported cost includes every call and elapsed time covers the whole episode, including reasoning, tool round trips, and all memory sessions. Request reliability is shared across calibration tasks: after two or more failed requests, the observed HTTP success rate caps the model’s task-quality signal. A clean task slice cannot erase repeated route failures elsewhere. One isolated failure remains in its own task score rather than penalizing every other task. This is route evidence from the evaluation period, not a promise about current provider availability.
+Quality comparisons use shared benchmark results, grouped by correlated family and task capability. Different cohorts are not compared through unrelated percentiles. BFCL uses native function-calling configurations with their original labels; correlated series are one family. Missing scores remain unknown and actual zero remains a score. Model/version matching remains conservative. A relative quality signal is not predicted accuracy or a statistical confidence interval.
 
-Profiles cover small text-field extraction, rule-based classification, single-function dispatch, read-only SQL, short numerical problems, long-text record retrieval, visual-field extraction, dependent tool actions, and external memory across sessions. The matcher also supports code-function operations, but their proposed measured profile and an additional medium-input extraction profile were withheld after fresh validation regressions; those tasks retain their prior evidence path. Matching requires the same typed operation, image/tool/schema requirements, and input/output size category. A new required capability outside the profile prevents a match. Retention scope (none, session, persistent) and whether execution uses an action loop must also match; tool dispatch, dependent actions, and cross-session memory are separate operations. At least five currently eligible tested models and six calibration cases per model are required. Availability and declared capabilities still come from the refreshed catalogue. Evidence expires after seven days; expired, undersampled and unmatched profiles fall back to public benchmarks.
+Matching calibration observations can supplement quality for their exact operation, format, modality, tools, retention and workload scope. Each admitted model needs at least six calibration cases, and evidence expires after seven days. A match no longer requires five tested models or excludes all untested eligible models. Observations remain tied to their tested router deployment. Repeated calibration request failures still cap effective quality. Calibration episode bills and durations remain source evidence; they are not substituted for only some candidates' reference costs or times. Held-out outcomes and answer keys never enter runtime ranking.
 
-When several profiles match, the most recently measured complete profile is used. Measured mode ranks **within its disclosed tested pool**, not every catalogue model. It applies the same pairwise policy to actual observed accuracy, charges and time. Memory quality uses correct updates, retrieval, and final store state across fresh conversations. In offline candidate evaluations, code uses executable hidden checks and tools use permitted actions and their prerequisites. If the task does not require strict JSON, evaluation-only JSON serialization does not erase correct actions or facts. Tool profiles score observable actions, prerequisites, and a nonempty final reply; they do not grade the precision of its prose. Memory facts accept a JSON object surrounded by prose or a fence. Raw JSON-valid counts remain separate. A strict-JSON task cannot match these non-strict profiles. A ten-percentage-point observed accuracy gap gives full pairwise quality preference. These are small development samples, not confidence intervals or proof of accuracy on an arbitrary prompt. Text cases are authored; the visual calibration set contains eight public Portuguese invoices. Classification, arithmetic and simple dispatch can saturate at perfect accuracy, leaving cost and variable latency to determine close positions. Public calibration aggregates and their protocol hash are available through each model’s **Measured task evidence** link. Raw answer keys and held-out test outcomes are not runtime inputs. The source images are attributed to [Cruz and Castelli's invoice dataset](https://zenodo.org/records/6371710), under that dataset's CC BY 4.0 terms.
+All eligible verified catalogue configurations with relevant evidence enter numerical comparison before model deduplication. A configuration can dominate another only within a comparable quality scope, with known quality, cost and time: it must be at least as good in all three and strictly better in one. Missing objectives and incomparable coverage remain provisional. This cannot establish a global three-objective frontier over every model in the world.
 
-The **benchmark fallback** compares each pair only on benchmarks both have run. It does not compare percentiles from different cohorts. Differences are averaged within correlated families, then across shared task capabilities. The first two percentage points are treated as an indifference band; a further twenty-point gap gives full quality preference. This deliberately conservative proxy policy prevents a small benchmark lead from outweighing a large cost/time premium. It is not a statistical significance test. Format-following evidence has one-quarter weight only when another substantive capability is **required**, rather than merely useful.
-Each comparison also considers relative cost and completion time. The default balanced priority uses **60% quality, 20% cost, 20% latency**. An explicit quality priority uses 90/5/5; cost uses 60/35/5; latency uses 60/5/35. These are transparent defaults, not learned optimal weights. Decisions can infer a priority from the prompt; there are no new controls or extra interpretation calls. Broad or ambiguous preferences can still be misclassified.
+Up to five distinct models represent the supported tradeoffs. Explicit priorities and soft metadata preferences order eligible frontier points, followed by supported quality and known cost/time. Default selection reserves no cheapest or fastest slot. Preferences do not broaden eligibility. There is no weighted quality/cost/time composite. Fewer than five points can remain even when more models are eligible; the API reports eligibility separately from the selected count. Stable IDs resolve exact ties.
 
-The strongest supported pairwise preferences are applied first, skipping any that would create a cycle, to order the whole candidate pool before taking five. This prevents a model's easy wins over weak candidates from automatically overriding its direct comparison with another finalist. Strength is reduced when fewer task capabilities have shared evidence. Unsupported comparisons add no pairwise preference. A conservative aggregate signal breaks otherwise unresolved orderings; stable IDs break exact ties. Missing measurements are never treated as zero cost or instant completion. Warnings expose limited comparison coverage.
+Reference input sizes are **1,024 / 4,096 / 32,768 tokens**, and visible output sizes are **128 / 512 / 2,048 tokens**, for short / medium / long. Supplied-label choices use a 16-token output reference. Reference cost is input tokens × advertised input rate + output tokens × advertised output rate. This excludes hidden reasoning, cache effects, image-specific tokenization and other modality charges. Decisions has no output-token charge. Unknown or self-hosted operating cost is not inferred to be zero.
 
-Reference input sizes are **1,024 / 4,096 / 32,768 tokens**, and visible output sizes are **128 / 512 / 2,048 tokens**, for short / medium / long. Cost is input tokens × advertised input rate + output tokens × advertised output rate. This excludes hidden reasoning, cache effects, image-specific tokenization and other modality charges. An image task's input size is an illustrative token workload, not a prediction of that image's bill. Separate rates and the assumptions are returned so clients can substitute their own measured usage.
+Reference completion time is **time to the first answer token + visible output tokens / throughput**, only when measurements identify the selected endpoint. Multiple published configurations retain paired timing before taking a median. Timing is not borrowed across provider routes, and early reasoning tokens are not confused with the first answer. Missing timing stays unknown. The page's request duration measures Scout matching, not a recommended model executing the visitor's task.
 
-Reference completion time is **time to the first answer token + visible output tokens / throughput**. When the source publishes multiple configurations, Scout calculates completion time for each paired measurement and takes the median. It does not mix one configuration's delay with another's throughput, or confuse early streamed reasoning with the first answer. Missing first-answer timing for a reasoning configuration stays unknown. A legacy snapshot with only first-token timing supplies no completion estimate; deployment replaces an older data format with the bundled compatible snapshot even if the legacy refresh timestamp is newer. These measurements are from Artificial Analysis, not the caller's OpenRouter route; provider and reasoning settings may differ. They are not guaranteed task latency.
+The shortlist requires evaluation on the visitor's own examples. Public benchmarks, small calibration samples and semantic applicability checks do not prove the optimal membership or order of five models for an arbitrary task. There are no visitor accounts, public prompt history or additional filter controls. Public methodology embeddings are stored in Vectorize; visitor query vectors are transient. Private prompt-and-evidence analytics are described below.
 
-Matching is deliberately narrow; code-function tests do not establish repository-level engineering skill, and small tool loops or key-value memory tasks do not establish long-horizon agent reliability. A profile label and quality basis are returned with every measured shortlist. No model-name bonuses, prompt-example rules, or task answer keys enter the ranking.
+## Current validation limits
 
-The goal is an ordered five-model shortlist, not just a first-place pick. Regression checks cover all five positions, all ten within-five pairs, cutoff membership, provider-name invariance, unknown cost/time, capability and modality boundaries, expiry, and rejection of held-out observations during evidence building. Paid evaluation uses separate calibration and held-out cases across multiple task types. Held-out measurements compare both membership and ordering within the tested candidate pool; they do not establish a global optimum or a perfect ordering for every prompt. Test the five candidates on your own examples before choosing a production model.
-If fewer than five eligible models have evidence, Scout returns fewer and explains why. There are no accounts, saved prompt history, vector database, user selection controls, or language-model reranker.
+The evidence architecture is validated against the published catalogue, not a million-model production load. Synthetic 10,000-configuration checks cover comparable and disconnected evidence scopes; they do not establish request latency with dense, mixed benchmark coverage. Discovery remains incremental and neither coverage counts nor a shortlist establish a globally verified optimum.
 
-## Current validation limits
+The identification correction was checked on the original bird-name prompt, plant-species identification, supplied indoor/outdoor choices and supplied ticket categories with strict JSON, using real prompt interpretation. Regression checks also cover missing/uncertain label sets, explicit endpoint scopes, specialist discovery, visual evidence and classification calibration boundaries. These checks verify interpretation and eligibility, not the accuracy of model-generated bird or plant names.
+
+The earlier 15 October endpoint-catalogue revision passed **20/20 fresh task-interpretation cases** with expectations frozen before dispatch. They covered image/text labels, predicates, rubric scores, single/all-instance boxes, masks, ambiguous separation, visual/text extraction, code, SQL, tool loops, memory, prose and unsupported generation. A prior 18-case development check exposed two routing errors; those cases were retired before the final set. Unit checks cover output-contract eligibility, exact endpoint scoping, name invariance, duplicate endpoints and unknown prices. This validates tested routing boundaries, not best-five task performance.
+
+A separate execution check used the same twelve 32×32 images from the public [CIFAR-10 test split](https://huggingface.co/datasets/uoft-cs/cifar10), rows 80–91, with ten category labels. Decisions `gpt-6-luna` returned valid choices for 12/12 and matched 5/12 labels, with 0.344s median request time and $0.0002148 total estimated input-token charges at the published rate. This is an estimate, not reconciled billing. A local CPU run of `Xenova/clip-vit-base-patch32`, ONNX q8 through Transformers.js, matched 7/12, with 0.022s median warm inference and about 8.6s initial loading. Local hardware and operating costs were not measured and are not zero. The deployment configuration and cold-start difference matter; these timings are not a universal speed comparison. Public test images may have appeared in training.
+
+These small, low-resolution results are **not ingested into production ranking** and do not establish the correct order of five endpoints. They also do not validate the original full-precision CLIP weights or other checkpoints. Jev is indexed from its published task/endpoint contract and was not executed without its separate credential. Task-fit mode keeps quality unknown instead of promoting cheap or specialized options through an invented accuracy score.
 
 The latest 15 October coding/extraction follow-up made **132 requests**: six new calibration cases and six disjoint new holdout cases per category, with six coding candidates and five extraction candidates. These are authored application scenarios, not private production records. Predictions, scorer and evidence were frozen before holdout dispatch. The v4 quality/cost/speed policy was unchanged. Candidate evidence used only the new calibration split; no model-specific rule or held-out result entered it.
 
@@ -170,11 +312,19 @@ The first test exposed cross-task request failures; those test outcomes were not
 
 ## Refresh and data handling
 
-The task description is sent to Scout and then to OpenAI Decisions for interpretation. Scout's application code processes it in memory without saving or logging it. Only the public index and published calibration aggregates are persisted; prompts are not sent to benchmark sites or recommended-model providers. [OpenAI's API data controls and retention policies](https://developers.openai.com/api/docs/guides/your-data) apply to that provider call. Scout does not claim zero retention by OpenAI. Hosting infrastructure can have separate request-metadata policies. Clipboard actions occur only when selected. Describe the task rather than pasting actual invoices, credentials or private customer records.
+The task description is sent to Scout and then to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, a transient query embedding and, when relevant, specialist compatibility/applicability checks. Hourly preparation also sends public repository metadata and benchmark definitions/methodology to Decisions; no user prompts are involved in that refresh. Scout stores the submitted prompt alongside its recommendations in private Cloudflare Analytics Engine records. Runtime request logging remains disabled. The public index, published calibration aggregates and the limited search analytics described below are persisted; prompts are not sent to benchmark sites or recommended-model providers. [OpenAI's API data controls and retention policies](https://developers.openai.com/api/docs/guides/your-data) apply to those provider calls. Scout does not claim zero retention by OpenAI. Hosting infrastructure can have separate request-metadata policies. Clipboard actions occur only when selected. Describe the task rather than pasting actual invoices, credentials or private customer records.
+
+Scout records one server-side Analytics Engine event for each valid recommendation attempt on the website/JSON API or MCP. It keeps the original submitted prompt (up to 4,000 characters, including whitespace), fixed task categories, capability/input/output requirements, priority and workload buckets together with the ordered model/endpoint shortlist (up to five), its quality/cost/time values and their calibration/reference/unknown basis, attached source evidence, ranking and interpreter versions, index date, scope category, freshness, outcome and server duration. Clarifications, unsupported tasks, empty results, admission limits and interpretation errors are distinct outcomes. A failure before interpretation has an unknown task category. Invalid requests rejected before recommendation, status checks and MCP discovery are not search events. Website and other JSON API calls share the `api` channel; MCP uses `mcp`.
+
+Apart from the submitted prompt and allowlisted public source evidence, analytics excludes free-text task summaries, prompt hashes, requested inventory IDs, endpoint API addresses, error messages, IP addresses, user agents, cookies, visitor IDs and session IDs. Prompt text is not redacted: personal information or secrets typed into it would be retained, so these records are not guaranteed to be anonymous. No visitor identity is collected separately or attached to a search. It adds no browser tracker or inference call. Analytics writes are best effort and cannot interrupt recommendations. Cloudflare [retains Analytics Engine events for three months](https://developers.cloudflare.com/analytics/analytics-engine/limits/) and may sample them; this is usage evidence, not an exhaustive audit log or evidence of task success. Hosting infrastructure and the existing Decisions provider call have their own data policies.
 
-Each query has one Decisions request with a four-second timeout and no automatic retry. Shared in-memory admission limits allow at most eight active requests and 120 starts per minute for the Scout service. Excess requests fail promptly instead of queuing. These counters reset with the process and are not an account-level billing cap. Provider-side limits can also make interpretation unavailable. The API key is held server-side; callers never receive it.
+Search records are private Cloudflare data, queried on demand through authenticated account access. Scout has no report generator, analytics dashboard, analytics sign-in, or public history/query endpoint. Each new record keeps the submitted prompt, the ordered five (or fewer), and a snapshot of their supporting evidence together: benchmark scores, source links/dates and the prepared capability/family interpretation version/time, applicable calibration measurements and protocol, advertised pricing, speed sources, and endpoint/model-card task-fit evidence, including the prepared semantic profile and compatibility decision for a selected Hub deployment. Capturing source facts at recommendation time avoids relying only on links whose contents can change. Unknown measurements remain unknown, and reported model-card results remain unverified claims.
 
-Sources refresh independently. A failed download, changed parser format, or unexpected loss of coverage preserves that source's previous evidence and timestamp while healthy sources and prices continue updating. Fresh catalogue removals still remove unavailable models. All eight Hub queries must succeed for its discovery snapshot to replace the previous one. A complete outage preserves the previous index. Failure and stale status are exposed in both interfaces.
+Earlier records are preserved as collected: version 1 has no prompt, and versions 1–2 have no attached evidence snapshot. Missing historical fields cannot be reconstructed. New records use lossless compression when needed to fit Cloudflare's event-size limit. Writes remain best effort; a record too large to fit is not silently truncated or presented as complete.
+
+Each query starts with batched Decisions questions and can make one verification pass for weak answers, followed by relevant specialist compatibility batches. Task interpretation and specialist comparison share a four-second request budget; catalogue reads and ranking add their own time. Optional verification has at most 750 milliseconds and leaves weak answers uncertain if unavailable. Independent specialist batches run in parallel, with at most 100 questions per call. Transient connection, timeout, rate-limit and server failures can receive one retry within the remaining deadline; valid `Retry-After` delays are honored. Authentication, exhausted-credit and quota failures are not retried. Failed interpretation returns an explicit unavailable response; failed specialist checks omit unverified specialist candidates. No keyword fallback is used. These bounds reduce avoidable waits and failures; they do not guarantee a successful search or a particular latency. In-memory admission allows 32 active requests and 600 starts per minute per Worker isolate; a separate edge limiter allows 60 POST requests per minute per IP. Excess requests fail promptly instead of queuing. These are operational limits, not an account-level billing cap or a throughput guarantee. Credentials remain server-side.
+
+Hourly sources refresh independently; reviewed typed-endpoint contracts retain their manual review dates. Freshness follows each source's prepared policy: public feeds have a two-hour freshness allowance, while incremental research uses the configured seven-day recheck interval. A source does not become stale merely because it follows a longer research interval. Missing or invalid check dates remain overdue, and actual collection or preparation failures remain degraded. A failed download, changed parser format, or unexpected loss of coverage preserves that source's previous evidence and timestamp while healthy sources and prices continue updating. Fresh catalogue removals still remove unavailable models. All fourteen Hub queries and required semantic preparation must succeed for its discovery snapshot to replace the previous one. A complete outage preserves the previous index. Failure and stale status are exposed in both interfaces.
 
 Source content is parsed as data, never executed. Downloads, decompression, and request bodies are size-bounded; source downloads time out. Links are restricted to HTTP(S). The publisher's OpenVLM data URL uses HTTP and lacks transport authentication; the remaining feeds use HTTPS. Scout does not independently authenticate benchmark submissions.
 
@@ -187,15 +337,19 @@ npm ci
 npm run scout
 ```
 
-Open `http://127.0.0.1:45833/scout/`. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.
+Open `http://127.0.0.1:45833/scout/`. This lightweight local preview uses the shared interpreter and matcher with bundled/source-feed data; it does not run the hosted Parallel research workflow or R2 publication system. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.
 
-`npm run scout:refresh` fetches the seven public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Refreshing public sources requires no provider key or inference budget. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid hourly evaluation job is installed. Recommendation queries require the server key and incur Decisions API usage.
+`npm run scout:refresh` fetches the eight hourly public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Source downloads need no provider inference, but preparing new or changed specialist profiles and benchmark interpretations requires `OPENAI_API_KEY` and incurs Decisions usage. Unchanged metadata reuses cached analysis. Without a usable key, required specialist preparation retains the previous specialist snapshot; affected new or changed benchmark definitions remain unclassified. Both failures are reported on the affected source. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid hourly evaluation job is installed. Recommendation queries require the server key and incur OpenAI interpretation/embedding usage. The local source-refresh command updates the bundled feeds; the hosted hourly Workflow additionally prepares and publishes its D1/Vectorize serving generation.
 
 ## Hosting and verification
 
-`wrangler.scout.jsonc` defines a separate Scout Worker, one SQLite Durable Object holding the index, and an hourly cron at minute 7 UTC. Configure `OPENAI_API_KEY` as a secret on this Scout Worker before deployment. Searches interpret the task once and use the prepared in-memory index. The bundled snapshot allows immediate startup; subsequent successful updates persist across restarts. Refreshes are serialized.
+`wrangler.scout.jsonc` defines the Scout Worker, a private R2 catalogue, an hourly Workflow and a SQLite Durable Object coordinating publication. The cron runs at minute 7 UTC. Configure `OPENAI_API_KEY` and `PARALLEL_API_KEY` as Worker secrets. Refreshes build immutable catalogue versions and prepared candidate pools; searches read those pools directly and never pass through the publication coordinator. A running refresh does not block searches against the active catalogue. The first hosted publication must be validated before serving searches; a missing catalogue fails explicitly. The Worker is `openmerit-scout`, the private catalogue bucket is `openmerit-scout-catalogue`, and the Workflow is `openmerit-scout-refresh`. Both API keys are stored as Cloudflare Worker secrets and read from server-side bindings; they are not shipped to the browser. Separate workers.dev and preview URLs are disabled.
+
+Publication records a durable completion receipt after updating the active pointer, pending-research queue and health status. Interrupted publication resumes unfinished work; a lost success reply can be acknowledged again after the lease is released. Replaying a completed publication cannot roll back a newer catalogue or clear a later refresh's lease. Completed research receipts leave the queue only when publication commits; unfinished research retains its original paid job for a later hourly run.
+
+Prepared capability pools can overlap. Searches use their prepared ID lists to choose a smaller covering set of pools, preserving the selected candidates and complete evidence while avoiding repeated downloads of the same model records. Explicit model and endpoint inventories retain their existing scope checks. These metadata reads do not fetch source pages or redo hourly research.
 
-`demo/domain-router.mjs` routes Scout paths to this service while retaining the existing website, protected memo, authentication, and dashboard services. The public guide is generated and published with the main documentation. Do not deploy the older root `wrangler.jsonc` to publish Scout. `npm run build:scout` packages only Scout, public documentation, and its shared assets; private memo and demo applications are excluded.
+`demo/domain-router.mjs` routes Scout paths to this service while retaining the existing website, protected memo, authentication, and terminal services. There is no Scout admin/report route. The public guide is generated and published with the main documentation. Do not deploy the older root `wrangler.jsonc` to publish Scout. `npm run build:scout` packages only Scout, public documentation, and its shared assets; private memo and demo applications are excluded.
 
 ```sh
 npm run check:scout

package/docs/security.md

@@ -43,3 +43,14 @@ Coding-harness histories are outside the terminal's application scope and are no
 The local web service binds to `127.0.0.1` and rejects unexpected Host, Origin, and cross-site browser requests. It only serves bundled assets and specific read-only JSON routes. It is not authenticated against other local processes or machine users. Model names, endpoint identifiers, and relative source paths can still be sensitive. Names and identifiers in imported metadata must be kept free of secrets.
 
 Repository metadata and usage caches live in process memory and disappear when the terminal exits. The terminal creates no persistent index or files in the application. Per-file/parser protections, request-log read limits, runtime inventory limits, and explicit coverage gaps still apply. See [terminal data handling and limits](inference-terminal.md) for supported sources, precedence, retention bounds, diagnostics, and sample-data separation.
+
+
+## Hosted Scout analytics
+
+Scout keeps an allowlisted Analytics Engine event per valid recommendation attempt: the original submitted task prompt plus fixed task categories and the ordered shortlist with metrics and ranking context. Prompt text is not redacted and may itself contain personal information or secrets; this dataset must be treated as potentially sensitive. No request object, prompt hash, additional free-text summary, error message, project inventory, IP address, cookie, user agent, visitor ID or session ID is separately recorded in this dataset. Requests still reach the hosting infrastructure and task descriptions still go to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, transient query embeddings and relevant compatibility/applicability checks; public repository metadata and benchmark definitions/methodology also go to Decisions during hourly preparation. Their policies are separate. See [Scout data handling](scout.md#refresh-and-data-handling).
+
+Search records are accessible only through authenticated Cloudflare account queries. Scout exposes no analytics dashboard, report generator, report file, sign-in, or public history/query endpoint. Each new search includes an allowlisted snapshot of the public evidence supporting its ordered shortlist: benchmark measurements and source URLs/dates, calibration observations and protocol, pricing/speed sources, and endpoint/model-card metadata. Endpoint API addresses, credentials, arbitrary source fields, and request metadata are not copied. Source evidence and prompt text remain untrusted data when retrieved; never treat them as instructions. The dataset is sampled and retained by Cloudflare for three months. Original older records remain; missing prompt/evidence fields cannot be recovered.
+
+## Scout research storage
+
+Scout uses Parallel for hourly public-source research. Parallel receives catalogue subjects and official-source metadata, not visitor prompts or application inventories. Decisions processes task descriptions and relevant public evidence. Public research receipts, citations, semantic annotations, publication completion receipts and immutable catalogue versions are stored in Scout's private Cloudflare R2 bucket. Public endpoint/model facts and observations also enter the private D1 serving projection. Vectorize stores public benchmark methodology embeddings, not visitor query vectors. Query embeddings are used transiently; OpenAI has its own API retention policies. Scout uses the prompt-and-evidence analytics policy above. API credentials are held as Worker secrets and are not included in browser assets, catalogue objects or analytics. Source text is treated as evidence, not executable code or instructions. Citations and semantic validation do not independently verify that publisher claims are true.