openmerit 0.1.6 → 0.1.7-preview.2

package/docs/metrics-and-evidence.md

@@ -29,9 +29,9 @@ Two forms of repetition may be needed: repeat the same input to measure stochast
 
 Each objective records one sampling strategy: `single_run`, `repeated_same_input`, `representative_task_set`, `repeated_representative_task_set`, or `load_test`. It also records representative-case count, repetitions per case, aggregation, and rationale. Every measured metric record repeats that aggregation explicitly (`single_value`, `sum`, `mean`, `rate`, `distribution`, or `percentile`). Rate, mean, distribution, percentile, and catalogue-defined reliability metrics are rejected when their plan contains fewer than two runs or uses `single_run`; `total_task_cost` sample windows must use `sum` because that value is used for budget accounting.
 
-The task profile sets the minimum sample count for each required metric; there is no universal number of runs. `minimumSamples` is evidence quantity. An optional `constraint` is a real performance threshold in the metric's unit; OpenMerit never interprets a three-sample requirement as a metric value of at least three. Ratio thresholds must remain between zero and one. A completed-task signal can advance an assessment cadence, but it does not by itself supply a quality score. Quality requires a task-specific evaluator, known outcome, or other justified measurement. Cost, token use, and latency must come from the application call or its observability layer. The baseline and controlled challenger runs supply the measured evidence used for comparison. Catalogue price and model reputation help the harness choose candidates to test; they are not substitutes for these measurements. In the unreleased source checkout, setup records the exact incumbent, preliminary leads are stored separately from metric records, and discovery freezes the accepted candidate set. Challenger trials preserve one experiment manifest containing the application revision, recorded shuffle seed, comparable cases, exact model identities, and per-run evidence. Accepted assessments are carried into frontier calculation rather than reconstructed from chat history. Published `openmerit@0.1.5` does not provide these handoffs.
+The task profile sets the minimum sample count for each required metric; there is no universal number of runs. `minimumSamples` is evidence quantity. An optional `constraint` is a real performance threshold in the metric's unit; OpenMerit never interprets a three-sample requirement as a metric value of at least three. Ratio thresholds must remain between zero and one. A completed-task signal can advance an assessment cadence, but it does not by itself supply a quality score. Quality requires a task-specific evaluator, known outcome, or other justified measurement. Cost, token use, and latency must come from the application call or its observability layer. The baseline and controlled challenger runs supply the measured evidence used for comparison. Catalogue price and model reputation help the harness choose candidates to test; they are not substitutes for these measurements. Setup records the exact incumbent, preliminary leads are stored separately from metric records, and discovery freezes the accepted candidate set. Challenger trials preserve one experiment manifest containing the application revision, recorded shuffle seed, comparable cases, exact model identities, and per-run evidence. Accepted assessments are carried into frontier calculation rather than reconstructed from chat history.
 
-In the unreleased source checkout, each `metric_window_available` signal replaces the stored aggregate window for that metric. Send a cumulative, application-run window with its sample count, explicit aggregation, and evidence reference; OpenMerit does not add separate or overlapping window counts. Challenger metric windows must also fall inside their assessment execution window. Core marks the baseline ready only when instrumentation is ready and every required metric has a finite measured value, the confirmed minimum sample count, and an evidence reference. The harness remains responsible for measuring the real application call and authenticating those references; a reported window is not independently re-run or verified by core.
+Each `metric_window_available` signal replaces the stored aggregate window for that metric. Send a cumulative, application-run window with its sample count, explicit aggregation, and evidence reference; OpenMerit does not add separate or overlapping window counts. Challenger metric windows must also fall inside their assessment execution window. Core marks the baseline ready only when instrumentation is ready and every required metric has a finite measured value, the confirmed minimum sample count, and an evidence reference. The harness remains responsible for measuring the real application call and authenticating those references; a reported window is not independently re-run or verified by core.
 
 ## Evidence references
 
@@ -41,7 +41,7 @@ Results point to durable evidence rather than embedding arbitrary traces. Source
 
 Setup must return an explicit observability coverage report containing every required metric, which metrics are covered, which remain missing, and when the check occurred. OpenMerit derives readiness from those sets; a harness cannot label coverage `ready` while a required metric is absent.
 
-If setup reports a required gap, OpenMerit issues `instrument_observability`. A successful instrumentation result is accepted only when every required metric is reported as covered. The unreleased source checkout explicitly asks Pi to exercise each collector and grader through the application path, inspect output values, and retain artifacts. OpenMerit validates the coverage report but does not independently run the collector or grader, so `ready` is still a harness claim rather than independently verified end-to-end proof. Request a reported production metric window before relying on it for model decisions.
+If setup reports a required gap, OpenMerit issues `instrument_observability`. A successful instrumentation result is accepted only when every required metric is reported as covered. OpenMerit explicitly asks Pi to exercise each collector and grader through the application path, inspect output values, and retain artifacts. OpenMerit validates the coverage report but does not independently run the collector or grader, so `ready` is still a harness claim rather than independently verified end-to-end proof. Request a reported production metric window before relying on it for model decisions.
 
 Product instrumentation and OpenMerit auditing are separate. The confirmed application LLM target emits measurements such as quality, success, tokens, cost, latency, retries, tool calls, and intervention. Every metric record carries that target ID. OpenMerit records the signals, policy decision, intent lifecycle, evidence identifiers, and readiness or frontier decision that followed.
 

package/docs/operations.md

@@ -9,8 +9,8 @@ Operational data lives under `.openmerit/` in the target product repository. It
 - `events.jsonl` — append-only audit events.
 - `audit-export-errors.jsonl` — failures from optional external event exporters; these failures never replace or erase the canonical local event.
 - `evidence/*.json` — evidence-reference manifests.
-- `scheduler-status.json` — last headless wakeup time and exit code in the unreleased macOS source implementation, when a wakeup actually ran.
-- `terminal.json` — optional source configuration and declared inventory for the Inference Terminal included in `0.1.6`.
+- `scheduler-status.json` — last headless wakeup time and exit code from the macOS background runner, when a wakeup actually ran.
+- `terminal.json` — optional source configuration and declared inventory for the Inference Terminal.
 - `inference.jsonl` — optional request-metadata export supplied by your existing tools for the terminal. The terminal reads this file; it does not instrument your application or create the feed.
 
 Snapshot writes are atomic. Files use owner-only permissions where POSIX modes are supported.
@@ -21,9 +21,7 @@ The [Inference Terminal](inference-terminal.md) runs independently of the orches
 
 Start with `/openmerit status`, which reports profile, metric and evaluation readiness, active work, next assessment, lifecycle, and pause state. `/openmerit doctor` confirms adapter loading without printing secrets.
 
-In published version 0.1.5, `/openmerit pause` suppresses lifecycle follow-up nudges but does not gate the automation-signal dispatch path. Task, measurement, and session-start signals can still issue due work while status says paused. The setting is process-local, resets on restart, and does not cancel active work. Explicit commands still work. Do not rely on that version's pause command to stop all OpenMerit work.
-
-In the unreleased source checkout, pause is stored in `.openmerit/state.json`, blocks automatic due decisions, and unloads an OpenMerit-managed macOS LaunchAgent for that project. Resume restores scheduling only if the confirmed policy still permits persistent execution and infrastructure changes. Incoming signals can still be recorded while paused, and an already-active intent is not cancelled. `/openmerit doctor` shows the LaunchAgent and status-file paths. A LaunchAgent runs only while the Mac user is logged in; a headless run requires Pi credentials available without a shell-only environment variable.
+Pause is stored in `.openmerit/state.json`, blocks automatic due decisions, and unloads an OpenMerit-managed macOS LaunchAgent for that project. Resume restores scheduling only if the confirmed policy still permits persistent execution and infrastructure changes. Incoming signals can still be recorded while paused, and an already-active intent is not cancelled. `/openmerit doctor` shows the LaunchAgent and status-file paths. A LaunchAgent runs only while the Mac user is logged in; a headless run requires Pi credentials available without a shell-only environment variable.
 
 Inspect `scheduler-status.json` after a headless wakeup. A successful baseline-only check records a decision without starting Pi. A nonzero `exitCode` or `activeIntentId` means model-driven work did not complete; open the project in interactive Pi, inspect `/openmerit status` and `.openmerit/events.jsonl`, then use `/openmerit cancel` only if you intend to abandon that work. Model-driven runs have a 15-minute limit. The runner does not silently mark incomplete work successful.
 

package/docs/pi-extension.md

@@ -19,17 +19,17 @@ Use `/openmerit doctor` to confirm that the adapter and intent executor are avai
 
 Use `/openmerit logs` to locate the project-local audit stream, exporter failures, evidence manifests, and state while testing an end-to-end run.
 
-On `session_start`, an interactive unconfigured project receives a bounded source preflight. In the unreleased source checkout, the adapter also repeats that bounded detection after a settled Pi turn. This lets Pi build an application from an empty directory and then receive an automatic `establish_evals` intent without requiring `/openmerit setup`. Durable active-intent state prevents duplicate setup nudges. Non-interactive runs do not attempt a confirmation flow.
+On `session_start`, an interactive unconfigured project receives a bounded source preflight. The adapter also repeats that bounded detection after a settled Pi turn. This lets Pi build an application from an empty directory and then receive an automatic `establish_evals` intent without requiring `/openmerit setup`. Durable active-intent state prevents duplicate setup nudges. Non-interactive runs do not attempt a confirmation flow.
 
-If Pi exits while an intent is active, the unreleased adapter restores its completion tool and resends that same persisted intent on the next interactive session so the harness can continue it.
+If Pi exits while an intent is active, the adapter restores its completion tool and resends that same persisted intent on the next interactive session so the harness can continue it.
 
-For configured projects with a confirmed application target, session start reports the available model catalogue and a scheduled tick. Application instrumentation reports idempotent completed-task and metric-window signals. OpenMerit evaluates the confirmed policy before issuing work. Published `openmerit@0.1.5` has no persistent Pi scheduler. In the unreleased source checkout on macOS, Pi can install a per-project user LaunchAgent after you confirm persistent execution and authorize infrastructure changes in the policy. It uses the selected authenticated Pi model, checks for due work locally every five minutes, and starts a headless Pi only when the due core check finds downstream harness work. If installation or background authentication fails, the adapter reports a scheduling gap. Other platforms still need an external scheduler.
+For configured projects with a confirmed application target, session start reports the available model catalogue and a scheduled tick. Application instrumentation reports idempotent completed-task and metric-window signals. OpenMerit evaluates the confirmed policy before issuing work. On macOS, Pi can install a per-project user LaunchAgent after you confirm persistent execution and authorize infrastructure changes in the policy. It uses the selected authenticated Pi model, checks for due work locally every five minutes, and starts a headless Pi only when the due core check finds downstream harness work. If installation or background authentication fails, the adapter reports a scheduling gap. Other platforms still need an external scheduler.
 
-The macOS background runner is not yet product-validated for candidate work. A local runner test verifies that an insufficient baseline is checked and recorded without starting Pi. An earlier live OpenRouter test reached the former due `run_assessment` intent and called Pi tools, but did not obtain a terminal result within the test window. A model-driven runner is capped at 15 minutes and records a nonzero exit and active intent ID on failure. Do not infer that unattended candidate discovery or swapping has passed end-to-end validation.
+The macOS background runner is not yet product-validated for candidate work. A local runner test verifies that an insufficient baseline is checked and recorded without starting Pi. A model-driven runner is capped at 15 minutes and records a nonzero exit and active intent ID on failure. Do not infer that unattended candidate discovery or swapping has passed end-to-end validation.
 
 The adapter does not convert Pi's own provider usage into application metrics. Application-task signals must carry the application model, target ID, and durable evidence; application telemetry may include provider-reported USD cost, token counts, and wall-clock latency. If automatic setup is active when interactive input arrives, the input is queued and released after setup completes instead of racing the setup turn.
 
-Pi also exposes `openmerit_report_signal` for verified `product_task_completed`, `metric_window_available`, and `verification_window_completed` events. Evals or observability work performed by Pi can submit measured application-target metric records through this typed ingress using stable signal IDs. For the unreleased core check, send cumulative metric windows with sample counts and evidence references. The background wakeup only checks whether a time-based action is due; it does not collect application results by itself. `/openmerit assess` checks those stored windows locally and starts candidate discovery only when all required metrics are ready.
+Pi also exposes `openmerit_report_signal` for verified `product_task_completed`, `metric_window_available`, and `verification_window_completed` events. Evals or observability work performed by Pi can submit measured application-target metric records through this typed ingress using stable signal IDs. For baseline checks, send cumulative metric windows with sample counts and evidence references. The background wakeup only checks whether a time-based action is due; it does not collect application results by itself. `/openmerit assess` checks those stored windows locally and starts candidate discovery only when all required metrics are ready.
 
 ## Intent delivery
 
@@ -39,7 +39,7 @@ Pi must finish by calling `openmerit_complete_intent` exactly once. Success with
 
 If Pi exits while an intent is active, the next session reports the recovered intent. Use `/openmerit cancel` to record an explicit cancellation before retrying; OpenMerit does not silently reinterpret interrupted work as success.
 
-The extension replaces the completion tool definition whenever an intent becomes active. Its `outputs` property therefore exposes the canonical schema for that exact intent instead of untyped JSON. Pi validates the arguments before execution and requests provider-side JSON Schema constrained sampling when supported. OpenMerit validates the canonical schema and semantic rules again when accepting the result. The source checkout records the exact incumbent during setup, freezes accepted candidates after discovery, and carries accepted assessments and the experiment manifest into frontier calculation so these transitions do not rely on conversational memory.
+The extension replaces the completion tool definition whenever an intent becomes active. Its `outputs` property therefore exposes the canonical schema for that exact intent instead of untyped JSON. Pi validates the arguments before execution and requests provider-side JSON Schema constrained sampling when supported. OpenMerit validates the canonical schema and semantic rules again when accepting the result. OpenMerit records the exact incumbent during setup, freezes accepted candidates after discovery, and carries accepted assessments and the experiment manifest into frontier calculation so these transitions do not rely on conversational memory.
 
 ## Model changes
 

package/docs/roadmap.md

@@ -2,27 +2,27 @@
 
 ## Current status
 
-OpenMerit is ready for an initial supervised product trial with Pi. Its protocol, lifecycle, persistence, metric catalogue, frontier verifier, and Pi adapter are implemented and tested. That is not evidence that it optimizes every product; each target workload requires its own profile and representative evaluations.
+The Inference Terminal provides a read-only view of existing application evidence. The Pi integration coordinates supervised model-comparison trials. Neither a populated dashboard nor passing implementation tests establishes that an application has better models, trustworthy evaluations, or reliable unattended automation.
 
 ## Known limitations
 
-- No second harness adapter exists.
-- Jev must be configured and invoked by the harness; OpenMerit intentionally provides no client.
-- Generic Pi task checkpoints do not replace product telemetry. Setup must create the infrastructure required by the profile.
-- Published 0.1.5 reports active-session signals and typed metric windows but cannot provision persistent background wakeups. The unreleased source checkout adds an opt-in macOS user LaunchAgent; non-macOS platforms still need an external scheduler.
-- The source LaunchAgent now performs a deterministic core baseline check without Pi when evidence is insufficient. This has a local runner test, not a representative product trial. Earlier headless Pi assessment attempts timed out; unattended candidate discovery and model decisions are not yet validated.
-- The source checkout can detect an application call created during a settled interactive Pi build turn, records the incumbent, and durably hands candidates and assessments between comparison stages. These paths have deterministic adapter tests but still need the representative live sandbox trial below.
-- In published 0.1.5, pause does not block automatic dispatch from task, measurement, or session-start signals and resets on process restart. The unreleased source checkout persists pause and gates automatic work; neither version cancels already-active work.
-- External evidence authenticity depends on the referenced system.
-- The core and protocol workspaces are private; the single `openmerit` npm
-  package exposes their public entry points.
+- The terminal does not instrument the application, call inference providers, route traffic, run evaluations, or apply model changes. It needs supported logs or exports for actual request history. Code/configuration declarations alone cannot supply past requests or spend.
+- History and inventory can be incomplete because of source availability and read/parser limits. “All time” means retained supported evidence, not a complete lifetime archive. Cloud account usage is not fetched.
+- Machine and runtime readings describe shared resources; they are not per-request resource measurements. Static endpoints do not prove deployment, availability, or physical location.
+- Hosted samples and the before/after dashboard demo are fictional. Their cost projections and public benchmark screens do not establish application quality or latency.
+- Pi 0.87 is the only implemented coding-harness adapter. Jev must be configured and invoked by the harness; OpenMerit provides no Jev client.
+- Setup asks the harness to establish task-specific evaluations and observability. Generic coding checkpoints cannot replace product telemetry, and structured-result validation cannot prove external evidence is truthful.
+- Background wakeups require a logged-in macOS user, working headless Pi authentication, and explicit persistent-execution/infrastructure permission. Other platforms require an external scheduler. Deterministic baseline checks have automated coverage; unattended candidate discovery and model changes still need representative product validation.
+- Pause persists and gates automatic work, but does not cancel active intents, stop explicit commands, or stop the standalone terminal.
+- Challenger accounting depends on reported costs. It cannot independently meter provider billing or stop an in-flight request. Use provider spending controls as well as the confirmed evaluation policy.
+- The source interface and hosted dashboard follow the device's light/dark preference. That UI change is not included in npm 0.1.6.
 
 ## Next validation milestones
 
-1. Run setup in one representative product.
-2. Verify generated eval and observability artifacts and their side effects.
+1. Establish working evaluations and observability in a representative product.
+2. Inspect generated artifacts, measurements, and side effects.
 3. Collect repeated evidence for the confirmed profile.
-4. Complete a budgeted A/B/C challenger cycle.
-5. Validate the harness-calculated frontier and recommendation.
-6. Perform a supervised swap, post-swap verification, and rollback drill.
-7. Repeat with a materially different product task before broad readiness claims.
+4. Complete a budgeted challenger cycle and inspect the frontier and recommendation.
+5. Perform a supervised swap, post-swap verification, and rollback drill.
+6. Validate background candidate work before relying on unattended operation.
+7. Repeat with materially different workloads before making broad reliability claims.

package/docs/scout.md

@@ -0,0 +1,206 @@
+# Scout by OpenMerit
+
+Describe what you need a model to do. Scout returns up to five available models using relevant public benchmark results, task-shaped reference cost, and completion time. The `/scout/` page, JSON API, and MCP tool all use the same index and ranking.
+
+Scout is a separate beta product. It does not change the OpenMerit npm package, Pi coordinator, or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
+
+## Use Scout
+
+Open [Scout](https://openmerit.ai/scout/), type a task, and select the arrow button (**Find models**) or press Enter. For example: “I want to upload an invoice and get the invoice number, date, vendor, subtotal, tax, and total as JSON.” The task field grows and shrinks as you type or paste. Very long prompts scroll within the field once it reaches its height limit. Shift + Enter adds a line. When the field is empty and unfocused, its placeholder cycles through detailed example tasks every ten seconds with a brief fade. Rotation pauses while you type, while the field is focused, or while the page is hidden. With reduced motion enabled, the example stays still. Placeholders never fill the input or send a request. The heading shows live counts of indexed models, model providers, benchmark series and source feeds. Provider aliases are counted once; these are model creators, not every hosting route. The evaluation guide remains available in the fixed bottom footer, with space reserved beneath the page content.
+
+The comparison table keeps recommendations in Scout rank order. When a matching measured task profile is available, it shows calibration accuracy, mean reported cost per 1,000 similar tasks, and median full request time. Otherwise it shows relative benchmark quality and reference cost/time estimates. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; missing marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **API & MCP**, **Capabilities**, and **Sources** expand beneath the search and results. Collection times and refresh failures are shown inside **Sources**.
+
+The capability index lists each area with its benchmark-series and model counts. After a search, task matches appear first with a check mark. Counts describe index coverage, not the number of models eligible for your task. The source panel describes what each feed measures, its coverage, source dates, archived results, and limitations. Collection time is separate from source dates, which can be publication dates rather than measurement dates. Scout task calibration appears separately with its tested pool, scope and seven-day expiry.
+
+**Provider page** in each model's evidence opens its provider's official pricing, model documentation, or website. Model links never fall back to a router. Token prices shown in Scout are still quoted via OpenRouter and may differ from direct provider rates. Benchmark links lead to the original evaluation sources.
+
+## Connect a coding agent
+
+Add this remote **Streamable HTTP MCP** endpoint to an agent that supports it:
+
+```text
+https://openmerit.ai/scout/mcp
+```
+
+Scout exposes one read-only tool, `scout_models`, with a required prompt:
+
+```json
+{"prompt":"Upload an invoice and extract six fields as JSON"}
+```
+
+No API key is required from the caller. Scout sends the task description to OpenAI Decisions to interpret requirements, then ranks its collected evidence. It does not execute the recommended models. The MCP endpoint supports protocol versions `2025-03-26`, `2025-06-18`, and `2025-11-25`, including initialization, tool discovery, and tool calls. It uses JSON responses without sessions or a long-lived SSE stream. Browser requests must have the same origin; ordinary agent clients omit the Origin header. An unsupported protocol version is rejected. Agent configuration is client-specific; connecting an endpoint does not automatically make an agent use it.
+
+Tell the agent: “Before choosing an inference model for this feature, call Scout with the task description. Compare the five candidates, read their evidence and warnings, and evaluate the shortlist on our own examples.”
+
+The MCP response includes up to four relevant evidence entries per model and the total evidence count. Evidence satisfying required memory, tool-calling, and visual-document capabilities is shown first. The JSON API exposes the complete evidence list. When specialist discovery finds matches, MCP includes up to three separate candidates, four reported evaluation records per candidate, and each candidate’s total indexed evaluation count.
+
+## JSON API
+
+```sh
+curl https://openmerit.ai/scout/api/recommend \
+  -H 'Content-Type: application/json' \
+  -d '{"prompt":"Upload an invoice and extract six fields as JSON"}'
+```
+
+`POST /scout/api/recommend` requires `prompt`: a non-empty string of at most 4,000 characters. The JSON body is limited to 18,000 bytes. Unknown properties are rejected.
+
+HTTP and MCP additionally accept optional `modelIds`, a unique list of up to 100 catalogue IDs. This filters eligible candidates before ranking; an empty or unknown-only list returns no models. The response echoes `selection.modelIds` (`null` for all). Scoped requests omit specialist discovery because it is outside the declared inventory. No endpoint URL or credential is accepted. The [agent evaluation runner](agent-evaluations.md) uses this for existing-project scope; the hosted API supports it as of 15 October 2026. The npm evaluation commands are available in `openmerit@0.1.7-preview.1` under the `preview` tag, separate from stable `0.1.6`.
+
+The response contains:
+
+- `recommendations`: up to five distinct model IDs with rank, provider, reason, quality signal, evidence coverage, input/output pricing, reference timing when available, supported inputs, and source-linked evidence. `reference` gives cost in USD and completion seconds (`null` when unavailable). Benchmark mode includes assumed input/output tokens; measured mode sets those token counts to `null` and supplies `measured` with observed field accuracy, the reliability-capped `effectiveQuality`, cross-task `availability`, valid outputs, sample size, charge coverage, provider routes and evidence provenance. `comparison` gives the number of comparable candidates and average shared-capability coverage. The compatibility `pricing.blendedPerMillion` field retains its old 3:1 calculation but is no longer scored.
+- `task`: interpreted capabilities and weights; image-input, structured-output, tool-calling and persistent-memory requirements; `requirements.tools` (`yes`, `no`, `unspecified`); `requirements.memory` (`none`, `session`, `persistent`, `unspecified`); consequential `uncertainty`; `workload.input` and `workload.output` (`short`, `medium`, `long`), `workload.priority` (`balanced`, `quality`, `cost`, `latency`), `kind` (`extract`, `classify`, `dispatch`, `code`, `act`, `remember`, `sql`, `calculate`, `retrieve`, `other`), and `interpreter: "openai-decisions"`. `keywords` is retained as an empty compatibility field. A general-purpose fallback means no specific capability was identified, not that the classifier was unavailable.
+- `specialists`: a separate discovery object with up to three Hub repositories, metadata-based task fit, declared licenses, pinned model-card revisions, live hosting metadata when supplied, and original reported evaluation records. `mode` is `discovery`; quality, cost and latency are `null`. These candidates are not additional ranked places. Unsupported task types can omit this field.
+- `specialistRepositories`: bounded Hub catalogue count, separate from the evidence-backed `models` count.
+- `calibration`: released task profiles, scopes, sample sizes, measurement dates and expiry status.
+- `checkedAt`, `stale`, `degraded`, `sources`: index and source health, feed purpose, limitations and coverage. Each source has its own last successful check. A failed source is marked degraded; any source unchecked for more than two hours makes the index stale.
+- `benchmarks`, `benchmarksWithResults`, `models`, `providers`, `availableModels`, `eligibleModels`: indexed series, series with matched results, evidence-backed models, distinct indexed model providers (including alias consolidation), catalogue models, and eligible candidates for this task.
+- `wordCloud`: capability vocabulary with benchmark and model coverage counts.
+- `ranking`: algorithm version, `mode` (`measured` or `benchmarks`), workload and tradeoff weights. A measured `profile` gives the task type, sample size, eligible tested pool, protocol hash, collection and expiry times, and public evidence URL. `benchmarkEligibleModels` reports the separate public-benchmark pool. Rank is the final order; the aggregate `score` is a supporting signal and can be non-monotonic because stronger direct comparisons take precedence.
+- `warnings`, `methodology`, `interpretationMs`, `elapsedMs`: limitations, ranking method, Decisions request time, and measured server ranking time when the runtime provides a sufficiently precise clock (`null` otherwise). Cloudflare freezes its clock during synchronous work, so the hosted page reports measured HTTP round-trip time instead of claiming zero-millisecond computation.
+
+`pricing.url` is the official provider destination, not the source of the quoted rates. It is `null` for a provider without a reviewed destination; the page then omits the link. The same rule applies to MCP responses. `pricing.basis` identifies the price source and calculation assumptions.
+
+`GET /scout/api/status` returns index coverage, source timestamps, freshness, and scheduler state without submitting a prompt. API responses are `no-store`. Errors return JSON with `error`: 400 for invalid input, 413 for an oversized request, 415 for a non-JSON body, 405 for an unsupported method, 404 for an unknown endpoint, 429 when shared admission limits are reached, and 503 if interpretation is unavailable. A failed, malformed, refused, or timed-out Decisions response never silently falls back to keyword ranking. MCP reports these failures as tool errors.
+
+Scout recommends models; it does not process an uploaded invoice, execute a task, install a model, or select a provider account.
+
+## Real sources and their limits
+
+The hourly updater fetches seven public sources (six existing feeds and eight bounded Hugging Face queries) and prepares the index before requests arrive. Queries make one Decisions call for intent classification; they do not browse benchmark sites or ask a model to invent rankings.
+
+| Feed | Used for |
+| --- | --- |
+| [Epoch AI](https://epoch.ai/benchmarks) | Benchmark definitions, published results, source links, and evaluation configurations. |
+| [Artificial Analysis](https://artificialanalysis.ai/leaderboards/models) | Benchmark results, measured first-answer delay and reference generation speed. Archived evaluations remain labelled when their models are still available. |
+| [OmniDocBench](https://github.com/opendatalab/OmniDocBench#end-to-end-evaluation) | Document parsing scores from the explicitly labelled `v1.6_full` leaderboard, including newer document models than the OpenVLM snapshot. |
+| [OpenCompass / VLMEvalKit](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) | Visual understanding and OCR benchmark results. Its feed currently identifies itself as 17 September 2025. |
+| [Berkeley Function Calling Leaderboard (BFCL v4)](https://gorilla.cs.berkeley.edu/leaderboard) | Native function-calling single-turn, live, and multi-turn scores; separate memory scores for key-value, vector, and recursive-summarization stores. |
+| [OpenRouter](https://openrouter.ai/api/v1/models) | Available model IDs, input modalities, declared tools and structured-output support, context limits, and advertised token prices. |
+| [Hugging Face](https://huggingface.co/models) | Public repository metadata, model-card results and structured evaluation records for specialist discovery. Separate from performance ranking. |
+
+Over 100 benchmark **series** are indexed, including source-specific implementations and versions. This is not 100 independent publishers, and coverage is not uniformly current. Some definitions have no results matched to an available model. Both counts are exposed. Missing metrics and ambiguous or unmatched model versions are omitted. Ranked availability is limited to OpenRouter's text-output catalogue, excluding batch/free variants and known routers. Specialist discovery also indexes repositories outside that catalogue.
+
+Collection time is not measurement time. A successful hourly check does not make an old result new. Some feeds omit measurement dates; those remain unknown. Model release dates are never substituted. OmniDocBench's repository version and leaderboard version differ; Scout uses the version actually printed on the score table. Document parsing is a proxy for invoice extraction, not an evaluation of your exact six fields. New models without comparable evidence may be absent from a task's shortlist.
+
+Epoch AI attribution: “Capabilities & benchmarking,” published at epoch.ai, normalized and matched by Scout under the source's [CC BY 4.0 terms](https://creativecommons.org/licenses/by/4.0/). Other sources are attributed in the source panel and individual evidence links. These are data-source attributions, not a statement about OpenMerit's licensing.
+
+## Specialist models and open weights
+
+Some specialist models have useful evaluations outside general LLM leaderboards. Scout reads both legacy model-card `model-index` results and the Hub’s [structured evaluation records](https://huggingface.co/docs/hub/eval-results). It retains the original dataset, task or metric, units, configuration, split, reported date, setup notes and source link when supplied. Scores are not averaged across unrelated datasets. A repository revision pins the metadata collected; it does not establish which checkpoint produced every reported score. Reported dates can also default to a commit date. Scout does not validate verification tokens or independently reproduce these scores.
+
+The current hourly discovery sample takes up to 30 repositories sorted by downloads for each of eight pipelines: document question answering, token classification, text classification, sentence similarity, text-to-text generation, text generation, image-to-text, and image-text-to-text. The two image queries are filtered to OCR. Duplicates, private or disabled repositories, adapters and quantization packages are excluded; fine-tunes remain separate. Downloads determine sampling coverage only, never quality. This is a bounded sample, not the whole Hub, and can miss low-download specialists. The initial snapshot contains 175 repositories, 49 with structured reported results; live counts change as sources refresh.
+
+When metadata matches the task, **Specialist models** expands beneath the five recommendations. Up to three matches cover different components, with task terms, metadata completeness and component diversity determining discovery order. Document OCR requires document-related metadata; incidental words such as “object” in “JSON object” do not select object detectors. Explicit supported-language declarations and known domain tags constrain matching, but metadata can still be incomplete or wrong. Tool execution and persistent-memory tasks do not receive unvalidated specialist matches. Entity taggers, OCR and embedding models are labelled as components requiring integration, not interchangeable chat APIs. Repeated repository names are de-emphasized for display diversity without merging their identities.
+
+Cards show declared licenses, access gates, language, library, parameter count when known, and live hosting routes listed by the Hub. Missing hosting metadata does not prove that no provider exists. Size is not a hardware-fit estimate. Public weights do not establish an open-source license or permission for a particular use; check the pinned model card and actual license. Scout neither downloads weights nor executes repository code, and it does not send task descriptions to Hugging Face.
+
+Missing evaluations mean **unknown**, not poor performance. Discovery models do not enter the five through download counts or self-reported scores. Promotion is not automated: it requires a runnable model or endpoint and comparable task checks, full workload cost and timing under a recorded deployment configuration. Self-hosted cost depends on hardware and utilization; a zero token price would be misleading. The same quality/cost/speed comparison can then be used without a provider preference. Model-specific exceptions and inferred free inference are not used.
+
+## Ranking
+
+One server-side call to [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions), using `gpt-6-luna`, interprets the work the selected model must perform. It distinguishes executing tools from writing code about them, incidental words such as “tool” or “agent,” supplied plain text versus visual inputs, and session state versus persistent memory. Eighteen typed choices become capability weights, requirements, reference input/output sizes, a tradeoff priority, and a narrow task operation. One function dispatch is distinct from a dependent action loop. Typed code, action, and external-memory operations keep the independent capability answers coherent: an action or memory-tool operation cannot silently become a reasoning-only task. SQL generation counts as coding. Required extraction is not downweighted merely because reasoning is marked useful. Long-context requirements imply a long reference input. Visual input uses at least the medium category, even when the task description is short. Decisions sees the task, not candidate names or rankings; the shortlist still comes from collected scores.
+
+Strong JSON/schema filtering also needs explicit format wording in the prompt. An uncertain format answer cannot turn an ordinary story or comparison into a JSON task. Low-confidence consequential classifications and unspecified retention/tool interfaces are exposed as uncertainty. This is still a probabilistic classifier: paraphrases can differ, ambiguous requests can be misread, and English format-grounding rules limit multilingual handling. Scout has no clarification dialogue or user-defined constraints yet.
+
+Image tasks require an image-capable model. JSON/schema tasks require declared `response_format` or `structured_outputs` support. Visual document tasks require document-specific evidence. Required tool use needs declared `tools` support and relevant tool-calling evidence. A matching measured task profile can supply this evidence; the benchmark fallback requires published results for the capability. Persistent memory needs declared tools support and dedicated memory evidence; generic reasoning or a large context window cannot substitute. Session-only state does not trigger this filter. Recognized image/video/audio generation and transcription tasks return no recommendations because those outputs are outside the current index.
+
+BFCL contributes only native function-calling (`FC`) configurations, preserving their published labels. Its six correlated series count as one benchmark family, not six independent sources. Missing scores stay unknown and zero remains a score. Model versions are matched conservatively; ambiguous versions are omitted. BFCL does not supply per-model measurement dates, so these remain unknown.
+
+The [BFCL memory evaluations](https://gorilla.cs.berkeley.edu/blogs/16_bfcl_v4_memory.html) test using external memory stores across sessions. They do **not** establish built-in model memory, automatic retention in your app, or performance on your storage system. The application still implements storage and tool execution. Without a matching measured profile, models without matched memory results are excluded when persistent memory is required, including newer models with other strong scores.
+
+Ranking version 4 first checks for a matching, recent **measured task profile**. The static evidence file contains only calibration aggregates: mean per-case task-check accuracy, mean reported charge, median full task time, valid-output counts, provider routes, and provenance. Malformed schemas and request failures receive zero accuracy; failure time remains in the latency median. Unknown charges are excluded from the cost mean and the known-charge count is disclosed. A model with no known charges earns no cost advantage. For multi-call episodes, reported cost includes every call and elapsed time covers the whole episode, including reasoning, tool round trips, and all memory sessions. Request reliability is shared across calibration tasks: after two or more failed requests, the observed HTTP success rate caps the model’s task-quality signal. A clean task slice cannot erase repeated route failures elsewhere. One isolated failure remains in its own task score rather than penalizing every other task. This is route evidence from the evaluation period, not a promise about current provider availability.
+
+Profiles cover small text-field extraction, rule-based classification, single-function dispatch, read-only SQL, short numerical problems, long-text record retrieval, visual-field extraction, dependent tool actions, and external memory across sessions. The matcher also supports code-function operations, but their proposed measured profile and an additional medium-input extraction profile were withheld after fresh validation regressions; those tasks retain their prior evidence path. Matching requires the same typed operation, image/tool/schema requirements, and input/output size category. A new required capability outside the profile prevents a match. Retention scope (none, session, persistent) and whether execution uses an action loop must also match; tool dispatch, dependent actions, and cross-session memory are separate operations. At least five currently eligible tested models and six calibration cases per model are required. Availability and declared capabilities still come from the refreshed catalogue. Evidence expires after seven days; expired, undersampled and unmatched profiles fall back to public benchmarks.
+
+When several profiles match, the most recently measured complete profile is used. Measured mode ranks **within its disclosed tested pool**, not every catalogue model. It applies the same pairwise policy to actual observed accuracy, charges and time. Memory quality uses correct updates, retrieval, and final store state across fresh conversations. In offline candidate evaluations, code uses executable hidden checks and tools use permitted actions and their prerequisites. If the task does not require strict JSON, evaluation-only JSON serialization does not erase correct actions or facts. Tool profiles score observable actions, prerequisites, and a nonempty final reply; they do not grade the precision of its prose. Memory facts accept a JSON object surrounded by prose or a fence. Raw JSON-valid counts remain separate. A strict-JSON task cannot match these non-strict profiles. A ten-percentage-point observed accuracy gap gives full pairwise quality preference. These are small development samples, not confidence intervals or proof of accuracy on an arbitrary prompt. Text cases are authored; the visual calibration set contains eight public Portuguese invoices. Classification, arithmetic and simple dispatch can saturate at perfect accuracy, leaving cost and variable latency to determine close positions. Public calibration aggregates and their protocol hash are available through each model’s **Measured task evidence** link. Raw answer keys and held-out test outcomes are not runtime inputs. The source images are attributed to [Cruz and Castelli's invoice dataset](https://zenodo.org/records/6371710), under that dataset's CC BY 4.0 terms.
+
+The **benchmark fallback** compares each pair only on benchmarks both have run. It does not compare percentiles from different cohorts. Differences are averaged within correlated families, then across shared task capabilities. The first two percentage points are treated as an indifference band; a further twenty-point gap gives full quality preference. This deliberately conservative proxy policy prevents a small benchmark lead from outweighing a large cost/time premium. It is not a statistical significance test. Format-following evidence has one-quarter weight only when another substantive capability is **required**, rather than merely useful.
+Each comparison also considers relative cost and completion time. The default balanced priority uses **60% quality, 20% cost, 20% latency**. An explicit quality priority uses 90/5/5; cost uses 60/35/5; latency uses 60/5/35. These are transparent defaults, not learned optimal weights. Decisions can infer a priority from the prompt; there are no new controls or extra interpretation calls. Broad or ambiguous preferences can still be misclassified.
+
+The strongest supported pairwise preferences are applied first, skipping any that would create a cycle, to order the whole candidate pool before taking five. This prevents a model's easy wins over weak candidates from automatically overriding its direct comparison with another finalist. Strength is reduced when fewer task capabilities have shared evidence. Unsupported comparisons add no pairwise preference. A conservative aggregate signal breaks otherwise unresolved orderings; stable IDs break exact ties. Missing measurements are never treated as zero cost or instant completion. Warnings expose limited comparison coverage.
+
+Reference input sizes are **1,024 / 4,096 / 32,768 tokens**, and visible output sizes are **128 / 512 / 2,048 tokens**, for short / medium / long. Cost is input tokens × advertised input rate + output tokens × advertised output rate. This excludes hidden reasoning, cache effects, image-specific tokenization and other modality charges. An image task's input size is an illustrative token workload, not a prediction of that image's bill. Separate rates and the assumptions are returned so clients can substitute their own measured usage.
+
+Reference completion time is **time to the first answer token + visible output tokens / throughput**. When the source publishes multiple configurations, Scout calculates completion time for each paired measurement and takes the median. It does not mix one configuration's delay with another's throughput, or confuse early streamed reasoning with the first answer. Missing first-answer timing for a reasoning configuration stays unknown. A legacy snapshot with only first-token timing supplies no completion estimate; deployment replaces an older data format with the bundled compatible snapshot even if the legacy refresh timestamp is newer. These measurements are from Artificial Analysis, not the caller's OpenRouter route; provider and reasoning settings may differ. They are not guaranteed task latency.
+
+Matching is deliberately narrow; code-function tests do not establish repository-level engineering skill, and small tool loops or key-value memory tasks do not establish long-horizon agent reliability. A profile label and quality basis are returned with every measured shortlist. No model-name bonuses, prompt-example rules, or task answer keys enter the ranking.
+
+The goal is an ordered five-model shortlist, not just a first-place pick. Regression checks cover all five positions, all ten within-five pairs, cutoff membership, provider-name invariance, unknown cost/time, capability and modality boundaries, expiry, and rejection of held-out observations during evidence building. Paid evaluation uses separate calibration and held-out cases across multiple task types. Held-out measurements compare both membership and ordering within the tested candidate pool; they do not establish a global optimum or a perfect ordering for every prompt. Test the five candidates on your own examples before choosing a production model.
+If fewer than five eligible models have evidence, Scout returns fewer and explains why. There are no accounts, saved prompt history, vector database, user selection controls, or language-model reranker.
+
+## Current validation limits
+
+The latest 15 October coding/extraction follow-up made **132 requests**: six new calibration cases and six disjoint new holdout cases per category, with six coding candidates and five extraction candidates. These are authored application scenarios, not private production records. Predictions, scorer and evidence were frozen before holdout dispatch. The v4 quality/cost/speed policy was unchanged. Candidate evidence used only the new calibration split; no model-specific rule or held-out result entered it.
+
+| Latest proposal | Top-five membership, before → candidate | Correct ordered pairs, before → candidate | Release |
+| --- | --- | --- | --- |
+| Small JavaScript functions | 4/5 → 5/5 | 7/10 → 8/10 | Withheld |
+| Text field extraction | 5/5 → 5/5 | 8/10 → 7/10 | Withheld |
+
+The per-category gate required the exact five in the same order as the unchanged policy applied to held-out quality, cost and time, without selected-quality regression. Neither passed. The proposed coding five all passed their six fresh cases, but cost and timing changed the first three positions. Extraction had valid JSON in all 30 held-out responses, yet two models missed fields; a perfect schema did not establish accurate extraction. All rankings remain on their previously released evidence path. This result does not validate exact-five ordering or whole-catalogue membership.
+
+Reported charges were $0.581873295 across calibration and holdout, excluding Decisions. Two empty, content-filtered calibration responses supplied no charge; their cost remains unknown and they scored zero. The harness reserved $0.30 for unknown charges inside its $2 guard. Free-text case, whitespace and terminal punctuation normalization was fixed from calibration before holdout; identifiers, dates and numbers stayed exact. Each holdout order is based on six cases per model and one endpoint run, so it remains sensitive to task mix and timing.
+
+The 15 October revision tested four proposed measured profiles on **72 fresh episodes: six models × three cases × four categories**. The earlier broad audit was explicitly retired to calibration data, alongside additional cases. The candidate policy improved pair agreement only from **24/40 to 25/40** and membership from **17/20 to 18/20**; none of its four complete lists matched the measured order. Coding membership and selected quality regressed; tool ordering regressed; extraction ordering regressed. Those three initial profiles were **not published**, and the experimental quality-smoothing rule was also withheld. The action pool was subsequently expanded from six to nine models by running the three unmeasured candidates from the corrected benchmark shortlist on the same six calibration cases. This adds comparable evidence rather than a model-specific score adjustment; a further fresh test checks the resulting shortlist.
+
+Of the initial four proposed profiles, only external memory passed the release gate: pair agreement **5/10 → 7/10**, membership **4/5 → 5/5**, and mean selected task-check quality **75.8% → 82.5%** within its six-model test pool. This is a validation-gated release decision, not an unbiased performance estimate after selection. The independent tool classification fix remains: a required external-action loop now receives tool and agent evidence instead of being treated as reasoning-only. The same pairwise policy and existing profiles remain otherwise unchanged. A separate two-case check of the memory shortlist matched all ten ordered pairs. Its quality checks include an ambiguous “table only” preference phrase that all five stored literally; accepting that interpretation leaves the order unchanged. This small follow-up is not a general accuracy guarantee.
+
+The expanded nine-model tool profile's final check used **two new cases × five selected models**. All ten ordered pairs matched the measured policy order, and all ten episodes took the permitted actions despite misleading tool-result instructions. This tests order within the five, not the optimal membership of the whole catalogue. Two episodes paused at the local evaluation spend guard; successful responses were replayed locally to continue them without repeating provider calls. Reported test timing sums active provider-call wall time for every model, excluding that harness pause. Production calibration timing remains full-episode wall time.
+
+The candidate action grader was corrected from calibration outputs after test dispatch began but before any held-out outputs were inspected. Final predictions and grader hashes were frozen before inspection; no test outcomes entered calibration. This was not a fully preregistered experiment. Tool action scoring does not assess final prose precision. The release still does **not** establish the correct order of all five models across tasks.
+
+The older results below describe ranking version 3 at that release, not a fresh validation of the current version.
+
+The 14 October development audit used 438 calibration observations, an initial 358-call test, then 159 fresh validation calls after a general reliability correction. The unchanged visual shortlist retained its original 40-call test; it was not rerun. The final validation has three fresh cases per text task and five document photos. The candidate pools were selected before testing and do not cover the whole catalogue.
+
+| Task | Top-five membership, before → after | Ordered pairs, before → after | Field accuracy, before → after |
+| --- | --- | --- | --- |
+| Text extraction | 5/5 → 4/5 | 6/10 → 6/10 | 43/45 → 42/45 |
+| Classification | 4/5 → 4/5 | 4/10 → 9/10 | 30/30 → 30/30 |
+| Single-tool dispatch | 1/5 → 5/5 | 6/10 → 8/10 | 15/15 → 15/15 |
+| SQL generation | 4/5 → 4/5 | 3/10 → 9/10 | 15/15 → 15/15 |
+| Numerical reasoning | 3/5 → 3/5 | 3/10 → 10/10 | 15/15 → 15/15 |
+| Long-text retrieval | 3/5 → 4/5 | 8/10 → 6/10 | 45/45 → 43/45 |
+| Document photos | 4/5 → 4/5 | 8/10 → 7/10 | 135/150 → 134/150 |
+
+Membership compares against the measured-policy top five within each tested pool. Ordered pairs compare all ten pairs within each shortlist, using the same 60/20/20 policy on actual test accuracy, charges and request time. Overall membership improved from 24/35 to 28/35 and pair agreement from 38/70 to 55/70, but none of the seven final lists matched the entire measured order. Text extraction, long-text retrieval and visual extraction still showed small accuracy regressions. These results do **not** establish that the text-extraction regression is fully resolved or that all five recommendations are optimal.
+
+The first test exposed cross-task request failures; those test outcomes were not ingested as calibration. The reliability correction uses calibration observations only. Provider rate limits and timeouts count as failures, and latency can change between runs. The final sample is small and the visual test contains four payment slips and one invoice, including missing identifiers and tax fields. Reported improvements are development evidence, not a SOTA claim.
+
+## Refresh and data handling
+
+The task description is sent to Scout and then to OpenAI Decisions for interpretation. Scout's application code processes it in memory without saving or logging it. Only the public index and published calibration aggregates are persisted; prompts are not sent to benchmark sites or recommended-model providers. [OpenAI's API data controls and retention policies](https://developers.openai.com/api/docs/guides/your-data) apply to that provider call. Scout does not claim zero retention by OpenAI. Hosting infrastructure can have separate request-metadata policies. Clipboard actions occur only when selected. Describe the task rather than pasting actual invoices, credentials or private customer records.
+
+Each query has one Decisions request with a four-second timeout and no automatic retry. Shared in-memory admission limits allow at most eight active requests and 120 starts per minute for the Scout service. Excess requests fail promptly instead of queuing. These counters reset with the process and are not an account-level billing cap. Provider-side limits can also make interpretation unavailable. The API key is held server-side; callers never receive it.
+
+Sources refresh independently. A failed download, changed parser format, or unexpected loss of coverage preserves that source's previous evidence and timestamp while healthy sources and prices continue updating. Fresh catalogue removals still remove unavailable models. All eight Hub queries must succeed for its discovery snapshot to replace the previous one. A complete outage preserves the previous index. Failure and stale status are exposed in both interfaces.
+
+Source content is parsed as data, never executed. Downloads, decompression, and request bodies are size-bounded; source downloads time out. Links are restricted to HTTP(S). The publisher's OpenVLM data URL uses HTTP and lacks transport authentication; the remaining feeds use HTTPS. Scout does not independently authenticate benchmark submissions.
+
+## Run locally
+
+Requires Node.js 22.19 or newer. Make `OPENAI_API_KEY` available in the server environment before starting; never put it in browser code. It must have access to the Decisions API:
+
+```sh
+npm ci
+npm run scout
+```
+
+Open `http://127.0.0.1:45833/scout/`. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.
+
+`npm run scout:refresh` fetches the seven public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Refreshing public sources requires no provider key or inference budget. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid hourly evaluation job is installed. Recommendation queries require the server key and incur Decisions API usage.
+
+## Hosting and verification
+
+`wrangler.scout.jsonc` defines a separate Scout Worker, one SQLite Durable Object holding the index, and an hourly cron at minute 7 UTC. Configure `OPENAI_API_KEY` as a secret on this Scout Worker before deployment. Searches interpret the task once and use the prepared in-memory index. The bundled snapshot allows immediate startup; subsequent successful updates persist across restarts. Refreshes are serialized.
+
+`demo/domain-router.mjs` routes Scout paths to this service while retaining the existing website, protected memo, authentication, and dashboard services. The public guide is generated and published with the main documentation. Do not deploy the older root `wrangler.jsonc` to publish Scout. `npm run build:scout` packages only Scout, public documentation, and its shared assets; private memo and demo applications are excluded.
+
+```sh
+npm run check:scout
+npm run verify
+npx wrangler@4.136.3 dev --config wrangler.scout.jsonc
+```
+
+Tests cover measured-profile matching, held-out data isolation, ordered top-five behavior, false-positive regressions, response validation, memory/tool eligibility, provenance, missing metrics, source outage recovery, catalogue removals, provider failure/timeout, admission limits, shared API/MCP interpretation, compact evidence and routing isolation. Real API diagnostic cases are separate opt-in checks that incur provider usage. These establish implementation behavior, not task-level model accuracy. Local checks and CI do not publish a deployment. Physical mobile hardware requires separate verification.

package/docs/security.md

@@ -1,5 +1,9 @@
 # Security
 
+The [preview agent-evaluation runner](agent-evaluations.md) is a separate execution path: it runs trusted local agents/graders, passes only basic OS variables and explicitly named environment variables, and preserves agent-owned authentication. It sends task descriptions and optional model IDs to Scout. Local metric reports and budget reservations persist; temporary task outputs are removed. Fresh directories are not security sandboxes. Its budget admission controls do not enforce provider billing limits.
+
+The [budget selector](agent-evaluations.md#budget-selection) in `0.1.7-preview.2` requires an explicit $10, custom or no-fixed-cap choice after model scope. No fixed cap removes monetary admission limits, including Claude's per-attempt dollar flag, while retaining tasks, repetition bounds, timeouts, native permissions and accounting. Saved choices apply across reruns; a foreground watcher can incur further charges on each change until stopped. Provider limits remain independent.
+
 ## Credential ownership
 
 OpenMerit accepts no model-provider or Jev credential. Credentials belong to the coding harness, its environment, or an external secret store.
@@ -12,9 +16,9 @@ Most intents carry `modelMutationAllowed: false`. Only application-target apply-
 
 Persistent wakeup provisioning is a separate authorization boundary. An adapter that needs to install a system scheduler, create a paid cloud agent, or change infrastructure must declare `wakeupProvisioning: "infrastructure_change"`; OpenMerit will not request that provisioning unless the confirmed check policy sets `infrastructureChangesAllowed: true`.
 
-The unreleased macOS Pi adapter installs a user LaunchAgent only under that authorization and persistent execution mode. The plist stores the project path, selected Pi model identifier, and executable paths, but no API key. It verifies that Pi authentication works without shell-only credentials before installing. The agent launches Pi headlessly only for due elapsed-time checks, so the configured model provider may receive project context and incur charges under the confirmed policy. Pause unloads the agent; already-running work is not cancelled.
+The macOS Pi adapter installs a user LaunchAgent only under that authorization and persistent execution mode. The plist stores the project path, selected Pi model identifier, and executable paths, but no API key. It verifies that Pi authentication works without shell-only credentials before installing. The agent launches Pi headlessly only for due elapsed-time checks, so the configured model provider may receive project context and incur charges under the confirmed policy. Pause unloads the agent; already-running work is not cancelled.
 
-The source runner imposes a 15-minute wall-clock limit and records a nonzero status if Pi fails to complete a due intent. This is a safety bound, not a provider-spend cap. Headless completion remains unverified in the current live test.
+The background runner imposes a 15-minute wall-clock limit and records a nonzero status if Pi fails to complete a due intent. This is a safety bound, not a provider-spend cap. Unattended candidate discovery and model changes still need representative product validation.
 
 ## Local state
 
@@ -28,13 +32,13 @@ OpenMerit validates structure and deterministic Pareto conformance, not the trut
 
 Do not include credentials, private traces, or proprietary evaluation data in public issues.
 
-## Inference Terminal (0.1.6)
+## Inference Terminal
 
 The terminal reads supported project metadata, request logs, explicit exports, and authorized loopback runtime inventory/metrics endpoints. It calls no inference endpoint and exposes no mutation API. A local syntax index reads supported Python, JavaScript/TypeScript, JSON, YAML, TOML, and project-local dotenv declarations. It does not execute or import project code, source shell files, or resolve arbitrary expressions. Known credential-variable presence can identify configuration; credential values are discarded and never used for provider calls. Endpoint addresses are sanitized before retention. Static URLs found by the syntax index never authorize network probes.
 
 Loopback runtime probes require supported project endpoint declarations or explicit terminal configuration. No standard ports, remote provider accounts, credential stores, running processes, or general home directories are scanned. Whole-machine readings are context, not per-request measurements; the compute view requires explicit links or supplied host evidence.
 
-Coding-harness histories are outside the terminal's application scope and are not scanned or imported. The retired Pi history reader and access prompts are removed; legacy grants in `~/.openmerit/terminal-access.json` are never consulted. Application request logs and explicit exports retain only allowlisted metadata, without prompts or responses. Sources must belong to the selected application; the terminal cannot infer the purpose of an unlabeled API response. Records explicitly marked `scope: "development"` are rejected.
+Coding-harness histories are outside the terminal's application scope and are not scanned or imported. Application request logs and explicit exports retain only allowlisted metadata, without prompts or responses. Sources must belong to the selected application; the terminal cannot infer the purpose of an unlabeled API response. Records explicitly marked `scope: "development"` are rejected.
 
 The local web service binds to `127.0.0.1` and rejects unexpected Host, Origin, and cross-site browser requests. It only serves bundled assets and specific read-only JSON routes. It is not authenticated against other local processes or machine users. Model names, endpoint identifiers, and relative source paths can still be sensitive. Names and identifiers in imported metadata must be kept free of secrets.
 

package/docs/site-demo.md

@@ -175,3 +175,7 @@ Final six-task validation passed `npm run verify`, including 33 demo tests and 1
 Published as Worker `f72b3b7b-2b0b-46ec-9bb6-8cf4b726d629`, build `demo-20260928-fieldkit-v1`, with container image `sha256:db1c69574b9777b7e4eaf13a4ef282b1a00377625fff280832ea03b5e557c090`. Cloudflare reports the container application ready. Thirty-six asset responses across both domains match the checked local files, including all five recordings and all six demo guide pages. Both configuration responses identify the new build, expose the six-stage routes and four-goal verification policy, and report `enabled: false`. Unconfirmed session-creation probes return HTTP 503, so no public inference session was started. The published demo and its Field kit guide were checked in the browser; the completed view shows the six distinct models on each side and the recorded metrics with no JavaScript warnings/errors. Temporary recording runners ended after evidence capture.
 
 The full-site publication was refreshed as Worker `6b9dfbd1-d9a0-4f13-9120-4e754774998a` on September 28, 2026. All 190 public-file responses across both domains match the local build, including all demo assets and guides. The container image and measured recordings are unchanged, and public inference remains disabled. No new paid run was started for this publication.
+
+## Device appearance — 14 October 2026
+
+The recorded demo's colors now follow the device's light/dark preference, including tabs, playback controls, results, outputs, and trace states. The newer deployed interactive demo's presentation stylesheet was recovered separately and themed across its workflow, inspector, code, switches, and status states. The interactive demo presentation is published; the recorded player remains a source-only historical example. Runtime behavior is unchanged. See [theme maintenance](site-themes.md) and the [publication record](site-publication-2026-10-14.md).

package/docs/site-design.md

@@ -1,5 +1,7 @@
 # OpenMerit website
 
+Domain operations: see the [primary domain migration](site-domain-migration.md) for current routing, email, auth, and deployment constraints. Historical publication records below retain their original domains.
+
 The website lives in `site/` and uses the existing Cloudflare asset configuration in `wrangler.jsonc`. The homepage is plain HTML, CSS, and JavaScript. The [documentation section](site-documentation.md) is generated from curated Markdown at build time; no browser framework is required. The [guided demo](site-demo.md) adds a Worker API and temporary Cloudflare Containers; Docker must be running for deployment.
 
 The playback-only revision is published as Worker `31c181c9-c083-4eb4-85a4-f47516eba877`, build `demo-20260928-fieldkit-v1`. It removes live sandbox controls, disables public session creation, adds a dedicated Demo documentation group, and adds the six-task Field kit recording. Each of the five examples has a persistent two-sentence description below its tabs. Both Field kit paths already use six distinct models; five assignments change while the feasibility model is retained. The recorded comparison shows 22% lower observed cost and 13% lower mean latency, with 4/4 evaluation and 4/4 separate verification goals passing. The deployment history below describes earlier published behavior.
@@ -34,7 +36,7 @@ Production verification: 54 URLs across both domains match the checked local fil
 - Reference: https://kobbe.io/ — quiet typography, warm neutral surfaces, a wide illustration, and generous whitespace. Copy and artwork are original to OpenMerit.
 - The page contains the hero, getting-started section, FAQ, and footer.
 - The standalone [memo](site-memo.md) is generated from `docs/memo.md` at `/memo/`, with a homepage navigation link and a reading layout using the site's typography and colors. It describes the company's direction separately from released product documentation. The memo, header artwork, and navigation link are published through the existing Cloudflare workflow.
-- The [documentation section](site-documentation.md) publishes 22 guides with grouped navigation, section outlines, lazy-loaded browser search, copyable examples, and mobile disclosures. It builds from maintained Markdown and validates local links before deployment.
+- The [documentation section](site-documentation.md) publishes 24 guides with grouped navigation, section outlines, lazy-loaded browser search, copyable examples, and mobile disclosures. It builds from maintained Markdown and validates local links before deployment.
 - The FAQ has 13 questions covering the harness boundary, supported adapters, evaluations, model tradeoffs, evidence, approvals, regressions, background checks, pause, data, cost, budget enforcement, and readiness. Claims are reviewed against the published release using the [FAQ maintenance procedure](site-faq-maintenance.md). The 0.1.5 pause limitation is explicit; budget and evidence checks are described without implying independent enforcement or evidence authentication.
 - Small miniature vignettes accompany setup (prepared tools), the FAQ (two makers comparing notes), and the footer (the workshop dog at rest). See [assets and generation prompts](site-section-artwork.md). They are decorative, lazy-loaded, and deliberately much smaller than the hero.
 - `site/illustrations.js` plays sequences of separately generated poses: the seated maker raises a hand plane, the toolbox lid closes and opens, the FAQ makers point across their notebook, and the dog lifts its head. The hero uses four individual 2172 × 724 images; eight poses for each supporting scene are packed into WebP atlases. A canvas copies complete frames with no displacement filters, morphing, crossfades, or procedural deformation. See [the sequence assets, exact prompts, and packaging notes](site-illustration-sequences.md) and [the hero resolution correction](site-hero-resolution.md).
@@ -92,3 +94,7 @@ Generated with the built-in imagegen tool. The uncompressed original is retained
 ## Full-site publication, September 28, 2026
 
 All current website changes were deployed as Worker `6b9dfbd1-d9a0-4f13-9120-4e754774998a`. `npm run verify` passed before publishing, and Cloudflare repeated its build, demo, and documentation checks. All 95 public files matched the local build on both domains (190 byte-for-byte checks), covering the homepage, memo and artwork, 22 documentation guides and their source downloads, all five demo recordings, scripts, styles, fonts, and image assets. Both demo configuration responses report `enabled: false`; no inference session was started. The existing container image was unchanged and reused.
+
+## Automatic appearance — published 14 October 2026
+
+The source styles now follow the device's light or dark preference without a website switch. Shared semantic colors cover the landing page, docs, memo, access forms, engineering article, and both demo presentations. The landing-page illustrations retain their poses and full resolution with [transparent backgrounds](site-illustration-transparency.md), so they sit directly on the page in either theme. The hero keeps its workshop fixed and registers only the moving maker to remove camera drift between generated poses. See [theme sources, verification, and publication constraints](site-themes.md); the [publication record](site-publication-2026-10-14.md) identifies the deployed versions.

package/docs/site-documentation.md

@@ -1,6 +1,8 @@
 # Website documentation
 
-The public documentation lives at https://openmerit.site/docs/. It uses the homepage's typography, colors, and Cloudflare hosting. Twenty-two curated guides, including a dedicated Demo group with an overview and five example walkthroughs, are generated from Markdown in `docs/`; internal website design and maintenance notes are excluded.
+Domain operations: see the [primary domain migration](site-domain-migration.md) for current routing, email, auth, and deployment constraints. Historical publication records below retain their original domains.
+
+The public documentation lives at https://openmerit.ai/docs/. It uses the homepage's typography, colors, and Cloudflare hosting. Twenty-four guides are generated from Markdown in `docs/`. The terminal is in Get started; demos distinguish hosted samples from the five historical recordings retained in main. Internal website design and maintenance notes are excluded.
 
 Article content has one maintained source: `docs/*.md`. Update those files alongside product changes, then rebuild and deploy. Do not edit the generated `site/docs/` files. The repository's `AGENTS.md` records this rule for future coding work.
 
@@ -15,7 +17,7 @@ The structure draws on [Pi's documentation](https://pi.dev/docs/latest): a group
 - All documentation pages currently share the homepage's default Open Graph/Twitter image. Separate card generation is disabled for now; the optional template is retained in `scripts/social-image.mjs`. See [sharing-image configuration](site-dynamic-og.md).
 - `site/docs/` contains generated HTML, downloadable Markdown, the search index, the self-hosted search library and its license, and a manifest. Generated files are ignored by Git.
 - Search loads its library and index on the first query of at least two characters. No search query is sent to a search service. Hashed filenames and versioned CSS/JavaScript URLs update with the source or supporting assets. If a tab still references assets removed by a newer deployment, search fetches the current manifest without using its cache and retries with the current library and index. Recovery uses a fresh module URL because browsers can cache failed imports. If recovery also fails, the error message lets the reader retry by typing again; retries do not loop in the background.
-- Each page displays the package version and compact actions for viewing its source on GitHub, editing that source, and reading the generated Markdown. This is documentation for the current release, with no historical version selector yet.
+- Each page displays the configured **Main branch** label and actions for viewing its source, editing it, and reading generated Markdown. The manifest retains the actual package version separately; a preview version in package metadata is not presented as the stable release. Current guides describe main and distinguish hosted/source appearance changes from npm 0.1.6. There is no historical version selector.
 
 ## Pi component structure
 
@@ -30,7 +32,7 @@ The 2026-09-23 component pass follows the rendered [Pi documentation layout](htt
 
 At intermediate widths, the documentation rail remains visible and the article outline moves above the article. Narrow layouts use the combined navigation component; its disclosures and section links work without JavaScript. Search and code copying remain available. Previous/next page links are omitted to match Pi's component structure. The component pass introduces no runtime framework or separate article content.
 
-The header contains the OpenMerit brand and Home/GitHub links. The sidebar begins directly with its navigation groups and ends with the package version. The repeated “Documentation” labels in the header and sidebar, and the sidebar's “Early development” link, were removed on 2026-09-23.
+The header contains the OpenMerit brand and Home/GitHub links. The sidebar begins directly with its navigation groups and ends with the documentation scope label. The repeated “Documentation” labels in the header and sidebar, and the sidebar's “Early development” link, were removed on 2026-09-23.
 
 ## Editing and previewing
 
@@ -81,3 +83,7 @@ Chrome browser checks covered all 16 pages at 1440px and 320px with HTTP 200 res
 WebKit also loaded all 16 pages without overflow at 1440px and 390px. Search, keyboard result navigation, Escape, mobile navigation, section links, and the clipboard fallback passed without runtime errors. Axe WCAG 2 A/AA and 2.1 AA checks passed for the overview, budget table, adapter code examples, and open search results at 1440px and 390px after increasing code-label contrast. These are browser and automated accessibility checks, not exhaustive accessibility certification or physical-device validation.
 
 Production smoke testing caught an event-ordering bug in pointer selection: a `focusout` microtask could close the result panel before the clicked link received focus. Closing the panel on a completed `focusin` outside search fixes that race. Mouse and touch result navigation, keyboard browsing, Escape, and moving focus outside search subsequently passed in both Chrome and WebKit. Include actual mouse and touch result selection in future browser reviews alongside keyboard tests.
+
+## Device appearance — published 14 October 2026
+
+Documentation now shares the automatic light/dark website palette. Search results, sidebars, table headers, code examples, mobile disclosures, and focus states follow the device preference without a reload or a theme control. Printing uses light colors. The generator emits appearance metadata before styles load; articles remain maintained exclusively in Markdown. See [theme maintenance](site-themes.md) for verification and the separate publication boundary.