openmerit 0.1.9 → 0.1.11

package/docs/scout.md

@@ -1,11 +1,93 @@
 # Scout by OpenMerit
 
-Describe what you need a model to do. Scout returns up to five model-and-endpoint candidates per task. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page and JSON API use the same index and ranking.
+Scout is OpenMerit's model search agent. Describe your task and get model recommendations. Scout returns up to five model-and-endpoint candidates per task. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page and JSON API use the same index and ranking.
 
 Scout is a separate hosted beta product. Npm 0.1.7 includes optional Claude Code setup for it; the hosted service does not change the Pi coordinator or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
 
+**Website rebuild — published 24 October 2026:** the rebuild preserves the existing landing page at `/` and Scout at `/scout/`, and takes all hosted `/docs/` pages offline, including this guide. `/connect/`, the JSON API/MCP and the standalone `/scout/data-policy/` remain available. The guide remains maintained in Markdown; this does not remove the local CLI or its documentation from the package. See the [cutover notes](site-domain-migration.md) for publication and verification details.
+
+**Small-screen handoff — source only, not deployed:** below 768 CSS pixels, the Scout page replaces its search interface with a note from Lazim asking visitors to use a larger screen. **Share link** attempts to copy the public Scout URL and opens the device’s native share sheet where supported. If clipboard access fails, the visible URL can be copied manually. At 768 CSS pixels and above, including full-width iPad layouts, the search interface remains available. This does not change JSON API or MCP access.
+
 **Coding-agent setup — 24 October 2026:** stable npm 0.1.7 includes [device sign-in setup](#connect-a-coding-agent). Hosted Scout serves `/connect` and `/scout/mcp`; the button opens Claude Desktop with a setup prompt. This supersedes the 23 October web-placeholder reset. Existing older client configuration and authorization grants are not automatically removed or revoked. Scout serving, main-site documentation and npm publication remain separate releases; dated receipts below describe their original versions, not acceptance of every later build.
 
+**Session restoration — unreleased source:** onboarding stays hidden while sign-in is checked, so returning signed-in visitors do not see the introduction flash on reload or reopening. Chat remains locked until an active session is confirmed. Signed-out visitors and failed checks reveal the introduction with sign-in still available; signing out restores it. No saved tour-completion flag grants access. This is not yet deployed and changes no homepage FAQ claim.
+
+**Product wording — unreleased source:** the homepage, Scout page, onboarding and search/social descriptions call Scout “OpenMerit's model search agent,” with “model search” as the shorthand and “Find the right AI models for your task” as its promise. “Model recommendations” describes its output; the web and coding agents are access channels, not separate product categories. This copy is not yet deployed. Dated receipts below retain the wording and behavior of their original releases.
+
+## Five-step introduction modal — deployed, 28 October 2026
+
+**Deployment receipt:** the owner approved commit, push and deployment. [PR #28](https://github.com/laz-aslam/openmerit/pull/28) merged as [a70103a](https://github.com/laz-aslam/openmerit/commit/a70103a1c8eb8a701ee262106ae6e6c026cf04b4). All PR and merged-commit CI checks passed. The guarded Scout-serving deploy moved `openmerit-scout-alpha` from `2cf2c2d3-8aef-40f0-9c44-800bcab247bf` to `17ab0166-9a5c-43e9-932b-c13bee040db7` at 20:45 UTC. Live Scout HTML, chat/auth/onboarding JavaScript, styles and Claude screenshot matched the release build. `/` and `/connect/` stayed byte-identical; other Workers were not redeployed. The catalogue remained healthy on snapshot `588855a0bf28212fad48a9513dbf3d5b`.
+
+Live Chrome desktop/mobile and WebKit mobile checks covered all five slides, locked future navigation, revisiting reached slides, real Google/GitHub sign-in controls with optional analytics blocked, dismissal keeping the page inert, signed-out reload returning to slide one and zero agent turns during the tour. Browser checks with simulated sign-in/replies covered all three example tasks, exact submitted messages, single submission after double activation, keyboard activation and New restoring examples; auth fixtures covered resumption after sign-in. Completed production OAuth, an authenticated production recommendation, native Claude installation and physical-phone behavior remain unverified.
+
+Signed-out visitors see a five-step introduction modal over the actual Scout page, with the page dimmed and softly blurred behind it. The first is a personal note signed by Lazim and Praveen: Scout is OpenMerit's model discovery product in early beta, and feedback on what is useful, confusing or could improve is welcome. This is a text-only note, not a feedback form or a new feedback channel. The second introduces Scout with bundled lab/Hub marks and coverage from the same optional status request as the page: indexed models, benchmarked labs, benchmarks and distinct source-feed IDs. Hub repositories appear separately because they can overlap the catalogue. Labs count model publishers with benchmark results, not every inference host; source feeds include research records, not distinct organizations. Failed or invalid status responses leave a general coverage description, never fabricated counts, and do not block the tour or sign-in.
+
+The third explains that Scout recommends rather than routes or resells inference tokens. Its illustration separates a recommendation from the visitor's app → provider/router → model runtime path. Recommendations can be constrained to the public endpoint inventory supplied through the [coding-agent project configuration](#using-scout-from-the-agent) or API; the slide does not import accounts, collect provider keys, connect endpoints or introduce a web configuration form. The fourth explains the task-to-shortlist workflow with an illustrative conversation; its model order, quality estimates and prices are example values, not a live recommendation or measured task accuracy. The fifth explains the optional Claude Code integration and shows cropped excerpts of an actual session, mentioning MCP, API and CLI availability. Its historical model recommendation is not a recommendation for the current visitor. The full-size capture opens separately. Viewing any step does not submit a task, run inference, install Scout or launch Claude.
+
+**Continue** advances through the first four steps; **Get started** on the fifth opens the existing Google/GitHub sign-in flow above the introduction. The five progress controls revisit only slides already reached through Continue; future slides are disabled and excluded from keyboard navigation. Going back preserves access to all reached slides, including those ahead of the current slide, for the current page visit. There is no separate back button. The modal has no separate header sign-in or footer navigation, and the conversation/Claude previews have no supplementary captions. The conversation preview's title bar uses decorative red/yellow/green window lights instead of the Scout wordmark and example label; these do not close, minimize or maximize the modal. Its accessible image description still identifies the values as illustrative. The underlying page stays visible but inert until Clerk reports an active session, including after an OAuth return; a dismissed, failed or pending sign-in never enables it. The introduction cannot be dismissed to bypass this gate. Successful onboarding sign-in dismisses the introduction and focuses chat without automatically sending the example. Existing explicit draft-submission resumption remains separate. Returning active sessions bypass the introduction. Signing out restores the modal and makes the page inert again.
+
+Tour progress stays in page memory: signed-out reloads and new page visits start at the first slide, even after opening or dismissing sign-in. The tour does not read or write a completion marker; old `scout-onboarding-v1` markers are ignored. Reloading with an active authenticated session still bypasses the introduction. Blocked storage does not prevent navigation or sign-in. Privacy controls and the data policy remain directly accessible without authentication at `/scout/privacy/` and `/scout/data-policy/`; their links stay in the main page footer, not the modal. No onboarding-specific analytics event is added; existing page/sign-in analytics and preferences remain effective.
+
+After sign-in, the empty chat shows three task examples under **Try these**: support-ticket summarization, invoice extraction and product-photo classification. Selecting one immediately submits that task through the normal composer/auth flow and starts a conversation; no separate Send action is needed. If the session has expired, the task stays as a draft while sign-in opens and resumes through the existing authenticated-submit flow. The examples disappear when the first conversation turn appears; **New** clears the conversation and shows them again. They remain inside the chat surface, above the fixed composer, and cannot bypass the onboarding/auth gate. The homepage makes no affected example-task claim and needs no FAQ change.
+
+Pointer navigation uses a short, non-looping opacity/position transition. Keyboard navigation and reduced motion change steps without movement. Keyboard focus stays inside the introduction, yielding to the sign-in modal while it is open. On narrow and short screens, the introduction scrolls within its rounded panel; the underlying page does not scroll. The homepage makes no affected onboarding or sign-in claim and needs no FAQ copy change.
+
+Design references: [Linear's welcome screen](https://mobbin.com/screens/62773d0f-3574-4191-ac91-eb8a51697a95), [Linear's onboarding flow](https://mobbin.com/flows/cbaf58a9-fd84-4afe-b214-e65b24484013), [Notion's focused setup screen](https://mobbin.com/screens/99396b7b-c21d-4cea-8956-4942c2b9aae6), [Neon's integration walkthrough](https://mobbin.com/screens/8b8b58fe-86fd-4a2a-a5ad-51b84614d7b2), and [Torph's text-continuity examples](https://torph.lochie.me/). The implementation keeps Scout's own graphite palette and system typography; it does not embed Mobbin or generated mockup assets or add an animation dependency. The Claude Code image comes from the owner's 24 October recorded session: two unaltered content crops of `claude-automatic-expanded-full.png`, `(1330, 1029)–(2810, 1103)` and `(1330, 1552)–(2810, 1770)`, stacked with a 42-pixel neutral gap and encoded as WebP. OS chrome, account information and unrelated session content are excluded.
+
+## Every model kind in the agent — source only, not deployed
+
+This describes source changes that are not yet published or deployed. Until both the publisher and the serving Worker are deployed and a refresh has published a new catalogue version, the agent behaves as described in the next section.
+
+Each published catalogue version will include every model in the merged catalogue, not only chat models: embedding, rerank, image, video, speech (text to speech), transcription (speech to text), decision, OCR, moderation, 3D and music models, plus models whose kind is unknown. Each model carries its kind, input and output modalities, context window, release date, open-weights status and up to 16 catalogue routes with their current base-tier prices. Models also carry their linked leaderboard results from the media, speech, decision and embedding leaderboards Scout imports (Arena, Artificial Analysis, TTS Arena, Open ASR, 3D Arena, VBench, MMEB and others), and from MTEB's final leaderboards: metric, whether higher or lower is better, score, rank out of every result on that leaderboard version, leaderboard version, date and source. Chat models keep Scout's reviewed routes and benchmark results unchanged. The agent reads only this published copy, so a refresh that is still running never changes what one conversation sees between search and the shortlist check.
+
+The agent's tools change as follows:
+
+- **search_catalogue** (new) searches models, benchmarks and leaderboards by free text. By default it combines keyword (BM25) and embedding results by reciprocal rank fusion; it can also use either alone, and filter by result type and model kind. Each result names the field that matched. Without embeddings (no OpenAI key, an embedding error, or a catalogue version published without them) it returns keyword results and says so. Its queries, written by the agent from your task, are embedded with OpenAI, as benchmark search already does.
+- **search_models** filters by model kind and reports the cheapest eligible price in each unit a model is billed in: per million tokens, per image, per second of audio or video, per million characters, per page or per request. Sorting by price compares one unit only (by default the usual unit for the requested kind, otherwise tokens); models not billed in that unit follow, unsorted. A route that lists only zero prices, such as a local runtime or a free tier, is not counted as a price.
+- **get_model_details** and **benchmark_leaderboard** include leaderboard results, and leaderboard IDs work wherever benchmark IDs do.
+- **submit_recommendation** accepts models of any kind. Fewer than five are accepted only when fewer models meet the stated hard constraints, which can now include the model kind. The per-task workload is given in billing units (input and output tokens, images, seconds of audio or video, characters, pages, requests) and cost per 1,000 tasks is computed from catalogue prices on the eligible route that prices most of the workload, then the cheapest. Quantities the route does not price are listed, and such a partial cost never leads on cost. Citations must be the model's own benchmark or leaderboard results.
+
+In the shortlist, the Input and Output columns keep per-million-token prices; a model billed in another unit shows that unit's price there instead, marked with its unit, and the expanded model shows the estimated cost per 1,000 tasks. Known limitations: per-second audio prices apply to audio seconds, but audio billed in tokens must be estimated as tokens; image prices are per generated image; prices are base-tier list prices from catalogue sources and can be wrong at the source.
+
+## Conversational Scout agent replaces the search page — deployed, 28 October 2026
+
+**Deployment receipt:** the owner approved the commit, push and Scout serving deployment. [PR #26](https://github.com/laz-aslam/openmerit/pull/26) merged as [332dda3](https://github.com/laz-aslam/openmerit/commit/332dda3d889a2da386f2d0bb59e60cc5170d29ce). After all PR and merged-commit CI checks passed, the guarded deployment wrapper moved `openmerit-scout-alpha` from `88e33d4c-4606-4eb1-be27-36a2b05309ff` to `2cf2c2d3-8aef-40f0-9c44-800bcab247bf` at 14:38 UTC (18:38 Dubai). The deployed Scout HTML, JavaScript, stylesheet and auth module matched the local release build. `/` and `/connect/` remained byte-identical; the publisher, router, terminal and memo service were not redeployed. The catalogue stayed healthy on snapshot `588855a0bf28212fad48a9513dbf3d5b`. Live desktop/mobile checks opened the real Google/GitHub sign-in controls with optional analytics blocked, preserved the task on dismissal and sent no agent turn before authentication. Mobile scroll checks reached the full-height chat before the footer. Completed OAuth, an authenticated production recommendation, native Claude installation and physical-phone keyboard behavior remain unverified. CI's first merged-commit attempt failed an existing assertion that mistook `900` inside random UUIDs for leaked error data; the failed jobs passed on retry without changing that test.
+
+**Claude Code action — deployed, 28 October 2026:** **Add to Claude Code**, with the existing Claude icon, sits below the intro's coverage counts. It restores the secret-free setup handoff described [below](#connect-a-coding-agent): copying and requesting the prefilled Claude link happen independently, without waiting for clipboard permission or requiring a chat sign-in. Feedback appears below the button and distinguishes a copied prompt from a requested launch; neither proves installation. Current task text is never included. DNT/GPC, saved usage refusals and account unlinking propagate into the installer command. A fresh setup ID is included only when browser usage collection and account linking are both enabled; blocked optional analytics does not manufacture a saved installer opt-out or prevent the handoff. On stacked screens, the intro and setup feedback scroll above the full-height chat. Browser-platform fixtures cover prompt/deep-link equivalence, privacy and failure handling; native Claude launch and completed installation remain unverified.
+
+**Header wordmark — deployed, 28 October 2026:** Scout uses the footer OpenMerit wordmark's 18px size, 550 weight and tight letter spacing, rendered lowercase. The **New** label aligns with the header divider's right end while retaining the button's padding; account controls precede it when present. Sign-in, account access and conversation reset are unchanged.
+
+**Chat surface and footer — deployed, 28 October 2026:** the conversation and composer share a subtle theme-aware surface with no outline. Spacing separates responses and task-brief fields without horizontal dividers; the model comparison header also has no divider. Continue is the task brief's only action, using a compact filled button with no explanatory line below it. Buttons are 32 CSS pixels high for mouse input and at least 44 pixels when a coarse pointer is available. Scout responses show the OpenMerit icon beside their Scout label; user messages keep their You label. The OpenMerit home logo sits at the footer's right edge. Data policy, Privacy & analytics, then Scout by OpenMerit stay vertically stacked at its left edge with the same spacing. Link destinations and privacy controls are unchanged.
+
+**Viewport layout — deployed, 28 October 2026:** above 900 CSS pixels, navigation, title/description/coverage and footer form a left column; the chat fills the right column's height, with its composer at the bottom and its conversation scrolling independently. At 900 pixels and below, navigation and intro scroll above a chat section that occupies the full dynamic viewport height. Native page scroll snapping stops at the chat's top; its transcript scrolls inside it while the composer stays at the bottom. Scrolling beyond the transcript's end can continue down the page to the footer, the only section below the chat. Focus the conversation to scroll with the keyboard, including long briefs and expanded model details. **New** stays in navigation instead of taking up a transcript row: a compact borderless button with a plus icon, labelled “New conversation” for assistive technology and in its tooltip. The intro and composer compact on short screens, and long drafts scroll inside a height-limited input instead of taking over the conversation. Touch-input text is at least 16 pixels to avoid focus zoom; safe-area padding protects the full-height chat's top and composer. Mobile viewport height follows browser chrome, with keyboard-driven layout resizing where the browser supports it. Physical-phone scrolling and keyboard behavior remain unverified.
+
+**Intro coverage — deployed, 28 October 2026:** the heading reads “Find the best AI models for your work.” A small overlapping lab/Hugging Face icon stack sits above live coverage from the public status endpoint. The counts distinguish catalogue models (`availableModels`), Hub repositories (`specialistRepositories`), labs with benchmark results (`providers`), indexed benchmark series (`benchmarks`, including descriptions without results), and distinct source-feed IDs (including retained research records, not unique organizations). Catalogue models and Hub repositories may overlap and are not added into a unique total. If status is unavailable, counts stay hidden without blocking sign-in or sending. These are catalogue coverage counts, not proof of task performance or an optimal recommendation. The homepage makes no affected coverage claim and needs no copy change.
+
+The conversational agent is the Scout page at `/scout/`, replacing the search box and its results table. The page stays open to everyone; sending a message requires sign-in, as searching did, and a message typed while signed out is kept while you sign in. The JSON search API (`/scout/api/recommend`), the MCP server (`/scout/mcp`), coding-agent setup at `/connect/` and the catalogue refresh are unchanged, so the CLI and installed coding agents keep their current behaviour. Sections below that describe the search page's prompt examples, results table and loading messages are historical; the Claude Code setup handoff remains available in the new page.
+
+You describe the work; Scout writes a task brief with five components (input, work, output, criteria and constraints), each marked stated, assumed or missing. Each component is editable in place: correct anything Scout got wrong, or fill in a missing one. All five fields share the same editable background, including Work; the page does not show per-component question text. Once sent, the brief collapses. **Continue** sends only the components you changed and requests recommendations immediately, using stated assumptions for missing details. It also works without edits; there is no separate Skip questions button. Pressing Enter in a brief field does the same; Shift+Enter adds a line break. The message box stays at the bottom of the page. Each turn ends with an editable brief or a shortlist of five models, followed by Scout's summary. While a turn runs, its steps are listed with the current one last; afterwards they stay as a collapsed record of the work. The shortlist reports Scout's 0–100 quality estimate with the cited benchmark ranks, input and output prices per million tokens on its cheapest eligible route and release date. Expanding a model shows its rationale, benchmark ranks, route and reference latency. A missing release date is shown as unknown. Reply in the same conversation to change constraints or trade-offs; **New** starts over.
+
+The agent runs in a Cloudflare Durable Object, one per conversation, using Pi Durable and the Agents SDK. Its model is set by `SCOUT_AGENT_MODEL`: `openai/<model>` calls OpenAI's Responses API directly with the Worker's `OPENAI_API_KEY` (default `openai/gpt-6-sol`), and any other value runs on Workers AI, such as `@cf/zai-org/glm-5.3`. Its tools search benchmark descriptions by embedding (OpenAI embeddings, as in search), read leaderboards and model records from the published catalogue, and use Workers AI web search when the catalogue cannot answer something decisive. Cost, latency and recency figures come from catalogue data, not from the model's text, and a shortlist that cites models or benchmarks outside the catalogue, or breaks a stated hard constraint, is rejected and resubmitted. A turn usually takes one to three minutes.
+
+The page uses the same sign-in and rate limits as search:
+
+- `POST /scout/api/agent` with `{"message": "…"}` starts a conversation and runs one turn. Continue it with `conversationId` plus `message`, `brief` (the edited components, such as `{"constraints": "Open weights only"}`; an empty string removes one) and/or `skip: true`. A client may supply its own UUID as `conversationId` with the first message.
+- `GET /scout/api/agent/<conversationId>` returns `busy`, the turn's `progress` steps and its latest `answer`. The page polls it while a turn runs and reads the answer from it if the POST connection drops.
+
+Conversations are keyed by the signed-in account, so one account cannot read another's. The conversation transcript is stored in its Durable Object with no expiry yet. With the default `openai/gpt-6-sol` agent model (not yet deployed), the task text and the conversation reach OpenAI, which also embeds benchmark and catalogue searches; web search runs on Workers AI. Before that change, the task text reached Workers AI and, for benchmark search, OpenAI. The [beta data policy](/scout/data-policy/) still needs updating to describe conversation storage. The deployed serving configuration includes the `SCOUT_AGENT` Durable Object (migration `agent-1`), the Workers AI binding and `nodejs_compat`.
+
+## Repeatable searches and task-specific evidence first — deployed, 27 October 2026
+
+**Repeatable results.** Scout's interpretation and evidence checks are AI calls whose answers vary between attempts, so an identical search could return a different shortlist: across 34 test cases run three times, repeats shared only 34–39% of their top five. A completed search now records each AI answer it received. For seven days, an identical search (same prompt, model/endpoint scope, catalogue generation, interpreter version and UTC day) replays those answers instead of asking again, so it returns the same shortlist. Failed or degraded searches record nothing. A new catalogue generation, interpreter version or day asks again. See [the data policy](/scout/data-policy/) for how the answers are stored. Relative dates in restrictions now resolve against the UTC day rather than the exact time. Interpreter version is 40.
+
+**Task-specific evidence first.** When some candidates in the quality band have a score on a benchmark of the task itself (direct evidence), they are ranked and labelled strongest, best value and fastest among themselves first. A model estimated only from general benchmarks can no longer outrank them or remove them as dominated; it can still fill a remaining place, and a warning names it. The same applies to domain evidence when no direct benchmark was accepted. For example, a model with only a high general score no longer leads a medical coding search above models measured on MedCode.
+
+The evidence assessment call now allows one retry within the search deadline (up to four seconds instead of 1.8), because a single slow answer previously failed the whole search.
+
+## Twice-daily catalogue refresh — deployed, 25 October 2026
+
+The hosted catalogue publisher refreshes twice a day, at 00:07 and 12:07 UTC (05:37 and 17:37 IST), instead of hourly. Each run reads the structured feeds and the D1 catalogue and applies the same publication checks. Since 26 October 2026 it no longer runs Parallel research jobs or the Hugging Face recently-modified listing discovery that fed them. Their 234 discovered benchmarks had no model results; the descriptions they collected are kept and are not refreshed, so they never mark the catalogue stale. Public feeds are reported stale 13 hours after their last successful check, which is one missed run plus an hour's grace. A single failed run can therefore leave up to roughly a day between source checks. The local `npm run scout` preview still refreshes its bundled feeds hourly. The owner-approved publisher and serving deployments went live on 25 October 2026 at 17:00 UTC.
+
 ## Beta evaluation logging — 22 October 2026 rollout
 
 The beta default records submitted prompts alongside recommendations for matching evaluation and debugging, not foundation-model training. The serving configuration enables `SCOUT_BETA_EVALUATION_LOGGING=true`, alongside `POSTHOG_CAPTURE_CONTENT=true` and complete account-sync/deletion configuration. The [beta data policy](/scout/data-policy/) is generated from Markdown and published with Scout, independently of the main website. This rollout supersedes the dated default-off contract below. The operator remains responsible for the applicable privacy basis and restricted PostHog access; a beta label does not establish consent or another lawful basis.
@@ -24,6 +106,12 @@ Examples go through the same interpretation and ranking as typed prompts, with n
 
 ## Product analytics
 
+### Scout setup analytics default — version 0.1.10
+
+Version 0.1.10 enables usage and installation diagnostics by default for Scout's `install`, `login`, `status`, `remove`, `logout`, `project` and `doctor` commands. The normal homepage and Scout handoff prompts use `npx -y openmerit@latest install --client claude-code` without an analytics-enabling environment prefix. Terminal and evaluation analytics remain opt-in. npm 0.1.9 uses the opt-in setup behavior below; publish 0.1.10 before deploying these prompts.
+
+`OPENMERIT_TELEMETRY=0` or `DO_NOT_TRACK=1` disables setup analytics, strips outgoing correlation IDs and carries the refusal into the approval URL and installed MCP headers. Scout's usage-disabled handoff still includes `DO_NOT_TRACK=1`, and its unlinked handoff includes `OPENMERIT_ANALYTICS_ACCOUNT=off`. Browser DNT/GPC, saved browser choices and server-side account opt-outs remain effective. Analytics-enabled, linked installations retain the random installation UUID described below. The homepage creates no browser setup ID. MCP prompt-content capture stays off; beta status does not establish consent or another legal basis. Package and hosted publication are independent; these changes alone do not establish live event ingestion.
+
 ### Coding-agent lifecycle instrumentation — npm 0.1.9
 
 Npm 0.1.9 includes these installer changes; browser/server instrumentation requires the matching independent Scout deployment. Release acceptance is recorded separately. Rerun the installer to update existing 0.1.8 installations; nothing backfills historical events. This contract supersedes the no-retry and process-only setup-identity descriptions below for lifecycle-enabled builds. Homepage/docs deployment remains separate.
@@ -111,6 +199,10 @@ Hosted Scout uses Clerk authentication for **Google and GitHub only**, without C
 
 The page and index status remain public. **Sign in** opens Clerk's account UI; submitting a task while signed out opens that same flow without invoking matching. A prompt draft is temporarily kept in this tab's `sessionStorage` to survive the OAuth redirect, then removed on return or successful sign-in. Storage-disabled browsers can sign in but may lose their draft during the redirect. Signing out or changing accounts clears the visible prompt/results and cancels the browser's pending request; it does not guarantee cancellation of provider work already accepted by the server.
 
+**Sign-in correction — deployed, 28 October 2026:** optional analytics downloads no longer block the conversational page's sign-in or submit handlers. Failed authentication initialization now displays an on-page error without clearing the task or starting an agent turn. Auth configuration and client initialization have 15-second limits; a sign-in modal that fails to render its controls within 30 seconds closes with a reload instruction. Closing a fully rendered modal does not count as an authentication error. Live desktop/mobile checks confirmed that the real Google/GitHub controls open with analytics blocked and preserve the draft on dismissal; these checks do not establish completed OAuth or an authenticated production conversation.
+
+Submitting a task while signed out resumes that task automatically once an active session is available, including after a same-tab OAuth return. A separate temporary submission marker contains the submitted task in this tab's `sessionStorage` and is consumed before resuming, so session updates and later reloads do not replay it. Header **Sign in** only restores the draft; it does not submit it. Editing the draft or dismissing sign-in with its close button, backdrop or Escape cancels automatic submission. With storage disabled, same-page sign-in can resume in memory, but redirect return cannot restore the task or submission intent. These cases have simulated-session coverage, not proof of completed provider OAuth.
+
 The Worker verifies Clerk session tokens before catalogue access for both `/scout/api/recommend` and `/scout/mcp`, including MCP discovery. Requests must send `Authorization: Bearer <session token>`; a cookie alone is not enough. Signature, expiry, issuer and authorized-party validation use Clerk's SDK, and pending sessions cannot search. Missing/invalid credentials return HTTP 401; missing server configuration returns 503 rather than enabling anonymous access. Status and `/scout/api/auth` remain public; the latter exposes only the publishable key and frontend API hostname. The secret key never reaches the browser.
 
 For an authenticated development/API check, obtain a current token from `await window.Clerk.session.getToken()` in your own signed-in browser. Treat it as a credential, do not share it, and set it privately as `SCOUT_TOKEN`. Then send:
@@ -189,7 +281,7 @@ Similar tasks can still receive identical shortlists when one benchmark is their
 
 This change replaces the cost-first policy below. Quality screening previously compared a model only with models that had exactly the same set of accepted benchmarks. That split the catalogue into many two- or three-model groups, so the best of a small, weak group passed the screen and cost-first ordering promoted it.
 
-Scout now fits **one latent quality estimate across every connected evaluation** for the task. Each benchmark has its own difficulty offset, score scale and noise, so a near-zero benchmark and a widely spread one contribute on comparable terms. Correlated series in one benchmark family share that family's weight. Each model receives an estimate with a standard error; a benchmark run by only one model links nothing. Applicable calibration observations join as one further evaluation using effective quality. Models with no evidence connected to the main group receive no estimate and remain provisional; they are never compared numerically. The estimate is reported as `comparison.qualityEstimate` and `comparison.qualityStandardError`; `quality` is its relative percentile among the estimated models.
+Scout now fits **one latent quality estimate across every connected evaluation** for the task. Each benchmark has its own difficulty offset, score scale and noise, so a near-zero benchmark and a widely spread one contribute on comparable terms. Correlated series in one benchmark family share that family's weight. Copies of one evaluation from different publishers, such as GPQA Diamond reported by Epoch AI, Artificial Analysis and model developers, form one family and count as one evaluation; their scores stay separate series. Evidence from the task's own domain carries at least three quarters of the weight when present, and evidence that scores the task's own kind of work counts twice as much as other domain evidence. General proxies fill in the rest and connect models that lack domain results. Each model receives an estimate with a standard error; a benchmark run by only one model links nothing. Applicable calibration observations join as one further evaluation using effective quality. Models with no evidence connected to the main group receive no estimate and remain provisional; they are never compared numerically. The estimate is reported as `comparison.qualityEstimate` and `comparison.qualityStandardError`; `quality` is its relative percentile among the estimated models.
 
 A configuration qualifies when it is not credibly worse than the strongest estimate: within two points plus one combined standard error. Supported Pareto selection over quality, reference cost and reference time follows. A balanced search keeps the strongest, lowest-known-cost and fastest frontier points, labelled in `pick` as `quality`, `value` and `speed`, then orders by estimated quality. An explicit `quality`, `cost` or `latency` priority orders the frontier accordingly. At most two models per provider are selected while other qualifying models remain; the cap never leaves a supported place empty. `ranking.version` is `10`, and `ranking.frontier` reports `version: 4`, the priority, the number of estimated models and benchmarks, and the tolerance.
 
@@ -255,9 +347,9 @@ The collection changes below remain **unreleased**; alpha reads the existing sta
 
 Research requests cited lifecycle/expiry information, capability updates, deployment regions, fine-tuning scope and documented hardware requirements. Standard rates remain separate from promotional offers, subsidies and expiry conditions. Editorial claims supply leads rather than numerical benchmark observations. Provider-specific facts must retain their scope; OpenRouter expiry is stored on its route rather than on every deployment of a model. Unknown or unsupported facts remain unavailable. No new paid collection run has validated this expanded coverage yet.
 
-OpenAI receives the submitted request for decomposition and derived work descriptions for interpretation and embedding. Parallel receives only public research subjects. Private hosted analytics records each model role separately with the original prompt, role position and allowlisted supporting evidence; the existing sampled three-month retention applies. Do not put secrets or personal information into prompts. Keys stay server-side.
+OpenAI receives the submitted request for decomposition and derived work descriptions for interpretation and embedding. Private hosted analytics records each model role separately with the original prompt, role position and allowlisted supporting evidence; the existing sampled three-month retention applies. Do not put secrets or personal information into prompts. Keys stay server-side.
 
-For development, `npm run scout` serves the page on port 45833 and requires `OPENAI_API_KEY` for matching. It refreshes structured feeds locally each hour; it does not run the hosted Parallel/R2/D1/Vectorize publication pipeline. Local evidence retrieval uses lexical matches and prepared memberships without a Vectorize binding. Production-like validation needs an isolated Worker with the storage bindings and both `OPENAI_API_KEY` and `PARALLEL_API_KEY`. Never put their values in browser assets, Git or a chat message.
+For development, `npm run scout` serves the page on port 45833 and requires `OPENAI_API_KEY` for matching. It refreshes structured feeds locally each hour; it does not run the hosted R2/D1/Vectorize publication pipeline. Local evidence retrieval uses lexical matches and prepared memberships without a Vectorize binding. Production-like validation needs an isolated Worker with the storage bindings and `OPENAI_API_KEY`. Never put its value in browser assets, Git or a chat message.
 
 ## Evidence architecture — hosted beta, 16 October 2026
 
@@ -273,7 +365,7 @@ Scout separates semantic interpretation from exact metadata checks and numerical
 | Evidence retrieval | A query embedding retrieves benchmark methodology, supplemented by hourly capability memberships. Semantic similarity supplies evidence candidates, never a model's score. |
 | Selection | All eligible verified catalogue models are considered before selecting up to five distinct models. One quality estimate spans connected evaluations; Pareto comparisons require known quality, cost and latency. |
 
-Mandatory restrictions use alternatives of complete condition groups. Soft preferences affect ordering, not eligibility. Relative dates use a supplied current UTC reference; arbitrary numeric bounds are retained, including impossible requirements. An unsupported mandatory condition cannot silently disappear. A no-match result describes the available verified catalogue and inventory, not every model in existence.
+Mandatory restrictions use alternatives of complete condition groups. Soft preferences affect ordering, not eligibility. Relative dates use a supplied current UTC day; arbitrary numeric bounds are retained, including impossible requirements. An unsupported mandatory condition cannot silently disappear. A no-match result describes the available verified catalogue and inventory, not every model in existence.
 
 Quality remains a relative benchmark signal, not predicted accuracy on an arbitrary task. Correlated benchmark families are grouped. Incomplete or incomparable objectives remain provisional and cannot establish a global three-objective frontier. Narrow typed tasks can still use a task-fit shortlist with unknown quality. Comparable direct calibration observations supplement quality evidence without excluding other eligible models. Calibration episode charges and timing do not replace the common token-workload reference costs and endpoint-bound timing; unknown values remain unknown.
 
@@ -327,7 +419,7 @@ The hosted implementation keeps the existing task and format Decisions, exact me
 
 Hourly staging prepares content-addressed specialist packs alongside profile indexes. Searches select the same scoped profiles and use those packs instead of many repository reads. A version-checked derived pack index can also be prepared for an existing immutable catalogue without repeating research. Existing generations remain readable through the original shard path when no matching pack index is available. Source research and the hourly trigger are unchanged.
 
-Successful searches may be reused within a Worker isolate for up to 30 seconds, bounded to 32 entries and approximately 4 MiB. Exact prompt text, model/endpoint inventories, publication, interpretation version and UTC calendar date identify a cache entry. Cache hits preserve the original `evaluatedAt` time; date conditions remain evaluated as of that time for at most 30 seconds, and results are not reused across UTC midnight. Identical concurrent requests share computation. Errors and results missing a specialist or applicability check are not cached. This temporary memory cache adds no external prompt storage and is not semantic similarity caching.
+Successful searches may be reused within a Worker isolate for up to 30 seconds, bounded to 32 entries and approximately 4 MiB. Exact prompt text, model/endpoint inventories, publication, interpretation version and UTC calendar date identify a cache entry. Cache hits preserve the original `evaluatedAt` time; date conditions remain evaluated as of that time for at most 30 seconds, and results are not reused across UTC midnight. Identical concurrent requests share computation. Errors and results missing a specialist or applicability check are not cached. This temporary memory cache is not semantic similarity caching. Since 26 October 2026, a separate seven-day answer cache in D1 replays the AI answers of an identical completed search; see the repeatable-searches section above.
 
 Private search records now include fixed stage durations, pending stages at a deadline, cache status and allowlisted provider status/error codes. They do not store raw errors, stack traces, headers or credentials. API results include `runtime` timings/cache status; cache hits report their current serving duration and zero new interpretation time. The page remains unchanged. A six-second matching deadline remains a reliability boundary; these changes do not establish a latency guarantee. The full repository release checks passed, including 318 Scout tests. An initial candidate timed out on books/posters; that failure was retained, and the interpretation was then reduced further. The revised semantic suite passed 17 of 17 cases, including cold repetitions and exact-cache reuse, and four final cold checks passed after the independent database reads were parallelized. These small samples do not establish a production percentile or a latency guarantee. Post-deployment HTTP/MCP checks passed five of five on their first attempts: supplied-label vision, creator restriction, impossible context, JSON invoice extraction through MCP and an exact repeat. Observed end-to-end durations were approximately 2.0–4.3 seconds for new searches and 146 ms for the repeated prompt served from the same isolate cache. The 127 UI source files and five live scripts/styles were unchanged. These are individual observations, not percentile guarantees.
 
@@ -398,7 +490,7 @@ Compatibility is eligibility, not predicted accuracy. Scout still enforces endpo
 
 Snapshot data version 7 prepares benchmark capabilities and correlated benchmark families with Decisions during the hourly refresh. It replaces benchmark-name regexes, manually assigned capability tags and the automatic reasoning fallback. Source adapters retain metric identifiers, score units and parsing contracts. Epoch descriptions and methodology are read from its published catalogue; Artificial Analysis and BFCL methodology are read from their public pages. Unavailable definitions remain explicit limitations, and names alone do not establish an opaque benchmark's capabilities.
 
-Each changed benchmark receives an isolated interpretation request so another benchmark's description cannot supply its capabilities. For shared publisher methodology, a preceding Decisions request selects the directly applicable definition for that benchmark and evaluated subset; unrelated sections are omitted from capability interpretation. Publisher-declared suite identities are retained as source facts; other family relationships are interpreted by Decisions. Capability choices and family references are validated, and positive choices below the existing 0.4 confidence threshold are not used. Family links group related evidence; numerical comparisons still require the exact same benchmark series. Unknown scopes receive no capability tags and cannot contribute capability-ranking evidence. This is semantic metadata interpretation, not a measurement of model quality or a guarantee that the classifications are correct.
+Each changed benchmark receives an isolated interpretation request so another benchmark's description cannot supply its capabilities. For shared publisher methodology, a preceding Decisions request selects the directly applicable definition for that benchmark and evaluated subset; unrelated sections are omitted from capability interpretation. A section titled with the benchmark's own name, such as "GPQA Diamond" on Artificial Analysis's methodology page, is always kept and listed first, ahead of shared sections such as the Intelligence Index overview. The publisher's methodology is used even when an earlier research description exists. Publisher-declared suite identities are retained as source facts; other family relationships are interpreted by Decisions. Capability choices and family references are validated, and positive choices below the existing 0.4 confidence threshold are not used. Family links group related evidence; numerical comparisons still require the exact same benchmark series. Unknown scopes receive no capability tags and cannot contribute capability-ranking evidence. This is semantic metadata interpretation, not a measurement of model quality or a guarantee that the classifications are correct.
 
 Interpretations are cached by definition, relevant publisher methodology, family inventory and interpreter version. Score, model, price and collection-time changes alone do not invalidate that cache. Family inventory changes trigger rechecking its references. If an interpretation fails, unchanged cached records remain usable, affected new or changed series stay unclassified, and source failure status is exposed until a later refresh succeeds. A methodology download failure retains the prior methodology when available and marks its source degraded. There is no keyword fallback.
 
@@ -408,11 +500,21 @@ The initial deployed snapshot processed all 146 benchmark series: 136 have suppo
 
 ## Use Scout
 
+**Source only, not deployed:** results use **Model Choices** in their count and no longer display rank numbers, the Rank column or the “Read left to right” instruction. The Assumptions & limitations disclosure, including its supplementary specialist list, is replaced with “Results could be wrong. Please verify before choosing a model.” Model evidence remains expandable; ranking, API evidence and the methodology documented here are unchanged. Feedback's reason selector and action buttons share a 40px height.
+
+**Source only, not deployed:** the pending search state places rotating search messages in the results-title position, preceded by a tiny pulsing orb. It does not repeat the submitted prompt or show a separate loading heading. Reduced-motion preferences disable both the orb pulse and the 180ms opacity/3px message transition.
+
+Loading messages describe search perspectives, not reported backend stages or completion percentages. Live timing starts at submission, including authentication and network wait, and freezes as **Took …s** when the response arrives. A cached response may legitimately be much faster. The existing 35-second request timeout still starts after authentication; replacing a search, signing out, completing or failing stops its loading updates.
+
+**Source only, not deployed:** focusing the task field changes its background without drawing an outline around the input container. Keyboard focus outlines on buttons and links are unchanged.
+
+**Source only, not deployed:** example prompts appear as rounded pills numbered 1, 2 and 3 beside **Try**, inside the bottom-left of the task field. They disappear as soon as the field contains any text, including whitespace, and return when it is cleared. Selecting a pill fills an editable draft without submitting it. The visible input-helper sentence is removed; Shift + Enter still adds a line. The Claude Code logo uses its existing transparent image without a white background in either theme.
+
 Open [Scout](https://openmerit.ai/scout/), type a task, and select the arrow button (**Find models**) or press Enter. For example: “I want to upload an invoice and get the invoice number, date, vendor, subtotal, tax, and total as JSON.” The task field grows and shrinks as you type or paste. Very long prompts scroll within the field once it reaches its height limit. Shift + Enter adds a line. When the field is empty and unfocused, its placeholder cycles through detailed example tasks every ten seconds with a brief fade. Rotation pauses while you type, while the field is focused, or while the page is hidden. With reduced motion enabled, the example stays still. Placeholders never fill the input or send a request. The heading shows live counts of API catalogue models, Hugging Face repositories, benchmark series and sources. API identities and Hub repositories are separate counts and can overlap; neither count promises a runnable endpoint for every entry. The evaluation guide remains available in the fixed bottom footer, with space reserved beneath the page content.
 
 The comparison table keeps recommendations in Scout rank order. Benchmark mode shows a relative quality signal and reference cost/time estimates. Applicable calibration contributes quality evidence within its recorded scope; its episode charges and timing do not replace the common reference workload. **Candidates to evaluate** denotes task-fit mode: quality is shown as unknown, alongside known reference costs and deployment requirements. When image separation is ambiguous, Scout first offers labels, bounding boxes or pixel masks/cutouts as output choices. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; Hugging Face results resolve the repository publisher, using a known lab mark or its bundled organization avatar; otherwise a Hugging Face mark identifies the repository platform. Missing or failed marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **JSON API**, **Capabilities**, and **Sources** expand beneath the search and results. **Sources** uses a compact list; expand a source for its coverage, methods, limits, collection times and refresh failures.
 
-The capability index lists each area with its benchmark-series and model counts. After a search, task matches appear first with a check mark. Counts describe index coverage, not the number of models eligible for your task. The source panel describes what each feed measures, its coverage, source dates, archived results, and limitations. Collection time is separate from source dates, which can be publication dates rather than measurement dates. Scout task calibration appears separately with its tested pool, scope and seven-day expiry.
+**Source only, not deployed:** the Scout page no longer exposes the JSON API setup, capability index or source catalogue disclosures, including the header API link. Privacy & analytics has its own [settings page](/scout/privacy/), linked from Scout's footer alongside the data policy. The task-logging notice and sharing controls live on that page; existing choices are preserved, and the former settings fragment forwards there. This is a presentation change, not API access control: the JSON API, status endpoint and MCP remain available under their existing contracts, and recommendation rows retain their evidence links and limitations.
 
 **Provider page** in each model's evidence opens its provider's official pricing, model documentation, or website. Model links never fall back to a router. Each quote identifies its source: a router route, independent provider metadata, or a reviewed provider contract. Routes have separate prices. Self-hosted infrastructure cost and incomplete quotes remain unknown. Hugging Face deployment links open the publisher’s model repository and pinned card. Benchmark links lead to the original evaluation sources.
 
@@ -479,6 +581,34 @@ Claude can load the `openmerit-scout` skill when choosing an application inferen
 
 The agent compares model and endpoint, evidence, reference cost/time and unknowns, and helps configure the chosen route. New providers, credentials, spending and deployments need separate approval. A skill is a default, not enforcement. The MCP tool, `scout_models`, is read-only and does not execute models.
 
+### Automatic instruction updates — version 0.1.10
+
+Version 0.1.10 adds default-on updates for the installed Scout skill and startup rule. **npm 0.1.9 does not contain the updater.** After 0.1.10 is published, existing installations need one explicit installer rerun in their original scope to acquire it. A website or Scout deployment cannot bootstrap files on someone else's machine. Compatible hosted Scout changes already reach agents through their next MCP request.
+
+Version 0.1.11 also recognizes updater entry points reached through symbolic links. Existing installations need an installer rerun to acquire this updater-code fix; automatic instruction updates do not replace the installed updater itself.
+
+Setup discloses automatic updates and installs asynchronous `SessionStart` and `Stop` command hooks. They check npm's stable `latest` release at most once per 24 hours per installation. Offline or failed checks retain the installed version and retry at the next eligible hook after 15 minutes. `Stop` means Claude finished responding, not that the user's task succeeded. No background service watches connectivity while Claude Code is closed, and checks never ask Claude to keep working. A one-shot `claude -p` process can exit before an asynchronous Stop hook finishes; startup checks and later sessions provide the retry path. Managed policies or disabled hooks can prevent automatic checks; an explicit manual update remains available.
+
+Completed failed checks emit a non-blocking Claude Code notice: the installed version was retained, automatic checks retry after 15 minutes during use, and the user can say **“update openmerit”** to retry manually. Hooks during that backoff stay silent; another eligible failed check emits the notice again. Current-version checks and disabled automatic checks stay silent. Delivery depends on Claude Code processing the completed asynchronous hook output; an exiting one-shot process may not show the notice.
+
+The updater lives beside the install record, independent of the npx cache. User scope adds hooks to Claude's `settings.json`; project scope uses `.claude/settings.local.json`. Existing settings and other hooks are preserved. It downloads only from the public npm registry, verifies the package's SHA-512 integrity against registry metadata, and reads a fixed instruction JSON bundle in memory. It does not install the downloaded package or execute its scripts. npm sees the normal network request, but receives no device key, installation ID, task, project path or transcript. Registry integrity protects against mismatched bytes, not a compromised publisher or registry.
+
+Only bundles declaring the supported instruction/permission contract update automatically. Releases that change provider permissions, authentication, analytics, hook code or MCP configuration need a reviewed installer upgrade; the updater leaves those components alone. The contract is a release-maintainer assertion, not an automated security review of prose. Maintainers must change the contract when those boundaries change. Updates preserve credentials, project policy and user-edited instructions, hooks and updater files. Writes are serialized and journaled so an interrupted instruction update can recover the previous complete pair; intervening user edits stop recovery instead of being overwritten.
+
+Start a **new Claude Code session** after an update to reliably load the new instructions. A running session may retain earlier instructions; reconnecting `/mcp` does not reload the startup rule. The installed rule tells Claude how to handle “update openmerit”; this natural-language trigger is guidance, not an enforced command.
+
+Commands in version 0.1.10 and later:
+
+```sh
+openmerit update                         # check immediately, even if auto updates are off
+openmerit update --auto off              # persistently disable automatic updates
+openmerit update --auto on               # enable them again
+openmerit update --scope project --json  # update this project's installation
+openmerit install --client claude-code --auto-update off  # opt out during setup
+```
+
+Use the same scope as installation. `OPENMERIT_AUTO_UPDATE=off` suppresses automatic checks in that process without disabling an explicit manual update. Installer reruns preserve a saved opt-out. `status` includes the automatic-update preference, hook readiness and instruction version; `remove` removes unchanged owned hooks/updater files and retains personal settings and edited files. Manual updates report conflicts, retryable failures or a required installer upgrade instead of claiming success. If lock recovery itself was interrupted, inspect the PID in `openmerit-update.lock.recovery` beside the install record and confirm that process has stopped before removing that recovery lock and retrying.
+
 ### Startup model-decision rule — npm 0.1.8
 
 The 0.1.8 installer adds `rules/openmerit-scout.md` alongside the skill. Unlike an on-demand skill, an unscoped [Claude Code rule](https://code.claude.com/docs/en/memory#organize-rules-with-claude/rules/) loads at session startup. The rule requires loading the skill and consulting Scout before any application-model choice, even when the user only asks for a feature. It explicitly includes pretrained/local/browser models, weight downloads, SDK defaults, embeddings, OCR, speech, classification, detection and segmentation. An explicit user model choice must be assessed, not silently replaced. Unchanged approved models and ordinary inference calls do not require another search.
@@ -525,33 +655,55 @@ Scout recommends models; it does not process an uploaded invoice, execute a task
 
 ## Real sources and their limits
 
-The hourly updater fetches eight independent public sources, including Models.dev and fourteen bounded Hugging Face pipeline queries. Two additional versioned provider-contract sources have explicit manual review dates. It prepares the index before requests arrive. Queries use batched Decisions questions for intent classification, a bounded verification pass for weak answers, and relevant specialist compatibility checks; they do not browse benchmark sites or ask a model to invent rankings.
+The twice-daily updater fetches eight independent public sources, including Models.dev and fourteen bounded Hugging Face pipeline queries, and adds hosted routes and models from OpenMerit's merged model catalogue. Two additional versioned provider-contract sources have explicit manual review dates. It prepares the index before requests arrive. Queries use batched Decisions questions for intent classification, a bounded verification pass for weak answers, and relevant specialist compatibility checks; they do not browse benchmark sites or ask a model to invent rankings.
 
 | Feed | Used for |
 | --- | --- |
 | [Epoch AI](https://epoch.ai/benchmarks) | Benchmark definitions, published results, source links, and evaluation configurations. |
-| [Artificial Analysis](https://artificialanalysis.ai/leaderboards/models) | Benchmark results, measured first-answer delay and reference generation speed. Archived evaluations remain labelled when their models are still available. |
+| [Artificial Analysis](https://artificialanalysis.ai/leaderboards/models) | Benchmark results for 32 evaluations (including HealthBench, HealthBench Professional, Humanity’s Last Exam Diamond, CritPt v2 Hard and FrontierCode 2.0, each its own series beside its earlier version), measured first-answer delay and reference generation speed. Elo-style ratings (GDPval-AA, AA-3D and Code2Video-AA) become the expected win rate against the median-rated model, as for Arena; AA-Briefcase is not read. Archived evaluations remain labelled when their models are still available. |
 | [OmniDocBench](https://github.com/opendatalab/OmniDocBench#end-to-end-evaluation) | Document parsing scores from the explicitly labelled `v1.6_full` leaderboard, including newer document models than the OpenVLM snapshot. |
 | [IDP Leaderboard](https://www.idp-leaderboard.org/) | Nanonets' MIT-licensed document processing leaderboard: OlmOCR, OmniDocBench v1.5 and IDP Core (key information extraction, handwriting, tables, visual QA) component scores. The overall average is not imported; per-model dates are not supplied. |
 | [OpenCompass / VLMEvalKit](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) | Visual understanding and OCR benchmark results. Its feed currently identifies itself as 17 September 2025. |
 | [Berkeley Function Calling Leaderboard (BFCL v4)](https://gorilla.cs.berkeley.edu/leaderboard) | Native function-calling single-turn, live, and multi-turn scores; separate memory scores for key-value, vector, and recursive-summarization stores. |
+| [Arena leaderboard dataset](https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset) | Style-controlled human-preference ratings (CC BY 4.0) from four boards: text request categories (including medicine and healthcare, legal and government, the sciences, and entertainment, sports and media), document prompts, vision (overall, OCR, diagrams, homework and creative writing about images) and Code Arena WebDev (overall, HTML, React and image-to-WebDev). Each becomes the expected win rate against the median-rated model of its category; ratings with confidence intervals wider than 60 points are left out. Each category keeps its own publication date. Human preference, not verified correctness. A separate text **factuality** board (overall, medicine and healthcare, legal and government, business and finance) blends those votes with Arena's automated web verification of each response's factual claims at 25% weight; it is converted the same way and covers web-verifiable claims only. Search and Agent Arena are not read: they rate models together with a search system or agent harness. |
+| [Vals AI](https://www.vals.ai/benchmarks) | Overall accuracy on 27 benchmarks Vals AI runs, many on private test sets: healthcare (MedQA, MedCode, MedScribe), legal (LegalBench, Legal Research Bench, HLAB), finance (CorpFin, TaxEval, MortgageTax, Finance Agent, Excel modelling, ClaimBench), public benefits, education, multilingual math, science, code migration, programming and security. Composite indexes, archived versions and evaluations other feeds already supply are not read. Vals AI publishes no data licence; Scout keeps only overall scores with links to each page. Per-model evaluation dates are not supplied. Agent products such as a model inside Claude Code or Codex are not matched to the model. |
+| [Scale Labs (SEAL)](https://labs.scale.com/leaderboard) | Published scores from 11 current boards Scale runs, mostly on private sets: Professional Reasoning (legal and finance), MCP Atlas, MultiChallenge, the Remote Labor Index, SWE Atlas (refactoring, test writing and codebase questions), HiL-Bench, DrugDiscoveryBench and Humanity's Sixth Sense (visual reasoning). Each result keeps the date Scale added it. Rows Scale flags as possibly contaminated or deprecated are left out, as are deprecated boards, safety boards, audio, Humanity's Last Exam Diamond (read from Artificial Analysis) and SWE-Bench Pro V2. Agent boards rate a model inside a named harness, such as Codex or Claude Code; the harness is kept as the result's configuration. Scale names configurations inconsistently, so some rows stay unmatched. |
+| [LiveBench](https://livebench.ai) | LiveBench 2.0 category scores (long agentic, reasoning, coding, browser, data analysis, research and design) from the newest release's published table, about 30 frontier models per release. No per-model dates; the overall score is not read. |
+| [τ-bench by Sierra](https://taubench.com) | pass^1 on τ²-bench airline, retail and telecom and τ-bench banking, from Sierra's own text runs on harness 1.x. Developer-submitted and custom-agent entries are left out, and banking keeps only the AllTools retrieval setup. Few models per domain. |
+| [OpenCompass](https://opencompass.org.cn) | Text-only knowledge, abstraction, reasoning, interaction and agent ability averages for about 20 models. The overall score and the safety board are not read. |
+| [HELM Arabic](https://crfm.stanford.edu/helm/) (Stanford CRFM) | Mean score across HELM's Arabic scenarios from the newest release, mostly multiple choice. Models missing a scenario are left out; no per-model dates. |
+| [BRIDGE Medical](https://huggingface.co/spaces/YLab-Open/BRIDGE-Medical-Leaderboard) (YLab) | Zero-shot average across clinical-text tasks in several languages, mostly open-weight models. Average only; no per-model dates. |
+| [SuperCLUE](https://www.superclueai.com) | Chinese general-board category scores: math and science reasoning, hallucination control, precise instruction following, agentic coding and planning, for about 20 models. Boards older than six months are not read. |
+| [LLM Stats](https://llm-stats.com/benchmarks) (vendor-reported) | Scores model developers report in announcements, papers and model cards, compiled by LLM Stats and reused with attribution under its terms. Scout reads LLM Stats' 100 most-reported benchmarks and keeps only results it marks self-reported, on 0–1 or 0–100 scales, as separate `· vendor-reported` series that never merge with third-party measurements of the same benchmark. Vendor-reported results count toward ranking. They are not independently verified, and settings such as tools, attempts and reasoning effort follow each vendor. Base (pretrained) checkpoints are never matched to instruct models. A benchmark that leaves the 100 keeps its last results. |
 | [OpenRouter](https://openrouter.ai/api/v1/models) | Available model IDs, input modalities, declared tools and structured-output support, context limits, and advertised token prices. |
 | [Models.dev](https://models.dev) | Independent native model IDs, declared capabilities and prices via ten direct-provider adapters: OpenAI, Anthropic, Google, DeepSeek, Mistral, xAI, Cohere, Moonshot, MiniMax and Z.AI. Not a performance benchmark or account-availability check. |
+| OpenMerit model catalogue | Hosted routes and advertised prices, merged field by field from [Models.dev](https://models.dev), [LiteLLM](https://github.com/BerriAI/litellm), OpenRouter's complete output-modality feed, [Vercel AI Gateway](https://vercel.com/ai-gateway/models), [DeepInfra's own model API](https://deepinfra.com/models), [genai-prices](https://github.com/pydantic/genai-prices) and the [TrueFoundry models registry](https://github.com/truefoundry/models) (MIT). [Eden AI's expert-model list](https://www.edenai.co/docs/v3/expert-models/listing-models) adds the gateway's own OCR and document-parsing, translation, text-to-speech, transcription, moderation and AI-content detection routes, including vendors no other source lists, such as Mindee, Veryfi, Affinda, DeepL, ModernMT, Gladia and Sightengine. Its prices are per character (per million), page, document file, image, request or second of audio or video, exclude Eden AI's platform fee, and come with no stated data licence. Eden's token-priced routes that do not say input or output, free quotes and its LLM catalogue are left out; only speech and OCR models with an existing catalogue ID link to it. TrueFoundry adds per-provider configs for Bedrock, Azure and Microsoft Foundry, Vertex, Gemini, OpenAI, Mistral, xAI, Cohere, Databricks, Deepgram, ElevenLabs and others, including speech, transcription, image, video, rerank, OCR, moderation and decision routes, with batch, priority and flex rates and regional prices where they differ from the base region; retired routes and context-tier surcharges are left out. OpenRouter contributes chat, decision, embedding, rerank, image, video, speech and transcription routes. DeepInfra contributes its chat, decision, embedding, rerank, image, video, speech, music and transcription inventory. Token, character, image, frame and input/output-time rates keep their provider-defined units; conditional overrides and unsupported multipliers remain in the archived source payload instead of being flattened into misleading base prices. Novita is not ingested in this revision: its public OpenAI model list mostly overlaps routes already present and does not supply source-specific pricing or modalities that this importer can normalize. A route is attached only through an exact model identity. Models with such routes that no other source lists become catalogue models, so benchmark results naming them are kept; a catalogue model whose name another listed model already uses is left out. Benchmark rows are also matched through the catalogue's other names for a listed model (host model IDs, listing names and Hugging Face repositories), and after removing run settings such as reasoning effort, thinking budgets and trailing parentheticals. A name shared by distinct catalogue models matches nothing, and a row under Scout's own name for a model outranks loosely matched rows for the same benchmark. Listings are metadata, not verified account access or a performance test. |
 | [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions) | Reviewed typed-choice, predicate and score endpoint contract, inputs and endpoint-specific rates. No general chat timing is inherited. |
 | [TypeSafe Jev](https://typesafe.ai/blog/jev-1-14) | Reviewed System One contract. Image and single-box output are experimental; headers, version and incomplete total pricing are disclosed. Jev is a candidate for compatible tasks, not Scout’s ranking engine. |
+| [Vercel AI Gateway decision routes](https://vercel.com/docs/ai-gateway/modalities/decision) | Decision models Vercel AI Gateway routes (Laya, Liquid d1, Perplexity Deciders, Jev, Microsoft Decision-1, Baseten Elephant and Mammoth and others), read from the OpenMerit catalogue each refresh with their input-token rates, context and image input. One reviewed contract covers them: `POST https://ai-gateway.vercel.sh/v1/evaluate` with supplied choices, ordered rubric scores and boolean probabilities. They join classification and judging shortlists as task-fit candidates; promotional and free-tier rates are shown as listed and can end. A free-tier ID with the same name as a paid one is listed once. Many decision rates are identical, so between routes with the same reference cost, the model's standing on [JevBench](https://benchmarkheaven.com/jev-models) (capability: intelligence and calibration, ranked systems only) and the [Artificial Analysis Decision Index](https://artificialanalysis.ai/evaluations/artificial-analysis-decision-index) breaks the tie: the share of each board's other entries it beats, averaged over the boards it appears on. Models without results come after those with them at that price; a cheaper route always comes first. Results link by exact model name. These boards measure general typed decisions, not accuracy on your task; the reason and `decisionBenchmarks` show each result. |
 | [Hugging Face](https://huggingface.co/models) | Public repository metadata, model-card results and structured evaluation records for specialist discovery. Separate from performance ranking. |
 
-Over 100 benchmark **series** are indexed, including source-specific implementations and versions. This is not 100 independent publishers, and coverage is not uniformly current. Some definitions have no results matched to an available model. Both counts are exposed. Missing metrics and ambiguous or unmatched model versions are omitted. Model discovery merges OpenRouter with independent native-provider records before matching public benchmark results. An OpenRouter outage does not prevent an independent catalogue from bootstrapping. Distinct endpoints retain prices and parameters; duplicate hosting entries do not create additional model votes or consume multiple places in the five. Endpoint listings are metadata, not verified access in the caller’s account.
+Over 100 benchmark **series** are indexed, including source-specific implementations and versions. This is not 100 independent publishers, and coverage is not uniformly current. Some definitions have no results matched to an available model. Both counts are exposed. Missing metrics and ambiguous or unmatched model versions are omitted. Model discovery merges OpenRouter with independent native-provider records and catalogue host routes before matching public benchmark results. An OpenRouter outage does not prevent an independent catalogue from bootstrapping. Distinct endpoints retain prices and parameters; duplicate hosting entries do not create additional model votes or consume multiple places in the five. Endpoint listings are metadata, not verified access in the caller’s account.
 
-Collection time is not measurement time. A successful hourly check does not make an old result new. Some feeds omit measurement dates; those remain unknown. Model release dates are never substituted. OmniDocBench's repository version and leaderboard version differ; Scout uses the version actually printed on the score table. Document parsing is a proxy for invoice extraction, not an evaluation of your exact six fields. New models without comparable evidence may be absent from a task's shortlist.
+Collection time is not measurement time. A successful refresh does not make an old result new. Some feeds omit measurement dates; those remain unknown. Model release dates are never substituted. OmniDocBench's repository version and leaderboard version differ; Scout uses the version actually printed on the score table. Document parsing is a proxy for invoice extraction, not an evaluation of your exact six fields. New models without comparable evidence may be absent from a task's shortlist.
 
 Epoch AI attribution: “Capabilities & benchmarking,” published at epoch.ai, normalized and matched by Scout under the source's [CC BY 4.0 terms](https://creativecommons.org/licenses/by/4.0/). Other sources are attributed in the source panel and individual evidence links. These are data-source attributions, not a statement about OpenMerit's licensing.
 
+### Pending embedding benchmark and media-price collection
+
+**Not yet deployed:** the publisher source adds [MTEB's final leaderboard summaries](https://huggingface.co/spaces/mteb/leaderboard), including historical/off-menu suites and text, image, audio and video embedding evaluations. It collects only published aggregate and per-task scores plus their model, experiment, suite and training-disclosure metadata—not benchmark datasets, embeddings, predictions or per-example outputs. Different suite versions and experiment settings remain separate; missing aggregates remain unknown. Results are source-reported, not independently verified by Scout. The results dataset declares CC0-1.0; that does not change the individual task datasets' licences.
+
+This is catalogue ingestion only. MTEB results do not yet enter the published chat benchmark index or recommendations, and embedding/rerank workload pricing and recommendation paths remain unfinished. The summaries do not supply evaluated checkpoint revisions or measurement dates; collection and release dates do not substitute for either. An unavailable or malformed suite retains its previous complete results and original collection timestamp.
+
+Also **not yet deployed:** three supplemental [Portkey pricing feeds](https://github.com/Portkey-AI/models) contribute reviewed Deepgram Aura, ElevenLabs speech and Stability AI image quotes. Speech generation rates are per input character, Scribe transcription rates per input-audio second, and image rates per image with any published quality/size tier. Portkey's token-like field names do not establish token billing. Unknown denominators, service aliases, custom calculations and zero-only media placeholders are excluded rather than advertised as free generation. Each response is archived with its source attribution; listed prices are not an account-specific quote or proof of availability. These supplemental prices do not add a media recommendation path.
+
+The pending publisher also discovers Hugging Face inference-provider routes across all tasks, including repositories from personal accounts. This is independent of the admitted-organisation crawl and does not admit an author's entire collection. Bounded pages and repository batches resume from durable cursors; incomplete collection never withdraws existing routes, and source-wide removals wait for completed application. Application batches can become visible incrementally, while the source retains its previous successful-check timestamp until completion. Only exact/reviewed identities link to canonical models, non-token task prices remain unknown, and provider listings do not establish working credentials or task accuracy. This does not widen the executable recommendation contracts.
+
 ## Specialist models and open weights
 
 Some specialist models have useful evaluations outside general LLM leaderboards. Scout reads both legacy model-card `model-index` results and the Hub’s [structured evaluation records](https://huggingface.co/docs/hub/eval-results). It retains the original dataset, task or metric, units, configuration, split, reported date, setup notes and source link when supplied. Scores are not averaged across unrelated datasets. A repository revision pins the metadata collected; it does not establish which checkpoint produced every reported score. Reported dates can also default to a commit date. Scout does not validate verification tokens or independently reproduce these scores.
 
-The hourly discovery sample takes up to 30 repositories sorted by downloads for each of fourteen pipelines: document question answering, token classification, text classification, sentence similarity, text-to-text generation, text generation, image-to-text, image-text-to-text, image classification, zero-shot image classification, object detection, zero-shot object detection, image segmentation and mask generation. The two image-to-text queries remain OCR-filtered. Bounded pinned configs add label vocabularies for up to six repositories in each fixed-label vision pipeline. Private/disabled repositories, adapters and quantization packages are excluded; fine-tunes remain separate. Downloads select coverage only, never quality. This bounded sample can miss capable low-download specialists.
+Each refresh's discovery sample takes up to 30 repositories sorted by downloads for each of fourteen pipelines: document question answering, token classification, text classification, sentence similarity, text-to-text generation, text generation, image-to-text, image-text-to-text, image classification, zero-shot image classification, object detection, zero-shot object detection, image segmentation and mask generation. The two image-to-text queries remain OCR-filtered. Bounded pinned configs add label vocabularies for up to six repositories in each fixed-label vision pipeline. Private/disabled repositories, adapters and quantization packages are excluded; fine-tunes remain separate. Downloads select coverage only, never quality. This bounded sample can miss capable low-download specialists.
 
 For document, code and retrieval components, **Specialist models** expands beneath the five recommendations when compatible prepared profiles match. For classification, detection and segmentation, compatible native pipelines may enter the five **Candidates to evaluate** instead. Fixed-label pipelines need complete semantic coverage of the requested categories; one overlapping label is insufficient. Zero-shot pipelines accept supplied labels; masks can require points or boxes. Pipeline metadata is not an accuracy guarantee. Up to three components are ordered by metadata completeness and component diversity after compatibility checks. Domain and language requirements come from Decisions, with no prompt keyword dictionary. Tool execution and persistent-memory tasks do not receive unvalidated specialist matches. Entity taggers, OCR and embedding models are components requiring integration, not interchangeable chat APIs. Repeated repository names are de-emphasized for display diversity without merging their identities. See the release status in [semantic specialist matching](#semantic-specialist-matching-hosted-beta-15-october-2026).
 
@@ -563,15 +715,15 @@ Missing evaluations mean **unknown**, not poor performance. Task-fit candidates
 
 [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions), using `gpt-6-luna`, interprets the work, capability requirements, operation, answer space, serialization and tradeoff priority. It distinguishes writing source from executing it, supplied categories from open identification, text-bearing objects from reading their contents, and session state from persistent memory. Typed answers are validated and checked for consistency. Uncertain answers remain disclosed; they cannot authorize narrow calibration evidence. No prompt keyword rules or model-name bonuses enter selection.
 
-A Structured Outputs call first extracts downstream work, explicit model/endpoint restrictions and preferences using the published metadata definitions and canonical vocabulary. Mandatory conditions use alternatives of complete AND groups; exclusions and arbitrary numeric/date bounds are preserved. Relative dates use a supplied current UTC reference. Generic typed comparisons enforce those conditions and intersect optional `modelIds`/`endpointIds` inventories. Missing facts cannot satisfy a restriction, including an exclusion. Unexpressible mandatory restrictions produce no verified matches rather than silently broadening the search. Names mentioned as task data, clients or comparison subjects are not automatic filters. Interpretation is probabilistic and can still misread a prompt.
+A Structured Outputs call first extracts downstream work, explicit model/endpoint restrictions and preferences using the published metadata definitions and canonical vocabulary. Mandatory conditions use alternatives of complete AND groups; exclusions and arbitrary numeric/date bounds are preserved. Relative dates use a supplied current UTC day. Generic typed comparisons enforce those conditions and intersect optional `modelIds`/`endpointIds` inventories. Missing facts cannot satisfy a restriction, including an exclusion. Unexpressible mandatory restrictions produce no verified matches rather than silently broadening the search. Names mentioned as task data, clients or comparison subjects are not automatic filters. Interpretation is probabilistic and can still misread a prompt.
 
 Image tasks require declared image input and relevant visual evidence in performance ranking. Document reading needs document evidence. JSON/schema tasks need declared `response_format` or `structured_outputs`; required tool execution needs tools support and relevant tool evidence. Persistent memory requires external-memory evidence rather than a large context window. [BFCL memory evaluations](https://gorilla.cs.berkeley.edu/blogs/16_bfcl_v4_memory.html) do not establish built-in memory or performance on the visitor's storage system. Recognized image/video/audio generation and transcription remain outside the supported output index. Ambiguous spatial output produces the existing clarification choices.
 
-Classification and judgment tasks first use applicable benchmark evidence. With no supported benchmark candidates, supplied-label classification and judgments retain the explicitly labelled **task-fit** fallback. Boxes and masks use their compatible task-fit path. Documented operation/output fit, deployment readiness and known reference costs/times determine unmeasured shortlists; quality stays unknown. Open or uncertain identification instead needs name-generating endpoints and relevant evidence; fixed-choice APIs cannot satisfy it. Specialist compatibility uses hourly-prepared public profiles. Unknown compatibility omits a repository with a warning, rather than assuming task accuracy.
+Classification and judgment tasks first use applicable benchmark evidence. With no supported benchmark candidates, supplied-label classification and judgments retain the explicitly labelled **task-fit** fallback. Boxes and masks use their compatible task-fit path. Documented operation/output fit, deployment readiness and known reference costs/times determine unmeasured shortlists; quality on the task stays unknown. Between decision models with the same reference cost, their standing on two general decision leaderboards breaks the tie, as described in the Vercel decision routes row of the source table. Open or uncertain identification instead needs name-generating endpoints and relevant evidence; fixed-choice APIs cannot satisfy it. Specialist compatibility uses hourly-prepared public profiles. Unknown compatibility omits a repository with a warning, rather than assuming task accuracy.
 
-For performance ranking, a query embedding retrieves benchmark methodologies, supplemented by their hourly capability memberships. Bounded independent Decisions predicates verify relevance and a deliberately scored required ability before a methodology contributes proxy evidence. Similarity never supplies a quality score. Rejected, unknown and unavailable evidence is withheld. If some checks fail, verified evidence can still be used with an explicit warning; if no applicability can be verified because checks are unavailable, the search returns unavailable rather than reporting a false no-match.
+For performance ranking, a query embedding retrieves benchmark methodologies, supplemented by capability memberships prepared at each refresh. Bounded independent Decisions predicates verify relevance and a deliberately scored required ability before a methodology contributes proxy evidence. A second Decisions question compares each methodology with the task's description: *direct* (the same kind of work in the same domain, such as MedCode for ICD-10 coding), *domain* (the task's subject domain or profession, but a different form of work), *generic* (a general ability the task uses) or *unrelated*. When direct and domain together hold at least 70% of that judgement, the benchmark is accepted as task-specific evidence, whatever the generic ability check says. A general proxy judged unrelated with at least 80%, such as competition mathematics for clinical questions, is withheld. Direct and domain evidence are listed first on each recommendation. The four nearest methodologies are always assessed, whatever their prepared capability tags, because a vertical benchmark such as TaxEval is often tagged for a different ability than the one the task names. When no single ability clearly determines quality, as with security review or tax checking, Scout uses the required ability the interpreter favours most and adds a warning, instead of failing the search. Similarity never supplies a quality score. Rejected, unknown and unavailable evidence is withheld. If some checks fail, verified evidence can still be used with an explicit warning; if no applicability can be verified because checks are unavailable, the search returns unavailable rather than reporting a false no-match.
 
-Quality comparisons use shared benchmark results, grouped by correlated family and task capability. Different cohorts are not compared through unrelated percentiles. BFCL uses native function-calling configurations with their original labels; correlated series are one family. Missing scores remain unknown and actual zero remains a score. Model/version matching remains conservative. A relative quality signal is not predicted accuracy or a statistical confidence interval.
+Quality comparisons use shared benchmark results, grouped by correlated family and task capability. Different cohorts are not compared through unrelated percentiles. BFCL uses native function-calling configurations with their original labels; correlated series are one family. Missing scores remain unknown and actual zero remains a score. Model/version matching remains conservative. A "+" in a model name is significant, so Command A+ and Command A are separate models rather than one ambiguous name. A catalogue model's dated ID (such as Cohere's `command-a-03-2025`) counts as a copy of the undated model. A relative quality signal is not predicted accuracy or a statistical confidence interval.
 
 Matching calibration observations can supplement quality for their exact operation, format, modality, tools, retention and workload scope. Each admitted model needs at least six calibration cases, and evidence expires after 30 days (seven days before 22 October 2026). An applicable profile past that age stays excluded and the result carries a warning naming the profile and its expiry date, rather than silently using benchmarks only. A match no longer requires five tested models or excludes all untested eligible models. Observations remain tied to their tested router deployment. Repeated calibration request failures still cap effective quality. Calibration episode bills and durations remain source evidence; they are not substituted for only some candidates' reference costs or times. Held-out outcomes and answer keys never enter runtime ranking.
 
@@ -644,7 +796,7 @@ Cloudflare [retains Analytics Engine events for three months](https://developers
 
 Each query starts with batched Decisions questions and can make one verification pass for weak answers, followed by relevant specialist compatibility batches. Task interpretation and specialist comparison share a four-second request budget; catalogue reads and ranking add their own time. Optional verification has at most 750 milliseconds and leaves weak answers uncertain if unavailable. Independent specialist batches run in parallel, with at most 100 questions per call. Transient connection, timeout, rate-limit and server failures can receive one retry within the remaining deadline; valid `Retry-After` delays are honored. Authentication, exhausted-credit and quota failures are not retried. Failed interpretation returns an explicit unavailable response; failed specialist checks omit unverified specialist candidates. No keyword fallback is used. These bounds reduce avoidable waits and failures; they do not guarantee a successful search or a particular latency. In-memory admission allows 32 active requests and 600 starts per minute per Worker isolate; a separate edge limiter allows 60 POST requests per minute per IP. Excess requests fail promptly instead of queuing. These are operational limits, not an account-level billing cap or a throughput guarantee. Credentials remain server-side.
 
-Hourly sources refresh independently; reviewed typed-endpoint contracts retain their manual review dates. Freshness follows each source's prepared policy: public feeds have a two-hour freshness allowance, while incremental research uses the configured seven-day recheck interval. A source does not become stale merely because it follows a longer research interval. Missing or invalid check dates remain overdue, and actual collection or preparation failures remain degraded. A failed download, changed parser format, or unexpected loss of coverage preserves that source's previous evidence and timestamp while healthy sources and prices continue updating. Fresh catalogue removals still remove unavailable models. All fourteen Hub queries and required semantic preparation must succeed for its discovery snapshot to replace the previous one. A complete outage preserves the previous index. Failure and stale status are exposed in both interfaces.
+Public sources refresh twice daily and independently; reviewed typed-endpoint contracts retain their manual review dates. Freshness follows each source's prepared policy: public feeds have a 13-hour freshness allowance (one twelve-hour cycle plus an hour), while incremental research uses the configured seven-day recheck interval. A source does not become stale merely because it follows a longer research interval. Missing or invalid check dates remain overdue, and actual collection or preparation failures remain degraded. A failed download, changed parser format, or unexpected loss of coverage preserves that source's previous evidence and timestamp while healthy sources and prices continue updating. Fresh catalogue removals still remove unavailable models. All fourteen Hub queries and required semantic preparation must succeed for its discovery snapshot to replace the previous one. A complete outage preserves the previous index. Failure and stale status are exposed in both interfaces.
 
 Source content is parsed as data, never executed. Downloads, decompression, and request bodies are size-bounded; source downloads time out. Links are restricted to HTTP(S). The publisher's OpenVLM data URL uses HTTP and lacks transport authentication; the remaining feeds use HTTPS. Scout does not independently authenticate benchmark submissions.
 
@@ -657,13 +809,13 @@ npm ci
 npm run scout
 ```
 
-Open `http://127.0.0.1:45833/scout/`. This lightweight local preview uses the shared interpreter and matcher with bundled/source-feed data; it does not run the hosted Parallel research workflow or R2 publication system. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.
+Open `http://127.0.0.1:45833/scout/`. This lightweight local preview uses the shared interpreter and matcher with bundled/source-feed data; it does not run the hosted refresh workflow or R2 publication system. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.
 
-`npm run scout:refresh` fetches the eight hourly public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Source downloads need no provider inference, but preparing new or changed specialist profiles and benchmark interpretations requires `OPENAI_API_KEY` and incurs Decisions usage. Unchanged metadata reuses cached analysis. Without a usable key, required specialist preparation retains the previous specialist snapshot; affected new or changed benchmark definitions remain unclassified. Both failures are reported on the affected source. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid hourly evaluation job is installed. Recommendation queries require the server key and incur OpenAI interpretation/embedding usage. The local source-refresh command updates the bundled feeds; the hosted hourly Workflow additionally prepares and publishes its D1/Vectorize serving generation.
+`npm run scout:refresh` fetches the eight public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Source downloads need no provider inference, but preparing new or changed specialist profiles and benchmark interpretations requires `OPENAI_API_KEY` and incurs Decisions usage. Unchanged metadata reuses cached analysis. Without a usable key, required specialist preparation retains the previous specialist snapshot; affected new or changed benchmark definitions remain unclassified. Both failures are reported on the affected source. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid scheduled evaluation job is installed. Recommendation queries require the server key and incur OpenAI interpretation/embedding usage. The local source-refresh command updates the bundled feeds; the hosted twice-daily Workflow additionally prepares and publishes its D1/Vectorize serving generation.
 
 ## Hosting and verification
 
-`wrangler.scout.jsonc` defines the Scout Worker, a private R2 catalogue, an hourly Workflow and a SQLite Durable Object coordinating publication. The cron runs at minute 7 UTC. Configure `OPENAI_API_KEY` and `PARALLEL_API_KEY` as Worker secrets. Refreshes build immutable catalogue versions and prepared candidate pools; searches read those pools directly and never pass through the publication coordinator. A running refresh does not block searches against the active catalogue. The first hosted publication must be validated before serving searches; a missing catalogue fails explicitly. The Worker is `openmerit-scout`, the private catalogue bucket is `openmerit-scout-catalogue`, and the Workflow is `openmerit-scout-refresh`. Both API keys are stored as Cloudflare Worker secrets and read from server-side bindings; they are not shipped to the browser. Separate workers.dev and preview URLs are disabled.
+`wrangler.scout.jsonc` defines the Scout Worker, a private R2 catalogue, a twice-daily Workflow and a SQLite Durable Object coordinating publication. The cron runs at 00:07 and 12:07 UTC (`7 0,12 * * *`). Configure `OPENAI_API_KEY` and `PARALLEL_API_KEY` as Worker secrets. Refreshes build immutable catalogue versions and prepared candidate pools; searches read those pools directly and never pass through the publication coordinator. A running refresh does not block searches against the active catalogue. The first hosted publication must be validated before serving searches; a missing catalogue fails explicitly. The Worker is `openmerit-scout`, the private catalogue bucket is `openmerit-scout-catalogue`, and the Workflow is `openmerit-scout-refresh`. Both API keys are stored as Cloudflare Worker secrets and read from server-side bindings; they are not shipped to the browser. Separate workers.dev and preview URLs are disabled.
 
 Publication records a durable completion receipt after updating the active pointer, pending-research queue and health status. Interrupted publication resumes unfinished work; a lost success reply can be acknowledged again after the lease is released. Replaying a completed publication cannot roll back a newer catalogue or clear a later refresh's lease. Completed research receipts leave the queue only when publication commits; unfinished research retains its original paid job for a later hourly run.
 
@@ -673,7 +825,7 @@ The 17 October post-deployment recovery reused all 24 saved research jobs and pu
 
 Prepared capability pools can overlap. Searches use their prepared ID lists to choose a smaller covering set of pools, preserving the selected candidates and complete evidence while avoiding repeated downloads of the same model records. Explicit model and endpoint inventories retain their existing scope checks. These metadata reads do not fetch source pages or redo hourly research.
 
-`demo/domain-router.mjs` routes Scout paths to this service while retaining the existing website, protected memo, authentication, and terminal services. There is no Scout admin/report route. The public guide is generated and published with the main documentation. Do not deploy the older root `wrangler.jsonc` to publish Scout. `npm run build:scout` packages only Scout, public documentation, and its shared assets; private memo and demo applications are excluded.
+`demo/domain-router.mjs` routes the landing page, Scout and `/connect/` to the Scout serving service. Hosted docs, demos, engineering articles and terminal pages/APIs are offline during the rebuild. The separately protected memo was restored through the router on 25 October 2026; see the [cutover record](site-domain-migration.md). There is no Scout admin/report route. Do not deploy the older root `wrangler.jsonc` to publish Scout. `npm run build:scout` packages the landing page, Scout, `/connect/`, the standalone Scout data policy and required shared assets; generated docs, memo and demo applications are excluded.
 
 ```sh
 npm run check:scout

package/docs/security.md

@@ -53,6 +53,8 @@ Hosted Scout uses Clerk Google/GitHub sign-in. The page and status are public; J
 
 The [coding-agent setup](scout.md#connect-a-coding-agent) in npm 0.1.7 uses hosted device sign-in instead of OAuth or tokens in prompts. The copied prompt contains no secret. A signed-in person approves an eight-character code on `/connect`; codes expire after ten minutes and are single-use. Approval mints one key per device and client, scoped to `scout:recommend` and accepted only on `/scout/mcp`. The server stores only SHA-256 hashes. Keys expire after 90 days without use and can be revoked on `/connect` or with `openmerit remove`/`logout`. Locally the key is in `~/.config/openmerit/credentials.json` (mode 600) and in Claude Code's plain-text MCP configuration; the CLI never prints it. As with any device-code flow, only approve a code you started.
 
+**Automatic instruction updates — version 0.1.10:** updater-enabled setup defaults on with an opt-out, installing asynchronous Claude Code startup/response-end hooks. Checks use npm's public stable registry without credentials, task content or installation IDs. SHA-512 verification checks package bytes against registry metadata; it does not defend against a compromised publisher/registry. The updater reads only a bounded instruction bundle, never executes downloaded code or package scripts, and leaves MCP configuration, keys, privacy settings and project policy unchanged. A supported release contract permits instruction updates; maintainers must change it when permissions or data handling change. This is a publisher assertion, not semantic validation of instructions. Edited files are retained, concurrent writes serialize, and journaled recovery stops on intervening edits. New sessions reliably load new instructions. See [automatic updates](scout.md#automatic-instruction-updates-version-0110) for cadence, offline retries, manual commands and the one-time bootstrap required for older installations. Availability follows npm publication independently of hosted releases.
+
 ### OpenMerit coding-agent reset — 23 October 2026
 
 **Historical reset, superseded by npm 0.1.7 and hosted device setup.** The reset temporarily removed the installer and MCP transport. The current [Claude Code setup](scout.md#connect-a-coding-agent) uses browser-approved, device-specific keys scoped to MCP recommendations, not pasted session tokens or agent OAuth. The key is stored locally in mode-600 credentials and Claude Code's plain-text MCP configuration. Setup installs an owned Scout skill; conflicts are not overwritten, and removal revokes the device key. Browser Google/GitHub sign-in remains separate.
@@ -63,6 +65,8 @@ Scout serving, main-site documentation and npm publication are separate releases
 
 ## Hosted Scout analytics
 
+**Version 0.1.10:** [Scout setup analytics](scout.md#scout-setup-analytics-default-version-0110) defaults on without a prompt prefix; npm publication and hosted prompt deployment remain separate. `OPENMERIT_TELEMETRY=0` and `DO_NOT_TRACK=1` suppress setup analytics and propagate to approval and installed MCP usage. Browser privacy choices and account opt-outs still apply. Terminal/evaluation analytics remain opt-in, and installed MCP prompt-content capture remains off. This supersedes the setup opt-in default below for version 0.1.10 and later.
+
 **Coding-agent lifecycle instrumentation — npm 0.1.9 and matching Scout deployment:** setup clicks, clipboard/deep-link attempts, device authorization states, opted-in installation/component verification and each dispatched MCP invocation gain typed milestones and fixed error codes. Linked setup uses random setup/installation UUIDs, never paths, machine names, device codes, keys or transcript content. The browser handoff explicitly carries usage/unlink choices into setup; standalone CLI analytics remains opt-in. Setup/installation IDs are omitted in unlinked mode, DNT wins, and installed MCP prompt-content capture stays off. Request-only suppression no longer overwrites the saved account-wide preference. Explicit account opt-out/deletion gates remain in force. See the [lifecycle contract](scout.md#coding-agent-lifecycle-instrumentation-npm-019) for event meanings, bounded delivery retries and diagnostic counters. Existing installations require an installer rerun; release acceptance is separate, and no historical completeness, replay, transcript collection or skipped-call detection is claimed.
 
 **Version-5 instrumentation:** the [serving contract](scout.md#version-5-instrumentation) adds validated native web events and explicit CTA-only `$autocapture` mapping, not general DOM capture. Acquisition/UTM/device values are categorical allowlists; public URL properties are reconstructed without raw URLs. Browser performance is sampled and support-dependent. Linked same-tab signup/search identity is separate from unlinking's request-only identifiers. Logical MCP errors count even with HTTP 200; stage/provider codes, actual attempt tokens and feedback/retry traces exclude raw messages and free text. CLI diagnostics are opt-in in npm 0.1.7, adding sanitized version/platform/runtime and bounded application basename frames, never absolute paths or environment contents. Prompt sharing, retention, deletion gates and necessary provider processing are unchanged. Replay is a separate, inactive spike; no experiment is active. Main-site publication remains blocked by documented source divergence; the Scout rollout does not publish npm or activate alert notifications.
@@ -93,4 +97,4 @@ Clerk webhook signatures are verified over bounded raw request bodies with the S
 
 ## Scout research storage
 
-Scout uses Parallel for hourly public-source research. Parallel receives catalogue subjects and official-source metadata, not visitor prompts or application inventories. Decisions processes task descriptions and relevant public evidence. Public research receipts, citations, semantic annotations, publication completion receipts and immutable catalogue versions are stored in Scout's private Cloudflare R2 bucket. Public endpoint/model facts and observations also enter the private D1 serving projection. Vectorize stores public benchmark methodology embeddings, not visitor query vectors. Query embeddings are used transiently; OpenAI has its own API retention policies. Scout uses the prompt-and-evidence analytics policy above. API credentials are held as Worker secrets and are not included in browser assets, catalogue objects or analytics. Source text is treated as evidence, not executable code or instructions. Citations and semantic validation do not independently verify that publisher claims are true.
+Scout's twice-daily refresh reads public sources directly and no longer uses a research service. Decisions processes task descriptions and relevant public evidence. Previously collected public research citations, semantic annotations, publication completion receipts and immutable catalogue versions are stored in Scout's private Cloudflare R2 bucket. Public endpoint/model facts and observations also enter the private D1 serving projection. Vectorize stores public benchmark methodology embeddings, not visitor query vectors. Query embeddings are used transiently; OpenAI has its own API retention policies. Scout uses the prompt-and-evidence analytics policy above. API credentials are held as Worker secrets and are not included in browser assets, catalogue objects or analytics. Source text is treated as evidence, not executable code or instructions. Citations and semantic validation do not independently verify that publisher claims are true.

package/docs/testing.md

@@ -56,7 +56,7 @@ npm run verify
 npm pack --dry-run
 ```
 
-`verify` runs workspace tests, syntax checks, TypeScript checks, the package build, demo tests, generated-documentation checks, and whitespace validation. It does not publish a package or deploy the website.
+`verify` runs workspace tests, syntax checks, strict TypeScript checks and the TypeScript migration ratchet, the package build, demo tests, generated-documentation checks, and whitespace validation. It does not publish a package or deploy the website.
 
 ## Coordinator and Pi coverage