openmerit 0.1.7-preview.4 → 0.1.7

package/docs/scout.md

@@ -1,8 +1,241 @@
 # Scout by OpenMerit
 
-Describe what you need a model to do. Scout returns up to five model-and-endpoint candidates. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page, JSON API, and MCP tool all use the same index and ranking.
+Describe what you need a model to do. Scout returns up to five model-and-endpoint candidates per task. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page and JSON API use the same index and ranking.
 
-Scout is a separate beta product. It does not change the OpenMerit npm package, Pi coordinator, or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
+Scout is a separate hosted beta product. Npm 0.1.7 includes optional Claude Code setup for it; the hosted service does not change the Pi coordinator or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
+
+**Coding-agent setup — 24 October 2026:** stable npm 0.1.7 includes [device sign-in setup](#connect-a-coding-agent). Hosted Scout serves `/connect` and `/scout/mcp`; the button opens Claude Desktop with a setup prompt. This supersedes the 23 October web-placeholder reset. Existing older client configuration and authorization grants are not automatically removed or revoked. Scout serving, main-site documentation and npm publication remain separate releases; dated receipts below describe their original versions, not acceptance of every later build.
+
+## Beta evaluation logging — 22 October 2026 rollout
+
+The beta default records submitted prompts alongside recommendations for matching evaluation and debugging, not foundation-model training. The serving configuration enables `SCOUT_BETA_EVALUATION_LOGGING=true`, alongside `POSTHOG_CAPTURE_CONTENT=true` and complete account-sync/deletion configuration. The [beta data policy](/scout/data-policy/) is generated from Markdown and published with Scout, independently of the main website. This rollout supersedes the dated default-off contract below. The operator remains responsible for the applicable privacy basis and restricted PostHog access; a beta label does not establish consent or another lawful basis.
+
+With the rollout enabled, browser prompt sharing defaults on only when no saved content choice exists. Existing refusals, general analytics opt-out, DNT and GPC still win; unreadable preference storage keeps content off. The notice beside the prompt reflects the actual setting before submission. Footer controls can disable usage or prompt sharing. No draft text is collected. API/MCP content remains explicit opt-in through `X-Scout-Analytics-Content: on`; `off` or omission excludes it. CLI planning continues to send `off`.
+
+In analytics version 4, the bounded, redacted prompt and up to three task excerpts/five-layer breakdowns exist **only** in `$ai_trace.$ai_input_state`, as `{prompt, tasks}`. The ordered recommendations appear in `$ai_output_state` and the existing content-free `scout_search_details` records. Public evidence, ranks, metric bases, quality-screen status, timings and interpreter/ranker/index versions remain available; they describe recommendations, not model executions. [PostHog Cloud documents 30-day retention](https://posthog.com/docs/ai-observability/data-retention) for these AI input/output fields in `posthog.ai_events`, stripped from ordinary events at ingestion. A synthetic live-project check confirmed this storage separation, email redaction, the recommendation payload and `retention_days=30`. Ordinary metadata follows normal project retention. No second prompt copy, custom expiry job or batch export is added. Actual expiry is not yet verified; old version-3 detail events and external exports are not retroactively erased by this change.
+
+**Don't link this browser's activity to my account** sends `X-Scout-Analytics-Account: off`. The server substitutes fresh identifiers and excludes account identity/merging, ignoring old browser identifiers and client-supplied trace IDs that might already be linked. Unlinked responses omit `X-Scout-Trace-Id` so later account-linked interactions cannot reuse it. This setting applies to future requests from this browser, not other devices or copied coding-agent configuration; API/MCP callers can send the same header themselves. General account opt-out/deletion gates still apply. Separate Clerk profile and lifecycle analytics can continue while usage analytics is enabled. Unlinking does not erase old associations, make identifying prompt text anonymous, or enable automatic account-based deletion of unlinked historical events. Cross-request funnels and returning-user analysis are unavailable for unlinked activity.
+
+## Example tasks — source only, not deployed
+
+Three buttons below the prompt offer invoice extraction (one task), ticket triage and reply writing (two tasks), and feedback analysis, fix prioritization and release notes (three tasks). Choosing one replaces the draft with an editable example and focuses the prompt; it does not submit, open sign-in or consume a search. Use **Find models** or Enter when ready. The normal sign-in requirement applies.
+
+Examples go through the same interpretation and ranking as typed prompts, with no fixed results or forced task counts. Each task returns up to five eligible candidates; the example does not guarantee five or establish that a recommended model solves it. When analytics is enabled, `scout_example_selected` records only one of three fixed example IDs, never the current draft. Existing privacy controls and separately opted-in submitted-prompt handling still apply.
+
+## Product analytics
+
+### Version-5 instrumentation
+
+This contract applies to version-5 Scout serving builds. The serving rollout is independent of main-site publication and the npm package; npm 0.1.7 includes the opt-in CLI additions, while the website collector remains source-only. Prompt-content choices, redaction, retention and account deletion controls are unchanged. Main-site publication remains blocked by [source divergence](site-documentation.md#website-analytics-source-only-unreleased). Earlier dated receipts below describe earlier releases, not acceptance of version 5.
+
+- First-party, validated web events map to native PostHog `$pageview`, `$pageleave`, `$web_vitals` and `$autocapture` events. The last represents only explicit allowlisted CTA IDs, not generic DOM autocapture. Page paths are restricted to public home, Scout and supported docs routes; queries, fragments, titles, text and raw URLs are excluded. A random pageview ID pairs each view with its leave. Leaves carry bounded duration, maximum scroll depth and the fraction of page content reached by the viewport.
+- Acquisition fields are fixed referrer channels/domains and allowlisted `utm_source`, `utm_medium`, `utm_campaign` values; unknown campaign values are dropped. Browser, OS, device and viewport are categories, not raw user agents or exact dimensions. Native URL properties are reconstructed from a fixed public origin and allowed path, never the visitor's raw URL.
+- Performance observations cover LCP, CLS, INP, FCP and TTFB through a self-hosted collector, where browser APIs support them. Treat these as sampled observations, not complete traffic coverage or a configured statistical sampling rate: unsupported browsers, privacy controls, delivery limits and lifecycle exits can omit measurements. No resource URL or attribution payload is sent.
+- When account linking is enabled, shared same-tab identity supports website → Scout signup/search attribution; verified sign-in can identify that tab's anonymous activity. It is not cross-device anonymous tracking or a unique-person count. Unlinking instead uses fresh request-only server identities, ignoring supplied browser/session/trace correlation; no cross-request funnel is available.
+- MCP outcomes inspect JSON-RPC errors and tool `isError`, including logical failures with HTTP 200. Server diagnostics add bounded stage, failure code and provider HTTP status, distinguishing timeout, transport/HTTP, rate/quota/billing, refusal, incomplete and validation/provider failures without raw messages.
+- AI computation spans describe actual bounded pipeline attempts, including retries and provider-reported input/output/cached-input tokens when available. Cache hits/shared followers do not invent their own attempts. Missing usage stays unknown; no cost is fabricated. Recommended models and their estimated evidence remain recommendations, not executions or actual charges.
+- Results offer `useful` / `not_useful` feedback, with an optional fixed reason (`wrong_task`, `missing_model`, `too_expensive`, `insufficient_evidence`, `other`), not free text. Linked feedback and result interactions use the request trace. A repeated failed prompt within five minutes in the same page can carry `retry_of`; the comparison stays in memory, not an analytics prompt/hash. Privacy changes invalidate prior feedback correlation. Delivery acknowledgement does not prove ingestion.
+- Opt-in CLI diagnostics add sanitized package version, platform/runtime categories, command correlation and allowlisted application basename/line/column frames; see [the CLI contract](inference-terminal.md#optional-cli-product-analytics). No arguments, absolute paths, raw messages or environment contents are collected.
+
+Replay is a separate spike and is **not activated or available**. No feature-flag/ranking experiment is active. Operational dashboards must distinguish the version-5 fields from older records with missing diagnostics. Alert cadence and recipients must be chosen and release ingestion acceptance completed before alerts can be enabled. See the [operator evaluation workflow](site-scout-architecture.md#reviewed-feedback-and-future-experiments).
+
+**Release boundary, 22 October 2026:** hosted Scout uses the [unified PostHog integration](#unified-product-and-search-analytics-deployed-22-october-2026) below. New Cloudflare Analytics Engine search writes have stopped; historical records are unchanged. Main-site/docs instrumentation and npm publication remain pending.
+
+Hosted Scout has PostHog product analytics enabled in the EU region. Other deployments remain **inactive without a PostHog project token, region and environment**. The latest rollout confirmed anonymous visits and rejected requests in PostHog storage, with three suppressed validation identities absent. Authenticated searches, profile synchronization and real Clerk lifecycle deliveries still require live acceptance; passing local tests or HTTP acknowledgements do not establish those outcomes. Npm 0.1.7 includes separately opt-in CLI telemetry.
+
+The operator's PostHog dashboard has eight charts covering visits, the sign-in journey, recommendation attempts and outcomes, successful-search p50/p95 latency, model-result interactions, API/MCP request outcomes and opt-in CLI completions. Charts exclude test/development environments; early production activity includes rollout checks. Complete authenticated-session tracking, successful searches and result interactions still await real-usage verification and are labelled accordingly. Empty pending charts are not verified zero usage. Reusable metric definitions are saved as unapproved proposals, not canonical metrics. The separate main-site documentation deployment is unchanged.
+
+When configured, explicit events cover visits, opening sign-in, authenticated sessions, rendered results, opening model evidence, successful model-ID copies, provider-link clicks, and bounded failure codes. Interaction events contain task position and model rank, **not the copied ID, link URL or task text**. `scout_search_completed` is server-owned: one event per accepted recommendation attempt, including cache hits and concurrent callers sharing computation, regardless of the number of tasks. It records outcome, task/candidate counts, server duration and cache status. `scout_request_completed` separately measures API/MCP requests, including authentication/validation failures and MCP tool failures returned with HTTP 200. Do not add request, rendered-result and search events together to count searches. A successful recommendation is not evidence of successful model execution.
+
+Authenticated events use a namespaced SHA-256 pseudonym of the **verified** Clerk user ID. The configured profile sync below can associate that identity with a name and verified email after an eligible visit; it is not anonymous data. Raw Clerk user/session IDs, bearer tokens and cookies are never forwarded to PostHog. The same verified account has the same pseudonym across web/API/MCP. `client` (`web`, `cli`, `agent`, `unknown`) is an allowlisted caller hint; `transport` (`http`, `mcp`) comes from the actual route. Neither is an authorization signal. Browser events are client-reported, not a security audit.
+
+The browser keeps a random anonymous ID and hashed account/session transition markers in this tab's `sessionStorage` so an OAuth return can join the visit-to-sign-in funnel. Sign-out/account changes rotate that anonymous ID. With storage disabled, these values remain in memory and redirect attribution may be lost. Anonymous users are not linked across separate tabs or visits; returning authenticated users can support account-level retention. CLI telemetry is separately [opt-in](inference-terminal.md#optional-cli-product-analytics), with a process-only identity and no implicit account linking.
+
+There is no PostHog browser SDK, autocapture, session replay, DOM/text collection, feature-flag polling, geography lookup or IP forwarding. Browser events pass a typed allowlist at `/scout/api/analytics` before the server sends them to PostHog. Server search diagnostics can include bounded public source URLs and separately opted-in, best-effort-redacted prompt text as described below. Explicit exception handlers send sanitized categories and application frames, not raw messages or arbitrary stacks. Disabling analytics does not disable necessary provider processing or erase historical records.
+
+Browser Do Not Track and Global Privacy Control suppress both browser and server-side PostHog events. To opt out for this browser independently, run `localStorage.setItem('scout-analytics-opt-out', '1')` on Scout and reload; remove that item and reload to restore collection. API/MCP clients can send `X-Scout-Analytics: off`, `DNT: 1` or `Sec-GPC: 1`. The event endpoint has a separate per-IP quota, so analytics clicks do not consume the search allowance. Delivery is best effort, without retries; the Worker uses background delivery with a 1.5-second timeout, and an outage cannot block recommendations. Receipt does not prove ingestion. PostHog retention, access, spend caps and any required consent are project/operator responsibilities; Cloudflare's three-month retention does not apply to PostHog.
+
+### Unified product and search analytics — deployed, 22 October 2026
+
+Hosted Scout now sends new search diagnostics to **PostHog only**, through the existing first-party ingestion boundary. It stops new Cloudflare Analytics Engine search writes; historical Cloudflare records are neither exported nor deleted. OpenAI processing, the short-lived result cache, catalogue storage and infrastructure logs are separate from analytics and unchanged. Anonymous event ingestion is confirmed; authenticated recommendation traces and errors have local contract coverage but still require live acceptance.
+
+One `scout_search_completed` still counts one recommendation attempt, including cache hits/shared callers. A `$ai_trace` describes that request and `scout_search_details` records each task/slot's ordered shortlist, public evidence, metric bases, quality-screen status and diagnostic context. The server creates a unique `trace_id`, returns it as `X-Scout-Trace-Id`, and the browser includes it on result views/copies/provider clicks. Identities and traces never enter shared computations or cached results. These are **recommendation traces**, not model executions: candidate costs/latencies are reference estimates, not actual LLM charges or task-success measurements. Missing measurements remain null. Evidence is bounded to 20 items per list, depth 10 and a 48,000-character string budget per task/slot; `evidence_truncated` and original benchmark counts disclose omissions while all five compact ranks/metrics remain. Prompt-derived semantic-match prose is excluded.
+
+**Historical initial content contract:** prompt text was off by default in this rollout; the beta revision above supersedes that browser default. API/MCP still require explicit `X-Scout-Analytics-Content: on`, the deployment content switch and complete account-sync/deletion configuration. CLI planning always sends `off`. General analytics opt-out, DNT or GPC overrides content sharing. Standard analytics and browser preference changes do not send input text; content is processed only when a task is submitted.
+
+Opted-in text receives best-effort removal of common credentials, authorization values, private keys, emails, IP/phone patterns, URLs and local home/temp paths. This cannot recognize every personal name, proprietary passage or secret. Never submit confidential records. Prompt text is limited to 4,000 characters; layer/excerpt text to 1,000 each. Content is linked to the verified account and is identifiable when profile sync is active. Turning sharing off stops future capture, not in-flight work or historical retention. Profile deletion also requests deletion of these account-linked events. Set suitable PostHog retention, access controls and spending limits before activation; no automatic retention setting or old-data backfill is included.
+
+Browser and server failures produce PostHog `$exception` events. Scout errors contain fixed categories, allowlisted exception classes, and up to 20 application-filename/line/column frames; messages, function names, absolute paths, source context, query strings, headers and arbitrary stacks are excluded. Browser exceptions are capped at three per page. Terminal/CLI and main-site errors currently carry categories/classes without frames. Deployment metadata labels serving releases. No session replay, DOM/text autocapture, fingerprinting or Sentry is added.
+
+Source-only homepage/docs events cover page categories and explicit navigation destinations, with a separate anonymous tab ID and no Clerk linking; that separate site deployment is pending. Opt-in local terminal events cover navigation/map interactions, inspector kinds, bounded aggregate inventory counts and errors, never local model/agent/project names or source content. The same process-only ID links terminal events to its CLI launch. Authenticated Scout web/API/MCP events share the verified account pseudonym, but anonymous website/CLI identities are not automatically resolved to that account. See [terminal telemetry](inference-terminal.md#optional-cli-product-analytics). The existing production dashboard has not been updated for this release; new charts, retention changes and historical backfill are not included.
+
+### Account profile synchronization — deployed, 22 October 2026
+
+Clerk-to-PostHog synchronization is **deployed and configured, but not yet verified against real Clerk deliveries**. Its signing secret, account-state binding and project-restricted PostHog person-deletion credential are installed. Without those prerequisites, personal profile export stays off. Clerk remains the identity system; Cloudflare coordinates synchronization, rather than providing a second login provider. There is no Clerk Billing integration. Account deletion is covered locally, not proven against a production account.
+
+After Scout receives an authenticated browser's enabled analytics preference, PostHog receives only the account's **name, verified primary email, signup time and connected Google/GitHub providers**. Values come from Clerk's current Backend API record, not browser-supplied profile data or an old webhook payload. Removed/unverified primary emails and cleared names are removed from the current PostHog profile. Existing accounts synchronize on their next eligible visit; there is no whole-directory backfill. Profiles and behavior use the same account pseudonym, so they are identifiable personal data. Neither profiles nor account IDs are added to the Cloudflare prompt archive.
+
+Verified `user.created`, `session.created` and `session.ended` deliveries produce `scout_signed_up`, `scout_signed_in` and `scout_signed_out`. Session removal and revocation have separate events. Browser session observations become `scout_session_observed`, which links the anonymous visit to the authenticated account but is not a new login. Before this change, `scout_signed_in` meant a browser session observation; dashboards must distinguish the release boundary and must not add the two events together. Lifecycle events use the Clerk event time and stable delivery IDs. Provider attribution is `unknown` for webhook lifecycle events. The browser reports Google/GitHub on session observations only when a completed, verified sign-in attempt identifies that exact session; connected providers and the client's latest sign-in strategy are not proof. Signup or restored sessions can remain unknown.
+
+The signed-in browser sends an authenticated boolean preference to `/scout/api/analytics/preference`, including when DNT, GPC or local opt-out is active. This privacy-control request contains no product event, anonymous ID or profile. Once received, an opt-out stops account-linked events and background profile/lifecycle forwarding and discards unsent lifecycle events. A later enabled preference from an authenticated browser can resume collection for that account; preferences are account-wide and the latest server-observed preference wins. Opt-out is not historical erasure. A browser that is offline, blocked or not signed in cannot update server-held preferences until it reconnects and authenticates. Request-local opt-outs still suppress that request immediately. These controls do not implement a consent banner or establish legal consent.
+
+Per-account Cloudflare Durable Objects hold preference state, hashed profile fingerprints, up to 256 recent delivery receipts and deletion tombstones. Before a preference is known, at most 128 pseudonymous lifecycle events wait up to seven days for an eligible visit; alarms expire the pending buffer. No names, emails, raw Clerk IDs, tokens, webhook bodies or prompts are stored there. Signed webhooks retry through Clerk/Svix on delivery failure; failed preference initialization retries on a later visit. Ordinary product-event delivery remains best effort. Search never waits for Clerk profile retrieval or PostHog.
+
+Deleting a Clerk account stores a permanent pseudonymous tombstone and requests deletion of its PostHog person, associated events and recordings, even if analytics is opted out or new capture is disabled. Late updates and still-valid session JWTs cannot send new account events through Scout. PostHog can still ingest an event accepted before deletion, so a Durable Object alarm repeats deletion daily, including after a response found no matching person. This reconciliation continues while the integration is configured and incurs background requests. PostHog deletion is asynchronous: an accepted request is **not proof that historical data has been erased**, and operators must check deletion status and replay failed webhook deliveries. Clearing a profile field alone does not remove old `$set` events. The separate Cloudflare prompt archive has no account identifier and is not erased by this sync. These changes do not authenticate or link anonymous CLI telemetry.
+
+## Account sign-in — published, 21 October 2026
+
+Hosted Scout uses Clerk authentication for **Google and GitHub only**, without Clerk Billing, subscriptions or payment collection. Production configuration checks confirmed both providers enabled, no additional required profile fields, and organizations and Billing disabled. The production domain and keys are configured and the serving Worker is deployed. Google/GitHub enablement is a Clerk dashboard setting, not something the source code alone establishes.
+
+The page and index status remain public. **Sign in** opens Clerk's account UI; submitting a task while signed out opens that same flow without invoking matching. A prompt draft is temporarily kept in this tab's `sessionStorage` to survive the OAuth redirect, then removed on return or successful sign-in. Storage-disabled browsers can sign in but may lose their draft during the redirect. Signing out or changing accounts clears the visible prompt/results and cancels the browser's pending request; it does not guarantee cancellation of provider work already accepted by the server.
+
+The Worker verifies Clerk session tokens before catalogue access for both `/scout/api/recommend` and `/scout/mcp`, including MCP discovery. Requests must send `Authorization: Bearer <session token>`; a cookie alone is not enough. Signature, expiry, issuer and authorized-party validation use Clerk's SDK, and pending sessions cannot search. Missing/invalid credentials return HTTP 401; missing server configuration returns 503 rather than enabling anonymous access. Status and `/scout/api/auth` remain public; the latter exposes only the publishable key and frontend API hostname. The secret key never reaches the browser.
+
+For an authenticated development/API check, obtain a current token from `await window.Clerk.session.getToken()` in your own signed-in browser. Treat it as a credential, do not share it, and set it privately as `SCOUT_TOKEN`. Then send:
+
+```sh
+curl https://openmerit.ai/scout/api/recommend \
+  -H "Authorization: Bearer $SCOUT_TOKEN" \
+  -H 'Content-Type: application/json' \
+  -d '{"prompt":"Extract six invoice fields as JSON"}'
+```
+
+Use the same header for MCP. These tokens are short-lived; clients must refresh them rather than store one as a permanent API key. This first integration does **not** implement agent OAuth discovery/registration, long-lived API keys, or unattended agent sign-in. Existing clients without a way to supply a current session token now receive HTTP 401 and need an update. Already-issued valid JWTs may remain usable until expiry after session revocation; this is not a per-request online session-status lookup.
+
+The existing edge limiter applies separately to IP addresses and authenticated user IDs: 60 protected requests per minute for each key, including MCP initialization/discovery. It is not a durable monthly usage ledger or a billing cap; the local preview does not emulate Cloudflare's limiter. Clerk handles account/profile and session data and loads its browser SDK/UI from its own frontend API domain. Its optional SDK telemetry is disabled. Raw account IDs are used transiently for rate limiting, not attached to the historical Cloudflare prompt archive. The 22 October analytics release adds pseudonymous account-state storage and the profile sync above; no public saved-search history or billing integration is introduced. PostHog and Clerk have separate retention policies from the historical three-month Cloudflare archive.
+
+Local development now also needs `CLERK_PUBLISHABLE_KEY` and `CLERK_SECRET_KEY` in the server environment. Configure a Clerk development instance with Google/GitHub sign-in and sign-up; disable other sign-in methods and unnecessary required profile fields. Use the production instance and its own OAuth applications for `openmerit.ai`. Never expose the secret key in client configuration or commit it. The deployed production modal was checked at desktop/mobile sizes, and both buttons reached provider authorization redirects requesting basic identity/email scopes only. Signed-out submissions opened sign-in without a recommendation request and preserved the tab's prompt draft. Hosted API/MCP checks rejected anonymous, invalid-token and cookie-only requests. Tests stopped before provider authentication: complete sign-in/sign-up, callback return and real-user authenticated searches remain unverified. Offline signature tests are not evidence of a complete Google/GitHub login.
+
+## Canonical promotion — published, 21 October 2026
+
+The [canonical Scout](https://openmerit.ai/scout/) now replaces the older interface and matcher with the former alpha: original prompt excerpts, one shared task-sequence table, automatic ambiguity resolution and cost-first selection after the relative quality screen. Canonical endpoints remain `/scout/api/status`, `/scout/api/recommend` and `/scout/mcp`. Existing `/scout/alpha/` page, asset and API/MCP URLs redirect to the corresponding canonical paths with HTTP 308, preserving query strings and POST methods/bodies. Clients should update their configured endpoint rather than rely on redirects. There is no separate alpha interface after promotion; retired `/scout/v1/` and `/scout/v2/` paths remain unavailable.
+
+Only serving changes. The existing hourly catalogue publisher, research budget, data and secrets remain intact; the unreleased collection changes below are not activated. Search retains the alpha analytics dataset and existing three-month data-handling policy. The homepage, terminal and protected main application are outside this deployment. Candidate counts can remain below five per task, timing may be unknown, and searches can exceed five seconds. Promotion does not establish task accuracy or jointly evaluated workflows. This promotion supersedes the alpha-only scope in the dated release records below.
+
+Fresh canonical browser searches returned five, nine and thirteen candidates for one, two and three tasks in 6.91, 6.44 and 6.71 seconds, using interpreter 37 without result-cache hits or clarification questions. A real MCP request through the old alpha URL redirected successfully to `/scout/mcp`. Redirects retained queries and POST bodies; untrusted origins remained rejected. Deployed page/script/style hashes matched the build, and desktop/mobile renders were inspected. The publisher's deployment and hourly schedule, main/router/terminal deployments and unrelated routes remained unchanged. Full verification passed 692 tests. These smoke checks establish the canonical routing and UI contracts, not a latency or task-accuracy guarantee.
+
+## Task sequences — published alpha, 21 October 2026
+
+The [alpha](https://openmerit.ai/scout/alpha/) UI uses a normal Model/Quality/Cost/Time table for one task. For two or three tasks, each task becomes a column in one shared table. Read each row left to right as a sequence of model choices, with quality, reference cost and time beneath each model. Column headings quote short excerpts from the submitted prompt rather than rewriting the work. The five-layer breakdown, role labels, frontier badges and capability tags remain hidden. The table scrolls internally when needed; narrow screens retain the side-by-side order instead of stacking separate task tables.
+
+Rows align each task's independently ranked alternatives; they are **not jointly evaluated workflows**, a global ranking of model combinations, or verified model-to-model dependencies. The top row takes each task's first choice under its existing ranking policy. Five is still a maximum, not a quota: a shorter shortlist leaves an explicit gap, and no model is repeated to complete a sequence. The same model can legitimately appear in different task columns with different evidence and workloads. Expand any model for its task-specific evidence. **Assumptions & limitations** retains quality-screen status, workload, warnings and specialist discovery. Unknown values remain unknown; calibration accuracy is distinct from relative benchmark scores. Costs are USD per 1,000 reference tasks, not exact dataset charges, and are not added into a workflow quote.
+
+Interpreter version 37 requests a short `tasks[].sourceText` excerpt in the existing decomposition call and checks that it occurs verbatim in the prompt. When every excerpt has a unique position, those positions establish task order while keeping restrictions and models attached; otherwise the interpreter's task order is retained. An invalid excerpt becomes `null`, without blocking the shortlist or asking a follow-up question; the UI then uses an unquoted interpreted title (or the original full prompt for a single task). Long headings are visually shortened, with their full text in the hover hint, accessible text and expanded model details. The existing five-layer data and `tasks[].slots[0]` remain; no additional model call, dependency analysis, recommended-model execution or ranking-policy change is introduced. This change is published only to alpha; stable Scout is unchanged.
+
+Fresh hosted browser checks returned five, ten and fourteen candidates for one, two and three tasks in 7.66, 7.43 and 7.12 seconds. All three searches used interpreter 37 without result-cache hits, preserved original prompt excerpts and cost-ordered model lists, and displayed no clarification questions. A separate uncached MCP request returned five coding models. The deployed HTML, JavaScript and styles matched the checked build; desktop and 390px mobile captures were inspected, including horizontal scrolling and model evidence expansion, with no browser errors or page overflow. Full verification passed 692 tests before publication; deployment repeated 349 Scout tests and 13 documentation tests. Stable Scout, the main site, domain routing and the hourly publisher were unchanged. These checks do not establish recommended-model task accuracy, a fixed five candidates per task, or the under-five-second target.
+
+## Newer document evidence — deployed, 22 October 2026
+
+Document and image-reading tasks previously relied on document benchmarks that covered only models from OpenVLM's 17 September 2025 snapshot and four OmniDocBench entries, so receipts and invoices could return GPT-4.1 or GPT-4o. The hourly refresh now imports the [IDP Leaderboard](https://www.idp-leaderboard.org/) component series as three benchmarks. Fourteen of its 26 entries match catalogue models, including GPT-5.4, GPT-5.2, GPT-5 Mini, Gemini 3.1 Pro, Gemini 3 Flash and Claude Opus, Sonnet 4.6 and Haiku 4.5. Benchmark interpretation tags all three series as document and image evidence. Its newest entries still trail the latest releases.
+
+A model with no document-reading result can now qualify for a document task on general image-understanding evidence. Such picks carry `missingEvidence: ["documents"]` and a warning naming them. Quality is still estimated only from accepted evidence; this widens eligibility, not the evidence itself.
+
+The first refresh with this source published catalogue `4efa445856c7eec97819270e48c0e658` (checked 22 October 2026, 13:40 UTC). Replays of invoice and receipt prompts against it returned current models such as GPT-6.1 Sol and Claude Opus 5.5 rather than GPT-4.1 and GPT-4o, but only one or two candidates, both admitted on general image evidence.
+
+In unreleased source, an accepted benchmark counts only for the ability the request-time applicability check confirmed for that task, not for every ability in its hourly interpretation. A spatial-reasoning benchmark accepted as image evidence therefore no longer counts as document evidence or carries document weight.
+
+Also unreleased: models with accepted document evidence are ranked among themselves first, so stronger general image scores elsewhere cannot push them out of the quality band. Image-only fallbacks are ranked separately and fill only the remaining places, after every document-evidenced model. They receive no strongest, best-value or fastest label, and their warning says they are listed after models with document results. With this change alone, replays against the same catalogue returned three or four document-evidenced models for the invoice and receipt prompts, with Claude Opus 5.5 as the only fallback; the margin change below brings both to five document-evidenced models. A cost-priority search lists the cheapest document-evidenced model first.
+
+## Writing, summary and analysis evidence — unreleased
+
+Task interpretation (interpreter 39) distinguishes two narrower abilities: **summarization & synthesis** (a child of writing) and **analysis & recommendations** (a child of reasoning). Searches previously reduced summarizing feedback, prioritizing fixes and writing release notes to the same writing weight, and their only accepted evidence was one creative-writing benchmark covering 20 mostly older models, so all three received one shortlist. The hourly refresh takes core abilities from the released registry; abilities discovered by research are kept.
+
+The hourly refresh adds two sources:
+
+- [Vectara Hallucination Leaderboard](https://github.com/vectara/hallucination-leaderboard) (Apache-2.0): the share of summaries of supplied documents that stay consistent with their source. Models answering fewer than 90% of documents are excluded, because their rate covers only the documents they answered. It measures faithfulness, not summary quality; smaller models often score higher.
+- [Arena leaderboard dataset](https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset) (CC BY 4.0): style-controlled human-preference ratings for eight request categories (creative writing; writing, literature and language; business, management and finance; instruction following; hard prompts; multi-turn; longer queries; expert) and the document arena. Ratings become an expected win rate against the median-rated model of the same leaderboard; ratings with confidence intervals wider than 60 points are excluded. Preference is not verified correctness.
+
+A test refresh matched 59 catalogue models to Vectara, about 142 to each Arena text category and 22 to the document arena.
+
+Three changes to quality qualification accompany them:
+
+- **Disagreeing evaluations.** With three or more connected evaluations, one whose scores correlate below 0.3 with the estimate from the others (over at least five shared models) is left out of the common quality estimate. A narrow score range gives such an evaluation very small fitted noise, so it previously dominated the estimate: models with only a Vectara result ranked above every frontier model.
+- **Margin and uncertainty.** The practical-equality margin is the larger of two points and 0.15 of the spread of model estimates on the task, so it means a similar share of model differences whichever evaluations a task uses. Each model's standard error also includes how much its own evaluations disagree about it, so one tightly measured evaluation cannot decide a result alone.
+- **Agentic evidence.** When a task calls no tools and needs no coding, evaluations interpreted as agentic or tool-calling are not used, unless they are the only accepted evidence; a warning names them. Coding evaluations normally run in agent harnesses and keep them.
+
+Replays of a three-task request (feedback themes, prioritized fixes, release notes) against a test refresh of catalogue `4efa445856c7eec97819270e48c0e658` gave three different shortlists. The summary task rests on Vectara alone and lists small, low-hallucination models. Invoice and receipt replays returned five document-evidenced models each.
+
+## Shortlist corrections from user reports — deployed, 22 October 2026
+
+Three reported results had separate causes. Image-reading tasks whose primary quality target was another ability, such as structured output, returned no candidates: evidence applicability accepted only benchmarks for the quality target, while ranking still required document or vision evidence for admission. Applicability now also offers the abilities the ranker requires (documents or vision, tools, memory), so a document benchmark can satisfy that requirement.
+
+A stated preference such as "prefer the lowest-cost model that can extract these fields accurately" was lost, because ranking priority was judged from the extracted downstream work, which deliberately omits preferences. Structured extraction now also returns the stated tradeoff from the whole request (`balanced`, `quality`, `cost` or `latency`), and that value sets `workload.priority`. Requests for the cheapest model that is still accurate are `cost`; candidates still pass the relative quality screen first. Interpreter version 38; no extra model call.
+
+Similar tasks can still receive identical shortlists when one benchmark is their only accepted evidence. The result now says so: a warning names the single benchmark, and later tasks note which earlier task has the same shortlist. These are interpretation and routing checks against the published catalogue, not executions of the recommended models.
+
+## Common quality estimate and labelled tradeoffs — deployed, 22 October 2026
+
+This change replaces the cost-first policy below. Quality screening previously compared a model only with models that had exactly the same set of accepted benchmarks. That split the catalogue into many two- or three-model groups, so the best of a small, weak group passed the screen and cost-first ordering promoted it.
+
+Scout now fits **one latent quality estimate across every connected evaluation** for the task. Each benchmark has its own difficulty offset, score scale and noise, so a near-zero benchmark and a widely spread one contribute on comparable terms. Correlated series in one benchmark family share that family's weight. Each model receives an estimate with a standard error; a benchmark run by only one model links nothing. Applicable calibration observations join as one further evaluation using effective quality. Models with no evidence connected to the main group receive no estimate and remain provisional; they are never compared numerically. The estimate is reported as `comparison.qualityEstimate` and `comparison.qualityStandardError`; `quality` is its relative percentile among the estimated models.
+
+A configuration qualifies when it is not credibly worse than the strongest estimate: within two points plus one combined standard error. Supported Pareto selection over quality, reference cost and reference time follows. A balanced search keeps the strongest, lowest-known-cost and fastest frontier points, labelled in `pick` as `quality`, `value` and `speed`, then orders by estimated quality. An explicit `quality`, `cost` or `latency` priority orders the frontier accordingly. At most two models per provider are selected while other qualifying models remain; the cap never leaves a supported place empty. `ranking.version` is `10`, and `ranking.frontier` reports `version: 4`, the priority, the number of estimated models and benchmarks, and the tolerance.
+
+Reference time previously required measurements bound to the selected endpoint. No published catalogue endpoint had such a binding, so time was unknown for every candidate and dominance never removed one. Endpoint-bound timing still takes precedence and never migrates to another route. Artificial Analysis model-level medians, which are not tied to any endpoint, now apply to each route of that model, marked `speed.scope: "model"`, with a warning that actual endpoint latency may differ.
+
+These remain relative benchmark signals, not task accuracy. An offline replay of the bundled 15 October catalogue, with simulated evidence retrieval, returned frontier coding and reasoning models first instead of the lowest-cost members of small cohorts; it did not execute any recommended model. Ranking added about 6 ms per search locally.
+
+## Cost-first selection — published alpha, 21 October 2026
+
+The source change above supersedes this policy once deployed.
+
+The [alpha](https://openmerit.ai/scout/alpha/) ranker puts the **lowest known reference cost first after a relative quality screen**. It keeps models within two proxy-score points of the strongest model with the same evaluation coverage, using the existing quality-indifference band. At least two distinct models must share that scope; duplicate routes do not create an independent comparison. Supported Pareto dominance still removes inferior configurations before the list is limited to five distinct models. Latency breaks price ties, followed by optional metadata preferences and quality. Extra quality, a soft preference or an inferred priority does not justify a higher price rank. Explicit metadata restrictions remain mandatory.
+
+This is a conservative interim policy, **not a verified “good enough for your task” threshold**. A best-in-cohort benchmark score may still be inadequate for the task. Unrelated evaluation scales are not compared numerically, and the relative display score is not used as an accuracy floor. Even a user-stated target such as 95% accuracy remains unverified without matching task evaluations; Scout neither invents such a target nor claims that the proxy band satisfies it.
+
+Missing or disconnected quality evidence cannot displace or pad a qualifying shortlist. If no comparable cohort exists, Scout retains an explicitly unverified discovery list rather than treating unknown quality as a pass. `ranking.frontier.qualityGate` reports `proxy-qualified` or `unverified`, the margin, and configuration counts. Unknown prices sort after known prices; unknown latency is not zero. Costs still use the disclosed reference token workload, not a quote for the user's exact images, execution loop or dataset. Native task-fit specialist lists remain unmeasured, not quality-qualified recommendations. No additional model calls, collection jobs or UI controls are introduced.
+
+Local real-provider browser checks returned five, eight and thirteen candidates for one-, two- and three-task searches in 6.69, 4.70 and 7.57 seconds, all cache misses. The invoice task retained three distinct models after screening and dominance; it was not padded to five. Every returned benchmark shortlist was ordered by reference cost. A single-model inventory displayed an unverified shortlist, not a quality pass. Desktop and narrow light-theme renders were inspected. These tests use the local catalogue and do not execute the recommended models, establish task accuracy, validate hosted storage or meet the under-five-second target universally.
+
+After deployment, hosted browser searches returned three, nine and fourteen candidates for one, two and three tasks in 5.50, 6.59 and 6.26 seconds. An additional MCP image-classification request returned four models in 5.08 seconds. All four requests used real OpenAI calls and production R2/D1/Vectorize without result-cache hits; every benchmark shortlist was cost-ordered and contained no clarification. A single-model inventory remained explicitly unverified. Desktop and mobile captures were inspected. The hosted catalogue and retrieved evaluation scopes differ from local checks; candidate counts and identities are not invariant across interpretations. Full verification passed 691 tests. Stable Scout, the main site and the hourly publisher were unchanged. These samples do not establish exact task accuracy, universal reliability or the five-second target.
+
+## Direct model lists — published alpha, 21 October 2026
+
+Alpha resolves ambiguity internally and returns one ranked shortlist per independent task, without clarification questions or answer buttons. One, two and three tasks target five, ten and fifteen candidates respectively. Each list contains distinct models; the same model may suit more than one task. Explicit restrictions and evidence requirements still apply, so fewer candidates can be returned when the verified frontier or selected inventory is too small. Scout does not invent models or measurements to fill a list.
+
+Steps toward one deliverable remain one task. Scout chooses the most likely intended output when it is unstated, marks inferred task layers, and leaves optional handling policies unspecified. It preserves explicit labels, boxes, masks, formats, provider restrictions and exclusions. Missing runtime data does not trigger a conversation. More than three independent tasks still returns HTTP 422; provider failures remain errors rather than fabricated recommendations.
+
+HTTP and MCP retain the existing response envelope: each `tasks[]` entry has its five layers and exactly one `slots[0]` containing the ranked models. The top-level compatibility shortlist remains available for a single task. `task.clarification` is always `null` in the evidence engine. Any residual semantic ambiguity is reconciled internally before evidence retrieval. The browser no longer displays model-role counts or clarification controls. The 30-second server and 35-second browser timeouts are unchanged; this is not a latency improvement.
+
+Before the cost-first quality screen, local browser checks with real OpenAI calls returned five, ten and fifteen models for one-, two- and three-task requests in 4.77, 5.42 and 6.48 seconds, without result-cache hits. Explicit bounding-box and pixel-mask requests retained their requested output types. One of the first eleven real-provider checks returned HTTP 503; twenty subsequent fresh repetitions of that prompt succeeded, but the intermittent failure's cause is unconfirmed. These checks use the local catalogue, not hosted storage, and do not establish universal search reliability or the under-five-second target.
+
+## Multi-task discovery — alpha, 21 October 2026
+
+The `scout-one` interface and evidence matcher are published as an experimental [alpha](https://openmerit.ai/scout/alpha/). The stable `/scout/` service is unchanged; the dated hosted-beta records below describe its earlier releases. This section records the initial role-based alpha contract; the direct task lists and cost-first policy above supersede it.
+
+Alpha serves `/scout/alpha/`, with `/scout/alpha/api/status`, `/scout/alpha/api/recommend` and `/scout/alpha/mcp`. It uses a separate Worker and reads the existing published catalogue; it does not run a second refresh or publish the expanded research policy. Alpha analytics uses a separate private dataset with the same data-handling policy. Initial publication checks confirmed the page, assets, status and MCP tool listing, but all three recommendation smoke checks failed: invoice and two-task HTTP requests returned unavailable in approximately 4.8 seconds, and the MCP summary request returned a tool error in 5.1 seconds. The subsequent timeout mitigation passed the four targeted hosted checks documented below; the latency target and nine-role limitation remain unresolved.
+
+Submit text or Markdown, up to 4,000 characters and three independent tasks. Scout describes each task as **input**, **work/steps**, **deliverable**, **criteria/evals**, and **constraints/guardrails**. Each layer distinguishes explicit information, necessary inference and missing information. Missing details are not automatically hard requirements. Steps toward one deliverable normally remain one task. Only separate model roles or incompatible endpoint needs produce additional roles, up to three per task. Requests exceeding either cap return HTTP 422 rather than silently losing work.
+
+The prompt field has no formatting toolbar or task-count hint; task decomposition happens automatically. The header shows catalogue counts without an hourly collection/status line. Source disclosures retain collection dates and refresh-failure details, and the status API still reports freshness.
+
+Each role has its own shortlist of up to five distinct models. Two roles can therefore display ten candidates, with overlap between roles. These are independent alternatives, not ten jointly evaluated combinations or a ranking of every possible model pairing. Comparable measured objectives support Pareto selection across quality, cost and latency. Dominated or unsupported candidates do not pad the list to five; unknown quality, cost and timing remain unknown. Provisional frontiers and task-fit shortlists are labelled separately. Benchmark proxies do not establish exact task accuracy or a global optimum.
+
+### API and MCP response
+
+The input remains `{ "prompt": "...", "modelIds": [...], "endpointIds": [...] }`, with both inventories optional and enforced on every role. Read `tasks[].layers` and `tasks[].slots[]`. Each slot contains `label`, `task`, `recommendations`, `ranking`, `warnings`, `specialists`, and diagnostic timings. Layers contain `text` and `origin` (`explicit`, `inferred`, or `missing`); missing text is `null`. A single-role response also retains the previous top-level result. For multiple roles, top-level `ranking.mode` is `multi-task` and `recommendations` is empty; clients must read the nested lists. MCP limits evidence excerpts per candidate as before. An ambiguous role can request clarification without discarding the other tasks in the UI.
+
+One Structured Outputs call decomposes the request and extracts scoped metadata restrictions. A single embedding batch and concurrent role-specific Decisions, eligibility and evidence checks follow. Keyword methodology retrieval supplements semantic vectors and prepared capability memberships; retrieval similarity never becomes a quality score. All roles share a 30-second server matching budget; the browser allows 35 seconds for transport overhead. A required provider failure or deadline returns unavailable, not a partially successful multi-task response. Exact-result caching remains bounded to 30 seconds and preserves evaluation dates.
+
+The under-five-second end-to-end target is **not met**. Before mitigation, the first six-case local check with real OpenAI calls returned five HTTP 503 deadline failures at approximately 4.5 seconds; only the four-task overflow case returned its expected HTTP 422. Task decomposition consumed most of the budget. Separate decomposition-only diagnostics with a longer timeout took approximately 5.5 seconds for one invoice task and 17.9 seconds for three tasks with nine roles. Those initial diagnostics did not change the serving deadline. These are local observations against the bundled snapshot, not hosted acceptance or recommended-model accuracy measurements. Fixture-based UI and contract checks alone do not establish successful live search.
+
+### Timeout mitigation — published 21 October 2026
+
+Alpha now separates the five-second performance target from a 30-second total matching timeout, with a 35-second browser timeout to allow for transport overhead. Every role still shares one deadline, and required interpretation, eligibility and evidence checks remain unchanged. Slow valid work can finish instead of being rejected at 4.5 seconds; this does not make searches faster or satisfy the latency target.
+
+The reported request to separate cats, dogs and snakes from 1,000 images returned HTTP 503 on alpha in 4.95 seconds. With the mitigation, the local browser returned five candidates in 6.23 seconds using real OpenAI calls and no result-cache hit. Local invoice and two-task checks returned candidates in 5.32 and 5.81 seconds; the same image task through MCP succeeded in 5.75 seconds. Four-task overflow still returned HTTP 422. A nine-role request still failed; a follow-up diagnostic found that a role's primary evidence capability could not be established after reconciliation. That semantic failure is not bypassed by the timeout change.
+
+After deployment, the exact image-classification request returned five candidates in the hosted browser in 7.00 seconds. Hosted invoice and two-task HTTP checks succeeded in 5.59 and 6.08 seconds, with five candidates per task; the image task through MCP succeeded in 4.75 seconds. All four checks used the existing R2/D1/Vectorize catalogue and real OpenAI calls without result-cache hits. Stable Scout was unchanged. These targeted checks establish successful search for those requests, not task accuracy, a universal success rate or compliance with the under-five-second target.
+
+### Hourly collection and coverage
+
+The collection changes below remain **unreleased**; alpha reads the existing stable catalogue. The branch retains the `7 * * * *` UTC trigger, six-way refresh concurrency and 24-job Parallel budget. Pending paid jobs resume first. Up to one quarter of the budget is reserved for six discovery categories: gateways (OpenRouter and Vercel AI Gateway); lab launches and updates; independent evaluations (Artificial Analysis and Arena); editorial leads (SemiAnalysis and Citrini); Hugging Face, hosted inference and self-hosted engines; and industry-specialist labs. Smaller budgets rotate categories. These are research starting points, not dedicated integrations or guaranteed hourly coverage of each site.
+
+Research requests cited lifecycle/expiry information, capability updates, deployment regions, fine-tuning scope and documented hardware requirements. Standard rates remain separate from promotional offers, subsidies and expiry conditions. Editorial claims supply leads rather than numerical benchmark observations. Provider-specific facts must retain their scope; OpenRouter expiry is stored on its route rather than on every deployment of a model. Unknown or unsupported facts remain unavailable. No new paid collection run has validated this expanded coverage yet.
+
+OpenAI receives the submitted request for decomposition and derived work descriptions for interpretation and embedding. Parallel receives only public research subjects. Private hosted analytics records each model role separately with the original prompt, role position and allowlisted supporting evidence; the existing sampled three-month retention applies. Do not put secrets or personal information into prompts. Keys stay server-side.
+
+For development, `npm run scout` serves the page on port 45833 and requires `OPENAI_API_KEY` for matching. It refreshes structured feeds locally each hour; it does not run the hosted Parallel/R2/D1/Vectorize publication pipeline. Local evidence retrieval uses lexical matches and prepared memberships without a Vectorize binding. Production-like validation needs an isolated Worker with the storage bindings and both `OPENAI_API_KEY` and `PARALLEL_API_KEY`. Never put their values in browser assets, Git or a chat message.
 
 ## Evidence architecture — hosted beta, 16 October 2026
 
@@ -16,7 +249,7 @@ Scout separates semantic interpretation from exact metadata checks and numerical
 | Task interpretation | Structured Outputs extracts metadata restrictions, alternatives, preferences and the downstream work. Decisions interprets that work for capabilities, operations and output serialization. No prompt keywords or provider/family exception lists are used. |
 | Eligibility | Generic typed equality, membership and range comparisons against documented facts, intersected with explicit inventories. Missing or incorrectly typed facts cannot establish a restriction. |
 | Evidence retrieval | A query embedding retrieves benchmark methodology, supplemented by hourly capability memberships. Semantic similarity supplies evidence candidates, never a model's score. |
-| Selection | All eligible verified catalogue models are considered before selecting up to five distinct models. Numerical Pareto comparisons apply only within comparable evaluation scopes with known quality, cost and latency. |
+| Selection | All eligible verified catalogue models are considered before selecting up to five distinct models. One quality estimate spans connected evaluations; Pareto comparisons require known quality, cost and latency. |
 
 Mandatory restrictions use alternatives of complete condition groups. Soft preferences affect ordering, not eligibility. Relative dates use a supplied current UTC reference; arbitrary numeric bounds are retained, including impossible requirements. An unsupported mandatory condition cannot silently disappear. A no-match result describes the available verified catalogue and inventory, not every model in existence.
 
@@ -34,7 +267,7 @@ Acceptance retained the earlier provider failures rather than counting them as s
 
 These corrections are deployed as a limited hosted beta. Hourly evidence documents retain verified research descriptions and directly selected methodology alongside feed metadata; embedding caches change when that evidence changes. The already verified descriptions are indexed directly, without a second model-generated summary; original citation excerpts remain in the source records rather than the semantic search text. Structured extraction separates the downstream work from model restrictions before Decisions interprets its capabilities and output contract. Name resolution can run alongside Decisions. This avoids interpreting model selection itself as a downstream capability; it does add a sequential interpretation dependency within the existing six-second budget.
 
-Query vectors describe the downstream work, excluding model-selection restrictions. They retrieve methodology candidates, with prepared memberships as a fallback when the query has no matches and a backstop for required capabilities missing from the retrieved set. The query order and hourly model/benchmark availability map propose up to sixteen methodology records. Decisions checks each record independently against the actual workload, without mixing unrelated methodologies in one input. Both general relevance and a deliberately scored required ability must be sufficiently supported. A broader ability can be a component proxy without testing the exact domain or answer format; incidental prerequisites cannot qualify. Ambiguous labels, broad capability overlap and unavailable checks cannot grant quality evidence. Fine-grained scope is checked against the full workload and methodology rather than asking every discovered capability again on each search. Core task/format interpretation stays bounded as the hourly directory grows.
+The initial task-relevance beta used query vectors to retrieve up to sixteen methodologies, then required both a general-relevance predicate and a scored-ability predicate for each. Although intended to admit component proxies, these overlapping questions could disagree and reject a benchmark because its domain differed from the requested task. The matching repair below replaces that assessment path; earlier acceptance results remain historical records.
 
 Classification and judgment searches consider applicable benchmark evidence before an unmeasured native endpoint contract. Without comparable candidates they retain an explicitly unmeasured task-fit fallback; contract compatibility alone does not demonstrate accuracy. Spatial pipelines keep their separate compatibility checks. Ambiguous output representations can use the existing clarification response, with model restrictions preserved. No task-name, object-name or model-specific exception is added.
 
@@ -52,6 +285,20 @@ One model confused two bird species; two invoice candidates added an extra zero
 
 Publication retained the latest completed hourly research and a healthy refreshed feed set. The final rollout check passed 9/11 cases on its first attempt: bird matching reached the deadline and explicit SVG generation returned no shortlist. Both passed one focused recheck without a matcher change. After deployment, the public HTTP suite again passed 9/11: the explicit SVG request returned no shortlist, and a Qwen-family code-authoring request admitted tool-calling evidence. Creator/family/context/date eligibility checks and impossible-context no-match passed. The live browser bird prompt displayed five recommendations without overflow. These are retained semantic and latency limitations of the beta, not a clean first-pass acceptance or proof of best-five performance.
 
+## Matching reliability repair — 17 October 2026
+
+The task Decisions batch now distinguishes the primary quality target from endpoint requirements such as image input or serialization. Retrieval and evidence assessment use that same target. Verified hourly memberships backstop the vector results even when some primary evidence was already retrieved, so an incomplete vector result cannot suppress other prepared evidence that fits the selected inventory. The existing sixteen-methodology request bound remains; this is not exhaustive methodology retrieval.
+
+If the primary target contradicts the required abilities, Scout makes one bounded Decisions reconciliation using the quoted task and previous answers. It has at most 750 milliseconds within the existing matching deadline. An unresolved contradiction returns unavailable instead of being presented as a verified no-match. This is a general consistency check, without task-name or format keywords.
+
+One Decisions batch identifies each proposed benchmark's deliberately scored ability from its own methodology and the registry's evaluation definitions. It does not require a general component benchmark to reproduce the task's exact domain, labels or output artifact. Only a recognized ability with sufficient confidence contributes proxy quality; incidental operations, unknown scope, rejected answers and unavailable checks do not. A primary-target proxy does not establish the quality of every component in a combined task or exact task accuracy. Endpoint modality, format and tool restrictions still apply independently. No model, provider or task-object exceptions are used.
+
+Exact metadata selection and query embedding start after extraction and canonical-name resolution, alongside task Decisions. Model evidence loads alongside the benchmark assessment; unaccepted observations are removed before ranking. These changes reuse the existing hourly catalogue, D1 projection and vectors, add no background service and retain the six-second matching deadline. The UI and API/MCP request inputs are unchanged. No new npm publication is included.
+
+Diagnostic stage timings overlap and must not be added together. `interpretationMs` includes the concurrently prepared eligibility/embedding result needed to complete interpretation; individual Decisions, extraction and database timings remain available in `runtime`. Cache hits retain the original evaluation time and report zero new interpretation time as before.
+
+Before deployment, full repository verification passed, including 336 Scout tests. Eight filter/ambiguity checks passed against the prepared metadata oracle: provider, family, context, date, impossible context, an incidental provider mention, an ambiguous diagram and explicit SVG source generation. The isolated Ubuntu VM run passed 203/203 assertions against staging. All seven workloads passed through both HTTP and the official MCP SDK; installed CLI email and SQL plans contained only the declared connections. An earlier staged run exposed the remaining email retrieval gap. A subsequent test setup reused an initialized directory and correctly failed `eval init`; the final isolated run used a fresh directory. Those attempts remain recorded separately. Final production acceptance also passed 203/203 VM assertions and all eight filter/ambiguity cases after the consistency correction. The HTTP workload median was 3.665 seconds versus the earlier 4.924-second sample; no final request timed out. These checks establish tested matching and integration contracts, not exact task accuracy or a latency guarantee. The [testing guide](testing.md#hosted-matching-repair-17-october-2026) retains the earlier failed rollout and separate final results.
+
 ## Latency improvements — hosted beta, 17 October 2026
 
 The hosted implementation keeps the existing task and format Decisions, exact metadata comparisons and numerical selection. Numeric and date vocabularies are omitted from interpretation input because their bounds are arbitrary; small canonical vocabularies remain in the first call. Larger vocabularies are deferred until an extracted restriction references them; a model-assisted resolution checks against the complete published vocabulary without changing unrelated conditions. Unknown mandatory names remain unresolved and cannot widen the result set. Stable reference data has an explicit provider prompt-cache boundary separate from the prompt and UTC reference time. Cache availability is measured rather than guaranteed.
@@ -141,7 +388,7 @@ The initial deployed snapshot processed all 146 benchmark series: 136 have suppo
 
 Open [Scout](https://openmerit.ai/scout/), type a task, and select the arrow button (**Find models**) or press Enter. For example: “I want to upload an invoice and get the invoice number, date, vendor, subtotal, tax, and total as JSON.” The task field grows and shrinks as you type or paste. Very long prompts scroll within the field once it reaches its height limit. Shift + Enter adds a line. When the field is empty and unfocused, its placeholder cycles through detailed example tasks every ten seconds with a brief fade. Rotation pauses while you type, while the field is focused, or while the page is hidden. With reduced motion enabled, the example stays still. Placeholders never fill the input or send a request. The heading shows live counts of API catalogue models, Hugging Face repositories, benchmark series and sources. API identities and Hub repositories are separate counts and can overlap; neither count promises a runnable endpoint for every entry. The evaluation guide remains available in the fixed bottom footer, with space reserved beneath the page content.
 
-The comparison table keeps recommendations in Scout rank order. Benchmark mode shows a relative quality signal and reference cost/time estimates. Applicable calibration contributes quality evidence within its recorded scope; its episode charges and timing do not replace the common reference workload. **Candidates to evaluate** denotes task-fit mode: quality is shown as unknown, alongside known reference costs and deployment requirements. When image separation is ambiguous, Scout first offers labels, bounding boxes or pixel masks/cutouts as output choices. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; Hugging Face results resolve the repository publisher, using a known lab mark or its bundled organization avatar; otherwise a Hugging Face mark identifies the repository platform. Missing or failed marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **API & MCP**, **Capabilities**, and **Sources** expand beneath the search and results. **Sources** uses a compact list; expand a source for its coverage, methods, limits, collection times and refresh failures.
+The comparison table keeps recommendations in Scout rank order. Benchmark mode shows a relative quality signal and reference cost/time estimates. Applicable calibration contributes quality evidence within its recorded scope; its episode charges and timing do not replace the common reference workload. **Candidates to evaluate** denotes task-fit mode: quality is shown as unknown, alongside known reference costs and deployment requirements. When image separation is ambiguous, Scout first offers labels, bounding boxes or pixel masks/cutouts as output choices. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; Hugging Face results resolve the repository publisher, using a known lab mark or its bundled organization avatar; otherwise a Hugging Face mark identifies the repository platform. Missing or failed marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **JSON API**, **Capabilities**, and **Sources** expand beneath the search and results. **Sources** uses a compact list; expand a source for its coverage, methods, limits, collection times and refresh failures.
 
 The capability index lists each area with its benchmark-series and model counts. After a search, task matches appear first with a check mark. Counts describe index coverage, not the number of models eligible for your task. The source panel describes what each feed measures, its coverage, source dates, archived results, and limitations. Collection time is separate from source dates, which can be publication dates rather than measurement dates. Scout task calibration appears separately with its tested pool, scope and seven-day expiry.
 
@@ -149,35 +396,77 @@ The capability index lists each area with its benchmark-series and model counts.
 
 ## Connect a coding agent
 
-Add this remote **Streamable HTTP MCP** endpoint to an agent that supports it:
+**Device sign-in setup — npm 0.1.7:** use the stable `latest` tag for the commands below. Hosted Scout serves `/connect` and `/scout/mcp` independently of npm. Claude Code is the only supported client; other agents are not yet supported. The older `0.1.7-preview.5` package does not contain these commands.
 
-```text
-https://openmerit.ai/scout/mcp
-```
+Setup takes four steps, and you act in each one only once:
+
+1. **Click Add to Claude Code** on the Scout page. It copies a short setup prompt, shows **Copied to clipboard** above the button and opens a `claude://code/new?q=…` link. Your browser may ask once whether to open the Claude desktop app. Desktop opens a new Code session with the prompt filled in; it never sends the prompt for you.
+2. **Send the prompt in Claude**, then allow the command it asks to run. The prompt asks it to run one command, with a 10-minute timeout:
 
-Scout exposes one read-only tool, `scout_models`, with a required prompt:
+   ```sh
+   npx -y openmerit@latest install --client claude-code
+   ```
 
-```json
-{"prompt":"Upload an invoice and extract six fields as JSON"}
+3. **Click Approve** on `https://openmerit.ai/connect`, which the command opens for you. Sign in first if needed. The page shows the client, the device name, the scope, an eight-character code that matches the command's output, and the only permission granted: Scout recommendations through MCP. Only approve a request you started yourself.
+4. **Done.** The command has kept waiting, so nothing needs rerunning. It stores the device key, adds the `openmerit` MCP server with `claude mcp add-json`, writes the Scout skill and checks the connection with a real MCP `initialize` and tool discovery (not a search). The approval page switches to *Claude Code is connected* once the command has collected its key, and Claude Code reports the result. Scout is available in new Claude Code sessions; in a session that is already open, reconnect `openmerit` in `/mcp`.
+
+The prompt contains no key, account data or task draft. If the Claude desktop app isn't installed or your browser blocks the link, paste the copied prompt into Claude Code instead. The command needs `npx` (Node.js 22.19 or newer) and the `claude` CLI on the PATH of that session; the installer runs `claude mcp add-json`. Having the desktop app alone does not establish that those commands are available. See [Claude Desktop](https://code.claude.com/docs/en/desktop).
+
+When the command can open a browser, it waits up to nine minutes for the Approve click. When it can't, for example over SSH, it prints the link and code and returns after 30 seconds with exit code 10 so the agent can show them. Rerunning the same command resumes the same request; it does not issue a new code. Codes expire after ten minutes and can be used once. `OPENMERIT_LOGIN_WAIT` sets the wait in seconds.
+
+`--client` defaults to the agent running the command (Claude Code sets `CLAUDECODE=1`). `--scope` defaults to `user`, which makes Scout available in every project. `--scope project` writes the skill to `.claude/skills/` in the current project and stores the MCP server in Claude Code's private per-project (`local`) configuration, so the key never enters the repository. `--plan` lists the changes without making them. `--json` prints one result object for agents.
+
+| Exit code | Meaning |
+|---|---|
+| 0 | Done |
+| 10 | Approval pending; open the link, approve, rerun |
+| 11 | Denied, expired or rejected key; rerun for a new code |
+| 20 | Conflict; nothing was changed |
+| 30 | OpenMerit unreachable |
+| 1 | Other error |
+
+**Conflicts are never overwritten.** Install stops if Claude Code already has an `openmerit` server it did not create, if an installed file was edited, or if `openmerit` exists in another Claude Code scope or `.mcp.json`. An install record stores hashes of what it wrote, so rerunning repairs missing files and updates unchanged ones. An unchanged rule file from the earlier source preview (`rules/openmerit.md`) is removed; an edited one is left alone.
+
+### Device keys
+
+Each approval creates one key for one device and client, limited to `scout:recommend`. It is accepted only on `/scout/mcp`; the JSON API and browser still require a Clerk session. The server stores only SHA-256 hashes of keys and device codes. A key expires after 90 days without use; each use extends it. The usual per-user rate limits and analytics preferences apply.
+
+On the device, the key is stored in `~/.config/openmerit/credentials.json` (mode 600; `OPENMERIT_CONFIG_DIR` or `XDG_CONFIG_HOME` override the location) and in Claude Code's MCP header configuration, which Claude Code stores in plain text. The CLI never prints the key and scrubs it from any failing `claude` output. Do not paste keys into a chat.
+
+Revoke devices under **Connected devices** on `/connect`. `openmerit remove` and `openmerit logout` also revoke this device's key.
+
+### Commands
+
+```sh
+openmerit install [--client claude-code] [--scope user|project] [--plan] [--json] [--no-open]
+openmerit status  [--client claude-code] [--scope user|project] [--check] [--json]
+openmerit remove  [--client claude-code] [--scope user|project] [--plan] [--json]
+openmerit login   # replace an expired or revoked key, then rerun install
+openmerit logout  # revoke and delete this device's key
+openmerit project --policy existing-only|existing-first [--endpoint-id <id> ...] [--plan]
+openmerit doctor  # Node.js, claude, reachability, key, MCP check and install state
 ```
 
-No API key is required from the caller. Scout sends the task description to OpenAI for Decisions interpretation, structured metadata extraction and a transient query embedding, then compares its collected evidence. It does not execute the recommended models. The MCP endpoint supports protocol versions `2025-03-26`, `2025-06-18`, and `2025-11-25`, including initialization, tool discovery, and tool calls. It uses JSON responses without sessions or a long-lived SSE stream. Browser requests must have the same origin; ordinary agent clients omit the Origin header. An unsupported protocol version is rejected. Agent configuration is client-specific; connecting an endpoint does not automatically make an agent use it.
+Removal deletes the MCP server, the unchanged skill and the install record, and revokes the key. It keeps edited files and `.openmerit/scout.json`.
+
+### Using Scout from the agent
 
-Tell the agent: “Before choosing an inference model for this feature, call Scout with the task description. Compare the five candidates, read their evidence and warnings, and evaluate the shortlist on our own examples.”
+The `openmerit-scout` skill loads when an application inference model is being chosen, even if you do not mention Scout. It is not loaded for unrelated edits, ordinary inference calls or the agent's own model. On first use in a project, it asks once whether the project is **existing-only** or **existing-first** and which public catalogue endpoint IDs it can reach, then saves them with `openmerit project`. Every search passes the saved `endpointIds`, including `[]`. Existing-only never broadens; existing-first asks before an unscoped search. Task descriptions go to Scout and OpenAI; do not send source files, secrets or customer records.
 
-The MCP response includes up to four relevant evidence entries per model and the total evidence count. Evidence satisfying required memory, tool-calling, and visual-document capabilities is shown first. The JSON API exposes the complete evidence list. When specialist discovery finds matches, MCP includes up to three separate candidates, four reported evaluation records per candidate, and each candidate’s total indexed evaluation count.
+The agent compares model and endpoint, evidence, reference cost/time and unknowns, and helps configure the chosen route. New providers, credentials, spending and deployments need separate approval. A skill is a default, not enforcement. The MCP tool, `scout_models`, is read-only and does not execute models.
 
 ## JSON API
 
 ```sh
 curl https://openmerit.ai/scout/api/recommend \
+  -H "Authorization: Bearer $SCOUT_TOKEN" \
   -H 'Content-Type: application/json' \
   -d '{"prompt":"Upload an invoice and extract six fields as JSON"}'
 ```
 
-`POST /scout/api/recommend` requires `prompt`: a non-empty string of at most 4,000 characters. The JSON body is limited to 18,000 bytes. Unknown properties are rejected.
+`POST /scout/api/recommend` requires a current Clerk session bearer token and `prompt`: a non-empty string of at most 4,000 characters. The JSON body is limited to 18,000 bytes. Unknown properties are rejected.
 
-HTTP and MCP additionally accept optional `modelIds`, a unique list of up to 100 legacy OpenRouter catalogue IDs. This preserves the existing-project inventory contract; a model ID alone does not grant a direct-provider endpoint. To select specific routes instead, use `endpointIds`, a unique list of up to 100 IDs returned in `recommendations[].endpoint.id`. Both filters intersect if supplied. This filters eligible candidates before ranking; an empty or unknown-only list returns no models. The response echoes `selection.modelIds` and `selection.endpointIds` (`null` for unscoped). Scoped requests omit specialist discovery because it is outside the declared inventory. No endpoint URL or credential is accepted. The [agent evaluation runner](agent-evaluations.md) uses this for existing-project scope; the hosted API supports it as of 15 October 2026. The npm evaluation commands are available in `openmerit@0.1.7-preview.3` under the `preview` tag, separate from stable `0.1.6`.
+The JSON API additionally accepts optional `modelIds`, a unique list of up to 100 legacy OpenRouter catalogue IDs. This preserves the existing-project inventory contract; a model ID alone does not grant a direct-provider endpoint. To select specific routes instead, use `endpointIds`, a unique list of up to 100 IDs returned in `recommendations[].endpoint.id`. Both filters intersect if supplied. This filters eligible candidates before ranking; an empty or unknown-only list returns no models. The response echoes `selection.modelIds` and `selection.endpointIds` (`null` for unscoped). Scoped requests omit specialist discovery because it is outside the declared inventory. No endpoint URL or credential is accepted. The [agent evaluation runner](agent-evaluations.md) uses this for existing-project scope; the hosted API supports it as of 15 October 2026. The npm evaluation commands are available in `openmerit@0.1.7-preview.3` under the `preview` tag, separate from stable `0.1.6`.
 
 Recommendation `id` preserves legacy router IDs; a new protocol receives a distinct catalogue ID. `modelId` is the canonical model identity, while `endpoint.modelId` is the identifier to send to that provider. Clients must use the returned protocol and deployment requirements rather than treating every option as a chat-completions model. Existing npm clients using legacy `modelIds` remain scoped to their router inventory.
 
@@ -194,9 +483,9 @@ The response contains:
 - `ranking`: algorithm version, `mode` (`measured`, `benchmarks`, `task-fit`, or `clarification`), workload and tradeoff weights. A measured `profile` gives the task type, sample size, eligible tested pool, protocol hash, collection and expiry times, and public evidence URL. `benchmarkEligibleModels` reports the separate public-benchmark pool. Rank is the final shortlist order. `ranking.frontier` reports compared configurations, selected distinct models and whether the objective scope is comparable or partial. The compatibility `score` can be `null`; there is no weighted quality/cost/time composite in frontier selection.
 - `warnings`, `methodology`, `interpretationMs`, `elapsedMs`: limitations, ranking method, Decisions request time, and measured server ranking time when the runtime provides a sufficiently precise clock (`null` otherwise). Cloudflare freezes its clock during synchronous work, so the hosted page reports measured HTTP round-trip time instead of claiming zero-millisecond computation.
 
-`pricing.url` is the official provider destination, not the source of the quoted rates. It is `null` for a provider without a reviewed destination; the page then omits the link. The same rule applies to MCP responses. `pricing.basis` identifies the price source and calculation assumptions.
+`pricing.url` is the official provider destination, not the source of the quoted rates. It is `null` for a provider without a reviewed destination; the page then omits the link. `pricing.basis` identifies the price source and calculation assumptions.
 
-`GET /scout/api/status` returns index coverage, source timestamps, freshness, and scheduler state without submitting a prompt. API responses are `no-store`. Errors return JSON with `error`: 400 for invalid input, 413 for an oversized request, 415 for a non-JSON body, 405 for an unsupported method, 404 for an unknown endpoint, 429 when shared admission limits are reached, and 503 if interpretation is unavailable. A failed, malformed, refused, or timed-out Decisions response never silently falls back to keyword ranking. MCP reports these failures as tool errors.
+`GET /scout/api/status` returns index coverage, source timestamps, freshness, and scheduler state without submitting a prompt. API responses are `no-store`. Errors return JSON with `error`: 400 for invalid input, 413 for an oversized request, 415 for a non-JSON body, 405 for an unsupported method, 404 for an unknown endpoint, 429 when shared admission limits are reached, and 503 if interpretation is unavailable. A failed, malformed, refused, or timed-out Decisions response never silently falls back to keyword ranking.
 
 Scout recommends models; it does not process an uploaded invoice, execute a task, install a model, or select a provider account.
 
@@ -209,6 +498,7 @@ The hourly updater fetches eight independent public sources, including Models.de
 | [Epoch AI](https://epoch.ai/benchmarks) | Benchmark definitions, published results, source links, and evaluation configurations. |
 | [Artificial Analysis](https://artificialanalysis.ai/leaderboards/models) | Benchmark results, measured first-answer delay and reference generation speed. Archived evaluations remain labelled when their models are still available. |
 | [OmniDocBench](https://github.com/opendatalab/OmniDocBench#end-to-end-evaluation) | Document parsing scores from the explicitly labelled `v1.6_full` leaderboard, including newer document models than the OpenVLM snapshot. |
+| [IDP Leaderboard](https://www.idp-leaderboard.org/) | Nanonets' MIT-licensed document processing leaderboard: OlmOCR, OmniDocBench v1.5 and IDP Core (key information extraction, handwriting, tables, visual QA) component scores. The overall average is not imported; per-model dates are not supplied. |
 | [OpenCompass / VLMEvalKit](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) | Visual understanding and OCR benchmark results. Its feed currently identifies itself as 17 September 2025. |
 | [Berkeley Function Calling Leaderboard (BFCL v4)](https://gorilla.cs.berkeley.edu/leaderboard) | Native function-calling single-turn, live, and multi-turn scores; separate memory scores for key-value, vector, and recursive-summarization stores. |
 | [OpenRouter](https://openrouter.ai/api/v1/models) | Available model IDs, input modalities, declared tools and structured-output support, context limits, and advertised token prices. |
@@ -249,17 +539,17 @@ For performance ranking, a query embedding retrieves benchmark methodologies, su
 
 Quality comparisons use shared benchmark results, grouped by correlated family and task capability. Different cohorts are not compared through unrelated percentiles. BFCL uses native function-calling configurations with their original labels; correlated series are one family. Missing scores remain unknown and actual zero remains a score. Model/version matching remains conservative. A relative quality signal is not predicted accuracy or a statistical confidence interval.
 
-Matching calibration observations can supplement quality for their exact operation, format, modality, tools, retention and workload scope. Each admitted model needs at least six calibration cases, and evidence expires after seven days. A match no longer requires five tested models or excludes all untested eligible models. Observations remain tied to their tested router deployment. Repeated calibration request failures still cap effective quality. Calibration episode bills and durations remain source evidence; they are not substituted for only some candidates' reference costs or times. Held-out outcomes and answer keys never enter runtime ranking.
+Matching calibration observations can supplement quality for their exact operation, format, modality, tools, retention and workload scope. Each admitted model needs at least six calibration cases, and evidence expires after 30 days (seven days before 22 October 2026). An applicable profile past that age stays excluded and the result carries a warning naming the profile and its expiry date, rather than silently using benchmarks only. A match no longer requires five tested models or excludes all untested eligible models. Observations remain tied to their tested router deployment. Repeated calibration request failures still cap effective quality. Calibration episode bills and durations remain source evidence; they are not substituted for only some candidates' reference costs or times. Held-out outcomes and answer keys never enter runtime ranking.
 
-All eligible verified catalogue configurations with relevant evidence enter numerical comparison before model deduplication. A configuration can dominate another only within a comparable quality scope, with known quality, cost and time: it must be at least as good in all three and strictly better in one. Missing objectives and incomparable coverage remain provisional. This cannot establish a global three-objective frontier over every model in the world.
+All eligible verified catalogue configurations with relevant evidence enter numerical comparison before model deduplication. A configuration can dominate another only when both have a common quality estimate and known cost and time: it must be at least as good in all three and strictly better in one. Missing objectives and incomparable coverage remain provisional. This cannot establish a global three-objective frontier over every model in the world.
 
-Up to five distinct models represent the supported tradeoffs. Explicit priorities and soft metadata preferences order eligible frontier points, followed by supported quality and known cost/time. Default selection reserves no cheapest or fastest slot. Preferences do not broaden eligibility. There is no weighted quality/cost/time composite. Fewer than five points can remain even when more models are eligible; the API reports eligibility separately from the selected count. Stable IDs resolve exact ties.
+Up to five distinct models represent the supported tradeoffs. Balanced searches keep the strongest, lowest-cost and fastest qualifying frontier points; explicit priorities order the frontier instead. Soft metadata preferences break exact ties. At most two models per provider are selected while alternatives remain. Preferences do not broaden eligibility. There is no weighted quality/cost/time composite. Fewer than five points can remain even when more models are eligible; the API reports eligibility separately from the selected count. Stable IDs resolve exact ties.
 
 Reference input sizes are **1,024 / 4,096 / 32,768 tokens**, and visible output sizes are **128 / 512 / 2,048 tokens**, for short / medium / long. Supplied-label choices use a 16-token output reference. Reference cost is input tokens × advertised input rate + output tokens × advertised output rate. This excludes hidden reasoning, cache effects, image-specific tokenization and other modality charges. Decisions has no output-token charge. Unknown or self-hosted operating cost is not inferred to be zero.
 
-Reference completion time is **time to the first answer token + visible output tokens / throughput**, only when measurements identify the selected endpoint. Multiple published configurations retain paired timing before taking a median. Timing is not borrowed across provider routes, and early reasoning tokens are not confused with the first answer. Missing timing stays unknown. The page's request duration measures Scout matching, not a recommended model executing the visitor's task.
+Reference completion time is **time to the first answer token + visible output tokens / throughput**. Endpoint-bound measurements take precedence; otherwise an unbound model-level median applies and is labelled `speed.scope: "model"`. Multiple published configurations retain paired timing before taking a median. Timing bound to one route is not borrowed by another, and early reasoning tokens are not confused with the first answer. Missing timing stays unknown. The page's request duration measures Scout matching, not a recommended model executing the visitor's task.
 
-The shortlist requires evaluation on the visitor's own examples. Public benchmarks, small calibration samples and semantic applicability checks do not prove the optimal membership or order of five models for an arbitrary task. There are no visitor accounts, public prompt history or additional filter controls. Public methodology embeddings are stored in Vectorize; visitor query vectors are transient. Private prompt-and-evidence analytics are described below.
+The shortlist requires evaluation on the visitor's own examples. Public benchmarks, small calibration samples and semantic applicability checks do not prove the optimal membership or order of five models for an arbitrary task. The hosted release uses Clerk accounts for sign-in but adds no public prompt history or additional filter controls. Public methodology embeddings are stored in Vectorize; visitor query vectors are transient. Private prompt-and-evidence analytics are described below.
 
 ## Current validation limits
 
@@ -312,15 +602,11 @@ The first test exposed cross-task request failures; those test outcomes were not
 
 ## Refresh and data handling
 
-The task description is sent to Scout and then to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, a transient query embedding and, when relevant, specialist compatibility/applicability checks. Hourly preparation also sends public repository metadata and benchmark definitions/methodology to Decisions; no user prompts are involved in that refresh. Scout stores the submitted prompt alongside its recommendations in private Cloudflare Analytics Engine records. Runtime request logging remains disabled. The public index, published calibration aggregates and the limited search analytics described below are persisted; prompts are not sent to benchmark sites or recommended-model providers. [OpenAI's API data controls and retention policies](https://developers.openai.com/api/docs/guides/your-data) apply to those provider calls. Scout does not claim zero retention by OpenAI. Hosting infrastructure can have separate request-metadata policies. Clipboard actions occur only when selected. Describe the task rather than pasting actual invoices, credentials or private customer records.
+The task description is sent to Scout and then to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, a transient query embedding and, when relevant, specialist compatibility/applicability checks. Hourly preparation also sends public repository metadata and benchmark definitions/methodology to Decisions; no user prompts are involved in that refresh. New search analytics use [PostHog](#product-analytics), with the separate browser/API content choices and beta defaults described above. Runtime request logging remains disabled. The public index and published calibration aggregates are persisted; prompts are not sent to benchmark sites or recommended-model providers. [OpenAI's API data controls and retention policies](https://developers.openai.com/api/docs/guides/your-data) apply to those provider calls. Scout does not claim zero retention by OpenAI. Hosting infrastructure can have separate request-metadata policies. Clipboard actions occur only when selected. Describe the task rather than pasting actual invoices, credentials or private customer records.
 
-Scout records one server-side Analytics Engine event for each valid recommendation attempt on the website/JSON API or MCP. It keeps the original submitted prompt (up to 4,000 characters, including whitespace), fixed task categories, capability/input/output requirements, priority and workload buckets together with the ordered model/endpoint shortlist (up to five), its quality/cost/time values and their calibration/reference/unknown basis, attached source evidence, ranking and interpreter versions, index date, scope category, freshness, outcome and server duration. Clarifications, unsupported tasks, empty results, admission limits and interpretation errors are distinct outcomes. A failure before interpretation has an unknown task category. Invalid requests rejected before recommendation, status checks and MCP discovery are not search events. Website and other JSON API calls share the `api` channel; MCP uses `mcp`.
+**Historical Cloudflare archive:** before the 22 October consolidation, Scout stored submitted prompts with task categories, ordered shortlists, metrics and supporting public evidence in private Analytics Engine records. New writes have stopped. Historical prompt text is unredacted and may contain personal information or secrets; no separately collected account/visitor identity was attached. These records remain accessible only through authenticated Cloudflare account queries and are neither exported nor erased by the PostHog integration.
 
-Apart from the submitted prompt and allowlisted public source evidence, analytics excludes free-text task summaries, prompt hashes, requested inventory IDs, endpoint API addresses, error messages, IP addresses, user agents, cookies, visitor IDs and session IDs. Prompt text is not redacted: personal information or secrets typed into it would be retained, so these records are not guaranteed to be anonymous. No visitor identity is collected separately or attached to a search. It adds no browser tracker or inference call. Analytics writes are best effort and cannot interrupt recommendations. Cloudflare [retains Analytics Engine events for three months](https://developers.cloudflare.com/analytics/analytics-engine/limits/) and may sample them; this is usage evidence, not an exhaustive audit log or evidence of task success. Hosting infrastructure and the existing Decisions provider call have their own data policies.
-
-Search records are private Cloudflare data, queried on demand through authenticated account access. Scout has no report generator, analytics dashboard, analytics sign-in, or public history/query endpoint. Each new record keeps the submitted prompt, the ordered five (or fewer), and a snapshot of their supporting evidence together: benchmark scores, source links/dates and the prepared capability/family interpretation version/time, applicable calibration measurements and protocol, advertised pricing, speed sources, and endpoint/model-card task-fit evidence, including the prepared semantic profile and compatibility decision for a selected Hub deployment. Capturing source facts at recommendation time avoids relying only on links whose contents can change. Unknown measurements remain unknown, and reported model-card results remain unverified claims.
-
-Earlier records are preserved as collected: version 1 has no prompt, and versions 1–2 have no attached evidence snapshot. Missing historical fields cannot be reconstructed. New records use lossless compression when needed to fit Cloudflare's event-size limit. Writes remain best effort; a record too large to fit is not silently truncated or presented as complete.
+Cloudflare [retains Analytics Engine events for three months](https://developers.cloudflare.com/analytics/analytics-engine/limits/) and may sample them; that policy does not apply to PostHog. Earlier records are preserved as collected: version 1 has no prompt, and versions 1–2 have no attached evidence snapshot. Missing historical fields cannot be reconstructed. Neither historical analytics nor current recommendation traces prove task success. Scout has no public history/query endpoint.
 
 Each query starts with batched Decisions questions and can make one verification pass for weak answers, followed by relevant specialist compatibility batches. Task interpretation and specialist comparison share a four-second request budget; catalogue reads and ranking add their own time. Optional verification has at most 750 milliseconds and leaves weak answers uncertain if unavailable. Independent specialist batches run in parallel, with at most 100 questions per call. Transient connection, timeout, rate-limit and server failures can receive one retry within the remaining deadline; valid `Retry-After` delays are honored. Authentication, exhausted-credit and quota failures are not retried. Failed interpretation returns an explicit unavailable response; failed specialist checks omit unverified specialist candidates. No keyword fallback is used. These bounds reduce avoidable waits and failures; they do not guarantee a successful search or a particular latency. In-memory admission allows 32 active requests and 600 starts per minute per Worker isolate; a separate edge limiter allows 60 POST requests per minute per IP. Excess requests fail promptly instead of queuing. These are operational limits, not an account-level billing cap or a throughput guarantee. Credentials remain server-side.
 
@@ -347,6 +633,10 @@ Open `http://127.0.0.1:45833/scout/`. This lightweight local preview uses the sh
 
 Publication records a durable completion receipt after updating the active pointer, pending-research queue and health status. Interrupted publication resumes unfinished work; a lost success reply can be acknowledged again after the lease is released. Replaying a completed publication cannot roll back a newer catalogue or clear a later refresh's lease. Completed research receipts leave the queue only when publication commits; unfinished research retains its original paid job for a later hourly run.
 
+Research rediscovery updates an existing record with the same stable primary key, even when semantic interpretation does not link it to another catalogue record. Different source/version identities remain separate unless the evidence supports a semantic link. Duplicate primary keys fail publication validation; searches continue against the last committed catalogue while a later refresh reuses durable research receipts.
+
+The 17 October post-deployment recovery reused all 24 saved research jobs and published successfully after this identity correction, with a matching completion receipt and empty backlog. Memory-limit and vector-index readiness retries occurred during that run. Its fresh npm VM acceptance also exposed unavailable books/posters and invoice HTTP responses, empty SQL and safety-rewrite shortlists, and empty installed-CLI plans. These remain beta limitations; the detailed results are in the [release validation record](testing.md#preview4-release-checks-17-october-2026).
+
 Prepared capability pools can overlap. Searches use their prepared ID lists to choose a smaller covering set of pools, preserving the selected candidates and complete evidence while avoiding repeated downloads of the same model records. Explicit model and endpoint inventories retain their existing scope checks. These metadata reads do not fetch source pages or redo hourly research.
 
 `demo/domain-router.mjs` routes Scout paths to this service while retaining the existing website, protected memo, authentication, and terminal services. There is no Scout admin/report route. The public guide is generated and published with the main documentation. Do not deploy the older root `wrangler.jsonc` to publish Scout. `npm run build:scout` packages only Scout, public documentation, and its shared assets; private memo and demo applications are excluded.