openmerit 0.1.10 → 0.1.12

package/docs/scout.md

@@ -1,16 +1,92 @@
 # Scout by OpenMerit
 
-Describe what you need a model to do. Scout returns up to five model-and-endpoint candidates per task. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page and JSON API use the same index and ranking.
+Scout is OpenMerit's model search agent. Describe your task and get model recommendations. Scout returns up to five model-and-endpoint candidates per task. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page and JSON API use the same index and ranking.
 
-Scout is a separate hosted beta product. Npm 0.1.7 includes optional Claude Code setup for it; the hosted service does not change the Pi coordinator or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
+Scout is a separate hosted beta product. Npm 0.1.7 introduced optional Claude Code setup for it. OpenMerit 0.1.12 adds native Scout access for Pi 1.5.0, replacing the old Pi evaluation adapter without changing the core coordinator, standalone evaluation CLI or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.
 
 **Website rebuild — published 24 October 2026:** the rebuild preserves the existing landing page at `/` and Scout at `/scout/`, and takes all hosted `/docs/` pages offline, including this guide. `/connect/`, the JSON API/MCP and the standalone `/scout/data-policy/` remain available. The guide remains maintained in Markdown; this does not remove the local CLI or its documentation from the package. See the [cutover notes](site-domain-migration.md) for publication and verification details.
 
 **Coding-agent setup — 24 October 2026:** stable npm 0.1.7 includes [device sign-in setup](#connect-a-coding-agent). Hosted Scout serves `/connect` and `/scout/mcp`; the button opens Claude Desktop with a setup prompt. This supersedes the 23 October web-placeholder reset. Existing older client configuration and authorization grants are not automatically removed or revoked. Scout serving, main-site documentation and npm publication remain separate releases; dated receipts below describe their original versions, not acceptance of every later build.
 
+**Session restoration — unreleased source:** onboarding stays hidden while sign-in is checked, so returning signed-in visitors do not see the introduction flash on reload or reopening. Chat remains locked until an active session is confirmed. Signed-out visitors and failed checks reveal the introduction with sign-in still available; signing out restores it. No saved tour-completion flag grants access. This is not yet deployed and changes no homepage FAQ claim.
+
+**Session tokens during agent turns — unreleased source:** each progress poll asks the sign-in client for a current session token instead of reusing the token captured at the start of the turn. This lets an active session renew its short-lived token while Scout works, including when polling retrieves an answer after the initial connection fails. It does not extend an expired or revoked session, resubmit the task or bypass server authentication. A browser-platform regression fixture covers token renewal and answer recovery; the fix is not yet deployed. No homepage FAQ claim changes.
+
+**Product wording — unreleased source:** the homepage, Scout page, onboarding and search/social descriptions call Scout “OpenMerit's model search agent,” with “model search” as the shorthand and “Find the right AI models for your task” as its promise. “Model recommendations” describes its output; the web and coding agents are access channels, not separate product categories. This copy is not yet deployed. Dated receipts below retain the wording and behavior of their original releases.
+
+## Five-step introduction modal — deployed, 28 October 2026
+
+**Deployment receipt:** the owner approved commit, push and deployment. [PR #28](https://github.com/laz-aslam/openmerit/pull/28) merged as [a70103a](https://github.com/laz-aslam/openmerit/commit/a70103a1c8eb8a701ee262106ae6e6c026cf04b4). All PR and merged-commit CI checks passed. The guarded Scout-serving deploy moved `openmerit-scout-alpha` from `2cf2c2d3-8aef-40f0-9c44-800bcab247bf` to `17ab0166-9a5c-43e9-932b-c13bee040db7` at 20:45 UTC. Live Scout HTML, chat/auth/onboarding JavaScript, styles and Claude screenshot matched the release build. `/` and `/connect/` stayed byte-identical; other Workers were not redeployed. The catalogue remained healthy on snapshot `588855a0bf28212fad48a9513dbf3d5b`.
+
+Live Chrome desktop/mobile and WebKit mobile checks covered all five slides, locked future navigation, revisiting reached slides, real Google/GitHub sign-in controls with optional analytics blocked, dismissal keeping the page inert, signed-out reload returning to slide one and zero agent turns during the tour. Browser checks with simulated sign-in/replies covered all three example tasks, exact submitted messages, single submission after double activation, keyboard activation and New restoring examples; auth fixtures covered resumption after sign-in. Completed production OAuth, an authenticated production recommendation, native Claude installation and physical-phone behavior remain unverified.
+
+Signed-out visitors see a five-step introduction modal over the actual Scout page, with the page dimmed and softly blurred behind it. The first is a personal note signed by Lazim and Praveen: Scout is OpenMerit's model discovery product in early beta, and feedback on what is useful, confusing or could improve is welcome. This is a text-only note, not a feedback form or a new feedback channel. The second introduces Scout with bundled lab/Hub marks and coverage from the same optional status request as the page: indexed models, benchmarked labs, benchmarks and distinct source-feed IDs. Hub repositories appear separately because they can overlap the catalogue. Labs count model publishers with benchmark results, not every inference host; source feeds include research records, not distinct organizations. Failed or invalid status responses leave a general coverage description, never fabricated counts, and do not block the tour or sign-in.
+
+The third explains that Scout recommends rather than routes or resells inference tokens. Its illustration separates a recommendation from the visitor's app → provider/router → model runtime path. Recommendations can be constrained to the public endpoint inventory supplied through the [coding-agent project configuration](#using-scout-from-the-agent) or API; the slide does not import accounts, collect provider keys, connect endpoints or introduce a web configuration form. The fourth explains the task-to-shortlist workflow with an illustrative conversation; its model order, quality estimates and prices are example values, not a live recommendation or measured task accuracy. The fifth explains the optional Claude Code integration and shows cropped excerpts of an actual session, mentioning MCP, API and CLI availability. Its historical model recommendation is not a recommendation for the current visitor. The full-size capture opens separately. Viewing any step does not submit a task, run inference, install Scout or launch Claude.
+
+**Continue** advances through the first four steps; **Get started** on the fifth opens the existing Google/GitHub sign-in flow above the introduction. The five progress controls revisit only slides already reached through Continue; future slides are disabled and excluded from keyboard navigation. Going back preserves access to all reached slides, including those ahead of the current slide, for the current page visit. There is no separate back button. The modal has no separate header sign-in or footer navigation, and the conversation/Claude previews have no supplementary captions. The conversation preview's title bar uses decorative red/yellow/green window lights instead of the Scout wordmark and example label; these do not close, minimize or maximize the modal. Its accessible image description still identifies the values as illustrative. The underlying page stays visible but inert until Clerk reports an active session, including after an OAuth return; a dismissed, failed or pending sign-in never enables it. The introduction cannot be dismissed to bypass this gate. Successful onboarding sign-in dismisses the introduction and focuses chat without automatically sending the example. Existing explicit draft-submission resumption remains separate. Returning active sessions bypass the introduction. Signing out restores the modal and makes the page inert again.
+
+Tour progress stays in page memory: signed-out reloads and new page visits start at the first slide, even after opening or dismissing sign-in. The tour does not read or write a completion marker; old `scout-onboarding-v1` markers are ignored. Reloading with an active authenticated session still bypasses the introduction. Blocked storage does not prevent navigation or sign-in. Privacy controls and the data policy remain directly accessible without authentication at `/scout/privacy/` and `/scout/data-policy/`; their links stay in the main page footer, not the modal. No onboarding-specific analytics event is added; existing page/sign-in analytics and preferences remain effective.
+
+After sign-in, the empty chat shows three task examples under **Try these**: support-ticket summarization, invoice extraction and product-photo classification. Selecting one immediately submits that task through the normal composer/auth flow and starts a conversation; no separate Send action is needed. If the session has expired, the task stays as a draft while sign-in opens and resumes through the existing authenticated-submit flow. The examples disappear when the first conversation turn appears; **New** clears the conversation and shows them again. They remain inside the chat surface, above the fixed composer, and cannot bypass the onboarding/auth gate. The homepage makes no affected example-task claim and needs no FAQ change.
+
+Pointer navigation uses a short, non-looping opacity/position transition. Keyboard navigation and reduced motion change steps without movement. Keyboard focus stays inside the introduction, yielding to the sign-in modal while it is open. On narrow and short screens, the introduction scrolls within its rounded panel; the underlying page does not scroll. The homepage makes no affected onboarding or sign-in claim and needs no FAQ copy change.
+
+Design references: [Linear's welcome screen](https://mobbin.com/screens/62773d0f-3574-4191-ac91-eb8a51697a95), [Linear's onboarding flow](https://mobbin.com/flows/cbaf58a9-fd84-4afe-b214-e65b24484013), [Notion's focused setup screen](https://mobbin.com/screens/99396b7b-c21d-4cea-8956-4942c2b9aae6), [Neon's integration walkthrough](https://mobbin.com/screens/8b8b58fe-86fd-4a2a-a5ad-51b84614d7b2), and [Torph's text-continuity examples](https://torph.lochie.me/). The implementation keeps Scout's own graphite palette and system typography; it does not embed Mobbin or generated mockup assets or add an animation dependency. The Claude Code image comes from the owner's 24 October recorded session: two unaltered content crops of `claude-automatic-expanded-full.png`, `(1330, 1029)–(2810, 1103)` and `(1330, 1552)–(2810, 1770)`, stacked with a 42-pixel neutral gap and encoded as WebP. OS chrome, account information and unrelated session content are excluded.
+
+## Every model kind in the agent — source only, not deployed
+
+This describes source changes that are not yet published or deployed. Until both the publisher and the serving Worker are deployed and a refresh has published a new catalogue version, the agent behaves as described in the next section.
+
+Each published catalogue version will include every model in the merged catalogue, not only chat models: embedding, rerank, image, video, speech (text to speech), transcription (speech to text), decision, OCR, moderation, 3D and music models, plus models whose kind is unknown. Each model carries its kind, input and output modalities, context window, release date, open-weights status and up to 16 catalogue routes with their current base-tier prices. Models also carry their linked leaderboard results from the media, speech, decision and embedding leaderboards Scout imports (Arena, Artificial Analysis, TTS Arena, Open ASR, 3D Arena, VBench, MMEB and others), and from MTEB's final leaderboards: metric, whether higher or lower is better, score, rank out of every result on that leaderboard version, leaderboard version, date and source. Chat models keep Scout's reviewed routes and benchmark results unchanged. The agent reads only this published copy, so a refresh that is still running never changes what one conversation sees between search and the shortlist check.
+
+The agent's tools change as follows:
+
+- **search_catalogue** (new) searches models, benchmarks and leaderboards by free text. By default it combines keyword (BM25) and embedding results by reciprocal rank fusion; it can also use either alone, and filter by result type and model kind. Each result names the field that matched. Without embeddings (no OpenAI key, an embedding error, or a catalogue version published without them) it returns keyword results and says so. Its queries, written by the agent from your task, are embedded with OpenAI, as benchmark search already does.
+- **search_models** filters by model kind and reports the cheapest eligible price in each unit a model is billed in: per million tokens, per image, per second of audio or video, per million characters, per page or per request. Sorting by price compares one unit only (by default the usual unit for the requested kind, otherwise tokens); models not billed in that unit follow, unsorted. A route that lists only zero prices, such as a local runtime or a free tier, is not counted as a price.
+- **get_model_details** and **benchmark_leaderboard** include leaderboard results, and leaderboard IDs work wherever benchmark IDs do.
+- **submit_recommendation** accepts models of any kind. Fewer than five are accepted only when fewer models meet the stated hard constraints, which can now include the model kind. The per-task workload is given in billing units (input and output tokens, images, seconds of audio or video, characters, pages, requests) and cost per 1,000 tasks is computed from catalogue prices on the eligible route that prices most of the workload, then the cheapest. Quantities the route does not price are listed, and such a partial cost never leads on cost. Citations must be the model's own benchmark or leaderboard results.
+
+In the shortlist, the Input and Output columns keep per-million-token prices; a model billed in another unit shows that unit's price there instead, marked with its unit, and the expanded model shows the estimated cost per 1,000 tasks. Known limitations: per-second audio prices apply to audio seconds, but audio billed in tokens must be estimated as tokens; image prices are per generated image; prices are base-tier list prices from catalogue sources and can be wrong at the source.
+
+## Conversational Scout agent replaces the search page — deployed, 28 October 2026
+
+**Deployment receipt:** the owner approved the commit, push and Scout serving deployment. [PR #26](https://github.com/laz-aslam/openmerit/pull/26) merged as [332dda3](https://github.com/laz-aslam/openmerit/commit/332dda3d889a2da386f2d0bb59e60cc5170d29ce). After all PR and merged-commit CI checks passed, the guarded deployment wrapper moved `openmerit-scout-alpha` from `88e33d4c-4606-4eb1-be27-36a2b05309ff` to `2cf2c2d3-8aef-40f0-9c44-800bcab247bf` at 14:38 UTC (18:38 Dubai). The deployed Scout HTML, JavaScript, stylesheet and auth module matched the local release build. `/` and `/connect/` remained byte-identical; the publisher, router, terminal and memo service were not redeployed. The catalogue stayed healthy on snapshot `588855a0bf28212fad48a9513dbf3d5b`. Live desktop/mobile checks opened the real Google/GitHub sign-in controls with optional analytics blocked, preserved the task on dismissal and sent no agent turn before authentication. Mobile scroll checks reached the full-height chat before the footer. Completed OAuth, an authenticated production recommendation, native Claude installation and physical-phone keyboard behavior remain unverified. CI's first merged-commit attempt failed an existing assertion that mistook `900` inside random UUIDs for leaked error data; the failed jobs passed on retry without changing that test.
+
+**Claude Code action — deployed, 28 October 2026:** **Add to Claude Code**, with the existing Claude icon, sits below the intro's coverage counts. It restores the secret-free setup handoff described [below](#connect-a-coding-agent): copying and requesting the prefilled Claude link happen independently, without waiting for clipboard permission or requiring a chat sign-in. Feedback appears below the button and distinguishes a copied prompt from a requested launch; neither proves installation. Current task text is never included. DNT/GPC, saved usage refusals and account unlinking propagate into the installer command. A fresh setup ID is included only when browser usage collection and account linking are both enabled; blocked optional analytics does not manufacture a saved installer opt-out or prevent the handoff. On stacked screens, the intro and setup feedback scroll above the full-height chat. Browser-platform fixtures cover prompt/deep-link equivalence, privacy and failure handling; native Claude launch and completed installation remain unverified.
+
+**Header wordmark — deployed, 28 October 2026:** Scout uses the footer OpenMerit wordmark's 18px size, 550 weight and tight letter spacing, rendered lowercase. The **New** label aligns with the header divider's right end while retaining the button's padding; account controls precede it when present. Sign-in, account access and conversation reset are unchanged.
+
+**Chat surface and footer — deployed, 28 October 2026:** the conversation and composer share a subtle theme-aware surface with no outline. Spacing separates responses and task-brief fields without horizontal dividers; the model comparison header also has no divider. Continue is the task brief's only action, using a compact filled button with no explanatory line below it. Buttons are 32 CSS pixels high for mouse input and at least 44 pixels when a coarse pointer is available. Scout responses show the OpenMerit icon beside their Scout label; user messages keep their You label. The OpenMerit home logo sits at the footer's right edge. Data policy, Privacy & analytics, then Scout by OpenMerit stay vertically stacked at its left edge with the same spacing. Link destinations and privacy controls are unchanged.
+
+**Viewport layout — deployed, 28 October 2026:** above 900 CSS pixels, navigation, title/description/coverage and footer form a left column; the chat fills the right column's height, with its composer at the bottom and its conversation scrolling independently. At 900 pixels and below, navigation and intro scroll above a chat section that occupies the full dynamic viewport height. Native page scroll snapping stops at the chat's top; its transcript scrolls inside it while the composer stays at the bottom. Scrolling beyond the transcript's end can continue down the page to the footer, the only section below the chat. Focus the conversation to scroll with the keyboard, including long briefs and expanded model details. **New** stays in navigation instead of taking up a transcript row: a compact borderless button with a plus icon, labelled “New conversation” for assistive technology and in its tooltip. The intro and composer compact on short screens, and long drafts scroll inside a height-limited input instead of taking over the conversation. Touch-input text is at least 16 pixels to avoid focus zoom; safe-area padding protects the full-height chat's top and composer. Mobile viewport height follows browser chrome, with keyboard-driven layout resizing where the browser supports it. Physical-phone scrolling and keyboard behavior remain unverified.
+
+**Intro coverage — deployed, 28 October 2026:** the heading reads “Find the best AI models for your work.” A small overlapping lab/Hugging Face icon stack sits above live coverage from the public status endpoint. The counts distinguish catalogue models (`availableModels`), Hub repositories (`specialistRepositories`), labs with benchmark results (`providers`), indexed benchmark series (`benchmarks`, including descriptions without results), and distinct source-feed IDs (including retained research records, not unique organizations). Catalogue models and Hub repositories may overlap and are not added into a unique total. If status is unavailable, counts stay hidden without blocking sign-in or sending. These are catalogue coverage counts, not proof of task performance or an optimal recommendation. The homepage makes no affected coverage claim and needs no copy change.
+
+The conversational agent is the Scout page at `/scout/`, replacing the search box and its results table. The page stays open to everyone; sending a message requires sign-in, as searching did, and a message typed while signed out is kept while you sign in. The JSON search API (`/scout/api/recommend`), the MCP server (`/scout/mcp`), coding-agent setup at `/connect/` and the catalogue refresh are unchanged, so the CLI and installed coding agents keep their current behaviour. Sections below that describe the search page's prompt examples, results table and loading messages are historical; the Claude Code setup handoff remains available in the new page.
+
+You describe the work; Scout writes a task brief with five components (input, work, output, criteria and constraints), each marked stated, assumed or missing. Each component is editable in place: correct anything Scout got wrong, or fill in a missing one. All five fields share the same editable background, including Work; the page does not show per-component question text. Once sent, the brief collapses. **Continue** sends only the components you changed and requests recommendations immediately, using stated assumptions for missing details. It also works without edits; there is no separate Skip questions button. Pressing Enter in a brief field does the same; Shift+Enter adds a line break. The message box stays at the bottom of the page. Each turn ends with an editable brief or a shortlist of five models, followed by Scout's summary. While a turn runs, its steps are listed with the current one last; afterwards they stay as a collapsed record of the work. The shortlist reports Scout's 0–100 quality estimate with the cited benchmark ranks, input and output prices per million tokens on its cheapest eligible route and release date. Expanding a model shows its rationale, benchmark ranks, route and reference latency. A missing release date is shown as unknown. Reply in the same conversation to change constraints or trade-offs; **New** starts over.
+
+The agent runs in a Cloudflare Durable Object, one per conversation, using Pi Durable and the Agents SDK. Its model is set by `SCOUT_AGENT_MODEL`: `openai/<model>` calls OpenAI's Responses API directly with the Worker's `OPENAI_API_KEY` (default `openai/gpt-6-sol`), and any other value runs on Workers AI, such as `@cf/zai-org/glm-5.3`. Its tools search benchmark descriptions by embedding (OpenAI embeddings, as in search), read leaderboards and model records from the published catalogue, and use Workers AI web search when the catalogue cannot answer something decisive. Cost, latency and recency figures come from catalogue data, not from the model's text, and a shortlist that cites models or benchmarks outside the catalogue, or breaks a stated hard constraint, is rejected and resubmitted. A turn usually takes one to three minutes.
+
+The page uses the same sign-in and rate limits as search:
+
+- `POST /scout/api/agent` with `{"message": "…"}` starts a conversation and runs one turn. Continue it with `conversationId` plus `message`, `brief` (the edited components, such as `{"constraints": "Open weights only"}`; an empty string removes one) and/or `skip: true`. A client may supply its own UUID as `conversationId` with the first message.
+- `GET /scout/api/agent/<conversationId>` returns `busy`, the turn's `progress` steps and its latest `answer`. The page polls it while a turn runs and reads the answer from it if the POST connection drops.
+
+Conversations are keyed by the signed-in account, so one account cannot read another's. The conversation transcript is stored in its Durable Object with no expiry yet. With the default `openai/gpt-6-sol` agent model (not yet deployed), the task text and the conversation reach OpenAI, which also embeds benchmark and catalogue searches; web search runs on Workers AI. Before that change, the task text reached Workers AI and, for benchmark search, OpenAI. The [beta data policy](/scout/data-policy/) still needs updating to describe conversation storage. The deployed serving configuration includes the `SCOUT_AGENT` Durable Object (migration `agent-1`), the Workers AI binding and `nodejs_compat`.
+
+## Repeatable searches and task-specific evidence first — deployed, 27 October 2026
+
+**Repeatable results.** Scout's interpretation and evidence checks are AI calls whose answers vary between attempts, so an identical search could return a different shortlist: across 34 test cases run three times, repeats shared only 34–39% of their top five. A completed search now records each AI answer it received. For seven days, an identical search (same prompt, model/endpoint scope, catalogue generation, interpreter version and UTC day) replays those answers instead of asking again, so it returns the same shortlist. Failed or degraded searches record nothing. A new catalogue generation, interpreter version or day asks again. See [the data policy](/scout/data-policy/) for how the answers are stored. Relative dates in restrictions now resolve against the UTC day rather than the exact time. Interpreter version is 40.
+
+**Task-specific evidence first.** When some candidates in the quality band have a score on a benchmark of the task itself (direct evidence), they are ranked and labelled strongest, best value and fastest among themselves first. A model estimated only from general benchmarks can no longer outrank them or remove them as dominated; it can still fill a remaining place, and a warning names it. The same applies to domain evidence when no direct benchmark was accepted. For example, a model with only a high general score no longer leads a medical coding search above models measured on MedCode.
+
+The evidence assessment call now allows one retry within the search deadline (up to four seconds instead of 1.8), because a single slow answer previously failed the whole search.
+
 ## Twice-daily catalogue refresh — deployed, 25 October 2026
 
-The hosted catalogue publisher refreshes twice a day, at 00:07 and 12:07 UTC (05:37 and 17:37 IST), instead of hourly. Each run is unchanged: the same feeds, discovery pages, up to 24 Parallel research jobs and the same publication checks. Daily paid research therefore drops from at most 576 to at most 48 jobs, and Hugging Face discovery pages advance twelve times more slowly. Public feeds are reported stale 13 hours after their last successful check, which is one missed run plus an hour's grace. A single failed run can therefore leave up to roughly a day between source checks. The local `npm run scout` preview still refreshes its bundled feeds hourly. The owner-approved publisher and serving deployments went live on 25 October 2026 at 17:00 UTC.
+The hosted catalogue publisher refreshes twice a day, at 00:07 and 12:07 UTC (05:37 and 17:37 IST), instead of hourly. Each run reads the structured feeds and the D1 catalogue and applies the same publication checks. Since 26 October 2026 it no longer runs Parallel research jobs or the Hugging Face recently-modified listing discovery that fed them. Their 234 discovered benchmarks had no model results; the descriptions they collected are kept and are not refreshed, so they never mark the catalogue stale. Public feeds are reported stale 13 hours after their last successful check, which is one missed run plus an hour's grace. A single failed run can therefore leave up to roughly a day between source checks. The local `npm run scout` preview still refreshes its bundled feeds hourly. The owner-approved publisher and serving deployments went live on 25 October 2026 at 17:00 UTC.
 
 ## Beta evaluation logging — 22 October 2026 rollout
 
@@ -123,6 +199,10 @@ Hosted Scout uses Clerk authentication for **Google and GitHub only**, without C
 
 The page and index status remain public. **Sign in** opens Clerk's account UI; submitting a task while signed out opens that same flow without invoking matching. A prompt draft is temporarily kept in this tab's `sessionStorage` to survive the OAuth redirect, then removed on return or successful sign-in. Storage-disabled browsers can sign in but may lose their draft during the redirect. Signing out or changing accounts clears the visible prompt/results and cancels the browser's pending request; it does not guarantee cancellation of provider work already accepted by the server.
 
+**Sign-in correction — deployed, 28 October 2026:** optional analytics downloads no longer block the conversational page's sign-in or submit handlers. Failed authentication initialization now displays an on-page error without clearing the task or starting an agent turn. Auth configuration and client initialization have 15-second limits; a sign-in modal that fails to render its controls within 30 seconds closes with a reload instruction. Closing a fully rendered modal does not count as an authentication error. Live desktop/mobile checks confirmed that the real Google/GitHub controls open with analytics blocked and preserve the draft on dismissal; these checks do not establish completed OAuth or an authenticated production conversation.
+
+Submitting a task while signed out resumes that task automatically once an active session is available, including after a same-tab OAuth return. A separate temporary submission marker contains the submitted task in this tab's `sessionStorage` and is consumed before resuming, so session updates and later reloads do not replay it. Header **Sign in** only restores the draft; it does not submit it. Editing the draft or dismissing sign-in with its close button, backdrop or Escape cancels automatic submission. With storage disabled, same-page sign-in can resume in memory, but redirect return cannot restore the task or submission intent. These cases have simulated-session coverage, not proof of completed provider OAuth.
+
 The Worker verifies Clerk session tokens before catalogue access for both `/scout/api/recommend` and `/scout/mcp`, including MCP discovery. Requests must send `Authorization: Bearer <session token>`; a cookie alone is not enough. Signature, expiry, issuer and authorized-party validation use Clerk's SDK, and pending sessions cannot search. Missing/invalid credentials return HTTP 401; missing server configuration returns 503 rather than enabling anonymous access. Status and `/scout/api/auth` remain public; the latter exposes only the publishable key and frontend API hostname. The secret key never reaches the browser.
 
 For an authenticated development/API check, obtain a current token from `await window.Clerk.session.getToken()` in your own signed-in browser. Treat it as a credential, do not share it, and set it privately as `SCOUT_TOKEN`. Then send:
@@ -201,7 +281,7 @@ Similar tasks can still receive identical shortlists when one benchmark is their
 
 This change replaces the cost-first policy below. Quality screening previously compared a model only with models that had exactly the same set of accepted benchmarks. That split the catalogue into many two- or three-model groups, so the best of a small, weak group passed the screen and cost-first ordering promoted it.
 
-Scout now fits **one latent quality estimate across every connected evaluation** for the task. Each benchmark has its own difficulty offset, score scale and noise, so a near-zero benchmark and a widely spread one contribute on comparable terms. Correlated series in one benchmark family share that family's weight. Each model receives an estimate with a standard error; a benchmark run by only one model links nothing. Applicable calibration observations join as one further evaluation using effective quality. Models with no evidence connected to the main group receive no estimate and remain provisional; they are never compared numerically. The estimate is reported as `comparison.qualityEstimate` and `comparison.qualityStandardError`; `quality` is its relative percentile among the estimated models.
+Scout now fits **one latent quality estimate across every connected evaluation** for the task. Each benchmark has its own difficulty offset, score scale and noise, so a near-zero benchmark and a widely spread one contribute on comparable terms. Correlated series in one benchmark family share that family's weight. Copies of one evaluation from different publishers, such as GPQA Diamond reported by Epoch AI, Artificial Analysis and model developers, form one family and count as one evaluation; their scores stay separate series. Evidence from the task's own domain carries at least three quarters of the weight when present, and evidence that scores the task's own kind of work counts twice as much as other domain evidence. General proxies fill in the rest and connect models that lack domain results. Each model receives an estimate with a standard error; a benchmark run by only one model links nothing. Applicable calibration observations join as one further evaluation using effective quality. Models with no evidence connected to the main group receive no estimate and remain provisional; they are never compared numerically. The estimate is reported as `comparison.qualityEstimate` and `comparison.qualityStandardError`; `quality` is its relative percentile among the estimated models.
 
 A configuration qualifies when it is not credibly worse than the strongest estimate: within two points plus one combined standard error. Supported Pareto selection over quality, reference cost and reference time follows. A balanced search keeps the strongest, lowest-known-cost and fastest frontier points, labelled in `pick` as `quality`, `value` and `speed`, then orders by estimated quality. An explicit `quality`, `cost` or `latency` priority orders the frontier accordingly. At most two models per provider are selected while other qualifying models remain; the cap never leaves a supported place empty. `ranking.version` is `10`, and `ranking.frontier` reports `version: 4`, the priority, the number of estimated models and benchmarks, and the tolerance.
 
@@ -267,9 +347,9 @@ The collection changes below remain **unreleased**; alpha reads the existing sta
 
 Research requests cited lifecycle/expiry information, capability updates, deployment regions, fine-tuning scope and documented hardware requirements. Standard rates remain separate from promotional offers, subsidies and expiry conditions. Editorial claims supply leads rather than numerical benchmark observations. Provider-specific facts must retain their scope; OpenRouter expiry is stored on its route rather than on every deployment of a model. Unknown or unsupported facts remain unavailable. No new paid collection run has validated this expanded coverage yet.
 
-OpenAI receives the submitted request for decomposition and derived work descriptions for interpretation and embedding. Parallel receives only public research subjects. Private hosted analytics records each model role separately with the original prompt, role position and allowlisted supporting evidence; the existing sampled three-month retention applies. Do not put secrets or personal information into prompts. Keys stay server-side.
+OpenAI receives the submitted request for decomposition and derived work descriptions for interpretation and embedding. Private hosted analytics records each model role separately with the original prompt, role position and allowlisted supporting evidence; the existing sampled three-month retention applies. Do not put secrets or personal information into prompts. Keys stay server-side.
 
-For development, `npm run scout` serves the page on port 45833 and requires `OPENAI_API_KEY` for matching. It refreshes structured feeds locally each hour; it does not run the hosted Parallel/R2/D1/Vectorize publication pipeline. Local evidence retrieval uses lexical matches and prepared memberships without a Vectorize binding. Production-like validation needs an isolated Worker with the storage bindings and both `OPENAI_API_KEY` and `PARALLEL_API_KEY`. Never put their values in browser assets, Git or a chat message.
+For development, `npm run scout` serves the page on port 45833 and requires `OPENAI_API_KEY` for matching. It refreshes structured feeds locally each hour; it does not run the hosted R2/D1/Vectorize publication pipeline. Local evidence retrieval uses lexical matches and prepared memberships without a Vectorize binding. Production-like validation needs an isolated Worker with the storage bindings and `OPENAI_API_KEY`. Never put its value in browser assets, Git or a chat message.
 
 ## Evidence architecture — hosted beta, 16 October 2026
 
@@ -279,13 +359,13 @@ Scout separates semantic interpretation from exact metadata checks and numerical
 | --- | --- |
 | Hourly discovery | Structured feeds and incremental Parallel research; resumable Hub discovery archives public listings without inventing usable endpoints or scores. |
 | Hourly evidence | Source-backed identities, deployment facts, benchmark methodology and specialist profiles; cached summaries fall back to the original source when support is uncertain. |
-| Hourly publication | Private R2 records, an indexed D1 fact/observation projection and versioned Vectorize benchmark embeddings. The active pointer changes only after the complete generation is ready. |
+| Hourly publication | Private R2 records, an indexed D1 fact/observation projection and Vectorize benchmark embeddings, stored once per content and listed per version. The active pointer changes only after the complete generation is ready. |
 | Task interpretation | Structured Outputs extracts metadata restrictions, alternatives, preferences and the downstream work. Decisions interprets that work for capabilities, operations and output serialization. No prompt keywords or provider/family exception lists are used. |
 | Eligibility | Generic typed equality, membership and range comparisons against documented facts, intersected with explicit inventories. Missing or incorrectly typed facts cannot establish a restriction. |
 | Evidence retrieval | A query embedding retrieves benchmark methodology, supplemented by hourly capability memberships. Semantic similarity supplies evidence candidates, never a model's score. |
 | Selection | All eligible verified catalogue models are considered before selecting up to five distinct models. One quality estimate spans connected evaluations; Pareto comparisons require known quality, cost and latency. |
 
-Mandatory restrictions use alternatives of complete condition groups. Soft preferences affect ordering, not eligibility. Relative dates use a supplied current UTC reference; arbitrary numeric bounds are retained, including impossible requirements. An unsupported mandatory condition cannot silently disappear. A no-match result describes the available verified catalogue and inventory, not every model in existence.
+Mandatory restrictions use alternatives of complete condition groups. Soft preferences affect ordering, not eligibility. Relative dates use a supplied current UTC day; arbitrary numeric bounds are retained, including impossible requirements. An unsupported mandatory condition cannot silently disappear. A no-match result describes the available verified catalogue and inventory, not every model in existence.
 
 Quality remains a relative benchmark signal, not predicted accuracy on an arbitrary task. Correlated benchmark families are grouped. Incomplete or incomparable objectives remain provisional and cannot establish a global three-objective frontier. Narrow typed tasks can still use a task-fit shortlist with unknown quality. Comparable direct calibration observations supplement quality evidence without excluding other eligible models. Calibration episode charges and timing do not replace the common token-workload reference costs and endpoint-bound timing; unknown values remain unknown.
 
@@ -339,7 +419,7 @@ The hosted implementation keeps the existing task and format Decisions, exact me
 
 Hourly staging prepares content-addressed specialist packs alongside profile indexes. Searches select the same scoped profiles and use those packs instead of many repository reads. A version-checked derived pack index can also be prepared for an existing immutable catalogue without repeating research. Existing generations remain readable through the original shard path when no matching pack index is available. Source research and the hourly trigger are unchanged.
 
-Successful searches may be reused within a Worker isolate for up to 30 seconds, bounded to 32 entries and approximately 4 MiB. Exact prompt text, model/endpoint inventories, publication, interpretation version and UTC calendar date identify a cache entry. Cache hits preserve the original `evaluatedAt` time; date conditions remain evaluated as of that time for at most 30 seconds, and results are not reused across UTC midnight. Identical concurrent requests share computation. Errors and results missing a specialist or applicability check are not cached. This temporary memory cache adds no external prompt storage and is not semantic similarity caching.
+Successful searches may be reused within a Worker isolate for up to 30 seconds, bounded to 32 entries and approximately 4 MiB. Exact prompt text, model/endpoint inventories, publication, interpretation version and UTC calendar date identify a cache entry. Cache hits preserve the original `evaluatedAt` time; date conditions remain evaluated as of that time for at most 30 seconds, and results are not reused across UTC midnight. Identical concurrent requests share computation. Errors and results missing a specialist or applicability check are not cached. This temporary memory cache is not semantic similarity caching. Since 26 October 2026, a separate seven-day answer cache in D1 replays the AI answers of an identical completed search; see the repeatable-searches section above.
 
 Private search records now include fixed stage durations, pending stages at a deadline, cache status and allowlisted provider status/error codes. They do not store raw errors, stack traces, headers or credentials. API results include `runtime` timings/cache status; cache hits report their current serving duration and zero new interpretation time. The page remains unchanged. A six-second matching deadline remains a reliability boundary; these changes do not establish a latency guarantee. The full repository release checks passed, including 318 Scout tests. An initial candidate timed out on books/posters; that failure was retained, and the interpretation was then reduced further. The revised semantic suite passed 17 of 17 cases, including cold repetitions and exact-cache reuse, and four final cold checks passed after the independent database reads were parallelized. These small samples do not establish a production percentile or a latency guarantee. Post-deployment HTTP/MCP checks passed five of five on their first attempts: supplied-label vision, creator restriction, impossible context, JSON invoice extraction through MCP and an exact repeat. Observed end-to-end durations were approximately 2.0–4.3 seconds for new searches and 146 ms for the repeated prompt served from the same isolate cache. The 127 UI source files and five live scripts/styles were unchanged. These are individual observations, not percentile guarantees.
 
@@ -410,7 +490,7 @@ Compatibility is eligibility, not predicted accuracy. Scout still enforces endpo
 
 Snapshot data version 7 prepares benchmark capabilities and correlated benchmark families with Decisions during the hourly refresh. It replaces benchmark-name regexes, manually assigned capability tags and the automatic reasoning fallback. Source adapters retain metric identifiers, score units and parsing contracts. Epoch descriptions and methodology are read from its published catalogue; Artificial Analysis and BFCL methodology are read from their public pages. Unavailable definitions remain explicit limitations, and names alone do not establish an opaque benchmark's capabilities.
 
-Each changed benchmark receives an isolated interpretation request so another benchmark's description cannot supply its capabilities. For shared publisher methodology, a preceding Decisions request selects the directly applicable definition for that benchmark and evaluated subset; unrelated sections are omitted from capability interpretation. Publisher-declared suite identities are retained as source facts; other family relationships are interpreted by Decisions. Capability choices and family references are validated, and positive choices below the existing 0.4 confidence threshold are not used. Family links group related evidence; numerical comparisons still require the exact same benchmark series. Unknown scopes receive no capability tags and cannot contribute capability-ranking evidence. This is semantic metadata interpretation, not a measurement of model quality or a guarantee that the classifications are correct.
+Each changed benchmark receives an isolated interpretation request so another benchmark's description cannot supply its capabilities. For shared publisher methodology, a preceding Decisions request selects the directly applicable definition for that benchmark and evaluated subset; unrelated sections are omitted from capability interpretation. A section titled with the benchmark's own name, such as "GPQA Diamond" on Artificial Analysis's methodology page, is always kept and listed first, ahead of shared sections such as the Intelligence Index overview. The publisher's methodology is used even when an earlier research description exists. Publisher-declared suite identities are retained as source facts; other family relationships are interpreted by Decisions. Capability choices and family references are validated, and positive choices below the existing 0.4 confidence threshold are not used. Family links group related evidence; numerical comparisons still require the exact same benchmark series. Unknown scopes receive no capability tags and cannot contribute capability-ranking evidence. This is semantic metadata interpretation, not a measurement of model quality or a guarantee that the classifications are correct.
 
 Interpretations are cached by definition, relevant publisher methodology, family inventory and interpreter version. Score, model, price and collection-time changes alone do not invalidate that cache. Family inventory changes trigger rechecking its references. If an interpretation fails, unchanged cached records remain usable, affected new or changed series stay unclassified, and source failure status is exposed until a later refresh succeeds. A methodology download failure retains the prior methodology when available and marks its source degraded. There is no keyword fallback.
 
@@ -420,17 +500,31 @@ The initial deployed snapshot processed all 146 benchmark series: 136 have suppo
 
 ## Use Scout
 
+**Source only, not deployed:** results use **Model Choices** in their count and no longer display rank numbers, the Rank column or the “Read left to right” instruction. The Assumptions & limitations disclosure, including its supplementary specialist list, is replaced with “Results could be wrong. Please verify before choosing a model.” Model evidence remains expandable; ranking, API evidence and the methodology documented here are unchanged. Feedback's reason selector and action buttons share a 40px height.
+
+**Source only, not deployed:** the pending search state places rotating search messages in the results-title position, preceded by a tiny pulsing orb. It does not repeat the submitted prompt or show a separate loading heading. Reduced-motion preferences disable both the orb pulse and the 180ms opacity/3px message transition.
+
+Loading messages describe search perspectives, not reported backend stages or completion percentages. Live timing starts at submission, including authentication and network wait, and freezes as **Took …s** when the response arrives. A cached response may legitimately be much faster. The existing 35-second request timeout still starts after authentication; replacing a search, signing out, completing or failing stops its loading updates.
+
+**Source only, not deployed:** focusing the task field changes its background without drawing an outline around the input container. Keyboard focus outlines on buttons and links are unchanged.
+
+**Source only, not deployed:** example prompts appear as rounded pills numbered 1, 2 and 3 beside **Try**, inside the bottom-left of the task field. They disappear as soon as the field contains any text, including whitespace, and return when it is cleared. Selecting a pill fills an editable draft without submitting it. The visible input-helper sentence is removed; Shift + Enter still adds a line. The Claude Code logo uses its existing transparent image without a white background in either theme.
+
 Open [Scout](https://openmerit.ai/scout/), type a task, and select the arrow button (**Find models**) or press Enter. For example: “I want to upload an invoice and get the invoice number, date, vendor, subtotal, tax, and total as JSON.” The task field grows and shrinks as you type or paste. Very long prompts scroll within the field once it reaches its height limit. Shift + Enter adds a line. When the field is empty and unfocused, its placeholder cycles through detailed example tasks every ten seconds with a brief fade. Rotation pauses while you type, while the field is focused, or while the page is hidden. With reduced motion enabled, the example stays still. Placeholders never fill the input or send a request. The heading shows live counts of API catalogue models, Hugging Face repositories, benchmark series and sources. API identities and Hub repositories are separate counts and can overlap; neither count promises a runnable endpoint for every entry. The evaluation guide remains available in the fixed bottom footer, with space reserved beneath the page content.
 
 The comparison table keeps recommendations in Scout rank order. Benchmark mode shows a relative quality signal and reference cost/time estimates. Applicable calibration contributes quality evidence within its recorded scope; its episode charges and timing do not replace the common reference workload. **Candidates to evaluate** denotes task-fit mode: quality is shown as unknown, alongside known reference costs and deployment requirements. When image separation is ambiguous, Scout first offers labels, bounding boxes or pixel masks/cutouts as output choices. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; Hugging Face results resolve the repository publisher, using a known lab mark or its bundled organization avatar; otherwise a Hugging Face mark identifies the repository platform. Missing or failed marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **JSON API**, **Capabilities**, and **Sources** expand beneath the search and results. **Sources** uses a compact list; expand a source for its coverage, methods, limits, collection times and refresh failures.
 
-The capability index lists each area with its benchmark-series and model counts. After a search, task matches appear first with a check mark. Counts describe index coverage, not the number of models eligible for your task. The source panel describes what each feed measures, its coverage, source dates, archived results, and limitations. Collection time is separate from source dates, which can be publication dates rather than measurement dates. Scout task calibration appears separately with its tested pool, scope and seven-day expiry.
+**Source only, not deployed:** the Scout page no longer exposes the JSON API setup, capability index or source catalogue disclosures, including the header API link. Privacy & analytics has its own [settings page](/scout/privacy/), linked from Scout's footer alongside the data policy. The task-logging notice and sharing controls live on that page; existing choices are preserved, and the former settings fragment forwards there. This is a presentation change, not API access control: the JSON API, status endpoint and MCP remain available under their existing contracts, and recommendation rows retain their evidence links and limitations.
 
 **Provider page** in each model's evidence opens its provider's official pricing, model documentation, or website. Model links never fall back to a router. Each quote identifies its source: a router route, independent provider metadata, or a reviewed provider contract. Routes have separate prices. Self-hosted infrastructure cost and incomplete quotes remain unknown. Hugging Face deployment links open the publisher’s model repository and pinned card. Benchmark links lead to the original evaluation sources.
 
 ## Connect a coding agent
 
-**Device sign-in setup — npm 0.1.8:** use the stable `latest` tag for the commands below. Version 0.1.8 adds the startup model-decision rule to the device setup introduced in 0.1.7. Hosted Scout serves `/connect` and `/scout/mcp` independently of npm. Claude Code is the only supported client; other agents are not yet supported. The older `0.1.7-preview.5` package does not contain these commands.
+**Pi native Scout adapter — npm 0.1.12:** Pi **1.5.0** joins Claude Code through the shared browser-approved installer. Run `npx -y openmerit@latest install --client pi [--scope user|project]`. The **Add to Pi** action below Claude copies setup text only; there is no Pi deep link. Approve the matching browser code, then start a new Pi session. Pi owns native MCP, tools, model, permissions and sessions; the adapter adds only the authenticated connection and shared Scout rule/skill. No `/openmerit` lifecycle commands, scheduler, LaunchAgent or automatic-update hooks are carried forward. Explicitly reinstall to update. See [Pi paths, migration and SDK embedding](pi-extension.md).
+
+Each new client/scope installation has independent credentials and approval; project keys are also separated by project and stored outside the repository. Shell `login`, `logout`, `status`, `remove` and `doctor` accept `--client pi|claude-code --scope user|project` to target that installation. Pi connection JSON references a private credential path rather than containing the key. The legacy Claude user credential path remains supported. Package publication and website deployment are independent; the versioned Claude setup below describes earlier releases.
+
+**Device sign-in setup — npm 0.1.8:** use the stable `latest` tag for the Claude commands below. Version 0.1.8 adds the startup model-decision rule to the device setup introduced in 0.1.7. Hosted Scout serves `/connect` and `/scout/mcp` independently of npm. Claude Code was the supported client in that release; Pi support requires 0.1.12 or newer. The older `0.1.7-preview.5` package does not contain these commands.
 
 Setup takes four steps, and you act in each one only once:
 
@@ -493,6 +587,8 @@ The agent compares model and endpoint, evidence, reference cost/time and unknown
 
 ### Automatic instruction updates — version 0.1.10
 
+**Claude Code only.** These hooks and `update` commands do not apply to Pi. Pi updates require an explicit installer rerun and a new session.
+
 Version 0.1.10 adds default-on updates for the installed Scout skill and startup rule. **npm 0.1.9 does not contain the updater.** After 0.1.10 is published, existing installations need one explicit installer rerun in their original scope to acquire it. A website or Scout deployment cannot bootstrap files on someone else's machine. Compatible hosted Scout changes already reach agents through their next MCP request.
 
 Setup discloses automatic updates and installs asynchronous `SessionStart` and `Stop` command hooks. They check npm's stable `latest` release at most once per 24 hours per installation. Offline or failed checks retain the installed version and retry at the next eligible hook after 15 minutes. `Stop` means Claude finished responding, not that the user's task succeeded. No background service watches connectivity while Claude Code is closed, and checks never ask Claude to keep working. A one-shot `claude -p` process can exit before an asynchronous Stop hook finishes; startup checks and later sessions provide the retry path. Managed policies or disabled hooks can prevent automatic checks; an explicit manual update remains available.
@@ -568,16 +664,28 @@ The twice-daily updater fetches eight independent public sources, including Mode
 | Feed | Used for |
 | --- | --- |
 | [Epoch AI](https://epoch.ai/benchmarks) | Benchmark definitions, published results, source links, and evaluation configurations. |
-| [Artificial Analysis](https://artificialanalysis.ai/leaderboards/models) | Benchmark results for 27 evaluations, measured first-answer delay and reference generation speed. Elo-style ratings (GDPval-AA and AA-3D) become the expected win rate against the median-rated model, as for Arena; AA-Briefcase is not read. Archived evaluations remain labelled when their models are still available. |
+| [Artificial Analysis](https://artificialanalysis.ai/leaderboards/models) | Benchmark results for 32 evaluations (including HealthBench, HealthBench Professional, Humanity’s Last Exam Diamond, CritPt v2 Hard and FrontierCode 2.0, each its own series beside its earlier version), measured first-answer delay and reference generation speed. Elo-style ratings (GDPval-AA, AA-3D and Code2Video-AA) become the expected win rate against the median-rated model, as for Arena; AA-Briefcase is not read. Archived evaluations remain labelled when their models are still available. |
 | [OmniDocBench](https://github.com/opendatalab/OmniDocBench#end-to-end-evaluation) | Document parsing scores from the explicitly labelled `v1.6_full` leaderboard, including newer document models than the OpenVLM snapshot. |
 | [IDP Leaderboard](https://www.idp-leaderboard.org/) | Nanonets' MIT-licensed document processing leaderboard: OlmOCR, OmniDocBench v1.5 and IDP Core (key information extraction, handwriting, tables, visual QA) component scores. The overall average is not imported; per-model dates are not supplied. |
 | [OpenCompass / VLMEvalKit](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) | Visual understanding and OCR benchmark results. Its feed currently identifies itself as 17 September 2025. |
 | [Berkeley Function Calling Leaderboard (BFCL v4)](https://gorilla.cs.berkeley.edu/leaderboard) | Native function-calling single-turn, live, and multi-turn scores; separate memory scores for key-value, vector, and recursive-summarization stores. |
+| [Arena leaderboard dataset](https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset) | Style-controlled human-preference ratings (CC BY 4.0) from four boards: text request categories (including medicine and healthcare, legal and government, the sciences, and entertainment, sports and media), document prompts, vision (overall, OCR, diagrams, homework and creative writing about images) and Code Arena WebDev (overall, HTML, React and image-to-WebDev). Each becomes the expected win rate against the median-rated model of its category; ratings with confidence intervals wider than 60 points are left out. Each category keeps its own publication date. Human preference, not verified correctness. A separate text **factuality** board (overall, medicine and healthcare, legal and government, business and finance) blends those votes with Arena's automated web verification of each response's factual claims at 25% weight; it is converted the same way and covers web-verifiable claims only. Search and Agent Arena are not read: they rate models together with a search system or agent harness. |
+| [Vals AI](https://www.vals.ai/benchmarks) | Overall accuracy on 27 benchmarks Vals AI runs, many on private test sets: healthcare (MedQA, MedCode, MedScribe), legal (LegalBench, Legal Research Bench, HLAB), finance (CorpFin, TaxEval, MortgageTax, Finance Agent, Excel modelling, ClaimBench), public benefits, education, multilingual math, science, code migration, programming and security. Composite indexes, archived versions and evaluations other feeds already supply are not read. Vals AI publishes no data licence; Scout keeps only overall scores with links to each page. Per-model evaluation dates are not supplied. Agent products such as a model inside Claude Code or Codex are not matched to the model. |
+| [Scale Labs (SEAL)](https://labs.scale.com/leaderboard) | Published scores from 11 current boards Scale runs, mostly on private sets: Professional Reasoning (legal and finance), MCP Atlas, MultiChallenge, the Remote Labor Index, SWE Atlas (refactoring, test writing and codebase questions), HiL-Bench, DrugDiscoveryBench and Humanity's Sixth Sense (visual reasoning). Each result keeps the date Scale added it. Rows Scale flags as possibly contaminated or deprecated are left out, as are deprecated boards, safety boards, audio, Humanity's Last Exam Diamond (read from Artificial Analysis) and SWE-Bench Pro V2. Agent boards rate a model inside a named harness, such as Codex or Claude Code; the harness is kept as the result's configuration. Scale names configurations inconsistently, so some rows stay unmatched. |
+| [LiveBench](https://livebench.ai) | LiveBench 2.0 category scores (long agentic, reasoning, coding, browser, data analysis, research and design) from the newest release's published table, about 30 frontier models per release. No per-model dates; the overall score is not read. |
+| [τ-bench by Sierra](https://taubench.com) | pass^1 on τ²-bench airline, retail and telecom and τ-bench banking, from Sierra's own text runs on harness 1.x. Developer-submitted and custom-agent entries are left out, and banking keeps only the AllTools retrieval setup. Few models per domain. |
+| [OpenCompass](https://opencompass.org.cn) | Text-only knowledge, abstraction, reasoning, interaction and agent ability averages for about 20 models. The overall score and the safety board are not read. |
+| [HELM Arabic](https://crfm.stanford.edu/helm/) (Stanford CRFM) | Mean score across HELM's Arabic scenarios from the newest release, mostly multiple choice. Models missing a scenario are left out; no per-model dates. |
+| [BRIDGE Medical](https://huggingface.co/spaces/YLab-Open/BRIDGE-Medical-Leaderboard) (YLab) | Zero-shot average across clinical-text tasks in several languages, mostly open-weight models. Average only; no per-model dates. |
+| [SuperCLUE](https://www.superclueai.com) | Chinese general-board category scores: math and science reasoning, hallucination control, precise instruction following, agentic coding and planning, for about 20 models. Boards older than six months are not read. |
+| [Fireworks Specialized Intelligence Index](https://fireworks.ai/specialized-intelligence-index/) | Domain benchmarks contributed by practitioner companies (legal, finance and tax, healthcare and life sciences, customer support, software and SRE, web search, marketing, hardware and more), from the index's published JSON. Each benchmark is its own `· Fireworks` series scored by its headline metric; only 0–100 headlines are read, so Elo, US-dollar and relative-score boards and the cost and duration columns are not. Unranked rows and entries Fireworks footnotes as run inside another product or with rejected tasks are left out. Most results are run by the contributor, some by Fireworks, which is itself an inference provider. No per-model dates. |
+| [LLM Stats](https://llm-stats.com/benchmarks) (vendor-reported) | Scores model developers report in announcements, papers and model cards, compiled by LLM Stats and reused with attribution under its terms. Scout reads LLM Stats' 100 most-reported benchmarks and keeps only results it marks self-reported, on 0–1 or 0–100 scales, as separate `· vendor-reported` series that never merge with third-party measurements of the same benchmark. Vendor-reported results count toward ranking. They are not independently verified, and settings such as tools, attempts and reasoning effort follow each vendor. Base (pretrained) checkpoints are never matched to instruct models. A benchmark that leaves the 100 keeps its last results. |
 | [OpenRouter](https://openrouter.ai/api/v1/models) | Available model IDs, input modalities, declared tools and structured-output support, context limits, and advertised token prices. |
 | [Models.dev](https://models.dev) | Independent native model IDs, declared capabilities and prices via ten direct-provider adapters: OpenAI, Anthropic, Google, DeepSeek, Mistral, xAI, Cohere, Moonshot, MiniMax and Z.AI. Not a performance benchmark or account-availability check. |
-| OpenMerit model catalogue | Additional hosted chat routes and token prices, merged from [Models.dev](https://models.dev), [LiteLLM](https://github.com/BerriAI/litellm), OpenRouter, [Vercel AI Gateway](https://vercel.com/ai-gateway/models) and [genai-prices](https://github.com/pydantic/genai-prices). Only reviewed inference hosts and model makers' own OpenAI-compatible APIs with positive per-token prices are included; resellers, routers, subscription plans and account-specific cloud endpoints are not. A route is attached only through an exact model identity. Chat models with such routes that no other source lists become catalogue models, so benchmark results naming them are kept; a catalogue model whose name another listed model already uses is left out. Listings are not verified access in your account. |
+| OpenMerit model catalogue | Hosted routes and advertised prices, merged field by field from [Models.dev](https://models.dev), [LiteLLM](https://github.com/BerriAI/litellm), OpenRouter's complete output-modality feed, [Vercel AI Gateway](https://vercel.com/ai-gateway/models), [DeepInfra's own model API](https://deepinfra.com/models), [genai-prices](https://github.com/pydantic/genai-prices) and the [TrueFoundry models registry](https://github.com/truefoundry/models) (MIT). [Eden AI's expert-model list](https://www.edenai.co/docs/v3/expert-models/listing-models) adds the gateway's own OCR and document-parsing, translation, text-to-speech, transcription, moderation and AI-content detection routes, including vendors no other source lists, such as Mindee, Veryfi, Affinda, DeepL, ModernMT, Gladia and Sightengine. Its prices are per character (per million), page, document file, image, request or second of audio or video, exclude Eden AI's platform fee, and come with no stated data licence. Eden's token-priced routes that do not say input or output, free quotes and its LLM catalogue are left out; only speech and OCR models with an existing catalogue ID link to it. TrueFoundry adds per-provider configs for Bedrock, Azure and Microsoft Foundry, Vertex, Gemini, OpenAI, Mistral, xAI, Cohere, Databricks, Deepgram, ElevenLabs and others, including speech, transcription, image, video, rerank, OCR, moderation and decision routes, with batch, priority and flex rates and regional prices where they differ from the base region; retired routes and context-tier surcharges are left out. OpenRouter contributes chat, decision, embedding, rerank, image, video, speech and transcription routes. DeepInfra contributes its chat, decision, embedding, rerank, image, video, speech, music and transcription inventory. Token, character, image, frame and input/output-time rates keep their provider-defined units; conditional overrides and unsupported multipliers remain in the archived source payload instead of being flattened into misleading base prices. Novita is not ingested in this revision: its public OpenAI model list mostly overlaps routes already present and does not supply source-specific pricing or modalities that this importer can normalize. A route is attached only through an exact model identity. Models with such routes that no other source lists become catalogue models, so benchmark results naming them are kept; a catalogue model whose name another listed model already uses is left out. Benchmark rows are also matched through the catalogue's other names for a listed model (host model IDs, listing names and Hugging Face repositories), and after removing run settings such as reasoning effort, thinking budgets and trailing parentheticals. A name shared by distinct catalogue models matches nothing, and a row under Scout's own name for a model outranks loosely matched rows for the same benchmark. Listings are metadata, not verified account access or a performance test. |
 | [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions) | Reviewed typed-choice, predicate and score endpoint contract, inputs and endpoint-specific rates. No general chat timing is inherited. |
 | [TypeSafe Jev](https://typesafe.ai/blog/jev-1-14) | Reviewed System One contract. Image and single-box output are experimental; headers, version and incomplete total pricing are disclosed. Jev is a candidate for compatible tasks, not Scout’s ranking engine. |
+| [Vercel AI Gateway decision routes](https://vercel.com/docs/ai-gateway/modalities/decision) | Decision models Vercel AI Gateway routes (Laya, Liquid d1, Perplexity Deciders, Jev, Microsoft Decision-1, Baseten Elephant and Mammoth and others), read from the OpenMerit catalogue each refresh with their input-token rates, context and image input. One reviewed contract covers them: `POST https://ai-gateway.vercel.sh/v1/evaluate` with supplied choices, ordered rubric scores and boolean probabilities. They join classification and judging shortlists as task-fit candidates; promotional and free-tier rates are shown as listed and can end. A free-tier ID with the same name as a paid one is listed once. Many decision rates are identical, so between routes with the same reference cost, the model's standing on [JevBench](https://benchmarkheaven.com/jev-models) (capability: intelligence and calibration, ranked systems only) and the [Artificial Analysis Decision Index](https://artificialanalysis.ai/evaluations/artificial-analysis-decision-index) breaks the tie: the share of each board's other entries it beats, averaged over the boards it appears on. Models without results come after those with them at that price; a cheaper route always comes first. Results link by exact model name. These boards measure general typed decisions, not accuracy on your task; the reason and `decisionBenchmarks` show each result. |
 | [Hugging Face](https://huggingface.co/models) | Public repository metadata, model-card results and structured evaluation records for specialist discovery. Separate from performance ranking. |
 
 Over 100 benchmark **series** are indexed, including source-specific implementations and versions. This is not 100 independent publishers, and coverage is not uniformly current. Some definitions have no results matched to an available model. Both counts are exposed. Missing metrics and ambiguous or unmatched model versions are omitted. Model discovery merges OpenRouter with independent native-provider records and catalogue host routes before matching public benchmark results. An OpenRouter outage does not prevent an independent catalogue from bootstrapping. Distinct endpoints retain prices and parameters; duplicate hosting entries do not create additional model votes or consume multiple places in the five. Endpoint listings are metadata, not verified access in the caller’s account.
@@ -586,6 +694,16 @@ Collection time is not measurement time. A successful refresh does not make an o
 
 Epoch AI attribution: “Capabilities & benchmarking,” published at epoch.ai, normalized and matched by Scout under the source's [CC BY 4.0 terms](https://creativecommons.org/licenses/by/4.0/). Other sources are attributed in the source panel and individual evidence links. These are data-source attributions, not a statement about OpenMerit's licensing.
 
+### Pending embedding benchmark and media-price collection
+
+**Not yet deployed:** the publisher source adds [MTEB's final leaderboard summaries](https://huggingface.co/spaces/mteb/leaderboard), including historical/off-menu suites and text, image, audio and video embedding evaluations. It collects only published aggregate and per-task scores plus their model, experiment, suite and training-disclosure metadata—not benchmark datasets, embeddings, predictions or per-example outputs. Different suite versions and experiment settings remain separate; missing aggregates remain unknown. Results are source-reported, not independently verified by Scout. The results dataset declares CC0-1.0; that does not change the individual task datasets' licences.
+
+This is catalogue ingestion only. MTEB results do not yet enter the published chat benchmark index or recommendations, and embedding/rerank workload pricing and recommendation paths remain unfinished. The summaries do not supply evaluated checkpoint revisions or measurement dates; collection and release dates do not substitute for either. An unavailable or malformed suite retains its previous complete results and original collection timestamp.
+
+Also **not yet deployed:** three supplemental [Portkey pricing feeds](https://github.com/Portkey-AI/models) contribute reviewed Deepgram Aura, ElevenLabs speech and Stability AI image quotes. Speech generation rates are per input character, Scribe transcription rates per input-audio second, and image rates per image with any published quality/size tier. Portkey's token-like field names do not establish token billing. Unknown denominators, service aliases, custom calculations and zero-only media placeholders are excluded rather than advertised as free generation. Each response is archived with its source attribution; listed prices are not an account-specific quote or proof of availability. These supplemental prices do not add a media recommendation path.
+
+The pending publisher also discovers Hugging Face inference-provider routes across all tasks, including repositories from personal accounts. This is independent of the admitted-organisation crawl and does not admit an author's entire collection. Bounded pages and repository batches resume from durable cursors; incomplete collection never withdraws existing routes, and source-wide removals wait for completed application. Application batches can become visible incrementally, while the source retains its previous successful-check timestamp until completion. Only exact/reviewed identities link to canonical models, non-token task prices remain unknown, and provider listings do not establish working credentials or task accuracy. This does not widen the executable recommendation contracts.
+
 ## Specialist models and open weights
 
 Some specialist models have useful evaluations outside general LLM leaderboards. Scout reads both legacy model-card `model-index` results and the Hub’s [structured evaluation records](https://huggingface.co/docs/hub/eval-results). It retains the original dataset, task or metric, units, configuration, split, reported date, setup notes and source link when supplied. Scores are not averaged across unrelated datasets. A repository revision pins the metadata collected; it does not establish which checkpoint produced every reported score. Reported dates can also default to a commit date. Scout does not validate verification tokens or independently reproduce these scores.
@@ -602,15 +720,15 @@ Missing evaluations mean **unknown**, not poor performance. Task-fit candidates
 
 [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions), using `gpt-6-luna`, interprets the work, capability requirements, operation, answer space, serialization and tradeoff priority. It distinguishes writing source from executing it, supplied categories from open identification, text-bearing objects from reading their contents, and session state from persistent memory. Typed answers are validated and checked for consistency. Uncertain answers remain disclosed; they cannot authorize narrow calibration evidence. No prompt keyword rules or model-name bonuses enter selection.
 
-A Structured Outputs call first extracts downstream work, explicit model/endpoint restrictions and preferences using the published metadata definitions and canonical vocabulary. Mandatory conditions use alternatives of complete AND groups; exclusions and arbitrary numeric/date bounds are preserved. Relative dates use a supplied current UTC reference. Generic typed comparisons enforce those conditions and intersect optional `modelIds`/`endpointIds` inventories. Missing facts cannot satisfy a restriction, including an exclusion. Unexpressible mandatory restrictions produce no verified matches rather than silently broadening the search. Names mentioned as task data, clients or comparison subjects are not automatic filters. Interpretation is probabilistic and can still misread a prompt.
+A Structured Outputs call first extracts downstream work, explicit model/endpoint restrictions and preferences using the published metadata definitions and canonical vocabulary. Mandatory conditions use alternatives of complete AND groups; exclusions and arbitrary numeric/date bounds are preserved. Relative dates use a supplied current UTC day. Generic typed comparisons enforce those conditions and intersect optional `modelIds`/`endpointIds` inventories. Missing facts cannot satisfy a restriction, including an exclusion. Unexpressible mandatory restrictions produce no verified matches rather than silently broadening the search. Names mentioned as task data, clients or comparison subjects are not automatic filters. Interpretation is probabilistic and can still misread a prompt.
 
 Image tasks require declared image input and relevant visual evidence in performance ranking. Document reading needs document evidence. JSON/schema tasks need declared `response_format` or `structured_outputs`; required tool execution needs tools support and relevant tool evidence. Persistent memory requires external-memory evidence rather than a large context window. [BFCL memory evaluations](https://gorilla.cs.berkeley.edu/blogs/16_bfcl_v4_memory.html) do not establish built-in memory or performance on the visitor's storage system. Recognized image/video/audio generation and transcription remain outside the supported output index. Ambiguous spatial output produces the existing clarification choices.
 
-Classification and judgment tasks first use applicable benchmark evidence. With no supported benchmark candidates, supplied-label classification and judgments retain the explicitly labelled **task-fit** fallback. Boxes and masks use their compatible task-fit path. Documented operation/output fit, deployment readiness and known reference costs/times determine unmeasured shortlists; quality stays unknown. Open or uncertain identification instead needs name-generating endpoints and relevant evidence; fixed-choice APIs cannot satisfy it. Specialist compatibility uses hourly-prepared public profiles. Unknown compatibility omits a repository with a warning, rather than assuming task accuracy.
+Classification and judgment tasks first use applicable benchmark evidence. With no supported benchmark candidates, supplied-label classification and judgments retain the explicitly labelled **task-fit** fallback. Boxes and masks use their compatible task-fit path. Documented operation/output fit, deployment readiness and known reference costs/times determine unmeasured shortlists; quality on the task stays unknown. Between decision models with the same reference cost, their standing on two general decision leaderboards breaks the tie, as described in the Vercel decision routes row of the source table. Open or uncertain identification instead needs name-generating endpoints and relevant evidence; fixed-choice APIs cannot satisfy it. Specialist compatibility uses hourly-prepared public profiles. Unknown compatibility omits a repository with a warning, rather than assuming task accuracy.
 
-For performance ranking, a query embedding retrieves benchmark methodologies, supplemented by capability memberships prepared at each refresh. Bounded independent Decisions predicates verify relevance and a deliberately scored required ability before a methodology contributes proxy evidence. Similarity never supplies a quality score. Rejected, unknown and unavailable evidence is withheld. If some checks fail, verified evidence can still be used with an explicit warning; if no applicability can be verified because checks are unavailable, the search returns unavailable rather than reporting a false no-match.
+For performance ranking, a query embedding retrieves benchmark methodologies, supplemented by capability memberships prepared at each refresh. Bounded independent Decisions predicates verify relevance and a deliberately scored required ability before a methodology contributes proxy evidence. A second Decisions question compares each methodology with the task's description: *direct* (the same kind of work in the same domain, such as MedCode for ICD-10 coding), *domain* (the task's subject domain or profession, but a different form of work), *generic* (a general ability the task uses) or *unrelated*. When direct and domain together hold at least 70% of that judgement, the benchmark is accepted as task-specific evidence, whatever the generic ability check says. A general proxy judged unrelated with at least 80%, such as competition mathematics for clinical questions, is withheld. Direct and domain evidence are listed first on each recommendation. The four nearest methodologies are always assessed, whatever their prepared capability tags, because a vertical benchmark such as TaxEval is often tagged for a different ability than the one the task names. When no single ability clearly determines quality, as with security review or tax checking, Scout uses the required ability the interpreter favours most and adds a warning, instead of failing the search. Similarity never supplies a quality score. Rejected, unknown and unavailable evidence is withheld. If some checks fail, verified evidence can still be used with an explicit warning; if no applicability can be verified because checks are unavailable, the search returns unavailable rather than reporting a false no-match.
 
-Quality comparisons use shared benchmark results, grouped by correlated family and task capability. Different cohorts are not compared through unrelated percentiles. BFCL uses native function-calling configurations with their original labels; correlated series are one family. Missing scores remain unknown and actual zero remains a score. Model/version matching remains conservative. A relative quality signal is not predicted accuracy or a statistical confidence interval.
+Quality comparisons use shared benchmark results, grouped by correlated family and task capability. Different cohorts are not compared through unrelated percentiles. BFCL uses native function-calling configurations with their original labels; correlated series are one family. Missing scores remain unknown and actual zero remains a score. Model/version matching remains conservative. A "+" in a model name is significant, so Command A+ and Command A are separate models rather than one ambiguous name. A catalogue model's dated ID (such as Cohere's `command-a-03-2025`) counts as a copy of the undated model. A relative quality signal is not predicted accuracy or a statistical confidence interval.
 
 Matching calibration observations can supplement quality for their exact operation, format, modality, tools, retention and workload scope. Each admitted model needs at least six calibration cases, and evidence expires after 30 days (seven days before 22 October 2026). An applicable profile past that age stays excluded and the result carries a warning naming the profile and its expiry date, rather than silently using benchmarks only. A match no longer requires five tested models or excludes all untested eligible models. Observations remain tied to their tested router deployment. Repeated calibration request failures still cap effective quality. Calibration episode bills and durations remain source evidence; they are not substituted for only some candidates' reference costs or times. Held-out outcomes and answer keys never enter runtime ranking.
 
@@ -696,7 +814,7 @@ npm ci
 npm run scout
 ```
 
-Open `http://127.0.0.1:45833/scout/`. This lightweight local preview uses the shared interpreter and matcher with bundled/source-feed data; it does not run the hosted Parallel research workflow or R2 publication system. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.
+Open `http://127.0.0.1:45833/scout/`. This lightweight local preview uses the shared interpreter and matcher with bundled/source-feed data; it does not run the hosted refresh workflow or R2 publication system. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.
 
 `npm run scout:refresh` fetches the eight public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Source downloads need no provider inference, but preparing new or changed specialist profiles and benchmark interpretations requires `OPENAI_API_KEY` and incurs Decisions usage. Unchanged metadata reuses cached analysis. Without a usable key, required specialist preparation retains the previous specialist snapshot; affected new or changed benchmark definitions remain unclassified. Both failures are reported on the affected source. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid scheduled evaluation job is installed. Recommendation queries require the server key and incur OpenAI interpretation/embedding usage. The local source-refresh command updates the bundled feeds; the hosted twice-daily Workflow additionally prepares and publishes its D1/Vectorize serving generation.
 

package/docs/security.md

@@ -1,5 +1,7 @@
 # Security
 
+**Pi replacement — unreleased, unpublished source:** Pi 1.5.0 uses native MCP plus shared Scout rule/skill sections, not the former evaluation adapter. It never initiates login, changes the host model/tools/session, or provisions background jobs. Browser-approved credentials are independent per client/scope/project, stored privately outside the repository, and passed as MCP HTTP headers, never instructions. Pi's installed connection JSON contains only the private credential path. See [Pi paths and targeted removal](pi-extension.md). The standalone evaluation CLI and core coordinator remain separate.
+
 The [preview agent-evaluation runner](agent-evaluations.md) is a separate execution path: it runs trusted local agents/graders, passes only basic OS variables and explicitly named environment variables, and preserves agent-owned authentication. It sends task descriptions and optional model IDs to Scout. Local metric reports and budget reservations persist; temporary task outputs are removed. Fresh directories are not security sandboxes. Its budget admission controls do not enforce provider billing limits.
 
 The [budget selector](agent-evaluations.md#budget-selection) in `0.1.7-preview.2` requires an explicit $10, custom or no-fixed-cap choice after model scope. No fixed cap removes monetary admission limits, including Claude's per-attempt dollar flag, while retaining tasks, repetition bounds, timeouts, native permissions and accounting. Saved choices apply across reruns; a foreground watcher can incur further charges on each change until stopped. Provider limits remain independent.
@@ -12,6 +14,8 @@ Never place secrets in profiles, policies, intent constraints, summaries, eviden
 
 ## Mutation authorization
 
+This section describes core/custom evaluation harness policy. Pi-specific lifecycle and background-runner behavior below is **historical Pi 0.87 functionality**, removed from the new Scout adapter. Installing the replacement does not establish that an old background job was removed; manage that job separately.
+
 Most intents carry `modelMutationAllowed: false`. Only application-target apply-swap and rollback intents can authorize a mutation, limited to the relevant proposal, prior configuration, and confirmed target ID. Pi's model is explicitly excluded. Automatic swaps require explicit policy opt-in and meeting the configured verified-swap threshold, which can be zero. Otherwise a proposal requires approval. Post-swap evaluation depends on the harness performing the checks and reporting their results. OpenMerit is not a sandbox around the harness's tools.
 
 Persistent wakeup provisioning is a separate authorization boundary. An adapter that needs to install a system scheduler, create a paid cloud agent, or change infrastructure must declare `wakeupProvisioning: "infrastructure_change"`; OpenMerit will not request that provisioning unless the confirmed check policy sets `infrastructureChangesAllowed: true`.
@@ -24,7 +28,7 @@ The background runner imposes a 15-minute wall-clock limit and records a nonzero
 
 `.openmerit/` may reveal task names, candidate identifiers, evidence locations, budgets, or model choices. Treat it as sensitive project metadata even though it must not contain credentials.
 
-Pi also records OpenMerit requests and results in its session history. Project context and evaluation data may be sent to the model providers, tools, and evidence systems configured in the harness. Local OpenMerit storage does not imply that all processing stays on your machine.
+The historical evaluation adapter also recorded OpenMerit requests and results in Pi history. With native Scout, Pi manages its own session/tool records and provider context; the adapter adds no separate evaluation-history store. Task descriptions sent to hosted Scout follow its data policy. Local OpenMerit storage does not imply that all processing stays on your machine.
 
 ## Evidence integrity
 
@@ -55,6 +59,8 @@ The [coding-agent setup](scout.md#connect-a-coding-agent) in npm 0.1.7 uses host
 
 **Automatic instruction updates — version 0.1.10:** updater-enabled setup defaults on with an opt-out, installing asynchronous Claude Code startup/response-end hooks. Checks use npm's public stable registry without credentials, task content or installation IDs. SHA-512 verification checks package bytes against registry metadata; it does not defend against a compromised publisher/registry. The updater reads only a bounded instruction bundle, never executes downloaded code or package scripts, and leaves MCP configuration, keys, privacy settings and project policy unchanged. A supported release contract permits instruction updates; maintainers must change it when permissions or data handling change. This is a publisher assertion, not semantic validation of instructions. Edited files are retained, concurrent writes serialize, and journaled recovery stops on intervening edits. New sessions reliably load new instructions. See [automatic updates](scout.md#automatic-instruction-updates-version-0110) for cadence, offline retries, manual commands and the one-time bootstrap required for older installations. Availability follows npm publication independently of hosted releases.
 
+These update hooks are Claude Code only. The new Pi adapter installs none; updates require explicit targeted reinstall. Its `login`, `logout`, `status`, `remove` and `doctor` shell commands target the selected client/scope without revoking another installation's independent key.
+
 ### OpenMerit coding-agent reset — 23 October 2026
 
 **Historical reset, superseded by npm 0.1.7 and hosted device setup.** The reset temporarily removed the installer and MCP transport. The current [Claude Code setup](scout.md#connect-a-coding-agent) uses browser-approved, device-specific keys scoped to MCP recommendations, not pasted session tokens or agent OAuth. The key is stored locally in mode-600 credentials and Claude Code's plain-text MCP configuration. Setup installs an owned Scout skill; conflicts are not overwritten, and removal revokes the device key. Browser Google/GitHub sign-in remains separate.
@@ -97,4 +103,4 @@ Clerk webhook signatures are verified over bounded raw request bodies with the S
 
 ## Scout research storage
 
-Scout uses Parallel for twice-daily public-source research. Parallel receives catalogue subjects and official-source metadata, not visitor prompts or application inventories. Decisions processes task descriptions and relevant public evidence. Public research receipts, citations, semantic annotations, publication completion receipts and immutable catalogue versions are stored in Scout's private Cloudflare R2 bucket. Public endpoint/model facts and observations also enter the private D1 serving projection. Vectorize stores public benchmark methodology embeddings, not visitor query vectors. Query embeddings are used transiently; OpenAI has its own API retention policies. Scout uses the prompt-and-evidence analytics policy above. API credentials are held as Worker secrets and are not included in browser assets, catalogue objects or analytics. Source text is treated as evidence, not executable code or instructions. Citations and semantic validation do not independently verify that publisher claims are true.
+Scout's twice-daily refresh reads public sources directly and no longer uses a research service. Decisions processes task descriptions and relevant public evidence. Previously collected public research citations, semantic annotations, publication completion receipts and immutable catalogue versions are stored in Scout's private Cloudflare R2 bucket. Public endpoint/model facts and observations also enter the private D1 serving projection. Vectorize stores public benchmark methodology embeddings, not visitor query vectors. Query embeddings are used transiently; OpenAI has its own API retention policies. Scout uses the prompt-and-evidence analytics policy above. API credentials are held as Worker secrets and are not included in browser assets, catalogue objects or analytics. Source text is treated as evidence, not executable code or instructions. Citations and semantic validation do not independently verify that publisher claims are true.

package/docs/testing.md

@@ -1,5 +1,7 @@
 # Testing and validation
 
+**Pi source replacement — unreleased and unpublished:** the new Pi 1.5.0 adapter uses native Scout MCP and shared guidance only. The dated Pi 0.87 lifecycle/scheduler results below remain historical receipts, not proof of the replacement or supported current commands. Package, browser-approval, native Pi and SDK acceptance must be reported for the new build separately; a successful documentation build alone establishes none of them. Core coordinator and standalone evaluation tests remain separate.
+
 ## Stable 0.1.7 acceptance — 24 October 2026
 
 Published `openmerit@0.1.7` to npm's `latest` channel after full repository verification: 757 passing tests, one existing skip, syntax/type checks, builds and documentation checks. The `preview` tag remains `0.1.7-preview.5`. A fresh registry installation on macOS ARM64 with Node.js 22.23.1 matched the tested 104-file tarball byte-for-byte. Packed core/protocol/Pi exports and the installed terminal runtime smoke checks passed, including passive collection, source removal, runtime outage/recovery and shutdown.

package/docs/troubleshooting.md

@@ -1,55 +1,50 @@
 # Troubleshooting
 
-Start with the current status and the local audit trail. These show what OpenMerit requested, what the harness reported, and which evidence references were recorded.
+The Pi Scout adapter requires OpenMerit **0.1.12** or newer and targets Pi **1.5.0**. Check the resolved package version when using `@latest`. The removed Pi 0.87 `/openmerit` workflow is not a diagnostic path for the replacement.
 
-```text
-/openmerit status
-/openmerit doctor
-/openmerit logs
-```
-
-## The command is missing
+## Scout is missing in Pi
 
-Check that OpenMerit is installed and enabled in Pi:
+Run targeted checks in your shell:
 
 ```sh
-pi list
+npx -y openmerit@latest status --client pi --scope user --check
+npx -y openmerit@latest doctor --client pi --scope user
 ```
 
-The published extension targets Pi 0.87 and Node.js 22.19 or newer. Follow [getting started](getting-started.md) to install it, then start Pi in your product repository.
-
-## Setup did not begin
+Use `--scope project` from the original project root for project setup. Check Pi and Node.js versions against [setup](getting-started.md), complete browser device approval, then start a **new session**. Installing the npm package or reconnecting an old session does not load new extension instructions. For SDK embedding, load Pi's `createMcpExtension()` explicitly alongside `createScoutPiExtension`; the Scout factory alone is not an MCP runtime.
 
-Automatic setup requires an unconfigured project, an interactive Pi UI, and a bounded source-detection match for an application LLM call. Non-interactive runs do not start the confirmation flow, and a project without a detected target remains idle until `/openmerit setup` is requested. Existing project state or active work may also prevent a fresh setup request.
+## The connection configuration is invalid
 
-Inspect status first. If no intent is active, use `/openmerit setup` to request setup explicitly; this is also the supported correction path when source detection or a prior setup recorded the wrong incumbent. A new setup preserves audit history but replaces the active task profile, candidate/assessment handoffs, frontier, and baseline state after the new result is accepted. Do not delete `.openmerit/` as a routine reset: it contains your policy, lifecycle state, evidence references, and audit history.
+The installed extension reads `openmerit-scout.json` beside its bundle; that file points to a private credential file outside the repository. Do not put the key in the extension or prompt. Check the scope and configuration-directory overrides against [installed paths](pi-extension.md#installed-files-and-credentials). Reinstall explicitly to repair missing owned files:
 
-## Evidence is missing or incomplete
+```sh
+npx -y openmerit@latest install --client pi --scope user
+```
 
-A task profile is not the same as working telemetry. Ask Pi which required metrics are uncovered, where their measurements should come from, and which evaluation artifacts have been run. Normal coding checkpoints do not replace those measurements.
+Expired or revoked credentials require targeted `login` followed by reinstall. The adapter reports a sanitized setup diagnostic; it never starts sign-in on its own. Approve only a browser code you initiated.
 
-The [metrics guide](metrics-and-evidence.md) explains missing and insufficient evidence. More samples or instrumentation may be needed before a candidate can be compared.
+## Installation reports a conflict
 
-## A result was rejected
+The installer does not overwrite edited or unowned files. Inspect and preserve your changes before resolving the conflict. Also check for overlapping Pi user/project installations or an existing `openmerit` MCP entry. Remove the unwanted owned installation using its own scope; do not revoke a different client's key as a repair. Disable the old `npm:openmerit` Pi registration when migrating from 0.87.
 
-Read the validation error and audit event. Successful work needs evidence references and the output structure required by that intent. Frontier results must match the task and product revisions, identify their assessments, and satisfy the comparison rules.
+## The adapter did not update automatically
 
-Have the harness correct the measurement or output problem. Do not mark an intent successful merely to advance the workflow.
+That is expected. Pi has no OpenMerit auto-update hooks. Explicitly rerun `install --client pi` with the original scope, then start a new session. Claude Code's update settings do not configure Pi.
 
-## Work appears stuck after a restart
+## `/openmerit` or evaluation tools are missing
 
-Pending or active intent state is durable. Inspect `state.json` and `events.jsonl` using the paths from `/openmerit logs`, and ask the harness to reconcile that work with the current session before requesting another intent. The current adapter may need supervised recovery; automatic completion after a restart is not guaranteed.
+They were removed, not renamed. The Pi adapter now supplies only native Scout MCP and shared guidance. It does not set up evaluations, recover intents, schedule work, swap models, or install a LaunchAgent. Existing legacy background jobs must be removed separately; installing the new adapter is not evidence that they stopped. Preserve old evaluation state and audit files if needed. For new evaluations use [the standalone CLI](agent-evaluations.md); custom harnesses may still use [the core coordinator](adapter-guide.md).
 
-## Checks are not running while Pi is closed
+## Scout was not consulted or its evidence is insufficient
 
-On macOS, check that the confirmed policy requests persistent execution and allows infrastructure changes, that the Mac user is logged in, and that Pi authentication works without shell-only credentials. `/openmerit doctor` reports scheduler diagnostics; `.openmerit/scheduler-status.json` records completed wakeups and failures. An active pause unloads the managed LaunchAgent. Other platforms need an external scheduler. A successful schedule does not establish that model-driven work completed. See [automatic checks](automation.md).
+The rule and skill are model guidance, not a guaranteed tool gate. Confirm they loaded in a new session and that the host permits the MCP tools. When project model policy is missing, ask the user; when Scout is unavailable or lacks suitable evidence, pause the model choice rather than invent a result. Host model/tool/session restrictions remain authoritative. Scout candidates are not measurements of your application's quality.
 
-## Work continues after pausing
+## Evaluation or terminal problems
 
-Pause is durable and gates automatic work, but an already-active intent can continue and explicit commands remain available. Inspect the active intent and audit timestamps to distinguish prior work from a new automatic dispatch. `/openmerit cancel` records abandonment; it does not terminate a provider request or harness process. The standalone terminal is independent of Pi pause; stop its CLI with Ctrl+C to stop collection. If a new automatic intent starts while the persisted pause is active, record a sanitized reproduction and your installed version.
+The standalone products are separate from Pi setup. Use [agent-evaluation diagnostics](agent-evaluations.md) for tasks, graders, budgets and execution blocks, or [terminal diagnostics](inference-terminal.md) for missing inventory and request evidence. A populated dashboard is not proof of successful evaluations. Never mark missing or rejected evidence successful merely to advance a workflow.
 
 ## Report a reproducible problem
 
-Include the OpenMerit, Pi, and Node.js versions, the triggering command or signal, the lifecycle stage, and a sanitized error. Remove credentials, private traces, and proprietary evaluation data before sharing anything in a public issue.
+Include OpenMerit, Pi and Node.js versions, the client/scope, the triggering command, and a sanitized error. Remove credentials, private paths, prompts and proprietary data before sharing. Do not attach credential or connection files containing private configuration.
 
 [Open an issue on GitHub](https://github.com/laz-aslam/openmerit/issues)