openmerit 0.3.10 → 0.3.11

package/CHANGELOG.md

@@ -1,11 +1,16 @@
 # Changelog
 
-## 0.3.10 — 2026-11-06 (release preparation)
+## 0.3.11 — 2026-11-09
+
+- The repository moved to `github.com/openmerit-ai/openmerit-core`. Package metadata (repository, homepage and issues) and the troubleshooting guide's issue link point there. No CLI, installer or MCP behavior changes.
+- Bring the bundled Scout guides up to date with hosted Scout. They describe the shortlist rules deployed on 9 November (unmeasured candidates beaten on both quality estimate and cost are left out; under the quality priority, unmeasured estimates within three points rank by cost) and the agent's requested benchmark evidence. OpenRouter decision models listed beside Vercel AI Gateway's, host route matching across version and lab spellings, and grouped Langfuse metric queries with `Retry-After` cooldowns are described as deployed on 6 November.
+
+## 0.3.10 — 2026-11-06
 
 - Refocus the Scout guide on supported setup, model selection, customer-owned evals and data handling. Preserve historical operator receipts in an internal archive excluded from npm, retain existing section links, remove repository-development instructions from the package README, and distinguish released Langfuse guidance from old preparation notes. This does not change model selection, credentials or permissions.
 - Align the data policy's opening aggregate-results description with its existing per-user, cross-conversation connection contract. Hosted policy publication requires the separate Scout-serving deployment; npm publication alone does not update it.
 
-## 0.3.9 — 2026-11-06 (release preparation)
+## 0.3.9 — 2026-11-06
 
 - Fix Claude Code project setup, status and conflict detection in linked git worktrees. Claude Code 2.1.63 and newer share private project MCP configuration under the main checkout; older versions use the linked worktree root. The installer follows the Claude CLI on PATH, leaves the project's skill and rule in place, and does not migrate or delete existing MCP entries.
 

package/dist/terminal/scout-update.json

@@ -1 +1 @@
-{"format":1,"contract":"scout-recommend-v2","version":"0.3.10","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Measuring candidates\n\nEach task also has an `evaluation`: the checks (per model call, agent run or\nend-to-end outcome), labelled examples per candidate and when to stop a losing\ncandidate. Offer to run it before the user commits to a model. First assess the\nproject's readiness for this task:\n\n- **No eval setup:** propose a customer-owned setup. Langfuse is the default\n  suggestion for its supported aggregate connector, not a requirement: existing\n  local/custom eval scripts can report aggregates without a hosted platform.\n  Phoenix/Arize and Braintrust are alternatives, not supported Scout connectors.\n  Ask which system the user wants before creating accounts or resources.\n- **Setup exists but evidence is insufficient:** keep that system. Write a\n  task-specific eval plan in the project's local Markdown decision record for\n  the user or their coding agent to implement. Name exact candidates/endpoints,\n  quality floor and task/agent/outcome checks, labelled boundary cases and\n  controls, sample size, early-stop rule, spend cap, aggregate mappings and\n  temporary run retention. Explain what the current evidence cannot decide.\n- **Suitable evals exist:** reuse their fixtures and definitions; add only the\n  approved candidate runs. Do not build a second eval or copy their datasets.\n\nOpenMerit does not host or run these evals. The customer or their coding agent\nimplements and maintains them with the customer's tools; Scout keeps Viewer-only\naccess. Ask before model spend or any platform write. Without measured evidence,\nScout may suggest candidates, but do not claim validated model quality.\n\nReport what you measured with `scout_models`: the same `conversationId` and\n`results`, one aggregate per model and metric (`level`, `task` title, `scope`,\n`model` endpoint or model ID, `metric`, `value`, `higherIsBetter`,\n`samples`, `source`). Set `kind` to `usd` or `seconds` for measured cost\nor time (per call at the task level, per run at the agent level), so Scout uses\nyour cost instead of its estimate. Report each agent's own checks at the agent\nlevel: an end-to-end result also measures the agents you did not swap. Send\naggregates only, never inputs, outputs or records.\nScout ranks them above public benchmarks for this user only; models whose\n`quality.measured` is true rest on them.\n\nSwap one agent at a time, holding the other agents, prompts, fixtures and controls\nfixed. Use each agent's own checks to attribute failures. If all candidates score\n100%, the eval cannot separate their quality: add harder labelled cases before\nclaiming equivalent quality. Cost may break the observed tie, not prove equality.\n\n## Preserve evidence before cleanup\n\nBefore removing temporary comparison runs, save a local Markdown decision record\nwith the package/service versions, date, task and policy, exact candidates,\nunchanged controls, metric definitions/directions, sample counts, aggregate\nquality/cost/time, failures and exclusions, chosen model, limitations and next\nreview. Record regret against the cheapest measured candidate meeting the same\nquality floor; write \"not measurable\" when comparable evidence is missing.\nExclude secrets, private record IDs, prompts, outputs and trace bodies.\n\nTemporary comparison traces/records are not permanent: after the record is saved\nand checked, propose the exact newly created comparison runs to remove. Obtain\nexplicit scoped approval, including any cascading deletion, before the customer\nor their coding agent deletes them with customer-controlled write access. Scout's\nViewer key cannot do this. Never delete pre-existing customer records, reusable\nfixtures, dataset definitions or eval definitions. Verify the approved cleanup\nand retained controls, then record the outcome locally. If not approved, leave\nrecords unchanged and state cleanup is pending; do not imply it happened.\n\nIf the user has Langfuse, use `openmerit connect langfuse --help`: the web UI is\noptional. Headless setup requires a separate connection-manager approval with\n`openmerit connect langfuse --authorize --no-open`. Relay its approval link and\nexpiry; after approval, rerun that command. Existing recommendation keys cannot\nmanage connections. The user creates a separate Viewer key; receive it through\nprotected stdin or a secret-manager-injected environment, never chat, tool\narguments, shell literals, traced commands or tracked files. The CLI accepts\nnon-secret settings via `--config mappings.json` and reads LANGFUSE_PUBLIC_KEY\nand LANGFUSE_SECRET_KEY from its environment, or reads full settings via --stdin.\nA separate trusted MCP connection with the management key exposes\n`scout_connection` (status/configure/sync/disconnect/clear). For configure, keys\nmust come from client-controlled HTTP headers outside model context; if the\nclient cannot do this securely, use protected CLI input or the web UI instead.\nOnce the user has confirmed the connection and score/model mappings (including\na connection already recorded in the project), call `scout_models` with\n`sync: \"langfuse\"` on the first consultation and before a final model decision.\nThe CLI equivalent is `openmerit scout --conversation <id> --sync langfuse`.\nRefreshing that approved connection reads aggregates and updates Scout's cache;\nit does not run candidates or create/change records in Langfuse. Do not ask for\nnew spend or platform-write approval just to refresh it. Do not reconstruct and\nre-report old measurements from project notes instead of using the connection.\nIf connection status is unknown, check `openmerit connect langfuse --status`\nbefore proposing setup; application tracing keys alone do not prove a Scout\nconnection. If sync fails, disclose the failure and any use of cached evidence;\ndo not describe it as fresh. Never read trace bodies to work around missing\nmappings or untagged runs; use the aggregate report path for those gaps instead.\nResults are stored per user, used in new conversations and\nexpire after 90 days. Disconnect deletes the Viewer key and imported aggregates;\nDelete all evaluation data also deletes manually reported aggregates. Obtain\nconsent before deleting: headless CLI requires --yes, MCP requires confirm: true.\nDeleting the connection does not revoke the separate management device key.\nPast conversation text is not erased. Never put record IDs, emails or secrets in\n`scope` or `source`; use labels such as \"coding agent\" and \"held-out eval\".\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. When an install or login waits for approval (exit code\n10), show the user the approval link and its UTC expiry in your reply, not only in\ncommand output, then rerun the same command once they approve. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}
+{"format":1,"contract":"scout-recommend-v2","version":"0.3.11","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Measuring candidates\n\nEach task also has an `evaluation`: the checks (per model call, agent run or\nend-to-end outcome), labelled examples per candidate and when to stop a losing\ncandidate. Offer to run it before the user commits to a model. First assess the\nproject's readiness for this task:\n\n- **No eval setup:** propose a customer-owned setup. Langfuse is the default\n  suggestion for its supported aggregate connector, not a requirement: existing\n  local/custom eval scripts can report aggregates without a hosted platform.\n  Phoenix/Arize and Braintrust are alternatives, not supported Scout connectors.\n  Ask which system the user wants before creating accounts or resources.\n- **Setup exists but evidence is insufficient:** keep that system. Write a\n  task-specific eval plan in the project's local Markdown decision record for\n  the user or their coding agent to implement. Name exact candidates/endpoints,\n  quality floor and task/agent/outcome checks, labelled boundary cases and\n  controls, sample size, early-stop rule, spend cap, aggregate mappings and\n  temporary run retention. Explain what the current evidence cannot decide.\n- **Suitable evals exist:** reuse their fixtures and definitions; add only the\n  approved candidate runs. Do not build a second eval or copy their datasets.\n\nOpenMerit does not host or run these evals. The customer or their coding agent\nimplements and maintains them with the customer's tools; Scout keeps Viewer-only\naccess. Ask before model spend or any platform write. Without measured evidence,\nScout may suggest candidates, but do not claim validated model quality.\n\nReport what you measured with `scout_models`: the same `conversationId` and\n`results`, one aggregate per model and metric (`level`, `task` title, `scope`,\n`model` endpoint or model ID, `metric`, `value`, `higherIsBetter`,\n`samples`, `source`). Set `kind` to `usd` or `seconds` for measured cost\nor time (per call at the task level, per run at the agent level), so Scout uses\nyour cost instead of its estimate. Report each agent's own checks at the agent\nlevel: an end-to-end result also measures the agents you did not swap. Send\naggregates only, never inputs, outputs or records.\nScout ranks them above public benchmarks for this user only; models whose\n`quality.measured` is true rest on them.\n\nSwap one agent at a time, holding the other agents, prompts, fixtures and controls\nfixed. Use each agent's own checks to attribute failures. If all candidates score\n100%, the eval cannot separate their quality: add harder labelled cases before\nclaiming equivalent quality. Cost may break the observed tie, not prove equality.\n\n## Preserve evidence before cleanup\n\nBefore removing temporary comparison runs, save a local Markdown decision record\nwith the package/service versions, date, task and policy, exact candidates,\nunchanged controls, metric definitions/directions, sample counts, aggregate\nquality/cost/time, failures and exclusions, chosen model, limitations and next\nreview. Record regret against the cheapest measured candidate meeting the same\nquality floor; write \"not measurable\" when comparable evidence is missing.\nExclude secrets, private record IDs, prompts, outputs and trace bodies.\n\nTemporary comparison traces/records are not permanent: after the record is saved\nand checked, propose the exact newly created comparison runs to remove. Obtain\nexplicit scoped approval, including any cascading deletion, before the customer\nor their coding agent deletes them with customer-controlled write access. Scout's\nViewer key cannot do this. Never delete pre-existing customer records, reusable\nfixtures, dataset definitions or eval definitions. Verify the approved cleanup\nand retained controls, then record the outcome locally. If not approved, leave\nrecords unchanged and state cleanup is pending; do not imply it happened.\n\nIf the user has Langfuse, use `openmerit connect langfuse --help`: the web UI is\noptional. Headless setup requires a separate connection-manager approval with\n`openmerit connect langfuse --authorize --no-open`. Relay its approval link and\nexpiry; after approval, rerun that command. Existing recommendation keys cannot\nmanage connections. The user creates a separate Viewer key; receive it through\nprotected stdin or a secret-manager-injected environment, never chat, tool\narguments, shell literals, traced commands or tracked files. The CLI accepts\nnon-secret settings via `--config mappings.json` and reads LANGFUSE_PUBLIC_KEY\nand LANGFUSE_SECRET_KEY from its environment, or reads full settings via --stdin.\nA separate trusted MCP connection with the management key exposes\n`scout_connection` (status/configure/sync/disconnect/clear). For configure, keys\nmust come from client-controlled HTTP headers outside model context; if the\nclient cannot do this securely, use protected CLI input or the web UI instead.\nOnce the user has confirmed the connection and score/model mappings (including\na connection already recorded in the project), call `scout_models` with\n`sync: \"langfuse\"` on the first consultation and before a final model decision.\nThe CLI equivalent is `openmerit scout --conversation <id> --sync langfuse`.\nRefreshing that approved connection reads aggregates and updates Scout's cache;\nit does not run candidates or create/change records in Langfuse. Do not ask for\nnew spend or platform-write approval just to refresh it. Do not reconstruct and\nre-report old measurements from project notes instead of using the connection.\nIf connection status is unknown, check `openmerit connect langfuse --status`\nbefore proposing setup; application tracing keys alone do not prove a Scout\nconnection. If sync fails, disclose the failure and any use of cached evidence;\ndo not describe it as fresh. Never read trace bodies to work around missing\nmappings or untagged runs; use the aggregate report path for those gaps instead.\nResults are stored per user, used in new conversations and\nexpire after 90 days. Disconnect deletes the Viewer key and imported aggregates;\nDelete all evaluation data also deletes manually reported aggregates. Obtain\nconsent before deleting: headless CLI requires --yes, MCP requires confirm: true.\nDeleting the connection does not revoke the separate management device key.\nPast conversation text is not erased. Never put record IDs, emails or secrets in\n`scope` or `source`; use labels such as \"coding agent\" and \"held-out eval\".\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. When an install or login waits for approval (exit code\n10), show the user the approval link and its UTC expiry in your reply, not only in\ncommand output, then rerun the same command once they approve. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}

package/docs/scout-close-loop.md

@@ -314,6 +314,18 @@ failed read cannot be retried for one minute. A failed import leaves the previou
 complete set intact. `conversationId` alone reads a saved answer, not a rerank;
 send a follow-up or `sync` to recompute with the current results.
 
+**Deployed 6 November 2026:** aggregate queries group mapped score names
+within each experiment and trace role, rather than requesting every metric
+separately. Experiment tags and explicit model aliases still isolate attribution.
+A rate-limited sync honors Langfuse's `Retry-After`
+duration, persists the cooldown and continues to report failure during it; a
+failed refresh is not a successful cache hit. Without a valid duration the
+fallback is one minute. Langfuse's Metrics quota can be hourly or daily and is
+shared across the organization's projects and keys, so repeatedly retrying or
+changing keys does not solve exhaustion. See [Langfuse API limits](https://langfuse.com/faq/all/api-limits).
+This server update does not grant access to trace or observation bodies and
+does not execute models or change Langfuse records.
+
 ### When a sync imports zero aggregates
 
 A connected Viewer key and a successful experiment read do not establish that

package/docs/scout.md

@@ -63,6 +63,13 @@ explicitly use `--skip` to continue with stated assumptions. A recommendation
 names exact model/endpoint IDs, quality evidence, estimated workload cost and gaps.
 Cheapest meeting the quality bar is the default; an explicit quality or recency
 priority changes the order. The bar is an estimate, not a measured accuracy floor.
+Under the quality priority, unmeasured quality estimates within three points of
+the best one not yet placed count as tied, and the cheaper model ranks first.
+Measured models keep their measured order.
+An unmeasured candidate that another unmeasured candidate beats on both quality
+estimate and estimated cost is left out, so a shortlist can have fewer than five
+models. Candidates at the same price stay, and measured candidates are never left
+out this way.
 Unknown processing location is not evidence of residency compliance.
 
 ## Close the loop
@@ -167,11 +174,51 @@ asynchronous and broader deletion may require contacting the policy's address.
 
 ## Refresh and data handling
 
+### Task-relevant catalogue evidence — deployed, 9 November 2026
+
+The agent's internal `get_model_details` tool accepts optional `benchmarkIds`:
+up to 40 benchmark or leaderboard IDs from task research, shared across up to
+eight requested models. When supplied, only those results are returned, even
+when their standings are weaker than unrelated evidence. Split larger selections
+across calls; an empty list selects no evidence. Without this option, the tool
+keeps its strongest-40 benchmark and strongest-30 leaderboard defaults.
+
+Each model's `evidenceCoverage` reports distinct result `total` and `omitted`
+counts separately for `benchmarks` and `leaderboards`. Omitted counts include
+results outside an explicit selection as well as results beyond default caps.
+`missingRequested` lists requested IDs with no result for that model, including
+unknown IDs; absence is not a zero score. Hard constraints and endpoint inventory
+still apply: details retain `constraintViolation` rather than hiding an
+ineligible model's record or making it eligible.
+
+Equal chat benchmark scores share competition ranks (1, 1, 3), including ranks
+shown in citations. Those ranks remain among all measured catalogue models, not
+just those meeting a filter. Leaderboards keep their own ranks and direction.
+Hosted Scout has served this since 9 November 2026; it does not by itself
+establish better shortlist quality. The homepage has no FAQ block; no homepage
+claim changes.
+
 Public benchmarks, catalogue prices and vendor metadata may be incomplete or
 outdated; citations do not certify model quality, access, residency or zero
 retention. Verify the exact endpoint in your application. No claim is made that
 all catalogue models can be run in your environment.
 
+**Deployed 6 November 2026:** decision models that OpenRouter routes (supplied
+choices, ordered rubric scores and boolean probabilities through its alpha
+Decisions API) are listed alongside Vercel AI Gateway's. An OpenRouter route
+whose expiration date has passed is no longer listed; a route with a removal
+date still ahead shows it as `retiresAt`, and Scout mentions it when it
+recommends a model through that route. Other hosts' deprecation dates do not
+remove a route. Moving aliases such as OpenRouter's `~lab/…-latest` IDs are
+never treated as one model.
+
+**Deployed 6 November 2026:** more host routes are matched to the model they
+serve when a host spells the version or names the lab differently, for example
+Amazon Bedrock's `eu.cohere.embed-v4:0`, Azure's `embed-v-4-0` or Fireworks'
+`glm-5p2`. A model that OpenRouter or Vercel AI Gateway listed under its own
+spelling, such as `cohere/embed-5-fast` beside `cohere/embed-v5.0-fast`, is
+merged into the existing model, so it no longer appears twice.
+
 On enabled deployments, reported and imported aggregates belong to your account
 and apply across conversations. They are not shared with other customers. The
 connector reads experiment headers/metadata and aggregate metrics, never trace

package/docs/troubleshooting.md

@@ -47,4 +47,4 @@ Both are retired in npm 0.3.0; see the [Inference Terminal](inference-terminal.m
 
 Include OpenMerit, Pi and Node.js versions, the client/scope, the triggering command, and a sanitized error. Remove credentials, private paths, prompts and proprietary data before sharing. Do not attach credential or connection files containing private configuration.
 
-[Open an issue on GitHub](https://github.com/laz-aslam/openmerit/issues)
+[Open an issue on GitHub](https://github.com/openmerit-ai/openmerit-core/issues)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.3.10",
+  "version": "0.3.11",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {
@@ -36,11 +36,11 @@
   },
   "repository": {
     "type": "git",
-    "url": "git+https://github.com/laz-aslam/openmerit.git"
+    "url": "git+https://github.com/openmerit-ai/openmerit-core.git"
   },
-  "homepage": "https://github.com/laz-aslam/openmerit#readme",
+  "homepage": "https://github.com/openmerit-ai/openmerit-core#readme",
   "bugs": {
-    "url": "https://github.com/laz-aslam/openmerit/issues"
+    "url": "https://github.com/openmerit-ai/openmerit-core/issues"
   },
   "publishConfig": {
     "access": "public"
@@ -56,7 +56,7 @@
     "check": "npm run check --workspaces --if-present",
     "test": "npm run test --workspaces --if-present",
     "pretypecheck": "npm run build:site",
-    "typecheck": "tsc -p tsconfig.json && tsc -p tsconfig.scout.json && tsc -p scout/agent/tsconfig.json",
+    "typecheck": "tsc -p tsconfig.json && tsc -p tsconfig.scout.json && tsc -p scout/agent/tsconfig.json && tsc -p email/tsconfig.json",
     "check:typescript": "node scripts/check-typescript.ts",
     "types:workers": "node scripts/worker-types.ts",
     "check:worker-types": "node scripts/worker-types.ts --check",
@@ -68,9 +68,10 @@
     "memo": "node demo/memo/preview.ts",
     "check:demo": "npm ci --ignore-scripts --prefix demo && npm run build:site && tsc -p tsconfig.memo.json && node --test demo/test/*.test.mjs demo/test/*.test.ts",
     "prepare": "npm run build",
-    "verify": "npm test && npm run check && npm run typecheck && npm run check:typescript && npm run build && npm run check:demo && npm run check:scout && npm run check:docs && npm run build:scout && git diff --check",
+    "verify": "npm test && npm run check && npm run typecheck && npm run check:typescript && npm run build && npm run check:demo && npm run check:scout && npm run check:email && npm run check:docs && npm run build:scout && git diff --check",
     "prepublishOnly": "npm run verify",
     "scout:refresh": "node scout/refresh.mjs",
+    "check:email": "node --test email/*.test.ts",
     "check:scout": "node --test scout/test/*.test.mjs scout/test/*.test.ts && node --check scout/worker.mjs",
     "build:scout": "node scripts/build-scout.mjs"
   },
@@ -97,7 +98,6 @@
     "typebox": "1.3.27",
     "typescript": "^5.9.3",
     "web-vitals": "6.2.3",
-    "wrangler": "4.136.3",
-    "yaml": "2.9.1"
+    "wrangler": "4.136.3"
   }
 }