openmerit 0.3.4 → 0.3.5

package/CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.3.5 — 2026-11-05
+
+- Add optional headless Langfuse setup through the CLI, HTTP API and `scout_connection` MCP tool. A separately approved `scout:connections` key manages the authenticated user's connection and evaluation data; recommendation keys retain their existing permission. Viewer credentials enter through protected CLI input or trusted client configuration, never model-visible tool arguments. The web connection UI remains available, with revised spacing, mobile controls and the official Langfuse logo. Hosted features require a matching Scout deployment; saving Langfuse credentials also requires its dedicated encryption secret. Publishing this package does not deploy or enable the hosted connector.
+
 ## 0.3.4 — 2026-11-05
 
 - `openmerit scout --results` refuses a file with more than 50 results before sending it, instead of failing with HTTP 413; split larger reports into files of up to 50.

package/dist/pi/scout-extension.mjs

@@ -141,6 +141,36 @@ aggregates only, never inputs, outputs or records.
 Scout ranks them above public benchmarks for this user only; models whose
 \`quality.measured\` is true rest on them.
 
+Swap one agent at a time, holding the other agents, prompts, fixtures and controls
+fixed. Use each agent's own checks to attribute failures. If all candidates score
+100%, the eval cannot separate their quality: add harder labelled cases before
+claiming equivalent quality. Cost may break the observed tie, not prove equality.
+
+If the user has Langfuse, use \`openmerit connect langfuse --help\`: the web UI is
+optional. Headless setup requires a separate connection-manager approval with
+\`openmerit connect langfuse --authorize --no-open\`. Relay its approval link and
+expiry; after approval, rerun that command. Existing recommendation keys cannot
+manage connections. The user creates a separate Viewer key; receive it through
+protected stdin or a secret-manager-injected environment, never chat, tool
+arguments, shell literals, traced commands or tracked files. The CLI accepts
+non-secret settings via \`--config mappings.json\` and reads LANGFUSE_PUBLIC_KEY
+and LANGFUSE_SECRET_KEY from its environment, or reads full settings via --stdin.
+A separate trusted MCP connection with the management key exposes
+\`scout_connection\` (status/configure/sync/disconnect/clear). For configure, keys
+must come from client-controlled HTTP headers outside model context; if the
+client cannot do this securely, use protected CLI input or the web UI instead.
+After they confirm connection and score/model mappings, call \`scout_models\` with
+\`sync: "langfuse"\` to refresh aggregate measurements before recommending. The CLI
+equivalent is \`openmerit scout --conversation <id> --sync langfuse\`. Never read
+trace bodies to work around missing mappings or untagged runs; use the aggregate
+report path instead. Results are stored per user, used in new conversations and
+expire after 90 days. Disconnect deletes the Viewer key and imported aggregates;
+Delete all evaluation data also deletes manually reported aggregates. Obtain
+consent before deleting: headless CLI requires --yes, MCP requires confirm: true.
+Deleting the connection does not revoke the separate management device key.
+Past conversation text is not erased. Never put record IDs, emails or secrets in
+\`scope\` or \`source\`; use labels such as "coding agent" and "held-out eval".
+
 ## Boundaries
 
 Ask before adding providers or credentials, spending, deploying or external

package/dist/terminal/scout-update.json

@@ -1 +1 @@
-{"format":1,"contract":"scout-recommend-v1","version":"0.3.4","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Measuring candidates\n\nEach task also has an `evaluation`: the checks (per model call, agent run or\nend-to-end outcome), labelled examples per candidate and when to stop a losing\ncandidate. Offer to run it before the user commits to a model. Use the evals the\nproject already has, such as datasets and experiments in its eval or tracing\nplatform, or its own eval scripts and fixtures: add the shortlisted candidates as\nextra runs of those evals. Do not build a second eval or copy their datasets.\nIf the project has no eval for the task, propose a minimal one built from its\nexisting traces and the decisions people already make, and create it only where\nand how the user agrees. Ask before spending on model calls or writing to an\neval platform.\n\nReport what you measured with `scout_models`: the same `conversationId` and\n`results`, one aggregate per model and metric (`level`, `task` title, `scope`,\n`model` endpoint or model ID, `metric`, `value`, `higherIsBetter`,\n`samples`, `source`). Set `kind` to `usd` or `seconds` for measured cost\nor time (per call at the task level, per run at the agent level), so Scout uses\nyour cost instead of its estimate. Report each agent's own checks at the agent\nlevel: an end-to-end result also measures the agents you did not swap. Send\naggregates only, never inputs, outputs or records.\nScout ranks them above public benchmarks for this user only; models whose\n`quality.measured` is true rest on them.\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. When an install or login waits for approval (exit code\n10), show the user the approval link and its UTC expiry in your reply, not only in\ncommand output, then rerun the same command once they approve. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}
+{"format":1,"contract":"scout-recommend-v2","version":"0.3.5","skill":"---\nname: openmerit-scout\ndescription: Consults OpenMerit Scout before choosing, adding, downloading, replacing or configuring any application inference model, including implicit choices while building features, pretrained browser/local models, SDK defaults, embeddings, OCR, speech, classification, detection, segmentation, LLMs and agents. Not for unrelated edits, ordinary inference calls or this coding agent's own model.\n---\n\n# OpenMerit · Scout\n\nBefore introducing, recommending, replacing or configuring an application\ninference model, call the `scout_models` tool on the `openmerit` MCP server.\nThis applies even when the user never mentions models or Scout. Downloading local\nweights, using a pretrained browser model or accepting an SDK default is a model\ndecision: consult Scout before model-specific code, installs or downloads.\nPreserve the user's explicit model choice; assess it rather than silently replacing\nit. Leave unchanged, approved application selections in place.\n\n## Project policy\n\nRead `.openmerit/scout.json` before searching. If it is absent, ask once whether this\nproject uses **existing-only** (only endpoints it can already reach) or\n**existing-first** (prefer them, ask before searching the whole catalogue), and\nwhich public catalogue endpoint IDs it can access. Save the answer:\n\n```sh\nnpx -y openmerit@latest project --policy existing-only --endpoint-id <id> ...\n```\n\nNever infer access from a model name, SDK or environment-variable name. Never\nrecord credentials, private URLs, prompts or customer data. An empty inventory\nis not permission to search all models.\n\n## Searching\n\nCall `scout_models` with a non-confidential task description as `prompt` and\n`endpointIds` set to the saved inventory, including `[]` when empty. Send\nonly task requirements, never source files, secrets or real customer records.\nIf the MCP tool is unavailable, run\n`npx -y openmerit@latest scout \"<task>\" --json`; it applies the saved inventory.\n\nScout is a conversation with the agent behind openmerit.ai/scout. When\n`status` is `needs_answers`, fill in what the user or project already states,\nshow the user Scout's brief and remaining questions, and wait for their answers.\nThen call again with the returned `conversationId` and `brief` set to the\nrewritten components, or `skip: true` if the user wants Scout to assume. Never\nanswer Scout's questions with guesses. Send later refinements to the same\n`conversationId`. A call with only a `conversationId` reads an interrupted turn.\n\nWhen `status` is `recommended`, Scout has split the work into `tasks`, one\nper model, each with up to five ranked `models`; cover every task. Each\nmodel's `cost` names the `endpointId` and `providerModelId` to call.\nExisting-only never broadens. For existing-first, if no existing candidate fits a\ntask, explain the gap and ask before a new conversation without `endpointIds`.\nCompare quality evidence, reference cost/time and watch-outs. Prefer the cheapest\nusable option the evidence supports, without claiming benchmarks prove\napplication accuracy. Use the project's actual SDK and protocol.\n\n## Measuring candidates\n\nEach task also has an `evaluation`: the checks (per model call, agent run or\nend-to-end outcome), labelled examples per candidate and when to stop a losing\ncandidate. Offer to run it before the user commits to a model. Use the evals the\nproject already has, such as datasets and experiments in its eval or tracing\nplatform, or its own eval scripts and fixtures: add the shortlisted candidates as\nextra runs of those evals. Do not build a second eval or copy their datasets.\nIf the project has no eval for the task, propose a minimal one built from its\nexisting traces and the decisions people already make, and create it only where\nand how the user agrees. Ask before spending on model calls or writing to an\neval platform.\n\nReport what you measured with `scout_models`: the same `conversationId` and\n`results`, one aggregate per model and metric (`level`, `task` title, `scope`,\n`model` endpoint or model ID, `metric`, `value`, `higherIsBetter`,\n`samples`, `source`). Set `kind` to `usd` or `seconds` for measured cost\nor time (per call at the task level, per run at the agent level), so Scout uses\nyour cost instead of its estimate. Report each agent's own checks at the agent\nlevel: an end-to-end result also measures the agents you did not swap. Send\naggregates only, never inputs, outputs or records.\nScout ranks them above public benchmarks for this user only; models whose\n`quality.measured` is true rest on them.\n\nSwap one agent at a time, holding the other agents, prompts, fixtures and controls\nfixed. Use each agent's own checks to attribute failures. If all candidates score\n100%, the eval cannot separate their quality: add harder labelled cases before\nclaiming equivalent quality. Cost may break the observed tie, not prove equality.\n\nIf the user has Langfuse, use `openmerit connect langfuse --help`: the web UI is\noptional. Headless setup requires a separate connection-manager approval with\n`openmerit connect langfuse --authorize --no-open`. Relay its approval link and\nexpiry; after approval, rerun that command. Existing recommendation keys cannot\nmanage connections. The user creates a separate Viewer key; receive it through\nprotected stdin or a secret-manager-injected environment, never chat, tool\narguments, shell literals, traced commands or tracked files. The CLI accepts\nnon-secret settings via `--config mappings.json` and reads LANGFUSE_PUBLIC_KEY\nand LANGFUSE_SECRET_KEY from its environment, or reads full settings via --stdin.\nA separate trusted MCP connection with the management key exposes\n`scout_connection` (status/configure/sync/disconnect/clear). For configure, keys\nmust come from client-controlled HTTP headers outside model context; if the\nclient cannot do this securely, use protected CLI input or the web UI instead.\nAfter they confirm connection and score/model mappings, call `scout_models` with\n`sync: \"langfuse\"` to refresh aggregate measurements before recommending. The CLI\nequivalent is `openmerit scout --conversation <id> --sync langfuse`. Never read\ntrace bodies to work around missing mappings or untagged runs; use the aggregate\nreport path instead. Results are stored per user, used in new conversations and\nexpire after 90 days. Disconnect deletes the Viewer key and imported aggregates;\nDelete all evaluation data also deletes manually reported aggregates. Obtain\nconsent before deleting: headless CLI requires --yes, MCP requires confirm: true.\nDeleting the connection does not revoke the separate management device key.\nPast conversation text is not erased. Never put record IDs, emails or secrets in\n`scope` or `source`; use labels such as \"coding agent\" and \"held-out eval\".\n\n## Boundaries\n\nAsk before adding providers or credentials, spending, deploying or external\nwrites. Report recommendation, endpoint connectivity and application checks as\nseparate results, each only after it succeeds. Do not switch this coding agent's\nown model.\n\nIf Scout fails or has no suitable candidate/evidence for the required deployment,\nsay so and pause the model decision; do not substitute a remembered recommendation\nor SDK default. Continue only model-independent work. Local/browser models are not\nexempt when the catalogue lacks coverage. When an install or login waits for approval (exit code\n10), show the user the approval link and its UTC expiry in your reply, not only in\ncommand output, then rerun the same command once they approve. If it reports that the device key is missing,\nexpired or revoked, run `npx -y openmerit@latest login` and then\n`npx -y openmerit@latest install` again. Never ask the user to paste a key.\n","rule":"# Application model decisions · OpenMerit Scout\n\nBefore choosing, recommending, adding, downloading, replacing or configuring\nANY application inference model, load the `openmerit-scout` skill and consult\n`scout_models` on the `openmerit` MCP server before committing to a model.\nThis applies even when the user only asks you to build a feature, never mentions\nmodels or Scout, or names a model themselves (assess it; do not silently replace it).\n\nA pretrained model, local weights, browser inference, an SDK's default model,\nembeddings, OCR, speech, image classification, detection and segmentation all\ncount as model choices, not implementation details. Identify this decision before\nwriting model-specific code, installing a model package or downloading weights.\n\nFollow the skill's project access policy. If it is missing, ask and save it first.\nScout may answer with clarifying questions: relay them to the user and continue\nthe same Scout conversation with their answers before choosing.\nIf Scout is unavailable, fails or has no suitable evidence/candidate, disclose the\ngap and pause the model decision; do not fall back to a remembered or SDK-default\nmodel. You may continue work that does not commit to a model.\n\nDo not consult Scout for unrelated edits, normal inference calls using an unchanged\napproved model, or this coding agent's own model. Do not reselect approved models\nunless the task introduces or changes a model decision.\n\nWhen the user says \"update openmerit\", run\n`npx -y openmerit@latest update --scope user` (use `--scope project` if this\nrule is in the project's .claude/rules directory). Report the result, including\nany conflict or manual-upgrade requirement. Do not delete edited files, bypass\npermissions or rerun login automatically. A new session loads updated instructions.\n"}

package/dist/terminal/src/browser.d.mts

@@ -0,0 +1,9 @@
+export function browserCommand(url: any, platform: NodeJS.Platform | undefined, env: NodeJS.ProcessEnv | undefined, allowed: any): (string | any[])[] | null;
+export function openBrowser(url: any, { platform, env, spawnProcess, timeoutMs, allowed }?: {
+    platform?: NodeJS.Platform | undefined;
+    env?: NodeJS.ProcessEnv | undefined;
+    spawnProcess?: typeof spawn | undefined;
+    timeoutMs?: number | undefined;
+    allowed?: ((url: string) => boolean) | undefined;
+}): Promise<any>;
+import { spawn } from 'node:child_process';

package/dist/terminal/src/browser.mjs

@@ -10,7 +10,7 @@ export function browserCommand(url, platform = process.platform, env = process.e
   return null;
 }
 
-export async function openBrowser(url, { platform = process.platform, env = process.env, spawnProcess = spawn, timeoutMs = 3000, allowed } = {}) {
+export async function openBrowser(url, { platform = process.platform, env = process.env, spawnProcess = spawn, timeoutMs = 3000, allowed = /** @type {(url: string) => boolean} */ (() => false) } = {}) {
   const command = browserCommand(url, platform, env, allowed);
   if (!command) return false;
   return new Promise(resolve => {

package/dist/terminal/src/cli.mjs

@@ -6,11 +6,14 @@ const help = `OpenMerit
   openmerit scout --help             Ask Scout which models fit your tasks
   openmerit install --help           Connect OpenMerit to your coding agent
   openmerit update --help            Update installed Scout instructions
+  openmerit connect langfuse --help  Connect through CLI or the optional web UI
+  openmerit disconnect langfuse      Delete the connection and imports
   openmerit --help                   Show this help
 `;
 
 async function main() {
   const args = process.argv.slice(2);
+  if (['connect', 'disconnect'].includes(args[0])) { const { connection } = await import('./connections.js'); process.exitCode = await connection(args); return; }
   if (['install', 'status', 'remove', 'login', 'logout', 'project', 'doctor', 'update'].includes(args[0])) { const { main } = await import('./install.mjs'); process.exitCode = await main(args); return; }
   if (args[0] === 'scout') { const { main } = await import('./scout.js'); process.exitCode = await main(args.slice(1)); return; }
   if (!args.length || args.includes('--help') || args.includes('-h')) { console.log(help); return; }

package/dist/terminal/src/connections.js

@@ -0,0 +1,145 @@
+import { readFile } from 'node:fs/promises';
+import { resolve } from 'node:path';
+import { credentialsPath, readCredentials } from "./scout-credentials.js";
+import { openBrowser } from './browser.mjs';
+import { main as install } from './install.mjs';
+const help = `OpenMerit · Langfuse connections
+
+  openmerit connect langfuse                       Open the optional web UI
+  openmerit connect langfuse --authorize [--no-open] [--json]
+  openmerit connect langfuse --config mappings.json [--json]
+  openmerit connect langfuse --stdin [--json]
+  openmerit connect langfuse --status|--sync [--json]
+  openmerit disconnect langfuse --yes [--json]
+  openmerit disconnect langfuse --delete-all --yes [--json]
+
+Authorize a separate connection-manager key once. First-time approval still
+requires account sign-in; --no-open prints a link for approval on any device.
+Rerun --authorize if pending (exit 10). This never replaces recommendation keys.
+The management key is saved separately with owner-only file permissions.
+
+--config reads non-secret region, from (YYYY-MM-DD), tagPrefix, mappings and models.
+Supply LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY through a secret manager.
+--stdin reads the full JSON settings including the Viewer key pair from a protected
+pipe (maximum 18 KB), never echoes them, and does not save them locally.
+Never put keys in arguments, chat, shell history or a tracked file. Use a separate
+Viewer key, not tracing keys. OPENMERIT_CONNECTION_KEY can supply an already
+approved management key to unattended jobs; otherwise the saved key is used.
+
+Disconnect deletes the key pair and imports, not reported results or conversations.
+--delete-all also removes reported results. Both require --yes for headless use.
+Without --yes, disconnect opens the optional browser confirmation.
+`;
+const object = (value) => !!value && typeof value === 'object' && !Array.isArray(value);
+async function settings(file, stdin, root) {
+    let text = '';
+    if (stdin) {
+        if (process.stdin.isTTY)
+            throw new Error('--stdin requires a protected JSON pipe, not interactive key entry.');
+        process.stdin.setEncoding('utf8');
+        let bytes = 0;
+        for await (const chunk of process.stdin) {
+            bytes += Buffer.byteLength(chunk);
+            if (bytes > 18_000)
+                throw new Error('Connection settings exceed 18 KB.');
+            text += String(chunk);
+        }
+    }
+    else {
+        try {
+            text = await readFile(resolve(root, file ?? ''), 'utf8');
+        }
+        catch {
+            throw new Error('Could not read connection settings.');
+        }
+    }
+    if (Buffer.byteLength(text) > 18_000)
+        throw new Error('Connection settings exceed 18 KB.');
+    let config;
+    try {
+        config = JSON.parse(text);
+    }
+    catch {
+        throw new Error('Connection settings must be a JSON object.');
+    }
+    const fields = ['region', 'from', 'tagPrefix', 'mappings', 'models', ...(stdin ? ['publicKey', 'secretKey'] : [])];
+    if (!object(config) || Object.keys(config).some(key => !fields.includes(key)))
+        throw new Error('Use non-secret settings in --config; credentials must come from a secret manager or --stdin.');
+    if (!stdin)
+        config = { ...config, publicKey: process.env.LANGFUSE_PUBLIC_KEY, secretKey: process.env.LANGFUSE_SECRET_KEY };
+    if (!object(config) || typeof config.publicKey !== 'string' || typeof config.secretKey !== 'string' || !config.publicKey || !config.secretKey)
+        throw new Error('Supply a separate Viewer key pair through a secret manager or --stdin.');
+    return config;
+}
+export async function connection(args, { root = process.cwd(), print = console.log, printError = console.error } = {}) {
+    if (args.includes('--help') || args.includes('-h')) {
+        print(help);
+        return 0;
+    }
+    const [command, provider, ...flags] = args;
+    if (!['connect', 'disconnect'].includes(command ?? '') || provider !== 'langfuse')
+        throw new Error('Use openmerit connect langfuse --help.');
+    let file;
+    const options = new Set();
+    for (let i = 0; i < flags.length; i++) {
+        const flag = flags[i] ?? '';
+        if (options.has(flag))
+            throw new Error('Duplicate connection option.');
+        options.add(flag);
+        if (flag === '--config') {
+            file = flags[++i];
+            if (!file || file.startsWith('-'))
+                throw new Error('--config needs a settings file.');
+        }
+        else if (!['--stdin', '--authorize', '--no-open', '--status', '--sync', '--yes', '--delete-all', '--json'].includes(flag))
+            throw new Error('Unknown connection option. Use --help; never pass keys as arguments.');
+    }
+    const operation = ['--config', '--stdin', '--authorize', '--status', '--sync'].filter(flag => options.has(flag));
+    if (operation.length > 1 || command === 'disconnect' && operation.length || command === 'connect' && (options.has('--yes') || options.has('--delete-all'))
+        || options.has('--no-open') && !options.has('--authorize'))
+        throw new Error('Choose one connection action. Use --help.');
+    if (options.has('--delete-all') && !options.has('--yes'))
+        throw new Error('--delete-all requires --yes.');
+    if (options.has('--authorize'))
+        return install(['login', '--client', 'connector', '--scope', 'user', ...(options.has('--no-open') ? ['--no-open'] : []), ...(options.has('--json') ? ['--json'] : [])], { root, print, printError });
+    if (!operation.length && !options.has('--yes')) {
+        const url = 'https://openmerit.ai/connect/?integration=langfuse';
+        print(`Manage Langfuse in your signed-in browser: ${url}`);
+        await openBrowser(url, { allowed: (value) => value === url }).catch(() => false);
+        return 0;
+    }
+    const server = (process.env.OPENMERIT_URL ?? 'https://openmerit.ai').replace(/\/+$/, '');
+    const url = new URL(server);
+    if (url.username || url.password || url.search || url.hash || url.pathname !== '/' || (url.protocol !== 'https:' && !(url.protocol === 'http:' && ['localhost', '127.0.0.1', '[::1]'].includes(url.hostname))))
+        throw new Error('Use an HTTPS OpenMerit origin, or localhost for development.');
+    const credentials = readCredentials(credentialsPath('connector', 'user', root));
+    const key = process.env.OPENMERIT_CONNECTION_KEY ?? (credentials?.server === server ? credentials.key : undefined);
+    if (!key || !/^om_[A-Za-z0-9_-]{43}$/.test(key))
+        throw new Error('Authorize connection management first: openmerit connect langfuse --authorize --no-open');
+    // Read secrets only after authorization. No credential values reach stdout,
+    // local files, tool arguments, analytics or errors, even on malformed input.
+    const config = file || options.has('--stdin') ? await settings(file, options.has('--stdin'), root) : undefined;
+    const path = options.has('--delete-all') ? 'evaluations' : options.has('--sync') ? 'langfuse/sync' : 'langfuse';
+    const method = command === 'disconnect' ? 'DELETE' : config || options.has('--sync') ? 'POST' : 'GET';
+    let response;
+    try {
+        response = await fetch(`${server}/connect/api/${path}`, { method, redirect: 'error', signal: AbortSignal.timeout(120_000),
+            headers: { Authorization: `Bearer ${key}`, ...(config ? { 'Content-Type': 'application/json' } : {}), 'X-Scout-Analytics': 'off' },
+            ...(config ? { body: JSON.stringify(config) } : {}),
+        });
+    }
+    catch {
+        throw new Error('Could not reach OpenMerit. Check connection status before retrying a change.');
+    }
+    if (!response.ok) {
+        await response.body?.cancel();
+        throw new Error(response.status === 401 || response.status === 403 ? 'Management key rejected. Run openmerit connect langfuse --authorize --no-open.'
+            : `Connection request failed (HTTP ${response.status}). Check Viewer permissions, region and mappings; check status before retrying.`);
+    }
+    const value = await response.json();
+    if (!object(value))
+        throw new Error('OpenMerit returned an invalid connection response.');
+    const state = Object.fromEntries(Object.entries(value).filter(([name]) => ['connected', 'connectedAt', 'syncedAt', 'experiments', 'unmapped', 'results', 'busy', 'cached', 'deleted'].includes(name)));
+    print(JSON.stringify(state, null, 2));
+    return 0;
+}

package/dist/terminal/src/install.d.mts

@@ -0,0 +1,2 @@
+export function main(args: any, dependencies?: {}): Promise<any>;
+export { rule, skill } from "./scout-instructions.ts";

package/dist/terminal/src/install.mjs

@@ -19,7 +19,7 @@ const exitCodes = { ok: 0, error: 1, pending: 10, denied: 11, conflict: 20, unre
 const exec = promisify(execFile);
 const hash = value => createHash('sha256').update(value).digest('hex');
 const policyPath = '.openmerit/scout.json';
-const clients = { 'claude-code': 'Claude Code', pi: 'Pi' };
+const clients = { 'claude-code': 'Claude Code', pi: 'Pi', connector: 'OpenMerit connection manager' };
 
 const help = `OpenMerit · coding-agent setup
 
@@ -45,6 +45,8 @@ It opens the approval page and waits up to 9 minutes for the Approve click
 pending, rerun the same command: it resumes the same request.
 --client defaults to the coding agent running the command. --scope defaults to user.
 --plan shows the changes without making them. Keys are never printed.
+login/logout also accept --client connector for a separate connection-management
+key. This grants no model recommendations and changes no coding-agent install.
 
 Exit codes: 0 done, 10 approval pending, 11 denied or expired, 20 conflict,
 30 OpenMerit unreachable, 1 other error.
@@ -471,6 +473,7 @@ async function runMain(args, { root = process.cwd(), print = console.log, printE
     options.client ??= detectClient() ?? (['install', 'status', 'remove'].includes(action) ? undefined : 'claude-code');
     if (!options.client) throw new Error('Choose the coding agent with --client claude-code or --client pi.');
     if (!clients[options.client]) throw new Error(`Unsupported client "${options.client}". Supported: ${Object.keys(clients).join(', ')}.`);
+    if (options.client === 'connector' && !['login', 'logout'].includes(action)) throw new Error('The connector client supports login/logout only. Use openmerit connect langfuse --help.');
   }
   root = resolve(root);
   telemetry.root = root;
@@ -510,7 +513,7 @@ async function runMain(args, { root = process.cwd(), print = console.log, printE
       : action === 'project' ? await project(root, options)
       : action === 'doctor' ? await doctor(root, options)
       : action === 'logout' ? { action, revoked: await revokeKey(telemetry), note: 'Signed out. This device key was revoked and deleted. Agents configured with it stop working until you run install again.' }
-      : (await revokeKey(telemetry), await login({ client: options.client, scope: options.scope, open: options.open, log, wait, telemetry }), { action, signedIn: true, note: `Signed in. Run "openmerit install --client ${options.client} --scope ${options.scope}" to refresh this installation.` });
+      : (await revokeKey(telemetry), await login({ client: options.client, scope: options.scope, open: options.open, log, wait, telemetry }), { action, signedIn: true, note: options.client === 'connector' ? 'Connection management authorized. Recommendation keys and coding-agent installations are unchanged.' : `Signed in. Run "openmerit install --client ${options.client} --scope ${options.scope}" to refresh this installation.` });
     if (result.conflicts?.length && telemetry.outcome !== 'error') { telemetry.outcome = 'conflict'; telemetry.properties.code = 'install_conflict'; }
     if (options.plan) await telemetry.step(telemetry.stage, telemetry.outcome, { applied: false });
     print(options.json ? JSON.stringify({ ok: action !== 'update' || !['conflict', 'retry', 'manual_required'].includes(result.status), ...result }, null, 2) : render(result));

package/dist/terminal/src/scout-instructions.js

@@ -107,6 +107,36 @@ aggregates only, never inputs, outputs or records.
 Scout ranks them above public benchmarks for this user only; models whose
 \`quality.measured\` is true rest on them.
 
+Swap one agent at a time, holding the other agents, prompts, fixtures and controls
+fixed. Use each agent's own checks to attribute failures. If all candidates score
+100%, the eval cannot separate their quality: add harder labelled cases before
+claiming equivalent quality. Cost may break the observed tie, not prove equality.
+
+If the user has Langfuse, use \`openmerit connect langfuse --help\`: the web UI is
+optional. Headless setup requires a separate connection-manager approval with
+\`openmerit connect langfuse --authorize --no-open\`. Relay its approval link and
+expiry; after approval, rerun that command. Existing recommendation keys cannot
+manage connections. The user creates a separate Viewer key; receive it through
+protected stdin or a secret-manager-injected environment, never chat, tool
+arguments, shell literals, traced commands or tracked files. The CLI accepts
+non-secret settings via \`--config mappings.json\` and reads LANGFUSE_PUBLIC_KEY
+and LANGFUSE_SECRET_KEY from its environment, or reads full settings via --stdin.
+A separate trusted MCP connection with the management key exposes
+\`scout_connection\` (status/configure/sync/disconnect/clear). For configure, keys
+must come from client-controlled HTTP headers outside model context; if the
+client cannot do this securely, use protected CLI input or the web UI instead.
+After they confirm connection and score/model mappings, call \`scout_models\` with
+\`sync: "langfuse"\` to refresh aggregate measurements before recommending. The CLI
+equivalent is \`openmerit scout --conversation <id> --sync langfuse\`. Never read
+trace bodies to work around missing mappings or untagged runs; use the aggregate
+report path instead. Results are stored per user, used in new conversations and
+expire after 90 days. Disconnect deletes the Viewer key and imported aggregates;
+Delete all evaluation data also deletes manually reported aggregates. Obtain
+consent before deleting: headless CLI requires --yes, MCP requires confirm: true.
+Deleting the connection does not revoke the separate management device key.
+Past conversation text is not erased. Never put record IDs, emails or secrets in
+\`scope\` or \`source\`; use labels such as "coding agent" and "held-out eval".
+
 ## Boundaries
 
 Ask before adding providers or credentials, spending, deploying or external

package/dist/terminal/src/scout.js

@@ -12,6 +12,9 @@ const help = `OpenMerit · Scout
   openmerit scout "<the work models need to do>" [--all-endpoints] [--json]
   openmerit scout --conversation <id> [--brief <component>=<text> ...] [--skip] ["<follow-up>"] [--json]
   openmerit scout --conversation <id> --results <file.json> [--json]
+  openmerit scout --conversation <id> --sync langfuse [--json]
+  openmerit connect langfuse
+  openmerit disconnect langfuse
 
 Asks Scout, the agent behind openmerit.ai/scout. Scout may first return a task
 brief with clarifying questions: answer them with --brief (components: input,
@@ -24,6 +27,13 @@ samples, source}), never inputs or outputs. kind is quality (the default), usd
 or seconds; measured usd replaces Scout's cost estimate. Scout ranks them above
 public benchmarks, for your recommendations only. A turn usually takes one to three minutes.
 
+Connect a separate Langfuse Viewer key through CLI or the optional web UI (see
+openmerit connect langfuse --help), then --sync langfuse refreshes mapped, tagged
+experiment aggregates before recommending.
+Results are stored per user across conversations for 90 days. Disconnect deletes
+the key and imports; use Delete all evaluation data to remove reported results too.
+Past conversations remain. Never put record IDs or emails in source or scope.
+
 A new conversation is limited to the endpoint IDs in ${policyFile}
 (set with openmerit project). --all-endpoints searches the whole catalogue and is
 refused under the existing-only policy. Uses the device key that
@@ -177,6 +187,11 @@ export async function main(args, { root = process.cwd(), print = console.log } =
             request.allEndpoints = true;
         else if (arg === '--json')
             json = true;
+        else if (arg === '--sync') {
+            if (value() !== 'langfuse')
+                throw new Error('--sync supports langfuse.');
+            request.sync = 'langfuse';
+        }
         else if (arg === '--results') {
             const file = value();
             let results;
@@ -190,7 +205,9 @@ export async function main(args, { root = process.cwd(), print = console.log } =
                 throw new Error(`${file} must hold a JSON list of results.`);
             // Scout accepts at most 50 per turn; each call is a turn that recommends again, so splitting is the caller's choice.
             if (results.length > 50)
-                throw new Error(`${file} holds ${results.length} results; Scout accepts at most 50 per call (200 per conversation). Split them into files of up to 50 and report each with --results in turn; Scout recommends again after each.`);
+                throw new Error(`${file} holds ${results.length} results; Scout accepts at most 50 per call. Split them into files of up to 50 and report each with --results in turn; Scout recommends again after each.`);
+            if (results.some(r => isObject(r) && [r.source, r.scope].some(v => typeof v === 'string' && /(?:\b(?:claim|patient|member|FICT|CLM)[-_ ]?\d|\b[0-9a-f]{8}-[0-9a-f-]{27,}\b|[^\s@]+@[^\s@]+\.[^\s@]+)/i.test(v))))
+                print('Warning: source or scope looks like a record ID or email. Send aggregate labels only; remove identifiers before sharing.');
             request.results = results;
         }
         else if (arg.startsWith('-'))

package/dist/terminal/src/telemetry.d.mts

@@ -0,0 +1,30 @@
+export function safeFrames(error: any): {
+    filename: string | undefined;
+    lineno: number;
+    colno: number;
+}[];
+export function errorProperties(error: any): {
+    error_type: any;
+    frames: {
+        filename: string | undefined;
+        lineno: number;
+        colno: number;
+    }[];
+};
+export function setupHeaders(ids?: {}, client?: string): {
+    'X-Scout-Installation-Id'?: any;
+    'X-Scout-Setup-Id'?: any;
+    'X-Scout-Analytics-Account'?: string;
+    'X-Scout-Analytics'?: string;
+    'X-Scout-Analytics-Content': string;
+    'X-Scout-Client-Version'?: any;
+    'X-Scout-Client': string;
+    'X-Scout-Agent': string;
+};
+export function installationProperties(input: any): {} | null;
+export function sendInstallationEvent(input: any): Promise<void>;
+export function commandTelemetry(args: any): (outcome: any, error: any, code: any) => Promise<void>;
+export let clientVersion: any;
+export function telemetryEnabled(defaultEnabled?: boolean): boolean;
+export function setupCorrelationEnabled(): boolean;
+export function setupUUID(value: any): boolean;

package/dist/terminal/src/updates.js

@@ -7,7 +7,7 @@ import { mkdir, open, readFile, rename, rm, stat, writeFile } from 'node:fs/prom
 import { dirname, join, resolve } from 'node:path';
 import { fileURLToPath } from 'node:url';
 import { gunzipSync } from 'node:zlib';
-export const updateContract = 'scout-recommend-v1';
+export const updateContract = 'scout-recommend-v2';
 const registry = 'https://registry.npmjs.org/openmerit/latest';
 const day = 24 * 60 * 60 * 1000;
 const retry = 15 * 60 * 1000;

package/docs/index.md

@@ -16,6 +16,9 @@ Follow [setup](getting-started.md#connect-scout-to-a-coding-agent) and [the Pi g
 
 Use your own representative task checks to evaluate model candidates. The [standalone evaluation CLI is retired in npm 0.3.0](agent-evaluations.md); existing local artifacts are not deleted.
 
+Follow [close the model-selection loop](scout-close-loop.md) to consult, measure,
+report and choose. It distinguishes npm 0.3.4 from the unreleased Langfuse connector.
+
 ## Operate and troubleshoot
 
 [Commands](commands.md) lists the Scout and setup commands. The [Inference Terminal is retired in npm 0.3.0](inference-terminal.md); existing local files are not deleted. [Troubleshooting](troubleshooting.md), [security and data](security.md), and [current limitations](roadmap.md) explain setup failures and product boundaries. [Testing and validation](testing.md) distinguishes automated coverage from product evidence.

package/docs/scout-close-loop.md

@@ -0,0 +1,258 @@
+# Close the model-selection loop
+
+Scout recommends candidates; your own evals decide whether they work. Never put
+inputs, outputs, patient records, source code or secrets in recommendation prompts
+or tool arguments. Connector credentials use the protected paths described below.
+
+**Version requirements:** the Langfuse setup commands below require npm 0.3.5 or
+newer and a matching Scout serving deployment. Saving Langfuse credentials also
+requires connection encryption to be enabled on the server. Publishing the npm
+package alone does not deploy or enable the hosted connector. npm 0.3.4 supports
+reporting aggregates per conversation, not this connector.
+
+## Install and consult
+
+Install the published package, approve the device code in your browser and start a
+new coding-agent session. See [setup](getting-started.md) for Pi and unattended
+reuse of an already-approved device key. Do not copy another user's key.
+
+```sh
+npx -y openmerit@latest install --client claude-code --scope project
+npx -y openmerit@latest scout "Extract invoice line items as structured JSON" --all-endpoints
+```
+
+Use your saved project inventory instead of `--all-endpoints` when policy limits
+access. Existing-only policy cannot be broadened by that flag. Answer Scout's
+questions in the same conversation with `--brief`, or explicitly choose `--skip`.
+Each recommended model names an exact endpoint and provider model ID.
+
+## Measure with your own evals
+
+Add the shortlist as runs of your existing evals, not a second platform. Obtain
+approval before model spend or platform writes. For agents, change **one agent at
+a time** while holding the other models, prompts, tools, fixtures and controls
+fixed. Use that agent's own checks, not just end-to-end success, to attribute
+failures. Include cost and latency per call or agent run.
+
+If every candidate scores 100%, the eval does not distinguish their quality.
+Add harder labelled boundary cases, rare failures and adversarial examples;
+cheaper on this test is not proof of equal quality in production.
+
+## Report aggregates
+
+Report up to 50 aggregates per call, one per model, metric and scope. Use public
+catalogue model or endpoint IDs. `scope` and `source` must be aggregate labels,
+not record IDs, emails or URLs containing private identifiers.
+
+```json
+[
+  {
+    "level": "agent", "kind": "quality", "task": "Invoice extraction",
+    "scope": "Invoice extraction", "model": "<catalogue-model-id>",
+    "metric": "fields-correct", "value": 0.96, "higherIsBetter": true,
+    "samples": 100, "source": "Held-out invoice eval"
+  }
+]
+```
+
+```sh
+openmerit scout --conversation <id> --results results.json
+```
+
+Use `kind: "usd"` or `"seconds"` for measured money or time; they are not quality
+scores. Once the per-user change is deployed, aggregates apply across your
+conversations. Re-reporting the same level/kind/task/scope/model/metric/source
+updates that aggregate rather than duplicating it.
+
+## Connect Langfuse
+
+After release, `openmerit connect langfuse` opens the optional signed-in
+[connection page](https://openmerit.ai/connect/?integration=langfuse). Create a
+**separate Viewer key pair** in Langfuse Project Settings → API Keys. Enter it
+in the password fields there, or use the protected headless paths below. Never
+paste it in chat, command arguments, issues or tracked files. Keep the
+application's tracing keys unchanged. US, EU and HIPAA Cloud hosts
+are supported; self-hosted instances are not. Selecting a Langfuse region does
+not promise that Scout stores or processes your data in that region.
+
+You can connect the key first with empty mappings. To import results, expand
+**Score and model mappings** and supply a start date, experiment-tag prefix,
+score mappings and model aliases. To change saved settings, disconnect and
+reconnect; this deletes the old imports, not reported results.
+
+### Headless CLI setup
+
+First authorize a **separate connection-management key**. This is not an upgrade
+to your existing recommendation key and does not change Claude Code or Pi setup.
+The first account authorization still requires a signed-in person to approve the
+displayed device code; `--no-open` lets them do that on another device. After
+approval, all connector operations can run without the web UI.
+
+```sh
+openmerit connect langfuse --authorize --no-open
+```
+
+If it returns exit 10 (approval pending), approve the matching code and rerun the
+same command. The resumable login stores the management key with mode 0600 in
+`connector/user/credentials.json` under the OpenMerit config directory (normally
+`~/.config/openmerit`). Never display or share that file. The key grants
+`scout:connections`, not `scout:recommend`; it can manage **all evaluation data
+for your account**, not only one project. It cannot approve devices, list other
+device keys, run recommendations or spend on models. Like recommendation keys,
+it expires after 90 days without use and can be revoked from the connection page
+or with `openmerit logout --client connector`. Disconnecting Langfuse does not
+revoke this management credential; revoke it separately when retiring a runner.
+
+Save only non-secret settings in `mappings.json`:
+
+```json
+{
+  "region": "us",
+  "from": "2026-11-01",
+  "tagPrefix": "experiment:",
+  "mappings": [],
+  "models": {}
+}
+```
+
+Have your secret manager inject `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY`
+into the CLI process environment, then run:
+
+```sh
+openmerit connect langfuse --config mappings.json
+openmerit connect langfuse --status
+openmerit connect langfuse --sync
+```
+
+Alternatively, `openmerit connect langfuse --stdin` reads a full JSON object with
+the same settings plus `publicKey` and `secretKey` from a protected pipe, up to
+18 KB. It does not echo or save the Viewer credentials locally. Do not use inline
+JSON, `echo` with literal secrets, command substitutions that print secrets into
+an agent transcript, or an interactive shell history entry. Authorize before
+piping a secret. Environment variables and pipes are not a vault: trust the
+runner and restrict process/debug access, and disable shell tracing and request
+logging. Unattended jobs may instead inject an already-approved
+`OPENMERIT_CONNECTION_KEY` through a secret manager; keep recommendation and
+management credentials separate.
+
+### MCP management
+
+The canonical `/scout/mcp` endpoint exposes `scout_connection` when authenticated
+with the separate management key (or a current browser session). Register a
+separate trusted MCP client connection using `Authorization: Bearer <management
+key>` without putting the actual key in chat or tool arguments. A recommendation
+key still discovers only `scout_models`; a management key discovers only
+`scout_connection`. This separation is enforced on calls, not just discovery.
+
+`scout_connection` supports `status`, `configure`, `sync`, `disconnect` and
+`clear`. `configure` takes the non-secret settings above as `config`. Supply the
+Viewer key pair through **client-controlled HTTP headers**
+`X-Langfuse-Public-Key` and `X-Langfuse-Secret-Key`, sourced from your secret manager,
+not from the model. These headers are consumed only for configuration and are
+never returned in tool results or sent to the recommendation model. Remove them
+from client configuration after connecting. Do not configure a client/proxy that
+records these headers. If your MCP client cannot securely inject headers outside
+model context, have the agent invoke the CLI with protected input instead, or
+use the web UI; do not fall back to putting credentials in tool arguments.
+
+```json
+{"action": "configure", "config": {"region": "us", "from": "2026-11-01", "tagPrefix": "experiment:", "mappings": [], "models": {}}}
+```
+
+This checks and saves the connection, not a sync. Send `{"action":"sync"}` to
+import mapped aggregates without starting inference. For deletion, obtain the
+user's consent and pass `confirm: true` with `disconnect` or `clear`. Tool
+confirmation is not a substitute for consent in your application.
+
+### HTTP API
+
+The same authenticated account store is available without MCP:
+
+| Method | Path | Operation |
+| --- | --- | --- |
+| GET | `/connect/api/langfuse` | Connection status and aggregate counts, never credentials |
+| POST | `/connect/api/langfuse` | Configure/replace using full JSON settings including `publicKey` and `secretKey` |
+| POST | `/connect/api/langfuse/sync` | Import mapped aggregates without inference |
+| DELETE | `/connect/api/langfuse` | Delete the Viewer key pair and imported aggregates |
+| DELETE | `/connect/api/evaluations` | Also delete manually reported aggregates |
+
+Send the management key in the `Authorization: Bearer` header and
+`Content-Type: application/json` for configuration. Browser session tokens still
+work. Recommendation keys, anonymous requests and cross-origin browser requests
+are rejected. No request accepts a target user ID: ownership comes from the
+authenticated identity. Inject credentials into the HTTP client, not command-line
+arguments; keep request bodies/headers out of logs. Check status after an ambiguous
+network failure before retrying a change. None of these paths bypasses the Viewer
+check, explicit mappings, encryption, retention limits or per-user isolation.
+
+### Attribute imported scores
+
+Every scored trace must carry a tag formed from the prefix plus its experiment
+ID, for example `experiment:<id>`. The experiment's metadata must identify the
+model for that agent. Scout does not infer model attribution from timestamps,
+trace contents or the currently selected model. Untagged historical runs are
+not imported: report their aggregates instead. Each mapping specifies an exact
+trace name so other agents in the same experiment cannot contaminate its score.
+
+Example score mapping (replace the task title with the exact Scout task):
+
+```json
+[
+  {
+    "metric": "coding-finding-found", "modelKey": "model_coding",
+    "task": "Coding review", "traceName": "review-coding",
+    "level": "agent", "kind": "quality", "higherIsBetter": true
+  }
+]
+```
+
+Model aliases explicitly map metadata values to Scout model or endpoint IDs:
+
+```json
+{"<experiment metadata model value>": "<Scout catalogue model or endpoint ID>"}
+```
+
+The first import supports numeric/boolean score averages and counts, not strings,
+raw scores, percentile distributions or automatic cost/latency extraction from
+generation bodies. At most eight mappings, 100 aliases and 100 experiments are
+read per sync. Narrow the date if a response or import exceeds its limit. Only
+experiment headers/metadata and aggregate Metrics v2 reads are requested. A
+deliberately invalid dataset-create request checks read-only permission without
+ever sending a valid write payload. Only an authenticated read followed by HTTP
+403 on that probe is accepted; other responses fail closed. Actual Viewer-key
+acceptance remains pending the release check.
+
+Click **Sync aggregates**, or request a fresh recommendation with:
+
+```sh
+openmerit scout --conversation <id> --sync langfuse
+```
+
+MCP uses `sync: "langfuse"`. Successful reads are cached for ten minutes; a
+failed read cannot be retried for one minute. A failed import leaves the previous
+complete set intact. `conversationId` alone reads a saved answer, not a rerank;
+send a follow-up or `sync` to recompute with the current results.
+
+## Choose and delete
+
+Check measured quality, the quality floor, price calibration and limitations
+before adopting a model. Benchmark estimates are not application acceptance.
+Test endpoint connectivity and application behavior separately.
+
+Results belong only to your Scout user account, not other customers. Up to 5,000
+aggregates or 1 MB are retained for 90 days from report/import; expired rows are
+excluded immediately and pruned daily (allow scheduling delay). An evaluation
+account inactive for 90 days is erased by the same cleanup. See the
+[data policy](scout-data-policy.md) for processors and backup limitations.
+
+`openmerit disconnect langfuse` opens the optional browser confirmation;
+`openmerit disconnect langfuse --yes` deletes headlessly, and adding
+`--delete-all` also deletes reported results. **Disconnect &
+delete imports** removes the encrypted key pair and imported aggregates, including
+an in-flight sync. **Delete all evaluation data** also removes manually reported
+aggregates. Neither erases past conversation text or previously generated
+recommendations; do not keep treating those answers as current measurements.
+Contact the address in the policy for broader deletion.
+
+This release changes the installed instruction-data contract. Existing automatic
+updaters stop at that boundary; explicitly reinstall to adopt the new instructions.

package/docs/scout-data-policy.md

@@ -25,6 +25,53 @@ Setup analytics defaults on. Set `OPENMERIT_TELEMETRY=0` or `DO_NOT_TRACK=1` to
 
 Turning sharing off stops future optional collection. It does not stop required processing or delete stored data.
 
+## Evaluation connections
+
+These features require a matching Scout serving deployment; saving Langfuse
+credentials also requires a dedicated server encryption secret. The npm package
+does not deploy or enable the hosted connector. On enabled deployments, reported
+and imported aggregates are held per authenticated user
+in Cloudflare Durable Objects and inform only that user's recommendations across
+conversations. No cross-customer sharing is enabled. Each aggregate records level,
+kind (quality, USD or seconds), task/scope, catalogue model/endpoint ID, metric,
+average/value, sample count, direction, source label and report/import time.
+Do not put record IDs, emails or sensitive information in labels; identifier-like
+source/scope labels trigger warnings, not guaranteed redaction.
+
+Connecting Langfuse sends a separate Viewer key pair and mappings from the
+signed-in browser, headless CLI, authenticated API or trusted MCP client to Scout,
+encrypted at rest with a dedicated Worker secret. Headless management uses a
+separately approved connection-management key; existing recommendation keys do
+not gain that permission. The CLI reads Viewer credentials from a protected stdin
+pipe or secret-manager-injected environment, not command arguments, and does not
+save those Viewer credentials locally. MCP accepts them through client-controlled
+HTTP headers, not model-visible tool arguments. Keep client, proxy and shell logs
+from recording credentials. The management key is stored separately on your
+device until revoked/deleted and can delete all evaluation data for your account;
+disconnecting Langfuse does not revoke it. First-time device authorization still
+requires account sign-in and approval. The web connection UI remains optional.
+Scout requests only experiment headers/metadata and aggregate metric averages
+and counts. It does not request inputs, outputs, trace/observation bodies,
+experiment items, raw scores or score comments. Arbitrary metadata is discarded;
+experiment source labels are hashed. A deliberately invalid write-permission
+probe does not send a valid dataset or create traces. Your Langfuse region does
+not constrain Scout's storage or AI-provider region.
+
+At most 5,000 aggregates or 1 MB are retained per user. Rows expire 90 days after
+report/import, are excluded from recommendations at expiry and pruned daily;
+cleanup may be delayed by platform scheduling. The evaluation account, including
+its encrypted connection, is erased after 90 days without a report, connection
+or successful sync. Backups and infrastructure logs follow provider policies;
+immediate backup erasure has not been independently verified.
+
+On the signed-in [connection page](/connect/?integration=langfuse), disconnecting
+deletes the key and imported aggregates, including imports from an in-flight
+sync. **Delete all evaluation data** also deletes reported aggregates. Past
+conversation text and recommendations remain and may mention prior results;
+request broader deletion through the contact below. Legacy per-conversation
+aggregates migrate when that conversation next receives a turn. Old conversations
+not reopened require the broader deletion request.
+
 ## Storage and retention
 
 - Your browser stores sign-in information, sharing preferences and temporary task drafts.

package/docs/scout.md

@@ -33,6 +33,7 @@ OpenMerit does not run evaluations or host an eval platform. Scout tells you wha
 - **Measured cost.** A result can set `kind` to `usd` or `seconds` (default `quality`): cost or time per model call at the task level, per agent run at the agent level, per outcome at the environment level. Cost and time results are never quality evidence. A model's agent- or task-level `usd` result on a task appears as `cost.measured` (`usdPer1kTasks`, `usdPer1kAgentRuns`, and the `result` it came from), and when every model in a group (measured or unmeasured) has one, the group is ranked by measured cost instead of Scout's catalogue estimate, which can be far off for agent runs.
 - **Environment results and attribution.** An environment-level result also measures every other agent and control. Scout no longer drops a candidate below the bar when its own task- and agent-level quality results on that task match a candidate that reaches it; an environment miss alone does not show the swapped model caused it.
 - **Since the Scout serving deploy of 5 November 2026 (version `a4d46878-5c00-4997-b7e6-0d0abd0187e7`) and npm 0.3.4:** models that tie under the conversation's priority, such as equal quality from the same measured results, are ordered by measured cost when both have one, so a quality priority no longer keeps the benchmark order among measured equals. Results stored before `kind` existed count as quality again, so older conversations keep `quality.measured`. `openmerit scout --results` refuses more than 50 results per call before sending them.
+- **Unreleased source (needs a Scout serving deploy):** the per-task workload is per model call, and Scout's instructions now say so, along with what one agent call usually sends and returns; before, the prompt asked for a per-run workload that was then multiplied by the calls per run. When you report measured cost on a task, every fully priced estimate in that task is scaled by the median ratio of measured to estimated cost among the models you measured, shown as `cost.calibration` (`factor` and the `results` it used); the order among estimates does not change. Catalogue prices still exclude prompt caching, so uncalibrated estimates for agents that reuse a cached prompt stay high. When any shortlisted model has measured cost, only measured cost earns the cheapest label (`best: ["cost"]`), so models that share one catalogue route price no longer all receive it.
 - **Scope.** Results are stored with the conversation for your account and inform only your recommendations. They are not shared with other accounts or used to change Scout's catalogue.
 
 The quality bar now considers what an error costs once your environment's controls, such as a person approving every payment, have acted. Cost remains the default ranking among models that reach the bar.

package/docs/security.md

@@ -12,6 +12,27 @@ OpenMerit accepts no model-provider or Jev credential. Credentials belong to the
 
 Never place secrets in evaluation configuration, task fixtures, Scout project policy, traces, or committed environment files.
 
+### Langfuse connector (unreleased)
+
+The [connection guide](scout-close-loop.md#connect-langfuse) describes the source
+implementation, not npm 0.3.4 or a deployed acceptance result. Only a separate
+Viewer key pair is accepted. A successful read plus a deliberately invalid
+dataset-create permission probe (HTTP 403 required) fails closed for Full keys;
+no valid write payload is sent. Positive Viewer-key acceptance still needs a real
+Viewer key. Fixed US/EU/HIPAA hosts, manual redirects, timeouts and bounded reads
+prevent credential forwarding and unbounded response bodies. The connector reads
+only experiment headers/metadata and aggregate Metrics v2 values, never trace,
+observation, experiment-item or raw-score bodies. Extra metadata fields are
+discarded; experiment source labels are hashes, not names or record IDs.
+
+Credentials and mappings are AES-GCM encrypted in one Cloudflare Durable Object
+per authenticated user, bound to that object's ID with a dedicated Worker secret.
+Keys never enter browser storage, URLs or analytics. Keep that encryption secret
+stable: replacing it invalidates stored connections and requires reconnecting.
+Disconnect deletes the key and imported aggregates; deleting all evaluation data
+also deletes reported aggregates. Neither erases conversation text. Retention,
+limits and processor caveats are in the [data policy](scout-data-policy.md).
+
 ## Local state
 
 `.openmerit/` may hold evaluation configuration, task fixtures, reports and budget usage, and Scout project policy, plus files left by the retired Inference Terminal (`terminal.json`, `inference.jsonl`), which upgrades do not delete. It may reveal task names, model choices, budgets or endpoint identifiers. Treat it as sensitive project metadata even though it must not contain credentials.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.3.4",
+  "version": "0.3.5",
   "description": "Evidence-driven model decisions for coding harnesses.",
   "type": "module",
   "bin": {