openmerit 0.1.3 → 0.1.4

package/README.md

@@ -142,6 +142,12 @@ Keep that checkout available because pi loads a local package from its path.
    switch. Then send a second message to see which model actually handles it.
    You do not need a watcher terminal or the standalone `trial` command.
 
+   Use `/openmerit pause` to stop the active comparison and suppress automatic
+   comparisons, `/openmerit resume` to enable them for future completed tasks,
+   and `/openmerit compare` to explicitly compare the latest completed task
+   even while automatic comparisons are paused. `/openmerit doctor` runs the
+   sanitized setup checks without leaving Pi.
+
    Candidate comparisons have no tools by default, even when the observed task
    used tools. See **Alpha boundaries** before explicitly enabling candidate
    tools.
@@ -150,6 +156,57 @@ Keep that checkout available because pi loads a local package from its path.
    `"mode": "auto"` and `"auto_apply.enabled": true`, review the score-gain
    and price-ratio thresholds, and run the next task.
 
+### Configure custom or local routes
+
+Pi normally supplies each route's price, context window, output limit, and
+modalities. Some custom and local providers omit that metadata. OpenMerit does
+not guess that a local model is free: add an override keyed by the exact
+`provider:modelId` route in `~/.openmerit/policy.json` instead:
+
+```json
+{
+  "route_overrides": {
+    "ollama:qwen3:8b": {
+      "cost": { "input": 0, "output": 0 },
+      "context_window": 32768,
+      "max_tokens": 4096,
+      "input": ["text"]
+    }
+  }
+}
+```
+
+All fields are optional, but both `cost.input` and `cost.output` are required
+when declaring cost. Prices use Pi's dollars-per-million-token units. The
+override applies only to that exact route; it does not change the stable
+`vendor/model` identity or another provider's route to the same model.
+
+If the provider itself is registered at runtime by a Pi extension, explicitly
+allow that provider-registration file in the same policy. Paths must be
+absolute, existing files:
+
+```json
+{
+  "pi": {
+    "provider_extensions": [
+      "/absolute/path/to/ollama-provider.ts"
+    ]
+  }
+}
+```
+
+OpenMerit still launches subprocesses with extension discovery disabled, then
+loads only these explicit files. Do not add OpenMerit's own extension. An
+allowlisted extension executes code in every candidate, judge, and strategist
+Pi subprocess, so list only provider extensions you trust. Run
+`npx openmerit doctor` or `/openmerit doctor` to validate the files and confirm
+that route overrides match Pi's visible routes.
+
+Existing `0.1.x` policy files remain valid: missing `route_overrides` and
+`pi.provider_extensions` fields normalize to empty safe defaults. `openmerit init`
+continues to preserve an existing policy, so add these fields manually
+only when you need them.
+
 ### Try an invoice-to-JSON task
 
 Use the same one-terminal setup. Start pi with a vision-capable model, for
@@ -193,7 +250,9 @@ the chosen frontier per task. `/openmerit` inside pi shows the current model,
 fallback, the model currently being compared, completed models with quality,
 cost, and latency, trial budget, pending recommendations, and any exact-task result
 measured in another session. When the gate declines an automatic swap, it
-prints reasons and `/openmerit apply` remains available.
+prints reasons and `/openmerit apply` remains available. It also lists every
+route skipped during the current comparison with the relevant policy, pricing,
+modality, or per-trial budget reason.
 
 The daily trial-count and dollar limits come from `~/.openmerit/policy.json`.
 If a limit is reached, the extension reports why it skipped the comparison;
@@ -255,7 +314,9 @@ Pi. An OpenRouter key only adds optional catalog and public-benchmark metadata.
   for automatic session comparisons plus the daily candidate count.
   `max_usd_per_trial` is a conservative admission estimate based on known Pi
   prices and a 4K answer; it is not a provider-side hard cap. Routes without
-  known pricing are excluded from automatic comparisons and cannot auto-apply.
+  known pricing are excluded as candidates and cannot auto-apply. Use an exact
+  `route_overrides` entry for a custom/local route whose price is known; zero
+  cost must be stated explicitly.
 - Each comparison has one observed baseline plus a small candidate slate and
   one quality score per answer. Treat recommendations as experimental evidence,
   not a universal model ranking.
@@ -264,11 +325,10 @@ Pi. An OpenRouter key only adds optional catalog and public-benchmark metadata.
   enrichment source. `HarnessAdapter`, `ModelProviderAdapter`,
   `ObservationSource`, and `EventSink` remain separate integration boundaries;
   Pi and local JSONL are the implementations shipped in this release.
-- Candidate subprocesses can use Pi built-ins and custom/local routes available
-  without loading extensions (for example routes from Pi's model
-  configuration). A provider registered only at runtime by another extension
-  is visible in the route snapshot but cannot yet be executed by the isolated
-  subprocess.
+- Candidate subprocesses can use Pi built-ins, configured custom/local routes,
+  and providers registered by explicitly allowlisted extension files. Normal
+  extension discovery remains disabled, and OpenMerit refuses to load its own
+  extension recursively.
 - JSON state snapshots are replaced atomically. Append-only readers skip and
   report malformed or interrupted lines while retaining later valid records.
   A job owned by a crashed process is reclaimable instead of remaining stuck
@@ -278,8 +338,8 @@ Pi. An OpenRouter key only adds optional catalog and public-benchmark metadata.
 
 | file | contents |
 |---|---|
-| `policy.json` | the policy file (gate thresholds, budgets, intervals) |
-| `harness-state.json` | current/fallback routes, Pi's eligible route snapshot, latest session and settled task |
+| `policy.json` | gate thresholds, budgets, intervals, exact-route metadata overrides, and allowlisted Pi provider extensions |
+| `harness-state.json` | current/fallback routes, Pi's eligible route snapshot, pause state, latest session and settled task |
 | `recommendations.jsonl` | append-only session-bound recommendations, routes, evidence, gate reasons, and status updates |
 | `trials.jsonl` | every model trial point, including its provider route when known |
 | `traces/observations.jsonl` | task observations extracted from session traces |

package/dist/daemon.js

@@ -11,10 +11,10 @@ import { buildRecommendation } from "./recommend.js";
 import { appendJsonl, paths, readJson, readJsonl, writeJson, } from "./store.js";
 import { budgetOk, ensureRubric, recordMeritSpend, recordTrialSpend, runTrial, trialBudgetOk } from "./trials.js";
 import { JUDGE_PREFS } from "./judge.js";
-import { runPiTrial, scorePiRun, settledActiveTask } from "./pi-trials.js";
+import { PiHarnessAdapter, runPiTrial, scorePiRun, settledActiveTask } from "./pi-trials.js";
 import { taskInputKey } from "./task-input.js";
 import { pickNext, STRAT_PREFS } from "./strategist.js";
-import { enrichRouteEntry, routeCatalog, routeKey, routeLabel } from "./routes.js";
+import { applyRouteOverrides, enrichRouteEntry, routeCatalog, routeKey, routeLabel } from "./routes.js";
 import { PiCliChatClient } from "./llm.js";
 import { LocalJsonlEventSink, PiTraceObservationSource, meritEvent } from "./integrations.js";
 function resolveRoutePref(cat, prefs, label, requireImages = false) {
@@ -172,9 +172,13 @@ export async function trialTick(key, policy) {
 /** Compare the one completed active pi task, sequentially, then target its session. */
 export async function autoTaskTick(key, policy, settledTask) {
     const sink = new LocalJsonlEventSink();
-    const task = settledTask === undefined ? settledActiveTask() : settledTask;
+    let task = settledTask === undefined ? settledActiveTask() : settledTask;
     if (!task)
         return null;
+    const routes = applyRouteOverrides(task.routes, policy.route_overrides);
+    const currentRoute = applyRouteOverrides([task.route], policy.route_overrides)[0];
+    task = { ...task, routes, route: currentRoute };
+    const tKey = taskInputKey(task.task, task.images, task.files);
     const marker = settledTask === undefined
         ? `${task.sessionFile}:${task.settledAt}`
         : `${task.sessionFile}:${task.sessionBytes ?? task.settledAt}`;
@@ -202,13 +206,28 @@ export async function autoTaskTick(key, policy, settledTask) {
                 console.log(`[openmerit] OpenRouter enrichment unavailable: ${error.message}`);
             }
         }
+        const excluded = [];
         for (const [id, entry] of [...cat]) {
-            if ((entry.priceKnown !== false && entry.price > policy.max_usd_per_m) ||
-                !providerAllowed(policy, entry.id, entry.route?.provider) ||
-                (task.images.length > 0 && !entry.inputModalities?.includes("image")) ||
-                (id !== routeKey(task.route) && !trialBudgetOk(policy, entry, task.task.length).ok))
+            const reasons = [];
+            if (entry.priceKnown !== false && entry.price > policy.max_usd_per_m)
+                reasons.push(`price $${entry.price.toFixed(2)}/M exceeds $${policy.max_usd_per_m.toFixed(2)}/M policy limit`);
+            if (!providerAllowed(policy, entry.id, entry.route?.provider))
+                reasons.push("provider is denied by policy");
+            if (task.images.length > 0 && !entry.inputModalities?.includes("image"))
+                reasons.push("route does not advertise image input");
+            if (id !== routeKey(task.route)) {
+                const admission = trialBudgetOk(policy, entry, task.task.length);
+                if (!admission.ok && admission.reason)
+                    reasons.push(admission.reason);
+            }
+            if (reasons.length) {
                 cat.delete(id);
+                excluded.push({ model: entry.route ? routeLabel(entry.route) : entry.id, reasons: [...new Set(reasons)] });
+            }
         }
+        if (settledTask !== undefined)
+            emitTrialProgress({ phase: "selection", taskKey: tKey,
+                eligible: [...cat.values()].map((entry) => entry.route ? routeLabel(entry.route) : entry.id), excluded });
         const activeEntry = cat.get(routeKey(task.route));
         if (!activeEntry)
             throw new Error(`active route ${task.route.provider}/${task.route.modelId} is excluded by policy or task capabilities`);
@@ -216,7 +235,8 @@ export async function autoTaskTick(key, policy, settledTask) {
             throw new Error("Pi exposes no eligible alternate model route for this task");
         const judge = resolveRoutePref(cat, [policy.judge_model, ...JUDGE_PREFS, task.model], "judge", task.images.length > 0);
         const strategist = resolveRoutePref(cat, [policy.strategist_model, ...STRAT_PREFS, task.model], "strategist");
-        const client = new PiCliChatClient(undefined, recordMeritSpend);
+        const client = new PiCliChatClient(undefined, recordMeritSpend, policy.pi.provider_extensions);
+        const harness = new PiHarnessAdapter(policy.pi.provider_extensions);
         const { category, benchmarks } = relevantBenchmarks(task.task);
         let benchmarkCandidates = [];
         try {
@@ -227,7 +247,6 @@ export async function autoTaskTick(key, policy, settledTask) {
             console.log(`[openmerit] benchmark shortlist unavailable: ${e.message}`);
         }
         const rubric = await ensureRubric(key ?? "", judge.route, task.task, new Map(), client);
-        const tKey = taskInputKey(task.task, task.images, task.files);
         const points = [];
         const tried = new Set();
         const failedVendors = new Set();
@@ -256,7 +275,7 @@ export async function autoTaskTick(key, policy, settledTask) {
                 emitTrialProgress({ phase: "start", taskKey: tKey,
                     model: pick.route ? routeLabel(pick.route) : pick.model, index: i + 1, total });
             const entry = pick.route ? cat.get(routeKey(pick.route)) : cat.get(pick.model);
-            const { point, costUsd, error } = await runPiTrial(key ?? "", judge.route, task.task, rubric, pick.model, entry, task.cwd, task.images, task.files, undefined, client);
+            const { point, costUsd, error } = await runPiTrial(key ?? "", judge.route, task.task, rubric, pick.model, entry, task.cwd, task.images, task.files, harness, client);
             tried.add(pick.route ? routeKey(pick.route) : pick.model);
             points.push(point);
             recordTrialSpend(costUsd);

package/dist/diagnostics.js

@@ -1,13 +1,14 @@
 /** Read-only diagnostics plus a provider-free self-test for first-run support. */
 import { spawnSync } from "node:child_process";
-import { appendFileSync, existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
+import { appendFileSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
 import { tmpdir } from "node:os";
 import { basename, join } from "node:path";
 import { LocalJsonlEventSink, meritEvent } from "./integrations.js";
 import { piExecutable, availablePiRoutes } from "./pi-trials.js";
 import { DEFAULT_POLICY, loadPolicy, providerAllowed } from "./policy.js";
 import { buildRecommendation } from "./recommend.js";
-import { modelRoute, routeCatalog, routeKey, routeLabel } from "./routes.js";
+import { applyRouteOverrides, modelRoute, routeCatalog, routeKey, routeLabel } from "./routes.js";
+import { validatePiProviderExtensions } from "./pi-config.js";
 import { appendJsonl, paths, readJsonReport, readJsonlReport, writeJson, } from "./store.js";
 function check(id, status, message, details) {
     return { id, status, message, ...(details?.length ? { details } : {}) };
@@ -32,13 +33,34 @@ export function collectDoctorReport() {
     checks.push(piVersion
         ? check("pi", "pass", `Pi detected (${piVersion})`)
         : check("pi", "fail", "Pi is not executable"));
+    let policy = DEFAULT_POLICY;
     try {
-        const policy = loadPolicy(paths.policy());
+        policy = loadPolicy(paths.policy());
         checks.push(check("policy", "pass", `Policy is valid (v${policy.version}, ${policy.mode})`));
     }
     catch (error) {
         checks.push(check("policy", "fail", `Policy is invalid: ${error.message}`));
     }
+    const extensions = validatePiProviderExtensions(policy.pi.provider_extensions);
+    const safeExtensionErrors = extensions.errors.map((message) => {
+        const configured = policy.pi.provider_extensions.find((file) => message.startsWith(`${file}:`));
+        return configured ? message.replace(configured, `local:${basename(configured)}`) : message;
+    });
+    let isolatedExtensionRoutes = null;
+    let extensionLoadError = null;
+    if (!safeExtensionErrors.length && policy.pi.provider_extensions.length && piVersion) {
+        try {
+            isolatedExtensionRoutes = availablePiRoutes(undefined, extensions.valid);
+        }
+        catch {
+            extensionLoadError = "Pi could not load the allowlisted provider extensions in isolated mode";
+        }
+    }
+    checks.push(safeExtensionErrors.length || extensionLoadError
+        ? check("provider-extensions", "fail", "One or more allowlisted Pi provider extensions are unavailable", [...safeExtensionErrors, ...(extensionLoadError ? [extensionLoadError] : [])])
+        : check("provider-extensions", "pass", policy.pi.provider_extensions.length
+            ? `${policy.pi.provider_extensions.length} explicit Pi provider extension${policy.pi.provider_extensions.length === 1 ? " loads" : "s load"} in isolated mode`
+            : "No Pi provider extensions are allowlisted"));
     const stateRead = readJsonReport(paths.harnessState(), {});
     checks.push(!stateRead.valid
         ? check("harness-state", "fail", "harness-state.json is malformed; start Pi to replace it atomically")
@@ -49,16 +71,17 @@ export function collectDoctorReport() {
     const liveSnapshot = routes.length > 0;
     if (!routes.length && piVersion) {
         try {
-            routes = availablePiRoutes();
+            routes = isolatedExtensionRoutes ?? availablePiRoutes(undefined, extensions.valid);
         }
         catch { /* the check below explains it */ }
     }
-    let policy = DEFAULT_POLICY;
-    try {
-        policy = loadPolicy(paths.policy());
-    }
-    catch { /* already reported */ }
+    routes = applyRouteOverrides(routes, policy.route_overrides);
+    const visibleRouteKeys = new Set(routes.map(routeKey));
     routes = routes.filter((route) => providerAllowed(policy, route.meritId, route.provider));
+    const unusedOverrides = Object.keys(policy.route_overrides).filter((key) => !visibleRouteKeys.has(key));
+    checks.push(unusedOverrides.length
+        ? check("route-overrides", "warn", `${unusedOverrides.length} route metadata override${unusedOverrides.length === 1 ? " does" : "s do"} not match a visible route`, unusedOverrides)
+        : check("route-overrides", "pass", `${Object.keys(policy.route_overrides).length} route metadata override${Object.keys(policy.route_overrides).length === 1 ? "" : "s"} applied or ready`));
     const priced = routes.filter((route) => !!route.cost);
     checks.push(routes.length >= 2
         ? check("routes", "pass", `${routes.length} Pi model routes are visible`, routes.slice(0, 8).map((route) => `${routeLabel(route)}${route.cost ? "" : " (price unknown)"}`))
@@ -156,6 +179,16 @@ export function runOfflineVerify() {
             recommendation.policy.reasons.length > 0
             ? check("provider-neutral", "pass", "Exact provider routes and policy reasons survive recommendation")
             : check("provider-neutral", "fail", "Provider route or policy evidence was lost"));
+        const configured = applyRouteOverrides([modelRoute("custom", "model-c")], {
+            "custom:model-c": { cost: { input: 0, output: 0 }, context_window: 4096,
+                max_tokens: 1024, input: ["text"] },
+        })[0];
+        const providerExtension = join(root, "custom-provider.ts");
+        writeFileSync(providerExtension, "export default function provider() {}\n");
+        const extension = validatePiProviderExtensions([providerExtension]);
+        checks.push(configured.cost?.input === 0 && configured.contextWindow === 4096 && extension.valid.length === 1
+            ? check("custom-routes", "pass", "Exact route metadata and allowlisted provider extensions validate")
+            : check("custom-routes", "fail", "Custom route or provider-extension configuration failed"));
         if (recommendation)
             appendJsonl(paths.recommendations(), recommendation);
         appendJsonl(paths.recommendations(), { schemaVersion: 0, id: "legacy", status: "dismissed" });

package/dist/llm.js

@@ -5,6 +5,7 @@ import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { paths } from "./store.js";
 import { legacyOpenRouterRoute } from "./routes.js";
+import { piProviderExtensionArgs } from "./pi-config.js";
 const OR = "https://openrouter.ai/api/v1";
 /** Load OPENROUTER_API_KEY from env or ~/.openmerit/.env. */
 export function loadKey() {
@@ -123,10 +124,12 @@ function textContent(content) {
 export class PiCliChatClient {
     executable;
     onSpend;
+    providerExtensions;
     id = "pi-cli";
-    constructor(executable = process.env.OPENMERIT_PI_BIN?.trim() || "pi", onSpend) {
+    constructor(executable = process.env.OPENMERIT_PI_BIN?.trim() || "pi", onSpend, providerExtensions = []) {
         this.executable = executable;
         this.onSpend = onSpend;
+        this.providerExtensions = providerExtensions;
     }
     chat(model, prompt, _maxTokens, _temperature) {
         return this.run(model, prompt, []);
@@ -148,7 +151,8 @@ export class PiCliChatClient {
             });
             const child = spawn(this.executable, [
                 "--provider", route.provider, "--model", route.modelId, "--mode", "json", "--offline",
-                "--no-extensions", "--no-tools", "--print", ...imageArgs, prompt,
+                "--no-extensions", ...piProviderExtensionArgs(this.providerExtensions),
+                "--no-tools", "--print", ...imageArgs, prompt,
             ], { cwd: dir, stdio: ["ignore", "pipe", "pipe"] });
             let stdout = "";
             let stderr = "";

package/dist/pi-config.js

@@ -0,0 +1,46 @@
+/** Safe, explicit Pi subprocess configuration. */
+import { existsSync, realpathSync, statSync } from "node:fs";
+import { basename, isAbsolute } from "node:path";
+/**
+ * Provider extensions execute as code. Require exact, existing, absolute files
+ * and refuse OpenMerit itself so an isolated trial cannot recursively launch
+ * another merit loop.
+ */
+export function validatePiProviderExtensions(paths) {
+    const valid = [];
+    const errors = [];
+    for (const configured of [...new Set(paths)]) {
+        if (!isAbsolute(configured)) {
+            errors.push(`${configured}: path is not absolute`);
+            continue;
+        }
+        if (!existsSync(configured)) {
+            errors.push(`${configured}: file does not exist`);
+            continue;
+        }
+        let actual = configured;
+        try {
+            actual = realpathSync(configured);
+            if (!statSync(actual).isFile()) {
+                errors.push(`${configured}: path is not a file`);
+                continue;
+            }
+        }
+        catch (error) {
+            errors.push(`${configured}: ${error.message}`);
+            continue;
+        }
+        if (/^openmerit\.[cm]?[jt]s$/i.test(basename(actual))) {
+            errors.push(`${configured}: refusing to load OpenMerit recursively`);
+            continue;
+        }
+        valid.push(actual);
+    }
+    return { valid, errors };
+}
+export function piProviderExtensionArgs(paths) {
+    const checked = validatePiProviderExtensions(paths);
+    if (checked.errors.length)
+        throw new Error(`invalid pi.provider_extensions: ${checked.errors.join("; ")}`);
+    return checked.valid.flatMap((file) => ["--extension", file]);
+}

package/dist/pi-trials.js

@@ -10,6 +10,7 @@ import { paths, readJson } from "./store.js";
 import { sessionFiles, sessionImages, taskInputKey } from "./task-input.js";
 import { legacyOpenRouterRoute, meritModelId, modelRoute, routeKey } from "./routes.js";
 import { directChatClient } from "./llm.js";
+import { piProviderExtensionArgs } from "./pi-config.js";
 function answerText(content) {
     if (typeof content === "string")
         return content;
@@ -20,8 +21,12 @@ function answerText(content) {
 }
 /** Pi-backed harness adapter. Future harnesses can implement the same seam. */
 export class PiHarnessAdapter {
+    providerExtensions;
+    constructor(providerExtensions = []) {
+        this.providerExtensions = providerExtensions;
+    }
     runTask(model, input) {
-        return executePiTask(model, input.task, input.cwd, input.images, input.files);
+        return executePiTask(model, input.task, input.cwd, input.images, input.files, this.providerExtensions);
     }
 }
 const defaultHarness = new PiHarnessAdapter();
@@ -192,10 +197,11 @@ function compactNumber(value) {
     return Math.round(Number(match[1]) * scale);
 }
 /** Read provider/model routes from pi when no live extension snapshot is available. */
-export function availablePiRoutes(snapshot) {
+export function availablePiRoutes(snapshot, providerExtensions = []) {
     if (snapshot?.length)
         return snapshot;
-    const result = spawnSync(piExecutable(), ["--offline", "--list-models"], { encoding: "utf8" });
+    const result = spawnSync(piExecutable(), ["--offline", "--no-extensions",
+        ...piProviderExtensionArgs(providerExtensions), "--list-models"], { encoding: "utf8" });
     if (result.error || result.status !== 0)
         throw new Error("could not list models from pi");
     const routes = [];
@@ -216,7 +222,7 @@ export function availablePiModels() {
     return new Set(availablePiRoutes().map((route) => route.meritId));
 }
 /** Each invocation is a fresh pi run, so model B does not inherit model A's answer. */
-export async function executePiTask(model, task, cwd = process.cwd(), images = [], files = []) {
+export async function executePiTask(model, task, cwd = process.cwd(), images = [], files = [], providerExtensions = []) {
     const route = typeof model === "string" ? legacyOpenRouterRoute(model) : model;
     const imageDir = images.length ? mkdtempSync(join(tmpdir(), "openmerit-pi-image-")) : null;
     const suffix = {
@@ -247,7 +253,8 @@ export async function executePiTask(model, task, cwd = process.cwd(), images = [
         const traceLines = [];
         const child = spawn(piExecutable(), [
             "--provider", route.provider, "--model", route.modelId, "--mode", "json",
-            "--offline", "--no-extensions", "--approve", ...piTrialToolArgs(),
+            "--offline", "--no-extensions", ...piProviderExtensionArgs(providerExtensions),
+            "--approve", ...piTrialToolArgs(),
             "--print", ...imageArgs, ...fileArgs, candidateTask,
         ], { cwd: workspace, stdio: ["ignore", "pipe", "pipe"] });
         let buffer = "";

package/dist/policy.js

@@ -1,7 +1,7 @@
 /** Policy loading, validation, and the swap gate. */
 import { existsSync, readFileSync } from "node:fs";
 export const DEFAULT_POLICY = {
-    version: 1,
+    version: 2,
     mode: "recommend",
     auto_apply: {
         enabled: false,
@@ -16,10 +16,78 @@ export const DEFAULT_POLICY = {
     judge_model: null,
     strategist_model: null,
     max_usd_per_m: 20.0,
+    route_overrides: {},
+    pi: { provider_extensions: [] },
 };
 function num(v, fallback) {
     return typeof v === "number" && Number.isFinite(v) ? v : fallback;
 }
+function finiteAtLeast(value, min, label) {
+    if (typeof value !== "number" || !Number.isFinite(value) || value < min)
+        throw new Error(`${label} must be a finite number >= ${min}`);
+    return value;
+}
+function positiveInteger(value, label) {
+    const result = finiteAtLeast(value, 1, label);
+    if (!Number.isInteger(result))
+        throw new Error(`${label} must be an integer`);
+    return result;
+}
+function routeOverrides(value) {
+    if (value === undefined)
+        return {};
+    if (!value || typeof value !== "object" || Array.isArray(value))
+        throw new Error("route_overrides must be an object keyed by provider:modelId");
+    const result = {};
+    for (const [key, raw] of Object.entries(value)) {
+        const separator = key.indexOf(":");
+        if (separator <= 0 || separator === key.length - 1)
+            throw new Error(`route override ${key} must use provider:modelId`);
+        if (!raw || typeof raw !== "object" || Array.isArray(raw))
+            throw new Error(`route override ${key} must be an object`);
+        const input = raw.input;
+        if (input !== undefined && (!Array.isArray(input) || input.length === 0 ||
+            input.some((item) => item !== "text" && item !== "image")))
+            throw new Error(`route override ${key}.input must contain only text or image`);
+        const cost = raw.cost;
+        let normalizedCost;
+        if (cost !== undefined) {
+            if (!cost || typeof cost !== "object" || Array.isArray(cost))
+                throw new Error(`route override ${key}.cost must be an object`);
+            const c = cost;
+            normalizedCost = {
+                input: finiteAtLeast(c.input, 0, `route override ${key}.cost.input`),
+                output: finiteAtLeast(c.output, 0, `route override ${key}.cost.output`),
+                ...(c.cacheRead === undefined ? {} : {
+                    cacheRead: finiteAtLeast(c.cacheRead, 0, `route override ${key}.cost.cacheRead`),
+                }),
+                ...(c.cacheWrite === undefined ? {} : {
+                    cacheWrite: finiteAtLeast(c.cacheWrite, 0, `route override ${key}.cost.cacheWrite`),
+                }),
+            };
+        }
+        const contextWindow = raw.context_window;
+        const maxTokens = raw.max_tokens;
+        result[key] = {
+            ...(normalizedCost ? { cost: normalizedCost } : {}),
+            ...(contextWindow === undefined ? {} : {
+                context_window: positiveInteger(contextWindow, `route override ${key}.context_window`),
+            }),
+            ...(maxTokens === undefined ? {} : {
+                max_tokens: positiveInteger(maxTokens, `route override ${key}.max_tokens`),
+            }),
+            ...(input === undefined ? {} : { input: [...new Set(input)] }),
+        };
+    }
+    return result;
+}
+function providerExtensions(value) {
+    if (value === undefined)
+        return [];
+    if (!Array.isArray(value) || value.some((item) => typeof item !== "string" || !item.trim()))
+        throw new Error("pi.provider_extensions must be an array of non-empty file paths");
+    return [...new Set(value.map((item) => String(item).trim()))];
+}
 /** Load a policy file, filling defaults for missing keys. Throws on invalid JSON. */
 export function loadPolicy(path) {
     if (!existsSync(path))
@@ -58,6 +126,8 @@ export function loadPolicy(path) {
         judge_model: raw.judge_model ?? null,
         strategist_model: raw.strategist_model ?? null,
         max_usd_per_m: num(raw.max_usd_per_m, d.max_usd_per_m),
+        route_overrides: routeOverrides(raw.route_overrides),
+        pi: { provider_extensions: providerExtensions(raw.pi?.provider_extensions) },
     };
 }
 /** Match either a logical vendor/model id or its concrete route provider. */

package/dist/routes.js

@@ -19,6 +19,21 @@ export function legacyOpenRouterRoute(meritId) {
 export function routeKey(route) {
     return `${route.provider}:${route.modelId}`;
 }
+/** Apply user metadata only to the exact provider route it names. */
+export function applyRouteOverrides(routes, overrides) {
+    return routes.map((route) => {
+        const override = overrides[routeKey(route)];
+        if (!override)
+            return route;
+        return {
+            ...route,
+            ...(override.cost ? { cost: { ...override.cost } } : {}),
+            ...(override.context_window === undefined ? {} : { contextWindow: override.context_window }),
+            ...(override.max_tokens === undefined ? {} : { maxTokens: override.max_tokens }),
+            ...(override.input ? { input: [...override.input] } : {}),
+        };
+    });
+}
 export function pointKey(point) {
     return point.route ? routeKey(point.route) : `openrouter:${point.model}`;
 }

package/dist/standalone.js

@@ -5,10 +5,10 @@ import { fetchCatalog, saveSnapshot } from "./catalog.js";
 import { paretoFrontier, pickBest, pickFallback } from "./frontier.js";
 import { JUDGE_PREFS } from "./judge.js";
 import { loadKey, PiCliChatClient } from "./llm.js";
-import { availablePiRoutes, recordedActiveTask, runPiTrial, scorePiRun } from "./pi-trials.js";
+import { availablePiRoutes, PiHarnessAdapter, recordedActiveTask, runPiTrial, scorePiRun } from "./pi-trials.js";
 import { loadPolicy, providerAllowed } from "./policy.js";
 import { buildRecommendation } from "./recommend.js";
-import { enrichRouteEntry, routeCatalog, routeKey, routeLabel } from "./routes.js";
+import { applyRouteOverrides, enrichRouteEntry, routeCatalog, routeKey, routeLabel } from "./routes.js";
 import { appendJsonl, paths, readJson, taskKey } from "./store.js";
 import { pickNext, STRAT_PREFS } from "./strategist.js";
 import { budgetOk, recordMeritSpend, recordTrialSpend, trialBudgetOk } from "./trials.js";
@@ -20,13 +20,16 @@ function optionalOpenRouterKey() {
         return null;
     }
 }
-function configuredRoutes() {
+function configuredRoutes(policy) {
     const state = readJson(paths.harnessState(), {});
-    let routes = state.routes?.length ? state.routes : availablePiRoutes();
+    let routes = state.routes?.length ? state.routes : availablePiRoutes(undefined, policy.pi.provider_extensions);
     if (state.currentRoute && !routes.some((route) => routeKey(route) === routeKey(state.currentRoute)))
         routes = [state.currentRoute, ...routes];
+    routes = applyRouteOverrides(routes, policy.route_overrides);
     const unique = new Map(routes.map((route) => [routeKey(route), route]));
-    return { routes: [...unique.values()], currentRoute: state.currentRoute ?? null };
+    const currentRoute = state.currentRoute
+        ? applyRouteOverrides([state.currentRoute], policy.route_overrides)[0] : null;
+    return { routes: [...unique.values()], currentRoute };
 }
 function selectorKey(selector) {
     if (!selector)
@@ -111,7 +114,7 @@ export async function runStandaloneTrial(taskFile, rounds) {
         throw new Error("--rounds must be a positive integer");
     const policy = loadPolicy(paths.policy());
     const key = optionalOpenRouterKey();
-    const { routes, currentRoute } = configuredRoutes();
+    const { routes, currentRoute } = configuredRoutes(policy);
     let cat = await buildCatalog(routes, key);
     for (const [id, entry] of [...cat]) {
         if (!entry.route || !providerAllowed(policy, entry.id, entry.route.provider) ||
@@ -123,7 +126,8 @@ export async function runStandaloneTrial(taskFile, rounds) {
     const initial = resolveInitial(cat, cfg, currentRoute);
     const judge = resolveMeritRoute(cat, [cfg.judge_model, policy.judge_model, ...JUDGE_PREFS, initial.id], "judge");
     const strategist = resolveMeritRoute(cat, [cfg.strategist_model, policy.strategist_model, ...STRAT_PREFS, initial.id], "strategist");
-    const client = new PiCliChatClient(undefined, recordMeritSpend);
+    const client = new PiCliChatClient(undefined, recordMeritSpend, policy.pi.provider_extensions);
+    const harness = new PiHarnessAdapter(policy.pi.provider_extensions);
     const { category, benchmarks } = relevantBenchmarks(cfg.task);
     let benchmarkCandidates = [];
     if (key) {
@@ -180,7 +184,7 @@ export async function runStandaloneTrial(taskFile, rounds) {
                 : "    active trace unavailable; running the initial route in a fresh Pi session");
         const outcome = observed
             ? await scorePiRun(key ?? "", judge.route, cfg.task, cfg.eval, entry.id, entry, observed, "trace", [], client)
-            : await runPiTrial(key ?? "", judge.route, cfg.task, cfg.eval, entry.id, entry, cfg.cwd ?? process.cwd(), [], [], undefined, client);
+            : await runPiTrial(key ?? "", judge.route, cfg.task, cfg.eval, entry.id, entry, cfg.cwd ?? process.cwd(), [], [], harness, client);
         tried.add(keyForRoute);
         results.push(outcome.point);
         // Reusing the active Pi answer is observation, not a new candidate call.

package/extension/openmerit.ts

@@ -37,7 +37,7 @@ const JOB_DIR = join(HOME, "watch", "jobs");
 const ENGINE_FILE = join(fileURLToPath(new URL("..", import.meta.url)), "dist", "cli.js");
 const PROGRESS_PREFIX = "[openmerit/progress] ";
 
-interface TrialProgress {
+interface TrialRunProgress {
   phase: "start" | "complete";
   taskKey: string;
   model: string;
@@ -53,19 +53,35 @@ interface TrialProgress {
   changedFiles?: number;
 }
 
+interface TrialSelectionProgress {
+  phase: "selection";
+  taskKey: string;
+  eligible: string[];
+  excluded: { model: string; reasons: string[] }[];
+}
+
+type TrialProgress = TrialRunProgress | TrialSelectionProgress;
+
 function parseTrialProgress(line: string): TrialProgress | null {
   if (!line.startsWith(PROGRESS_PREFIX)) return null;
   try {
     const value = JSON.parse(line.slice(PROGRESS_PREFIX.length)) as TrialProgress;
-    if ((value.phase !== "start" && value.phase !== "complete") ||
-        typeof value.taskKey !== "string" || typeof value.model !== "string" ||
+    if (typeof value.taskKey !== "string") return null;
+    if (value.phase === "selection") {
+      if (!Array.isArray(value.eligible) || !Array.isArray(value.excluded) ||
+          value.eligible.some((item) => typeof item !== "string") ||
+          value.excluded.some((item) => typeof item?.model !== "string" || !Array.isArray(item.reasons) ||
+            item.reasons.some((reason) => typeof reason !== "string"))) return null;
+      return value;
+    }
+    if ((value.phase !== "start" && value.phase !== "complete") || typeof value.model !== "string" ||
         !Number.isInteger(value.index) || !Number.isInteger(value.total) ||
         value.index < 1 || value.total < value.index) return null;
     return value;
   } catch { return null; }
 }
 
-function completedTrialText(trial: TrialProgress): string {
+function completedTrialText(trial: TrialRunProgress): string {
   return `${trial.model}: quality ${(trial.score ?? 0).toFixed(2)}, ` +
     `$${(trial.price ?? 0).toFixed(2)}/M, ${Math.round(trial.latencyMs ?? 0)}ms, ` +
     `run $${(trial.costUsd ?? 0).toFixed(4)}, tools ${trial.toolCalls ?? 0}` +
@@ -77,6 +93,12 @@ interface ExtensionPolicy {
   mode?: string;
   fallback?: { apply_on_error?: boolean };
   budgets?: { max_trials_per_day?: number; max_usd_per_day?: number };
+  route_overrides?: Record<string, {
+    cost?: { input?: number; output?: number; cacheRead?: number; cacheWrite?: number };
+    context_window?: number;
+    max_tokens?: number;
+    input?: ("text" | "image")[];
+  }>;
 }
 
 function trialBudget(): { summary: string; reason: string | null } {
@@ -237,6 +259,7 @@ function latestSessionTaskKey(ctx: ExtensionContext): string | null {
 }
 
 let settledTaskKey: string | null = null;
+let comparisonsPaused = false;
 
 function pendingRecommendations(ctx?: ExtensionContext): Recommendation[] {
   const pending = latestRecommendations().filter((r) => r.status === "pending");
@@ -295,8 +318,8 @@ let fallbackRoute: ModelRoute | null = null;
 
 function routeFor(model: { provider: string; id: string; input?: ("text" | "image")[];
   cost?: { input?: number; output?: number; cacheRead?: number; cacheWrite?: number };
-  contextWindow?: number; maxTokens?: number }): ModelRoute {
-  return {
+  contextWindow?: number; maxTokens?: number }, overrides = loadPolicy().route_overrides ?? {}): ModelRoute {
+  const route: ModelRoute = {
     provider: model.provider,
     modelId: model.id,
     meritId: canonicalModel(model.provider, model.id),
@@ -308,6 +331,33 @@ function routeFor(model: { provider: string; id: string; input?: ("text" | "imag
     contextWindow: model.contextWindow,
     maxTokens: model.maxTokens,
   };
+  const override = overrides[`${route.provider}:${route.modelId}`];
+  if (!override) return route;
+  const finite = (value: unknown, min: number) =>
+    typeof value === "number" && Number.isFinite(value) && value >= min ? value : undefined;
+  const integer = (value: unknown) => {
+    const normalized = finite(value, 1);
+    return normalized !== undefined && Number.isInteger(normalized) ? normalized : undefined;
+  };
+  const input = Array.isArray(override.input) && override.input.length > 0 &&
+    override.input.every((item) => item === "text" || item === "image")
+    ? [...new Set(override.input)] : undefined;
+  const costInput = finite(override.cost?.input, 0);
+  const costOutput = finite(override.cost?.output, 0);
+  return {
+    ...route,
+    ...(costInput === undefined || costOutput === undefined ? {} : { cost: {
+      input: costInput,
+      output: costOutput,
+      cacheRead: finite(override.cost?.cacheRead, 0),
+      cacheWrite: finite(override.cost?.cacheWrite, 0),
+    }}),
+    ...(integer(override.context_window) === undefined ? {} : {
+      contextWindow: integer(override.context_window),
+    }),
+    ...(integer(override.max_tokens) === undefined ? {} : { maxTokens: integer(override.max_tokens) }),
+    ...(input ? { input } : {}),
+  };
 }
 
 function eligibleRoutes(ctx: ExtensionContext): ModelRoute[] {
@@ -317,8 +367,9 @@ function eligibleRoutes(ctx: ExtensionContext): ModelRoute[] {
   if (ctx.model && !models.some((model) => model.provider === ctx.model!.provider && model.id === ctx.model!.id))
     models.unshift(ctx.model);
   const byRoute = new Map<string, ModelRoute>();
+  const overrides = loadPolicy().route_overrides ?? {};
   for (const model of models) {
-    const route = routeFor(model);
+    const route = routeFor(model, overrides);
     byRoute.set(`${route.provider}:${route.modelId}`, route);
   }
   return [...byRoute.values()];
@@ -347,7 +398,7 @@ function reportHarnessState(ctx: ExtensionContext, settled = false): void {
     const settledAt = settled ? new Date().toISOString()
       : prior.sessionFile === sessionFile ? prior.settledAt ?? null : null;
     writeState({
-        schemaVersion: 1,
+        schemaVersion: 2,
         currentModel: model,
         currentRoute,
         routes: eligibleRoutes(ctx),
@@ -356,6 +407,7 @@ function reportHarnessState(ctx: ExtensionContext, settled = false): void {
         sessionFile,
         settledTaskKey,
         settledAt,
+        comparisonsPaused,
         cwd: ctx.cwd,
         updatedAt: new Date().toISOString(),
       });
@@ -465,6 +517,34 @@ function ensureModel(pi: ExtensionAPI, ctx: ExtensionContext, openrouterId: stri
   return registry.find?.(INJECT_PROVIDER, openrouterId);
 }
 
+async function doctorSummary(): Promise<{ text: string; ok: boolean }> {
+  if (!existsSync(ENGINE_FILE)) return {
+    text: "trial engine missing; reinstall OpenMerit or run npm ci in the checkout",
+    ok: false,
+  };
+  const child = spawn(process.execPath, [ENGINE_FILE, "doctor", "--json"], {
+    env: process.env, stdio: ["ignore", "pipe", "pipe"],
+  });
+  let stdout = "";
+  let stderr = "";
+  child.stdout.on("data", (chunk: Buffer) => { stdout += chunk.toString(); });
+  child.stderr.on("data", (chunk: Buffer) => { stderr += chunk.toString(); });
+  const code = await new Promise<number>((resolve, reject) => {
+    child.on("error", reject);
+    child.on("close", (value) => resolve(value ?? 1));
+  });
+  try {
+    const report = JSON.parse(stdout) as { ok?: boolean; checks?: {
+      id?: string; status?: string; message?: string;
+    }[] };
+    const lines = (report.checks ?? []).map((item) =>
+      `${(item.status ?? "?").toUpperCase().padEnd(4)} ${item.id ?? "check"}: ${item.message ?? ""}`);
+    return { text: lines.join("\n") || "doctor returned no checks", ok: report.ok === true };
+  } catch {
+    return { text: stderr.trim().slice(0, 500) || stdout.trim().slice(0, 500) || `doctor exited ${code}`, ok: false };
+  }
+}
+
 export default function openmerit(pi: ExtensionAPI) {
   let pollTimer: ReturnType<typeof setInterval> | null = null;
   let trialJob: ChildProcessByStdio<null, Readable, Readable> | null = null;
@@ -472,8 +552,8 @@ export default function openmerit(pi: ExtensionAPI) {
     key: string; cwd: string; bytes: number }[] = [];
   const startedJobs = new Set<string>();
   let activeSession: string | null = null;
-  let progress: { sessionFile: string; taskKey: string; current: TrialProgress | null;
-    completed: TrialProgress[] } | null = null;
+  let progress: { sessionFile: string; taskKey: string; current: TrialRunProgress | null;
+    completed: TrialRunProgress[]; eligible: string[]; excluded: TrialSelectionProgress["excluded"] } | null = null;
 
   function stopTrialJob(): void {
     if (!trialJob) return;
@@ -501,14 +581,20 @@ export default function openmerit(pi: ExtensionAPI) {
       detached: process.platform !== "win32",
     });
     trialJob = child;
-    progress = { sessionFile: job.sessionFile, taskKey: job.key, current: null, completed: [] };
+    progress = { sessionFile: job.sessionFile, taskKey: job.key, current: null, completed: [],
+      eligible: [], excluded: [] };
     if (ctx.hasUI) ctx.ui.notify(`openmerit: comparing models for task ${job.key} (A=${job.model})`, "info");
     let stdout = "";
     let stderr = "";
     function onLine(line: string): void {
       const trial = parseTrialProgress(line);
       if (trial && job.sessionFile === activeSession && progress?.taskKey === trial.taskKey) {
-        if (trial.phase === "start") {
+        if (trial.phase === "selection") {
+          progress.eligible = trial.eligible;
+          progress.excluded = trial.excluded;
+          if (ctx.hasUI) ctx.ui.notify(`openmerit: ${trial.eligible.length} routes eligible; ` +
+            `${trial.excluded.length} skipped (/openmerit for reasons)`, "info");
+        } else if (trial.phase === "start") {
           progress.current = trial;
           if (ctx.hasUI) ctx.ui.notify(
             `openmerit: trial ${trial.index}/${trial.total} comparing ${trial.model}` +
@@ -551,11 +637,13 @@ export default function openmerit(pi: ExtensionAPI) {
     });
   }
 
-  function queueSettledTask(ctx: ExtensionContext): void {
+  function queueSettledTask(ctx: ExtensionContext, force = false): string {
+    if (comparisonsPaused && !force) return "comparisons are paused";
     const sessionFile = ctx.sessionManager.getSessionFile();
     const model = ctx.model ? canonicalModel(ctx.model.provider, ctx.model.id) : null;
     const route = ctx.model ? routeFor(ctx.model) : null;
-    if (!sessionFile || !model || !route || !settledTaskKey || !existsSync(sessionFile)) return;
+    if (!sessionFile || !model || !route || !settledTaskKey || !existsSync(sessionFile))
+      return "no saved completed task is available";
     // The alpha compares completed response tasks. The engine validates images,
     // tool use and the baseline answer before any provider calls.
     let completed = false;
@@ -568,19 +656,20 @@ export default function openmerit(pi: ExtensionAPI) {
             userTaskKeyFromAnswer(entry.message.content)) completed = true;
       } catch { /* partial session line */ }
     }
-    if (!completed) return;
+    if (!completed) return "the latest task has not completed successfully";
     const settledAt = new Date().toISOString();
     const bytes = readFileSync(sessionFile).length;
     const marker = `${sessionFile}:${settledTaskKey}:${model}:${bytes}`;
-    if (startedJobs.has(marker)) return;
+    if (startedJobs.has(marker)) return "this task is already queued or was compared in this Pi session";
     startedJobs.add(marker);
     const budget = trialBudget();
     if (budget.reason) {
       if (ctx.hasUI) ctx.ui.notify(`openmerit: comparison skipped: ${budget.reason}. ${budget.summary}`, "warning");
-      return;
+      return budget.reason;
     }
     queuedJobs.push({ sessionFile, settledAt, model, route, key: settledTaskKey, cwd: ctx.cwd, bytes });
     startNextJob(ctx);
+    return "comparison queued";
   }
 
   function userTaskKeyFromAnswer(content: unknown): boolean {
@@ -667,11 +756,12 @@ export default function openmerit(pi: ExtensionAPI) {
     restoreFallback();
     try {
       const st = JSON.parse(readFileSync(STATE_FILE, "utf8")) as {
-        sessionFile?: string | null; settledTaskKey?: string | null;
+        sessionFile?: string | null; settledTaskKey?: string | null; comparisonsPaused?: boolean;
       };
+      comparisonsPaused = st.comparisonsPaused === true;
       settledTaskKey = st.sessionFile === ctx.sessionManager.getSessionFile()
         ? st.settledTaskKey ?? null : null;
-    } catch { settledTaskKey = null; }
+    } catch { settledTaskKey = null; comparisonsPaused = false; }
     reportHarnessState(ctx);
     await surface(ctx);
     if (pollTimer) clearInterval(pollTimer);
@@ -748,7 +838,7 @@ export default function openmerit(pi: ExtensionAPI) {
   });
 
   pi.registerCommand("openmerit", {
-    description: "OpenMerit: status, pending recommendations, apply/dismiss",
+    description: "OpenMerit: status, compare, pause/resume, doctor, apply/dismiss",
     handler: async (args, ctx) => {
       const pending = pendingRecommendations(ctx);
       const prior = priorExactTaskSuggestion(ctx);
@@ -782,6 +872,37 @@ export default function openmerit(pi: ExtensionAPI) {
         }
         return;
       }
+      if (sub === "pause") {
+        comparisonsPaused = true;
+        queuedJobs.length = 0;
+        stopTrialJob();
+        startedJobs.clear();
+        progress = null;
+        reportHarnessState(ctx);
+        ctx.ui.notify("openmerit: automatic comparisons paused; `/openmerit compare` remains available", "info");
+        return;
+      }
+      if (sub === "resume") {
+        comparisonsPaused = false;
+        reportHarnessState(ctx);
+        ctx.ui.notify("openmerit: automatic comparisons resumed for future completed tasks", "info");
+        return;
+      }
+      if (sub === "compare") {
+        settledTaskKey = latestSessionTaskKey(ctx);
+        const result = queueSettledTask(ctx, true);
+        if (result !== "comparison queued") ctx.ui.notify(`openmerit: ${result}`, "warning");
+        return;
+      }
+      if (sub === "doctor") {
+        try {
+          const result = await doctorSummary();
+          ctx.ui.notify(`openmerit doctor:\n${result.text}`, result.ok ? "info" : "warning");
+        } catch (error) {
+          ctx.ui.notify(`openmerit doctor failed: ${(error as Error).message}`, "error");
+        }
+        return;
+      }
 
       const currentRoute = ctx.model ? routeFor(ctx.model) : undefined;
       const current = currentRoute ? routeDisplay(currentRoute, currentRoute.meritId) : "unknown";
@@ -791,6 +912,7 @@ export default function openmerit(pi: ExtensionAPI) {
         ? `running — ${liveProgress.current.model} (trial ${liveProgress.current.index}/${liveProgress.current.total})`
         : "running — selecting next model"
         : queuedJobs.length ? `queued (${queuedJobs.length})`
+        : comparisonsPaused ? "paused"
         : latestSessionJob(ctx.sessionManager.getSessionFile() ?? undefined) ?? "not started";
       const lines = [
         `current model : ${current}`,
@@ -803,6 +925,11 @@ export default function openmerit(pi: ExtensionAPI) {
         lines.push("", "models tested:");
         for (const trial of liveProgress.completed) lines.push(`  ${completedTrialText(trial)}`);
       }
+      if (liveProgress?.excluded.length) {
+        lines.push("", "routes skipped:");
+        for (const route of liveProgress.excluded)
+          lines.push(`  ${route.model}: ${route.reasons.join("; ")}`);
+      }
       if (prior) lines.push("", suggestionText(prior.recommendation, prior.trials));
       for (const r of pending.slice(-5)) {
         lines.push(

package/instructions/OPENMERIT.md

@@ -39,6 +39,12 @@ the best model for the task at hand, at the best price, with a vetted fallback.
   recommendations.
 - `/openmerit apply` / `/openmerit dismiss` — act on a pending recommendation
   when automatic application is declined by the policy gate.
+- `/openmerit pause` / `/openmerit resume` — stop or restart automatic task
+  comparisons without changing the saved policy.
+- `/openmerit compare` — explicitly compare the latest completed task, even
+  while automatic comparisons are paused.
+- `/openmerit doctor` — inspect sanitized installation, route, pricing, state,
+  and provider-extension diagnostics inside Pi.
 - Policy changes (auto-apply thresholds, budgets, provider allow-lists) are
   made by the user in `~/.openmerit/policy.json`, not by you.
 
@@ -52,3 +58,6 @@ the best model for the task at hand, at the best price, with a vetted fallback.
 - The shipped policy is supervised. Swaps happen automatically only after the
   user opts in and the configured quality/cost guardrails pass; otherwise they
   remain recommendations for a human to approve.
+- Isolated subprocesses disable extension discovery. Provider-registration
+  extensions run only when the user lists their absolute file paths in
+  `policy.json`; OpenMerit refuses to load itself recursively.

package/instructions/openmerit.policy.json

@@ -1,5 +1,5 @@
 {
-  "version": 1,
+  "version": 2,
   "mode": "recommend",
   "auto_apply": {
     "enabled": false,
@@ -29,5 +29,9 @@
   },
   "judge_model": null,
   "strategist_model": null,
-  "max_usd_per_m": 20.0
+  "max_usd_per_m": 20.0,
+  "route_overrides": {},
+  "pi": {
+    "provider_extensions": []
+  }
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.1.3",
+  "version": "0.1.4",
   "description": "Find better models for each pi task by comparing quality, cost, and latency.",
   "type": "module",
   "keywords": [

package/rules.md

@@ -16,6 +16,8 @@
 
 - **Respect the harness's eligible model pool.** Prefer models already available and configured in the active harness; external catalogs may enrich or expand discovery but must not silently override harness scope or credentials.
 
+- **Require explicit metadata instead of price guesses.** Missing price, context, output, or modality data for a custom route may be supplied by an exact-route override; never assume a local route is free or copy metadata across providers.
+
 - **Treat public benchmarks as priors, not proof.** Benchmarks help shortlist candidates, but merit comes from trials on the user's actual task.
 
 - **Keep four integration boundaries distinct.** Model execution (`ModelProviderAdapter`), agent execution (`HarnessAdapter`), incoming traces (`ObservationSource`), and outgoing telemetry (`EventSink`) solve different problems and must not be coupled.
@@ -30,6 +32,8 @@
 
 - **Default to safe, explicit trials.** Candidate tool access stays off unless deliberately allowed, trials remain isolated, and temporary workspaces are cleaned up because trying a model must not expose or damage a user's project by surprise.
 
+- **Allowlist subprocess extensions.** Keep Pi extension discovery disabled in isolated trials and load only explicitly named provider-registration files; never recursively load OpenMerit or inherit unrelated harness extensions.
+
 - **Keep the shadow track isolated, not necessarily concurrent.** Candidate trials may run sequentially to respect cost, rate, and safety limits while remaining separate from the user's live task.
 
 - **Treat local JSONL as the default integration, not a lock-in.** Local traces and events should work without an external service; systems such as Langfuse can later plug in as observation sources or event sinks.