openmerit 0.1.2 → 0.1.4

package/README.md

@@ -54,13 +54,14 @@ package into pi, then initialize its policy and state:
 ```bash
 pi install npm:openmerit
 npx --yes openmerit init
+npx --yes openmerit verify
 pi list
 ```
 
 `pi install` makes the extension and its trial engine available to pi. The
 one-off `npx` command creates `~/.openmerit/policy.json`; install OpenMerit
 globally with `npm install --global openmerit` only if you also want persistent
-shell access to `openmerit status`, `doctor`, `frontier`, or the optional watcher. Install
+shell access to `openmerit status`, `doctor`, `verify`, `frontier`, or the optional watcher. Install
 the extension from only one source—remove any older copied `openmerit.ts` first
 so pi does not load it twice.
 
@@ -120,8 +121,9 @@ Keep that checkout available because pi loads a local package from its path.
 3. Verify the extension appears in `pi list`. The instruction file
    [`instructions/OPENMERIT.md`](instructions/OPENMERIT.md) can be added to a
    pi project's AGENTS.md for agent context, but the extension does
-   not require it. Run `npx openmerit doctor` after starting Pi once to inspect
-   the eligible route snapshot and configuration without printing secrets.
+   not require it. Run `npx openmerit verify` for a provider-free core self-test,
+   then `npx openmerit doctor` after starting Pi once to inspect the eligible
+   route snapshot and configuration without printing secrets.
 
 4. Start pi in one terminal with any configured model. For example:
 
@@ -140,6 +142,12 @@ Keep that checkout available because pi loads a local package from its path.
    switch. Then send a second message to see which model actually handles it.
    You do not need a watcher terminal or the standalone `trial` command.
 
+   Use `/openmerit pause` to stop the active comparison and suppress automatic
+   comparisons, `/openmerit resume` to enable them for future completed tasks,
+   and `/openmerit compare` to explicitly compare the latest completed task
+   even while automatic comparisons are paused. `/openmerit doctor` runs the
+   sanitized setup checks without leaving Pi.
+
    Candidate comparisons have no tools by default, even when the observed task
    used tools. See **Alpha boundaries** before explicitly enabling candidate
    tools.
@@ -148,6 +156,57 @@ Keep that checkout available because pi loads a local package from its path.
    `"mode": "auto"` and `"auto_apply.enabled": true`, review the score-gain
    and price-ratio thresholds, and run the next task.
 
+### Configure custom or local routes
+
+Pi normally supplies each route's price, context window, output limit, and
+modalities. Some custom and local providers omit that metadata. OpenMerit does
+not guess that a local model is free: add an override keyed by the exact
+`provider:modelId` route in `~/.openmerit/policy.json` instead:
+
+```json
+{
+  "route_overrides": {
+    "ollama:qwen3:8b": {
+      "cost": { "input": 0, "output": 0 },
+      "context_window": 32768,
+      "max_tokens": 4096,
+      "input": ["text"]
+    }
+  }
+}
+```
+
+All fields are optional, but both `cost.input` and `cost.output` are required
+when declaring cost. Prices use Pi's dollars-per-million-token units. The
+override applies only to that exact route; it does not change the stable
+`vendor/model` identity or another provider's route to the same model.
+
+If the provider itself is registered at runtime by a Pi extension, explicitly
+allow that provider-registration file in the same policy. Paths must be
+absolute, existing files:
+
+```json
+{
+  "pi": {
+    "provider_extensions": [
+      "/absolute/path/to/ollama-provider.ts"
+    ]
+  }
+}
+```
+
+OpenMerit still launches subprocesses with extension discovery disabled, then
+loads only these explicit files. Do not add OpenMerit's own extension. An
+allowlisted extension executes code in every candidate, judge, and strategist
+Pi subprocess, so list only provider extensions you trust. Run
+`npx openmerit doctor` or `/openmerit doctor` to validate the files and confirm
+that route overrides match Pi's visible routes.
+
+Existing `0.1.x` policy files remain valid: missing `route_overrides` and
+`pi.provider_extensions` fields normalize to empty safe defaults. `openmerit init`
+continues to preserve an existing policy, so add these fields manually
+only when you need them.
+
 ### Try an invoice-to-JSON task
 
 Use the same one-terminal setup. Start pi with a vision-capable model, for
@@ -179,14 +238,21 @@ schema, so its published scores are not directly comparable to these pi runs.
 
 ### Inspect or troubleshoot a run
 
-`npx openmerit doctor` checks Pi, policy, eligible routes, and optional
-OpenRouter enrichment. `npx openmerit status` shows the latest pi model and pending recommendations;
+`npx openmerit doctor` checks Pi, policy, eligible routes and prices, saved
+session state, duplicate package sources, append-only files, and optional
+OpenRouter enrichment. Add `--json` for a sanitized diagnostic report suitable
+for a bug report; it contains no credentials, prompts, or trace contents.
+`npx openmerit verify` runs an offline self-test of atomic state, JSONL recovery,
+route preservation, policy evidence, and neutral events without contacting a
+provider. `npx openmerit status` shows the latest pi model and pending recommendations;
 `npx openmerit frontier` shows measured quality, blended price, latency, and
 the chosen frontier per task. `/openmerit` inside pi shows the current model,
 fallback, the model currently being compared, completed models with quality,
 cost, and latency, trial budget, pending recommendations, and any exact-task result
 measured in another session. When the gate declines an automatic swap, it
-prints reasons and `/openmerit apply` remains available.
+prints reasons and `/openmerit apply` remains available. It also lists every
+route skipped during the current comparison with the relevant policy, pricing,
+modality, or per-trial budget reason.
 
 The daily trial-count and dollar limits come from `~/.openmerit/policy.json`.
 If a limit is reached, the extension reports why it skipped the comparison;
@@ -203,10 +269,21 @@ Closing or switching the session cancels the active job.
 Candidate runs use the same text and uploaded image bytes, but they do not
 replay earlier answers or file changes. An exact task in another session
 (including identical image bytes) appears as **advice**, not a pending swap;
-the new session still gets its own comparison. The optional `trial` command
-remains for controlled text-task runs from a task JSON file; that legacy
-standalone command is still OpenRouter-specific in 0.1.2. The automatic Pi
-session path is provider-neutral.
+the new session still gets its own comparison.
+
+The optional `trial` command runs controlled text-task comparisons through the
+same exact Pi provider routes and credential store as the automatic session
+path:
+
+```bash
+npx openmerit trial examples/task.example.json --rounds 3
+```
+
+`initial_model` is the stable `vendor/model` identity. When Pi exposes that
+model through more than one provider, set `initial_route` to
+`provider:model-id` (for example `openai:gpt-4o-mini` or
+`openrouter:openai/gpt-4o-mini`). Judge and strategist calls also run through
+Pi. An OpenRouter key only adds optional catalog and public-benchmark metadata.
 
 ## Alpha boundaries
 
@@ -237,7 +314,9 @@ session path is provider-neutral.
   for automatic session comparisons plus the daily candidate count.
   `max_usd_per_trial` is a conservative admission estimate based on known Pi
   prices and a 4K answer; it is not a provider-side hard cap. Routes without
-  known pricing are excluded from automatic comparisons and cannot auto-apply.
+  known pricing are excluded as candidates and cannot auto-apply. Use an exact
+  `route_overrides` entry for a custom/local route whose price is known; zero
+  cost must be stated explicitly.
 - Each comparison has one observed baseline plus a small candidate slate and
   one quality score per answer. Treat recommendations as experimental evidence,
   not a universal model ranking.
@@ -246,18 +325,21 @@ session path is provider-neutral.
   enrichment source. `HarnessAdapter`, `ModelProviderAdapter`,
   `ObservationSource`, and `EventSink` remain separate integration boundaries;
   Pi and local JSONL are the implementations shipped in this release.
-- Candidate subprocesses can use Pi built-ins and custom/local routes available
-  without loading extensions (for example routes from Pi's model
-  configuration). A provider registered only at runtime by another extension
-  is visible in the route snapshot but cannot yet be executed by the isolated
-  subprocess.
+- Candidate subprocesses can use Pi built-ins, configured custom/local routes,
+  and providers registered by explicitly allowlisted extension files. Normal
+  extension discovery remains disabled, and OpenMerit refuses to load its own
+  extension recursively.
+- JSON state snapshots are replaced atomically. Append-only readers skip and
+  report malformed or interrupted lines while retaining later valid records.
+  A job owned by a crashed process is reclaimable instead of remaining stuck
+  in `running`; completed jobs remain final.
 
 ## State layout (`~/.openmerit/`)
 
 | file | contents |
 |---|---|
-| `policy.json` | the policy file (gate thresholds, budgets, intervals) |
-| `harness-state.json` | current/fallback routes, Pi's eligible route snapshot, latest session and settled task |
+| `policy.json` | gate thresholds, budgets, intervals, exact-route metadata overrides, and allowlisted Pi provider extensions |
+| `harness-state.json` | current/fallback routes, Pi's eligible route snapshot, pause state, latest session and settled task |
 | `recommendations.jsonl` | append-only session-bound recommendations, routes, evidence, gate reasons, and status updates |
 | `trials.jsonl` | every model trial point, including its provider route when known |
 | `traces/observations.jsonl` | task observations extracted from session traces |

package/dist/cli.js

@@ -1,36 +1,18 @@
 #!/usr/bin/env node
-/** openmerit CLI: init / watch / trial / frontier / recommend / status / doctor. */
-import { copyFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
+/** openmerit CLI: init / watch / trial / frontier / recommend / status / doctor / verify. */
+import { copyFileSync, existsSync, mkdirSync } from "node:fs";
 import { dirname, join } from "node:path";
 import { fileURLToPath } from "node:url";
 import { loadPolicy } from "./policy.js";
 import { loadKey } from "./llm.js";
-import { fetchCatalog } from "./catalog.js";
 import { paretoFrontier, pickBest, pickFallback } from "./frontier.js";
-import { buildRecommendation } from "./recommend.js";
-import { appendJsonl, paths, readJson, readJsonl, taskKey } from "./store.js";
+import { paths, readJson, readJsonl } from "./store.js";
 import { autoTaskTick, recommendTick, runDaemon, tickOnce } from "./daemon.js";
-import { budgetOk, recordTrialSpend } from "./trials.js";
-import { availablePiModels, availablePiRoutes, piExecutable, recordedActiveTask, runPiTrial, scorePiRun, settledActiveTask } from "./pi-trials.js";
-import { pickNext, STRAT_PREFS } from "./strategist.js";
-import { JUDGE_PREFS } from "./judge.js";
-import { openRouterBenchmarkCandidates, relevantBenchmarks } from "./benchmarks.js";
+import { settledActiveTask } from "./pi-trials.js";
 import { modelRoute, routeLabel } from "./routes.js";
-import { spawnSync } from "node:child_process";
+import { runStandaloneTrial } from "./standalone.js";
+import { collectDoctorReport, renderDiagnosticReport, runOfflineVerify } from "./diagnostics.js";
 const ROOT = dirname(dirname(fileURLToPath(import.meta.url)));
-function resolvePref(cat, prefs, label) {
-    for (const p of prefs)
-        if (p && cat.has(p))
-            return p;
-    for (const p of prefs) {
-        if (!p)
-            continue;
-        for (const id of [...cat.keys()].sort())
-            if (id.includes(p))
-                return id;
-    }
-    throw new Error(`could not resolve ${label} model`);
-}
 function cmdInit() {
     const policyPath = paths.policy();
     mkdirSync(dirname(policyPath), { recursive: true });
@@ -55,129 +37,16 @@ function optionalOpenRouterKey() {
         return null;
     }
 }
-function cmdDoctor() {
-    const executable = piExecutable();
-    const version = spawnSync(executable, ["--version"], { encoding: "utf8" });
-    const state = readJson(paths.harnessState(), {});
-    console.log(`pi executable : ${executable}`);
-    console.log(`pi detected   : ${version.status === 0 ? `yes (${version.stdout.trim()})` : "no"}`);
-    try {
-        const policy = loadPolicy(paths.policy());
-        console.log(`policy        : valid (v${policy.version}, ${policy.mode})`);
-    }
-    catch (error) {
-        console.log(`policy        : invalid (${error.message})`);
-    }
-    let routes = state.routes ?? [];
-    const hasLiveSnapshot = routes.length > 0;
-    if (routes.length === 0 && version.status === 0) {
-        try {
-            routes = availablePiRoutes();
-        }
-        catch { /* reported below */ }
-    }
-    console.log(`eligible routes: ${routes.length}`);
-    for (const route of routes.slice(0, 8))
-        console.log(`  - ${routeLabel(route)}${route.cost ? "" : " (price unknown)"}`);
-    if (routes.length > 8)
-        console.log(`  … ${routes.length - 8} more`);
-    console.log(`active route  : ${state.currentRoute ? routeLabel(state.currentRoute) : "not reported; start pi once"}`);
-    const pricedRoutes = routes.filter((route) => !!route.cost).length;
-    console.log(`alternates    : ${hasLiveSnapshot
-        ? pricedRoutes >= 2 ? "ready" : "need at least two priced eligible pi routes"
-        : "start Pi once to capture route prices and capabilities"}`);
-    console.log(`OpenRouter enrichment: ${optionalOpenRouterKey() ? "available" : "not configured (optional)"}`);
+function printReport(report, json) {
+    console.log(json ? JSON.stringify(report, null, 2) : renderDiagnosticReport(report));
+    if (!report.ok)
+        process.exitCode = 1;
 }
-async function cmdTrial(taskFile, rounds) {
-    const cfg = JSON.parse(readFileSync(taskFile, "utf8"));
-    const policy = loadPolicy(paths.policy());
-    const key = loadKey();
-    console.log("fetching OpenRouter catalog...");
-    const cat = await fetchCatalog(key);
-    const piModels = availablePiModels();
-    for (const id of [...cat.keys()])
-        if (!piModels.has(id))
-            cat.delete(id);
-    if (!cat.has(cfg.initial_model))
-        throw new Error(`initial_model ${cfg.initial_model} is unavailable in pi's OpenRouter registry`);
-    console.log(`catalog: ${cat.size} models selectable in pi`);
-    const judge = resolvePref(cat, [cfg.judge_model ?? policy.judge_model, ...JUDGE_PREFS], "judge");
-    const strat = resolvePref(cat, [cfg.strategist_model ?? policy.strategist_model, ...STRAT_PREFS], "strategist");
-    const { category, benchmarks } = relevantBenchmarks(cfg.task);
-    let benchmarkCandidates = [];
-    try {
-        benchmarkCandidates = await openRouterBenchmarkCandidates(key, category, cat);
-    }
-    catch (e) {
-        console.log(`OpenRouter benchmark shortlist unavailable: ${e.message}`);
-    }
-    console.log(`judge: ${judge} | strategist: ${strat} | benchmarks: ${benchmarks.join(", ")}`);
-    console.log(`OpenRouter benchmark candidates: ${benchmarkCandidates.slice(0, 5).join(", ") || "none"}`);
-    const tKey = taskKey(cfg.task);
-    const maxPrice = cfg.max_usd_per_m ?? policy.max_usd_per_m;
-    const tried = new Set();
-    const results = [];
-    const failedVendors = new Set();
-    for (let i = 0; i < rounds; i++) {
-        const budget = budgetOk(policy);
-        if (!budget.ok) {
-            console.log(`budget: ${budget.reason}; stopping`);
-            break;
-        }
-        let model;
-        let why;
-        if (i === 0) {
-            model = cfg.initial_model;
-            why = "initial model";
-        }
-        else {
-            const pick = await pickNext(key, strat, cfg.task, cfg.eval, benchmarks, results, cat, tried, maxPrice, failedVendors, benchmarkCandidates);
-            if (!pick)
-                break;
-            model = pick.model;
-            why = pick.why;
-        }
-        if (tried.has(model))
-            break;
-        console.log(`[round ${i + 1}/${rounds}] ${model} (${why}) ...`);
-        const observed = i === 0 ? recordedActiveTask(cfg.task, model) : null;
-        if (i === 0)
-            console.log(observed ? "    using model A's active pi session trace" : "    active trace unavailable; running model A in a fresh pi session");
-        const { point, costUsd, error, sessionId } = observed
-            ? await scorePiRun(key, judge, cfg.task, cfg.eval, model, cat.get(model), observed, "trace")
-            : await runPiTrial(key, judge, cfg.task, cfg.eval, model, cat.get(model));
-        tried.add(model);
-        results.push(point);
-        recordTrialSpend(costUsd);
-        appendJsonl(paths.trials(), { ...point, taskKey: tKey });
-        if (sessionId)
-            console.log(`    pi trace session: ${sessionId}`);
-        if (error) {
-            failedVendors.add(model.split("/")[0]);
-            console.log(`    FAILED: ${error.slice(0, 160)}`);
-        }
-        else {
-            console.log(`    score=${point.score.toFixed(2)}  $/M=${point.price}  est_cost=$${costUsd.toFixed(4)}  ${(point.why ?? "").slice(0, 110)}`);
-        }
-    }
-    const pts = results.filter((p) => p.score > 0);
-    if (pts.length === 0) {
-        console.log("no trials completed");
-        process.exit(1);
-    }
-    const chain = paretoFrontier(pts);
-    const best = pickBest(pts);
-    const fb = pickFallback(pts, best, policy.fallback.min_score);
-    console.log("\n=== pareto frontier (nondominated: score up, price down, latency down) ===");
-    for (const p of chain)
-        console.log(`  ${p.model.padEnd(44)} score=${p.score.toFixed(2)}  $${p.price.toFixed(2)}/M`);
-    console.log(`\nBEST FIT : ${best.model}  (score ${best.score.toFixed(2)}, $${best.price.toFixed(2)}/M)`);
-    console.log(fb ? `FALLBACK : ${fb.model}  (score ${fb.score.toFixed(2)}, $${fb.price.toFixed(2)}/M)` : "FALLBACK : n/a");
-    const rec = buildRecommendation(tKey, cfg.task.slice(0, 120), cfg.initial_model, pts, policy);
-    if (rec) {
-        appendJsonl(paths.recommendations(), rec);
-        console.log(`\nrecommendation -> ${paths.recommendations()} (auto=${rec.policy.autoApply})`);
-    }
+function cmdDoctor(json) {
+    printReport(collectDoctorReport(), json);
+}
+function cmdVerify(json) {
+    printReport(runOfflineVerify(), json);
 }
 function loadFrontiers() {
     const trials = readJsonl(paths.trials());
@@ -243,12 +112,12 @@ function cmdStatus() {
 }
 async function main() {
     const [cmd, ...args] = process.argv.slice(2);
-    const policy = loadPolicy(paths.policy());
     switch (cmd) {
         case "init":
             cmdInit();
             break;
         case "session-trial": {
+            const policy = loadPolicy(paths.policy());
             const [sessionFile, settledAt, currentModel, settledTaskKey, cwd, bytesText, provider, providerModelId] = args;
             if (!sessionFile || !settledAt || !currentModel || !settledTaskKey || !cwd || !bytesText)
                 throw new Error("usage: openmerit session-trial <session-file> <settled-at> <model> <task-key> <cwd> <session-bytes>");
@@ -269,29 +138,32 @@ async function main() {
         }
         case "watch":
             if (args.includes("--once"))
-                await tickOnce(policy);
+                await tickOnce(loadPolicy(paths.policy()));
             else
-                await runDaemon(policy);
+                await runDaemon(loadPolicy(paths.policy()));
             break;
         case "trial": {
             const file = args[0];
             if (!file)
                 throw new Error("usage: openmerit trial <task.json> [--rounds N]");
             const rIdx = args.indexOf("--rounds");
-            await cmdTrial(file, rIdx >= 0 ? Number(args[rIdx + 1]) : 5);
+            await runStandaloneTrial(file, rIdx >= 0 ? Number(args[rIdx + 1]) : 5);
             break;
         }
         case "frontier":
             cmdFrontier(args[0]);
             break;
         case "recommend":
-            recommendTick(policy);
+            recommendTick(loadPolicy(paths.policy()));
             break;
         case "status":
             cmdStatus();
             break;
         case "doctor":
-            cmdDoctor();
+            cmdDoctor(args.includes("--json"));
+            break;
+        case "verify":
+            cmdVerify(args.includes("--json"));
             break;
         default:
             console.log(`openmerit — external model-merit harness
@@ -304,7 +176,8 @@ usage: openmerit <command>
   frontier [taskKey]   print pareto frontier(s)
   recommend            emit recommendations now
   status               harness model, queued models, pending recommendations
-  doctor               verify pi, policy, routes, and optional enrichment`);
+  doctor [--json]      diagnose pi, policy, routes, state, and installation
+  verify [--json]      run an offline, provider-free core self-test`);
             if (cmd && cmd !== "help" && cmd !== "--help")
                 process.exitCode = 1;
     }

package/dist/daemon.js

@@ -11,10 +11,10 @@ import { buildRecommendation } from "./recommend.js";
 import { appendJsonl, paths, readJson, readJsonl, writeJson, } from "./store.js";
 import { budgetOk, ensureRubric, recordMeritSpend, recordTrialSpend, runTrial, trialBudgetOk } from "./trials.js";
 import { JUDGE_PREFS } from "./judge.js";
-import { runPiTrial, scorePiRun, settledActiveTask } from "./pi-trials.js";
+import { PiHarnessAdapter, runPiTrial, scorePiRun, settledActiveTask } from "./pi-trials.js";
 import { taskInputKey } from "./task-input.js";
 import { pickNext, STRAT_PREFS } from "./strategist.js";
-import { enrichRouteEntry, routeCatalog, routeKey, routeLabel } from "./routes.js";
+import { applyRouteOverrides, enrichRouteEntry, routeCatalog, routeKey, routeLabel } from "./routes.js";
 import { PiCliChatClient } from "./llm.js";
 import { LocalJsonlEventSink, PiTraceObservationSource, meritEvent } from "./integrations.js";
 function resolveRoutePref(cat, prefs, label, requireImages = false) {
@@ -42,6 +42,28 @@ function resolveRoutePref(cat, prefs, label, requireImages = false) {
         return fallback;
     throw new Error(`could not resolve ${label} route from Pi's eligible models`);
 }
+function processIsAlive(pid) {
+    if (!Number.isInteger(pid) || pid <= 0)
+        return false;
+    try {
+        process.kill(pid, 0);
+        return true;
+    }
+    catch (error) {
+        return error.code === "EPERM";
+    }
+}
+/** Completed jobs stay final; live owners retain their claim; abandoned claims can retry. */
+export function taskJobIsClaimed(prior, marker, now = Date.now()) {
+    if (!prior || prior.marker !== marker || prior.status === "failed")
+        return false;
+    if (prior.status !== "running")
+        return true;
+    if (prior.ownerPid !== undefined)
+        return processIsAlive(prior.ownerPid);
+    const updated = Date.parse(prior.updatedAt);
+    return Number.isFinite(updated) && now - updated < 15 * 60_000;
+}
 function emitTrialProgress(progress) {
     console.log(`[openmerit/progress] ${JSON.stringify(progress)}`);
 }
@@ -150,20 +172,25 @@ export async function trialTick(key, policy) {
 /** Compare the one completed active pi task, sequentially, then target its session. */
 export async function autoTaskTick(key, policy, settledTask) {
     const sink = new LocalJsonlEventSink();
-    const task = settledTask === undefined ? settledActiveTask() : settledTask;
+    let task = settledTask === undefined ? settledActiveTask() : settledTask;
     if (!task)
         return null;
+    const routes = applyRouteOverrides(task.routes, policy.route_overrides);
+    const currentRoute = applyRouteOverrides([task.route], policy.route_overrides)[0];
+    task = { ...task, routes, route: currentRoute };
+    const tKey = taskInputKey(task.task, task.images, task.files);
     const marker = settledTask === undefined
         ? `${task.sessionFile}:${task.settledAt}`
         : `${task.sessionFile}:${task.sessionBytes ?? task.settledAt}`;
     const processedPath = settledTask === undefined ? paths.watchProcessed() : paths.sessionJob(marker);
     const prior = readJson(processedPath, null);
-    if (prior?.marker === marker && prior.status !== "failed")
+    if (taskJobIsClaimed(prior, marker))
         return null;
     if (!budgetOk(policy).ok)
         return null;
     // Claim the settled turn before provider calls so the next poll cannot duplicate it.
-    writeJson(processedPath, { marker, status: "running", updatedAt: new Date().toISOString() });
+    writeJson(processedPath, { marker, status: "running", ownerPid: process.pid,
+        updatedAt: new Date().toISOString() });
     try {
         let cat = routeCatalog(task.routes);
         if (!cat.has(routeKey(task.route)))
@@ -179,13 +206,28 @@ export async function autoTaskTick(key, policy, settledTask) {
                 console.log(`[openmerit] OpenRouter enrichment unavailable: ${error.message}`);
             }
         }
+        const excluded = [];
         for (const [id, entry] of [...cat]) {
-            if ((entry.priceKnown !== false && entry.price > policy.max_usd_per_m) ||
-                !providerAllowed(policy, entry.id, entry.route?.provider) ||
-                (task.images.length > 0 && !entry.inputModalities?.includes("image")) ||
-                (id !== routeKey(task.route) && !trialBudgetOk(policy, entry, task.task.length).ok))
+            const reasons = [];
+            if (entry.priceKnown !== false && entry.price > policy.max_usd_per_m)
+                reasons.push(`price $${entry.price.toFixed(2)}/M exceeds $${policy.max_usd_per_m.toFixed(2)}/M policy limit`);
+            if (!providerAllowed(policy, entry.id, entry.route?.provider))
+                reasons.push("provider is denied by policy");
+            if (task.images.length > 0 && !entry.inputModalities?.includes("image"))
+                reasons.push("route does not advertise image input");
+            if (id !== routeKey(task.route)) {
+                const admission = trialBudgetOk(policy, entry, task.task.length);
+                if (!admission.ok && admission.reason)
+                    reasons.push(admission.reason);
+            }
+            if (reasons.length) {
                 cat.delete(id);
+                excluded.push({ model: entry.route ? routeLabel(entry.route) : entry.id, reasons: [...new Set(reasons)] });
+            }
         }
+        if (settledTask !== undefined)
+            emitTrialProgress({ phase: "selection", taskKey: tKey,
+                eligible: [...cat.values()].map((entry) => entry.route ? routeLabel(entry.route) : entry.id), excluded });
         const activeEntry = cat.get(routeKey(task.route));
         if (!activeEntry)
             throw new Error(`active route ${task.route.provider}/${task.route.modelId} is excluded by policy or task capabilities`);
@@ -193,7 +235,8 @@ export async function autoTaskTick(key, policy, settledTask) {
             throw new Error("Pi exposes no eligible alternate model route for this task");
         const judge = resolveRoutePref(cat, [policy.judge_model, ...JUDGE_PREFS, task.model], "judge", task.images.length > 0);
         const strategist = resolveRoutePref(cat, [policy.strategist_model, ...STRAT_PREFS, task.model], "strategist");
-        const client = new PiCliChatClient(undefined, recordMeritSpend);
+        const client = new PiCliChatClient(undefined, recordMeritSpend, policy.pi.provider_extensions);
+        const harness = new PiHarnessAdapter(policy.pi.provider_extensions);
         const { category, benchmarks } = relevantBenchmarks(task.task);
         let benchmarkCandidates = [];
         try {
@@ -204,7 +247,6 @@ export async function autoTaskTick(key, policy, settledTask) {
             console.log(`[openmerit] benchmark shortlist unavailable: ${e.message}`);
         }
         const rubric = await ensureRubric(key ?? "", judge.route, task.task, new Map(), client);
-        const tKey = taskInputKey(task.task, task.images, task.files);
         const points = [];
         const tried = new Set();
         const failedVendors = new Set();
@@ -233,7 +275,7 @@ export async function autoTaskTick(key, policy, settledTask) {
                 emitTrialProgress({ phase: "start", taskKey: tKey,
                     model: pick.route ? routeLabel(pick.route) : pick.model, index: i + 1, total });
             const entry = pick.route ? cat.get(routeKey(pick.route)) : cat.get(pick.model);
-            const { point, costUsd, error } = await runPiTrial(key ?? "", judge.route, task.task, rubric, pick.model, entry, task.cwd, task.images, task.files, undefined, client);
+            const { point, costUsd, error } = await runPiTrial(key ?? "", judge.route, task.task, rubric, pick.model, entry, task.cwd, task.images, task.files, harness, client);
             tried.add(pick.route ? routeKey(pick.route) : pick.model);
             points.push(point);
             recordTrialSpend(costUsd);
@@ -258,12 +300,12 @@ export async function autoTaskTick(key, policy, settledTask) {
         else
             console.log(`[openmerit] task ${tKey}: active model remains best`);
         writeJson(processedPath, { marker, status: rec ? "complete" : "no_swap",
-            updatedAt: new Date().toISOString() });
+            ownerPid: process.pid, updatedAt: new Date().toISOString() });
         return rec;
     }
     catch (error) {
         writeJson(processedPath, { marker, status: "failed",
-            updatedAt: new Date().toISOString() });
+            ownerPid: process.pid, updatedAt: new Date().toISOString() });
         throw error;
     }
 }