openmerit 0.1.1 → 0.1.3

package/README.md

@@ -32,10 +32,10 @@ pi task A → saved pi trace → extension trial job → pi trials B, C → scor
    ([`src/frontier.ts`](src/frontier.ts), 3 objectives: quality ↑, price ↓,
    latency ↓). The aggregate across all of an agent's tasks is the union of
    its task frontiers (`openmerit frontier`).
-4. **Model discovery** — the OpenRouter catalog is snapshotted and diffed;
-   newly appeared models are queued for future release experiments. The
-   per-task loop shortlists currently available pi/OpenRouter models with
-   OpenRouter benchmark signals and the existing strategist
+4. **Model discovery** — the active Pi session's scoped models (or Pi's
+   authenticated available-model registry when unscoped) are the candidate
+   pool. OpenRouter is an optional route and enrichment source: its catalog can
+   be snapshotted and its benchmark signals can improve the shortlist
    ([`src/catalog.ts`](src/catalog.ts) + [`src/daemon.ts`](src/daemon.ts)).
 5. **Informing the main harness** — recommendations land in
    `~/.openmerit/recommendations.jsonl`. The pi extension
@@ -47,19 +47,21 @@ pi task A → saved pi trace → extension trial job → pi trials B, C → scor
 
 ## Install from npm
 
-You need Node 22.18+, pi 0.85.1+, and an OpenRouter API key with available
-credits. Install the package into pi, then initialize its policy and state:
+You need Node 22.18+, pi 0.85.1+, and at least two eligible model routes
+authenticated in Pi. OpenRouter is supported but not required. Install the
+package into pi, then initialize its policy and state:
 
 ```bash
 pi install npm:openmerit
 npx --yes openmerit init
+npx --yes openmerit verify
 pi list
 ```
 
 `pi install` makes the extension and its trial engine available to pi. The
 one-off `npx` command creates `~/.openmerit/policy.json`; install OpenMerit
 globally with `npm install --global openmerit` only if you also want persistent
-shell access to `openmerit status`, `frontier`, or the optional watcher. Install
+shell access to `openmerit status`, `doctor`, `verify`, `frontier`, or the optional watcher. Install
 the extension from only one source—remove any older copied `openmerit.ts` first
 so pi does not load it twice.
 
@@ -85,40 +87,45 @@ Keep that checkout available because pi loads a local package from its path.
    policy**. The shipped policy uses `"mode": "recommend"` and does not change
    the active model without approval.
 
-2. Give **Pi and the extension trial job** OpenRouter access. The job reads
-   `OPENROUTER_API_KEY` from its environment or `~/.openmerit/.env`. If using
-   the file, edit it to contain `OPENROUTER_API_KEY=your-key` and protect it:
+2. Authenticate the providers you want to compare in Pi, using `/login` or
+   Pi's normal environment/model configuration. OpenMerit takes its eligible
+   model routes, capabilities, prices, and credentials from Pi; judge and
+   strategist calls use the same routes and do not require duplicate keys.
+
+   For example, verify one provider without printing its credential:
 
    ```bash
-   nano ~/.openmerit/.env
-   chmod 600 ~/.openmerit/.env
+   pi auth check --provider openai --json
    ```
 
-   Pi does **not** read OpenMerit's `.env` file. Log in to OpenRouter with
-   `/login openrouter` inside pi, or export `OPENROUTER_API_KEY` in the shell
-   that starts pi. Verify pi's side without printing the key:
+   Add an OpenRouter route the same way if you want its routed catalog:
 
    ```bash
    pi auth check --provider openrouter --json
    ```
 
-   A ready configuration prints `{"status":"ready","provider":"openrouter",...}`.
-   Older `model_search/.env` files are not read by the trial engine.
+   An `OPENROUTER_API_KEY` in the process environment or
+   `~/.openmerit/.env` additionally enables OpenRouter catalog and public-
+   benchmark enrichment. It is optional for native-provider comparisons. If
+   using the file, protect it:
 
-   OpenRouter is currently required for candidate runs and catalog discovery.
-   Judge and strategist calls can use OpenAI or Gemini directly, but require
-   `OPENMERIT_DIRECT_PROVIDER=openai` or
-   `OPENMERIT_DIRECT_PROVIDER=google` plus the corresponding provider key;
-   OpenRouter model IDs remain on OpenRouter by default. The harness and
-   provider interfaces are extension seams for future integrations, not a
-   claim that other harnesses already work.
+   ```bash
+   nano ~/.openmerit/.env
+   chmod 600 ~/.openmerit/.env
+   ```
+
+   Pi does not read OpenMerit's `.env` file for its ordinary sessions, so an
+   OpenRouter route still needs Pi authentication. Older `model_search/.env`
+   files are not read.
 
 3. Verify the extension appears in `pi list`. The instruction file
    [`instructions/OPENMERIT.md`](instructions/OPENMERIT.md) can be added to a
    pi project's AGENTS.md for agent context, but the extension does
-   not require it.
+   not require it. Run `npx openmerit verify` for a provider-free core self-test,
+   then `npx openmerit doctor` after starting Pi once to inspect the eligible
+   route snapshot and configuration without printing secrets.
 
-4. Start pi in one terminal with an OpenRouter model:
+4. Start pi in one terminal with any configured model. For example:
 
    ```bash
    pi --provider openrouter --model openai/gpt-4o-mini
@@ -127,7 +134,8 @@ Keep that checkout available because pi loads a local package from its path.
    For a first text task, ask: “Give the shortest valid word ladder from cat
    to dog. Each step changes one letter and must be a common English word.
    Return only the path.” Wait for A to finish and leave the pi session open.
-   The extension automatically starts B and C **sequentially through pi**,
+   The extension automatically starts B and C **sequentially through their
+   selected Pi routes**,
    reports each score in Pi, and writes a recommendation for this exact
    session. With the shipped supervised policy, use
    `/openmerit` inside pi to inspect the evidence and `/openmerit apply` to
@@ -173,7 +181,13 @@ schema, so its published scores are not directly comparable to these pi runs.
 
 ### Inspect or troubleshoot a run
 
-`npx openmerit status` shows the latest pi model and pending recommendations;
+`npx openmerit doctor` checks Pi, policy, eligible routes and prices, saved
+session state, duplicate package sources, append-only files, and optional
+OpenRouter enrichment. Add `--json` for a sanitized diagnostic report suitable
+for a bug report; it contains no credentials, prompts, or trace contents.
+`npx openmerit verify` runs an offline self-test of atomic state, JSONL recovery,
+route preservation, policy evidence, and neutral events without contacting a
+provider. `npx openmerit status` shows the latest pi model and pending recommendations;
 `npx openmerit frontier` shows measured quality, blended price, latency, and
 the chosen frontier per task. `/openmerit` inside pi shows the current model,
 fallback, the model currently being compared, completed models with quality,
@@ -189,23 +203,37 @@ candidate-spend threshold.
 
 If no comparison starts after Pi settles, confirm that pi loaded the extension
 (`pi list`), the task finished, and the pi session is saved (do not use
-`--no-session`). Image candidates must advertise
-image input in OpenRouter's current model catalog. The extension queues
+`--no-session`). Image candidates must advertise image input in Pi's model
+registry. The extension queues
 completed tasks from its current session and runs one comparison at a time.
 Closing or switching the session cancels the active job.
 Candidate runs use the same text and uploaded image bytes, but they do not
 replay earlier answers or file changes. An exact task in another session
 (including identical image bytes) appears as **advice**, not a pending swap;
-the new session still gets its own comparison. The optional `trial` command
-remains for controlled text-task runs from a task JSON file.
+the new session still gets its own comparison.
+
+The optional `trial` command runs controlled text-task comparisons through the
+same exact Pi provider routes and credential store as the automatic session
+path:
+
+```bash
+npx openmerit trial examples/task.example.json --rounds 3
+```
+
+`initial_model` is the stable `vendor/model` identity. When Pi exposes that
+model through more than one provider, set `initial_route` to
+`provider:model-id` (for example `openai:gpt-4o-mini` or
+`openrouter:openai/gpt-4o-mini`). Judge and strategist calls also run through
+Pi. An OpenRouter key only adds optional catalog and public-benchmark metadata.
 
 ## Alpha boundaries
 
-- With the extension installed and an OpenRouter key configured, each supported
-  settled task can start comparison calls automatically. Candidate, judge, and
-  strategist requests send the task text, attached files or images, and
-  candidate output to OpenRouter and can incur charges. Use non-sensitive test
-  tasks and conservative account limits while evaluating this alpha.
+- With the extension installed and at least two eligible Pi routes, each
+  supported settled task can start comparison calls automatically. Candidate,
+  judge, and strategist requests send the task text, attached files or images,
+  and candidate output to the configured providers and can incur charges. Use
+  non-sensitive test tasks and conservative account limits while evaluating
+  this alpha.
 - The extension queues tasks from its current pi session and compares one at a
   time. Candidate runs use isolated temporary copies of the original working
   directory and have no Pi tools by default. Their temporary changes are
@@ -223,25 +251,39 @@ remains for controlled text-task runs from a task JSON file.
   candidate sandbox and passed back to Pi as `@` file inputs. This covers PDFs,
   CSVs, spreadsheets, and other files that Pi can open; OpenMerit does not
   implement a separate parser for them.
-- `ledger.json` counts reported candidate-trial spend and the daily number of
-  trials. It does not yet include rubric, judge, or strategist request cost,
-  and `max_usd_per_trial` is not yet a hard provider-side cap.
+- `ledger.json` counts reported candidate, rubric, judge, and strategist spend
+  for automatic session comparisons plus the daily candidate count.
+  `max_usd_per_trial` is a conservative admission estimate based on known Pi
+  prices and a 4K answer; it is not a provider-side hard cap. Routes without
+  known pricing are excluded from automatic comparisons and cannot auto-apply.
 - Each comparison has one observed baseline plus a small candidate slate and
   one quality score per answer. Treat recommendations as experimental evidence,
   not a universal model ranking.
-- Candidate execution and catalog discovery currently use Pi through
-  OpenRouter. `HarnessAdapter`, `ModelProvider`, and `CatalogProvider` are
-  implementation boundaries for future harness/provider integrations.
+- Candidate execution, judging, and strategy use the provider/model routes
+  exposed by Pi. OpenRouter remains an optional route plus catalog/benchmark
+  enrichment source. `HarnessAdapter`, `ModelProviderAdapter`,
+  `ObservationSource`, and `EventSink` remain separate integration boundaries;
+  Pi and local JSONL are the implementations shipped in this release.
+- Candidate subprocesses can use Pi built-ins and custom/local routes available
+  without loading extensions (for example routes from Pi's model
+  configuration). A provider registered only at runtime by another extension
+  is visible in the route snapshot but cannot yet be executed by the isolated
+  subprocess.
+- JSON state snapshots are replaced atomically. Append-only readers skip and
+  report malformed or interrupted lines while retaining later valid records.
+  A job owned by a crashed process is reclaimable instead of remaining stuck
+  in `running`; completed jobs remain final.
 
 ## State layout (`~/.openmerit/`)
 
 | file | contents |
 |---|---|
 | `policy.json` | the policy file (gate thresholds, budgets, intervals) |
-| `harness-state.json` | written by the pi extension: current model, fallback, latest session and settled task |
-| `recommendations.jsonl` | shared append-only channel: session-bound swap recommendations + status updates |
-| `trials.jsonl` | every model trial point (source, score, price, latency) |
-| `observations.jsonl` | task observations extracted from session traces |
+| `harness-state.json` | current/fallback routes, Pi's eligible route snapshot, latest session and settled task |
+| `recommendations.jsonl` | append-only session-bound recommendations, routes, evidence, gate reasons, and status updates |
+| `trials.jsonl` | every model trial point, including its provider route when known |
+| `traces/observations.jsonl` | task observations extracted from session traces |
+| `events.jsonl` | versioned provider-neutral observation, trial, and recommendation events for future sinks |
 | `traces/trials/*.jsonl` | raw Pi JSON event streams for candidate trials |
 | `catalog/snapshot.json` + `candidates.json` | catalog snapshot (including input modalities) + new-model queue |
 | `benchmarks/digest.json` | public-benchmark scores per model (seed + refresh) |

package/dist/cli.js

@@ -1,34 +1,18 @@
 #!/usr/bin/env node
-/** openmerit CLI: init / watch / trial / frontier / recommend / status. */
-import { copyFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
+/** openmerit CLI: init / watch / trial / frontier / recommend / status / doctor / verify. */
+import { copyFileSync, existsSync, mkdirSync } from "node:fs";
 import { dirname, join } from "node:path";
 import { fileURLToPath } from "node:url";
 import { loadPolicy } from "./policy.js";
 import { loadKey } from "./llm.js";
-import { fetchCatalog } from "./catalog.js";
 import { paretoFrontier, pickBest, pickFallback } from "./frontier.js";
-import { buildRecommendation } from "./recommend.js";
-import { appendJsonl, paths, readJson, readJsonl, taskKey } from "./store.js";
+import { paths, readJson, readJsonl } from "./store.js";
 import { autoTaskTick, recommendTick, runDaemon, tickOnce } from "./daemon.js";
-import { budgetOk, recordTrialSpend } from "./trials.js";
-import { availablePiModels, recordedActiveTask, runPiTrial, scorePiRun, settledActiveTask } from "./pi-trials.js";
-import { pickNext, STRAT_PREFS } from "./strategist.js";
-import { JUDGE_PREFS } from "./judge.js";
-import { openRouterBenchmarkCandidates, relevantBenchmarks } from "./benchmarks.js";
+import { settledActiveTask } from "./pi-trials.js";
+import { modelRoute, routeLabel } from "./routes.js";
+import { runStandaloneTrial } from "./standalone.js";
+import { collectDoctorReport, renderDiagnosticReport, runOfflineVerify } from "./diagnostics.js";
 const ROOT = dirname(dirname(fileURLToPath(import.meta.url)));
-function resolvePref(cat, prefs, label) {
-    for (const p of prefs)
-        if (p && cat.has(p))
-            return p;
-    for (const p of prefs) {
-        if (!p)
-            continue;
-        for (const id of [...cat.keys()].sort())
-            if (id.includes(p))
-                return id;
-    }
-    throw new Error(`could not resolve ${label} model`);
-}
 function cmdInit() {
     const policyPath = paths.policy();
     mkdirSync(dirname(policyPath), { recursive: true });
@@ -43,99 +27,27 @@ function cmdInit() {
     console.log(`ext.    -> ${join(ROOT, "extension", "openmerit.ts")}`);
     console.log(`pi npm  -> pi install npm:openmerit`);
     console.log(`pi local-> pi install "${ROOT}"`);
-    console.log("\nNext: set OPENROUTER_API_KEY (env or ~/.openmerit/.env), install one extension source into pi, then start pi; the extension launches comparisons automatically with candidate tools disabled by default.");
+    console.log("\nNext: authenticate at least two models in pi, install one extension source, then start pi. OpenRouter is optional enrichment; candidate tools are disabled by default.");
 }
-async function cmdTrial(taskFile, rounds) {
-    const cfg = JSON.parse(readFileSync(taskFile, "utf8"));
-    const policy = loadPolicy(paths.policy());
-    const key = loadKey();
-    console.log("fetching OpenRouter catalog...");
-    const cat = await fetchCatalog(key);
-    const piModels = availablePiModels();
-    for (const id of [...cat.keys()])
-        if (!piModels.has(id))
-            cat.delete(id);
-    if (!cat.has(cfg.initial_model))
-        throw new Error(`initial_model ${cfg.initial_model} is unavailable in pi's OpenRouter registry`);
-    console.log(`catalog: ${cat.size} models selectable in pi`);
-    const judge = resolvePref(cat, [cfg.judge_model ?? policy.judge_model, ...JUDGE_PREFS], "judge");
-    const strat = resolvePref(cat, [cfg.strategist_model ?? policy.strategist_model, ...STRAT_PREFS], "strategist");
-    const { category, benchmarks } = relevantBenchmarks(cfg.task);
-    let benchmarkCandidates = [];
+function optionalOpenRouterKey() {
     try {
-        benchmarkCandidates = await openRouterBenchmarkCandidates(key, category, cat);
-    }
-    catch (e) {
-        console.log(`OpenRouter benchmark shortlist unavailable: ${e.message}`);
-    }
-    console.log(`judge: ${judge} | strategist: ${strat} | benchmarks: ${benchmarks.join(", ")}`);
-    console.log(`OpenRouter benchmark candidates: ${benchmarkCandidates.slice(0, 5).join(", ") || "none"}`);
-    const tKey = taskKey(cfg.task);
-    const maxPrice = cfg.max_usd_per_m ?? policy.max_usd_per_m;
-    const tried = new Set();
-    const results = [];
-    const failedVendors = new Set();
-    for (let i = 0; i < rounds; i++) {
-        const budget = budgetOk(policy);
-        if (!budget.ok) {
-            console.log(`budget: ${budget.reason}; stopping`);
-            break;
-        }
-        let model;
-        let why;
-        if (i === 0) {
-            model = cfg.initial_model;
-            why = "initial model";
-        }
-        else {
-            const pick = await pickNext(key, strat, cfg.task, cfg.eval, benchmarks, results, cat, tried, maxPrice, failedVendors, benchmarkCandidates);
-            if (!pick)
-                break;
-            model = pick.model;
-            why = pick.why;
-        }
-        if (tried.has(model))
-            break;
-        console.log(`[round ${i + 1}/${rounds}] ${model} (${why}) ...`);
-        const observed = i === 0 ? recordedActiveTask(cfg.task, model) : null;
-        if (i === 0)
-            console.log(observed ? "    using model A's active pi session trace" : "    active trace unavailable; running model A in a fresh pi session");
-        const { point, costUsd, error, sessionId } = observed
-            ? await scorePiRun(key, judge, cfg.task, cfg.eval, model, cat.get(model), observed, "trace")
-            : await runPiTrial(key, judge, cfg.task, cfg.eval, model, cat.get(model));
-        tried.add(model);
-        results.push(point);
-        recordTrialSpend(costUsd);
-        appendJsonl(paths.trials(), { ...point, taskKey: tKey });
-        if (sessionId)
-            console.log(`    pi trace session: ${sessionId}`);
-        if (error) {
-            failedVendors.add(model.split("/")[0]);
-            console.log(`    FAILED: ${error.slice(0, 160)}`);
-        }
-        else {
-            console.log(`    score=${point.score.toFixed(2)}  $/M=${point.price}  est_cost=$${costUsd.toFixed(4)}  ${(point.why ?? "").slice(0, 110)}`);
-        }
+        return loadKey();
     }
-    const pts = results.filter((p) => p.score > 0);
-    if (pts.length === 0) {
-        console.log("no trials completed");
-        process.exit(1);
-    }
-    const chain = paretoFrontier(pts);
-    const best = pickBest(pts);
-    const fb = pickFallback(pts, best, policy.fallback.min_score);
-    console.log("\n=== pareto frontier (nondominated: score up, price down, latency down) ===");
-    for (const p of chain)
-        console.log(`  ${p.model.padEnd(44)} score=${p.score.toFixed(2)}  $${p.price.toFixed(2)}/M`);
-    console.log(`\nBEST FIT : ${best.model}  (score ${best.score.toFixed(2)}, $${best.price.toFixed(2)}/M)`);
-    console.log(fb ? `FALLBACK : ${fb.model}  (score ${fb.score.toFixed(2)}, $${fb.price.toFixed(2)}/M)` : "FALLBACK : n/a");
-    const rec = buildRecommendation(tKey, cfg.task.slice(0, 120), cfg.initial_model, pts, policy);
-    if (rec) {
-        appendJsonl(paths.recommendations(), rec);
-        console.log(`\nrecommendation -> ${paths.recommendations()} (auto=${rec.policy.autoApply})`);
+    catch {
+        return null;
     }
 }
+function printReport(report, json) {
+    console.log(json ? JSON.stringify(report, null, 2) : renderDiagnosticReport(report));
+    if (!report.ok)
+        process.exitCode = 1;
+}
+function cmdDoctor(json) {
+    printReport(collectDoctorReport(), json);
+}
+function cmdVerify(json) {
+    printReport(runOfflineVerify(), json);
+}
 function loadFrontiers() {
     const trials = readJsonl(paths.trials());
     const observations = readJsonl(paths.observations());
@@ -189,7 +101,8 @@ function cmdStatus() {
         latestById.set(r.id, r);
     const recs = [...latestById.values()].filter((r) => r.status === "pending");
     const frontiers = loadFrontiers();
-    console.log(`harness model : ${harness.currentModel ?? "unknown"} (as of ${harness.updatedAt ?? "n/a"})`);
+    console.log(`harness model : ${harness.currentRoute ? routeLabel(harness.currentRoute) : harness.currentModel ?? "unknown"} ` +
+        `(as of ${harness.updatedAt ?? "n/a"})`);
     console.log(`tasks tracked : ${frontiers.length}`);
     console.log(`new models queued for trial: ${queue.newModels.length}${queue.newModels.length ? " — " + queue.newModels.slice(0, 5).map((m) => m.id).join(", ") + (queue.newModels.length > 5 ? "…" : "") : ""}`);
     console.log(`pending recommendations: ${recs.length}`);
@@ -199,47 +112,59 @@ function cmdStatus() {
 }
 async function main() {
     const [cmd, ...args] = process.argv.slice(2);
-    const policy = loadPolicy(paths.policy());
     switch (cmd) {
         case "init":
             cmdInit();
             break;
         case "session-trial": {
-            const [sessionFile, settledAt, currentModel, settledTaskKey, cwd, bytesText] = args;
+            const policy = loadPolicy(paths.policy());
+            const [sessionFile, settledAt, currentModel, settledTaskKey, cwd, bytesText, provider, providerModelId] = args;
             if (!sessionFile || !settledAt || !currentModel || !settledTaskKey || !cwd || !bytesText)
                 throw new Error("usage: openmerit session-trial <session-file> <settled-at> <model> <task-key> <cwd> <session-bytes>");
             const sessionBytes = Number(bytesText);
             if (!Number.isSafeInteger(sessionBytes) || sessionBytes <= 0)
                 throw new Error("invalid session byte limit");
-            const task = settledActiveTask({ sessionFile, settledAt, currentModel, settledTaskKey, cwd, sessionBytes });
+            const state = readJson(paths.harnessState(), {});
+            const currentRoute = provider && providerModelId
+                ? state.routes?.find((route) => route.provider === provider && route.modelId === providerModelId) ??
+                    modelRoute(provider, providerModelId)
+                : state.currentRoute;
+            const task = settledActiveTask({ sessionFile, settledAt, currentModel, currentRoute,
+                routes: state.routes, settledTaskKey, cwd, sessionBytes });
             if (!task)
                 throw new Error("the specified pi session has no completed, supported task matching this marker");
-            await autoTaskTick(loadKey(), policy, task);
+            await autoTaskTick(optionalOpenRouterKey(), policy, task);
             break;
         }
         case "watch":
             if (args.includes("--once"))
-                await tickOnce(policy);
+                await tickOnce(loadPolicy(paths.policy()));
             else
-                await runDaemon(policy);
+                await runDaemon(loadPolicy(paths.policy()));
             break;
         case "trial": {
             const file = args[0];
             if (!file)
                 throw new Error("usage: openmerit trial <task.json> [--rounds N]");
             const rIdx = args.indexOf("--rounds");
-            await cmdTrial(file, rIdx >= 0 ? Number(args[rIdx + 1]) : 5);
+            await runStandaloneTrial(file, rIdx >= 0 ? Number(args[rIdx + 1]) : 5);
             break;
         }
         case "frontier":
             cmdFrontier(args[0]);
             break;
         case "recommend":
-            recommendTick(policy);
+            recommendTick(loadPolicy(paths.policy()));
             break;
         case "status":
             cmdStatus();
             break;
+        case "doctor":
+            cmdDoctor(args.includes("--json"));
+            break;
+        case "verify":
+            cmdVerify(args.includes("--json"));
+            break;
         default:
             console.log(`openmerit — external model-merit harness
 
@@ -250,7 +175,9 @@ usage: openmerit <command>
   trial <task.json>    iterative model search on one task spec [--rounds N]
   frontier [taskKey]   print pareto frontier(s)
   recommend            emit recommendations now
-  status               harness model, queued models, pending recommendations`);
+  status               harness model, queued models, pending recommendations
+  doctor [--json]      diagnose pi, policy, routes, state, and installation
+  verify [--json]      run an offline, provider-free core self-test`);
             if (cmd && cmd !== "help" && cmd !== "--help")
                 process.exitCode = 1;
     }