openmerit 0.1.0 → 0.1.2

package/README.md

@@ -32,10 +32,10 @@ pi task A → saved pi trace → extension trial job → pi trials B, C → scor
    ([`src/frontier.ts`](src/frontier.ts), 3 objectives: quality ↑, price ↓,
    latency ↓). The aggregate across all of an agent's tasks is the union of
    its task frontiers (`openmerit frontier`).
-4. **Model discovery** — the OpenRouter catalog is snapshotted and diffed;
-   newly appeared models are queued for future release experiments. The
-   per-task loop shortlists currently available pi/OpenRouter models with
-   OpenRouter benchmark signals and the existing strategist
+4. **Model discovery** — the active Pi session's scoped models (or Pi's
+   authenticated available-model registry when unscoped) are the candidate
+   pool. OpenRouter is an optional route and enrichment source: its catalog can
+   be snapshotted and its benchmark signals can improve the shortlist
    ([`src/catalog.ts`](src/catalog.ts) + [`src/daemon.ts`](src/daemon.ts)).
 5. **Informing the main harness** — recommendations land in
    `~/.openmerit/recommendations.jsonl`. The pi extension
@@ -47,8 +47,9 @@ pi task A → saved pi trace → extension trial job → pi trials B, C → scor
 
 ## Install from npm
 
-You need Node 22.18+, pi 0.85.1+, and an OpenRouter API key with available
-credits. Install the package into pi, then initialize its policy and state:
+You need Node 22.18+, pi 0.85.1+, and at least two eligible model routes
+authenticated in Pi. OpenRouter is supported but not required. Install the
+package into pi, then initialize its policy and state:
 
 ```bash
 pi install npm:openmerit
@@ -59,7 +60,7 @@ pi list
 `pi install` makes the extension and its trial engine available to pi. The
 one-off `npx` command creates `~/.openmerit/policy.json`; install OpenMerit
 globally with `npm install --global openmerit` only if you also want persistent
-shell access to `openmerit status`, `frontier`, or the optional watcher. Install
+shell access to `openmerit status`, `doctor`, `frontier`, or the optional watcher. Install
 the extension from only one source—remove any older copied `openmerit.ts` first
 so pi does not load it twice.
 
@@ -85,47 +86,64 @@ Keep that checkout available because pi loads a local package from its path.
    policy**. The shipped policy uses `"mode": "recommend"` and does not change
    the active model without approval.
 
-2. Give **Pi and the extension trial job** OpenRouter access. The job reads
-   `OPENROUTER_API_KEY` from its environment or `~/.openmerit/.env`. If using
-   the file, edit it to contain `OPENROUTER_API_KEY=your-key` and protect it:
+2. Authenticate the providers you want to compare in Pi, using `/login` or
+   Pi's normal environment/model configuration. OpenMerit takes its eligible
+   model routes, capabilities, prices, and credentials from Pi; judge and
+   strategist calls use the same routes and do not require duplicate keys.
+
+   For example, verify one provider without printing its credential:
 
    ```bash
-   nano ~/.openmerit/.env
-   chmod 600 ~/.openmerit/.env
+   pi auth check --provider openai --json
    ```
 
-   Pi does **not** read OpenMerit's `.env` file. Log in to OpenRouter with
-   `/login openrouter` inside pi, or export `OPENROUTER_API_KEY` in the shell
-   that starts pi. Verify pi's side without printing the key:
+   Add an OpenRouter route the same way if you want its routed catalog:
 
    ```bash
    pi auth check --provider openrouter --json
    ```
 
-   A ready configuration prints `{"status":"ready","provider":"openrouter",...}`.
-   Older `model_search/.env` files are not read by the trial engine.
+   An `OPENROUTER_API_KEY` in the process environment or
+   `~/.openmerit/.env` additionally enables OpenRouter catalog and public-
+   benchmark enrichment. It is optional for native-provider comparisons. If
+   using the file, protect it:
+
+   ```bash
+   nano ~/.openmerit/.env
+   chmod 600 ~/.openmerit/.env
+   ```
+
+   Pi does not read OpenMerit's `.env` file for its ordinary sessions, so an
+   OpenRouter route still needs Pi authentication. Older `model_search/.env`
+   files are not read.
 
 3. Verify the extension appears in `pi list`. The instruction file
    [`instructions/OPENMERIT.md`](instructions/OPENMERIT.md) can be added to a
    pi project's AGENTS.md for agent context, but the extension does
-   not require it.
+   not require it. Run `npx openmerit doctor` after starting Pi once to inspect
+   the eligible route snapshot and configuration without printing secrets.
 
-4. Start pi in one terminal with an OpenRouter model and tools disabled:
+4. Start pi in one terminal with any configured model. For example:
 
    ```bash
-   pi --provider openrouter --model openai/gpt-4o-mini --no-tools
+   pi --provider openrouter --model openai/gpt-4o-mini
    ```
 
    For a first text task, ask: “Give the shortest valid word ladder from cat
    to dog. Each step changes one letter and must be a common English word.
    Return only the path.” Wait for A to finish and leave the pi session open.
-   The extension automatically starts B and C **sequentially through pi**,
+   The extension automatically starts B and C **sequentially through their
+   selected Pi routes**,
    reports each score in Pi, and writes a recommendation for this exact
    session. With the shipped supervised policy, use
    `/openmerit` inside pi to inspect the evidence and `/openmerit apply` to
    switch. Then send a second message to see which model actually handles it.
    You do not need a watcher terminal or the standalone `trial` command.
 
+   Candidate comparisons have no tools by default, even when the observed task
+   used tools. See **Alpha boundaries** before explicitly enabling candidate
+   tools.
+
    To opt in to automatic swaps, edit `~/.openmerit/policy.json`, set
    `"mode": "auto"` and `"auto_apply.enabled": true`, review the score-gain
    and price-ratio thresholds, and run the next task.
@@ -133,7 +151,7 @@ Keep that checkout available because pi loads a local package from its path.
 ### Try an invoice-to-JSON task
 
 Use the same one-terminal setup. Start pi with a vision-capable model, for
-example `openai/gpt-4o-mini`, and `--no-tools`. In pi, type `@` to select
+example `openai/gpt-4o-mini`. In pi, type `@` to select
 [`benchmark/invoice_ocr/data/invoice_01_row_2.jpg`](benchmark/invoice_ocr/data/invoice_01_row_2.jpg)
 and paste the **entire, unchanged** text from
 [`examples/invoice-prompt.txt`](examples/invoice-prompt.txt) into the same
@@ -161,7 +179,8 @@ schema, so its published scores are not directly comparable to these pi runs.
 
 ### Inspect or troubleshoot a run
 
-`npx openmerit status` shows the latest pi model and pending recommendations;
+`npx openmerit doctor` checks Pi, policy, eligible routes, and optional
+OpenRouter enrichment. `npx openmerit status` shows the latest pi model and pending recommendations;
 `npx openmerit frontier` shows measured quality, blended price, latency, and
 the chosen frontier per task. `/openmerit` inside pi shows the current model,
 fallback, the model currently being compared, completed models with quality,
@@ -176,43 +195,74 @@ for local experiments while keeping `budgets.max_usd_per_day` as a conservative
 candidate-spend threshold.
 
 If no comparison starts after Pi settles, confirm that pi loaded the extension
-(`pi list`), the task finished, the pi session is saved (do not use
-`--no-session`), and the task used no tools. Image candidates must advertise
-image input in OpenRouter's current model catalog. The extension queues
+(`pi list`), the task finished, and the pi session is saved (do not use
+`--no-session`). Image candidates must advertise image input in Pi's model
+registry. The extension queues
 completed tasks from its current session and runs one comparison at a time.
 Closing or switching the session cancels the active job.
 Candidate runs use the same text and uploaded image bytes, but they do not
 replay earlier answers or file changes. An exact task in another session
 (including identical image bytes) appears as **advice**, not a pending swap;
 the new session still gets its own comparison. The optional `trial` command
-remains for controlled text-task runs from a task JSON file.
+remains for controlled text-task runs from a task JSON file; that legacy
+standalone command is still OpenRouter-specific in 0.1.2. The automatic Pi
+session path is provider-neutral.
 
 ## Alpha boundaries
 
-- With the extension installed and an OpenRouter key configured, each supported
-  settled task can start comparison calls automatically. Candidate, judge, and
-  strategist requests send the task text, attached images, and candidate output
-  to OpenRouter and can incur charges. Use non-sensitive test tasks and
-  conservative account limits while evaluating this alpha.
-- The extension queues tasks from its current pi session, compares one at a
-  time, and skips tasks that use tools. Candidate sessions replay the latest
-  text and images, not earlier conversation context or workspace changes.
-- `ledger.json` counts reported candidate-trial spend and the daily number of
-  trials. It does not yet include rubric, judge, or strategist request cost,
-  and `max_usd_per_trial` is not yet a hard provider-side cap.
+- With the extension installed and at least two eligible Pi routes, each
+  supported settled task can start comparison calls automatically. Candidate,
+  judge, and strategist requests send the task text, attached files or images,
+  and candidate output to the configured providers and can incur charges. Use
+  non-sensitive test tasks and conservative account limits while evaluating
+  this alpha.
+- The extension queues tasks from its current pi session and compares one at a
+  time. Candidate runs use isolated temporary copies of the original working
+  directory and have no Pi tools by default. Their temporary changes are
+  discarded and recorded as counts, and raw Pi JSON event streams are saved
+  under `~/.openmerit/traces/trials/`.
+  Candidate sessions replay the task text and images, not earlier conversation
+  context or workspace changes.
+- Set `OPENMERIT_PI_TRIAL_TOOLS` to an explicit comma-separated allowlist such
+  as `read,grep,find,ls` to let candidate models use tools; `none` keeps them
+  disabled. Any tool access is an advanced opt-in: the copied working directory
+  prevents ordinary project writes from touching the original, but it is not an
+  OS sandbox. Tools may accept absolute paths, and shell tools may access the
+  network. Keep the no-tools default for untrusted or sensitive projects.
+- File attachments that Pi records as `<file name="…">` are copied into every
+  candidate sandbox and passed back to Pi as `@` file inputs. This covers PDFs,
+  CSVs, spreadsheets, and other files that Pi can open; OpenMerit does not
+  implement a separate parser for them.
+- `ledger.json` counts reported candidate, rubric, judge, and strategist spend
+  for automatic session comparisons plus the daily candidate count.
+  `max_usd_per_trial` is a conservative admission estimate based on known Pi
+  prices and a 4K answer; it is not a provider-side hard cap. Routes without
+  known pricing are excluded from automatic comparisons and cannot auto-apply.
 - Each comparison has one observed baseline plus a small candidate slate and
   one quality score per answer. Treat recommendations as experimental evidence,
   not a universal model ranking.
+- Candidate execution, judging, and strategy use the provider/model routes
+  exposed by Pi. OpenRouter remains an optional route plus catalog/benchmark
+  enrichment source. `HarnessAdapter`, `ModelProviderAdapter`,
+  `ObservationSource`, and `EventSink` remain separate integration boundaries;
+  Pi and local JSONL are the implementations shipped in this release.
+- Candidate subprocesses can use Pi built-ins and custom/local routes available
+  without loading extensions (for example routes from Pi's model
+  configuration). A provider registered only at runtime by another extension
+  is visible in the route snapshot but cannot yet be executed by the isolated
+  subprocess.
 
 ## State layout (`~/.openmerit/`)
 
 | file | contents |
 |---|---|
 | `policy.json` | the policy file (gate thresholds, budgets, intervals) |
-| `harness-state.json` | written by the pi extension: current model, fallback, latest session and settled task |
-| `recommendations.jsonl` | shared append-only channel: session-bound swap recommendations + status updates |
-| `trials.jsonl` | every model trial point (source, score, price, latency) |
-| `observations.jsonl` | task observations extracted from session traces |
+| `harness-state.json` | current/fallback routes, Pi's eligible route snapshot, latest session and settled task |
+| `recommendations.jsonl` | append-only session-bound recommendations, routes, evidence, gate reasons, and status updates |
+| `trials.jsonl` | every model trial point, including its provider route when known |
+| `traces/observations.jsonl` | task observations extracted from session traces |
+| `events.jsonl` | versioned provider-neutral observation, trial, and recommendation events for future sinks |
+| `traces/trials/*.jsonl` | raw Pi JSON event streams for candidate trials |
 | `catalog/snapshot.json` + `candidates.json` | catalog snapshot (including input modalities) + new-model queue |
 | `benchmarks/digest.json` | public-benchmark scores per model (seed + refresh) |
 | `ledger.json` | daily trial spend (budget enforcement) |

package/dist/catalog.js

@@ -1,7 +1,17 @@
 /** OpenRouter catalog fetch + snapshot diffing: the "new model release" watch. */
 import { readJson, paths, writeJson } from "./store.js";
 const OR = "https://openrouter.ai/api/v1";
-export async function fetchCatalog(key) {
+export class OpenRouterCatalogProvider {
+    key;
+    id = "openrouter";
+    constructor(key) {
+        this.key = key;
+    }
+    fetchCatalog() { return fetchCatalog(this.key); }
+}
+export async function fetchCatalog(key, provider = "openrouter") {
+    if (provider !== "openrouter")
+        return new Map();
     const res = await fetch(`${OR}/models`, {
         headers: { Authorization: `Bearer ${key}` },
     });

package/dist/cli.js

@@ -1,5 +1,5 @@
 #!/usr/bin/env node
-/** openmerit CLI: init / watch / trial / frontier / recommend / status. */
+/** openmerit CLI: init / watch / trial / frontier / recommend / status / doctor. */
 import { copyFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
 import { dirname, join } from "node:path";
 import { fileURLToPath } from "node:url";
@@ -11,10 +11,12 @@ import { buildRecommendation } from "./recommend.js";
 import { appendJsonl, paths, readJson, readJsonl, taskKey } from "./store.js";
 import { autoTaskTick, recommendTick, runDaemon, tickOnce } from "./daemon.js";
 import { budgetOk, recordTrialSpend } from "./trials.js";
-import { availablePiModels, recordedActiveTask, runPiTrial, scorePiRun, settledActiveTask } from "./pi-trials.js";
+import { availablePiModels, availablePiRoutes, piExecutable, recordedActiveTask, runPiTrial, scorePiRun, settledActiveTask } from "./pi-trials.js";
 import { pickNext, STRAT_PREFS } from "./strategist.js";
 import { JUDGE_PREFS } from "./judge.js";
 import { openRouterBenchmarkCandidates, relevantBenchmarks } from "./benchmarks.js";
+import { modelRoute, routeLabel } from "./routes.js";
+import { spawnSync } from "node:child_process";
 const ROOT = dirname(dirname(fileURLToPath(import.meta.url)));
 function resolvePref(cat, prefs, label) {
     for (const p of prefs)
@@ -43,7 +45,48 @@ function cmdInit() {
     console.log(`ext.    -> ${join(ROOT, "extension", "openmerit.ts")}`);
     console.log(`pi npm  -> pi install npm:openmerit`);
     console.log(`pi local-> pi install "${ROOT}"`);
-    console.log("\nNext: set OPENROUTER_API_KEY (env or ~/.openmerit/.env), install one extension source into pi, then start pi; the extension launches comparisons automatically.");
+    console.log("\nNext: authenticate at least two models in pi, install one extension source, then start pi. OpenRouter is optional enrichment; candidate tools are disabled by default.");
+}
+function optionalOpenRouterKey() {
+    try {
+        return loadKey();
+    }
+    catch {
+        return null;
+    }
+}
+function cmdDoctor() {
+    const executable = piExecutable();
+    const version = spawnSync(executable, ["--version"], { encoding: "utf8" });
+    const state = readJson(paths.harnessState(), {});
+    console.log(`pi executable : ${executable}`);
+    console.log(`pi detected   : ${version.status === 0 ? `yes (${version.stdout.trim()})` : "no"}`);
+    try {
+        const policy = loadPolicy(paths.policy());
+        console.log(`policy        : valid (v${policy.version}, ${policy.mode})`);
+    }
+    catch (error) {
+        console.log(`policy        : invalid (${error.message})`);
+    }
+    let routes = state.routes ?? [];
+    const hasLiveSnapshot = routes.length > 0;
+    if (routes.length === 0 && version.status === 0) {
+        try {
+            routes = availablePiRoutes();
+        }
+        catch { /* reported below */ }
+    }
+    console.log(`eligible routes: ${routes.length}`);
+    for (const route of routes.slice(0, 8))
+        console.log(`  - ${routeLabel(route)}${route.cost ? "" : " (price unknown)"}`);
+    if (routes.length > 8)
+        console.log(`  … ${routes.length - 8} more`);
+    console.log(`active route  : ${state.currentRoute ? routeLabel(state.currentRoute) : "not reported; start pi once"}`);
+    const pricedRoutes = routes.filter((route) => !!route.cost).length;
+    console.log(`alternates    : ${hasLiveSnapshot
+        ? pricedRoutes >= 2 ? "ready" : "need at least two priced eligible pi routes"
+        : "start Pi once to capture route prices and capabilities"}`);
+    console.log(`OpenRouter enrichment: ${optionalOpenRouterKey() ? "available" : "not configured (optional)"}`);
 }
 async function cmdTrial(taskFile, rounds) {
     const cfg = JSON.parse(readFileSync(taskFile, "utf8"));
@@ -189,7 +232,8 @@ function cmdStatus() {
         latestById.set(r.id, r);
     const recs = [...latestById.values()].filter((r) => r.status === "pending");
     const frontiers = loadFrontiers();
-    console.log(`harness model : ${harness.currentModel ?? "unknown"} (as of ${harness.updatedAt ?? "n/a"})`);
+    console.log(`harness model : ${harness.currentRoute ? routeLabel(harness.currentRoute) : harness.currentModel ?? "unknown"} ` +
+        `(as of ${harness.updatedAt ?? "n/a"})`);
     console.log(`tasks tracked : ${frontiers.length}`);
     console.log(`new models queued for trial: ${queue.newModels.length}${queue.newModels.length ? " — " + queue.newModels.slice(0, 5).map((m) => m.id).join(", ") + (queue.newModels.length > 5 ? "…" : "") : ""}`);
     console.log(`pending recommendations: ${recs.length}`);
@@ -205,16 +249,22 @@ async function main() {
             cmdInit();
             break;
         case "session-trial": {
-            const [sessionFile, settledAt, currentModel, settledTaskKey, cwd, bytesText] = args;
+            const [sessionFile, settledAt, currentModel, settledTaskKey, cwd, bytesText, provider, providerModelId] = args;
             if (!sessionFile || !settledAt || !currentModel || !settledTaskKey || !cwd || !bytesText)
                 throw new Error("usage: openmerit session-trial <session-file> <settled-at> <model> <task-key> <cwd> <session-bytes>");
             const sessionBytes = Number(bytesText);
             if (!Number.isSafeInteger(sessionBytes) || sessionBytes <= 0)
                 throw new Error("invalid session byte limit");
-            const task = settledActiveTask({ sessionFile, settledAt, currentModel, settledTaskKey, cwd, sessionBytes });
+            const state = readJson(paths.harnessState(), {});
+            const currentRoute = provider && providerModelId
+                ? state.routes?.find((route) => route.provider === provider && route.modelId === providerModelId) ??
+                    modelRoute(provider, providerModelId)
+                : state.currentRoute;
+            const task = settledActiveTask({ sessionFile, settledAt, currentModel, currentRoute,
+                routes: state.routes, settledTaskKey, cwd, sessionBytes });
             if (!task)
                 throw new Error("the specified pi session has no completed, supported task matching this marker");
-            await autoTaskTick(loadKey(), policy, task);
+            await autoTaskTick(optionalOpenRouterKey(), policy, task);
             break;
         }
         case "watch":
@@ -240,6 +290,9 @@ async function main() {
         case "status":
             cmdStatus();
             break;
+        case "doctor":
+            cmdDoctor();
+            break;
         default:
             console.log(`openmerit — external model-merit harness
 
@@ -250,7 +303,8 @@ usage: openmerit <command>
   trial <task.json>    iterative model search on one task spec [--rounds N]
   frontier [taskKey]   print pareto frontier(s)
   recommend            emit recommendations now
-  status               harness model, queued models, pending recommendations`);
+  status               harness model, queued models, pending recommendations
+  doctor               verify pi, policy, routes, and optional enrichment`);
             if (cmd && cmd !== "help" && cmd !== "--help")
                 process.exitCode = 1;
     }