openmerit 0.1.0 → 0.1.1

package/README.md

@@ -105,15 +105,23 @@ Keep that checkout available because pi loads a local package from its path.
    A ready configuration prints `{"status":"ready","provider":"openrouter",...}`.
    Older `model_search/.env` files are not read by the trial engine.
 
+   OpenRouter is currently required for candidate runs and catalog discovery.
+   Judge and strategist calls can use OpenAI or Gemini directly, but require
+   `OPENMERIT_DIRECT_PROVIDER=openai` or
+   `OPENMERIT_DIRECT_PROVIDER=google` plus the corresponding provider key;
+   OpenRouter model IDs remain on OpenRouter by default. The harness and
+   provider interfaces are extension seams for future integrations, not a
+   claim that other harnesses already work.
+
 3. Verify the extension appears in `pi list`. The instruction file
    [`instructions/OPENMERIT.md`](instructions/OPENMERIT.md) can be added to a
    pi project's AGENTS.md for agent context, but the extension does
    not require it.
 
-4. Start pi in one terminal with an OpenRouter model and tools disabled:
+4. Start pi in one terminal with an OpenRouter model:
 
    ```bash
-   pi --provider openrouter --model openai/gpt-4o-mini --no-tools
+   pi --provider openrouter --model openai/gpt-4o-mini
    ```
 
    For a first text task, ask: “Give the shortest valid word ladder from cat
@@ -126,6 +134,10 @@ Keep that checkout available because pi loads a local package from its path.
    switch. Then send a second message to see which model actually handles it.
    You do not need a watcher terminal or the standalone `trial` command.
 
+   Candidate comparisons have no tools by default, even when the observed task
+   used tools. See **Alpha boundaries** before explicitly enabling candidate
+   tools.
+
    To opt in to automatic swaps, edit `~/.openmerit/policy.json`, set
    `"mode": "auto"` and `"auto_apply.enabled": true`, review the score-gain
    and price-ratio thresholds, and run the next task.
@@ -133,7 +145,7 @@ Keep that checkout available because pi loads a local package from its path.
 ### Try an invoice-to-JSON task
 
 Use the same one-terminal setup. Start pi with a vision-capable model, for
-example `openai/gpt-4o-mini`, and `--no-tools`. In pi, type `@` to select
+example `openai/gpt-4o-mini`. In pi, type `@` to select
 [`benchmark/invoice_ocr/data/invoice_01_row_2.jpg`](benchmark/invoice_ocr/data/invoice_01_row_2.jpg)
 and paste the **entire, unchanged** text from
 [`examples/invoice-prompt.txt`](examples/invoice-prompt.txt) into the same
@@ -176,8 +188,8 @@ for local experiments while keeping `budgets.max_usd_per_day` as a conservative
 candidate-spend threshold.
 
 If no comparison starts after Pi settles, confirm that pi loaded the extension
-(`pi list`), the task finished, the pi session is saved (do not use
-`--no-session`), and the task used no tools. Image candidates must advertise
+(`pi list`), the task finished, and the pi session is saved (do not use
+`--no-session`). Image candidates must advertise
 image input in OpenRouter's current model catalog. The extension queues
 completed tasks from its current session and runs one comparison at a time.
 Closing or switching the session cancels the active job.
@@ -191,18 +203,35 @@ remains for controlled text-task runs from a task JSON file.
 
 - With the extension installed and an OpenRouter key configured, each supported
   settled task can start comparison calls automatically. Candidate, judge, and
-  strategist requests send the task text, attached images, and candidate output
-  to OpenRouter and can incur charges. Use non-sensitive test tasks and
-  conservative account limits while evaluating this alpha.
-- The extension queues tasks from its current pi session, compares one at a
-  time, and skips tasks that use tools. Candidate sessions replay the latest
-  text and images, not earlier conversation context or workspace changes.
+  strategist requests send the task text, attached files or images, and
+  candidate output to OpenRouter and can incur charges. Use non-sensitive test
+  tasks and conservative account limits while evaluating this alpha.
+- The extension queues tasks from its current pi session and compares one at a
+  time. Candidate runs use isolated temporary copies of the original working
+  directory and have no Pi tools by default. Their temporary changes are
+  discarded and recorded as counts, and raw Pi JSON event streams are saved
+  under `~/.openmerit/traces/trials/`.
+  Candidate sessions replay the task text and images, not earlier conversation
+  context or workspace changes.
+- Set `OPENMERIT_PI_TRIAL_TOOLS` to an explicit comma-separated allowlist such
+  as `read,grep,find,ls` to let candidate models use tools; `none` keeps them
+  disabled. Any tool access is an advanced opt-in: the copied working directory
+  prevents ordinary project writes from touching the original, but it is not an
+  OS sandbox. Tools may accept absolute paths, and shell tools may access the
+  network. Keep the no-tools default for untrusted or sensitive projects.
+- File attachments that Pi records as `<file name="…">` are copied into every
+  candidate sandbox and passed back to Pi as `@` file inputs. This covers PDFs,
+  CSVs, spreadsheets, and other files that Pi can open; OpenMerit does not
+  implement a separate parser for them.
 - `ledger.json` counts reported candidate-trial spend and the daily number of
   trials. It does not yet include rubric, judge, or strategist request cost,
   and `max_usd_per_trial` is not yet a hard provider-side cap.
 - Each comparison has one observed baseline plus a small candidate slate and
   one quality score per answer. Treat recommendations as experimental evidence,
   not a universal model ranking.
+- Candidate execution and catalog discovery currently use Pi through
+  OpenRouter. `HarnessAdapter`, `ModelProvider`, and `CatalogProvider` are
+  implementation boundaries for future harness/provider integrations.
 
 ## State layout (`~/.openmerit/`)
 
@@ -213,6 +242,7 @@ remains for controlled text-task runs from a task JSON file.
 | `recommendations.jsonl` | shared append-only channel: session-bound swap recommendations + status updates |
 | `trials.jsonl` | every model trial point (source, score, price, latency) |
 | `observations.jsonl` | task observations extracted from session traces |
+| `traces/trials/*.jsonl` | raw Pi JSON event streams for candidate trials |
 | `catalog/snapshot.json` + `candidates.json` | catalog snapshot (including input modalities) + new-model queue |
 | `benchmarks/digest.json` | public-benchmark scores per model (seed + refresh) |
 | `ledger.json` | daily trial spend (budget enforcement) |

package/dist/catalog.js

@@ -1,7 +1,17 @@
 /** OpenRouter catalog fetch + snapshot diffing: the "new model release" watch. */
 import { readJson, paths, writeJson } from "./store.js";
 const OR = "https://openrouter.ai/api/v1";
-export async function fetchCatalog(key) {
+export class OpenRouterCatalogProvider {
+    key;
+    id = "openrouter";
+    constructor(key) {
+        this.key = key;
+    }
+    fetchCatalog() { return fetchCatalog(this.key); }
+}
+export async function fetchCatalog(key, provider = "openrouter") {
+    if (provider !== "openrouter")
+        return new Map();
     const res = await fetch(`${OR}/models`, {
         headers: { Authorization: `Bearer ${key}` },
     });

package/dist/cli.js

@@ -43,7 +43,7 @@ function cmdInit() {
     console.log(`ext.    -> ${join(ROOT, "extension", "openmerit.ts")}`);
     console.log(`pi npm  -> pi install npm:openmerit`);
     console.log(`pi local-> pi install "${ROOT}"`);
-    console.log("\nNext: set OPENROUTER_API_KEY (env or ~/.openmerit/.env), install one extension source into pi, then start pi; the extension launches comparisons automatically.");
+    console.log("\nNext: set OPENROUTER_API_KEY (env or ~/.openmerit/.env), install one extension source into pi, then start pi; the extension launches comparisons automatically with candidate tools disabled by default.");
 }
 async function cmdTrial(taskFile, rounds) {
     const cfg = JSON.parse(readFileSync(taskFile, "utf8"));

package/dist/daemon.js

@@ -168,7 +168,8 @@ export async function autoTaskTick(key, policy, settledTask) {
             emitTrialProgress({ phase: "complete", taskKey: tKey,
                 model: task.model, index: 1, total, score: baseline.point.score,
                 price: baseline.point.price, latencyMs: baseline.point.latencyMs,
-                costUsd: baseline.costUsd });
+                costUsd: baseline.costUsd, toolCalls: baseline.point.toolCalls,
+                toolErrors: baseline.point.toolErrors, changedFiles: baseline.point.changedFiles });
         for (let i = 1; i < total; i++) {
             if (!budgetOk(policy).ok)
                 break;
@@ -178,18 +179,21 @@ export async function autoTaskTick(key, policy, settledTask) {
             if (settledTask !== undefined)
                 emitTrialProgress({ phase: "start", taskKey: tKey,
                     model: pick.model, index: i + 1, total });
-            const { point, costUsd, error } = await runPiTrial(key, judge, task.task, rubric, pick.model, cat.get(pick.model), task.cwd, task.images);
+            const { point, costUsd, error } = await runPiTrial(key, judge, task.task, rubric, pick.model, cat.get(pick.model), task.cwd, task.images, task.files);
             tried.add(pick.model);
             points.push(point);
             recordTrialSpend(costUsd);
             appendJsonl(paths.trials(), { ...point, taskKey: tKey, sessionFile: task.sessionFile });
             if (error)
                 failedVendors.add(pick.model.split("/")[0]);
-            console.log(`[openmerit] task ${tKey}: ${pick.model} score=${point.score.toFixed(2)} cost=$${costUsd.toFixed(4)}${error ? ` error=${error.slice(0, 80)}` : ""}`);
+            console.log(`[openmerit] task ${tKey}: ${pick.model} score=${point.score.toFixed(2)} cost=$${costUsd.toFixed(4)} ` +
+                `tools=${point.toolCalls ?? 0} toolErrors=${point.toolErrors ?? 0} changedFiles=${point.changedFiles ?? 0}` +
+                `${error ? ` error=${error.slice(0, 80)}` : ""}`);
             if (settledTask !== undefined)
                 emitTrialProgress({ phase: "complete", taskKey: tKey,
                     model: pick.model, index: i + 1, total, score: point.score, price: point.price,
-                    latencyMs: point.latencyMs, costUsd, error: error?.slice(0, 120) });
+                    latencyMs: point.latencyMs, costUsd, error: error?.slice(0, 120),
+                    toolCalls: point.toolCalls, toolErrors: point.toolErrors, changedFiles: point.changedFiles });
         }
         const rec = buildRecommendation(tKey, task.task.slice(0, 120), task.model, points, policy, task.sessionFile);
         if (rec) {

package/dist/harness.js

@@ -0,0 +1 @@
+export {};

package/dist/llm.js

@@ -1,4 +1,4 @@
-/** Minimal OpenRouter chat client for catalog and trial requests. */
+/** Minimal provider clients for judge and strategist requests. */
 import { existsSync, readFileSync } from "node:fs";
 import { paths } from "./store.js";
 const OR = "https://openrouter.ai/api/v1";
@@ -20,9 +20,9 @@ export function loadKey() {
         throw new Error("missing OPENROUTER_API_KEY (env or ~/.openmerit/.env)");
     return key;
 }
-async function request(key, path, body, tries = 3) {
+async function request(key, path, body, baseUrl = OR, tries = 3) {
     for (let attempt = 0; attempt < tries; attempt++) {
-        const res = await fetch(OR + path, {
+        const res = await fetch(baseUrl + path, {
             method: body === undefined ? "GET" : "POST",
             headers: {
                 Authorization: `Bearer ${key}`,
@@ -41,26 +41,62 @@ async function request(key, path, body, tries = 3) {
     }
     throw new Error("unreachable");
 }
+class OpenAICompatibleProvider {
+    id;
+    baseUrl;
+    key;
+    constructor(id, baseUrl, key) {
+        this.id = id;
+        this.baseUrl = baseUrl;
+        this.key = key;
+    }
+    model(model) { return this.id === "openrouter" ? model : model.split("/", 2).at(-1) ?? model; }
+    async chat(model, prompt, maxTokens, temperature) {
+        const r = await request(this.key, "/chat/completions", { model: this.model(model), messages: [{ role: "user", content: prompt }], max_tokens: maxTokens, temperature }, this.baseUrl);
+        return { content: r.choices[0]?.message?.content ?? "", usage: r.usage ?? {} };
+    }
+    async chatWithImages(model, prompt, images, maxTokens) {
+        const content = [{ type: "text", text: prompt }, ...images.map((image) => ({ type: "image_url", image_url: { url: `data:${image.mimeType};base64,${image.data}` } }))];
+        const r = await request(this.key, "/chat/completions", { model: this.model(model), messages: [{ role: "user", content }], max_tokens: maxTokens, temperature: 0 }, this.baseUrl);
+        return { content: r.choices[0]?.message?.content ?? "", usage: r.usage ?? {} };
+    }
+}
+class GeminiProvider {
+    key;
+    id = "google";
+    constructor(key) {
+        this.key = key;
+    }
+    async call(model, prompt, images = [], maxTokens) {
+        const name = model.split("/", 2).at(-1) ?? model;
+        const parts = [{ text: prompt }, ...images.map((image) => ({ inline_data: { mime_type: image.mimeType, data: image.data } }))];
+        const res = await fetch(`https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(name)}:generateContent?key=${encodeURIComponent(this.key)}`, {
+            method: "POST", headers: { "Content-Type": "application/json" },
+            body: JSON.stringify({ contents: [{ role: "user", parts }], generationConfig: { maxOutputTokens: maxTokens, temperature: images.length ? 0 : undefined } }),
+        });
+        if (!res.ok)
+            throw new Error(`Google Gemini HTTP ${res.status}: ${(await res.text()).slice(0, 300)}`);
+        const raw = await res.json();
+        return { content: raw.candidates?.[0]?.content?.parts?.map((p) => p.text ?? "").join("") ?? "", usage: { prompt_tokens: raw.usageMetadata?.promptTokenCount, completion_tokens: raw.usageMetadata?.candidatesTokenCount, total_tokens: raw.usageMetadata?.totalTokenCount } };
+    }
+    chat(model, prompt, maxTokens) { return this.call(model, prompt, [], maxTokens); }
+    chatWithImages(model, prompt, images, maxTokens) { return this.call(model, prompt, images, maxTokens); }
+}
+export function providerFor(model, fallbackKey) {
+    const id = model.split("/", 1)[0] || "openrouter";
+    // OpenRouter model IDs are vendor/model (for example openai/gpt-4o-mini).
+    // Direct providers require an explicit opt-in to avoid routing those IDs
+    // away from OpenRouter when users have multiple credentials configured.
+    if (id === "openai" && process.env.OPENMERIT_DIRECT_PROVIDER === "openai" && process.env.OPENAI_API_KEY)
+        return new OpenAICompatibleProvider(id, "https://api.openai.com/v1", process.env.OPENAI_API_KEY);
+    if (id === "google" && process.env.OPENMERIT_DIRECT_PROVIDER === "google" && process.env.GEMINI_API_KEY)
+        return new GeminiProvider(process.env.GEMINI_API_KEY);
+    return new OpenAICompatibleProvider("openrouter", OR, fallbackKey);
+}
 export async function chat(key, model, prompt, maxTokens, temperature) {
-    const r = await request(key, "/chat/completions", {
-        model,
-        messages: [{ role: "user", content: prompt }],
-        max_tokens: maxTokens,
-        temperature,
-    });
-    return { content: r.choices[0]?.message?.content ?? "", usage: r.usage ?? {} };
+    return providerFor(model, key).chat(model, prompt, maxTokens, temperature);
 }
 /** Send a visual grading request through the same OpenRouter account. */
 export async function chatWithImages(key, model, prompt, images, maxTokens) {
-    const r = await request(key, "/chat/completions", {
-        model,
-        messages: [{ role: "user", content: [
-                    { type: "text", text: prompt },
-                    ...images.map((image) => ({ type: "image_url",
-                        image_url: { url: `data:${image.mimeType};base64,${image.data}` } })),
-                ] }],
-        max_tokens: maxTokens,
-        temperature: 0,
-    });
-    return { content: r.choices[0]?.message?.content ?? "", usage: r.usage ?? {} };
+    return providerFor(model, key).chatWithImages(model, prompt, images, maxTokens);
 }

package/dist/pi-trials.js

@@ -1,12 +1,13 @@
 /** Run a task through pi for each model, using pi's JSON event stream as the measurement source. */
 import { spawn, spawnSync } from "node:child_process";
-import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { cpSync, existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
 import { tmpdir } from "node:os";
-import { join } from "node:path";
+import { dirname, join, relative } from "node:path";
+import { createHash } from "node:crypto";
 import { judge, judgeWithImages } from "./judge.js";
 import { knownInvoiceScore } from "./invoice-eval.js";
 import { paths, readJson } from "./store.js";
-import { sessionImages, taskInputKey } from "./task-input.js";
+import { sessionFiles, sessionImages, taskInputKey } from "./task-input.js";
 function answerText(content) {
     if (typeof content === "string")
         return content;
@@ -15,6 +16,59 @@ function answerText(content) {
     return content.filter((x) => !!x && typeof x === "object" && x.type === "text")
         .map((x) => x.text ?? "").join("\n");
 }
+/** Pi-backed harness adapter. Future harnesses can implement the same seam. */
+export class PiHarnessAdapter {
+    runTask(model, input) {
+        return executePiTask(model, input.task, input.cwd, input.images, input.files);
+    }
+}
+const defaultHarness = new PiHarnessAdapter();
+const KNOWN_TRIAL_TOOLS = new Set([
+    "read", "grep", "find", "ls", "bash", "powershell", "edit", "write",
+]);
+/**
+ * Candidate runs have no tools by default. Users may explicitly opt into a
+ * list, but a copied cwd is not an OS sandbox for absolute paths or network.
+ */
+export function piTrialToolArgs(setting = process.env.OPENMERIT_PI_TRIAL_TOOLS) {
+    if (!setting?.trim())
+        return ["--no-tools"];
+    const requested = [...new Set(setting.split(",").map((tool) => tool.trim()).filter(Boolean))];
+    if (requested.length === 1 && requested[0] === "none")
+        return ["--no-tools"];
+    const unknown = requested.filter((tool) => !KNOWN_TRIAL_TOOLS.has(tool));
+    if (requested.length === 0 || unknown.length > 0) {
+        throw new Error(`invalid OPENMERIT_PI_TRIAL_TOOLS${unknown.length ? `: ${unknown.join(", ")}` : ""}`);
+    }
+    return ["--tools", requested.join(",")];
+}
+function fileSnapshot(root) {
+    const out = new Map();
+    function walk(dir) {
+        for (const name of readdirSync(dir)) {
+            if (name === ".git" || name === "node_modules" || name === ".openmerit")
+                continue;
+            const file = join(dir, name);
+            const rel = relative(root, file);
+            const st = statSync(file);
+            if (st.isDirectory())
+                walk(file);
+            else if (st.isFile() && st.size < 20_000_000) {
+                out.set(rel, createHash("sha1").update(readFileSync(file)).digest("hex"));
+            }
+        }
+    }
+    walk(root);
+    return out;
+}
+function isolatedWorkspace(cwd) {
+    const root = mkdtempSync(join(tmpdir(), "openmerit-pi-workspace-"));
+    cpSync(cwd, root, { recursive: true, filter: (src) => {
+            const rel = relative(cwd, src);
+            return !rel.split("/").some((part) => part === ".git" || part === "node_modules" || part === ".openmerit");
+        } });
+    return root;
+}
 /** Use model A's completed result from the active pi session when it matches this task. */
 export function recordedActiveTask(task, model, images = [], sessionFile, sessionBytes) {
     const state = sessionFile ? null : readJson(paths.harnessState(), {});
@@ -23,6 +77,8 @@ export function recordedActiveTask(task, model, images = [], sessionFile, sessio
         return null;
     let user = null;
     const assistants = [];
+    let toolCalls = 0;
+    let toolErrors = 0;
     for (const line of readFileSync(file).subarray(0, sessionBytes).toString("utf8").split("\n")) {
         if (!line.trim())
             continue;
@@ -38,9 +94,17 @@ export function recordedActiveTask(task, model, images = [], sessionFile, sessio
         if (entry.message.role === "user") {
             user = entry.message;
             assistants.length = 0;
+            toolCalls = 0;
+            toolErrors = 0;
         }
-        else if (user && entry.message.role === "assistant")
+        else if (user && entry.message.role === "assistant") {
             assistants.push(entry.message);
+            if (Array.isArray(entry.message.content))
+                toolCalls += entry.message.content.filter((part) => part?.type === "toolCall").length;
+        }
+        else if (user && entry.message.role === "toolResult" &&
+            entry.message.isError)
+            toolErrors++;
     }
     const userImages = user ? sessionImages(user.content) : null;
     if (!user || !userImages || taskInputKey(answerText(user.content), userImages) !==
@@ -59,6 +123,9 @@ export function recordedActiveTask(task, model, images = [], sessionFile, sessio
         costUsd: matching.reduce((n, m) => n + (m.usage?.cost?.total ?? 0), 0),
         latencyMs: Math.max(0, last - first),
         errors: 0,
+        toolCalls,
+        toolErrors,
+        changedFiles: 0,
     };
 }
 /** Read the exact latest completed text task from the pi extension's session marker. */
@@ -91,19 +158,20 @@ export function settledActiveTask(snapshot) {
             Array.isArray(entry.message.content) && entry.message.content.some((b) => b?.type === "toolCall"))
             usedTools = true;
     }
-    if (!userContent || usedTools)
+    if (!userContent)
         return null;
     const images = sessionImages(userContent);
     if (!images)
         return null;
     const task = answerText(userContent);
-    if (!task.trim() || taskInputKey(task, images) !== st.settledTaskKey)
+    const files = sessionFiles(task);
+    if (!task.trim() || taskInputKey(task, images, files) !== st.settledTaskKey)
         return null;
     const run = recordedActiveTask(task, st.currentModel, images, st.sessionFile, st.sessionBytes);
     if (!run)
         return null;
     return { task, images, model: st.currentModel, run, sessionFile: st.sessionFile,
-        settledAt: st.settledAt, sessionBytes: st.sessionBytes, cwd: st.cwd ?? process.cwd() };
+        settledAt: st.settledAt, sessionBytes: st.sessionBytes, cwd: st.cwd ?? process.cwd(), usedTools, files };
 }
 /** Limit candidates to models this installed pi can actually select. */
 export function availablePiModels() {
@@ -115,7 +183,7 @@ export function availablePiModels() {
         .filter((id) => !!id && !id.startsWith("~")));
 }
 /** Each invocation is a fresh pi run, so model B does not inherit model A's answer. */
-export async function executePiTask(model, task, cwd = process.cwd(), images = []) {
+export async function executePiTask(model, task, cwd = process.cwd(), images = [], files = []) {
     const imageDir = images.length ? mkdtempSync(join(tmpdir(), "openmerit-pi-image-")) : null;
     const suffix = {
         "image/jpeg": "jpg", "image/png": "png", "image/webp": "webp", "image/gif": "gif",
@@ -127,73 +195,109 @@ export async function executePiTask(model, task, cwd = process.cwd(), images = [
         writeFileSync(file, Buffer.from(image.data, "base64"));
         return `@${file}`;
     });
-    const child = spawn("pi", [
-        "--provider", "openrouter", "--model", model, "--mode", "json",
-        "--offline", "--no-extensions", "--no-tools", "--print", ...imageArgs, task,
-    ], { cwd, stdio: ["ignore", "pipe", "pipe"] });
-    let buffer = "";
-    let stderr = "";
-    let answer = "";
-    let costUsd = 0;
-    let errors = 0;
-    let sessionId;
-    const started = Date.now();
-    function consume(line) {
-        if (!line.trim())
-            return;
-        let event;
-        try {
-            event = JSON.parse(line);
-        }
-        catch {
-            return;
-        }
-        if (event.type === "session" && event.id)
-            sessionId = event.id;
-        if (event.type !== "message_end" || event.message?.role !== "assistant")
-            return;
-        answer = answerText(event.message.content) || answer;
-        costUsd += event.message.usage?.cost?.total ?? 0;
-        if (event.message.stopReason === "error")
-            errors++;
-    }
-    child.stdout.on("data", (chunk) => {
-        buffer += chunk.toString("utf8");
-        let n;
-        while ((n = buffer.indexOf("\n")) >= 0) {
-            consume(buffer.slice(0, n));
-            buffer = buffer.slice(n + 1);
-        }
-    });
-    child.stderr.on("data", (chunk) => { stderr += chunk.toString("utf8"); });
-    let exitCode;
+    const workspace = isolatedWorkspace(cwd);
     try {
-        exitCode = await new Promise((resolve, reject) => {
+        const attachmentDir = join(workspace, ".openmerit-attachments");
+        mkdirSync(attachmentDir, { recursive: true });
+        const attachmentPaths = new Map();
+        const fileArgs = files.map((file, i) => {
+            const safeName = `${String(i).padStart(3, "0")}-${file.name.replace(/[^A-Za-z0-9._-]/g, "_")}`;
+            const target = join(attachmentDir, safeName);
+            cpSync(file.path, target);
+            attachmentPaths.set(file.path, target);
+            return `@${target}`;
+        });
+        const candidateTask = [...attachmentPaths.entries()].reduce((text, [source, target]) => text.split(source).join(target), task);
+        const before = fileSnapshot(workspace);
+        const traceFile = paths.trialTrace(`${model}:${task}:${Date.now()}`);
+        const traceLines = [];
+        const child = spawn("pi", [
+            "--provider", "openrouter", "--model", model, "--mode", "json",
+            "--offline", "--no-extensions", "--approve", ...piTrialToolArgs(),
+            "--print", ...imageArgs, ...fileArgs, candidateTask,
+        ], { cwd: workspace, stdio: ["ignore", "pipe", "pipe"] });
+        let buffer = "";
+        let stderr = "";
+        let answer = "";
+        let costUsd = 0;
+        let errors = 0;
+        let sessionId;
+        let toolCalls = 0;
+        let toolErrors = 0;
+        const started = Date.now();
+        function consume(line) {
+            if (!line.trim())
+                return;
+            traceLines.push(line);
+            let event;
+            try {
+                event = JSON.parse(line);
+            }
+            catch {
+                return;
+            }
+            if (event.type === "session" && event.id)
+                sessionId = event.id;
+            const content = event.message?.content;
+            if (event.message?.role === "assistant" && Array.isArray(content))
+                toolCalls += content.filter((part) => part?.type === "toolCall").length;
+            if (event.message?.role === "toolResult" && event.message.isError)
+                toolErrors++;
+            if (event.type !== "message_end" || event.message?.role !== "assistant")
+                return;
+            answer = answerText(event.message.content) || answer;
+            costUsd += event.message.usage?.cost?.total ?? 0;
+            if (event.message.stopReason === "error")
+                errors++;
+        }
+        child.stdout.on("data", (chunk) => {
+            buffer += chunk.toString("utf8");
+            let n;
+            while ((n = buffer.indexOf("\n")) >= 0) {
+                consume(buffer.slice(0, n));
+                buffer = buffer.slice(n + 1);
+            }
+        });
+        child.stderr.on("data", (chunk) => { stderr += chunk.toString("utf8"); });
+        const exitCode = await new Promise((resolve, reject) => {
             child.on("error", reject);
             child.on("close", (code) => resolve(code ?? 1));
         });
+        consume(buffer);
+        mkdirSync(dirname(traceFile), { recursive: true });
+        writeFileSync(traceFile, traceLines.join("\n") + (traceLines.length ? "\n" : ""));
+        const after = fileSnapshot(workspace);
+        let changedFiles = 0;
+        for (const [file, hash] of after)
+            if (before.get(file) !== hash)
+                changedFiles++;
+        for (const file of before.keys())
+            if (!after.has(file))
+                changedFiles++;
+        if (exitCode !== 0 || !answer.trim()) {
+            throw new Error(`pi trial ${model} failed: ${stderr.trim().slice(0, 180) || `exit ${exitCode}, empty answer`}`);
+        }
+        return { answer, costUsd, latencyMs: Date.now() - started, errors, sessionId,
+            toolCalls, toolErrors, changedFiles, traceFile };
     }
     finally {
         if (imageDir)
             rmSync(imageDir, { recursive: true, force: true });
+        rmSync(workspace, { recursive: true, force: true });
     }
-    consume(buffer);
-    if (exitCode !== 0 || !answer.trim()) {
-        throw new Error(`pi trial ${model} failed: ${stderr.trim().slice(0, 180) || `exit ${exitCode}, empty answer`}`);
-    }
-    return { answer, costUsd, latencyMs: Date.now() - started, errors, sessionId };
 }
 /** Quality uses the same OpenMerit judge; cost and latency come from pi itself. */
-export async function runPiTrial(key, judgeModel, task, rubric, model, entry, cwd = process.cwd(), images = []) {
+export async function runPiTrial(key, judgeModel, task, rubric, model, entry, cwd = process.cwd(), images = [], files = [], harness = defaultHarness) {
     try {
-        const run = await executePiTask(model, task, cwd, images);
+        const run = await harness.runTask(model, { task, cwd, images, files });
         return await scorePiRun(key, judgeModel, task, rubric, model, entry, run, "pi_trial", images);
     }
     catch (e) {
         const error = String(e);
         return {
             point: { model, score: 0, price: entry?.price ?? 0,
-                ts: new Date().toISOString(), source: "pi_trial", why: error.slice(0, 160) },
+                ts: new Date().toISOString(), source: "pi_trial", toolCalls: 0, toolErrors: 1,
+                changedFiles: 0, why: error.slice(0, 160) },
             costUsd: 0, error,
         };
     }
@@ -216,7 +320,9 @@ export async function scorePiRun(key, judgeModel, task, rubric, model, entry, ru
         point: {
             model, score, price: entry?.price ?? 0,
             latencyMs: run.latencyMs, ts: new Date().toISOString(), source,
-            why: `${why}${run.errors ? `; pi errors: ${run.errors}` : ""}`,
+            why: `${why}${run.errors ? `; pi errors: ${run.errors}` : ""}; ` +
+                `tools=${run.toolCalls}, toolErrors=${run.toolErrors}, changedFiles=${run.changedFiles}`,
+            toolCalls: run.toolCalls, toolErrors: run.toolErrors, changedFiles: run.changedFiles,
         },
         costUsd: run.costUsd,
         sessionId: run.sessionId,

package/dist/providers.js

@@ -0,0 +1 @@
+export {};

package/dist/store.js

@@ -22,6 +22,7 @@ export const paths = {
     ledger: () => join(stateDir(), "ledger.json"),
     watchProcessed: () => join(stateDir(), "watch", "processed.json"),
     sessionJob: (marker) => join(stateDir(), "watch", "jobs", sha1(marker) + ".json"),
+    trialTrace: (id) => join(stateDir(), "traces", "trials", `${sha1(id)}.jsonl`),
     envFile: () => join(stateDir(), ".env"),
 };
 export function sha1(text) {

package/dist/task-input.js

@@ -1,6 +1,29 @@
 /** A pi text task may carry images embedded in its saved session message. */
 import { createHash } from "node:crypto";
+import { existsSync, readFileSync, statSync } from "node:fs";
+import { basename } from "node:path";
 import { taskKey } from "./store.js";
+/** Recover files Pi expanded into a task message (for example @invoice.pdf). */
+export function sessionFiles(text) {
+    const files = [];
+    const seen = new Set();
+    const re = /<file\s+name=["']([^"']+)["'][^>]*>/g;
+    for (const match of text.matchAll(re)) {
+        const file = match[1];
+        if (!file || seen.has(file) || !existsSync(file))
+            continue;
+        try {
+            const stat = statSync(file);
+            if (!stat.isFile() || stat.size > 100_000_000)
+                continue;
+            files.push({ path: file, name: basename(file), size: stat.size,
+                sha256: createHash("sha256").update(readFileSync(file)).digest("hex") });
+            seen.add(file);
+        }
+        catch { /* inaccessible attachment */ }
+    }
+    return files;
+}
 const SUPPORTED = new Set(["image/jpeg", "image/png", "image/webp", "image/gif"]);
 export function sessionImages(content) {
     if (!Array.isArray(content))
@@ -20,11 +43,12 @@ export function sessionImages(content) {
     return images;
 }
 /** Text-only keys remain compatible; image bytes distinguish same-prompt documents. */
-export function taskInputKey(text, images) {
-    if (images.length === 0)
+export function taskInputKey(text, images, files = []) {
+    if (images.length === 0 && files.length === 0)
         return taskKey(text);
     const normalized = text.toLowerCase().replace(/\s+/g, " ").trim();
     const hashes = images.map((image) => `${image.mimeType}:` +
         createHash("sha256").update(Buffer.from(image.data, "base64")).digest("hex"));
-    return taskKey(`${normalized}\nimages:${hashes.join(",")}`);
+    const fileHashes = files.map((file) => `${file.name}:${file.sha256}`).join(",");
+    return taskKey(`${normalized}\nimages:${hashes.join(",")}\nfiles:${fileHashes}`);
 }

package/extension/openmerit.ts

@@ -19,7 +19,7 @@
 import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
 import { appendFileSync, existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from "node:fs";
 import { homedir } from "node:os";
-import { join } from "node:path";
+import { basename, join } from "node:path";
 import { createHash } from "node:crypto";
 import { spawn, type ChildProcessByStdio } from "node:child_process";
 import type { Readable } from "node:stream";
@@ -46,6 +46,9 @@ interface TrialProgress {
   latencyMs?: number;
   costUsd?: number;
   error?: string;
+  toolCalls?: number;
+  toolErrors?: number;
+  changedFiles?: number;
 }
 
 function parseTrialProgress(line: string): TrialProgress | null {
@@ -63,7 +66,9 @@ function parseTrialProgress(line: string): TrialProgress | null {
 function completedTrialText(trial: TrialProgress): string {
   return `${trial.model}: quality ${(trial.score ?? 0).toFixed(2)}, ` +
     `$${(trial.price ?? 0).toFixed(2)}/M, ${Math.round(trial.latencyMs ?? 0)}ms, ` +
-    `run $${(trial.costUsd ?? 0).toFixed(4)}` + (trial.error ? ` (failed: ${trial.error})` : "");
+    `run $${(trial.costUsd ?? 0).toFixed(4)}, tools ${trial.toolCalls ?? 0}` +
+    (trial.toolErrors || trial.changedFiles ? `, tool errors ${trial.toolErrors ?? 0}, changed files ${trial.changedFiles ?? 0}` : "") +
+    (trial.error ? ` (failed: ${trial.error})` : "");
 }
 
 interface ExtensionPolicy {
@@ -171,10 +176,18 @@ function userTaskKey(content: unknown): string | null {
   const images = parts.filter((b) => b?.type === "image");
   if (parts.length !== parts.filter((b) => b?.type === "text" || b?.type === "image").length ||
       images.some((b) => typeof b.data !== "string" || typeof b.mimeType !== "string")) return null;
-  if (images.length === 0) return taskKey(text);
+  const files: string[] = [];
+  const fileRe = /<file\s+name=["']([^"']+)["'][^>]*>/g;
+  for (const match of text.matchAll(fileRe)) {
+    const file = match[1];
+    if (!file || !existsSync(file) || files.includes(file)) continue;
+    try { files.push(`${basename(file)}:` + createHash("sha256").update(readFileSync(file)).digest("hex")); }
+    catch { /* inaccessible attachment */ }
+  }
+  if (images.length === 0 && files.length === 0) return taskKey(text);
   const hashes = images.map((b) => `${b.mimeType}:` +
     createHash("sha256").update(Buffer.from(b.data, "base64")).digest("hex"));
-  return taskKey(`${text.toLowerCase().replace(/\s+/g, " ").trim()}\nimages:${hashes.join(",")}`);
+  return taskKey(`${text.toLowerCase().replace(/\s+/g, " ").trim()}\nimages:${hashes.join(",")}\nfiles:${files.join(",")}`);
 }
 
 function latestSessionTaskKey(ctx: ExtensionContext): string | null {

package/instructions/OPENMERIT.md

@@ -8,8 +8,9 @@ the best model for the task at hand, at the best price, with a vetted fallback.
 
 1. **Observes** this harness's session traces (models used, tokens, cost,
    latency, errors) without intercepting or slowing down your work.
-2. **Evaluates** candidate models for each completed text or image task sequentially
-   through pi and scores their outputs with a judge model.
+2. **Evaluates** candidate models for each completed task sequentially through
+   pi and scores their outputs with a judge model. Candidate tools are disabled
+   by default.
 3. **Maintains a pareto frontier** per task (quality vs. cost vs. latency) and
    an aggregate frontier across all of your tasks.
 4. **Watches for new model releases** (provider catalogs + public benchmarks)
@@ -43,9 +44,11 @@ the best model for the task at hand, at the best price, with a vetted fallback.
 
 ## Guarantees
 
-- Per-task comparisons send the task text, uploaded images (when present), and
-  candidate answers through pi/OpenRouter for model runs and judging. Use
-  non-sensitive examples while evaluating this alpha.
+- Per-task comparisons send the task text, uploaded files or images (when
+  present), and candidate answers through pi/OpenRouter for model runs and
+  judging. Candidate Pi runs default to no tools; any tool access is an
+  explicit user opt-in. Use non-sensitive examples while
+  evaluating this alpha.
 - The shipped policy is supervised. Swaps happen automatically only after the
   user opts in and the configured quality/cost guardrails pass; otherwise they
   remain recommendations for a human to approve.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openmerit",
-  "version": "0.1.0",
+  "version": "0.1.1",
   "description": "Find better models for each pi task by comparing quality, cost, and latency.",
   "type": "module",
   "keywords": [