agent-trials 1.0.0 → 1.0.1

package/README.md

@@ -2,7 +2,7 @@
 
 Interview your coding agent before you ship with it.
 
-`agent-trials` runs the same task suite through every coding agent CLI you have installed and prints a scoreboard: who passed, how long they took, what they spent. Every task runs in a throwaway copy of your repo, so agents can do their worst and your working tree stays clean.
+`agent-trials` runs the same task suite through every coding agent CLI you have installed and prints a scoreboard: who passed, how long they took, what they spent. Each task gets a temporary working copy. Commands still run on your host.
 
 ![demo](demo.png)
 
@@ -10,6 +10,10 @@ Interview your coding agent before you ship with it.
 
 Public benchmarks score agents on tasks someone else picked. Your repo is not that benchmark. The question that matters before you let an agent loose on a codebase: does it fix YOUR bugs, in YOUR stack, at a sane price? Write the tasks once, run every candidate, keep the receipts.
 
+## execution safety
+
+A temporary working copy is not a security sandbox. Agents and suite commands run with your host permissions and environment; built-in Claude and Gemini commands disable approval prompts. Repository dotfiles and secrets may be copied. Use only trusted suites in a disposable VM with no sensitive files or credentials. Do not run this on a workstation containing private data.
+
 ## install
 
 ```sh
@@ -31,6 +35,7 @@ agent-trials run            # run the suite, print the board, write TRIALS.md
 ```json
 {
   "verify": "npm test",
+  "verifyPaths": ["verify"],
   "timeoutSec": 600,
   "tasks": [
     {
@@ -47,10 +52,13 @@ agent-trials run            # run the suite, print the board, write TRIALS.md
 
 Each cell (agent x task) works like this:
 
-1. your repo, minus `node_modules` and `.git`, is copied to a temp sandbox
+1. your repo, minus `node_modules` and `.git`, is copied to a temporary working copy
 2. optional `setup` command runs once (`npm ci` and friends)
-3. the agent runs headless on the task prompt, permission checks off, timeout-guarded
-4. your `verify` command runs in the sandbox; exit 0 means pass
+3. the agent runs headless on the task prompt, using the selected CLI flags, timeout-guarded
+4. checker paths are restored from your repo, so the agent cannot rewrite its own test
+5. your `verify` command runs in the working copy; exit 0 means pass
+
+Step 4 restores `verifyPaths` from the suite, or a top-level `verify/` dir when `verifyPaths` is not set. A verify command that reads mutable state outside those paths can still be gamed; put your checkers in `verify/` or list them.
 
 Builtin agents: `claude`, `codex`, `gemini`. Custom agents are one argv array away: anything installed, any flags, `{prompt}` gets replaced with the task prompt.
 
@@ -73,11 +81,13 @@ cost             $1.11       $1.10         $2.06
 winner: haiku (3/3, median 60.9s)
 ```
 
+Read this board with caveats. `opus`, `sonnet`, and `haiku` are requested CLI aliases; that run predates model-identity capture (`--json` now records `modelUsed` per cell when the agent CLI reports it). One run of three toy tasks is a smoke test, not a comparative benchmark. Costs are CLI-reported, not invoice costs.
+
 What the board caught:
 
-- haiku, the cheap one, went 3/3 with the fastest median. It also spent the most: $2.06, because it ran more turns. Pass rate and price do not move together.
-- an earlier run caught the agent asking a clarifying question ("should -0 count as zero?") into an empty terminal, then exiting 0 having changed nothing. Exit code means nothing; your verify command is the ground truth.
-- sonnet lost slug-test outright. The task was write a test file that runs clean under plain node; it didn't.
+- haiku went 3/3 with the fastest median while spending the most: $2.06. Pass rate and price did not move together.
+- in one earlier run, the agent asked a clarifying question ("should -0 count as zero?") into an empty terminal, then exited 0 having changed nothing. One observed example, not a general claim. This showed why exit code alone cannot be the verdict; the verify command is what counts.
+- sonnet failed slug-test.
 
 Full results with per-cell costs: [LEADERBOARD.md](LEADERBOARD.md)
 
@@ -86,7 +96,7 @@ Full results with per-cell costs: [LEADERBOARD.md](LEADERBOARD.md)
 - pass/fail is your verify command's exit code. The tool cannot judge taste, only outcomes.
 - runs spend your agent quota and your auth; costs are reported when the agent's JSON output includes them (claude does)
 - agent CLIs must be installed and logged in
-- macOS/Linux; the sandbox copy and verify assume `sh`. Windows untested
+- macOS/Linux; the working copy and verify assume `sh`. Windows untested
 - scoring is binary per task. No partial credit for "almost fixed it"
 
 ## flags
@@ -99,7 +109,7 @@ Full results with per-cell costs: [LEADERBOARD.md](LEADERBOARD.md)
 | `--jobs <n>` | `2` | parallel agent x task cells |
 | `--out <file>` | `TRIALS.md` | markdown report path, `none` skips |
 | `--json <file>` | - | raw per-cell results |
-| `--keep` | off | keep sandboxes under /tmp for debugging |
+| `--keep` | off | keep working copies under /tmp for debugging |
 
 ## why the suite format is the point
 

package/dist/cli.js

@@ -2,7 +2,7 @@
 import { existsSync, writeFileSync } from "node:fs";
 import path from "node:path";
 import { agentInstalled, BUILTIN_AGENTS, loadSuite, renderBoard, renderMarkdown, resolveAgents, runSuite, writeMarkdown, verdictOf, } from "./trials.js";
-const VERSION = "1.0.0";
+const VERSION = "1.0.1";
 const HELP = `agent-trials ${VERSION}
 Interview your coding agent before you ship with it.
 
@@ -18,7 +18,7 @@ run flags:
   --jobs <n>        parallel cells (default: 2)
   --out <file>      markdown report path (default: TRIALS.md, "none" skips)
   --json <file>     also write raw results JSON
-  --keep            keep sandbox dirs for debugging
+  --keep            keep working copies for debugging
 `;
 const STARTER_SUITE = {
     name: "my repo trials",

package/dist/trials.js

@@ -22,6 +22,9 @@ export function loadSuite(file) {
         if (!t.verify && !raw.verify)
             throw new Error(`${file}: task "${t.id}" has no verify and suite has no default`);
     }
+    if (raw.verifyPaths !== undefined && !Array.isArray(raw.verifyPaths)) {
+        throw new Error(`${file}: "verifyPaths" must be an array of repo-relative paths`);
+    }
     return raw;
 }
 export function resolveAgents(names, suite) {
@@ -83,12 +86,28 @@ function parseAgentJson(stdout) {
     try {
         const j = JSON.parse(stdout.trim());
         const cost = j.total_cost_usd ?? j.cost_usd ?? j.stats?.total_cost_usd;
-        return typeof cost === "number" ? { costUsd: cost } : {};
+        const model = j.model ?? Object.keys(j.modelUsage ?? {})[0];
+        const out = {};
+        if (typeof cost === "number")
+            out.costUsd = cost;
+        if (typeof model === "string" && model)
+            out.modelUsed = model;
+        return out;
     }
     catch {
         return {};
     }
 }
+// The checker files are restored from the source repo after the agent runs,
+// so an agent cannot rewrite its own test. The suite marks them with
+// verifyPaths; a top-level verify/ dir is the convention fallback.
+function protectedPaths(root, suite) {
+    const rels = suite.verifyPaths ?? (existsSync(path.join(root, "verify")) ? ["verify"] : []);
+    return rels.filter((rel) => !path.isAbsolute(rel) &&
+        !rel.split(/[\\/]+/).includes("..") &&
+        !SKIP_DIRS.has(rel.split(/[\\/]+/)[0]) &&
+        existsSync(path.join(root, rel)));
+}
 export async function runSuite(opts) {
     const { root, suite, agents, jobs, keep, onCell } = opts;
     const timeoutMs = (suite.timeoutSec ?? 600) * 1000;
@@ -131,6 +150,12 @@ export async function runSuite(opts) {
                     out.error = agentRes.timedOut ? "timeout" : `spawn failed: ${agentRes.stderr.slice(-160)}`;
                 }
                 const verifyCmd = task.verify ?? suite.verify;
+                const restored = protectedPaths(root, suite);
+                for (const rel of restored) {
+                    cpSync(path.join(root, rel), path.join(sandbox, rel), { recursive: true });
+                }
+                if (restored.length)
+                    out.verifyRestored = restored;
                 const v = await runCmd(["sh", "-c", verifyCmd], sandbox, timeoutMs);
                 out.verifyExit = v.code;
                 out.pass = out.agentExit === 0 && v.code === 0;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-trials",
-  "version": "1.0.0",
+  "version": "1.0.1",
   "description": "Interview your coding agent before you ship with it. Run the same tasks through every agent CLI you have installed and get a scoreboard.",
   "type": "module",
   "bin": {