agent-trials 0.0.0-stage → 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Prince Jain
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,108 @@
-# Temporary Holding Version
+# agent-trials
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Interview your coding agent before you ship with it.
+
+`agent-trials` runs the same task suite through every coding agent CLI you have installed and prints a scoreboard: who passed, how long they took, what they spent. Every task runs in a throwaway copy of your repo, so agents can do their worst and your working tree stays clean.
+
+![demo](demo.png)
+
+## why
+
+Public benchmarks score agents on tasks someone else picked. Your repo is not that benchmark. The question that matters before you let an agent loose on a codebase: does it fix YOUR bugs, in YOUR stack, at a sane price? Write the tasks once, run every candidate, keep the receipts.
+
+## install
+
+```sh
+npx agent-trials init        # writes trials.json in your repo
+```
+
+Works straight from GitHub too: `npx github:princejain756/agent-trials`.
+
+## use
+
+```sh
+agent-trials init           # starter suite
+agent-trials doctor         # which agent CLIs are installed
+agent-trials run            # run the suite, print the board, write TRIALS.md
+```
+
+## trials.json
+
+```json
+{
+  "verify": "npm test",
+  "timeoutSec": 600,
+  "tasks": [
+    {
+      "id": "fix-cache",
+      "prompt": "The cache in src/cache.js never expires entries. Add a ttl option. Keep the API.",
+      "verify": "node checks/cache-ttl.mjs"
+    }
+  ],
+  "agents": {
+    "sonnet": { "run": ["claude", "-p", "{prompt}", "--dangerously-skip-permissions", "--model", "sonnet"] }
+  }
+}
+```
+
+Each cell (agent x task) works like this:
+
+1. your repo, minus `node_modules` and `.git`, is copied to a temp sandbox
+2. optional `setup` command runs once (`npm ci` and friends)
+3. the agent runs headless on the task prompt, permission checks off, timeout-guarded
+4. your `verify` command runs in the sandbox; exit 0 means pass
+
+Builtin agents: `claude`, `codex`, `gemini`. Custom agents are one argv array away: anything installed, any flags, `{prompt}` gets replaced with the task prompt.
+
+## a real run
+
+`examples/tinylib` is a 3-function library with planted bugs. Same 3 tasks, 3 model variants of the same agent CLI (`claude --model`):
+
+```
+agent-trials · 3 tasks · 3 agents
+
+                  opus      sonnet       haiku
+fix-clamp    pass 141.3s  pass 44.2s   pass 60.9s
+divide-guard  fail 96.1s  pass 72.1s  pass 118.6s
+slug-test    pass 253.4s  fail 124.4s    pass 50s
+
+score              2/3         2/3           3/3
+median          141.3s       72.1s         60.9s
+cost             $1.11       $1.10         $2.06
+
+winner: haiku (3/3, median 60.9s)
+```
+
+What the board caught:
+
+- haiku, the cheap one, went 3/3 with the fastest median. It also spent the most: $2.06, because it ran more turns. Pass rate and price do not move together.
+- an earlier run caught the agent asking a clarifying question ("should -0 count as zero?") into an empty terminal, then exiting 0 having changed nothing. Exit code means nothing; your verify command is the ground truth.
+- sonnet lost slug-test outright. The task was write a test file that runs clean under plain node; it didn't.
+
+Full results with per-cell costs: [LEADERBOARD.md](LEADERBOARD.md)
+
+## limits, honestly
+
+- pass/fail is your verify command's exit code. The tool cannot judge taste, only outcomes.
+- runs spend your agent quota and your auth; costs are reported when the agent's JSON output includes them (claude does)
+- agent CLIs must be installed and logged in
+- macOS/Linux; the sandbox copy and verify assume `sh`. Windows untested
+- scoring is binary per task. No partial credit for "almost fixed it"
+
+## flags
+
+| flag | default | what it does |
+|---|---|---|
+| `--suite <file>` | `trials.json` | suite file to run |
+| `--agents a,b` | all known | restrict the agent list |
+| `--root <dir>` | `.` | the repo being tested |
+| `--jobs <n>` | `2` | parallel agent x task cells |
+| `--out <file>` | `TRIALS.md` | markdown report path, `none` skips |
+| `--json <file>` | - | raw per-cell results |
+| `--keep` | off | keep sandboxes under /tmp for debugging |
+
+## why the suite format is the point
+
+Tasks are plain JSON and `verify` is plain shell, so a suite doubles as a smoke-test pack for humans. Commit `trials.json` next to the code it describes. When you swap agents or bump model versions, rerun and diff the board.
+
+MIT

package/dist/cli.js

@@ -0,0 +1,144 @@
+#!/usr/bin/env node
+import { existsSync, writeFileSync } from "node:fs";
+import path from "node:path";
+import { agentInstalled, BUILTIN_AGENTS, loadSuite, renderBoard, renderMarkdown, resolveAgents, runSuite, writeMarkdown, verdictOf, } from "./trials.js";
+const VERSION = "1.0.0";
+const HELP = `agent-trials ${VERSION}
+Interview your coding agent before you ship with it.
+
+usage:
+  agent-trials init                 write a starter trials.json
+  agent-trials doctor               check which agent CLIs are installed
+  agent-trials run                  run the suite and print the scoreboard
+
+run flags:
+  --suite <file>    suite file (default: trials.json)
+  --agents a,b      only these agents (default: all known + custom)
+  --root <dir>      repo to run on (default: cwd)
+  --jobs <n>        parallel cells (default: 2)
+  --out <file>      markdown report path (default: TRIALS.md, "none" skips)
+  --json <file>     also write raw results JSON
+  --keep            keep sandbox dirs for debugging
+`;
+const STARTER_SUITE = {
+    name: "my repo trials",
+    setup: "",
+    verify: "npm test",
+    timeoutSec: 600,
+    tasks: [
+        {
+            id: "example-task",
+            prompt: "Describe what this repo does in one sentence, then make the smallest possible improvement to the README.",
+            verify: "git diff --quiet || true",
+        },
+    ],
+    agents: {},
+};
+function parseArgs(argv) {
+    const out = {};
+    for (let i = 0; i < argv.length; i++) {
+        const a = argv[i];
+        if (a.startsWith("--")) {
+            const key = a.slice(2);
+            if (i + 1 < argv.length && !argv[i + 1].startsWith("--")) {
+                out[key] = argv[++i];
+            }
+            else {
+                out[key] = true;
+            }
+        }
+    }
+    return out;
+}
+function fail(msg) {
+    console.error(`agent-trials: ${msg}`);
+    process.exit(1);
+}
+async function main() {
+    const [cmd, ...rest] = process.argv.slice(2);
+    const flags = parseArgs(rest);
+    if (!cmd || flags.help || cmd === "help") {
+        console.log(HELP);
+        return;
+    }
+    if (cmd === "--version" || cmd === "-v" || flags.version) {
+        console.log(VERSION);
+        return;
+    }
+    if (cmd === "init") {
+        const file = typeof flags.suite === "string" ? flags.suite : "trials.json";
+        if (existsSync(file) && !flags.force) {
+            fail(`${file} already exists (use --force to overwrite)`);
+        }
+        writeFileSync(file, JSON.stringify(STARTER_SUITE, null, 2) + "\n");
+        console.log(`wrote ${file}\n\nedit tasks[] with real work for your repo, then:\n  agent-trials run`);
+        return;
+    }
+    if (cmd === "doctor") {
+        const suiteFile = typeof flags.suite === "string" ? flags.suite : "trials.json";
+        const suite = existsSync(suiteFile)
+            ? loadSuite(suiteFile)
+            : { tasks: [] };
+        const agents = resolveAgents(typeof flags.agents === "string" ? flags.agents.split(",") : undefined, suite);
+        console.log("agent-trials doctor\n");
+        for (const [name, spec] of Object.entries(agents)) {
+            const bin = spec.argv[0];
+            const ok = agentInstalled(bin);
+            const src = BUILTIN_AGENTS[name] ? "builtin" : "custom";
+            console.log(`${ok ? "ok  " : "miss"} ${name.padEnd(10)} ${bin}${ok ? "" : "  (not on PATH)"}  [${src}]`);
+        }
+        console.log("\nmissing agents are skipped at run time only if you pass --agents");
+        return;
+    }
+    if (cmd === "run") {
+        const suiteFile = typeof flags.suite === "string" ? flags.suite : "trials.json";
+        if (!existsSync(suiteFile)) {
+            fail(`no ${suiteFile}. run \`agent-trials init\` or pass --suite <file>`);
+        }
+        const suite = loadSuite(suiteFile);
+        const root = path.resolve(typeof flags.root === "string" ? flags.root : ".");
+        const agentNames = typeof flags.agents === "string" ? flags.agents.split(",") : undefined;
+        let agents = resolveAgents(agentNames, suite);
+        const missing = Object.entries(agents)
+            .filter(([, s]) => !agentInstalled(s.argv[0]))
+            .map(([n]) => n);
+        if (missing.length && !agentNames) {
+            for (const n of missing)
+                delete agents[n];
+            console.log(`skipping missing agents: ${missing.join(", ")}`);
+        }
+        const names = Object.keys(agents);
+        if (names.length === 0)
+            fail("no agents available. install claude/codex/gemini or define a custom agent");
+        const jobs = Math.max(1, Number(flags.jobs ?? 2) || 2);
+        const keep = Boolean(flags.keep);
+        const started = Date.now();
+        const results = await runSuite({
+            root,
+            suite,
+            agents,
+            jobs,
+            keep,
+            onCell: (c) => {
+                const verdict = verdictOf(c);
+                console.log(`  [${verdict}] ${c.agent} · ${c.task} · ${c.sec}s`);
+            },
+        });
+        console.log("");
+        const board = renderBoard(suite, results, names);
+        console.log(board);
+        console.log(`\ntotal wall time ${Math.round((Date.now() - started) / 1000)}s`);
+        const out = flags.out === undefined ? "TRIALS.md" : flags.out;
+        if (out !== "none" && typeof out === "string") {
+            writeMarkdown(out, renderMarkdown(suite, results, names));
+            console.log(`wrote ${out}`);
+        }
+        if (typeof flags.json === "string") {
+            writeFileSync(flags.json, JSON.stringify(results, null, 2) + "\n");
+            console.log(`wrote ${flags.json}`);
+        }
+        return;
+    }
+    fail(`unknown command "${cmd}". try agent-trials help`);
+}
+main().catch((err) => fail(err instanceof Error ? err.message : String(err)));

package/dist/trials.js

@@ -0,0 +1,266 @@
+import { spawn, spawnSync } from "node:child_process";
+import { cpSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import path from "node:path";
+export const BUILTIN_AGENTS = {
+    claude: {
+        argv: ["claude", "-p", "{prompt}", "--dangerously-skip-permissions", "--output-format", "json"],
+        resultJson: true,
+    },
+    codex: { argv: ["codex", "exec", "--full-auto", "{prompt}"] },
+    gemini: { argv: ["gemini", "-p", "{prompt}", "--yolo", "-o", "json"], resultJson: true },
+};
+const SKIP_DIRS = new Set(["node_modules", ".git", ".agent-trials"]);
+export function loadSuite(file) {
+    const raw = JSON.parse(readFileSync(file, "utf-8"));
+    if (!Array.isArray(raw.tasks) || raw.tasks.length === 0) {
+        throw new Error(`${file}: needs a non-empty "tasks" array`);
+    }
+    for (const t of raw.tasks) {
+        if (!t.id || !t.prompt)
+            throw new Error(`${file}: every task needs "id" and "prompt"`);
+        if (!t.verify && !raw.verify)
+            throw new Error(`${file}: task "${t.id}" has no verify and suite has no default`);
+    }
+    return raw;
+}
+export function resolveAgents(names, suite) {
+    const custom = suite.agents ?? {};
+    const all = { ...BUILTIN_AGENTS };
+    for (const [name, def] of Object.entries(custom)) {
+        if (!Array.isArray(def.run) || def.run.length === 0) {
+            throw new Error(`agent "${name}": "run" must be an argv array, e.g. ["sh","./agent.sh","{prompt}"]`);
+        }
+        all[name] = { argv: def.run, resultJson: def.resultJson === true };
+    }
+    if (!names || names.length === 0)
+        return all;
+    const picked = {};
+    for (const n of names) {
+        if (!all[n])
+            throw new Error(`unknown agent "${n}". known: ${Object.keys(all).join(", ")}`);
+        picked[n] = all[n];
+    }
+    return picked;
+}
+export function copyWorkdir(root, dest) {
+    cpSync(root, dest, {
+        recursive: true,
+        filter: (src) => {
+            const rel = path.relative(root, src);
+            if (!rel)
+                return true;
+            const first = rel.split(path.sep)[0];
+            return !SKIP_DIRS.has(first);
+        },
+    });
+}
+function runCmd(argv, cwd, timeoutMs) {
+    return new Promise((resolve) => {
+        const child = spawn(argv[0], argv.slice(1), { cwd, env: process.env });
+        let stdout = "";
+        let stderr = "";
+        let timedOut = false;
+        const cap = (s, chunk) => (s.length < 512_000 ? s + chunk.toString() : s);
+        child.stdout.on("data", (c) => (stdout = cap(stdout, c)));
+        child.stderr.on("data", (c) => (stderr = cap(stderr, c)));
+        const killer = setTimeout(() => {
+            timedOut = true;
+            child.kill("SIGTERM");
+            setTimeout(() => child.kill("SIGKILL"), 5000).unref();
+        }, timeoutMs);
+        child.on("error", (err) => {
+            clearTimeout(killer);
+            resolve({ code: null, stdout, stderr: String(err), timedOut });
+        });
+        child.on("close", (code) => {
+            clearTimeout(killer);
+            resolve({ code, stdout, stderr, timedOut });
+        });
+    });
+}
+function parseAgentJson(stdout) {
+    try {
+        const j = JSON.parse(stdout.trim());
+        const cost = j.total_cost_usd ?? j.cost_usd ?? j.stats?.total_cost_usd;
+        return typeof cost === "number" ? { costUsd: cost } : {};
+    }
+    catch {
+        return {};
+    }
+}
+export async function runSuite(opts) {
+    const { root, suite, agents, jobs, keep, onCell } = opts;
+    const timeoutMs = (suite.timeoutSec ?? 600) * 1000;
+    const cells = [];
+    for (const [agent, spec] of Object.entries(agents)) {
+        for (const task of suite.tasks)
+            cells.push({ agent, spec, task });
+    }
+    const results = [];
+    let idx = 0;
+    async function worker() {
+        while (idx < cells.length) {
+            const cell = cells[idx++];
+            const { agent, spec, task } = cell;
+            const sandbox = mkdtempSync(path.join(tmpdir(), `agent-trials-${task.id}-`));
+            const t0 = Date.now();
+            const out = {
+                agent,
+                task: task.id,
+                pass: false,
+                agentExit: null,
+                verifyExit: null,
+                sec: 0,
+            };
+            try {
+                copyWorkdir(root, sandbox);
+                if (suite.setup) {
+                    const s = await runCmd(["sh", "-c", suite.setup], sandbox, timeoutMs);
+                    if (s.code !== 0)
+                        throw new Error(`setup failed (${s.code}): ${s.stderr.slice(-200)}`);
+                }
+                const argv = spec.argv.map((a) => (a === "{prompt}" ? task.prompt : a));
+                if (!spec.argv.includes("{prompt}"))
+                    argv.push(task.prompt);
+                const agentRes = await runCmd(argv, sandbox, timeoutMs);
+                out.agentExit = agentRes.code;
+                if (spec.resultJson)
+                    Object.assign(out, parseAgentJson(agentRes.stdout));
+                if (agentRes.code === null) {
+                    out.error = agentRes.timedOut ? "timeout" : `spawn failed: ${agentRes.stderr.slice(-160)}`;
+                }
+                const verifyCmd = task.verify ?? suite.verify;
+                const v = await runCmd(["sh", "-c", verifyCmd], sandbox, timeoutMs);
+                out.verifyExit = v.code;
+                out.pass = out.agentExit === 0 && v.code === 0;
+            }
+            catch (err) {
+                out.error = err instanceof Error ? err.message : String(err);
+            }
+            finally {
+                out.sec = Math.round((Date.now() - t0) / 100) / 10;
+                if (!keep)
+                    rmSync(sandbox, { recursive: true, force: true });
+                results.push(out);
+                onCell?.(out);
+            }
+        }
+    }
+    await Promise.all(Array.from({ length: Math.max(1, jobs) }, worker));
+    return results;
+}
+export function median(nums) {
+    if (nums.length === 0)
+        return 0;
+    const s = [...nums].sort((a, b) => a - b);
+    const mid = Math.floor(s.length / 2);
+    return s.length % 2 ? s[mid] : (s[mid - 1] + s[mid]) / 2;
+}
+export function summarize(results, agentNames) {
+    return agentNames.map((agent) => {
+        const cells = results.filter((r) => r.agent === agent);
+        const passes = cells.filter((r) => r.pass).length;
+        const costs = cells.map((r) => r.costUsd).filter((c) => typeof c === "number");
+        return {
+            agent,
+            passes,
+            total: cells.length,
+            medianSec: median(cells.map((r) => r.sec)),
+            totalCost: costs.length ? costs.reduce((a, b) => a + b, 0) : undefined,
+        };
+    });
+}
+export function verdictOf(c) {
+    if (c.error || c.agentExit !== 0)
+        return "err";
+    return c.pass ? "pass" : "fail";
+}
+export function renderBoard(suite, results, agentNames) {
+    const taskIds = suite.tasks.map((t) => t.id);
+    const find = (a, t) => results.find((r) => r.agent === a && r.task === t);
+    const colW = agentNames.map((a) => Math.max(a.length, ...taskIds.map((t) => {
+        const c = find(a, t);
+        return c ? `${verdictOf(c)} ${c.sec}s`.length : 1;
+    })) + 2);
+    const taskW = Math.max(...taskIds.map((t) => t.length), "median".length) + 2;
+    const lines = [];
+    lines.push(`agent-trials · ${taskIds.length} task${taskIds.length === 1 ? "" : "s"} · ${agentNames.length} agent${agentNames.length === 1 ? "" : "s"}`);
+    lines.push("");
+    lines.push("".padEnd(taskW) + agentNames.map((a, i) => a.padStart(colW[i])).join(""));
+    for (const tid of taskIds) {
+        const row = agentNames
+            .map((a, i) => {
+            const c = find(a, tid);
+            const label = c ? `${verdictOf(c)} ${c.sec}s` : "-";
+            return label.padStart(colW[i]);
+        })
+            .join("");
+        lines.push(tid.padEnd(taskW) + row);
+    }
+    lines.push("");
+    const sums = summarize(results, agentNames);
+    lines.push("score".padEnd(taskW) +
+        sums.map((s, i) => `${s.passes}/${s.total}`.padStart(colW[i])).join(""));
+    lines.push("median".padEnd(taskW) +
+        sums.map((s, i) => `${s.medianSec}s`.padStart(colW[i])).join(""));
+    if (sums.some((s) => s.totalCost !== undefined)) {
+        lines.push("cost".padEnd(taskW) +
+            sums
+                .map((s) => (s.totalCost === undefined ? "-" : `$${s.totalCost.toFixed(2)}`))
+                .map((x, i) => x.padStart(colW[i]))
+                .join(""));
+    }
+    const best = [...sums].sort((a, b) => b.passes - a.passes || a.medianSec - b.medianSec)[0];
+    if (best && sums.length > 1) {
+        lines.push("");
+        lines.push(`winner: ${best.agent} (${best.passes}/${best.total}, median ${best.medianSec}s)`);
+    }
+    return lines.join("\n");
+}
+export function renderMarkdown(suite, results, agentNames) {
+    const sums = summarize(results, agentNames);
+    const find = (a, t) => results.find((r) => r.agent === a && r.task === t);
+    const head = `| task | ${agentNames.join(" | ")} |`;
+    const sep = `|${"---|".repeat(agentNames.length + 1)}`;
+    const rows = suite.tasks.map((t) => {
+        const cells = agentNames
+            .map((a) => {
+            const c = find(a, t.id);
+            if (!c)
+                return "-";
+            const verdict = verdictOf(c);
+            const cost = c.costUsd !== undefined ? ` · $${c.costUsd.toFixed(2)}` : "";
+            return `${verdict} · ${c.sec}s${cost}`;
+        })
+            .join(" | ");
+        return `| ${t.id} | ${cells} |`;
+    });
+    const totals = `| **score** | ${sums.map((s) => `**${s.passes}/${s.total}**`).join(" | ")} |`;
+    const med = `| median | ${sums.map((s) => `${s.medianSec}s`).join(" | ")} |`;
+    const when = new Date().toISOString().slice(0, 10);
+    return [
+        `# agent-trials results`,
+        ``,
+        `${suite.name ?? "suite"} · run ${when} · ${suite.tasks.length} tasks × ${agentNames.length} agents`,
+        ``,
+        head,
+        sep,
+        ...rows,
+        totals,
+        med,
+        ``,
+        `Generated by [agent-trials](https://github.com/princejain756/agent-trials)`,
+    ].join("\n");
+}
+export function writeMarkdown(file, content) {
+    writeFileSync(file, content + "\n");
+}
+export function agentInstalled(argv0) {
+    const which = process.platform === "win32" ? "where" : "which";
+    const res = spawnSync(which, [argv0], { stdio: "ignore" });
+    return res.status === 0;
+}
+export function suiteFileExists(file) {
+    return existsSync(file);
+}

package/package.json

@@ -1,6 +1,40 @@
 {
   "name": "agent-trials",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "1.0.0",
+  "description": "Interview your coding agent before you ship with it. Run the same tasks through every agent CLI you have installed and get a scoreboard.",
+  "type": "module",
+  "bin": {
+    "agent-trials": "dist/cli.js"
+  },
+  "files": [
+    "dist"
+  ],
+  "scripts": {
+    "build": "tsc",
+    "prepare": "npm run build",
+    "test": "node test/run.mjs"
+  },
+  "engines": {
+    "node": ">=18"
+  },
+  "keywords": [
+    "ai-agents",
+    "evals",
+    "benchmark",
+    "claude-code",
+    "codex",
+    "gemini-cli",
+    "coding-agents",
+    "llm"
+  ],
+  "author": "Prince Jain",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/princejain756/agent-trials.git"
+  },
+  "devDependencies": {
+    "@types/node": "^26.6.5",
+    "typescript": "^7.0.2"
+  }
+}