firegoat 0.13.0 → 0.15.0

package/README.md

@@ -7,7 +7,7 @@ Status: early (v0.11). See `plan.md`.
 ## Quickstart
 
 ```bash
-npx firegoat init     # creates firegoat.yaml and .firegoat/, installs the skills, copies a prompt for your agent
+npx firegoat init     # creates firegoat.yaml and .firegoat/, installs the firegoat-onboarding skill, copies a prompt for your agent
 # edit firegoat.yaml (or paste the prompt into your coding agent and let it)
 npx firegoat eval     # runs your claude on each test, in .firegoat/runs/<runId>/<test>/<n>/work
 ```
@@ -82,12 +82,28 @@ tests:
 npx firegoat eval --var SHA=$(git rev-parse HEAD)
 ```
 
+### Same tests on several models
+
+```yaml
+model: [claude-haiku-5-5, claude-sonnet-5-5]
+runs: 2
+tests:
+  - name: quickstart
+    prompt: Get the acme SDK's hello-world example running and save its output to output.txt.
+    end_goal: output.txt holds the example's output.
+    checks:
+      - run: test -s output.txt
+```
+
+`model:` is passed to claude as `--model`. A list is a matrix: each test runs `runs` times per model, numbered `1..runs×models` (here `quickstart/1`, `2` on haiku, `3`, `4` on sonnet). `npx firegoat eval --model claude-opus-5-5` replaces it for one eval; repeat the flag for a matrix. Unset, firegoat passes no `--model` and claude uses its default. Each run's `run.json` records `model` (what firegoat asked for, or null) and `claudeModel` (what claude reported starting with), and the output labels each run with its model.
+
 ### Env vars for the agent: a home dir per run, a feature flag
 
 ```yaml
 env:
   ACME_HOME: "/tmp/acme-{{run.id}}-{{run.n}}"   # sessions/config never collide across runs
   ACME_TELEMETRY: "off"
+  UV_FIND_LINKS: "$FIREGOAT_OUT"                # resolve the wheel setup built
 tests:
   - name: login
     prompt: Log in to acme with the CLI and print the current user.
@@ -98,18 +114,19 @@ tests:
       - run: ls "$ACME_HOME"/sessions | grep -q .
 ```
 
-`env:` goes to the agent, and to `workdir_setup` and checks of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs), `{{run.workdir}}`, `{{test.name}}`. `setup` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}` and `{{run.id}}`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set.
+`env:` goes to the agent, and to `workdir_setup` and checks of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` (also `${...}`), the same dirs `setup`, `workdir_setup` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `setup` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`.
 
 ## What eval does
 
 - claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers. claude saves its session there, in its own format.
-- No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. In `.firegoat/`, git ignores `runs/` and `.claude/`; keep test fixtures (scripts, sample data) in `.firegoat/fixtures/` and commit them with `firegoat.yaml`.
+- No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. In `.firegoat/`, git ignores everything except `fixtures/`: keep test fixtures (scripts, sample data) in `.firegoat/fixtures/` and commit them with `firegoat.yaml`.
+- In a linked git worktree (`git worktree add`), `.firegoat/runs/` and `.firegoat/.claude/` are in the main checkout, so runs (and their session files) survive `git worktree remove`; `firegoat.yaml`, fixtures and `setup:` stay in the worktree. Each `summary.json` records `dir` (where eval ran) and `git: {branch, commit, dirty}`, and `.firegoat/runs/index.jsonl` gets one line per eval (`runId, startedAt, dir, branch, commit, passed, failed, notGraded, cost`) to tell runs from different branches apart.
 - The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, setup, workdir_setup, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it.
-- Runs (tests × `runs`) go 4 at a time, each in its own folder with its own session; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
+- Runs (tests × models × `runs`) go 4 at a time, each in its own folder with its own session; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
 - Script checks run in the agent's folder afterwards and set the exit code. `end_goal` is not graded by firegoat: it's for you, or your coding agent, to judge from `run.json`, the session and the folder.
 - Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`), `run.json` (status, prompt, end_goal, check results, paths), `work/`, logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
 
-`firegoat.yaml` keys: `runs`, `setup`, `workdir_setup`, `vars`, `env`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
+`firegoat.yaml` keys: `runs`, `model`, `setup`, `workdir_setup`, `vars`, `env`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
 
 Want a plain `npm i acme-sdk` to get your build, with no hint in the prompt? See [`examples/local-npm-registry`](examples/local-npm-registry/README.md).
 
@@ -128,7 +145,7 @@ Something in firegoat got in your way (or your agent's)? Tell us from the termin
 npx firegoat feedback "eval hid claude's spend-limit error" --type bug --body "ran npx firegoat eval, got 'not graded' x3"
 ```
 
-It goes to the firegoat team with the firegoat version, your OS and node version, and which coding agent sent it. Don't include secrets: a report that looks like it has one is refused.
+It goes to the firegoat team with the firegoat version, your OS and node version, and which coding agent sent it. Don't include secrets: a report that looks like it has one is refused. The first send registers your device and saves its token to `~/.config/firegoat/feedback-device.json` (`$XDG_CONFIG_HOME` respected), which raises the daily report limit.
 
 ## Develop
 

package/dist/cli.js

@@ -6947,14 +6947,14 @@ var require_public_api = __commonJS(function(exports) {
 // package.json
 var package_default = {
   name: "firegoat",
-  version: "0.13.0",
+  version: "0.15.0",
   repository: {
     type: "git",
     url: "git+https://github.com/firegoat-ai/firegoat-cli.git"
   },
   devDependencies: {
     "@biomejs/biome": "^2.5.15",
-    "@firegoat/feedback": "0.0.0",
+    "@firegoat/feedback": "0.2.0",
     "@types/bun": "latest",
     typescript: "^7",
     yaml: "^2.9.1"
@@ -7075,6 +7075,7 @@ function parseConfig(text) {
     fail("(root)", "must be a mapping");
   onlyKeys(raw, "(root)", [
     "runs",
+    "model",
     "setup",
     "workdir_setup",
     "vars",
@@ -7082,6 +7083,7 @@ function parseConfig(text) {
     "tests"
   ]);
   const runs = posInt(raw.runs, "runs", 1);
+  const models = parseModels(raw.model);
   const setup = commands(raw.setup, "setup");
   const workdirSetup = commands(raw.workdir_setup, "workdir_setup");
   const vars = parseVars(raw.vars);
@@ -7122,6 +7124,7 @@ function parseConfig(text) {
   });
   return {
     runs,
+    models,
     setup,
     workdirSetup,
     vars,
@@ -7139,6 +7142,22 @@ function commands(raw, path) {
     fail(path, "must be a shell command or a list of them");
   return raw.map((c, i) => str(c, `${path}[${i}]`));
 }
+function parseModels(raw) {
+  if (raw === undefined)
+    return [];
+  if (typeof raw === "string")
+    return [str(raw, "model")];
+  if (!Array.isArray(raw) || raw.length === 0)
+    fail("model", "must be a model name or a non-empty list of them");
+  const seen = new Set;
+  return raw.map((m, i) => {
+    const name = str(m, `model[${i}]`);
+    if (seen.has(name))
+      fail(`model[${i}]`, `duplicate model ${JSON.stringify(name)}`);
+    seen.add(name);
+    return name;
+  });
+}
 var VAR_KEY = /^[A-Za-z_][A-Za-z0-9_]*$/;
 function parseVars(raw) {
   if (raw === undefined)
@@ -7160,9 +7179,11 @@ var RESERVED_ENV = new Set([
   "CLAUDE_CONFIG_DIR",
   "CLAUDE_SECURESTORAGE_CONFIG_DIR"
 ]);
-var RUN_PLACEHOLDERS = ["run.id", "run.n", "run.workdir", "test.name"];
+var RUN_PLACEHOLDERS = ["run.id", "run.n", "test.name"];
 var EVAL_PLACEHOLDERS = ["run.id"];
 var ANY_PLACEHOLDER = /\{\{\s*([^{}]*?)\s*\}\}/g;
+var DIR_REF = /\$(?:\{(FIREGOAT_OUT|FIREGOAT_WORKDIR)\}|(FIREGOAT_OUT|FIREGOAT_WORKDIR)(?![A-Za-z0-9_]))/g;
+var ENV_REF = new RegExp(`${ANY_PLACEHOLDER.source}|${DIR_REF.source}`, "g");
 function parseEnv(raw, path) {
   if (raw === undefined)
     return {};
@@ -7173,6 +7194,8 @@ function parseEnv(raw, path) {
     const p = `${path}.${k}`;
     if (!VAR_KEY.test(k))
       fail(p, "name must be letters, digits and _, not starting with a digit");
+    if (k === "ANTHROPIC_MODEL")
+      fail(p, "is reserved: choose the model with top-level model: (or eval --model)");
     if (RESERVED_ENV.has(k) || k.startsWith("FIREGOAT_"))
       fail(p, "is set by firegoat and cannot be overridden (reserved: PATH, CLAUDE_CONFIG_DIR, CLAUDE_SECURESTORAGE_CONFIG_DIR, FIREGOAT_*)");
     if (typeof v !== "string")
@@ -7187,7 +7210,8 @@ function parseEnv(raw, path) {
   return out;
 }
 function isEvalLevel(value) {
-  return [...value.matchAll(ANY_PLACEHOLDER)].every((m) => {
+  const dirsOk = [...value.matchAll(DIR_REF)].every((m) => (m[1] ?? m[2]) === "FIREGOAT_OUT");
+  return dirsOk && [...value.matchAll(ANY_PLACEHOLDER)].every((m) => {
     const name = m[1];
     return name.startsWith("vars.") || EVAL_PLACEHOLDERS.includes(name);
   });
@@ -7195,8 +7219,9 @@ function isEvalLevel(value) {
 function renderEnv(env, vars, run) {
   const out = {};
   for (const [k, v] of Object.entries(env))
-    out[k] = v.replace(ANY_PLACEHOLDER, (whole, name) => {
-      const filled = name.startsWith("vars.") ? vars[name.slice("vars.".length)] : run[name];
+    out[k] = v.replace(ENV_REF, (whole, name, braced, bare) => {
+      const key = name ?? braced ?? bare;
+      const filled = key.startsWith("vars.") ? vars[key.slice("vars.".length)] : run[key];
       if (filled === undefined)
         throw new ConfigError(`env.${k}: no value for ${whole}`);
       return filled;
@@ -7225,13 +7250,14 @@ function loadConfig(dir) {
 // src/eval.ts
 import { randomBytes } from "node:crypto";
 import {
+  appendFileSync,
   createWriteStream as createWriteStream2,
   existsSync as existsSync2,
   mkdirSync as mkdirSync2,
   readdirSync as readdirSync2,
   writeFileSync as writeFileSync3
 } from "node:fs";
-import { basename, delimiter as delimiter2, join as join4 } from "node:path";
+import { basename as basename2, delimiter as delimiter2, join as join4 } from "node:path";
 
 // src/claude.ts
 import { spawn, spawnSync } from "node:child_process";
@@ -7293,7 +7319,12 @@ async function runClaude(opts) {
   let pending = "";
   const exited = await run({
     cmd: opts.bin,
-    args: ["-p", opts.prompt, ...CLAUDE_FLAGS],
+    args: [
+      "-p",
+      opts.prompt,
+      ...CLAUDE_FLAGS,
+      ...opts.model === undefined ? [] : ["--model", opts.model]
+    ],
     cwd: opts.cwd,
     env: opts.env,
     timeoutMs: opts.timeoutMs,
@@ -7317,6 +7348,7 @@ async function runClaude(opts) {
   if (exited.error)
     return {
       events: [],
+      claudeModel: null,
       ok: false,
       reason: `could not start ${opts.bin}: ${exited.error.message}`
     };
@@ -7334,15 +7366,18 @@ async function runClaude(opts) {
 `)[0]?.slice(0, 300);
     return {
       events,
+      claudeModel: null,
       ok: false,
       reason: `claude did not start a session (exit ${exited.code ?? "by signal"})${said ? `: ${said}` : ", see stderr.txt"}`
     };
   }
   const sessionId = String(init.session_id);
+  const claudeModel = typeof init.model === "string" ? init.model : null;
   const result = events.findLast((e) => e?.type === "result");
   if (exited.timedOut)
     return {
       events,
+      claudeModel,
       ok: false,
       reason: `timed out after ${opts.timeoutMs / 1000}s`,
       result,
@@ -7351,6 +7386,7 @@ async function runClaude(opts) {
   if (!result)
     return {
       events,
+      claudeModel,
       ok: false,
       reason: `claude exited ${exited.code ?? "by signal"} without a result, see stderr.txt`,
       sessionId
@@ -7358,12 +7394,13 @@ async function runClaude(opts) {
   if (result.is_error === true || result.subtype !== "success")
     return {
       events,
+      claudeModel,
       ok: false,
       reason: `claude result is an error (subtype=${result.subtype})${result.result ? `: ${result.result.slice(0, 300)}` : ""}`,
       result,
       sessionId
     };
-  return { events, ok: true, result, sessionId };
+  return { events, claudeModel, ok: true, result, sessionId };
 }
 async function runShell(opts) {
   opts.log.write(`$ ${opts.run}
@@ -7415,8 +7452,9 @@ var EXIT = { pass: 0, fail: 1, notGraded: 2, usage: 3 };
 // src/firegoat-dir.ts
 import { mkdirSync, writeFileSync } from "node:fs";
 import { join } from "node:path";
-var GITIGNORE = `runs/
-.claude/
+var GITIGNORE = `*
+!fixtures/
+!fixtures/**
 `;
 function ensureFiregoatDir(root) {
   const dir = join(root, ".firegoat");
@@ -7425,6 +7463,51 @@ function ensureFiregoatDir(root) {
   return dir;
 }
 
+// src/git.ts
+import { spawnSync as spawnSync2 } from "node:child_process";
+import { basename, dirname } from "node:path";
+function git(dir, args) {
+  const r = spawnSync2("git", args, { cwd: dir, encoding: "utf8" });
+  if (r.error)
+    throw r.error;
+  return { code: r.status ?? -1, stdout: r.stdout.trim() };
+}
+function gitRepo(dir) {
+  const r = spawnSync2("git", ["rev-parse", "--path-format=absolute", "--git-dir", "--git-common-dir"], { cwd: dir, encoding: "utf8" });
+  if (r.error?.code === "ENOENT")
+    return null;
+  if (r.error)
+    throw r.error;
+  if (r.status !== 0) {
+    if (/not a git repository/.test(r.stderr))
+      return null;
+    throw new Error(`git rev-parse in ${dir} failed: ${r.stderr.trim()}`);
+  }
+  const [gitDir, commonDir] = r.stdout.trim().split(`
+`);
+  if (!gitDir || !commonDir)
+    throw new Error(`git rev-parse in ${dir} printed ${r.stdout}`);
+  let mainCheckout = null;
+  if (gitDir !== commonDir) {
+    if (basename(commonDir) !== ".git")
+      throw new Error(`${dir} is a linked git worktree, but its common git dir ${commonDir} is not a checkout's .git (a bare repo?), so there is no main checkout to keep .firegoat/runs in`);
+    mainCheckout = dirname(commonDir);
+  }
+  const branch = git(dir, ["symbolic-ref", "-q", "--short", "HEAD"]);
+  const commit = git(dir, ["rev-parse", "-q", "--verify", "HEAD"]);
+  const status = git(dir, ["status", "--porcelain"]);
+  if (status.code !== 0)
+    throw new Error(`git status in ${dir} failed`);
+  return {
+    info: {
+      branch: branch.code === 0 ? branch.stdout : null,
+      commit: commit.code === 0 ? commit.stdout : null,
+      dirty: status.stdout !== ""
+    },
+    mainCheckout
+  };
+}
+
 // src/help.ts
 var EXIT_CODES = `Exit codes:
   0  every run's script checks passed
@@ -7437,7 +7520,7 @@ function topHelp(version) {
 folder, with none of your claude config; firegoat saves the session and runs your
 script checks.
 
-  npx firegoat init     create firegoat.yaml and .firegoat/, install the skills
+  npx firegoat init     create firegoat.yaml and .firegoat/, install the skill
   npx firegoat eval     run the tests (claude must be logged in: run \`claude\` once)
   npx firegoat feedback "<what got in the way>"   tell the firegoat team
 
@@ -7454,16 +7537,20 @@ ${EXIT_CODES}`;
 var INIT_HELP = `Usage: firegoat init [--force] [--no-skills]
 
 In the current folder: creates .firegoat/ and a starter firegoat.yaml, installs the
-firegoat-onboarding and suggesting-eval-prompts skills into .claude/skills/ (npx skills
+firegoat-onboarding skill into .claude/skills/ (npx skills
 add firegoat-ai/skills; asks first in a terminal), then prints a prompt to paste into
-your coding agent. In .firegoat/, git ignores runs/ and .claude/; keep test fixtures
+your coding agent. In .firegoat/, git ignores everything but fixtures/: keep test fixtures
 (helper scripts, sample data) in .firegoat/fixtures/ and commit them.
 
   --force       replace an existing firegoat.yaml
-  --no-skills   don't install the skills
+  --no-skills   don't install the skill
 
 firegoat.yaml reference:
   runs: 1                     runs per test
+  model: <name> | [<name>, ...]
+                              claude --model; a list runs every test runs times
+                              per model (n counts 1..runs x models). Unset: no
+                              --model, claude's default. eval --model overrides
   setup: <sh> | [<sh>, ...]   once per eval, in this folder; what it writes to
                               $FIREGOAT_OUT is first on the agent's PATH
   workdir_setup: <sh> | [...] in each fresh folder before the agent starts
@@ -7472,9 +7559,15 @@ firegoat.yaml reference:
   env: {NAME: value}          env vars for the agent, and for workdir_setup and
                               checks of the same run; values are strings and can
                               use {{vars.KEY}}, {{run.id}}, {{run.n}},
-                              {{run.workdir}}, {{test.name}}. setup gets only the
-                              entries that use nothing but {{vars.*}}/{{run.id}}.
-                              PATH, CLAUDE_CONFIG_DIR and FIREGOAT_* are reserved.
+                              {{test.name}}, $FIREGOAT_OUT, $FIREGOAT_WORKDIR
+                              (no other $ expansion: $HOME stays literal).
+                              setup gets only the entries that use nothing but
+                              {{vars.*}}, {{run.id}} and $FIREGOAT_OUT.
+                              PATH, CLAUDE_CONFIG_DIR, FIREGOAT_* and
+                              ANTHROPIC_MODEL (use model:) are reserved.
+                              Runs go 4 at a time: give a tool with a global home
+                              a per-run one (or use eval --parallel 1), e.g.
+                    MYTOOL_HOME: "/tmp/mytool-{{run.id}}-{{test.name}}-{{run.n}}"
   tests:
     - name: quickstart        letters, digits, '.', '_', '-'; unique
       prompt: ...             what a user would ask the agent; one concrete outcome
@@ -7513,9 +7606,9 @@ Example:
         - run: |
             ! grep -Eq '$ .*npm (install|i|add) .*acme' "$FIREGOAT_TRANSCRIPT"
 
-Exit codes: 0 done, 2 the skills install failed, 3 usage error.`;
-var EVAL_HELP = `Usage: firegoat eval [test..] [--runs N] [--parallel N] [--var KEY=value]..
-                     [--json] [--claude-bin PATH]
+Exit codes: 0 done, 2 the skill install failed, 3 usage error.`;
+var EVAL_HELP = `Usage: firegoat eval [test..] [--runs N] [--model NAME].. [--parallel N]
+                     [--var KEY=value].. [--json] [--claude-bin PATH]
 
 Runs each test in firegoat.yaml (in the current folder): your claude gets the prompt
 in a fresh, empty folder, .firegoat/runs/<runId>/<test>/<n>/work, with setup:'s
@@ -7526,18 +7619,31 @@ with a fresh config dir, .firegoat/.claude: none of your settings, CLAUDE.md,
 plugins, skills or MCP servers. claude saves its session there. There is no
 sandbox: the agent can reach the rest of your machine, including the repo.
 
+In a linked git worktree (git worktree add), .firegoat/runs/ and .firegoat/.claude
+live in the main checkout instead, so runs outlive the worktree; firegoat.yaml,
+fixtures and setup: stay in the current folder.
+
   test..             only these tests (by name)
   --runs N           runs per test (overrides runs: in firegoat.yaml)
-  --parallel N       runs at once (default 4; 1 runs them one after another)
+  --model NAME       claude --model, replacing model: in firegoat.yaml; repeat it
+                     (--model a --model b) to run each test runs times per model
+  --parallel N       runs at once (default 4; 1 runs them one after another).
+                     Tools with global on-disk state collide: give each run its
+                     own home via env: ({{run.id}}-{{test.name}}-{{run.n}}) or use 1
   --var KEY=value    fills {{vars.KEY}}; env for setup, workdir_setup and checks
   --json             the summary as JSON on stdout (or {error, exitCode} if eval
                      stops early); progress goes to stderr
   --claude-bin PATH  claude binary (env FIREGOAT_CLAUDE_BIN; default: claude on PATH)
 
 Output: .firegoat/runs/<runId>/
-  summary.json            every run, plus totals
-  <test>/<n>/run.json     one run: status and reason (from the checks), prompt,
-                          end_goal, checks [{run, code}], and paths: workdir,
+  summary.json            every run, plus totals, dir (where eval ran) and
+                          git {branch, commit, dirty} (no git outside a repo)
+  ../index.jsonl          one line per eval: runId, startedAt, dir, branch,
+                          commit, passed, failed, notGraded, cost
+  <test>/<n>/run.json     one run: status and reason (from the checks), model
+                          (asked for, or null) and claudeModel (what claude's
+                          init line reported), prompt, end_goal,
+                          checks [{run, code}], and paths: workdir,
                           transcript, session (claude's session file), stderr,
                           checksLog
   <test>/<n>/transcript.txt  what the agent did: each tool call with its command,
@@ -7551,6 +7657,8 @@ yours to judge: read transcript.txt first (a few KB); the session file has every
 Examples:
   npx firegoat eval --json                  all tests, JSON summary on stdout
   npx firegoat eval quickstart --runs 3     one test, three runs
+  npx firegoat eval --model claude-haiku-5-5 --model claude-sonnet-5-5
+                                            each test on both models
   npx firegoat eval --var SHA=$(git rev-parse HEAD)
 
 A run can take minutes (default timeout 600s per test).
@@ -7568,7 +7676,9 @@ title, then what you ran and what happened in --body (- reads stdin).
 
 Sent along: the firegoat version, your OS and node version, and which agent you are
 (claude-code/codex/cursor, from the environment). Don't include secrets: a report that
-looks like it contains one is refused.
+looks like it contains one is refused. The first send registers this device and saves
+its token to ~/.config/firegoat/feedback-device.json ($XDG_CONFIG_HOME respected),
+which raises the daily report limit.
 
 Example:
   npx firegoat feedback "checks can't grep commands with quotes" --type bug \\
@@ -7692,10 +7802,14 @@ ${indent(truncate(text, OUTPUT_LINES, OUTPUT_CHARS))}` : `    → ${status}`);
 }
 
 // src/ui.ts
-import { relative as relative2 } from "node:path";
+import { isAbsolute, relative as relative2 } from "node:path";
 var RAMP = "▁▂▃▄▅▆▇";
 var ANSI = new RegExp(`${String.fromCharCode(27)}\\[[0-9;]*m`, "g");
 var SPIN = ["◐", "◓", "◑", "◒"];
+function shortPath(path) {
+  const rel = relative2(process.cwd(), path);
+  return rel.startsWith("..") || isAbsolute(rel) ? path : rel;
+}
 function painter(color) {
   return (code, text) => color ? `\x1B[${code}m${text}\x1B[0m` : text;
 }
@@ -7786,7 +7900,8 @@ function plainUI(write) {
         printed.add(r);
         const label = r.state === "error" ? "ERROR" : r.state.toUpperCase();
         const u = `$${(r.cost ?? 0).toFixed(2)}, ${r.turns} turns, ${r.seconds ?? 0}s`;
-        write(`${label} ${r.test}#${r.n} ${r.reason} (${u})`);
+        const model = r.model === null ? "" : ` [${r.model}]`;
+        write(`${label} ${r.test}#${r.n}${model} ${r.reason} (${u})`);
       }
     },
     done: ({ runsDir, rows: all }) => {
@@ -7853,10 +7968,12 @@ function ttyUI(out, opts) {
       const mark = setup.state === "ok" ? p(C.green, "✓") : setup.state === "fail" ? p(C.red, "✗") : p(C.ember, SPIN[frame % SPIN.length]);
       out.push(`  ${setup.state === "running" ? p(C.ember, "◐") : p(C.green, "●")}  setup${" ".repeat(Math.max(1, w - 20))}${mark}${setup.seconds !== undefined ? ` ${setup.seconds}s` : ""}`);
     }
+    const modelCols = Math.min(24, Math.max(0, ...rows.map((r) => r.model?.length ?? 0)));
     rows.forEach((r, i) => {
       out.push(p(C.gray, "  │"));
       const num = String(i + 1).padStart(2, "0");
       const name = cut(`${r.test}${r.n > 1 ? `#${r.n}` : ""}`, 16).padEnd(16);
+      const model = modelCols > 0 ? ` ${p(C.cyan, cut(r.model ?? "", modelCols).padEnd(modelCols))}` : "";
       let detail = "";
       if (r.state === "running")
         detail = p(C.gray, `turn ${r.turns}`);
@@ -7864,7 +7981,7 @@ function ttyUI(out, opts) {
         detail = `${checksPassed(r).padEnd(11)} ${p(C.gray, usage(r))}`;
       else if (r.state === "error")
         detail = p(C.gray, usage(r));
-      out.push(`  ${glyph(r)}  ${p(C.gray, num)}  ${p(C.bold, name)} ${p(C.ember, ramp(r))} ${statusWord(r)}  ${detail}`);
+      out.push(`  ${glyph(r)}  ${p(C.gray, num)}  ${p(C.bold, name)}${model} ${p(C.ember, ramp(r))} ${statusWord(r)}  ${detail}`);
       if (r.state === "running" && r.command)
         out.push(p(C.gray, `  ┊        └ $ ${cut(r.command, w - 16)}`));
       if (r.state === "fail")
@@ -7891,7 +8008,7 @@ function ttyUI(out, opts) {
     const right = goated ? `GOATED · tier ${tr}` : tr ? `tier ${tr}` : "";
     out.push(`  ${goated ? p(C.green, left) : left}${" ".repeat(Math.max(2, w - left.length - right.length - 3))}${goated ? p(C.ember, right) : right}`);
     out.push(p(C.gray, `  total $${t.cost.toFixed(2)} · ${t.turns} turns · ${duration(finished.seconds)}`));
-    out.push(p(C.gray, `  runs  ${relative2(process.cwd(), finished.runsDir)}/`));
+    out.push(p(C.gray, `  runs  ${shortPath(finished.runsDir)}/`));
     out.push(`  checks are a score, not the verdict: judge each end_goal from ${p(C.cyan, "<test>/<n>/transcript.txt")}`);
     return out;
   };
@@ -7963,6 +8080,7 @@ class UsageError extends Error {
 function parseArgs(args) {
   const opts = {
     names: [],
+    models: [],
     parallel: 4,
     json: false,
     vars: {},
@@ -7996,6 +8114,13 @@ function parseArgs(args) {
       if (!/^\d+$/.test(v) || n < 1)
         throw new UsageError(`--runs must be a positive integer, got ${v}`);
       opts.runs = n;
+    } else if (flag === "--model") {
+      const v = value();
+      if (v.trim() === "")
+        throw new UsageError("--model needs a model name");
+      if (opts.models.includes(v))
+        throw new UsageError(`--model ${v} given twice`);
+      opts.models.push(v);
     } else if (flag === "--parallel") {
       const v = value();
       const n = Number(v);
@@ -8022,6 +8147,10 @@ function findSession(configDir, sessionId) {
   }
   return null;
 }
+function grepsSessionJson(run) {
+  const unescaped = run.replace(/\\/g, "");
+  return unescaped.includes("FIREGOAT_SESSION") && /"command":"|"name":"Write"/.test(unescaped);
+}
 function newRunId() {
   const ts = new Date().toISOString().replace(/\.\d+Z$/, "Z").replace(/[-:]/g, "");
   return `${ts}-${randomBytes(2).toString("hex")}`;
@@ -8035,6 +8164,8 @@ async function runOne(opts) {
   const r = {
     test: test.name,
     n,
+    model: opts.model,
+    claudeModel: null,
     status: "error",
     reason: "",
     prompt: render(test.prompt, opts.vars),
@@ -8061,8 +8192,9 @@ async function runOne(opts) {
   const env = renderEnv(test.env, opts.vars, {
     "run.id": opts.runId,
     "run.n": String(n),
-    "run.workdir": workdir,
-    "test.name": test.name
+    "test.name": test.name,
+    FIREGOAT_OUT: opts.out,
+    FIREGOAT_WORKDIR: workdir
   });
   const stepEnv = { ...opts.stepEnv, ...env, FIREGOAT_WORKDIR: workdir };
   if (opts.workdirSetup.length > 0) {
@@ -8084,14 +8216,16 @@ async function runOne(opts) {
     env: { ...claudeEnv(opts.configDir, env), PATH: opts.stepEnv.PATH },
     stderr,
     timeoutMs: test.timeout * 1000,
+    model: opts.model ?? undefined,
     onEvent: opts.onEvent
   });
+  r.claudeModel = outcome.claudeModel;
   await new Promise((res) => stderr.end(res));
   if (outcome.sessionId)
     r.session = findSession(opts.configDir, outcome.sessionId);
   writeTranscript({
     path: r.transcript,
-    title: `${test.name}#${n}`,
+    title: `${test.name}#${n}${opts.model ? ` (model ${opts.model})` : ""}`,
     prompt: r.prompt,
     events: outcome.events,
     workdir
@@ -8146,7 +8280,7 @@ async function evalCommand(args, root) {
   }
   const say = opts.json ? console.error : console.log;
   const stream = opts.json ? process.stderr : process.stdout;
-  const ui = stream.isTTY ? ttyUI(stream, { project: basename(root), color: !process.env.NO_COLOR }) : plainUI(say);
+  const ui = stream.isTTY ? ttyUI(stream, { project: basename2(root), color: !process.env.NO_COLOR }) : plainUI(say);
   let config;
   try {
     config = loadConfig(root);
@@ -8163,7 +8297,10 @@ async function evalCommand(args, root) {
       return fatal(EXIT.usage, `unknown test ${unknown.map((n) => JSON.stringify(n)).join(", ")} (known: ${[...known].join(", ")})`);
     tests = tests.filter((t) => opts.names.includes(t.name));
   }
+  if (tests.some((t) => t.checks.some((c) => grepsSessionJson(c.run))))
+    console.error('firegoat eval: warning: a check greps $FIREGOAT_SESSION for session JSON ("command":" / "name":"Write"), which breaks on escaping. Grep $FIREGOAT_TRANSCRIPT instead: one "[n] $ <command>" line per shell call, "[n] Write <path>" per file write.');
   const runs = opts.runs ?? config.runs;
+  const models = opts.models.length > 0 ? opts.models : config.models;
   const vars = { ...config.vars, ...opts.vars };
   const missing = [
     ...new Set(tests.flatMap((t) => [
@@ -8177,7 +8314,13 @@ async function evalCommand(args, root) {
   const claudePath = resolveBin(opts.claudeBin);
   if (!claudePath)
     return fatal(EXIT.notGraded, `not graded: claude binary ${JSON.stringify(opts.claudeBin)} not found. Install Claude Code (https://claude.com/claude-code) or pass --claude-bin PATH / FIREGOAT_CLAUDE_BIN.`);
-  const firegoatDir = ensureFiregoatDir(root);
+  let repo;
+  try {
+    repo = gitRepo(root);
+  } catch (e) {
+    return fatal(EXIT.usage, e.message);
+  }
+  const firegoatDir = ensureFiregoatDir(repo?.mainCheckout ?? root);
   const configDir = join4(firegoatDir, ".claude");
   mkdirSync2(configDir, { recursive: true });
   ui.auth(process.env.ANTHROPIC_API_KEY ? "ANTHROPIC_API_KEY (claude bills the API key; unset it to use your claude login)" : "your claude login");
@@ -8190,7 +8333,8 @@ async function evalCommand(args, root) {
     return fatal(EXIT.notGraded, `not graded: ${loginProblem}`);
   const evalStarted = Date.now();
   const runId = newRunId();
-  const runDir = join4(firegoatDir, "runs", runId);
+  const runsRoot = join4(firegoatDir, "runs");
+  const runDir = join4(runsRoot, runId);
   const out = join4(runDir, "out");
   mkdirSync2(out, { recursive: true });
   const stepEnv = {
@@ -8199,12 +8343,14 @@ async function evalCommand(args, root) {
     FIREGOAT_OUT: out,
     PATH: [out, process.env.PATH ?? ""].join(delimiter2)
   };
-  const rows = tests.flatMap((t) => Array.from({ length: runs }, (_, i) => ({
+  const matrix = models.length > 0 ? models : [null];
+  const rows = tests.flatMap((t) => matrix.flatMap((model, m) => Array.from({ length: runs }, (_, i) => ({
     test: t.name,
-    n: i + 1,
+    n: m * runs + i + 1,
+    model,
     state: "queued",
     turns: 0
-  })));
+  }))));
   if (config.setup.length > 0) {
     const setupStarted = Date.now();
     ui.setup("running");
@@ -8214,7 +8360,7 @@ async function evalCommand(args, root) {
       cwd: root,
       env: {
         ...stepEnv,
-        ...renderEnv(Object.fromEntries(Object.entries(config.env).filter(([, v]) => isEvalLevel(v))), vars, { "run.id": runId })
+        ...renderEnv(Object.fromEntries(Object.entries(config.env).filter(([, v]) => isEvalLevel(v))), vars, { "run.id": runId, FIREGOAT_OUT: out })
       },
       logPath: join4(runDir, "setup.txt")
     });
@@ -8247,11 +8393,13 @@ async function evalCommand(args, root) {
       },
       test,
       n: row.n,
+      model: row.model,
       runId,
       runDir,
       claudeBin: claudePath,
       configDir,
       workdirSetup: config.workdirSetup,
+      out,
       stepEnv,
       vars
     });
@@ -8276,8 +8424,27 @@ async function evalCommand(args, root) {
   const totals = { pass: 0, fail: 0, error: 0 };
   for (const r of results)
     totals[r.status]++;
-  const doc = { runId, runsDir: runDir, totals, runs: results };
+  const doc = {
+    runId,
+    runsDir: runDir,
+    dir: root,
+    ...repo && { git: repo.info },
+    totals,
+    runs: results
+  };
   writeFileSync3(join4(runDir, "summary.json"), `${JSON.stringify(doc, null, 2)}
+`);
+  appendFileSync(join4(runsRoot, "index.jsonl"), `${JSON.stringify({
+    runId,
+    startedAt: new Date(evalStarted).toISOString(),
+    dir: root,
+    branch: repo?.info.branch ?? null,
+    commit: repo?.info.commit ?? null,
+    passed: totals.pass,
+    failed: totals.fail,
+    notGraded: totals.error,
+    cost: results.reduce((sum, r) => sum + (r.cost ?? 0), 0)
+  })}
 `);
   ui.done({
     runId,
@@ -8294,7 +8461,68 @@ async function evalCommand(args, root) {
 import { readFileSync as readFileSync2 } from "node:fs";
 
 // node_modules/@firegoat/feedback/dist/index.js
-var VERSION = "0.0.0";
+function memoryDeviceStore() {
+  const tokens = {};
+  return {
+    load: async (baseUrl) => tokens[baseUrl],
+    save: async (baseUrl, token) => {
+      tokens[baseUrl] = token;
+    },
+    clear: async (baseUrl) => {
+      delete tokens[baseUrl];
+    }
+  };
+}
+async function defaultDevicePath() {
+  const { homedir } = await import("node:os");
+  const { join } = await import("node:path");
+  const configHome = process.env.XDG_CONFIG_HOME || join(homedir(), ".config");
+  return join(configHome, "firegoat", "feedback-device.json");
+}
+function fileDeviceStore(path) {
+  const resolvePath = async () => path ?? await defaultDevicePath();
+  async function read() {
+    const { readFile } = await import("node:fs/promises");
+    const file = await resolvePath();
+    let text;
+    try {
+      text = await readFile(file, "utf8");
+    } catch (err) {
+      if (err.code === "ENOENT")
+        return {};
+      throw err;
+    }
+    try {
+      return JSON.parse(text);
+    } catch {
+      throw new Error(`@firegoat/feedback: ${file} is not valid JSON; delete it to register a new device`);
+    }
+  }
+  async function write(tokens) {
+    const { mkdir, writeFile } = await import("node:fs/promises");
+    const { dirname } = await import("node:path");
+    const file = await resolvePath();
+    await mkdir(dirname(file), { recursive: true });
+    await writeFile(file, `${JSON.stringify(tokens, null, 2)}
+`, {
+      mode: 384
+    });
+  }
+  return {
+    load: async (baseUrl) => (await read())[baseUrl],
+    save: async (baseUrl, token) => write({ ...await read(), [baseUrl]: token }),
+    clear: async (baseUrl) => {
+      const tokens = await read();
+      delete tokens[baseUrl];
+      await write(tokens);
+    }
+  };
+}
+function defaultDeviceStore() {
+  const isNode = Boolean(globalThis.process?.versions?.node);
+  return isNode ? fileDeviceStore() : memoryDeviceStore();
+}
+var VERSION = "0.2.0";
 var DEFAULT_BASE_URL = "https://firegoat.ai/evals";
 var TIMEOUT_MS = 15000;
 function envBaseUrl() {
@@ -8323,6 +8551,7 @@ class FeedbackClient {
   baseUrl;
   token;
   fetchImpl;
+  device;
   constructor(options) {
     if (!options.domain?.trim()) {
       throw new Error("FeedbackClient: `domain` is required (your product's domain, e.g. example.com)");
@@ -8331,6 +8560,7 @@ class FeedbackClient {
     this.baseUrl = (options.baseUrl ?? envBaseUrl() ?? DEFAULT_BASE_URL).replace(/\/+$/, "");
     this.token = options.token;
     this.fetchImpl = options.fetch ?? fetch;
+    this.device = options.token || options.device === false ? undefined : options.device ?? defaultDeviceStore();
   }
   async registerDevice() {
     const data = await this.post("/api/feedback/device", undefined);
@@ -8338,6 +8568,7 @@ class FeedbackClient {
       throw new FeedbackError(201, "invalid_response", "server response missing token");
     }
     this.token = data.token;
+    await this.device?.save(this.baseUrl, data.token);
     return data.token;
   }
   async send(input) {
@@ -8350,18 +8581,43 @@ class FeedbackClient {
       cliVersion: `@firegoat/feedback@${VERSION}`,
       feedbackType: input.type
     };
-    const data = await this.post("/api/feedback", {
+    const payload = {
       domain: this.domain,
       title,
       body: input.body ?? "",
       meta,
       maskSecrets: input.maskSecrets ?? false
-    });
+    };
+    await this.resolveToken();
+    let data;
+    try {
+      data = await this.post("/api/feedback", payload);
+    } catch (err) {
+      if (!(this.device && isInvalidToken(err)))
+        throw err;
+      await this.device.clear(this.baseUrl);
+      this.token = undefined;
+      await this.registerDevice();
+      data = await this.post("/api/feedback", payload);
+    }
     if (typeof data.id !== "string" || !data.id) {
       throw new FeedbackError(201, "invalid_response", "server response missing id");
     }
     return { id: data.id, similarCount: data.similarCount ?? 0 };
   }
+  async resolveToken() {
+    if (this.token || !this.device)
+      return;
+    this.token = await this.device.load(this.baseUrl);
+    if (this.token)
+      return;
+    try {
+      await this.registerDevice();
+    } catch (err) {
+      if (!(err instanceof FeedbackError && err.code === "rate_limited"))
+        throw err;
+    }
+  }
   async post(path, body) {
     const headers = {
       "content-type": "application/json"
@@ -8387,6 +8643,9 @@ class FeedbackClient {
     return data;
   }
 }
+function isInvalidToken(err) {
+  return err instanceof FeedbackError && err.code === "invalid_token";
+}
 
 // src/feedback.ts
 var TYPES = ["bug", "feature_request", "feedback"];
@@ -8466,9 +8725,9 @@ firegoat ${package_default.version}` : `firegoat ${package_default.version}`,
 }
 
 // src/init.ts
-import { spawnSync as spawnSync2 } from "node:child_process";
+import { spawnSync as spawnSync3 } from "node:child_process";
 import { existsSync as existsSync3, writeFileSync as writeFileSync4 } from "node:fs";
-import { basename as basename2, join as join5 } from "node:path";
+import { basename as basename3, join as join5 } from "node:path";
 import { createInterface } from "node:readline/promises";
 var SKILLS_ARGV = [
   "skills",
@@ -8476,8 +8735,6 @@ var SKILLS_ARGV = [
   "firegoat-ai/skills",
   "--skill",
   "firegoat-onboarding",
-  "--skill",
-  "suggesting-eval-prompts",
   "--agent",
   "claude-code",
   "--copy",
@@ -8491,6 +8748,10 @@ function template(product) {
 # How many times to run each test.
 runs: 1
 
+# Optional: claude's --model. A list runs every test \`runs\` times per model.
+# Unset: claude's default. eval --model NAME overrides it.
+# model: [claude-haiku-5-5, claude-sonnet-5-5]
+
 # Optional: test your local build instead of the published package.
 # setup: runs once per eval, here; what it writes to $FIREGOAT_OUT is put on the
 # agent's PATH. workdir_setup: runs in each fresh folder before the agent starts.
@@ -8514,9 +8775,13 @@ runs: 1
 
 # Optional: env vars for the agent (and for workdir_setup and checks of the same
 # run). The agent gets none of vars:, only these. Placeholders: {{vars.KEY}},
-# {{run.id}}, {{run.n}}, {{run.workdir}}, {{test.name}}. A test can add its own env:.
+# {{run.id}}, {{run.n}}, {{test.name}}, and $FIREGOAT_OUT / $FIREGOAT_WORKDIR (no
+# other $ expansion: $HOME stays literal). A test can add its own env:.
+# Runs go 4 at a time (--parallel), so give a tool with a global home (~/.mytool)
+# a per-run one, as below, or use --parallel 1.
 # env:
-#   MYTOOL_HOME: "/tmp/mytool-{{run.id}}-{{run.n}}"
+#   MYTOOL_HOME: "/tmp/mytool-{{run.id}}-{{test.name}}-{{run.n}}"
+#   UV_FIND_LINKS: "$FIREGOAT_OUT"
 
 tests:
   - name: quickstart
@@ -8545,7 +8810,7 @@ var CLIPBOARD = [
 ];
 function copyToClipboard(text) {
   return CLIPBOARD.some(([cmd, args]) => {
-    const r = spawnSync2(cmd, args, { input: text });
+    const r = spawnSync3(cmd, args, { input: text });
     return !r.error && r.status === 0;
   });
 }
@@ -8592,7 +8857,7 @@ ${INIT_HELP}`);
   const configPath = join5(cwd, CONFIG_FILE);
   const keep = existsSync3(configPath) && !force;
   if (!keep)
-    writeFileSync4(configPath, template(basename2(cwd)));
+    writeFileSync4(configPath, template(basename3(cwd)));
   if (interactive) {
     step(1, CONFIG_FILE, keep ? "kept (--force replaces it)" : "created");
     console.log(paint("90", "  │"));
@@ -8609,7 +8874,7 @@ ${INIT_HELP}`);
       console.log(`installing skills: npx ${SKILLS_ARGV.join(" ")}`);
     const env = { ...process.env };
     delete env.npm_config_package;
-    const r = spawnSync2("npx", ["-y", ...SKILLS_ARGV], {
+    const r = spawnSync3("npx", ["-y", ...SKILLS_ARGV], {
       cwd,
       env,
       stdio: "pipe",
@@ -8617,18 +8882,18 @@ ${INIT_HELP}`);
     });
     if (r.error || r.status !== 0) {
       process.stderr.write(`${r.stdout ?? ""}${r.stderr ?? ""}`);
-      console.error(`firegoat init: installing the skills failed (${r.error?.message ?? `exit ${r.status}`}). Run it yourself: npx ${SKILLS_ARGV.join(" ")}`);
+      console.error(`firegoat init: installing the skill failed (${r.error?.message ?? `exit ${r.status}`}). Run it yourself: npx ${SKILLS_ARGV.join(" ")}`);
       return EXIT.notGraded;
     }
     if (interactive)
       step(3, "firegoat-onboarding skill", "installed → .claude/skills/");
     else
-      console.log("installed skills firegoat-onboarding, suggesting-eval-prompts in .claude/skills/");
-    const ignored = spawnSync2("git", ["check-ignore", "-q", ".claude/skills"], {
+      console.log("installed skill firegoat-onboarding in .claude/skills/");
+    const ignored = spawnSync3("git", ["check-ignore", "-q", ".claude/skills"], {
       cwd
     });
     if (ignored.status === 0)
-      console.error("warning: git ignores .claude/skills, so teammates and CI won't get the skills. Commit them with: git add -f .claude/skills");
+      console.error("warning: git ignores .claude/skills, so teammates and CI won't get the skill. Commit them with: git add -f .claude/skills");
   } else if (interactive)
     step(3, "firegoat-onboarding skill", "skipped");
   if (!interactive) {

package/package.json

@@ -1,13 +1,13 @@
 {
 	"name": "firegoat",
-	"version": "0.13.0",
+	"version": "0.15.0",
 	"repository": {
 		"type": "git",
 		"url": "git+https://github.com/firegoat-ai/firegoat-cli.git"
 	},
 	"devDependencies": {
 		"@biomejs/biome": "^2.5.15",
-		"@firegoat/feedback": "0.0.0",
+		"@firegoat/feedback": "0.2.0",
 		"@types/bun": "latest",
 		"typescript": "^7",
 		"yaml": "^2.9.1"