firegoat 0.14.0 → 0.15.0

package/README.md

@@ -7,7 +7,7 @@ Status: early (v0.11). See `plan.md`.
 ## Quickstart
 
 ```bash
-npx firegoat init     # creates firegoat.yaml and .firegoat/, installs the skills, copies a prompt for your agent
+npx firegoat init     # creates firegoat.yaml and .firegoat/, installs the firegoat-onboarding skill, copies a prompt for your agent
 # edit firegoat.yaml (or paste the prompt into your coding agent and let it)
 npx firegoat eval     # runs your claude on each test, in .firegoat/runs/<runId>/<test>/<n>/work
 ```
@@ -82,6 +82,21 @@ tests:
 npx firegoat eval --var SHA=$(git rev-parse HEAD)
 ```
 
+### Same tests on several models
+
+```yaml
+model: [claude-haiku-5-5, claude-sonnet-5-5]
+runs: 2
+tests:
+  - name: quickstart
+    prompt: Get the acme SDK's hello-world example running and save its output to output.txt.
+    end_goal: output.txt holds the example's output.
+    checks:
+      - run: test -s output.txt
+```
+
+`model:` is passed to claude as `--model`. A list is a matrix: each test runs `runs` times per model, numbered `1..runs×models` (here `quickstart/1`, `2` on haiku, `3`, `4` on sonnet). `npx firegoat eval --model claude-opus-5-5` replaces it for one eval; repeat the flag for a matrix. Unset, firegoat passes no `--model` and claude uses its default. Each run's `run.json` records `model` (what firegoat asked for, or null) and `claudeModel` (what claude reported starting with), and the output labels each run with its model.
+
 ### Env vars for the agent: a home dir per run, a feature flag
 
 ```yaml
@@ -99,19 +114,19 @@ tests:
       - run: ls "$ACME_HOME"/sessions | grep -q .
 ```
 
-`env:` goes to the agent, and to `workdir_setup` and checks of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` (also `${...}`), the same dirs `setup`, `workdir_setup` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `setup` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set.
+`env:` goes to the agent, and to `workdir_setup` and checks of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` (also `${...}`), the same dirs `setup`, `workdir_setup` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `setup` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`.
 
 ## What eval does
 
 - claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers. claude saves its session there, in its own format.
-- No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. In `.firegoat/`, git ignores `runs/` and `.claude/`; keep test fixtures (scripts, sample data) in `.firegoat/fixtures/` and commit them with `firegoat.yaml`.
+- No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. In `.firegoat/`, git ignores everything except `fixtures/`: keep test fixtures (scripts, sample data) in `.firegoat/fixtures/` and commit them with `firegoat.yaml`.
 - In a linked git worktree (`git worktree add`), `.firegoat/runs/` and `.firegoat/.claude/` are in the main checkout, so runs (and their session files) survive `git worktree remove`; `firegoat.yaml`, fixtures and `setup:` stay in the worktree. Each `summary.json` records `dir` (where eval ran) and `git: {branch, commit, dirty}`, and `.firegoat/runs/index.jsonl` gets one line per eval (`runId, startedAt, dir, branch, commit, passed, failed, notGraded, cost`) to tell runs from different branches apart.
 - The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, setup, workdir_setup, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it.
-- Runs (tests × `runs`) go 4 at a time, each in its own folder with its own session; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
+- Runs (tests × models × `runs`) go 4 at a time, each in its own folder with its own session; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
 - Script checks run in the agent's folder afterwards and set the exit code. `end_goal` is not graded by firegoat: it's for you, or your coding agent, to judge from `run.json`, the session and the folder.
 - Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`), `run.json` (status, prompt, end_goal, check results, paths), `work/`, logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
 
-`firegoat.yaml` keys: `runs`, `setup`, `workdir_setup`, `vars`, `env`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
+`firegoat.yaml` keys: `runs`, `model`, `setup`, `workdir_setup`, `vars`, `env`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
 
 Want a plain `npm i acme-sdk` to get your build, with no hint in the prompt? See [`examples/local-npm-registry`](examples/local-npm-registry/README.md).
 
@@ -130,7 +145,7 @@ Something in firegoat got in your way (or your agent's)? Tell us from the termin
 npx firegoat feedback "eval hid claude's spend-limit error" --type bug --body "ran npx firegoat eval, got 'not graded' x3"
 ```
 
-It goes to the firegoat team with the firegoat version, your OS and node version, and which coding agent sent it. Don't include secrets: a report that looks like it has one is refused.
+It goes to the firegoat team with the firegoat version, your OS and node version, and which coding agent sent it. Don't include secrets: a report that looks like it has one is refused. The first send registers your device and saves its token to `~/.config/firegoat/feedback-device.json` (`$XDG_CONFIG_HOME` respected), which raises the daily report limit.
 
 ## Develop
 

package/dist/cli.js

@@ -6947,14 +6947,14 @@ var require_public_api = __commonJS(function(exports) {
 // package.json
 var package_default = {
   name: "firegoat",
-  version: "0.14.0",
+  version: "0.15.0",
   repository: {
     type: "git",
     url: "git+https://github.com/firegoat-ai/firegoat-cli.git"
   },
   devDependencies: {
     "@biomejs/biome": "^2.5.15",
-    "@firegoat/feedback": "0.0.0",
+    "@firegoat/feedback": "0.2.0",
     "@types/bun": "latest",
     typescript: "^7",
     yaml: "^2.9.1"
@@ -7075,6 +7075,7 @@ function parseConfig(text) {
     fail("(root)", "must be a mapping");
   onlyKeys(raw, "(root)", [
     "runs",
+    "model",
     "setup",
     "workdir_setup",
     "vars",
@@ -7082,6 +7083,7 @@ function parseConfig(text) {
     "tests"
   ]);
   const runs = posInt(raw.runs, "runs", 1);
+  const models = parseModels(raw.model);
   const setup = commands(raw.setup, "setup");
   const workdirSetup = commands(raw.workdir_setup, "workdir_setup");
   const vars = parseVars(raw.vars);
@@ -7122,6 +7124,7 @@ function parseConfig(text) {
   });
   return {
     runs,
+    models,
     setup,
     workdirSetup,
     vars,
@@ -7139,6 +7142,22 @@ function commands(raw, path) {
     fail(path, "must be a shell command or a list of them");
   return raw.map((c, i) => str(c, `${path}[${i}]`));
 }
+function parseModels(raw) {
+  if (raw === undefined)
+    return [];
+  if (typeof raw === "string")
+    return [str(raw, "model")];
+  if (!Array.isArray(raw) || raw.length === 0)
+    fail("model", "must be a model name or a non-empty list of them");
+  const seen = new Set;
+  return raw.map((m, i) => {
+    const name = str(m, `model[${i}]`);
+    if (seen.has(name))
+      fail(`model[${i}]`, `duplicate model ${JSON.stringify(name)}`);
+    seen.add(name);
+    return name;
+  });
+}
 var VAR_KEY = /^[A-Za-z_][A-Za-z0-9_]*$/;
 function parseVars(raw) {
   if (raw === undefined)
@@ -7175,6 +7194,8 @@ function parseEnv(raw, path) {
     const p = `${path}.${k}`;
     if (!VAR_KEY.test(k))
       fail(p, "name must be letters, digits and _, not starting with a digit");
+    if (k === "ANTHROPIC_MODEL")
+      fail(p, "is reserved: choose the model with top-level model: (or eval --model)");
     if (RESERVED_ENV.has(k) || k.startsWith("FIREGOAT_"))
       fail(p, "is set by firegoat and cannot be overridden (reserved: PATH, CLAUDE_CONFIG_DIR, CLAUDE_SECURESTORAGE_CONFIG_DIR, FIREGOAT_*)");
     if (typeof v !== "string")
@@ -7298,7 +7319,12 @@ async function runClaude(opts) {
   let pending = "";
   const exited = await run({
     cmd: opts.bin,
-    args: ["-p", opts.prompt, ...CLAUDE_FLAGS],
+    args: [
+      "-p",
+      opts.prompt,
+      ...CLAUDE_FLAGS,
+      ...opts.model === undefined ? [] : ["--model", opts.model]
+    ],
     cwd: opts.cwd,
     env: opts.env,
     timeoutMs: opts.timeoutMs,
@@ -7322,6 +7348,7 @@ async function runClaude(opts) {
   if (exited.error)
     return {
       events: [],
+      claudeModel: null,
       ok: false,
       reason: `could not start ${opts.bin}: ${exited.error.message}`
     };
@@ -7339,15 +7366,18 @@ async function runClaude(opts) {
 `)[0]?.slice(0, 300);
     return {
       events,
+      claudeModel: null,
       ok: false,
       reason: `claude did not start a session (exit ${exited.code ?? "by signal"})${said ? `: ${said}` : ", see stderr.txt"}`
     };
   }
   const sessionId = String(init.session_id);
+  const claudeModel = typeof init.model === "string" ? init.model : null;
   const result = events.findLast((e) => e?.type === "result");
   if (exited.timedOut)
     return {
       events,
+      claudeModel,
       ok: false,
       reason: `timed out after ${opts.timeoutMs / 1000}s`,
       result,
@@ -7356,6 +7386,7 @@ async function runClaude(opts) {
   if (!result)
     return {
       events,
+      claudeModel,
       ok: false,
       reason: `claude exited ${exited.code ?? "by signal"} without a result, see stderr.txt`,
       sessionId
@@ -7363,12 +7394,13 @@ async function runClaude(opts) {
   if (result.is_error === true || result.subtype !== "success")
     return {
       events,
+      claudeModel,
       ok: false,
       reason: `claude result is an error (subtype=${result.subtype})${result.result ? `: ${result.result.slice(0, 300)}` : ""}`,
       result,
       sessionId
     };
-  return { events, ok: true, result, sessionId };
+  return { events, claudeModel, ok: true, result, sessionId };
 }
 async function runShell(opts) {
   opts.log.write(`$ ${opts.run}
@@ -7420,8 +7452,9 @@ var EXIT = { pass: 0, fail: 1, notGraded: 2, usage: 3 };
 // src/firegoat-dir.ts
 import { mkdirSync, writeFileSync } from "node:fs";
 import { join } from "node:path";
-var GITIGNORE = `runs/
-.claude/
+var GITIGNORE = `*
+!fixtures/
+!fixtures/**
 `;
 function ensureFiregoatDir(root) {
   const dir = join(root, ".firegoat");
@@ -7487,7 +7520,7 @@ function topHelp(version) {
 folder, with none of your claude config; firegoat saves the session and runs your
 script checks.
 
-  npx firegoat init     create firegoat.yaml and .firegoat/, install the skills
+  npx firegoat init     create firegoat.yaml and .firegoat/, install the skill
   npx firegoat eval     run the tests (claude must be logged in: run \`claude\` once)
   npx firegoat feedback "<what got in the way>"   tell the firegoat team
 
@@ -7504,16 +7537,20 @@ ${EXIT_CODES}`;
 var INIT_HELP = `Usage: firegoat init [--force] [--no-skills]
 
 In the current folder: creates .firegoat/ and a starter firegoat.yaml, installs the
-firegoat-onboarding and suggesting-eval-prompts skills into .claude/skills/ (npx skills
+firegoat-onboarding skill into .claude/skills/ (npx skills
 add firegoat-ai/skills; asks first in a terminal), then prints a prompt to paste into
-your coding agent. In .firegoat/, git ignores runs/ and .claude/; keep test fixtures
+your coding agent. In .firegoat/, git ignores everything but fixtures/: keep test fixtures
 (helper scripts, sample data) in .firegoat/fixtures/ and commit them.
 
   --force       replace an existing firegoat.yaml
-  --no-skills   don't install the skills
+  --no-skills   don't install the skill
 
 firegoat.yaml reference:
   runs: 1                     runs per test
+  model: <name> | [<name>, ...]
+                              claude --model; a list runs every test runs times
+                              per model (n counts 1..runs x models). Unset: no
+                              --model, claude's default. eval --model overrides
   setup: <sh> | [<sh>, ...]   once per eval, in this folder; what it writes to
                               $FIREGOAT_OUT is first on the agent's PATH
   workdir_setup: <sh> | [...] in each fresh folder before the agent starts
@@ -7526,7 +7563,11 @@ firegoat.yaml reference:
                               (no other $ expansion: $HOME stays literal).
                               setup gets only the entries that use nothing but
                               {{vars.*}}, {{run.id}} and $FIREGOAT_OUT.
-                              PATH, CLAUDE_CONFIG_DIR and FIREGOAT_* are reserved.
+                              PATH, CLAUDE_CONFIG_DIR, FIREGOAT_* and
+                              ANTHROPIC_MODEL (use model:) are reserved.
+                              Runs go 4 at a time: give a tool with a global home
+                              a per-run one (or use eval --parallel 1), e.g.
+                    MYTOOL_HOME: "/tmp/mytool-{{run.id}}-{{test.name}}-{{run.n}}"
   tests:
     - name: quickstart        letters, digits, '.', '_', '-'; unique
       prompt: ...             what a user would ask the agent; one concrete outcome
@@ -7565,9 +7606,9 @@ Example:
         - run: |
             ! grep -Eq '$ .*npm (install|i|add) .*acme' "$FIREGOAT_TRANSCRIPT"
 
-Exit codes: 0 done, 2 the skills install failed, 3 usage error.`;
-var EVAL_HELP = `Usage: firegoat eval [test..] [--runs N] [--parallel N] [--var KEY=value]..
-                     [--json] [--claude-bin PATH]
+Exit codes: 0 done, 2 the skill install failed, 3 usage error.`;
+var EVAL_HELP = `Usage: firegoat eval [test..] [--runs N] [--model NAME].. [--parallel N]
+                     [--var KEY=value].. [--json] [--claude-bin PATH]
 
 Runs each test in firegoat.yaml (in the current folder): your claude gets the prompt
 in a fresh, empty folder, .firegoat/runs/<runId>/<test>/<n>/work, with setup:'s
@@ -7584,7 +7625,11 @@ fixtures and setup: stay in the current folder.
 
   test..             only these tests (by name)
   --runs N           runs per test (overrides runs: in firegoat.yaml)
-  --parallel N       runs at once (default 4; 1 runs them one after another)
+  --model NAME       claude --model, replacing model: in firegoat.yaml; repeat it
+                     (--model a --model b) to run each test runs times per model
+  --parallel N       runs at once (default 4; 1 runs them one after another).
+                     Tools with global on-disk state collide: give each run its
+                     own home via env: ({{run.id}}-{{test.name}}-{{run.n}}) or use 1
   --var KEY=value    fills {{vars.KEY}}; env for setup, workdir_setup and checks
   --json             the summary as JSON on stdout (or {error, exitCode} if eval
                      stops early); progress goes to stderr
@@ -7595,8 +7640,10 @@ Output: .firegoat/runs/<runId>/
                           git {branch, commit, dirty} (no git outside a repo)
   ../index.jsonl          one line per eval: runId, startedAt, dir, branch,
                           commit, passed, failed, notGraded, cost
-  <test>/<n>/run.json     one run: status and reason (from the checks), prompt,
-                          end_goal, checks [{run, code}], and paths: workdir,
+  <test>/<n>/run.json     one run: status and reason (from the checks), model
+                          (asked for, or null) and claudeModel (what claude's
+                          init line reported), prompt, end_goal,
+                          checks [{run, code}], and paths: workdir,
                           transcript, session (claude's session file), stderr,
                           checksLog
   <test>/<n>/transcript.txt  what the agent did: each tool call with its command,
@@ -7610,6 +7657,8 @@ yours to judge: read transcript.txt first (a few KB); the session file has every
 Examples:
   npx firegoat eval --json                  all tests, JSON summary on stdout
   npx firegoat eval quickstart --runs 3     one test, three runs
+  npx firegoat eval --model claude-haiku-5-5 --model claude-sonnet-5-5
+                                            each test on both models
   npx firegoat eval --var SHA=$(git rev-parse HEAD)
 
 A run can take minutes (default timeout 600s per test).
@@ -7627,7 +7676,9 @@ title, then what you ran and what happened in --body (- reads stdin).
 
 Sent along: the firegoat version, your OS and node version, and which agent you are
 (claude-code/codex/cursor, from the environment). Don't include secrets: a report that
-looks like it contains one is refused.
+looks like it contains one is refused. The first send registers this device and saves
+its token to ~/.config/firegoat/feedback-device.json ($XDG_CONFIG_HOME respected),
+which raises the daily report limit.
 
 Example:
   npx firegoat feedback "checks can't grep commands with quotes" --type bug \\
@@ -7849,7 +7900,8 @@ function plainUI(write) {
         printed.add(r);
         const label = r.state === "error" ? "ERROR" : r.state.toUpperCase();
         const u = `$${(r.cost ?? 0).toFixed(2)}, ${r.turns} turns, ${r.seconds ?? 0}s`;
-        write(`${label} ${r.test}#${r.n} ${r.reason} (${u})`);
+        const model = r.model === null ? "" : ` [${r.model}]`;
+        write(`${label} ${r.test}#${r.n}${model} ${r.reason} (${u})`);
       }
     },
     done: ({ runsDir, rows: all }) => {
@@ -7916,10 +7968,12 @@ function ttyUI(out, opts) {
       const mark = setup.state === "ok" ? p(C.green, "✓") : setup.state === "fail" ? p(C.red, "✗") : p(C.ember, SPIN[frame % SPIN.length]);
       out.push(`  ${setup.state === "running" ? p(C.ember, "◐") : p(C.green, "●")}  setup${" ".repeat(Math.max(1, w - 20))}${mark}${setup.seconds !== undefined ? ` ${setup.seconds}s` : ""}`);
     }
+    const modelCols = Math.min(24, Math.max(0, ...rows.map((r) => r.model?.length ?? 0)));
     rows.forEach((r, i) => {
       out.push(p(C.gray, "  │"));
       const num = String(i + 1).padStart(2, "0");
       const name = cut(`${r.test}${r.n > 1 ? `#${r.n}` : ""}`, 16).padEnd(16);
+      const model = modelCols > 0 ? ` ${p(C.cyan, cut(r.model ?? "", modelCols).padEnd(modelCols))}` : "";
       let detail = "";
       if (r.state === "running")
         detail = p(C.gray, `turn ${r.turns}`);
@@ -7927,7 +7981,7 @@ function ttyUI(out, opts) {
         detail = `${checksPassed(r).padEnd(11)} ${p(C.gray, usage(r))}`;
       else if (r.state === "error")
         detail = p(C.gray, usage(r));
-      out.push(`  ${glyph(r)}  ${p(C.gray, num)}  ${p(C.bold, name)} ${p(C.ember, ramp(r))} ${statusWord(r)}  ${detail}`);
+      out.push(`  ${glyph(r)}  ${p(C.gray, num)}  ${p(C.bold, name)}${model} ${p(C.ember, ramp(r))} ${statusWord(r)}  ${detail}`);
       if (r.state === "running" && r.command)
         out.push(p(C.gray, `  ┊        └ $ ${cut(r.command, w - 16)}`));
       if (r.state === "fail")
@@ -8026,6 +8080,7 @@ class UsageError extends Error {
 function parseArgs(args) {
   const opts = {
     names: [],
+    models: [],
     parallel: 4,
     json: false,
     vars: {},
@@ -8059,6 +8114,13 @@ function parseArgs(args) {
       if (!/^\d+$/.test(v) || n < 1)
         throw new UsageError(`--runs must be a positive integer, got ${v}`);
       opts.runs = n;
+    } else if (flag === "--model") {
+      const v = value();
+      if (v.trim() === "")
+        throw new UsageError("--model needs a model name");
+      if (opts.models.includes(v))
+        throw new UsageError(`--model ${v} given twice`);
+      opts.models.push(v);
     } else if (flag === "--parallel") {
       const v = value();
       const n = Number(v);
@@ -8085,6 +8147,10 @@ function findSession(configDir, sessionId) {
   }
   return null;
 }
+function grepsSessionJson(run) {
+  const unescaped = run.replace(/\\/g, "");
+  return unescaped.includes("FIREGOAT_SESSION") && /"command":"|"name":"Write"/.test(unescaped);
+}
 function newRunId() {
   const ts = new Date().toISOString().replace(/\.\d+Z$/, "Z").replace(/[-:]/g, "");
   return `${ts}-${randomBytes(2).toString("hex")}`;
@@ -8098,6 +8164,8 @@ async function runOne(opts) {
   const r = {
     test: test.name,
     n,
+    model: opts.model,
+    claudeModel: null,
     status: "error",
     reason: "",
     prompt: render(test.prompt, opts.vars),
@@ -8148,14 +8216,16 @@ async function runOne(opts) {
     env: { ...claudeEnv(opts.configDir, env), PATH: opts.stepEnv.PATH },
     stderr,
     timeoutMs: test.timeout * 1000,
+    model: opts.model ?? undefined,
     onEvent: opts.onEvent
   });
+  r.claudeModel = outcome.claudeModel;
   await new Promise((res) => stderr.end(res));
   if (outcome.sessionId)
     r.session = findSession(opts.configDir, outcome.sessionId);
   writeTranscript({
     path: r.transcript,
-    title: `${test.name}#${n}`,
+    title: `${test.name}#${n}${opts.model ? ` (model ${opts.model})` : ""}`,
     prompt: r.prompt,
     events: outcome.events,
     workdir
@@ -8227,7 +8297,10 @@ async function evalCommand(args, root) {
       return fatal(EXIT.usage, `unknown test ${unknown.map((n) => JSON.stringify(n)).join(", ")} (known: ${[...known].join(", ")})`);
     tests = tests.filter((t) => opts.names.includes(t.name));
   }
+  if (tests.some((t) => t.checks.some((c) => grepsSessionJson(c.run))))
+    console.error('firegoat eval: warning: a check greps $FIREGOAT_SESSION for session JSON ("command":" / "name":"Write"), which breaks on escaping. Grep $FIREGOAT_TRANSCRIPT instead: one "[n] $ <command>" line per shell call, "[n] Write <path>" per file write.');
   const runs = opts.runs ?? config.runs;
+  const models = opts.models.length > 0 ? opts.models : config.models;
   const vars = { ...config.vars, ...opts.vars };
   const missing = [
     ...new Set(tests.flatMap((t) => [
@@ -8270,12 +8343,14 @@ async function evalCommand(args, root) {
     FIREGOAT_OUT: out,
     PATH: [out, process.env.PATH ?? ""].join(delimiter2)
   };
-  const rows = tests.flatMap((t) => Array.from({ length: runs }, (_, i) => ({
+  const matrix = models.length > 0 ? models : [null];
+  const rows = tests.flatMap((t) => matrix.flatMap((model, m) => Array.from({ length: runs }, (_, i) => ({
     test: t.name,
-    n: i + 1,
+    n: m * runs + i + 1,
+    model,
     state: "queued",
     turns: 0
-  })));
+  }))));
   if (config.setup.length > 0) {
     const setupStarted = Date.now();
     ui.setup("running");
@@ -8318,6 +8393,7 @@ async function evalCommand(args, root) {
       },
       test,
       n: row.n,
+      model: row.model,
       runId,
       runDir,
       claudeBin: claudePath,
@@ -8385,7 +8461,68 @@ async function evalCommand(args, root) {
 import { readFileSync as readFileSync2 } from "node:fs";
 
 // node_modules/@firegoat/feedback/dist/index.js
-var VERSION = "0.0.0";
+function memoryDeviceStore() {
+  const tokens = {};
+  return {
+    load: async (baseUrl) => tokens[baseUrl],
+    save: async (baseUrl, token) => {
+      tokens[baseUrl] = token;
+    },
+    clear: async (baseUrl) => {
+      delete tokens[baseUrl];
+    }
+  };
+}
+async function defaultDevicePath() {
+  const { homedir } = await import("node:os");
+  const { join } = await import("node:path");
+  const configHome = process.env.XDG_CONFIG_HOME || join(homedir(), ".config");
+  return join(configHome, "firegoat", "feedback-device.json");
+}
+function fileDeviceStore(path) {
+  const resolvePath = async () => path ?? await defaultDevicePath();
+  async function read() {
+    const { readFile } = await import("node:fs/promises");
+    const file = await resolvePath();
+    let text;
+    try {
+      text = await readFile(file, "utf8");
+    } catch (err) {
+      if (err.code === "ENOENT")
+        return {};
+      throw err;
+    }
+    try {
+      return JSON.parse(text);
+    } catch {
+      throw new Error(`@firegoat/feedback: ${file} is not valid JSON; delete it to register a new device`);
+    }
+  }
+  async function write(tokens) {
+    const { mkdir, writeFile } = await import("node:fs/promises");
+    const { dirname } = await import("node:path");
+    const file = await resolvePath();
+    await mkdir(dirname(file), { recursive: true });
+    await writeFile(file, `${JSON.stringify(tokens, null, 2)}
+`, {
+      mode: 384
+    });
+  }
+  return {
+    load: async (baseUrl) => (await read())[baseUrl],
+    save: async (baseUrl, token) => write({ ...await read(), [baseUrl]: token }),
+    clear: async (baseUrl) => {
+      const tokens = await read();
+      delete tokens[baseUrl];
+      await write(tokens);
+    }
+  };
+}
+function defaultDeviceStore() {
+  const isNode = Boolean(globalThis.process?.versions?.node);
+  return isNode ? fileDeviceStore() : memoryDeviceStore();
+}
+var VERSION = "0.2.0";
 var DEFAULT_BASE_URL = "https://firegoat.ai/evals";
 var TIMEOUT_MS = 15000;
 function envBaseUrl() {
@@ -8414,6 +8551,7 @@ class FeedbackClient {
   baseUrl;
   token;
   fetchImpl;
+  device;
   constructor(options) {
     if (!options.domain?.trim()) {
       throw new Error("FeedbackClient: `domain` is required (your product's domain, e.g. example.com)");
@@ -8422,6 +8560,7 @@ class FeedbackClient {
     this.baseUrl = (options.baseUrl ?? envBaseUrl() ?? DEFAULT_BASE_URL).replace(/\/+$/, "");
     this.token = options.token;
     this.fetchImpl = options.fetch ?? fetch;
+    this.device = options.token || options.device === false ? undefined : options.device ?? defaultDeviceStore();
   }
   async registerDevice() {
     const data = await this.post("/api/feedback/device", undefined);
@@ -8429,6 +8568,7 @@ class FeedbackClient {
       throw new FeedbackError(201, "invalid_response", "server response missing token");
     }
     this.token = data.token;
+    await this.device?.save(this.baseUrl, data.token);
     return data.token;
   }
   async send(input) {
@@ -8441,18 +8581,43 @@ class FeedbackClient {
       cliVersion: `@firegoat/feedback@${VERSION}`,
       feedbackType: input.type
     };
-    const data = await this.post("/api/feedback", {
+    const payload = {
       domain: this.domain,
       title,
       body: input.body ?? "",
       meta,
       maskSecrets: input.maskSecrets ?? false
-    });
+    };
+    await this.resolveToken();
+    let data;
+    try {
+      data = await this.post("/api/feedback", payload);
+    } catch (err) {
+      if (!(this.device && isInvalidToken(err)))
+        throw err;
+      await this.device.clear(this.baseUrl);
+      this.token = undefined;
+      await this.registerDevice();
+      data = await this.post("/api/feedback", payload);
+    }
     if (typeof data.id !== "string" || !data.id) {
       throw new FeedbackError(201, "invalid_response", "server response missing id");
     }
     return { id: data.id, similarCount: data.similarCount ?? 0 };
   }
+  async resolveToken() {
+    if (this.token || !this.device)
+      return;
+    this.token = await this.device.load(this.baseUrl);
+    if (this.token)
+      return;
+    try {
+      await this.registerDevice();
+    } catch (err) {
+      if (!(err instanceof FeedbackError && err.code === "rate_limited"))
+        throw err;
+    }
+  }
   async post(path, body) {
     const headers = {
       "content-type": "application/json"
@@ -8478,6 +8643,9 @@ class FeedbackClient {
     return data;
   }
 }
+function isInvalidToken(err) {
+  return err instanceof FeedbackError && err.code === "invalid_token";
+}
 
 // src/feedback.ts
 var TYPES = ["bug", "feature_request", "feedback"];
@@ -8567,8 +8735,6 @@ var SKILLS_ARGV = [
   "firegoat-ai/skills",
   "--skill",
   "firegoat-onboarding",
-  "--skill",
-  "suggesting-eval-prompts",
   "--agent",
   "claude-code",
   "--copy",
@@ -8582,6 +8748,10 @@ function template(product) {
 # How many times to run each test.
 runs: 1
 
+# Optional: claude's --model. A list runs every test \`runs\` times per model.
+# Unset: claude's default. eval --model NAME overrides it.
+# model: [claude-haiku-5-5, claude-sonnet-5-5]
+
 # Optional: test your local build instead of the published package.
 # setup: runs once per eval, here; what it writes to $FIREGOAT_OUT is put on the
 # agent's PATH. workdir_setup: runs in each fresh folder before the agent starts.
@@ -8607,8 +8777,10 @@ runs: 1
 # run). The agent gets none of vars:, only these. Placeholders: {{vars.KEY}},
 # {{run.id}}, {{run.n}}, {{test.name}}, and $FIREGOAT_OUT / $FIREGOAT_WORKDIR (no
 # other $ expansion: $HOME stays literal). A test can add its own env:.
+# Runs go 4 at a time (--parallel), so give a tool with a global home (~/.mytool)
+# a per-run one, as below, or use --parallel 1.
 # env:
-#   MYTOOL_HOME: "/tmp/mytool-{{run.id}}-{{run.n}}"
+#   MYTOOL_HOME: "/tmp/mytool-{{run.id}}-{{test.name}}-{{run.n}}"
 #   UV_FIND_LINKS: "$FIREGOAT_OUT"
 
 tests:
@@ -8710,18 +8882,18 @@ ${INIT_HELP}`);
     });
     if (r.error || r.status !== 0) {
       process.stderr.write(`${r.stdout ?? ""}${r.stderr ?? ""}`);
-      console.error(`firegoat init: installing the skills failed (${r.error?.message ?? `exit ${r.status}`}). Run it yourself: npx ${SKILLS_ARGV.join(" ")}`);
+      console.error(`firegoat init: installing the skill failed (${r.error?.message ?? `exit ${r.status}`}). Run it yourself: npx ${SKILLS_ARGV.join(" ")}`);
       return EXIT.notGraded;
     }
     if (interactive)
       step(3, "firegoat-onboarding skill", "installed → .claude/skills/");
     else
-      console.log("installed skills firegoat-onboarding, suggesting-eval-prompts in .claude/skills/");
+      console.log("installed skill firegoat-onboarding in .claude/skills/");
     const ignored = spawnSync3("git", ["check-ignore", "-q", ".claude/skills"], {
       cwd
     });
     if (ignored.status === 0)
-      console.error("warning: git ignores .claude/skills, so teammates and CI won't get the skills. Commit them with: git add -f .claude/skills");
+      console.error("warning: git ignores .claude/skills, so teammates and CI won't get the skill. Commit them with: git add -f .claude/skills");
   } else if (interactive)
     step(3, "firegoat-onboarding skill", "skipped");
   if (!interactive) {

package/package.json

@@ -1,13 +1,13 @@
 {
 	"name": "firegoat",
-	"version": "0.14.0",
+	"version": "0.15.0",
 	"repository": {
 		"type": "git",
 		"url": "git+https://github.com/firegoat-ai/firegoat-cli.git"
 	},
 	"devDependencies": {
 		"@biomejs/biome": "^2.5.15",
-		"@firegoat/feedback": "0.0.0",
+		"@firegoat/feedback": "0.2.0",
 		"@types/bun": "latest",
 		"typescript": "^7",
 		"yaml": "^2.9.1"