firegoat 0.18.0 → 0.20.0

package/README.md

@@ -7,12 +7,12 @@ Status: early. See `plan.md`.
 ## Quickstart
 
 ```bash
-npx firegoat init     # Creates firegoat.yaml and .firegoat/, installs the firegoat-onboarding skill, copies a prompt for your agent
+npx firegoat@latest init     # Creates firegoat.yaml and .firegoat/, installs the firegoat-onboarding skill, copies a prompt for your agent
 # Edit firegoat.yaml (or paste the prompt into your coding agent and let it)
-npx firegoat eval     # Runs your claude on each test, in .firegoat/runs/<runId>/<test>/<n>/work
+npx firegoat@latest eval     # Runs your claude on each test, in .firegoat/runs/<runId>/<test>/<n>/work
 ```
 
-That's it: two commands. `claude` must be installed and logged in (run `claude` once).
+That's it: two commands. Use `npx firegoat@latest`: plain `npx firegoat` can keep running an old cached version. `eval` prints its version first (`firegoat 0.18.0`), and `summary.json` records it as `firegoatVersion`. `claude` must be installed and logged in (run `claude` once).
 
 `init` installs the firegoat-onboarding skill globally, into `~/.claude/skills/` (in a terminal it asks: global, this repo, or none). `--local-skills` puts it in this repo's `.claude/skills/` instead (commit it so teammates and CI get it); `--no-skills` skips it.
 
@@ -98,6 +98,21 @@ tests:
 npx firegoat eval --var SHA=$(git rev-parse HEAD)
 ```
 
+### Start from a template project
+
+```yaml
+build: cp -R fixtures/starter-app "$FIREGOAT_OUT/starter-app"
+install: cp -R "$FIREGOAT_OUT/starter-app/." .
+tests:
+  - name: add-auth
+    prompt: Add acme login to this app. Keep the existing pages working.
+    end_goal: The app signs users in with acme, and the existing pages still render.
+    checks:
+      - run: npm test
+```
+
+No template feature needed: keep the project committed in `fixtures/starter-app/`, `build` copies it once per eval, and `install` copies it into each fresh folder, so every run begins in its own copy of that project.
+
 ### Same tests on several models
 
 ```yaml
@@ -113,11 +128,56 @@ tests:
 
 `model:` is passed to claude as `--model`. A list is a matrix: each test runs `runs` times per model, numbered `1..runs×models` (here `quickstart/1`, `2` on haiku, `3`, `4` on sonnet). `npx firegoat eval --model claude-opus-5-5` replaces it for one eval; repeat the flag for a matrix. Unset, firegoat passes no `--model` and claude uses its default. Each run's `run.json` records `model` (what firegoat asked for, or null) and `claudeModel` (what claude reported starting with), and the output labels each run with its model.
 
-### Env vars for the agent: a home dir per run, a feature flag
+### Compare two setups
+
+```yaml
+build: npm pack --pack-destination "$FIREGOAT_OUT"
+tests:
+  - name: deploy-local-build
+    prompt: &deploy Deploy this app to an acme preview environment and print the URL.
+    end_goal: &deployed The app runs on an acme preview URL, and the URL is printed.
+    install: npm install "$FIREGOAT_OUT"/acme-*.tgz
+  - name: deploy-npm-release
+    prompt: *deploy
+    end_goal: *deployed
+    install: npm install acme@latest
+```
+
+There is no variants feature: to compare setups (your local build vs the npm release, with vs without `--chrome`, docs A vs docs B), write one test per setup with the same prompt and a different per-test `install:`, `env:` or `claude_args:`. A YAML anchor (`&deploy`) and alias (`*deploy`) keep the prompt in one place. Skills installed into the run's folder are not loaded as skills: eval runs claude with `--setting-sources ""` and `--disable-slash-commands`, so a with/without-skill comparison isn't possible yet.
+
+### Extra claude flags
+
+```yaml
+claude_args: [--plugin-dir, ./plugins/acme]   # Every test
+tests:
+  - name: dashboard-key
+    prompt: Get an acme API key from the acme dashboard (I'm logged in in Chrome) and put it in .env as ACME_API_KEY.
+    end_goal: .env holds a key copied from the dashboard, not a made-up one.
+    claude_args: [--chrome]                    # Claude in Chrome, this test only
+```
+
+`claude_args:` (top level, and per test after the top-level ones) is passed to claude after firegoat's own flags and `--model`. Each entry is one argument, as-is, with no shell splitting: `[--plugin-dir, ./plugins/acme]`, not `["--plugin-dir ./plugins/acme"]`. Use it for `--chrome`, `--plugin-dir`, `--mcp-config`, `--agents`, `--append-system-prompt`. The flags firegoat sets itself are config errors, also as `--flag=value`: `-p`, `--print`, `--output-format`, `--verbose`, `--setting-sources`, `--strict-mcp-config`, `--disable-slash-commands`, `--dangerously-skip-permissions`, `--model` (use `model:`), and `--session-id`, `--resume`, `-r`, `--continue`, `-c` (every run is a fresh session). Entries expand like `env:` values: `$FIREGOAT_WORKDIR`, `$FIREGOAT_HOME`, `$FIREGOAT_OUT` (also `${...}`), `{{vars.KEY}}`, `{{run.id}}`, `{{run.n}}`, `{{test.name}}`; no other `$` expansion. Each run's `run.json` records the expanded `claudeArgs`.
+
+### Testing an MCP server
+
+```yaml
+install: |
+  cat > .mcp.json <<EOF
+  {"mcpServers": {"acme": {"command": "npx", "args": ["-y", "acme-mcp"],
+    "env": {"HOME": "$FIREGOAT_HOME"}}}}
+  EOF
+claude_args: [--mcp-config, .mcp.json]
+```
+
+Write the MCP config per run in `install:` (or copy a committed one to `$FIREGOAT_OUT` in `build:` and pass `"$FIREGOAT_OUT/mcp.json"`, but then it can't hold the run's home), and pass it with `claude_args: [--mcp-config, .mcp.json]`: relative paths are from the run's folder, claude's cwd. `--strict-mcp-config` is always on, so only the configs you pass load. MCP servers that claude starts inherit claude's own env, which keeps your real `HOME` (claude needs it for its login): set `"env": {"HOME": "<the run's $FIREGOAT_HOME>"}` on the server, as above, or it writes to your real home.
+
+With `--chrome`, claude points the Claude in Chrome extension's native host (`~/Library/Application Support/<browser>/NativeMessagingHosts/com.anthropic.claude_code_browser_extension.json`) at `.firegoat/.claude`. If you delete `.firegoat/`, run a normal `claude --chrome` once to point it back.
+
+### Env vars for the agent: a feature flag, a tool that ignores HOME
 
 ```yaml
 env:
-  ACME_HOME: "/tmp/acme-{{run.id}}-{{run.n}}"   # Sessions/config never collide across runs
+  ACME_HOME: "$FIREGOAT_HOME/.acme"             # Only for a tool that ignores HOME
   ACME_TELEMETRY: "off"
   UV_FIND_LINKS: "$FIREGOAT_OUT"                # Resolve the wheel build: made
 tests:
@@ -130,21 +190,35 @@ tests:
       - run: ls "$ACME_HOME"/sessions | grep -q .
 ```
 
-`env:` goes to the agent, and to `install`, checks and `cleanup` of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` (also `${...}`), the same dirs `build`, `install` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `build` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`.
+`env:` goes to the agent, and to `install`, checks and `cleanup` of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` / `$FIREGOAT_HOME` (also `${...}`), the same dirs `build`, `install` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `build` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`. `HOME`, `XDG_CONFIG_HOME`, `XDG_CACHE_HOME`, `XDG_DATA_HOME`, `XDG_STATE_HOME` and `CLAUDE_ENV_FILE` are reserved too: every run gets its own home.
+
+### Each run gets its own HOME
+
+Every run has its own home, `.firegoat/runs/<runId>/<test>/<n>/home/`, with `.config`, `.cache`, `.local/share` and `.local/state` for the XDG dirs. A tool that keeps logins, config or caches in `~` starts fresh in every run, and parallel runs don't collide. Point a tool's own `*_HOME` var at `$FIREGOAT_HOME` only if the tool ignores `HOME`.
+
+- The agent's shell commands see it: firegoat writes `<n>/home.env` (exporting `HOME` and `XDG_CONFIG_HOME`, `XDG_CACHE_HOME`, `XDG_DATA_HOME`, `XDG_STATE_HOME`) and passes it to claude as `CLAUDE_ENV_FILE`, which claude sources before every Bash command. claude itself keeps your real `HOME`, where its login is (the macOS keychain lookup, `~/.claude/.credentials.json` on Linux).
+- `install`, checks and `cleanup` get `HOME` and the XDG vars set to the run's home, plus `$FIREGOAT_HOME`. `build` keeps your real home.
+- `run.json` records it as `home`.
+
+Limits:
 
-Setting `HOME`: on macOS claude reads its login from `$HOME/Library`, so with a per-run `HOME` every run says "Not logged in" unless `install` runs `ln -s <real home>/Library "$HOME/Library"`. Get the real home in `build`, which doesn't get entries using `{{test.name}}` or `{{run.n}}` (e.g. `build: echo "$HOME" > "$FIREGOAT_OUT/.real-home"`).
+- claude's own Read/Write/Edit tools expand `~` to your real home; only Bash commands get the run's home.
+- Lookups by user id (`os.userInfo().homedir`, python's `pwd`, `getpwuid`) still return your real home.
+- Keychain-backed logins (`gh`, `docker`, git credential helpers) look logged out inside a run. Log tools in from `install:`, or pass tokens with `env:`.
+- The fresh home has no `.gitconfig`: if the agent commits, set `git config --global user.name ...` / `user.email` in `install:`.
+- claude's shell snapshot still sources your real `~/.zshrc` (or `~/.bashrc`).
 
 ## What eval does
 
-- claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers. claude saves its session there, in its own format.
+- claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers (add flags with `claude_args:`, see above). claude saves its session there, in its own format.
 - No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. `.firegoat/` holds local run output and ignores itself in git. Keep test fixtures (scripts, sample data) outside it, e.g. in `fixtures/` next to `firegoat.yaml`, and commit them.
 - In a linked git worktree (`git worktree add`), `.firegoat/runs/` and `.firegoat/.claude/` are in the main checkout, so runs (and their session files) survive `git worktree remove`; `firegoat.yaml`, fixtures and `build:` stay in the worktree. Each `summary.json` records `dir` (where eval ran) and `git: {branch, commit, dirty}`, and `.firegoat/runs/index.jsonl` gets one line per eval (`runId, startedAt, dir, branch, commit, passed, failed, notGraded, cost, uploaded, uploadFailed`) to tell runs from different branches apart.
-- The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, build, install, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it. Every `FIREGOAT_*` var firegoat was started with (`FIREGOAT_API_KEY` above all) is removed too.
-- Runs (tests × models × `runs`) go 4 at a time, each in its own folder with its own session; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
+- The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, build, install, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it. Every `FIREGOAT_*` var firegoat was started with (`FIREGOAT_API_KEY` above all) is removed too, and so are the vars a parent Claude Code session sets (when firegoat runs from claude, or its agent runs firegoat), so the claude under test doesn't act as a nested session: `CLAUDECODE`, `CLAUDE_CODE_*`, `CLAUDE_PID`, `CLAUDE_ENV_FILE`, `CLAUDE_SESSION_ID`, `CLAUDE_PROJECT_DIR`, `CLAUDE_EFFORT`, `CLAUDE_AGENT_SDK_*`, `CLAUDE_PLUGIN_ROOT`, `CLAUDE_PLUGIN_DATA`, `CLAUDE_SKILL_DIR`. Login and provider vars stay: `ANTHROPIC_*`, `CLAUDE_CODE_OAUTH_TOKEN`, `CLAUDE_CODE_OAUTH_REFRESH_TOKEN`, `CLAUDE_CODE_USE_*` (Bedrock, Vertex, ...), `CLAUDE_CODE_SKIP_*_AUTH`, `CLAUDE_CODE_CLIENT_CERT`, `CLAUDE_CODE_CLIENT_KEY`, `CLAUDE_CODE_CLIENT_KEY_PASSPHRASE`.
+- Runs (tests × models × `runs`) go 4 at a time, each in its own folder with its own session and HOME; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
 - Script checks run in the agent's folder afterwards and set the exit code. `end_goal` is not graded by firegoat: it's for you, or your coding agent, to judge from `run.json`, the session and the folder.
-- Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`; greppable line by line: every line of a shell call starts with `[n] $ `, every line of the agent's text and final answer with `agent: `, output is indented, so `grep -E '^\[[0-9]+\] \$ .*npm install'` matches only commands, also inside heredocs and multi-line scripts), `run.json` (status, prompt, end_goal, check results, paths), `work/`, logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
+- Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`; greppable line by line: every line of a shell call starts with `[n] $ `, every line of the agent's text and final answer with `agent: `, output is indented, so `grep -E '^\[[0-9]+\] \$ .*npm install'` matches only commands, also inside heredocs and multi-line scripts), `run.json` (status, prompt, end_goal, check results, paths), `work/`, `home/` and `home.env` (the run's HOME), logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
 
-`firegoat.yaml` keys: `runs`, `model`, `build`, `install`, `cleanup`, `vars`, `env`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`, `install`, `cleanup`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
+`firegoat.yaml` keys: `runs`, `model`, `build`, `install`, `cleanup`, `vars`, `env`, `claude_args`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`, `install`, `cleanup`, `claude_args`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
 
 ### Stop what the agent left running, seed one test
 

package/dist/cli.js

@@ -6947,7 +6947,7 @@ var require_public_api = __commonJS(function(exports) {
 // package.json
 var package_default = {
   name: "firegoat",
-  version: "0.18.0",
+  version: "0.20.0",
   repository: {
     type: "git",
     url: "git+https://github.com/firegoat-ai/firegoat-cli.git"
@@ -7081,6 +7081,7 @@ function parseConfig(text) {
     "cleanup",
     "vars",
     "env",
+    "claude_args",
     "tests"
   ]);
   const runs = posInt(raw.runs, "runs", 1);
@@ -7090,6 +7091,7 @@ function parseConfig(text) {
   const cleanup = commands(raw.cleanup, "cleanup");
   const vars = parseVars(raw.vars);
   const env = parseEnv(raw.env, "env");
+  const claudeArgs = parseClaudeArgs(raw.claude_args, "claude_args");
   if (!Array.isArray(raw.tests) || raw.tests.length === 0)
     fail("tests", "must be a non-empty list");
   const names = new Set;
@@ -7105,7 +7107,8 @@ function parseConfig(text) {
       "timeout",
       "env",
       "install",
-      "cleanup"
+      "cleanup",
+      "claude_args"
     ]);
     const name = str(t.name, `${p}.name`);
     if (!NAME.test(name))
@@ -7132,7 +7135,11 @@ function parseConfig(text) {
       timeout: posInt(t.timeout, `${p}.timeout`, DEFAULT_TIMEOUT),
       env: { ...env, ...parseEnv(t.env, `${p}.env`) },
       install: commands(t.install, `${p}.install`),
-      cleanup: commands(t.cleanup, `${p}.cleanup`)
+      cleanup: commands(t.cleanup, `${p}.cleanup`),
+      claudeArgs: [
+        ...claudeArgs,
+        ...parseClaudeArgs(t.claude_args, `${p}.claude_args`)
+      ]
     };
   });
   return {
@@ -7172,6 +7179,42 @@ function parseModels(raw) {
     return name;
   });
 }
+var OWNED_FLAGS = [
+  [
+    [
+      "-p",
+      "--print",
+      "--output-format",
+      "--verbose",
+      "--setting-sources",
+      "--strict-mcp-config",
+      "--disable-slash-commands",
+      "--dangerously-skip-permissions"
+    ],
+    "firegoat sets it for every run"
+  ],
+  [["--model"], "choose the model with top-level model: (or eval --model)"],
+  [
+    ["--session-id", "--resume", "-r", "--continue", "-c"],
+    "every run is a fresh session"
+  ]
+];
+function parseClaudeArgs(raw, path) {
+  if (raw === undefined)
+    return [];
+  if (!Array.isArray(raw))
+    fail(path, "must be a list of claude arguments, one per entry, e.g. [--chrome]");
+  return raw.map((a, i) => {
+    const p = `${path}[${i}]`;
+    const arg = str(a, p);
+    for (const [flags, why] of OWNED_FLAGS)
+      for (const f of flags)
+        if (arg === f || arg.startsWith(`${f}=`))
+          fail(p, `${f} is not allowed: ${why}`);
+    checkPlaceholders(arg, p);
+    return arg;
+  });
+}
 var VAR_KEY = /^[A-Za-z_][A-Za-z0-9_]*$/;
 function parseVars(raw) {
   if (raw === undefined)
@@ -7193,10 +7236,18 @@ var RESERVED_ENV = new Set([
   "CLAUDE_CONFIG_DIR",
   "CLAUDE_SECURESTORAGE_CONFIG_DIR"
 ]);
+var HOME_ENV = new Set([
+  "HOME",
+  "XDG_CONFIG_HOME",
+  "XDG_CACHE_HOME",
+  "XDG_DATA_HOME",
+  "XDG_STATE_HOME",
+  "CLAUDE_ENV_FILE"
+]);
 var RUN_PLACEHOLDERS = ["run.id", "run.n", "test.name"];
 var EVAL_PLACEHOLDERS = ["run.id"];
 var ANY_PLACEHOLDER = /\{\{\s*([^{}]*?)\s*\}\}/g;
-var DIR_REF = /\$(?:\{(FIREGOAT_OUT|FIREGOAT_WORKDIR)\}|(FIREGOAT_OUT|FIREGOAT_WORKDIR)(?![A-Za-z0-9_]))/g;
+var DIR_REF = /\$(?:\{(FIREGOAT_OUT|FIREGOAT_WORKDIR|FIREGOAT_HOME)\}|(FIREGOAT_OUT|FIREGOAT_WORKDIR|FIREGOAT_HOME)(?![A-Za-z0-9_]))/g;
 var ENV_REF = new RegExp(`${ANY_PLACEHOLDER.source}|${DIR_REF.source}`, "g");
 function parseEnv(raw, path) {
   if (raw === undefined)
@@ -7210,19 +7261,24 @@ function parseEnv(raw, path) {
       fail(p, "name must be letters, digits and _, not starting with a digit");
     if (k === "ANTHROPIC_MODEL")
       fail(p, "is reserved: choose the model with top-level model: (or eval --model)");
+    if (HOME_ENV.has(k))
+      fail(p, "Reserved: every run gets its own home ($FIREGOAT_HOME)");
     if (RESERVED_ENV.has(k) || k.startsWith("FIREGOAT_"))
       fail(p, "is set by firegoat and cannot be overridden (reserved: PATH, CLAUDE_CONFIG_DIR, CLAUDE_SECURESTORAGE_CONFIG_DIR, FIREGOAT_*)");
     if (typeof v !== "string")
       fail(p, "must be a string (quote numbers and booleans)");
-    for (const m of v.matchAll(ANY_PLACEHOLDER)) {
-      const name = m[1];
-      if (!/^vars\.[A-Za-z0-9_]+$/.test(name) && !RUN_PLACEHOLDERS.includes(name))
-        fail(p, `unknown placeholder ${m[0]} (allowed: {{vars.KEY}}, ${RUN_PLACEHOLDERS.map((n) => `{{${n}}}`).join(", ")})`);
-    }
+    checkPlaceholders(v, p);
     out[k] = v;
   }
   return out;
 }
+function checkPlaceholders(v, p) {
+  for (const m of v.matchAll(ANY_PLACEHOLDER)) {
+    const name = m[1];
+    if (!/^vars\.[A-Za-z0-9_]+$/.test(name) && !RUN_PLACEHOLDERS.includes(name))
+      fail(p, `unknown placeholder ${m[0]} (allowed: {{vars.KEY}}, ${RUN_PLACEHOLDERS.map((n) => `{{${n}}}`).join(", ")})`);
+  }
+}
 function isEvalLevel(value) {
   const dirsOk = [...value.matchAll(DIR_REF)].every((m) => (m[1] ?? m[2]) === "FIREGOAT_OUT");
   return dirsOk && [...value.matchAll(ANY_PLACEHOLDER)].every((m) => {
@@ -7230,16 +7286,19 @@ function isEvalLevel(value) {
     return name.startsWith("vars.") || EVAL_PLACEHOLDERS.includes(name);
   });
 }
+function renderValue(value, vars, run, where) {
+  return value.replace(ENV_REF, (whole, name, braced, bare) => {
+    const key = name ?? braced ?? bare;
+    const filled = key.startsWith("vars.") ? vars[key.slice("vars.".length)] : run[key];
+    if (filled === undefined)
+      throw new ConfigError(`${where}: no value for ${whole}`);
+    return filled;
+  });
+}
 function renderEnv(env, vars, run) {
   const out = {};
   for (const [k, v] of Object.entries(env))
-    out[k] = v.replace(ENV_REF, (whole, name, braced, bare) => {
-      const key = name ?? braced ?? bare;
-      const filled = key.startsWith("vars.") ? vars[key.slice("vars.".length)] : run[key];
-      if (filled === undefined)
-        throw new ConfigError(`env.${k}: no value for ${whole}`);
-      return filled;
-    });
+    out[k] = renderValue(v, vars, run, `env.${k}`);
   return out;
 }
 var PLACEHOLDER = /\{\{\s*vars\.([A-Za-z0-9_]+)\s*\}\}/g;
@@ -7267,11 +7326,11 @@ import {
   appendFileSync,
   createWriteStream as createWriteStream2,
   existsSync as existsSync3,
-  mkdirSync as mkdirSync3,
+  mkdirSync as mkdirSync4,
   readdirSync as readdirSync2,
-  writeFileSync as writeFileSync3
+  writeFileSync as writeFileSync4
 } from "node:fs";
-import { basename as basename3, delimiter as delimiter2, join as join5 } from "node:path";
+import { basename as basename3, delimiter as delimiter2, join as join6 } from "node:path";
 
 // src/claude.ts
 import { spawn, spawnSync } from "node:child_process";
@@ -7286,15 +7345,18 @@ var CLAUDE_FLAGS = [
   "--disable-slash-commands"
 ];
 var NPM_INJECTED = /^(npm_config_|npm_lifecycle_|npm_package_)|^(npm_execpath|npm_node_execpath|npm_command|INIT_CWD)$/;
+var CLAUDE_SESSION = /^(CLAUDECODE|CLAUDE_CODE_.+|CLAUDE_PID|CLAUDE_ENV_FILE|CLAUDE_SESSION_ID|CLAUDE_PROJECT_DIR|CLAUDE_EFFORT|CLAUDE_AGENT_SDK_.+|CLAUDE_PLUGIN_ROOT|CLAUDE_PLUGIN_DATA|CLAUDE_SKILL_DIR)$/;
+var CLAUDE_AUTH = /^CLAUDE_CODE_(OAUTH_TOKEN|OAUTH_REFRESH_TOKEN|USE_[A-Z_]+|SKIP_[A-Z]+_AUTH|CLIENT_CERT|CLIENT_KEY|CLIENT_KEY_PASSPHRASE)$/;
 function hostEnv() {
-  return Object.fromEntries(Object.entries(process.env).filter(([k]) => !NPM_INJECTED.test(k) && !k.startsWith("FIREGOAT_")));
+  return Object.fromEntries(Object.entries(process.env).filter(([k]) => !NPM_INJECTED.test(k) && !k.startsWith("FIREGOAT_") && (!CLAUDE_SESSION.test(k) || CLAUDE_AUTH.test(k))));
 }
-function claudeEnv(configDir, env = {}) {
+function claudeEnv(configDir, env = {}, envFile) {
   return {
     ...hostEnv(),
     ...env,
     CLAUDE_CONFIG_DIR: configDir,
-    CLAUDE_SECURESTORAGE_CONFIG_DIR: ""
+    CLAUDE_SECURESTORAGE_CONFIG_DIR: "",
+    ...envFile === undefined ? {} : { CLAUDE_ENV_FILE: envFile }
   };
 }
 function run(opts) {
@@ -7337,7 +7399,8 @@ async function runClaude(opts) {
       "-p",
       opts.prompt,
       ...CLAUDE_FLAGS,
-      ...opts.model === undefined ? [] : ["--model", opts.model]
+      ...opts.model === undefined ? [] : ["--model", opts.model],
+      ...opts.extraArgs
     ],
     cwd: opts.cwd,
     env: opts.env,
@@ -7828,15 +7891,19 @@ function topHelp(version) {
 folder, with none of your claude config; firegoat saves the session and runs your
 script checks.
 
-  npx firegoat init     Create firegoat.yaml and .firegoat/, install the skill
-  npx firegoat eval     Run the tests (claude must be logged in: run \`claude\` once)
-  npx firegoat login    Log in to firegoat.ai: eval then uploads each session there
-  npx firegoat logout   Forget that login
-  npx firegoat feedback "<what got in the way>"   Tell the firegoat team
+  npx firegoat@latest init     Create firegoat.yaml and .firegoat/, install the skill
+  npx firegoat@latest eval     Run the tests (claude must be logged in: run \`claude\`)
+  npx firegoat@latest login    Log in to firegoat.ai: eval then uploads each session
+  npx firegoat@latest logout   Forget that login
+  npx firegoat@latest feedback "<what got in the way>"
+                               Tell the firegoat team
+
+Use npx firegoat@latest: plain npx firegoat can keep running an old cached version.
+Each eval prints its version first.
 
 Typical flow for a coding agent driving firegoat:
-  1. Read npx firegoat init --help, then edit firegoat.yaml
-  2. npx firegoat eval --json
+  1. Read npx firegoat@latest init --help, then edit firegoat.yaml
+  2. npx firegoat@latest eval --json
   3. Judge each run's end_goal from its run.json, session and workdir: firegoat only
      runs the script checks
 
@@ -7880,20 +7947,32 @@ firegoat.yaml reference:
   env: {NAME: value}          Env vars for the agent, and for install, checks
                               and cleanup of the same run; values are strings and can
                               use {{vars.KEY}}, {{run.id}}, {{run.n}},
-                              {{test.name}}, $FIREGOAT_OUT, $FIREGOAT_WORKDIR
-                              (no other $ expansion: $HOME stays literal).
-                              The build step gets only the entries that use only
-                              {{vars.*}}, {{run.id}} and $FIREGOAT_OUT.
+                              {{test.name}}, $FIREGOAT_OUT, $FIREGOAT_WORKDIR,
+                              $FIREGOAT_HOME (no other $ expansion: $HOME stays
+                              literal). The build step gets only the entries that
+                              use only {{vars.*}}, {{run.id}} and $FIREGOAT_OUT.
                               PATH, CLAUDE_CONFIG_DIR, FIREGOAT_* and
-                              ANTHROPIC_MODEL (use model:) are reserved.
-                              Runs go 4 at a time: give a tool with a global home
-                              a per-run one (or use eval --parallel 1), e.g.
-                    MYTOOL_HOME: "/tmp/mytool-{{run.id}}-{{test.name}}-{{run.n}}"
-                              Setting HOME: on macOS claude reads its login from
-                              $HOME/Library, so a per-run HOME needs
-                              ln -s <real home>/Library "$HOME/Library" in
-                              install (get the real home in build, which
-                              doesn't get entries using {{test.name}}/{{run.n}}).
+                              ANTHROPIC_MODEL (use model:) are reserved, and so
+                              are HOME, XDG_CONFIG_HOME, XDG_CACHE_HOME,
+                              XDG_DATA_HOME, XDG_STATE_HOME and CLAUDE_ENV_FILE:
+                              every run gets its own home ($FIREGOAT_HOME), so a
+                              tool that keeps state in ~ starts fresh in each
+                              run. Only a tool that ignores HOME needs its own
+                              var, e.g. MYTOOL_HOME: "$FIREGOAT_HOME/.mytool"
+  claude_args: [<arg>, ...]   Extra claude arguments, after firegoat's own flags
+                              and --model. Each entry is one argument, passed
+                              as-is (no shell splitting): [--chrome] (Claude in
+                              Chrome, for tests that need a browser),
+                              [--plugin-dir, ./plugins/x], --mcp-config, --agents,
+                              --append-system-prompt. Entries expand like env:
+                              values ($FIREGOAT_WORKDIR, $FIREGOAT_HOME,
+                              $FIREGOAT_OUT, {{vars.KEY}}, {{run.*}}); run.json
+                              claudeArgs has the result. Flags firegoat sets are
+                              errors, also as --flag=value: -p, --print,
+                              --output-format, --verbose, --setting-sources,
+                              --strict-mcp-config, --disable-slash-commands,
+                              --dangerously-skip-permissions, --model (use
+                              model:), --session-id, --resume, -r, --continue, -c
   tests:
     - name: quickstart        Letters, digits, '.', '_', '-'; unique
       prompt: ...             What a user would ask the agent; one concrete outcome
@@ -7905,14 +7984,20 @@ firegoat.yaml reference:
       env: {NAME: value}      Optional, on top of the top-level env:
       install: <sh> | [...]   Optional, runs after the top-level install
       cleanup: <sh> | [...]   Optional, runs before the top-level cleanup
+      claude_args: [...]      Optional, after the top-level claude_args
 
   Step order: build once per eval; then per run: install (top-level, then the
   test's), the agent, checks, cleanup (the test's, then top-level).
 
   Unknown keys are errors. Keep helper files out of firegoat.yaml: put them in
-  e.g. fixtures/ (committed) and copy them in build or install. The build, install
-  and cleanup commands and the checks get the vars as env vars, plus $FIREGOAT_OUT and
-  $FIREGOAT_WORKDIR. Checks also get $FIREGOAT_TRANSCRIPT, a short plain-text
+  e.g. fixtures/ (committed) and copy them in build or install. To start each run
+  from a template project, commit it as e.g. fixtures/starter-app/, then:
+    build: cp -R fixtures/starter-app "$FIREGOAT_OUT/starter-app"
+    install: cp -R "$FIREGOAT_OUT/starter-app/." .
+  The build, install and cleanup commands and the checks get the vars as env
+  vars, plus $FIREGOAT_OUT and $FIREGOAT_WORKDIR; install, checks and cleanup also
+  get $FIREGOAT_HOME, the run's own home, as HOME (build keeps your real HOME).
+  Checks also get $FIREGOAT_TRANSCRIPT, a short plain-text
   transcript, greppable line by line: "[1] $ <command>" on every line of a shell
   call (one line per line of the command, same n), "[2] Write <path>" for other
   tools, each output indented, and "agent: " on every line of the agent's text and
@@ -7925,6 +8010,18 @@ firegoat.yaml reference:
   every env firegoat starts, unless env: sets them. So is every FIREGOAT_* var
   (FIREGOAT_API_KEY above all) firegoat itself was started with.
 
+Testing an MCP server: write its config per run in install: and pass it with
+claude_args (relative paths are from the run's folder, claude's cwd). Only the
+configs you pass load: --strict-mcp-config is always on. An MCP server inherits
+claude's env, which keeps your real HOME (claude's login is there), so set the
+server's HOME to the run's home, or it writes to yours:
+  install: |
+    cat > .mcp.json <<EOF
+    {"mcpServers": {"acme": {"command": "npx", "args": ["-y", "acme-mcp"],
+      "env": {"HOME": "$FIREGOAT_HOME"}}}}
+    EOF
+  claude_args: [--mcp-config, .mcp.json]
+
 Example:
   build: npm pack --pack-destination "$FIREGOAT_OUT"
   install: npm init -y >/dev/null && npm install "$FIREGOAT_OUT"/*.tgz
@@ -7942,6 +8039,12 @@ Example:
         - run: |
             test -s "$FIREGOAT_TRANSCRIPT" &&
               ! grep -Eq '^\\[[0-9]+\\] \\$ .*npm (install|i|add) .*acme' "$FIREGOAT_TRANSCRIPT"
+    - name: dashboard-key
+      prompt: >-
+        Get an acme API key from the acme dashboard (I'm logged in in Chrome) and put
+        it in .env as ACME_API_KEY.
+      end_goal: .env holds a key copied from the dashboard, not a made-up one.
+      claude_args: [--chrome]
 
 Exit codes: 0 done, 1 the firegoat login failed (the files stay), 2 the skill
 install failed, 3 usage error.`;
@@ -7993,9 +8096,30 @@ changes the run's status or the exit code.
 
 claude runs on its own login (or ANTHROPIC_API_KEY if set, as claude always does)
 with a fresh config dir, .firegoat/.claude: none of your settings, CLAUDE.md,
-plugins, skills or MCP servers. claude saves its session there. There is no
+plugins, skills or MCP servers (claude_args: in firegoat.yaml adds claude flags,
+e.g. --chrome or --plugin-dir). claude saves its session there. There is no
 sandbox: the agent can reach the rest of your machine, including the repo.
 
+Every run gets its own HOME, .firegoat/runs/<runId>/<test>/<n>/home (with
+XDG_CONFIG_HOME, XDG_CACHE_HOME, XDG_DATA_HOME, XDG_STATE_HOME inside it), as
+$FIREGOAT_HOME: tools that keep state in ~ start fresh and parallel runs don't
+collide. claude itself keeps your real HOME (its login is there) and sources
+<n>/home.env (CLAUDE_ENV_FILE) before each Bash command the agent runs; install,
+checks and cleanup get the run's HOME directly; build keeps yours. Limits: claude's
+own Read/Write tools expand ~ to your real home; getpwuid lookups (os.userInfo,
+python pwd) still see it; keychain logins (gh, docker, git credential helpers) look
+logged out inside a run, so log tools in from install: or with env: tokens; the
+fresh home has no .gitconfig, so set user.name/user.email in install: if the agent
+commits; claude's shell snapshot still sources your real ~/.zshrc.
+
+Vars a parent Claude Code session sets (CLAUDECODE, CLAUDE_CODE_*, CLAUDE_PID,
+CLAUDE_ENV_FILE, CLAUDE_SESSION_ID, CLAUDE_PROJECT_DIR, CLAUDE_EFFORT,
+CLAUDE_AGENT_SDK_*, CLAUDE_PLUGIN_ROOT, CLAUDE_PLUGIN_DATA, CLAUDE_SKILL_DIR) are
+removed from every env firegoat starts, so the claude under test doesn't act as a
+nested session. Login and provider vars stay: ANTHROPIC_*, CLAUDE_CODE_OAUTH_TOKEN,
+CLAUDE_CODE_OAUTH_REFRESH_TOKEN, CLAUDE_CODE_USE_*, CLAUDE_CODE_SKIP_*_AUTH,
+CLAUDE_CODE_CLIENT_CERT, CLAUDE_CODE_CLIENT_KEY, CLAUDE_CODE_CLIENT_KEY_PASSPHRASE.
+
 In a linked git worktree (git worktree add), .firegoat/runs/ and .firegoat/.claude
 live in the main checkout instead, so runs outlive the worktree; firegoat.yaml,
 fixtures and build: stay in the current folder.
@@ -8005,8 +8129,8 @@ fixtures and build: stay in the current folder.
   --model NAME       claude --model, replacing model: in firegoat.yaml; repeat it
                      (--model a --model b) to run each test runs times per model
   --parallel N       Runs at once (default 4; 1 runs them one after another).
-                     Tools with global on-disk state collide: give each run its
-                     own home via env: ({{run.id}}-{{test.name}}-{{run.n}}) or use 1
+                     Each run has its own HOME; state outside it (a fixed port,
+                     /tmp paths) can still collide: use 1 then
   --var KEY=value    Fills {{vars.KEY}}; env for build, install, checks and
                      cleanup
   --json             The summary as JSON on stdout (or {error, exitCode} if eval
@@ -8034,9 +8158,10 @@ Output: .firegoat/runs/<runId>/
                           uploadFailed
   <test>/<n>/run.json     One run: status and reason (from the checks), model
                           (asked for, or null) and claudeModel (what claude's
-                          init line reported), prompt, end_goal,
+                          init line reported), claudeArgs (claude_args
+                          passed), prompt, end_goal,
                           checks [{run, code}], cleanup [{run, code}],
-                          and paths: workdir,
+                          and paths: workdir, home (the run's HOME),
                           transcript, session (claude's session file), stderr,
                           checksLog, cleanupLog; startedAt; upload ({status: "uploaded",
                           url} | {status: "failed", error} | {status:
@@ -8045,6 +8170,8 @@ Output: .firegoat/runs/<runId>/
                           ("[n] $ " on every line of a shell call), short output
                           (indented), its text and final answer ("agent: " on
                           every line), files it made
+  <test>/<n>/home/, home.env  The run's HOME, and the file claude sources
+                          before the agent's Bash commands
   build.txt, <test>/<n>/install.txt, checks.txt, cleanup.txt   Logs
 
 Grading: firegoat runs only the script checks; they set the exit code. end_goal is
@@ -8083,6 +8210,30 @@ Example:
 
 Exit codes: 0 sent, 2 the service failed (or rate limit), 3 usage error.`;
 
+// src/home.ts
+import { mkdirSync as mkdirSync3, writeFileSync as writeFileSync2 } from "node:fs";
+import { join as join3 } from "node:path";
+function homeEnv(home) {
+  return {
+    HOME: home,
+    XDG_CONFIG_HOME: join3(home, ".config"),
+    XDG_CACHE_HOME: join3(home, ".cache"),
+    XDG_DATA_HOME: join3(home, ".local", "share"),
+    XDG_STATE_HOME: join3(home, ".local", "state")
+  };
+}
+var shellQuote = (s) => `'${s.replaceAll("'", `'\\''`)}'`;
+function makeRunHome(dir) {
+  const home = join3(dir, "home");
+  const env = homeEnv(home);
+  for (const d of Object.values(env))
+    mkdirSync3(d, { recursive: true });
+  const envFile = join3(dir, "home.env");
+  writeFileSync2(envFile, Object.entries(env).map(([k, v]) => `export ${k}=${shellQuote(v)}
+`).join(""));
+  return { home, envFile };
+}
+
 // src/steps.ts
 import { createWriteStream } from "node:fs";
 var STEP_TIMEOUT_MS = 600000;
@@ -8124,8 +8275,8 @@ async function runAllSteps(opts) {
 }
 
 // src/transcript.ts
-import { readdirSync, statSync, writeFileSync as writeFileSync2 } from "node:fs";
-import { join as join3, relative } from "node:path";
+import { readdirSync, statSync, writeFileSync as writeFileSync3 } from "node:fs";
+import { join as join4, relative } from "node:path";
 var OUTPUT_LINES = 15;
 var OUTPUT_CHARS = 1500;
 var TEXT_CHARS = 1500;
@@ -8169,7 +8320,7 @@ function workdirFiles(dir) {
     for (const name of readdirSync(d)) {
       if (SKIP_DIRS.has(name))
         continue;
-      const p = join3(d, name);
+      const p = join4(d, name);
       if (statSync(p).isDirectory())
         walk(p);
       else
@@ -8212,7 +8363,7 @@ ${indent(truncate(text, OUTPUT_LINES, OUTPUT_CHARS))}` : `    → ${status}`);
     out.pop();
   out.push("", "## final answer", final ? prose(final) : "(none)");
   out.push("", "## files in the workdir", ...workdirFiles(opts.workdir));
-  writeFileSync2(opts.path, `${out.join(`
+  writeFileSync3(opts.path, `${out.join(`
 `)}
 `);
 }
@@ -8392,7 +8543,7 @@ function ttyUI(out, opts) {
     const doneCount = c.pass + c.fail + c.error;
     const badge = finished ? `DONE · RUN ${finished.runId.slice(-4)}` : `RUNNING · ${doneCount}/${rows.length}`;
     const out = [
-      ...logoHeader(p, `firegoat eval · ${opts.project}`, p(finished ? C.green : C.ember, badge), w),
+      ...logoHeader(p, `firegoat ${opts.version} eval · ${opts.project}`, p(finished ? C.green : C.ember, badge), w),
       rule
     ];
     if (authText)
@@ -8653,7 +8804,7 @@ async function uploadRun(t, r) {
 
 // src/which.ts
 import { accessSync, constants, statSync as statSync2 } from "node:fs";
-import { delimiter, join as join4, resolve } from "node:path";
+import { delimiter, join as join5, resolve } from "node:path";
 function resolveBin(bin) {
   const executable = (p) => {
     try {
@@ -8666,8 +8817,8 @@ function resolveBin(bin) {
   if (bin.includes("/"))
     return executable(bin) ? resolve(bin) : undefined;
   for (const dir of (process.env.PATH ?? "").split(delimiter)) {
-    if (dir && executable(join4(dir, bin)))
-      return join4(dir, bin);
+    if (dir && executable(join5(dir, bin)))
+      return join5(dir, bin);
   }
   return;
 }
@@ -8740,11 +8891,11 @@ function parseArgs(args) {
   return opts;
 }
 function findSession(configDir, sessionId) {
-  const projects = join5(configDir, "projects");
+  const projects = join6(configDir, "projects");
   if (!existsSync3(projects))
     return null;
   for (const d of readdirSync2(projects)) {
-    const f = join5(projects, d, `${sessionId}.jsonl`);
+    const f = join6(projects, d, `${sessionId}.jsonl`);
     if (existsSync3(f))
       return f;
   }
@@ -8755,7 +8906,7 @@ function newRunId() {
   return `${ts}-${randomBytes2(2).toString("hex")}`;
 }
 function writeRunJson(r) {
-  writeFileSync3(join5(r.dir, "run.json"), `${JSON.stringify(r, null, 2)}
+  writeFileSync4(join6(r.dir, "run.json"), `${JSON.stringify(r, null, 2)}
 `);
 }
 async function uploadPreflight(opts, ui, project, runGroup, git) {
@@ -8797,14 +8948,24 @@ async function uploadPreflight(opts, ui, project, runGroup, git) {
 async function runOne(opts) {
   const { test, n } = opts;
   const started = Date.now();
-  const dir = join5(opts.runDir, test.name, String(n));
-  const workdir = join5(dir, "work");
-  mkdirSync3(workdir, { recursive: true });
+  const dir = join6(opts.runDir, test.name, String(n));
+  const workdir = join6(dir, "work");
+  mkdirSync4(workdir, { recursive: true });
+  const { home, envFile } = makeRunHome(dir);
+  const fill = {
+    "run.id": opts.runId,
+    "run.n": String(n),
+    "test.name": test.name,
+    FIREGOAT_OUT: opts.out,
+    FIREGOAT_WORKDIR: workdir,
+    FIREGOAT_HOME: home
+  };
   const r = {
     test: test.name,
     n,
     model: opts.model,
     claudeModel: null,
+    claudeArgs: test.claudeArgs.map((a, i) => renderValue(a, opts.vars, fill, `claude_args[${i}]`)),
     status: "error",
     reason: "",
     prompt: render(test.prompt, opts.vars),
@@ -8812,27 +8973,28 @@ async function runOne(opts) {
     checks: [],
     dir,
     workdir,
+    home,
     session: null,
-    transcript: join5(dir, "transcript.txt"),
-    stderr: join5(dir, "stderr.txt"),
-    checksLog: join5(dir, "checks.txt"),
+    transcript: join6(dir, "transcript.txt"),
+    stderr: join6(dir, "stderr.txt"),
+    checksLog: join6(dir, "checks.txt"),
     cleanup: [],
-    cleanupLog: join5(dir, "cleanup.txt"),
+    cleanupLog: join6(dir, "cleanup.txt"),
     num_turns: null,
     cost: null,
     duration_ms: 0,
     startedAt: new Date(started).toISOString(),
     upload: null
   };
-  const env = renderEnv(test.env, opts.vars, {
-    "run.id": opts.runId,
-    "run.n": String(n),
-    "test.name": test.name,
-    FIREGOAT_OUT: opts.out,
-    FIREGOAT_WORKDIR: workdir
-  });
-  const stepEnv = { ...opts.stepEnv, ...env, FIREGOAT_WORKDIR: workdir };
-  const [status, reason] = await attempt(opts, r, env, stepEnv);
+  const env = renderEnv(test.env, opts.vars, fill);
+  const stepEnv = {
+    ...opts.stepEnv,
+    ...env,
+    ...homeEnv(home),
+    FIREGOAT_WORKDIR: workdir,
+    FIREGOAT_HOME: home
+  };
+  const [status, reason] = await attempt(opts, r, env, stepEnv, envFile);
   const cleanup = [...test.cleanup, ...opts.cleanup];
   if (cleanup.length > 0)
     r.cleanup = await runAllSteps({
@@ -8847,7 +9009,7 @@ async function runOne(opts) {
   writeRunJson(r);
   return r;
 }
-async function attempt(opts, r, env, stepEnv) {
+async function attempt(opts, r, env, stepEnv, homeEnvFile) {
   const { test, n } = opts;
   const { dir, workdir } = r;
   const install = [...opts.install, ...test.install];
@@ -8857,7 +9019,7 @@ async function attempt(opts, r, env, stepEnv) {
       commands: install,
       cwd: workdir,
       env: stepEnv,
-      logPath: join5(dir, "install.txt")
+      logPath: join6(dir, "install.txt")
     });
     if (error)
       return ["error", `not graded: ${error}`];
@@ -8867,10 +9029,14 @@ async function attempt(opts, r, env, stepEnv) {
     bin: opts.claudeBin,
     prompt: r.prompt,
     cwd: workdir,
-    env: { ...claudeEnv(opts.configDir, env), PATH: opts.stepEnv.PATH },
+    env: {
+      ...claudeEnv(opts.configDir, env, homeEnvFile),
+      PATH: opts.stepEnv.PATH
+    },
     stderr,
     timeoutMs: test.timeout * 1000,
     model: opts.model ?? undefined,
+    extraArgs: r.claudeArgs,
     onEvent: opts.onEvent
   });
   r.claudeModel = outcome.claudeModel;
@@ -8921,6 +9087,20 @@ async function attempt(opts, r, env, stepEnv) {
     test.checks.length === 0 ? "no script checks" : `${r.checks.length}/${r.checks.length} checks passed`
   ];
 }
+var MAX_LISTED_URLS = 5;
+function uploadedSummary(origin, results, runDir) {
+  const urls = results.flatMap((r) => r.upload?.status === "uploaded" ? [r.upload.url] : []);
+  if (urls.length === 0)
+    return;
+  const head = `Uploaded ${urls.length} session${urls.length === 1 ? "" : "s"} to ${new URL(origin).host} (private)`;
+  if (urls.length > MAX_LISTED_URLS) {
+    console.log(`${head}: each run's upload.url in ${join6(runDir, "summary.json")}`);
+    return;
+  }
+  console.log(`${head}:`);
+  for (const u of urls)
+    console.log(`  ${u}`);
+}
 async function evalCommand(args, root) {
   if (args.includes("--help") || args.includes("-h")) {
     console.log(EVAL_HELP);
@@ -8943,7 +9123,13 @@ async function evalCommand(args, root) {
   }
   const say = opts.json ? console.error : console.log;
   const stream = opts.json ? process.stderr : process.stdout;
-  const ui = stream.isTTY ? ttyUI(stream, { project: basename3(root), color: !process.env.NO_COLOR }) : plainUI(say);
+  const ui = stream.isTTY ? ttyUI(stream, {
+    project: basename3(root),
+    version: package_default.version,
+    color: !process.env.NO_COLOR
+  }) : plainUI(say);
+  if (!stream.isTTY)
+    say(`firegoat ${package_default.version}`);
   let config;
   try {
     config = loadConfig(root);
@@ -8967,7 +9153,8 @@ async function evalCommand(args, root) {
     ...new Set(tests.flatMap((t) => [
       t.prompt,
       ...t.checks.map((c) => c.run),
-      ...Object.values(t.env)
+      ...Object.values(t.env),
+      ...t.claudeArgs
     ]).flatMap(varRefs).filter((k) => !(k in vars)))
   ];
   if (missing.length > 0)
@@ -8982,8 +9169,8 @@ async function evalCommand(args, root) {
     return fatal(EXIT.usage, e.message);
   }
   const firegoatDir = ensureFiregoatDir(repo?.mainCheckout ?? root);
-  const configDir = join5(firegoatDir, ".claude");
-  mkdirSync3(configDir, { recursive: true });
+  const configDir = join6(firegoatDir, ".claude");
+  mkdirSync4(configDir, { recursive: true });
   ui.auth(process.env.ANTHROPIC_API_KEY ? "ANTHROPIC_API_KEY (claude bills the API key; unset it to use your claude login)" : "your claude login");
   const loginProblem = checkLogin({
     bin: claudePath,
@@ -9005,10 +9192,10 @@ async function evalCommand(args, root) {
       return fatal(EXIT.usage, e.message);
     throw e;
   }
-  const runsRoot = join5(firegoatDir, "runs");
-  const runDir = join5(runsRoot, runId);
-  const out = join5(runDir, "out");
-  mkdirSync3(out, { recursive: true });
+  const runsRoot = join6(firegoatDir, "runs");
+  const runDir = join6(runsRoot, runId);
+  const out = join6(runDir, "out");
+  mkdirSync4(out, { recursive: true });
   const stepEnv = {
     ...hostEnv(),
     ...vars,
@@ -9034,7 +9221,7 @@ async function evalCommand(args, root) {
         ...stepEnv,
         ...renderEnv(Object.fromEntries(Object.entries(config.env).filter(([, v]) => isEvalLevel(v))), vars, { "run.id": runId, FIREGOAT_OUT: out })
       },
-      logPath: join5(runDir, "build.txt")
+      logPath: join6(runDir, "build.txt")
     });
     const secs = Math.round((Date.now() - buildStarted) / 1000);
     ui.build(error ? "fail" : "ok", secs);
@@ -9107,6 +9294,7 @@ async function evalCommand(args, root) {
     totals[r.status]++;
   const uploadsFailed = results.filter((r) => r.upload?.status === "failed").length;
   const doc = {
+    firegoatVersion: package_default.version,
     runId,
     runsDir: runDir,
     dir: root,
@@ -9115,9 +9303,9 @@ async function evalCommand(args, root) {
     totals,
     runs: results
   };
-  writeFileSync3(join5(runDir, "summary.json"), `${JSON.stringify(doc, null, 2)}
+  writeFileSync4(join6(runDir, "summary.json"), `${JSON.stringify(doc, null, 2)}
 `);
-  appendFileSync(join5(runsRoot, "index.jsonl"), `${JSON.stringify({
+  appendFileSync(join6(runsRoot, "index.jsonl"), `${JSON.stringify({
     runId,
     startedAt: new Date(evalStarted).toISOString(),
     dir: root,
@@ -9137,6 +9325,8 @@ async function evalCommand(args, root) {
     rows,
     seconds: Math.round((Date.now() - evalStarted) / 1000)
   });
+  if (upload && !opts.json)
+    uploadedSummary(upload.origin, results, runDir);
   for (const r of results)
     for (const c of r.cleanup)
       if (c.code !== 0)
@@ -9417,8 +9607,8 @@ firegoat ${package_default.version}` : `firegoat ${package_default.version}`,
 
 // src/init.ts
 import { spawnSync as spawnSync3 } from "node:child_process";
-import { existsSync as existsSync4, writeFileSync as writeFileSync4 } from "node:fs";
-import { basename as basename4, join as join6 } from "node:path";
+import { existsSync as existsSync4, writeFileSync as writeFileSync5 } from "node:fs";
+import { basename as basename4, join as join7 } from "node:path";
 import { createInterface } from "node:readline/promises";
 
 // src/login.ts
@@ -9613,7 +9803,7 @@ function skillsDir(scope) {
   if (scope === "local")
     return ".claude/skills/";
   const config = process.env.CLAUDE_CONFIG_DIR?.trim();
-  return config ? `${join6(config, "skills")}/` : "~/.claude/skills/";
+  return config ? `${join7(config, "skills")}/` : "~/.claude/skills/";
 }
 function template(product) {
   return `# firegoat.yaml: evals for how well coding agents use ${product}.
@@ -9647,6 +9837,10 @@ runs: 1
 # Helper files (scripts, sample data) go in e.g. fixtures/ (committed), not inline here.
 # The build step runs in this folder, so: build: cp -R fixtures/. "$FIREGOAT_OUT"
 # Then e.g. install: cp "$FIREGOAT_OUT/wizard.mjs" .
+#
+# To start each run from a template project, commit it as e.g. fixtures/starter-app/:
+# build: cp -R fixtures/starter-app "$FIREGOAT_OUT/starter-app"
+# install: cp -R "$FIREGOAT_OUT/starter-app/." .
 
 # Optional: cleanup: runs in each run's folder once the run is over (after the
 # checks, also after an error or timeout), e.g. to stop a server the agent left
@@ -9658,28 +9852,30 @@ runs: 1
 
 # Optional: env vars for the agent (and for install, checks and cleanup of the
 # same run). The agent gets none of vars:, only these. Placeholders: {{vars.KEY}},
-# {{run.id}}, {{run.n}}, {{test.name}}, and $FIREGOAT_OUT / $FIREGOAT_WORKDIR (no
-# other $ expansion: $HOME stays literal). A test can add its own env:.
-# Runs go 4 at a time (--parallel), so give a tool with a global home (~/.mytool)
-# a per-run one, as below, or use --parallel 1.
+# {{run.id}}, {{run.n}}, {{test.name}}, and $FIREGOAT_OUT / $FIREGOAT_WORKDIR /
+# $FIREGOAT_HOME (no other $ expansion: $HOME stays literal). A test can add its
+# own env:. Every run gets its own HOME ($FIREGOAT_HOME), so tools that keep state
+# in ~ are isolated per run already; set a tool's own *_HOME var only if the tool
+# ignores HOME, as below.
 # env:
-#   MYTOOL_HOME: "/tmp/mytool-{{run.id}}-{{test.name}}-{{run.n}}"
+#   MYTOOL_HOME: "$FIREGOAT_HOME/.mytool"
 #   UV_FIND_LINKS: "$FIREGOAT_OUT"
 
 tests:
-  - name: quickstart
-    # What a user would ask their agent. Edit this.
-    prompt: >-
-      I want to try ${product}. Get the smallest working example going in a new
-      project, run it, and save what it prints to output.txt. Keep it minimal.
+  # A smoke test that firegoat and claude work. Replace it with tests for your
+  # product: the firegoat-onboarding skill writes them.
+  - name: goat
+    # What a user would ask their agent.
+    prompt: Who's the GOAT? Answer in one short sentence.
     # True when the user's problem is solved. Not graded by firegoat: it goes to
     # your agent (eval --json, run.json) to judge.
-    end_goal: >-
-      A minimal project uses ${product}, runs without errors, and output.txt holds
-      its output.
+    end_goal: Claude answers that you (the user) are the goat, e.g. YOU'RE THE GOAT.
     # Shell commands run in the agent's folder afterwards; exit 0 = pass.
+    # $FIREGOAT_TRANSCRIPT has "agent: " on every line of the agent's text.
     checks:
-      - run: test -s output.txt
+      - run: >-
+          test -s "$FIREGOAT_TRANSCRIPT" &&
+          grep -Eiq '^agent: .*goat' "$FIREGOAT_TRANSCRIPT"
     # Seconds before the agent run is stopped.
     timeout: 600
 `;
@@ -9798,16 +9994,16 @@ ${INIT_HELP}`);
     console.log(rule);
   }
   ensureFiregoatDir(cwd);
-  const configPath = join6(cwd, CONFIG_FILE);
+  const configPath = join7(cwd, CONFIG_FILE);
   const keep = existsSync4(configPath) && !force;
   if (!keep)
-    writeFileSync4(configPath, template(basename4(cwd)));
+    writeFileSync5(configPath, template(basename4(cwd)));
   if (interactive) {
     step(1, CONFIG_FILE, keep ? "Kept (--force replaces it)" : "Created");
     console.log(paint("90", "  │"));
     step(2, ".firegoat/", "Ready");
   } else
-    console.log(keep ? `Kept ${CONFIG_FILE} (--force replaces it)` : `Wrote ${CONFIG_FILE}: edit the quickstart test to fit your users`);
+    console.log(keep ? `Kept ${CONFIG_FILE} (--force replaces it)` : `Wrote ${CONFIG_FILE}: replace the goat smoke test with tests for your product`);
   if (interactive && !noSkills)
     console.log(paint("90", "  │"));
   if (interactive && !noSkills && !localSkills) {
@@ -9861,7 +10057,7 @@ Prompt for your coding agent: ${AGENT_PROMPT}`);
   console.log(AGENT_PROMPT);
   console.log("");
   const copied = copyToClipboard(AGENT_PROMPT);
-  console.log(`  ${copied ? `${paint("32", "✓ Prompt copied!")}   ` : ""}Then: ${paint("36", "npx firegoat eval")}`);
+  console.log(`  ${copied ? `${paint("32", "✓ Prompt copied!")}   ` : ""}Then: ${paint("36", "npx firegoat@latest eval")}`);
   return EXIT.pass;
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "firegoat",
-	"version": "0.18.0",
+	"version": "0.20.0",
 	"repository": {
 		"type": "git",
 		"url": "git+https://github.com/firegoat-ai/firegoat-cli.git"