firegoat 0.17.1 → 0.19.0

package/README.md

@@ -7,12 +7,12 @@ Status: early. See `plan.md`.
 ## Quickstart
 
 ```bash
-npx firegoat init     # Creates firegoat.yaml and .firegoat/, installs the firegoat-onboarding skill, copies a prompt for your agent
+npx firegoat@latest init     # Creates firegoat.yaml and .firegoat/, installs the firegoat-onboarding skill, copies a prompt for your agent
 # Edit firegoat.yaml (or paste the prompt into your coding agent and let it)
-npx firegoat eval     # Runs your claude on each test, in .firegoat/runs/<runId>/<test>/<n>/work
+npx firegoat@latest eval     # Runs your claude on each test, in .firegoat/runs/<runId>/<test>/<n>/work
 ```
 
-That's it: two commands. `claude` must be installed and logged in (run `claude` once).
+That's it: two commands. Use `npx firegoat@latest`: plain `npx firegoat` can keep running an old cached version. `eval` prints its version first (`firegoat 0.18.0`), and `summary.json` records it as `firegoatVersion`. `claude` must be installed and logged in (run `claude` once).
 
 `init` installs the firegoat-onboarding skill globally, into `~/.claude/skills/` (in a terminal it asks: global, this repo, or none). `--local-skills` puts it in this repo's `.claude/skills/` instead (commit it so teammates and CI get it); `--no-skills` skips it.
 
@@ -98,6 +98,21 @@ tests:
 npx firegoat eval --var SHA=$(git rev-parse HEAD)
 ```
 
+### Start from a template project
+
+```yaml
+build: cp -R fixtures/starter-app "$FIREGOAT_OUT/starter-app"
+install: cp -R "$FIREGOAT_OUT/starter-app/." .
+tests:
+  - name: add-auth
+    prompt: Add acme login to this app. Keep the existing pages working.
+    end_goal: The app signs users in with acme, and the existing pages still render.
+    checks:
+      - run: npm test
+```
+
+No template feature needed: keep the project committed in `fixtures/starter-app/`, `build` copies it once per eval, and `install` copies it into each fresh folder, so every run begins in its own copy of that project.
+
 ### Same tests on several models
 
 ```yaml
@@ -113,11 +128,43 @@ tests:
 
 `model:` is passed to claude as `--model`. A list is a matrix: each test runs `runs` times per model, numbered `1..runs×models` (here `quickstart/1`, `2` on haiku, `3`, `4` on sonnet). `npx firegoat eval --model claude-opus-5-5` replaces it for one eval; repeat the flag for a matrix. Unset, firegoat passes no `--model` and claude uses its default. Each run's `run.json` records `model` (what firegoat asked for, or null) and `claudeModel` (what claude reported starting with), and the output labels each run with its model.
 
-### Env vars for the agent: a home dir per run, a feature flag
+### Compare two setups
+
+```yaml
+build: npm pack --pack-destination "$FIREGOAT_OUT"
+tests:
+  - name: deploy-local-build
+    prompt: &deploy Deploy this app to an acme preview environment and print the URL.
+    end_goal: &deployed The app runs on an acme preview URL, and the URL is printed.
+    install: npm install "$FIREGOAT_OUT"/acme-*.tgz
+  - name: deploy-npm-release
+    prompt: *deploy
+    end_goal: *deployed
+    install: npm install acme@latest
+```
+
+There is no variants feature: to compare setups (your local build vs the npm release, with vs without `--chrome`, docs A vs docs B), write one test per setup with the same prompt and a different per-test `install:`, `env:` or `claude_args:`. A YAML anchor (`&deploy`) and alias (`*deploy`) keep the prompt in one place. Skills installed into the run's folder are not loaded as skills: eval runs claude with `--setting-sources ""` and `--disable-slash-commands`, so a with/without-skill comparison isn't possible yet.
+
+### Extra claude flags
+
+```yaml
+claude_args: [--plugin-dir, ./plugins/acme]   # Every test
+tests:
+  - name: dashboard-key
+    prompt: Get an acme API key from the acme dashboard (I'm logged in in Chrome) and put it in .env as ACME_API_KEY.
+    end_goal: .env holds a key copied from the dashboard, not a made-up one.
+    claude_args: [--chrome]                    # Claude in Chrome, this test only
+```
+
+`claude_args:` (top level, and per test after the top-level ones) is passed to claude after firegoat's own flags and `--model`. Each entry is one argument, as-is, with no shell splitting: `[--plugin-dir, ./plugins/acme]`, not `["--plugin-dir ./plugins/acme"]`. Use it for `--chrome`, `--plugin-dir`, `--mcp-config`, `--agents`, `--append-system-prompt`. The flags firegoat sets itself are config errors, also as `--flag=value`: `-p`, `--print`, `--output-format`, `--verbose`, `--setting-sources`, `--strict-mcp-config`, `--disable-slash-commands`, `--dangerously-skip-permissions`, `--model` (use `model:`), and `--session-id`, `--resume`, `-r`, `--continue`, `-c` (every run is a fresh session). Each run's `run.json` records `claudeArgs`.
+
+With `--chrome`, claude points the Claude in Chrome extension's native host (`~/Library/Application Support/<browser>/NativeMessagingHosts/com.anthropic.claude_code_browser_extension.json`) at `.firegoat/.claude`. If you delete `.firegoat/`, run a normal `claude --chrome` once to point it back.
+
+### Env vars for the agent: a feature flag, a tool that ignores HOME
 
 ```yaml
 env:
-  ACME_HOME: "/tmp/acme-{{run.id}}-{{run.n}}"   # Sessions/config never collide across runs
+  ACME_HOME: "$FIREGOAT_HOME/.acme"             # Only for a tool that ignores HOME
   ACME_TELEMETRY: "off"
   UV_FIND_LINKS: "$FIREGOAT_OUT"                # Resolve the wheel build: made
 tests:
@@ -130,21 +177,50 @@ tests:
       - run: ls "$ACME_HOME"/sessions | grep -q .
 ```
 
-`env:` goes to the agent, and to `install` and checks of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` (also `${...}`), the same dirs `build`, `install` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `build` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`.
+`env:` goes to the agent, and to `install`, checks and `cleanup` of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` / `$FIREGOAT_HOME` (also `${...}`), the same dirs `build`, `install` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `build` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`. `HOME`, `XDG_CONFIG_HOME`, `XDG_CACHE_HOME`, `XDG_DATA_HOME`, `XDG_STATE_HOME` and `CLAUDE_ENV_FILE` are reserved too: every run gets its own home.
+
+### Each run gets its own HOME
+
+Every run has its own home, `.firegoat/runs/<runId>/<test>/<n>/home/`, with `.config`, `.cache`, `.local/share` and `.local/state` for the XDG dirs. A tool that keeps logins, config or caches in `~` starts fresh in every run, and parallel runs don't collide. Point a tool's own `*_HOME` var at `$FIREGOAT_HOME` only if the tool ignores `HOME`.
+
+- The agent's shell commands see it: firegoat writes `<n>/home.env` (exporting `HOME` and `XDG_CONFIG_HOME`, `XDG_CACHE_HOME`, `XDG_DATA_HOME`, `XDG_STATE_HOME`) and passes it to claude as `CLAUDE_ENV_FILE`, which claude sources before every Bash command. claude itself keeps your real `HOME`, where its login is (the macOS keychain lookup, `~/.claude/.credentials.json` on Linux).
+- `install`, checks and `cleanup` get `HOME` and the XDG vars set to the run's home, plus `$FIREGOAT_HOME`. `build` keeps your real home.
+- `run.json` records it as `home`.
 
-Setting `HOME`: on macOS claude reads its login from `$HOME/Library`, so with a per-run `HOME` every run says "Not logged in" unless `install` runs `ln -s <real home>/Library "$HOME/Library"`. Get the real home in `build`, which doesn't get entries using `{{test.name}}` or `{{run.n}}` (e.g. `build: echo "$HOME" > "$FIREGOAT_OUT/.real-home"`).
+Limits:
+
+- claude's own Read/Write/Edit tools expand `~` to your real home; only Bash commands get the run's home.
+- Lookups by user id (`os.userInfo().homedir`, python's `pwd`, `getpwuid`) still return your real home.
+- Keychain-backed logins (`gh`, `docker`, git credential helpers) look logged out inside a run. Log tools in from `install:`, or pass tokens with `env:`.
+- The fresh home has no `.gitconfig`: if the agent commits, set `git config --global user.name ...` / `user.email` in `install:`.
+- claude's shell snapshot still sources your real `~/.zshrc` (or `~/.bashrc`).
 
 ## What eval does
 
-- claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers. claude saves its session there, in its own format.
-- No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. Commit `.firegoat/` with `firegoat.yaml`: its `.gitignore` ignores only `runs/` and `.claude/`, test fixtures (scripts, sample data) go in `.firegoat/fixtures/`, and its `README.md` says what each folder is.
+- claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers (add flags with `claude_args:`, see above). claude saves its session there, in its own format.
+- No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. `.firegoat/` holds local run output and ignores itself in git. Keep test fixtures (scripts, sample data) outside it, e.g. in `fixtures/` next to `firegoat.yaml`, and commit them.
 - In a linked git worktree (`git worktree add`), `.firegoat/runs/` and `.firegoat/.claude/` are in the main checkout, so runs (and their session files) survive `git worktree remove`; `firegoat.yaml`, fixtures and `build:` stay in the worktree. Each `summary.json` records `dir` (where eval ran) and `git: {branch, commit, dirty}`, and `.firegoat/runs/index.jsonl` gets one line per eval (`runId, startedAt, dir, branch, commit, passed, failed, notGraded, cost, uploaded, uploadFailed`) to tell runs from different branches apart.
-- The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, build, install, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it. Every `FIREGOAT_*` var firegoat was started with (`FIREGOAT_API_KEY` above all) is removed too.
-- Runs (tests × models × `runs`) go 4 at a time, each in its own folder with its own session; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
+- The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, build, install, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it. Every `FIREGOAT_*` var firegoat was started with (`FIREGOAT_API_KEY` above all) is removed too, and so are the vars a parent Claude Code session sets (when firegoat runs from claude, or its agent runs firegoat), so the claude under test doesn't act as a nested session: `CLAUDECODE`, `CLAUDE_CODE_*`, `CLAUDE_PID`, `CLAUDE_ENV_FILE`, `CLAUDE_SESSION_ID`, `CLAUDE_PROJECT_DIR`, `CLAUDE_EFFORT`, `CLAUDE_AGENT_SDK_*`, `CLAUDE_PLUGIN_ROOT`, `CLAUDE_PLUGIN_DATA`, `CLAUDE_SKILL_DIR`. Login and provider vars stay: `ANTHROPIC_*`, `CLAUDE_CODE_OAUTH_TOKEN`, `CLAUDE_CODE_OAUTH_REFRESH_TOKEN`, `CLAUDE_CODE_USE_*` (Bedrock, Vertex, ...), `CLAUDE_CODE_SKIP_*_AUTH`, `CLAUDE_CODE_CLIENT_CERT`, `CLAUDE_CODE_CLIENT_KEY`, `CLAUDE_CODE_CLIENT_KEY_PASSPHRASE`.
+- Runs (tests × models × `runs`) go 4 at a time, each in its own folder with its own session and HOME; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
 - Script checks run in the agent's folder afterwards and set the exit code. `end_goal` is not graded by firegoat: it's for you, or your coding agent, to judge from `run.json`, the session and the folder.
-- Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`; greppable line by line: every line of a shell call starts with `[n] $ `, every line of the agent's text and final answer with `agent: `, output is indented, so `grep -E '^\[[0-9]+\] \$ .*npm install'` matches only commands, also inside heredocs and multi-line scripts), `run.json` (status, prompt, end_goal, check results, paths), `work/`, logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
+- Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`; greppable line by line: every line of a shell call starts with `[n] $ `, every line of the agent's text and final answer with `agent: `, output is indented, so `grep -E '^\[[0-9]+\] \$ .*npm install'` matches only commands, also inside heredocs and multi-line scripts), `run.json` (status, prompt, end_goal, check results, paths), `work/`, `home/` and `home.env` (the run's HOME), logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
+
+`firegoat.yaml` keys: `runs`, `model`, `build`, `install`, `cleanup`, `vars`, `env`, `claude_args`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`, `install`, `cleanup`, `claude_args`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
+
+### Stop what the agent left running, seed one test
+
+```yaml
+install: npm init -y >/dev/null
+cleanup: pkill -f "$FIREGOAT_WORKDIR" || true   # A dev server the agent started
+tests:
+  - name: migrate
+    prompt: Migrate the app in this folder to acme v2 and start it.
+    end_goal: The app runs on acme v2.
+    install: cp -R "$FIREGOAT_OUT"/v1-app/. .     # This test only, after the top-level install
+    cleanup: acme down || true                    # This test only, before the top-level cleanup
+```
 
-`firegoat.yaml` keys: `runs`, `model`, `build`, `install`, `vars`, `env`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
+Step order: `build` once per eval, in your folder; then per run, in its folder: `install` (top-level, then the test's), the agent, checks, `cleanup` (the test's, then top-level). `cleanup` runs whenever the run's folder exists: after the checks, and also after a failed install, a claude error or a timeout. Every cleanup step runs; a failing one prints a warning and is recorded in `run.json` (`cleanup: [{run, code}]`, output in `cleanup.txt`), but never changes the run's status or eval's exit code.
 
 Want a plain `npm i acme-sdk` to get your build, with no hint in the prompt? See [`examples/local-npm-registry`](examples/local-npm-registry/README.md).
 

package/dist/cli.js

@@ -6947,7 +6947,7 @@ var require_public_api = __commonJS(function(exports) {
 // package.json
 var package_default = {
   name: "firegoat",
-  version: "0.17.1",
+  version: "0.19.0",
   repository: {
     type: "git",
     url: "git+https://github.com/firegoat-ai/firegoat-cli.git"
@@ -7078,16 +7078,20 @@ function parseConfig(text) {
     "model",
     "build",
     "install",
+    "cleanup",
     "vars",
     "env",
+    "claude_args",
     "tests"
   ]);
   const runs = posInt(raw.runs, "runs", 1);
   const models = parseModels(raw.model);
   const build = commands(raw.build, "build");
   const install = commands(raw.install, "install");
+  const cleanup = commands(raw.cleanup, "cleanup");
   const vars = parseVars(raw.vars);
   const env = parseEnv(raw.env, "env");
+  const claudeArgs = parseClaudeArgs(raw.claude_args, "claude_args");
   if (!Array.isArray(raw.tests) || raw.tests.length === 0)
     fail("tests", "must be a non-empty list");
   const names = new Set;
@@ -7095,7 +7099,17 @@ function parseConfig(text) {
     const p = `tests[${i}]`;
     if (!isRecord(t))
       fail(p, "must be a mapping");
-    onlyKeys(t, p, ["name", "prompt", "end_goal", "checks", "timeout", "env"]);
+    onlyKeys(t, p, [
+      "name",
+      "prompt",
+      "end_goal",
+      "checks",
+      "timeout",
+      "env",
+      "install",
+      "cleanup",
+      "claude_args"
+    ]);
     const name = str(t.name, `${p}.name`);
     if (!NAME.test(name))
       fail(`${p}.name`, "may only contain letters, digits, '.', '_' and '-'");
@@ -7119,7 +7133,13 @@ function parseConfig(text) {
       end_goal: str(t.end_goal, `${p}.end_goal`),
       checks,
       timeout: posInt(t.timeout, `${p}.timeout`, DEFAULT_TIMEOUT),
-      env: { ...env, ...parseEnv(t.env, `${p}.env`) }
+      env: { ...env, ...parseEnv(t.env, `${p}.env`) },
+      install: commands(t.install, `${p}.install`),
+      cleanup: commands(t.cleanup, `${p}.cleanup`),
+      claudeArgs: [
+        ...claudeArgs,
+        ...parseClaudeArgs(t.claude_args, `${p}.claude_args`)
+      ]
     };
   });
   return {
@@ -7127,6 +7147,7 @@ function parseConfig(text) {
     models,
     build,
     install,
+    cleanup,
     vars,
     env,
     tests
@@ -7158,6 +7179,41 @@ function parseModels(raw) {
     return name;
   });
 }
+var OWNED_FLAGS = [
+  [
+    [
+      "-p",
+      "--print",
+      "--output-format",
+      "--verbose",
+      "--setting-sources",
+      "--strict-mcp-config",
+      "--disable-slash-commands",
+      "--dangerously-skip-permissions"
+    ],
+    "firegoat sets it for every run"
+  ],
+  [["--model"], "choose the model with top-level model: (or eval --model)"],
+  [
+    ["--session-id", "--resume", "-r", "--continue", "-c"],
+    "every run is a fresh session"
+  ]
+];
+function parseClaudeArgs(raw, path) {
+  if (raw === undefined)
+    return [];
+  if (!Array.isArray(raw))
+    fail(path, "must be a list of claude arguments, one per entry, e.g. [--chrome]");
+  return raw.map((a, i) => {
+    const p = `${path}[${i}]`;
+    const arg = str(a, p);
+    for (const [flags, why] of OWNED_FLAGS)
+      for (const f of flags)
+        if (arg === f || arg.startsWith(`${f}=`))
+          fail(p, `${f} is not allowed: ${why}`);
+    return arg;
+  });
+}
 var VAR_KEY = /^[A-Za-z_][A-Za-z0-9_]*$/;
 function parseVars(raw) {
   if (raw === undefined)
@@ -7179,10 +7235,18 @@ var RESERVED_ENV = new Set([
   "CLAUDE_CONFIG_DIR",
   "CLAUDE_SECURESTORAGE_CONFIG_DIR"
 ]);
+var HOME_ENV = new Set([
+  "HOME",
+  "XDG_CONFIG_HOME",
+  "XDG_CACHE_HOME",
+  "XDG_DATA_HOME",
+  "XDG_STATE_HOME",
+  "CLAUDE_ENV_FILE"
+]);
 var RUN_PLACEHOLDERS = ["run.id", "run.n", "test.name"];
 var EVAL_PLACEHOLDERS = ["run.id"];
 var ANY_PLACEHOLDER = /\{\{\s*([^{}]*?)\s*\}\}/g;
-var DIR_REF = /\$(?:\{(FIREGOAT_OUT|FIREGOAT_WORKDIR)\}|(FIREGOAT_OUT|FIREGOAT_WORKDIR)(?![A-Za-z0-9_]))/g;
+var DIR_REF = /\$(?:\{(FIREGOAT_OUT|FIREGOAT_WORKDIR|FIREGOAT_HOME)\}|(FIREGOAT_OUT|FIREGOAT_WORKDIR|FIREGOAT_HOME)(?![A-Za-z0-9_]))/g;
 var ENV_REF = new RegExp(`${ANY_PLACEHOLDER.source}|${DIR_REF.source}`, "g");
 function parseEnv(raw, path) {
   if (raw === undefined)
@@ -7196,6 +7260,8 @@ function parseEnv(raw, path) {
       fail(p, "name must be letters, digits and _, not starting with a digit");
     if (k === "ANTHROPIC_MODEL")
       fail(p, "is reserved: choose the model with top-level model: (or eval --model)");
+    if (HOME_ENV.has(k))
+      fail(p, "Reserved: every run gets its own home ($FIREGOAT_HOME)");
     if (RESERVED_ENV.has(k) || k.startsWith("FIREGOAT_"))
       fail(p, "is set by firegoat and cannot be overridden (reserved: PATH, CLAUDE_CONFIG_DIR, CLAUDE_SECURESTORAGE_CONFIG_DIR, FIREGOAT_*)");
     if (typeof v !== "string")
@@ -7252,12 +7318,12 @@ import { randomBytes as randomBytes2 } from "node:crypto";
 import {
   appendFileSync,
   createWriteStream as createWriteStream2,
-  existsSync as existsSync4,
-  mkdirSync as mkdirSync3,
+  existsSync as existsSync3,
+  mkdirSync as mkdirSync4,
   readdirSync as readdirSync2,
-  writeFileSync as writeFileSync3
+  writeFileSync as writeFileSync4
 } from "node:fs";
-import { basename as basename3, delimiter as delimiter2, join as join5 } from "node:path";
+import { basename as basename3, delimiter as delimiter2, join as join6 } from "node:path";
 
 // src/claude.ts
 import { spawn, spawnSync } from "node:child_process";
@@ -7272,15 +7338,18 @@ var CLAUDE_FLAGS = [
   "--disable-slash-commands"
 ];
 var NPM_INJECTED = /^(npm_config_|npm_lifecycle_|npm_package_)|^(npm_execpath|npm_node_execpath|npm_command|INIT_CWD)$/;
+var CLAUDE_SESSION = /^(CLAUDECODE|CLAUDE_CODE_.+|CLAUDE_PID|CLAUDE_ENV_FILE|CLAUDE_SESSION_ID|CLAUDE_PROJECT_DIR|CLAUDE_EFFORT|CLAUDE_AGENT_SDK_.+|CLAUDE_PLUGIN_ROOT|CLAUDE_PLUGIN_DATA|CLAUDE_SKILL_DIR)$/;
+var CLAUDE_AUTH = /^CLAUDE_CODE_(OAUTH_TOKEN|OAUTH_REFRESH_TOKEN|USE_[A-Z_]+|SKIP_[A-Z]+_AUTH|CLIENT_CERT|CLIENT_KEY|CLIENT_KEY_PASSPHRASE)$/;
 function hostEnv() {
-  return Object.fromEntries(Object.entries(process.env).filter(([k]) => !NPM_INJECTED.test(k) && !k.startsWith("FIREGOAT_")));
+  return Object.fromEntries(Object.entries(process.env).filter(([k]) => !NPM_INJECTED.test(k) && !k.startsWith("FIREGOAT_") && (!CLAUDE_SESSION.test(k) || CLAUDE_AUTH.test(k))));
 }
-function claudeEnv(configDir, env = {}) {
+function claudeEnv(configDir, env = {}, envFile) {
   return {
     ...hostEnv(),
     ...env,
     CLAUDE_CONFIG_DIR: configDir,
-    CLAUDE_SECURESTORAGE_CONFIG_DIR: ""
+    CLAUDE_SECURESTORAGE_CONFIG_DIR: "",
+    ...envFile === undefined ? {} : { CLAUDE_ENV_FILE: envFile }
   };
 }
 function run(opts) {
@@ -7323,7 +7392,8 @@ async function runClaude(opts) {
       "-p",
       opts.prompt,
       ...CLAUDE_FLAGS,
-      ...opts.model === undefined ? [] : ["--model", opts.model]
+      ...opts.model === undefined ? [] : ["--model", opts.model],
+      ...opts.extraArgs
     ],
     cwd: opts.cwd,
     env: opts.env,
@@ -7732,28 +7802,14 @@ function resolveKey(base) {
 var EXIT = { pass: 0, fail: 1, notGraded: 2, usage: 3 };
 
 // src/firegoat-dir.ts
-import { existsSync as existsSync2, mkdirSync as mkdirSync2, writeFileSync } from "node:fs";
+import { mkdirSync as mkdirSync2, writeFileSync } from "node:fs";
 import { join as join2 } from "node:path";
-var GITIGNORE = `runs/
-.claude/
-`;
-var README = `# .firegoat
-
-Files for the evals in firegoat.yaml (run them with \`npx firegoat eval\`).
-
-- \`fixtures/\`: test fixtures (helper scripts, sample data) for build:, install: and checks. Commit them.
-- \`runs/\`: one folder per eval (transcripts, sessions, check output). Ignored by git.
-- \`.claude/\`: the empty claude config eval runs the agent with. Ignored by git.
-
-Reference: \`npx firegoat init --help\`.
+var GITIGNORE = `*
 `;
 function ensureFiregoatDir(root) {
   const dir = join2(root, ".firegoat");
-  mkdirSync2(join2(dir, "fixtures"), { recursive: true });
+  mkdirSync2(dir, { recursive: true });
   writeFileSync(join2(dir, ".gitignore"), GITIGNORE);
-  const readme = join2(dir, "README.md");
-  if (!existsSync2(readme))
-    writeFileSync(readme, README);
   return dir;
 }
 
@@ -7822,21 +7878,25 @@ var EXIT_CODES = `Exit codes:
   3  Usage or config error (bad flag, unknown test, invalid or missing firegoat.yaml,
      firegoat.ai answered 401/403 for the upload key); any other failure of
      that key check exits 2
-  Upload failures don't change the exit code.`;
+  Upload and cleanup failures don't change the exit code.`;
 function topHelp(version) {
   return `firegoat ${version}: local AX evals. Your own claude gets one prompt in a fresh
 folder, with none of your claude config; firegoat saves the session and runs your
 script checks.
 
-  npx firegoat init     Create firegoat.yaml and .firegoat/, install the skill
-  npx firegoat eval     Run the tests (claude must be logged in: run \`claude\` once)
-  npx firegoat login    Log in to firegoat.ai: eval then uploads each session there
-  npx firegoat logout   Forget that login
-  npx firegoat feedback "<what got in the way>"   Tell the firegoat team
+  npx firegoat@latest init     Create firegoat.yaml and .firegoat/, install the skill
+  npx firegoat@latest eval     Run the tests (claude must be logged in: run \`claude\`)
+  npx firegoat@latest login    Log in to firegoat.ai: eval then uploads each session
+  npx firegoat@latest logout   Forget that login
+  npx firegoat@latest feedback "<what got in the way>"
+                               Tell the firegoat team
+
+Use npx firegoat@latest: plain npx firegoat can keep running an old cached version.
+Each eval prints its version first.
 
 Typical flow for a coding agent driving firegoat:
-  1. Read npx firegoat init --help, then edit firegoat.yaml
-  2. npx firegoat eval --json
+  1. Read npx firegoat@latest init --help, then edit firegoat.yaml
+  2. npx firegoat@latest eval --json
   3. Judge each run's end_goal from its run.json, session and workdir: firegoat only
      runs the script checks
 
@@ -7850,9 +7910,9 @@ In the current folder: creates .firegoat/ and a starter firegoat.yaml, installs
 firegoat-onboarding skill globally, into ~/.claude/skills/ (npx skills add
 firegoat-ai/skills -g; in a terminal it asks: global, this repo, or none), offers
 firegoat login (in a terminal only; otherwise prints how), then prints a prompt to
-paste into your coding agent. Commit .firegoat/ with firegoat.yaml: its .gitignore
-ignores only runs/ and .claude/, and test fixtures (helper scripts, sample data)
-go in .firegoat/fixtures/. Its README.md says what each folder is.
+paste into your coding agent. .firegoat/ holds local run output and ignores itself
+in git. Keep test fixtures (helper scripts, sample data) outside it, e.g. in
+fixtures/ next to firegoat.yaml, and commit them.
 
   --force          Replace an existing firegoat.yaml
   --local-skills   Install the skill into this repo's .claude/skills/ instead
@@ -7868,26 +7928,41 @@ firegoat.yaml reference:
                               --model, claude's default. eval --model overrides
   build: <sh> | [<sh>, ...]   Once per eval, in this folder; what it writes to
                               $FIREGOAT_OUT is first on the agent's PATH
-  install: <sh> | [<sh>, ...] In each fresh folder, before the agent starts
+  install: <sh> | [<sh>, ...] In each fresh folder, before the agent starts; a
+                              failure: that run is not graded
+  cleanup: <sh> | [<sh>, ...] In each run's folder once the run is over: after the
+                              checks, and also after an error or timeout. Stops
+                              what the agent left running (servers, daemons).
+                              Every step runs; a failure is a warning (run.json
+                              cleanup, <test>/<n>/cleanup.txt), never the verdict
   vars: {KEY: value}          Defaults for {{vars.KEY}} in prompts and checks;
                               --var KEY=value overrides
-  env: {NAME: value}          Env vars for the agent, and for install and
-                              checks of the same run; values are strings and can
+  env: {NAME: value}          Env vars for the agent, and for install, checks
+                              and cleanup of the same run; values are strings and can
                               use {{vars.KEY}}, {{run.id}}, {{run.n}},
-                              {{test.name}}, $FIREGOAT_OUT, $FIREGOAT_WORKDIR
-                              (no other $ expansion: $HOME stays literal).
-                              The build step gets only the entries that use only
-                              {{vars.*}}, {{run.id}} and $FIREGOAT_OUT.
+                              {{test.name}}, $FIREGOAT_OUT, $FIREGOAT_WORKDIR,
+                              $FIREGOAT_HOME (no other $ expansion: $HOME stays
+                              literal). The build step gets only the entries that
+                              use only {{vars.*}}, {{run.id}} and $FIREGOAT_OUT.
                               PATH, CLAUDE_CONFIG_DIR, FIREGOAT_* and
-                              ANTHROPIC_MODEL (use model:) are reserved.
-                              Runs go 4 at a time: give a tool with a global home
-                              a per-run one (or use eval --parallel 1), e.g.
-                    MYTOOL_HOME: "/tmp/mytool-{{run.id}}-{{test.name}}-{{run.n}}"
-                              Setting HOME: on macOS claude reads its login from
-                              $HOME/Library, so a per-run HOME needs
-                              ln -s <real home>/Library "$HOME/Library" in
-                              install (get the real home in build, which
-                              doesn't get entries using {{test.name}}/{{run.n}}).
+                              ANTHROPIC_MODEL (use model:) are reserved, and so
+                              are HOME, XDG_CONFIG_HOME, XDG_CACHE_HOME,
+                              XDG_DATA_HOME, XDG_STATE_HOME and CLAUDE_ENV_FILE:
+                              every run gets its own home ($FIREGOAT_HOME), so a
+                              tool that keeps state in ~ starts fresh in each
+                              run. Only a tool that ignores HOME needs its own
+                              var, e.g. MYTOOL_HOME: "$FIREGOAT_HOME/.mytool"
+  claude_args: [<arg>, ...]   Extra claude arguments, after firegoat's own flags
+                              and --model. Each entry is one argument, passed
+                              as-is (no shell splitting): [--chrome] (Claude in
+                              Chrome, for tests that need a browser),
+                              [--plugin-dir, ./plugins/x], --mcp-config, --agents,
+                              --append-system-prompt. Flags firegoat sets are
+                              errors, also as --flag=value: -p, --print,
+                              --output-format, --verbose, --setting-sources,
+                              --strict-mcp-config, --disable-slash-commands,
+                              --dangerously-skip-permissions, --model (use
+                              model:), --session-id, --resume, -r, --continue, -c
   tests:
     - name: quickstart        Letters, digits, '.', '_', '-'; unique
       prompt: ...             What a user would ask the agent; one concrete outcome
@@ -7897,11 +7972,22 @@ firegoat.yaml reference:
         - run: ...            Afterwards; exit 0 = pass; they set eval's exit code
       timeout: 600            Optional, seconds
       env: {NAME: value}      Optional, on top of the top-level env:
+      install: <sh> | [...]   Optional, runs after the top-level install
+      cleanup: <sh> | [...]   Optional, runs before the top-level cleanup
+      claude_args: [...]      Optional, after the top-level claude_args
+
+  Step order: build once per eval; then per run: install (top-level, then the
+  test's), the agent, checks, cleanup (the test's, then top-level).
 
   Unknown keys are errors. Keep helper files out of firegoat.yaml: put them in
-  .firegoat/fixtures/ and copy them in build or install. The build and install
-  commands and the checks get the vars as env vars, plus $FIREGOAT_OUT and
-  $FIREGOAT_WORKDIR. Checks also get $FIREGOAT_TRANSCRIPT, a short plain-text
+  e.g. fixtures/ (committed) and copy them in build or install. To start each run
+  from a template project, commit it as e.g. fixtures/starter-app/, then:
+    build: cp -R fixtures/starter-app "$FIREGOAT_OUT/starter-app"
+    install: cp -R "$FIREGOAT_OUT/starter-app/." .
+  The build, install and cleanup commands and the checks get the vars as env
+  vars, plus $FIREGOAT_OUT and $FIREGOAT_WORKDIR; install, checks and cleanup also
+  get $FIREGOAT_HOME, the run's own home, as HOME (build keeps your real HOME).
+  Checks also get $FIREGOAT_TRANSCRIPT, a short plain-text
   transcript, greppable line by line: "[1] $ <command>" on every line of a shell
   call (one line per line of the command, same n), "[2] Write <path>" for other
   tools, each output indented, and "agent: " on every line of the agent's text and
@@ -7917,6 +8003,7 @@ firegoat.yaml reference:
 Example:
   build: npm pack --pack-destination "$FIREGOAT_OUT"
   install: npm init -y >/dev/null && npm install "$FIREGOAT_OUT"/*.tgz
+  cleanup: pkill -f "$FIREGOAT_WORKDIR" || true
   tests:
     - name: first-call
       prompt: >-
@@ -7930,6 +8017,12 @@ Example:
         - run: |
             test -s "$FIREGOAT_TRANSCRIPT" &&
               ! grep -Eq '^\\[[0-9]+\\] \\$ .*npm (install|i|add) .*acme' "$FIREGOAT_TRANSCRIPT"
+    - name: dashboard-key
+      prompt: >-
+        Get an acme API key from the acme dashboard (I'm logged in in Chrome) and put
+        it in .env as ACME_API_KEY.
+      end_goal: .env holds a key copied from the dashboard, not a made-up one.
+      claude_args: [--chrome]
 
 Exit codes: 0 done, 1 the firegoat login failed (the files stay), 2 the skill
 install failed, 3 usage error.`;
@@ -7973,11 +8066,38 @@ Runs each test in firegoat.yaml (in the current folder): your claude gets the pr
 in a fresh, empty folder, .firegoat/runs/<runId>/<test>/<n>/work, with build:'s
 outputs on PATH. Then the test's checks run in that folder.
 
+Step order: build: once, in this folder; then per run, in its folder: install:
+(top-level, then the test's), the agent, checks, cleanup: (the test's, then
+top-level). cleanup: runs whenever the folder exists, also after a failed install,
+a claude error or a timeout; a failing cleanup step prints a warning and never
+changes the run's status or the exit code.
+
 claude runs on its own login (or ANTHROPIC_API_KEY if set, as claude always does)
 with a fresh config dir, .firegoat/.claude: none of your settings, CLAUDE.md,
-plugins, skills or MCP servers. claude saves its session there. There is no
+plugins, skills or MCP servers (claude_args: in firegoat.yaml adds claude flags,
+e.g. --chrome or --plugin-dir). claude saves its session there. There is no
 sandbox: the agent can reach the rest of your machine, including the repo.
 
+Every run gets its own HOME, .firegoat/runs/<runId>/<test>/<n>/home (with
+XDG_CONFIG_HOME, XDG_CACHE_HOME, XDG_DATA_HOME, XDG_STATE_HOME inside it), as
+$FIREGOAT_HOME: tools that keep state in ~ start fresh and parallel runs don't
+collide. claude itself keeps your real HOME (its login is there) and sources
+<n>/home.env (CLAUDE_ENV_FILE) before each Bash command the agent runs; install,
+checks and cleanup get the run's HOME directly; build keeps yours. Limits: claude's
+own Read/Write tools expand ~ to your real home; getpwuid lookups (os.userInfo,
+python pwd) still see it; keychain logins (gh, docker, git credential helpers) look
+logged out inside a run, so log tools in from install: or with env: tokens; the
+fresh home has no .gitconfig, so set user.name/user.email in install: if the agent
+commits; claude's shell snapshot still sources your real ~/.zshrc.
+
+Vars a parent Claude Code session sets (CLAUDECODE, CLAUDE_CODE_*, CLAUDE_PID,
+CLAUDE_ENV_FILE, CLAUDE_SESSION_ID, CLAUDE_PROJECT_DIR, CLAUDE_EFFORT,
+CLAUDE_AGENT_SDK_*, CLAUDE_PLUGIN_ROOT, CLAUDE_PLUGIN_DATA, CLAUDE_SKILL_DIR) are
+removed from every env firegoat starts, so the claude under test doesn't act as a
+nested session. Login and provider vars stay: ANTHROPIC_*, CLAUDE_CODE_OAUTH_TOKEN,
+CLAUDE_CODE_OAUTH_REFRESH_TOKEN, CLAUDE_CODE_USE_*, CLAUDE_CODE_SKIP_*_AUTH,
+CLAUDE_CODE_CLIENT_CERT, CLAUDE_CODE_CLIENT_KEY, CLAUDE_CODE_CLIENT_KEY_PASSPHRASE.
+
 In a linked git worktree (git worktree add), .firegoat/runs/ and .firegoat/.claude
 live in the main checkout instead, so runs outlive the worktree; firegoat.yaml,
 fixtures and build: stay in the current folder.
@@ -7987,9 +8107,10 @@ fixtures and build: stay in the current folder.
   --model NAME       claude --model, replacing model: in firegoat.yaml; repeat it
                      (--model a --model b) to run each test runs times per model
   --parallel N       Runs at once (default 4; 1 runs them one after another).
-                     Tools with global on-disk state collide: give each run its
-                     own home via env: ({{run.id}}-{{test.name}}-{{run.n}}) or use 1
-  --var KEY=value    Fills {{vars.KEY}}; env for build, install and checks
+                     Each run has its own HOME; state outside it (a fixed port,
+                     /tmp paths) can still collide: use 1 then
+  --var KEY=value    Fills {{vars.KEY}}; env for build, install, checks and
+                     cleanup
   --json             The summary as JSON on stdout (or {error, exitCode} if eval
                      stops early); progress goes to stderr
   --no-upload        Keep runs local even when logged in to firegoat.ai
@@ -8015,17 +8136,21 @@ Output: .firegoat/runs/<runId>/
                           uploadFailed
   <test>/<n>/run.json     One run: status and reason (from the checks), model
                           (asked for, or null) and claudeModel (what claude's
-                          init line reported), prompt, end_goal,
-                          checks [{run, code}], and paths: workdir,
+                          init line reported), claudeArgs (claude_args
+                          passed), prompt, end_goal,
+                          checks [{run, code}], cleanup [{run, code}],
+                          and paths: workdir, home (the run's HOME),
                           transcript, session (claude's session file), stderr,
-                          checksLog; startedAt; upload ({status: "uploaded",
+                          checksLog, cleanupLog; startedAt; upload ({status: "uploaded",
                           url} | {status: "failed", error} | {status:
                           "skipped", reason}, null when not uploading)
   <test>/<n>/transcript.txt  What the agent did: each tool call with its command
                           ("[n] $ " on every line of a shell call), short output
                           (indented), its text and final answer ("agent: " on
                           every line), files it made
-  build.txt, <test>/<n>/install.txt, checks.txt   Logs
+  <test>/<n>/home/, home.env  The run's HOME, and the file claude sources
+                          before the agent's Bash commands
+  build.txt, <test>/<n>/install.txt, checks.txt, cleanup.txt   Logs
 
 Grading: firegoat runs only the script checks; they set the exit code. end_goal is
 yours to judge: read transcript.txt first (a few KB); the session file has everything
@@ -8063,6 +8188,30 @@ Example:
 
 Exit codes: 0 sent, 2 the service failed (or rate limit), 3 usage error.`;
 
+// src/home.ts
+import { mkdirSync as mkdirSync3, writeFileSync as writeFileSync2 } from "node:fs";
+import { join as join3 } from "node:path";
+function homeEnv(home) {
+  return {
+    HOME: home,
+    XDG_CONFIG_HOME: join3(home, ".config"),
+    XDG_CACHE_HOME: join3(home, ".cache"),
+    XDG_DATA_HOME: join3(home, ".local", "share"),
+    XDG_STATE_HOME: join3(home, ".local", "state")
+  };
+}
+var shellQuote = (s) => `'${s.replaceAll("'", `'\\''`)}'`;
+function makeRunHome(dir) {
+  const home = join3(dir, "home");
+  const env = homeEnv(home);
+  for (const d of Object.values(env))
+    mkdirSync3(d, { recursive: true });
+  const envFile = join3(dir, "home.env");
+  writeFileSync2(envFile, Object.entries(env).map(([k, v]) => `export ${k}=${shellQuote(v)}
+`).join(""));
+  return { home, envFile };
+}
+
 // src/steps.ts
 import { createWriteStream } from "node:fs";
 var STEP_TIMEOUT_MS = 600000;
@@ -8085,10 +8234,27 @@ async function runSteps(opts) {
     await new Promise((r) => log.end(r));
   }
 }
+async function runAllSteps(opts) {
+  const log = createWriteStream(opts.logPath);
+  const outcomes = [];
+  try {
+    for (const run of opts.commands)
+      outcomes.push(await runShell({
+        run,
+        cwd: opts.cwd,
+        env: opts.env,
+        timeoutMs: STEP_TIMEOUT_MS,
+        log
+      }));
+    return outcomes;
+  } finally {
+    await new Promise((r) => log.end(r));
+  }
+}
 
 // src/transcript.ts
-import { readdirSync, statSync, writeFileSync as writeFileSync2 } from "node:fs";
-import { join as join3, relative } from "node:path";
+import { readdirSync, statSync, writeFileSync as writeFileSync3 } from "node:fs";
+import { join as join4, relative } from "node:path";
 var OUTPUT_LINES = 15;
 var OUTPUT_CHARS = 1500;
 var TEXT_CHARS = 1500;
@@ -8132,7 +8298,7 @@ function workdirFiles(dir) {
     for (const name of readdirSync(d)) {
       if (SKIP_DIRS.has(name))
         continue;
-      const p = join3(d, name);
+      const p = join4(d, name);
       if (statSync(p).isDirectory())
         walk(p);
       else
@@ -8175,7 +8341,7 @@ ${indent(truncate(text, OUTPUT_LINES, OUTPUT_CHARS))}` : `    → ${status}`);
     out.pop();
   out.push("", "## final answer", final ? prose(final) : "(none)");
   out.push("", "## files in the workdir", ...workdirFiles(opts.workdir));
-  writeFileSync2(opts.path, `${out.join(`
+  writeFileSync3(opts.path, `${out.join(`
 `)}
 `);
 }
@@ -8355,7 +8521,7 @@ function ttyUI(out, opts) {
     const doneCount = c.pass + c.fail + c.error;
     const badge = finished ? `DONE · RUN ${finished.runId.slice(-4)}` : `RUNNING · ${doneCount}/${rows.length}`;
     const out = [
-      ...logoHeader(p, `firegoat eval · ${opts.project}`, p(finished ? C.green : C.ember, badge), w),
+      ...logoHeader(p, `firegoat ${opts.version} eval · ${opts.project}`, p(finished ? C.green : C.ember, badge), w),
       rule
     ];
     if (authText)
@@ -8459,7 +8625,7 @@ function ttyUI(out, opts) {
 }
 
 // src/upload.ts
-import { existsSync as existsSync3, readFileSync as readFileSync3 } from "node:fs";
+import { existsSync as existsSync2, readFileSync as readFileSync3 } from "node:fs";
 import { basename as basename2 } from "node:path";
 var MAX_FILE = 20 * 1024 * 1024;
 var MAX_CHECKS = 200;
@@ -8535,7 +8701,7 @@ async function uploadRun(t, r) {
   };
   const bytes = {};
   for (const name of FILES) {
-    const b = existsSync3(paths[name]) ? readFileSync3(paths[name]) : Buffer.alloc(0);
+    const b = existsSync2(paths[name]) ? readFileSync3(paths[name]) : Buffer.alloc(0);
     if (b.byteLength > MAX_FILE)
       return {
         status: "failed",
@@ -8616,7 +8782,7 @@ async function uploadRun(t, r) {
 
 // src/which.ts
 import { accessSync, constants, statSync as statSync2 } from "node:fs";
-import { delimiter, join as join4, resolve } from "node:path";
+import { delimiter, join as join5, resolve } from "node:path";
 function resolveBin(bin) {
   const executable = (p) => {
     try {
@@ -8629,8 +8795,8 @@ function resolveBin(bin) {
   if (bin.includes("/"))
     return executable(bin) ? resolve(bin) : undefined;
   for (const dir of (process.env.PATH ?? "").split(delimiter)) {
-    if (dir && executable(join4(dir, bin)))
-      return join4(dir, bin);
+    if (dir && executable(join5(dir, bin)))
+      return join5(dir, bin);
   }
   return;
 }
@@ -8703,12 +8869,12 @@ function parseArgs(args) {
   return opts;
 }
 function findSession(configDir, sessionId) {
-  const projects = join5(configDir, "projects");
-  if (!existsSync4(projects))
+  const projects = join6(configDir, "projects");
+  if (!existsSync3(projects))
     return null;
   for (const d of readdirSync2(projects)) {
-    const f = join5(projects, d, `${sessionId}.jsonl`);
-    if (existsSync4(f))
+    const f = join6(projects, d, `${sessionId}.jsonl`);
+    if (existsSync3(f))
       return f;
   }
   return null;
@@ -8718,7 +8884,7 @@ function newRunId() {
   return `${ts}-${randomBytes2(2).toString("hex")}`;
 }
 function writeRunJson(r) {
-  writeFileSync3(join5(r.dir, "run.json"), `${JSON.stringify(r, null, 2)}
+  writeFileSync4(join6(r.dir, "run.json"), `${JSON.stringify(r, null, 2)}
 `);
 }
 async function uploadPreflight(opts, ui, project, runGroup, git) {
@@ -8760,14 +8926,16 @@ async function uploadPreflight(opts, ui, project, runGroup, git) {
 async function runOne(opts) {
   const { test, n } = opts;
   const started = Date.now();
-  const dir = join5(opts.runDir, test.name, String(n));
-  const workdir = join5(dir, "work");
-  mkdirSync3(workdir, { recursive: true });
+  const dir = join6(opts.runDir, test.name, String(n));
+  const workdir = join6(dir, "work");
+  mkdirSync4(workdir, { recursive: true });
+  const { home, envFile } = makeRunHome(dir);
   const r = {
     test: test.name,
     n,
     model: opts.model,
     claudeModel: null,
+    claudeArgs: test.claudeArgs,
     status: "error",
     reason: "",
     prompt: render(test.prompt, opts.vars),
@@ -8775,51 +8943,77 @@ async function runOne(opts) {
     checks: [],
     dir,
     workdir,
+    home,
     session: null,
-    transcript: join5(dir, "transcript.txt"),
-    stderr: join5(dir, "stderr.txt"),
-    checksLog: join5(dir, "checks.txt"),
+    transcript: join6(dir, "transcript.txt"),
+    stderr: join6(dir, "stderr.txt"),
+    checksLog: join6(dir, "checks.txt"),
+    cleanup: [],
+    cleanupLog: join6(dir, "cleanup.txt"),
     num_turns: null,
     cost: null,
     duration_ms: 0,
     startedAt: new Date(started).toISOString(),
     upload: null
   };
-  const done = (status, reason) => {
-    r.status = status;
-    r.reason = reason;
-    r.duration_ms = Date.now() - started;
-    writeRunJson(r);
-    return r;
-  };
   const env = renderEnv(test.env, opts.vars, {
     "run.id": opts.runId,
     "run.n": String(n),
     "test.name": test.name,
     FIREGOAT_OUT: opts.out,
-    FIREGOAT_WORKDIR: workdir
+    FIREGOAT_WORKDIR: workdir,
+    FIREGOAT_HOME: home
   });
-  const stepEnv = { ...opts.stepEnv, ...env, FIREGOAT_WORKDIR: workdir };
-  if (opts.install.length > 0) {
+  const stepEnv = {
+    ...opts.stepEnv,
+    ...env,
+    ...homeEnv(home),
+    FIREGOAT_WORKDIR: workdir,
+    FIREGOAT_HOME: home
+  };
+  const [status, reason] = await attempt(opts, r, env, stepEnv, envFile);
+  const cleanup = [...test.cleanup, ...opts.cleanup];
+  if (cleanup.length > 0)
+    r.cleanup = await runAllSteps({
+      commands: cleanup,
+      cwd: workdir,
+      env: stepEnv,
+      logPath: r.cleanupLog
+    });
+  r.status = status;
+  r.reason = reason;
+  r.duration_ms = Date.now() - started;
+  writeRunJson(r);
+  return r;
+}
+async function attempt(opts, r, env, stepEnv, homeEnvFile) {
+  const { test, n } = opts;
+  const { dir, workdir } = r;
+  const install = [...opts.install, ...test.install];
+  if (install.length > 0) {
     const error = await runSteps({
       what: "install",
-      commands: opts.install,
+      commands: install,
       cwd: workdir,
       env: stepEnv,
-      logPath: join5(dir, "install.txt")
+      logPath: join6(dir, "install.txt")
     });
     if (error)
-      return done("error", `not graded: ${error}`);
+      return ["error", `not graded: ${error}`];
   }
   const stderr = createWriteStream2(r.stderr);
   const outcome = await runClaude({
     bin: opts.claudeBin,
     prompt: r.prompt,
     cwd: workdir,
-    env: { ...claudeEnv(opts.configDir, env), PATH: opts.stepEnv.PATH },
+    env: {
+      ...claudeEnv(opts.configDir, env, homeEnvFile),
+      PATH: opts.stepEnv.PATH
+    },
     stderr,
     timeoutMs: test.timeout * 1000,
     model: opts.model ?? undefined,
+    extraArgs: r.claudeArgs,
     onEvent: opts.onEvent
   });
   r.claudeModel = outcome.claudeModel;
@@ -8836,9 +9030,12 @@ async function runOne(opts) {
   r.num_turns = outcome.result?.num_turns ?? null;
   r.cost = outcome.result?.total_cost_usd ?? null;
   if (!outcome.ok)
-    return done("error", `not graded: ${outcome.reason}`);
+    return ["error", `not graded: ${outcome.reason}`];
   if (!r.session)
-    return done("error", `not graded: claude saved no session file for ${outcome.sessionId} under ${opts.configDir}/projects`);
+    return [
+      "error",
+      `not graded: claude saved no session file for ${outcome.sessionId} under ${opts.configDir}/projects`
+    ];
   const log = createWriteStream2(r.checksLog);
   try {
     for (const c of test.checks)
@@ -8858,8 +9055,14 @@ async function runOne(opts) {
   }
   const failed = r.checks.filter((c) => c.code !== 0);
   if (failed.length > 0)
-    return done("fail", failed.map((c) => `check failed (${c.error ?? `exit ${c.code}`}): ${c.run}`).join("; "));
-  return done("pass", test.checks.length === 0 ? "no script checks" : `${r.checks.length}/${r.checks.length} checks passed`);
+    return [
+      "fail",
+      failed.map((c) => `check failed (${c.error ?? `exit ${c.code}`}): ${c.run}`).join("; ")
+    ];
+  return [
+    "pass",
+    test.checks.length === 0 ? "no script checks" : `${r.checks.length}/${r.checks.length} checks passed`
+  ];
 }
 async function evalCommand(args, root) {
   if (args.includes("--help") || args.includes("-h")) {
@@ -8883,7 +9086,13 @@ async function evalCommand(args, root) {
   }
   const say = opts.json ? console.error : console.log;
   const stream = opts.json ? process.stderr : process.stdout;
-  const ui = stream.isTTY ? ttyUI(stream, { project: basename3(root), color: !process.env.NO_COLOR }) : plainUI(say);
+  const ui = stream.isTTY ? ttyUI(stream, {
+    project: basename3(root),
+    version: package_default.version,
+    color: !process.env.NO_COLOR
+  }) : plainUI(say);
+  if (!stream.isTTY)
+    say(`firegoat ${package_default.version}`);
   let config;
   try {
     config = loadConfig(root);
@@ -8922,8 +9131,8 @@ async function evalCommand(args, root) {
     return fatal(EXIT.usage, e.message);
   }
   const firegoatDir = ensureFiregoatDir(repo?.mainCheckout ?? root);
-  const configDir = join5(firegoatDir, ".claude");
-  mkdirSync3(configDir, { recursive: true });
+  const configDir = join6(firegoatDir, ".claude");
+  mkdirSync4(configDir, { recursive: true });
   ui.auth(process.env.ANTHROPIC_API_KEY ? "ANTHROPIC_API_KEY (claude bills the API key; unset it to use your claude login)" : "your claude login");
   const loginProblem = checkLogin({
     bin: claudePath,
@@ -8945,10 +9154,10 @@ async function evalCommand(args, root) {
       return fatal(EXIT.usage, e.message);
     throw e;
   }
-  const runsRoot = join5(firegoatDir, "runs");
-  const runDir = join5(runsRoot, runId);
-  const out = join5(runDir, "out");
-  mkdirSync3(out, { recursive: true });
+  const runsRoot = join6(firegoatDir, "runs");
+  const runDir = join6(runsRoot, runId);
+  const out = join6(runDir, "out");
+  mkdirSync4(out, { recursive: true });
   const stepEnv = {
     ...hostEnv(),
     ...vars,
@@ -8974,7 +9183,7 @@ async function evalCommand(args, root) {
         ...stepEnv,
         ...renderEnv(Object.fromEntries(Object.entries(config.env).filter(([, v]) => isEvalLevel(v))), vars, { "run.id": runId, FIREGOAT_OUT: out })
       },
-      logPath: join5(runDir, "build.txt")
+      logPath: join6(runDir, "build.txt")
     });
     const secs = Math.round((Date.now() - buildStarted) / 1000);
     ui.build(error ? "fail" : "ok", secs);
@@ -9011,6 +9220,7 @@ async function evalCommand(args, root) {
       claudeBin: claudePath,
       configDir,
       install: config.install,
+      cleanup: config.cleanup,
       out,
       stepEnv,
       vars
@@ -9046,6 +9256,7 @@ async function evalCommand(args, root) {
     totals[r.status]++;
   const uploadsFailed = results.filter((r) => r.upload?.status === "failed").length;
   const doc = {
+    firegoatVersion: package_default.version,
     runId,
     runsDir: runDir,
     dir: root,
@@ -9054,9 +9265,9 @@ async function evalCommand(args, root) {
     totals,
     runs: results
   };
-  writeFileSync3(join5(runDir, "summary.json"), `${JSON.stringify(doc, null, 2)}
+  writeFileSync4(join6(runDir, "summary.json"), `${JSON.stringify(doc, null, 2)}
 `);
-  appendFileSync(join5(runsRoot, "index.jsonl"), `${JSON.stringify({
+  appendFileSync(join6(runsRoot, "index.jsonl"), `${JSON.stringify({
     runId,
     startedAt: new Date(evalStarted).toISOString(),
     dir: root,
@@ -9076,6 +9287,10 @@ async function evalCommand(args, root) {
     rows,
     seconds: Math.round((Date.now() - evalStarted) / 1000)
   });
+  for (const r of results)
+    for (const c of r.cleanup)
+      if (c.code !== 0)
+        console.error(`firegoat eval: Warning: ${r.test}#${r.n} cleanup failed (${c.error ?? `exit ${c.code}`}): ${c.run} (log: ${r.cleanupLog})`);
   if (uploadsFailed > 0)
     console.error(`firegoat eval: ${uploadsFailed} upload${uploadsFailed === 1 ? "" : "s"} failed (each run's upload.error in summary.json)`);
   if (opts.json)
@@ -9352,8 +9567,8 @@ firegoat ${package_default.version}` : `firegoat ${package_default.version}`,
 
 // src/init.ts
 import { spawnSync as spawnSync3 } from "node:child_process";
-import { existsSync as existsSync5, writeFileSync as writeFileSync4 } from "node:fs";
-import { basename as basename4, join as join6 } from "node:path";
+import { existsSync as existsSync4, writeFileSync as writeFileSync5 } from "node:fs";
+import { basename as basename4, join as join7 } from "node:path";
 import { createInterface } from "node:readline/promises";
 
 // src/login.ts
@@ -9548,7 +9763,7 @@ function skillsDir(scope) {
   if (scope === "local")
     return ".claude/skills/";
   const config = process.env.CLAUDE_CONFIG_DIR?.trim();
-  return config ? `${join6(config, "skills")}/` : "~/.claude/skills/";
+  return config ? `${join7(config, "skills")}/` : "~/.claude/skills/";
 }
 function template(product) {
   return `# firegoat.yaml: evals for how well coding agents use ${product}.
@@ -9579,34 +9794,48 @@ runs: 1
 # (An npm/pypi CLI that users run via npx/uvx: pack + install it as above instead;
 # a binary on PATH does not stop "npx mycli" from fetching the registry release)
 #
-# Helper files (scripts, sample data) go in .firegoat/fixtures/ (committed), not inline here.
-# The build step runs in this folder, so: build: cp -R .firegoat/fixtures/. "$FIREGOAT_OUT"
+# Helper files (scripts, sample data) go in e.g. fixtures/ (committed), not inline here.
+# The build step runs in this folder, so: build: cp -R fixtures/. "$FIREGOAT_OUT"
 # Then e.g. install: cp "$FIREGOAT_OUT/wizard.mjs" .
-
-# Optional: env vars for the agent (and for install and checks of the same
-# run). The agent gets none of vars:, only these. Placeholders: {{vars.KEY}},
-# {{run.id}}, {{run.n}}, {{test.name}}, and $FIREGOAT_OUT / $FIREGOAT_WORKDIR (no
-# other $ expansion: $HOME stays literal). A test can add its own env:.
-# Runs go 4 at a time (--parallel), so give a tool with a global home (~/.mytool)
-# a per-run one, as below, or use --parallel 1.
+#
+# To start each run from a template project, commit it as e.g. fixtures/starter-app/:
+# build: cp -R fixtures/starter-app "$FIREGOAT_OUT/starter-app"
+# install: cp -R "$FIREGOAT_OUT/starter-app/." .
+
+# Optional: cleanup: runs in each run's folder once the run is over (after the
+# checks, also after an error or timeout), e.g. to stop a server the agent left
+# running. A failing cleanup step is a warning, never the verdict.
+# cleanup:
+#   - pkill -f "$FIREGOAT_WORKDIR" || true
+#
+# A test can add its own install: (after this one) and cleanup: (before this one).
+
+# Optional: env vars for the agent (and for install, checks and cleanup of the
+# same run). The agent gets none of vars:, only these. Placeholders: {{vars.KEY}},
+# {{run.id}}, {{run.n}}, {{test.name}}, and $FIREGOAT_OUT / $FIREGOAT_WORKDIR /
+# $FIREGOAT_HOME (no other $ expansion: $HOME stays literal). A test can add its
+# own env:. Every run gets its own HOME ($FIREGOAT_HOME), so tools that keep state
+# in ~ are isolated per run already; set a tool's own *_HOME var only if the tool
+# ignores HOME, as below.
 # env:
-#   MYTOOL_HOME: "/tmp/mytool-{{run.id}}-{{test.name}}-{{run.n}}"
+#   MYTOOL_HOME: "$FIREGOAT_HOME/.mytool"
 #   UV_FIND_LINKS: "$FIREGOAT_OUT"
 
 tests:
-  - name: quickstart
-    # What a user would ask their agent. Edit this.
-    prompt: >-
-      I want to try ${product}. Get the smallest working example going in a new
-      project, run it, and save what it prints to output.txt. Keep it minimal.
+  # A smoke test that firegoat and claude work. Replace it with tests for your
+  # product: the firegoat-onboarding skill writes them.
+  - name: goat
+    # What a user would ask their agent.
+    prompt: Who's the GOAT? Answer in one short sentence.
     # True when the user's problem is solved. Not graded by firegoat: it goes to
     # your agent (eval --json, run.json) to judge.
-    end_goal: >-
-      A minimal project uses ${product}, runs without errors, and output.txt holds
-      its output.
+    end_goal: Claude answers that you (the user) are the goat, e.g. YOU'RE THE GOAT.
     # Shell commands run in the agent's folder afterwards; exit 0 = pass.
+    # $FIREGOAT_TRANSCRIPT has "agent: " on every line of the agent's text.
     checks:
-      - run: test -s output.txt
+      - run: >-
+          test -s "$FIREGOAT_TRANSCRIPT" &&
+          grep -Eiq '^agent: .*goat' "$FIREGOAT_TRANSCRIPT"
     # Seconds before the agent run is stopped.
     timeout: 600
 `;
@@ -9725,16 +9954,16 @@ ${INIT_HELP}`);
     console.log(rule);
   }
   ensureFiregoatDir(cwd);
-  const configPath = join6(cwd, CONFIG_FILE);
-  const keep = existsSync5(configPath) && !force;
+  const configPath = join7(cwd, CONFIG_FILE);
+  const keep = existsSync4(configPath) && !force;
   if (!keep)
-    writeFileSync4(configPath, template(basename4(cwd)));
+    writeFileSync5(configPath, template(basename4(cwd)));
   if (interactive) {
     step(1, CONFIG_FILE, keep ? "Kept (--force replaces it)" : "Created");
     console.log(paint("90", "  │"));
     step(2, ".firegoat/", "Ready");
   } else
-    console.log(keep ? `Kept ${CONFIG_FILE} (--force replaces it)` : `Wrote ${CONFIG_FILE}: edit the quickstart test to fit your users`);
+    console.log(keep ? `Kept ${CONFIG_FILE} (--force replaces it)` : `Wrote ${CONFIG_FILE}: replace the goat smoke test with tests for your product`);
   if (interactive && !noSkills)
     console.log(paint("90", "  │"));
   if (interactive && !noSkills && !localSkills) {
@@ -9788,7 +10017,7 @@ Prompt for your coding agent: ${AGENT_PROMPT}`);
   console.log(AGENT_PROMPT);
   console.log("");
   const copied = copyToClipboard(AGENT_PROMPT);
-  console.log(`  ${copied ? `${paint("32", "✓ Prompt copied!")}   ` : ""}Then: ${paint("36", "npx firegoat eval")}`);
+  console.log(`  ${copied ? `${paint("32", "✓ Prompt copied!")}   ` : ""}Then: ${paint("36", "npx firegoat@latest eval")}`);
   return EXIT.pass;
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "firegoat",
-	"version": "0.17.1",
+	"version": "0.19.0",
 	"repository": {
 		"type": "git",
 		"url": "git+https://github.com/firegoat-ai/firegoat-cli.git"