firegoat 0.17.0 → 0.18.0

package/README.md

@@ -130,21 +130,36 @@ tests:
       - run: ls "$ACME_HOME"/sessions | grep -q .
 ```
 
-`env:` goes to the agent, and to `install` and checks of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` (also `${...}`), the same dirs `build`, `install` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `build` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`.
+`env:` goes to the agent, and to `install`, checks and `cleanup` of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` (also `${...}`), the same dirs `build`, `install` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `build` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`.
 
 Setting `HOME`: on macOS claude reads its login from `$HOME/Library`, so with a per-run `HOME` every run says "Not logged in" unless `install` runs `ln -s <real home>/Library "$HOME/Library"`. Get the real home in `build`, which doesn't get entries using `{{test.name}}` or `{{run.n}}` (e.g. `build: echo "$HOME" > "$FIREGOAT_OUT/.real-home"`).
 
 ## What eval does
 
 - claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers. claude saves its session there, in its own format.
-- No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. In `.firegoat/`, git ignores everything except `fixtures/` and its own `.gitignore`: keep test fixtures (scripts, sample data) in `.firegoat/fixtures/` and commit them and `.firegoat/.gitignore` with `firegoat.yaml`.
+- No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. `.firegoat/` holds local run output and ignores itself in git. Keep test fixtures (scripts, sample data) outside it, e.g. in `fixtures/` next to `firegoat.yaml`, and commit them.
 - In a linked git worktree (`git worktree add`), `.firegoat/runs/` and `.firegoat/.claude/` are in the main checkout, so runs (and their session files) survive `git worktree remove`; `firegoat.yaml`, fixtures and `build:` stay in the worktree. Each `summary.json` records `dir` (where eval ran) and `git: {branch, commit, dirty}`, and `.firegoat/runs/index.jsonl` gets one line per eval (`runId, startedAt, dir, branch, commit, passed, failed, notGraded, cost, uploaded, uploadFailed`) to tell runs from different branches apart.
 - The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, build, install, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it. Every `FIREGOAT_*` var firegoat was started with (`FIREGOAT_API_KEY` above all) is removed too.
 - Runs (tests × models × `runs`) go 4 at a time, each in its own folder with its own session; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
 - Script checks run in the agent's folder afterwards and set the exit code. `end_goal` is not graded by firegoat: it's for you, or your coding agent, to judge from `run.json`, the session and the folder.
 - Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`; greppable line by line: every line of a shell call starts with `[n] $ `, every line of the agent's text and final answer with `agent: `, output is indented, so `grep -E '^\[[0-9]+\] \$ .*npm install'` matches only commands, also inside heredocs and multi-line scripts), `run.json` (status, prompt, end_goal, check results, paths), `work/`, logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
 
-`firegoat.yaml` keys: `runs`, `model`, `build`, `install`, `vars`, `env`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
+`firegoat.yaml` keys: `runs`, `model`, `build`, `install`, `cleanup`, `vars`, `env`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`, `install`, `cleanup`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
+
+### Stop what the agent left running, seed one test
+
+```yaml
+install: npm init -y >/dev/null
+cleanup: pkill -f "$FIREGOAT_WORKDIR" || true   # A dev server the agent started
+tests:
+  - name: migrate
+    prompt: Migrate the app in this folder to acme v2 and start it.
+    end_goal: The app runs on acme v2.
+    install: cp -R "$FIREGOAT_OUT"/v1-app/. .     # This test only, after the top-level install
+    cleanup: acme down || true                    # This test only, before the top-level cleanup
+```
+
+Step order: `build` once per eval, in your folder; then per run, in its folder: `install` (top-level, then the test's), the agent, checks, `cleanup` (the test's, then top-level). `cleanup` runs whenever the run's folder exists: after the checks, and also after a failed install, a claude error or a timeout. Every cleanup step runs; a failing one prints a warning and is recorded in `run.json` (`cleanup: [{run, code}]`, output in `cleanup.txt`), but never changes the run's status or eval's exit code.
 
 Want a plain `npm i acme-sdk` to get your build, with no hint in the prompt? See [`examples/local-npm-registry`](examples/local-npm-registry/README.md).
 

package/dist/cli.js

@@ -6947,7 +6947,7 @@ var require_public_api = __commonJS(function(exports) {
 // package.json
 var package_default = {
   name: "firegoat",
-  version: "0.17.0",
+  version: "0.18.0",
   repository: {
     type: "git",
     url: "git+https://github.com/firegoat-ai/firegoat-cli.git"
@@ -7078,6 +7078,7 @@ function parseConfig(text) {
     "model",
     "build",
     "install",
+    "cleanup",
     "vars",
     "env",
     "tests"
@@ -7086,6 +7087,7 @@ function parseConfig(text) {
   const models = parseModels(raw.model);
   const build = commands(raw.build, "build");
   const install = commands(raw.install, "install");
+  const cleanup = commands(raw.cleanup, "cleanup");
   const vars = parseVars(raw.vars);
   const env = parseEnv(raw.env, "env");
   if (!Array.isArray(raw.tests) || raw.tests.length === 0)
@@ -7095,7 +7097,16 @@ function parseConfig(text) {
     const p = `tests[${i}]`;
     if (!isRecord(t))
       fail(p, "must be a mapping");
-    onlyKeys(t, p, ["name", "prompt", "end_goal", "checks", "timeout", "env"]);
+    onlyKeys(t, p, [
+      "name",
+      "prompt",
+      "end_goal",
+      "checks",
+      "timeout",
+      "env",
+      "install",
+      "cleanup"
+    ]);
     const name = str(t.name, `${p}.name`);
     if (!NAME.test(name))
       fail(`${p}.name`, "may only contain letters, digits, '.', '_' and '-'");
@@ -7119,7 +7130,9 @@ function parseConfig(text) {
       end_goal: str(t.end_goal, `${p}.end_goal`),
       checks,
       timeout: posInt(t.timeout, `${p}.timeout`, DEFAULT_TIMEOUT),
-      env: { ...env, ...parseEnv(t.env, `${p}.env`) }
+      env: { ...env, ...parseEnv(t.env, `${p}.env`) },
+      install: commands(t.install, `${p}.install`),
+      cleanup: commands(t.cleanup, `${p}.cleanup`)
     };
   });
   return {
@@ -7127,6 +7140,7 @@ function parseConfig(text) {
     models,
     build,
     install,
+    cleanup,
     vars,
     env,
     tests
@@ -7735,9 +7749,6 @@ var EXIT = { pass: 0, fail: 1, notGraded: 2, usage: 3 };
 import { mkdirSync as mkdirSync2, writeFileSync } from "node:fs";
 import { join as join2 } from "node:path";
 var GITIGNORE = `*
-!.gitignore
-!fixtures/
-!fixtures/**
 `;
 function ensureFiregoatDir(root) {
   const dir = join2(root, ".firegoat");
@@ -7811,7 +7822,7 @@ var EXIT_CODES = `Exit codes:
   3  Usage or config error (bad flag, unknown test, invalid or missing firegoat.yaml,
      firegoat.ai answered 401/403 for the upload key); any other failure of
      that key check exits 2
-  Upload failures don't change the exit code.`;
+  Upload and cleanup failures don't change the exit code.`;
 function topHelp(version) {
   return `firegoat ${version}: local AX evals. Your own claude gets one prompt in a fresh
 folder, with none of your claude config; firegoat saves the session and runs your
@@ -7839,9 +7850,9 @@ In the current folder: creates .firegoat/ and a starter firegoat.yaml, installs
 firegoat-onboarding skill globally, into ~/.claude/skills/ (npx skills add
 firegoat-ai/skills -g; in a terminal it asks: global, this repo, or none), offers
 firegoat login (in a terminal only; otherwise prints how), then prints a prompt to
-paste into your coding agent. In .firegoat/, git ignores everything but fixtures/
-and the .gitignore: keep test fixtures (helper scripts, sample data) in
-.firegoat/fixtures/ and commit them, and .firegoat/.gitignore, with firegoat.yaml.
+paste into your coding agent. .firegoat/ holds local run output and ignores itself
+in git. Keep test fixtures (helper scripts, sample data) outside it, e.g. in
+fixtures/ next to firegoat.yaml, and commit them.
 
   --force          Replace an existing firegoat.yaml
   --local-skills   Install the skill into this repo's .claude/skills/ instead
@@ -7857,11 +7868,17 @@ firegoat.yaml reference:
                               --model, claude's default. eval --model overrides
   build: <sh> | [<sh>, ...]   Once per eval, in this folder; what it writes to
                               $FIREGOAT_OUT is first on the agent's PATH
-  install: <sh> | [<sh>, ...] In each fresh folder, before the agent starts
+  install: <sh> | [<sh>, ...] In each fresh folder, before the agent starts; a
+                              failure: that run is not graded
+  cleanup: <sh> | [<sh>, ...] In each run's folder once the run is over: after the
+                              checks, and also after an error or timeout. Stops
+                              what the agent left running (servers, daemons).
+                              Every step runs; a failure is a warning (run.json
+                              cleanup, <test>/<n>/cleanup.txt), never the verdict
   vars: {KEY: value}          Defaults for {{vars.KEY}} in prompts and checks;
                               --var KEY=value overrides
-  env: {NAME: value}          Env vars for the agent, and for install and
-                              checks of the same run; values are strings and can
+  env: {NAME: value}          Env vars for the agent, and for install, checks
+                              and cleanup of the same run; values are strings and can
                               use {{vars.KEY}}, {{run.id}}, {{run.n}},
                               {{test.name}}, $FIREGOAT_OUT, $FIREGOAT_WORKDIR
                               (no other $ expansion: $HOME stays literal).
@@ -7886,10 +7903,15 @@ firegoat.yaml reference:
         - run: ...            Afterwards; exit 0 = pass; they set eval's exit code
       timeout: 600            Optional, seconds
       env: {NAME: value}      Optional, on top of the top-level env:
+      install: <sh> | [...]   Optional, runs after the top-level install
+      cleanup: <sh> | [...]   Optional, runs before the top-level cleanup
+
+  Step order: build once per eval; then per run: install (top-level, then the
+  test's), the agent, checks, cleanup (the test's, then top-level).
 
   Unknown keys are errors. Keep helper files out of firegoat.yaml: put them in
-  .firegoat/fixtures/ and copy them in build or install. The build and install
-  commands and the checks get the vars as env vars, plus $FIREGOAT_OUT and
+  e.g. fixtures/ (committed) and copy them in build or install. The build, install
+  and cleanup commands and the checks get the vars as env vars, plus $FIREGOAT_OUT and
   $FIREGOAT_WORKDIR. Checks also get $FIREGOAT_TRANSCRIPT, a short plain-text
   transcript, greppable line by line: "[1] $ <command>" on every line of a shell
   call (one line per line of the command, same n), "[2] Write <path>" for other
@@ -7906,6 +7928,7 @@ firegoat.yaml reference:
 Example:
   build: npm pack --pack-destination "$FIREGOAT_OUT"
   install: npm init -y >/dev/null && npm install "$FIREGOAT_OUT"/*.tgz
+  cleanup: pkill -f "$FIREGOAT_WORKDIR" || true
   tests:
     - name: first-call
       prompt: >-
@@ -7962,6 +7985,12 @@ Runs each test in firegoat.yaml (in the current folder): your claude gets the pr
 in a fresh, empty folder, .firegoat/runs/<runId>/<test>/<n>/work, with build:'s
 outputs on PATH. Then the test's checks run in that folder.
 
+Step order: build: once, in this folder; then per run, in its folder: install:
+(top-level, then the test's), the agent, checks, cleanup: (the test's, then
+top-level). cleanup: runs whenever the folder exists, also after a failed install,
+a claude error or a timeout; a failing cleanup step prints a warning and never
+changes the run's status or the exit code.
+
 claude runs on its own login (or ANTHROPIC_API_KEY if set, as claude always does)
 with a fresh config dir, .firegoat/.claude: none of your settings, CLAUDE.md,
 plugins, skills or MCP servers. claude saves its session there. There is no
@@ -7978,7 +8007,8 @@ fixtures and build: stay in the current folder.
   --parallel N       Runs at once (default 4; 1 runs them one after another).
                      Tools with global on-disk state collide: give each run its
                      own home via env: ({{run.id}}-{{test.name}}-{{run.n}}) or use 1
-  --var KEY=value    Fills {{vars.KEY}}; env for build, install and checks
+  --var KEY=value    Fills {{vars.KEY}}; env for build, install, checks and
+                     cleanup
   --json             The summary as JSON on stdout (or {error, exitCode} if eval
                      stops early); progress goes to stderr
   --no-upload        Keep runs local even when logged in to firegoat.ai
@@ -8005,16 +8035,17 @@ Output: .firegoat/runs/<runId>/
   <test>/<n>/run.json     One run: status and reason (from the checks), model
                           (asked for, or null) and claudeModel (what claude's
                           init line reported), prompt, end_goal,
-                          checks [{run, code}], and paths: workdir,
+                          checks [{run, code}], cleanup [{run, code}],
+                          and paths: workdir,
                           transcript, session (claude's session file), stderr,
-                          checksLog; startedAt; upload ({status: "uploaded",
+                          checksLog, cleanupLog; startedAt; upload ({status: "uploaded",
                           url} | {status: "failed", error} | {status:
                           "skipped", reason}, null when not uploading)
   <test>/<n>/transcript.txt  What the agent did: each tool call with its command
                           ("[n] $ " on every line of a shell call), short output
                           (indented), its text and final answer ("agent: " on
                           every line), files it made
-  build.txt, <test>/<n>/install.txt, checks.txt   Logs
+  build.txt, <test>/<n>/install.txt, checks.txt, cleanup.txt   Logs
 
 Grading: firegoat runs only the script checks; they set the exit code. end_goal is
 yours to judge: read transcript.txt first (a few KB); the session file has everything
@@ -8074,6 +8105,23 @@ async function runSteps(opts) {
     await new Promise((r) => log.end(r));
   }
 }
+async function runAllSteps(opts) {
+  const log = createWriteStream(opts.logPath);
+  const outcomes = [];
+  try {
+    for (const run of opts.commands)
+      outcomes.push(await runShell({
+        run,
+        cwd: opts.cwd,
+        env: opts.env,
+        timeoutMs: STEP_TIMEOUT_MS,
+        log
+      }));
+    return outcomes;
+  } finally {
+    await new Promise((r) => log.end(r));
+  }
+}
 
 // src/transcript.ts
 import { readdirSync, statSync, writeFileSync as writeFileSync2 } from "node:fs";
@@ -8768,19 +8816,14 @@ async function runOne(opts) {
     transcript: join5(dir, "transcript.txt"),
     stderr: join5(dir, "stderr.txt"),
     checksLog: join5(dir, "checks.txt"),
+    cleanup: [],
+    cleanupLog: join5(dir, "cleanup.txt"),
     num_turns: null,
     cost: null,
     duration_ms: 0,
     startedAt: new Date(started).toISOString(),
     upload: null
   };
-  const done = (status, reason) => {
-    r.status = status;
-    r.reason = reason;
-    r.duration_ms = Date.now() - started;
-    writeRunJson(r);
-    return r;
-  };
   const env = renderEnv(test.env, opts.vars, {
     "run.id": opts.runId,
     "run.n": String(n),
@@ -8789,16 +8832,35 @@ async function runOne(opts) {
     FIREGOAT_WORKDIR: workdir
   });
   const stepEnv = { ...opts.stepEnv, ...env, FIREGOAT_WORKDIR: workdir };
-  if (opts.install.length > 0) {
+  const [status, reason] = await attempt(opts, r, env, stepEnv);
+  const cleanup = [...test.cleanup, ...opts.cleanup];
+  if (cleanup.length > 0)
+    r.cleanup = await runAllSteps({
+      commands: cleanup,
+      cwd: workdir,
+      env: stepEnv,
+      logPath: r.cleanupLog
+    });
+  r.status = status;
+  r.reason = reason;
+  r.duration_ms = Date.now() - started;
+  writeRunJson(r);
+  return r;
+}
+async function attempt(opts, r, env, stepEnv) {
+  const { test, n } = opts;
+  const { dir, workdir } = r;
+  const install = [...opts.install, ...test.install];
+  if (install.length > 0) {
     const error = await runSteps({
       what: "install",
-      commands: opts.install,
+      commands: install,
       cwd: workdir,
       env: stepEnv,
       logPath: join5(dir, "install.txt")
     });
     if (error)
-      return done("error", `not graded: ${error}`);
+      return ["error", `not graded: ${error}`];
   }
   const stderr = createWriteStream2(r.stderr);
   const outcome = await runClaude({
@@ -8825,9 +8887,12 @@ async function runOne(opts) {
   r.num_turns = outcome.result?.num_turns ?? null;
   r.cost = outcome.result?.total_cost_usd ?? null;
   if (!outcome.ok)
-    return done("error", `not graded: ${outcome.reason}`);
+    return ["error", `not graded: ${outcome.reason}`];
   if (!r.session)
-    return done("error", `not graded: claude saved no session file for ${outcome.sessionId} under ${opts.configDir}/projects`);
+    return [
+      "error",
+      `not graded: claude saved no session file for ${outcome.sessionId} under ${opts.configDir}/projects`
+    ];
   const log = createWriteStream2(r.checksLog);
   try {
     for (const c of test.checks)
@@ -8847,8 +8912,14 @@ async function runOne(opts) {
   }
   const failed = r.checks.filter((c) => c.code !== 0);
   if (failed.length > 0)
-    return done("fail", failed.map((c) => `check failed (${c.error ?? `exit ${c.code}`}): ${c.run}`).join("; "));
-  return done("pass", test.checks.length === 0 ? "no script checks" : `${r.checks.length}/${r.checks.length} checks passed`);
+    return [
+      "fail",
+      failed.map((c) => `check failed (${c.error ?? `exit ${c.code}`}): ${c.run}`).join("; ")
+    ];
+  return [
+    "pass",
+    test.checks.length === 0 ? "no script checks" : `${r.checks.length}/${r.checks.length} checks passed`
+  ];
 }
 async function evalCommand(args, root) {
   if (args.includes("--help") || args.includes("-h")) {
@@ -9000,6 +9071,7 @@ async function evalCommand(args, root) {
       claudeBin: claudePath,
       configDir,
       install: config.install,
+      cleanup: config.cleanup,
       out,
       stepEnv,
       vars
@@ -9065,6 +9137,10 @@ async function evalCommand(args, root) {
     rows,
     seconds: Math.round((Date.now() - evalStarted) / 1000)
   });
+  for (const r of results)
+    for (const c of r.cleanup)
+      if (c.code !== 0)
+        console.error(`firegoat eval: Warning: ${r.test}#${r.n} cleanup failed (${c.error ?? `exit ${c.code}`}): ${c.run} (log: ${r.cleanupLog})`);
   if (uploadsFailed > 0)
     console.error(`firegoat eval: ${uploadsFailed} upload${uploadsFailed === 1 ? "" : "s"} failed (each run's upload.error in summary.json)`);
   if (opts.json)
@@ -9568,12 +9644,20 @@ runs: 1
 # (An npm/pypi CLI that users run via npx/uvx: pack + install it as above instead;
 # a binary on PATH does not stop "npx mycli" from fetching the registry release)
 #
-# Helper files (scripts, sample data) go in .firegoat/fixtures/ (committed), not inline here.
-# The build step runs in this folder, so: build: cp -R .firegoat/fixtures/. "$FIREGOAT_OUT"
+# Helper files (scripts, sample data) go in e.g. fixtures/ (committed), not inline here.
+# The build step runs in this folder, so: build: cp -R fixtures/. "$FIREGOAT_OUT"
 # Then e.g. install: cp "$FIREGOAT_OUT/wizard.mjs" .
 
-# Optional: env vars for the agent (and for install and checks of the same
-# run). The agent gets none of vars:, only these. Placeholders: {{vars.KEY}},
+# Optional: cleanup: runs in each run's folder once the run is over (after the
+# checks, also after an error or timeout), e.g. to stop a server the agent left
+# running. A failing cleanup step is a warning, never the verdict.
+# cleanup:
+#   - pkill -f "$FIREGOAT_WORKDIR" || true
+#
+# A test can add its own install: (after this one) and cleanup: (before this one).
+
+# Optional: env vars for the agent (and for install, checks and cleanup of the
+# same run). The agent gets none of vars:, only these. Placeholders: {{vars.KEY}},
 # {{run.id}}, {{run.n}}, {{test.name}}, and $FIREGOAT_OUT / $FIREGOAT_WORKDIR (no
 # other $ expansion: $HOME stays literal). A test can add its own env:.
 # Runs go 4 at a time (--parallel), so give a tool with a global home (~/.mytool)

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "firegoat",
-	"version": "0.17.0",
+	"version": "0.18.0",
 	"repository": {
 		"type": "git",
 		"url": "git+https://github.com/firegoat-ai/firegoat-cli.git"