firegoat 0.21.0 → 0.22.1

package/README.md

@@ -1,40 +1,28 @@
 # firegoat
 
-Local AX evals: how well coding agents use your SDK, CLI, API and docs. firegoat gives your own `claude` one prompt in a fresh folder, saves the session, and runs your checks.
+Firegoat is an agent experience platform that lets devtools see how agents use their product and improve the agent experience.
 
-Status: early. See `plan.md`.
+It's for developers who build for agents as customers: SDKs, CLIs, APIs, MCP servers, cloud products, local products, and the people writing documentation.
 
-## Quickstart
+So far we've got the CLI. It lets anyone use a simplified version of our product, running on their local machine with their own Claude subscription and the Claude Code they already have installed. Support for other harnesses is coming soon.
 
-```bash
-npx firegoat@latest init     # Creates firegoat.yaml and .firegoat/, installs the firegoat-onboarding skill, copies a prompt for your agent
-# Edit firegoat.yaml (or paste the prompt into your coding agent and let it)
-npx firegoat@latest eval     # Runs your claude on each test, in .firegoat/runs/<runId>/<test>/<n>/work
-```
+## What you get
 
-That's it: two commands. Use `npx firegoat@latest`: plain `npx firegoat` can keep running an old cached version. `eval` prints its version first (`firegoat X.Y.Z`), and `summary.json` records it as `firegoatVersion`. `claude` must be installed and logged in (run `claude` once).
+When you run `firegoat eval`, you get visibility into how agents use your product: whether they fail or succeed, and how much time they take doing it. You can point them at your local codebase or at your cloud product, and see how Claude acts on the simple tasks your product aims to do.
 
-`init` installs the firegoat-onboarding skill globally, into `~/.claude/skills/` (in a terminal it asks: global, this repo, or none). `--local-skills` puts it in this repo's `.claude/skills/` instead (commit it so teammates and CI get it); `--no-skills` skips it.
-
-## Upload to firegoat.ai
+## Start
 
 ```bash
-npx firegoat login    # Device flow: open the URL, enter the code; key saved to ~/.firegoat/credentials.json (0600)
-npx firegoat eval     # Now also uploads each run as a private trace and prints its URL
-npx firegoat logout   # Forgets the login (revoke the key itself on the firegoat.ai traces page)
+npx firegoat@latest init
 ```
 
-Logging in means agreeing to the firegoat.ai [Terms of Service](https://firegoat.ai/terms) and [Privacy Policy](https://firegoat.ai/privacy); login says so before showing the code. `init` offers the login in a terminal (`--no-login` skips it). In CI, set `FIREGOAT_API_KEY=fire_...` instead. `eval --no-upload` keeps runs local; without a key, runs stay local and eval says so in one line.
-
-- Per run, eval uploads claude's session (`session.jsonl`: prompts, tool calls and outputs, file contents, so possibly secrets from your env or repo), `checks.txt` and `stderr.txt` (check output sees `vars:`), plus the run's test, status, checks, the model asked for and the one claude reported, and (in a git repo) the branch, commit and whether the tree was dirty. firegoat.ai masks secrets when it shows a trace, but stores what was sent. Traces start private; you share one from its page.
-- Before any run, eval checks the key (`GET /api/traces/me`) and prints `Uploading sessions to <origin> as <email>`: a key answered with 401/403 exits 3; any other failure (unreachable, 429, 5xx, a wrong `FIREGOAT_BASE_URL` path) exits 2.
-- Each run prints `✓ <test>#<n> <url>` or `✗ <test>#<n> Upload failed: <why>`, and `run.json` / `summary.json` get `upload` (`{status: "uploaded", url}`, `{status: "failed", error}`, `{status: "skipped", reason}`, or null when not uploading). A failed upload never changes a run's status or eval's exit code (0/1/2 stay the eval's verdict); eval ends with `N uploads failed` on stderr. Each file may be at most 20 MiB: a bigger one fails that run's upload. Runs without a session file are skipped.
-- The key is never passed to the agent, build, install or checks (every `FIREGOAT_*` var is removed from their env), is never written to run files, and goes only to the base URL's origin, never through a redirect; presigned upload URLs get no auth header. But there is no sandbox: the agent runs as you, so it can read `~/.firegoat/credentials.json`, and with `FIREGOAT_API_KEY` it can read the key from firegoat's own process environment (`ps eww $PPID`, `/proc/<pid>/environ`), which no program can scrub. In CI, use a dedicated key and revoke it if a run looks off. The key can only upload and read your own traces.
-- `FIREGOAT_BASE_URL` (default `https://firegoat.ai/evals`) points firegoat elsewhere: https only, http only for `localhost` / `127.0.0.1`. A saved login only works for the base URL it was made for.
+Then paste the prompt it prints into your agent, and let it set up the evals that point agents at your product. Then run them and see the results:
 
-## Examples
+```bash
+npx firegoat@latest eval
+```
 
-### Can an agent get started with my npm package?
+## Example
 
 ```yaml
 build: npm pack --pack-destination "$FIREGOAT_OUT"
@@ -49,226 +37,15 @@ tests:
       - run: grep -qE 'evt_[A-Za-z0-9]+' "$FIREGOAT_TRANSCRIPT"
 ```
 
-`build` builds your package once; `install` installs that build into each fresh folder, so the agent uses your local code, not the registry.
-
-### Does my CLI make sense to an agent?
-
-```yaml
-build: go build -o "$FIREGOAT_OUT/acme" ./cmd/acme
-tests:
-  - name: deploy
-    prompt: Deploy the hello-world template to a preview environment with the acme CLI and print the URL.
-    end_goal: The agent finds the right acme commands and ends with a working preview URL.
-    checks:
-      - run: grep -q '\$ acme deploy' "$FIREGOAT_TRANSCRIPT"
-```
-
-Whatever `build` writes to `$FIREGOAT_OUT` is first on the agent's `PATH`. That only helps for binaries called by bare name: if users run your CLI via `npx`/`uvx`, pack and install it like the npm example, or `npx` fetches the registry release.
-
-### Can an agent find the right package from my docs alone?
-
-```yaml
-tests:
-  - name: which-packages
-    prompt: >-
-      I'm building a mobile app with Acme. It needs a local database, maps and Sign in
-      with Apple. Which Acme packages should I use? Write the list to packages.txt.
-    end_goal: packages.txt names the right Acme package for each need, found in Acme's docs.
-    checks:
-      - run: test -s packages.txt
-```
-
-### Python, three runs, and a value from the command line
-
-```yaml
-runs: 3
-build: uv build --wheel --out-dir "$FIREGOAT_OUT"
-install: uv venv && uv pip install "$FIREGOAT_OUT"/*.whl
-tests:
-  - name: sandbox-hello
-    prompt: >-
-      Using the acme Python SDK (already installed), create a sandbox on commit
-      {{vars.SHA}}, run `echo hello-from-acme` in it, print the output, then delete it.
-    end_goal: The script prints hello-from-acme from a sandbox and deletes the sandbox.
-    checks:
-      - run: grep -rq hello-from-acme "$FIREGOAT_WORKDIR"
-```
-
-```bash
-npx firegoat eval --var SHA=$(git rev-parse HEAD)
-```
-
-### Start from a template project
-
-```yaml
-build: cp -R fixtures/starter-app "$FIREGOAT_OUT/starter-app"
-install: cp -R "$FIREGOAT_OUT/starter-app/." .
-tests:
-  - name: add-auth
-    prompt: Add acme login to this app. Keep the existing pages working.
-    end_goal: The app signs users in with acme, and the existing pages still render.
-    checks:
-      - run: npm test
-```
-
-No template feature needed: keep the project committed in `fixtures/starter-app/`, `build` copies it once per eval, and `install` copies it into each fresh folder, so every run begins in its own copy of that project.
-
-### Same tests on several models
-
-```yaml
-model: [claude-haiku-5-5, claude-sonnet-5-5]
-runs: 2
-tests:
-  - name: quickstart
-    prompt: Get the acme SDK's hello-world example running and save its output to output.txt.
-    end_goal: output.txt holds the example's output.
-    checks:
-      - run: test -s output.txt
-```
-
-`model:` is passed to claude as `--model`. A list is a matrix: each test runs `runs` times per model, numbered `1..runs×models` (here `quickstart/1`, `2` on haiku, `3`, `4` on sonnet). `npx firegoat eval --model claude-opus-5-5` replaces it for one eval; repeat the flag for a matrix. Unset, firegoat passes no `--model` and claude uses its default. Each run's `run.json` records `model` (what firegoat asked for, or null) and `claudeModel` (what claude reported starting with), and the output labels each run with its model.
-
-### Compare two setups
-
-```yaml
-build: npm pack --pack-destination "$FIREGOAT_OUT"
-tests:
-  - name: deploy-local-build
-    prompt: &deploy Deploy this app to an acme preview environment and print the URL.
-    end_goal: &deployed The app runs on an acme preview URL, and the URL is printed.
-    install: npm install "$FIREGOAT_OUT"/acme-*.tgz
-  - name: deploy-npm-release
-    prompt: *deploy
-    end_goal: *deployed
-    install: npm install acme@latest
-```
-
-There is no variants feature: to compare setups (your local build vs the npm release, with vs without `--chrome`, docs A vs docs B), write one test per setup with the same prompt and a different per-test `install:`, `env:` or `claude_args:`. A YAML anchor (`&deploy`) and alias (`*deploy`) keep the prompt in one place. Skills installed into the run's folder are not loaded as skills: eval runs claude with `--setting-sources ""` and `--disable-slash-commands`, so a with/without-skill comparison isn't possible yet.
-
-### Extra claude flags
-
-```yaml
-claude_args: [--plugin-dir, ./plugins/acme]   # Every test
-tests:
-  - name: dashboard-key
-    prompt: Get an acme API key from the acme dashboard (I'm logged in in Chrome) and put it in .env as ACME_API_KEY.
-    end_goal: .env holds a key copied from the dashboard, not a made-up one.
-    claude_args: [--chrome]                    # Claude in Chrome, this test only
-```
-
-`claude_args:` (top level, and per test after the top-level ones) is passed to claude after firegoat's own flags and `--model`. Each entry is one argument, as-is, with no shell splitting: `[--plugin-dir, ./plugins/acme]`, not `["--plugin-dir ./plugins/acme"]`. Use it for `--chrome`, `--plugin-dir`, `--mcp-config`, `--agents`, `--append-system-prompt`. The flags firegoat sets itself are config errors, also as `--flag=value`: `-p`, `--print`, `--output-format`, `--verbose`, `--setting-sources`, `--strict-mcp-config`, `--disable-slash-commands`, `--dangerously-skip-permissions`, `--model` (use `model:`), and `--session-id`, `--resume`, `-r`, `--continue`, `-c` (every run is a fresh session). Entries expand like `env:` values: `$FIREGOAT_WORKDIR`, `$FIREGOAT_HOME`, `$FIREGOAT_OUT` (also `${...}`), `{{vars.KEY}}`, `{{run.id}}`, `{{run.n}}`, `{{test.name}}`; no other `$` expansion. Each run's `run.json` records the expanded `claudeArgs`.
-
-### Testing an MCP server
-
-Commit the server's config as `fixtures/mcp.json`:
-
-```json
-{"mcpServers": {"acme": {"command": "./node_modules/.bin/acme-mcp",
-  "env": {"HOME": "${FIREGOAT_HOME}"}}}}
-```
+## Uploads
 
-Copy it in `build:` and pass it with `claude_args:`:
-
-```yaml
-build:
-  - npm pack --pack-destination "$FIREGOAT_OUT"
-  - cp -R fixtures/. "$FIREGOAT_OUT"
-install: npm init -y >/dev/null && npm install "$FIREGOAT_OUT"/*.tgz
-claude_args: [--mcp-config, $FIREGOAT_OUT/mcp.json]
-```
-
-claude gets `FIREGOAT_HOME`, `FIREGOAT_WORKDIR` and `FIREGOAT_OUT` in its own env, set per run, and expands `${VAR}` in an MCP config's `command`, `args`, `env`, `url` and `headers` from its env (an unset var stays the literal text). Relative paths in the config are from the run's folder, claude's cwd: `./node_modules/.bin/acme-mcp` is the build under test, installed by `install:`. MCP servers that claude starts inherit claude's own env, which keeps your real `HOME` (claude needs it for its login): set `"env": {"HOME": "${FIREGOAT_HOME}"}` on the server, as above, or it writes to your real home. Per-run state the server keeps elsewhere goes the same way, e.g. `"ACME_DB": "${FIREGOAT_WORKDIR}/acme.db"`. `--strict-mcp-config` is always on, so only the configs you pass load. To check that the agent used a tool, grep the transcript: `grep -Eq '^\[[0-9]+\] mcp__acme__search' "$FIREGOAT_TRANSCRIPT"`.
-
-With `--chrome`, claude points the Claude in Chrome extension's native host (`~/Library/Application Support/<browser>/NativeMessagingHosts/com.anthropic.claude_code_browser_extension.json`) at `.firegoat/.claude`. If you delete `.firegoat/`, run a normal `claude --chrome` once to point it back.
-
-### Env vars for the agent: a feature flag, a tool that ignores HOME
-
-```yaml
-env:
-  ACME_HOME: "$FIREGOAT_HOME/.acme"             # Only for a tool that ignores HOME
-  ACME_TELEMETRY: "off"
-  UV_FIND_LINKS: "$FIREGOAT_OUT"                # Resolve the wheel build: made
-tests:
-  - name: login
-    prompt: Log in to acme with the CLI and print the current user.
-    end_goal: The agent completes acme's login flow and prints the user.
-    env:
-      ACME_BETA_LOGIN: "1"                        # This test only, on top of env:
-    checks:
-      - run: ls "$ACME_HOME"/sessions | grep -q .
-```
+Log in to your firegoat.ai account and each trace is uploaded to our system automatically. Secrets are masked in everything you see, but the uploaded files are stored as sent. You can opt out of uploading with `eval --no-upload`. You can also delete your traces on the dashboard, and the files and the record are gone from our site.
 
-`env:` goes to the agent, and to `install`, checks and `cleanup` of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` / `$FIREGOAT_HOME` (also `${...}`), the same dirs `build`, `install` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `build` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`. `HOME`, `XDG_CONFIG_HOME`, `XDG_CACHE_HOME`, `XDG_DATA_HOME`, `XDG_STATE_HOME` and `CLAUDE_ENV_FILE` are reserved too: every run gets its own home.
+## Links
 
-### Each run gets its own HOME
+- Everything else (every config key, more examples, how eval runs claude): [docs/reference.md](docs/reference.md) and `npx firegoat@latest eval --help`
+- The skill your agent uses to write evals: [firegoat-ai/skills](https://github.com/firegoat-ai/skills)
+- Tell us what got in your way: `npx firegoat@latest feedback "<what happened>"`
+- [firegoat.ai](https://firegoat.ai)
 
-Every run has its own home, `.firegoat/runs/<runId>/<test>/<n>/home/`, with `.config`, `.cache`, `.local/share` and `.local/state` for the XDG dirs. A tool that keeps logins, config or caches in `~` starts fresh in every run, and parallel runs don't collide. Point a tool's own `*_HOME` var at `$FIREGOAT_HOME` only if the tool ignores `HOME`.
-
-- The agent's shell commands see it: firegoat writes `<n>/home.env` (exporting `HOME` and `XDG_CONFIG_HOME`, `XDG_CACHE_HOME`, `XDG_DATA_HOME`, `XDG_STATE_HOME`) and passes it to claude as `CLAUDE_ENV_FILE`, which claude sources before every Bash command. claude itself keeps your real `HOME`, where its login is (the macOS keychain lookup, `~/.claude/.credentials.json` on Linux).
-- `install`, checks and `cleanup` get `HOME` and the XDG vars set to the run's home, plus `$FIREGOAT_HOME`. `build` keeps your real home.
-- `run.json` records it as `home`.
-
-Limits:
-
-- claude's own Read/Write/Edit tools expand `~` to your real home; only Bash commands get the run's home.
-- Lookups by user id (`os.userInfo().homedir`, python's `pwd`, `getpwuid`) still return your real home.
-- Keychain-backed logins (`gh`, `docker`, git credential helpers) look logged out inside a run. Log tools in from `install:`, or pass tokens with `env:`.
-- The fresh home has no `.gitconfig`: if the agent commits, set `git config --global user.name ...` / `user.email` in `install:`.
-- claude's shell snapshot still sources your real `~/.zshrc` (or `~/.bashrc`).
-
-## What eval does
-
-- claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers (add flags with `claude_args:`, see above). claude saves its session there, in its own format.
-- No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. `.firegoat/` holds local run output and ignores itself in git. Keep test fixtures (scripts, sample data) in `fixtures/` next to `firegoat.yaml`, committed.
-- In a linked git worktree (`git worktree add`), `.firegoat/runs/` and `.firegoat/.claude/` are in the main checkout, so runs (and their session files) survive `git worktree remove`; `firegoat.yaml`, fixtures and `build:` stay in the worktree. Each `summary.json` records `dir` (where eval ran) and `git: {branch, commit, dirty}`, and `.firegoat/runs/index.jsonl` gets one line per eval (`runId, startedAt, dir, branch, commit, passed, failed, notGraded, cost, uploaded, uploadFailed`) to tell runs from different branches apart.
-- The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, build, install, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it. Every `FIREGOAT_*` var firegoat was started with (`FIREGOAT_API_KEY` above all) is removed too (claude then gets the run's own `FIREGOAT_OUT`, `FIREGOAT_WORKDIR` and `FIREGOAT_HOME`, for MCP configs; the agent's commands see them too), and so are the vars a parent Claude Code session sets (when firegoat runs from claude, or its agent runs firegoat), so the claude under test doesn't act as a nested session: `CLAUDECODE`, `CLAUDE_CODE_*`, `CLAUDE_PID`, `CLAUDE_ENV_FILE`, `CLAUDE_SESSION_ID`, `CLAUDE_PROJECT_DIR`, `CLAUDE_EFFORT`, `CLAUDE_AGENT_SDK_*`, `CLAUDE_PLUGIN_ROOT`, `CLAUDE_PLUGIN_DATA`, `CLAUDE_SKILL_DIR`. Login and provider vars stay: `ANTHROPIC_*`, `CLAUDE_CODE_OAUTH_TOKEN`, `CLAUDE_CODE_OAUTH_REFRESH_TOKEN`, `CLAUDE_CODE_USE_*` (Bedrock, Vertex, ...), `CLAUDE_CODE_SKIP_*_AUTH`, `CLAUDE_CODE_CLIENT_CERT`, `CLAUDE_CODE_CLIENT_KEY`, `CLAUDE_CODE_CLIENT_KEY_PASSPHRASE`.
-- Runs (tests × models × `runs`) go 4 at a time, each in its own folder with its own session and HOME; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
-- Script checks run in the agent's folder afterwards and set the exit code. `end_goal` is not graded by firegoat: it's for you, or your coding agent, to judge from `run.json`, the session and the folder.
-- Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`; greppable line by line: every line of a shell call starts with `[n] $ `, every line of the agent's text and final answer with `agent: `, output is indented, so `grep -E '^\[[0-9]+\] \$ .*npm install'` matches only commands, also inside heredocs and multi-line scripts), `run.json` (status, prompt, end_goal, check results, paths), `work/`, `home/` and `home.env` (the run's HOME), logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
-
-`firegoat.yaml` keys: `runs`, `model`, `build`, `install`, `cleanup`, `vars`, `env`, `claude_args`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`, `install`, `cleanup`, `claude_args`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
-
-### Stop what the agent left running, seed one test
-
-```yaml
-install: npm init -y >/dev/null
-cleanup: pkill -f "$FIREGOAT_WORKDIR" || true   # A dev server the agent started
-tests:
-  - name: migrate
-    prompt: Migrate the app in this folder to acme v2 and start it.
-    end_goal: The app runs on acme v2.
-    install: cp -R "$FIREGOAT_OUT"/v1-app/. .     # This test only, after the top-level install
-    cleanup: acme down || true                    # This test only, before the top-level cleanup
-```
-
-Step order: `build` once per eval, in your folder; then per run, in its folder: `install` (top-level, then the test's), the agent, checks, `cleanup` (the test's, then top-level). `cleanup` runs whenever the run's folder exists: after the checks, and also after a failed install, a claude error or a timeout. Every cleanup step runs; a failing one prints a warning and is recorded in `run.json` (`cleanup: [{run, code}]`, output in `cleanup.txt`), but never changes the run's status or eval's exit code.
-
-Want a plain `npm i acme-sdk` to get your build, with no hint in the prompt? See [`examples/local-npm-registry`](examples/local-npm-registry/README.md).
-
-| exit code | meaning |
-|---|---|
-| 0 | Every run's checks passed |
-| 1 | A check failed |
-| 2 | At least one run was not graded (claude missing or logged out, crash, timeout, build failed); wins over 1 |
-| 3 | Usage or config error (including firegoat.ai rejecting the upload key) |
-
-Upload failures don't change the exit code.
-
-## Feedback
-
-Something in firegoat got in your way (or your agent's)? Tell us from the terminal:
-
-```sh
-npx firegoat feedback "eval hid claude's spend-limit error" --type bug --body "ran npx firegoat eval, got 'not graded' x3"
-```
-
-It goes to the firegoat team with the firegoat version, your OS and node version, and which coding agent sent it. Don't include secrets: a report that looks like it has one is refused. The first send registers your device and saves its token to `~/.config/firegoat/feedback-device.json` (`$XDG_CONFIG_HOME` respected), which raises the daily report limit.
-
-## Develop
-
-```bash
-bun install
-bun run format
-bun test                     # End-to-end: packs the package and drives the installed bin
-FIREGOAT_E2E_REAL=1 bun test # Also runs one eval against your real claude
-```
+If every run passes, firegoat calls you GOATED. Who's the GOAT? You are.

package/dist/cli.js

@@ -6947,7 +6947,7 @@ var require_public_api = __commonJS(function(exports) {
 // package.json
 var package_default = {
   name: "firegoat",
-  version: "0.21.0",
+  version: "0.22.1",
   repository: {
     type: "git",
     url: "git+https://github.com/firegoat-ai/firegoat-cli.git"
@@ -8010,11 +8010,9 @@ firegoat.yaml reference:
   vars, plus $FIREGOAT_OUT and $FIREGOAT_WORKDIR; install, checks and cleanup also
   get $FIREGOAT_HOME, the run's own home, as HOME (build keeps your real HOME).
   Checks also get $FIREGOAT_TRANSCRIPT, a short plain-text
-  transcript, greppable line by line: "[1] $ <command>" on every line of a shell
-  call (one line per line of the command, same n), "[2] Write <path>" for other
-  tools, each output indented, and "agent: " on every line of the agent's text and
-  final answer. So '^\\[[0-9]+\\] \\$ .*foo' matches only commands. Checks also
-  get $FIREGOAT_SESSION (claude's full session file, JSON lines).
+  transcript, greppable line by line (format: eval --help, Transcript). So
+  '^\\[[0-9]+\\] \\$ .*foo' matches only commands. Checks also get
+  $FIREGOAT_SESSION (claude's full session file, JSON lines).
 
   The agent never gets vars, only env:. The vars npm/npx set for firegoat itself
   (npm_config_*, npm_lifecycle_*, npm_package_*, npm_execpath, npm_node_execpath,
@@ -8098,7 +8096,8 @@ page.
 Exit codes: 0 logged out (or was not logged in), 3 the file or ~/.firegoat is a
 symlink, not a regular file/directory, or someone else's (left alone).`;
 var EVAL_HELP = `Usage: firegoat eval [test..] [--runs N] [--model NAME].. [--parallel N]
-                     [--var KEY=value].. [--json] [--no-upload] [--claude-bin PATH]
+                     [--var KEY=value].. [--json] [--no-upload] [--dry-run]
+                     [--claude-bin PATH]
 
 Runs each test in firegoat.yaml (in the current folder): your claude gets the prompt
 in a fresh, empty folder, .firegoat/runs/<runId>/<test>/<n>/work, with build:'s
@@ -8152,6 +8151,14 @@ fixtures and build: stay in the current folder.
   --json             The summary as JSON on stdout (or {error, exitCode} if eval
                      stops early); progress goes to stderr
   --no-upload        Keep runs local even when logged in to firegoat.ai
+  --dry-run          Check firegoat.yaml as eval would (test names, --var,
+                     --model, missing vars), print each planned run
+                     ("<test>#<n> [model]" with its install, checks and cleanup
+                     counts and rendered claude_args, paths under
+                     runs/<runId>/) and exit 0. Builds nothing, starts no
+                     claude, uploads nothing, writes no files. Config errors
+                     exit 3. With --json: {dryRun, build, runs: [{test, n,
+                     model, claudeArgs, install, checks, cleanup}]}
   --claude-bin PATH  Path to claude (default: claude on PATH)
 
 Upload: after firegoat login (or with FIREGOAT_API_KEY), each run's claude session,
@@ -8181,14 +8188,25 @@ Output: .firegoat/runs/<runId>/
                           checksLog, cleanupLog; startedAt; upload ({status: "uploaded",
                           url} | {status: "failed", error} | {status:
                           "skipped", reason}, null when not uploading)
-  <test>/<n>/transcript.txt  What the agent did: each tool call with its command
-                          ("[n] $ " on every line of a shell call), short output
-                          (indented), its text and final answer ("agent: " on
-                          every line), files it made
+  <test>/<n>/transcript.txt  What the agent did, below
   <test>/<n>/home/, home.env  The run's HOME, and the file claude sources
                           before the agent's Bash commands
   build.txt, <test>/<n>/install.txt, checks.txt, cleanup.txt   Logs
 
+Transcript ($FIREGOAT_TRANSCRIPT), greppable line by line; n counts tool calls:
+  [n] $ <command>         A shell call, on every line of the command, uncut
+  [n] <Tool> <target>     A tool (MCP too) with a file_path, path, url, pattern
+                          or query input: that value, e.g. "[2] Write src/a.ts"
+  [n] <Tool> <json>       Any other tool, MCP ones included: its input as one
+                          line of JSON, e.g. [3] mcp__<server>__<tool> {"q":"x"};
+                          cut at 4000 chars
+      → ok | → error      Then the tool's output, indented 4 spaces, cut at 15
+                          lines or 1500 chars ("… (N more chars)")
+  agent: <text>           Every line of the agent's text (cut at 30 lines or
+                          1500 chars) and of its final answer (uncut)
+  Then the files in the workdir (not node_modules, .git, .venv, __pycache__).
+  A written file's body is not in the transcript: check the file in the workdir.
+
 Grading: firegoat runs only the script checks; they set the exit code. end_goal is
 yours to judge: read transcript.txt first (a few KB); the session file has everything
 (often 50x larger).
@@ -8199,6 +8217,7 @@ Examples:
   npx firegoat eval --model claude-haiku-5-5 --model claude-sonnet-5-5
                                             Each test on both models
   npx firegoat eval --var SHA=$(git rev-parse HEAD)
+  npx firegoat eval --dry-run               Validate firegoat.yaml, list the runs
 
 A run can take minutes (default timeout 600s per test).
 
@@ -8295,6 +8314,7 @@ import { join as join4, relative } from "node:path";
 var OUTPUT_LINES = 15;
 var OUTPUT_CHARS = 1500;
 var TEXT_CHARS = 1500;
+var INPUT_CHARS = 4000;
 var SKIP_DIRS = new Set(["node_modules", ".git", ".venv", "__pycache__"]);
 function truncate(text, maxLines, maxChars) {
   let out = text.length > maxChars ? text.slice(0, maxChars) : text;
@@ -8319,7 +8339,7 @@ function describeCall(step, name, input) {
   for (const key of ["file_path", "path", "url", "pattern", "query"])
     if (typeof input[key] === "string")
       return `[${step}] ${name} ${input[key]}`;
-  return `[${step}] ${name} ${truncate(JSON.stringify(input), 1, 300)}`;
+  return `[${step}] ${name} ${truncate(JSON.stringify(input), 1, INPUT_CHARS)}`;
 }
 function describeToolUse(step, b) {
   if (typeof b.name !== "string" || b.name === "")
@@ -8879,6 +8899,7 @@ function parseArgs(args) {
     parallel: 4,
     json: false,
     noUpload: false,
+    dryRun: false,
     vars: {},
     claudeBin: "claude"
   };
@@ -8899,6 +8920,8 @@ function parseArgs(args) {
       opts.json = true;
     else if (flag === "--no-upload")
       opts.noUpload = true;
+    else if (flag === "--dry-run")
+      opts.dryRun = true;
     else if (flag === "--var") {
       const kv = value();
       const at = kv.indexOf("=");
@@ -8989,6 +9012,51 @@ async function uploadPreflight(opts, ui, project, runGroup, git) {
     };
   }
 }
+function runFill(runId, n, testName, out, dir) {
+  return {
+    "run.id": runId,
+    "run.n": String(n),
+    "test.name": testName,
+    FIREGOAT_OUT: out,
+    FIREGOAT_WORKDIR: join6(dir, "work"),
+    FIREGOAT_HOME: join6(dir, "home")
+  };
+}
+function renderClaudeArgs(test, vars, fill) {
+  return test.claudeArgs.map((a, i) => renderValue(a, vars, fill, `claude_args[${i}]`));
+}
+function planRuns(tests, models, runs) {
+  const matrix = models.length > 0 ? models : [null];
+  return tests.flatMap((t) => matrix.flatMap((model, m) => Array.from({ length: runs }, (_, i) => ({
+    test: t.name,
+    n: m * runs + i + 1,
+    model
+  }))));
+}
+var DRY_RUN_ID = "<runId>";
+function dryRun(opts) {
+  const out = join6(opts.runsRoot, DRY_RUN_ID, "out");
+  const runs = opts.plan.map((p) => {
+    const test = opts.tests.find((t) => t.name === p.test);
+    const fill = runFill(DRY_RUN_ID, p.n, test.name, out, join6(opts.runsRoot, DRY_RUN_ID, test.name, String(p.n)));
+    renderEnv(test.env, opts.vars, fill);
+    return {
+      ...p,
+      claudeArgs: renderClaudeArgs(test, opts.vars, fill),
+      install: [...opts.config.install, ...test.install],
+      checks: test.checks.map((c) => render(c.run, opts.vars)),
+      cleanup: [...test.cleanup, ...opts.config.cleanup]
+    };
+  });
+  if (opts.json) {
+    console.log(JSON.stringify({ dryRun: true, build: opts.config.build, runs }, null, 2));
+    return;
+  }
+  const count = (n, what) => `${n} ${what}${n === 1 ? "" : "s"}`;
+  opts.say(`Dry run: ${count(runs.length, "run")}, build ${count(opts.config.build.length, "step")}. Nothing was started.`);
+  for (const r of runs)
+    opts.say(`${r.test}#${r.n}${r.model === null ? "" : ` [${r.model}]`} install ${r.install.length}, checks ${r.checks.length}, cleanup ${r.cleanup.length}${r.claudeArgs.length > 0 ? `, claude_args ${JSON.stringify(r.claudeArgs)}` : ""}`);
+}
 async function runOne(opts) {
   const { test, n } = opts;
   const started = Date.now();
@@ -8996,20 +9064,13 @@ async function runOne(opts) {
   const workdir = join6(dir, "work");
   mkdirSync4(workdir, { recursive: true });
   const { home, envFile } = makeRunHome(dir);
-  const fill = {
-    "run.id": opts.runId,
-    "run.n": String(n),
-    "test.name": test.name,
-    FIREGOAT_OUT: opts.out,
-    FIREGOAT_WORKDIR: workdir,
-    FIREGOAT_HOME: home
-  };
+  const fill = runFill(opts.runId, n, test.name, opts.out, dir);
   const r = {
     test: test.name,
     n,
     model: opts.model,
     claudeModel: null,
-    claudeArgs: test.claudeArgs.map((a, i) => renderValue(a, opts.vars, fill, `claude_args[${i}]`)),
+    claudeArgs: renderClaudeArgs(test, opts.vars, fill),
     status: "error",
     reason: "",
     prompt: render(test.prompt, opts.vars),
@@ -9206,6 +9267,30 @@ async function evalCommand(args, root) {
   ];
   if (missing.length > 0)
     return fatal(EXIT.usage, `No value for ${missing.map((k) => `{{vars.${k}}}`).join(", ")}: pass --var ${missing[0]}=... or set it under vars: in firegoat.yaml`);
+  if (opts.dryRun) {
+    let repo;
+    try {
+      repo = gitRepo(root);
+    } catch (e) {
+      return fatal(EXIT.usage, e.message);
+    }
+    try {
+      dryRun({
+        config,
+        tests,
+        plan: planRuns(tests, models, runs),
+        vars,
+        runsRoot: join6(repo?.mainCheckout ?? root, ".firegoat", "runs"),
+        json: opts.json,
+        say
+      });
+    } catch (e) {
+      if (e instanceof ConfigError)
+        return fatal(EXIT.usage, e.message);
+      throw e;
+    }
+    return EXIT.pass;
+  }
   const claudePath = resolveBin(opts.claudeBin);
   if (!claudePath)
     return fatal(EXIT.notGraded, `Not graded: claude binary ${JSON.stringify(opts.claudeBin)} not found. Install Claude Code (https://claude.com/claude-code) or pass --claude-bin PATH.`);
@@ -9249,14 +9334,11 @@ async function evalCommand(args, root) {
     FIREGOAT_OUT: out,
     PATH: [out, process.env.PATH ?? ""].join(delimiter2)
   };
-  const matrix = models.length > 0 ? models : [null];
-  const rows = tests.flatMap((t) => matrix.flatMap((model, m) => Array.from({ length: runs }, (_, i) => ({
-    test: t.name,
-    n: m * runs + i + 1,
-    model,
+  const rows = planRuns(tests, models, runs).map((p) => ({
+    ...p,
     state: "queued",
     turns: 0
-  }))));
+  }));
   if (config.build.length > 0) {
     const buildStarted = Date.now();
     ui.build("running");
@@ -9664,7 +9746,8 @@ var TIMEOUT_MS2 = 30000;
 var MAX_EXPIRES_S = 900;
 var MAX_INTERVAL_S = 60;
 var FORM = { "Content-Type": "application/x-www-form-urlencoded" };
-var UPLOAD_NOTICE = "Each eval uploads every agent session (prompts, tool outputs, file contents, possibly secrets from your env or repo) to firegoat.ai as private traces; --no-upload keeps runs local. The agent runs as you and can read ~/.firegoat (and a FIREGOAT_API_KEY in firegoat's own process env).";
+var UPLOADS_ON = "Eval uploads each session to firegoat.ai as a private trace; --no-upload keeps runs local.";
+var UPLOAD_NOTICE = `${UPLOADS_ON} A session holds prompts, tool outputs and file contents: possibly secrets from your env or repo. The agent runs as you and can read ~/.firegoat (and a FIREGOAT_API_KEY in firegoat's own process env).`;
 var TERMS = "By logging in, you agree to the firegoat.ai Terms of Service (https://firegoat.ai/terms) and Privacy Policy (https://firegoat.ai/privacy).";
 function str2(o, k, step) {
   const v = o?.[k];
@@ -9967,13 +10050,16 @@ async function askScope(question) {
 var LOGIN_HINT = "Runs stay local: npx firegoat login uploads them to firegoat.ai";
 async function loginStep(interactive, done, paint) {
   const label = "firegoat.ai login";
+  const uploadsOn = () => console.log(`      ${paint("90", UPLOADS_ON)}`);
   const key = resolveKey(baseUrl());
   if (key) {
     const who = key.source === "env" ? "FIREGOAT_API_KEY is set" : `Logged in as ${key.email}`;
-    if (interactive)
+    if (interactive) {
       done(label, who);
-    else
-      console.log(who);
+      uploadsOn();
+    } else
+      console.log(`${who}
+${UPLOADS_ON}`);
     return EXIT.pass;
   }
   if (!interactive) {
@@ -9989,6 +10075,7 @@ async function loginStep(interactive, done, paint) {
   try {
     const email = await loginFlow((l) => console.log(`  ${l}`), true);
     done(label, `Logged in as ${email}`);
+    uploadsOn();
     return EXIT.pass;
   } catch (e) {
     if (!(e instanceof ApiError))

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "firegoat",
-	"version": "0.21.0",
+	"version": "0.22.1",
 	"repository": {
 		"type": "git",
 		"url": "git+https://github.com/firegoat-ai/firegoat-cli.git"