firegoat 0.15.1 → 0.17.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +47 -27
  2. package/dist/cli.js +1099 -226
  3. package/package.json +2 -2
package/README.md CHANGED
@@ -2,27 +2,43 @@
2
2
 
3
3
  Local AX evals: how well coding agents use your SDK, CLI, API and docs. firegoat gives your own `claude` one prompt in a fresh folder, saves the session, and runs your checks.
4
4
 
5
- Status: early (v0.11). See `plan.md`.
5
+ Status: early. See `plan.md`.
6
6
 
7
7
  ## Quickstart
8
8
 
9
9
  ```bash
10
- npx firegoat init # creates firegoat.yaml and .firegoat/, installs the firegoat-onboarding skill, copies a prompt for your agent
11
- # edit firegoat.yaml (or paste the prompt into your coding agent and let it)
12
- npx firegoat eval # runs your claude on each test, in .firegoat/runs/<runId>/<test>/<n>/work
10
+ npx firegoat init # Creates firegoat.yaml and .firegoat/, installs the firegoat-onboarding skill, copies a prompt for your agent
11
+ # Edit firegoat.yaml (or paste the prompt into your coding agent and let it)
12
+ npx firegoat eval # Runs your claude on each test, in .firegoat/runs/<runId>/<test>/<n>/work
13
13
  ```
14
14
 
15
15
  That's it: two commands. `claude` must be installed and logged in (run `claude` once).
16
16
 
17
- Coming next: `npx firegoat login` (asked during `init`, `--no-login` skips it), and `eval` uploading each session and printing its firegoat.ai URL (`--no-upload` skips it).
17
+ `init` installs the firegoat-onboarding skill globally, into `~/.claude/skills/` (in a terminal it asks: global, this repo, or none). `--local-skills` puts it in this repo's `.claude/skills/` instead (commit it so teammates and CI get it); `--no-skills` skips it.
18
+
19
+ ## Upload to firegoat.ai
20
+
21
+ ```bash
22
+ npx firegoat login # Device flow: open the URL, enter the code; key saved to ~/.firegoat/credentials.json (0600)
23
+ npx firegoat eval # Now also uploads each run as a private trace and prints its URL
24
+ npx firegoat logout # Forgets the login (revoke the key itself on the firegoat.ai traces page)
25
+ ```
26
+
27
+ Logging in means agreeing to the firegoat.ai [Terms of Service](https://firegoat.ai/terms) and [Privacy Policy](https://firegoat.ai/privacy); login says so before showing the code. `init` offers the login in a terminal (`--no-login` skips it). In CI, set `FIREGOAT_API_KEY=fire_...` instead. `eval --no-upload` keeps runs local; without a key, runs stay local and eval says so in one line.
28
+
29
+ - Per run, eval uploads claude's session (`session.jsonl`: prompts, tool calls and outputs, file contents, so possibly secrets from your env or repo), `checks.txt` and `stderr.txt` (check output sees `vars:`), plus the run's test, status, checks, the model asked for and the one claude reported, and (in a git repo) the branch, commit and whether the tree was dirty. firegoat.ai masks secrets when it shows a trace, but stores what was sent. Traces start private; you share one from its page.
30
+ - Before any run, eval checks the key (`GET /api/traces/me`) and prints `Uploading sessions to <origin> as <email>`: a key answered with 401/403 exits 3; any other failure (unreachable, 429, 5xx, a wrong `FIREGOAT_BASE_URL` path) exits 2.
31
+ - Each run prints `✓ <test>#<n> <url>` or `✗ <test>#<n> Upload failed: <why>`, and `run.json` / `summary.json` get `upload` (`{status: "uploaded", url}`, `{status: "failed", error}`, `{status: "skipped", reason}`, or null when not uploading). A failed upload never changes a run's status or eval's exit code (0/1/2 stay the eval's verdict); eval ends with `N uploads failed` on stderr. Each file may be at most 20 MiB: a bigger one fails that run's upload. Runs without a session file are skipped.
32
+ - The key is never passed to the agent, build, install or checks (every `FIREGOAT_*` var is removed from their env), is never written to run files, and goes only to the base URL's origin, never through a redirect; presigned upload URLs get no auth header. But there is no sandbox: the agent runs as you, so it can read `~/.firegoat/credentials.json`, and with `FIREGOAT_API_KEY` it can read the key from firegoat's own process environment (`ps eww $PPID`, `/proc/<pid>/environ`), which no program can scrub. In CI, use a dedicated key and revoke it if a run looks off. The key can only upload and read your own traces.
33
+ - `FIREGOAT_BASE_URL` (default `https://firegoat.ai/evals`) points firegoat elsewhere: https only, http only for `localhost` / `127.0.0.1`. A saved login only works for the base URL it was made for.
18
34
 
19
35
  ## Examples
20
36
 
21
37
  ### Can an agent get started with my npm package?
22
38
 
23
39
  ```yaml
24
- setup: npm pack --pack-destination "$FIREGOAT_OUT"
25
- workdir_setup: npm init -y >/dev/null && npm install "$FIREGOAT_OUT"/*.tgz
40
+ build: npm pack --pack-destination "$FIREGOAT_OUT"
41
+ install: npm init -y >/dev/null && npm install "$FIREGOAT_OUT"/*.tgz
26
42
  tests:
27
43
  - name: quickstart
28
44
  prompt: >-
@@ -33,12 +49,12 @@ tests:
33
49
  - run: grep -qE 'evt_[A-Za-z0-9]+' "$FIREGOAT_TRANSCRIPT"
34
50
  ```
35
51
 
36
- `setup` builds your package once; `workdir_setup` installs that build into each fresh folder, so the agent uses your local code, not the registry.
52
+ `build` builds your package once; `install` installs that build into each fresh folder, so the agent uses your local code, not the registry.
37
53
 
38
54
  ### Does my CLI make sense to an agent?
39
55
 
40
56
  ```yaml
41
- setup: go build -o "$FIREGOAT_OUT/acme" ./cmd/acme
57
+ build: go build -o "$FIREGOAT_OUT/acme" ./cmd/acme
42
58
  tests:
43
59
  - name: deploy
44
60
  prompt: Deploy the hello-world template to a preview environment with the acme CLI and print the URL.
@@ -47,7 +63,7 @@ tests:
47
63
  - run: grep -q '\$ acme deploy' "$FIREGOAT_TRANSCRIPT"
48
64
  ```
49
65
 
50
- Whatever `setup` writes to `$FIREGOAT_OUT` is first on the agent's `PATH`. That only helps for binaries called by bare name: if users run your CLI via `npx`/`uvx`, pack and install it like the npm example, or `npx` fetches the registry release.
66
+ Whatever `build` writes to `$FIREGOAT_OUT` is first on the agent's `PATH`. That only helps for binaries called by bare name: if users run your CLI via `npx`/`uvx`, pack and install it like the npm example, or `npx` fetches the registry release.
51
67
 
52
68
  ### Can an agent find the right package from my docs alone?
53
69
 
@@ -66,8 +82,8 @@ tests:
66
82
 
67
83
  ```yaml
68
84
  runs: 3
69
- setup: uv build --wheel --out-dir "$FIREGOAT_OUT"
70
- workdir_setup: uv venv && uv pip install "$FIREGOAT_OUT"/*.whl
85
+ build: uv build --wheel --out-dir "$FIREGOAT_OUT"
86
+ install: uv venv && uv pip install "$FIREGOAT_OUT"/*.whl
71
87
  tests:
72
88
  - name: sandbox-hello
73
89
  prompt: >-
@@ -101,41 +117,45 @@ tests:
101
117
 
102
118
  ```yaml
103
119
  env:
104
- ACME_HOME: "/tmp/acme-{{run.id}}-{{run.n}}" # sessions/config never collide across runs
120
+ ACME_HOME: "/tmp/acme-{{run.id}}-{{run.n}}" # Sessions/config never collide across runs
105
121
  ACME_TELEMETRY: "off"
106
- UV_FIND_LINKS: "$FIREGOAT_OUT" # resolve the wheel setup built
122
+ UV_FIND_LINKS: "$FIREGOAT_OUT" # Resolve the wheel build: made
107
123
  tests:
108
124
  - name: login
109
125
  prompt: Log in to acme with the CLI and print the current user.
110
126
  end_goal: The agent completes acme's login flow and prints the user.
111
127
  env:
112
- ACME_BETA_LOGIN: "1" # this test only, on top of env:
128
+ ACME_BETA_LOGIN: "1" # This test only, on top of env:
113
129
  checks:
114
130
  - run: ls "$ACME_HOME"/sessions | grep -q .
115
131
  ```
116
132
 
117
- `env:` goes to the agent, and to `workdir_setup` and checks of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` (also `${...}`), the same dirs `setup`, `workdir_setup` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `setup` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`.
133
+ `env:` goes to the agent, and to `install` and checks of the same run, so checks see the same values. Placeholders: `{{vars.KEY}}`, `{{run.id}}` (the eval's run id), `{{run.n}}` (1..runs, or 1..runs×models), `{{test.name}}`, and `$FIREGOAT_OUT` / `$FIREGOAT_WORKDIR` (also `${...}`), the same dirs `build`, `install` and checks see. There is no other `$` expansion: `$HOME` stays the literal text `$HOME`. `build` runs once, before any run: it gets only the top-level entries that use nothing but `{{vars.*}}`, `{{run.id}}` and `$FIREGOAT_OUT`. Values are strings (quote numbers). `PATH`, `CLAUDE_CONFIG_DIR` and `FIREGOAT_*` are firegoat's and can't be set; `ANTHROPIC_MODEL` neither: choose the model with `model:`.
134
+
135
+ Setting `HOME`: on macOS claude reads its login from `$HOME/Library`, so with a per-run `HOME` every run says "Not logged in" unless `install` runs `ln -s <real home>/Library "$HOME/Library"`. Get the real home in `build`, which doesn't get entries using `{{test.name}}` or `{{run.n}}` (e.g. `build: echo "$HOME" > "$FIREGOAT_OUT/.real-home"`).
118
136
 
119
137
  ## What eval does
120
138
 
121
139
  - claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers. claude saves its session there, in its own format.
122
- - No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. In `.firegoat/`, git ignores everything except `fixtures/`: keep test fixtures (scripts, sample data) in `.firegoat/fixtures/` and commit them with `firegoat.yaml`.
123
- - In a linked git worktree (`git worktree add`), `.firegoat/runs/` and `.firegoat/.claude/` are in the main checkout, so runs (and their session files) survive `git worktree remove`; `firegoat.yaml`, fixtures and `setup:` stay in the worktree. Each `summary.json` records `dir` (where eval ran) and `git: {branch, commit, dirty}`, and `.firegoat/runs/index.jsonl` gets one line per eval (`runId, startedAt, dir, branch, commit, passed, failed, notGraded, cost`) to tell runs from different branches apart.
124
- - The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, setup, workdir_setup, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it.
140
+ - No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. In `.firegoat/`, git ignores everything except `fixtures/` and its own `.gitignore`: keep test fixtures (scripts, sample data) in `.firegoat/fixtures/` and commit them and `.firegoat/.gitignore` with `firegoat.yaml`.
141
+ - In a linked git worktree (`git worktree add`), `.firegoat/runs/` and `.firegoat/.claude/` are in the main checkout, so runs (and their session files) survive `git worktree remove`; `firegoat.yaml`, fixtures and `build:` stay in the worktree. Each `summary.json` records `dir` (where eval ran) and `git: {branch, commit, dirty}`, and `.firegoat/runs/index.jsonl` gets one line per eval (`runId, startedAt, dir, branch, commit, passed, failed, notGraded, cost, uploaded, uploadFailed`) to tell runs from different branches apart.
142
+ - The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, build, install, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it. Every `FIREGOAT_*` var firegoat was started with (`FIREGOAT_API_KEY` above all) is removed too.
125
143
  - Runs (tests × models × `runs`) go 4 at a time, each in its own folder with its own session; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
126
144
  - Script checks run in the agent's folder afterwards and set the exit code. `end_goal` is not graded by firegoat: it's for you, or your coding agent, to judge from `run.json`, the session and the folder.
127
- - Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`), `run.json` (status, prompt, end_goal, check results, paths), `work/`, logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
145
+ - Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`; greppable line by line: every line of a shell call starts with `[n] $ `, every line of the agent's text and final answer with `agent: `, output is indented, so `grep -E '^\[[0-9]+\] \$ .*npm install'` matches only commands, also inside heredocs and multi-line scripts), `run.json` (status, prompt, end_goal, check results, paths), `work/`, logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
128
146
 
129
- `firegoat.yaml` keys: `runs`, `model`, `setup`, `workdir_setup`, `vars`, `env`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
147
+ `firegoat.yaml` keys: `runs`, `model`, `build`, `install`, `vars`, `env`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`, `env`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
130
148
 
131
149
  Want a plain `npm i acme-sdk` to get your build, with no hint in the prompt? See [`examples/local-npm-registry`](examples/local-npm-registry/README.md).
132
150
 
133
151
  | exit code | meaning |
134
152
  |---|---|
135
- | 0 | every run's checks passed |
136
- | 1 | a check failed |
137
- | 2 | at least one run was not graded (claude missing or logged out, crash, timeout, setup failed); wins over 1 |
138
- | 3 | usage or config error |
153
+ | 0 | Every run's checks passed |
154
+ | 1 | A check failed |
155
+ | 2 | At least one run was not graded (claude missing or logged out, crash, timeout, build failed); wins over 1 |
156
+ | 3 | Usage or config error (including firegoat.ai rejecting the upload key) |
157
+
158
+ Upload failures don't change the exit code.
139
159
 
140
160
  ## Feedback
141
161
 
@@ -152,6 +172,6 @@ It goes to the firegoat team with the firegoat version, your OS and node version
152
172
  ```bash
153
173
  bun install
154
174
  bun run format
155
- bun test # e2e: packs the package and drives the installed bin
156
- FIREGOAT_E2E_REAL=1 bun test # also runs one eval against your real claude
175
+ bun test # End-to-end: packs the package and drives the installed bin
176
+ FIREGOAT_E2E_REAL=1 bun test # Also runs one eval against your real claude
157
177
  ```