firegoat 0.15.1 → 0.16.0

package/README.md

@@ -2,7 +2,7 @@
 
 Local AX evals: how well coding agents use your SDK, CLI, API and docs. firegoat gives your own `claude` one prompt in a fresh folder, saves the session, and runs your checks.
 
-Status: early (v0.11). See `plan.md`.
+Status: early. See `plan.md`.
 
 ## Quickstart
 
@@ -14,7 +14,21 @@ npx firegoat eval     # runs your claude on each test, in .firegoat/runs/<runId>
 
 That's it: two commands. `claude` must be installed and logged in (run `claude` once).
 
-Coming next: `npx firegoat login` (asked during `init`, `--no-login` skips it), and `eval` uploading each session and printing its firegoat.ai URL (`--no-upload` skips it).
+## Upload to firegoat.ai
+
+```bash
+npx firegoat login    # device flow: open the URL, enter the code; key saved to ~/.firegoat/credentials.json (0600)
+npx firegoat eval     # now also uploads each run as a private trace and prints its URL
+npx firegoat logout   # forgets the login (revoke the key itself on the firegoat.ai traces page)
+```
+
+Logging in means agreeing to the firegoat.ai [Terms of Service](https://firegoat.ai/terms) and [Privacy Policy](https://firegoat.ai/privacy); login says so before showing the code. `init` offers the login in a terminal (`--no-login` skips it). In CI, set `FIREGOAT_API_KEY=fire_...` instead. `eval --no-upload` keeps runs local; without a key, runs stay local and eval says so in one line.
+
+- Per run, eval uploads claude's session (`session.jsonl`: prompts, tool calls and outputs, file contents, so possibly secrets from your env or repo), `checks.txt` and `stderr.txt` (check output sees `vars:`), plus the run's test, status, checks, the model asked for and the one claude reported, and (in a git repo) the branch, commit and whether the tree was dirty. firegoat.ai masks secrets when it shows a trace, but stores what was sent. Traces start private; you share one from its page.
+- Before any run, eval checks the key (`GET /api/traces/me`) and prints `Uploading sessions to <origin> as <email>`: a key answered with 401/403 exits 3; any other failure (unreachable, 429, 5xx, a wrong `FIREGOAT_BASE_URL` path) exits 2.
+- Each run prints `✓ <test>#<n> <url>` or `✗ <test>#<n> Upload failed: <why>`, and `run.json` / `summary.json` get `upload` (`{status: "uploaded", url}`, `{status: "failed", error}`, `{status: "skipped", reason}`, or null when not uploading). A failed upload never changes a run's status or eval's exit code (0/1/2 stay the eval's verdict); eval ends with `N uploads failed` on stderr. Each file may be at most 20 MiB: a bigger one fails that run's upload. Runs without a session file are skipped.
+- The key is never passed to the agent, setup, workdir_setup or checks (every `FIREGOAT_*` var is removed from their env), is never written to run files, and goes only to the base URL's origin, never through a redirect; presigned upload URLs get no auth header. But there is no sandbox: the agent runs as you, so it can read `~/.firegoat/credentials.json`, and with `FIREGOAT_API_KEY` it can read the key from firegoat's own process environment (`ps eww $PPID`, `/proc/<pid>/environ`), which no program can scrub. In CI, use a dedicated key and revoke it if a run looks off. The key can only upload and read your own traces.
+- `FIREGOAT_BASE_URL` (default `https://firegoat.ai/evals`) points firegoat elsewhere: https only, http only for `localhost` / `127.0.0.1`. A saved login only works for the base URL it was made for.
 
 ## Examples
 
@@ -120,8 +134,8 @@ tests:
 
 - claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers. claude saves its session there, in its own format.
 - No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. In `.firegoat/`, git ignores everything except `fixtures/`: keep test fixtures (scripts, sample data) in `.firegoat/fixtures/` and commit them with `firegoat.yaml`.
-- In a linked git worktree (`git worktree add`), `.firegoat/runs/` and `.firegoat/.claude/` are in the main checkout, so runs (and their session files) survive `git worktree remove`; `firegoat.yaml`, fixtures and `setup:` stay in the worktree. Each `summary.json` records `dir` (where eval ran) and `git: {branch, commit, dirty}`, and `.firegoat/runs/index.jsonl` gets one line per eval (`runId, startedAt, dir, branch, commit, passed, failed, notGraded, cost`) to tell runs from different branches apart.
-- The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, setup, workdir_setup, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it.
+- In a linked git worktree (`git worktree add`), `.firegoat/runs/` and `.firegoat/.claude/` are in the main checkout, so runs (and their session files) survive `git worktree remove`; `firegoat.yaml`, fixtures and `setup:` stay in the worktree. Each `summary.json` records `dir` (where eval ran) and `git: {branch, commit, dirty}`, and `.firegoat/runs/index.jsonl` gets one line per eval (`runId, startedAt, dir, branch, commit, passed, failed, notGraded, cost, uploaded, uploadFailed`) to tell runs from different branches apart.
+- The agent gets your environment plus `env:`, never `vars:`. Vars that npm/npx set for firegoat itself (`npm_config_*`, `npm_lifecycle_*`, `npm_package_*`, `npm_execpath`, `npm_node_execpath`, `npm_command`, `INIT_CWD`) are removed from every env firegoat starts (agent, setup, workdir_setup, checks): `npx -y firegoat` sets `npm_config_yes=true`, which would make the agent's `npm init` / `npx create-*` skip their prompts. Set one under `env:` to keep it. Every `FIREGOAT_*` var firegoat was started with (`FIREGOAT_API_KEY` above all) is removed too.
 - Runs (tests × models × `runs`) go 4 at a time, each in its own folder with its own session; `--parallel N` changes that, `--parallel 1` runs them one after another. `summary.json` lists them in `firegoat.yaml` order.
 - Script checks run in the agent's folder afterwards and set the exit code. `end_goal` is not graded by firegoat: it's for you, or your coding agent, to judge from `run.json`, the session and the folder.
 - Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `transcript.txt` (each tool call with its command and short output, the final answer, files made; checks get it as `$FIREGOAT_TRANSCRIPT`), `run.json` (status, prompt, end_goal, check results, paths), `work/`, logs. Read the transcript to judge `end_goal`; claude's full session file is ~50x larger. `summary.json` covers the whole eval; `--json` prints it.
@@ -135,7 +149,9 @@ Want a plain `npm i acme-sdk` to get your build, with no hint in the prompt? See
 | 0 | every run's checks passed |
 | 1 | a check failed |
 | 2 | at least one run was not graded (claude missing or logged out, crash, timeout, setup failed); wins over 1 |
-| 3 | usage or config error |
+| 3 | usage or config error (including firegoat.ai rejecting the upload key) |
+
+Upload failures don't change the exit code.
 
 ## Feedback