firegoat 0.8.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +72 -50
  2. package/dist/cli.js +473 -1344
  3. package/package.json +4 -6
package/README.md CHANGED
@@ -1,82 +1,104 @@
1
1
  # firegoat
2
2
 
3
- Local-first AX evals: how well coding agents use your SDK, CLI, API and docs.
3
+ Local AX evals: how well coding agents use your SDK, CLI, API and docs. firegoat gives your own `claude` one prompt in a fresh folder, saves the session, and runs your checks.
4
4
 
5
- Status: early (v0.8). See `plan.md`.
5
+ Status: early (v0.9). See `plan.md`.
6
6
 
7
- ## Use
7
+ ## Quickstart
8
8
 
9
9
  ```bash
10
- npx firegoat init # writes firegoat.yaml with starter evals, asks how the agent signs in;
11
- # --skills adds the firegoat-onboarding skill (from team2027/skills)
12
- npx firegoat claude login # sign-in option 1, once per machine: claude in ~/.firegoat/claude
13
- npx firegoat eval # runs each test with your local `claude`, sandboxed, in a fresh folder
14
- npx firegoat open # opens the latest run's folder (--print to only print it)
10
+ npx firegoat init # creates firegoat.yaml and .firegoat/, installs the skills, copies a prompt for your agent
11
+ # edit firegoat.yaml (or paste the prompt into your coding agent and let it)
12
+ npx firegoat eval # runs your claude on each test, in .firegoat/runs/<runId>/<test>/<n>/work
15
13
  ```
16
14
 
17
- ### Local builds: setup
15
+ That's it: two commands. `claude` must be installed and logged in (run `claude` once).
18
16
 
19
- Plain shell, any language. `setup:` runs once per `eval` in the repo, on your machine; whatever it writes to `$FIREGOAT_OUT` is copied into each run's folder and put on the agent's PATH. `workdir_setup:` runs in each fresh workdir before the agent starts. The agent gets the build, never the source. Each takes one command or a list.
17
+ Coming next: `npx firegoat login` (asked during `init`, `--no-login` skips it), and `eval` uploading each session and printing its firegoat.ai URL (`--no-upload` skips it).
18
+
19
+ ## Examples
20
+
21
+ ### Can an agent get started with my npm package?
20
22
 
21
23
  ```yaml
22
- # npm
23
24
  setup: npm pack --pack-destination "$FIREGOAT_OUT"
24
25
  workdir_setup: npm init -y >/dev/null && npm install "$FIREGOAT_OUT"/*.tgz
25
-
26
- # python
27
- setup: uv build --wheel --out-dir "$FIREGOAT_OUT"
28
- workdir_setup: uv venv && uv pip install "$FIREGOAT_OUT"/*.whl
29
-
30
- # a CLI binary: on PATH as is
31
- setup: go build -o "$FIREGOAT_OUT/acme" ./cmd/acme
26
+ tests:
27
+ - name: quickstart
28
+ prompt: >-
29
+ acme-sdk is already installed in this project. Use it to send one test event
30
+ and print the event id. Credentials are in ACME_API_KEY. Keep it minimal.
31
+ end_goal: A script sends one event with the SDK's client and prints its id.
32
+ checks:
33
+ - run: grep -qE 'evt_[A-Za-z0-9]+' "$FIREGOAT_SESSION"
32
34
  ```
33
35
 
34
- Then say so in the prompt: "The package is already installed from the build under test."
35
-
36
- - `{{vars.KEY}}` in prompts and checks comes from `--var KEY=value` or `vars:` in `firegoat.yaml`. Vars are also env vars for `setup:`, `workdir_setup:` and checks, not for the agent. A missing var is a usage error; other `{{...}}` text is left alone.
37
- - Checks also get `$FIREGOAT_OUT` (the run's copy).
38
- - A failed `setup:` or `workdir_setup:` means not graded (exit 2); logs are `.firegoat/runs/<runId>/setup.txt` and `<test>/<n>/workdir_setup.txt`.
39
-
40
- Want the agent to run a plain `npm i acme-sdk` (or `pip install`) and still get your build, with no hint in the prompt? See the recipe in [`examples/local-npm-registry`](examples/local-npm-registry/README.md): a throwaway local registry started from `setup:`.
41
-
42
- ### How the agent signs in
43
-
44
- `firegoat init` asks (or pass `--auth`):
36
+ `setup` builds your package once; `workdir_setup` installs that build into each fresh folder, so the agent uses your local code, not the registry.
45
37
 
46
- 1. **Login with Claude** (`--auth claude`, same as `npx firegoat claude login`): your Claude plan, in firegoat's own config dir `~/.firegoat/claude`, never your `~/.claude`.
47
- 2. **Paste an Anthropic API key** (`--auth api-key`, piped on stdin or typed hidden): stored for firegoat only in `~/.firegoat/anthropic-api-key` (mode 0600), never in `firegoat.yaml` or the repo, and passed to the sandboxed claude as `ANTHROPIC_API_KEY`.
38
+ ### Does my CLI make sense to an agent?
48
39
 
49
- `eval` prints which one it uses. In CI (`CI` set), `ANTHROPIC_API_KEY` from the env wins. Outside CI, a key in your shell is not passed to the agent, so it never silently replaces the Claude login. `firegoat login` still works as a deprecated alias of `firegoat claude login`; the name is reserved for firegoat.ai.
40
+ ```yaml
41
+ setup: go build -o "$FIREGOAT_OUT/acme" ./cmd/acme
42
+ tests:
43
+ - name: deploy
44
+ prompt: Deploy the hello-world template to a preview environment with the acme CLI and print the URL.
45
+ end_goal: The agent finds the right acme commands and ends with a working preview URL.
46
+ checks:
47
+ - run: grep -q '"command":"acme deploy' "$FIREGOAT_SESSION"
48
+ ```
50
49
 
51
- ### Isolation
50
+ Whatever `setup` writes to `$FIREGOAT_OUT` is first on the agent's `PATH`.
52
51
 
53
- On macOS, `eval` runs `claude` inside a Seatbelt (`sandbox-exec`) profile:
52
+ ### Can an agent find the right package from my docs alone?
54
53
 
55
- - it signs in as chosen above, never with your `~/.claude`;
56
- - it cannot read your home dir or the repo, except toolchains (node, bun, cargo, ...) it needs to run;
57
- - it can write only in its run dir (`work/`, its own `TMPDIR` and package caches) and system temp;
58
- - `ANTHROPIC_API_KEY` is set only for the API-key sign-in; firegoat checks claude's reported key source on every run.
54
+ ```yaml
55
+ tests:
56
+ - name: which-packages
57
+ prompt: >-
58
+ I'm building a mobile app with Acme. It needs a local database, maps and Sign in
59
+ with Apple. Which Acme packages should I use? Write the list to packages.txt.
60
+ end_goal: packages.txt names the right Acme package for each need, found in Acme's docs.
61
+ checks:
62
+ - run: test -s packages.txt
63
+ ```
59
64
 
60
- A probe checks the sandbox before every run; if it does not hold, the run is not graded. Network is open.
65
+ ### Python, three runs, and a value from the command line
61
66
 
62
- Off macOS there is no sandbox yet: run inside a throwaway container and pass `--host`.
67
+ ```yaml
68
+ runs: 3
69
+ setup: uv build --wheel --out-dir "$FIREGOAT_OUT"
70
+ workdir_setup: uv venv && uv pip install "$FIREGOAT_OUT"/*.whl
71
+ tests:
72
+ - name: sandbox-hello
73
+ prompt: >-
74
+ Using the acme Python SDK (already installed), create a sandbox on commit
75
+ {{vars.SHA}}, run `echo hello-from-acme` in it, print the output, then delete it.
76
+ end_goal: The script prints hello-from-acme from a sandbox and deletes the sandbox.
77
+ checks:
78
+ - run: grep -rq hello-from-acme "$FIREGOAT_WORKDIR"
79
+ ```
63
80
 
64
- `eval` saves each run's transcript to `.firegoat/runs/<runId>/<test>/<n>/session.jsonl`, runs the test's `checks` in the agent's folder, and writes `run.json` per run plus `summary.json` (prompt, end_goal, check results, paths). Nothing is uploaded.
81
+ ```bash
82
+ npx firegoat eval --var SHA=$(git rev-parse HEAD)
83
+ ```
65
84
 
66
- ### Grading: your agent judges the end goal
85
+ ## What eval does
67
86
 
68
- firegoat runs only the script checks; they set the exit code. Each test's `end_goal` is handed over, not graded: whoever drives firegoat (usually your coding agent) judges it with its own process from `npx firegoat eval --json` / `run.json`, the transcript and the workdir. firegoat records no verdict beyond the checks.
87
+ - claude runs on its own login (or `ANTHROPIC_API_KEY`, if set) with `CLAUDE_CONFIG_DIR=.firegoat/.claude` and `--setting-sources "" --strict-mcp-config --disable-slash-commands`: none of your settings, CLAUDE.md, plugins, skills or MCP servers. claude saves its session there, in its own format.
88
+ - No sandbox: the agent's folder is inside `.firegoat/runs/`, and it can reach the rest of your machine. `.firegoat/` ignores itself in git.
89
+ - Script checks run in the agent's folder afterwards and set the exit code. `end_goal` is not graded by firegoat: it's for you, or your coding agent, to judge from `run.json`, the session and the folder.
90
+ - Output per run, in `.firegoat/runs/<runId>/<test>/<n>/`: `run.json` (status, prompt, end_goal, check results, paths), `work/`, logs. `summary.json` covers the whole eval; `--json` prints it.
69
91
 
70
- Every command has `--help` written for an agent; start with `npx firegoat --help`. `firegoat.yaml` is looked up in the current directory, then parents up to the git root.
92
+ `firegoat.yaml` keys: `runs`, `setup`, `workdir_setup`, `vars`, and `tests` (`name`, `prompt`, `end_goal`, `checks`, `timeout`). Unknown keys are errors. Full reference: `npx firegoat init --help`.
71
93
 
72
- Exit codes:
94
+ Want a plain `npm i acme-sdk` to get your build, with no hint in the prompt? See [`examples/local-npm-registry`](examples/local-npm-registry/README.md).
73
95
 
74
- | code | meaning |
96
+ | exit code | meaning |
75
97
  |---|---|
76
- | 0 | every run's script checks passed |
77
- | 1 | a script check failed |
78
- | 2 | at least one run was not graded (claude missing, auth, crash, timeout); wins over 1 |
79
- | 3 | usage or config error (bad flag, unknown test, invalid or missing `firegoat.yaml`) |
98
+ | 0 | every run's checks passed |
99
+ | 1 | a check failed |
100
+ | 2 | at least one run was not graded (claude missing or logged out, crash, timeout, setup failed); wins over 1 |
101
+ | 3 | usage or config error |
80
102
 
81
103
  ## Develop
82
104