firegoat 0.5.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +26 -8
- package/dist/cli.js +658 -334
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -2,13 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
Local-first AX evals: how well coding agents use your SDK, CLI, API and docs.
|
|
4
4
|
|
|
5
|
-
Status: early (v0.
|
|
5
|
+
Status: early (v0.7). See `plan.md`.
|
|
6
6
|
|
|
7
7
|
## Use
|
|
8
8
|
|
|
9
9
|
```bash
|
|
10
|
-
npx firegoat init # writes firegoat.yaml with starter evals
|
|
11
|
-
|
|
10
|
+
npx firegoat init # writes firegoat.yaml with starter evals, asks how the agent signs in;
|
|
11
|
+
# --skills adds the firegoat-onboarding skill (from team2027/skills)
|
|
12
|
+
npx firegoat claude login # sign-in option 1, once per machine: claude in ~/.firegoat/claude
|
|
12
13
|
npx firegoat eval # runs each test with your local `claude`, sandboxed, in a fresh folder
|
|
13
14
|
npx firegoat open # opens the latest run's folder (--print to only print it)
|
|
14
15
|
```
|
|
@@ -36,27 +37,44 @@ Then say so in the prompt: "The package is already installed from the build unde
|
|
|
36
37
|
- Checks also get `$FIREGOAT_OUT` (the run's copy).
|
|
37
38
|
- A failed `setup:` or `workdir_setup:` means not graded (exit 2); logs are `.firegoat/runs/<runId>/setup.txt` and `<test>/<n>/workdir_setup.txt`.
|
|
38
39
|
|
|
40
|
+
Want the agent to run a plain `npm i acme-sdk` (or `pip install`) and still get your build, with no hint in the prompt? See the recipe in [`examples/local-npm-registry`](examples/local-npm-registry/README.md): a throwaway local registry started from `setup:`.
|
|
41
|
+
|
|
42
|
+
### How the agent signs in
|
|
43
|
+
|
|
44
|
+
`firegoat init` asks (or pass `--auth`):
|
|
45
|
+
|
|
46
|
+
1. **Login with Claude** (`--auth claude`, same as `npx firegoat claude login`): your Claude plan, in firegoat's own config dir `~/.firegoat/claude`, never your `~/.claude`.
|
|
47
|
+
2. **Paste an Anthropic API key** (`--auth api-key`, piped on stdin or typed hidden): stored for firegoat only in `~/.firegoat/anthropic-api-key` (mode 0600), never in `firegoat.yaml` or the repo, and passed to the sandboxed claude as `ANTHROPIC_API_KEY`.
|
|
48
|
+
|
|
49
|
+
`eval` prints which one it uses. In CI (`CI` set), `ANTHROPIC_API_KEY` from the env wins. Outside CI, a key in your shell is not passed to the agent, so it never silently replaces the Claude login. `firegoat login` still works as a deprecated alias of `firegoat claude login`; the name is reserved for firegoat.ai.
|
|
50
|
+
|
|
39
51
|
### Isolation
|
|
40
52
|
|
|
41
53
|
On macOS, `eval` runs `claude` inside a Seatbelt (`sandbox-exec`) profile:
|
|
42
54
|
|
|
43
|
-
- it signs in
|
|
55
|
+
- it signs in as chosen above, never with your `~/.claude`;
|
|
44
56
|
- it cannot read your home dir or the repo, except toolchains (node, bun, cargo, ...) it needs to run;
|
|
45
57
|
- it can write only in its run dir (`work/`, its own `TMPDIR` and package caches) and system temp;
|
|
46
|
-
- `ANTHROPIC_API_KEY` is
|
|
58
|
+
- `ANTHROPIC_API_KEY` is set only for the API-key sign-in; firegoat checks claude's reported key source on every run.
|
|
47
59
|
|
|
48
60
|
A probe checks the sandbox before every run; if it does not hold, the run is not graded. Network is open.
|
|
49
61
|
|
|
50
62
|
Off macOS there is no sandbox yet: run inside a throwaway container and pass `--host`.
|
|
51
63
|
|
|
52
|
-
`eval` saves each run's transcript to `.firegoat/runs/<runId>/<test>/<n>/session.jsonl
|
|
64
|
+
`eval` saves each run's transcript to `.firegoat/runs/<runId>/<test>/<n>/session.jsonl`, runs the test's `checks` in the agent's folder, and writes `run.json` per run plus `summary.json` (prompt, goal, rubric, check results, paths). Nothing is uploaded.
|
|
65
|
+
|
|
66
|
+
### Grading: your agent judges the goal
|
|
67
|
+
|
|
68
|
+
firegoat runs only the script checks; they set the exit code. The `goal` and an optional `rubric:` list are handed over, not graded: whoever drives firegoat (usually your coding agent) judges them with its own process from `npx firegoat eval --json` / `run.json`, the transcript and the workdir. firegoat records no verdict beyond the checks.
|
|
69
|
+
|
|
70
|
+
Every command has `--help` written for an agent; start with `npx firegoat --help`. `firegoat.yaml` is looked up in the current directory, then parents up to the git root.
|
|
53
71
|
|
|
54
72
|
Exit codes:
|
|
55
73
|
|
|
56
74
|
| code | meaning |
|
|
57
75
|
|---|---|
|
|
58
|
-
| 0 | every run passed |
|
|
59
|
-
| 1 | a
|
|
76
|
+
| 0 | every run's script checks passed |
|
|
77
|
+
| 1 | a script check failed |
|
|
60
78
|
| 2 | at least one run was not graded (claude missing, auth, crash, timeout); wins over 1 |
|
|
61
79
|
| 3 | usage or config error (bad flag, unknown test, invalid or missing `firegoat.yaml`) |
|
|
62
80
|
|