firegoat 0.0.1 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +30 -1
- package/dist/cli.js +7962 -0
- package/package.json +40 -13
- package/skills/firegoat-onboarding/SKILL.md +105 -0
- package/skills/firegoat-onboarding/references/examples.md +114 -0
package/README.md
CHANGED
|
@@ -1,3 +1,32 @@
|
|
|
1
1
|
# firegoat
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Local-first AX evals: how well coding agents use your SDK, CLI, API and docs.
|
|
4
|
+
|
|
5
|
+
Status: early (v0.1). See `plan.md`.
|
|
6
|
+
|
|
7
|
+
## Use
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npx firegoat init # writes firegoat.yaml with starter evals; --skills adds the onboarding skill
|
|
11
|
+
npx firegoat eval # runs each test with your local `claude` in a fresh empty folder
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
`eval` saves each run's transcript to `.firegoat/runs/<runId>/<test>/<n>/session.jsonl` and a `summary.json`, then runs the test's `checks` in the agent's folder. Nothing is uploaded. `firegoat.yaml` is looked up in the current directory, then parents up to the git root.
|
|
15
|
+
|
|
16
|
+
Exit codes:
|
|
17
|
+
|
|
18
|
+
| code | meaning |
|
|
19
|
+
|---|---|
|
|
20
|
+
| 0 | every run passed |
|
|
21
|
+
| 1 | a goal or check failed |
|
|
22
|
+
| 2 | at least one run was not graded (claude missing, auth, crash, timeout); wins over 1 |
|
|
23
|
+
| 3 | usage or config error (bad flag, unknown test, invalid or missing `firegoat.yaml`) |
|
|
24
|
+
|
|
25
|
+
## Develop
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
bun install
|
|
29
|
+
bun run format
|
|
30
|
+
bun test # e2e: packs the package and drives the installed bin
|
|
31
|
+
FIREGOAT_E2E_REAL=1 bun test # also runs one eval against your real claude
|
|
32
|
+
```
|