firegoat 0.0.2 → 0.2.0

package/README.md

@@ -0,0 +1,46 @@
+# firegoat
+
+Local-first AX evals: how well coding agents use your SDK, CLI, API and docs.
+
+Status: early (v0.2). See `plan.md`.
+
+## Use
+
+```bash
+npx firegoat init            # writes firegoat.yaml with starter evals; --skills adds the onboarding skill
+npx firegoat login           # once per machine: signs claude in to ~/.firegoat/claude
+npx firegoat eval            # runs each test with your local `claude`, sandboxed, in a fresh empty folder
+```
+
+### Isolation
+
+On macOS, `eval` runs `claude` inside a Seatbelt (`sandbox-exec`) profile:
+
+- it signs in with `~/.firegoat/claude` (from `firegoat login`), never your `~/.claude`;
+- it cannot read your home dir or the repo, except toolchains (node, bun, cargo, ...) it needs to run;
+- it can write only in its run dir (`work/`, its own `TMPDIR` and package caches) and system temp;
+- `ANTHROPIC_API_KEY` is removed, so runs bill your subscription, not the API.
+
+A probe checks the sandbox before every run; if it does not hold, the run is not graded. Network is open.
+
+Off macOS there is no sandbox yet: run inside a throwaway container and pass `--host`.
+
+`eval` saves each run's transcript to `.firegoat/runs/<runId>/<test>/<n>/session.jsonl` and a `summary.json`, then runs the test's `checks` in the agent's folder. Nothing is uploaded. `firegoat.yaml` is looked up in the current directory, then parents up to the git root.
+
+Exit codes:
+
+| code | meaning |
+|---|---|
+| 0 | every run passed |
+| 1 | a goal or check failed |
+| 2 | at least one run was not graded (claude missing, auth, crash, timeout); wins over 1 |
+| 3 | usage or config error (bad flag, unknown test, invalid or missing `firegoat.yaml`) |
+
+## Develop
+
+```bash
+bun install
+bun run format
+bun test                     # e2e: packs the package and drives the installed bin
+FIREGOAT_E2E_REAL=1 bun test # also runs one eval against your real claude
+```