@brainbase-labs/cli 0.27.1 → 0.29.0

package/README.md

@@ -22,10 +22,72 @@ brainbase template search       # find templates published by your team
 brainbase template onboard <creator/slug>   # install or refresh a template
 brainbase agent create          # claim a local brainbase.agent.yaml
 brainbase task create --message "Review this project and propose next steps"
+brainbase benchmark list        # list benchmarks for the linked agent
 ```
 
 Run `brainbase help` to see every command.
 
+## Benchmark management
+
+`benchmark init`, `benchmark validate`, and command help are local and need no
+login or linked agent. Control-plane commands use the claimed agent in
+`brainbase.agent.yaml` by default; pass `--agent <id>` to override it.
+
+```sh
+# Scaffold and validate locally.
+brainbase benchmark init ./benchmarks/support-quality
+brainbase benchmark validate ./benchmarks/support-quality
+
+# Create accepts manifests and directories without attached files.
+brainbase benchmark create ./benchmarks/support-quality --json
+
+# A directory containing cases/*/files/** must be imported as a bundle.
+brainbase benchmark import bundle ./benchmarks/support-quality --json
+
+# Publish and run the resulting draft.
+brainbase benchmark publish <benchmark-id> --expected-version <version> --json
+brainbase benchmark run plan <benchmark-id> \
+  --revision <revision-id> \
+  --variant default \
+  --json
+brainbase benchmark run start <plan-id> --yes --json
+brainbase benchmark run watch <run-id> --jsonl
+
+# Inspect and export terminal results.
+brainbase benchmark diagnoses <run-id> --json
+brainbase benchmark export-results <run-id> --output ./results.zip
+```
+
+`benchmark pull` and `benchmark push` round-trip the local directory layout:
+
+```text
+benchmark.yaml
+cases/<case-key>/case.yaml
+cases/<case-key>/files/**
+bundle/<original-bundle-path>
+```
+
+`benchmark.yaml` carries a local-only `case_order` list so pull and push
+preserve case ordering; it is not sent as part of the remote manifest.
+Case-local files stay beside their case. Shared, hidden, and root-level bundle
+paths are stored below `bundle/` while retaining their original remote paths.
+Pull destinations must be absent or empty. Use a new directory to refresh a
+benchmark instead of overwriting local edits.
+
+Human-readable output is the default. `--json` emits one versioned envelope.
+For paginated commands, it returns one page and its `next_cursor` unless
+`--all` is set. `--jsonl` emits item and page-boundary records, while watch
+emits state-change events. Resume with `--cursor`; structured modes never
+prompt.
+
+Idempotency is explicit-only: requests omit the idempotency header unless
+`--idempotency-key` is supplied. Reuse a key only for an identical retry.
+Starting a run requires `--yes`; confirming the same plan ID is replay-safe.
+Interrupting `run watch` does not cancel the remote run.
+
+Use `brainbase benchmark --help`, `brainbase benchmark run --help`, or
+`brainbase benchmark <command> --help` for local command help.
+
 ## Benchmark runtime commands
 
 Managed benchmark workers invoke two machine-only commands inside the same
@@ -102,6 +164,13 @@ release workflows read that file. The generated `dist/index.js` is committed
 and byte-checked in CI, and Bun patch releases can produce different bundle
 output.
 
+The suite is version-sensitive too, so the pin is enforced at each entry point:
+`bun run verify` and `bun run build` refuse another Bun outright, and `bun test`
+prints a warning and carries on. Each names the pinned version and the command
+to get it: `npx bun@<pinned> …`. Use `npx` rather than `bunx` — bunx runs most
+published Bun versions but cannot resolve a bin for 1.3.10 specifically, which
+is the current pin. Set `SKIP_BUN_VERSION_CHECK=1` to override any of them.
+
 ### Releasing
 
 1. Run **Prepare Release** from `main` with the exact version. It refuses a
@@ -186,6 +255,7 @@ it talks to:
 | `agent`, `orchestration`, `link`, `unlink`, `sync`, `status`, `team` | `BRAINBASE_TOKEN`, then the `auth.json` session |
 | `template`, `skill`, `token` | `BRAINBASE_TOKEN`, then the session, then `token.json` |
 | `task create` | `BRAINBASE_TOKEN`, then the session, then `token.json` — the last only when no session is configured |
+| `benchmark` control-plane commands | `BRAINBASE_TOKEN`, then the session |
 
 `brainbase whoami` (or `--json`) reports which one is active.