@brainbase-labs/cli 0.27.0 → 0.28.0

package/README.md

@@ -22,10 +22,72 @@ brainbase template search       # find templates published by your team
 brainbase template onboard <creator/slug>   # install or refresh a template
 brainbase agent create          # claim a local brainbase.agent.yaml
 brainbase task create --message "Review this project and propose next steps"
+brainbase benchmark list        # list benchmarks for the linked agent
 ```
 
 Run `brainbase help` to see every command.
 
+## Benchmark management
+
+`benchmark init`, `benchmark validate`, and command help are local and need no
+login or linked agent. Control-plane commands use the claimed agent in
+`brainbase.agent.yaml` by default; pass `--agent <id>` to override it.
+
+```sh
+# Scaffold and validate locally.
+brainbase benchmark init ./benchmarks/support-quality
+brainbase benchmark validate ./benchmarks/support-quality
+
+# Create accepts manifests and directories without attached files.
+brainbase benchmark create ./benchmarks/support-quality --json
+
+# A directory containing cases/*/files/** must be imported as a bundle.
+brainbase benchmark import bundle ./benchmarks/support-quality --json
+
+# Publish and run the resulting draft.
+brainbase benchmark publish <benchmark-id> --expected-version <version> --json
+brainbase benchmark run plan <benchmark-id> \
+  --revision <revision-id> \
+  --variant default \
+  --json
+brainbase benchmark run start <plan-id> --yes --json
+brainbase benchmark run watch <run-id> --jsonl
+
+# Inspect and export terminal results.
+brainbase benchmark diagnoses <run-id> --json
+brainbase benchmark export-results <run-id> --output ./results.zip
+```
+
+`benchmark pull` and `benchmark push` round-trip the local directory layout:
+
+```text
+benchmark.yaml
+cases/<case-key>/case.yaml
+cases/<case-key>/files/**
+bundle/<original-bundle-path>
+```
+
+`benchmark.yaml` carries a local-only `case_order` list so pull and push
+preserve case ordering; it is not sent as part of the remote manifest.
+Case-local files stay beside their case. Shared, hidden, and root-level bundle
+paths are stored below `bundle/` while retaining their original remote paths.
+Pull destinations must be absent or empty. Use a new directory to refresh a
+benchmark instead of overwriting local edits.
+
+Human-readable output is the default. `--json` emits one versioned envelope.
+For paginated commands, it returns one page and its `next_cursor` unless
+`--all` is set. `--jsonl` emits item and page-boundary records, while watch
+emits state-change events. Resume with `--cursor`; structured modes never
+prompt.
+
+Idempotency is explicit-only: requests omit the idempotency header unless
+`--idempotency-key` is supplied. Reuse a key only for an identical retry.
+Starting a run requires `--yes`; confirming the same plan ID is replay-safe.
+Interrupting `run watch` does not cancel the remote run.
+
+Use `brainbase benchmark --help`, `brainbase benchmark run --help`, or
+`brainbase benchmark <command> --help` for local command help.
+
 ## Benchmark runtime commands
 
 Managed benchmark workers invoke two machine-only commands inside the same
@@ -97,9 +159,28 @@ remains their final cleanup boundary. Isolated evaluator workspace copies omit
 
 ## Development
 
-Use Bun 1.3.10 when building this repository. The generated
-`dist/index.js` is committed and byte-checked in CI, and Bun patch releases
-can produce different bundle output.
+Build with the Bun version in `.bun-version` (currently 1.3.10) — CI and both
+release workflows read that file. The generated `dist/index.js` is committed
+and byte-checked in CI, and Bun patch releases can produce different bundle
+output.
+
+### Releasing
+
+1. Run **Prepare Release** from `main` with the exact version. It refuses a
+   version already on npm, or a `release/*` branch or tag that already exists,
+   then pushes the branch and prints a link to open the pull request. The org
+   does not let Actions open pull requests, so that last step is yours.
+2. Merge the release pull request. CI checks any pull request that moves the
+   version, or comes from a `release/*` branch, against npm — so a hand-made
+   release pull request is held to the same bar as a generated one.
+3. Run **Publish**, naming the same version. It refuses to run if the commit
+   it checked out declares a different one, and the tarball's `dist/index.js`
+   must match the committed bundle.
+
+Prepare and publish share a `release` concurrency group, so two releases queue
+instead of interleaving. A published version is immutable: if a release goes
+out without something you meant to include, ship the next version rather than
+re-cutting the number.
 
 ## Agent runtime configuration
 
@@ -167,6 +248,7 @@ it talks to:
 | `agent`, `orchestration`, `link`, `unlink`, `sync`, `status`, `team` | `BRAINBASE_TOKEN`, then the `auth.json` session |
 | `template`, `skill`, `token` | `BRAINBASE_TOKEN`, then the session, then `token.json` |
 | `task create` | `BRAINBASE_TOKEN`, then the session, then `token.json` — the last only when no session is configured |
+| `benchmark` control-plane commands | `BRAINBASE_TOKEN`, then the session |
 
 `brainbase whoami` (or `--json`) reports which one is active.