@brainbase-labs/cli 0.19.0 → 0.20.0

package/README.md

@@ -26,6 +26,68 @@ brainbase task create --message "Review this project and propose next steps"
 
 Run `brainbase help` to see every command.
 
+## Benchmark runtime commands
+
+Managed benchmark workers invoke two machine-only commands inside the same
+sandbox as the agent task:
+
+```sh
+brainbase benchmark hydrate --spec /path/to/hydrate.json --result /path/to/result.json --json
+brainbase benchmark evaluate --spec /path/to/evaluate.json --result /path/to/result.json --json
+brainbase benchmark capabilities --json
+```
+
+The CLI executes local, versioned phase specifications. It does not claim work,
+authorize users, or persist benchmark database state. The capabilities response
+advertises both the CLI version and supported phase schema versions.
+
+Both specs use `schema_version: "1"`, an `attempt_id`, a stable `phase_id`,
+absolute workspace/staging/log paths, and a phase budget. Staging must be:
+
+```text
+<workspace>/.brainbase/benchmark/<attempt_id>/incoming
+```
+
+Phase log/test roots must be absent, empty, or carry the CLI's matching
+attempt/phase ownership marker; the CLI never recursively clears an unowned
+non-empty directory.
+
+Hydration verifies every staged size and SHA-256 before copying a file or
+safely extracting a gzip tar into the candidate workspace. `.brainbase`, `.git`,
+and `brainbase.agent.yaml` are reserved. Setup commands use argv arrays, bounded
+time/output, an allowlisted base environment, literal values explicitly marked
+`sensitive: false`, and only command-selected secret environment variables from
+the spec's declared bindings. Expected answers and evaluator definitions are
+not valid hydrate fields.
+
+Evaluation starts only after MAS has ended the agent turn and staged a separate
+evaluation spec. It verifies final-output, trajectory, reference, and evaluator
+bundle digests; writes a pre-evaluation workspace manifest; preserves declared
+candidate artifacts; optionally creates a candidate archive; then materializes
+hidden references outside the candidate workspace and runs evaluators
+synchronously. Schema v1 supports:
+
+- `output_assertion`: exact, contains, or regex checks over final output.
+- `trajectory_assertion`: bounded counts of canonical event types.
+- `workspace_assertion`: file existence, absence, SHA-256, or content checks.
+- `sandbox_command`: a bounded argv command rooted in the workspace or hidden
+  tests directory. Exit zero passes; any other exit code is a valid failed
+  verdict. Schema v1 allows one command evaluator and runs it after all
+  read-only assertions so evaluator mutations cannot change candidate verdicts.
+
+Valid failed verdicts still produce a successful evaluation phase. Invalid
+specs, digest/path violations, missing environment, launch failures, timeouts,
+and output-budget violations fail the phase. Results are written atomically
+with mode `0600`, include the raw spec digest and checksummed evidence/log
+input/output references with explicit `staging`, `workspace`, `tests`, or
+`logs` roots, redact declared secret values from command logs, require the
+result path to be `<logs_root>/result.json`, and treat a matching successful
+phase result as authoritative on replay.
+
+On any phase-level execution failure, MAS must tear down the task sandbox. The
+CLI kills the command process group and descendants it can observe, but the
+runtime lifecycle remains the final cleanup boundary for daemonized processes.
+
 ## Development
 
 Use Bun 1.3.10 when building this repository. The generated