@caveman-ai/cloud 0.0.1 → 0.1.0-rc.22

package/README.md

@@ -3,8 +3,12 @@
 Every Caveman Cloud operation from the terminal, plus the Cloud MCP server.
 
 ```sh
-npm i -g @caveman-ai/cloud
-cvm login                 # runs `caveman login`; one sign-in for both CLIs
+npm i -g "@caveman-ai/cloud@>=0.1.0-rc.22"   # the first release with setup; `latest` is still 0.0.1
+cvm setup --project PROJECT_ID   # signs in, picks the project, installs the Caveman Cloud skill
+```
+
+```sh
+cvm login                 # browser sign-in; one sign-in for cvm and caveman
 cvm projects list
 cvm traces search --help
 cvm sql "SELECT model, count() FROM requests GROUP BY model"
@@ -12,11 +16,79 @@ cvm tools list            # every operation this Cloud offers you
 cvm mcp                   # Cloud MCP server over stdio
 ```
 
-- **Sign-in.** cvm uses the credential `caveman login` saved. If `caveman` isn't installed, `cvm login` fetches it once with npx. In CI, set `CAVE_TOKEN` (and `CAVE_API_URL` for a private install).
+- **Sign-in.** `cvm login` signs in with the browser device flow: it prints a code and a link, opens the browser (`--no-browser` or `CAVE_NO_BROWSER=1` to skip) and waits for you to confirm. It stores the credential where `caveman login` does (the macOS Keychain, else `~/.caveman/credentials`), so either CLI's sign-in works for both; `cvm logout` signs both out. `--instance URL` (or `CAVE_API_URL`) signs in to a private install. In CI, set `CAVE_TOKEN` (and `CAVE_API_URL` for a private install).
+- **Environment.** A flag wins over the environment, and the environment over the saved login: `CAVE_API_URL`, `CAVE_PROJECT_ID`, `CAVE_TOKEN`. The saved credential is only ever sent to the address it was saved with; another `CAVE_API_URL` needs its own `CAVE_TOKEN`.
 - **Projects.** Use `--project UUID`, `CAVE_PROJECT_ID`, `cvm context bind` (writes `.caveman-cloud.json` in the repository) or the saved default.
 - **Output.** JSON when piped, readable when on a terminal; `--format json|jsonl|table`.
 - **Exit codes.** 0 ok · 1 request failed · 2 usage or invalid input (HTTP 400/422) · 3 signed out or forbidden (401/403) · 4 conflict (409) · 5 a comparison or gate failed · 6 timeout, still running or not enough data · 7 rate limited (429).
 
+## Setup
+
+`cvm setup` does three things, each safe to run again, and reports each:
+
+1. **Signs in**, unless the saved sign-in or `CAVE_TOKEN` already answers.
+2. **Picks the project**: `--project UUID`, else `CAVE_PROJECT_ID`, the repository's binding or the saved default, else your only project (with more than one it lists them and exits 2). It checks the project and saves it as the default, as `cvm context set` does.
+3. **Installs the Caveman Cloud skill** for the coding agents it finds: Claude Code in `~/.claude/skills/caveman-cloud` (`$CLAUDE_CONFIG_DIR/skills` when set), Codex and Cursor in `~/.agents/skills/caveman-cloud`. With none found it installs to `~/.agents/skills`. `--agent claude|codex|cursor` installs for that agent only.
+
+For Claude Code or Codex, setup also reports the finished steps to the console's onboarding screen (best effort). Setup writes nothing in the repository. It replaces only a skill it installed itself (marked by a `.cvm-version` file) and leaves any other `caveman-cloud` skill alone (exit 1). `cvm setup --remove` deletes the skills it installed and keeps you signed in. Every later `cvm` command refreshes an installed skill when cvm was upgraded. Then open your agent in the repository and ask it: "Set up Caveman Cloud for this repository."
+
+## An eval in one file
+
+```sh
+cvm evals init                                             # writes example.eval.json: three cases, a task, two criteria, a strict gate, a budget
+cvm evals validate example.eval.json                       # the schema, and its names against the project
+cvm evals run example.eval.json --set-baseline-on-pass     # run it and save the baseline
+cvm evals ci evals/*.eval.json                             # in CI: gate a change against that baseline
+```
+
+An eval file is JSON (`*.eval.json`; schema `https://caveman.so/schemas/eval-file.schema.json`):
+
+```json
+{
+  "name": "support-answers",
+  "dataset": {"cases": [{"id": "refund", "input": {"question": "How do I get a refund?"}, "expected": "Within 30 days.", "tags": ["billing"]}]},
+  "task": {"command": ["node", "answer.mjs"]},
+  "criteria": ["Grounded", {"type": "contains", "fragments": ["30 days"]}, {"judge": {"question": "Is the answer polite?", "pass": "Courteous", "fail": "Curt or rude"}}],
+  "trials": 3,
+  "gate": {"baseline": "saved", "policy": "strict"},
+  "budget_usd": 2
+}
+```
+
+- **name** is the experiment. The same name is the same experiment and the same dataset every time, so runs line up.
+- **dataset**: `cases` written in the file, or `{"name": "Support tickets", "version": 3}` for a dataset of the project. A command task runs here on the dataset's rows, so a named dataset with a command task needs a sign-in that may read them (an owner or admin); a CI key reads none. With a task Caveman runs, no row is read here and any sign-in that may run evals will do.
+- **task**: `{"command": [...]}` runs here, once per case and trial, directly and not through a shell: the case's input arrives as JSON on stdin, the output is read from stdout (JSON when it parses, else the text), and a non-zero exit or a timeout is a task error. Only a task error is tried again (`retries`, default 2; `timeout_seconds`, default 120; `concurrency`, default 4). `{"recorded": "output"}` grades outputs already in a dataset column. `{"kind": "model" | "prompt" | "http", "config": {...}}` is a target Caveman runs.
+- **criteria**: a library criterion by name, a fixed check (`{"type": "contains", ...}`; `cvm criteria catalog` lists the types), or a judge question.
+- **gate**: present, a full run is compared on the server with the experiment's saved baseline (or the run id you name). `strict` fails when any criterion got worse on any case, and decides on a handful of cases. `statistical` fails on a drop the numbers support and passes only when they support that too: that takes about 30 cases, and under that it reads need more data (exit 6), which fails a CI job. A run stores the file's policy on the experiment, so the console and the SDKs read the same two runs by the same rule; a gate that names no policy is read by the experiment's stored one (statistical until one is stored), and the result names the policy that decided. Absent, the run is reported and never compared. Two runs compare only on the same cases and criteria: after changing either on purpose, save the new run as the baseline (`cvm evals set_baseline --workbench-id ID --run-id ID`; the refusal prints the command).
+- **budget_usd**: the most the run may spend. Required when a project key runs the file, and over one trial.
+
+`cvm evals validate FILE...` exits 2 on an invalid file and every error names its JSON path (`$.criteria[1].fragments`): the schema, and each fixed check's type and options. `--offline` stops there and needs no sign-in; without it the dataset and library criterion names are looked up in the project too.
+
+`cvm evals run FILE...` flags: `--dry-run` runs the task here and sends nothing; `--trials N`; `--first N` or `--sample N --seed S` run some of the cases, and such a partial run is graded but never compared and can never become a baseline (the run says so itself, so the server refuses it too); `--set-baseline-on-pass` saves a passed run as the baseline, and the first run of an experiment, which has nothing to be compared with; `--junit-out PATH` writes a JUnit report with a test case per case and criterion; `--timeout SECONDS` bounds the wait for grading (default 1800).
+
+- **Exit.** 0 passed, or nothing to gate (no gate, no baseline yet, a partial or dry run) · 5 the gate failed · 6 no verdict: more data needed, the runs are not comparable, or not every output could be graded · 2 an invalid file · 1 the run crashed · 4 the server refused the run (a project key's daily budget is held). Several files exit by the worst of them, and nothing runs when one is invalid.
+- **Output.** On a terminal, a table: a row per case, a column per criterion (`pass`, `fail` or `error`, and `4/5` over trials), why the failing ones failed with the judge's own reason under it when it gave one, pass rates, the server's verdict sentence with its interval and every criterion that regressed, the baseline and the policy it was read by, cost (what Caveman measured; for a task you ran, its own judging cost, with what your task reported apart), and the console address when `CAVE_CONSOLE_URL` is set. Piped, one JSON object per file. `--format json|table` overrides.
+
+`cvm evals ci FILE...` is the same run for CI: each result also carries the commit, branch and pull request the CI system names, the run keeps them (`scope.commit` on its read, shown in the console), and `--advisory` spares the job the verdict (never an invalid file or a crash). It needs no GitHub: an eval file gates on GitLab CI, Buildkite, Jenkins or a laptop alike. Set `CAVE_TOKEN` to a project key with the eval scope, `CAVE_PROJECT_ID`, and `CAVE_API_URL` for a private install. On GitHub Actions the Caveman action's `files:` input does this. A file a key runs names `budget_usd` (each run's budget is held against the key's limit for the day) and writes its cases in the file, or uses a task Caveman runs. What a key reads of a run holds no judge reason and no error text, so the `Failing:` lines and the JUnit failures name the case and criterion alone; the reasons are on the run in the console.
+
+## Pull request checks from any CI system
+
+```sh
+cvm evals ci --suite evals --junit 'reports/junit*.xml' -- npm test
+```
+
+With a test command after `--`, `cvm evals ci` wraps a test suite instead of running eval files. It starts a CI run, runs the test command with the run's spend-capped key (`CAVE_API_KEY`, `CAVE_GATEWAY_URL`, `CAVE_CI_RUN_ID`), uploads the JUnit reports, waits for the verdict against the base branch and prints it. The commit, branch, pull request and job come from GitHub Actions, GitLab CI, Buildkite or Jenkins variables, or the checkout; `--sha`, `--ref`, `--base-sha`, `--base-ref`, `--pr`, `--job`, `--repository-id` override them.
+
+- `--advisory` never fails on Caveman's account: not on the verdict, not when Caveman cannot be reached.
+- `--max-quality-drop-pp N` and `--max-cost-increase-pct N` add limits of your own. They can only add a failure.
+- `--junit-out PATH` writes the verdict as a JUnit report for the CI system's test tab (`-` prints it).
+- `--max-usd N` caps the run's key; `--timeout SECONDS` bounds the wait (default 180, 0 does not wait).
+- Exit: the test command's own code when it fails; else 0, 5 when the gate failed, 6 when a gating suite had no verdict in time.
+- **Credential.** Set `CAVE_TOKEN` to a project API key made for running evals: in the console, New key with "Runs evals from CI" on and a spend limit per day (over the API, scope `eval:ci` alone with `budget_usd` and `budget_period: "daily"`). The key reaches only its own project's datasets, criteria, experiments, runs, verdicts and CI runs, reads no dataset rows or request bodies, and honours its IP allow-list. Every run it starts is given a budget (`--max-usd` here, `scope.budget_usd` on `cvm evals run`); the day's budgets add up to the key's limit, and the next is refused with `cave_key_daily_budget_exceeded` (exit 4) until the next UTC day. A person's session token still works. On GitHub Actions use the Caveman action instead, which needs no secret and writes the check run.
+- **Repository.** This check on your test suite gates only for a project with a GitHub repository connected, because whether a run comes from a fork is GitHub's answer. Any other project is refused with `cave_ci_repository_required` (exit 4): on GitLab, Buildkite or Jenkins without that connection, gate on the comparison verdict instead, which needs no repository: eval files, or `cvm evals run --wait` then `cvm evals compare` (exit 5 when the comparison fails).
+
+`cvm evals watch RUN_ID [--timeout SECONDS]` waits for any run's verdict: exit 0 ship, pass or skipped, 5 don't ship or a failed gate, 6 need more data or still running. `cvm evals watch WORKBENCH_ID RUN_ID` is `cvm evals wait`. Operations that start a long run take `--wait` (`cvm evals run --wait`).
+
 Commands come from the Cloud release they ship with: `cvm <family> <verb>` runs the operation `family.verb`. `cvm tools describe <family>.<verb>` prints its input schema.
 
 Source: Caveman Cloud `public/cloud-cli/`, which bundles `cloud/agent-surface` at the release commit.

package/bin/ci.mjs

@@ -0,0 +1,236 @@
+// `cvm evals ci` and `cvm evals watch`: the pull request check from any CI
+// system, and a wait on one run's verdict. The GitHub Action
+// (public/integrations/github-action) does the same with the job's OIDC
+// token; here the caller is a signed-in person, so nothing is written on
+// GitHub and this command's exit code is the gate.
+import {globSync,readFileSync,writeFileSync} from 'node:fs';
+import {execFileSync,spawnSync} from 'node:child_process';
+import {randomUUID} from 'node:crypto';
+import {join} from 'node:path';
+import {parseArgs} from 'node:util';
+import {setTimeout as delay} from 'node:timers/promises';
+import {CloudClient,CloudError,inertJSON,runCloudCLI} from '../dist/cloud-client.mjs';
+
+const MAX_REPORTS=20,MAX_REPORT_BYTES=10*1024*1024; // the finish route's own limits
+const say=line=>process.stderr.write(line+'\n');
+const sha40=value=>/^[0-9a-f]{40}$/.test(value??'')&&!/^0+$/.test(value);
+const digits=value=>/^\d+$/.test(value??'')?value:undefined;
+const git=(...args)=>{try{return execFileSync('git',args,{encoding:'utf8',stdio:['ignore','pipe','ignore']}).trim();}catch{return '';}};
+export const xml=text=>String(text).replace(/[<>&"]/g,c=>({'<':'&lt;','>':'&gt;','&':'&amp;','"':'&quot;'})[c]).replace(/[^\x09\x0a\x0d\x20-퟿-�]/g,'');
+
+// Flags, then CAVE_PROJECT_ID, then the repository's binding, then the saved default.
+export async function connect(host,project){
+ const cfg=await host();
+ let bound;try{bound=JSON.parse(readFileSync('.caveman-cloud.json','utf8')).project_id;}catch{}
+ const client=new CloudClient({...cfg,projectId:project??process.env.CAVE_PROJECT_ID??bound??cfg.projectId});
+ // A server that no longer speaks this build's operation contract answers a
+ // bare 409: say what to do about it in one line.
+ const request=client.request.bind(client);
+ client.request=async(...args)=>{try{return await request(...args);}catch(error){if(error?.code==='cave_operation_version_conflict')error.message=VERSION_CONFLICT;throw error;}};
+ return client;
+}
+export const VERSION_CONFLICT='This cvm and the Caveman server speak different API versions. Upgrade cvm (npm install -g @caveman-ai/cloud@latest), or install the cvm version that matches a self-hosted server, then run the command again.';
+function projectPath(client){
+ if(!/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(client.projectId??''))throw new CloudError('cave_project_required','Select a project with --project, CAVE_PROJECT_ID or cvm context bind.',2);
+ return `/api/v1/projects/${client.projectId}`;
+}
+export function seconds(raw,fallback){
+ const value=Number(raw??fallback);
+ if(!Number.isFinite(value)||value<0||value>86400)throw new CloudError('cave_usage','--timeout is seconds, 0 to 86400.',2);
+ return value*1000;
+}
+function limit(raw,name){
+ if(raw===undefined)return undefined;
+ const value=Number(raw);
+ if(!Number.isFinite(value)||value<0)throw new CloudError('cave_usage',`--${name} must be a number of at least 0.`,2);
+ return value;
+}
+// until reads again, 1 s apart growing to 10 s, until done says so or the
+// time is up (undefined).
+async function until(read,done,timeoutMs,waiting){
+ const deadline=Date.now()+timeoutMs;
+ for(let ms=1000;;ms=Math.min(ms*1.5,10000)){
+  const value=await read();
+  if(done(value))return value;
+  if(Date.now()+ms>deadline)return undefined;
+  waiting(value);
+  await delay(ms);
+ }
+}
+
+// detectRun reads the commit, branch, pull request and job from the CI
+// system's own variables: GitHub Actions, GitLab CI, Buildkite, Jenkins, then
+// the checkout itself. A flag wins over all of them.
+export function detectRun(env,flags={},{gitRead=git,read=readFileSync}={}){
+ const first=(...values)=>values.find(value=>value!==undefined&&value!=='');
+ let event={};try{if(env.GITHUB_EVENT_PATH)event=JSON.parse(read(env.GITHUB_EVENT_PATH,'utf8'));}catch{}
+ const branch=first(env.CI_COMMIT_REF_NAME,env.BUILDKITE_BRANCH,env.BRANCH_NAME,env.GIT_BRANCH);
+ const pr=first(flags.pr,/^refs\/pull\/(\d+)\//.exec(env.GITHUB_REF??'')?.[1],env.CI_MERGE_REQUEST_IID,digits(env.BUILDKITE_PULL_REQUEST),env.CHANGE_ID);
+ const run={
+  sha:flags.sha??[event.pull_request?.head?.sha,env.GITHUB_SHA,env.CI_COMMIT_SHA,env.BUILDKITE_COMMIT,env.GIT_COMMIT].find(sha40)??gitRead('rev-parse','HEAD'),
+  ref:first(flags.ref,env.GITHUB_REF,branch&&`refs/heads/${branch}`)??`refs/heads/${gitRead('rev-parse','--abbrev-ref','HEAD')||'HEAD'}`,
+  base_sha:first(flags['base-sha'],[event.pull_request?.base?.sha,event.merge_group?.base_sha,env.CI_MERGE_REQUEST_DIFF_BASE_SHA,event.before].find(sha40)),
+  base_ref:first(flags['base-ref'],env.GITHUB_BASE_REF,env.CI_MERGE_REQUEST_TARGET_BRANCH_NAME,env.BUILDKITE_PULL_REQUEST_BASE_BRANCH,env.CHANGE_TARGET),
+  pr_number:pr===undefined?undefined:Number(pr),
+  event:first(flags.event,env.GITHUB_EVENT_NAME==='merge_group'?'merge_group':undefined,pr===undefined?'push':'pull_request'),
+  job:first(flags.job,env.GITHUB_JOB,env.CI_JOB_NAME,env.BUILDKITE_STEP_KEY,env.BUILDKITE_LABEL,env.JOB_BASE_NAME,'ci').replace(/[^A-Za-z0-9_-]/g,'-').slice(0,100),
+  // run_id and run_attempt only name the run to its reader; the request id is what makes it one run.
+  run_id:first(flags['run-id'],digits(env.GITHUB_RUN_ID),digits(env.CI_PIPELINE_ID),digits(env.BUILDKITE_BUILD_NUMBER),digits(env.BUILD_NUMBER),String(Date.now())),
+  run_attempt:Number(first(flags['run-attempt'],env.GITHUB_RUN_ATTEMPT,digits(env.BUILDKITE_RETRY_COUNT)&&Number(env.BUILDKITE_RETRY_COUNT)+1,1)),
+  repository_id:Number(first(flags['repository-id'],env.GITHUB_REPOSITORY_ID,env.CI_PROJECT_ID,0)),
+ };
+ for(const key of Object.keys(run))if(run[key]===undefined)delete run[key];
+ return run;
+}
+
+// collectReports reads the JUnit files the globs match, inside the upload's
+// limits. What does not fit is left out and named, never silently truncated.
+function collectReports(patterns,cwd){
+ const files=[...new Set(patterns.flatMap(pattern=>globSync(pattern,{cwd,exclude:entry=>(entry.name??entry)==='node_modules'})))].sort();
+ const reports=[],skipped=[];let bytes=0;
+ for(const file of files){
+  const text=readFileSync(join(cwd,file),'utf8'),size=Buffer.byteLength(text);
+  if(reports.length>=MAX_REPORTS||bytes+size>MAX_REPORT_BYTES){skipped.push(file);continue;}
+  bytes+=size;reports.push(text);
+ }
+ return {reports,skipped};
+}
+
+// judge reads a scored run: what fails the gate and why. A gating suite's
+// failing check does; so does a limit the caller set. The limits only add a
+// failure and --advisory only spares the job: the verdict stays Caveman's.
+export function judge(answer,{advisory=false,maxQualityDropPP,maxCostIncreasePct}={}){
+ const reasons=[];
+ if(answer.mode==='gate'&&['failure','action_required'].includes(answer.conclusion))reasons.push(`the gating suite's check concluded ${answer.conclusion}`);
+ const {quality,cost}=answer.check??{};
+ if(maxQualityDropPP!==undefined&&quality&&quality.estimate<-maxQualityDropPP)reasons.push(`test pass rate moved ${quality.estimate.toFixed(1)} pp, past --max-quality-drop-pp ${maxQualityDropPP}`);
+ if(maxCostIncreasePct!==undefined&&cost&&cost.estimate>maxCostIncreasePct)reasons.push(`cost moved ${cost.estimate>0?'+':''}${cost.estimate.toFixed(1)}%, past --max-cost-increase-pct ${maxCostIncreasePct}`);
+ return {reasons,failed:reasons.length>0&&!advisory};
+}
+
+// verdictJUnit is the verdict as one JUnit test case, for a CI system that
+// shows test reports: failed when the gate failed, skipped with no verdict.
+export function verdictJUnit(suite,answer,verdict){
+ const check=answer.check,name=check?.title??'No verdict yet';
+ const body=!check?`<skipped message="${xml('No verdict within the wait; read it with cvm evals watch '+answer.run.id)}"/>`
+  :verdict.failed?`<failure message="${xml(verdict.reasons.join('; '))}">${xml(check.sentence)}</failure>`:'';
+ return `<?xml version="1.0" encoding="UTF-8"?>\n<testsuite name="${xml('Caveman / '+suite)}" tests="1" failures="${check&&verdict.failed?1:0}" skipped="${check?0:1}">\n`+
+  `  <testcase classname="${xml('caveman.'+suite)}" name="${xml(name)}">${body}<system-out>${xml([check?.summary,answer.details_url].filter(Boolean).join('\n\n'))}</system-out></testcase>\n</testsuite>\n`;
+}
+
+const ciUsage='cvm evals ci FILE.eval.json... (eval files: cvm evals ci FILE --help)  or  cvm evals ci [--suite NAME] [--junit GLOB]... [--junit-out PATH|-] [--advisory] [--max-usd N] [--max-quality-drop-pp N] [--max-cost-increase-pct N] [--timeout SECONDS] [--repository-id N] [--sha SHA] [--ref REF] [--base-sha SHA] [--base-ref REF] [--pr N] [--event push|pull_request|merge_group] [--job NAME] [--run-id N] [--run-attempt N] [--project UUID] -- TEST COMMAND';
+const text={type:'string'};
+
+export async function evalsCI(argv,{context}){
+ const split=argv.indexOf('--');
+ // Eval files instead of a test command: each is run and gated by cvm evals run's own path.
+ if(split<0&&argv.some(arg=>!arg.startsWith('-')&&arg.endsWith('.json')))return (await import('./evals.mjs')).evalsRun(argv,{context},{ci:true});
+ try{
+  const {values}=parseArgs({args:split<0?argv:argv.slice(0,split),strict:true,allowPositionals:false,options:{suite:text,junit:{type:'string',multiple:true},'junit-out':text,advisory:{type:'boolean'},'max-usd':text,
+   'max-quality-drop-pp':text,'max-cost-increase-pct':text,timeout:text,'repository-id':text,sha:text,ref:text,'base-sha':text,'base-ref':text,pr:text,event:text,job:text,'run-id':text,'run-attempt':text,project:text,help:{type:'boolean'}}});
+  if(values.help){process.stdout.write(inertJSON({usage:ciUsage,exit_codes:'the test command\'s own when it fails; else 0 no gate failed (always with --advisory), 5 the gate failed, 6 a gating suite had no verdict within --timeout'},2)+'\n');return 0;}
+  const command=split<0?[]:argv.slice(split+1);
+  if(!command.length)throw new CloudError('cave_usage',ciUsage,2);
+  const timeoutMs=seconds(values.timeout,180);
+  const limits={advisory:!!values.advisory,maxQualityDropPP:limit(values['max-quality-drop-pp'],'max-quality-drop-pp'),maxCostIncreasePct:limit(values['max-cost-increase-pct'],'max-cost-increase-pct')};
+  const run=detectRun(process.env,values);
+  const suite=values.suite??run.job;
+  const body={request_id:randomUUID(),...run,suite:{name:suite},...(values['max-usd']===undefined?{}:{max_usd:limit(values['max-usd'],'max-usd')})};
+  if(!sha40(run.sha))throw new CloudError('cave_usage','No commit found: run inside the checkout or pass --sha with the full commit.',2);
+  if(!(run.repository_id>0))throw new CloudError('cave_usage','Pass --repository-id: the numeric id of the repository connected to this project (gh api repos/OWNER/NAME --jq .id). GitHub Actions and GitLab CI supply it themselves.',2);
+
+  // Start. With --advisory Caveman never fails the job, an unreachable or
+  // refusing API included: the tests then run unmeasured.
+  let client,base,started;
+  try{
+   client=await connect(context,values.project);base=projectPath(client)+'/ci/runs';
+   started=await client.request(base,{body});
+  }catch(error){
+   if(!limits.advisory||error.exitCode===2&&error.code==='cave_usage')throw error;
+   say(`caveman: ${error.code??'cave_request_failed'}: ${error.message} The tests run unmeasured (--advisory).`);
+  }
+  const env={...process.env};
+  if(started?.key){
+   Object.assign(env,{CAVE_API_KEY:started.key.token,CAVE_CI_RUN_ID:started.run.id},started.key.gateway_url?{CAVE_GATEWAY_URL:started.key.gateway_url}:{});
+   say(`caveman: run ${started.run.id} started, ${started.mode} mode, baseline ${started.baseline?.match}, key capped at $${started.key.max_usd}.`);
+  }else if(started)say(`caveman: run ${started.run.id} has no key; its model calls are not measured.`);
+
+  // Windows runs a .cmd shim through cmd.exe with every argument quoted.
+  const child=process.platform==='win32'
+   ?spawnSync(process.env.ComSpec??'cmd.exe',['/d','/s','/c',`"${command.map(arg=>`"${String(arg).replaceAll('"','""')}"`).join(' ')}"`],{stdio:'inherit',env,windowsVerbatimArguments:true})
+   :spawnSync(command[0],command.slice(1),{stdio:'inherit',env});
+  if(child.error)say(`caveman: could not run ${command[0]}: ${child.error.message}`);
+  const commandExit=child.status??(child.error?127:1);
+  if(!started)return commandExit;
+
+  const status=child.signal?'cancelled':commandExit===0?'completed':'failed';
+  const patterns=values.junit?.length?values.junit:['**/junit*.xml'];
+  const {reports,skipped}=collectReports(patterns,process.cwd());
+  if(skipped.length)say(`caveman: ${skipped.length} JUnit reports were over the upload limit (20 files, 10 MiB) and were left out: ${skipped.join(', ')}`);
+  if(!reports.length)say(`caveman: no JUnit report matched ${patterns.join(', ')}; there are no tests to compare.`);
+  const finish=`${base}/${started.run.id}/finish`;
+  let answer;
+  try{
+   answer=await client.request(finish,{body:{status,junit:reports}});
+   say(`caveman: uploaded ${reports.length} JUnit reports for run ${started.run.id}.`);
+   // A closed run is returned unchanged: repeating the finish reads its verdict.
+   if(status!=='cancelled'&&timeoutMs>0)answer=await until(()=>client.request(finish,{body:{status}}),a=>a.run?.status==='complete',timeoutMs,()=>say('caveman: scoring…'))??answer;
+  }catch(error){
+   if(!limits.advisory)throw error;
+   say(`caveman: ${error.code??'cave_request_failed'}: ${error.message} (--advisory: the job is not failed).`);
+   return commandExit;
+  }
+  const mode=answer.mode??started.mode;
+  const verdict=answer.check?judge({...answer,mode},limits):{reasons:[],failed:false};
+  if(answer.check){
+   say(`caveman: ${answer.check.title} (${mode} mode). ${answer.check.sentence}`);
+   if(answer.check.summary)say('\n'+answer.check.summary+'\n');
+  }else if(status!=='cancelled')say(`caveman: no verdict for run ${started.run.id} yet. Read it later with: cvm evals watch ${started.run.id} --project ${client.projectId}`);
+  for(const reason of verdict.reasons)say(`caveman: ${reason}${verdict.failed?'':' (--advisory: the job is not failed)'}.`);
+  if(answer.details_url)say(`caveman: ${answer.details_url}`);
+  // A gating suite with no verdict has not passed: this exit code is its gate.
+  const unread=!answer.check&&status!=='cancelled'&&timeoutMs>0&&mode==='gate'&&!limits.advisory;
+  const exit=commandExit!==0?commandExit:verdict.failed?5:unread?6:0;
+  const report=verdictJUnit(suite,answer,verdict);
+  if(values['junit-out']&&values['junit-out']!=='-')writeFileSync(values['junit-out'],report);
+  process.stdout.write(values['junit-out']==='-'?report:inertJSON({schema_version:1,ok:exit===0,run_id:started.run.id,suite,mode,status:answer.run?.status,conclusion:answer.conclusion??null,
+   verdict:answer.check?{kind:answer.check.kind,title:answer.check.title,sentence:answer.check.sentence,quality:answer.check.quality??null,cost:answer.check.cost??null}:null,
+   details_url:answer.details_url??null,advisory:limits.advisory,failed:verdict.failed,reasons:verdict.reasons,command_exit:commandExit})+'\n');
+  return exit;
+ }catch(error){
+  const e=error instanceof CloudError?error:new CloudError('cave_usage',error.message??'Command failed.',2);
+  process.stdout.write(inertJSON({schema_version:1,ok:false,status:'failed',error:{code:e.code,message:e.message}})+'\n');return e.exitCode;
+ }
+}
+
+const watchUsage='cvm evals watch RUN_ID [--source eval_run|experiment|scenario_run|factory_comparison] [--timeout SECONDS] [--project UUID]  or  cvm evals watch WORKBENCH_ID RUN_ID [--timeout SECONDS] [--project UUID]';
+
+// evalsWatch waits for one run's verdict. Exit 0 ship, pass or skipped; 5
+// don't ship, or a gating check that failed; 6 need more data, or no verdict
+// within --timeout (the run continues).
+export async function evalsWatch(argv,{context,login}){
+ try{
+  const {values,positionals}=parseArgs({args:argv,strict:true,allowPositionals:true,options:{source:text,timeout:text,project:text,format:text,help:{type:'boolean'}}});
+  if(values.help){process.stdout.write(inertJSON({usage:watchUsage,exit_codes:'0 ship, pass or skipped; 5 don\'t ship or a failed gate; 6 need more data, or no verdict within --timeout'},2)+'\n');return 0;}
+  const timeoutMs=seconds(values.timeout,3600);
+  // A workbench run named with its workbench is the client's own wait, cell progress included.
+  if(positionals.length===2)return runCloudCLI(['evals','wait',...positionals,'--wait-timeout',String(Math.max(1,timeoutMs)),...(values.project?['--project',values.project]:[]),...(values.format?['--format',values.format]:[])],{context,login});
+  if(positionals.length!==1)throw new CloudError('cave_usage',watchUsage,2);
+  const client=await connect(context,values.project),input={source:values.source??'eval_run',run_id:positionals[0]};
+  projectPath(client);
+  let last='';
+  const tell=line=>{if(line!==last)say(`caveman: ${last=line}`);};
+  // A run that has not produced a verdict row yet reads as not found.
+  const read=()=>client.execute('verdicts.get',input).catch(error=>{if(error.code!=='cave_not_found')throw error;return undefined;});
+  const result=await until(read,r=>r&&r.data.state!=='running',timeoutMs,r=>tell(r?`${r.data.label}: running`:'no verdict yet'));
+  if(!result)throw new CloudError('cave_wait_timeout',`No verdict for run ${input.run_id} within the wait; the run continues. Reconnect with cvm evals watch ${input.run_id}.`,6);
+  const {verdict,gate,label,sentence}=result.data;
+  tell(`${label}: ${verdict}. ${sentence}`);
+  process.stdout.write(inertJSON(result)+'\n');
+  if(verdict==='dont_ship'||gate?.mode==='gate'&&['failure','action_required'].includes(gate.conclusion))return 5;
+  return ['ship','pass','skipped'].includes(verdict)?0:6;
+ }catch(error){
+  const e=error instanceof CloudError?error:new CloudError('cave_usage',error.message??'Command failed.',2);
+  process.stdout.write(inertJSON({schema_version:1,ok:false,status:'failed',error:{code:e.code,message:e.message}})+'\n');return e.exitCode;
+ }
+}