@caveman-ai/cloud 0.1.0-rc.21 → 0.1.0-rc.23

package/README.md

@@ -3,8 +3,12 @@
 Every Caveman Cloud operation from the terminal, plus the Cloud MCP server.
 
 ```sh
-npm i -g @caveman-ai/cloud
-cvm login                 # runs `caveman login`; one sign-in for both CLIs
+npm i -g "@caveman-ai/cloud@>=0.1.0-rc.22"   # the first release with setup; `latest` is still 0.0.1
+cvm setup --project PROJECT_ID   # signs in, picks the project, installs the Caveman Cloud skill
+```
+
+```sh
+cvm login                 # browser sign-in; one sign-in for cvm and caveman
 cvm projects list
 cvm traces search --help
 cvm sql "SELECT model, count() FROM requests GROUP BY model"
@@ -12,26 +16,76 @@ cvm tools list            # every operation this Cloud offers you
 cvm mcp                   # Cloud MCP server over stdio
 ```
 
-- **Sign-in.** cvm uses the credential `caveman login` saved. If `caveman` isn't installed, `cvm login` fetches it once with npx. In CI, set `CAVE_TOKEN` (and `CAVE_API_URL` for a private install).
+- **Sign-in.** `cvm login` signs in with the browser device flow: it prints a code and a link, opens the browser (`--no-browser` or `CAVE_NO_BROWSER=1` to skip) and waits for you to confirm. It stores the credential where `caveman login` does (the macOS Keychain, else `~/.caveman/credentials`), so either CLI's sign-in works for both; `cvm logout` signs both out. `--instance URL` (or `CAVE_API_URL`) signs in to a private install. In CI, set `CAVE_TOKEN` (and `CAVE_API_URL` for a private install).
 - **Environment.** A flag wins over the environment, and the environment over the saved login: `CAVE_API_URL`, `CAVE_PROJECT_ID`, `CAVE_TOKEN`. The saved credential is only ever sent to the address it was saved with; another `CAVE_API_URL` needs its own `CAVE_TOKEN`.
 - **Projects.** Use `--project UUID`, `CAVE_PROJECT_ID`, `cvm context bind` (writes `.caveman-cloud.json` in the repository) or the saved default.
 - **Output.** JSON when piped, readable when on a terminal; `--format json|jsonl|table`.
 - **Exit codes.** 0 ok · 1 request failed · 2 usage or invalid input (HTTP 400/422) · 3 signed out or forbidden (401/403) · 4 conflict (409) · 5 a comparison or gate failed · 6 timeout, still running or not enough data · 7 rate limited (429).
 
+## Setup
+
+`cvm setup` does three things, each safe to run again, and reports each:
+
+1. **Signs in**, unless the saved sign-in or `CAVE_TOKEN` already answers.
+2. **Picks the project**: `--project UUID`, else `CAVE_PROJECT_ID`, the repository's binding or the saved default, else your only project (with more than one it lists them and exits 2). It checks the project and saves it as the default, as `cvm context set` does.
+3. **Installs the Caveman Cloud skill** for the coding agents it finds: Claude Code in `~/.claude/skills/caveman-cloud` (`$CLAUDE_CONFIG_DIR/skills` when set), Codex and Cursor in `~/.agents/skills/caveman-cloud`. With none found it installs to `~/.agents/skills`. `--agent claude|codex|cursor` installs for that agent only.
+
+For Claude Code or Codex, setup also reports the finished steps to the console's onboarding screen (best effort). Setup writes nothing in the repository. It replaces only a skill it installed itself (marked by a `.cvm-version` file) and leaves any other `caveman-cloud` skill alone (exit 1). `cvm setup --remove` deletes the skills it installed and keeps you signed in. Every later `cvm` command refreshes an installed skill when cvm was upgraded. Then open your agent in the repository and ask it: "Set up Caveman Cloud for this repository."
+
+## An eval in one file
+
+```sh
+cvm evals init                                             # writes example.eval.json: three cases, a task, two criteria, a strict gate, a budget
+cvm evals validate example.eval.json                       # the schema, and its names against the project
+cvm evals run example.eval.json --set-baseline-on-pass     # run it and save the baseline
+cvm evals ci evals/*.eval.json                             # in CI: gate a change against that baseline
+```
+
+An eval file is JSON (`*.eval.json`; schema `https://caveman.so/schemas/eval-file.schema.json`):
+
+```json
+{
+  "name": "support-answers",
+  "dataset": {"cases": [{"id": "refund", "input": {"question": "How do I get a refund?"}, "expected": "Within 30 days.", "tags": ["billing"]}]},
+  "task": {"command": ["node", "answer.mjs"]},
+  "criteria": ["Grounded", {"type": "contains", "fragments": ["30 days"]}, {"judge": {"question": "Is the answer polite?", "pass": "Courteous", "fail": "Curt or rude"}}],
+  "trials": 3,
+  "gate": {"baseline": "saved", "policy": "strict"},
+  "budget_usd": 2
+}
+```
+
+- **name** is the experiment. The same name is the same experiment and the same dataset every time, so runs line up.
+- **dataset**: `cases` written in the file, or `{"name": "Support tickets", "version": 3}` for a dataset of the project. A command task runs here on the dataset's rows, so a named dataset with a command task needs a sign-in that may read them (an owner or admin); a CI key reads none. With a task Caveman runs, no row is read here and any sign-in that may run evals will do.
+- **task**: `{"command": [...]}` runs here, once per case and trial, directly and not through a shell: the case's input arrives as JSON on stdin, the output is read from stdout (JSON when it parses, else the text), and a non-zero exit or a timeout is a task error. Only a task error is tried again (`retries`, default 2; `timeout_seconds`, default 120; `concurrency`, default 4). `{"recorded": "output"}` grades outputs already in a dataset column. `{"kind": "model" | "prompt" | "http", "config": {...}}` is a target Caveman runs.
+- **criteria**: a library criterion by name, a fixed check (`{"type": "contains", ...}`; `cvm criteria catalog` lists the types), or a judge question.
+- **gate**: present, a full run is compared on the server with the experiment's saved baseline (or the run id you name). `strict` fails when any criterion got worse on any case, and decides on a handful of cases. `statistical` fails on a drop the numbers support and passes only when they support that too: that takes about 30 cases, and under that it reads need more data (exit 6), which fails a CI job. A run stores the file's policy on the experiment, so the console and the SDKs read the same two runs by the same rule; a gate that names no policy is read by the experiment's stored one (statistical until one is stored), and the result names the policy that decided. Absent, the run is reported and never compared. Two runs compare only on the same cases and criteria: after changing either on purpose, save the new run as the baseline (`cvm evals set_baseline --workbench-id ID --run-id ID`; the refusal prints the command).
+- **budget_usd**: the most the run may spend. Required when a project key runs the file, and over one trial.
+
+`cvm evals validate FILE...` exits 2 on an invalid file and every error names its JSON path (`$.criteria[1].fragments`): the schema, and each fixed check's type and options. `--offline` stops there and needs no sign-in; without it the dataset and library criterion names are looked up in the project too.
+
+`cvm evals run FILE...` flags: `--dry-run` runs the task here and sends nothing; `--trials N`; `--first N` or `--sample N --seed S` run some of the cases, and such a partial run is graded but never compared and can never become a baseline (the run says so itself, so the server refuses it too); `--set-baseline-on-pass` saves a passed run as the baseline, and the first run of an experiment, which has nothing to be compared with; `--junit-out PATH` writes a JUnit report with a test case per case and criterion; `--timeout SECONDS` bounds the wait for grading (default 1800).
+
+- **Exit.** 0 passed, or nothing to gate (no gate, no baseline yet, a partial or dry run) · 5 the gate failed · 6 no verdict: more data needed, the runs are not comparable, or not every output could be graded · 2 an invalid file · 1 the run crashed · 4 the server refused the run (a project key's daily budget is held). Several files exit by the worst of them, and nothing runs when one is invalid.
+- **Output.** On a terminal, a table: a row per case, a column per criterion (`pass`, `fail` or `error`, and `4/5` over trials), why the failing ones failed with the judge's own reason under it when it gave one, pass rates, the server's verdict sentence with its interval and every criterion that regressed, the baseline and the policy it was read by, cost (what Caveman measured; for a task you ran, its own judging cost, with what your task reported apart), and the console address when `CAVE_CONSOLE_URL` is set. Piped, one JSON object per file. `--format json|table` overrides.
+
+`cvm evals ci FILE...` is the same run for CI: each result also carries the commit, branch and pull request the CI system names, the run keeps them (`scope.commit` on its read, shown in the console), and `--advisory` spares the job the verdict (never an invalid file or a crash). It needs no GitHub: an eval file gates on GitLab CI, Buildkite, Jenkins or a laptop alike. Set `CAVE_TOKEN` to a project key with the eval scope, `CAVE_PROJECT_ID`, and `CAVE_API_URL` for a private install. On GitHub Actions the Caveman action's `files:` input does this. A file a key runs names `budget_usd` (each run's budget is held against the key's limit for the day) and writes its cases in the file, or uses a task Caveman runs. What a key reads of a run holds no judge reason and no error text, so the `Failing:` lines and the JUnit failures name the case and criterion alone; the reasons are on the run in the console.
+
 ## Pull request checks from any CI system
 
 ```sh
 cvm evals ci --suite evals --junit 'reports/junit*.xml' -- npm test
 ```
 
-Starts a CI run, runs the test command with the run's spend-capped key (`CAVE_API_KEY`, `CAVE_GATEWAY_URL`, `CAVE_CI_RUN_ID`), uploads the JUnit reports, waits for the verdict against the base branch and prints it. The commit, branch, pull request and job come from GitHub Actions, GitLab CI, Buildkite or Jenkins variables, or the checkout; `--sha`, `--ref`, `--base-sha`, `--base-ref`, `--pr`, `--job`, `--repository-id` override them.
+With a test command after `--`, `cvm evals ci` wraps a test suite instead of running eval files. It starts a CI run, runs the test command with the run's spend-capped key (`CAVE_API_KEY`, `CAVE_GATEWAY_URL`, `CAVE_CI_RUN_ID`), uploads the JUnit reports, waits for the verdict against the base branch and prints it. The commit, branch, pull request and job come from GitHub Actions, GitLab CI, Buildkite or Jenkins variables, or the checkout; `--sha`, `--ref`, `--base-sha`, `--base-ref`, `--pr`, `--job`, `--repository-id` override them.
 
 - `--advisory` never fails on Caveman's account: not on the verdict, not when Caveman cannot be reached.
 - `--max-quality-drop-pp N` and `--max-cost-increase-pct N` add limits of your own. They can only add a failure.
 - `--junit-out PATH` writes the verdict as a JUnit report for the CI system's test tab (`-` prints it).
 - `--max-usd N` caps the run's key; `--timeout SECONDS` bounds the wait (default 180, 0 does not wait).
 - Exit: the test command's own code when it fails; else 0, 5 when the gate failed, 6 when a gating suite had no verdict in time.
-- It signs in as a person (`CAVE_TOKEN` from a session, or `caveman login`): a project API key is refused. On GitHub Actions use the Caveman action instead, which needs no secret and writes the check run.
+- **Credential.** Set `CAVE_TOKEN` to a project API key made for running evals: in the console, New key with "Runs evals from CI" on and a spend limit per day (over the API, scope `eval:ci` alone with `budget_usd` and `budget_period: "daily"`). The key reaches only its own project's datasets, criteria, experiments, runs, verdicts and CI runs, reads no dataset rows or request bodies, and honours its IP allow-list. Every run it starts is given a budget (`--max-usd` here, `scope.budget_usd` on `cvm evals run`); the day's budgets add up to the key's limit, and the next is refused with `cave_key_daily_budget_exceeded` (exit 4) until the next UTC day. A person's session token still works. On GitHub Actions use the Caveman action instead, which needs no secret and writes the check run.
+- **Repository.** This check on your test suite gates only for a project with a GitHub repository connected, because whether a run comes from a fork is GitHub's answer. Any other project is refused with `cave_ci_repository_required` (exit 4): on GitLab, Buildkite or Jenkins without that connection, gate on the comparison verdict instead, which needs no repository: eval files, or `cvm evals run --wait` then `cvm evals compare` (exit 5 when the comparison fails).
 
 `cvm evals watch RUN_ID [--timeout SECONDS]` waits for any run's verdict: exit 0 ship, pass or skipped, 5 don't ship or a failed gate, 6 need more data or still running. `cvm evals watch WORKBENCH_ID RUN_ID` is `cvm evals wait`. Operations that start a long run take `--wait` (`cvm evals run --wait`).
 

package/bin/ci.mjs

@@ -16,19 +16,25 @@ const say=line=>process.stderr.write(line+'\n');
 const sha40=value=>/^[0-9a-f]{40}$/.test(value??'')&&!/^0+$/.test(value);
 const digits=value=>/^\d+$/.test(value??'')?value:undefined;
 const git=(...args)=>{try{return execFileSync('git',args,{encoding:'utf8',stdio:['ignore','pipe','ignore']}).trim();}catch{return '';}};
-const xml=text=>String(text).replace(/[<>&"]/g,c=>({'<':'&lt;','>':'&gt;','&':'&amp;','"':'&quot;'})[c]).replace(/[^\x09\x0a\x0d\x20-퟿-�]/g,'');
+export const xml=text=>String(text).replace(/[<>&"]/g,c=>({'<':'&lt;','>':'&gt;','&':'&amp;','"':'&quot;'})[c]).replace(/[^\x09\x0a\x0d\x20-퟿-�]/g,'');
 
 // Flags, then CAVE_PROJECT_ID, then the repository's binding, then the saved default.
-async function connect(host,project){
+export async function connect(host,project){
  const cfg=await host();
  let bound;try{bound=JSON.parse(readFileSync('.caveman-cloud.json','utf8')).project_id;}catch{}
- return new CloudClient({...cfg,projectId:project??process.env.CAVE_PROJECT_ID??bound??cfg.projectId});
+ const client=new CloudClient({...cfg,projectId:project??process.env.CAVE_PROJECT_ID??bound??cfg.projectId});
+ // A server that no longer speaks this build's operation contract answers a
+ // bare 409: say what to do about it in one line.
+ const request=client.request.bind(client);
+ client.request=async(...args)=>{try{return await request(...args);}catch(error){if(error?.code==='cave_operation_version_conflict')error.message=VERSION_CONFLICT;throw error;}};
+ return client;
 }
+export const VERSION_CONFLICT='This cvm and the Caveman server speak different API versions. Upgrade cvm (npm install -g @caveman-ai/cloud@latest), or install the cvm version that matches a self-hosted server, then run the command again.';
 function projectPath(client){
  if(!/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(client.projectId??''))throw new CloudError('cave_project_required','Select a project with --project, CAVE_PROJECT_ID or cvm context bind.',2);
  return `/api/v1/projects/${client.projectId}`;
 }
-function seconds(raw,fallback){
+export function seconds(raw,fallback){
  const value=Number(raw??fallback);
  if(!Number.isFinite(value)||value<0||value>86400)throw new CloudError('cave_usage','--timeout is seconds, 0 to 86400.',2);
  return value*1000;
@@ -112,11 +118,13 @@ export function verdictJUnit(suite,answer,verdict){
   `  <testcase classname="${xml('caveman.'+suite)}" name="${xml(name)}">${body}<system-out>${xml([check?.summary,answer.details_url].filter(Boolean).join('\n\n'))}</system-out></testcase>\n</testsuite>\n`;
 }
 
-const ciUsage='cvm evals ci [--suite NAME] [--junit GLOB]... [--junit-out PATH|-] [--advisory] [--max-usd N] [--max-quality-drop-pp N] [--max-cost-increase-pct N] [--timeout SECONDS] [--repository-id N] [--sha SHA] [--ref REF] [--base-sha SHA] [--base-ref REF] [--pr N] [--event push|pull_request|merge_group] [--job NAME] [--run-id N] [--run-attempt N] [--project UUID] -- TEST COMMAND';
+const ciUsage='cvm evals ci FILE.eval.json... (eval files: cvm evals ci FILE --help)  or  cvm evals ci [--suite NAME] [--junit GLOB]... [--junit-out PATH|-] [--advisory] [--max-usd N] [--max-quality-drop-pp N] [--max-cost-increase-pct N] [--timeout SECONDS] [--repository-id N] [--sha SHA] [--ref REF] [--base-sha SHA] [--base-ref REF] [--pr N] [--event push|pull_request|merge_group] [--job NAME] [--run-id N] [--run-attempt N] [--project UUID] -- TEST COMMAND';
 const text={type:'string'};
 
 export async function evalsCI(argv,{context}){
  const split=argv.indexOf('--');
+ // Eval files instead of a test command: each is run and gated by cvm evals run's own path.
+ if(split<0&&argv.some(arg=>!arg.startsWith('-')&&arg.endsWith('.json')))return (await import('./evals.mjs')).evalsRun(argv,{context},{ci:true});
  try{
   const {values}=parseArgs({args:split<0?argv:argv.slice(0,split),strict:true,allowPositionals:false,options:{suite:text,junit:{type:'string',multiple:true},'junit-out':text,advisory:{type:'boolean'},'max-usd':text,
    'max-quality-drop-pp':text,'max-cost-increase-pct':text,timeout:text,'repository-id':text,sha:text,ref:text,'base-sha':text,'base-ref':text,pr:text,event:text,job:text,'run-id':text,'run-attempt':text,project:text,help:{type:'boolean'}}});

package/bin/cvm.mjs

@@ -1,19 +1,24 @@
 #!/usr/bin/env node
-// cvm, the Caveman Cloud CLI. `caveman login` owns sign-in: cvm reads the
-// credential it saved (~/.caveman-cloud/config.json points at the macOS
-// Keychain item or the 0600 credentials file) and writes a refreshed one back
-// to the same place, so neither CLI ever signs the other out.
-import {readFileSync,writeFileSync,mkdirSync,chmodSync,renameSync,existsSync,realpathSync} from 'node:fs';
-import {execFileSync,spawnSync} from 'node:child_process';
+// cvm, the Caveman Cloud CLI. It signs in with the browser device flow and
+// stores the credential exactly where `caveman login` does
+// (~/.caveman-cloud/config.json points at the macOS Keychain item or the 0600
+// credentials file), so either CLI's sign-in serves both, and a refreshed pair
+// goes back to the same place, so neither CLI ever signs the other out.
+import {readFileSync,writeFileSync,mkdirSync,chmodSync,renameSync,existsSync,realpathSync,cpSync,rmSync,readdirSync,statSync} from 'node:fs';
+import {execFileSync,spawn} from 'node:child_process';
 import {randomUUID} from 'node:crypto';
 import {homedir} from 'node:os';
 import {join,dirname,delimiter} from 'node:path';
+import {fileURLToPath} from 'node:url';
+import {parseArgs} from 'node:util';
+import {setTimeout as pause} from 'node:timers/promises';
 
-const win=process.platform==='win32';
-const cmdQuote=arg=>`"${String(arg).replaceAll('"','""')}"`;
+const version=JSON.parse(readFileSync(new URL('../package.json',import.meta.url),'utf8')).version;
+const client=()=>import('../dist/cloud-client.mjs');
 const configPath=join(homedir(),'.caveman-cloud','config.json');
 const credentialsPath=join(process.env.CAVEMAN_HOME??join(homedir(),'.caveman'),'credentials');
 const keychain=['-s','caveman','-a','token'];
+const origin=url=>(url??'').replace(/\/+$/,'');
 
 function readConfig(){try{return JSON.parse(readFileSync(configPath,'utf8'));}catch{return {};}}
 function writePrivate(path,text){
@@ -30,7 +35,8 @@ function storedSecret(meta){
 }
 // A stored secret is the credentials JSON, or a bare access token in older stores.
 function decode(raw){try{const parsed=JSON.parse(raw);if(typeof parsed.access_token==='string')return parsed;}catch{}return {access_token:raw};}
-function expiresSoon(token){try{return JSON.parse(Buffer.from(token.split('.')[0],'base64url').toString('utf8')).exp<=Date.now()/1000+60;}catch{return false;}}
+const claims=token=>{try{return JSON.parse(Buffer.from(token.split('.')[0],'base64url').toString('utf8'))??{};}catch{return {};}};
+const expiresSoon=token=>claims(token).exp<=Date.now()/1000+60;
 function store(credentials){
  const secret=JSON.stringify(credentials);
  if(process.platform==='darwin'&&!process.env.CAVE_NO_KEYCHAIN){try{execFileSync('security',['add-generic-password','-U',...keychain,'-w',secret],{stdio:'ignore'});return 'keychain';}catch{}}
@@ -60,7 +66,7 @@ async function refresh(meta,credentials,baseURL){
 async function context(){
  // The environment wins over the saved login, and a flag over both (--project,
  // read by the client). logout leaves baseURL empty.
- const meta=readConfig(),origin=url=>(url??'').replace(/\/+$/,''),baseURL=process.env.CAVE_API_URL||meta.baseURL||'https://api.caveman.so';
+ const meta=readConfig(),baseURL=process.env.CAVE_API_URL||meta.baseURL||'https://api.caveman.so';
  const projectId=process.env.CAVE_PROJECT_ID||meta.projectId;
  if(process.env.CAVE_TOKEN)return {baseURL,token:process.env.CAVE_TOKEN,projectId};
  // The saved credential was issued by the saved address and is sent nowhere
@@ -70,31 +76,250 @@ async function context(){
  if(!meta.logoutPendingLocalCleanup&&credentials.refresh_token&&expiresSoon(credentials.access_token))credentials=await refresh(meta,credentials,baseURL);
  return {baseURL,token:credentials.access_token,projectId:projectId??credentials.project_id};
 }
-// Sign-in stays in caveman: run the installed one, or fetch it once with npx.
-function login(args){
- const installed=(process.env.PATH??'').split(delimiter).some(dir=>dir&&existsSync(join(dir,win?'caveman.cmd':'caveman')));
- const [command,prefix]=installed?['caveman',['login']]:['npx',['-y','@caveman-ai/cli@latest','login']];
- // Windows runs the .cmd shim through cmd.exe with every argument quoted.
- const result=win
-  ?spawnSync(process.env.ComSpec??'cmd.exe',['/d','/s','/c',`"${[command,...prefix,...args].map(cmdQuote).join(' ')}"`],{stdio:'inherit',windowsVerbatimArguments:true})
-  :spawnSync(command,[...prefix,...args],{stdio:'inherit'});
- if(result.status!==0)process.exit(result.status??1);
+const post=(url,body,headers={})=>fetch(url,{method:'POST',redirect:'error',headers:{'content-type':'application/json','x-cave-client':'cli',...headers},body:JSON.stringify(body),signal:AbortSignal.timeout(10000)});
+function openBrowser(url){
+ const [command,args]=process.platform==='darwin'?['open',[url]]:process.platform==='win32'?['rundll32.exe',['url.dll,FileProtocolHandler',url]]:['xdg-open',[url]];
+ try{spawn(command,args,{detached:true,stdio:'ignore'}).on('error',()=>{}).unref();}catch{}
+}
+// RFC 8628 device grant: the requests `caveman login` makes, and the
+// credential stored where it stores one.
+async function login(args=[]){
+ const {CloudError}=await client();
+ let values;try{({values}=parseArgs({args,strict:true,options:{instance:{type:'string'},'no-browser':{type:'boolean'}}}));}catch{throw new CloudError('cave_usage','Use cvm login [--instance URL] [--no-browser].',2);}
+ const baseURL=origin(values.instance||process.env.CAVE_API_URL||readConfig().baseURL||'https://api.caveman.so');
+ let url;try{url=new URL(baseURL);}catch{}
+ if(!url||url.username||url.password||url.pathname!=='/'||url.search||url.hash||url.protocol!=='https:'&&!(url.protocol==='http:'&&['localhost','127.0.0.1','[::1]'].includes(url.hostname)))throw new CloudError('cave_endpoint_invalid','Sign in over HTTPS (plain HTTP only on localhost), to an origin with no path, credentials, query or fragment.',2);
+ const started=await post(`${baseURL}/api/v1/auth/device/code`,{}).catch(()=>{throw new CloudError('cave_network_error',`Could not reach ${baseURL}.`,1);});
+ const code=await started.json().catch(()=>({}));
+ if(started.status===403&&code.error?.code==='cave_device_login_disabled')throw new CloudError('cave_device_login_disabled','Device login is not open on this installation.',3);
+ if(!started.ok||typeof code.device_code!=='string')throw new CloudError(code.error?.code??'cave_device_login_failed',`Could not start sign-in at ${baseURL} (HTTP ${started.status}).`,1);
+ const link=code.verification_uri_complete??code.verification_uri;
+ process.stderr.write(`Confirm the code ${code.user_code} in your browser: ${link}\n`);
+ if(!values['no-browser']&&!process.env.CAVE_NO_BROWSER&&/^https?:\/\//.test(link))openBrowser(link);
+ let interval=Number(code.interval??5)*1000;const deadline=Date.now()+Number(code.expires_in??600)*1000;
+ const expired=new CloudError('expired_token','The code expired before it was confirmed. Run cvm login again.',3);
+ while(Date.now()<deadline){
+  await pause(Math.max(interval,200));
+  let response,grant;
+  // A dropped poll, a rate limit or a busy server is retried until the code expires.
+  try{response=await post(`${baseURL}/api/v1/auth/device/token`,{device_code:code.device_code});if(response.status>=500||response.status===429)continue;grant=await response.json();}catch{continue;}
+  if(grant.error==='slow_down'){interval+=5000;continue;}
+  if(grant.error==='authorization_pending')continue;
+  if(grant.error==='expired_token')throw expired;
+  if(grant.error==='access_denied')throw new CloudError('access_denied','Sign-in was denied in the browser.',3);
+  if(typeof grant.access_token!=='string'||!grant.access_token)throw new CloudError(typeof grant.error==='string'?grant.error:'cave_device_login_failed',`Sign-in failed: ${typeof grant.error==='string'?grant.error:`HTTP ${response.status}`}.`,1);
+  const credentials=Object.fromEntries(['access_token','refresh_token','gateway_api_key','gateway_key_id','project_id'].filter(key=>typeof grant[key]==='string'&&grant[key]).map(key=>[key,grant[key]]));
+  const tokenStore=store(credentials),{oid}=claims(credentials.access_token);
+  // No gatewayUrl: with one, `caveman enable claude` switches to managed mode
+  // and writes a gateway base URL with no key into Claude Code's settings.
+  const {token,logoutPendingLocalCleanup,gatewayUrl,organizationId,projectId,...rest}=readConfig();
+  writePrivate(configPath,JSON.stringify({...rest,baseURL,tokenStore,...(typeof oid==='string'?{organizationId:oid}:{}),...(credentials.project_id?{projectId:credentials.project_id}:{})},null,2)+'\n');
+  // The receipt: the server replays a durable grant until told it was stored,
+  // and may revoke one that never is.
+  if(Object.keys(credentials).length>1){
+   for(let attempt=0;;attempt++){
+    const ack=typeof grant.delivery_ack_token==='string'&&await post(`${baseURL}/api/v1/auth/device/ack`,{device_code:code.device_code,ack_token:grant.delivery_ack_token},{authorization:`Bearer ${credentials.access_token}`}).catch(()=>undefined);
+    if(ack?.ok)break;
+    if(attempt===4||ack===false||ack&&ack.status<500&&ack.status!==429)throw new CloudError('cave_device_ack_failed','Signed in, but the server did not confirm this device stored the credential and may revoke it. Run cvm login again.',1);
+    await pause(200*2**attempt);
+   }
+  }
+  return {baseURL,projectId:credentials.project_id??null};
+ }
+ throw expired;
+}
+async function loginCommand(args){
+ const {baseURL,projectId}=await login(args);
+ if(process.stdout.isTTY)process.stdout.write(`Signed in to ${baseURL}.\n`);
+ else{process.stderr.write(`Signed in to ${baseURL}.\n`);process.stdout.write(JSON.stringify({signed_in:true,api:baseURL,project_id:projectId})+'\n');}
+ return 0;
+}
+// `caveman logout`'s steps: revoke the device's gateway key and the session,
+// then forget the credential and the address.
+async function logout(){
+ const meta=readConfig(),baseURL=origin(meta.baseURL);
+ let credentials=decode(storedSecret(meta));
+ if(credentials.access_token&&baseURL&&!meta.logoutPendingLocalCleanup){
+  const {CloudError}=await client();
+  if(credentials.refresh_token&&expiresSoon(credentials.access_token))credentials=await refresh(meta,credentials,baseURL);
+  const kept=new CloudError('cave_logout_failed','Could not revoke this sign-in on the server; you are still signed in. Run cvm logout again.',1);
+  const revoke=(path,body,headers)=>post(baseURL+path,body,{authorization:`Bearer ${credentials.access_token}`,...headers}).catch(()=>{throw kept;});
+  const project=meta.projectId??credentials.project_id;
+  // The gateway key is gone only when the server says so (404 cave_key_not_found):
+  // after any other answer this is the last copy of a live key, so it stays.
+  if(project&&credentials.gateway_key_id){
+   const response=await revoke(`/api/v1/projects/${encodeURIComponent(project)}/keys/${encodeURIComponent(credentials.gateway_key_id)}/revoke`,{},{'x-cave-csrf':'cli'});
+   if(!response.ok&&!(response.status===404&&(await response.json().catch(()=>({}))).error?.code==='cave_key_not_found'))throw kept;
+  }
+  const response=await revoke('/api/v1/auth/logout',credentials.refresh_token?{refresh_token:credentials.refresh_token}:{});
+  if(!response.ok&&response.status!==401)throw kept; // 401: the session already ended
+ }
+ // Remote revocation is recorded before the only local credential goes, so a
+ // failed cleanup is retried without presenting a revoked session.
+ writePrivate(configPath,JSON.stringify({...meta,logoutPendingLocalCleanup:true},null,2)+'\n');
+ if(meta.tokenStore==='keychain'||!meta.tokenStore&&process.platform==='darwin'&&!process.env.CAVE_NO_KEYCHAIN){
+  try{execFileSync('security',['delete-generic-password',...keychain],{stdio:'ignore'});}catch(error){if(error.status!==44)throw new Error('Could not remove the credential from the macOS Keychain.');} // 44: no such item
+ }
+ if(meta.tokenStore!=='keychain')rmSync(credentialsPath,{force:true});
+ const {token,tokenStore,projectId,organizationId,gatewayUrl,logoutPendingLocalCleanup,...rest}=meta;
+ writePrivate(configPath,JSON.stringify({...rest,baseURL:''},null,2)+'\n');
+ process.stdout.write(process.stdout.isTTY?'Signed out.\n':JSON.stringify({logged_out:true})+'\n');
+ return 0;
+}
+
+// The Caveman Cloud skill, built from public/skills into this package. Claude
+// Code reads its own skills directory; Codex and Cursor read the open Agent
+// Skills one.
+const skillDirs={claude:join(process.env.CLAUDE_CONFIG_DIR||join(homedir(),'.claude'),'skills','caveman-cloud'),agents:join(homedir(),'.agents','skills','caveman-cloud')};
+const marker='.cvm-version',labels={claude:'Claude Code',codex:'Codex',cursor:'Cursor'};
+// One cvm at a time changes a skills directory, and first clears what an
+// interrupted one left. A lock over a minute old is a crashed cvm's (two cvms
+// taking over the same stale lock at one moment can both go ahead).
+function locked(dir,work){
+ const parent=dirname(dir),lock=join(parent,'.caveman-cloud.lock'),busy='another cvm is installing it';
+ try{mkdirSync(lock);}catch(error){
+  if(error.code!=='EEXIST')throw error;
+  if(Date.now()-statSync(lock).mtimeMs<60000)return busy;
+  rmSync(lock,{recursive:true,force:true});
+  try{mkdirSync(lock);}catch{return busy;}
+ }
+ try{
+  for(const name of readdirSync(parent))if(name.startsWith('.caveman-cloud-'))rmSync(join(parent,name),{recursive:true,force:true});
+  return work();
+ }finally{rmSync(lock,{recursive:true,force:true});}
+}
+// Only a directory cvm wrote (its marker says so) is replaced, by renames: an
+// agent reading it finds the old skill or the new one, never half of either.
+function installSkill(dir){
+ mkdirSync(dirname(dir),{recursive:true});
+ return locked(dir,()=>{
+  if(existsSync(dir)&&!existsSync(join(dir,marker)))return 'it holds a caveman-cloud skill cvm did not install';
+  const next=join(dirname(dir),`.caveman-cloud-${randomUUID()}`),old=next+'-old';
+  try{
+   cpSync(fileURLToPath(new URL('../dist/skills/caveman-cloud',import.meta.url)),next,{recursive:true});writeFileSync(join(next,marker),version);
+   if(existsSync(dir))renameSync(dir,old);
+   renameSync(next,dir);
+  }catch(error){
+   rmSync(next,{recursive:true,force:true});
+   if(!existsSync(dir)&&existsSync(old))renameSync(old,dir);
+   throw error;
+  }
+  rmSync(old,{recursive:true,force:true});
+ });
+}
+// Runs a client command for its JSON answer, the client's own printing held back.
+async function call(...args){
+ const {runCloudCLI}=await client();
+ let out='';const write=process.stdout.write;process.stdout.write=chunk=>(out+=chunk,true);
+ try{const code=await runCloudCLI([...args,'--format','json'],{context,login:loginCommand});return {...JSON.parse(out),code};}
+ finally{process.stdout.write=write;}
+}
+// What the onboarding screen shows of the repository, each only when git
+// answers it in a form the server accepts.
+function repository(){
+ const git=(...args)=>{try{return execFileSync('git',args,{encoding:'utf8',stdio:['ignore','pipe','ignore'],timeout:5000}).trim();}catch{return '';}};
+ const [,owner,name]=git('remote','get-url','origin').match(/([^/:]+)\/([^/]+?)(?:\.git)?\/?$/)??[];
+ const facts={repo_owner:owner,repo_name:name,default_branch:git('symbolic-ref','--short','refs/remotes/origin/HEAD').replace(/^origin\//,'')||git('branch','--show-current'),head_sha:git('rev-parse','HEAD')};
+ const valid={repo_owner:/^[A-Za-z0-9._-]{1,100}$/,repo_name:/^[A-Za-z0-9._-]{1,100}$/,default_branch:/^(?!-)(?!.*\.\.)[^\s~^:?*[\\\x00-\x1f\x7f]{1,255}$/,head_sha:/^[0-9a-f]{7,64}$/};
+ return Object.entries(facts).filter(([key,value])=>value&&valid[key].test(value));
+}
+async function setup(args){
+ const {CloudError}=await client();
+ const usage='Use cvm setup [--project UUID] [--agent claude|codex|cursor] [--remove] [--format json|table].';
+ let values;try{({values}=parseArgs({args,strict:true,options:{project:{type:'string'},agent:{type:'string'},remove:{type:'boolean'},format:{type:'string'}}}));}catch{throw new CloudError('cave_usage',usage,2);}
+ const json=(values.format??(process.stdout.isTTY?'table':'json'))==='json';
+ if(values.format&&!['json','table'].includes(values.format)||values.agent&&!labels[values.agent])throw new CloudError('cave_usage',usage,2);
+ const plain=text=>String(text).replace(/[\x00-\x1f\x7f-\x9f\p{Cf}\u2028\u2029]/gu,' ');
+ const out=(value,lines)=>process.stdout.write((json?JSON.stringify(value):lines.join('\n'))+'\n');
+ const failed=({code,...result})=>(process.stdout.write(JSON.stringify(result)+'\n'),code);
+ if(values.remove){
+  const removed=Object.values(skillDirs).filter(dir=>existsSync(dirname(dir))&&locked(dir,()=>existsSync(join(dir,marker))&&(rmSync(dir,{recursive:true,force:true}),true))===true);
+  out({skill:{removed}},[...removed.map(dir=>`Removed ${dir}`),...removed.length?[]:['No Caveman Cloud skill installed by cvm.'],'cvm logout signs you out.']);
+  return 0;
+ }
+ // 1. A credential that answers, or a sign-in.
+ let me=await call('auth','status');
+ if(me.code===3&&!process.env.CAVE_TOKEN){await login();me=await call('auth','status');}
+ if(me.code)return failed(me);
+ // 2. The project, resolved as every cvm command resolves it, saved the way
+ // `cvm context set` saves it.
+ const shown=await call('context','show',...(values.project?['--project',values.project]:[]));
+ if(shown.code)return failed(shown);
+ let project;
+ if(shown.project_id){const got=await call('projects','get',shown.project_id);if(got.code)return failed(got);project=got.data;}
+ else{
+  const list=await call('projects','list');if(list.code)return failed(list);
+  const projects=(list.data.data??[]).map(({id,name})=>({id,name}));
+  if(projects.length!==1){
+   out({schema_version:1,ok:false,status:'failed',error:{code:'cave_project_required',message:`This sign-in reaches ${projects.length} projects: pass --project UUID.`},projects},[`This sign-in reaches ${projects.length} projects: pass --project UUID.`,...projects.map(p=>`  ${p.id}  ${plain(p.name)}`)]);
+   return 2;
+  }
+  project=projects[0];
+ }
+ const saved=await call('context','set',project.id);if(saved.code)return failed(saved);
+ // 3. The skill, for the agent named, or for every agent found here.
+ const has=name=>(process.env.PATH??'').split(delimiter).some(dir=>dir&&['','.cmd','.exe'].some(ext=>existsSync(join(dir,name+ext))));
+ const found=values.agent?[values.agent]:[['claude',dirname(dirname(skillDirs.claude))],['codex',join(homedir(),'.codex')],['cursor',join(homedir(),'.cursor')]].filter(([name,dir])=>has(name)||existsSync(dir)).map(([name])=>name);
+ const agents=found.length?found:['codex','cursor']; // none found: the open directory, which both read
+ const dirOf=agent=>agent==='claude'?skillDirs.claude:skillDirs.agents,outcome=new Map();
+ for(const dir of new Set(agents.map(dirOf))){try{outcome.set(dir,installSkill(dir));}catch(error){outcome.set(dir,error.message);}}
+ const installed=agents.filter(agent=>!outcome.get(dirOf(agent))).map(agent=>({agent,path:dirOf(agent)}));
+ const refused=[...outcome].filter(([,reason])=>reason).map(([path,reason])=>({path,reason}));
+ // 4. The onboarding screen follows along; it knows the two clients below.
+ // Best effort: no report is no failure.
+ const reporter=values.agent??found.find(agent=>agent!=='cursor');
+ let reported=false;
+ if(['claude','codex'].includes(reporter)&&installed.length&&!refused.length){
+  try{
+   const session=await call('wizard','start','--client',reporter,...repository().map(([key,value])=>`--${key.replaceAll('_','-')}=${value}`),'--project',project.id);
+   const tasks=[['sign_in','Sign in',me.data.user.email||new URL(shown.endpoint).host],['project','Choose the project',project.name],['skill','Install the Caveman Cloud skill',[...new Set(installed.map(i=>labels[i.agent]))].join(', ')]].map(([id,label,detail])=>({id,label,status:'done',detail:String(detail).slice(0,240)}));
+   reported=!session.code&&!(await call('wizard','update',session.data.id,'--phase','completed','--tasks',JSON.stringify(tasks),'--project',project.id)).code;
+  }catch{}
+ }
+ const next=`Open ${labels[reporter??found[0]]??'your coding agent'} in your repository and ask: "Set up Caveman Cloud for this repository."`;
+ out({ok:!refused.length,signed_in:true,api:shown.endpoint,project:{id:project.id,name:project.name},skill:{version,installed,refused},onboarding_reported:reported,next},[
+  `Signed in as ${plain(me.data.user.email)} at ${shown.endpoint}`,`Project ${plain(project.name)} (${project.id})`,
+  ...[...new Set(installed.map(i=>i.path))].map(path=>`Installed the Caveman Cloud skill for ${installed.filter(i=>i.path===path).map(i=>labels[i.agent]).join(' and ')}: ${path}`),
+  ...refused.map(r=>`Left ${r.path} alone: ${r.reason}`),'',next]);
+ return refused.length?1:0;
 }
 
 const argv=process.argv.slice(2);
+// What only the operation form of `evals validate` and `evals run` takes.
+// A bare id is the experiment to run, unless it is the value of --project.
+const operationInput=(arg,i,args)=>/^--(file|stdin|workbench-id|scope|concurrency|outputs|wait|wait-timeout|idempotency-key|out)(=|$)/.test(arg)||args[i-1]!=='--project'&&/^[a-f0-9]{8}(-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(arg);
+// `cvm setup` with flags or nothing is onboarding; `cvm setup show|assert|hold` are operations.
+const onboarding=argv[0]==='setup'&&!(argv[1]&&!argv[1].startsWith('-'));
+// A skill cvm installed follows the installed cvm: an upgrade refreshes it on
+// the next command. Nothing is created here, and nothing here fails a command.
+if(!['--version','-v'].includes(argv[0])&&!(onboarding&&argv.includes('--remove'))){
+ for(const dir of Object.values(skillDirs))try{if(readFileSync(join(dir,marker),'utf8')!==version)installSkill(dir);}catch{}
+}
 if(argv[0]==='--version'||argv[0]==='-v'){
- process.stdout.write(JSON.parse(readFileSync(new URL('../package.json',import.meta.url),'utf8')).version+'\n');
-}else if(argv[0]==='login'){
- login(argv.slice(1));
+ process.stdout.write(version+'\n');
+}else if(['login','logout'].includes(argv[0])||onboarding){
+ // No TTY needed: a coding agent runs these through its shell, and the person
+ // confirms the sign-in in the browser.
+ try{process.exitCode=await {login:loginCommand,logout,setup}[argv[0]](argv.slice(1));}
+ catch(error){
+  const exit=error.exitCode??1;
+  process.stdout.write(JSON.stringify({schema_version:1,ok:false,status:'failed',error:{code:error.exitCode?error.code:'cave_failed',message:error.message}})+'\n');
+  process.exitCode=exit;
+ }
 }else if(argv[0]==='mcp'&&argv.length===1){
  const {serveCloudMCP}=await import('../dist/cloud-mcp.mjs');
  // A launcher may narrow the server to the operations one task needs.
  const granted=(process.env.CAVE_MCP_OPERATIONS??'').split(',').map(id=>id.trim()).filter(Boolean);
  await serveCloudMCP({context,...(granted.length?{grantedOperations:granted}:{})});
+}else if(argv[0]==='evals'&&(argv[1]==='init'||['validate','run'].includes(argv[1])&&!argv.slice(2).some(operationInput))){
+ // An eval file (*.eval.json) from nothing to a gated run. `evals validate
+ // --file MANIFEST` and `evals run WORKBENCH_ID` stay the client's own.
+ const evals=await import('./evals.mjs');
+ process.exitCode=await {init:evals.evalsInit,validate:evals.evalsValidate,run:evals.evalsRun}[argv[1]](argv.slice(2),{context});
 }else if(argv[0]==='evals'&&['ci','watch'].includes(argv[1])){
  const {evalsCI,evalsWatch}=await import('./ci.mjs');
- process.exitCode=await (argv[1]==='ci'?evalsCI:evalsWatch)(argv.slice(2),{context,login});
+ process.exitCode=await (argv[1]==='ci'?evalsCI:evalsWatch)(argv.slice(2),{context,login:loginCommand});
 }else{
- const {runCloudCLI}=await import('../dist/cloud-client.mjs');
- process.exitCode=await runCloudCLI(argv,{context,login});
+ const {runCloudCLI}=await client();
+ process.exitCode=await runCloudCLI(argv,{context,login:loginCommand});
 }