@vibe-agent-toolkit/cli 0.1.41-rc.4 → 0.1.41-rc.6

package/docs/skill-test.md

@@ -0,0 +1,213 @@
+# vat skill test - Eval Harness Commands
+
+## Overview
+
+The `vat skill test` commands run a packaged skill's eval suite in a headless,
+context-isolated Claude session. Each eval runs in two roles: a blind **executor**
+(the skill under test) performs the task, then a separate **grader** judges the
+captured transcript and emits a verdict.
+
+```bash
+vat skill test run <skill>        # execute the eval suite
+vat skill test configure <skill>  # persist knobs into vibe-agent-toolkit.config.yaml
+```
+
+> **This command executes skill code.** It is context-isolated, not an OS sandbox.
+> `--i-understand-this-runs-skill-code` is required to acknowledge that.
+
+Note the singular `vat skill` — distinct from the `vat skills` (plural) packaging
+commands documented in [skills.md](./skills.md).
+
+## Two config homes
+
+Most knobs exist both as a CLI flag (one-off) and a config key (persisted). **Flags
+override config.** There are two places config lives:
+
+| Home | Location | Holds |
+|---|---|---|
+| **Per-skill** | `skills.config.<skill>.test` | Everything about testing *one* skill — model under test, budgets, evals, companions |
+| **Global** | top-level `test:` node | Judge/pipeline knobs shared by every skill — `graderModel`, `concurrency` |
+
+```yaml
+# vibe-agent-toolkit.config.yaml
+test:                          # GLOBAL — the judge and the pipeline
+  graderModel: claude-sonnet-5
+  concurrency: 4
+
+skills:
+  config:
+    my-skill:
+      test:                    # PER-SKILL — the thing under test
+        model: claude-opus-5
+        timeout: 1500
+        auth: subscription
+        requireAuth: subscription
+        evals: evals/suite.json
+```
+
+Both blocks are validated under a **strict** schema — an unknown key is a config
+error, not a silent no-op. (This is the deliberate inverse of `evals.json`, which
+VAT reads liberally as adopter-authored data.)
+
+`vat skill test configure <skill>` writes the **per-skill** block using a
+comment-preserving YAML upsert, and validates values before writing. Prefer it over
+hand-editing.
+
+## Per-skill knobs (`skills.config.<skill>.test`)
+
+Every field is optional; an omitted knob falls back to its default.
+
+| Config key | Type / unit | Default | CLI flag | Purpose |
+|---|---|---|---|---|
+| `model` | string | claude's own default | `--model <id>` | Model for the **executor** (the skill under test). Passed **verbatim** to `claude --model` — VAT does no mapping or validation. Pin it for reproducible runs. |
+| `maxTurns` | integer > 0 | none | `--max-turns <n>` | Per-spawn cap on executor/grader turns. |
+| `maxBudgetUsd` | number > 0 | none | `--max-budget-usd <n>` | Hard USD budget cap passed to the CLI. |
+| `timeout` | integer, **seconds** | scales with eval count: ~`2min + 2min/eval`, floored at 5min, capped at 1h | `--timeout <s>` | Wall-clock timeout. An explicit value always overrides the scaled default. |
+| `stall` | integer, **seconds** | none | `--stall <s>` | Stall watchdog — kill the spawn after this long with no stream output. |
+| `evals` | string (path or npm specifier) | auto-detected `evals/evals.json` (see below) | `--evals <path>` | Which `evals.json` to grade against. `test.evals` resolves **relative to the skill source**; `--evals` resolves **relative to the current directory**. Either may be absolute, and either may be an npm bare specifier. An explicit value always wins over the default convention. |
+| `auth` | `inherit` \| `subscription` \| `api-key` \| `auto` | `inherit` | `--auth <mode>` | Auth mechanism for the spawned session. |
+| `requireAuth` | `subscription` \| `api-key` | none | `--require-auth <mech>` | Fail-fast guard: preflight exits `2` if the effective mechanism isn't this. |
+| `baseline` | boolean | `false` | `--baseline` | Run the opt-in with/without A/B skill-lift comparison. |
+| `skillCreator` | source descriptor | `{ vendored: true }` | — | Source for the vendored skill-creator rubric. |
+| `with` | array of source descriptors | none | `--with name=<src>` | **Required** companion skills staged alongside the subject, invocable by it. |
+| `optional` | array of source descriptors | none | `--with-optional name=<src>` | **Optional** companions — skipped with a warning if unresolvable. |
+| `env` | map string→string | none | `--env KEY=VALUE` | Env vars injected into the **executor** spawn. Values interpolate `${fixturesDir}`, `${stagedSkillDir}`, `${harnessRoot}`, `${resultsDir}`. `${fixturesDir}` is **per-eval** — it names that eval's own staged workspace (`fixtures/` under the executor's working directory), so the eval must declare input `files`; using it on an eval without them fails the run (exit 2). Protected names (PATH, auth, model, admin) cannot be overridden. |
+| `passEnv` | array of strings | none | `--pass-env KEY` | Names of host env vars to forward to the executor spawn if present. Protected names are ignored with a warning. |
+| `build` | string (shell command) | none | — | Command run **once, before staging**, to generate build artifacts. Runs with `cwd` = config root. A non-zero exit aborts the run (exit `2`). |
+
+A **source descriptor** is one of `{ workspace: <pkg> }`, `{ npm: <spec> }`,
+`{ url: <u>, sha256?: <hash> }`, `{ path: <dir> }`, or `{ vendored: true }`. On the
+command line the same sources are written as `name=workspace:<pkg>`,
+`name=npm:<spec>`, `name=url:<u>`, `name=path:<dir>`, or `name=vendored` — the CLI
+form requires an explicit companion `name=`, the config form derives it from the
+resolved skill.
+
+### Companion staging and builds
+
+A `path:` companion whose source directory maps to a **declared** skill is **built**
+first, exactly like the subject, so its `files:` build artifacts are injected — a
+companion backed by a bundled executable stages functional rather than inert. A
+required companion's build failure fails the run; an optional one falls back to raw
+unbuilt source only when the failure is non-destructive. Each declared skill builds
+**at most once per run**.
+
+Staging the same name twice across the subject, `--with`, and `--with-optional` is a
+duplicate-name error (exit `2`).
+
+### The default eval-suite convention
+
+**You do not need a `test:` block to have an eval suite.** If a skill has a file at:
+
+```
+<skill-source-dir>/evals/evals.json
+```
+
+VAT treats that as the skill's suite — the same path `vat skill test run` has always defaulted to. Two things follow from it, and they are deliberately the same two things an explicit `test:` block would give you:
+
+1. `vat skill test run <skill>` finds and runs the suite.
+2. **The suite directory is excluded from packaged output.** It holds the `expected_output` / `expectations` answer key, so shipping it would both publish the answers and let a skill under test read its own key.
+
+This is **auto-detection, not a requirement**. It is keyed on the suite *file* existing:
+
+| On disk | Treated as a suite? |
+|---|---|
+| `<skill>/evals/evals.json` | **Yes** — run by the harness, excluded from the bundle |
+| `<skill>/evals/` with no `evals.json` | No — ordinary content, ships normally |
+| `<skill>/docs/evals/…` (any other location named `evals`) | No — ordinary content, ships normally |
+| No `evals/` directory at all | No — nothing happens, no error, no warning |
+
+A skill with no suite is completely unaffected: it packages exactly as it would have, with no error and nothing excluded. The convention never makes evals mandatory.
+
+**Set `test.evals` explicitly** when your suite lives anywhere else — a shared directory, a per-skill subdirectory (VAT's own repo uses `evals/<skill-name>/evals.json`), or outside the skill tree. An explicit path always overrides the convention.
+
+> Both lanes honor this. Before it was shared, the harness auto-detected the suite while the packager did not, so a skill relying on the default had its answer key published even though the harness was protecting it.
+
+### Testing a skill you did not author
+
+A correctly packaged skill ships **no** eval suite — the suite is the answer key, and excluding it from the bundle is the whole point of the convention above. So for any skill you install rather than write, there is nothing in its tree to grade against, and you have to supply the suite yourself:
+
+```bash
+vat skill test run npm:@vendor/their-skill \
+  --evals ./our-audit-corpus/their-skill.json \
+  --i-understand-this-runs-skill-code
+```
+
+`--evals` resolves against the directory you run it from, so the path means what you typed. It may point anywhere — including outside the skill's tree entirely, which is the normal case here. It also accepts an npm bare specifier (`--evals @acme/skill-evals/their-skill.json`), honoring that package's `exports` map, so a shared corpus can be distributed as a package.
+
+The suite is never copied into anything the executor can reach. Each eval's declared input `files` are resolved relative to the **suite's** directory and staged into that eval's own workspace, so fixtures travel with the suite rather than with the skill.
+
+> **On sensitive fixtures.** `workspaces/` and `results/` are created `0700`, so a fixture that was never in your repo is not left readable by other local users. That is necessary but **not** sufficient: `grading.json` quotes the executor transcript verbatim as evidence, so anything the skill *reads out of* a fixture is written into `results/` as text — and `results/` survives `--keep` by design. Treat the harness output as being as sensitive as the fixtures you feed it.
+
+## Global knobs (top-level `test:`)
+
+| Config key | Type | Default | CLI flag | Purpose |
+|---|---|---|---|---|
+| `graderModel` | string | `claude-sonnet-5` | `--grader-model <id>` | Model for the fixed grader/judge. Passed verbatim to `claude --model`. Independent of `model` — you can run the skill under one model and grade under another. |
+| `concurrency` | integer > 0 | `4` | `--concurrency <n>` | Width of the bounded-parallel executor→grader pipeline. |
+
+## Run-only flags (no config key)
+
+| Flag | Purpose |
+|---|---|
+| `--i-understand-this-runs-skill-code` | **Required.** Acknowledges the command executes skill code. |
+| `--no-build` | Stage existing `dist` instead of building. Errors if absent for the subject or a **required** companion; an optional companion falls back to raw source with a warning. |
+| `--refresh` | Force a full re-stage, ignoring existing staged content. |
+| `--keep` | Keep the harness directory after the run (needed to inspect `results/`). |
+| `--dry-run` | Build and stage exactly as a real run would, then stop without spawning Claude — no session, no tokens. It **does build** (when `--i-understand-this-runs-skill-code` is passed), because the question a dry run answers is "what happens if I drop this flag", and a preview built from a stale `dist/` answers it wrongly. **Without** the acknowledgement it does not build — building runs the repo's `test.build` hook, an arbitrary shell command — so it falls back to an existing `dist/` and warns it may be stale. `--no-build` skips the build either way. |
+| `--out <dir>` / `--workdir <dir>` | Override the harness output / working directory. |
+| `--allow-eval-failure` | Opt out of fail-closed: exit `0` even when an eval fails. For interactive iteration. |
+| `--allow-unverified-skill-source` | Skip the vendored manifest integrity check. |
+| `--debug` | Enable debug logging. |
+
+## Exit codes
+
+The harness deliberately separates "the evals failed" from "the harness broke", so
+CI can tolerate one and gate on the other.
+
+- `0` - Run completed, all expectations passed
+- `1` - Harness broke (internal error, stall, timeout, grader nonce failure)
+- `2` - Preflight failure (bad config, unresolvable required companion, auth guard, `build` hook failed)
+- `3` - Bootstrap failure
+- `4` - **Eval failure** — the run completed but expectations did not all pass
+
+```bash
+vat skill test run my-skill --i-understand-this-runs-skill-code
+case $? in
+  0) ;;
+  4) echo "evals failed (tolerated)" ;;
+  *) exit 1 ;;    # harness broke — fail the build
+esac
+```
+
+## Results
+
+With `--keep`, the harness directory retains a `results/` tree that VAT is the sole
+writer of:
+
+- `grading.json` - per-expectation verdicts and the pass/total summary
+- `friction.json` - packaging issues observed during the run (advisory)
+- `tool-eval.json` - tool-expectation verdicts; always written, so check
+  `.evals.length` rather than file existence
+
+## Examples
+
+```bash
+# Run a declared skill's suite, honoring its test: config
+vat skill test run my-skill --i-understand-this-runs-skill-code
+
+# Pin both models independently, keep the harness for inspection
+vat skill test run my-skill --model claude-opus-5 --grader-model claude-sonnet-5 \
+  --keep --i-understand-this-runs-skill-code
+
+# Stage a required companion the subject is expected to invoke
+vat skill test run router-skill --with helper=path:./skills/helper \
+  --i-understand-this-runs-skill-code
+
+# Persist knobs instead of passing them every time
+vat skill test configure my-skill --auth subscription --require-auth subscription
+```
+
+## See Also
+
+- [skills.md](./skills.md) - `vat skills` (plural): packaging, validation, install
+- [index.md](./index.md) - full CLI command index

package/docs/skills.md

@@ -315,14 +315,22 @@ for terminology.
 
 **What it does:**
 1. Resolves the source (local directory, ZIP, .tgz, or npm package)
-2. Discovers one or more skills inside the source
+2. Discovers one or more skill directories inside the source
 3. Pre-validates every skill with `validateSkill()` — zero files written if validation fails
-4. Builds an install plan, detecting conflicts before touching the filesystem
-5. Copies skill(s) to the resolved platform directory (all-or-nothing semantics)
-6. Emits a YAML summary on stdout
+4. Reads each skill's declared name, which is the name it installs under
+5. Builds an install plan, detecting conflicts before touching the filesystem
+6. Copies skill(s) to the resolved platform directory (all-or-nothing semantics)
+7. Emits a YAML summary on stdout
 
 Both `--target` and `--scope` are **required** — there are no defaults.
 
+**Installed name:** a skill installs under the `name` its `SKILL.md` frontmatter
+declares — the same identity `vat skills build` and the plugin build key on — not
+the name of the directory it came from. A ZIP or npm source has no meaningful
+directory name to use. If two skills in one source declare the same name, the
+whole install fails rather than one silently overwriting the other. Use `--name`
+to override (single-skill sources only).
+
 **Supported Sources:**
 - **Local directory:** `./path/to/skill-dir` — must contain `SKILL.md` at root or in subdirectories
 - **Local ZIP file:** `./path/to/skill.zip` — extracted to a temp directory, then installed
@@ -354,6 +362,16 @@ Both `--target` and `--scope` are **required** — there are no defaults.
 | `windsurf` | `~/.codeium/windsurf/skills/` | `.windsurf/skills/` |
 | `agents` | `~/.agents/skills/` | `.agents/skills/` |
 
+These paths were last reviewed against vendor conventions on 2026-04-08 and are
+not verified at build or test time — the tests assert the table's shape, never
+that a path is where a platform actually looks. If a vendor moves its convention,
+installs land somewhere unread and every VAT check still passes. Re-check the
+vendor's documentation before relying on a non-`claude` target.
+
+**Visibility:** VAT's own inspection commands are Claude-scoped — `vat skills
+list --user` and `vat audit --user` read `~/.claude` only. A skill installed to
+any other target lands correctly but is invisible to them.
+
 **Exit Codes:**
 - `0` - Install successful (or dry-run complete)
 - `1` - Install error (validation failed, conflict without `--force`, source not found, bad target/scope)
@@ -438,6 +456,10 @@ for terminology.
 2. Reports validation status for each skill
 3. Shows skill metadata (name, description, path)
 
+The reported `name` is the one the skill's frontmatter declares — matching what
+`vat skills install` would install it as — falling back to the directory name
+only when the `SKILL.md` declares none.
+
 **Modes:**
 
 **Project mode (default):**
@@ -729,6 +751,8 @@ name: my-help  # Instead of "help"
 
 ## Related Commands
 
+- `vat skill test` - Run a skill's eval suite (note the **singular** `skill`); its
+  `test:` config surface is documented separately in [skill-test.md](./skill-test.md)
 - `vat audit` - Comprehensive validation for plugins, marketplaces, and skills
 - `vat resources validate` - Validate markdown resources (links, anchors)
 - `vat doctor` - Check environment and installation health
@@ -738,5 +762,6 @@ name: my-help  # Instead of "help"
 ## See Also
 
 - [CLI Reference](./index.md) - Complete CLI documentation
+- [Skill Test Command](./skill-test.md) - `vat skill test` eval harness and its config knobs
 - [Audit Command](./audit.md) - Comprehensive validation
 - [Resources Command](./resources.md) - Markdown resource validation

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vibe-agent-toolkit/cli",
-  "version": "0.1.41-rc.4",
+  "version": "0.1.41-rc.6",
   "description": "Command-line interface for vibe-agent-toolkit",
   "type": "module",
   "bin": {
@@ -41,17 +41,17 @@
   "//dependencies-note": "DO NOT add example agent packages or adopter packages here. Adopter packages (like vat-development-agents) depend on the CLI binary at build time — declaring them as deps creates a circular build dependency. They are built in a second turbo pass via package#task overrides in turbo.json.",
   "dependencies": {
     "@anthropic-ai/sdk": "^0.71.2",
-    "@vibe-agent-toolkit/agent-config": "0.1.41-rc.4",
-    "@vibe-agent-toolkit/agent-schema": "0.1.41-rc.4",
-    "@vibe-agent-toolkit/agent-skills": "0.1.41-rc.4",
-    "@vibe-agent-toolkit/claude-marketplace": "0.1.41-rc.4",
-    "@vibe-agent-toolkit/discovery": "0.1.41-rc.4",
-    "@vibe-agent-toolkit/gateway-mcp": "0.1.41-rc.4",
-    "@vibe-agent-toolkit/rag": "0.1.41-rc.4",
-    "@vibe-agent-toolkit/rag-lancedb": "0.1.41-rc.4",
-    "@vibe-agent-toolkit/resources": "0.1.41-rc.4",
-    "@vibe-agent-toolkit/utils": "0.1.41-rc.4",
-    "adm-zip": "^0.5.16",
+    "@vibe-agent-toolkit/agent-config": "0.1.41-rc.6",
+    "@vibe-agent-toolkit/agent-schema": "0.1.41-rc.6",
+    "@vibe-agent-toolkit/agent-skills": "0.1.41-rc.6",
+    "@vibe-agent-toolkit/claude-marketplace": "0.1.41-rc.6",
+    "@vibe-agent-toolkit/discovery": "0.1.41-rc.6",
+    "@vibe-agent-toolkit/gateway-mcp": "0.1.41-rc.6",
+    "@vibe-agent-toolkit/rag": "0.1.41-rc.6",
+    "@vibe-agent-toolkit/rag-lancedb": "0.1.41-rc.6",
+    "@vibe-agent-toolkit/resources": "0.1.41-rc.6",
+    "@vibe-agent-toolkit/utils": "0.1.41-rc.6",
+    "adm-zip": "^0.6.0",
     "commander": "^12.1.0",
     "picomatch": "^4.0.3",
     "semver": "^7.7.3",
@@ -65,7 +65,7 @@
     "@types/picomatch": "^4.0.2",
     "@types/semver": "^7.7.1",
     "@types/tar": "^6.1.13",
-    "@vibe-agent-toolkit/dev-tools": "0.1.41-rc.4",
+    "@vibe-agent-toolkit/dev-tools": "0.1.41-rc.6",
     "rimraf": "^6.0.1",
     "typescript": "^5.7.3",
     "vitest": "^3.2.4"