@icntswm/skillcheck 1.2.0 → 1.3.0

package/README.md

@@ -8,50 +8,49 @@
 **Regression tests for Claude Code skills.**
 
 You edited a skill description, installed a plugin or switched models, and now
-some of your requests load the wrong skill. Nothing warns you: the agent still
+some requests load the wrong skill. Nothing warns you: the agent still
 answers, just with the wrong instructions. skillcheck catches this before your
 users do.
 
 ![skillcheck catching a routing regression](docs/assets/demo.gif)
 
-A real run on the [demo](examples/demo): one skill's description got wider,
-and it started taking requests that belong to its neighbour. The free `lint`
-flags a suspicious description, and one batch call finds the two misrouted
-requests.
+<sub>A real run on the [demo](examples/demo): one description got wider and
+started taking its neighbour's requests. The free `lint` flags it, one batch
+call finds both misrouted requests.</sub>
 
-## What you get
+## Quick start
 
-- 🎯 **The real routing decision.** skillcheck runs Claude Code itself and reads
-  which skills the model actually loads. Nothing is simulated or guessed from
-  keywords.
-- 💸 **Cheap by design.** Free static checks first. Then one batch call covers
-  25 requests. A full run stops Claude Code the moment it picks a skill, so
-  you pay for the decision, not for the work.
-- 🔍 **Points at the fix.** The confusion block shows which skill took whose
-  requests. When the model names the right skill but doesn't load it, the
-  report says so separately.
-- 🛡️ **Safe on any project.** Runs go in plan mode with edits, shell, web and
-  MCP tools disabled. Your files are never touched.
-- 🧪 **Honest about randomness.** `--repeat` and `threshold` tell a flaky case
-  from a broken one instead of letting you guess.
-- 🏷️ **Catches renames and typos.** Case names are checked against the skills
-  the agent really has, so a renamed skill can't pass silently.
-- ⚙️ **CI-ready.** A GitHub Action (`uses: icntswm/skillcheck@v1`) that
-  comments the report on the pull request, JUnit, JSON and Markdown reports,
-  clear exit codes, a spending cap (`--budget`), and `--config-dir` to test
-  only the skills in your repository.
-- 📄 **Plain YAML, one dependency.** Cases are readable by anyone on the team
-  and live next to the skills they test.
-
-## When to run it
-
-- after you rewrite or shorten a skill description;
-- when you add a skill that sounds like an existing one;
-- after installing a plugin that brings its own skills;
-- before switching to another model;
-- on every pull request that touches `skills/`.
-
-## How it works in one minute
+Needs Node 20+ and [Claude Code](https://docs.anthropic.com/en/docs/claude-code)
+(`claude`) in `PATH`, logged in.
+
+```sh
+npm install -g @icntswm/skillcheck
+
+skillcheck init               # skillcheck.yaml listing your skills; add real requests
+skillcheck lint               # free static checks, no model calls
+skillcheck run --batch        # cheap pre-check: one call per 25 cases
+skillcheck run --only 2,5     # confirm what the batch flagged with real runs
+```
+
+| Command | What it does |
+|---|---|
+| `init` | starter `skillcheck.yaml` with your skill names, free |
+| `gen` | the model drafts cases from your descriptions, for you to review |
+| `import` | turns a skill-creator trigger eval set into cases |
+| `check` | validates the file and catches misspelled or renamed skills, free |
+| `lint` | short or look-alike descriptions, free |
+| `list` | the skills and commands Claude Code sees, free |
+| `run` | runs the cases; `--batch` for the cheap pre-check |
+
+Every option, the default file lookup and exit codes are in
+[Commands](docs/commands.md).
+
+No cases yet? `skillcheck gen -o skillcheck.yaml` lets the model draft them from
+your skill descriptions, for you to review. Coming from Anthropic's
+skill-creator? `skillcheck import eval_set.json --skill <name>` converts its
+trigger eval set. To try without installing: `npx @icntswm/skillcheck lint`.
+
+## How it works
 
 Write requests the way you actually type them, and say which skill must, or
 must not, load:
@@ -66,84 +65,66 @@ cases:
 ```
 
 skillcheck sends each request to a headless `claude -p`, watches which skills
-the model loads, and stops the process as soon as the answer is clear.
+the model loads, and stops the process as soon as the answer is clear. Runs go
+in plan mode with edits, shell, web and MCP tools off, so your files are never
+touched.
 
-## Three levels, from free to exact
-
-Start at the cheapest level and go up only when it finds nothing.
+Three levels, from free to exact. Go up only when the cheaper one finds nothing:
 
 | Command | Model calls | What it tells you |
 |---|---|---|
-| `skillcheck lint` | none | short descriptions, look-alike skills, cases whose expected skill shares few words with the request |
+| `skillcheck lint` | none | short or look-alike descriptions, cases that share few words with their skill |
 | `skillcheck run --batch` | one per 25 cases | which skill the model *says* it would load |
 | `skillcheck run` | one per case | which skill the model *actually* loads |
 
-On the demo suite one batch call covered all 12 cases for $0.05–0.10 and agreed
-with the normal run in 34 checks out of 34. More in [docs/cost.md](docs/cost.md).
-
-## Does it really catch regressions?
+On the [demo](examples/demo) (six skills, twelve cases, a bad edit of two
+descriptions) `lint` flagged the short description, and both `run --batch` and
+`run` caught the two misrouted cases. One batch call cost $0.05–0.10 and agreed
+with the full run in 34 checks out of 34.
 
-Yes, and the [demo](examples/demo) shows it: six skills, twelve cases and a bad
-edit of two descriptions.
-
-| | good skills | bad edit |
-|---|---|---|
-| `lint` | no problems | flags the too-short description |
-| `run --batch` | 12/12 passed | catches both misrouted cases |
-| `run` | 5/5 passed | catches the same two cases |
-
-The demo README also lists edits that did *not* break routing, and why.
-
-## Install
-
-Needs Node 20+ and [Claude Code](https://docs.anthropic.com/en/docs/claude-code)
-(`claude`) in `PATH`, logged in.
-
-```
-npm install -g @icntswm/skillcheck
-```
+## What you get
 
-The command is `skillcheck`. To try it without installing:
-`npx @icntswm/skillcheck lint`.
+- 🎯 **The real routing decision**, read from Claude Code itself, not guessed from keywords.
+- 💸 **You pay for the decision, not the work**: a run stops the moment a skill is picked.
+- 🔍 **Points at the fix**: the confusion block shows which skill took whose requests.
+- 🧪 **Honest about randomness**: `--repeat` and `threshold` tell a flaky case from a broken one.
+- 🏷️ **Catches renames**: case names are checked against the skills the agent really has.
+- 📄 **Plain YAML, one dependency**: cases live next to the skills they test.
 
-## Quick start
+## In CI
 
+```yaml
+- uses: icntswm/skillcheck@v1
+  env:
+    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+  with:
+    model: sonnet
+    budget: 2
 ```
-skillcheck init              # writes skillcheck.yaml listing your skills
-                             # then add a few real requests per skill
-skillcheck check             # validates the file, no model calls
-skillcheck lint              # free static checks
-skillcheck run --batch       # cheap pre-check, one call
-skillcheck run --only 2,5    # confirm what the batch flagged
-```
-
-`skillcheck list` prints the skills and slash commands Claude Code sees. It
-stops Claude Code before the first model call, so it costs nothing.
 
-Already tuned a description with Anthropic's skill-creator? `skillcheck import
-eval_set.json --skill <name>` turns its trigger eval set into cases (see
-[Writing cases](docs/writing-cases.md#from-skill-creator)).
+The action tests only the skills in your repository, comments the report on the
+pull request and writes JUnit, JSON and Markdown reports. `--budget` caps the
+spend; `--baseline` with `--only-new-failures` compares with `main` and fails
+only on new failures. Run it after you rewrite a description, add a skill that
+sounds like an existing one, install a plugin, or switch models.
 
 ## Documentation
 
 | | |
 |---|---|
-| [Writing cases](docs/writing-cases.md) | the file format and what makes a case catch regressions |
+| [Commands](docs/commands.md) | every command and option, exit codes |
+| [Writing cases](docs/writing-cases.md) | the file format, what makes a good case, `gen` and `import` |
 | [Cost](docs/cost.md) | what a run costs and how to spend less |
-| [Reports and CI](docs/ci.md) | confusion block, JSON, JUnit, Markdown, GitHub Actions, exit codes |
+| [Reports and CI](docs/ci.md) | confusion block, reports, GitHub Actions, comparing with `main` |
 | [How it works](docs/how-it-works.md) | what happens inside a run, and the limits of each level |
 
-`skillcheck --help` lists every command and option.
-
 ## Status
 
-Works with Claude Code. Agents sit behind a small adapter interface, so others
-that support skills can be added.
-
-Requests can be in any language: `run` asks the model itself. `lint` compares
-words, so it is tuned for English and Russian, works roughly for other
-languages that put spaces between words, and is of little use for Chinese,
-Japanese or Korean.
+Works with Claude Code; other agents with skills can be added behind a small
+adapter interface. Requests can be in any language: `run` asks the model
+itself. `lint` compares words, so it is tuned for English and Russian, works
+roughly for other languages with spaces between words, and is of little use for
+Chinese, Japanese or Korean.
 
 ## License
 

package/dist/baseline.js

@@ -0,0 +1,59 @@
+const STATUSES = ["passed", "failed", "skipped"];
+/** Parse a --json report; checked up front so a bad file fails before any model call. */
+export function parseBaseline(text) {
+    const value = JSON.parse(text);
+    if (value === null || typeof value !== "object" || value.tool !== "skillcheck" || !Array.isArray(value.cases)) {
+        throw new Error("not a skillcheck --json report");
+    }
+    value.cases.forEach((c, i) => {
+        const entry = c;
+        if (typeof entry !== "object" || entry === null || typeof entry.query !== "string" || !STATUSES.includes(entry.status) ||
+            (entry.id !== null && typeof entry.id !== "string")) {
+            throw new Error(`case ${i + 1} is not a report case (needs query, id and status passed, failed or skipped)`);
+        }
+    });
+    return value;
+}
+/**
+ * Compare current cases with a previous report. suite is every case of the cases file, so cases
+ * left out by --only or --skill do not count as removed.
+ */
+export function compare(current, baseline, file, suite = current) {
+    const previous = new Map();
+    for (const c of baseline.cases) {
+        const key = caseKey(c);
+        if (!previous.has(key))
+            previous.set(key, c);
+    }
+    let regressed = 0;
+    let fixed = 0;
+    let newCount = 0;
+    const changes = current.map((c) => {
+        const old = previous.get(caseKey(c));
+        if (c.status === "skipped")
+            return null;
+        // a case the baseline budget skipped has no result to compare with
+        if (old === undefined || old.status === "skipped") {
+            newCount++;
+            return "new";
+        }
+        if (old.status === "passed" && c.status === "failed") {
+            regressed++;
+            return "regressed";
+        }
+        if (old.status === "failed" && c.status === "passed") {
+            fixed++;
+            return "fixed";
+        }
+        return null;
+    });
+    const suiteKeys = new Set(suite.map(caseKey));
+    let removed = 0;
+    for (const key of previous.keys())
+        if (!suiteKeys.has(key))
+            removed++;
+    return { changes, summary: { file, regressed, fixed, new: newCount, removed } };
+}
+function caseKey(c) {
+    return c.id !== null ? `id:${c.id}` : `query:${c.query}`;
+}

package/dist/cli.js

@@ -4,12 +4,15 @@ import { readFile } from "node:fs/promises";
 import * as path from "node:path";
 import { pathToFileURL } from "node:url";
 import { parseArgs } from "node:util";
+import { isSeq, parse as parseYaml, parseDocument } from "yaml";
 import { abortActiveRuns, ClaudeAdapter, DEFAULT_DIRECTIVE } from "./agents/claude.js";
 import { BATCH_SCHEMA, buildBatchPrompt, parseBatchAnswer } from "./batch.js";
+import { parseBaseline } from "./baseline.js";
 import { Budget } from "./budget.js";
-import { BUILTIN_SKILLS, ConfigError, findDefaultCasesFile, knownSkillNames, loadSuite, unknownNames, } from "./cases.js";
+import { BUILTIN_SKILLS, ConfigError, findDefaultCasesFile, knownSkillNames, loadSuite, unknownNames, parseSuite, } from "./cases.js";
 import { confusion } from "./confusion.js";
 import { loadSkillDocs } from "./describe.js";
+import { GEN_SCHEMA, buildGenPrompt, caseEntry, genSuite, orderCases, parseGenAnswer } from "./gen.js";
 import { aggregate, judge } from "./judge.js";
 import { evalSetToSuite } from "./import.js";
 import { toJunit } from "./junit.js";
@@ -29,6 +32,7 @@ Usage:
   skillcheck init [file] [options]    write a starter cases file with those names
   skillcheck import <file> --skill <name>
                                       turn a skill-creator trigger eval set into cases
+  skillcheck gen [options]            draft cases from skill descriptions (one model call per 8 skills)
 
 run options:
   -a, --agent <name>     agent to route with (default: suite or claude)
@@ -47,6 +51,8 @@ run options:
                          and the terminal report to stderr)
       --junit <path>     JUnit XML report (GitLab/GitHub test reporters) to path
       --markdown <path>  Markdown summary (PR comments, GitHub job summary) to path
+      --baseline <path>  earlier --json report: mark regressed, fixed and new cases
+      --only-new-failures  exit 1 only for regressed or new failing cases, or budget skips (needs --baseline)
 run/check options:
       --skill <a,b>      keep only cases that mention these skills
 check options:
@@ -65,6 +71,15 @@ init options:
 import options:
       --skill <name>     the skill the eval set is about
   -o, --out <file>       write the cases there instead of stdout (--force overwrites)
+
+gen options:
+      --skill <a,b>      skills to draft (default: user/project skills)
+      --plugin <name>    skills of this installed plugin
+      --per-skill <n>    positive requests per skill (default: 4)
+  -o, --out <file>       write the cases there instead of stdout (--force overwrites)
+      --append <file>    add the draft to an existing cases file (default: skills it lacks)
+  -m, -j, --timeout and --config-dir work as for run
+
 common:
   -h, --help             show this help
       --version          show version
@@ -75,12 +90,12 @@ nothing reaches the model and nothing is billed, even when not logged in.
 
 Exit codes: 0 all passed, 1 some case failed or was skipped, 2 config or environment error.
 `;
-const COMMANDS = ["run", "check", "lint", "list", "init", "import"];
+const COMMANDS = ["run", "check", "lint", "list", "init", "import", "gen"];
 const OPTIONS = {
     help: { type: "boolean", short: "h", default: false },
     version: { type: "boolean", default: false },
-    agent: { type: "string" },
-    model: { type: "string" },
+    agent: { type: "string", short: "a" },
+    model: { type: "string", short: "m" },
     jobs: { type: "string", short: "j" },
     only: { type: "string" },
     repeat: { type: "string" },
@@ -96,22 +111,34 @@ const OPTIONS = {
     json: { type: "string" },
     junit: { type: "string" },
     markdown: { type: "string" },
+    baseline: { type: "string" },
+    "only-new-failures": { type: "boolean", default: false },
     top: { type: "string" },
     overlap: { type: "string" },
     strict: { type: "boolean", default: false },
     "config-dir": { type: "string" },
     force: { type: "boolean", default: false },
     out: { type: "string", short: "o" },
+    "per-skill": { type: "string" },
+    plugin: { type: "string" },
+    append: { type: "string" },
 };
 /** Config/environment problems: message to stderr, exit 2. */
 class UsageError extends Error {
 }
 function parseFlags(values, cwd) {
+    if (values.plugin !== undefined && values.skill !== undefined)
+        throw new UsageError("--plugin and --skill do not go together");
+    if (values.append !== undefined && values.out !== undefined)
+        throw new UsageError("--append and --out do not go together");
     if (values["batch-size"] !== undefined && !values.batch)
         throw new UsageError("--batch-size requires --batch");
     if (values.batch && (values.directive !== undefined || values["no-early-stop"])) {
         throw new UsageError("--directive and --no-early-stop do not apply to --batch");
     }
+    if (values["only-new-failures"] && values.baseline === undefined) {
+        throw new UsageError("--only-new-failures requires --baseline");
+    }
     return {
         help: values.help,
         version: values.version,
@@ -128,16 +155,21 @@ function parseFlags(values, cwd) {
         batch: values.batch,
         batchSize: intFlag(values["batch-size"], "batch-size", 1) ?? 25,
         skill: values.skill,
+        plugin: values.plugin,
+        append: values.append,
         budget: numberFlag(values.budget),
         json: values.json,
         junit: values.junit,
         markdown: values.markdown,
+        baseline: values.baseline,
+        onlyNewFailures: values["only-new-failures"],
         top: intFlag(values.top, "top", 1) ?? 5,
         overlap: ratioFlag(values.overlap) ?? 0.3,
         strict: values.strict,
         configDir: values["config-dir"] !== undefined ? resolveConfigDir(values["config-dir"], cwd) : undefined,
         force: values.force,
         out: values.out,
+        perSkill: intFlag(values["per-skill"], "per-skill", 1) ?? 4,
     };
 }
 /** --config-dir: absolute, must be an existing directory. */
@@ -225,6 +257,8 @@ export async function main(argv, io = { stdout: process.stdout, stderr: process.
             return await initCommand(positionals[1], flags, io, deps);
         if (command === "import")
             return importCommand(positionals[1], flags, io);
+        if (command === "gen")
+            return await genCommand(flags, io, deps);
         const file = casesFile(positionals[1], io);
         if (command === "run")
             return await runCommand(file, flags, io, deps);
@@ -428,6 +462,164 @@ function importCommand(positional, flags, io) {
     io.stdout.write(`wrote ${flags.out} (${total} cases: ${suite.positive} should load ${skill}, ${suite.negative} should not)\n`);
     return 0;
 }
+async function genCommand(flags, io, deps) {
+    const docs = loadSkillDocs({ cwd: io.cwd, configDir: flags.configDir });
+    const byName = new Map(docs.map((doc) => [doc.name, doc]));
+    const append = flags.append === undefined ? null : path.resolve(io.cwd, flags.append);
+    const existing = append === null ? null : await loadCases(append, io);
+    if (append !== null && existing === null)
+        return 2;
+    let targets;
+    if (flags.skill !== undefined) {
+        const names = flags.skill.split(",").map((name) => name.trim()).filter((name) => name !== "");
+        const unknown = names.filter((name, i) => !byName.has(name) && names.indexOf(name) === i);
+        if (unknown.length > 0)
+            throw new UsageError(`unknown skill: ${unknown.join(", ")}`);
+        targets = names.map((name) => byName.get(name)).filter((doc, i, all) => all.findIndex((other) => other.name === doc.name) === i);
+    }
+    else if (flags.plugin !== undefined) {
+        const pluginDocs = docs.filter((doc) => doc.plugin === flags.plugin);
+        if (pluginDocs.length === 0) {
+            const installed = [...new Set(docs.flatMap((doc) => doc.plugin === null ? [] : [doc.plugin]))].sort();
+            throw new UsageError(`unknown plugin: ${flags.plugin} (installed: ${installed.length > 0 ? installed.join(", ") : "none installed"})`);
+        }
+        targets = pluginDocs.filter((doc) => doc.kind === "skill");
+        if (targets.length === 0)
+            throw new UsageError(`plugin ${flags.plugin} has no skills`);
+    }
+    else if (append !== null && existing !== null) {
+        const covered = new Set(existing.cases.flatMap((item) => [...item.expect, ...item.expect_any, ...item.forbid, ...(item.first ? [item.first] : [])]));
+        targets = docs.filter((doc) => doc.kind === "skill" && doc.plugin === null && !covered.has(doc.name));
+        if (targets.length === 0) {
+            io.stderr.write(`nothing to draft: every skill already has cases in ${flags.append} (pass --skill to draft more)\n`);
+            return 0;
+        }
+    }
+    else {
+        targets = docs.filter((doc) => doc.kind === "skill" && doc.plugin === null);
+    }
+    if (targets.length === 0)
+        throw new UsageError("no skills found: put them in .claude/skills or pass --config-dir");
+    // before any model call: a refused file must not cost anything
+    const out = flags.out === undefined ? null : path.resolve(io.cwd, flags.out);
+    if (out !== null && fs.existsSync(out) && !flags.force)
+        throw new UsageError(`${flags.out} exists, use --force to overwrite`);
+    const adapter = pickAdapter(flags.agent ?? "claude", io, deps);
+    if (!adapter)
+        return 2;
+    if (!adapter.runBatch)
+        throw new UsageError(`agent ${adapter.name} cannot generate cases`);
+    const groups = [];
+    for (let i = 0; i < targets.length; i += 8)
+        groups.push(targets.slice(i, i + 8));
+    const results = await runPool(groups, flags.jobs, async (group) => {
+        try {
+            const result = await adapter.runBatch({
+                prompt: buildGenPrompt(group, docs, flags.perSkill),
+                schema: GEN_SCHEMA,
+                model: flags.model,
+                timeoutMs: flags.timeoutSec * 1000,
+                configDir: flags.configDir,
+            });
+            if (result.error) {
+                io.stderr.write(`skillcheck: gen failed for ${group.map((doc) => doc.name).join(", ")}: ${result.error}\n`);
+                return { cases: [], dropped: 0, costUsd: result.costUsd, failed: true };
+            }
+            const parsed = parseGenAnswer(result.structured, new Set(docs.map((doc) => doc.name)), new Set(group.map((doc) => doc.name)));
+            return { ...parsed, costUsd: result.costUsd, failed: false };
+        }
+        catch (e) {
+            const error = e.message;
+            io.stderr.write(`skillcheck: gen failed for ${group.map((doc) => doc.name).join(", ")}: ${error}\n`);
+            return { cases: [], dropped: 0, costUsd: null, failed: true };
+        }
+    });
+    const successful = results.filter((result) => result !== undefined && !result.failed);
+    if (successful.length === 0)
+        return 2;
+    // each answer is deduplicated on its own; groups can still repeat each other
+    const seen = new Set();
+    const cases = successful.flatMap((result) => result.cases).filter((item) => {
+        const key = item.query.trim();
+        if (seen.has(key))
+            return false;
+        seen.add(key);
+        return true;
+    });
+    const dropped = successful.reduce((total, result) => total + result.dropped, 0);
+    if (dropped > 0)
+        io.stderr.write(`note: dropped ${dropped} proposed cases naming unknown skills or none of the requested ones\n`);
+    const cost = genCost(results.map((result) => result?.costUsd ?? null));
+    if (cases.length === 0) {
+        io.stderr.write(`skillcheck: gen got no usable cases from the model (cost $${cost})\n`);
+        return 2;
+    }
+    if (append !== null) {
+        const oldQueries = new Set(existing.cases.map((item) => item.query.trim()));
+        const unique = cases.filter((item) => !oldQueries.has(item.query.trim()));
+        const duplicates = cases.length - unique.length;
+        if (unique.length > 0)
+            appendCases(append, unique, targets.map((doc) => doc.name), flags.model ?? null);
+        const suffix = duplicates > 0 ? `, ${duplicates} duplicates skipped` : "";
+        io.stdout.write(`appended ${unique.length} cases to ${flags.append} (${targets.length} skills${suffix}, cost $${cost})\n`);
+        return 0;
+    }
+    const text = genSuite(cases, { model: flags.model ?? null, skills: targets.map((doc) => doc.name), json: out !== null && path.extname(out).toLowerCase() === ".json" });
+    const counts = `${cases.length} cases for ${targets.length} skills, cost $${cost}`;
+    if (out === null) {
+        io.stdout.write(text);
+        io.stderr.write(`drafted ${counts}\n`);
+        return 0;
+    }
+    try {
+        fs.writeFileSync(out, text);
+    }
+    catch (e) {
+        throw new UsageError(`cannot write ${flags.out}: ${e.message}`);
+    }
+    io.stdout.write(`wrote ${flags.out} (${counts})\n`);
+    return 0;
+}
+function genCost(costs) {
+    const known = costs.filter((cost) => cost !== null);
+    const unknown = costs.length - known.length;
+    return known.length === 0 ? "?" : known.reduce((total, value) => total + value, 0).toFixed(2) +
+        (unknown > 0 ? ` + ${unknown} call${unknown === 1 ? "" : "s"} of unknown cost` : "");
+}
+function appendCases(file, cases, skills, model) {
+    const ext = path.extname(file).toLowerCase();
+    const ordered = orderCases(cases, skills);
+    const text = fs.readFileSync(file, "utf8");
+    let nextText;
+    if (ext === ".json") {
+        const data = JSON.parse(text);
+        if (Array.isArray(data))
+            data.push(...ordered.map(caseEntry));
+        else
+            data.cases.push(...ordered.map(caseEntry));
+        nextText = JSON.stringify(data, null, 2) + "\n";
+    }
+    else {
+        const doc = parseDocument(text);
+        // a bare list of cases is the older format, still accepted
+        const sequence = (isSeq(doc.contents) ? doc.contents : doc.get("cases", true));
+        for (const [i, item] of ordered.entries()) {
+            const node = doc.createNode(caseEntry(item));
+            // [name] like the rest of a cases file, not a block list
+            for (const key of ["expect", "forbid"]) {
+                const list = node.get(key, true);
+                if (isSeq(list))
+                    list.flow = true;
+            }
+            if (i === 0)
+                node.commentBefore = ` gen draft (model: ${model ?? "default"}): review these cases`;
+            sequence.add(node);
+        }
+        nextText = doc.toString({ flowCollectionPadding: false });
+    }
+    parseSuite(ext === ".json" ? JSON.parse(nextText) : parseYaml(nextText));
+    fs.writeFileSync(file, nextText);
+}
 const UNCOVERED_WIDTH = 78;
 function printLint(out, report, docs, suite) {
     const casesPart = suite ? `${report.cases} cases` : "no cases file";
@@ -503,6 +695,15 @@ async function runCommand(file, flags, io, deps) {
     const suite = await loadCases(file, io);
     if (!suite)
         return 2;
+    let baseline;
+    if (flags.baseline !== undefined) {
+        try {
+            baseline = parseBaseline(fs.readFileSync(path.resolve(io.cwd, flags.baseline), "utf8"));
+        }
+        catch (e) {
+            throw new UsageError(`cannot read baseline ${flags.baseline}: ${e.message}`);
+        }
+    }
     const agent = flags.agent ?? suite.agent;
     const adapter = pickAdapter(agent, io, deps);
     if (!adapter)
@@ -546,6 +747,24 @@ async function runCommand(file, flags, io, deps) {
     const expected = expectedNames(selected);
     const unavailable = outcome.sawAvailability ? expected.filter((name) => !outcome.available.has(name)) : [];
     const pairs = confusion(ordered);
+    const report = buildReport({
+        file,
+        agent,
+        model: model ?? null,
+        startedAtMs,
+        durationMs: Date.now() - startedAtMs,
+        cases: items.map((item) => ({ c: item.c, threshold: item.threshold, result: outcome.results[item.jobIndex] ?? null })),
+        unavailable,
+        confusion: pairs,
+        batch: flags.batch,
+        estimatedCostUsd: budget.spent,
+        skippedRunCosts,
+        budgetUsd: flags.budget ?? null,
+        budgetReached: skipped.length > 0,
+        baseline,
+        baselineFile: flags.baseline,
+        suiteCases: suite.cases.map((c) => ({ id: c.id ?? null, query: c.query })),
+    });
     reporter.summary(ordered, {
         unavailable,
         confusion: pairs,
@@ -553,6 +772,11 @@ async function runCommand(file, flags, io, deps) {
         budget: skipped.length > 0 ? { limitUsd: flags.budget, spent: budget.spent, notStartedRuns } : undefined,
         estimatedUsd: budget.spent,
         skippedRunCosts,
+        baseline: report.baseline ? {
+            summary: report.baseline,
+            regressed: report.cases.filter((c) => c.change === "regressed").map((c) => `#${c.id ?? c.index}`),
+            fixed: report.cases.filter((c) => c.change === "fixed").map((c) => `#${c.id ?? c.index}`),
+        } : undefined,
     });
     if (flags.batch) {
         // cases that only errored have no answer to confirm
@@ -562,21 +786,12 @@ async function runCommand(file, flags, io, deps) {
         const hint = failed.length > 0 ? ` (--only ${failed.join(",")})` : "";
         reporter.note(`batch mode: answers are the model's stated choice, not an actual Skill call — confirm failures with a normal run${hint}`);
     }
-    const report = buildReport({
-        file,
-        agent,
-        model: model ?? null,
-        startedAtMs,
-        durationMs: Date.now() - startedAtMs,
-        cases: items.map((item) => ({ c: item.c, threshold: item.threshold, result: outcome.results[item.jobIndex] ?? null })),
-        unavailable,
-        confusion: pairs,
-        batch: flags.batch,
-        estimatedCostUsd: budget.spent,
-        skippedRunCosts,
-        budgetUsd: flags.budget ?? null,
-        budgetReached: skipped.length > 0,
-    });
+    if (baseline && baseline.model !== null && report.model !== null && baseline.model !== report.model) {
+        reporter.note(`baseline was run with model ${baseline.model}, this run with ${report.model}`);
+    }
+    if (baseline && baseline.batch !== report.batch) {
+        reporter.note(baseline.batch ? "baseline was a batch run, this one is not" : "this is a batch run, the baseline was not");
+    }
     if (flags.json !== undefined) {
         const payload = JSON.stringify(report, null, 2) + "\n";
         if (flags.json === "-")
@@ -588,6 +803,9 @@ async function runCommand(file, flags, io, deps) {
         writeReportFile(flags.junit, toJunit(report), "--junit");
     if (flags.markdown !== undefined)
         writeReportFile(flags.markdown, toMarkdown(report), "--markdown");
+    if (flags.onlyNewFailures) {
+        return report.cases.some((c) => c.status === "skipped" || (c.status === "failed" && (c.change === "regressed" || c.change === "new"))) ? 1 : 0;
+    }
     return ordered.some((r) => !r.ok) || skipped.length > 0 ? 1 : 0;
 }
 function writeReportFile(path, text, flag) {

package/dist/gen.js

@@ -0,0 +1,119 @@
+import { parse as parseYaml } from "yaml";
+import { parseSuite } from "./cases.js";
+import { yamlName } from "./import.js";
+export const GEN_SCHEMA = {
+    type: "object",
+    properties: {
+        cases: {
+            type: "array",
+            items: {
+                type: "object",
+                properties: {
+                    query: { type: "string" },
+                    skill: { type: ["string", "null"] },
+                    avoid: { type: ["string", "null"] },
+                },
+                required: ["query", "skill", "avoid"],
+            },
+        },
+    },
+    required: ["cases"],
+};
+/** Order generated cases by their requested skills and convert one to a suite entry. */
+export function orderCases(cases, skills) {
+    const buckets = new Map(skills.map((skill) => [skill, []]));
+    const rest = [];
+    for (const item of cases) {
+        const target = item.avoid ?? item.skill;
+        const bucket = target === null ? undefined : buckets.get(target);
+        if (bucket)
+            bucket.push(item);
+        else
+            rest.push(item);
+    }
+    return [...skills.flatMap((skill) => buckets.get(skill) ?? []), ...rest];
+}
+/** A near miss owned by a neighbour gets both: expect the neighbour, forbid the target. */
+export function caseEntry(item) {
+    return {
+        query: item.query,
+        ...(item.skill !== null ? { expect: [item.skill] } : {}),
+        ...(item.avoid !== null ? { forbid: [item.avoid] } : {}),
+    };
+}
+/** Build the case-writing prompt from installed descriptions. */
+export function buildGenPrompt(targets, context, perSkill) {
+    const contextLines = context.map((doc) => `- ${doc.name}: ${JSON.stringify(doc.description.replace(/\s+/g, " ").slice(0, 300))}`);
+    const near = Math.ceil(perSkill / 2);
+    const targetLines = targets.map((doc) => `For ${doc.name}: write ${perSkill} positive requests and ${near} near misses. ` +
+        `Positive requests have skill ${JSON.stringify(doc.name)} and avoid null. ` +
+        `Near misses have avoid ${JSON.stringify(doc.name)}, and skill is another listed skill or null.`);
+    return [
+        "This is a test-writing task. Do not load any skill, do not use tools other than the structured output.",
+        "Context skills:",
+        ...contextLines,
+        "For each target skill, write realistic first messages a user would send. They should vary in wording and length, must not name the skill, and should be written in the language of the skill description.",
+        ...targetLines,
+        "Return only the structured output. Each request must be a JSON string.",
+    ].join("\n");
+}
+// targets: the skills asked for; every case must load or avoid one of them
+export function parseGenAnswer(structured, known, targets = known) {
+    if (typeof structured !== "object" || structured === null || Array.isArray(structured))
+        return { cases: [], dropped: 0 };
+    const rawCases = structured.cases;
+    if (!Array.isArray(rawCases))
+        return { cases: [], dropped: 0 };
+    const cases = [];
+    const seen = new Set();
+    let dropped = 0;
+    for (const raw of rawCases) {
+        if (typeof raw !== "object" || raw === null || Array.isArray(raw)) {
+            dropped++;
+            continue;
+        }
+        const item = raw;
+        const query = item.query;
+        const skill = item.skill === null ? null : item.skill;
+        const avoid = item.avoid === null ? null : item.avoid;
+        if (typeof query !== "string" || query.trim() === "" ||
+            (skill !== null && (typeof skill !== "string" || !known.has(skill))) ||
+            (avoid !== null && (typeof avoid !== "string" || !known.has(avoid))) ||
+            (skill === null && avoid === null) || skill === avoid || seen.has(query) ||
+            !((skill !== null && targets.has(skill)) || (avoid !== null && targets.has(avoid)))) {
+            dropped++;
+            continue;
+        }
+        seen.add(query);
+        cases.push({ query, skill: skill, avoid: avoid });
+    }
+    return { cases, dropped };
+}
+export function genSuite(cases, meta) {
+    const ordered = orderCases(cases, meta.skills);
+    if (meta.json) {
+        // JSON has no comments: the review note lives only in the command output
+        const data = { agent: "claude", repeat: 1, threshold: 1, cases: ordered.map(caseEntry) };
+        parseSuite(data);
+        return JSON.stringify(data, null, 2) + "\n";
+    }
+    const lines = [
+        `# Draft cases written by skillcheck gen (model: ${meta.model ?? "default"}) for: ${meta.skills.join(", ")}.`,
+        "# Review every case: the model guessed what should route where. Delete what is wrong,",
+        "# keep what matches how people really ask, then `skillcheck run --batch`.",
+        "agent: claude",
+        "repeat: 1",
+        "threshold: 1.0",
+        "cases:",
+    ];
+    for (const item of ordered) {
+        lines.push(`  - query: ${JSON.stringify(item.query)}`);
+        if (item.skill !== null)
+            lines.push(`    expect: [${yamlName(item.skill)}]`);
+        if (item.avoid !== null)
+            lines.push(`    forbid: [${yamlName(item.avoid)}]`);
+    }
+    const text = lines.join("\n") + "\n";
+    parseSuite(parseYaml(text));
+    return text;
+}

package/dist/import.js

@@ -2,6 +2,9 @@ import * as path from "node:path";
 import { parse as parseYaml } from "yaml";
 import { ConfigError, parseSuite } from "./cases.js";
 const PLAIN_NAME = /^[A-Za-z0-9._:/-]+$/;
+export function yamlName(name) {
+    return PLAIN_NAME.test(name) && parseYaml(name) === name ? name : JSON.stringify(name);
+}
 /**
  * A skill-creator trigger eval set ([{query, should_trigger}], one skill) as a
  * cases file: should_trigger true becomes expect, false becomes forbid.
@@ -29,7 +32,7 @@ export function evalSetToSuite(data, skill, source) {
     if (errors.length > 0)
         throw new ConfigError(errors);
     // plain only when YAML reads it back as the same string: not true, null, 123
-    const name = PLAIN_NAME.test(skill) && parseYaml(skill) === skill ? skill : JSON.stringify(skill);
+    const name = yamlName(skill);
     const lines = [
         `# skillcheck cases imported from ${path.basename(source)} (skill-creator trigger eval set).`,
         "# should_trigger: true became expect, false became forbid: another skill may",

package/dist/junit.js

@@ -19,6 +19,12 @@ export function toJunit(report) {
         properties.push(`      <property name="unknownCostRuns" value="${report.summary.unknownCostRuns}"/>`);
         properties.push(`      <property name="estimatedCostUsd" value="${report.summary.estimatedCostUsd.toFixed(2)}"/>`);
     }
+    if (report.baseline) {
+        properties.push(`      <property name="baselineRegressed" value="${report.baseline.regressed}"/>`);
+        properties.push(`      <property name="baselineFixed" value="${report.baseline.fixed}"/>`);
+        properties.push(`      <property name="baselineNew" value="${report.baseline.new}"/>`);
+        properties.push(`      <property name="baselineRemoved" value="${report.baseline.removed}"/>`);
+    }
     properties.push("    </properties>");
     const cases = rows.map(({ c, kind }) => testcase(c, kind, suiteName));
     return [

package/dist/markdown.js

@@ -6,6 +6,10 @@ export function toMarkdown(report) {
     const lines = [MARKDOWN_MARKER, header(report)];
     lines.push("");
     lines.push(metaLine(report));
+    if (report.baseline) {
+        lines.push("");
+        lines.push(`> Since baseline: ${baselineText(report.baseline)}`);
+    }
     if (report.batch) {
         lines.push("");
         lines.push("> Batch mode: answers are the model's stated choice, not an actual Skill call. Confirm failures with a normal run.");
@@ -101,6 +105,10 @@ function reasonCell(c) {
     const bad = c.runs.filter((r) => !r.ok);
     const parts = c.runs.length > 1 ? [`${c.passed}/${c.runs.length}`, bad[0]?.reason ?? ""] : [bad[0]?.reason ?? ""];
     let text = parts.join(" · ");
+    if (c.change === "regressed")
+        text = `**regressed** · ${text}`;
+    else if (c.change === "new")
+        text = `new · ${text}`;
     const diagnosis = bad.find((r) => r.diagnosis !== null)?.diagnosis;
     if (diagnosis)
         text += ` — _${diagnosis}_`;
@@ -110,9 +118,18 @@ function reasonCell(c) {
 }
 function passedRow(c) {
     const score = c.runs.length > 1 ? ` (${c.passed}/${c.runs.length})` : "";
-    const cells = [`${caseLabel(c)}${score}`, oneLine(c.query), loaded(c)];
+    const cells = [`${caseLabel(c)}${score}`, oneLine(c.query), `${loaded(c)}${c.change === "fixed" ? " · fixed" : ""}`];
     return `| ${cells.map(esc).join(" | ")} |`;
 }
+function baselineText(summary) {
+    const parts = [
+        summary.regressed > 0 ? `**${summary.regressed} regressed**` : null,
+        summary.fixed > 0 ? `${summary.fixed} fixed` : null,
+        summary.new > 0 ? `${summary.new} new` : null,
+        summary.removed > 0 ? `${summary.removed} removed` : null,
+    ].filter((p) => p !== null);
+    return parts.length > 0 ? `${parts.join(", ")}.` : "no changes.";
+}
 function caseLabel(c) {
     return `#${c.id ?? c.index}`;
 }

package/dist/report.js

@@ -72,6 +72,16 @@ export class Reporter {
         const skippedPart = skipped > 0 ? `, ${skipped} skipped` : "";
         const costs = [...verdicts.map((v) => v.costUsd), ...(extra?.skippedRunCosts ?? [])];
         this.write(`${failed} failed${skippedPart} of ${results.length + skipped} · runs ${costs.length} · ${costLine(costs, extra?.estimatedUsd)}\n`);
+        if (extra?.baseline) {
+            const b = extra.baseline;
+            const parts = [
+                b.summary.regressed > 0 ? this.paint("31", `${b.summary.regressed} regressed (${b.regressed.join(", ")})`) : null,
+                b.summary.fixed > 0 ? this.paint("32", `${b.summary.fixed} fixed (${b.fixed.join(", ")})`) : null,
+                b.summary.new > 0 ? `${b.summary.new} new` : null,
+                b.summary.removed > 0 ? `${b.summary.removed} removed` : null,
+            ].filter((p) => p !== null);
+            this.write(`vs baseline: ${parts.length > 0 ? parts.join(", ") : "no changes"}\n`);
+        }
     }
     paint(code, text) {
         return this.useColor ? `\u001b[${code}m${text}${RESET}` : text;

package/dist/results.js

@@ -1,9 +1,12 @@
+import { compare } from "./baseline.js";
 import { readVersion } from "./version.js";
 /** The single source of truth for --json and --junit; computed once per run. */
 export function buildReport(input) {
     const verdicts = input.cases.flatMap((e) => e.result?.runs ?? []);
     const costs = [...verdicts.map((v) => v.costUsd), ...(input.skippedRunCosts ?? [])];
     const known = costs.filter((c) => c !== null);
+    const cases = input.cases.map(toCaseReport);
+    const comparison = input.baseline ? compare(cases, input.baseline, input.baselineFile ?? input.baseline.file, input.suiteCases ?? cases) : null;
     return {
         tool: "skillcheck",
         version: readVersion(),
@@ -27,7 +30,8 @@ export function buildReport(input) {
         },
         unavailable: input.unavailable,
         confusion: input.confusion,
-        cases: input.cases.map(toCaseReport),
+        cases: comparison ? cases.map((c, i) => ({ ...c, change: comparison.changes[i] ?? null })) : cases,
+        baseline: comparison?.summary ?? null,
     };
 }
 function toCaseReport(e) {
@@ -46,5 +50,6 @@ function toCaseReport(e) {
         passed: e.result?.passed ?? 0,
         threshold: e.result?.threshold ?? e.threshold,
         runs: e.result?.runs ?? [],
+        change: null,
     };
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@icntswm/skillcheck",
-  "version": "1.2.0",
+  "version": "1.3.0",
   "description": "Regression tests for Claude Code skills: check that every request still loads the right skill",
   "type": "module",
   "bin": {