skilldrop-cli 1.4.0 → 1.6.0

package/README.md

@@ -25,12 +25,12 @@ Both blind judges picked the skill's answer: *"A keeps each comment to a single
 
 ## The receipts
 
-Every skill runs as a real agent session twice: once with the skill, once without. The results are [published either way](https://sananthanarayan.github.io/skilldrop/evals/). In the last full run (23 October 2026: 117 skills, 520 evals, `claude-sonnet-5`):
+Every skill runs as a real agent session twice: once with the skill, once without. The results are [published either way](https://sananthanarayan.github.io/skilldrop/evals/). In the latest run (5 November 2026: 111 skills, 300 held-out evals written blind from each skill's description, `claude-sonnet-5-5`, scripts free to run):
 
-- **With a skill, answers met 73% of their checks; without one, 53%.** The checks were written by each skill's author, so the fairer numbers are the next ones.
-- **Two blind judges preferred the skill's answer in 62% and 61% of pairs.** Each judge saw only the request and both answers, in shuffled order. On the 176 held-out evals that no revision or admission was based on, it was 49% and 48%: no preference.
-- **On Sonnet 5.5, now the standard tier, the judges preferred the plain agent.** On all 308 held-out evals the skill won 40% and 43% of pairs ([what the benchmark showed](guides/explanation/what-the-benchmark-showed.md#standard-on-sonnet-55)).
-- **4 skills that didn't beat the plain agent were [retired](https://sananthanarayan.github.io/skilldrop/evals/#retired)**, and they're listed. Each new skill has to win on held-out evals before it ships.
+- **With a skill, answers met 84% of their checks; without one, 76%.**
+- **Two blind judges preferred the skill's answer in 44% and 42% of pairs.** Each judge saw only the request and both answers, in shuffled order. Skills that ship a script came out slightly ahead of the plain agent (52%); skills that are instructions alone lost about two pairs in three ([what the benchmark showed](guides/explanation/what-the-benchmark-showed.md#a-new-baseline-with-scripts-free-to-run)).
+- **Loops are measured against their own skills, used without the loop.** Three are clearly ahead of both their skills and the plain agent: [`release`](packs/dev-team/loops/release/LOOP.md), [`modernize`](packs/dev-team/loops/modernize/LOOP.md) and [`ship-agent`](packs/ai-engineering/loops/ship-agent/LOOP.md). Start there: `npx skilldrop-cli install --loop release`. Four loops that lost to their own skills were retired ([RFC-0074](docs/rfcs/0074-loops-measured-four-retired.md)).
+- **Ten skills and four loops that didn't beat the alternative were [retired](https://sananthanarayan.github.io/skilldrop/evals/#retired)**, and they're listed. Each new skill has to win on held-out evals before it ships.
 
 [Get started](https://sananthanarayan.github.io/skilldrop/start/) · [Browse the packs](https://sananthanarayan.github.io/skilldrop/packs/) · [Search all skills](https://sananthanarayan.github.io/skilldrop/catalogue/) · [Docs](https://sananthanarayan.github.io/skilldrop/docs/) · [What stays stable in 1.x](guides/reference/versioning-and-stability.md) · [Contribute](CONTRIBUTING.md)
 
@@ -51,7 +51,7 @@ A good result runs your lint, typecheck and tests through a gate script, then a
 In Claude Code you can use the plugin marketplace instead: run `/plugin marketplace add sananthanarayan/skilldrop`, then `/plugin install dev-team@skilldrop`. [Every install route](guides/how-to/install.md) covers single skills, other IDEs, loops, reviewer subagents, profiles and updates.
 
 <details>
-<summary><b>Choose your role:</b> 19 packs, from product management and architecture to SRE, GRC and job applications</summary>
+<summary><b>Choose your role:</b> 20 packs, from product management and architecture to SRE, GRC, job applications and building design</summary>
 
 Install one pack. Every role pack also brings `core` (intake, critique, review council, output hygiene, and the decisions behind a codebase). Each pack page has a starter prompt to paste and says what a good result looks like.
 
@@ -73,6 +73,7 @@ Install one pack. Every role pack also brings `core` (intake, critique, review c
 - **Frontend:** `frontend` — what makes a page slow for real users, what makes a bundle heavy, and how each route should be rendered. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/frontend/)
 - **Writing agent skills:** `skill-engineering` — author a portable skill with evals, and review one before it ships. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/skill-engineering/)
 - **Applying for a job:** `career` — an honest fit check against the posting, a tailored resume and cover letter that claim nothing you didn't, and the application form's answers. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/career/)
+- **Designing buildings:** `civil-architect` — a dimensioned concept floor plan from a room list and a footprint, checked for overlapping rooms, doors on no wall and rooms no one can reach. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/civil-architect/)
 - **Everyone:** `core` on its own. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/core/)
 
 </details>
@@ -80,7 +81,7 @@ Install one pack. Every role pack also brings `core` (intake, critique, review c
 ## How it works
 
 - **A skill produces one file you own.** Each ships a quality bar, named anti-patterns and acceptance evals, so the output is an artifact, not a conversation. See [every skill, by category](guides/reference/skill-catalogue.md).
-- **Loops order skills, and a gate decides when work moves on.** Five loops cover the lifecycle (`discover`, `design`, `build`, `release`, `operate`), plus the `ship-a-draft` wrapper for any document and loops for jobs beside it, such as `research`, `comply`, `adopt` for a team starting with AI, `ship-agent` for an AI agent and `provision` for a Terraform change. Each gate is a script, a review panel or a person, chosen by how expensive the mistake is to undo. See [the operating model](guides/explanation/operating-model.md) for every stage and gate on one page, [the loops, stage by stage](guides/reference/loops.md), and [why loops](guides/explanation/loops.md).
+- **Loops order skills, and a gate decides when work moves on.** Loops cover three lifecycle stages (`design`, `build`, `release`), plus the `ship-a-draft` wrapper for any document and loops for jobs beside it, such as `research`, `comply`, `ship-agent` for an AI agent and `provision` for a Terraform change. Every loop is measured against its own skills used without it; four that lost were retired. Each gate is a script, a review panel or a person, chosen by how expensive the mistake is to undo. See [the operating model](guides/explanation/operating-model.md) for every stage and gate on one page, [the loops, stage by stage](guides/reference/loops.md), and [why loops](guides/explanation/loops.md).
 - **Measured, not asserted.** `run_bench.py` runs each acceptance eval with the skill and without it, in a sandbox, and reports the lift, a blind preference from two judges, and the cost per run. The results are published whether or not they flatter the skills. See [how skills are checked](https://sananthanarayan.github.io/skilldrop/evals/) and [RFC-0040](docs/rfcs/0040-skill-benchmark.md).
 - **Copy, never transform.** A skill is a plain `SKILL.md` folder in the [Agent Skills](https://agentskills.io) format, at `packs/<pack>/skills/<name>/`. What runs in your agent is byte-identical to what is reviewed here. There is no runtime, and the tooling has zero dependencies.
 - **Updates keep your edits.** `npx skilldrop-cli update` replaces files you haven't touched. For any file you changed, it leaves the new version beside it as `<file>.upstream`. See [upgrade installed skills](guides/how-to/upgrade-skills.md).

package/bin/skilldrop.js

@@ -264,11 +264,16 @@ function readPacksCatalog(dir, source) {
 }
 
 /* RFC-0046: a skill that left the catalogue keeps a record in catalogue.json `retired`, so an
-   installed copy can say why it stopped updating instead of looking like an unreachable source. */
-function retiredNote(cat, s) {
+   installed copy can say why it stopped updating instead of looking like an unreachable source.
+   RFC-0074: a loop can be retired too; its record says "kind": "loop". `isLoop` picks the kind
+   the caller is asking about; left out, a record of either kind answers (search). */
+function retiredNote(cat, s, isLoop) {
   const r = cat && cat.retired && cat.retired[s];
   if (!r) return null;
-  return `'${s}' was retired in ${r.in}: ${r.why}` + (r.instead ? ` Use ${r.instead} instead.` : "");
+  const kind = r.kind || "skill";
+  if (isLoop !== undefined && kind !== (isLoop ? "loop" : "skill")) return null;
+  const what = kind === "loop" ? `The loop '${s}'` : `'${s}'`;
+  return `${what} was retired in ${r.in}: ${r.why}` + (r.instead ? ` Use ${r.instead} instead.` : "");
 }
 
 /* Minimal TOML read — the [pack] table's basic-string scalars (name, version, description).
@@ -1225,7 +1230,7 @@ function expandNames(args, cat) {
     return packMembers(ps, args.flags.pack, "skills");
   }
   if (!args._.length) die("nothing to install — pass skill names, --pack <name>, or --all");
-  for (const s of args._) if (!skillExists(cat, s)) die(retiredNote(cat, s) || `unknown skill '${s}' in catalog '${cat.source}'`);
+  for (const s of args._) if (!skillExists(cat, s)) die(retiredNote(cat, s, false) || `unknown skill '${s}' in catalog '${cat.source}'`);
   return args._.slice();
 }
 
@@ -1567,7 +1572,7 @@ function installedRows(flags) {
     const loop = isLoopEntry(entry);
     try {
       cat = resolveCatalog(src === BUNDLED ? undefined : src);
-      if (!loop && !unitExists(cat, s, loop)) retired = retiredNote(cat, s);
+      if (!unitExists(cat, s, loop)) retired = retiredNote(cat, s, loop);
       if (unitExists(cat, s, loop)) {
         current = unitMeta(cat, s, loop).version || null;
         // Same version, different files: the catalog changed the skill without bumping it.
@@ -1711,7 +1716,7 @@ function outdated(args) {
     console.log(`\n${drift.length} skill(s) changed upstream with the same version. Read the change first (skilldrop diff <skill>),` +
                 `\nthen take it with: skilldrop update --changed`);
   if (gone.length)
-    console.log(`\n${gone.length} retired skill(s) will not update. Keep them as they are, or remove them: skilldrop uninstall ${gone.map((r) => r.s).join(" ")}${targetHint(args.flags)}`);
+    console.log(`\n${gone.length} retired skill(s) or loop(s) will not update. Keep them as they are, or remove them: skilldrop uninstall ${gone.map((r) => r.s).join(" ")}${targetHint(args.flags)}`);
   console.log(stale.length ? `\n${stale.length} outdated — run: skilldrop update` : (drift.length || gone.length ? "" : "everything up to date."));
 }
 
@@ -1767,7 +1772,7 @@ function diff(args) {
   const loop = isLoopEntry(entry);
   const src = args.flags.from || (entry ? lsrc(entry) : BUNDLED);
   const cat = resolveCatalog(src === BUNDLED ? undefined : src);
-  if (!unitExists(cat, s, loop)) die(loop ? `'${s}' is no longer in catalog '${cat.source}'` : (retiredNote(cat, s) || `'${s}' is no longer in catalog '${cat.source}'`));
+  if (!unitExists(cat, s, loop)) die(retiredNote(cat, s, loop) || `'${s}' is no longer in catalog '${cat.source}'`);
   const theirs = unitHashes(cat, s, loop), mine = fileHashes(path.join(dest, s));
   const base = (entry && entry.files) || {};
   const changed = Object.keys(theirs).filter((f) => f in mine && mine[f] !== theirs[f]).sort();
@@ -1811,7 +1816,7 @@ function doctor(args) {
   else try { data = JSON.parse(raw); } catch (e) { add("error", `${lp} is not valid JSON`, "fix or delete it; skilldrop can't track updates until then"); }
   const names = Object.keys(data);
   for (const n of names) {
-    const note = !isLoopEntry(data[n]) && lsrc(data[n]) === BUNDLED && retiredNote(resolveCatalog(), n);
+    const note = lsrc(data[n]) === BUNDLED && retiredNote(resolveCatalog(), n, isLoopEntry(data[n]));
     if (note) add("warn", note, `skilldrop uninstall ${n}${targetHint(args.flags)}, or keep it as it is; it will not update`);
     if (!fs.existsSync(path.join(dest, n))) add("error", `${n} is in the ledger but its folder is gone`, `skilldrop install ${n} (or uninstall ${n} to drop the record)`);
     else if (!data[n].file) {
@@ -2293,7 +2298,7 @@ function installLoops(args) {
   }
   if (!names.length && args.flags.all) names = available.slice();
   if (!names.length) die("nothing to install — pass loop names, --pack <name>, or --all");
-  for (const n of names) if (!available.includes(n)) die(`unknown loop '${n}' in catalog '${cat.source}'`);
+  for (const n of names) if (!available.includes(n)) die(retiredNote(cat, n, true) || `unknown loop '${n}' in catalog '${cat.source}'`);
 
   let bad = 0;
   for (const n of names) {
@@ -2569,7 +2574,7 @@ function info(args) {
   if (args.flags.pack) return infoPack(args);
   const cat = resolveCatalog(args.flags.from);
   const s = args._[0] || die("pass a skill name, or --pack <name>");
-  if (!skillExists(cat, s)) die(retiredNote(cat, s) || `unknown skill '${s}' in catalog '${cat.source}'`);
+  if (!skillExists(cat, s)) die(retiredNote(cat, s, false) || `unknown skill '${s}' in catalog '${cat.source}'`);
   const m = manifestOf(cat, s);
   const ps = packsOf(cat) || {};
   const inPacks = Object.entries(ps).filter(([, p]) => p.skills.includes(s)).map(([n]) => n);

package/catalogue.json

@@ -1,7 +1,7 @@
 {
   "$comment": "RFC-0034. The catalogue index: pack display order and the outcome browse axis (RFC-0026). Pack metadata lives in packs/<pack>/pack.json; membership is the folders — a skill is in a pack because it sits at packs/<pack>/skills/<name>/.",
   "version": "1.0.0",
-  "packs": ["core", "solution-architect", "product-manager", "dev-team", "sre-oncall", "stakeholder-comms", "design", "ai-engineering", "api-builder", "converters", "experience-design", "research", "trackers", "data-analytics", "grc", "infra-as-code", "frontend", "skill-engineering", "career"],
+  "packs": ["core", "solution-architect", "product-manager", "dev-team", "sre-oncall", "stakeholder-comms", "design", "ai-engineering", "api-builder", "converters", "experience-design", "research", "trackers", "data-analytics", "grc", "infra-as-code", "frontend", "skill-engineering", "career", "civil-architect"],
   "outcomes": {
     "decide-what-to-build": {
       "for": "Product managers · founders · strategists",
@@ -205,6 +205,13 @@
         "cover-letter",
         "application-form-answers"
       ]
+    },
+    "design-a-building": {
+      "for": "Architects · building designers · owners planning a build",
+      "description": "Go from a room list and a footprint to a first floor plan: every room at its size, doors on real walls, every room reachable, drawn to scale and checked before anyone sees it.",
+      "skills": [
+        "floor-plan-layout"
+      ]
     }
   },
   "retired": {
@@ -217,6 +224,10 @@
     "marketing-flyer": { "in": "1.2.0", "why": "On its held-out evals, in each of two trials and with both blind judges, the agent without it built the whole flyer with placeholders where the skill stopped or used a generic template; it adds no rule the agent breaks." },
     "deck-builder": { "in": "1.2.0", "why": "On its held-out evals the agent without it built the better deck in each of two trials; rebuilt rule-first, with its script able to run, it still lost on fresh evals (judges 2 and 1 of 4; 69% of checks against 81%).", "instead": "md-to-pptx" },
     "ai-usage-report": { "in": "1.4.0", "why": "On its held-out evals, over three trials and with both blind judges, the agent without it made the better report: it kept every rule the skill teaches (counts show activity, not quality; missing data is not zero; no leaderboard), while the skill refused to rank when asked, hid names the manager had supplied, and its script read neither eval's export." },
-    "lean-canvas": { "in": "1.4.0", "why": "On its held-out evals, in each of three trials and with both blind judges, the agent without it made the better canvas: it caught the same errors and went further, while the skill's lead on checks came from its own format and its script's break-even left out the cost of replacing lost customers.", "instead": "business-case" }
+    "lean-canvas": { "in": "1.4.0", "why": "On its held-out evals, in each of three trials and with both blind judges, the agent without it made the better canvas: it caught the same errors and went further, while the skill's lead on checks came from its own format and its script's break-even left out the cost of replacing lost customers.", "instead": "business-case" },
+    "adopt": { "in": "1.5.0", "kind": "loop", "why": "On its held-out evals its own four skills, installed without the loop, were preferred in every pair by both blind judges: the loop narrowed the policy to the first use case when the team had no rules at all, its gate swapped out the use case the sponsor had chosen, and its reply carried gate jargon. Install ai-use-case-triage, ai-readiness-assessment, ai-usage-policy and ai-adoption-rollout and run them in the pack's order." },
+    "apply": { "in": "1.5.0", "kind": "loop", "why": "On its held-out evals its own skills, installed without the loop, were preferred in every pair by both blind judges: its fit gate stopped before writing anything even when the candidate had already decided to apply, and its confirm gate left form answers blank. Use job-fit-analysis to decide, then resume-tailor, cover-letter and application-form-answers." },
+    "discover": { "in": "1.5.0", "kind": "loop", "why": "On its held-out evals the plain agent was preferred in every pair and its own skills in three of four: its ratification gate refused to record an approval the requester had already given, and its reply carried gate jargon. Use feedback-synthesis on raw signal, then prd-draft." },
+    "operate": { "in": "1.5.0", "kind": "loop", "why": "On its held-out evals its own skills, installed without the loop, were preferred in every pair by both blind judges: the skills already link instrumentation, runbook, comms and postmortem, and the loop added gate jargon and broke its own one rule. Use observability-plan, runbook-generator, incident-comms and postmortem-generator in that order." }
   }
 }

package/contracts/catalogue.schema.json

@@ -31,13 +31,14 @@
     },
     "retired": {
       "type": "object",
-      "description": "RFC-0046. Skills removed from the catalogue, so an installed copy can say why it stopped updating. A retired name has no folder and nothing references it; validate.py checks both.",
+      "description": "RFC-0046. Skills (and, RFC-0074, loops) removed from the catalogue, so an installed copy can say why it stopped updating. A retired name has no folder and nothing references it; validate.py checks both.",
       "additionalProperties": {
         "type": "object",
         "additionalProperties": false,
         "required": ["in", "why"],
         "properties": {
           "in": { "type": "string", "pattern": "^\\d+\\.\\d+\\.\\d+$", "description": "The skilldrop-cli version that removed it." },
+          "kind": { "enum": ["skill", "loop"], "description": "RFC-0074. What was retired; a record without it is a skill." },
           "why": { "type": "string", "description": "One sentence, shown by outdated, update, doctor, diff, info and install." },
           "instead": { "type": "string", "description": "A skill that covers the same job, when one does. Omit it when the agent without a skill does as well." }
         }

package/model-routing.json

@@ -183,8 +183,8 @@
       "rationale": "Extraction + tagging [explicit]/[implied]/[inferred]. The tagging needs care — above light, below heavy."
     },
     "bug-triage": {
-      "tier": "standard",
-      "rationale": "Extraction plus judgment calls (severity vs priority, hypothesis ranking) — same shape as brief-intake: above light, below heavy."
+      "tier": "heavy",
+      "rationale": "Measured (RFC-0073): on its held-out evals the heavy tier met 92% and 92% of checks in two tier trials, against standard's 76% and light's 70% and 86%, more than 5 points ahead both times; moved once its examples/ oracle shipped (RFC-0074)."
     },
     "bundle-size-review": {
       "tier": "standard",
@@ -287,6 +287,10 @@
       "tier": "light",
       "rationale": "A deterministic stdlib script does the extraction; the model runs it, relays the dropped-content report and hands off. No generation beyond a short summary of losses."
     },
+    "floor-plan-layout": {
+      "tier": "heavy",
+      "rationale": "Spatial layout is design judgment: zoning, adjacencies, daylight and circulation traded against areas, with a checker to iterate against."
+    },
     "guide-builder": {
       "tier": "standard",
       "rationale": "Shapes notes into a guide without adding to them. Generation, not adversarial review."
@@ -372,8 +376,8 @@
       "rationale": "Measured (RFC-0073): on its held-out evals the light tier met 85% and 85% of checks in two tier trials, against standard's 85% and heavy's 77% and 85%, within 5 points of the best both times."
     },
     "prd-draft": {
-      "tier": "standard",
-      "rationale": "Synthesizes a structured requirements doc from a brief. The solution-language discipline needs care; bounded by the template. Escalate via ambiguous-input on contested scope."
+      "tier": "heavy",
+      "rationale": "Measured (RFC-0073): on its held-out evals the heavy tier met 85% and 76% of checks in two tier trials, against standard's 68% and light's 68% and 68%, more than 5 points ahead both times; moved once its examples/ oracle shipped (RFC-0074)."
     },
     "pre-merge-review": {
       "tier": "light",
@@ -384,8 +388,8 @@
       "rationale": "Synthesizes a press release and FAQs from a product concept against a fixed format. The specificity discipline needs care but the template bounds it; escalate via ambiguous-input when the concept itself is contested."
     },
     "prompt-caching-advisor": {
-      "tier": "standard",
-      "rationale": "Structural analysis of prompt with threshold arithmetic and pattern matching — above light, no need for heavy."
+      "tier": "heavy",
+      "rationale": "Measured (RFC-0073): on its held-out evals the heavy tier met 90% and 90% of checks in two tier trials, against standard's 63% and light's 83% and 73%, more than 5 points ahead both times; moved once its examples/ oracle shipped (RFC-0074)."
     },
     "refactor-plan": {
       "tier": "standard",
@@ -480,8 +484,8 @@
       "rationale": "Framework selection plus evidenced multi-factor judgment ending in a ranked, committal verdict. Cheaper models fill cells with generic filler, skip the TOWS crossings, and refuse to take a side. Never downgrade."
     },
     "subagent-design": {
-      "tier": "standard",
-      "rationale": "Design synthesis against a fixed topology catalog and role-card template. The fan-out-or-refuse judgment is bounded by the three-reasons rule; escalate via ambiguous-input on genuinely novel workflow shapes."
+      "tier": "heavy",
+      "rationale": "Measured (RFC-0073): on its held-out evals the heavy tier met 100% and 100% of checks in two tier trials, against standard's 86% and light's 85% and 86%, more than 5 points ahead both times; moved once its examples/ oracle shipped (RFC-0074)."
     },
     "success-metrics": {
       "tier": "standard",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "skilldrop-cli",
-  "version": "1.4.0",
+  "version": "1.6.0",
   "description": "Portable AI-agent skills, measured against the agent without them. ADRs, PRDs, runbooks, decks and reviews, installable into Claude Code, Cursor, Kiro, and any IDE that takes custom instructions. One command per skill or per role-based pack.",
   "author": {
     "name": "Sanjay Ananthanarayan",

package/packs/ai-engineering/skills/judge-audit/SKILL.md

@@ -19,7 +19,10 @@ cases for a skill is `eval-harness-generator`.
 - **Use only what you were given.** Don't add facts, names, numbers, incidents, history, steps or sections the input doesn't contain. What you need and don't have is left out of the artifact and listed once at the end under "To confirm".
 - **Deliver from what you have.** When the request gives you something to work on, state your assumptions in a line and produce the result. When it gives you nothing to work on, ask for it in one or two plain sentences and say what you will do once you have it.
 - **Write for someone who has never heard of this skill.** No skill names, no paths or scripts from this folder, no internal terms, and nothing about how the run was set up. A next step is one plain sentence at the end that describes the work.
-- **For this skill:** The pass mark is the user's, never yours. Judge the numbers against the gate or criterion the user or their files state; when there is none, say what the numbers do to their decision. A threshold the input doesn't contain ("kappa ≥ 0.6", "n ≥ 100") never decides the answer.
+- **For this skill, three more:**
+  - **Every repeated run is its own row, then the flips by item id.** Name each item that changed verdict between runs, and say which of them the experts label positive. A pooled figure over two runs hides the items a decision turns on.
+  - **Answer the decision as the user will run it.** Judge their plan (their panel, their rollout, their threshold), not a stricter version you'd prefer. A swap into a three-person panel is judged as that swap.
+  - **The pass mark is the user's, never yours. ** Judge the numbers against the gate or criterion the user or their files state; when there is none, say what the numbers do to their decision. A threshold the input doesn't contain ("kappa ≥ 0.6", "n ≥ 100") never decides the answer.
 
 1. **Pin the decision, the user's criterion, and the comparison.** Find the decision (rank
    variants, publish a number, gate a release, act on single items) and the criterion the user
@@ -43,8 +46,10 @@ cases for a skill is `eval-harness-generator`.
    Several files are one run each, joined on the item id with columns resolved per file;
    `--gold` reads expert labels from their own file; `--judges` reads one column per judge;
    `--map` renames labels that mean the same; `--col` maps any other column. Given only counts,
-   rebuild one row per item and run it anyway; without a shell, use the formulas in
-   [`reference.md`](reference.md) and show the arithmetic.
+   rebuild one row per item and run it anyway. When the script can't run, for any reason,
+   count by hand from the files with the formulas in [`reference.md`](reference.md), show the
+   arithmetic, and deliver the same answer. The reply never mentions the script, a tool, an
+   approval or how the run was set up.
 
 3. **Quote every number from the output, with its subset, run and n.** All-rows and
    labelled-rows figures are different numbers; so are run 1 and run 2. ✅ *"On the 20 items the
@@ -108,20 +113,28 @@ cases for a skill is `eval-harness-generator`.
     spans it). "Which judge can we use" names one: ✅ *"Switch to the mid tier: it is the cheapest
     judge whose error doesn't move with length (bias shift −2 points against a 2.0-point gate);
     the small tier says relevant to 9 of 10 long passages the reviewer rejected."* — ❌ *"None
-    can be recommended until 100 more labels exist."* Conditions are actions with an n (label 30
-    more long passages); a small sample widens the interval, it doesn't withhold the answer.
+    can be recommended until 100 more labels exist."* Conditions are actions with an n (label more long
+    passages, sized as below); a small sample widens the interval, it doesn't withhold the answer.
     When the judge in use today also reads *can't tell*, the sample can't certify anyone: compare
     the candidate with the incumbent on the same items (the accuracy difference and its interval).
+    **More labels are sized, never stock.** When a condition is "label more", work out how many
+    from the interval: the n at which an interval of the width you observed would stop spanning
+    the user's bar, with the arithmetic in one line. ✅ *"At 4 of 6 caught, the interval is
+    30%–90%; about 40 more violations would narrow it to ±15 points, enough to tell 80% recall
+    from 50%."* — ❌ *"Label ~30 more."* When the file lacks a column the condition needs (a
+    severity, a timestamp), say so instead of sizing it.
     Kappa bands (Landis–Koch "moderate") appear only as labelled context, never as the reason.
     **One evidence standard for every subgroup.** If 30 cases are too few to clear one segment,
     12 are too few to clear another: give each its interval and decide them by the same rule.
-    The headline and the verdict word say the same thing: never "No" above *USE WITH
-    CONDITIONS*.
-    Verdict words: **TRUSTWORTHY FOR THIS DECISION**, **USE WITH CONDITIONS**, **DO NOT RELY ON IT**.
+    The headline and the verdict say the same thing: never "No" above "usable with conditions".
+    Say the verdict in plain words, in sentence case: *trustworthy for this decision*, *usable
+    with conditions* or *don't rely on it*. Never in capitals: a shouted verdict reads as alarm,
+    not evidence.
 
 13. **Write the reply short.** First line: the answer (yes/no, which judge, which arm it
     favours and by how much). Then the two to four numbers behind it, each with its subset and
-    run; what can't be measured; one next step that fits the user's deadline. No kappa tutorial,
+    run; with repeated runs, the per-run rows and the flipped item ids; one line on what the
+    data can't measure; one next step that fits the user's deadline. No kappa tutorial,
     no tables unless asked. A chat question gets a chat answer. Write only the files the user
     asked for, and delete any scratch files you made.
 
@@ -135,7 +148,7 @@ counts`, because a verdict without them is invented.
 
 - [`reference.md`](reference.md) — turning the user's gate into the number that answers it, kappa (unweighted, weighted, Fleiss') and Wilson formulas with worked tables, the both-orders protocol, and every input shape the script reads
 - [`scripts/judge_stats.py`](scripts/judge_stats.py) — per-group, per-run and pooled agreement, weighted kappa, rare-class precision and recall, flips joined on item id, position and length bias, `--gate` (stdlib only; `--self-test` checks the maths)
-- [`examples/summary-panel.md`](examples/summary-panel.md) — worked example: a three-judge panel on summary faithfulness, a USE WITH CONDITIONS verdict
+- [`examples/summary-panel.md`](examples/summary-panel.md) — worked example: a three-judge panel on summary faithfulness, a usable-with-conditions verdict
 
 ## Quality bar
 
@@ -144,7 +157,9 @@ counts`, because a verdict without them is invented.
 - **Statistics are stratified by the compared variable** before any pooled figure is quoted.
 - **Ordinal scores get weighted kappa, named as quadratic or linear.**
 - **A rare class gets its count, confusion, recall and precision with intervals,** not accuracy.
-- **Each run is reported on its own, with the flips,** joined on item id.
+- **Each run is reported on its own, then the flipped items by id,** and which of them are positives.
+- **The decision judged is the user's plan,** not a stricter stand-in.
+- **A "label more" condition carries an n worked out from the interval,** never a stock number.
 - **Every number matches the output for the subset and run it is labelled with,** n included.
 - **What the data cannot measure is named,** never estimated.
 
@@ -171,4 +186,6 @@ counts`, because a verdict without them is invented.
 - ❌ **Accuracy on a rare class.** 96% when 95% is the always-"fine" score.
 - ❌ **A pooled kappa for a comparison,** or unweighted kappa on a 1–5 scale.
 - ❌ **Position bias stated backwards,** or judgements counted as the unit instead of questions.
+- ❌ **"Label about 30 more."** A number with no arithmetic behind it reads as a guess, because it is one.
+- ❌ **Stopping, or narrating, when a tool won't run.** Count by hand and answer; the reader never needs to hear about the tooling.
 - ❌ **Leading with the stats dump.** The reader asked whether to act; tell them, then show why.

package/packs/ai-engineering/skills/judge-audit/evals/evals.json

@@ -160,6 +160,36 @@
         "Points out that 'catch most of the really bad stuff' has no number and asks for one or proposes one, and offers a safer interim option, such as using a judge to flag comments into a human review queue instead of removing them automatically.",
         "Invents nothing: every count, comment ID and rate it cites comes from mod_sample_scored.csv or the prompt, with no made-up model names, prices, comment text or extra data."
       ]
+    },
+    {
+      "id": 12,
+      "held_out": "2026-11-05",
+      "prompt": "We're testing a new system prompt (v2) for our order-support bot against the current one (v1). 120 test conversations, each answered by both prompts, and an LLM judge grades every answer pass/fail. The rule we agreed up front: ship v2 if it beats v1 by at least 5 points. Run 1 came out 83% vs 71%, so a clear yes, but the judge runs at temperature 0.7 so I re-ran it twice more and the numbers moved, and now the team is arguing about it. Rosa from support hand-graded every answer where the three runs didn't agree. Everything is in judge_runs.csv and rosa_labels.csv. Can we ship v2 or not?",
+      "assertions": [
+        "Reports each of the three runs separately for both prompts: v2 vs v1 at 83.3% vs 70.8%, 73.3% vs 72.5% and 75.0% vs 71.7%, so the gap is +12.5, +0.8 and +3.3 points, and only run 1 clears the 5-point rule",
+        "Finds that the flips are lopsided: 24 of 120 v2 answers changed verdict across runs against 8 of 120 v1 answers, so the judge's instability falls mostly on v2",
+        "Finds the planted length effect: 22 of the 24 v2 flips are long answers (over about 350 output tokens). 22 of the 40 long v2 answers flipped, against 2 of the 80 short ones",
+        "Uses Rosa's labels on the flipped answers: only 7 of the 24 v2 flips pass against 4 of the 8 v1 flips, which gives about 87 vs 86 passes (72.5% vs 71.7%, +0.8 points), well short of the 5-point rule",
+        "Does not accept a majority vote as the fix: majority-of-three gives 94 vs 86 (+6.7 points), but Rosa passes only 4 of the 14 v2 flips that the majority calls pass",
+        "Notes that on the answers judged the same in all three runs, v1 passes 82 and v2 passes 80, so v2's apparent lead comes entirely from answers the judge cannot decide on",
+        "Gives a direct answer that v2 does not meet the agreed 5-point rule on this evidence and should not ship on the strength of run 1, and names a concrete next step, such as fixing the judge's instability on long answers or setting temperature to 0 before re-comparing",
+        "Does not invent figures that are not in the files, such as Rosa's verdicts on answers she did not grade or the judge's accuracy on the answers whose verdict never changed"
+      ]
+    },
+    {
+      "id": 13,
+      "held_out": "2026-11-05",
+      "prompt": "We moderate product reviews on our marketplace. Right now judge-L (the big model) scores every review 1-5 for 'should this be hidden', and a person checks anything that scores 4 or 5. The plan for next sprint is to move the Spanish queue only to judge-S, which costs about a tenth as much, and to turn on auto-hide for anything judge-S scores 4 or higher, with no person in the loop. Policy signed off on auto-hide as long as it's at least 95% precise on what it hides and catches at least 80% of the reviews that should be hidden. I compared the two judges on our labelled set and they agree on hide vs keep 97.5% of the time, so I think we're fine? The file is moderation_eval.csv, and expert_label comes from the trust and safety team. Can we go ahead with the plan?",
+      "assertions": [
+        "Tests the plan as stated (judge-S, Spanish rows only, auto-hide at a score of 4 or more) against the trust and safety labels: 14 of the 20 Spanish reviews it would hide should be hidden (70% precision), and it catches 14 of the 20 that should be hidden (70% recall), so it fails both the 95% bar and the 80% bar",
+        "Explains why the 97.5% figure misleads: it is S-vs-L agreement pooled over all 600 reviews, two-thirds of them English and nine-tenths of them keep. On Spanish alone the two judges agree 95.0% of the time, with Cohen's kappa of about 0.71",
+        "Reports chance-corrected agreement with the expert labels on Spanish: judge-S's 94.0% raw agreement is a kappa of about 0.67, against about 0.94 for judge-L",
+        "Checks whether a different cut-off saves judge-S on Spanish and finds none that does: a score of 5 or more gives 88.9% precision (8 of 9) but 40% recall, and 3 or more gives 90% recall but 48.6% precision (18 of 37)",
+        "Shows that the problem is specific to Spanish: on English, judge-S at 4 or more reaches 97.4% precision (38 of 39) and 95% recall (38 of 40)",
+        "Reports judge-L on Spanish at 18 of 18 precision and 90% recall (18 of 20), meeting both bars on the point estimates. It also notes that with only 20 Spanish reviews that should be hidden, the 95% confidence interval on that precision has a lower bound of about 81-82%, so even judge-L is not shown to meet 95% with confidence",
+        "Gives a direct answer: do not move the Spanish queue to judge-S with auto-hide. It names a workable alternative, such as keeping judge-L with human review, or using judge-S only to route reviews to people, together with labelling more Spanish reviews that should be hidden",
+        "Does not invent figures that are not in the file, such as judge prices beyond 'about a tenth', Spanish queue volumes, or the text of individual reviews"
+      ]
     }
   ]
 }