skilldrop-cli 0.16.36 → 0.16.37

package/README.md

@@ -32,7 +32,7 @@ Install one pack. Every role pack also brings `core` (intake, critique, review c
 - **Data and analytics:** `data-analytics` — metric definitions, SQL review, dashboard specs. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/data-analytics/)
 - **Governance, risk and compliance:** `grc` — DPIAs, SOC 2 evidence maps, risk registers, security questionnaire answers. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/grc/)
 - **Infrastructure as code:** `infra-as-code` — Terraform modules with secure defaults, and plan reviews before apply. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/infra-as-code/)
-- **Frontend:** `frontend` — what makes a page slow for real users, and how each route should be rendered. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/frontend/)
+- **Frontend:** `frontend` — what makes a page slow for real users, what makes a bundle heavy, and how each route should be rendered. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/frontend/)
 - **Writing agent skills:** `skill-engineering` — author a portable skill with evals, and review one before it ships. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/skill-engineering/)
 - **Applying for a job:** `career` — an honest fit check against the posting, a tailored resume and cover letter that claim nothing you didn't, and the application form's answers. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/career/)
 - **Everyone:** `core` on its own. [Open pack](https://sananthanarayan.github.io/skilldrop/packs/core/)

package/catalogue.json

@@ -70,7 +70,9 @@
         "refactor-plan",
         "behaviour-equivalence",
         "rendering-strategy",
-        "web-performance-review"
+        "web-performance-review",
+        "bundle-size-review",
+        "judge-audit"
       ]
     },
     "build-on-the-claude-api": {

package/model-routing.json

@@ -190,6 +190,10 @@
       "tier": "standard",
       "rationale": "Extraction plus judgment calls (severity vs priority, hypothesis ranking) — same shape as brief-intake: above light, below heavy."
     },
+    "bundle-size-review": {
+      "tier": "standard",
+      "rationale": "Sizes, duplicates and what pulls each module in are computed from the bundler's stats file by its script; the judgment is which removal is safe."
+    },
     "business-case": {
       "tier": "heavy",
       "rationale": "Weighted multi-option judgment with a committal recommendation — cheaper models mis-weight costs, inflate benefits, and miss the flip-assumption. Never downgrade."
@@ -314,6 +318,10 @@
       "tier": "standard",
       "rationale": "Matches a posting's requirements to resume lines and reports the gaps. The rule is mechanical: no line in the resume, no credit."
     },
+    "judge-audit": {
+      "tier": "standard",
+      "rationale": "The agreement and stability statistics are computed by its script; the model's work is reading them against the decision the judge serves."
+    },
     "launch-readiness": {
       "tier": "standard",
       "rationale": "Checks evidence against a fixed checklist. Bounded synthesis, not adversarial review."

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "skilldrop-cli",
-  "version": "0.16.36",
+  "version": "0.16.37",
   "description": "Portable AI-agent skills, measured against the agent without them. ADRs, PRDs, runbooks, decks and reviews, installable into Claude Code, Cursor, Kiro, and any IDE that takes custom instructions. One command per skill or per role-based pack.",
   "author": {
     "name": "Sanjay Ananthanarayan",

package/packs/ai-engineering/skills/judge-audit/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: judge-audit
+description: Audit whether an LLM-as-judge, or a panel of judges, can be trusted for the decision its scores drive, before anyone acts on them — above all whether the judge errs differently on the variants being compared (prompt v1 vs v2, model A vs B, short vs long inputs), plus chance-corrected and weighted agreement with expert labels, precision and recall on a rare class, each run reported separately with the items that flipped, position and length bias, and confidence intervals, checked against the user's own gate or criterion and ending in a direct answer (ship or not, which judge to use). Use when the user asks "can we trust our LLM judge", "is our grader reliable enough to gate releases", "the grader says v2 is better, can we ship it", "which is the cheapest judge we can switch to", "our two judges agree 90%, is that good", "can we turn on auto-remove", "check our judge for position bias", or has a file of judge scores and wants to know whether to act on them.
+---
+
+# judge-audit
+
+Decides whether a judge's scores are good enough for one named decision, from the judgements
+themselves, and answers the question the user asked. A grader can match an expert well overall
+and still over-score the very variant it is being used to pick. This skill audits judgements that
+already exist; designing the eval set and grading plan is `llm-eval-harness`, generating eval
+cases for a skill is `eval-harness-generator`, and measuring AI tool usage is `ai-usage-report`.
+
+## How to respond
+
+**Four rules come before the steps and outrank them:**
+
+- **Answer what was asked, first.** Open with the answer, the decision or the artifact, in plain words. Scores, matrices, frameworks and tags come after it, and anything that doesn't change the answer is cut.
+- **Use only what you were given.** Don't add facts, names, numbers, incidents, history, steps or sections the input doesn't contain. What you need and don't have is left out of the artifact and listed once at the end under "To confirm".
+- **Deliver from what you have.** When the request gives you something to work on, state your assumptions in a line and produce the result. When it gives you nothing to work on, ask for it in one or two plain sentences and say what you will do once you have it.
+- **Write for someone who has never heard of this skill.** No skill names, no paths or scripts from this folder, no internal terms, and nothing about how the run was set up. A next step is one plain sentence at the end that describes the work.
+- **For this skill:** The pass mark is the user's, never yours. Judge the numbers against the gate or criterion the user or their files state; when there is none, say what the numbers do to their decision. A threshold the input doesn't contain ("kappa ≥ 0.6", "n ≥ 100") never decides the answer.
+
+1. **Pin the decision, the user's criterion, and the comparison.** Find the decision (rank
+   variants, publish a number, gate a release, act on single items) and the criterion the user
+   or their files give: ✅ *"Gate: a build ships if its precision@5 is within 2.0 points of
+   production; the change compares short against long documents."* Read every file for it
+   (READMEs, run notes) before asking. No criterion stated: no pass mark, and the answer says
+   what the errors would do to the decision. Name the variable the decision compares: version,
+   model, arm, or a property the planned change shifts (length, language).
+
+2. **Compute, don't estimate.** Run the script, grouped by the compared variable, with the
+   user's criterion as `--gate`:
+
+   ```bash
+   # Claude Code
+   python3 "${CLAUDE_SKILL_DIR}/scripts/judge_stats.py" grades.csv --group-by prompt_version
+   python3 "${CLAUDE_SKILL_DIR}/scripts/judge_stats.py" run_1.jsonl run_2.jsonl --gold gold.csv --map true=violation,false=ok --positive violation
+   # Other IDEs (from the skill folder)
+   python3 scripts/judge_stats.py wide.csv --judges big,small --positive relevant --group-by "tokens>=400" --gate "bias_diff<=2.0"
+   ```
+
+   Several files are one run each, joined on the item id with columns resolved per file;
+   `--gold` reads expert labels from their own file; `--judges` reads one column per judge;
+   `--map` renames labels that mean the same; `--col` maps any other column. Given only counts,
+   rebuild one row per item and run it anyway; without a shell, use the formulas in
+   [`reference.md`](reference.md) and show the arithmetic.
+
+3. **Quote every number from the output, with its subset, run and n.** All-rows and
+   labelled-rows figures are different numbers; so are run 1 and run 2. ✅ *"On the 20 items the
+   reviewer scored, the grader averages 3.7 against her 3.1."* — ❌ a run-1 false-positive count
+   presented as the result when run 2 has more. No figure appears that the output doesn't print.
+
+4. **Use the scale and the metric the user uses.** For 1–5 scores report quadratic-weighted
+   kappa (say "quadratic"), exact agreement and mean bias; unweighted kappa treats 4-vs-5 like
+   1-vs-5. Pass `--threshold` only when the user stated a pass score. When the user quotes a
+   figure ("kappa 0.8", "they agree 80%"), reproduce it first, say which statistic it is on which
+   rows, then show what it hides.
+
+5. **Lead with the differential.** Per variant compare exact agreement, bias judge − expert
+   with its interval, how often the judge scores higher and lower, and the bias-vs-length
+   correlation; then the judge's v2 − v1 gain against the expert's own on the labelled rows. ✅
+   *"It matches the reviewer on 18 of 20 A answers but 9 of 20 B answers and over-scores B by
+   +0.55 [+0.30, +0.80]; her own scores show B ahead by +0.10 (interval spans zero)."* When the
+   bias tracks a property the planned change shifts, the shift in bias is what lands on the gate.
+
+6. **Chance-correct agreement, and on a rare class count the rare class.** Two raters who each
+   say "pass" 90% of the time agree 82% by chance: ✅ *"Raw 88%, chance 80%, kappa 0.40."* With
+   one class under about 15%, lead with how many expert positives exist, the confusion counts,
+   recall and precision with intervals; accuracy only beside the always-majority baseline. ✅
+   *"6 of 120 posts are violations, so always saying 'fine' scores 95%. It caught 4 of 6 (30%–90%)
+   and flagged 2 fine posts."* — ❌ *"96% accurate, so it's reliable."*
+
+7. **With repeated runs, report each run, then the flips.** One row per run: precision, recall,
+   false positives. Then run-against-run agreement joined on the item id, the items that
+   flipped and which of them the expert labels positive, and what was caught in both runs. ✅
+   *"Run 1 wrongly flagged 1 post, run 2 flagged 2; 5 posts changed verdict, 2 of them real
+   violations."* Files in different row orders or formats are joined on id, never by line.
+
+8. **For pairwise judges, read the order column before the win rate.** State the direction from
+   the counts: ✅ *"The first-shown answer won 70 of 100; the new variant went first on 15 of 25
+   questions."* Report the order-balanced win rate per judge; the unit is the question, not the
+   judgement. An interval excluding 50% does not fix an order confound; judging both orders does.
+
+9. **Measure stability and judge-to-judge agreement across every trial.** Flip rate per judge,
+   with the flipped items named; trial-matched kappa between judges, and Fleiss' across every
+   judge × trial rating. Judges sharing a blind spot agree on it: expert agreement ranks first.
+
+10. **Say what the data cannot measure, in one line.** One run: no flip rate. One expert: no
+    expert-expert baseline. No expert labels: nothing shows the judge is right. Never fill a gap
+    with a guess.
+
+11. **Answer the question asked, against the user's criterion.** Each gate check reads
+    *clears* (whole interval meets it), *fails* (none of it does) or *can't tell* (the interval
+    spans it). "Which judge can we use" names one: ✅ *"Switch to the mid tier: it is the cheapest
+    judge whose error doesn't move with length (bias shift −2 points against a 2.0-point gate);
+    the small tier says relevant to 9 of 10 long passages the reviewer rejected."* — ❌ *"None
+    can be recommended until 100 more labels exist."* Conditions are actions with an n (label 30
+    more long passages); a small sample widens the interval, it doesn't withhold the answer.
+    When the judge in use today also reads *can't tell*, the sample can't certify anyone: compare
+    the candidate with the incumbent on the same items (the accuracy difference and its interval).
+    Kappa bands (Landis–Koch "moderate") appear only as labelled context, never as the reason.
+    Verdict words: **TRUSTWORTHY FOR THIS DECISION**, **USE WITH CONDITIONS**, **DO NOT RELY ON IT**.
+
+12. **Write the reply short.** First line: the answer (yes/no, which judge, which arm it
+    favours and by how much). Then the two to four numbers behind it, each with its subset and
+    run; what can't be measured; one next step that fits the user's deadline. No kappa tutorial,
+    no tables unless asked. A chat question gets a chat answer.
+
+**Non-interactive runs** (subagent, CI, headless): no criterion in the input becomes
+`[assumption] no stated gate; the reply says what the errors do to the decision`; a compared
+variable the input does not name becomes `[assumption] no grouping`. No judgements and no counts
+emit `BLOCKED: need the per-item judgements (item, judge, label or score) or the agreement
+counts`, because a verdict without them is invented.
+
+## Useful references in this skill
+
+- [`reference.md`](reference.md) — turning the user's gate into the number that answers it, kappa (unweighted, weighted, Fleiss') and Wilson formulas with worked tables, the both-orders protocol, and every input shape the script reads
+- [`scripts/judge_stats.py`](scripts/judge_stats.py) — per-group, per-run and pooled agreement, weighted kappa, rare-class precision and recall, flips joined on item id, position and length bias, `--gate` (stdlib only; `--self-test` checks the maths)
+- [`examples/summary-panel.md`](examples/summary-panel.md) — worked example: a three-judge panel on summary faithfulness, a USE WITH CONDITIONS verdict
+
+## Quality bar
+
+- **The first line answers the question asked:** ship or not, which judge, which arm it favours.
+- **The criterion is the user's;** with none, no invented pass mark decides anything.
+- **Statistics are stratified by the compared variable** before any pooled figure is quoted.
+- **Ordinal scores get weighted kappa, named as quadratic or linear.**
+- **A rare class gets its count, confusion, recall and precision with intervals,** not accuracy.
+- **Each run is reported on its own, with the flips,** joined on item id.
+- **Every number matches the output for the subset and run it is labelled with,** n included.
+- **What the data cannot measure is named,** never estimated.
+
+## When to use this skill
+
+- ✅ "The grader says v2 beats v1 — can we ship on it?" with a file of scores
+- ✅ "Which is the cheapest judge we can switch to?" with several judges on the same items
+- ✅ "It's 96% accurate on two runs — can we turn on auto-remove?"
+- ✅ A pairwise A/B judged by a model, before shipping the winner
+- ✅ "Our two graders agree 90%, good enough?"
+
+## When NOT to use this skill
+
+- ❌ Designing the eval set, grading methods and pass bar for an LLM feature — `llm-eval-harness`
+- ❌ Generating eval cases for a skill's `evals/` folder — `eval-harness-generator`
+- ❌ Measuring how much teams use AI tools — `ai-usage-report`
+- ❌ Inter-rater reliability for humans only, with no model in the loop — the statistics transfer, but the bias checks here assume a model
+
+## Anti-patterns to avoid
+
+- ❌ **An invented gate.** "Kappa must be ≥ 0.6 with n ≥ 100" when the user's gate is a 1.0-point tolerance.
+- ❌ **Refusing to pick.** "Which is cheapest?" answered with "none yet" instead of a name and a condition.
+- ❌ **Run 1 as the result.** Two runs are two rows, then the flips.
+- ❌ **Joining runs by line number.** Rows in a different order make a line-by-line kappa meaningless.
+- ❌ **Accuracy on a rare class.** 96% when 95% is the always-"fine" score.
+- ❌ **A pooled kappa for a comparison,** or unweighted kappa on a 1–5 scale.
+- ❌ **Position bias stated backwards,** or judgements counted as the unit instead of questions.
+- ❌ **Leading with the stats dump.** The reader asked whether to act; tell them, then show why.

package/packs/ai-engineering/skills/judge-audit/evals/eval_queries.json

@@ -0,0 +1,11 @@
+[
+  { "query": "Our LLM grader and a second model agree 88% on pass/fail. Is that enough to use it as a release gate?", "should_trigger": true },
+  { "query": "Here's a CSV of judge scores with three trials per item and some human labels. Can we trust the judge?", "should_trigger": true },
+  { "query": "Check our pairwise LLM judge for position bias before we pick the winning prompt", "should_trigger": true },
+  { "query": "Compute Cohen's kappa between our model judge and the annotators and tell me whether the judge is usable", "should_trigger": true },
+  { "query": "We want to swap our expensive judge model for a cheaper one. Here are both models' grades on the same 100 answers plus expert labels.", "should_trigger": true },
+  { "query": "We want an LLM judge to grade helpfulness on our chatbot. How should we set that up?", "should_trigger": false },
+  { "query": "Design an eval set and pass bar for our RAG assistant before we change the prompt", "should_trigger": false },
+  { "query": "Generate eval cases for this skilldrop SKILL.md so I can drop them in its evals folder", "should_trigger": false },
+  { "query": "Summarise how much each team used AI coding assistants last quarter from this telemetry export", "should_trigger": false }
+]

package/packs/ai-engineering/skills/judge-audit/evals/evals.json

@@ -0,0 +1,136 @@
+{
+  "skill_name": "judge-audit",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Tomás reckons our LLM judges are reliable enough to gate the reply drafter. Everything is in notes.md and judgements.csv. Can we use judge-b as the release gate from next sprint? Keep it short.",
+      "assertions": [
+        "The first lines say judge-b should not be relied on as the release gate as things stand, and name that gate as the decision",
+        "It does not accept the 91% agreement as evidence: it gives the chance-corrected figure (Cohen's kappa about 0.54, with about 81% agreement expected by chance) instead of raw agreement alone",
+        "It compares the judges with the support lead's 40 labels and reports that judge-b passes 10 of the 12 replies she failed (about 83%), with an interval or a note that 12 failures is a small base",
+        "It translates the gate: on her 40 replies the judges pass about 88% while she passes 70%, so an 85% judge pass rate does not mean 85% of replies are sendable",
+        "It reports judge-b's flip rate between trials (12 of 80, about 15%) against judge-a's (3 of 80) and says this matters for choosing the cheaper judge",
+        "It points out that the two judges share errors (both passed the same long replies R29–R37 that the support lead failed), so their agreement with each other does not validate them",
+        "It notes that passes rise with reply length while the support lead's do not, and recommends judging policy correctness as a separate criterion to gate on instead of one 'good reply' score",
+        "It does not invent figures, names, items or labels that are not in the files"
+      ]
+    },
+    {
+      "id": 2,
+      "prompt": "Growth wants to switch our onboarding emails to prompt v2 because the grader picked it 53 to 43. The setup is in run-notes.md and the raw results in pairwise.csv. Is that solid enough to ship v2 on?",
+      "assertions": [
+        "The first lines say the grader result is not enough to ship v2 on",
+        "It finds the grader picks whichever email is shown first about 70% of the time (67 of 96 decided comparisons)",
+        "It finds that v2 was shown first in 60 of the 96 decided comparisons despite the randomisation, so the raw 55% win rate is inflated by position",
+        "It gives an order-balanced figure of about 50% (v2 wins about 70% when shown first and about 31% when shown second), so the data shows no evidence v2 is better",
+        "It reports that the grader's pick changed between the two runs on about half the profiles (26 of 50)",
+        "It recommends judging each pair in both orders and counting a win only when it holds in both, or an equivalent fix, and says that an interval on the win rate (computed over the 50 profiles, not the 100 judgements) cannot remove the order effect",
+        "It notes there are no human labels, so even a consistent grader has not been shown to pick the better email",
+        "It does not invent numbers, profiles or human judgements that are not in the files"
+      ]
+    },
+    {
+      "id": 3,
+      "prompt": "Quick one. We're piloting two LLM judges to grade our support agents' ticket replies as good or poor, and the grades will feed into quarterly bonus decisions. On 20 replies the two judges agreed on 18: both said good on 16, both said poor on 2, and they split on the other 2 (one each way). That's 90% agreement, so good enough to go ahead? A couple of lines is fine.",
+      "assertions": [
+        "The first line or two say no, not for bonus decisions",
+        "It corrects the 90% for chance: with each judge saying good on 17 of 20, about 75% agreement is expected by chance, giving a kappa of about 0.6",
+        "It says 20 replies is too few to conclude, for example that the uncertainty on the agreement figure runs from near zero to near perfect",
+        "It says the two judges agreeing does not show either is right, and that their grades need comparing with a sample graded by a person such as a team lead or QA reviewer",
+        "It ties the answer to the decision: grades that affect individual people's bonuses need a much higher bar than tracking an overall trend",
+        "It stays short: a few lines or bullets, no long report or tables",
+        "It does not invent results, reply counts or statistics beyond the counts given"
+      ]
+    },
+    {
+      "id": 4,
+      "held_out": "2026-10-20",
+      "prompt": "hey, sanity check before thursday. we moved the support reply prompt to v2 and the grader model says it's way better (avg went ~3.25 -> ~4.0 on the 1-5 scale). Priya hand-scored 32 of them last week and I got a weighted kappa of 0.85 between her and the grader, so I'm thinking we just use the grader as the release gate and ship v2 to all queues. grades.csv is the export. am I good??",
+      "assertions": [
+        "Answers the actual question up front with a clear verdict: the grader should NOT be relied on to gate the v2 rollout as things stand (do-not-rely, or use-with-conditions where the conditions rule out using it alone for this v2-vs-v1 call). It does not endorse shipping on the grader's say-so.",
+        "Shows that the pooled quadratic-weighted kappa (about 0.85 on the 32 labelled rows) hides a split by version: the grader matches Priya exactly on 14 of 16 v1 rows but only 7 of 16 v2 rows, and on v2 it scores higher than her on 9 of 16 rows and lower on none (mean grader-minus-Priya +0.625 on v2 vs 0.00 on v1).",
+        "Spots the length confound: v2 replies are much longer (mean about 201 words vs about 80 for v1), and the grader's over-scoring grows with reply length (e.g. grader-minus-human error correlates positively with reply_words, r about 0.6 on the labelled rows), so the +0.77 average gain may be the grader rewarding length rather than quality.",
+        "Points out that on the human-labelled rows Priya's own scores show no v2 improvement (mean about 3.25 for v2 vs about 3.44 for v1), and notes that with only 16 labels per version this human difference is itself very uncertain (a CI that spans zero), so it neither proves v2 better nor proves it worse.",
+        "Mentions what this data cannot measure: there is a single grader run (no test-retest / flip rate) and a single human labeller (no human-human baseline), and suggests checking these rather than claiming them.",
+        "Gives concrete next steps that fit the Thursday deadline, e.g. hand-label a fresh random sample of v2 replies (especially long ones), re-score with a rubric or prompt that controls for length, or keep a human check on the gate, rather than a generic 'collect more data'.",
+        "Is sized to a quick 'am I good?' sanity check: leads with the verdict and the two or three numbers that drive it, without a long tutorial on kappa theory.",
+        "Invents nothing: every number, name and file it cites comes from grades.csv or the message (no made-up grader model name, extra labellers, columns, or statistics that the data doesn't support)."
+      ]
+    },
+    {
+      "id": 5,
+      "held_out": "2026-10-20",
+      "prompt": "I'm comparing our current RAG pipeline (A) against a new reranker variant (B) on 30 eval questions. Two judge models, orca-judge-large and wren-judge-small, each judged every A/B pair twice (trial 1 and trial 2). Across all 120 judgements B won 73, A won 40, 7 ties, and the two judges agree on roughly two thirds of verdicts. I have to tell the eng review on Friday whether B is a real winner or noise. Raw data is in pairwise_judgements.jsonl. Can you check whether these judges are trustworthy enough to make that call, and whether B's win is real?",
+      "assertions": [
+        "Answers both questions directly: the judges, as this experiment was run, are not trustworthy enough to declare B the winner, and B's win is not established (verdict do-not-rely, or use-with-conditions where the condition is a position-balanced rerun). It does not tell the review that B is a real winner.",
+        "Catches the order imbalance in the first_shown field: B was shown first on 22 of the 30 questions (88 of the 120 judgements), and each question kept the same order in both trials, so B's win rate is confounded with position.",
+        "Quantifies the position bias from the data: the first-shown answer won 90 of 120 judgements (75%; orca-judge-large 50/60, wren-judge-small 40/60), and B won only 6 of 32 judgements when A was shown first versus 67 of 88 when B was shown first.",
+        "Reports chance-corrected agreement between the two judges rather than the raw ~65% (39 of 60 paired verdicts): Cohen's kappa about 0.33 pooled (about 0.38 in trial 1, 0.29 in trial 2), or Fleiss' kappa about 0.32 across all four ratings, and calls that fair or weak agreement, not strong.",
+        "Reports test-retest instability from the two same-order trials: orca-judge-large changed its verdict on 8 of 30 questions and wren-judge-small on 11 of 30.",
+        "Treats the 30 questions, not the 120 judgements, as the unit of analysis (judgements on the same question are not independent), so any confidence interval on B's win rate is clustered or bootstrapped by question, and it explains that even an interval excluding 50% would not fix the position confound.",
+        "Recommends a specific fix: rerun every question in both orders (A-first and B-first), count a win only when it holds in both orders (flips count as ties), and add some expert or human labels, noting that the file has no ground-truth labels so agreement with experts cannot be checked yet.",
+        "Invents nothing: every count, rate and name it cites comes from pairwise_judgements.jsonl or the message; it does not make up expert labels, extra judges, or results of a rerun that hasn't happened."
+      ]
+    },
+    {
+      "id": 6,
+      "held_out": "2026-10-20",
+      "prompt": "We run a nightly relevance eval on our RAG retriever and its labels gate whether retriever changes ship. atlas-xl is the judge today, but at ~20k judgments a night finance wants the bill down. I ran all three Atlas tiers over the same 300 query-passage pairs and Priya hand-labelled 60 of them - everything, including prices and how the gate works, is in rag-judge/. Which is the cheapest judge we can switch to without making the gate meaningless? Dmitri's position is that nano agrees with xl about 80% of the time so nano is fine. Is he right?",
+      "assertions": [
+        "Answers the actual question directly: recommends atlas-m (not atlas-nano) as the cheapest judge to switch to, with conditions, and says plainly that Dmitri's '80% agreement' argument does not justify nano.",
+        "Reports chance-corrected agreement with Priya's 60 expert labels for each judge, matching the data to within about 0.02: Cohen's kappa ~0.80 for atlas-xl, ~0.72 for atlas-m, ~0.37-0.38 for atlas-nano (raw accuracy 55/60, 53/60, 47/60).",
+        "Explains why raw agreement misleads here: 42 of the 60 expert-labelled pairs (70%) are relevant, so always answering 'relevant' already scores 70%; nano's ~78% accuracy and its ~80% raw agreement with xl (kappa ~0.42 over all 300 pairs) sit close to that baseline.",
+        "Identifies nano's failure mode: it over-labels 'relevant' (12 of the 18 expert-irrelevant pairs judged relevant), and the false positives are concentrated on long passages (10 of 11 long irrelevant passages vs 2 of 7 short ones, using the ~400-token split in passage_tokens).",
+        "Connects that length bias to the decision: the planned Q1 re-chunk to ~900-token chunks would be judged by a model that favours long passages, so nano would inflate the new retriever's precision@5 and could wave it through the gate.",
+        "States the uncertainty from a 60-label (only 18 irrelevant) sample - e.g. wide confidence intervals on kappa or accuracy - and says the xl-vs-m difference is not clearly distinguishable on this data; recommends a concrete next step such as more expert labels (especially irrelevant and long passages) before or alongside the switch.",
+        "Notes what the data cannot show rather than asserting it: each judge ran once (no test-retest or flip rate), and the expert labels come from a single annotator; does not report a position-bias result for these pointwise judgments.",
+        "Invents nothing not in the input: no fabricated judge names, prices, label counts, extra annotators or files; any cost figures follow from the README (e.g. roughly $96/night for xl vs $22 for m and $3.60 for nano at 20k judgments)."
+      ]
+    },
+    {
+      "id": 7,
+      "held_out": "2026-10-20",
+      "prompt": "ok quick one - we ran our mod judge (sentinel-2) twice over the same 100 forum posts from the review queue and it's like 96-97% accurate vs what T&S labelled. can we turn on auto-remove for the forums monday?? files are in moderation/, sorry the run 2 export came out in a different format",
+      "assertions": [
+        "Answers the actual question directly: no, do not turn on auto-remove on this evidence (a 'do not rely on it' verdict for auto-removal), optionally offering a narrower use such as ranking or routing posts to human review.",
+        "Points out that only 5 of the 100 posts are violations, so a judge that flags nothing scores 95% - the 96-97% accuracy (97% run 1, 96% run 2) is barely above that baseline.",
+        "Reports the per-run confusion against the gold labels: each run catches 3 of 5 violations (recall 60%); run 1 has 1 false positive (precision 3/4), run 2 has 2 (precision 3/5); chance-corrected agreement with gold is moderate (Cohen's kappa ~0.65 run 1, ~0.58 run 2).",
+        "Joins run_1.jsonl and run_2.jsonl on the post ID (item_id vs id, flagged true/false vs verdict VIOLATION/OK) rather than by line order - run 2 is in a different row order - and gets test-retest agreement of 95/100 with Cohen's kappa ~0.42; does not report the near-zero kappa (~ -0.05) that a line-by-line comparison would produce.",
+        "Shows the instability where it matters: 5 posts flip between runs, including 2 real violations (fp-35782 and fp-31250), and of the 7 posts flagged in either run only 2 (fp-30923, fp-38748) were flagged both times.",
+        "Flags that the self-harm violation fp-34265 was missed in both runs, and that false positives (fp-36620, fp-31299, fp-31504) mean auto-remove would delete legitimate posts.",
+        "Quantifies how little 5 positives can show - e.g. a 95% interval on 3/5 recall spanning roughly 23% to 88% - and recommends a concrete next step such as labelling a larger, violation-enriched sample before reconsidering auto-remove.",
+        "Invents nothing not in the input: no fabricated post IDs, counts, policy categories, thresholds or files, and the reply stays proportionate to a quick Slack-style question (a clear verdict plus the key numbers, not a long report)."
+      ]
+    },
+    {
+      "id": 8,
+      "held_out": "2026-10-20",
+      "prompt": "quick one - our code review bot has an LLM judge that scores every generated patch 0-10 for correctness, been running since july. my manager wants to turn on auto-merge next sprint for anything the judge scores above 8. i dumped what we have into patches.csv (tests col = CI unit test result, only where the repo actually has tests; we also re-ran the judge on some patches last week, that's the rerun col). from eyeballing it the 9s and 10s basically all pass tests so i think we're good?? can u sanity check before i tell him yes",
+      "assertions": [
+        "Opens with a direct answer to 'can I tell him yes': no, not as proposed (a 'do not rely on it' or 'use only with conditions' verdict), and does not endorse auto-merging every patch scored above 8 across all repos.",
+        "Reads 'above 8' as scores 9-10 (or says which reading it used, or shows both >8 and >=8) and computes the decision metric from patches.csv: of the labelled patches scored 9-10, 20 of 24 passed tests (about 83%), so 'basically all pass' is wrong; it names or counts the 4 high-scored failures (PR-1101, PR-1085, PR-1139, PR-1004) that would have been auto-merged.",
+        "Puts that 20/24 in context with uncertainty and a chance baseline: gives a confidence interval (e.g. Wilson about 64-93%, or Clopper-Pearson about 63-95%) and compares it with the base pass rate of 107/153 labelled patches (about 70%), making the point that the judge adds only modest lift over merging blind.",
+        "Catches that infra-scripts has no test results at all yet supplies 15 of the 40 patches scored above 8, so the judge's precision there is unmeasured and the 20/24 figure says nothing about that repo.",
+        "Detects size/length bias: judge scores climb with lines_changed whether or not tests pass (e.g. rank correlation around 0.56 overall, or failing patches scoring high when large); all 4 high-scored failures are 230+ lines, while all 14 labelled patches under 200 lines that scored above 8 passed.",
+        "Uses the judge_score_rerun column for test-retest: of the 60 patches scored twice, only 17 got the same score and 9 land on opposite sides of the 'above 8' line; 14 were above 8 in at least one run but only 5 in both. It concludes the gate itself is unstable.",
+        "Handles the 4 rows whose judge_score is ERR (PR-1222, PR-1161, PR-1127, PR-1133) by excluding them and saying so, not by treating them as 0 or dropping them silently. Any conditions it proposes follow from the findings (e.g. only small patches in repos with tests, CI must still be green, two consistent judge runs, and a labelled sample from infra-scripts before including it).",
+        "Invents nothing: every number, patch ID, repo and test result it cites can be derived from patches.csv. It does not name or guess which LLM the bot uses, and does not claim to have run the bot or CI. The reply is sized as a sanity check (a verdict plus a handful of key numbers and conditions), not a long methodology essay."
+      ]
+    },
+    {
+      "id": 9,
+      "held_out": "2026-10-20",
+      "prompt": "I run localization QA at Halvorsen Language Services. Every segment we deliver is scored for adequacy on a 1-4 scale (1 = meaning lost, 2 = major errors, 3 = minor errors, 4 = complete) by three linguists, and any segment whose average is below 2.5 goes back to the translator for retranslation. A vendor, Quillmark, is pitching an LLM rater; their deck says it \"matches human linguists within one point 92% of the time\". We ran a pilot: adequacy_ratings.csv has 150 segments across en-de, en-es and en-ja, each rated by the three linguists we use for that language (linguist_1/2/3 are slots, not the same people across languages) and by the vendor's LLM. The steering committee meets Thursday and the question is simple: can the LLM replace one of our three linguists? I need a yes / no / yes-with-conditions answer and the evidence behind it, one page at most.",
+      "assertions": [
+        "Answers the committee's question in the first lines with one verdict. It is not an unqualified yes: either no, or yes-with-conditions where the conditions at least exclude en-ja. The whole reply fits roughly one page, as asked.",
+        "Takes apart the vendor's metric: it confirms LLM-linguist within-one-point agreement of about 92% (92.4% pooled) but shows the number is near-meaningless on a 4-point scale, because chance alone gives about 72-73% from these rating distributions and the linguists agree within one point about 98% of the time with each other.",
+        "Uses chance-corrected agreement that suits an ordinal scale (weighted kappa, or Fleiss/Krippendorff with an ordinal treatment) and benchmarks the LLM against the linguists rather than an absolute cutoff. Pooled, mean quadratic-weighted kappa is about 0.85 linguist-to-linguist versus about 0.64 LLM-to-linguist (or equivalent figures from a comparable method), so the LLM is a noticeably weaker rater than any linguist.",
+        "Breaks results out by language pair and flags en-ja. There, LLM-to-linguist quadratic-weighted kappa is about 0.47 versus about 0.88 between linguists (or three-linguist Fleiss kappa about 0.69 dropping to about 0.32-0.36 when the LLM replaces any one slot), while within-one agreement stays about 92% and hides the problem. It notes en-ja has only 40 segments (via an interval or an explicit sample-size caveat).",
+        "Notices the LLM compresses the scale and almost never says 'meaning lost': it gave a 1 only 3 times in 150 (never in en-ja), against 25-29 times per linguist slot. On the 28 segments where at least two linguists gave a 1, the LLM gave a 1 only twice.",
+        "Connects the agreement findings to the retranslation rule (average below 2.5): e.g. three linguists send back 44 segments, and replacing one slot with the LLM changes the send-back decision on 4-5 of the 150 segments. It also says that small number is cushioned by the two remaining linguists and does not show the LLM is as good as a human. A reply that reports agreement statistics only fails this.",
+        "Does not invent test-retest or position-bias numbers. It notes the pilot ran the LLM once per segment, so its stability is unknown, and recommends a repeat run (and, if relevant, a check by segment type or length) before any rollout.",
+        "Invents nothing: every figure, segment ID and language breakdown comes from adequacy_ratings.csv. It attributes no capabilities, model names or other claims to Quillmark beyond the one quoted in the request."
+      ]
+    }
+  ]
+}