paperlint 2.0.0

package/skills/argument-arc/SKILL.md

@@ -0,0 +1,177 @@
+---
+name: argument-arc
+description: Build or repair the paper's argument architecture — the one-sentence-per-section outline, the bottom-up inevitability pass, and the name/number budget. Run it when the reader says the paper throws ideas at them, when a structural objection repeats, or before any large rewrite. Not a prose or length skill.
+allowed-tools: [Read, Write, Grep, Glob, Agent, Bash(node .claude/skills/paper-pipeline/scripts/announce.mjs:*), Bash(node .claude/skills/paper-pipeline/scripts/ledger.mjs:*)]
+---
+
+<!-- vigiles:sha256:05a1eaa79bb596d5 compiled from skills/argument-arc/SKILL.md.spec.ts -->
+
+# argument-arc — does the paper carry the reader to one conclusion
+
+> **Which structure skill?** `tighten-paper` = length, sag, what to cut. `grade-paper-writing` =
+> sentences, jargon, where a reader stalls. **`argument-arc` (you are here) = the load-bearing order
+> of ideas.** The first two assume the argument is right and the delivery is wrong. This one asks
+> whether the argument exists. Run it *before* them — cutting words inside a broken arc is how a
+> session burns a day and ships the same paper.
+
+## Run me
+
+🔴 FIRST, before any other step:
+
+```
+node .claude/skills/paper-pipeline/scripts/announce.mjs argument-arc <paper-dir>
+```
+
+An advisory pass cannot be observed failing — silence is both its error state and its normal
+state — so starting is an event, and events get written down.
+
+The failure it exists to catch has a signature: every paragraph is defensible, every number is real,
+and the reader still finishes the section unable to say what it was for. That is not a prose problem
+and no amount of rewriting sentences fixes it.
+
+## Two modes — and one of them runs BEFORE the study exists
+
+| Mode | Where it runs | Scorecard row | What it produces |
+|---|---|---|---|
+| **frame** | SETUP, before any run | **`frame`** | **One paragraph: what will this paper claim, and what would have to be true for that claim to hold?** |
+| **rebuild** | the LOOP, on a draft whose framing is wrong | `arc` | A new arc, built from scratch through steps 1–4 |
+
+### Frame mode (SETUP) — one paragraph, before the first run
+
+Write it before the study is designed. Not an abstract, not an outline — **a paragraph saying what
+the paper will claim** and what would have to be true for that claim to hold.
+
+**Why it is here and not later:** the frame decides which results matter. A study designed without a
+stated claim measures what the harness can already see, and then the claim gets fitted to whatever
+came out — which is how a run gets thrown away. That is not hypothetical: it was observed on
+`the reference paper`, and it is the reason this skill has a rebuild mode at all. **Reframing after the
+data is collected costs the data. Costs a paragraph; saves a study.**
+
+`build-benchmark` reads this paragraph before designing, and refuses to proceed if `frame` is empty.
+`pipeline-check.mjs` reports a study that ran with no stated claim.
+
+*Added 2026-08-03 alongside the replacement of the numbered stage list — this skill previously existed
+only as a repair, which meant nobody stated the claim while stating it was still cheap.*
+
+## When the arc pass fires (row `arc`, in the LOOP)
+
+- 🔴 **A structural objection repeats.** Not "this is unclear" but *"it throws ideas at me"*,
+  *"I can't hold this in my head"*, *"where is this going"*. **The second time you hear the same
+  objection, stop editing and run this.** The third time means you already ignored the second.
+- Before a rewrite touching more than one section.
+- After a claim dies. When a refutation pass kills a load-bearing claim, the arc built on it is
+  usually dead too, and it will not announce itself — the sections still read fine one at a time.
+- Before `tighten-paper` / `grade-paper-writing` / `pc-panel-review` on any draft whose thesis has
+  changed since those gates last ran.
+
+## How to run it
+
+### 1. Write the conclusion first, in one sentence, in the author's own words
+Not the abstract. The sentence you want the reader thinking as they close the paper. If it takes two
+sentences, the paper has two papers in it and that is the finding.
+
+Ask the author for it if there is any doubt. A conclusion you inferred is a conclusion you will
+defend against them.
+
+### 2. One sentence per section: what does the reader carry out
+Build the whole outline as a flat list, section by section, each line answering **only** *what does
+the reader now believe that they did not believe before this section*. Not "what this section
+covers" — coverage is a table of contents and it hides the defect.
+
+Then read the list on its own, without the paper:
+- **Two lines carrying the same belief** → the sections merge. No exceptions; "but they use different
+  evidence" means one section with two pieces of evidence.
+- **A line you cannot write** → that section has no job. It goes to the artifact or it goes away.
+- **A line that is a topic, not a belief** ("we survey related work") → rewrite it as a belief or
+  admit the section is filler.
+
+### 3. The inevitability pass — bottom to top
+Start at the conclusion and walk **upward**, asking at each section: *does this make the conclusion
+harder to escape?* A section that merely supports the conclusion is not enough; the test is whether
+removing it lets the reader wriggle out.
+
+This direction matters. Top-down you will narrate the paper you wrote. Bottom-up you find the
+sections that are true, interesting and load-bearing for nothing.
+
+### 4. Count the names
+Count every named system, benchmark, corpus and acronym in the body. For each, ask: **must the reader
+remember this later?** If the answer is no, it becomes an unnamed instance ("one corpus study",
+"a trace monitor") or a citation, and the name moves to the artifact or a footnote.
+
+Report the count. A reader who says they cannot hold the paper in their head is usually telling you
+this number, not the word count. There is no correct threshold — but a body carrying more names than
+sections is worth arguing about.
+
+### 5. 🔴 Show the outline to the author BEFORE editing anything
+The deliverable of this skill is the outline and the verdict, **not a rewritten paper**. Hand over:
+the one-sentence conclusion · the section list · which sections merge, move, die · the name count ·
+the single biggest arc defect in one sentence.
+
+Editing before this is agreed is how five patch rounds happen. The author asked for an architecture;
+returning a diff answers a question they did not ask.
+
+## The rebuild mode — when the frame itself is wrong
+
+Every other gate in the pipeline assumes a settled draft and improves it. None has a mode for *the
+framing is wrong, start the arc over*, which is why gates feel inappropriate exactly when the paper
+most needs help. This skill does:
+
+- **Do not defend the current form because it exists.** The question is *"if all this work had been
+  done yesterday by someone else, would I choose this shape?"* Time already spent is not evidence.
+- **Salvage explicitly, in writing:** which measurements survive the reframe untouched, which need
+  re-analysis, which die with the old frame. Measurements usually survive; framing paragraphs rarely do.
+- **A deadline bounds how much you rebuild. It is never an argument that the current shape is right.**
+  If the author says the paper is *weak*, that is a quality judgment, not procrastination, and citing
+  the calendar against it is a category error.
+- **Rebuild produces a new outline through steps 1–4 above, and it too gets shown before editing.**
+
+## Record the verdict
+
+🔴 LAST step, once the deliverable exists:
+
+```
+node .claude/skills/paper-pipeline/scripts/ledger.mjs record argument-arc <paper-dir> FINDING <count> <report-path>
+node .claude/skills/paper-pipeline/scripts/ledger.mjs record argument-arc <paper-dir> ABSTAINED <reason> "<one line>"
+```
+
+🔴 **There is no PASS.** "The arc carries the reader" is not a value this pass can write down; it is
+what a reader may conclude from the absence of findings, and the concluding is theirs.
+
+**FINDING** — the arc is broken; `<count>` is the number of sections the new outline moves, merges
+or deletes, and `<report-path>` is that outline. Add `--blocking` when there is no conclusion
+sentence to build an arc from: that refusal is a finding about the draft, not an aborted run, and
+`<report-path>` is the one-line statement of what is missing.
+**ABSTAINED** — `no-witness`: the outline was read end to end and nothing moved. `blocked`: there
+was no outline to read at all.
+
+An `ABSTAINED no-witness` on a draft the author has twice called a pile is a missed finding, and
+`status.mjs` will start asking about a check that has only ever abstained.
+
+## Rules
+
+- **The arc is judged on the outline, never on the draft.** If you find yourself rereading paragraphs
+  to decide whether a section belongs, you are grading prose again.
+- **Merge is the default remedy, deletion the second.** Most arc defects are two sections doing one
+  job, not a section doing nothing.
+- **Never answer a structural objection with a local edit.** Moving a paragraph and reporting "done"
+  is the exact failure this skill was built from.
+- **The author's confusion is data about the paper, not about the author.** If they ask twice what a
+  section is for, the section is the problem.
+- **Say what the reader must hold in their head at each point.** If that set only grows, the arc is
+  a pile.
+
+## Compose with
+
+- **Before** `tighten-paper` (length/sag) and `grade-paper-writing` (sentences/stalls) — both assume
+  the arc holds. Their verdicts are the gate inputs `pc-panel-review` demands.
+- **After** any refutation pass that killed a load-bearing claim (see the paper's `CLAIMS.md`).
+- `sweep-design-space` when the outline shows the paper has no distinctive move to make — that is a
+  design problem, not an arc problem.
+
+## Provenance
+
+Built 2026-07-30 after a full-day rewrite of `the reference paper` in which the author said the same
+thing five times — *"does not guide the reader sequentially through the ideas, but simply throws a bunch of
+different ideas, references, and benchmarks at them"* — and got five local edits in reply. The pipeline had a skill
+for length, a skill for sentences and a skill for defects; the question *does this paper carry the
+reader to one conclusion* belonged to nobody, and that is the one that failed.

package/skills/argument-arc/SKILL.md.spec.ts

@@ -0,0 +1,192 @@
+// Compiled to SKILL.md by `vigiles compile`. Edit THIS file, never the markdown.
+//
+// Adopted 2026-08-17 as part of the second batch. The body is the previous SKILL.md
+// VERBATIM, so the compiled diff shows only what the compiler adds. No `disallowedTools`
+// fence yet: the field exists on `SkillSpec` as of the branch `claude/skill-disallowed-tools`
+// but is not in a release `mine` installs, so adding it here would not compile.
+import { experimental_skill } from "vigiles/spec";
+
+export default experimental_skill({
+  name: "argument-arc",
+  description:
+    "Build or repair the paper's argument architecture — the one-sentence-per-section outline, the bottom-up inevitability pass, and the name/number budget. Run it when the reader says the paper throws ideas at them, when a structural objection repeats, or before any large rewrite. Not a prose or length skill.",
+  tools: [
+    "Read",
+    "Write",
+    "Grep",
+    "Glob",
+    "Agent",
+    "Bash(node .claude/skills/paper-pipeline/scripts/announce.mjs:*)",
+    "Bash(node .claude/skills/paper-pipeline/scripts/ledger.mjs:*)",
+  ],
+  body: `
+# argument-arc — does the paper carry the reader to one conclusion
+
+> **Which structure skill?** \`tighten-paper\` = length, sag, what to cut. \`grade-paper-writing\` =
+> sentences, jargon, where a reader stalls. **\`argument-arc\` (you are here) = the load-bearing order
+> of ideas.** The first two assume the argument is right and the delivery is wrong. This one asks
+> whether the argument exists. Run it *before* them — cutting words inside a broken arc is how a
+> session burns a day and ships the same paper.
+
+## Run me
+
+🔴 FIRST, before any other step:
+
+\`\`\`
+node .claude/skills/paper-pipeline/scripts/announce.mjs argument-arc <paper-dir>
+\`\`\`
+
+An advisory pass cannot be observed failing — silence is both its error state and its normal
+state — so starting is an event, and events get written down.
+
+The failure it exists to catch has a signature: every paragraph is defensible, every number is real,
+and the reader still finishes the section unable to say what it was for. That is not a prose problem
+and no amount of rewriting sentences fixes it.
+
+## Two modes — and one of them runs BEFORE the study exists
+
+| Mode | Where it runs | Scorecard row | What it produces |
+|---|---|---|---|
+| **frame** | SETUP, before any run | **\`frame\`** | **One paragraph: what will this paper claim, and what would have to be true for that claim to hold?** |
+| **rebuild** | the LOOP, on a draft whose framing is wrong | \`arc\` | A new arc, built from scratch through steps 1–4 |
+
+### Frame mode (SETUP) — one paragraph, before the first run
+
+Write it before the study is designed. Not an abstract, not an outline — **a paragraph saying what
+the paper will claim** and what would have to be true for that claim to hold.
+
+**Why it is here and not later:** the frame decides which results matter. A study designed without a
+stated claim measures what the harness can already see, and then the claim gets fitted to whatever
+came out — which is how a run gets thrown away. That is not hypothetical: it was observed on
+\`the reference paper\`, and it is the reason this skill has a rebuild mode at all. **Reframing after the
+data is collected costs the data. Costs a paragraph; saves a study.**
+
+\`build-benchmark\` reads this paragraph before designing, and refuses to proceed if \`frame\` is empty.
+\`pipeline-check.mjs\` reports a study that ran with no stated claim.
+
+*Added 2026-08-03 alongside the replacement of the numbered stage list — this skill previously existed
+only as a repair, which meant nobody stated the claim while stating it was still cheap.*
+
+## When the arc pass fires (row \`arc\`, in the LOOP)
+
+- 🔴 **A structural objection repeats.** Not "this is unclear" but *"it throws ideas at me"*,
+  *"I can't hold this in my head"*, *"where is this going"*. **The second time you hear the same
+  objection, stop editing and run this.** The third time means you already ignored the second.
+- Before a rewrite touching more than one section.
+- After a claim dies. When a refutation pass kills a load-bearing claim, the arc built on it is
+  usually dead too, and it will not announce itself — the sections still read fine one at a time.
+- Before \`tighten-paper\` / \`grade-paper-writing\` / \`pc-panel-review\` on any draft whose thesis has
+  changed since those gates last ran.
+
+## How to run it
+
+### 1. Write the conclusion first, in one sentence, in the author's own words
+Not the abstract. The sentence you want the reader thinking as they close the paper. If it takes two
+sentences, the paper has two papers in it and that is the finding.
+
+Ask the author for it if there is any doubt. A conclusion you inferred is a conclusion you will
+defend against them.
+
+### 2. One sentence per section: what does the reader carry out
+Build the whole outline as a flat list, section by section, each line answering **only** *what does
+the reader now believe that they did not believe before this section*. Not "what this section
+covers" — coverage is a table of contents and it hides the defect.
+
+Then read the list on its own, without the paper:
+- **Two lines carrying the same belief** → the sections merge. No exceptions; "but they use different
+  evidence" means one section with two pieces of evidence.
+- **A line you cannot write** → that section has no job. It goes to the artifact or it goes away.
+- **A line that is a topic, not a belief** ("we survey related work") → rewrite it as a belief or
+  admit the section is filler.
+
+### 3. The inevitability pass — bottom to top
+Start at the conclusion and walk **upward**, asking at each section: *does this make the conclusion
+harder to escape?* A section that merely supports the conclusion is not enough; the test is whether
+removing it lets the reader wriggle out.
+
+This direction matters. Top-down you will narrate the paper you wrote. Bottom-up you find the
+sections that are true, interesting and load-bearing for nothing.
+
+### 4. Count the names
+Count every named system, benchmark, corpus and acronym in the body. For each, ask: **must the reader
+remember this later?** If the answer is no, it becomes an unnamed instance ("one corpus study",
+"a trace monitor") or a citation, and the name moves to the artifact or a footnote.
+
+Report the count. A reader who says they cannot hold the paper in their head is usually telling you
+this number, not the word count. There is no correct threshold — but a body carrying more names than
+sections is worth arguing about.
+
+### 5. 🔴 Show the outline to the author BEFORE editing anything
+The deliverable of this skill is the outline and the verdict, **not a rewritten paper**. Hand over:
+the one-sentence conclusion · the section list · which sections merge, move, die · the name count ·
+the single biggest arc defect in one sentence.
+
+Editing before this is agreed is how five patch rounds happen. The author asked for an architecture;
+returning a diff answers a question they did not ask.
+
+## The rebuild mode — when the frame itself is wrong
+
+Every other gate in the pipeline assumes a settled draft and improves it. None has a mode for *the
+framing is wrong, start the arc over*, which is why gates feel inappropriate exactly when the paper
+most needs help. This skill does:
+
+- **Do not defend the current form because it exists.** The question is *"if all this work had been
+  done yesterday by someone else, would I choose this shape?"* Time already spent is not evidence.
+- **Salvage explicitly, in writing:** which measurements survive the reframe untouched, which need
+  re-analysis, which die with the old frame. Measurements usually survive; framing paragraphs rarely do.
+- **A deadline bounds how much you rebuild. It is never an argument that the current shape is right.**
+  If the author says the paper is *weak*, that is a quality judgment, not procrastination, and citing
+  the calendar against it is a category error.
+- **Rebuild produces a new outline through steps 1–4 above, and it too gets shown before editing.**
+
+## Record the verdict
+
+🔴 LAST step, once the deliverable exists:
+
+\`\`\`
+node .claude/skills/paper-pipeline/scripts/ledger.mjs record argument-arc <paper-dir> FINDING <count> <report-path>
+node .claude/skills/paper-pipeline/scripts/ledger.mjs record argument-arc <paper-dir> ABSTAINED <reason> "<one line>"
+\`\`\`
+
+🔴 **There is no PASS.** "The arc carries the reader" is not a value this pass can write down; it is
+what a reader may conclude from the absence of findings, and the concluding is theirs.
+
+**FINDING** — the arc is broken; \`<count>\` is the number of sections the new outline moves, merges
+or deletes, and \`<report-path>\` is that outline. Add \`--blocking\` when there is no conclusion
+sentence to build an arc from: that refusal is a finding about the draft, not an aborted run, and
+\`<report-path>\` is the one-line statement of what is missing.
+**ABSTAINED** — \`no-witness\`: the outline was read end to end and nothing moved. \`blocked\`: there
+was no outline to read at all.
+
+An \`ABSTAINED no-witness\` on a draft the author has twice called a pile is a missed finding, and
+\`status.mjs\` will start asking about a check that has only ever abstained.
+
+## Rules
+
+- **The arc is judged on the outline, never on the draft.** If you find yourself rereading paragraphs
+  to decide whether a section belongs, you are grading prose again.
+- **Merge is the default remedy, deletion the second.** Most arc defects are two sections doing one
+  job, not a section doing nothing.
+- **Never answer a structural objection with a local edit.** Moving a paragraph and reporting "done"
+  is the exact failure this skill was built from.
+- **The author's confusion is data about the paper, not about the author.** If they ask twice what a
+  section is for, the section is the problem.
+- **Say what the reader must hold in their head at each point.** If that set only grows, the arc is
+  a pile.
+
+## Compose with
+
+- **Before** \`tighten-paper\` (length/sag) and \`grade-paper-writing\` (sentences/stalls) — both assume
+  the arc holds. Their verdicts are the gate inputs \`pc-panel-review\` demands.
+- **After** any refutation pass that killed a load-bearing claim (see the paper's \`CLAIMS.md\`).
+- \`sweep-design-space\` when the outline shows the paper has no distinctive move to make — that is a
+  design problem, not an arc problem.
+
+## Provenance
+
+Built 2026-07-30 after a full-day rewrite of \`the reference paper\` in which the author said the same
+thing five times — *"does not guide the reader sequentially through the ideas, but simply throws a bunch of
+different ideas, references, and benchmarks at them"* — and got five local edits in reply. The pipeline had a skill
+for length, a skill for sentences and a skill for defects; the question *does this paper carry the
+reader to one conclusion* belonged to nobody, and that is the one that failed.`,
+});

package/skills/argument-arc/argument-arc.eval.mjs

@@ -0,0 +1,19 @@
+/**
+ * argument-arc — the PAID tier: does this skill's description actually fire?
+ *
+ * COLOCATED ON PURPOSE (vigiles decides coverage by placement as of 2026-08-11).
+ * The prompts live in `.claude/lib/skill-trigger-cases.mjs` so all 21 cases
+ * are reviewed as one table where collisions between siblings are visible;
+ * copying them here would recreate the drift that rule exists to prevent.
+ *
+ * Measures recall (fires on its own territory) AND precision (stays quiet on a
+ * colliding sibling's territory), against the REAL `.claude` harness so the skill
+ * competes with every other installed description — an isolated run overstates
+ * recall and understates false positives.
+ *
+ * Costs money; not CI.
+ *   node .claude/skills/argument-arc/argument-arc.eval.mjs [trials]
+ */
+import { runSkillTriggerEval } from "../../lib/skill-eval-kit.mjs";
+
+await runSkillTriggerEval("argument-arc");

package/skills/argument-arc/argument-arc.harness.mjs

@@ -0,0 +1,23 @@
+/**
+ * argument-arc — the free, deterministic tier. No model, no network.
+ *
+ * COLOCATED ON PURPOSE. vigiles decides coverage by PLACEMENT as of 2026-08-11:
+ * a test that merely names a surface no longer counts, because that tier was
+ * crediting surfaces nothing touched. So each skill needs a file inside its own
+ * directory — this one.
+ *
+ * The assertions live in `.claude/lib/skill-checks.mjs` and are CALLED here with
+ * this skill's name. They are not copied: 22 copies of the same checks is the drift that
+ * module exists to avoid. (Until 2026-08-11 this was an env-var side channel into a
+ * 614-line file named after no surface; it is a function call now.)
+ *
+ * What this proves: this skill's frontmatter parses as strict YAML, its declared
+ * tool contract is sane, its pipeline wiring points at scripts that exist, and it
+ * announces/records under ITS OWN identity rather than a sibling's.
+ *
+ * What it does NOT prove: that the skill fires, or that its guidance produces a
+ * good result. Those need a real model — see `argument-arc.eval.mjs`.
+ */
+import { checkSkill } from "../../lib/skill-checks.mjs";
+
+await checkSkill("argument-arc");

package/skills/build-benchmark/SKILL.md

@@ -0,0 +1,213 @@
+---
+name: build-benchmark
+description: Design and run the empirical study behind a measurement paper, and ship a reviewer-proof reproduction artifact. Covers the study design (invert the pitfall you're critiquing), honest statistics (paired/Welch t, CIs, Bonferroni, small-n spread as a result, not noise), a structural bound that outlives the specific artifacts tested, and a self-checking artifact that recomputes every headline number and exits non-zero on drift. Use when the paper's contribution is a way to MEASURE something and you're building the evidence + the thing reviewers will run. Compose with research-ideate (upstream), draft-paper, pc-panel-review (its artifact-runner executes this), submit-paper.
+context: fork
+allowed-tools: [Read, Write, Edit, Grep, Glob, Bash, Agent]
+---
+
+<!-- vigiles:sha256:067ada6f8aa44d5b compiled from skills/build-benchmark/SKILL.md.spec.ts -->
+
+# build-benchmark — the study + the artifact reviewers can run
+
+## Run me
+
+🔴 FIRST, before any other step:
+
+```
+node .claude/skills/paper-pipeline/scripts/announce.mjs build-benchmark <paper-dir>
+```
+
+An advisory pass cannot be observed failing — silence is both its error state and its normal
+state — so starting is an event, and events get written down.
+
+A measurement paper is only as strong as the artifact a reviewer can `cd` into and re-run. This skill
+covers both halves: the study design that makes the finding true, and the self-checking artifact that
+makes it *verifiable*. The contribution is the **method**, not the one tool you happened to test — build
+so both survive review.
+
+## 0. 🔴 READ THE STATED CLAIM FIRST — and refuse to design without it
+
+**Before designing anything, open `<paper-dir>/PIPELINE-STATUS.md` and read row `frame`** — the one
+paragraph saying what this paper will claim (written by `argument-arc` in **frame mode**, in SETUP).
+
+🔴 **If `frame` is empty, stop. Do not design the study.** Go run `argument-arc` frame mode, get the
+paragraph, then come back. This is a refusal, not a recommendation:
+
+- **The frame decides which results matter.** A study designed without a stated claim measures
+  whatever the available harness can already see, and the claim gets fitted afterwards to whatever
+  came out. That is backwards, and it is expensive in exactly one direction — **reframing after the
+  data is collected is how runs get thrown away**, observed on `compile-rules-2026`.
+- **The claim is what makes §1 answerable.** "What would make the critiqued pitfall impossible here?"
+  has no answer until you have written down what you are claiming.
+- **It costs a paragraph and it saves a study.** There is no version of this trade that favours
+  starting the runs first.
+
+Cross-check the claim against the prevention rule in the `CLAUDE.md` beside the papers: if the claim is
+about *preventing* something, write the one sentence saying **what physically stops the bad outcome in
+the experimental arm**. "There is different text in it" means you are measuring persuasion, not
+prevention, and the design is wrong before a single run.
+
+## 1. Design that inverts the pitfall you're critiquing
+
+The strongest measurement papers are a corrected version of the mistake they name. Pick the design that
+is the *inverse* of the flaw:
+
+- **Critiquing a proxy metric?** Measure the real thing, gated on correctness. The cost study is a
+  **cost-aware, correctness-gated A/B**: two arms × two tools, every run scored pass/fail first, then
+  the *dollar* bill compared — because the whole point was that the token-proxy and the dollar disagree.
+  Never let a cheaper-but-broken run count as a win; correctness is the gate before cost is even read.
+- **Measuring something that runs untrusted/destructive code?** **Parse, don't execute.** The guard eval
+  is a **SAFE STATIC evaluation**: transcribe each guard's predicate by hand and reason over it against a
+  disaster battery — never run the guard or the attack. See Safety below; this is non-negotiable.
+
+Write the design as the answer to "what would make the critiqued pitfall impossible here?"
+
+## 2. Honest statistics — the part reviewers attack first
+
+- **Paired where the design is paired; Welch where variances differ.** Use a **paired-t** when the same
+  task is run under both arms (blocks task difficulty); use **Welch's t** for unequal-variance unpaired
+  comparisons. Do NOT label a test "paired" unless the pairing is real — a mislabeled test is a blocker.
+- **Report CIs and k/n, never bare percentages over tiny n.** "2/10 guards covered" beats "20% coverage";
+  "median 2/10 (range 0–5)" beats a single mean. A percentage over n=10 hides the n.
+- **Report the SOURCE CONCENTRATION of a mined corpus — the single-source share is a number, not a
+  caveat.** For any corpus mined from several repositories / trackers / vendors, compute and print
+  **n per source and the max single-source share**, and put it in the paper (a table row or one
+  sentence), not only in Threats. A corpus advertised as covering *k* sources but dominated by one is
+  a top reason a mined-corpus paper gets rejected, and it is invisible to every other check — the
+  totals, the percentages and the internal consistency all pass, because the cut was simply never
+  made. **Mechanical leg:** the artifact emits `source, n, share` for the corpus and the top share
+  appears in the paper. If the top source exceeds ~50%, say so in the abstract's scope or narrow the
+  claimed population to what you actually sampled.
+- **Inter-rater agreement is void if the rater was trained by, and scored against, the codebook's
+  author.** Agreement with the person who wrote the scheme measures *trainability*, not construct
+  reliability — it is the human form of the failure this suite already names in code (a checker and
+  its self-test written in one pass by one model). Requirements, all mechanical: (i) reference labels
+  come from a rater who did **not** author the codebook and did **not** see the hypothesis; (ii)
+  report the agreement **denominator as a share of the full corpus** — "κ=0.93 on 69 of 547 (12.6%)",
+  never a bare κ; (iii) use a coefficient that matches the design — Cohen's κ is single-label, so
+  multi-label coding needs per-label κ or Krippendorff's α; (iv) treat **κ = 1.00 as a red flag to
+  investigate, not a result to report** — perfect agreement on a many-category scheme usually means
+  the validation set was easy or calibration was de-facto joint coding.
+- **Correct for the family.** Multiple comparisons across a family of tasks/tools → **Bonferroni** (or
+  state the correction you used). Report it; don't p-hack the one significant cell.
+- **Small-n spread is a first-class result, not noise.** If five trials of the same task swing wildly,
+  that variance IS the finding (the thing under test is unstable) — report it, don't average it away.
+- Repeated trials: fix trial count up front (e.g. per-task × trials × arms × tools), report the full n,
+  and treat every run — including failures — as data.
+
+## 3. A structural bound that outlives the artifacts
+
+Numbers about today's tools rot. A **mechanism or structural bound** doesn't. Give the paper one claim
+that holds regardless of which specific tool/version you measured:
+
+- the cost study's bound: token-efficiency and dollar cost are decoupled by pricing structure, so a
+  token-optimizing tool *cannot* be assumed to cut the bill — a property of the pricing, not the tool.
+- the guard eval's axes: **mutation-evasion** (does a trivial rephrase of the attack slip the guard?),
+  **held-out generalization** (does coverage transfer to commands the guard wasn't written for?), and
+  **per-step causal ablation** (remove one guard step, measure the coverage delta — which step actually
+  does the work). These are properties of the *defense class*, reusable against the next guard set.
+
+State the bound explicitly; it's what makes the paper a benchmark and not a product review.
+
+## 4. Build a SELF-CHECKING artifact
+
+This is the single biggest accept-probability lever, and pc-panel-review's artifact-runner WILL execute
+it. Do **not** restate the requirements here — build to the canonical checklist:
+
+→ **`paper-pipeline/references/artifact-checklist.md`** (self-recompute from raw data, exit non-zero on
+drift, stdlib-only, no network, a README that maps each paper-number to where it prints, LICENSE, honest
+badge scope, de-anon scanned).
+
+The load-bearing property: each script **recomputes** every headline number from raw data and
+**self-asserts** — exit non-zero the moment a cell drifts from the paper (`reproduce.py` on the cost
+study; `evaluate.py`/`mutate.py`/`reproduce.mjs`/`ablation.mjs` on the guard eval). Derive from base
+fields, never echo a stored/precomputed field — a reviewer calls that circular and they're right. For
+de-anonymization before hosting, follow **`paper-pipeline/references/anonymization.md`**; host per
+`submit-paper` (OSF anonymized view-only link; automation at `papers/osf_upload.py`).
+
+## Safety — never execute untrusted or destructive commands to measure them
+
+If the object of study is code that could delete, exfiltrate, or run attacker-controlled input, you
+**transcribe and parse its predicate** — you never execute it, and you never run the attack against it.
+The guard eval scores 46 real command-guards by reading each guard's logic against a 10-command disaster
+battery statically; nothing in the battery is ever run. A benchmark that has to detonate the payload to
+score it is a liability, not evidence. Same rule inside the artifact: no network, no shelling out to the
+thing under test.
+
+## Validate the transcription against real-code execution, not just a blind re-derivation
+
+When you score third-party code by **transcribing its logic** (e.g. a guard's regexes) into your own
+evaluator, a *blind re-derivation* (a second person re-reads the code and re-transcribes it) only checks
+transcription-vs-human-reading — it does **not** check transcription-vs-real-behavior. There's a cheap,
+zero-execution-risk way to close that gap: **run the actual scraped code against its REAL agent input
+contract** and reconcile its live verdict with your static prediction.
+
+- **Feed the real contract, not a bare string.** For Claude Code PreToolUse hooks, the command arrives as
+  a **JSON envelope on stdin** with the command at `.tool_input.command`. Feed each battery/benign command
+  as a **string inside that envelope** to the real guard's stdin and read its verdict (exit 2 = block; or
+  stdout JSON `permissionDecision` deny/ask/block). The command is **never executed** — the guard only
+  inspects the string.
+- **SAFETY: sandbox + stub-PATH first.** Run inside a throwaway sandbox with a **stub PATH** (fake
+  `rm`/`dd`/`git`/`curl`/… → `exit 0`) as defense-in-depth, so even a guard that shells out can't do
+  damage. Invoke the real interpreters by absolute path.
+- **Reconcile per code×input pair.** Every mismatch between live verdict and static prediction is a
+  **finding** — either a transcription nit to fix, or a real behavior your static model structurally
+  cannot capture.
+
+**Concrete payoff (the motivating example):** this method surfaced a **"whole-stdin confound"** — a
+widely-copied guard that greps its **entire raw stdin** rather than the extracted command. The JSON
+envelope's trailing bytes defeated its anchored `rm …/$` rule, so it **FAILED TO BLOCK `rm -rf /`** live,
+even though it "blocked" the bare string. A pure transcription/static model can't see this; only
+real-code-through-real-contract execution reveals it.
+
+Before submitting, verify the artifact you built with the reviewer-side reproduction protocol:
+**`references/adversarial-cold-repro.md`** (cold re-run, three-way reconcile, robustness attacks).
+
+## Record the verdict
+
+🔴 LAST step, once the deliverable exists:
+
+```
+node .claude/skills/paper-pipeline/scripts/ledger.mjs record build-benchmark <paper-dir> FINDING <count> <report-path>
+node .claude/skills/paper-pipeline/scripts/ledger.mjs record build-benchmark <paper-dir> ABSTAINED <reason> "<one line>"
+```
+
+🔴 **There is no PASS.** An artifact that recomputes every headline number and exits 0 has produced
+no finding; it has not certified the study. Record the absence, do not name it a success.
+
+**FINDING** — `<count>` numbers did not reproduce, or reproduce only with caveats; `<report-path>`
+is the artifact output. Add `--blocking` for the §0 refusal: `frame` is empty, so the study was not
+designed.
+**ABSTAINED** — `no-witness`: the artifact ran and every number reproduced. `input-missing`: the
+raw data is not in the repo. `crashed`: the harness itself died.
+
+🔴 That §0 refusal is a **finding, not an aborted run** — it is a fact about the work, so it is a
+FINDING and not an abstention. Record it and stop. A study that was never designed leaves exactly
+the same silence as one that is still running, and the difference costs a week to rediscover.
+
+🔴 **THREE MECHANICAL CHECKS FILE UNDER THIS SKILL** and each has its own ledger row:
+`build-benchmark/check-provenance`, `build-benchmark/arm-permutation`,
+`build-benchmark/delivered-pdf` (see `.claude/skills/paper-pipeline/scripts/run-mechanical.mjs`). Until 2026-08-10 the
+ledger keyed on the SKILL, so a clean run of one erased a finding of another from every derived
+view — which is why two of the three sat unwired. This block records the JUDGEMENT pass, under the
+bare key `build-benchmark`; never file a mechanical result here.
+
+## Compose with
+- **argument-arc (frame mode)** — 🔴 **hard input.** Owns row `frame`, the stated claim. No `frame`, no study.
+- **research-ideate** (upstream) — supplies the contribution and the pitfall to invert; this skill turns
+  it into evidence.
+- **draft-paper** — the study's stats, bound, and artifact-number table feed Methods/Results/Availability.
+- **pc-panel-review** — its artifact-runner reviewer `cd`s in and re-executes every harness; build so
+  that pass is deterministic.
+- **submit-paper** — hosts the scrubbed artifact anonymously and links it in Availability.
+
+## Provenance
+- **AgenticDev 2026 @ ASE — "Measuring the Wrong Number":** a cost-aware, correctness-gated A/B harness
+  over 140 runs (7 tasks × 5 trials × 2 arms × 2 tools); Welch t + paired-t CIs + Bonferroni; the finding
+  that token-efficiency tools don't cut the dollar bill. Artifact = a stdlib-only `reproduce.py` that
+  recomputes every headline number from raw data and exits non-zero on drift.
+- **AISec 2026 @ ACM CCS — "Safety Theater":** a 10-command disaster battery against 46 real
+  command-guards, SAFE STATIC evaluation (transcribe each predicate, never execute), median coverage
+  2/10, plus mutation-evasion, held-out generalization, and per-step causal ablation axes. Artifact =
+  `evaluate.py`/`mutate.py`/`reproduce.mjs`/`ablation.mjs`, each self-asserting on run. Both artifacts
+  are hosted anonymized on OSF.