paperlint 9.0.1 → 10.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (196) hide show
  1. package/CONTRIBUTING.md +16 -4
  2. package/README.md +68 -128
  3. package/dist/build.d.ts +1 -1
  4. package/dist/build.js +1 -1
  5. package/dist/cli.d.ts +5 -1
  6. package/dist/cli.d.ts.map +1 -1
  7. package/dist/cli.js +52 -6
  8. package/dist/cli.js.map +1 -1
  9. package/dist/domain/layout-tree.d.ts +19 -0
  10. package/dist/domain/layout-tree.d.ts.map +1 -0
  11. package/dist/domain/layout-tree.js +32 -0
  12. package/dist/domain/layout-tree.js.map +1 -0
  13. package/dist/domain/lookup-cache.d.ts +2 -2
  14. package/dist/domain/lookup-cache.js +2 -2
  15. package/dist/domain/paper-layout.d.ts +81 -0
  16. package/dist/domain/paper-layout.d.ts.map +1 -0
  17. package/dist/domain/paper-layout.js +259 -0
  18. package/dist/domain/paper-layout.js.map +1 -0
  19. package/dist/domain/repro-layout.d.ts +80 -0
  20. package/dist/domain/repro-layout.d.ts.map +1 -0
  21. package/dist/domain/repro-layout.js +247 -0
  22. package/dist/domain/repro-layout.js.map +1 -0
  23. package/dist/domain/scaffold-layout.d.ts +28 -0
  24. package/dist/domain/scaffold-layout.d.ts.map +1 -0
  25. package/dist/domain/scaffold-layout.js +81 -0
  26. package/dist/domain/scaffold-layout.js.map +1 -0
  27. package/dist/eslint-rules/js-needs-reason.d.ts +55 -0
  28. package/dist/eslint-rules/js-needs-reason.d.ts.map +1 -0
  29. package/dist/eslint-rules/js-needs-reason.js +35 -0
  30. package/dist/eslint-rules/js-needs-reason.js.map +1 -0
  31. package/dist/eslint-rules/prereg-coverage.d.ts +98 -0
  32. package/dist/eslint-rules/prereg-coverage.d.ts.map +1 -0
  33. package/dist/eslint-rules/prereg-coverage.js +274 -0
  34. package/dist/eslint-rules/prereg-coverage.js.map +1 -0
  35. package/dist/folder-layout-hook.d.ts +21 -0
  36. package/dist/folder-layout-hook.d.ts.map +1 -0
  37. package/dist/folder-layout-hook.js +117 -0
  38. package/dist/folder-layout-hook.js.map +1 -0
  39. package/dist/folder-layout-rule.d.ts +22 -0
  40. package/dist/folder-layout-rule.d.ts.map +1 -0
  41. package/dist/folder-layout-rule.js +107 -0
  42. package/dist/folder-layout-rule.js.map +1 -0
  43. package/dist/init.js +1 -1
  44. package/dist/init.js.map +1 -1
  45. package/dist/new-paper.d.ts +4 -1
  46. package/dist/new-paper.d.ts.map +1 -1
  47. package/dist/new-paper.js +40 -6
  48. package/dist/new-paper.js.map +1 -1
  49. package/dist/package-dirs.d.ts +6 -0
  50. package/dist/package-dirs.d.ts.map +1 -1
  51. package/dist/package-dirs.js +6 -0
  52. package/dist/package-dirs.js.map +1 -1
  53. package/dist/references.d.ts +1 -1
  54. package/dist/references.d.ts.map +1 -1
  55. package/dist/references.js +2 -2
  56. package/dist/references.js.map +1 -1
  57. package/dist/repro-layout-rule.d.ts +16 -0
  58. package/dist/repro-layout-rule.d.ts.map +1 -0
  59. package/dist/repro-layout-rule.js +81 -0
  60. package/dist/repro-layout-rule.js.map +1 -0
  61. package/dist/skills/label-agent-runs/scripts/labels-commands.d.ts +13 -0
  62. package/dist/skills/label-agent-runs/scripts/labels.d.ts +2 -0
  63. package/dist/skills/run-agent-grid/scripts/grid-guard-commands.d.ts +20 -0
  64. package/dist/skills/run-agent-grid/scripts/grid-guard.d.ts +2 -0
  65. package/docs/ci.md +60 -0
  66. package/docs/configuration.md +2 -0
  67. package/docs/e2e.md +2 -2
  68. package/docs/install.md +20 -16
  69. package/docs/layout.md +154 -0
  70. package/docs/optional-rules.md +5 -4
  71. package/docs/references.md +5 -2
  72. package/docs/rules/paper/allowed-files-only.md +192 -0
  73. package/docs/rules/repro/allowed-files-only.md +128 -0
  74. package/docs/rules/study/declare-sample.md +146 -0
  75. package/docs/rules.md +53 -50
  76. package/docs/skills.md +8 -3
  77. package/docs/testing.md +13 -0
  78. package/docs/venues.md +94 -0
  79. package/eslint-rules/js-needs-reason.test.ts +141 -0
  80. package/eslint-rules/js-needs-reason.ts +77 -0
  81. package/eslint-rules/prereg-coverage.test.ts +448 -0
  82. package/eslint-rules/prereg-coverage.ts +419 -0
  83. package/eslint.config.mjs +22 -2
  84. package/lib/skill-trigger-cases.harness.mjs +51 -5
  85. package/lib/skill-trigger-cases.mjs +53 -15
  86. package/lib/skill-trigger-cases.test.ts +211 -0
  87. package/package.json +8 -4
  88. package/plugin/hooks/hooks.json +9 -0
  89. package/scripts/coverage-config.test.ts +5 -3
  90. package/skills/analyze-sibling-paper/SKILL.md +2 -2
  91. package/skills/analyze-sibling-paper/SKILL.md.spec.ts +1 -1
  92. package/skills/build-benchmark/SKILL.md +17 -1
  93. package/skills/build-benchmark/SKILL.md.spec.ts +16 -0
  94. package/skills/build-label-page/SKILL.md +120 -0
  95. package/skills/build-label-page/SKILL.md.spec.ts +131 -0
  96. package/skills/build-label-page/build-label-page.eval.mjs +15 -0
  97. package/skills/build-label-page/build-label-page.test.ts +959 -0
  98. package/skills/build-label-page/references/checklist.md +148 -0
  99. package/skills/build-label-page/references/example-generator.py +803 -0
  100. package/skills/find-venue/SKILL.md +5 -3
  101. package/skills/find-venue/SKILL.md.spec.ts +4 -2
  102. package/skills/grade-paper-writing/SKILL.md +7 -7
  103. package/skills/grade-paper-writing/SKILL.md.spec.ts +6 -6
  104. package/skills/label-agent-runs/SKILL.md +103 -0
  105. package/skills/label-agent-runs/SKILL.md.spec.ts +106 -0
  106. package/skills/label-agent-runs/label-agent-runs.eval.ts +55 -0
  107. package/skills/label-agent-runs/label-agent-runs.harness.ts +123 -0
  108. package/skills/label-agent-runs/references/formats.md +64 -0
  109. package/skills/label-agent-runs/scripts/labels-commands.js +13 -0
  110. package/skills/label-agent-runs/scripts/labels-commands.ts +13 -0
  111. package/skills/label-agent-runs/scripts/labels.js +396 -0
  112. package/skills/label-agent-runs/scripts/labels.test.ts +538 -0
  113. package/skills/label-agent-runs/scripts/labels.ts +501 -0
  114. package/skills/map-prior-work/SKILL.md +2 -2
  115. package/skills/map-prior-work/SKILL.md.spec.ts +1 -1
  116. package/skills/paper-pipeline/SKILL.md +15 -6
  117. package/skills/paper-pipeline/SKILL.md.spec.ts +14 -5
  118. package/skills/paper-pipeline/grade-paper-writing-ablation.eval.mjs +19 -9
  119. package/skills/paper-pipeline/references/acceptance-gate.md +19 -19
  120. package/skills/paper-pipeline/references/artifact-checklist.md +5 -0
  121. package/skills/paper-pipeline/references/pipeline-status-template.md +30 -1
  122. package/skills/paper-pipeline/repro/2026-08-07-fork-check.log +1 -1
  123. package/skills/paper-pipeline/repro/2026-08-07-fork-check2.log +1 -1
  124. package/skills/paper-pipeline/repro/2026-08-07-language-eval-pilot.log +1 -1
  125. package/skills/paper-pipeline/repro/2026-08-07-language-eval-raw.log +1 -1
  126. package/skills/paper-pipeline/repro/2026-08-08-framing-vs-vocabulary-oracle.log +48 -48
  127. package/skills/paper-pipeline/repro/2026-08-08-grade-paper-writing-ablation-A6-oracle.log +8 -8
  128. package/skills/paper-pipeline/repro/2026-08-08-grade-paper-writing-ablation-oracle.log +48 -48
  129. package/skills/paper-pipeline/repro/2026-08-08-parent-replication-gpw.log +2 -2
  130. package/skills/paper-pipeline/repro/2026-08-08-parent-replication.mjs +3 -3
  131. package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-ablation-preflight-after.log +81 -0
  132. package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-ablation-preflight-before.log +81 -0
  133. package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-ablation-setupdiff.log +59 -0
  134. package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-trigger.jsonl +1 -0
  135. package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-trigger.log +22 -0
  136. package/skills/paper-pipeline/repro/README.md +4 -0
  137. package/skills/paper-pipeline/scripts/extract-ref-facts.harness.mjs +2 -2
  138. package/skills/paper-pipeline/scripts/extract-ref-facts.mjs +2 -2
  139. package/skills/paper-pipeline/scripts/extract-ref-facts.test.mjs +4 -2
  140. package/skills/paper-pipeline/scripts/generated-code.harness.mjs +7 -2
  141. package/skills/paper-pipeline/scripts/generated-code.mjs +8 -4
  142. package/skills/paper-pipeline/scripts/generated-code.test.ts +15 -1
  143. package/skills/paper-status/SKILL.md +2 -2
  144. package/skills/paper-status/SKILL.md.spec.ts +1 -1
  145. package/skills/run-agent-grid/SKILL.md +181 -0
  146. package/skills/run-agent-grid/SKILL.md.spec.ts +184 -0
  147. package/skills/run-agent-grid/references/failure-catalog.md +36 -0
  148. package/skills/run-agent-grid/references/registry-and-ledger.md +142 -0
  149. package/skills/run-agent-grid/run-agent-grid.eval.ts +59 -0
  150. package/skills/run-agent-grid/run-agent-grid.harness.ts +115 -0
  151. package/skills/run-agent-grid/scripts/grid-guard-commands.js +18 -0
  152. package/skills/run-agent-grid/scripts/grid-guard-commands.ts +21 -0
  153. package/skills/run-agent-grid/scripts/grid-guard.js +700 -0
  154. package/skills/run-agent-grid/scripts/grid-guard.test.ts +1090 -0
  155. package/skills/run-agent-grid/scripts/grid-guard.ts +869 -0
  156. package/skills/study-accepted-papers/SKILL.md +163 -161
  157. package/skills/study-accepted-papers/SKILL.md.spec.ts +167 -165
  158. package/skills/study-accepted-papers/references/base-rates.md +69 -0
  159. package/skills/study-accepted-papers/references/mechanisms.md +181 -0
  160. package/skills/verify-citations/SKILL.md +2 -2
  161. package/skills/verify-citations/SKILL.md.spec.ts +1 -1
  162. package/src/build.ts +1 -1
  163. package/src/cli.harness.mjs +42 -0
  164. package/src/cli.test.ts +90 -6
  165. package/src/cli.ts +78 -5
  166. package/src/domain/layout-tree.test.ts +32 -0
  167. package/src/domain/layout-tree.ts +47 -0
  168. package/src/domain/lookup-cache.ts +2 -2
  169. package/src/domain/paper-layout.test.ts +264 -0
  170. package/src/domain/paper-layout.ts +336 -0
  171. package/src/domain/repro-layout.test.ts +222 -0
  172. package/src/domain/repro-layout.ts +311 -0
  173. package/src/domain/scaffold-layout.ts +104 -0
  174. package/src/folder-layout-hook.test.ts +214 -0
  175. package/src/folder-layout-hook.ts +141 -0
  176. package/src/folder-layout-rule.test.ts +235 -0
  177. package/src/folder-layout-rule.ts +151 -0
  178. package/src/hooks-settings.harness.ts +9 -8
  179. package/src/init.ts +1 -1
  180. package/src/layout-docs.test.ts +52 -0
  181. package/src/new-paper.harness.mjs +23 -1
  182. package/src/new-paper.test.ts +299 -6
  183. package/src/new-paper.ts +51 -5
  184. package/src/package-dirs.ts +8 -0
  185. package/src/paper-files.test.ts +9 -6
  186. package/src/references.test.ts +5 -5
  187. package/src/references.ts +2 -2
  188. package/src/repro-layout-rule.test.ts +114 -0
  189. package/src/repro-layout-rule.ts +121 -0
  190. package/src/rule-cases.test.ts +98 -0
  191. package/templates/paper/PIPELINE-STATUS.md +41 -7
  192. package/tsconfig.base.json +22 -0
  193. package/tsconfig.json +6 -24
  194. package/tsconfig.pkg.json +3 -1
  195. package/tsconfig.skills.json +14 -0
  196. package/tsconfig.src.json +12 -0
@@ -0,0 +1,501 @@
1
+ #!/usr/bin/env node
2
+ /**
3
+ * labels — the checks of the label-agent-runs skill.
4
+ *
5
+ * validate <labels.tsv> --codebook <codebook.json> [--ledger <attempts.jsonl>]
6
+ * [--analysed-at <git rev> --notes <notes.md>]
7
+ * The label file is well formed, uses only codebook labels, names each cell once, and —
8
+ * with a ledger — covers exactly the completed runs. The codebook is committed, unchanged,
9
+ * and was committed no later than the label file. With --analysed-at: every cell whose
10
+ * label differs from that revision is explained in a dated section of the notes, with a
11
+ * verbatim quote.
12
+ * agree <a.tsv> <b.tsv>
13
+ * Agreement between two label files over the same cells: n, percent agreement with a 95%
14
+ * Wilson interval, Cohen's kappa, and every disagreement. Use it for the automatic pre-sort
15
+ * against the hand labels, and for one rater against another.
16
+ *
17
+ * A label file is TSV with exactly this header: agent, model, cell, label. `cell` is
18
+ * `<unit>/<arm>/<seed>`, the cell of the run-agent-grid ledger.
19
+ *
20
+ * TypeScript, compiled beside itself to `labels.js` by `npm run build`: a consumer runs the skill
21
+ * from `node_modules`, where Node does not strip types. The skill names the `.js`.
22
+ *
23
+ * EXIT CODES: 0 valid / reported · 1 a finding · 2 usage.
24
+ */
25
+ import { spawnSync } from "node:child_process";
26
+ import { readFileSync } from "node:fs";
27
+ import { dirname, relative, resolve, sep } from "node:path";
28
+ import { parseArgs } from "node:util";
29
+ import { z } from "zod";
30
+ import { COMMANDS } from "./labels-commands.ts";
31
+
32
+ class Refusal extends Error {
33
+ readonly exitCode: number;
34
+ constructor(exitCode: number, message: string) {
35
+ super(message);
36
+ this.exitCode = exitCode;
37
+ }
38
+ }
39
+ const refuse = (exitCode: number, message: string): never => {
40
+ throw new Refusal(exitCode, message);
41
+ };
42
+ const say = (line: string): void => {
43
+ process.stdout.write(`${line}\n`);
44
+ };
45
+ const prop = (e: unknown, key: string): unknown => Reflect.get(Object(e), key);
46
+
47
+ const HEADER = ["agent", "model", "cell", "label"];
48
+ interface Row {
49
+ readonly agent: string;
50
+ readonly model: string;
51
+ readonly cell: string;
52
+ readonly label: string;
53
+ readonly n: number;
54
+ }
55
+ type CellId = Pick<Row, "agent" | "model" | "cell">;
56
+ const keyOf = (r: CellId): string => `${r.agent}\t${r.model}\t${r.cell}`;
57
+ const show = (r: CellId): string => `${r.agent} ${r.model} ${r.cell}`;
58
+
59
+ interface Line {
60
+ readonly fields: readonly string[];
61
+ readonly n: number;
62
+ }
63
+
64
+ /** A line with exactly four fields as a row; anything else is not one. */
65
+ const asRow = ({ fields, n }: Line): Row[] => {
66
+ const [agent, model, cell, label] = fields;
67
+ return fields.length === 4
68
+ ? [
69
+ {
70
+ agent: String(agent),
71
+ model: String(model),
72
+ cell: String(cell),
73
+ label: String(label),
74
+ n,
75
+ },
76
+ ]
77
+ : [];
78
+ };
79
+
80
+ /** Rows of a label file, and the problems that stop a line being one. */
81
+ function readLabels(file: string): { rows: Row[]; problems: string[] } {
82
+ const [first, ...body] = readFileSync(file, "utf8").split("\n");
83
+ const lines: Line[] = body
84
+ .map((line, i) => ({ line, fields: line.split("\t"), n: i + 2 }))
85
+ .filter(({ line }) => line.trim() !== "");
86
+ const headerOk = String(first).replace(/\r$/, "") === HEADER.join("\t");
87
+ return {
88
+ rows: lines.flatMap(asRow),
89
+ problems: [
90
+ ...(headerOk
91
+ ? []
92
+ : [`${file}:1: the header must be "${HEADER.join("\\t")}"`]),
93
+ ...lines
94
+ .filter(({ fields }) => fields.length !== 4)
95
+ .map(({ n }) => `${file}:${String(n)}: not four tab-separated fields`),
96
+ ],
97
+ };
98
+ }
99
+
100
+ const git = (cwd: string, ...args: string[]) =>
101
+ spawnSync("git", ["-C", cwd, ...args], { encoding: "utf8" });
102
+ /** Where `abs` sits in its git repository, or null outside one. */
103
+ const repoPath = (abs: string): { top: string; rel: string } | null => {
104
+ const top = git(dirname(abs), "rev-parse", "--show-toplevel");
105
+ return top.status === 0
106
+ ? {
107
+ top: top.stdout.trim(),
108
+ rel: relative(top.stdout.trim(), abs).split(sep).join("/"),
109
+ }
110
+ : null;
111
+ };
112
+ /** Unix times of the commits that touched `abs` — only the ones that ADDED it, with `added`. */
113
+ const commitTimes = (abs: string, added: boolean): number[] => {
114
+ const at = repoPath(abs);
115
+ const filter = added ? ["--diff-filter=A"] : [];
116
+ return at
117
+ ? git(at.top, "log", ...filter, "--format=%ct", "--", at.rel)
118
+ .stdout.trim()
119
+ .split("\n")
120
+ .filter(Boolean)
121
+ .map(Number)
122
+ : [];
123
+ };
124
+
125
+ const Codebook = z.looseObject({
126
+ labels: z
127
+ .array(
128
+ z.looseObject({
129
+ id: z.unknown().optional(),
130
+ definition: z.unknown().optional(),
131
+ borderline: z.unknown().optional(),
132
+ }),
133
+ )
134
+ .default([]),
135
+ });
136
+ const filled = (v: unknown): boolean =>
137
+ typeof v === "string" && v.trim() !== "";
138
+
139
+ /** Is the committed copy of `abs` the one on disk? A message when it is not. */
140
+ function committedProblem(file: string, abs: string, text: string): string[] {
141
+ const at = repoPath(abs);
142
+ const head = at ? git(at.top, "show", `HEAD:${at.rel}`) : null;
143
+ if (!head || head.status !== 0)
144
+ return [
145
+ `${file} is not committed: commit the codebook before the first label`,
146
+ ];
147
+ return head.stdout === text
148
+ ? []
149
+ : [
150
+ `${file} differs from its committed version: the codebook is fixed before labelling`,
151
+ ];
152
+ }
153
+
154
+ type CodebookLabel = z.infer<typeof Codebook>["labels"][number];
155
+
156
+ /** Labels that lack an id, a definition or a borderline example. */
157
+ const incomplete = (file: string, labels: readonly CodebookLabel[]): string[] =>
158
+ labels
159
+ .filter(
160
+ (l) => !(filled(l.id) && filled(l.definition) && filled(l.borderline)),
161
+ )
162
+ .map(
163
+ (l) =>
164
+ `${file}: label "${String(l.id)}" needs an id, a definition and a borderline example`,
165
+ );
166
+
167
+ function codebookProblems(
168
+ file: string,
169
+ labelsFile: string,
170
+ ): { known: Set<unknown>; problems: string[] } {
171
+ const abs = resolve(file);
172
+ const text = readFileSync(abs, "utf8");
173
+ const { labels } = Codebook.parse(JSON.parse(text));
174
+ // EVERY commit of the codebook, not only the one that added it: an edit after labelling began
175
+ // changes what the labels already given mean.
176
+ const bookLast = Math.max(-Infinity, ...commitTimes(abs, false));
177
+ const labelsFirst = Math.min(
178
+ Infinity,
179
+ ...commitTimes(resolve(labelsFile), true),
180
+ );
181
+ const late = bookLast > labelsFirst;
182
+ return {
183
+ known: new Set(labels.map((l) => l.id)),
184
+ problems: [
185
+ ...committedProblem(file, abs, text),
186
+ ...(labels.length ? [] : [`${file} defines no labels`]),
187
+ ...incomplete(file, labels),
188
+ ...(late
189
+ ? [
190
+ `${file} was committed after ${labelsFile} was first committed: the codebook is fixed before labelling; a changed codebook needs a new label file`,
191
+ ]
192
+ : []),
193
+ ],
194
+ };
195
+ }
196
+
197
+ const Attempt = z.looseObject({
198
+ attempt: z.string(),
199
+ agent: z.string().optional(),
200
+ model: z.string().optional(),
201
+ unit: z.string().optional(),
202
+ arm: z.string().optional(),
203
+ seed: z.union([z.string(), z.number()]).optional(),
204
+ status: z.string().optional(),
205
+ });
206
+ type Attempt = z.infer<typeof Attempt>;
207
+
208
+ /** The cells of the completed attempts in a run-agent-grid ledger. */
209
+ function ledgerCells(file: string): CellId[] {
210
+ const byId = new Map<string, Attempt>();
211
+ readFileSync(file, "utf8")
212
+ .split("\n")
213
+ .filter((l) => l.trim())
214
+ .map((l) => Attempt.parse(JSON.parse(l)))
215
+ .forEach((r) => {
216
+ byId.set(r.attempt, { ...byId.get(r.attempt), ...r });
217
+ });
218
+ return [...byId.values()]
219
+ .filter((r) => r.status === "complete")
220
+ .map((r) => ({
221
+ agent: String(r.agent),
222
+ model: String(r.model),
223
+ cell: `${String(r.unit)}/${String(r.arm)}/${String(r.seed)}`,
224
+ }));
225
+ }
226
+
227
+ /** The labels a revision of the label file held, by cell; null when it did not exist there. */
228
+ function labelsAt(labelsFile: string, rev: string): Map<string, string> | null {
229
+ const at = repoPath(resolve(labelsFile));
230
+ const then = at ? git(at.top, "show", `${rev}:${at.rel}`) : null;
231
+ if (!then || then.status !== 0) return null;
232
+ return new Map(
233
+ then.stdout
234
+ .split("\n")
235
+ .slice(1)
236
+ .filter((l) => l.trim())
237
+ .map((l) => l.split("\t"))
238
+ .map(([agent, model, cell, label]) => [
239
+ keyOf({
240
+ agent: String(agent),
241
+ model: String(model),
242
+ cell: String(cell),
243
+ }),
244
+ String(label),
245
+ ]),
246
+ );
247
+ }
248
+
249
+ /** The sections of the notes under a "## YYYY-MM-DD" heading, each up to the next "## ". */
250
+ const datedSections = (notesFile: string): string[] =>
251
+ readFileSync(notesFile, "utf8")
252
+ .split(/^(?=## )/m)
253
+ .filter((s) => /^## \d{4}-\d{2}-\d{2}\b/.test(s));
254
+
255
+ /** A cell key back as the three fields `show` prints. */
256
+ const fromKey = (key: string): CellId => {
257
+ const [agent, model, cell] = key.split("\t");
258
+ return { agent: String(agent), model: String(model), cell: String(cell) };
259
+ };
260
+
261
+ const WHY = " with no dated note quoting the run";
262
+
263
+ /** Rows whose label is new or different since the analysis, and not explained. */
264
+ const touchedSince = (
265
+ rows: readonly Row[],
266
+ before: ReadonlyMap<string, string>,
267
+ noted: (cell: string) => boolean,
268
+ ): string[] =>
269
+ rows
270
+ .filter((r) => before.get(keyOf(r)) !== r.label && !noted(r.cell))
271
+ .map((r) => {
272
+ const was = before.get(keyOf(r));
273
+ return was === undefined
274
+ ? `${show(r)}: added after the analysis (${r.label}),${WHY}`
275
+ : `${show(r)}: relabelled ${was} → ${r.label} after the analysis,${WHY}`;
276
+ });
277
+
278
+ /** Cells the analysis read that are gone now, and not explained. */
279
+ const removedSince = (
280
+ rows: readonly Row[],
281
+ before: ReadonlyMap<string, string>,
282
+ noted: (cell: string) => boolean,
283
+ ): string[] => {
284
+ const now = new Set(rows.map(keyOf));
285
+ return [...before]
286
+ .filter(([key]) => !now.has(key) && !noted(fromKey(key).cell))
287
+ .map(
288
+ ([key, was]) =>
289
+ `${show(fromKey(key))}: removed after the analysis (was ${was}),${WHY}`,
290
+ );
291
+ };
292
+
293
+ /**
294
+ * Every cell whose label differs from what the analysis read at `rev` — changed, ADDED after it,
295
+ * or REMOVED since — and is not explained in a dated, quoted note naming the cell.
296
+ */
297
+ function relabelProblems(
298
+ labelsFile: string,
299
+ rev: string,
300
+ notesFile: string,
301
+ rows: readonly Row[],
302
+ ): string[] {
303
+ const before = labelsAt(labelsFile, rev);
304
+ if (!before)
305
+ return [
306
+ `${labelsFile} does not exist at ${rev}: name the revision the analysis read`,
307
+ ];
308
+ const sections = datedSections(notesFile);
309
+ const noted = (cell: string): boolean =>
310
+ sections.some((s) => s.includes(cell) && /^> \S/m.test(s));
311
+ return [
312
+ ...touchedSince(rows, before, noted),
313
+ ...removedSince(rows, before, noted),
314
+ ];
315
+ }
316
+
317
+ /** Cells a file labels more than once, with the lines. */
318
+ function duplicates(file: string, rows: readonly Row[]): string[] {
319
+ const seen = new Map<string, number[]>();
320
+ rows.forEach((r) => seen.set(keyOf(r), [...(seen.get(keyOf(r)) ?? []), r.n]));
321
+ return [...seen]
322
+ .filter(([, ns]) => ns.length > 1)
323
+ .map(
324
+ ([k, ns]) =>
325
+ `${file}: ${show(fromKey(k))} is labelled ${String(ns.length)} times (lines ${ns.join(", ")})`,
326
+ );
327
+ }
328
+
329
+ /** Rows naming a cell twice, and coverage against the ledger. */
330
+ function rowProblems(file: string, rows: readonly Row[], ledger?: string) {
331
+ const runs = ledger === undefined ? null : ledgerCells(ledger);
332
+ const ran = new Set((runs ?? []).map(keyOf));
333
+ const labelled = new Set(rows.map(keyOf));
334
+ return [
335
+ ...duplicates(file, rows),
336
+ ...(runs
337
+ ? [
338
+ ...rows
339
+ .filter((r) => !ran.has(keyOf(r)))
340
+ .map(
341
+ (r) =>
342
+ `${file}:${String(r.n)}: ${show(r)} has no completed run in the ledger`,
343
+ ),
344
+ ...runs
345
+ .filter((r) => !labelled.has(keyOf(r)))
346
+ .map((r) => `${show(r)}: a completed run with no label`),
347
+ ]
348
+ : []),
349
+ ];
350
+ }
351
+
352
+ function validate(file: string, o: Values): void {
353
+ const { rows, problems } = readLabels(file);
354
+ const book = codebookProblems(String(o.codebook), file);
355
+ const all = [
356
+ ...problems,
357
+ ...book.problems,
358
+ ...rows
359
+ .filter((r) => !book.known.has(r.label))
360
+ .map(
361
+ (r) => `${file}:${String(r.n)}: "${r.label}" is not a codebook label`,
362
+ ),
363
+ ...rowProblems(file, rows, o.ledger),
364
+ ...(o["analysed-at"] === undefined
365
+ ? []
366
+ : relabelProblems(file, o["analysed-at"], String(o.notes), rows)),
367
+ ];
368
+ if (all.length) refuse(1, all.join("\n"));
369
+ say(`${file}: ${String(rows.length)} labels, valid`);
370
+ }
371
+
372
+ // ── agreement ───────────────────────────────────────────────────────────────────────────────
373
+ const Z = 1.959963984540054;
374
+ /** 95% Wilson score interval for k successes in n. */
375
+ const wilson = (k: number, n: number): readonly [number, number] => {
376
+ const p = k / n;
377
+ const denom = 1 + (Z * Z) / n;
378
+ const centre = (p + (Z * Z) / (2 * n)) / denom;
379
+ const half =
380
+ (Z * Math.sqrt((p * (1 - p)) / n + (Z * Z) / (4 * n * n))) / denom;
381
+ return [centre - half, centre + half];
382
+ };
383
+
384
+ /** Cohen's kappa over paired labels; null when chance agreement is 1 (one label only). */
385
+ function cohenKappa(
386
+ pairs: readonly (readonly [string, string])[],
387
+ ): number | null {
388
+ const n = pairs.length;
389
+ const po = pairs.filter(([a, b]) => a === b).length / n;
390
+ const labels = [...new Set(pairs.flat())];
391
+ const share = (i: 0 | 1, l: string): number =>
392
+ pairs.filter((p) => p[i] === l).length / n;
393
+ const pe = labels.reduce((s, l) => s + share(0, l) * share(1, l), 0);
394
+ return pe === 1 ? null : (po - pe) / (1 - pe);
395
+ }
396
+
397
+ /** Cells of either file that the other does not label. */
398
+ const onlyIn = (
399
+ fileA: string,
400
+ a: readonly Row[],
401
+ b: readonly Row[],
402
+ fileB: string,
403
+ ): string[] => {
404
+ const inA = new Set(a.map(keyOf));
405
+ const inB = new Set(b.map(keyOf));
406
+ return [
407
+ ...a
408
+ .filter((r) => !inB.has(keyOf(r)))
409
+ .map((r) => `${show(r)}: only in ${fileA}`),
410
+ ...b
411
+ .filter((r) => !inA.has(keyOf(r)))
412
+ .map((r) => `${show(r)}: only in ${fileB}`),
413
+ ];
414
+ };
415
+
416
+ /** The second file's label for each cell of the first; a refusal when the cells differ. */
417
+ function matched(
418
+ fileA: string,
419
+ fileB: string,
420
+ ): readonly (readonly [Row, string])[] {
421
+ const a = readLabels(fileA);
422
+ const b = readLabels(fileB);
423
+ // A cell labelled twice has no single label to compare: refused before any statistic.
424
+ const bad = [
425
+ ...a.problems,
426
+ ...b.problems,
427
+ ...duplicates(fileA, a.rows),
428
+ ...duplicates(fileB, b.rows),
429
+ ];
430
+ if (bad.length) refuse(1, bad.join("\n"));
431
+ const inB = new Map(b.rows.map((r) => [keyOf(r), r.label]));
432
+ const unmatched = onlyIn(fileA, a.rows, b.rows, fileB);
433
+ if (unmatched.length)
434
+ refuse(
435
+ 1,
436
+ `the two files label different cells — rate the same runs:\n${unmatched.join("\n")}`,
437
+ );
438
+ if (a.rows.length === 0) refuse(1, "no labels to compare");
439
+ return a.rows.map((r) => [r, String(inB.get(keyOf(r)))] as const);
440
+ }
441
+
442
+ function agree(fileA: string, fileB: string): void {
443
+ const rows = matched(fileA, fileB);
444
+ const pairs = rows.map(([r, other]) => [r.label, other] as const);
445
+ const n = pairs.length;
446
+ const k = pairs.filter(([x, y]) => x === y).length;
447
+ const [lo, hi] = wilson(k, n);
448
+ const kappa = cohenKappa(pairs);
449
+ const f = (x: number): string => x.toFixed(3);
450
+ say(
451
+ [
452
+ `n ${String(n)}`,
453
+ `agreement ${String(k)}/${String(n)} = ${f(k / n)} (95% Wilson ${f(lo)}–${f(hi)})`,
454
+ `cohen kappa ${kappa === null ? "undefined (one label only)" : f(kappa)}`,
455
+ ...rows
456
+ .filter(([r, other]) => r.label !== other)
457
+ .map(([r, other]) => `disagree ${show(r)}: ${r.label} | ${other}`),
458
+ ].join("\n"),
459
+ );
460
+ }
461
+
462
+ // ── command line ────────────────────────────────────────────────────────────────────────────
463
+ const OPTIONS = {
464
+ codebook: { type: "string" },
465
+ ledger: { type: "string" },
466
+ "analysed-at": { type: "string" },
467
+ notes: { type: "string" },
468
+ } as const;
469
+ type Values = ReturnType<
470
+ typeof parseArgs<{ options: typeof OPTIONS; allowPositionals: true }>
471
+ >["values"];
472
+
473
+ const USAGE =
474
+ "usage: labels.js validate <labels.tsv> --codebook <f> [--ledger <f>] [--analysed-at <rev> --notes <f>] | agree <a.tsv> <b.tsv>";
475
+
476
+ function main(): void {
477
+ const { values, positionals } = (() => {
478
+ try {
479
+ return parseArgs({ options: OPTIONS, allowPositionals: true });
480
+ } catch (e) {
481
+ return refuse(2, `${String(prop(e, "message"))}\n${USAGE}`);
482
+ }
483
+ })();
484
+ const [cmd, ...paths] = positionals;
485
+ const validateOk =
486
+ cmd === COMMANDS.validate &&
487
+ paths.length === 1 &&
488
+ values.codebook !== undefined &&
489
+ (values["analysed-at"] === undefined || values.notes !== undefined);
490
+ if (validateOk) validate(String(paths[0]), values);
491
+ else if (cmd === COMMANDS.agree && paths.length === 2)
492
+ agree(String(paths[0]), String(paths[1]));
493
+ else refuse(2, USAGE);
494
+ }
495
+
496
+ try {
497
+ main();
498
+ } catch (e) {
499
+ process.stderr.write(`labels: ${String(prop(e, "message"))}\n`);
500
+ process.exitCode = e instanceof Refusal ? e.exitCode : 1;
501
+ }
@@ -1,11 +1,11 @@
1
1
  ---
2
2
  name: map-prior-work
3
- description: Sweep the competitive landscape for a paper idea BEFORE drafting — find everyone already working on it, date each against your submission (prior vs concurrent is a five-month question, not a vibe), triage by whether they threaten your claimed contribution, deep-read the ones that do, and emit a verdict on WHAT YOU CAN STILL CLAIM plus a related-work skeleton. Saves the full map, not a summary. Use right after research-ideate and before draft-paper, and re-run as a cheap delta pass before submit (hot fields move). Distinct from analyze-sibling-paper (one known competitor, deep) — this one FINDS the competitors and decides which deserve that treatment; distinct from verify-citations (checks cites you already wrote, too late to reshape a contribution); distinct from study-accepted-papers (mines a venue's bar, not your rivals). Compose with research-ideate (upstream), analyze-sibling-paper (it hands off), draft-paper and find-venue (downstream).
3
+ description: Sweep the competitive landscape for a paper idea BEFORE drafting — find everyone already working on it, date each against your submission (prior vs concurrent is a five-month question, not a vibe), triage by whether they threaten your claimed contribution, deep-read the ones that do, and emit a verdict on WHAT YOU CAN STILL CLAIM plus a related-work skeleton. Saves the full map, not a summary. Use right after research-ideate and before draft-paper, and re-run as a cheap delta pass before submit (hot fields move). Distinct from analyze-sibling-paper (one known competitor, deep) — this one FINDS the competitors and decides which deserve that treatment; distinct from verify-citations (checks cites you already wrote, too late to reshape a contribution); distinct from study-accepted-papers (mines what a venue rewards, not your rivals). Compose with research-ideate (upstream), analyze-sibling-paper (it hands off), draft-paper and find-venue (downstream).
4
4
  context: fork
5
5
  allowed-tools: [WebSearch, WebFetch, Read, Write, Grep, Glob, Bash, Agent]
6
6
  ---
7
7
 
8
- <!-- vigiles:sha256:55971644308c470f compiled from skills/map-prior-work/SKILL.md.spec.ts -->
8
+ <!-- vigiles:sha256:474b3d7b8e0542ae compiled from skills/map-prior-work/SKILL.md.spec.ts -->
9
9
 
10
10
  # map-prior-work — find out who already did it, while you can still change course
11
11
 
@@ -9,7 +9,7 @@ import { experimental_skill } from "vigiles/spec";
9
9
  export default experimental_skill({
10
10
  name: "map-prior-work",
11
11
  description:
12
- "Sweep the competitive landscape for a paper idea BEFORE drafting — find everyone already working on it, date each against your submission (prior vs concurrent is a five-month question, not a vibe), triage by whether they threaten your claimed contribution, deep-read the ones that do, and emit a verdict on WHAT YOU CAN STILL CLAIM plus a related-work skeleton. Saves the full map, not a summary. Use right after research-ideate and before draft-paper, and re-run as a cheap delta pass before submit (hot fields move). Distinct from analyze-sibling-paper (one known competitor, deep) — this one FINDS the competitors and decides which deserve that treatment; distinct from verify-citations (checks cites you already wrote, too late to reshape a contribution); distinct from study-accepted-papers (mines a venue's bar, not your rivals). Compose with research-ideate (upstream), analyze-sibling-paper (it hands off), draft-paper and find-venue (downstream).",
12
+ "Sweep the competitive landscape for a paper idea BEFORE drafting — find everyone already working on it, date each against your submission (prior vs concurrent is a five-month question, not a vibe), triage by whether they threaten your claimed contribution, deep-read the ones that do, and emit a verdict on WHAT YOU CAN STILL CLAIM plus a related-work skeleton. Saves the full map, not a summary. Use right after research-ideate and before draft-paper, and re-run as a cheap delta pass before submit (hot fields move). Distinct from analyze-sibling-paper (one known competitor, deep) — this one FINDS the competitors and decides which deserve that treatment; distinct from verify-citations (checks cites you already wrote, too late to reshape a contribution); distinct from study-accepted-papers (mines what a venue rewards, not your rivals). Compose with research-ideate (upstream), analyze-sibling-paper (it hands off), draft-paper and find-venue (downstream).",
13
13
  context: "fork",
14
14
  tools: [
15
15
  "WebSearch",
@@ -4,7 +4,7 @@ description: The orchestrator for writing a research paper end-to-end, from idea
4
4
  allowed-tools: [Read, Write, Edit, Grep, Glob, Bash, WebSearch, WebFetch, Agent, Skill]
5
5
  ---
6
6
 
7
- <!-- vigiles:sha256:67a3989629952dc1 compiled from skills/paper-pipeline/SKILL.md.spec.ts -->
7
+ <!-- vigiles:sha256:da7c692384fb79aa compiled from skills/paper-pipeline/SKILL.md.spec.ts -->
8
8
 
9
9
  # paper-pipeline — the conductor for the whole organism
10
10
 
@@ -150,7 +150,7 @@ Ordered, because each really does gate the next. Two pairs run in parallel; that
150
150
  | **Validate the idea** | **`research-ideate`** | Is it a reusable method (survives review) in your lane, and does it feed the credit criteria? Go/no-go + the MVP finding. |
151
151
  | **Map the competition** ‖ | **`map-prior-work`** | 🔴 **Who already did this?** Multi-angle sweep; date each hit against your submission (5 months = prior, not concurrent); triage by which layer of your claim it threatens; deep-read the dangerous ones. Output: **what you can still claim** + a related-work skeleton. *Added 2026-07-27 after a five-month-older paper surfaced at T−10 days on a finished draft.* |
152
152
  | **Pick the venue** ‖ | **`find-venue`** | Rank by credit-weight (peer-reviewed + indexed) × fit × accept-odds × deadline × remote. |
153
- | **Learn the venue's bar** | **`study-accepted-papers`** | 🔴 **What does an ACCEPTED paper look like HERE?** Measure the venue's whole accepted volume, deep-read 8–12 of your contribution type, and extract what made them strong: evidence scale, figures, artifact release, how the abstract is built. File the venue card (`presets/<venue>.md` in the package, or the project's own beside its preset) and confirm family rules (`references/acl-venue-rules.md`). *Promoted to its own step 2026-08-03: on `compile-rules-2026` every judgement of the paper's quality — rubric, persona, simulated panel, accept probability — came from instruments this pipeline itself generated, with no external grounding, and the author had to say so at T−3 days. A closed loop optimises against its own model of a reviewer.* |
153
+ | **Learn what the venue rewards** | **`study-accepted-papers`** (mode A idea, B design) | 🔴 **What does this venue REWARD, before the study is designed?** The venue's award winners against its ordinary accepted papers over 5–7 years (plus award papers on the topic at other venues, labelled apart), coded for substance as well as form, written as **requirements for the study**: what the oracle under the headline is, which belief stated on record the claim targets, which factor is a column. The corpus goes to `<papers-root>/venues/<venue>/`, shared by every paper for that venue; the stage is row `venuebar`. **The same skill comes back once text exists** (mode C draft: the strong-accept scorecard, M1–M10, in `PIPELINE-STATUS.md`; mode D final: the abstract and introduction). The skill always asks which mode. Venue format facts go in its preset (`presets/<venue>.jsonc` in the package, or the project's own) and the venue card beside it (`presets/<venue>.md`); confirm family rules (`references/acl-venue-rules.md`). *Promoted to its own step 2026-08-03: every judgement of a paper's quality had come from instruments this pipeline itself generated, with no external grounding. Moved to modes on 2026-10-12 (#207): run only on a draft, its main findings arrived after they could change the design.* |
154
154
  | 🔴 **Can you physically submit?** | **`plan-paper-timeline`** | **The step that owns the question no stage used to own.** Create the portal account NOW; drive the profile to **ACTIVE**, not "pending moderation"; confirm the form's fields and the artifact-hosting requirement. OpenReview moderation runs **up to two weeks**, and an author with no institutional email has no expedite route. Scorecard row `access`. *Added 2026-08-03, when this was the sole critical-path item on a finished paper at T−4 days: the science was done and there was nowhere to put it.* |
155
155
  | **Schedule it** | **`plan-paper-timeline`** | Deadlines → calendar with buffers, in the author's own zone (AoE − 12h ≠ your clock). Space papers apart. |
156
156
  | 🔴 **State the claim** | **`argument-arc`** (frame mode) | **One paragraph: what will this paper claim, and what would have to be true?** Before any run. The frame decides which results matter, and reframing after the data is collected is how experiments get thrown away — observed on `compile-rules-2026`, and the reason `argument-arc` has a rebuild mode at all. Costs a paragraph; saves a study. Scorecard row `frame`. |
@@ -170,6 +170,15 @@ No numbers. Nothing here happens once, and nothing here is "next".
170
170
 
171
171
  - **`build-benchmark`** — a design that inverts the pitfall; honest stats; a self-checking artifact
172
172
  (`references/artifact-checklist.md`).
173
+ - **`run-agent-grid`** — when the study is a grid of agent runs (units × arms × seeds, one agent CLI
174
+ run per cell): the registry and the author's approval gate every counted run, a sandbox and secret
175
+ probe, a smoke per model, a locked and resumable driver, an overlap audit, finished runs committed.
176
+ Between the design and the analysis.
177
+ - **`label-agent-runs`** — when the runs are in and before any outcome number: a codebook fixed
178
+ first, the automatic classifier only a pre-sort, every uncertain run read and labelled by hand, a
179
+ blind second rater with its agreement, labels frozen after the analysis.
180
+ - **`build-label-page`** — when the study needs a person to label items (gold labels, a blind review):
181
+ a phone page with a shared answer store, the answers exported into the study's `data/`.
173
182
  - **`draft-paper`** — section skeleton + the accept-and-get-cited properties
174
183
  (`references/writing-craft.md`).
175
184
  - **`argument-arc`** — 🔴 **does the draft carry the reader to ONE conclusion?** One sentence per
@@ -221,7 +230,7 @@ Use an atom for a quick spot-check; **run the panel for the decision.**
221
230
  skill: 🔴 **argument architecture** — does the whole thing carry the reader to one conclusion
222
231
  (`argument-arc`, in the LOOP; the axis that failed on `compile-rules-2026` precisely because it was
223
232
  nobody's) · **content strength** — does the contribution clear *this venue's* bar (`study-accepted-papers`,
224
- mines the accepted corpus for Accept→Strong levers; in SETUP, before the prose is set) ·
233
+ from the venue's own award winners: in SETUP as requirements for the study, and again on the draft as the strong-accept scorecard, M1–M10) ·
225
234
  **scientific defects** — what a hostile reviewer attacks (`paper-adversarial-review` / `pc-panel-review`)
226
235
  · **writing craft** — does it read like a human wrote it (`grade-paper-writing`, the 10-dim rubric in
227
236
  `references/writing-craft.md`; called by `harden-paper`). A paper can ace one and fail another — grade all four.
@@ -294,7 +303,7 @@ Over {{corpus.n}} real rules files, with the extractor's proposed mapping
294
303
  taken as if an author had written it, {{annotated.failedContradicted}} fail admission.
295
304
  ```
296
305
 
297
- `<paper>/repro/numbers.tsv` gives each name its value, the JSON path or **formula** that recomputes
306
+ `<paper>/repro/data/numbers.tsv` gives each name its value, the JSON path or **formula** that recomputes
298
307
  it, and two guards. The build refuses a PDF unless: the name resolves · the value still matches the
299
308
  data · the **paragraph** names the arm (`requires`) · the **sentence** does not claim a different one
300
309
  (`forbids`). Requires and forbids are separate on purpose — the pairing is auditing's, where a control
@@ -310,7 +319,7 @@ binds every row from now on, and the existing rows are *enumerated rather than e
310
319
  says out loud how many figures still have no machine-checked path to data. On the paper that has it:
311
320
  7 guarded, 248 grandfathered. That ratio is a finding, not a score.
312
321
 
313
- **Adopting it in another paper:** copy `repro/paper_numbers.py`, `repro/paper_numbers.selftest.py`
322
+ **Adopting it in another paper:** copy `repro/tools/paper_numbers.py`, `repro/tools/paper_numbers.selftest.py`
314
323
  and the build step that calls `expand()` on `paper.tex` before `pdflatex`; generate the
315
324
  grandfather list from the current draft; register the numbers a reviewer will actually check first.
316
325
 
@@ -350,7 +359,7 @@ printing two unrelated 22s broke it the same hour. Key on the occurrence.
350
359
  🔴 **Hand the gate to a red team the same day you build it.** This one was, and seven holes came
351
360
  back, the worst being that the original defect could simply be **retyped as digits** — its value was
352
361
  grandfathered, so the guard applied to the macro and not to the claim. Every hole is now a case in
353
- `repro/paper_numbers.selftest.py` (18/18, including three that must stay **quiet** — a gate that
362
+ `repro/tools/paper_numbers.selftest.py` (18/18, including three that must stay **quiet** — a gate that
354
363
  fires on ordinary writing gets muted, and then its silence reads as coverage).
355
364
 
356
365
  ## 🔴 The page limit is a SUBMISSION gate, not an EDIT gate