paperlint 9.0.1 → 10.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +16 -4
- package/README.md +68 -128
- package/dist/build.d.ts +1 -1
- package/dist/build.js +1 -1
- package/dist/cli.d.ts +5 -1
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +52 -6
- package/dist/cli.js.map +1 -1
- package/dist/domain/layout-tree.d.ts +19 -0
- package/dist/domain/layout-tree.d.ts.map +1 -0
- package/dist/domain/layout-tree.js +32 -0
- package/dist/domain/layout-tree.js.map +1 -0
- package/dist/domain/lookup-cache.d.ts +2 -2
- package/dist/domain/lookup-cache.js +2 -2
- package/dist/domain/paper-layout.d.ts +81 -0
- package/dist/domain/paper-layout.d.ts.map +1 -0
- package/dist/domain/paper-layout.js +259 -0
- package/dist/domain/paper-layout.js.map +1 -0
- package/dist/domain/repro-layout.d.ts +80 -0
- package/dist/domain/repro-layout.d.ts.map +1 -0
- package/dist/domain/repro-layout.js +247 -0
- package/dist/domain/repro-layout.js.map +1 -0
- package/dist/domain/scaffold-layout.d.ts +28 -0
- package/dist/domain/scaffold-layout.d.ts.map +1 -0
- package/dist/domain/scaffold-layout.js +81 -0
- package/dist/domain/scaffold-layout.js.map +1 -0
- package/dist/eslint-rules/js-needs-reason.d.ts +55 -0
- package/dist/eslint-rules/js-needs-reason.d.ts.map +1 -0
- package/dist/eslint-rules/js-needs-reason.js +35 -0
- package/dist/eslint-rules/js-needs-reason.js.map +1 -0
- package/dist/eslint-rules/prereg-coverage.d.ts +98 -0
- package/dist/eslint-rules/prereg-coverage.d.ts.map +1 -0
- package/dist/eslint-rules/prereg-coverage.js +274 -0
- package/dist/eslint-rules/prereg-coverage.js.map +1 -0
- package/dist/folder-layout-hook.d.ts +21 -0
- package/dist/folder-layout-hook.d.ts.map +1 -0
- package/dist/folder-layout-hook.js +117 -0
- package/dist/folder-layout-hook.js.map +1 -0
- package/dist/folder-layout-rule.d.ts +22 -0
- package/dist/folder-layout-rule.d.ts.map +1 -0
- package/dist/folder-layout-rule.js +107 -0
- package/dist/folder-layout-rule.js.map +1 -0
- package/dist/init.js +1 -1
- package/dist/init.js.map +1 -1
- package/dist/new-paper.d.ts +4 -1
- package/dist/new-paper.d.ts.map +1 -1
- package/dist/new-paper.js +40 -6
- package/dist/new-paper.js.map +1 -1
- package/dist/package-dirs.d.ts +6 -0
- package/dist/package-dirs.d.ts.map +1 -1
- package/dist/package-dirs.js +6 -0
- package/dist/package-dirs.js.map +1 -1
- package/dist/references.d.ts +1 -1
- package/dist/references.d.ts.map +1 -1
- package/dist/references.js +2 -2
- package/dist/references.js.map +1 -1
- package/dist/repro-layout-rule.d.ts +16 -0
- package/dist/repro-layout-rule.d.ts.map +1 -0
- package/dist/repro-layout-rule.js +81 -0
- package/dist/repro-layout-rule.js.map +1 -0
- package/dist/skills/label-agent-runs/scripts/labels-commands.d.ts +13 -0
- package/dist/skills/label-agent-runs/scripts/labels.d.ts +2 -0
- package/dist/skills/run-agent-grid/scripts/grid-guard-commands.d.ts +20 -0
- package/dist/skills/run-agent-grid/scripts/grid-guard.d.ts +2 -0
- package/docs/ci.md +60 -0
- package/docs/configuration.md +2 -0
- package/docs/e2e.md +2 -2
- package/docs/install.md +20 -16
- package/docs/layout.md +154 -0
- package/docs/optional-rules.md +5 -4
- package/docs/references.md +5 -2
- package/docs/rules/paper/allowed-files-only.md +192 -0
- package/docs/rules/repro/allowed-files-only.md +128 -0
- package/docs/rules/study/declare-sample.md +146 -0
- package/docs/rules.md +53 -50
- package/docs/skills.md +8 -3
- package/docs/testing.md +13 -0
- package/docs/venues.md +94 -0
- package/eslint-rules/js-needs-reason.test.ts +141 -0
- package/eslint-rules/js-needs-reason.ts +77 -0
- package/eslint-rules/prereg-coverage.test.ts +448 -0
- package/eslint-rules/prereg-coverage.ts +419 -0
- package/eslint.config.mjs +22 -2
- package/lib/skill-trigger-cases.harness.mjs +51 -5
- package/lib/skill-trigger-cases.mjs +53 -15
- package/lib/skill-trigger-cases.test.ts +211 -0
- package/package.json +8 -4
- package/plugin/hooks/hooks.json +9 -0
- package/scripts/coverage-config.test.ts +5 -3
- package/skills/analyze-sibling-paper/SKILL.md +2 -2
- package/skills/analyze-sibling-paper/SKILL.md.spec.ts +1 -1
- package/skills/build-benchmark/SKILL.md +17 -1
- package/skills/build-benchmark/SKILL.md.spec.ts +16 -0
- package/skills/build-label-page/SKILL.md +120 -0
- package/skills/build-label-page/SKILL.md.spec.ts +131 -0
- package/skills/build-label-page/build-label-page.eval.mjs +15 -0
- package/skills/build-label-page/build-label-page.test.ts +959 -0
- package/skills/build-label-page/references/checklist.md +148 -0
- package/skills/build-label-page/references/example-generator.py +803 -0
- package/skills/find-venue/SKILL.md +5 -3
- package/skills/find-venue/SKILL.md.spec.ts +4 -2
- package/skills/grade-paper-writing/SKILL.md +7 -7
- package/skills/grade-paper-writing/SKILL.md.spec.ts +6 -6
- package/skills/label-agent-runs/SKILL.md +103 -0
- package/skills/label-agent-runs/SKILL.md.spec.ts +106 -0
- package/skills/label-agent-runs/label-agent-runs.eval.ts +55 -0
- package/skills/label-agent-runs/label-agent-runs.harness.ts +123 -0
- package/skills/label-agent-runs/references/formats.md +64 -0
- package/skills/label-agent-runs/scripts/labels-commands.js +13 -0
- package/skills/label-agent-runs/scripts/labels-commands.ts +13 -0
- package/skills/label-agent-runs/scripts/labels.js +396 -0
- package/skills/label-agent-runs/scripts/labels.test.ts +538 -0
- package/skills/label-agent-runs/scripts/labels.ts +501 -0
- package/skills/map-prior-work/SKILL.md +2 -2
- package/skills/map-prior-work/SKILL.md.spec.ts +1 -1
- package/skills/paper-pipeline/SKILL.md +15 -6
- package/skills/paper-pipeline/SKILL.md.spec.ts +14 -5
- package/skills/paper-pipeline/grade-paper-writing-ablation.eval.mjs +19 -9
- package/skills/paper-pipeline/references/acceptance-gate.md +19 -19
- package/skills/paper-pipeline/references/artifact-checklist.md +5 -0
- package/skills/paper-pipeline/references/pipeline-status-template.md +30 -1
- package/skills/paper-pipeline/repro/2026-08-07-fork-check.log +1 -1
- package/skills/paper-pipeline/repro/2026-08-07-fork-check2.log +1 -1
- package/skills/paper-pipeline/repro/2026-08-07-language-eval-pilot.log +1 -1
- package/skills/paper-pipeline/repro/2026-08-07-language-eval-raw.log +1 -1
- package/skills/paper-pipeline/repro/2026-08-08-framing-vs-vocabulary-oracle.log +48 -48
- package/skills/paper-pipeline/repro/2026-08-08-grade-paper-writing-ablation-A6-oracle.log +8 -8
- package/skills/paper-pipeline/repro/2026-08-08-grade-paper-writing-ablation-oracle.log +48 -48
- package/skills/paper-pipeline/repro/2026-08-08-parent-replication-gpw.log +2 -2
- package/skills/paper-pipeline/repro/2026-08-08-parent-replication.mjs +3 -3
- package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-ablation-preflight-after.log +81 -0
- package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-ablation-preflight-before.log +81 -0
- package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-ablation-setupdiff.log +59 -0
- package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-trigger.jsonl +1 -0
- package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-trigger.log +22 -0
- package/skills/paper-pipeline/repro/README.md +4 -0
- package/skills/paper-pipeline/scripts/extract-ref-facts.harness.mjs +2 -2
- package/skills/paper-pipeline/scripts/extract-ref-facts.mjs +2 -2
- package/skills/paper-pipeline/scripts/extract-ref-facts.test.mjs +4 -2
- package/skills/paper-pipeline/scripts/generated-code.harness.mjs +7 -2
- package/skills/paper-pipeline/scripts/generated-code.mjs +8 -4
- package/skills/paper-pipeline/scripts/generated-code.test.ts +15 -1
- package/skills/paper-status/SKILL.md +2 -2
- package/skills/paper-status/SKILL.md.spec.ts +1 -1
- package/skills/run-agent-grid/SKILL.md +181 -0
- package/skills/run-agent-grid/SKILL.md.spec.ts +184 -0
- package/skills/run-agent-grid/references/failure-catalog.md +36 -0
- package/skills/run-agent-grid/references/registry-and-ledger.md +142 -0
- package/skills/run-agent-grid/run-agent-grid.eval.ts +59 -0
- package/skills/run-agent-grid/run-agent-grid.harness.ts +115 -0
- package/skills/run-agent-grid/scripts/grid-guard-commands.js +18 -0
- package/skills/run-agent-grid/scripts/grid-guard-commands.ts +21 -0
- package/skills/run-agent-grid/scripts/grid-guard.js +700 -0
- package/skills/run-agent-grid/scripts/grid-guard.test.ts +1090 -0
- package/skills/run-agent-grid/scripts/grid-guard.ts +869 -0
- package/skills/study-accepted-papers/SKILL.md +163 -161
- package/skills/study-accepted-papers/SKILL.md.spec.ts +167 -165
- package/skills/study-accepted-papers/references/base-rates.md +69 -0
- package/skills/study-accepted-papers/references/mechanisms.md +181 -0
- package/skills/verify-citations/SKILL.md +2 -2
- package/skills/verify-citations/SKILL.md.spec.ts +1 -1
- package/src/build.ts +1 -1
- package/src/cli.harness.mjs +42 -0
- package/src/cli.test.ts +90 -6
- package/src/cli.ts +78 -5
- package/src/domain/layout-tree.test.ts +32 -0
- package/src/domain/layout-tree.ts +47 -0
- package/src/domain/lookup-cache.ts +2 -2
- package/src/domain/paper-layout.test.ts +264 -0
- package/src/domain/paper-layout.ts +336 -0
- package/src/domain/repro-layout.test.ts +222 -0
- package/src/domain/repro-layout.ts +311 -0
- package/src/domain/scaffold-layout.ts +104 -0
- package/src/folder-layout-hook.test.ts +214 -0
- package/src/folder-layout-hook.ts +141 -0
- package/src/folder-layout-rule.test.ts +235 -0
- package/src/folder-layout-rule.ts +151 -0
- package/src/hooks-settings.harness.ts +9 -8
- package/src/init.ts +1 -1
- package/src/layout-docs.test.ts +52 -0
- package/src/new-paper.harness.mjs +23 -1
- package/src/new-paper.test.ts +299 -6
- package/src/new-paper.ts +51 -5
- package/src/package-dirs.ts +8 -0
- package/src/paper-files.test.ts +9 -6
- package/src/references.test.ts +5 -5
- package/src/references.ts +2 -2
- package/src/repro-layout-rule.test.ts +114 -0
- package/src/repro-layout-rule.ts +121 -0
- package/src/rule-cases.test.ts +98 -0
- package/templates/paper/PIPELINE-STATUS.md +41 -7
- package/tsconfig.base.json +22 -0
- package/tsconfig.json +6 -24
- package/tsconfig.pkg.json +3 -1
- package/tsconfig.skills.json +14 -0
- package/tsconfig.src.json +12 -0
|
@@ -0,0 +1,501 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* labels — the checks of the label-agent-runs skill.
|
|
4
|
+
*
|
|
5
|
+
* validate <labels.tsv> --codebook <codebook.json> [--ledger <attempts.jsonl>]
|
|
6
|
+
* [--analysed-at <git rev> --notes <notes.md>]
|
|
7
|
+
* The label file is well formed, uses only codebook labels, names each cell once, and —
|
|
8
|
+
* with a ledger — covers exactly the completed runs. The codebook is committed, unchanged,
|
|
9
|
+
* and was committed no later than the label file. With --analysed-at: every cell whose
|
|
10
|
+
* label differs from that revision is explained in a dated section of the notes, with a
|
|
11
|
+
* verbatim quote.
|
|
12
|
+
* agree <a.tsv> <b.tsv>
|
|
13
|
+
* Agreement between two label files over the same cells: n, percent agreement with a 95%
|
|
14
|
+
* Wilson interval, Cohen's kappa, and every disagreement. Use it for the automatic pre-sort
|
|
15
|
+
* against the hand labels, and for one rater against another.
|
|
16
|
+
*
|
|
17
|
+
* A label file is TSV with exactly this header: agent, model, cell, label. `cell` is
|
|
18
|
+
* `<unit>/<arm>/<seed>`, the cell of the run-agent-grid ledger.
|
|
19
|
+
*
|
|
20
|
+
* TypeScript, compiled beside itself to `labels.js` by `npm run build`: a consumer runs the skill
|
|
21
|
+
* from `node_modules`, where Node does not strip types. The skill names the `.js`.
|
|
22
|
+
*
|
|
23
|
+
* EXIT CODES: 0 valid / reported · 1 a finding · 2 usage.
|
|
24
|
+
*/
|
|
25
|
+
import { spawnSync } from "node:child_process";
|
|
26
|
+
import { readFileSync } from "node:fs";
|
|
27
|
+
import { dirname, relative, resolve, sep } from "node:path";
|
|
28
|
+
import { parseArgs } from "node:util";
|
|
29
|
+
import { z } from "zod";
|
|
30
|
+
import { COMMANDS } from "./labels-commands.ts";
|
|
31
|
+
|
|
32
|
+
class Refusal extends Error {
|
|
33
|
+
readonly exitCode: number;
|
|
34
|
+
constructor(exitCode: number, message: string) {
|
|
35
|
+
super(message);
|
|
36
|
+
this.exitCode = exitCode;
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
const refuse = (exitCode: number, message: string): never => {
|
|
40
|
+
throw new Refusal(exitCode, message);
|
|
41
|
+
};
|
|
42
|
+
const say = (line: string): void => {
|
|
43
|
+
process.stdout.write(`${line}\n`);
|
|
44
|
+
};
|
|
45
|
+
const prop = (e: unknown, key: string): unknown => Reflect.get(Object(e), key);
|
|
46
|
+
|
|
47
|
+
const HEADER = ["agent", "model", "cell", "label"];
|
|
48
|
+
interface Row {
|
|
49
|
+
readonly agent: string;
|
|
50
|
+
readonly model: string;
|
|
51
|
+
readonly cell: string;
|
|
52
|
+
readonly label: string;
|
|
53
|
+
readonly n: number;
|
|
54
|
+
}
|
|
55
|
+
type CellId = Pick<Row, "agent" | "model" | "cell">;
|
|
56
|
+
const keyOf = (r: CellId): string => `${r.agent}\t${r.model}\t${r.cell}`;
|
|
57
|
+
const show = (r: CellId): string => `${r.agent} ${r.model} ${r.cell}`;
|
|
58
|
+
|
|
59
|
+
interface Line {
|
|
60
|
+
readonly fields: readonly string[];
|
|
61
|
+
readonly n: number;
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
/** A line with exactly four fields as a row; anything else is not one. */
|
|
65
|
+
const asRow = ({ fields, n }: Line): Row[] => {
|
|
66
|
+
const [agent, model, cell, label] = fields;
|
|
67
|
+
return fields.length === 4
|
|
68
|
+
? [
|
|
69
|
+
{
|
|
70
|
+
agent: String(agent),
|
|
71
|
+
model: String(model),
|
|
72
|
+
cell: String(cell),
|
|
73
|
+
label: String(label),
|
|
74
|
+
n,
|
|
75
|
+
},
|
|
76
|
+
]
|
|
77
|
+
: [];
|
|
78
|
+
};
|
|
79
|
+
|
|
80
|
+
/** Rows of a label file, and the problems that stop a line being one. */
|
|
81
|
+
function readLabels(file: string): { rows: Row[]; problems: string[] } {
|
|
82
|
+
const [first, ...body] = readFileSync(file, "utf8").split("\n");
|
|
83
|
+
const lines: Line[] = body
|
|
84
|
+
.map((line, i) => ({ line, fields: line.split("\t"), n: i + 2 }))
|
|
85
|
+
.filter(({ line }) => line.trim() !== "");
|
|
86
|
+
const headerOk = String(first).replace(/\r$/, "") === HEADER.join("\t");
|
|
87
|
+
return {
|
|
88
|
+
rows: lines.flatMap(asRow),
|
|
89
|
+
problems: [
|
|
90
|
+
...(headerOk
|
|
91
|
+
? []
|
|
92
|
+
: [`${file}:1: the header must be "${HEADER.join("\\t")}"`]),
|
|
93
|
+
...lines
|
|
94
|
+
.filter(({ fields }) => fields.length !== 4)
|
|
95
|
+
.map(({ n }) => `${file}:${String(n)}: not four tab-separated fields`),
|
|
96
|
+
],
|
|
97
|
+
};
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
const git = (cwd: string, ...args: string[]) =>
|
|
101
|
+
spawnSync("git", ["-C", cwd, ...args], { encoding: "utf8" });
|
|
102
|
+
/** Where `abs` sits in its git repository, or null outside one. */
|
|
103
|
+
const repoPath = (abs: string): { top: string; rel: string } | null => {
|
|
104
|
+
const top = git(dirname(abs), "rev-parse", "--show-toplevel");
|
|
105
|
+
return top.status === 0
|
|
106
|
+
? {
|
|
107
|
+
top: top.stdout.trim(),
|
|
108
|
+
rel: relative(top.stdout.trim(), abs).split(sep).join("/"),
|
|
109
|
+
}
|
|
110
|
+
: null;
|
|
111
|
+
};
|
|
112
|
+
/** Unix times of the commits that touched `abs` — only the ones that ADDED it, with `added`. */
|
|
113
|
+
const commitTimes = (abs: string, added: boolean): number[] => {
|
|
114
|
+
const at = repoPath(abs);
|
|
115
|
+
const filter = added ? ["--diff-filter=A"] : [];
|
|
116
|
+
return at
|
|
117
|
+
? git(at.top, "log", ...filter, "--format=%ct", "--", at.rel)
|
|
118
|
+
.stdout.trim()
|
|
119
|
+
.split("\n")
|
|
120
|
+
.filter(Boolean)
|
|
121
|
+
.map(Number)
|
|
122
|
+
: [];
|
|
123
|
+
};
|
|
124
|
+
|
|
125
|
+
const Codebook = z.looseObject({
|
|
126
|
+
labels: z
|
|
127
|
+
.array(
|
|
128
|
+
z.looseObject({
|
|
129
|
+
id: z.unknown().optional(),
|
|
130
|
+
definition: z.unknown().optional(),
|
|
131
|
+
borderline: z.unknown().optional(),
|
|
132
|
+
}),
|
|
133
|
+
)
|
|
134
|
+
.default([]),
|
|
135
|
+
});
|
|
136
|
+
const filled = (v: unknown): boolean =>
|
|
137
|
+
typeof v === "string" && v.trim() !== "";
|
|
138
|
+
|
|
139
|
+
/** Is the committed copy of `abs` the one on disk? A message when it is not. */
|
|
140
|
+
function committedProblem(file: string, abs: string, text: string): string[] {
|
|
141
|
+
const at = repoPath(abs);
|
|
142
|
+
const head = at ? git(at.top, "show", `HEAD:${at.rel}`) : null;
|
|
143
|
+
if (!head || head.status !== 0)
|
|
144
|
+
return [
|
|
145
|
+
`${file} is not committed: commit the codebook before the first label`,
|
|
146
|
+
];
|
|
147
|
+
return head.stdout === text
|
|
148
|
+
? []
|
|
149
|
+
: [
|
|
150
|
+
`${file} differs from its committed version: the codebook is fixed before labelling`,
|
|
151
|
+
];
|
|
152
|
+
}
|
|
153
|
+
|
|
154
|
+
type CodebookLabel = z.infer<typeof Codebook>["labels"][number];
|
|
155
|
+
|
|
156
|
+
/** Labels that lack an id, a definition or a borderline example. */
|
|
157
|
+
const incomplete = (file: string, labels: readonly CodebookLabel[]): string[] =>
|
|
158
|
+
labels
|
|
159
|
+
.filter(
|
|
160
|
+
(l) => !(filled(l.id) && filled(l.definition) && filled(l.borderline)),
|
|
161
|
+
)
|
|
162
|
+
.map(
|
|
163
|
+
(l) =>
|
|
164
|
+
`${file}: label "${String(l.id)}" needs an id, a definition and a borderline example`,
|
|
165
|
+
);
|
|
166
|
+
|
|
167
|
+
function codebookProblems(
|
|
168
|
+
file: string,
|
|
169
|
+
labelsFile: string,
|
|
170
|
+
): { known: Set<unknown>; problems: string[] } {
|
|
171
|
+
const abs = resolve(file);
|
|
172
|
+
const text = readFileSync(abs, "utf8");
|
|
173
|
+
const { labels } = Codebook.parse(JSON.parse(text));
|
|
174
|
+
// EVERY commit of the codebook, not only the one that added it: an edit after labelling began
|
|
175
|
+
// changes what the labels already given mean.
|
|
176
|
+
const bookLast = Math.max(-Infinity, ...commitTimes(abs, false));
|
|
177
|
+
const labelsFirst = Math.min(
|
|
178
|
+
Infinity,
|
|
179
|
+
...commitTimes(resolve(labelsFile), true),
|
|
180
|
+
);
|
|
181
|
+
const late = bookLast > labelsFirst;
|
|
182
|
+
return {
|
|
183
|
+
known: new Set(labels.map((l) => l.id)),
|
|
184
|
+
problems: [
|
|
185
|
+
...committedProblem(file, abs, text),
|
|
186
|
+
...(labels.length ? [] : [`${file} defines no labels`]),
|
|
187
|
+
...incomplete(file, labels),
|
|
188
|
+
...(late
|
|
189
|
+
? [
|
|
190
|
+
`${file} was committed after ${labelsFile} was first committed: the codebook is fixed before labelling; a changed codebook needs a new label file`,
|
|
191
|
+
]
|
|
192
|
+
: []),
|
|
193
|
+
],
|
|
194
|
+
};
|
|
195
|
+
}
|
|
196
|
+
|
|
197
|
+
const Attempt = z.looseObject({
|
|
198
|
+
attempt: z.string(),
|
|
199
|
+
agent: z.string().optional(),
|
|
200
|
+
model: z.string().optional(),
|
|
201
|
+
unit: z.string().optional(),
|
|
202
|
+
arm: z.string().optional(),
|
|
203
|
+
seed: z.union([z.string(), z.number()]).optional(),
|
|
204
|
+
status: z.string().optional(),
|
|
205
|
+
});
|
|
206
|
+
type Attempt = z.infer<typeof Attempt>;
|
|
207
|
+
|
|
208
|
+
/** The cells of the completed attempts in a run-agent-grid ledger. */
|
|
209
|
+
function ledgerCells(file: string): CellId[] {
|
|
210
|
+
const byId = new Map<string, Attempt>();
|
|
211
|
+
readFileSync(file, "utf8")
|
|
212
|
+
.split("\n")
|
|
213
|
+
.filter((l) => l.trim())
|
|
214
|
+
.map((l) => Attempt.parse(JSON.parse(l)))
|
|
215
|
+
.forEach((r) => {
|
|
216
|
+
byId.set(r.attempt, { ...byId.get(r.attempt), ...r });
|
|
217
|
+
});
|
|
218
|
+
return [...byId.values()]
|
|
219
|
+
.filter((r) => r.status === "complete")
|
|
220
|
+
.map((r) => ({
|
|
221
|
+
agent: String(r.agent),
|
|
222
|
+
model: String(r.model),
|
|
223
|
+
cell: `${String(r.unit)}/${String(r.arm)}/${String(r.seed)}`,
|
|
224
|
+
}));
|
|
225
|
+
}
|
|
226
|
+
|
|
227
|
+
/** The labels a revision of the label file held, by cell; null when it did not exist there. */
|
|
228
|
+
function labelsAt(labelsFile: string, rev: string): Map<string, string> | null {
|
|
229
|
+
const at = repoPath(resolve(labelsFile));
|
|
230
|
+
const then = at ? git(at.top, "show", `${rev}:${at.rel}`) : null;
|
|
231
|
+
if (!then || then.status !== 0) return null;
|
|
232
|
+
return new Map(
|
|
233
|
+
then.stdout
|
|
234
|
+
.split("\n")
|
|
235
|
+
.slice(1)
|
|
236
|
+
.filter((l) => l.trim())
|
|
237
|
+
.map((l) => l.split("\t"))
|
|
238
|
+
.map(([agent, model, cell, label]) => [
|
|
239
|
+
keyOf({
|
|
240
|
+
agent: String(agent),
|
|
241
|
+
model: String(model),
|
|
242
|
+
cell: String(cell),
|
|
243
|
+
}),
|
|
244
|
+
String(label),
|
|
245
|
+
]),
|
|
246
|
+
);
|
|
247
|
+
}
|
|
248
|
+
|
|
249
|
+
/** The sections of the notes under a "## YYYY-MM-DD" heading, each up to the next "## ". */
|
|
250
|
+
const datedSections = (notesFile: string): string[] =>
|
|
251
|
+
readFileSync(notesFile, "utf8")
|
|
252
|
+
.split(/^(?=## )/m)
|
|
253
|
+
.filter((s) => /^## \d{4}-\d{2}-\d{2}\b/.test(s));
|
|
254
|
+
|
|
255
|
+
/** A cell key back as the three fields `show` prints. */
|
|
256
|
+
const fromKey = (key: string): CellId => {
|
|
257
|
+
const [agent, model, cell] = key.split("\t");
|
|
258
|
+
return { agent: String(agent), model: String(model), cell: String(cell) };
|
|
259
|
+
};
|
|
260
|
+
|
|
261
|
+
const WHY = " with no dated note quoting the run";
|
|
262
|
+
|
|
263
|
+
/** Rows whose label is new or different since the analysis, and not explained. */
|
|
264
|
+
const touchedSince = (
|
|
265
|
+
rows: readonly Row[],
|
|
266
|
+
before: ReadonlyMap<string, string>,
|
|
267
|
+
noted: (cell: string) => boolean,
|
|
268
|
+
): string[] =>
|
|
269
|
+
rows
|
|
270
|
+
.filter((r) => before.get(keyOf(r)) !== r.label && !noted(r.cell))
|
|
271
|
+
.map((r) => {
|
|
272
|
+
const was = before.get(keyOf(r));
|
|
273
|
+
return was === undefined
|
|
274
|
+
? `${show(r)}: added after the analysis (${r.label}),${WHY}`
|
|
275
|
+
: `${show(r)}: relabelled ${was} → ${r.label} after the analysis,${WHY}`;
|
|
276
|
+
});
|
|
277
|
+
|
|
278
|
+
/** Cells the analysis read that are gone now, and not explained. */
|
|
279
|
+
const removedSince = (
|
|
280
|
+
rows: readonly Row[],
|
|
281
|
+
before: ReadonlyMap<string, string>,
|
|
282
|
+
noted: (cell: string) => boolean,
|
|
283
|
+
): string[] => {
|
|
284
|
+
const now = new Set(rows.map(keyOf));
|
|
285
|
+
return [...before]
|
|
286
|
+
.filter(([key]) => !now.has(key) && !noted(fromKey(key).cell))
|
|
287
|
+
.map(
|
|
288
|
+
([key, was]) =>
|
|
289
|
+
`${show(fromKey(key))}: removed after the analysis (was ${was}),${WHY}`,
|
|
290
|
+
);
|
|
291
|
+
};
|
|
292
|
+
|
|
293
|
+
/**
|
|
294
|
+
* Every cell whose label differs from what the analysis read at `rev` — changed, ADDED after it,
|
|
295
|
+
* or REMOVED since — and is not explained in a dated, quoted note naming the cell.
|
|
296
|
+
*/
|
|
297
|
+
function relabelProblems(
|
|
298
|
+
labelsFile: string,
|
|
299
|
+
rev: string,
|
|
300
|
+
notesFile: string,
|
|
301
|
+
rows: readonly Row[],
|
|
302
|
+
): string[] {
|
|
303
|
+
const before = labelsAt(labelsFile, rev);
|
|
304
|
+
if (!before)
|
|
305
|
+
return [
|
|
306
|
+
`${labelsFile} does not exist at ${rev}: name the revision the analysis read`,
|
|
307
|
+
];
|
|
308
|
+
const sections = datedSections(notesFile);
|
|
309
|
+
const noted = (cell: string): boolean =>
|
|
310
|
+
sections.some((s) => s.includes(cell) && /^> \S/m.test(s));
|
|
311
|
+
return [
|
|
312
|
+
...touchedSince(rows, before, noted),
|
|
313
|
+
...removedSince(rows, before, noted),
|
|
314
|
+
];
|
|
315
|
+
}
|
|
316
|
+
|
|
317
|
+
/** Cells a file labels more than once, with the lines. */
|
|
318
|
+
function duplicates(file: string, rows: readonly Row[]): string[] {
|
|
319
|
+
const seen = new Map<string, number[]>();
|
|
320
|
+
rows.forEach((r) => seen.set(keyOf(r), [...(seen.get(keyOf(r)) ?? []), r.n]));
|
|
321
|
+
return [...seen]
|
|
322
|
+
.filter(([, ns]) => ns.length > 1)
|
|
323
|
+
.map(
|
|
324
|
+
([k, ns]) =>
|
|
325
|
+
`${file}: ${show(fromKey(k))} is labelled ${String(ns.length)} times (lines ${ns.join(", ")})`,
|
|
326
|
+
);
|
|
327
|
+
}
|
|
328
|
+
|
|
329
|
+
/** Rows naming a cell twice, and coverage against the ledger. */
|
|
330
|
+
function rowProblems(file: string, rows: readonly Row[], ledger?: string) {
|
|
331
|
+
const runs = ledger === undefined ? null : ledgerCells(ledger);
|
|
332
|
+
const ran = new Set((runs ?? []).map(keyOf));
|
|
333
|
+
const labelled = new Set(rows.map(keyOf));
|
|
334
|
+
return [
|
|
335
|
+
...duplicates(file, rows),
|
|
336
|
+
...(runs
|
|
337
|
+
? [
|
|
338
|
+
...rows
|
|
339
|
+
.filter((r) => !ran.has(keyOf(r)))
|
|
340
|
+
.map(
|
|
341
|
+
(r) =>
|
|
342
|
+
`${file}:${String(r.n)}: ${show(r)} has no completed run in the ledger`,
|
|
343
|
+
),
|
|
344
|
+
...runs
|
|
345
|
+
.filter((r) => !labelled.has(keyOf(r)))
|
|
346
|
+
.map((r) => `${show(r)}: a completed run with no label`),
|
|
347
|
+
]
|
|
348
|
+
: []),
|
|
349
|
+
];
|
|
350
|
+
}
|
|
351
|
+
|
|
352
|
+
function validate(file: string, o: Values): void {
|
|
353
|
+
const { rows, problems } = readLabels(file);
|
|
354
|
+
const book = codebookProblems(String(o.codebook), file);
|
|
355
|
+
const all = [
|
|
356
|
+
...problems,
|
|
357
|
+
...book.problems,
|
|
358
|
+
...rows
|
|
359
|
+
.filter((r) => !book.known.has(r.label))
|
|
360
|
+
.map(
|
|
361
|
+
(r) => `${file}:${String(r.n)}: "${r.label}" is not a codebook label`,
|
|
362
|
+
),
|
|
363
|
+
...rowProblems(file, rows, o.ledger),
|
|
364
|
+
...(o["analysed-at"] === undefined
|
|
365
|
+
? []
|
|
366
|
+
: relabelProblems(file, o["analysed-at"], String(o.notes), rows)),
|
|
367
|
+
];
|
|
368
|
+
if (all.length) refuse(1, all.join("\n"));
|
|
369
|
+
say(`${file}: ${String(rows.length)} labels, valid`);
|
|
370
|
+
}
|
|
371
|
+
|
|
372
|
+
// ── agreement ───────────────────────────────────────────────────────────────────────────────
|
|
373
|
+
const Z = 1.959963984540054;
|
|
374
|
+
/** 95% Wilson score interval for k successes in n. */
|
|
375
|
+
const wilson = (k: number, n: number): readonly [number, number] => {
|
|
376
|
+
const p = k / n;
|
|
377
|
+
const denom = 1 + (Z * Z) / n;
|
|
378
|
+
const centre = (p + (Z * Z) / (2 * n)) / denom;
|
|
379
|
+
const half =
|
|
380
|
+
(Z * Math.sqrt((p * (1 - p)) / n + (Z * Z) / (4 * n * n))) / denom;
|
|
381
|
+
return [centre - half, centre + half];
|
|
382
|
+
};
|
|
383
|
+
|
|
384
|
+
/** Cohen's kappa over paired labels; null when chance agreement is 1 (one label only). */
|
|
385
|
+
function cohenKappa(
|
|
386
|
+
pairs: readonly (readonly [string, string])[],
|
|
387
|
+
): number | null {
|
|
388
|
+
const n = pairs.length;
|
|
389
|
+
const po = pairs.filter(([a, b]) => a === b).length / n;
|
|
390
|
+
const labels = [...new Set(pairs.flat())];
|
|
391
|
+
const share = (i: 0 | 1, l: string): number =>
|
|
392
|
+
pairs.filter((p) => p[i] === l).length / n;
|
|
393
|
+
const pe = labels.reduce((s, l) => s + share(0, l) * share(1, l), 0);
|
|
394
|
+
return pe === 1 ? null : (po - pe) / (1 - pe);
|
|
395
|
+
}
|
|
396
|
+
|
|
397
|
+
/** Cells of either file that the other does not label. */
|
|
398
|
+
const onlyIn = (
|
|
399
|
+
fileA: string,
|
|
400
|
+
a: readonly Row[],
|
|
401
|
+
b: readonly Row[],
|
|
402
|
+
fileB: string,
|
|
403
|
+
): string[] => {
|
|
404
|
+
const inA = new Set(a.map(keyOf));
|
|
405
|
+
const inB = new Set(b.map(keyOf));
|
|
406
|
+
return [
|
|
407
|
+
...a
|
|
408
|
+
.filter((r) => !inB.has(keyOf(r)))
|
|
409
|
+
.map((r) => `${show(r)}: only in ${fileA}`),
|
|
410
|
+
...b
|
|
411
|
+
.filter((r) => !inA.has(keyOf(r)))
|
|
412
|
+
.map((r) => `${show(r)}: only in ${fileB}`),
|
|
413
|
+
];
|
|
414
|
+
};
|
|
415
|
+
|
|
416
|
+
/** The second file's label for each cell of the first; a refusal when the cells differ. */
|
|
417
|
+
function matched(
|
|
418
|
+
fileA: string,
|
|
419
|
+
fileB: string,
|
|
420
|
+
): readonly (readonly [Row, string])[] {
|
|
421
|
+
const a = readLabels(fileA);
|
|
422
|
+
const b = readLabels(fileB);
|
|
423
|
+
// A cell labelled twice has no single label to compare: refused before any statistic.
|
|
424
|
+
const bad = [
|
|
425
|
+
...a.problems,
|
|
426
|
+
...b.problems,
|
|
427
|
+
...duplicates(fileA, a.rows),
|
|
428
|
+
...duplicates(fileB, b.rows),
|
|
429
|
+
];
|
|
430
|
+
if (bad.length) refuse(1, bad.join("\n"));
|
|
431
|
+
const inB = new Map(b.rows.map((r) => [keyOf(r), r.label]));
|
|
432
|
+
const unmatched = onlyIn(fileA, a.rows, b.rows, fileB);
|
|
433
|
+
if (unmatched.length)
|
|
434
|
+
refuse(
|
|
435
|
+
1,
|
|
436
|
+
`the two files label different cells — rate the same runs:\n${unmatched.join("\n")}`,
|
|
437
|
+
);
|
|
438
|
+
if (a.rows.length === 0) refuse(1, "no labels to compare");
|
|
439
|
+
return a.rows.map((r) => [r, String(inB.get(keyOf(r)))] as const);
|
|
440
|
+
}
|
|
441
|
+
|
|
442
|
+
function agree(fileA: string, fileB: string): void {
|
|
443
|
+
const rows = matched(fileA, fileB);
|
|
444
|
+
const pairs = rows.map(([r, other]) => [r.label, other] as const);
|
|
445
|
+
const n = pairs.length;
|
|
446
|
+
const k = pairs.filter(([x, y]) => x === y).length;
|
|
447
|
+
const [lo, hi] = wilson(k, n);
|
|
448
|
+
const kappa = cohenKappa(pairs);
|
|
449
|
+
const f = (x: number): string => x.toFixed(3);
|
|
450
|
+
say(
|
|
451
|
+
[
|
|
452
|
+
`n ${String(n)}`,
|
|
453
|
+
`agreement ${String(k)}/${String(n)} = ${f(k / n)} (95% Wilson ${f(lo)}–${f(hi)})`,
|
|
454
|
+
`cohen kappa ${kappa === null ? "undefined (one label only)" : f(kappa)}`,
|
|
455
|
+
...rows
|
|
456
|
+
.filter(([r, other]) => r.label !== other)
|
|
457
|
+
.map(([r, other]) => `disagree ${show(r)}: ${r.label} | ${other}`),
|
|
458
|
+
].join("\n"),
|
|
459
|
+
);
|
|
460
|
+
}
|
|
461
|
+
|
|
462
|
+
// ── command line ────────────────────────────────────────────────────────────────────────────
|
|
463
|
+
const OPTIONS = {
|
|
464
|
+
codebook: { type: "string" },
|
|
465
|
+
ledger: { type: "string" },
|
|
466
|
+
"analysed-at": { type: "string" },
|
|
467
|
+
notes: { type: "string" },
|
|
468
|
+
} as const;
|
|
469
|
+
type Values = ReturnType<
|
|
470
|
+
typeof parseArgs<{ options: typeof OPTIONS; allowPositionals: true }>
|
|
471
|
+
>["values"];
|
|
472
|
+
|
|
473
|
+
const USAGE =
|
|
474
|
+
"usage: labels.js validate <labels.tsv> --codebook <f> [--ledger <f>] [--analysed-at <rev> --notes <f>] | agree <a.tsv> <b.tsv>";
|
|
475
|
+
|
|
476
|
+
function main(): void {
|
|
477
|
+
const { values, positionals } = (() => {
|
|
478
|
+
try {
|
|
479
|
+
return parseArgs({ options: OPTIONS, allowPositionals: true });
|
|
480
|
+
} catch (e) {
|
|
481
|
+
return refuse(2, `${String(prop(e, "message"))}\n${USAGE}`);
|
|
482
|
+
}
|
|
483
|
+
})();
|
|
484
|
+
const [cmd, ...paths] = positionals;
|
|
485
|
+
const validateOk =
|
|
486
|
+
cmd === COMMANDS.validate &&
|
|
487
|
+
paths.length === 1 &&
|
|
488
|
+
values.codebook !== undefined &&
|
|
489
|
+
(values["analysed-at"] === undefined || values.notes !== undefined);
|
|
490
|
+
if (validateOk) validate(String(paths[0]), values);
|
|
491
|
+
else if (cmd === COMMANDS.agree && paths.length === 2)
|
|
492
|
+
agree(String(paths[0]), String(paths[1]));
|
|
493
|
+
else refuse(2, USAGE);
|
|
494
|
+
}
|
|
495
|
+
|
|
496
|
+
try {
|
|
497
|
+
main();
|
|
498
|
+
} catch (e) {
|
|
499
|
+
process.stderr.write(`labels: ${String(prop(e, "message"))}\n`);
|
|
500
|
+
process.exitCode = e instanceof Refusal ? e.exitCode : 1;
|
|
501
|
+
}
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: map-prior-work
|
|
3
|
-
description: Sweep the competitive landscape for a paper idea BEFORE drafting — find everyone already working on it, date each against your submission (prior vs concurrent is a five-month question, not a vibe), triage by whether they threaten your claimed contribution, deep-read the ones that do, and emit a verdict on WHAT YOU CAN STILL CLAIM plus a related-work skeleton. Saves the full map, not a summary. Use right after research-ideate and before draft-paper, and re-run as a cheap delta pass before submit (hot fields move). Distinct from analyze-sibling-paper (one known competitor, deep) — this one FINDS the competitors and decides which deserve that treatment; distinct from verify-citations (checks cites you already wrote, too late to reshape a contribution); distinct from study-accepted-papers (mines a venue
|
|
3
|
+
description: Sweep the competitive landscape for a paper idea BEFORE drafting — find everyone already working on it, date each against your submission (prior vs concurrent is a five-month question, not a vibe), triage by whether they threaten your claimed contribution, deep-read the ones that do, and emit a verdict on WHAT YOU CAN STILL CLAIM plus a related-work skeleton. Saves the full map, not a summary. Use right after research-ideate and before draft-paper, and re-run as a cheap delta pass before submit (hot fields move). Distinct from analyze-sibling-paper (one known competitor, deep) — this one FINDS the competitors and decides which deserve that treatment; distinct from verify-citations (checks cites you already wrote, too late to reshape a contribution); distinct from study-accepted-papers (mines what a venue rewards, not your rivals). Compose with research-ideate (upstream), analyze-sibling-paper (it hands off), draft-paper and find-venue (downstream).
|
|
4
4
|
context: fork
|
|
5
5
|
allowed-tools: [WebSearch, WebFetch, Read, Write, Grep, Glob, Bash, Agent]
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
<!-- vigiles:sha256:
|
|
8
|
+
<!-- vigiles:sha256:474b3d7b8e0542ae compiled from skills/map-prior-work/SKILL.md.spec.ts -->
|
|
9
9
|
|
|
10
10
|
# map-prior-work — find out who already did it, while you can still change course
|
|
11
11
|
|
|
@@ -9,7 +9,7 @@ import { experimental_skill } from "vigiles/spec";
|
|
|
9
9
|
export default experimental_skill({
|
|
10
10
|
name: "map-prior-work",
|
|
11
11
|
description:
|
|
12
|
-
"Sweep the competitive landscape for a paper idea BEFORE drafting — find everyone already working on it, date each against your submission (prior vs concurrent is a five-month question, not a vibe), triage by whether they threaten your claimed contribution, deep-read the ones that do, and emit a verdict on WHAT YOU CAN STILL CLAIM plus a related-work skeleton. Saves the full map, not a summary. Use right after research-ideate and before draft-paper, and re-run as a cheap delta pass before submit (hot fields move). Distinct from analyze-sibling-paper (one known competitor, deep) — this one FINDS the competitors and decides which deserve that treatment; distinct from verify-citations (checks cites you already wrote, too late to reshape a contribution); distinct from study-accepted-papers (mines a venue
|
|
12
|
+
"Sweep the competitive landscape for a paper idea BEFORE drafting — find everyone already working on it, date each against your submission (prior vs concurrent is a five-month question, not a vibe), triage by whether they threaten your claimed contribution, deep-read the ones that do, and emit a verdict on WHAT YOU CAN STILL CLAIM plus a related-work skeleton. Saves the full map, not a summary. Use right after research-ideate and before draft-paper, and re-run as a cheap delta pass before submit (hot fields move). Distinct from analyze-sibling-paper (one known competitor, deep) — this one FINDS the competitors and decides which deserve that treatment; distinct from verify-citations (checks cites you already wrote, too late to reshape a contribution); distinct from study-accepted-papers (mines what a venue rewards, not your rivals). Compose with research-ideate (upstream), analyze-sibling-paper (it hands off), draft-paper and find-venue (downstream).",
|
|
13
13
|
context: "fork",
|
|
14
14
|
tools: [
|
|
15
15
|
"WebSearch",
|
|
@@ -4,7 +4,7 @@ description: The orchestrator for writing a research paper end-to-end, from idea
|
|
|
4
4
|
allowed-tools: [Read, Write, Edit, Grep, Glob, Bash, WebSearch, WebFetch, Agent, Skill]
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
<!-- vigiles:sha256:
|
|
7
|
+
<!-- vigiles:sha256:da7c692384fb79aa compiled from skills/paper-pipeline/SKILL.md.spec.ts -->
|
|
8
8
|
|
|
9
9
|
# paper-pipeline — the conductor for the whole organism
|
|
10
10
|
|
|
@@ -150,7 +150,7 @@ Ordered, because each really does gate the next. Two pairs run in parallel; that
|
|
|
150
150
|
| **Validate the idea** | **`research-ideate`** | Is it a reusable method (survives review) in your lane, and does it feed the credit criteria? Go/no-go + the MVP finding. |
|
|
151
151
|
| **Map the competition** ‖ | **`map-prior-work`** | 🔴 **Who already did this?** Multi-angle sweep; date each hit against your submission (5 months = prior, not concurrent); triage by which layer of your claim it threatens; deep-read the dangerous ones. Output: **what you can still claim** + a related-work skeleton. *Added 2026-07-27 after a five-month-older paper surfaced at T−10 days on a finished draft.* |
|
|
152
152
|
| **Pick the venue** ‖ | **`find-venue`** | Rank by credit-weight (peer-reviewed + indexed) × fit × accept-odds × deadline × remote. |
|
|
153
|
-
| **Learn the venue
|
|
153
|
+
| **Learn what the venue rewards** | **`study-accepted-papers`** (mode A idea, B design) | 🔴 **What does this venue REWARD, before the study is designed?** The venue's award winners against its ordinary accepted papers over 5–7 years (plus award papers on the topic at other venues, labelled apart), coded for substance as well as form, written as **requirements for the study**: what the oracle under the headline is, which belief stated on record the claim targets, which factor is a column. The corpus goes to `<papers-root>/venues/<venue>/`, shared by every paper for that venue; the stage is row `venuebar`. **The same skill comes back once text exists** (mode C draft: the strong-accept scorecard, M1–M10, in `PIPELINE-STATUS.md`; mode D final: the abstract and introduction). The skill always asks which mode. Venue format facts go in its preset (`presets/<venue>.jsonc` in the package, or the project's own) and the venue card beside it (`presets/<venue>.md`); confirm family rules (`references/acl-venue-rules.md`). *Promoted to its own step 2026-08-03: every judgement of a paper's quality had come from instruments this pipeline itself generated, with no external grounding. Moved to modes on 2026-10-12 (#207): run only on a draft, its main findings arrived after they could change the design.* |
|
|
154
154
|
| 🔴 **Can you physically submit?** | **`plan-paper-timeline`** | **The step that owns the question no stage used to own.** Create the portal account NOW; drive the profile to **ACTIVE**, not "pending moderation"; confirm the form's fields and the artifact-hosting requirement. OpenReview moderation runs **up to two weeks**, and an author with no institutional email has no expedite route. Scorecard row `access`. *Added 2026-08-03, when this was the sole critical-path item on a finished paper at T−4 days: the science was done and there was nowhere to put it.* |
|
|
155
155
|
| **Schedule it** | **`plan-paper-timeline`** | Deadlines → calendar with buffers, in the author's own zone (AoE − 12h ≠ your clock). Space papers apart. |
|
|
156
156
|
| 🔴 **State the claim** | **`argument-arc`** (frame mode) | **One paragraph: what will this paper claim, and what would have to be true?** Before any run. The frame decides which results matter, and reframing after the data is collected is how experiments get thrown away — observed on `compile-rules-2026`, and the reason `argument-arc` has a rebuild mode at all. Costs a paragraph; saves a study. Scorecard row `frame`. |
|
|
@@ -170,6 +170,15 @@ No numbers. Nothing here happens once, and nothing here is "next".
|
|
|
170
170
|
|
|
171
171
|
- **`build-benchmark`** — a design that inverts the pitfall; honest stats; a self-checking artifact
|
|
172
172
|
(`references/artifact-checklist.md`).
|
|
173
|
+
- **`run-agent-grid`** — when the study is a grid of agent runs (units × arms × seeds, one agent CLI
|
|
174
|
+
run per cell): the registry and the author's approval gate every counted run, a sandbox and secret
|
|
175
|
+
probe, a smoke per model, a locked and resumable driver, an overlap audit, finished runs committed.
|
|
176
|
+
Between the design and the analysis.
|
|
177
|
+
- **`label-agent-runs`** — when the runs are in and before any outcome number: a codebook fixed
|
|
178
|
+
first, the automatic classifier only a pre-sort, every uncertain run read and labelled by hand, a
|
|
179
|
+
blind second rater with its agreement, labels frozen after the analysis.
|
|
180
|
+
- **`build-label-page`** — when the study needs a person to label items (gold labels, a blind review):
|
|
181
|
+
a phone page with a shared answer store, the answers exported into the study's `data/`.
|
|
173
182
|
- **`draft-paper`** — section skeleton + the accept-and-get-cited properties
|
|
174
183
|
(`references/writing-craft.md`).
|
|
175
184
|
- **`argument-arc`** — 🔴 **does the draft carry the reader to ONE conclusion?** One sentence per
|
|
@@ -221,7 +230,7 @@ Use an atom for a quick spot-check; **run the panel for the decision.**
|
|
|
221
230
|
skill: 🔴 **argument architecture** — does the whole thing carry the reader to one conclusion
|
|
222
231
|
(`argument-arc`, in the LOOP; the axis that failed on `compile-rules-2026` precisely because it was
|
|
223
232
|
nobody's) · **content strength** — does the contribution clear *this venue's* bar (`study-accepted-papers`,
|
|
224
|
-
|
|
233
|
+
from the venue's own award winners: in SETUP as requirements for the study, and again on the draft as the strong-accept scorecard, M1–M10) ·
|
|
225
234
|
**scientific defects** — what a hostile reviewer attacks (`paper-adversarial-review` / `pc-panel-review`)
|
|
226
235
|
· **writing craft** — does it read like a human wrote it (`grade-paper-writing`, the 10-dim rubric in
|
|
227
236
|
`references/writing-craft.md`; called by `harden-paper`). A paper can ace one and fail another — grade all four.
|
|
@@ -294,7 +303,7 @@ Over {{corpus.n}} real rules files, with the extractor's proposed mapping
|
|
|
294
303
|
taken as if an author had written it, {{annotated.failedContradicted}} fail admission.
|
|
295
304
|
```
|
|
296
305
|
|
|
297
|
-
`<paper>/repro/numbers.tsv` gives each name its value, the JSON path or **formula** that recomputes
|
|
306
|
+
`<paper>/repro/data/numbers.tsv` gives each name its value, the JSON path or **formula** that recomputes
|
|
298
307
|
it, and two guards. The build refuses a PDF unless: the name resolves · the value still matches the
|
|
299
308
|
data · the **paragraph** names the arm (`requires`) · the **sentence** does not claim a different one
|
|
300
309
|
(`forbids`). Requires and forbids are separate on purpose — the pairing is auditing's, where a control
|
|
@@ -310,7 +319,7 @@ binds every row from now on, and the existing rows are *enumerated rather than e
|
|
|
310
319
|
says out loud how many figures still have no machine-checked path to data. On the paper that has it:
|
|
311
320
|
7 guarded, 248 grandfathered. That ratio is a finding, not a score.
|
|
312
321
|
|
|
313
|
-
**Adopting it in another paper:** copy `repro/paper_numbers.py`, `repro/paper_numbers.selftest.py`
|
|
322
|
+
**Adopting it in another paper:** copy `repro/tools/paper_numbers.py`, `repro/tools/paper_numbers.selftest.py`
|
|
314
323
|
and the build step that calls `expand()` on `paper.tex` before `pdflatex`; generate the
|
|
315
324
|
grandfather list from the current draft; register the numbers a reviewer will actually check first.
|
|
316
325
|
|
|
@@ -350,7 +359,7 @@ printing two unrelated 22s broke it the same hour. Key on the occurrence.
|
|
|
350
359
|
🔴 **Hand the gate to a red team the same day you build it.** This one was, and seven holes came
|
|
351
360
|
back, the worst being that the original defect could simply be **retyped as digits** — its value was
|
|
352
361
|
grandfathered, so the guard applied to the macro and not to the claim. Every hole is now a case in
|
|
353
|
-
`repro/paper_numbers.selftest.py` (18/18, including three that must stay **quiet** — a gate that
|
|
362
|
+
`repro/tools/paper_numbers.selftest.py` (18/18, including three that must stay **quiet** — a gate that
|
|
354
363
|
fires on ordinary writing gets muted, and then its silence reads as coverage).
|
|
355
364
|
|
|
356
365
|
## 🔴 The page limit is a SUBMISSION gate, not an EDIT gate
|