paperlint 9.0.1 → 10.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -128
- package/dist/build.d.ts +1 -1
- package/dist/build.js +1 -1
- package/dist/cli.d.ts +5 -1
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +52 -6
- package/dist/cli.js.map +1 -1
- package/dist/domain/layout-tree.d.ts +19 -0
- package/dist/domain/layout-tree.d.ts.map +1 -0
- package/dist/domain/layout-tree.js +32 -0
- package/dist/domain/layout-tree.js.map +1 -0
- package/dist/domain/lookup-cache.d.ts +2 -2
- package/dist/domain/lookup-cache.js +2 -2
- package/dist/domain/paper-layout.d.ts +81 -0
- package/dist/domain/paper-layout.d.ts.map +1 -0
- package/dist/domain/paper-layout.js +259 -0
- package/dist/domain/paper-layout.js.map +1 -0
- package/dist/domain/repro-layout.d.ts +80 -0
- package/dist/domain/repro-layout.d.ts.map +1 -0
- package/dist/domain/repro-layout.js +247 -0
- package/dist/domain/repro-layout.js.map +1 -0
- package/dist/domain/scaffold-layout.d.ts +28 -0
- package/dist/domain/scaffold-layout.d.ts.map +1 -0
- package/dist/domain/scaffold-layout.js +81 -0
- package/dist/domain/scaffold-layout.js.map +1 -0
- package/dist/eslint-rules/prereg-coverage.d.ts +98 -0
- package/dist/eslint-rules/prereg-coverage.d.ts.map +1 -0
- package/dist/eslint-rules/prereg-coverage.js +274 -0
- package/dist/eslint-rules/prereg-coverage.js.map +1 -0
- package/dist/folder-layout-hook.d.ts +21 -0
- package/dist/folder-layout-hook.d.ts.map +1 -0
- package/dist/folder-layout-hook.js +117 -0
- package/dist/folder-layout-hook.js.map +1 -0
- package/dist/folder-layout-rule.d.ts +22 -0
- package/dist/folder-layout-rule.d.ts.map +1 -0
- package/dist/folder-layout-rule.js +107 -0
- package/dist/folder-layout-rule.js.map +1 -0
- package/dist/init.js +1 -1
- package/dist/init.js.map +1 -1
- package/dist/new-paper.d.ts +4 -1
- package/dist/new-paper.d.ts.map +1 -1
- package/dist/new-paper.js +40 -6
- package/dist/new-paper.js.map +1 -1
- package/dist/references.d.ts +1 -1
- package/dist/references.d.ts.map +1 -1
- package/dist/references.js +2 -2
- package/dist/references.js.map +1 -1
- package/dist/repro-layout-rule.d.ts +16 -0
- package/dist/repro-layout-rule.d.ts.map +1 -0
- package/dist/repro-layout-rule.js +81 -0
- package/dist/repro-layout-rule.js.map +1 -0
- package/docs/ci.md +60 -0
- package/docs/configuration.md +2 -0
- package/docs/e2e.md +2 -2
- package/docs/install.md +20 -16
- package/docs/layout.md +154 -0
- package/docs/optional-rules.md +5 -4
- package/docs/references.md +5 -2
- package/docs/rules/paper/allowed-files-only.md +192 -0
- package/docs/rules/repro/allowed-files-only.md +128 -0
- package/docs/rules/study/declare-sample.md +146 -0
- package/docs/rules.md +53 -50
- package/docs/skills.md +6 -3
- package/docs/venues.md +94 -0
- package/eslint-rules/prereg-coverage.test.ts +448 -0
- package/eslint-rules/prereg-coverage.ts +419 -0
- package/lib/skill-trigger-cases.harness.mjs +51 -5
- package/lib/skill-trigger-cases.mjs +53 -15
- package/package.json +1 -1
- package/plugin/hooks/hooks.json +9 -0
- package/skills/analyze-sibling-paper/SKILL.md +2 -2
- package/skills/analyze-sibling-paper/SKILL.md.spec.ts +1 -1
- package/skills/build-benchmark/SKILL.md +14 -1
- package/skills/build-benchmark/SKILL.md.spec.ts +13 -0
- package/skills/build-label-page/SKILL.md +118 -0
- package/skills/build-label-page/SKILL.md.spec.ts +129 -0
- package/skills/build-label-page/build-label-page.eval.mjs +15 -0
- package/skills/build-label-page/build-label-page.test.ts +959 -0
- package/skills/build-label-page/references/checklist.md +148 -0
- package/skills/build-label-page/references/example-generator.py +803 -0
- package/skills/find-venue/SKILL.md +5 -3
- package/skills/find-venue/SKILL.md.spec.ts +4 -2
- package/skills/grade-paper-writing/SKILL.md +7 -7
- package/skills/grade-paper-writing/SKILL.md.spec.ts +6 -6
- package/skills/map-prior-work/SKILL.md +2 -2
- package/skills/map-prior-work/SKILL.md.spec.ts +1 -1
- package/skills/paper-pipeline/SKILL.md +8 -6
- package/skills/paper-pipeline/SKILL.md.spec.ts +7 -5
- package/skills/paper-pipeline/grade-paper-writing-ablation.eval.mjs +19 -9
- package/skills/paper-pipeline/references/acceptance-gate.md +19 -19
- package/skills/paper-pipeline/references/artifact-checklist.md +5 -0
- package/skills/paper-pipeline/references/pipeline-status-template.md +30 -1
- package/skills/paper-pipeline/repro/2026-08-07-fork-check.log +1 -1
- package/skills/paper-pipeline/repro/2026-08-07-fork-check2.log +1 -1
- package/skills/paper-pipeline/repro/2026-08-07-language-eval-pilot.log +1 -1
- package/skills/paper-pipeline/repro/2026-08-07-language-eval-raw.log +1 -1
- package/skills/paper-pipeline/repro/2026-08-08-framing-vs-vocabulary-oracle.log +48 -48
- package/skills/paper-pipeline/repro/2026-08-08-grade-paper-writing-ablation-A6-oracle.log +8 -8
- package/skills/paper-pipeline/repro/2026-08-08-grade-paper-writing-ablation-oracle.log +48 -48
- package/skills/paper-pipeline/repro/2026-08-08-parent-replication-gpw.log +2 -2
- package/skills/paper-pipeline/repro/2026-08-08-parent-replication.mjs +3 -3
- package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-ablation-preflight-after.log +81 -0
- package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-ablation-preflight-before.log +81 -0
- package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-ablation-setupdiff.log +59 -0
- package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-trigger.jsonl +1 -0
- package/skills/paper-pipeline/repro/2026-10-13-grade-paper-writing-trigger.log +22 -0
- package/skills/paper-pipeline/repro/README.md +4 -0
- package/skills/paper-pipeline/scripts/extract-ref-facts.harness.mjs +2 -2
- package/skills/paper-pipeline/scripts/extract-ref-facts.mjs +2 -2
- package/skills/paper-pipeline/scripts/extract-ref-facts.test.mjs +4 -2
- package/skills/paper-pipeline/scripts/generated-code.harness.mjs +7 -2
- package/skills/paper-pipeline/scripts/generated-code.mjs +8 -4
- package/skills/paper-pipeline/scripts/generated-code.test.ts +15 -1
- package/skills/paper-status/SKILL.md +2 -2
- package/skills/paper-status/SKILL.md.spec.ts +1 -1
- package/skills/study-accepted-papers/SKILL.md +163 -161
- package/skills/study-accepted-papers/SKILL.md.spec.ts +167 -165
- package/skills/study-accepted-papers/references/base-rates.md +69 -0
- package/skills/study-accepted-papers/references/mechanisms.md +181 -0
- package/skills/verify-citations/SKILL.md +2 -2
- package/skills/verify-citations/SKILL.md.spec.ts +1 -1
- package/src/build.ts +1 -1
- package/src/cli.harness.mjs +42 -0
- package/src/cli.test.ts +90 -6
- package/src/cli.ts +78 -5
- package/src/domain/layout-tree.test.ts +32 -0
- package/src/domain/layout-tree.ts +47 -0
- package/src/domain/lookup-cache.ts +2 -2
- package/src/domain/paper-layout.test.ts +264 -0
- package/src/domain/paper-layout.ts +336 -0
- package/src/domain/repro-layout.test.ts +222 -0
- package/src/domain/repro-layout.ts +311 -0
- package/src/domain/scaffold-layout.ts +104 -0
- package/src/folder-layout-hook.test.ts +214 -0
- package/src/folder-layout-hook.ts +141 -0
- package/src/folder-layout-rule.test.ts +235 -0
- package/src/folder-layout-rule.ts +151 -0
- package/src/hooks-settings.harness.ts +9 -8
- package/src/init.ts +1 -1
- package/src/layout-docs.test.ts +52 -0
- package/src/new-paper.harness.mjs +23 -1
- package/src/new-paper.test.ts +299 -6
- package/src/new-paper.ts +51 -5
- package/src/paper-files.test.ts +9 -6
- package/src/references.test.ts +5 -5
- package/src/references.ts +2 -2
- package/src/repro-layout-rule.test.ts +114 -0
- package/src/repro-layout-rule.ts +121 -0
- package/src/rule-cases.test.ts +98 -0
- package/templates/paper/PIPELINE-STATUS.md +41 -7
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
# repro/allowed-files-only
|
|
2
|
+
|
|
3
|
+
**Level:** error · **Reads:** the listing of the paper's `repro/`, and whether each study folder in
|
|
4
|
+
it holds a `README.md` · **Reported on:** the first line of the paper's `PIPELINE-STATUS.md`, once
|
|
5
|
+
per misplaced entry
|
|
6
|
+
|
|
7
|
+
## What it catches
|
|
8
|
+
|
|
9
|
+
An entry at the top of a paper's `repro/` that is not part of its layout, and a study folder there
|
|
10
|
+
without a `README.md`. `repro/` holds an index, the tools and data the whole paper shares, and one
|
|
11
|
+
dated folder per study. Stage 3 of the pipeline, the study, writes here; the rule runs on every
|
|
12
|
+
`paperlint lint` and in CI, and a hook refuses a new loose entry before it is written. The whole map:
|
|
13
|
+
[`layout.md`](../../layout.md).
|
|
14
|
+
|
|
15
|
+
<!-- layout:begin — drawn from src/domain/repro-layout.ts (REPRO_LAYOUT_TREE); src/layout-docs.test.ts fails when this block differs -->
|
|
16
|
+
|
|
17
|
+
```text
|
|
18
|
+
repro/
|
|
19
|
+
├── README.md ← the index: one line per study
|
|
20
|
+
├── .gitignore ← keeps the scripts' byproducts out of git
|
|
21
|
+
├── tools/ ← scripts used across the paper (numbers, figures, anonymity check, harnesses)
|
|
22
|
+
├── data/ ← shared data (tables, caches)
|
|
23
|
+
└── <slug>-YYYY-MM-DD/ ← one folder per study or spike (<slug>-YYYY-MM for one that ran over weeks)
|
|
24
|
+
└── README.md ← required: what it asks, how to run it, what it found
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
<!-- layout:end -->
|
|
28
|
+
|
|
29
|
+
`paperlint new` makes `repro/README.md` (a stub index), `repro/tools/` and `repro/data/` (each with a
|
|
30
|
+
`.gitkeep` until it holds a file). `repro/` itself gets no `.gitkeep`: its top allows only the entries
|
|
31
|
+
above, and it already holds the index.
|
|
32
|
+
|
|
33
|
+
A loose note belongs in the folder of the study it is about, or in the paper's `research/` when it is
|
|
34
|
+
research or a decision rather than the record of a run.
|
|
35
|
+
|
|
36
|
+
**The date in a study's name** is the day the study began: `ablation-2026-10-12`. A month is
|
|
37
|
+
accepted too, `human-labels-2026-10`: a study that ran over weeks (labelling, reading a corpus) has a
|
|
38
|
+
month as its honest date, and both forms sort by time. The date must be a real one — no month 13, no
|
|
39
|
+
30 February. The slug is lowercase letters and digits, words joined by single hyphens.
|
|
40
|
+
|
|
41
|
+
Never reported: what interpreters, package managers, test runners and the operating system leave in
|
|
42
|
+
the folder they run in — `__pycache__/`, `.pytest_cache/`, `.mypy_cache/`, `.ruff_cache/`,
|
|
43
|
+
`.ipynb_checkpoints/`, `node_modules/`, `.venv/`, `.DS_Store`, `Thumbs.db`.
|
|
44
|
+
|
|
45
|
+
## Why
|
|
46
|
+
|
|
47
|
+
`repro/` is where the evidence for every number lives, and it fills up like a desk. One real paper
|
|
48
|
+
had 120 entries at its top: about 70 study folders, some with no date, and about 50 loose scripts,
|
|
49
|
+
data files and notes. Nobody could say which script produces which number, which study is current,
|
|
50
|
+
or where a new run goes. Each kind of material now has one place, and each study says, in its
|
|
51
|
+
`README.md`, what it is.
|
|
52
|
+
|
|
53
|
+
## Examples
|
|
54
|
+
|
|
55
|
+
Failing: a script loose at the top.
|
|
56
|
+
|
|
57
|
+
```text
|
|
58
|
+
papers/agent-rule-drift/PIPELINE-STATUS.md
|
|
59
|
+
1:1 error `repro/paper_numbers.py` is loose at the top of the paper's repro/ folder, which holds
|
|
60
|
+
only an index, the shared tools and data, and one dated folder per study. Where it
|
|
61
|
+
goes: repro/tools/ — the scripts used across the paper (numbers, figures, the anonymity
|
|
62
|
+
check, harnesses); a script only one study runs goes in that study's folder instead.
|
|
63
|
+
What repro/ holds (what interpreters and package managers leave there, like __pycache__/ and node_modules/, is never reported):
|
|
64
|
+
repro/
|
|
65
|
+
├── README.md ← the index: one line per study
|
|
66
|
+
…
|
|
67
|
+
Rule repro/allowed-files-only: https://github.com/zernie/paperlint/blob/main/docs/rules/repro/allowed-files-only.md repro/allowed-files-only
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Each message names the entry and where it goes, and draws the layout above, so it reads on its own. Where
|
|
71
|
+
an entry goes depends on what it is:
|
|
72
|
+
|
|
73
|
+
| entry | where it goes |
|
|
74
|
+
| ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
|
|
75
|
+
| a script (`.py`, `.ipynb`, `.sh`, `.js`, `.mjs`, `.ts`, `.r`, `.jl`, `.sql` …), `Makefile`, `requirements.txt`, `package.json` | `repro/tools/`, or the folder of the one study that runs it |
|
|
76
|
+
| data (`.tsv`, `.csv`, `.json`, `.jsonl`, `.yaml`, `.txt`, `.log`, `.parquet`, `.zip` …) | `repro/data/`, or the folder of the one study that produced it |
|
|
77
|
+
| a note (`.md`) | the dated folder of the study it is about, or the paper's `research/` |
|
|
78
|
+
| a folder whose name is not `<slug>-YYYY-MM-DD` | rename it to `<slug>-YYYY-MM-DD` and give it a `README.md`; its first commit's date is a fine default |
|
|
79
|
+
| a study folder without `README.md` | add one: what the study asks, how to run it, what it found; and a line in `repro/README.md` |
|
|
80
|
+
| anything else | by what it is, as above |
|
|
81
|
+
|
|
82
|
+
Three files paperlint itself once kept at the top of `repro/` are named with their new place and the
|
|
83
|
+
command that moves them: `references-cache.json` → `repro/data/` (read by `paperlint build`),
|
|
84
|
+
`refs-cache.json` → `repro/data/`, `generated-code-grandfathered.txt` → `repro/tools/`.
|
|
85
|
+
|
|
86
|
+
To date a folder by its first commit, run in the paper folder:
|
|
87
|
+
|
|
88
|
+
```sh
|
|
89
|
+
git log --reverse --format=%as -- repro/old-baseline | head -1
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Passing: a paper with no `repro/`, and a `repro/` holding only the entries above.
|
|
93
|
+
|
|
94
|
+
## Options / preset fields
|
|
95
|
+
|
|
96
|
+
No rule options.
|
|
97
|
+
|
|
98
|
+
## What it does not check
|
|
99
|
+
|
|
100
|
+
- What is inside `tools/`, `data/` and the study folders.
|
|
101
|
+
- Whether `repro/README.md` exists, or has a line for every study.
|
|
102
|
+
- What a study's `README.md` says.
|
|
103
|
+
- `repro/` itself: whether the paper has one is `paper/allowed-files-only`'s concern, and that rule never
|
|
104
|
+
looks inside it. Each entry is judged by exactly one of the two rules.
|
|
105
|
+
|
|
106
|
+
## The hook
|
|
107
|
+
|
|
108
|
+
The hook `paper-folder-layout` refuses, before it is written, a new file that would add an entry
|
|
109
|
+
this rule reports — a loose file at the top of `repro/`, an undated folder, or a study folder whose
|
|
110
|
+
first file is not its `README.md` — with this rule's message as the reason. It never refuses a file
|
|
111
|
+
that already exists, and it cannot see a shell `mv` or `cp`: the lint is the guarantee. What it does
|
|
112
|
+
and does not do is on [`paper/allowed-files-only`'s page](../paper/allowed-files-only.md#the-hook-a-new-stray-entry-is-refused-before-it-is-written).
|
|
113
|
+
|
|
114
|
+
## How to fix
|
|
115
|
+
|
|
116
|
+
Move or rename the entry where its message says, and update whatever names its path:
|
|
117
|
+
|
|
118
|
+
```sh
|
|
119
|
+
git mv papers/agent-rule-drift/repro/paper_numbers.py papers/agent-rule-drift/repro/tools/
|
|
120
|
+
git mv papers/agent-rule-drift/repro/baseline papers/agent-rule-drift/repro/baseline-2026-08-05
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
To switch the rule off for one paper — say, one whose `repro/` predates the layout — set it in that
|
|
124
|
+
paper's `paperlint.json`; `paper/allowed-files-only` still judges the paper's root:
|
|
125
|
+
|
|
126
|
+
```json
|
|
127
|
+
{ "rules": { "repro/allowed-files-only": "off" } }
|
|
128
|
+
```
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# study/declare-sample
|
|
2
|
+
|
|
3
|
+
> Experimental, off by default. Whether this belongs in a lint rule or in the study skills is open:
|
|
4
|
+
> see [#214](https://github.com/zernie/paperlint/issues/214).
|
|
5
|
+
|
|
6
|
+
**Level:** off unless turned on; a paper turns it on with `"rules": { "study/declare-sample": "error" }`
|
|
7
|
+
in its `paperlint.json` · **Reads:** `repro/<study>/prereg.json`, for every study folder at the top of the
|
|
8
|
+
paper's `repro/` that holds one (every folder there but the shared `tools/` and `data/`, whatever
|
|
9
|
+
its name) · **Reported on:** the first line of the paper's
|
|
10
|
+
`PIPELINE-STATUS.md`, once per problem
|
|
11
|
+
|
|
12
|
+
## What it catches
|
|
13
|
+
|
|
14
|
+
A study's pre-registration that does not say how its sample relates to the population the paper
|
|
15
|
+
talks about. Every `prereg.json` in a study folder (`repro/<slug>-YYYY-MM-DD/`) must carry five
|
|
16
|
+
fields; every other key in the file is the author's and is not read.
|
|
17
|
+
|
|
18
|
+
| field | holds |
|
|
19
|
+
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
20
|
+
| `population` | `about`: what the paper's claim is about, in its own words · `source`: where the units come from (census, crawl, registry, with its date) · `size`: how many units, a whole number above 0 (an estimate is fine; say how in `source`) |
|
|
21
|
+
| `strata` | `by`: the property the screens are counted by (rule type, language, …) · `values`: its values, at least one, each once |
|
|
22
|
+
| `screens` | one entry per screen applied between the population and the sample: `name` (what it keeps, as applied) and `removed`, an object with a count for **every** declared stratum value (0 where it removed none) and no other key. `[]` when no screen was applied |
|
|
23
|
+
| `coverage` | `share`: the share of the population the sample can represent, 0 to 1, or `null` with a `reason` · `unreachable`: the strata or kinds of units the sample cannot reach (`[]` only when it reaches them all; a `share` of 1 with a non-empty list is refused) |
|
|
24
|
+
| `author_check` | `who` looked at the drawn units before the run, `when` (YYYY-MM-DD), `units_seen` of `units_drawn` (the units drawn — rules, cases — not the items made from them), and the `verdict` to "is this what you meet in practice?": `representative`, `partly` or `not-representative` |
|
|
25
|
+
|
|
26
|
+
A verdict other than `representative` needs `author_check.decision`: `proceed` (true or false), `by`
|
|
27
|
+
whom, and `why`. A doubtful verdict with no recorded decision is an error; with one, either way, it
|
|
28
|
+
passes: the rule asks that the decision be made and written down, not which way it goes.
|
|
29
|
+
|
|
30
|
+
A study folder without a `prereg.json` owes nothing here.
|
|
31
|
+
|
|
32
|
+
**When it acts.** Inside stage 3, the study: `build-benchmark` writes the `prereg.json` before the
|
|
33
|
+
sample is drawn, the author checks the drawn units, and only then the run and any labelling
|
|
34
|
+
(`build-label-page`) follow. Once a paper turns the rule on, it runs on every `paperlint lint` and in CI, so a study whose
|
|
35
|
+
pre-registration lacks a field fails from the moment the file exists. The flow and the map:
|
|
36
|
+
[`layout.md`](../../layout.md).
|
|
37
|
+
|
|
38
|
+
## Why
|
|
39
|
+
|
|
40
|
+
A measurement study of how language models read coding rules drew its rules from three pools, kept
|
|
41
|
+
only the rules judgeable from one code excerpt of at most 40 lines, restricted one pool to a
|
|
42
|
+
classifier's "enforceable" lane, and let a model pick short excerpts. Each screen was reasonable
|
|
43
|
+
alone; together they left a sample of crisp rules. The labeller found 93 % of the items designed as
|
|
44
|
+
borderline clear-cut, and the rules looked nothing like the ones met in practice. A spot-check of
|
|
45
|
+
ten items flagged it (seven of ten clear-cut), and the run went ahead.
|
|
46
|
+
|
|
47
|
+
Nothing in that design forced anyone to state what share of the real population the sample covered,
|
|
48
|
+
or which screens removed which kinds of rules. Those are the statements this rule requires, before
|
|
49
|
+
the sample is frozen, in the file that freezes it. The counts per stratum are what make the drift
|
|
50
|
+
visible: a screen that removes 120 crisp rules and 1,630 vague ones is a screen on the construct.
|
|
51
|
+
The author's check is on the drawn **units**, because ten items made from crisp rules look fine one
|
|
52
|
+
by one.
|
|
53
|
+
|
|
54
|
+
## Examples
|
|
55
|
+
|
|
56
|
+
Failing, with the rule turned on: a pre-registration with hypotheses and arms, and none of the five fields.
|
|
57
|
+
|
|
58
|
+
```text
|
|
59
|
+
papers/rule-reading/PIPELINE-STATUS.md
|
|
60
|
+
1:1 error `repro/draw-2026-10-12/prereg.json`: `population` is missing — add `population`: what
|
|
61
|
+
the paper's claim is about (`about`), where its units come from (`source`) and how many
|
|
62
|
+
there are (`size`). Why: a study once screened its sample down to rules that are easy
|
|
63
|
+
to judge — its labeller found 93 % of the items designed as borderline clear-cut — and
|
|
64
|
+
nothing in its design made anyone state what share of the real population the sample
|
|
65
|
+
covered or which screens removed which kinds of units. Rule study/declare-sample: …
|
|
66
|
+
study/declare-sample
|
|
67
|
+
1:1 error … `strata` is missing …
|
|
68
|
+
1:1 error … `screens` is missing …
|
|
69
|
+
1:1 error … `coverage` is missing …
|
|
70
|
+
1:1 error … `author_check` is missing …
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Failing: a screen counted under a value `strata` does not declare, and a doubtful check with no
|
|
74
|
+
decision.
|
|
75
|
+
|
|
76
|
+
```text
|
|
77
|
+
1:1 error `repro/draw-2026-10-12/prereg.json`: `screens[0].removed.procedural` is not a declared
|
|
78
|
+
stratum value — `strata.values` declares crisp, parameterised, vague; add it there or
|
|
79
|
+
count it under one of them. …
|
|
80
|
+
1:1 error `repro/draw-2026-10-12/prereg.json`: `author_check.decision` is missing — the verdict
|
|
81
|
+
is «partly», and a doubtful verdict with no recorded decision is how a sample the author
|
|
82
|
+
doubted gets run anyway: add `proceed` (true or false), `by` whom and `why`. …
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Passing:
|
|
86
|
+
|
|
87
|
+
```json
|
|
88
|
+
{
|
|
89
|
+
"hypotheses": ["…the author's own keys, not read…"],
|
|
90
|
+
"population": {
|
|
91
|
+
"about": "rules written in agent instruction files of public repositories",
|
|
92
|
+
"source": "a census of instruction files, crawled 2026-07-28",
|
|
93
|
+
"size": 4210
|
|
94
|
+
},
|
|
95
|
+
"strata": {
|
|
96
|
+
"by": "rule type",
|
|
97
|
+
"values": ["crisp", "parameterised", "vague"]
|
|
98
|
+
},
|
|
99
|
+
"screens": [
|
|
100
|
+
{
|
|
101
|
+
"name": "judgeable from one code excerpt of at most 40 lines",
|
|
102
|
+
"removed": { "crisp": 120, "parameterised": 410, "vague": 1630 }
|
|
103
|
+
}
|
|
104
|
+
],
|
|
105
|
+
"coverage": {
|
|
106
|
+
"share": 0.31,
|
|
107
|
+
"unreachable": ["vague", "rules about process rather than code"]
|
|
108
|
+
},
|
|
109
|
+
"author_check": {
|
|
110
|
+
"who": "first author",
|
|
111
|
+
"when": "2026-10-12",
|
|
112
|
+
"units_seen": 24,
|
|
113
|
+
"units_drawn": 24,
|
|
114
|
+
"verdict": "partly",
|
|
115
|
+
"decision": {
|
|
116
|
+
"proceed": true,
|
|
117
|
+
"by": "first author",
|
|
118
|
+
"why": "the claim is narrowed to crisp and parameterised rules; vague rules are named as out of reach"
|
|
119
|
+
}
|
|
120
|
+
}
|
|
121
|
+
}
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
## Options / preset fields
|
|
125
|
+
|
|
126
|
+
No rule options.
|
|
127
|
+
|
|
128
|
+
## What it does not check
|
|
129
|
+
|
|
130
|
+
- Whether the numbers are true: that `removed` adds up to what the pools held, that `share` follows
|
|
131
|
+
from them, or that the author really looked. It checks that each was stated, in a shape a reader
|
|
132
|
+
and a script can use.
|
|
133
|
+
- That the paper's text matches `population.about` or names what `coverage.unreachable` lists. That
|
|
134
|
+
is reading, and the paper skills do it (`build-benchmark`, `draft-paper`'s threats section).
|
|
135
|
+
- A `prereg.json` deeper than a study folder's top (`repro/<study>/sub/prereg.json`), at the top of
|
|
136
|
+
`repro/` itself, or under any other name. The study's folder and its name are `repro/allowed-files-only`'s.
|
|
137
|
+
- A study without a `prereg.json`.
|
|
138
|
+
|
|
139
|
+
## How to fix
|
|
140
|
+
|
|
141
|
+
Add the five fields to the study's `prereg.json` before the sample is frozen: count what each screen
|
|
142
|
+
removed per stratum value from the screening script's own output, and show the author the drawn
|
|
143
|
+
units with the question "is this what you meet in practice?". A pre-registration already frozen and
|
|
144
|
+
run gets the fields as a dated addendum, with the values measured now; a paper that cannot give them
|
|
145
|
+
leaves the rule off, which is the default; one that turned it on and cannot give them sets it back
|
|
146
|
+
to `"off"` in its `paperlint.json`, which says so in the record.
|
package/docs/rules.md
CHANGED
|
@@ -3,54 +3,57 @@
|
|
|
3
3
|
The README lists what each check catches in one line. This page says exactly which file each one
|
|
4
4
|
reads and when it fails. Errors fail `paperlint lint`; warnings print and do not.
|
|
5
5
|
|
|
6
|
-
| Rule | Level
|
|
7
|
-
| --------------------------------------------------------------------- |
|
|
8
|
-
| `paper/stages` | error
|
|
9
|
-
| `paper/source` | error
|
|
10
|
-
| [`
|
|
11
|
-
| [`cycle/
|
|
12
|
-
| [`cycle/
|
|
13
|
-
| [`cycle/
|
|
14
|
-
| [`
|
|
15
|
-
| [`paper/
|
|
16
|
-
| `paper/
|
|
17
|
-
| `
|
|
18
|
-
| `paper/
|
|
19
|
-
| `paper/
|
|
20
|
-
|
|
|
21
|
-
| `paper/
|
|
22
|
-
| `paper/
|
|
23
|
-
| `
|
|
24
|
-
| `
|
|
25
|
-
| `
|
|
26
|
-
|
|
|
27
|
-
|
|
|
28
|
-
|
|
|
29
|
-
| [`
|
|
30
|
-
| [`
|
|
31
|
-
| [`tex/
|
|
32
|
-
| [`
|
|
33
|
-
| [`
|
|
34
|
-
| [`tex/
|
|
35
|
-
| [`tex/claim-
|
|
36
|
-
| [`tex/
|
|
37
|
-
| `
|
|
38
|
-
| `
|
|
39
|
-
| `
|
|
40
|
-
| `
|
|
41
|
-
| `
|
|
42
|
-
| `pdf/
|
|
43
|
-
|
|
|
44
|
-
|
|
|
45
|
-
| `pdf/
|
|
46
|
-
| [`
|
|
47
|
-
| [`
|
|
48
|
-
|
|
|
49
|
-
| [`talk/
|
|
50
|
-
| [`talk/
|
|
51
|
-
| [`talk/
|
|
52
|
-
| [`talk/
|
|
53
|
-
| [`talk/
|
|
6
|
+
| Rule | Level | Reads | Fails when |
|
|
7
|
+
| --------------------------------------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
8
|
+
| `paper/stages` | error | `PIPELINE-STATUS.md` | a declared stage's PDF is missing, or the `bytes:` it names does not match the file's real size, or a frozen PDF exists that no stage declares |
|
|
9
|
+
| `paper/source` | error | `PIPELINE-STATUS.md` | a declared stage has no frozen `.tex` beside its PDF (a commit hash does not count — squash and gc destroy it) |
|
|
10
|
+
| [`study/declare-sample`](rules/study/declare-sample.md) | experimental; off unless a paper turns it on (`"error"`) | each study's `repro/<study>/prereg.json` | a study's pre-registration does not declare its `population` (what the claim is about, source, size), the `strata` and what each of its `screens` removed per stratum value, its `coverage` (share 0–1 or null with a reason, and the strata it cannot reach), or the `author_check` of the drawn units (a doubtful verdict needs a recorded `decision`); reported on `PIPELINE-STATUS.md`, once per problem — [page](rules/study/declare-sample.md) |
|
|
11
|
+
| [`cycle/record`](rules/cycle/record.md) | error | `paperlint.json` (`cycles`) | the paper's attempts at venues (`cycles`, one entry per attempt) hold one source's deadlines out of order (registration ≤ submission ≤ resubmission < notification < camera-ready), or copy a portal reading the venue's preset carries; reported on `PIPELINE-STATUS.md`. A record that does not parse — a bare date, two open cycles, an unknown key — is refused by `paperlint lint` itself before any rule runs, exit 2 — [page](rules/cycle/record.md) |
|
|
12
|
+
| [`cycle/evidence`](rules/cycle/evidence.md) | error | `paperlint.json` (`cycles`) → the paper folder | a closed cycle (`accepted`, `rejected`, `withdrawn`) or a deadline's `override` names an `evidence` file — the decision mail, the reviews, the withdrawal note, the chairs' mail — that is not in the paper folder; reported on `PIPELINE-STATUS.md` — [page](rules/cycle/evidence.md) |
|
|
13
|
+
| [`cycle/stage`](rules/cycle/stage.md) | error | `PIPELINE-STATUS.md` → `paperlint.json` (`cycles`) | with `cycles` declared, a stage names no `cycle: <id>`, names one no cycle has, or a `submitted`/`camera-ready` stage belongs to a cycle whose `phase` is still `porting` — [page](rules/cycle/stage.md) |
|
|
14
|
+
| [`cycle/port-done`](rules/cycle/port-done.md) | error | `paper.tex` → `paperlint.json` (`cycles`) → the preset's `template` | the current cycle still declares `"phase": "porting"` (which silences the venue's format rules) while the `\documentclass` is already the venue's template: the port is done, drop the phase — [page](rules/cycle/port-done.md) |
|
|
15
|
+
| [`paper/folder-venue-leftover`](rules/paper/folder-venue-leftover.md) | warn | the folder name → `paperlint.json` | the paper's folder is named after another shipped venue than the one it extends (`papers/aisec-2026/` for a paper now set for AIDC); reported on `PIPELINE-STATUS.md` — [page](rules/paper/folder-venue-leftover.md) |
|
|
16
|
+
| [`paper/allowed-files-only`](rules/paper/allowed-files-only.md) | error | the paper folder's root listing → `paperlint.json` | a file or folder at the paper's root is not part of the layout (`PLAN.md`, `archive/`, a stray PDF); reported on `PIPELINE-STATUS.md`, once per entry, with where it goes; `extra/` takes anything — [page](rules/paper/allowed-files-only.md) |
|
|
17
|
+
| [`repro/allowed-files-only`](rules/repro/allowed-files-only.md) | error | the listing of the paper's `repro/` | an entry at the top of `repro/` is not `README.md`, `.gitignore`, `tools/`, `data/` or a study folder named `<slug>-YYYY-MM-DD` (or `<slug>-YYYY-MM`), or a study folder has no `README.md`; reported on `PIPELINE-STATUS.md`, once per entry, with where it goes by kind (a script → `repro/tools/`, data → `repro/data/`, a note → its study or `research/`) — [page](rules/repro/allowed-files-only.md) |
|
|
18
|
+
| [`paper/sources-fresh`](rules/paper/sources-fresh.md) | warn | `paper.tex` → `_build/sources.json` | the paper has not been built (no record of the files TeX read), or a file changed since the last build (its SHA-256 differs) — run `npx paperlint build` |
|
|
19
|
+
| `paper/author-list` | error | `paper.tex` → `_build/references.json`, `_build/sources.json` | an entry's authors are not those of the version it cites — typically the arXiv preprint's list under a published venue. Checked online by `paperlint build` (DBLP), reported on the entry's own line (or at the top of `paper.tex`, naming the entry's file); silent without a current build record |
|
|
20
|
+
| `paper/section-word` | warn | `paper.tex` and its body files | `§` or `\S\ref` instead of the word Section. **Fixable:** `--fix` writes `Section~\ref{…}` / `Section 5` |
|
|
21
|
+
| `paper/leading-zero` | warn | `paper.tex` and its body files | a decimal below 1 without its leading zero (`.05` instead of `0.05`, IEEE / ISO 80000-1 style), counted only where the reader sees it — prose, math and table cells, not macro options, column specs, comments, code or tikz. A number right after `¶` or `§` (`¶¶.42`) is a paragraph designator and is not reported; in a paragraph that cites numbers that way, another bare decimal is reported without a fix, with the zero as a suggestion. **Fixable** |
|
|
22
|
+
| `paper/figure-ref-style` | warn | `paper.tex` and its body files | `Fig.~\ref` and `Figure~\ref` mixed in one document; each minority occurrence is reported. **Fixable:** `--fix` writes the majority form |
|
|
23
|
+
| [`bib/reachable-entry`](rules/bib/reachable-entry.md) | warn | `paper.tex` → `_build/sources.json`: the databases the last build's bibtex opened | an entry has no doi, url or arXiv id — a reader has nothing to follow. Reported on the entry in the `filecontents` block of `paper.tex` that wrote its `.bib`, or at the top of `paper.tex` with the entry's file and line first; silent until a build records the paper; no fix, the link has to be looked up — [page](rules/bib/reachable-entry.md) |
|
|
24
|
+
| `paper/cite-exists` | error | `paper.tex` → `_build/references.json` | an entry's doi or arXiv id resolves to nothing, or to a different work. Checked online by `paperlint build`; a work that was merely not found is not a finding |
|
|
25
|
+
| `paper/refs-fresh` | error | `paper.tex` → `_build/references.json` | the databases bibtex opened changed since the build checked them (their SHA-256 differs) — run `npx paperlint build` |
|
|
26
|
+
| `paper/refs-checked` | warn | `paper.tex` → `_build/references.json` | a build recorded the paper's databases and none has checked their references, or the last build could not (no network), or its bibtex typeset a key paperlint's reader did not read (an entry behind `%`, inside `@comment{…}`), or bibtex found a database outside the paper directory (`BIBINPUTS`) — so the two rules above did not run on it. Silent without a record of the build (`paper/sources-fresh` speaks) |
|
|
27
|
+
| `tex/future-promise` | warn | `paper.tex` and its body files | a camera-ready build still says "will be released" about something already handed over |
|
|
28
|
+
| `tex/acm-frontmatter-override` | error | `paper.tex` | an `acmart` build overrides ACM's front-matter commands and drops template elements from page 1 |
|
|
29
|
+
| [`tex/template`](rules/tex/template.md) | error | `paper.tex` → `paperlint.json` | the `\documentclass` is not the preset's `template` class, or lacks an option it names — [page](rules/tex/template.md) |
|
|
30
|
+
| [`tex/required-section`](rules/tex/required-section.md) | error | `paper.tex` → `paperlint.json` | a section the preset requires (AIDC: «LLM Usage Statement», last) is missing or not where it must be — [page](rules/tex/required-section.md) |
|
|
31
|
+
| [`tex/heading-case`](rules/tex/heading-case.md) | optional; the AgenticDev preset sets `error` | `paper.tex` and the files it includes | a word in the paper's `\title` or in a `\section`, `\subsection`, `\subsubsection`, `\paragraph` or `\subparagraph` title is not in the style its scope asks: `{ "title", "headings", "levels"? }`, each `chicago-headline` (Conference Publishing's quote of the Chicago Manual of Style), `sentence` (first word only) or `any`. Words that are a preposition in one title and a particle in another (`up`, `on`, `in`) are not read mid-title. **Fixable:** `--fix` — [page](rules/tex/heading-case.md) |
|
|
32
|
+
| [`format/layout-override`](rules/format/layout-override.md) | error | `paper.tex` → `paperlint.json` | the source changes the page layout the venue's template sets for the whole document — `geometry`, a layout `\setlength`, `\linespread`, `\baselinestretch` — [page](rules/format/layout-override.md) |
|
|
33
|
+
| [`anonymity/identity`](rules/anonymity/identity.md) | error | `paper.tex` → `_build/paper.facts.json` | a double-blind venue (`"blind": true`) and no `identity` declared, or a declared name, handle, email or project in the PDF's text, metadata or link targets — [page](rules/anonymity/identity.md) |
|
|
34
|
+
| [`tex/venue-leftover`](rules/tex/venue-leftover.md) | warn | `paper.tex` → `paperlint.json` | the text names another shipped venue than the one the paper extends («Submission to VenueA 2026» in a paper now set for VenueB) — [page](rules/tex/venue-leftover.md) |
|
|
35
|
+
| [`tex/claim-provenance`](rules/tex/claim-provenance.md) | warn | `paper.tex` | a sentence of the body states a number and does not say whose it is — no citation or link, no _we_/_our_, no `\ref` or "Table 2", no sample ("n = 134", "48 runs") — [page](rules/tex/claim-provenance.md) |
|
|
36
|
+
| [`tex/register`](rules/tex/register.md) | warn | `paper.tex` and the files it includes | the body runs to a post's register: more than 0.8 sentences per 1000 words open with _And_, _So_, _But_, _Nor_, _Or_ or _Yet_ — the limit is an option — [page](rules/tex/register.md) |
|
|
37
|
+
| [`tex/contrast-frames`](rules/tex/contrast-frames.md) | warn | `paper.tex` and the files it includes → `paperlint.json` | contrast frames (_X, not Y_, _not X but Y_, _rather than_, _instead of_, _as opposed to_) per 10,000 words fall outside the band the venue preset's anchors set (AIDC: 0–10.4, from five accepted ACSAC papers); above it, each frame is reported too — [page](rules/tex/contrast-frames.md) |
|
|
38
|
+
| [`tex/claim-emphasis`](rules/tex/claim-emphasis.md) | warn | `paper.tex` and the files it includes → `paperlint.json` | claims set in bold inside a sentence (six words or more, or holding a number; not a term, not a label opening a paragraph) per 10,000 words fall outside the preset's band (AIDC: 0–17.8); above it, each is reported too — [page](rules/tex/claim-emphasis.md) |
|
|
39
|
+
| [`tex/relation-markers`](rules/tex/relation-markers.md) | warn | `paper.tex` and the files it includes → `paperlint.json` | words that name how a sentence relates to the one before (_because_, _therefore_, _however_, _for example_ …) per 10,000 words fall outside the preset's band (AIDC: 25.4–135.0) — the floor that catches relations left to infer — [page](rules/tex/relation-markers.md) |
|
|
40
|
+
| `review/frontmatter` | error | `reviews/*.md` | the review's frontmatter fails paperlint's JSON Schema — a `findings` record without `id`/`status`, an unknown field, or an **open** finding with no `cause` (`skill-defect` · `missing-skill` · `hook` · `rule`). A review with no `findings` key is not checked for findings |
|
|
41
|
+
| `sibling/frontmatter` | warn | `siblings/*.md` (not `README.md`) | a sibling card's frontmatter has no `read:` saying how much of the competing paper was read (`full` · `abstract` · `none`) |
|
|
42
|
+
| `pdf/fresh` | error | `paper.tex` → `_build/paper.facts.json` | the paper names a venue and the facts cannot be judged: not JSON, a schema other than 3, or they describe a PDF that is gone or differs from the one on disk (its SHA-256) |
|
|
43
|
+
| `pdf/profile` | error | `paper.tex` → `paperlint.json` | `paperlint.json` is not JSON or has an unknown key, its `extends` does not resolve (not found, a cycle, a chain longer than four, a preset that fails the schema, an npm name), it names no `kind` while the preset has kinds, or names a kind the preset does not have |
|
|
44
|
+
| `pdf/fonts` | error | `paper.tex` → `_build/paper.facts.json` | a font the pages draw is Type 3 or not embedded, or no font starts with the family the venue preset names for body text (`fonts_text`) or headings (`fonts_title`) |
|
|
45
|
+
| `pdf/geometry` | error | `paper.tex` → `_build/paper.facts.json` | the page width or height is more than `dimTol` (default 0.05 in) off the preset's, or the column count differs |
|
|
46
|
+
| [`format/page-limit`](rules/format/page-limit.md) | error | `paper.tex` → `_build/paper.facts.json` | body or reference pages exceed the limits of the paper's kind. With `body_ends_at: "references"` (AIDC) the body ends at the references or the appendix, whichever comes first — found by hyperref's anchors, then the heading line — and the page it ends on counts when body text stands above the heading; with `appendix_in_body` too (MSR) the appendix is body wherever it sits and the reference pages are the ones holding only references; an end it cannot find, or signals that disagree, are reported — [page](rules/format/page-limit.md) |
|
|
47
|
+
| [`pdf/body-size`](rules/pdf/body-size.md) | error | `paper.tex` → `_build/paper.facts.json` | the body font size is more than `body_pt_tol` off the preset's, or the reference font size is outside the preset's range widened by `body_pt_tol`. banal reads 0.3 pt above the size the glyphs are set in; checked on five built accepted papers (none reported) and three variants one size off (each reported) — [page](rules/pdf/body-size.md) |
|
|
48
|
+
| `pdf/measured` | warn | `paper.tex` → `_build/paper.facts.json` | the paper's `paperlint.json` names no venue preset yet; or it names one and there are no facts (it was not built), or the facts carry no page geometry (banal was not found) — so the checks above did not run |
|
|
49
|
+
| [`talk/profile`](rules/talk/profile.md) | error | `paper.tex` → `paperlint.json` | the paper declares `talk` and names no preset, the preset has no `talk` block, the mode is not one the venue allows, or the venue sets no talk slot for the paper's kind — [page](rules/talk/profile.md) |
|
|
50
|
+
| [`talk/required-files`](rules/talk/required-files.md) | error | `paper.tex` → `paperlint.json`, the talk folder | a file the venue requires for the paper's talk mode (video, one-slide, captions…) is not in the talk folder — [page](rules/talk/required-files.md) |
|
|
51
|
+
| [`talk/undeclared`](rules/talk/undeclared.md) | warn | `paper.tex` → the paper's `talk/` folder | an `.mp4` sits in `talk/` and the paper declares no `talk`, so nothing checks it — [page](rules/talk/undeclared.md) |
|
|
52
|
+
| [`talk/duration`](rules/talk/duration.md) | error | `paper.tex` → the talk video | the video runs longer than the slot's `talk_s_max`, or its audio track is more than 1 s off the video's length — [page](rules/talk/duration.md) |
|
|
53
|
+
| [`talk/duration-floor`](rules/talk/duration-floor.md) | warn | `paper.tex` → the talk video | the video runs shorter than the slot's `talk_s_min` — [page](rules/talk/duration-floor.md) |
|
|
54
|
+
| [`talk/video-format`](rules/talk/video-format.md) | error | `paper.tex` → the talk video | the video is not ISO media, is QuickTime where the venue asks for MP4, is under `min_height_px`, or over `max_bytes` — [page](rules/talk/video-format.md) |
|
|
55
|
+
| [`talk/one-slide-size`](rules/talk/one-slide-size.md) | error | `paper.tex` → the one-slide PNG | the one-slide image is not the size the paper or the preset states, or not a PNG — [page](rules/talk/one-slide-size.md) |
|
|
56
|
+
| [`talk/captions-cover`](rules/talk/captions-cover.md) | warn | `paper.tex` → the captions and the video | the captions start late, end early or leave a long gap (uncalibrated thresholds) — [page](rules/talk/captions-cover.md) |
|
|
54
57
|
|
|
55
58
|
**Body files** are the files of the paper's directory that TeX read after `\begin{document}` in the
|
|
56
59
|
last build, as `paperlint build` recorded them in `_build/sources.json` — not a preamble file, not a
|
|
@@ -72,8 +75,8 @@ The `talk/*` rules judge a talk's finished files and are silent for a paper that
|
|
|
72
75
|
`talk` (but `talk/undeclared`): [`talk.md`](talk.md).
|
|
73
76
|
|
|
74
77
|
Optional rules — off unless you turn them on in the `rules` setting, because only some venues need
|
|
75
|
-
them — are on their own page: [`optional-rules.md`](optional-rules.md). Today there are
|
|
76
|
-
`pdf/last-page-balance` and `
|
|
78
|
+
them — are on their own page: [`optional-rules.md`](optional-rules.md). Today there are three,
|
|
79
|
+
`pdf/last-page-balance`, `tex/heading-case` and the experimental `study/declare-sample`.
|
|
77
80
|
|
|
78
81
|
## A deliberate exception: a disable directive, with the reason
|
|
79
82
|
|