paperlint 10.2.0 → 10.4.0

package/docs/rules/repro/outputs-before-decision.md

@@ -0,0 +1,119 @@
+# repro/outputs-before-decision
+
+**Level:** error · **Reads:** `repro/studies.json`, each study's `plan.json`, the files under a
+decide-first study's folder, and git's history of them · **Reported on:** the first line of the
+paper's `PIPELINE-STATUS.md`, once per finding
+
+## What it catches
+
+A study decided before its run — its plan's `mode` is `decide_before_run`, the default
+`paperlint studies new` writes for a measurement — whose history does not show that order:
+
+```json
+{
+  "mode": {
+    "kind": "decide_before_run",
+    "outputs": ["out/**", "RESULTS.md"],
+    "decision": {
+      "file": "repro/P1-S01-outcome/decision.md",
+      "note": "exclusions, veto, and the sample approved"
+    }
+  }
+}
+```
+
+- **`output-before-decision`** — an output (a file the `outputs` globs match, under the study's
+  folder) whose FIRST commit is not a strict descendant of the decision file's FIRST commit: earlier,
+  on a line of history the decision is not on, or in the very same commit, which says nothing about
+  which came first. An output committed and later deleted still counts.
+- **`undecided-output`** — an output in the tree, or one ever committed and since deleted, while the
+  decision file has never been committed.
+- **`decision-is-output`** — the decision file lies in the study's folder and an output glob matches
+  it: the decision would be a file its own run produces.
+- **`decision-deleted`** — the decision file was committed and later deleted: it is not in HEAD, so
+  the study has no decision its outputs could follow, and the order is not judged against it.
+- **`decision-not-file`** — the decision path names a directory or a symbolic link: git answers for
+  a directory with the commits of anything under it, and records a link rather than the file it
+  points to, so no order is read from either. (A path that names the paper's folder
+  itself, `.`, or ends in `/` is refused when the plan is parsed.)
+- **`shallow`** — the clone is shallow, so every file's first commit is the clone's: no order can be
+  read, and the rule says so instead of judging.
+- **`no-history`** — git could not answer (not installed, not a repository).
+
+The study's own files — `plan.json`, `README.md` and the decision file — are never outputs. A symbolic
+link in the study's folder is one entry, as git stores it, and is never followed: a link to the folder
+itself cannot loop the walk, and one to a tree outside brings none of its files into the study.
+
+## Why
+
+A measurement's inputs — which runs count, which units are excluded, whether the sample is approved
+— must be decided before anyone has seen an outcome; otherwise the decision can follow the result.
+"Decide first, then compute" needs no key and no encryption: the decision is committed, the run
+starts only after (`paperlint studies gate` exits 0 only then), and git's history is the proof of
+order. This rule re-reads that proof on every lint, so a reviewer who clones the repository can run
+the same check and get the same answer. The design is
+[`docs/design/study-modes.md`](../../design/study-modes.md).
+
+**It asks git for an order, never pins a sha**, which is why it can live as a lint rule (rules in
+`eslint-rules/` may not run git: a pinned sha is deleted by a squash or a gc). A squash merge keeps
+the order when the decision was merged before the run — the decide-first workflow — and a shallow
+checkout is named, not read as a violation.
+
+## Examples
+
+Failing:
+
+```text
+papers/example/PIPELINE-STATUS.md
+  1:1  error  `repro/P1-S01-outcome/RESULTS.md`, an output of `P1-S01`, entered history in
+              4f2a9c0e1b7d (2026-10-16T10:00:00+05:00), the very commit that first recorded its
+              decision, 4f2a9c0e1b7d (2026-10-16T10:00:00+05:00). The study is declared decided
+              before its run, and its history does not show that order. What to do: if the decision
+              really came first, nothing in this history can prove it — declare the study
+              `exploratory` with the reason, or run it again after the decision; never rewrite the
+              history to make it pass.
+Rule repro/outputs-before-decision: https://github.com/zernie/paperlint/blob/main/docs/rules/repro/outputs-before-decision.md  repro/outputs-before-decision
+```
+
+Passing: `decision.md` committed in one commit; `RESULTS.md` and `out/` committed in a later commit
+that descends from it. An `exploratory` study is not this rule's.
+
+## Options / preset fields
+
+No rule options. The study declares its mode in its plan.
+
+## What it does not check
+
+- What the decision says. The gate is that it is committed, not its content.
+- **A decision amended after the outputs**: the order is between FIRST commits. A later edit of the
+  decision file is visible in its history (`git log -- <file>`) but not reported.
+- That nobody ran the study before the decision and kept the outputs out of git. History records
+  what was committed; a run on a machine that never committed it leaves no trace.
+- A study whose row names a folder that is not one inside `repro/` (`../x`), or a folder that is a
+  symbolic link: nothing is read through it, so it is no study here;
+  [`repro/study-registered`](study-registered.md) reports the row or reads its plan as missing.
+- A study whose plan is still a draft: only a ready plan's mode decides anything, and
+  [`repro/study-registered`](study-registered.md) reports an unfilled plan once the study runs.
+- Commit dates. They are shown in the message, but the order is ancestry, which a rebase can
+  rewrite only by making a new history that a reviewer comparing against the published one sees.
+
+## How to fix
+
+Commit the decision first, then let the run start:
+
+```sh
+git add papers/example/repro/P1-S01-outcome/decision.md && git commit -m "P1-S01: decision"
+npx paperlint studies gate papers/example P1-S01 && ./run.sh
+```
+
+In CI, check out the whole history: `actions/checkout` with `fetch-depth: 0`. A study whose outputs
+were made before its decision cannot be repaired by editing history; declare it exploratory:
+
+```json
+{
+  "mode": {
+    "kind": "exploratory",
+    "reason": "outputs existed before the exclusions were fixed"
+  }
+}
+```

package/docs/rules/repro/study-plan-name.md

@@ -0,0 +1,70 @@
+# repro/study-plan-name
+
+**Level:** warning · **Reads:** `repro/studies.json` · **Reported on:** the first line of the
+paper's `PIPELINE-STATUS.md`, once per study
+
+## What it catches
+
+A study whose row in `repro/studies.json` names its plan as `prereg.json` — the name the plan had
+before `paperlint studies new` wrote it as `plan.json`.
+
+```json
+{
+  "id": "P1-S01",
+  "folder": "P1-S01-rule-census",
+  "plan": "prereg.json",
+  "…": "…"
+}
+```
+
+The file is still read as the study's plan — by a loose schema, since it was written by hand before
+the plan had one: keys of its own are accepted, a seed makes its draw `seeded` and none makes it a
+draw of `all`, a size is not read, and an absent check is `null` —
+and [`repro/study-registered`](study-registered.md) judges it like any `plan.json`. This rule only
+asks for the new name. A `prereg.json` moved to `plan.json` is parsed strictly: its own keys go in
+`extra`.
+
+## Why
+
+A study's plan was once a hand-written `prereg.json`. The name misled: it is not a pre-registration
+with an outside registry, it is the study's own plan, fixed before the runs, with the author's check
+of the drawn units. Papers that wrote one before `plan.json` existed keep working; this warning tells
+them where the file moves, without failing their build.
+
+## Examples
+
+Failing (a warning):
+
+```text
+papers/example/PIPELINE-STATUS.md
+  1:1  warning  The plan of `P1-S01` is `prereg.json`, its old name; it is read as the plan. What to
+                do: rename it to `plan.json` in the study's folder (`git mv`) and set the row's `plan`
+                to `plan.json`.
+Rule repro/study-plan-name: https://github.com/zernie/paperlint/blob/main/docs/rules/repro/study-plan-name.md  repro/study-plan-name
+```
+
+Passing: the row names `"plan": "plan.json"`, as `paperlint studies new` writes it; or the row
+names no plan.
+
+## Options / preset fields
+
+No rule options.
+
+## What it does not check
+
+- A `prereg.json` in a study folder whose row names no `plan`: the file is not the study's plan
+  until the row says so, and is not read.
+- Anything inside the plan: that is [`repro/study-registered`](study-registered.md).
+
+## How to fix
+
+```sh
+git mv papers/example/repro/P1-S01-rule-census/prereg.json papers/example/repro/P1-S01-rule-census/plan.json
+```
+
+then set the row's `"plan": "plan.json"` in `repro/studies.json`. To keep the old name for one
+paper, switch the rule off in its `paperlint.json`:
+
+```json
+{ "rules": { "repro/study-plan-name": "off" } }
+```

package/docs/rules/repro/study-registered.md

@@ -1,8 +1,8 @@
 # repro/study-registered
 
 **Level:** error · **Reads:** the listing of the paper's `repro/`, `repro/studies.json`,
-`repro/README.md`, and each `result_file` the registry names · **Reported on:** the first line of the
-paper's `PIPELINE-STATUS.md`, once per finding
+`repro/README.md`, and each `result_file` and `plan` the registry names · **Reported on:** the first
+line of the paper's `PIPELINE-STATUS.md`, once per finding
 
 ## What it catches
 
@@ -46,9 +46,142 @@ is named `<id>-<slug>` after its row — `P1-S03-baseline`. The design: [`study-
 | `superseded_by` | optional: the id of the study that replaced it; required when `superseded`       |
 | `in_paper`      | `yes`, `no`, or `registry` (listed in the registry, not in the paper's text)     |
 
-A row may carry more fields — a paper's own measurement metadata (`role`, `frame`, `method`,
-`outcome`, …) — and `paperlint studies new` keeps them when it rewrites the file. The top level holds
-only `prefix` and `studies`.
+| `measurement` | optional: the id of the measurement the study belongs to (below) |
+| `plan` | optional: the study's plan, a file inside its folder — `plan.json` (below) |
+
+A row may carry more fields — a paper's own metadata (`frame`, `method`, `outcome`, …) — and
+`paperlint studies new` keeps them when it rewrites the file. Beside `prefix` and `studies`, the top
+level holds only the optional `author`, `requirements` and `measurements`; any other key is an error.
+
+### Measurements, requirements, the author
+
+One measurement — a census, a benchmark — often spans several studies: a pilot, the main run, a
+replication. A measurement is declared once, at the top level, and each of its studies' rows names
+it in `measurement`. The author's requirements on the measurements are kept in their words, beside
+them:
+
+```json
+{
+  "prefix": "P1",
+  "author": "A. Author",
+  "requirements": [
+    {
+      "id": "R1",
+      "text": "Every sample is a random draw from all rules found, no filter by difficulty.",
+      "date": "2026-10-10",
+      "source": "chat"
+    }
+  ],
+  "measurements": [
+    {
+      "id": "M1",
+      "title": "Rule census",
+      "rq": "RQ1",
+      "role": "headline",
+      "approval": {
+        "by": "A. Author",
+        "date": "2026-10-12",
+        "quote": "This is the one."
+      },
+      "requirements": ["R1"]
+    }
+  ],
+  "studies": [
+    {
+      "id": "P1-S01",
+      "folder": "P1-S01-census-pilot",
+      "measurement": "M1",
+      "plan": "plan.json",
+      "…": "…"
+    },
+    {
+      "id": "P1-S02",
+      "folder": "P1-S02-census",
+      "measurement": "M1",
+      "plan": "plan.json",
+      "…": "…"
+    }
+  ]
+}
+```
+
+| field                         | what it holds                                                                    |
+| ----------------------------- | -------------------------------------------------------------------------------- |
+| `author`                      | optional: the paper's author, the one person who approves a headline measurement |
+| `requirements[].id`           | the requirement's id, named by the measurements that must meet it                |
+| `requirements[].text`         | the author's words, verbatim                                                     |
+| `requirements[].date`         | `YYYY-MM-DD`                                                                     |
+| `requirements[].source`       | where they said it: a chat, a message, a review comment                          |
+| `measurements[].id`           | the measurement's id, named by its studies' rows                                 |
+| `measurements[].title`        | a plain name                                                                     |
+| `measurements[].rq`           | optional: the research question it answers                                       |
+| `measurements[].role`         | `headline` (the paper's claim rests on it) or `baseline`                         |
+| `measurements[].approval`     | the author's approval: `by`, `date`, their words in `quote`, optional `source`   |
+| `measurements[].requirements` | optional: the ids of the requirements it must meet                               |
+
+A measurement may carry more fields, as a row does; a requirement and an approval are exactly
+their fields. **A registry without `measurements` is judged as before**: a row's `measurement` is
+then a field of the paper's own, and nothing checks it.
+
+### The study's plan, `plan.json`
+
+Each study's plan is a JSON file in its folder, fixed before the runs: what it measures, the
+population and how the sample is drawn from it, the arms compared, the primary outcome, and the
+author's check of the drawn units. `paperlint studies new` writes the template beside the study's
+`README.md` and names it in the row's `plan`:
+
+```json
+{
+  "question": "",
+  "population": {
+    "source": "",
+    "draw": { "kind": "all", "method": "" }
+  },
+  "arms": [],
+  "primary_outcome": "",
+  "author_check": null,
+  "mode": {
+    "kind": "decide_before_run",
+    "outputs": [],
+    "decision": { "file": "repro/P1-S02-census/decision.md", "note": "" }
+  }
+}
+```
+
+| field                    | what it holds                                                                    |
+| ------------------------ | -------------------------------------------------------------------------------- |
+| `question`               | what the study measures, one sentence                                            |
+| `population.source`      | where the units come from: the census, crawl or registry, with its date          |
+| `population.draw.kind`   | `all` (the whole population, a census: no seed) or `seeded` (a sample)           |
+| `population.draw.method` | how the sample is drawn: `census`, `random`, `stratified` …                      |
+| `population.draw.seed`   | `seeded` only, and required there: the draw's seed, a string or a number         |
+| `arms`                   | the conditions compared, each `{ "name": … }`, each name once; `[]` for a census |
+| `primary_outcome`        | the one number the study reports first                                           |
+| `author_check`           | `null` until the author has seen the drawn units; then `by`, `date`, `quote`     |
+| `extra`                  | optional: the paper's own keys (hypotheses, models, budget), kept unread         |
+| `mode`                   | a measurement's: exactly one of `decide_before_run` or `exploratory`             |
+
+`mode` is required of a `measurement` (other kinds may omit it): how its outcomes are kept from the
+decisions on its inputs — decided before the run (the default the template writes; its `outputs`,
+and its `decision` file relative to the paper, committed before the run) or `exploratory` with its
+`reason`. Without it a measurement's plan is a draft whose empty text is `mode`. The modes, and the
+rule that holds the order in git, [`repro/outputs-before-decision`](outputs-before-decision.md), are
+in [`docs/design/study-modes.md`](../../design/study-modes.md).
+
+The template's draw is of all the population; a sampled study changes it to
+`{ "kind": "seeded", "method": "random", "seed": 7 }`. The population's size is not stored: it is
+counted from the source, and a stored copy could only contradict it.
+
+The plan is closed at every level, and holds no contradiction. Each of these makes the file "not a
+study plan" (`bad-plan`, with the reason): a misspelt key; a key of the paper's own outside `extra`,
+or a key of `extra` that is a field of the plan; a missing key — `null` is the one way to say "not
+yet"; a `seeded` draw without its seed, or an `all` draw with one; two arms of one name; and an
+`author_check` on a plan whose required texts are still empty — the author approves a complete plan.
+
+A plan named `prereg.json`, its old name, is parsed by a loose schema — keys of its own accepted at
+every level, those at the top kept in `extra`; a seed makes the draw `seeded`, none (or `null`)
+makes it `all`; a size is not read; an absent check is `null` — and
+[`repro/study-plan-name`](study-plan-name.md) warns that it moves to `plan.json`.
 
 The rule reports:
 
@@ -56,12 +189,13 @@ The rule reports:
   managers leave there, like `__pycache__/` and `node_modules/`, are not studies);
 - a row whose folder does not exist, and a folder named by more than one row;
 - an id that is not `<prefix>-S<digits>`, an id on more than one row, and a folder that is not
-  `<id>-<slug>`;
+  `<id>-<slug>` — a row naming `../x` among them, through which no file is read;
 - a title over 8 words;
 - `done` without `result` or `result_file`, and a `result_file` that is not a file inside the folder;
 - `superseded` without `superseded_by`, or naming an id that is not in the registry;
-- an index out of sync: the block of `repro/README.md` between the markers below must equal the
-  text the registry generates, character for character;
+- an index out of sync: the block of `repro/README.md` between the markers below must hold the
+  cells the registry and the plans generate, whatever padding a formatter gave the table — every
+  column but the two `studies index` reads from git (below);
 
   ```text
   <!-- studies:begin — generated from studies.json by `paperlint studies index`; do not edit -->
@@ -69,7 +203,21 @@ The rule reports:
   ```
 
 - a `repro/` with study folders and no `repro/studies.json` (one finding naming the file), and a
-  `studies.json` that is not JSON or not of the shape above (one finding with the reasons).
+  `studies.json` that is not JSON or not of the shape above (one finding with the reasons);
+- when the registry has `measurements`: a row whose `measurement` is not one of them
+  (`unknown-measurement`), a measurement no row names (`empty-measurement`), and a measurement that
+  must meet a requirement the registry does not hold (`unknown-requirement`);
+- a `headline` measurement with no `approval`, or, when the registry names its `author`, one approved
+  by anyone else (`headline-approval`);
+- a row's `plan` that is outside its folder, not a file, or not of the plan's shape — and, once the
+  study has run (`running`, `done`, `stopped`, `superseded`), a plan that leaves `question`,
+  `population.source`, `population.draw.method` or `primary_outcome` empty, or — for a measurement —
+  declares no `mode` or leaves one of its texts empty (`bad-plan`);
+- a study of a `headline` measurement that has run with no `author_check` in its plan — or no plan —
+  or, when the registry names its `author`, a check by anyone else (`author-check`).
+
+A study that is still `designing` owes neither a filled plan nor the author's check: the template
+`studies new` writes is clean until the first run. A `dropped` study owes neither.
 
 ## Why
 
@@ -79,9 +227,24 @@ study began, not what it is. The registry gives each study a number that does no
 person can read, and says in one place where every study stands. A rule holds it, not a guideline: a
 row the rule does not check is a row that drifts.
 
-The index in `repro/README.md` is generated, so it is never edited by hand and never parsed: the rule
-compares strings. It is a table padded the way prettier writes one, so formatting the README leaves
-it in sync.
+The index in `repro/README.md` is generated, so it is never edited by hand: the rule compares its
+cells. It is a table padded the way prettier writes one, so formatting the README leaves it in sync.
+
+Three of its columns say how each study kept its outcomes from its decisions
+([`docs/design/study-modes.md`](../../design/study-modes.md)):
+
+| column         | cell                                                                                 |
+| -------------- | ------------------------------------------------------------------------------------ |
+| `mode`         | `decide_before_run`, `exploratory: <reason>`, or `—` for a study whose plan has none |
+| `decision`     | the decision file's first commit (12 characters) and its date, or `not committed`    |
+| `first output` | the first commit of the oldest output and its date, or `none`                        |
+
+The `mode` comes from the plan, which is in the tree, so the rule compares it. The two commits are
+what git's history said when `paperlint studies index` last ran: a squash merge or a rebase gives the
+same files new commits, and a shallow CI clone has none to read, so the rule never compares those two
+columns. Regenerate the index before reading them; the order itself is judged by
+[`repro/outputs-before-decision`](outputs-before-decision.md), not by the table. In a shallow clone
+`studies index` writes `unknown (shallow clone)`.
 
 ## Examples
 
@@ -98,6 +261,20 @@ Rule repro/study-registered: https://github.com/zernie/paperlint/blob/main/docs/
 
 Failing: a row set to `done` by hand, with no result, and the index not regenerated — two findings.
 
+Failing: the census of the example above set to `running` while its plan is still the template —
+the plan is reported for its empty fields, and the study for the author's check it lacks:
+
+```text
+papers/example/PIPELINE-STATUS.md
+  1:1  error  The plan of `P1-S02`, `plan.json` in its folder, leaves `question`, `population.source`,
+              `population.draw.method`, `primary_outcome`, `mode.outputs` and `mode.decision.note`
+              empty, and the study is `running`: a plan is written before the runs. What to do: fill
+              them in, or set the study back to `designing`.
+  1:1  error  `P1-S02` is `running` in the headline measurement `M1`, and its plan has no
+              `author_check`. What to do: show the author the drawn units, then record their answer
+              in the plan's `author_check` — `by`, `date`, and their words in `quote`.
+```
+
 Passing: a paper with no study folder; and a `repro/` made only by `paperlint studies new`, its rows
 edited as the studies end and its index regenerated:
 
@@ -107,6 +284,7 @@ repro/
 ├── studies.json
 ├── P1-S01-baseline-pass-rate/
 │   ├── README.md
+│   ├── plan.json
 │   └── results.tsv
 ├── tools/
 └── data/
@@ -115,9 +293,9 @@ repro/
 ```markdown
 <!-- studies:begin — generated from studies.json by `paperlint studies index`; do not edit -->
 
-| id     | title              | status | result             | in paper | folder                                                   |
-| ------ | ------------------ | ------ | ------------------ | -------- | -------------------------------------------------------- |
-| P1-S01 | Baseline pass rate | done   | 41 of 60 runs pass | yes      | [P1-S01-baseline-pass-rate/](P1-S01-baseline-pass-rate/) |
+| id     | title              | status | result             | in paper | mode              | decision                | first output            | folder                                                   |
+| ------ | ------------------ | ------ | ------------------ | -------- | ----------------- | ----------------------- | ----------------------- | -------------------------------------------------------- |
+| P1-S01 | Baseline pass rate | done   | 41 of 60 runs pass | yes      | decide_before_run | 3f2a9c1d7e40 2026-10-14 | 8b51e0aa42c9 2026-10-15 | [P1-S01-baseline-pass-rate/](P1-S01-baseline-pass-rate/) |
 
 <!-- studies:end -->
 ```
@@ -130,7 +308,15 @@ No rule options.
 
 - Whether `question` is one sentence and `result` one line, beyond their being text.
 - What a study's `README.md` says, or whether its frontmatter matches the row.
+- Whether a plan's texts say anything true, beyond their being filled in; whether `arms` should be
+  empty; whether the person in `approval` or `author_check` really said it — the rule sees the
+  name, not who typed it.
+- A repeated measurement or requirement id, and a requirement no measurement names.
+- A `prereg.json` or `plan.json` that no row names.
 - Whether the paper cites a study: `in_paper` is declared, not detected.
+- The index's `decision` and `first output` cells: they name commits a squash merge renames.
+- Inside a study folder that is a symbolic link: nothing is read through it, so its plan and its
+  result file read as missing.
 - What may stand at the top of `repro/` besides the study folders, and whether a study folder has a
   `README.md`: that is [`repro/allowed-files-only`](allowed-files-only.md).
 
@@ -143,6 +329,11 @@ Create every new study with the command, which takes the next id, makes the fold
 npx paperlint studies new papers/example "Baseline pass rate" --kind measurement
 ```
 
+`studies new` also writes the study's `plan.json`: fill it in before the first run, and, for a
+study of a headline measurement, show the author the drawn units and record their answer in its
+`author_check`. A measurement is added by hand to `measurements`, with the author's `approval` when
+it is the headline, and each of its studies' rows names it in `measurement`.
+
 When a study ends, edit its row in `repro/studies.json` — `status`, `result`, `result_file`,
 `in_paper`, and `superseded_by` when another study replaced it — then regenerate the index: