paperlint 13.1.0 → 13.3.0

package/docs/rules/repro/study-registered.md

@@ -132,6 +132,7 @@ author's check of the drawn units. `paperlint studies new` writes the template b
 
 ```json
 {
+  "$schema": "https://unpkg.com/paperlint/schemas/plan.v2.schema.json",
   "question": "",
   "population": {
     "source": "",
@@ -143,23 +144,36 @@ author's check of the drawn units. `paperlint studies new` writes the template b
   "mode": {
     "kind": "decide_before_run",
     "outputs": [],
-    "decision": { "file": "repro/P1-S02-census/decision.md", "note": "" }
-  }
+    "decision": { "file": "repro/P1-S02-census/decision.md", "note": "" },
+    "we_lose_if": null
+  },
+  "second_check": null,
+  "self_test": null,
+  "funnel": null,
+  "recall_audit": null
 }
 ```
 
-| field                    | what it holds                                                                    |
-| ------------------------ | -------------------------------------------------------------------------------- |
-| `question`               | what the study measures, one sentence                                            |
-| `population.source`      | where the units come from: the census, crawl or registry, with its date          |
-| `population.draw.kind`   | `all` (the whole population, a census: no seed) or `seeded` (a sample)           |
-| `population.draw.method` | how the sample is drawn: `census`, `random`, `stratified` …                      |
-| `population.draw.seed`   | `seeded` only, and required there: the draw's seed, a string or a number         |
-| `arms`                   | the conditions compared, each `{ "name": … }`, each name once; `[]` for a census |
-| `primary_outcome`        | the one number the study reports first                                           |
-| `author_check`           | `null` until the author has seen the drawn units; then `by`, `date`, `quote`     |
-| `extra`                  | optional: the paper's own keys (hypotheses, models, budget), kept unread         |
-| `mode`                   | a measurement's: exactly one of `decide_before_run` or `exploratory`             |
+That is a measurement's template, a plan v2. Another kind's has no `$schema`, `mode` or gates.
+
+| field                    | what it holds                                                                        |
+| ------------------------ | ------------------------------------------------------------------------------------ |
+| `question`               | what the study measures, one sentence                                                |
+| `population.source`      | where the units come from: the census, crawl or registry, with its date              |
+| `population.draw.kind`   | `all` (the whole population, a census: no seed) or `seeded` (a sample)               |
+| `population.draw.method` | how the sample is drawn: `census`, `random`, `stratified` …                          |
+| `population.draw.seed`   | `seeded` only, and required there: the draw's seed, a string or a number             |
+| `arms`                   | the conditions compared, each `{ "name": … }`, each name once; `[]` for a census     |
+| `primary_outcome`        | the one number the study reports first                                               |
+| `author_check`           | `null` until the author has seen the drawn units; then `by`, `date`, `quote`         |
+| `extra`                  | optional: the paper's own keys (hypotheses, models, budget), kept unread             |
+| `mode`                   | a measurement's: exactly one of `decide_before_run` or `exploratory`                 |
+| `$schema`                | a plan v2 says so here; without it the plan is v1                                    |
+| `mode.we_lose_if`        | v2, decide-first only: the number its claim loses by, `below` or `above` a threshold |
+| `second_check`           | v2: per labelling step, the least kappa between two raters, or `"none"`              |
+| `self_test`              | v2: how the study's check is run on known answers first, or `"none"`                 |
+| `funnel`                 | v2: the reasons a drawn unit may be dropped, or `"none"`                             |
+| `recall_audit`           | v2: files a person reads in full, and the least recall of the extractor, or `"none"` |
 
 `mode` is required of a `measurement` (other kinds may omit it): how its outcomes are kept from the
 decisions on its inputs — decided before the run (the default the template writes; its `outputs`,
@@ -168,6 +182,12 @@ and its `decision` file relative to the paper, committed before the run) or `exp
 rule that holds the order in git, [`repro/outputs-before-decision`](outputs-before-decision.md), are
 in [`docs/design/study-modes.md`](../../design/study-modes.md).
 
+The four gates of a plan v2 — what each is for, what the machine checks and what it cannot — are in
+[`docs/design/study-gates.md`](../../design/study-gates.md). Each is `null` until decided (the plan is
+a draft), `"none"` when the study has none of it, or its value; a plan v2 that leaves one out is not
+a plan. A plan without `$schema` is v1 and is judged as before, and may carry none of the gates. The
+result a done study's plan v2 asks for is judged by [`repro/study-result`](study-result.md).
+
 The template's draw is of all the population; a sampled study changes it to
 `{ "kind": "seeded", "method": "random", "seed": 7 }`. The population's size is not stored: it is
 counted from the source, and a stored copy could only contradict it.
@@ -212,7 +232,10 @@ The rule reports:
 - a row's `plan` that is outside its folder, not a file, reached through a symbolic link, or not of the plan's shape — and, once the
   study has run (`running`, `done`, `stopped`, `superseded`), a plan that leaves `question`,
   `population.source`, `population.draw.method` or `primary_outcome` empty, or — for a measurement —
-  declares no `mode` or leaves one of its texts empty (`bad-plan`);
+  declares no `mode` or leaves one of its texts empty, or — for a plan v2 — leaves a gate `null`
+  (`bad-plan`);
+- a measurement registered on or after 2026-10-31 whose plan is v1, without `$schema`
+  (`plan-version`): its gates are not declared. A measurement registered earlier keeps its plan v1;
 - a study of a `headline` measurement that has run with no `author_check` in its plan — or no plan —
   or, when the registry names its `author`, a check by anyone else (`author-check`).
 

package/docs/rules/repro/study-result.md

@@ -0,0 +1,100 @@
+# repro/study-result
+
+**Level:** error · **Reads:** `repro/studies.json`, each `done` study's `plan.json` (v2) and
+`result.json` · **Reported on:** `PIPELINE-STATUS.md`, line 1
+
+A study whose plan is v2 fixed, before its run, what makes its number believable: when its claim
+loses, how well two raters must agree, that its checker is tried on known answers, and why a unit may
+be dropped ([the gates](../../design/study-gates.md)). Once the study is `done`, its `result.json`
+must show each of them.
+
+## What it catches
+
+For each study that is `done`, whose plan is ready and v2:
+
+| problem             | when                                                                                                                                                                                                                                                                              |
+| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `no-result-file`    | the study's folder has no `result.json`                                                                                                                                                                                                                                           |
+| `result-linked`     | `result.json` is reached through a symbolic link                                                                                                                                                                                                                                  |
+| `result-unreadable` | `result.json` is not JSON, or `kappa`, `self_test`, `funnel`, `recall_audit` or `claim` is not of its shape (a kappa outside -1..1, such as a percentage written as `75`, is not one; an audit counting more units extracted than the person found, or no unit found, is not one) |
+| `no-number`         | the number `mode.we_lose_if.result` names is not a number in `result.json`                                                                                                                                                                                                        |
+| `claim-lost`        | that number crosses the threshold, and `result.json` does not say `"claim": "lost"`                                                                                                                                                                                               |
+| `claim-holds`       | `result.json` says `"claim": "lost"`, and the number does not cross the threshold                                                                                                                                                                                                 |
+| `kappa`             | a step of `second_check` has no kappa in `result.json`'s `kappa`, or one below its `min_kappa`                                                                                                                                                                                    |
+| `self-test`         | `self_test` is declared, and `result.json`'s `self_test` is missing or `failed` is not 0                                                                                                                                                                                          |
+| `self-test-empty`   | `self_test` is declared, and `result.json`'s `self_test` ran no case: `"passed": 0, "failed": 0`                                                                                                                                                                                  |
+| `funnel`            | a reason of `funnel` has no count in `result.json`'s `funnel`                                                                                                                                                                                                                     |
+| `no-recall-audit`   | `recall_audit` is declared, and `result.json` has no `recall_audit` counts                                                                                                                                                                                                        |
+| `recall-sample`     | the audit read fewer `files` than the plan's `n_files`                                                                                                                                                                                                                            |
+| `recall`            | `extracted / units_in_files` is below the plan's `min_recall`: the extractor misses too much                                                                                                                                                                                      |
+
+A gate that is `"none"` asks nothing; an exploratory study has no `we_lose_if`.
+
+## Why
+
+A threshold fixed before the run and checked by hand after it can be moved, forgotten or quietly
+reinterpreted. Here the plan holds the threshold and the result holds the number, and the engine
+compares them on every lint. A claim that lost stays red until the result says so in so many words —
+`"claim": "lost"` — where the paper reads its numbers from, so the loss cannot sit unnoticed in a
+table. The same holds for a kappa below the bar set before labelling, a checker that missed a known
+answer, and a dropped unit nobody counted.
+
+## Examples
+
+Failing — the plan declares the funnel reason `timed out`, the result counts none:
+
+```json
+{
+  "pass_rate": 0.7,
+  "kappa": { "pass": 0.71 },
+  "self_test": { "passed": 4, "failed": 0 },
+  "funnel": {}
+}
+```
+
+```text
+PIPELINE-STATUS.md
+  1:1  error  The `result.json` of `P1-S01` counts no drop for `timed out`, declared in its plan's
+              `funnel`. What to do: write each in `funnel`, 0 when none was dropped for it.
+```
+
+Failing — `we_lose_if` is `{ "result": "pass_rate", "below": 0.5 }` and the rate is 0.4:
+
+```text
+  1:1  error  `P1-S01`'s claim LOST: its `pass_rate` is 0.4, below 0.5, the threshold its plan fixed
+              before the run. What to do: report the loss in the paper, and say so in `result.json`
+              with `"claim": "lost"`; the threshold is not moved after the run.
+```
+
+Passing — every gate shown:
+
+```json
+{
+  "pass_rate": 0.7,
+  "kappa": { "pass": 0.71 },
+  "self_test": { "passed": 4, "failed": 0 },
+  "funnel": { "timed out": 2 }
+}
+```
+
+## Options / preset fields
+
+None. The rule is on at `error` in paperlint's config; a paper turns it off in its `paperlint.json`.
+
+## What it does not check
+
+- A study that is not `done`, a plan v1, a draft or a plan that does not parse
+  ([`repro/study-registered`](study-registered.md) reports those).
+- Whether a gate's `"none"` is true, whether the threshold is a sensible one, or whether the number
+  is computed right: the author's check of the plan, and the study's own artifact.
+- Whether the gates were fixed before the outputs: [`repro/outputs-before-decision`](outputs-before-decision.md)
+  holds them to the plan as it stood at the first output.
+- Whether the person of the recall audit read the files in full, or counted right: the audit's own
+  labels, exported and committed like any other.
+
+## How to fix
+
+Have the study's script write `result.json` beside its plan, with the number `we_lose_if` names at
+the top, and `kappa`, `self_test`, `funnel` and `recall_audit` as the plan declares them
+([the shape](../../design/study-gates.md#resultjson)). A lost claim is reported, not hidden: add
+`"claim": "lost"` and say so in the paper. Never edit the plan's thresholds after the run.

package/docs/rules.md

@@ -21,6 +21,7 @@ reads and when it fails. Errors fail `paperlint lint`; warnings print and do not
 | [`repro/study-registered`](rules/repro/study-registered.md)               | error                                                    | the listing of the paper's `repro/`, `repro/studies.json`, `repro/README.md`, each study's `plan.json` | a study folder with no row in `repro/studies.json`, a row whose folder is missing or not `<id>-<slug>`, a bad or repeated id, a title over 8 words, `done` without a result and an existing result file, `superseded` without an existing successor, a row naming a measurement the registry does not declare, a measurement with no study or an unknown requirement, a headline measurement not approved by the author, a plan that is missing, malformed or — once the study runs — unfilled, a headline study that has run without the author's check in its plan, or the index in `repro/README.md` out of sync with the registry; `paperlint studies new` and `paperlint studies index` write both — [page](rules/repro/study-registered.md) |
 | [`repro/study-plan-name`](rules/repro/study-plan-name.md)                 | warn                                                     | `repro/studies.json`                                                                                   | a study's plan still named `prereg.json`, its old name: read as the plan, and to be renamed `plan.json` — [page](rules/repro/study-plan-name.md)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | [`repro/outputs-before-decision`](rules/repro/outputs-before-decision.md) | error                                                    | `repro/studies.json`, each decide-first study's `plan.json` and folder, git's history                  | an output of a study decided before its run whose first commit does not descend from its decision's first commit, an output while the decision is uncommitted, a decision its outputs match, a shallow or unreadable history; `paperlint studies gate` opens the run only once the decision is committed — [page](rules/repro/outputs-before-decision.md)                                                                                                                                                                                                                                                                                                                                                                                         |
+| [`repro/study-result`](rules/repro/study-result.md)                       | error                                                    | `repro/studies.json`, each done study's `plan.json` (v2) and `result.json`                             | a done study whose plan is v2 and whose `result.json` is missing, lacks the number its claim is judged by, crosses that threshold without saying `"claim": "lost"`, lacks a labelling step's kappa or has one below the plan's bar, shows a failed self-test, counts no drop for a declared funnel reason, or audits its extractor on too few files or finds its recall below the plan's bar — [page](rules/repro/study-result.md)                                                                                                                                                                                                                                                                                                                |
 | [`paper/sources-fresh`](rules/paper/sources-fresh.md)                     | warn                                                     | `paper.tex` → `_build/sources.json`                                                                    | the paper has not been built (no record of the files TeX read), or a file changed since the last build (its SHA-256 differs) — run `npx paperlint build`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
 | `paper/author-list`                                                       | error                                                    | `paper.tex` → `_build/references.json`, `_build/sources.json`                                          | an entry's authors are not those of the version it cites — typically the arXiv preprint's list under a published venue. Checked online by `paperlint build` (DBLP), reported on the entry's own line (or at the top of `paper.tex`, naming the entry's file); silent without a current build record                                                                                                                                                                                                                                                                                                                                                                                                                                               |
 | `paper/section-word`                                                      | warn                                                     | `paper.tex` and its body files                                                                         | `§` or `\S\ref` instead of the word Section. **Fixable:** `--fix` writes `Section~\ref{…}` / `Section 5`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |

package/docs/toolchain.md

@@ -16,7 +16,7 @@ after deciding to use the tool, not before.
 | `texcount`           | TeX Live (`paperlint toolchain`) | render-paper, grade-paper-writing        | the length checks cannot run                                          |
 | `checkcites`         | TeX Live (`paperlint toolchain`) | render-paper                             | nothing asks whether a bibliography entry is uncited                  |
 | `java`               | any JRE (21 works)               | render-paper                             | TeXtidote does not run, and **nothing else spell-checks the text**    |
-| `python3`            | your system                      | the analysis and report scripts          | those scripts do not start                                            |
+| `python3`            | your system                      | assess-novelty, the analysis scripts     | those scripts do not start (`crop_quote.py` also needs `pymupdf`)     |
 | `tlmgr`              | TeX Live                         | the TeX installer itself                 | you cannot add a TeX package                                          |
 | `banal`              | HotCRP (`paperlint toolchain`)   | `paperlint build`, extract-pdf-facts     | page size, columns and font sizes are `null` in the facts file        |
 | `perl`               | your system (macOS has it)       | banal                                    | banal cannot run — the same `null`, and `paperlint toolchain` refuses |

package/eslint.config.mjs

@@ -135,6 +135,36 @@ const RATCHET = {
   "skills/plan-paper-timeline/fixtures/fake-google-calendar.ts": {
     "max-lines-per-function": 31,
   },
+  "skills/assess-novelty/scripts/candidate_page.test.ts": {
+    "max-lines-per-function": 343,
+    "max-nested-callbacks": 4,
+  },
+  "skills/assess-novelty/scripts/check_novelty.test.ts": {
+    "max-lines-per-function": 419,
+  },
+  "skills/assess-novelty/scripts/check_novelty_coverage.test.ts": {
+    "max-lines-per-function": 131,
+  },
+  "skills/assess-novelty/scripts/crop_quote.test.ts": {
+    "max-lines-per-function": 96,
+  },
+  "skills/assess-novelty/scripts/novelty_candidates.test.ts": {
+    "max-lines-per-function": 179,
+  },
+  "skills/assess-novelty/scripts/novelty_cells.test.ts": {
+    "max-lines-per-function": 117,
+  },
+  "skills/assess-novelty/scripts/novelty_reader_brief.test.ts": {
+    "max-lines-per-function": 180,
+    complexity: 11,
+  },
+  "skills/assess-novelty/scripts/candidate_page.ts": {
+    "max-lines-per-function": 55,
+  },
+  "skills/assess-novelty/scripts/novelty_reader_brief.ts": {
+    "max-lines-per-function": 32,
+    "max-params": 5,
+  },
   "src/adapters/curl/download.io.ts": { "max-lines-per-function": 39 },
   "src/adapters/references/index.ts": { "max-lines-per-function": 36 },
   "src/build-engine.ts": { "max-lines-per-function": 33 },
@@ -667,6 +697,28 @@ export default [
     files: TEST_FILES,
     rules: { "max-lines-per-function": ["error", MAX_LINES_TESTS] },
   },
+  // 🔴 DEBT, NOT A PERMISSION. The novelty-check scripts and their harnesses were moved from a consumer
+  // repository AS THEY WERE (the 2026-11-02 decision): untyped JSON in, `any` throughout. Making them pass the
+  // type-safety rules below means rewriting them, which is paperlint#339 (TypeScript rewrite of the Python
+  // checkers) and paperlint#338 (the regex priority-sentence detector). Only the rules that need typed data
+  // are off, only here; the rules about behaviour (`complexity`, `max-depth`, `no-empty`, ...) still apply.
+  // Delete this block with the rewrite.
+  {
+    files: ["skills/assess-novelty/scripts/*.ts"],
+    rules: {
+      "@typescript-eslint/no-unsafe-member-access": "off",
+      "@typescript-eslint/no-unsafe-call": "off",
+      "@typescript-eslint/no-unsafe-assignment": "off",
+      "@typescript-eslint/no-unsafe-argument": "off",
+      "@typescript-eslint/no-unsafe-return": "off",
+      "@typescript-eslint/no-explicit-any": "off",
+      "@typescript-eslint/consistent-type-assertions": "off",
+      "@typescript-eslint/restrict-template-expressions": "off",
+      "@typescript-eslint/no-unnecessary-condition": "off",
+      "@typescript-eslint/no-confusing-void-expression": "off",
+      "@typescript-eslint/no-non-null-assertion": "off",
+    },
+  },
   // Hexagonal layers, both axes (src/CLAUDE.md): knowledge by element, purity by file category.
   IO_GLOBALS,
   NO_CLOCK,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "paperlint",
-  "version": "13.1.0",
+  "version": "13.3.0",
   "description": "A linter for scientific papers written in LaTeX: catches mistakes before you submit to a conference or journal.",
   "type": "module",
   "license": "MIT",
@@ -58,6 +58,7 @@
     "eslint-plugin-project-structure": "^3.14.5",
     "eslint-plugin-sonarjs": "^4.2.1",
     "jsonc-parser": "^3.3.1",
+    "playwright-core": "^1.64.1",
     "prettier": "^3.8.1",
     "semantic-release": "25.0.9",
     "typescript-eslint": "^8.70.1",
@@ -119,6 +120,7 @@
     "presets",
     "plugin",
     "scripts",
+    "schemas",
     "skills",
     "src",
     "templates",

package/presets/agentverify.md

@@ -26,8 +26,10 @@ researchr page `conf.researchr.org/home/agentverify-2027` only redirects to the
   preset keeps the family's `\documentclass[conference]{IEEEtran}`.
 - **Publisher**: «Accepted papers will appear in the ICSE 2027 workshop proceedings by default.»
   The ICSE workshops page: «The workshop papers will be published by the IEEE.» The CFP does not
-  say what a non-default route would be, or whether it is archival.
-- **Presenting**: «At least one author of each accepted paper must register and present.»
+  say what a non-default route would be, or whether it is archival. The organizers: no other route
+  is offered (below).
+- **Presenting**: «At least one author of each accepted paper must register and present.» Remote
+  presentation is allowed (below).
 - **Deadlines (AoE)**: submission 13 Nov 2026, reviews due 5 Dec, final notification 11 Dec 2026,
   camera-ready 29 Jan 2027.
 
@@ -65,10 +67,19 @@ and are over the limit. Count the PDF's pages yourself before submitting.
   scheduled workflow; notification and camera-ready are the site's. No `registration`: there is no
   abstract deadline.
 
+## Answered by the organizers (emails to an author, not public)
+
+- **Remote presentation** (2026-10-23): «In general, remote presentations will be allowed for
+  authors who are unable to attend the workshop in person. However, at least one author of each
+  accepted paper must register for the workshop to ensure its inclusion in the proceedings.»
+- **APC** (2026-10-27), for every paper kind: «APC. It is free completely.»
+- **Proceedings** (2026-10-27): «Workshop papers are normally published in an ICSE companion
+  IEEE/ACM proceedings.» Asked whether a non-archival option exists: «The non-trivial [sic:
+  non-archival] option is not offered.» So every accepted paper is archival.
+
 ## Not verified
 
-- **Remote presentation**: not stated (only «must register and present»).
-- **Archival / non-archival choice**: «by default» suggests an alternative; none is described.
+- **Digital library**: whether the 2027 companion volume goes to IEEE Xplore or the ACM DL.
 - **AI / LLM usage policy and required sections**: none stated.
 
 ## What the preset resolves to