paperlint 8.0.0 → 9.0.0

package/docs/design/paper-sources.md

@@ -0,0 +1,197 @@
+# Paper sources — TeX answers "which files, which bibliography"
+
+As built in PR #174. Everything that asks which files make up a paper, or which bibliography TeX reads,
+reads the record the last build wrote from TeX's own files; nothing in paperlint reads TeX source to
+decide either, and nothing reads a `.bib` but one adapter. TeX's own answer for each planted paper is
+recorded beside it (§8), and the consumers are tested against that, not against prose.
+
+## 1. TeX is the one source of truth
+
+Which files make up the paper, and which bibliography it uses, are answered by TeX, from the last
+build — never by reading TeX by hand. There is no static decision: no declaration scan, no
+`filecontents` runs, no include ordering. Lint never runs TeX; it reads what the build recorded.
+
+**What the build records — `_build/sources.json`, schema 1.** Written by the compile step, after
+the last pdflatex pass and bibtex, from TeX's own files in the paper directory:
+
+| field     | from                      | holds                                                                                                                                                                               |
+| --------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `inputs`  | `paper.fls` (`-recorder`) | every file in the paper directory TeX read, in first-read order, each `role: preamble` or `body` — before or after TeX opens `paper.aux` for writing (`\begin{document}`, measured) |
+| `written` | `paper.fls` `OUTPUT`      | the `.bib` and `.tex` files TeX wrote (a `filecontents` block)                                                                                                                      |
+| `bibdata` | `paper.aux` `\bibdata{…}` | the databases bibtex was told to read, after every switch, macro and `\input`                                                                                                       |
+| `bibtex`  | `paper.blg`, `paper.bbl`  | the database files bibtex opened, the keys it typeset, its exit and errors                                                                                                          |
+| `sha256`  | the files themselves      | every `inputs` file and every database bibtex opened (one TeX wrote too), by path and bytes                                                                                         |
+
+A file TeX wrote is an input only when it is a `.tex`: a `filecontents` block that writes `body.tex` the
+body `\input`s puts that text in the PDF, so the file is hashed and spliced like any other (measured:
+pass 1 `OUTPUT body.tex` then `INPUT ./body.tex` after `\begin{document}`). Every other file a pass
+wrote — `.aux`, `.out`, `.toc`, a block's `.bib` — is TeX's bookkeeping or a database, not an input.
+The written `.tex` is not handed to ESLint as a file of its own: its words are the block's, and the
+block's body is no prose to the rules, so the text is judged once, where `readPaper` splices it.
+
+Every path is relative to the paper directory and spelled as the directory entries spell it: TeX
+logs the name it opened, which on a file system that ignores case is the source's spelling, so the
+record takes each segment's spelling from the disk and one paper gets one record on every platform.
+
+A record of its own, not `references.json`: it is written by every build (the references step is
+optional and may record "not checked"), it is read by rules that have nothing to do with references,
+and it goes stale when the SOURCES change, not when one `.bib` does — one staleness key per file.
+`references.json` keeps its verdicts, one per entry of `sources.json`'s databases, and loses its
+`bibtex` field.
+
+**Staleness.** A rule recomputes `sha256` over the files the record lists. No record, or a different
+hash (a file edited, added to an `\input`, deleted), is one finding of a new rule,
+`paper/sources-fresh` — "not built" / "the paper changed since the last build — run
+`npx paperlint build`" — and every rule below is silent until a build records the paper again.
+
+**The consumers.**
+
+| consumer                                                         | reads                                                               | no record / stale                                                  |
+| ---------------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------------ |
+| `bib/reachable-entry`, `paper/author-list`, `paper/cite-exists`  | the entries of the databases bibtex opened, through the `BibReader` | silent (`paper/sources-fresh` speaks)                              |
+| `paper/refs-checked`                                             | keys bibtex typeset that the reader did not read (`%`, `@comment`)  | silent                                                             |
+| `extract-ref-facts`, `bib-authors`, `verify-cites` (dir, `.tex`) | `bibliographyAt` → the record's databases                           | refused: "run `npx paperlint build` first"                         |
+| `bibliographyAt` with a `.bib`                                   | that file alone, unchanged                                          | —                                                                  |
+| `paperlint lint`'s files (`paper-includes.ts`), `paper-context`  | the `body` `.tex` files of `inputs` TeX did not write               | `paper.tex` only; `paper/sources-fresh` says the rest is unlinted  |
+| the prose rules' text (`readPaper`)                              | `paper.tex` with the files of `inputs` spliced where they stand     | `paper.tex` alone; `paper/sources-fresh` says the rest is unlinted |
+
+An entry in a `.bib` TeX wrote is reported at the `filecontents` block whose body bibtex takes as
+the same database (`sameDatabase`) — a lookup of where the text stands, not a decision of what TeX
+reads; an entry in any other `.bib` at the top of `paper.tex`, `refs.bib:12:1:` first.
+
+**What stays syntactic: where an include stands, and nothing else.** The prose rules read
+`paper.tex` with the files of `inputs` spliced in (`readPaper`): a projection of the text for rules to
+read in context, which decides nothing above. Which files are spliced is the record's answer — an
+include of a file `inputs` does not list (behind `\iffalse`, or past `\end{document}`) contributes
+nothing, as it did to TeX, and no include is looked for on disk. Where the macro stands in the text
+is the one thing read from the text, because `.fls` records no position: files are spliced in the
+order the text names them, which is the order TeX read them, each ended the way TeX ends the last
+line of a file it reads (a newline when the file has none — measured: `a\input{f}b` with `f` holding
+`foo` typesets "afoo b"). With no record, or a stale one, the text is `paper.tex` alone. A file the
+record lists that no include macro in the text stands for (an `\input` inside a macro's body, #177)
+is linted as a file of its own and is absent from the whole-paper text.
+
+**Ground truth.** `test/e2e/tex/paper-sources.e2e.ts` runs the build's own record step on each
+planted paper and snapshots its `sources.json` as `tex-truth.json`; the unit tests run the
+consumers on the fixture with that record. TeX is the expectation and the input at once.
+
+## 4. One bibtex reader
+
+The port is `BibReader` (`src/ports/bib-reader.ts`: `read(path, text, body, inherited?)` for a block,
+`readFile(path, text, inherited?)` for a `.bib`). bibtex reads a paper's databases in sequence with one
+table of `@string`s, so a `@string` of `abbrev.bib` expands an entry of `refs.bib` under
+`\bibliography{abbrev,refs}` (measured; in the other order bibtex warns it is undefined).
+`recordedBibliography` reads the databases in the order bibtex opened them, each with the `strings` the
+one before it left in force (`inherited`), and reads a block with the `@string`s its database was read
+with; each database is still parsed as its own text, so each entry keeps its place in its own file.
+
+The adapter, `src/adapters/bibtex/`, is the only code that knows `@retorquere/bibtex-parser`. It reads each database twice:
+
+- **`entries`** — LaTeX read into text (`{\"u}` → `ü`, `\url{x}` → a link), names split into parts,
+  `@string`s expanded where they are used. What the checks judge.
+- **`written`** — `raw: true`, every field verbatim, no outer braces dropped: each field as bibtex
+  takes it, `@string`s expanded where used, LaTeX as written. What `sameDatabase` compares, with the
+  `@preamble` commands, so `{\"o}` and `ö` stay two values, as they are in the `.bbl`.
+
+**The fence is lint, not prose.** `eslint.config.mjs` sets `n/no-restricted-import` on
+`@retorquere/bibtex-parser` (and its subpaths) for every file but `src/adapters/bibtex/**`
+(`test/eslint-bib-parser.test.ts` plants imports in the app, the domain, another adapter and a
+skill's script).
+
+## 5. A path a person gave → `bibliographyAt`
+
+`extract-ref-facts`, `bib-authors` and `verify-cites` (its `.bib` input) take a path on the command
+line. All three ask `bibliographyAt(path, deps)`: a `.bib` is read alone, as named; a `.tex`, or a
+directory holding `paper.tex`, is the bibliography the last build's bibtex opened for that paper
+(`recordedBibliography`) — for a database TeX wrote, the text of the `filecontents` block (or included
+file) that holds it, so an entry stands where the author edits it. A refusal is a `BibliographyUnread`
+(`missing`, `not-bib-or-tex`, `no-paper`, `not-built`, `no-database`, `outside-paper`), and
+`bibliographyUnreadWhy` is its one sentence; `outside-paper` is a paper whose bibtex also opened a
+database the paper directory does not hold (BIBINPUTS) — refused rather than read in part;
+`not-built` covers a missing, unreadable and stale record alike: run `npx paperlint build` first. Each script keeps its own CLI frame around it.
+
+## 6. The references a build checks
+
+The build's references step (`src/references.ts`) checks entries, not keys: two databases may each
+define a key with other metadata.
+
+- **What is checked is `readBibliography`**: the databases the build's bibtex opened, in the order it
+  opened them, each once, read by the bibtex reader — a view over `recordedBibliography`, the one reader
+  of the record's databases. Without a fresh record the step checks nothing and its note names the
+  reason; a database bibtex opened that is not in the paper directory (bibtex found it on its search
+  path, `BIBINPUTS`) is named, not skipped. One removed after the build — a `.bib` TeX wrote, then a
+  clean — makes the record stale: every database bibtex opened is hashed.
+- **One verdict per entry, in order.** The `CheckReferences` port returns verdict `i` for entry `i`;
+  the adapter runs verify-cites and bib-authors per entry and pairs them by position.
+- **`_build/references.json`, schema 3**: `bib {sources, sha256}`, `status`, `entries`. `sources` are
+  the database names as bibtex names them and `sha256` covers their text. Lint lays the verdicts on the
+  entries of the same databases and reports each at its entry (`entryReports`); a record that is not one
+  verdict per entry, in order, about the bytes the databases hold now, is not about this bibliography
+  (`paper/refs-fresh`).
+
+## 7. The post-build check — where bibtex reads differently
+
+bibtex has no comment syntax and recovers from malformed text as no parser does, so the reader does
+not emulate it. Each difference has an owner:
+
+| shape                                                          | answered by                                                            |
+| -------------------------------------------------------------- | ---------------------------------------------------------------------- |
+| malformed: an entry or brace never closed, `%` inside an entry | the build — bibtex exits 2, and `paperlint build` fails with its lines |
+| an entry behind `%`, or inside `@comment{…}`                   | `paper/refs-checked`, from the keys bibtex typeset                     |
+| `@`, `)` or `{"}` inside a quoted field                        | the reader, which reads it as bibtex does                              |
+
+`paper/refs-checked` names every key bibtex typeset (`paper.bbl`) that no entry of the databases it
+opened has (`unseenKeys`; keys compare case-insensitively, as bibtex's do), and every database bibtex
+opened that is not in the paper directory.
+
+## 8. Ground truth — `tex-truth.json`
+
+Each planted paper of `fixtures/paper-sources/` carries `tex-truth.json`, written by TeX:
+`test/e2e/tex/paper-sources.e2e.ts` builds it with `pdflatex -recorder` and `bibtex` and snapshots the
+record the build writes. `-u` re-records; a TeX that answers differently fails. The unit tests of the
+consumers lay that file at `_build/sources.json` (`test/recorded-fixture.ts`). The accepted papers of
+`fixtures/accepted-papers/` carry theirs the same way (`test/e2e/tex/accepted-papers-record.e2e.ts`).
+The files are the numbers; this document does not copy them.
+
+## 9. Decisions
+
+- **The record, not a reading** — which files and databases a paper has is what TeX did on the last
+  build; a static reading of the text agrees only on the cases its author thought of.
+- **No record, no verdict** — a rule that needs the record is silent without a current one, and one
+  rule (`paper/sources-fresh`) says so once; a stale answer is worse than none.
+- **Prose rules splice only the files the record lists** — an include behind `\iffalse` contributes
+  nothing; without a record the text is `paper.tex` alone and `paper/sources-fresh` says the rest went
+  unlinted.
+- **A lookup of where a text stands is not a decision of what TeX reads** — `filecontents` blocks are
+  found in the text only to put a finding on the line that holds an entry (`sameDatabase`).
+- **One verdict per entry, not per key** — a key does not identify an entry; a verdict that cannot say
+  which entry it is about reports at the wrong one.
+- **"Same database" is what bibtex takes** — raw bytes and the final `@string` map are a proxy that
+  misses a `@string` redefined after use; the text between entries reaches no `.bbl`.
+- **One `.bib` reader, fenced by lint** — three readers grew before there was one; the parser's name
+  may appear in one folder only.
+- **One path → bibliography function** — three scripts each resolved it by hand, each with its own
+  refusals.
+- **Recorded TeX truth, not hand-written expectations** — the consumers' tests compare with what TeX
+  did, so a wrong belief about TeX cannot be written into both the code and its test.
+- **The post-build check, not emulation** — no parser recovers as bibtex does; bibtex's own output
+  is compared instead.
+
+## 10. Scope: what this PR does not decide
+
+- A finding for a `filecontents` block without `[overwrite]` (#176).
+- A listed body file that no include macro in the text stands for (an `\input` inside a macro's body,
+  #177): linted as a file of its own, absent from the whole-paper text.
+- `eslint --cache`: `bib/reachable-entry` depends on files other than the linted one, so a cached
+  result can be stale after a `.bib` changes. `paperlint lint` does not cache.
+- Reading past `\end{document}` in `headings.ts`, `layout.ts` and `rendered.ts` (#178).
+- A file a `filecontents` block writes under another extension than `.tex` (a `.pgf` the body
+  `\input`s) is not an input: its text reaches the PDF, but the record neither hashes nor splices it.
+- A database TeX wrote is read from the file it left on disk, which the record does not hash: edited
+  after a build, that file is not detected as stale.
+- Two copies compared where bibtex and the parser differ in what no check reads: a `@preamble` is
+  compared as written, so a `@string` it uses that is redefined between the copies is not seen; and
+  `keywords` are compared as the parser splits, dedupes and sorts them — it does so for that field
+  name unconditionally, with no option to keep it as written.
+- Q3 (one projection of the live text, replacing `paper-typography`'s `skippedRanges` and
+  `texToMdast`'s own parse), the hooks, and an enforcement rule for Q1–Q3: later PRs.

package/docs/package-shape-options.md

@@ -138,7 +138,7 @@ lockfile in that array was not.
 zero: `pipeline-check.mjs`, `extract-ref-facts.mjs` and `bib-authors.mjs`, all reaching
 `markdown-it`. Which retires a number this note carried: the closure is **two** third-party
 packages over **eight** referenced scripts, not one over seven — `extract-ref-facts.mjs` reaches
-`@retorquere/bibtex-parser` through a dynamic `await import()` that a static grep does not see.
+`@retorquere/bibtex-parser`, now through `#src/adapters/bibtex/index`.
 
 ⏳ **Not measured, recorded as such:** the documented 60-second install timeout; the personal
 `~/.claude/skills` and `--add-dir` channels (2 of 4 locations verified); and a genuine

package/docs/references.md

@@ -7,7 +7,11 @@ what happens in CI and without a network.
 
 ## What is checked, and where
 
-For each entry in the bibliography (the `filecontents` block inside `paper.tex`, else `refs.bib`):
+For each entry of the databases the build's bibtex **opened** — the `.bib` files its `\bibliography` /
+`\addbibresource` named after every switch and macro, and the file a `filecontents` block wrote. The
+build records them in `<paper>/_build/sources.json` from TeX's own files (`paper.blg`), so nothing here
+guesses which `.bib` TeX reads by reading the `.tex`. Their text is read by the one bibtex reader
+(`src/ports/bib-reader.ts`):
 
 | question                                                                                                 | asked of                                    |
 | -------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
@@ -23,18 +27,40 @@ A work that is simply not found is `unresolvable`, never a failure: plenty of re
 indexed. Only a positive disproof fails an entry, for example a DOI that resolves to a different
 paper, or one the DOI authority says does not exist.
 
-The verdicts land in `<paper>/_build/references.json`. `paperlint lint` reads that file and reports:
+The verdicts land in `<paper>/_build/references.json`: one verdict per entry of those databases, in the
+order bibtex opened them, and a hash of their text. `paperlint lint` reads that file and reports:
 
-| rule                 | when                                                              |
-| -------------------- | ----------------------------------------------------------------- |
-| `paper/cite-exists`  | an entry's identifier provably does not resolve to the work cited |
-| `paper/author-list`  | an entry's authors are the preprint's under a published venue     |
-| `paper/refs-fresh`   | the bibliography changed since the build checked it — build again |
-| `paper/refs-checked` | no build has checked the references, or the last one could not    |
+| rule                 | when                                                                                      |
+| -------------------- | ----------------------------------------------------------------------------------------- |
+| `paper/cite-exists`  | an entry's identifier provably does not resolve to the work cited                         |
+| `paper/author-list`  | an entry's authors are the preprint's under a published venue                             |
+| `paper/refs-fresh`   | the databases changed since the build checked them — build again                          |
+| `paper/refs-checked` | no build has checked the references, or the last one could not, or the reader missed some |
+
+An entry of a `.bib` is reported at the first line of `paper.tex`, with the entry's file, line and
+column at the front of the message (`refs.bib:12:1: …`): ESLint does not lint a `.bib`.
 
 The step never fails the build. Without a network the PDF is still built and lint warns that the
 references were not checked.
 
+**No record of the build, no answer.** Which databases bibtex opened is in `sources.json`, so a paper
+with no record (never built), or one whose files changed since (`paper/sources-fresh`), has none to
+check: the step records nothing, and the four rules are silent while `paper/sources-fresh` says once
+that the paper must be built.
+
+**What bibtex typeset, against what paperlint read.** The record also holds the keys bibtex typeset
+(its `paper.bbl`). bibtex has no comment syntax, so an entry behind `%` or inside `@comment{…}` is one
+it typesets and paperlint's reader does not; `paper/refs-checked` names each such key as not checked.
+(bibtex's keys are case-insensitive: `\cite{SMITH}` typesets `\bibitem{SMITH}` from an entry `smith`,
+which is the same entry.) A malformed `.bib` never gets that far: bibtex fails on it, and `paperlint
+build` fails with bibtex's own lines.
+
+A database TeX wrote itself (a `filecontents` block) is read from the file it left beside the paper.
+The build hashes it like any other, so when that file is edited or removed after the build the record
+is stale and `paper/sources-fresh` says so: build again to write it. A database bibtex found outside
+the paper directory, on its search path (`BIBINPUTS`), is not read: the step says which and
+`paper/refs-checked` warns.
+
 ## `repro/references-cache.json` — commit it
 
 Asking five services about every entry is slow. DBLP asks clients to pause between requests, and a

package/docs/rules/bib/reachable-entry.md

@@ -0,0 +1,64 @@
+# bib/reachable-entry
+
+**Level:** warn · **Reads:** `paper.tex`, and `_build/sources.json` — the databases the last build's
+bibtex opened, read through the bibtex reader
+
+## What it catches
+
+A bibliography entry with no `doi`, no `url` and no arXiv id: nothing a reader can follow. The entries
+judged are those of the databases bibtex opened in the last build — TeX's own answer, recorded by
+`paperlint build`, not a reading of the source for `\bibliography`.
+
+## Why
+
+Reviewer A on HotCRP #20 (2026-08-23): "not every entry has a DOI/link" — one paper had 0 of 47
+entries reachable. Not "no doi": ICLR, NeurIPS and TMLR issue no DOIs at all (measured 2026-08-24),
+so a url or an arXiv id counts.
+
+## Examples
+
+Reported on the entry's own line when the `.bib` bibtex opened is one TeX wrote from a `filecontents`
+block of `paper.tex`; any other entry — a `.bib` file the author keeps, or a block in an included file —
+at the top of `paper.tex`, with the entry's file, line and column first:
+
+```bibtex
+@book{lost2026, title = {A Book}, year = {2026}}
+```
+
+> refs.bib:12:1: `lost2026` has no doi, url or arXiv id — a reader has nothing to follow. If none
+> exists, keep the exception with `% eslint-disable-next-line bib/reachable-entry -- <why>` above
+> the entry
+
+Not reported:
+
+```bibtex
+@article{a, doi = {10.1/x}}
+@misc{b, url = {https://example.org}}
+@misc{c, note = {arXiv:2310.05736}}
+```
+
+## Options / preset fields
+
+None.
+
+## What it does not check
+
+- Whether the link resolves: that is the references check of `paperlint build`.
+- `@string` and `@preamble`, which are not entries.
+- A paper with no record of a build, or one changed since: the rule is silent, and
+  [`paper/sources-fresh`](../paper/sources-fresh.md) says once that it was not built.
+- A database bibtex found outside the paper directory, and an entry bibtex reads that the `.bib` reader
+  does not (behind `%`, inside `@comment{…}`): that is the post-build check's.
+- With `eslint --cache`, a change to a `.bib` alone does not re-run the rule: its findings depend on
+  files other than `paper.tex`. `paperlint lint` does not cache.
+
+## How to fix
+
+Add the entry's `doi`, `url` or arXiv id. An entry that has none — a talk, a personal communication
+— keeps its exception where it is written, on the line above the entry, in the block or in the
+`.bib`:
+
+```bibtex
+% eslint-disable-next-line bib/reachable-entry -- an invited talk, no recording exists
+@misc{smith2024talk, …}
+```

package/docs/rules/paper/sources-fresh.md

@@ -0,0 +1,81 @@
+# paper/sources-fresh
+
+**Level:** warn · **Reads:** `paper.tex` and `_build/sources.json` · **Reported on:** the first line of
+`paper.tex`
+
+## What it catches
+
+The paper has no usable record of the last build, so paperlint does not know which files TeX reads for
+it. One finding, in one of three forms:
+
+- **not built** — there is no `_build/sources.json`;
+- **unreadable** — the file is not JSON, is of another schema number, or does not have the shape of
+  schema 1;
+- **stale** — a file the build read has other bytes now, was deleted, or appeared where the build saw
+  none. The message names each: `sections/intro.tex edited, refs.bib deleted`.
+
+## Why
+
+`paperlint build` asks TeX itself which files it read (`pdflatex -recorder`) and which databases bibtex
+opened, and writes the answer, with a SHA-256 of every file, to `_build/sources.json`. Nothing in
+paperlint answers "which files make up this paper" or "which bibliography does it use" by reading the
+TeX source: `\input` behind a macro, a `.bib` written by `filecontents`, a database named by a
+switch are TeX's to decide, and TeX has already decided.
+
+The rules that need that answer — the ones over the bibliography, and the prose rules over the files
+`paper.tex` includes — read the record. With no record, or one about an earlier paper, they would
+judge files that are not the paper's, so they stay silent, and this rule speaks once in their place.
+Without it a paper that was never built would lint clean because nothing was looked at.
+
+## Examples
+
+Failing — a paper that was never built:
+
+```text
+papers/my-paper/paper.tex
+  1:1  warning  the paper has not been built — run `npx paperlint build`, which records the files TeX
+                reads. Until it has, the files `paper.tex` includes are not linted, and the rules that
+                read the bibliography say nothing
+                paper/sources-fresh
+```
+
+Failing — `sections/intro.tex` was edited after the last build:
+
+```text
+papers/my-paper/paper.tex
+  1:1  warning  the paper changed since the last build (sections/intro.tex edited) — run `npx paperlint
+                build`; until then the files `paper.tex` includes are not linted, and the rules that
+                read the bibliography say nothing
+                paper/sources-fresh
+```
+
+Passing: a paper whose files are byte for byte what the last build read. Editing a file TeX did not
+read (a figure source the paper does not include, a note beside it) changes nothing.
+
+## Options / preset fields
+
+No rule options. The level is `warn` for every paper; set it in the paper's `paperlint.json`:
+
+```json
+{ "rules": { "paper/sources-fresh": "error" } }
+```
+
+`error` suits a CI job that builds first; `off` suits one that only lints and would rather not hear it.
+
+## What it does not check
+
+- That the build was clean. A record is written after a build whose pdflatex and bibtex runs ended;
+  what they reported is `paper/refs-checked`'s and the build's own.
+- Files TeX never read. A file the paper does not include is not in the record, so its edits are not
+  changes, and it is not linted.
+- A file edited in an editor and not saved: the rule compares the bytes on disk with the record.
+- `.bib` files TeX wrote itself (`filecontents`): their bytes are the block's, which is in a file the
+  record hashes.
+
+## How to fix
+
+Build the paper again; the record is rewritten from the files as they are:
+
+```sh
+npx paperlint build papers/my-paper
+```