paperlint 2.0.0

package/docs/texlive-install-decision.md

@@ -0,0 +1,230 @@
+---
+title: "Where CI gets TeX Live from: three methods, measurements, what was chosen and why"
+created: 2026-09-01
+updated: 2026-09-24
+tags: [ci, texlive, render-paper, resheno-i-otkloneno, zamery]
+---
+
+# What to install TeX with in CI — a measured breakdown
+
+**Why this file exists.** Over one night on 2026-09-01 the method for installing TeX in the
+`build` job changed **three times**, each change cost a CI run, and two of the three methods
+failed for reasons invisible in the documentation. Without this record, the next attempt would
+start from the same first try.
+
+The corpus owner's question that prompted the file: _"write down why we chose one action over
+another"_.
+
+> **Where this ended up (2026-09-24).** Method 5 below won, and it is no longer a CI script: it is
+> `paperlint toolchain` (`src/toolchain.ts`), the command users run and the `build-e2e` job runs on Linux
+> and macOS. The package list and the file contract (`REQUIRED_FILES` in the deleted
+> `ensure-toolchain.sh`) moved into the venue profiles — each `skills/submit-paper/references/venues/*.jsonc`
+> declares a `tex` block of CTAN package → proof files, and `tex-base.jsonc` the set every paper
+> gets. This file stays as the record of why tlmgr and not apt or a container; the scripts it names
+> (`ensure-toolchain.sh`, `ci-install-texlive.sh`) were deleted in the same change.
+
+---
+
+## Summary
+
+| method                                      | size        | cold install              | cacheable | verdict             |
+| ------------------------------------------- | ----------- | ------------------------- | --------- | ------------------- |
+| `texlive/texlive:latest` container          | **2700 MB** | 2 m 01 s **every** run    | ❌ never  | 🟡 works, expensive |
+| `texlive/texlive:latest-medium` container   | 940 MB      | hangs 14 min              | ❌        | 🔴 rejected         |
+| apt + `awalsh128/cache-apt-pkgs-action`     | 2100 MB     | installed **zero**        | ✅        | 🔴 rejected         |
+| tlmgr + `teatimeguest/setup-texlive-action` | ~250 MB     | —                         | ✅        | 🔴 **blocked**      |
+| tlmgr + `actions/cache` (own script)        | ~250 MB     | 77 s + follow-up installs | ✅        | 🟢 candidate        |
+
+---
+
+## 1. `texlive/texlive:latest` container — what worked
+
+**How:** `container: image: texlive/texlive:latest` on the job.
+
+**Cost, measured 2026-09-01** (run `33455680908`): the "Initialize containers" step —
+**2 minutes 01 seconds**, and that **on every run**. GitHub does **not cache the job container
+image** under any circumstances; a homemade cache via `docker save`/`load` for a multi-gigabyte
+image costs more than the pool time it would replace.
+
+Of that 2.7 GB, our papers touch **47 MB of fonts** — the runtime subset of three families,
+without `doc/` and `source/` (issue #37; this line said "71 MB" until 2026-09-24, which counted the
+documentation and sources too):
+
+| family              | size, with doc/ and source/ | what it's for          |
+| ------------------- | --------------------------- | ---------------------- |
+| libertine           | 29 MB                       | acmart's main typeface |
+| inconsolata (`zi4`) | 23 MB                       | monospace              |
+| newtx               | 19 MB                       | math                   |
+
+So overhead is ~98%. A conscious tradeoff, not an unnoticed one.
+
+## 2. `latest-medium` — rejected 2026-09-01
+
+**Idea:** a 940 MB image instead of 2700, with the gap filled in by `tlmgr`.
+
+**What happened:** the follow-up class-install step hung for **14 minutes** (reaching out to
+CTAN), got cut off by the job's bond, the job came back `cancelled`, and the gate never ran at
+all. The trade of "save 2 minutes of pool time" cost fifteen minutes and an unrun gate.
+
+**The lesson isn't about the image, it's about bonds:** any step that reaches the network needs
+its **own** `timeout-minutes`. A job-level bond doesn't distinguish "the step is stuck" from "the
+set is slow" and hands the whole job over to the hang.
+
+## 3. apt + `awalsh128/cache-apt-pkgs-action` — rejected, and rejected QUIETLY
+
+**Idea:** GitHub caches `.deb` files, unlike a container image. Install the nine packages from
+`PACKAGES` in `ensure-toolchain.sh`.
+
+**What happened** (run `33456559102`, job `99697684739`):
+
+```
+E: Failed to fetch mirror+file:/etc/apt/apt-mirrors.txt/pool/main/o/openjdk-21/
+   openjdk-21-jre-headless_21.0.11+10-1~24.04.2_amd64.deb  404  Not Found
+E: Unable to fetch some archives
+Caching 0 installed packages...
+Skipped all manifest write. No packages to install.
+##[end-action …;outcome=success;conclusion=success]
+```
+
+`texlive-latex-extra` pulls in JRE via its dependencies, its version in the runner's index had
+gone stale, the mirror returned 404 — and apt installed **nothing**.
+
+🔴 **The main point here isn't the 404, it's that the step was GREEN.** The action reported
+`outcome=success`, cached 32 KB, and moved on. Only a separate `kpsewhich` check in the next step
+caught it, printing all 14 required files as missing. This is the third instance in the knowledge
+base of the rule **"`exit=0` and empty output is NOT 'clean'"**, and it's also the reason the
+post-install check lives in its own separate step: **you cannot trust the installer**.
+
+**The second reason for rejecting it, independent of the 404 — granularity.** apt's minimal unit
+is the distribution package: for three font styles it pulls in the **whole** of
+`texlive-fonts-extra`.
+
+|             |                                      |
+| ----------- | ------------------------------------ |
+| needed      | 47 MB (three families, runtime; #37) |
+| installed   | **1691 MB** (`texlive-fonts-extra`)  |
+| waste share | **97%**                              |
+
+apt can't go finer than that — in Debian these three families aren't split into separate
+packages.
+
+⚠️ **A side effect: a stale note surfaced in our own file.** While justifying the move, I quoted a
+comment from `ensure-toolchain.sh`: "172 MB fetched, 482 MB on disk." That measurement is dated
+17.08, and `texlive-fonts-extra` was added to the list on **24.08** — meaning the comment predates
+half the set. Remeasured: **2.1 GB**. A comment in our own file is not a verified measurement.
+
+## 4. `teatimeguest/setup-texlive-action` — BLOCKED, not rejected
+
+**Idea:** the right class of tool. Installs the **upstream** TeX Live from a list of CTAN
+packages, cache built in. The LaTeX ecosystem's standard answer to "I need three fonts, not two
+gigabytes."
+
+**What happened** (run `33457399377`): the job died at the **"Set up job"** step in 2 seconds.
+
+```
+Prepare all required actions
+Getting action download info
+##[error]Repository access blocked
+```
+
+The account has a **third-party GitHub Actions allowlist** turned on, and this action isn't on
+it. Confirmed alongside it: `awalsh128/cache-apt-pkgs-action@v1` and `zernie/vigiles@v20.0.0` in
+the same file start up normally.
+
+🔴 **Fixed NOT by code:** Settings → Actions → General → Allow specified actions. Not reachable
+from the session — checked, not assumed:
+
+| channel                       | result                                                                                                                                         |
+| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `GH_TOKEN` in the environment | present, and valid (`GET /user` → `zernie`)                                                                                                    |
+| bare `curl` to api.github.com | **403 on everything**, including `GET /repos/<owner>/<private-repo>`: `Access to this GitHub Actions path is not permitted through this proxy` |
+| GitHub MCP                    | the only reachable channel, and it has **no** repository-settings tool                                                                         |
+
+**Status: not rejected, unreachable.** If the action ever gets allowlisted, it's the best option,
+and the code for it sits in the history of the `claude/article-deadline-check-0unptg` branch
+(commit `be6f33ec8`), restorable in one commit.
+
+### Why comparing the actions was, in itself, the wrong move
+
+I picked `cache-apt-pkgs-action` **first** because I searched for "how to cache an install," and a
+generic tool for a generic query turned up immediately. The right question was different:
+**"what does this ecosystem actually use to install TeX specifically."** LaTeX-specific actions
+exist, and there are three of them:
+
+| action                                  | what it does                                    | why not this one                                                                       |
+| --------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------- |
+| **`teatimeguest/setup-texlive-action`** | upstream TL from a package list, cache built in | ✅ best on the merits, blocked by the allowlist                                        |
+| `xu-cheng/latex-action`                 | the most popular of all                         | wraps the same `texlive/texlive` **image** underneath — same pool cost, solves nothing |
+| `zauguin/install-texlive`               | same idea as the first                          | less maintained, no advantage                                                          |
+
+**Lesson:** before reaching for a general-purpose tool, ask whether a specialized one exists for
+this exact subject. The general one gives you its own domain's granularity (apt → distribution
+package), the specialized one gives you its own (tlmgr → CTAN package), and here the difference
+is **twelvefold** by weight.
+
+## 5. tlmgr + `actions/cache` with our own script — candidate
+
+**Idea:** bypass the allowlist entirely. `actions/cache` is first-party (`actions/*`), allowed
+under any "Allow select actions" policy, and we **already** use it to cache TeXtidote's
+224-megabyte jar.
+
+**Measurement, run LOCALLY in the session container on 2026-09-01** (not on CI — that's the whole
+point):
+
+| step                                 | result                     |
+| ------------------------------------ | -------------------------- |
+| `install-tl-unx.tar.gz`              | 5.3 MB                     |
+| `scheme-basic` per profile           | **147 MB, 65 seconds**     |
+| `tlmgr install` 19 packages          | +75 MB, 12 seconds         |
+| **total before the first build**     | **222 MB, 77 seconds**     |
+| binary directory                     | `texlive/bin/x86_64-linux` |
+| `kpsewhich` against `REQUIRED_FILES` | **14 of 14** ✅            |
+
+⚠️ **`--cacert` is mandatory:** in this container `curl` to CTAN fails certificate verification
+(the proxy), fixed with `CURL_CA_BUNDLE=/root/.ccr/ca-bundle.crt`. Not needed on a GitHub runner —
+written down so the next spike doesn't lose ten minutes to it.
+
+### 🔴 And this is where `kpsewhich` lied — and the build didn't
+
+14 of 14 files resolved, and `pdflatex` failed on the real paper:
+
+```
+! LaTeX Error: File `xstring.sty' not found.
+!  ==> Fatal error occurred, no output PDF file produced!
+```
+
+`xstring` is pulled in by `acmart`, and it's in neither the package list nor `REQUIRED_FILES`.
+What followed was the knowledge base's familiar **endless queue**: `xstring` → `everyshi` →
+`hyperxmp` → `ncctools` → `cmap` → `float` → `comment` → `upquote` → `doclicense` → …
+
+**This is exactly the case that `CLAUDE.md`'s "install by COLLECTION, not by name" rule is for.**
+But that same rule also names the reason: _"names surface one at a time, and each one costs a full
+CI run (~25 min of quota)"_.
+
+🔑 **Here that reason disappears.** The queue is run **locally**, in a loop of "build → pull out
+the missing file → `tlmgr search --global --file` → install → repeat," and it costs **zero
+minutes of quota**. What changes is not the fact of the queue, but its cost — and the conclusion
+"names can't be listed one at a time" rested precisely on that cost.
+
+**The rule this implies, and it's broader than TeX:** before accepting "must list only in bulk,
+never one at a time," check **whether the iteration itself can be made free.** An expensive
+iteration is a property of where it's run, not a property of the task.
+
+⚠️ **The honest cost of an exact list:** a new paper pulling in a new package will break CI with a
+missing filename. That's acceptable, because the `REQUIRED_FILES` check prints the fix straight
+into the log, and because the alternative — pulling in the whole of `collection-latexextra` —
+means going back to the same hundreds of megabytes of waste this move was meant to escape.
+
+---
+
+## What to check on the next attempt
+
+1. **Build a real paper, not just `kpsewhich`.** Filename resolution and a successful build are
+   **different claims**, and here the second turned out to be stricter than the first by nine
+   packages.
+2. **Watch for `Class acmart Warning` in `paper.log`.** Missing libertine/zi4/newtxmath does
+   **not** crash acmart — it silently falls back to Computer Modern, the PDF builds and looks
+   normal, but is typeset in the wrong font, and a different metric gives a different pagination.
+3. **After installing, check the result, not the installer's exit code.** See §3.
+4. **Run iterations locally.** The spike above cost zero minutes of quota and caught three errors,
+   each of which would have cost a CI run.

package/docs/toolchain.md

@@ -0,0 +1,152 @@
+# External programs the skills need
+
+`paperlint lint` needs nothing but Node — it reads your files and reports. **The skills are a different
+matter**: they build PDFs, read them back, and run external checkers, so they call programs this
+package does not ship.
+
+This page is the prerequisite list and how to satisfy it cheaply. It was moved out of the README
+on 2026-09-19: it answers "how do I make the render stage work", which is a question you have
+after deciding to use the tool, not before.
+
+## The programs
+
+| program              | comes from                       | which skills call it                     | what happens without it                                               |
+| -------------------- | -------------------------------- | ---------------------------------------- | --------------------------------------------------------------------- |
+| `pdflatex`, `bibtex` | TeX Live (`paperlint toolchain`) | render-paper, submit-paper, camera-ready | no PDF is produced — loud                                             |
+| `texcount`           | TeX Live (`paperlint toolchain`) | render-paper, grade-paper-writing        | the length checks cannot run                                          |
+| `checkcites`         | TeX Live (`paperlint toolchain`) | render-paper                             | nothing asks whether a bibliography entry is uncited                  |
+| `java`               | any JRE (21 works)               | render-paper                             | TeXtidote does not run, and **nothing else spell-checks the text**    |
+| `python3`            | your system                      | the analysis and report scripts          | those scripts do not start                                            |
+| `tlmgr`              | TeX Live                         | the TeX installer itself                 | you cannot add a TeX package                                          |
+| `banal`              | HotCRP (`paperlint toolchain`)   | `paperlint build`, extract-pdf-facts     | page size, columns and font sizes are `null` in the facts file        |
+| `perl`               | your system (macOS has it)       | banal                                    | banal cannot run — the same `null`, and `paperlint toolchain` refuses |
+
+🔴 **Most of these fail QUIETLY**, which is why they are listed rather than left to be discovered.
+A missing checker and a passing checker look identical from outside, so every script here states in
+its last line which checks actually ran — read that line, not the exit code.
+
+## TeX Live and banal: `paperlint toolchain`
+
+```sh
+npx paperlint toolchain            # install, or add what is missing; a second run does nothing
+npx paperlint toolchain --check    # report, change nothing; exit 1 when a declared package or banal is missing
+```
+
+The command has two halves, TeX Live and banal (below). Both always run, so one failing does not
+hide the other, and the exit code is 0 only when both are ready.
+
+It downloads `install-tl` from a CTAN mirror (four in turn; each download has its own time limit,
+TLS is always verified), installs `scheme-basic` into `~/.cache/rpp/texlive/<TeX Live year>`
+(`$XDG_CACHE_HOME/rpp/texlive` when that is set, `$RPP_TEXLIVE_DIR` over both), and `tlmgr install`s
+every package the venue profiles declare. Then it checks the RESULT: `kpsewhich` must find every
+file each profile names, and each declared tool must be an executable in the bin directory. A gap
+fails the command and names the package and its file. Linux and macOS; on Windows it refuses, and
+`paperlint build` uses a TeX Live on PATH that has the packages. `RPP_CTAN_MIRROR` names one mirror
+(a `…/systems/texlive/tlnet` URL) to use instead of the list.
+
+Measured 2026-09-24 from an empty directory: **3 min 04 s, 269 MB** (du: 298 MB), TeX Live 2026, 47
+packages verified (46 files, 2 tools). The second run: 0.55 s, "nothing to do".
+
+**When CTAN moves to a new TeX Live year.** A complete tree is left alone: a new release on CTAN is
+not by itself a reason to download 270 MB. It matters only when a venue declares a package the
+tree lacks — tlmgr then refuses to install into an older year (`Local TeX Live (2026) is older
+than remote repository (2027)`). `paperlint toolchain` reads the two years from that refusal, installs
+a fresh tree for the new year into its own `<TeX Live year>` directory, installs every declared
+package there and verifies it, and from then on that tree is used. The old tree is not deleted;
+the run names it with its size, and removing it is your call. `paperlint build` uses the newest tree
+that has every declared package, so a new-year install that was interrupted never replaces a
+complete older one.
+
+**Where the package list lives:** in the venue profiles, not here and not in a script.
+`skills/submit-paper/references/venues/<venue>.jsonc` carries a `tex` block — CTAN package → the
+files that prove it is installed — and `tex-base.jsonc` carries what every paper gets, and all a
+paper with an unknown venue gets. The shape is `venue-profile.schema.json` in the same directory.
+A venue that needs a new package gets one line there.
+
+**`paperlint build` uses it without being asked to.** It looks for a TeX Live with every package the
+paper's venue declares — paperlint's own first, then the `pdflatex` on PATH — and on a terminal offers to
+install one. Without a terminal it stops with one line naming `npx paperlint toolchain`. The last line
+`paperlint toolchain` prints is `bin: <dir>`: put that first on PATH to call `pdflatex`, `texcount` or
+`checkcites` by hand, or from a skill's script.
+
+Why upstream TeX Live by package name and not the distribution packages: `texlive-fonts-extra`
+alone is **1691 MB**, and acmart papers use **47 MB** of it — apt cannot install less. The measured
+comparison of every method tried (containers, apt, a GitHub Action, this) is
+[`texlive-install-decision.md`](texlive-install-decision.md).
+
+## Reading the PDF: nothing to install
+
+`paperlint build` and `extract-pdf-facts.mjs` read the finished PDF — page count, the fonts it draws text
+with, the last page's words — with **pdf.js**, which arrives with paperlint as the npm dependency
+[`unpdf`](https://www.npmjs.com/package/unpdf). Until 2026-09-24 this took three poppler programs
+(`pdfinfo`, `pdffonts`, `pdftotext`) from the system package manager; a measurement on 25 PDFs
+found pdf.js equal on page counts and Type 3 fonts, and the last page's column heights within
+0.2 pt except on an all-Type-3 page (7.2 pt). Poppler is no longer needed by anything paperlint runs.
+
+## Page geometry: banal, without poppler
+
+The page size, column count, body and reference font sizes and page types in the facts file come
+from **banal**, the page-geometry script HotCRP's format checker runs
+([`src/banal`](https://github.com/kohler/hotcrp/blob/master/src/banal), by Geoffrey M. Voelker and
+Eddie Kohler). paperlint runs the real banal, unmodified, so the numbers are the ones HotCRP shows at
+upload.
+
+banal normally reads a PDF through poppler's `pdftohtml -xml`. paperlint does not: it writes that XML
+itself from the same pdf.js read (`src/adapters/banal/xml.ts`) and hands banal the `.xml` file, which banal
+accepts as input. Measured on 50 PDFs / 598 pages (2026-09-25): banal on paperlint's XML and banal on
+real `pdftohtml` agree on every field the facts file keeps, with no venue verdict changed. They
+agree only because paperlint leaves out rotated and invisible text (as `pdftohtml` does), writes each
+text's colour so banal drops light text by its own rule, and writes sizes and coordinates at the
+zoom and precision banal expects. `test/e2e/banal.mjs` checks each of those against the real banal
+on the committed fixtures.
+
+**How it is installed — and the licence boundary.** banal is **GPL-2.0-or-later**; paperlint is MIT. paperlint
+therefore does not contain banal. `paperlint toolchain` downloads it from HotCRP at a pinned commit:
+
+|         |                                                                                                      |
+| ------- | ---------------------------------------------------------------------------------------------------- |
+| URL     | `https://raw.githubusercontent.com/kohler/hotcrp/f3e4352133f3184c7c42b0d5e6501124bead18e6/src/banal` |
+| version | banal 1.2                                                                                            |
+| sha256  | `fd8cc4ae189b9da02460ae442a34f14434e5784210489fb668313ac671006911`                                   |
+
+It refuses a file with any other sha256, stores it in `~/.cache/rpp/banal/<commit>/banal`
+(`$XDG_CACHE_HOME/rpp/banal` when set, `$RPP_BANAL_DIR` over both), and accepts it only after banal
+has run on a one-page probe and measured it. paperlint then runs it as a separate program — `perl banal
+-no-time -json <file>.xml` — and reads its JSON output. Nothing of banal is copied, linked or
+translated into this package.
+
+**perl is required** for that, and is checked: without it `paperlint toolchain` fails and says how to
+install it (Debian/Ubuntu `apt-get install perl`; macOS ships it). banal asks `pdftohtml -v` before
+it reads any input, so paperlint points banal's `$PDFTOHTML` at a small stub that answers with the
+`pdftohtml` version whose XML paperlint writes. Poppler itself is not needed by anything paperlint runs.
+
+**Which banal is used**, in order: `$BANAL` (a path you name), `vendor/banal` in the project (a copy
+you vendor), then the one `paperlint toolchain` installed. Without any, `paperlint build` still succeeds with
+the geometry fields `null` and says so with the command that fixes it; `extract-pdf-facts.mjs
+--strict` fails.
+
+pdf.js needs **Node 22.13 or later**, which paperlint requires anyway: on Node 20 it opens the same PDFs
+and reports zero fonts without an error, and paperlint refuses such a read instead of reporting a
+clean font list.
+
+## The external checkers
+
+`aclpubcheck` (the official ACL format checker), TeXtidote (spelling) and `rebiber` are not TeX
+packages and not npm packages. One idempotent command installs them and then **proves each one
+starts**:
+
+```sh
+bash node_modules/paperlint/skills/render-paper/ensure-checkers.sh
+```
+
+```
+   ✅ aclpubcheck
+   ✅ rebiber
+   ✅ jinja2
+   ✅ textidote (/opt/textidote/textidote.jar)
+✅ all checkers are installed AND run
+```
+
+It checks that the tools RUN, not that pip exited zero — `aclpubcheck --help` prints usage and
+exits zero on an interpreter where its own dependencies do not import, so "installed" and "works"
+are separate questions here.