@geonosis/ratchet 0.4.0 → 1.0.0

package/README.md

@@ -178,12 +178,30 @@ counter that reads a file rather than running a command (`lawLineCount`) prints
 
 ### `testFailures` reads the runner's summary, never the exit code
 
-The counter looks for the runner's own count — `N failed` — and **refuses when it cannot parse
-one**. It never reads the process exit code, because a test runner exiting 0 over a red suite is
-commoner than anyone expects: `@cloudflare/vitest-pool-workers` 0.22 on vitest 4.1 exited 0 with
-failing tests on every workerd suite of a consumer, and every gate that trusted the exit code
-reported green over red for weeks. A refusal is loud and stops the run; a trusted 0 is silent and
-banks the red as a win.
+The counter looks for the runner's own count and **refuses when it cannot parse one**. It never
+reads the process exit code, because a test runner exiting 0 over a red suite is commoner than
+anyone expects: `@cloudflare/vitest-pool-workers` 0.22 on vitest 4.1 exited 0 with failing tests on
+every workerd suite of a consumer, and every gate that trusted the exit code reported green over red
+for weeks. A refusal is loud and stops the run; a trusted 0 is silent and banks the red as a win.
+
+**The lines it reads are the runner's summary lines, whole, and nothing else** — two dialects, both
+captured from the binary rather than guessed at:
+
+| Runner | The line | Read as |
+| --- | --- | --- |
+| vitest 3.2.7 | `      Tests  1 failed \| 1 passed (2)` | 1 |
+| vitest 3.2.7 | `      Tests  3 passed (3)` · `      Tests  2 skipped (2)` | 0 |
+| vitest 3.2.7 | `      Tests  no tests` (a suite that threw before collecting) | **refused** |
+| bun 1.4.0 | ` 1 fail` on its own line | 1 |
+| bun 1.4.0 | ` 0 fail` | 0 |
+
+Nothing else in the output counts. `Test Files  1 failed (1)` is a count of files, not of tests;
+`(fail) one [11.71ms]` is bun naming one; `naming 4 failing test(s)` is some wrapper counting its
+own findings. All three used to be read as a failure count — a **green run regressing a baseline**,
+the false RED mirroring the false green above (backlog #44). A run that prints no summary line at
+all is unmeasured, and the refusal names the two dialects so a third runner's output is a message
+rather than a wrong number. Summaries **add up**: a command that invokes the runner twice prints
+two, and reading the first and stopping banks the second suite's failures as a win.
 
 For the same reason, **give every test package its own `testFailures` entry**, each with its own
 `key`. One entry over one workspace measures one workspace; the suites it does not run are not zero
@@ -196,6 +214,38 @@ failures, they are unmeasured — and unmeasured reads exactly like green.
 `examples/during-day.ratchet.json` is four test entries for that reason: `apps/web` plus the three
 workerd packages that were previously outside every gate.
 
+### …or the runner's JSON report, which is stronger
+
+`report: "vitest-json"` reads the runner's own machine-readable answer instead of its prose:
+
+```jsonc
+{
+  "counter": "testFailures",
+  "key": "testFailuresApi",
+  "command": "cd apps/api && bunx vitest run --reporter=json --outputFile={report}",
+  "report": "vitest-json"
+}
+```
+
+`{report}` is replaced with a path in a temp directory the counter makes and removes; give the entry
+a `reportPath` instead when the report belongs somewhere your CI already collects. A command in this
+mode with neither is refused, naming what is missing — a run whose report goes nowhere is a run
+nobody can read.
+
+The counter then reads `numFailedTests`, and **refuses** rather than returning a number when:
+
+- the file is not there — a crash before the reporter wrote is not a pass, and it prints no summary
+  line either, so the summary parser had nothing to refuse on;
+- the file is not JSON, or has no `numFailedTests` in it;
+- the report says `success: false` and names **0** failing tests — the shape a pool that dies
+  mid-run writes. The run did not finish, so there is no number to bank.
+
+The summary mode stays the default; nothing changes for an entry that does not ask for a report.
+`--prove` proves all three: `testFailures` ships one probe per reading mode — `summary (vitest)`,
+`summary (bun)`, `vitest-json` — and prints them by name (`PROVEN testFailuresApi (vitest-json)`),
+because proving the mode nobody configured says nothing about the mode they did, and a summary
+dialect nobody proved is a dialect nobody has been shown to read.
+
 ### `oxlintRule` counts a warned rule twice, on purpose
 
 A rule parked at `"warn"` as ratcheted debt appears in two numbers: once inside `oxlintWarnings`,
@@ -206,12 +256,36 @@ That is not double-counting the total; the second key exists so the debt is visi
 of hidden inside a lump sum that a different rule's warning could mask. dielime today: `oxlintWarnings`
 12 → 135 when `no-raw-html-atoms` was armed, with its own key at 123.
 
+### `--prove` also proves the lock
+
+`--exclusive` is a claim about the machine, so it is measured on the machine, every `--prove`:
+
+```
+PROVEN exclusive: two runs of 400ms serialised, the second starting 161ms after the first finished
+```
+
+The self-test runs two children of this CLI over `--hold <ms>` — an instrument that takes the lock,
+says when it started, waits, says when it finished, and gives the lock back — and requires the second
+to have started after the first finished. Anything else prints `CANNOT FAIL exclusive: interleaved`
+and exits 2. It uses a lock file of its own in a temp directory, so proving the lock never takes the
+real one out from under the runs it exists to serialise.
+
+It is here because the lock did not work for two releases and every gate was green throughout:
+`openSync(path, 'wx')` is atomic about the NAME and not about the holder, and a run polling in that
+window read an unparsable lock, called it stale, and took it. A lock nobody has watched fail has not
+been shown to work — and this one guards the running time of every other gate.
+
 ## Running it where it will actually run
 
 - **Counters run in the caller's environment.** They inherit the shell the ratchet was started in,
   nothing more. If a child command needs `NODE_OPTIONS` — a TypeScript shim, a loader — put it on
   the script that invokes `geonosis-ratchet`, not on the counter's own line, and not only in your
   interactive shell.
+- **A path in a counter's command must be absolute, or `--prove` cannot run it.** A probe runs in a
+  scratch directory with none of your repo in it, so a loader or shim named relatively — `node
+  --import ./scripts/ts5-shim.mjs …` — resolves to nothing there and the counter comes back
+  "command did not run". The command is the same string in both places; write the path as
+  `$PWD/scripts/ts5-shim.mjs` and it works in the repo and in the probe alike.
 - **A `typecheckErrors` counter needs the same precondition CI gives it: build the workspace
   packages first.** In a fresh worktree the `dist/*.d.ts` files do not exist yet, and `tsc` reports
   a false +N of missing-module errors that has nothing to do with the change under test. `pnpm build
@@ -266,15 +340,60 @@ Every counter takes its `command` from the config, so the toolchain stays the re
 | `oxlintErrors` / `oxlintWarnings` | findings under any shape oxlint prints — `--format=unix`, the compact `agent` format, the graphical `default` — cross-checked against the tool's own summary | `command`, `expectFormat` |
 | `oxlintRule` | one named rule's findings, counted only on lines the run reported as findings and refused when none of them attributes itself readably; with `config`, after forcing the rule to `error` in a temp copy — `"warn"`, `"off"` and the `["off", { … }]` array form alike — so debt cannot grow behind a downgrade | `rule`, `config`, `command`, `expectFormat` |
 | `typecheckErrors` | `error TS` occurrences, refusing when TS2305/TS2307 name a workspace package of this repo — an unbuilt sibling is a missing build, not debt | `command` |
-| `testFailures` | the runner's own failure summary; throws when neither a pass nor a fail count is readable | `command` |
+| `testFailures` | the runner's own failure summary — or, with `report: "vitest-json"`, `numFailedTests` out of the JSON report the command wrote; throws when nothing is readable, when the report is absent, and when the report failed with nothing failing | `command`, `report`, `reportPath` |
 | `unformattedFiles` | paths `--list-different` names that exist on disk | `command` |
 | `cloneCount` | jscpd's `Found N clones` | `command` |
 | `knipIssues` | the totals under knip's unused-* headings | `command`, `headings` |
 | `boundaryIssues` | `N issues found` from a boundary scan | `command` |
-| `archViolations` | lines matching a marker your own architecture scan prints | `command`, `match` |
+| `archViolations` | lines matching a marker your own architecture scan prints, refusing a non-zero exit that printed none of them — a scan that could not run is not a clean scan | `command`, `match` |
 | `sumOfCounts` | the total of one capture group across a per-file census (`grep -rc`) | `command`, `match` |
 | `lawLineCount` | the lines of the law file — a ceiling that can only come down | `path` |
 | `runtimeCodeShipped` | 0 when a change shipped runtime code, 1 when it shipped none | `command`, `patterns` |
+| `disabledCiJobs` | lines of `if: false` across the workflow files — a job switched off to get a release through, still off. A condition that merely mentions `false` is not one. No workflows directory at all reads 0 | `dir` |
+| `bundleBytes` | one integer out of whatever your sizing command printed, separators and all; with `match`, the group that pattern names rather than the last integer, refusing when it matches nothing. **Tolerates.** | `command`, `match`, `tolerance` |
+| `fastTierMs` | `finishedAt − startedAt` from the gate report `geonosis-verify` wrote, refusing a report of another tier rather than timing the wrong gate. **Tolerates.** | `report`, `tier`, `tolerance` |
+| `testsWithoutRunner` | workspaces holding `*.test.*`, `*.spec.*` or `__tests__/` with no `test` script — the suites nobody runs, which read exactly like suites that pass | `script` |
+| `packagesWithoutTypecheck` | workspaces with no `typecheck` script | `script` |
+| `walkFindings` | the defects in the report `geonosis-walk` wrote, over every page; with `classes`, only those classes, refusing a class the walk does not have. A missing or unparsable report is a refusal — the walk writes none when it could not run | `report`, `classes` |
+
+### `tolerance`, and the two counters that accept one
+
+Most counters count findings, where one more is one too many. `bundleBytes` and `fastTierMs` measure
+something that moves on a dependency patch nobody chose, and a gate that fails on +40 bytes is a
+gate that gets switched off within a week. `tolerance` is the fraction of the baseline such a number
+may drift up before the ratchet calls it growth:
+
+```jsonc
+{ "counter": "bundleBytes", "command": "…", "tolerance": 0.1 }  // +10 % is noise, +11 % is debt
+```
+
+**Only a counter that declares `tolerates` accepts one, and only `bundleBytes` and `fastTierMs` do.**
+Anything else stops the run:
+
+```
+geonosis-ratchet: "oxlintErrors" does not accept a tolerance — only a counter that measures a
+quantity declares one
+```
+
+That refusal is the point. A band around a findings count is not a tolerance, it is the gate turned
+off from the config file: `"tolerance": 100` on `oxlintErrors` and the run still prints PASS. It is
+law 2 — never downgrade a rule — wearing a friendlier name, and it would be the easiest edit in the
+repo to get past a reviewer. Whether a number is a quantity or a count is the counter's to know, not
+the config's, so the counter declares it.
+
+It forgives noise upward, never downward: a shrink of any size is still a shrink and still lowers
+the baseline, or the number would stop following the artefact down. A `tolerance` that is not a
+non-negative number stops the run and says which entry it was — one that silently became `NaN`
+would make every comparison false, which reads exactly like a counter that can never grow.
+
+The `geonosis` Claude Code plugin puts any write that introduces or raises one in front of a human
+(`permissionDecision: "ask"`): a band somebody chose and a band somebody's agent chose are not the
+same thing.
+
+Tolerance is also why a rewrite lowers only the numbers that **shrank**. While `held` implied
+`now === baseline`, writing every measured key back was a harmless no-op; with a tolerance it is
+not, and a counter that grew inside its tolerance would have had that growth laundered into its new
+floor by the first unrelated win.
 
 Two example configs from real repos live in `examples/` in the repository.
 
@@ -289,4 +408,18 @@ process.stdout.write(formatReport(result))
 
 `counters` is a plain array, so a repo can add its own `{ id, run }` beside the built-ins.
 
+## Two things this cannot see about itself
+
+A baseline is lowered **in place** when a number shrinks; that is what locks a win in, and it is
+also what lets a branch write any number it likes and stay green on every gate it runs. And a
+workspace whose suite no `testFailures` entry covers is unmeasured, which reads exactly like green.
+
+[`@geonosis/doctor`](https://www.npmjs.com/package/@geonosis/doctor) asks both from outside — the
+baseline at HEAD against another ref, and which workspaces have a test script nothing reads a report
+from:
+
+```bash
+npx geonosis-doctor --baseline-against origin/main --strict
+```
+
 Apache-2.0.