@codyswann/lisa 4.4.18 → 4.4.20

package/typescript/create-only/.github/required-checks.json

@@ -13,9 +13,11 @@
     "`_example_required_contexts` below is a STARTING POINT FOR TYPING, never read by the guard. Verify every line against your own ruleset before promoting any of it.",
     "THE FAMILY: required-and-red is loud; required-and-vacuous is not; advisory-and-stale is invisible. All three are one gate reporting satisfied without proving anything. `required_contexts` + `skip_job_declarations` above cover the SKIPPED variant; `evidence_bearing_checks` below covers the VACUOUS one.",
     "VACUOUS, measured (CodySwannGT/lisa#2497): a required `CodeRabbit` context posted `success` with the description `Review rate limited`, having reviewed nothing, on two security-relevant PRs that then merged and shipped. `gh pr checks` prints `pass` for that exactly as it does for a real review — only the DESCRIPTION tells them apart.",
-    "Run it per PR: `npm run check:vacuous-required-checks -- --pr=1234` (or `node scripts/check-skipped-required-checks.mjs --pr=1234`). It reads `gh pr checks --json name,state,bucket,description`, which is the only route that carries the description for a legacy commit status like CodeRabbit's.",
+    "NOTHING HAS TO REMEMBER A FLAG. `.github/workflows/review-evidence.yml` runs this arm on every pull request, and `npm run check:vacuous-required-checks` resolves the pull request itself: `--pr`, else the `pull_request` payload at GITHUB_EVENT_PATH, else `refs/pull/N/merge` in GITHUB_REF, else the checked-out branch. For one specific pull request by hand: `npm run check:vacuous-required-checks -- --pr=1234`.",
+    "It reads `gh pr checks --json name,state,bucket,description`, the only route that carries the description for a legacy commit status like CodeRabbit's, and falls back to the commit-status API when the token lacks `actions: read` (`gh pr checks` resolves the rollup through `checkSuite.workflowRun`, and without that scope it exits non-zero with EMPTY STDOUT — a failure shaped like a content problem that never says the word \"permission\").",
+    "IT REFUSES RATHER THAN REPORTING AN EMPTY INSPECTION. An unresolvable pull request, a `gh` it could not read, a roster with no checks in it, or an empty `evidence_bearing_checks` all print the same \"nothing vacuous here\" that a genuinely clean run prints — so the guard fails instead and names which of the four it hit (`vacuity_pr_unresolved`, `vacuity_checks_unreadable`, `vacuity_no_checks_reported`, `vacuity_none_declared`). A refusal is NOT a finding: NOBODY LOOKED and THE REVIEW WAS FAKE are opposite facts, and a single red job would otherwise report them identically.",
     "`evidence_bearing_checks` names the checks whose GREEN is supposed to mean something reviewed the code. Use `{}` to accept the shipped description vocabulary, or add `proof` / `no_work` arrays to extend it — extensions ADD to the defaults, they do not replace them. Undeclared checks are never examined, because most CI jobs ship an empty description and flagging them all would bury the one finding that matters.",
-    "This arm REPORTS AND NEVER BLOCKS, in every enforcement mode. A review bot can go hollow because an org-wide SPENDING CAP was hit, and a gate that reddens every PR on a billing state is worse than the one it criticises. What it changes is what you may CLAIM: a PR carrying a `vacuous_required_check` finding has not been shown to be reviewed, so do not record it as reviewed."
+    "This arm REPORTS AND NEVER BLOCKS, in every enforcement mode. A review bot can go hollow because an org-wide SPENDING CAP was hit, and a gate that reddens every PR on a billing state is worse than the one it criticises. What it changes is what you may CLAIM: a PR carrying a `vacuous_required_check` finding has not been shown to be reviewed, so do not record it as reviewed. Once you have made the governance call, `--fail-on-vacuous` is the supported way to ask for an exit code — add it to the command in `review-evidence.yml`, instead of writing a wrapper that reads `--json`."
   ],
   "enforcement": "warn",
   "ruleset": {

package/typescript/create-only/.github/workflows/review-evidence.yml

@@ -0,0 +1,122 @@
+# Seeded by Lisa on first setup — this file is YOURS.
+# Lisa will not overwrite it. (copy-overwrite assets ARE replaced each run.)
+# -----------------------------------------------------------------------------
+# Review evidence — the VACUITY arm, wired to a pull request (#2928)
+# -----------------------------------------------------------------------------
+# `.github/required-checks.json` names, in `evidence_bearing_checks`, the checks
+# whose GREEN is supposed to mean something reviewed the code. A review bot can
+# report `success` with the description `Review rate limited` having reviewed
+# NOTHING — `gh pr checks` prints `pass` for that exactly as it does for a real
+# review, and only the description tells them apart.
+#
+# WHY THIS WORKFLOW EXISTS. The rule that detects it shipped long before
+# anything invoked it: `quality.yml` runs the guard's OFFLINE arm with no `--pr`,
+# `required-checks-drift.yml` runs `--remote` on a schedule, and the package
+# script named for the vacuous check was a bare invocation. So the detection was
+# `declared-but-uncallable` — the very family the guard exists to describe,
+# one level up inside its own shipping. Measured on the repository that owns it:
+# 23 of the last 25 merged pull requests had a REQUIRED review check that
+# reviewed nothing, and nothing anywhere said so.
+#
+# WHY IT IS ITS OWN WORKFLOW AND NOT A JOB IN `ci.yml`. Reading checks needs
+# scopes the reusable-quality caller does not grant, and a called workflow may
+# only DOWNGRADE the caller's grant — asking for more is a startup_failure for
+# the entire run (#2049). A standalone workflow declares its own permissions and
+# constrains nothing else.
+#
+# THIS NEVER BLOCKS A MERGE, on purpose. A review bot can go hollow because an
+# org-wide SPENDING CAP was hit, and a gate that reddens every pull request on a
+# billing state is worse than the one it criticises. What the finding changes is
+# what you may CLAIM: a pull request carrying it has not been shown to be
+# reviewed, so do not record it as reviewed. Once you have made that governance
+# call, add `--fail-on-vacuous` to the command below — that is the supported way
+# to ask for an exit code, and it replaces the `--json`-reading wrapper every
+# consumer used to have to write.
+#
+# It CAN still go red, for one reason that is not a finding: the arm inspected
+# NOTHING. An unresolvable pull request, a `gh` it could not read, an empty
+# check roster, or an empty `evidence_bearing_checks` all print the same
+# "nothing vacuous here" as a genuinely clean run, so the guard refuses instead
+# and names which one it hit. Read the `vacuity_*` token before reporting a red
+# run here as a hollow review: `NOBODY LOOKED` and `THE REVIEW WAS FAKE` are
+# opposite facts.
+
+name: 🕵️ Review Evidence
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review]
+  workflow_dispatch:
+    inputs:
+      pr:
+        description: 'Pull request number to inspect'
+        required: true
+        type: string
+
+# `actions: read` is what `gh pr checks` needs — it resolves the rollup through
+# `checkSuite.workflowRun`, and without it `gh` exits non-zero with EMPTY
+# STDOUT, a failure shaped like a content problem that never says the word
+# "permission". `checks: read` + `statuses: read` are what the guard's
+# permission-light fallback needs; a legacy commit status like CodeRabbit's is
+# the only route that carries a description at all.
+permissions:
+  contents: read
+  actions: read
+  checks: read
+  statuses: read
+  pull-requests: read
+
+concurrency:
+  group: review-evidence-${{ github.event.pull_request.number || inputs.pr || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  vacuity:
+    name: 🕵️ Did the required review checks do any work?
+    runs-on: ubuntu-latest
+    # The guard waits for the declared checks to settle before judging them: a
+    # review bot posts `pending — "Review queued"` and `pending — "Review in
+    # progress"` first, both of which are in the no-work vocabulary, so judging
+    # them on arrival manufactures a finding on EVERY pull request. Its default
+    # ceiling is five minutes; this leaves room for the wait plus the run.
+    timeout-minutes: 10
+
+    steps:
+      - name: 📥 Checkout repository
+        uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+
+      - name: 🔧 Setup Node.js
+        uses: actions/setup-node@v6
+        with:
+          package-manager-cache: false
+          node-version: '22.21.1'
+
+      - name: 🕵️ Refuse a required check that reviewed nothing
+        env:
+          GH_TOKEN: ${{ github.token }}
+          # Absent on a `pull_request` run, where the guard reads the event
+          # payload instead; set only for the manual dispatch path.
+          VACUITY_PR: ${{ inputs.pr }}
+        run: |
+          set -eu
+          # An absent input is a FAILURE, never a skip — the same flip #2933
+          # made to the `🔒 Skipped Required Checks` job in quality.yml, for the
+          # same reason: this workflow is installed by the same `lisa apply`
+          # that installs the prover and the seed, so neither can be absent
+          # without the install being broken, and a green run that inspected
+          # nothing is the exact defect being checked for.
+          if [ ! -f scripts/check-skipped-required-checks.mjs ]; then
+            echo "::error title=Vacuity prover missing::scripts/check-skipped-required-checks.mjs does not exist, so no required check was read on this pull request. Run \`lisa apply\` to install the Lisa-managed script."
+            exit 1
+          fi
+          if [ ! -f .github/required-checks.json ]; then
+            echo "::error title=Required-checks declaration missing::.github/required-checks.json does not exist, so nothing names which checks are supposed to carry review evidence. It is create-only and Lisa ships a seed — run \`lisa apply\` to get it."
+            exit 1
+          fi
+          if [ -n "${VACUITY_PR:-}" ]; then
+            node scripts/check-skipped-required-checks.mjs --vacuity --pr="$VACUITY_PR"
+          else
+            node scripts/check-skipped-required-checks.mjs --vacuity
+          fi

package/typescript/package-lisa/package.lisa.json

@@ -22,7 +22,7 @@
       "nightly:health": "node scripts/check-nightly-e2e-health.mjs",
       "check:skipped-required-checks": "node scripts/check-skipped-required-checks.mjs",
       "check:skipped-required-checks:remote": "node scripts/check-skipped-required-checks.mjs --remote",
-      "check:vacuous-required-checks": "node scripts/check-skipped-required-checks.mjs",
+      "check:vacuous-required-checks": "node scripts/check-skipped-required-checks.mjs --vacuity",
       "test:node": "node scripts/lisa-test-node.mjs",
       "check:work-item": "node scripts/lisa-work-item.mjs validate-pr",
       "check:conflict-markers": "node scripts/check-conflict-markers.mjs",