@codyswann/lisa 4.4.15 → 4.4.17

package/plugins/src/base/rules/reference/falsifiable-checks.md

@@ -10,7 +10,7 @@ It is the more dangerous failure of the two, because the first leaves you uncert
 
 The empirical origin: a single defect-sweep run produced four false-passing checks and zero false fixes. Every fix was correct; every one of the four *instruments* was broken. Reviewers then found three real defects the checks had cleared. The error concentrated entirely in verification, which is why the countermeasure belongs at the check level rather than the code level.
 
-## The four failure modes, in detail
+## The five failure modes, in detail
 
 ### 1. Self-matching guard
 
@@ -51,6 +51,79 @@ Observed: an uncalled-method detector returned 0 hits across 4,656 files, run on
 
 Countermeasure: every detector reporting a zero must first be shown to find known instances on a ref where they exist (`origin/<base>`, the pre-fix commit, or a synthetic fixture).
 
+### 5. Pager-shadowed status
+
+The check is correct, it ran, and it failed. The *reading* of it is what reported green.
+
+A shell pipeline's exit status is the status of its **last** stage. So this reports on `tail`, not on the gate:
+
+```text
+$ node scripts/plugin-parity-drift.mjs 2>&1 | tail -4; echo "exit=$?"
+| ... | safety-net@cc-marketplace | 1.0.6 | 2.0.4 | stale |
+1 of 5 synced skills drifted
+exit=0                      <-- tail's exit code
+
+$ node scripts/plugin-parity-drift.mjs > drift.log 2>&1; echo "REAL exit code: $?"
+REAL exit code: 1
+```
+
+This one is worth stating separately from the other four because of *where* it occurs. The other four are defects in a check. This is a defect in **how a correct check gets read**, and the reading happens at exactly the moment someone is deciding whether they are blocked. It is also self-similar in an unhelpful way: it defeats verification while looking like verification.
+
+The pipe is there to keep the output short — which means the failure line is the line most likely to fall outside the window. In the transcript above the tool happened to print its verdict inside `tail -4`, so a careful reader could catch it. That is luck about one chatty tool. A quiet gate that fails with no output produces `exit=0` and an empty, plausible-looking tail, and gets recorded as green.
+
+#### The one spelling to use
+
+Capture the status first; shorten afterwards.
+
+```sh
+status=0
+gate >gate.log 2>&1 || status=$?
+tail -n 20 gate.log
+[ "$status" -eq 0 ] || echo "FAILED ($status) — full output in gate.log"
+```
+
+The status capture must be the very next thing after the command: `$?` holds only the most recent status, and even `echo` overwrites it.
+
+##### `|| status=$?`, never `; status=$?`
+
+The `;` spelling is the same class of defect one level down — a status read that cannot report a failure — reached by a different route. Under `set -e`, errexit terminates the shell on the failing command, so the assignment never runs and neither does the branch that would print `FAILED`. The bug is invisible while the gate passes and appears only in the case the code exists to handle. Measured:
+
+```text
+sh -e -c 'status=0; false >/dev/null 2>&1; status=$?; echo "reported=$status"' -> prints nothing, exit 1
+sh -e -c 'status=0; false >/dev/null 2>&1 || status=$?; echo "reported=$status"' -> prints reported=1, exit 0
+```
+
+The left-hand side of `||` is exempt from errexit, which is why the `||` form behaves identically with and without `set -e`. `status=0` is required for the same reason the branch is: on success nothing assigns it, and an unset `status` under `set -u` aborts at the comparison.
+
+##### Return the status when the block's own status is read
+
+Inside a hook, a CI step, or anything invoked with `&&`, add `exit "$status"` (`return "$status"` in a function). Without it the last command executed is the `echo`, so the block exits 0 immediately after printing `FAILED` — pager-shadowing displaced by one level, with the trailing `echo` in the role of `tail`.
+
+#### Why this one and not the other two
+
+| Spelling | Works in | Does not work in | Keeps full output |
+|---|---|---|---|
+| redirect, then read `$?` | every POSIX shell, `sh` included | — | yes |
+| `set -o pipefail` | bash, zsh, ksh, POSIX Issue 8 shells | **not POSIX `sh`** — dash has no such option, and GitHub Actions' default `run:` shell is `bash -e {0}`, with `-e` but no pipefail | no |
+| `${PIPESTATUS[0]}` | bash | zsh spells it `$pipestatus[1]` and indexes from 1; absent from `sh`. Overwritten by the next command, so it must be read immediately | no |
+
+The redirect form wins on both columns that matter here. It is the only one that survives `sh` — this repository runs hooks under `sh` — and it is the only one that still has the untruncated output at the moment the status tells you to go look at it. `pipefail` is the right thing to add to a *script* you control; it is not a substitute for this when you are reading a gate at a prompt.
+
+#### It is not hypothetical, and not only about `tail`
+
+`security-floors.yml` ran `node scripts/check-security-floors.mjs --strict | tee -a "$GITHUB_STEP_SUMMARY"` with no `pipefail`. Every failure `--strict` exists to raise — a dependency floor below a live advisory, a rate-limited inconclusive run, an unresolved `$name` — was discarded, and the job reported green. Measured:
+
+```text
+bash -e             -c 'node -e "process.exit(1)" | tee /dev/null' -> 0
+bash -e -o pipefail -c 'node -e "process.exit(1)" | tee /dev/null' -> 1
+```
+
+`tee` is the same mechanism as `head`/`tail`: a last stage that reports on its own writing rather than on the command upstream.
+
+#### The executable half
+
+`scripts/check-pipeline-status-reads.mjs` sweeps shipped shell scripts and workflow `run:` blocks for the pattern. It reports **how many pipelines it inspected** and exits 2 — not 0 — when that count is zero, because an empty inspection and a clean tree otherwise print the same tick.
+
 ## What a negative result is scoped to
 
 A clean result is a statement about what the check can perceive, not about the code. State the boundary:

package/plugins/src/base/skills/lisa-improve-test-coverage/SKILL.md

@@ -16,8 +16,11 @@ If no argument provided, prompt the user for a target.
 1. **Find coverage config** (jest.config.js, vitest.config.ts, .nycrc, etc.)
 2. **Run coverage report** to get current state:
    ```bash
-   bun run test:cov 2>&1 | head -100
+   status=0
+   bun run test:cov >coverage.log 2>&1 || status=$?
+   head -n 100 coverage.log; echo "exit=$status"
    ```
+   The status is captured before the pipe, because a pipeline reports its LAST stage's exit code — `head` always succeeds, so a failing run reads as `exit=0` (`falsifiable-checks`, pager-shadowed status). `|| status=$?` rather than `; status=$?`: under `set -e` the `;` form exits before the assignment, so the failure is never reported at all.
 3. **Identify the 20 files with lowest coverage**, noting:
    - File path
    - Current coverage % (lines, branches, functions)

package/plugins/src/base/skills/lisa-improve-tests/SKILL.md

@@ -14,8 +14,11 @@ If no argument provided, scan the full test suite.
 
 1. **Run test suite** to establish baseline:
    ```bash
-   bun run test 2>&1 | tail -20
+   status=0
+   bun run test >test.log 2>&1 || status=$?
+   tail -n 20 test.log; echo "exit=$status"
    ```
+   The status is captured before the pipe, because a pipeline reports its LAST stage's exit code — `tail` always succeeds, so a failing run reads as `exit=0` (`falsifiable-checks`, pager-shadowed status). `|| status=$?` rather than `; status=$?`: under `set -e` the `;` form exits before the assignment, so the failure is never reported at all.
 2. **Scan test files** for quality issues:
    - Weak assertions (`toBeTruthy`, `toBeDefined` instead of specific values)
    - Missing edge cases (no boundary values, no error paths)

package/plugins/src/base/skills/lisa-nightly-lower-code-complexity/SKILL.md

@@ -20,9 +20,17 @@ The caller provides pre-computed context:
 4. Fix violations one file at a time. Read only the specific function that violates — do not pre-read all files upfront. Fix it, then move to the next.
 5. For cognitive complexity violations: use early returns, extract helper functions, replace conditionals with lookup tables
 6. For max-lines-per-function violations: split large functions, extract helper functions, separate concerns
-7. After each file edit, run the project's formatter (e.g., `bun run format` or `npx prettier --write <file>`) to ensure line counts reflect the final formatted state before moving on
+7. After each file edit, run the project's formatter **with the provided package manager** (e.g., `npm run format`, `yarn format`, or `bun run format`) to ensure line counts reflect the final formatted state before moving on. Do not reach for a bare `npx prettier`: when the binary is absent `npx` silently installs and executes whatever the registry currently publishes under that name, which is an unpinned dependency introduced by a formatting step. If the project has no format script, run the lockfile-pinned binary directly (`./node_modules/.bin/prettier --write <file>`)
 8. Re-run the lint script with the provided package manager to verify all violations are resolved (both the target metric AND max-lines)
-9. Run the TypeScript compiler to catch type errors early: `npx tsc --noEmit 2>&1 | head -30`. If there are type errors, fix them now — do NOT wait until the commit step. Pre-commit hooks run type checking, and discovering errors at commit time wastes turns.
+9. Run the project's typecheck script with the provided package manager to catch type errors early — same reasoning as step 7, so `npm run typecheck` / `yarn typecheck` / `bun run typecheck`, falling back to `./node_modules/.bin/tsc --noEmit` rather than `npx tsc`:
+
+   ```sh
+   status=0
+   bun run typecheck >tsc.log 2>&1 || status=$?
+   head -n 30 tsc.log; echo "exit=$status"
+   ```
+
+   Capture the status before the pipe — a pipeline reports its LAST stage's exit code, and `head` always succeeds, so `tsc --noEmit | head -30` reads as clean however many errors it printed (`falsifiable-checks`, pager-shadowed status). `|| status=$?` rather than `; status=$?`: under `set -e` the `;` form exits before the assignment, so the failure is never reported at all. If there are type errors, fix them now — do NOT wait until the commit step. Pre-commit hooks run type checking, and discovering errors at commit time wastes turns.
 10. Run the project's test script with the provided package manager (e.g., `npm run test`, `yarn test`, or `bun run test`) to verify no tests are broken by the refactoring
 11. Commit all changes (refactored code + updated eslint.thresholds.json) with conventional commit messages
 12. Create a PR with `gh pr create` with a title like "refactor: reduce code complexity: [metrics being reduced]" summarizing the changes

package/plugins/src/rails/skills/improve-code-complexity/SKILL.md

@@ -17,8 +17,11 @@ Reduces the CyclomaticComplexity threshold by 2 and fixes all violations.
    bundle exec rubocop --only Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity --format json 2>&1
    ```
    ```bash
-   bundle exec flog --all --group app/ 2>&1 | head -50
+   status=0
+   bundle exec flog --all --group app/ >flog.log 2>&1 || status=$?
+   head -n 50 flog.log; echo "exit=$status"
    ```
+   The status is captured before the pipe, because a pipeline reports its LAST stage's exit code — `head` always succeeds, so a failing run reads as `exit=0` (`falsifiable-checks`, pager-shadowed status). `|| status=$?` rather than `; status=$?`: under `set -e` the `;` form exits before the assignment, so the failure is never reported at all.
 4. **Note for each violation**:
    - File path and line number
    - Method name

package/plugins/src/rails/skills/improve-test-coverage/SKILL.md

@@ -16,8 +16,11 @@ If no argument provided, prompt the user for a target.
 1. **Find coverage config** (`.simplecov` or `spec/spec_helper.rb`)
 2. **Run test suite with coverage** to get current state:
    ```bash
-   bundle exec rspec 2>&1 | tail -50
+   status=0
+   bundle exec rspec >rspec.log 2>&1 || status=$?
+   tail -n 50 rspec.log; echo "exit=$status"
    ```
+   The status is captured before the pipe, because a pipeline reports its LAST stage's exit code — `tail` always succeeds, so a failing run reads as `exit=0` (`falsifiable-checks`, pager-shadowed status). `|| status=$?` rather than `; status=$?`: under `set -e` the `;` form exits before the assignment, so the failure is never reported at all.
 3. **Check SimpleCov output** in `coverage/index.html` or console output
 4. **Identify the 20 files with lowest coverage**, noting:
    - File path

package/plugins/src/rails/skills/ops-verify-telemetry/SKILL.md

@@ -58,7 +58,36 @@ grep -i "otel\|opentelemetry\|trace_id" log/development.log | tail -10
 ### Check Collector Sidecar (if using Docker Compose)
 
 ```bash
-docker compose logs otel-collector 2>/dev/null | tail -20 || echo "No otel-collector service in Docker Compose"
+# Capture the status BEFORE the pipe. A pipeline reports its LAST stage's exit
+# code, so `docker compose logs ... | tail -20 || echo "No otel-collector"`
+# never prints the fallback: `tail` succeeds even when there is no such
+# service, and the absent sidecar reads as a silent pass.
+#
+# `|| status=$?`, not `; status=$?`: under `set -e` the `;` form exits before
+# the assignment, so the branch that reports the failure never runs.
+#
+# Ask whether the service EXISTS separately from whether its logs could be
+# read. `docker compose logs` also exits non-zero for a stopped daemon, a
+# permission failure, and an unparseable compose file; answering all of those
+# with "No otel-collector service" turns a broken telemetry stack into a
+# reassuring sentence — the same silent-measurement failure one level up.
+services_status=0
+services=$(docker compose config --services 2>&1) || services_status=$?
+if [ "$services_status" -ne 0 ]; then
+  echo "FAILED ($services_status) — docker compose config could not be read:"
+  printf '%s\n' "$services"
+elif ! printf '%s\n' "$services" | grep -qx otel-collector; then
+  echo "No otel-collector service in Docker Compose"
+else
+  logs_status=0
+  docker compose logs otel-collector >otel.log 2>&1 || logs_status=$?
+  if [ "$logs_status" -eq 0 ]; then
+    tail -n 20 otel.log
+  else
+    echo "FAILED ($logs_status) — otel-collector is configured but its logs could not be read:"
+    cat otel.log
+  fi
+fi
 ```
 
 ## Remote Verification (AWS X-Ray)

package/rails/copy-overwrite/scripts/threshold-ratchet-compare.mjs

@@ -12,6 +12,7 @@
 import {
   extractAllowEntries,
   extractK6Constraints,
+  extractLighthouseAssertions,
   extractNumericLeaves,
   extractRubocopThresholds,
   extractStrykerConstraints,
@@ -299,6 +300,12 @@ export function compareFile(relPath, baselineText, currentText) {
       return compareStryker(relPath, base, current);
     case "k6":
       return compareK6(relPath, base, current);
+    case "lighthouse":
+      return compareConstraints(
+        relPath,
+        extractLighthouseAssertions(base),
+        extractLighthouseAssertions(current)
+      );
     case KIND_ALLOW_LIST:
       return compareAllowList(relPath, base, current);
     default:

package/rails/copy-overwrite/scripts/threshold-ratchet-families.mjs

@@ -80,8 +80,61 @@ export const FAMILIES = [
     match: /(^|\/)\.lisa\.config\.json$/,
     kind: "allow-list",
   },
+  {
+    id: "lighthouse",
+    match: /(^|\/)lighthouserc-config\.json$/,
+    kind: "lighthouse",
+  },
 ];
 
+/**
+ * Which way each Lighthouse assertion key ratchets.
+ *
+ * Lighthouse mixes floors and ceilings in one file, so assigning one direction
+ * to the whole family would silently approve half of the possible weakenings.
+ * Unknown keys stay unwatched until their direction is explicit here.
+ * @type {Readonly<Record<string, "min"|"max">>}
+ */
+export const LIGHTHOUSE_ASSERTION_DIRECTIONS = {
+  minScore: "min",
+  maxNumericValue: "max",
+  maxLength: "max",
+};
+
+/**
+ * Extract numeric Lighthouse assertions with a direction chosen per key.
+ *
+ * A malformed array is deliberately ignored instead of being walked as an
+ * object whose numeric indices look like audit names.
+ * @param {unknown} conf Parsed lighthouserc-config.json
+ * @returns {Map<string, { value: number, direction: "min"|"max" }>} Dotted
+ *   `assertion.key` path to numeric constraint
+ */
+export function extractLighthouseAssertions(conf) {
+  const out = new Map();
+  const assertions =
+    conf && typeof conf === "object" && !Array.isArray(conf)
+      ? /** @type {Record<string, unknown>} */ (conf).assertions
+      : undefined;
+  if (
+    !assertions ||
+    typeof assertions !== "object" ||
+    Array.isArray(assertions)
+  ) {
+    return out;
+  }
+  for (const [audit, spec] of Object.entries(assertions)) {
+    if (!spec || typeof spec !== "object" || Array.isArray(spec)) continue;
+    for (const [key, value] of Object.entries(spec)) {
+      const direction = LIGHTHOUSE_ASSERTION_DIRECTIONS[key];
+      if (direction && typeof value === "number" && Number.isFinite(value)) {
+        out.set(`${audit}.${key}`, { value, direction });
+      }
+    }
+  }
+  return out;
+}
+
 /**
  * Find the family a repo-relative path belongs to.
  * @param {string} relPath Repo-relative path (forward slashes)