staysfixed 0.14.0 → 0.15.0

package/CHANGELOG.md

@@ -6,6 +6,38 @@ numbers follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.15.0] — 2026-09-02
+
+### A guard can say it cannot be answered here
+
+Fifty guards were written for a real product in one night and put through a real run. Several
+failed on that machine for reasons that had nothing to do with the product: one wanted a paired
+second machine, one an Android phone, several a session in a state nothing had put it in.
+Nothing was broken. They were being asked somewhere that could not answer them.
+
+Until now such a guard had two ways out and both were wrong. **Passing is a lie** — it reports
+that a bug did not come back, having looked for nothing, which is the false all-clear this whole
+tool exists to prevent, wearing the friendliest face it has. **Failing is a false alarm**, and
+worse than it sounds: the line printed over a failed guard is "bugs that were already fixed are
+back", so somebody hunts a regression that never happened, and after twice they stop believing
+any of them.
+
+- `cannotRunHere(why)` on a guard's run context stops it and reports **not proved**, with the
+  reason and what would let it run. Counted as neither a pass nor a failure, and excluded from
+  what the run says it looked at.
+- It **cannot** be used after an expectation has already failed. A door out of a red run is a
+  door somebody would eventually walk through.
+- It is for what the MACHINE is missing, never for what the product is doing — using it because
+  the product looks wrong would turn a real finding into silence.
+- `not proved` is its own verdict everywhere it is printed, distinct from `left out on purpose`,
+  which means somebody deliberately switched a guard off. The story of the original bug is
+  deliberately not printed underneath it: under a guard that never asked its question, "why this
+  guard exists" reads as that bug being back.
+
+On the run that prompted it, 56 guards came back 40 held, 11 not proved, 5 failed — and all five
+failures were real.
+
+
 ## [0.14.0] — 2026-09-02
 
 Four defects, every one of them found by WATCHING this tool work rather than by reading it.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "staysfixed",
-  "version": "0.14.0",
+  "version": "0.15.0",
   "description": "Prove that what already worked still works after an agent changed the code. Picture checks, guards for fixed bugs, a pre-release walkthrough, and known-good markers \u2014 as a CLI and as an MCP server.",
   "type": "module",
   "license": "MIT",

package/src/guard/api.js

@@ -32,6 +32,34 @@ export class ExpectationFailed extends Error {
   }
 }
 
+/**
+ * This guard cannot answer its question on this machine, and that is not a verdict.
+ *
+ * Added 2026-09-02, after fifty guards were written for a real product in one night and four
+ * of them failed for reasons that had nothing to do with the product: one wanted a paired
+ * machine, one an Android phone, two a session in a particular state. Nothing was broken. The
+ * guards were simply being run somewhere that could not answer them.
+ *
+ * Until this existed a guard in that position had two ways out and both were wrong. Passing is
+ * a lie — it reports that a bug did not come back, having checked nothing. Failing is a false
+ * alarm, and worse than it sounds: the line a run prints for a failed guard is "bugs that were
+ * already fixed are back", so somebody goes hunting a regression that never happened, and after
+ * it happens twice they stop believing any of them.
+ *
+ * "Not proved" is a third answer and it is the honest one. It is the same distance from a pass
+ * as "nothing was compared" is from "nothing changed", which is the distinction this whole tool
+ * is built on, and it had a hole in it exactly here.
+ */
+export class GuardCannotRunHere extends Error {
+  /** @param {string} why  Plain English: what is missing, and what would let it run. */
+  constructor(why) {
+    super(why);
+    this.name = 'GuardCannotRunHere';
+    /** @type {string} */
+    this.why = why;
+  }
+}
+
 /**
  * The run has given up on this guard, and the guard is still going.
  *
@@ -199,6 +227,21 @@ export function makeGuardApi(page, project, opts = {}) {
     page: refusable(page),
     project,
 
+    /**
+     * Say this guard cannot be answered here, and stop.
+     *
+     * Use it for what the machine is missing, never for what the product is doing: no phone
+     * attached, no second machine paired, no signed-in account to switch between. A guard that
+     * reaches for this because the product looks wrong has turned a real finding into silence.
+     *
+     * @param {string} why  What is missing, and what would let it run.
+     * @returns {never}
+     */
+    cannotRunHere(why) {
+      announce(ACTION, `cannot run here: ${short(String(why))}`)('ok');
+      throw new GuardCannotRunHere(String(why));
+    },
+
     /**
      * @param {string} to
      * @returns {Promise<void>}

package/src/guard/run.js

@@ -16,7 +16,7 @@
  * this one is not" is most of the value of running it at all.
  */
 
-import { makeGuardApi, ExpectationFailed, GuardAbandoned } from './api.js';
+import { makeGuardApi, ExpectationFailed, GuardAbandoned, GuardCannotRunHere } from './api.js';
 import { resetWindow } from '../drive/launch.js';
 import { emitEvent } from '../core/events.js';
 
@@ -29,6 +29,7 @@ const FRESH_KEY = 'fresh';
  * @typedef {import('../types.js').GuardResult & {
  *   retriedToPass?: boolean,
  *   assertedNothing?: boolean,
+ *   cannotRunHere?: boolean,
  *   timedOut?: boolean,
  *   checks?: import('../types.js').CheckStep[],
  * }} GuardRunResult
@@ -41,6 +42,9 @@ const FRESH_KEY = 'fresh';
  * @property {string} [failedAt]
  * @property {boolean} [timedOut]  The clock ran out before the guard answered. Not the same
  *                                 as the answer being no — see the note on the result below.
+ * @property {boolean} [cannotRunHere]  The guard said this machine cannot answer it and stopped.
+ *                                 `ok` is true because nothing went wrong; the flag is what stops
+ *                                 it being counted as a bug that did not come back.
  */
 
 /** @typedef {(step: import('../types.js').CheckStep) => void} StepSink */
@@ -150,12 +154,16 @@ export async function runGuards(project, app, guards, opts = {}) {
     // Its OWN questions, not the runner's. Every guard gets a "fresh start" step from this
     // file whether it asks anything or not, so counting the whole list would always find one.
     const asked = checks.filter((c) => c.key !== FRESH_KEY && !String(c.key ?? '').endsWith(`-${FRESH_KEY}`));
-    const assertedNothing = outcome.ok && asked.length === 0;
+    // A guard that stopped because this machine cannot answer it did not "check nothing" in the
+    // sense the rule below is about — it deliberately declined to claim anything, which is the
+    // opposite failure and has to be told apart from it.
+    const couldNotRun = outcome.cannotRunHere === true;
+    const assertedNothing = outcome.ok && asked.length === 0 && !couldNotRun;
 
     /** @type {GuardRunResult} */
     const result = {
       name: guard.name,
-      status: outcome.ok && !assertedNothing ? 'passed' : 'failed',
+      status: couldNotRun ? 'skipped' : outcome.ok && !assertedNothing ? 'passed' : 'failed',
       file: guard.file,
       because: guard.because,
       durationMs: Date.now() - startedAt,
@@ -163,7 +171,13 @@ export async function runGuards(project, app, guards, opts = {}) {
     };
     if (checks.length > 0) result.checks = checks;
 
-    if (assertedNothing) {
+    if (couldNotRun) {
+      // Neither a pass nor a failure, and it must not be summarised as either. The message
+      // carries what is missing so somebody reading a run can tell at a glance whether it is
+      // worth plugging a phone in, or whether this guard will never run on a build machine.
+      result.cannotRunHere = true;
+      result.message = outcome.message ?? 'This guard could not be answered on this machine.';
+    } else if (assertedNothing) {
       result.assertedNothing = true;
       // The story of the bug is not repeated in here. It travels on `because`, and the console,
       // the HTML report, the MCP answer and the live panel each print it themselves — so
@@ -368,6 +382,22 @@ async function attemptGuard(project, app, guard, baseUrl, timeoutMs, onStep) {
         message: `This should still be true, and it is not: "${error.claim}".${consoleNote(app)}`,
       };
     }
+    // NOT PROVED, which is neither of the two answers above it.
+    //
+    // The guard asked for something this machine does not have — a paired machine, a phone, a
+    // second signed-in account — and stopped. Nothing about the product was learned, so nothing
+    // about the product may be reported. Passing here would be the false all-clear this whole
+    // tool exists to prevent, and failing would say "a bug that was already fixed is back" about
+    // a bug nobody looked for.
+    if (error instanceof GuardCannotRunHere) {
+      return {
+        ok: true,
+        cannotRunHere: true,
+        message:
+          `Not proved here: ${error.why} Nothing about this bug was checked on this run, so nothing here ` +
+          `says it has not come back.`,
+      };
+    }
     // Out of time is its own answer, and it is not "no". The guard was still going when the
     // clock stopped, so all anyone knows is that nobody asked it anything it managed to
     // finish. Said in those words rather than as a returned bug.

package/src/report/console.js

@@ -127,17 +127,25 @@ function shorten(s, max) {
  *  - `unanswered` — nobody got an answer: it ran out of time, or it asserted nothing at all.
  *                   Not a pass, and not a returned bug either.
  *  - `left out`   — marked skip. Never ran.
+ *  - `not proved` — the guard asked for something this machine has not got and declined: no
+ *                   phone attached, no machine paired, no server. Distinct from `left out`,
+ *                   which is somebody deliberately switching a guard off. Somebody reading
+ *                   "left out on purpose" against a guard that WANTED to run and could not
+ *                   would go looking for the person who disabled it.
  *  - `held`       — asked, and the answer was yes.
  *
  * Anything unrecognised counts as `unanswered`, never as `held`: the one thing that must
  * never happen here is a result nobody understood being read as a clean bill of health.
  *
  * @param {import('../types.js').GuardResult} guard
- * @returns {'held'|'back'|'unanswered'|'left out'}
+ * @returns {'held'|'back'|'unanswered'|'left out'|'not proved'}
  */
 export function guardVerdict(guard) {
   const g = /** @type {any} */ (guard ?? {});
   if (g.status === 'passed') return 'held';
+  // Read before the plain 'skipped' below it: both wear that status and they mean opposite
+  // things about whether anybody wanted this guard to run.
+  if (g.cannotRunHere === true) return 'not proved';
   if (g.status === 'skipped') return 'left out';
   if (g.timedOut === true || g.assertedNothing === true) return 'unanswered';
   if (g.status === 'failed') return 'back';
@@ -162,6 +170,10 @@ function guardOutcome(g) {
       return 'still holds';
     case 'left out':
       return 'left out on purpose';
+    case 'not proved':
+      // The reason, not a fixed phrase: what is missing is the only thing worth reading here,
+      // and it is the difference between "plug a phone in" and "this will never run on CI".
+      return shorten(String(g.message || 'could not be answered on this machine'), 90);
     case 'unanswered':
       if (any.timedOut === true) return 'ran out of time — nothing was proved either way';
       if (any.assertedNothing === true) return 'checks nothing, so it is protecting nothing';
@@ -387,6 +399,16 @@ export function printGuardResult(r) {
     say(`${paint.grey(sym(mark.info))} ${paint.grey(`${name} left out on purpose`)}`);
     return;
   }
+  // NOT PROVED gets its own line, and deliberately NOT the story of the bug underneath it.
+  // The story is printed to say whether a failure matters; under a guard that never asked its
+  // question, "why this guard exists: long messages used to vanish" reads as that bug being
+  // back — which is exactly the impression this whole file exists to prevent. What belongs
+  // here is what is missing, so somebody can decide whether to go and plug it in.
+  if (verdict === 'not proved') {
+    say(`${paint.yellow(sym(mark.warn))} ${paint.yellow(`${name} ${r.message || 'could not be answered on this machine'}`)} ${time}`);
+    if (r.file) detail(`    ${shortPath(r.file)}`);
+    return;
+  }
   // A question nobody answered is not painted like a bug coming back. It still keeps the run
   // out of the green — `allClear` counts it — but the colour a person scans for should not
   // say "regression" about something the run has no opinion on.

package/src/types.js

@@ -273,6 +273,9 @@
 
 /**
  * @typedef {object} GuardApi
+ * @property {(why: string) => never} cannotRunHere               Stop, saying this machine cannot answer
+ *                                                                this guard. For what the MACHINE is
+ *                                                                missing, never for what the product does.
  * @property {PageApi} page                                       Full page control.
  * @property {(path: string) => Promise<void>} open               Shorthand for page.goto.
  * @property {(selector: string) => Promise<void>} click
@@ -506,6 +509,11 @@ export {};
  *                                          as a pass — but it is not a bug coming back, and anything
  *                                          drawing this stream has to be able to tell the two apart.
  * @property {boolean} [assertedNothing]      A guard that finished without asking a single question.
+ * @property {boolean} [cannotRunHere]       A guard that stopped because this machine cannot answer it —
+ *                                          no phone attached, no second machine paired, no second account
+ *                                          to switch between. Reported with the status 'skipped': it is
+ *                                          not a pass, because nothing about the bug was checked, and it
+ *                                          is not a failure, because nothing about the product went wrong.
  * @property {string} [thumbnail]             A small JPEG as a data: URI — an instant preview, shown
  *                                          while the real file is still being written.
  * @property {string} [shotFile]              file:// URL of the FULL-RESOLUTION picture just taken.