blastproof 0.18.0 → 0.20.0

package/README.md

@@ -86,6 +86,8 @@ The name is matched **exactly first**, then by substring if nothing matches exac
 | `<button aria-label="Delete note">` | an icon-only button with no name |
 | `<select>` with `<option>`s | an ARIA-less custom dropdown |
 
+**An editable data grid is the shape most likely to defeat this, and every row above understates it.** Two adversarial runs against a personal-finance application failed on the journey such an application exists for — entering a transaction. The row's cells carry no accessible name, but the Payee *column header* does, so the tree does not merely lack the target: it offers a control with exactly the right name that is the wrong control. The agent clicks the header, the row editor never opens, and the next `type` lands in whatever input does exist — there, the page's global search box. The step fails describing a textbox nobody mentioned, roughly 70 model calls in. Nobody has yet tried driving such a grid by keyboard (`Tab` into the row, type, `Tab` on), so the honest claim is that **the obvious authoring fails, and fails confusingly** — not that it is impossible. If your core journey is a row editor, spend one test on it before adopting ([#67](https://github.com/hamc/blastproof/issues/67)).
+
 **Run an accessibility checker on your app before installing anything.** The result predicts how well this will work better than anything else you could measure — and the fixes it suggests are worth making regardless of whether you adopt this tool.
 
 ### 2. Does your journey need anything on this list?
@@ -133,7 +135,7 @@ steps:
   - complete checkout
 ```
 
-`priority` is P0–P2 (default P1). `tags`, `setup` steps and `auth` are optional — `auth: false` runs the test signed out, which a login test needs. `routes` declares the URLs a test covers, which is what `--impacted` selects on; write route strings consistently, since they compare by exact equality (`/cart` ≠ `/cart/`). `run` warns to stderr — non-fatal — when a test declares a route no `routes:` mapping declares, since that route contributes nothing to `--impacted` selection.
+`priority` is P0–P2 (default P1). `tags`, `setup` steps and `auth` are optional — `auth: false` runs the test signed out, which a login test needs, and a selection in which *every* test declares it skips the login altogether. `routes` declares the URLs a test covers, which is what `--impacted` selects on; write route strings consistently, since they compare by exact equality (`/cart` ≠ `/cart/`). `run` warns to stderr — non-fatal — when a test declares a route no `routes:` mapping declares, since that route contributes nothing to `--impacted` selection.
 
 ### Say what each step should produce
 
@@ -227,6 +229,10 @@ The key is the file glob and the value is the routes it can affect — the oppos
 
 Every changed file lands in one of three buckets: it matches `routes:` and contributes them, matches `ignore:` and is knowingly irrelevant, or **matches neither — nobody has said what it affects**. `--fail-on-unmapped` blocks on that third case, naming the files and both ways to resolve them.
 
+**The diff is `git diff <base>...HEAD`, so it compares commits.** Work you have not committed — edited, staged or untracked — is not in it. That is the right comparison in CI, where everything is committed by the time the workflow runs, and the wrong one on the machine you are writing the change on: a run against a dirty working tree selects from your last commit, not from your editor.
+
+It says so rather than leaving you to notice. Any working-tree change the diff excluded is named on stderr, with the routes it maps to, on `run --impacted` and `plan --base` alike — non-fatal, and it changes no exit code and nothing about what is selected. Files your `ignore:` globs already cover stay silent, and so do files already in the diff, whose routes are selected either way.
+
 **Nothing is ignored by default**, on purpose: a default that guesses on your behalf would hide the first files worth thinking about. The flag is additive — a run can meet `--min-score` and still be blocked here, because "the tests I ran passed" and "something changed that nobody classified" are different claims.
 
 Its limit is worth knowing: it catches files that are *unclassified*, not *misclassified*. A shared module mapped to one route when it can break five still slips through. Impact by import graph is the fix, and blastproof does not do it yet.
@@ -269,9 +275,11 @@ The application under test is not trusted input: its page content reaches the mo
 
 If your application legitimately spans hosts (an identity provider, a hosted payment step), declare them. A suite that was quietly walking onto a foreign page will now fail and name the origin to add.
 
-**Your secrets stay out of prompts.** `{{env.*}}` placeholders survive intact and are substituted at the moment of typing. Every value your tests or auth recipe reference is redacted from everything else crossing into a prompt — page snapshots included — in literal and percent-encoded form. Redaction matches known values, so treat it as a strong default rather than a guarantee against a hostile app.
+**Your secrets stay out of prompts.** `{{env.*}}` placeholders survive intact and are substituted at the moment of typing. Every value your tests or auth recipe reference is redacted from everything else crossing into a prompt — page snapshots included — in literal and percent-encoded form. Redaction matches a known value case-insensitively and tolerantly of whitespace, so a page that uppercases what you typed is still covered. It does not chase a value your application re-encodes, hashes or truncates — no list of transforms can be complete, and one that pretends to be would make you careless. When a redaction is found only by that wider comparison, the run prints one warning naming the variable, because the same page could just as easily have returned a form nothing here can recognise. Treat all of it as a strong default rather than a guarantee against a hostile app.
+
+**Redaction covers text, not pixels.** A failure screenshot shows whatever was on screen, including a value typed from `{{env.*}}` or echoed back by the page. So when anything in a run references `{{env.*}}`, the HTML report does not embed screenshots: each failure links to its PNG under `.blastproof/reports/` instead. That keeps `report.html` safe to attach to a pull request. The PNGs themselves are not masked, so do not upload `.blastproof/reports/` anywhere you would not put the secret ([#110](https://github.com/hamc/blastproof/issues/110)).
 
-**A redacted value cannot be asserted on.** The masking is thorough by design, and page snapshots are not exempt — so a step that verifies text which happens to equal an `{{env.*}}` value can never pass, because the judge is shown `***` where the page shows the thing. The failure is the most misleading shape available: the test is right, the application is right, and the report blames the application. Put a value in `{{env.*}}` because it is a secret or because it varies by environment, but do not then write a step that asserts on it ([#87](https://github.com/hamc/blastproof/issues/87)).
+**A redacted value cannot be asserted on.** The masking is thorough by design, and page snapshots are not exempt — so a step that verifies text which happens to equal an `{{env.*}}` value can never pass, because the judge is shown `***` where the page shows the thing. The failure is the most misleading shape available: the test is right, the application is right, and the report blames the application. Put a value in `{{env.*}}` because it is a secret or because it varies by environment, but do not then write a step that asserts on it ([#87](https://github.com/hamc/blastproof/issues/87)). Matching is case-insensitive, so this covers a little more of the page than the value's exact spelling — one more reason to keep assertions off it.
 
 The system prompt also tells the model that page content is data, never instruction. That raises the cost of casual injection and is **not** a boundary — the origin constraint is. Do not point blastproof at an application you would not run locally.
 

package/dist/cli.js

@@ -336,16 +336,30 @@ function referencedEnvVars(text) {
 function escapeRegExp(value) {
   return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
 }
-function maskSecrets(text, secrets) {
-  const ordered = [...secrets].filter(Boolean).sort((a, b) => b.length - a.length);
+function secretPattern(secret) {
+  return new RegExp(escapeRegExp(secret).replace(/\s+/g, "\\s+"), "gi");
+}
+function compile(secrets) {
+  return [...secrets].filter(Boolean).sort((a, b) => b.length - a.length).map((value) => ({ value, pattern: secretPattern(value) }));
+}
+function maskCompiled(text, compiled, onNearMiss) {
   let masked = text;
-  for (const secret of ordered) {
-    masked = masked.replace(new RegExp(escapeRegExp(secret), "g"), "***");
+  for (const { value, pattern } of compiled) {
+    pattern.lastIndex = 0;
+    masked = masked.replace(pattern, (found) => {
+      if (found !== value) onNearMiss?.(value, found);
+      return "***";
+    });
   }
   return masked;
 }
 var SecretsMask = class {
   secrets = /* @__PURE__ */ new Set();
+  /** Value -> the `{{env.*}}` variable it came from, so a warning can name it. */
+  names = /* @__PURE__ */ new Map();
+  nearMissed = /* @__PURE__ */ new Set();
+  /** Built on first use after a registration, not on every masked string. */
+  compiled;
   /** Registers the current values of the env vars referenced in `text`. Throws MissingEnvError on unset vars. */
   registerFrom(text, env = process.env) {
     for (const name of referencedEnvVars(text)) {
@@ -353,7 +367,7 @@ var SecretsMask = class {
       if (value === void 0) {
         throw new MissingEnvError(name);
       }
-      this.add(value);
+      this.add(value, name);
     }
   }
   /**
@@ -362,14 +376,39 @@ var SecretsMask = class {
    * space stopped matching a literal search and passed through unmasked.
    * This cannot cover every transform a page might apply; see the README.
    */
-  add(value) {
+  add(value, name) {
     if (!value) return;
     this.secrets.add(value);
+    this.compiled = void 0;
+    if (name !== void 0 && !this.names.has(value)) this.names.set(value, name);
     const encoded = encodeURIComponent(value);
-    if (encoded !== value) this.secrets.add(encoded);
+    if (encoded !== value) {
+      this.secrets.add(encoded);
+      if (name !== void 0 && !this.names.has(encoded)) this.names.set(encoded, name);
+    }
   }
   mask(text) {
-    return maskSecrets(text, this.secrets);
+    this.compiled ??= compile(this.secrets);
+    return maskCompiled(text, this.compiled, (secret) => {
+      const name = this.names.get(secret);
+      if (name !== void 0) this.nearMissed.add(name);
+    });
+  }
+  /**
+   * The `{{env.*}}` variables whose value was redacted in a form differing from
+   * the one supplied — the case a literal comparison would have missed
+   * (design D2). Names only: a caller reporting this must never hold the value.
+   */
+  nearMissedVariables() {
+    return [...this.nearMissed];
+  }
+  /**
+   * Whether no value is registered — i.e. nothing in the run referenced
+   * `{{env.*}}`. Says nothing about which values, so a caller deciding what a
+   * report may carry never holds a secret to decide it.
+   */
+  isEmpty() {
+    return this.secrets.size === 0;
   }
 };
 
@@ -1316,6 +1355,20 @@ async function getChangedFiles(baseRef, cwd = process.cwd()) {
     throw toDiffError(error, baseRef);
   }
 }
+async function getUncommittedFiles(cwd = process.cwd()) {
+  try {
+    const status = await simpleGit(cwd).status();
+    const paths = /* @__PURE__ */ new Set();
+    for (const file of status.files) {
+      paths.add(file.path.replace(/\\/g, "/"));
+      const from = file.from;
+      if (from) paths.add(from.replace(/\\/g, "/"));
+    }
+    return [...paths].sort();
+  } catch {
+    return [];
+  }
+}
 
 // src/impact.ts
 import picomatch from "picomatch";
@@ -1354,6 +1407,25 @@ function mapImpact(changedFiles, routes, ignore = []) {
   };
 }
 
+// src/report/uncommitted.ts
+function printUncommitted(uncommitted, changedFiles, config, baseRef) {
+  const inDiff = new Set(changedFiles.map((file) => file.replace(/\\/g, "/")));
+  const routes = config.routes ?? {};
+  const ignore = config.ignore ?? [];
+  const findings = uncommitted.filter((file) => !inDiff.has(file)).map((file) => ({ file, impact: mapImpact([file], routes, ignore) })).filter(({ impact }) => impact.ignoredFiles.length === 0);
+  if (findings.length === 0) return;
+  console.error(
+    `
+warning: ${findings.length} file(s) changed in the working tree are not in the diff against '${baseRef}':`
+  );
+  for (const { file, impact } of findings) {
+    console.error(
+      impact.affectedRoutes.length > 0 ? `  ${file} -> ${impact.affectedRoutes.join(", ")}` : `  ${file}  (matched by no routes: or ignore: glob)`
+    );
+  }
+  console.error("Nothing above was considered. Commit or stash them to include them.");
+}
+
 // src/llm/brain.ts
 import { generateObject } from "ai";
 
@@ -2038,7 +2110,11 @@ function renderSteps(steps) {
         ${items}
       </ol>`;
 }
-async function renderTest(result, cwd) {
+function withheldScreenshot(file, reportDir) {
+  const href = path7.relative(reportDir, file).split(path7.sep).join("/");
+  return `      <p class="note">Screenshot withheld: this run handled secrets ({{env.*}}), and a screenshot cannot be masked. <a href="${escapeHtml(href)}">${escapeHtml(href)}</a></p>`;
+}
+async function renderTest(result, screenshots, cwd) {
   const failed = result.status === "failed";
   const notRun = result.status === "not-run";
   const tagClass = failed ? "fail" : notRun ? "skip" : "pass";
@@ -2060,7 +2136,9 @@ async function renderTest(result, cwd) {
     parts.push(`      <p class="note">${escapeHtml(result.reason)}</p>`);
   }
   parts.push(`      ${renderSteps(result.steps)}`);
-  if (failed && result.screenshot) {
+  if (failed && result.screenshot && screenshots !== "embed") {
+    parts.push(withheldScreenshot(result.screenshot, screenshots.withheldRelativeTo));
+  } else if (failed && result.screenshot) {
     const embedded = await embedScreenshot(result.screenshot);
     parts.push(
       embedded ? `      <img class="shot" alt="Screenshot at failure" src="${embedded}">` : '      <p class="note">Screenshot unavailable.</p>'
@@ -2075,7 +2153,7 @@ async function renderHtml(results, skipped, meta) {
   const notRun = results.filter((r) => r.status === "not-run");
   const generatedAt = (meta.generatedAt ?? /* @__PURE__ */ new Date()).toISOString();
   const ordered = [...failures, ...notRun, ...results.filter((r) => r.status === "passed")];
-  const detail = (await Promise.all(ordered.map((r) => renderTest(r, meta.cwd)))).join("\n");
+  const detail = (await Promise.all(ordered.map((r) => renderTest(r, meta.screenshots, meta.cwd)))).join("\n");
   const verdict = meta.minScore === void 0 ? "" : meta.score >= meta.minScore ? `<span class="verdict pass">min-score ${meta.minScore}: pass</span>` : `<span class="verdict fail">min-score ${meta.minScore}: FAIL</span>`;
   const banner = meta.incomplete === void 0 ? "" : `  <section class="incomplete">Run stopped: ${escapeHtml(meta.incomplete)}</section>
 
@@ -2119,7 +2197,7 @@ ${rows}${rows && skippedRows ? "\n" : ""}${skippedRows}
 
 ${detail}
 
-  <footer>Generated by blastproof. Screenshots are embedded, so this file works offline.</footer>
+  <footer>Generated by blastproof. ${meta.screenshots === "embed" ? "Screenshots are embedded, so this file works offline." : "Screenshots were withheld because this run handled secrets ({{env.*}}); failures link to the file instead."}</footer>
 </main>
 </body>
 </html>
@@ -2489,8 +2567,16 @@ function printImpactReport(impact, selection, cwd) {
   }
   console.log("---------------------------------------------------------------");
 }
-async function finalize(results, skipped, options, sessionDir, durationMs, impact, incomplete, spend) {
+function reportNearMissedSecrets(variables) {
+  for (const variable of variables) {
+    console.error(
+      `warning: the application returned ${variable} in a different form than the value supplied (case or spacing). It was redacted, but a form this tool cannot recognise \u2014 an encoding, a hash \u2014 would not have been. Check what that page does with the value.`
+    );
+  }
+}
+async function finalize(results, skipped, options, sessionDir, secretsHeld, nearMissedSecrets, durationMs, impact, incomplete, spend) {
   if (results.length > 0) printSummary(results);
+  reportNearMissedSecrets(nearMissedSecrets);
   if (spend) console.log(formatSpendLine(spend));
   const score = computeScore(results);
   console.log(
@@ -2516,7 +2602,13 @@ async function finalize(results, skipped, options, sessionDir, durationMs, impac
       minScore: options.minScore,
       cwd: options.cwd,
       incomplete: incomplete?.message,
-      spend
+      spend,
+      // A screenshot cannot be masked, so a run that held any `{{env.*}}` value
+      // links to it instead of carrying it (design
+      // withhold-a-screenshot-that-saw-a-secret, D1): the run's mask, not the
+      // failed test's own placeholders, because a login credential is echoed by
+      // the pages of every test that comes after it.
+      screenshots: secretsHeld ? { withheldRelativeTo: path9.dirname(target) } : "embed"
     });
     await writeHtml(target, html);
     console.log(`HTML report: ${path9.relative(options.cwd, target)}`);
@@ -2574,7 +2666,8 @@ Dry run: ${selected.length} test(s) selected, base_url=${config.base_url}`);
   for (const test of selected) {
     console.log(`  ${test.summary} [${test.priority}] (${path9.relative(cwd, test.path)})`);
   }
-  const authSteps = config.auth?.steps ?? [];
+  const needsLogin = selected.some((test) => test.auth);
+  const authSteps = needsLogin ? config.auth?.steps ?? [] : [];
   const withAuth = authSteps.length > 0 ? [...selected, { steps: authSteps }] : selected;
   const ceiling = estimateMaxModelCalls(
     withAuth,
@@ -2619,6 +2712,7 @@ async function runCommand(options) {
       throw error;
     }
     impact = mapImpact(changedFiles, config.routes ?? {}, config.ignore ?? []);
+    printUncommitted(await getUncommittedFiles(options.cwd), changedFiles, config, base);
   }
   const testsDir = path9.join(options.cwd, TESTS_RELATIVE_DIR);
   let files;
@@ -2685,7 +2779,7 @@ ${results.length} test file(s) could not be parsed:`);
     console.log(
       impact ? "No impacted tests to run." : "No tests matched the given filters."
     );
-    return finalize(results, selection.unroutedSkipped, options, sessionDir, Date.now() - startedAt, impact);
+    return finalize(results, selection.unroutedSkipped, options, sessionDir, false, [], Date.now() - startedAt, impact);
   }
   try {
     createModel(config.llm);
@@ -2711,7 +2805,7 @@ ${results.length} test file(s) could not be parsed:`);
   let incomplete;
   try {
     let session;
-    if (config.auth) {
+    if (config.auth && selected.some((test) => test.auth)) {
       try {
         console.log("Authenticating...");
         session = await authenticate({
@@ -2783,6 +2877,8 @@ ${results.length} test file(s) could not be parsed:`);
     selection.unroutedSkipped,
     options,
     sessionDir,
+    !runMask.isEmpty(),
+    runMask.nearMissedVariables(),
     Date.now() - startedAt,
     impact,
     incomplete,
@@ -2832,6 +2928,7 @@ async function resolveTargets(options, config, suite) {
   const base = options.base ?? "main";
   const changedFiles = await getChangedFiles(base, options.cwd);
   const impact = mapImpact(changedFiles, config.routes ?? {}, config.ignore ?? []);
+  printUncommitted(await getUncommittedFiles(options.cwd), changedFiles, config, base);
   const filesForRoute = (route) => changedFiles.filter(
     (file) => mapImpact([file], config.routes ?? {}, config.ignore ?? []).affectedRoutes.includes(route)
   );
@@ -3109,7 +3206,7 @@ function parsePositiveNumber(flag) {
   };
 }
 var program = new Command();
-program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.18.0");
+program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.20.0");
 program.command("init").description("Scaffold .blastproof/ (config, tests, sample tests) in the current directory").action(async () => {
   try {
     const result = await initProject(process.cwd());