blastproof 0.19.0 → 0.20.0

package/README.md

@@ -86,6 +86,8 @@ The name is matched **exactly first**, then by substring if nothing matches exac
 | `<button aria-label="Delete note">` | an icon-only button with no name |
 | `<select>` with `<option>`s | an ARIA-less custom dropdown |
 
+**An editable data grid is the shape most likely to defeat this, and every row above understates it.** Two adversarial runs against a personal-finance application failed on the journey such an application exists for — entering a transaction. The row's cells carry no accessible name, but the Payee *column header* does, so the tree does not merely lack the target: it offers a control with exactly the right name that is the wrong control. The agent clicks the header, the row editor never opens, and the next `type` lands in whatever input does exist — there, the page's global search box. The step fails describing a textbox nobody mentioned, roughly 70 model calls in. Nobody has yet tried driving such a grid by keyboard (`Tab` into the row, type, `Tab` on), so the honest claim is that **the obvious authoring fails, and fails confusingly** — not that it is impossible. If your core journey is a row editor, spend one test on it before adopting ([#67](https://github.com/hamc/blastproof/issues/67)).
+
 **Run an accessibility checker on your app before installing anything.** The result predicts how well this will work better than anything else you could measure — and the fixes it suggests are worth making regardless of whether you adopt this tool.
 
 ### 2. Does your journey need anything on this list?
@@ -273,9 +275,11 @@ The application under test is not trusted input: its page content reaches the mo
 
 If your application legitimately spans hosts (an identity provider, a hosted payment step), declare them. A suite that was quietly walking onto a foreign page will now fail and name the origin to add.
 
-**Your secrets stay out of prompts.** `{{env.*}}` placeholders survive intact and are substituted at the moment of typing. Every value your tests or auth recipe reference is redacted from everything else crossing into a prompt — page snapshots included — in literal and percent-encoded form. Redaction matches known values, so treat it as a strong default rather than a guarantee against a hostile app.
+**Your secrets stay out of prompts.** `{{env.*}}` placeholders survive intact and are substituted at the moment of typing. Every value your tests or auth recipe reference is redacted from everything else crossing into a prompt — page snapshots included — in literal and percent-encoded form. Redaction matches a known value case-insensitively and tolerantly of whitespace, so a page that uppercases what you typed is still covered. It does not chase a value your application re-encodes, hashes or truncates — no list of transforms can be complete, and one that pretends to be would make you careless. When a redaction is found only by that wider comparison, the run prints one warning naming the variable, because the same page could just as easily have returned a form nothing here can recognise. Treat all of it as a strong default rather than a guarantee against a hostile app.
+
+**Redaction covers text, not pixels.** A failure screenshot shows whatever was on screen, including a value typed from `{{env.*}}` or echoed back by the page. So when anything in a run references `{{env.*}}`, the HTML report does not embed screenshots: each failure links to its PNG under `.blastproof/reports/` instead. That keeps `report.html` safe to attach to a pull request. The PNGs themselves are not masked, so do not upload `.blastproof/reports/` anywhere you would not put the secret ([#110](https://github.com/hamc/blastproof/issues/110)).
 
-**A redacted value cannot be asserted on.** The masking is thorough by design, and page snapshots are not exempt — so a step that verifies text which happens to equal an `{{env.*}}` value can never pass, because the judge is shown `***` where the page shows the thing. The failure is the most misleading shape available: the test is right, the application is right, and the report blames the application. Put a value in `{{env.*}}` because it is a secret or because it varies by environment, but do not then write a step that asserts on it ([#87](https://github.com/hamc/blastproof/issues/87)).
+**A redacted value cannot be asserted on.** The masking is thorough by design, and page snapshots are not exempt — so a step that verifies text which happens to equal an `{{env.*}}` value can never pass, because the judge is shown `***` where the page shows the thing. The failure is the most misleading shape available: the test is right, the application is right, and the report blames the application. Put a value in `{{env.*}}` because it is a secret or because it varies by environment, but do not then write a step that asserts on it ([#87](https://github.com/hamc/blastproof/issues/87)). Matching is case-insensitive, so this covers a little more of the page than the value's exact spelling — one more reason to keep assertions off it.
 
 The system prompt also tells the model that page content is data, never instruction. That raises the cost of casual injection and is **not** a boundary — the origin constraint is. Do not point blastproof at an application you would not run locally.
 

package/dist/cli.js

@@ -336,16 +336,30 @@ function referencedEnvVars(text) {
 function escapeRegExp(value) {
   return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
 }
-function maskSecrets(text, secrets) {
-  const ordered = [...secrets].filter(Boolean).sort((a, b) => b.length - a.length);
+function secretPattern(secret) {
+  return new RegExp(escapeRegExp(secret).replace(/\s+/g, "\\s+"), "gi");
+}
+function compile(secrets) {
+  return [...secrets].filter(Boolean).sort((a, b) => b.length - a.length).map((value) => ({ value, pattern: secretPattern(value) }));
+}
+function maskCompiled(text, compiled, onNearMiss) {
   let masked = text;
-  for (const secret of ordered) {
-    masked = masked.replace(new RegExp(escapeRegExp(secret), "g"), "***");
+  for (const { value, pattern } of compiled) {
+    pattern.lastIndex = 0;
+    masked = masked.replace(pattern, (found) => {
+      if (found !== value) onNearMiss?.(value, found);
+      return "***";
+    });
   }
   return masked;
 }
 var SecretsMask = class {
   secrets = /* @__PURE__ */ new Set();
+  /** Value -> the `{{env.*}}` variable it came from, so a warning can name it. */
+  names = /* @__PURE__ */ new Map();
+  nearMissed = /* @__PURE__ */ new Set();
+  /** Built on first use after a registration, not on every masked string. */
+  compiled;
   /** Registers the current values of the env vars referenced in `text`. Throws MissingEnvError on unset vars. */
   registerFrom(text, env = process.env) {
     for (const name of referencedEnvVars(text)) {
@@ -353,7 +367,7 @@ var SecretsMask = class {
       if (value === void 0) {
         throw new MissingEnvError(name);
       }
-      this.add(value);
+      this.add(value, name);
     }
   }
   /**
@@ -362,14 +376,39 @@ var SecretsMask = class {
    * space stopped matching a literal search and passed through unmasked.
    * This cannot cover every transform a page might apply; see the README.
    */
-  add(value) {
+  add(value, name) {
     if (!value) return;
     this.secrets.add(value);
+    this.compiled = void 0;
+    if (name !== void 0 && !this.names.has(value)) this.names.set(value, name);
     const encoded = encodeURIComponent(value);
-    if (encoded !== value) this.secrets.add(encoded);
+    if (encoded !== value) {
+      this.secrets.add(encoded);
+      if (name !== void 0 && !this.names.has(encoded)) this.names.set(encoded, name);
+    }
   }
   mask(text) {
-    return maskSecrets(text, this.secrets);
+    this.compiled ??= compile(this.secrets);
+    return maskCompiled(text, this.compiled, (secret) => {
+      const name = this.names.get(secret);
+      if (name !== void 0) this.nearMissed.add(name);
+    });
+  }
+  /**
+   * The `{{env.*}}` variables whose value was redacted in a form differing from
+   * the one supplied — the case a literal comparison would have missed
+   * (design D2). Names only: a caller reporting this must never hold the value.
+   */
+  nearMissedVariables() {
+    return [...this.nearMissed];
+  }
+  /**
+   * Whether no value is registered — i.e. nothing in the run referenced
+   * `{{env.*}}`. Says nothing about which values, so a caller deciding what a
+   * report may carry never holds a secret to decide it.
+   */
+  isEmpty() {
+    return this.secrets.size === 0;
   }
 };
 
@@ -2071,7 +2110,11 @@ function renderSteps(steps) {
         ${items}
       </ol>`;
 }
-async function renderTest(result, cwd) {
+function withheldScreenshot(file, reportDir) {
+  const href = path7.relative(reportDir, file).split(path7.sep).join("/");
+  return `      <p class="note">Screenshot withheld: this run handled secrets ({{env.*}}), and a screenshot cannot be masked. <a href="${escapeHtml(href)}">${escapeHtml(href)}</a></p>`;
+}
+async function renderTest(result, screenshots, cwd) {
   const failed = result.status === "failed";
   const notRun = result.status === "not-run";
   const tagClass = failed ? "fail" : notRun ? "skip" : "pass";
@@ -2093,7 +2136,9 @@ async function renderTest(result, cwd) {
     parts.push(`      <p class="note">${escapeHtml(result.reason)}</p>`);
   }
   parts.push(`      ${renderSteps(result.steps)}`);
-  if (failed && result.screenshot) {
+  if (failed && result.screenshot && screenshots !== "embed") {
+    parts.push(withheldScreenshot(result.screenshot, screenshots.withheldRelativeTo));
+  } else if (failed && result.screenshot) {
     const embedded = await embedScreenshot(result.screenshot);
     parts.push(
       embedded ? `      <img class="shot" alt="Screenshot at failure" src="${embedded}">` : '      <p class="note">Screenshot unavailable.</p>'
@@ -2108,7 +2153,7 @@ async function renderHtml(results, skipped, meta) {
   const notRun = results.filter((r) => r.status === "not-run");
   const generatedAt = (meta.generatedAt ?? /* @__PURE__ */ new Date()).toISOString();
   const ordered = [...failures, ...notRun, ...results.filter((r) => r.status === "passed")];
-  const detail = (await Promise.all(ordered.map((r) => renderTest(r, meta.cwd)))).join("\n");
+  const detail = (await Promise.all(ordered.map((r) => renderTest(r, meta.screenshots, meta.cwd)))).join("\n");
   const verdict = meta.minScore === void 0 ? "" : meta.score >= meta.minScore ? `<span class="verdict pass">min-score ${meta.minScore}: pass</span>` : `<span class="verdict fail">min-score ${meta.minScore}: FAIL</span>`;
   const banner = meta.incomplete === void 0 ? "" : `  <section class="incomplete">Run stopped: ${escapeHtml(meta.incomplete)}</section>
 
@@ -2152,7 +2197,7 @@ ${rows}${rows && skippedRows ? "\n" : ""}${skippedRows}
 
 ${detail}
 
-  <footer>Generated by blastproof. Screenshots are embedded, so this file works offline.</footer>
+  <footer>Generated by blastproof. ${meta.screenshots === "embed" ? "Screenshots are embedded, so this file works offline." : "Screenshots were withheld because this run handled secrets ({{env.*}}); failures link to the file instead."}</footer>
 </main>
 </body>
 </html>
@@ -2522,8 +2567,16 @@ function printImpactReport(impact, selection, cwd) {
   }
   console.log("---------------------------------------------------------------");
 }
-async function finalize(results, skipped, options, sessionDir, durationMs, impact, incomplete, spend) {
+function reportNearMissedSecrets(variables) {
+  for (const variable of variables) {
+    console.error(
+      `warning: the application returned ${variable} in a different form than the value supplied (case or spacing). It was redacted, but a form this tool cannot recognise \u2014 an encoding, a hash \u2014 would not have been. Check what that page does with the value.`
+    );
+  }
+}
+async function finalize(results, skipped, options, sessionDir, secretsHeld, nearMissedSecrets, durationMs, impact, incomplete, spend) {
   if (results.length > 0) printSummary(results);
+  reportNearMissedSecrets(nearMissedSecrets);
   if (spend) console.log(formatSpendLine(spend));
   const score = computeScore(results);
   console.log(
@@ -2549,7 +2602,13 @@ async function finalize(results, skipped, options, sessionDir, durationMs, impac
       minScore: options.minScore,
       cwd: options.cwd,
       incomplete: incomplete?.message,
-      spend
+      spend,
+      // A screenshot cannot be masked, so a run that held any `{{env.*}}` value
+      // links to it instead of carrying it (design
+      // withhold-a-screenshot-that-saw-a-secret, D1): the run's mask, not the
+      // failed test's own placeholders, because a login credential is echoed by
+      // the pages of every test that comes after it.
+      screenshots: secretsHeld ? { withheldRelativeTo: path9.dirname(target) } : "embed"
     });
     await writeHtml(target, html);
     console.log(`HTML report: ${path9.relative(options.cwd, target)}`);
@@ -2720,7 +2779,7 @@ ${results.length} test file(s) could not be parsed:`);
     console.log(
       impact ? "No impacted tests to run." : "No tests matched the given filters."
     );
-    return finalize(results, selection.unroutedSkipped, options, sessionDir, Date.now() - startedAt, impact);
+    return finalize(results, selection.unroutedSkipped, options, sessionDir, false, [], Date.now() - startedAt, impact);
   }
   try {
     createModel(config.llm);
@@ -2818,6 +2877,8 @@ ${results.length} test file(s) could not be parsed:`);
     selection.unroutedSkipped,
     options,
     sessionDir,
+    !runMask.isEmpty(),
+    runMask.nearMissedVariables(),
     Date.now() - startedAt,
     impact,
     incomplete,
@@ -3145,7 +3206,7 @@ function parsePositiveNumber(flag) {
   };
 }
 var program = new Command();
-program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.19.0");
+program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.20.0");
 program.command("init").description("Scaffold .blastproof/ (config, tests, sample tests) in the current directory").action(async () => {
   try {
     const result = await initProject(process.cwd());