blastproof 0.20.0 → 0.22.0

package/README.md

@@ -275,11 +275,11 @@ The application under test is not trusted input: its page content reaches the mo
 
 If your application legitimately spans hosts (an identity provider, a hosted payment step), declare them. A suite that was quietly walking onto a foreign page will now fail and name the origin to add.
 
-**Your secrets stay out of prompts.** `{{env.*}}` placeholders survive intact and are substituted at the moment of typing. Every value your tests or auth recipe reference is redacted from everything else crossing into a prompt — page snapshots included — in literal and percent-encoded form. Redaction matches a known value case-insensitively and tolerantly of whitespace, so a page that uppercases what you typed is still covered. It does not chase a value your application re-encodes, hashes or truncates — no list of transforms can be complete, and one that pretends to be would make you careless. When a redaction is found only by that wider comparison, the run prints one warning naming the variable, because the same page could just as easily have returned a form nothing here can recognise. Treat all of it as a strong default rather than a guarantee against a hostile app.
+**Your secrets stay out of prompts.** `{{env.*}}` placeholders survive intact and are substituted at the moment of typing. Every value your tests or auth recipe reference is redacted from everything else crossing into a prompt — page snapshots included — in literal and percent-encoded form, and replaced with a label naming its variable: `[redacted TEST_PASSWORD]`. The label carries the name and nothing derived from the value, so it tells a reader, and the model, *which* secret was there without telling anyone what it was. Redaction matches a known value case-insensitively and tolerantly of whitespace, so a page that uppercases what you typed is still covered. It does not chase a value your application re-encodes, hashes or truncates — no list of transforms can be complete, and one that pretends to be would make you careless. When a redaction is found only by that wider comparison, the run prints one warning naming the variable, because the same page could just as easily have returned a form nothing here can recognise. Treat all of it as a strong default rather than a guarantee against a hostile app.
 
 **Redaction covers text, not pixels.** A failure screenshot shows whatever was on screen, including a value typed from `{{env.*}}` or echoed back by the page. So when anything in a run references `{{env.*}}`, the HTML report does not embed screenshots: each failure links to its PNG under `.blastproof/reports/` instead. That keeps `report.html` safe to attach to a pull request. The PNGs themselves are not masked, so do not upload `.blastproof/reports/` anywhere you would not put the secret ([#110](https://github.com/hamc/blastproof/issues/110)).
 
-**A redacted value cannot be asserted on.** The masking is thorough by design, and page snapshots are not exempt — so a step that verifies text which happens to equal an `{{env.*}}` value can never pass, because the judge is shown `***` where the page shows the thing. The failure is the most misleading shape available: the test is right, the application is right, and the report blames the application. Put a value in `{{env.*}}` because it is a secret or because it varies by environment, but do not then write a step that asserts on it ([#87](https://github.com/hamc/blastproof/issues/87)). Matching is case-insensitive, so this covers a little more of the page than the value's exact spelling — one more reason to keep assertions off it.
+**A redacted value is asserted on by identity.** A step that verifies text equal to an `{{env.*}}` value reaches the judge through the mask, like the page does, so both sides read `[redacted NAME]`. The same label on both sides is a match; different labels are not. Until 0.21.0 every secret was redacted to the same `***`, and a step verifying one secret passed against a page showing another — measured, 3 times out of 3 ([#87](https://github.com/hamc/blastproof/issues/87)). What the judge still cannot check is anything *about* the value beyond which variable it is: its length, its format, a prefix. Assert on those with a value that is not in `{{env.*}}`.
 
 The system prompt also tells the model that page content is data, never instruction. That raises the cost of casual injection and is **not** a boundary — the origin constraint is. Do not point blastproof at an application you would not run locally.
 

package/dist/cli.js

@@ -118,7 +118,7 @@ var SAMPLE_LOGIN_TEMPLATE = `# TEMPLATE \u2014 rename to login.yaml once these s
 #
 # Credentials come from the environment. Export them before running:
 #   export TEST_EMAIL=... TEST_PASSWORD=...
-# Substituted values are masked as *** everywhere and never reach the model.
+# Substituted values are masked as [redacted NAME] everywhere and never reach the model.
 summary: Login with valid credentials succeeds
 priority: P0
 tags: [smoke, auth]
@@ -333,22 +333,35 @@ function referencedEnvVars(text) {
   }
   return [...names];
 }
+function placeholdersAsLabels(text) {
+  return text.replace(ENV_PLACEHOLDER, (_match, name) => redactionLabel(name));
+}
 function escapeRegExp(value) {
   return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
 }
 function secretPattern(secret) {
   return new RegExp(escapeRegExp(secret).replace(/\s+/g, "\\s+"), "gi");
 }
-function compile(secrets) {
-  return [...secrets].filter(Boolean).sort((a, b) => b.length - a.length).map((value) => ({ value, pattern: secretPattern(value) }));
+var REDACTION_PREFIX = "[redacted";
+function redactionLabel(name) {
+  return name === void 0 ? `${REDACTION_PREFIX}]` : `${REDACTION_PREFIX} ${name}]`;
+}
+var LABEL = /\[redacted ([A-Za-z_][A-Za-z0-9_]*)\]/g;
+function labelledVariables(text) {
+  const names = /* @__PURE__ */ new Set();
+  for (const match of text.matchAll(LABEL)) if (match[1]) names.add(match[1]);
+  return [...names];
+}
+function compile(secrets, names) {
+  return [...secrets].filter(Boolean).sort((a, b) => b.length - a.length).map((value) => ({ value, pattern: secretPattern(value), label: redactionLabel(names?.get(value)) }));
 }
 function maskCompiled(text, compiled, onNearMiss) {
   let masked = text;
-  for (const { value, pattern } of compiled) {
+  for (const { value, pattern, label } of compiled) {
     pattern.lastIndex = 0;
     masked = masked.replace(pattern, (found) => {
       if (found !== value) onNearMiss?.(value, found);
-      return "***";
+      return label;
     });
   }
   return masked;
@@ -388,7 +401,7 @@ var SecretsMask = class {
     }
   }
   mask(text) {
-    this.compiled ??= compile(this.secrets);
+    this.compiled ??= compile(this.secrets, this.names);
     return maskCompiled(text, this.compiled, (secret) => {
       const name = this.names.get(secret);
       if (name !== void 0) this.nearMissed.add(name);
@@ -677,7 +690,26 @@ var StepRecovery = class {
    * and #28 has now produced one on three applications.
    */
   refusalFor(action) {
-    return this.repeatedCommitRefusal(action) ?? this.unsourcedValueRefusal(action);
+    return this.repeatedCommitRefusal(action) ?? this.redactionLabelRefusal(action) ?? this.unsourcedValueRefusal(action);
+  }
+  /**
+   * Refuses a typed value containing a redaction label (design
+   * label-a-redaction-with-its-variable, D3).
+   *
+   * Checked before the source check because the source check would admit it:
+   * `observe` credits the model with the masked snapshot, and the label is in
+   * the masked snapshot. It is the mask's own writing, not the application's, so
+   * typing it would put the words "[redacted TEST_EMAIL]" into a real field.
+   * A prefix match rather than the exact label, because a model that copies
+   * part of one, or rebuilds one around another name, is equally wrong.
+   *
+   * Translating it back into its placeholder was the alternative, and it is the
+   * exemption #66 closed: the model would be choosing a secret to type.
+   */
+  redactionLabelRefusal(action) {
+    if (!SOURCED_VALUE_ACTIONS.has(action.action)) return void 0;
+    if (!action.value?.includes(REDACTION_PREFIX)) return void 0;
+    return `refused: the value was NOT typed, because it contains a redaction label. ${REDACTION_PREFIX} NAME] stands for the value of {{env.NAME}}, withheld from you; it is never a value the application should receive. To enter an environment value, use the {{env.*}} placeholder this step names. If the step names none, use a value it supplies, or fail the step.`;
   }
   repeatedCommitRefusal(action) {
     if (!COMMIT_ACTIONS.has(action.action)) return void 0;
@@ -1449,7 +1481,7 @@ Rules:
 - An action reported as "blocked" is the exception to that rule: it means another element is on top of your target, not that you picked the wrong target. Re-targeting cannot fix it. Whatever is covering the page is in the snapshot \u2014 a dialog, a cookie banner, an onboarding overlay \u2014 so dismiss that first, with its own close or accept control, or by pressing Escape with no target, and then act on your original target again. Overlays can be stacked: clearing one may reveal another, and that is progress, not failure.
 - Never invent a value. A value you type must come from the step, from the page, or from an {{env.*}} placeholder. This one is enforced, not merely asked: a fill or select whose value is in none of those is refused and not performed. If a step needs a value it does not give you, that is a failing step, not a gap for you to fill in.
 - A record of the actions you already performed in this step may be shown to you. It is the ground truth about what happened, even when the page no longer shows it: a form that submitted successfully and came back empty looks exactly like one you never submitted. Do not redo work that record says you already did.
-- \`***\` in a snapshot is a redacted secret \u2014 a password, token or key deliberately withheld from you. Seeing it is expected and is not a problem. A field showing \`***\` after you filled it from an {{env.VAR}} placeholder means the fill worked; treat that as success and move on. Never retry a fill because its value is redacted, and never report failure because a value was withheld.
+- \`[redacted NAME]\` in a snapshot is the value of {{env.NAME}}, deliberately withheld from you. Seeing one is expected and is not a problem. The same label is the same value; two different labels are two different values. A field showing \`[redacted NAME]\` after you filled it from {{env.NAME}} means the fill worked; treat that as success and move on. Never retry a fill because its value is redacted, never report failure because a value was withheld, and never type a label as a value: to enter an environment value, use the {{env.*}} placeholder the step names.
 - Keep reasoning to one short sentence.`;
 }
 function agentUserPrompt(input) {
@@ -1485,7 +1517,7 @@ The record is not evidence that the step's outcome holds. An action reported as
 
 Be strict about what the step asks, not about withholding a pass you can plainly see is earned. Answer with pass=true/false and a one-sentence reason.
 
-\`***\` marks a secret deliberately withheld from you \u2014 a password, token or key. Seeing it is expected. A field holding \`***\` is filled, not empty, so do not fail a step on the grounds that a value was redacted. This applies only to the redaction itself: everything else the step asks for must still be visibly satisfied by the snapshot, and a step you genuinely cannot check against what you were shown still fails.`;
+\`[redacted NAME]\` marks the value of {{env.NAME}}, deliberately withheld from you. Seeing one is expected. A field holding a label is filled, not empty, so do not fail a step on the grounds that a value was redacted. The same label is the same value, and two different labels are two different values: a step expecting \`[redacted A]\` is not satisfied by a page showing \`[redacted B]\`. This applies only to the redaction itself: everything else the step asks for must still be visibly satisfied by the snapshot, and a step you genuinely cannot check against what you were shown still fails.`;
 }
 function assertUserPrompt(step, expectation, snapshot, stepHistory) {
   const parts = [`Step under test: ${step}`];
@@ -1520,7 +1552,7 @@ Rules:
 - **The test starts at the application's base URL, not at this route.** Begin with a step that navigates to the route and says what should be visible once it loads \u2014 "navigate to /support and verify the heading "Contact support" is shown". Without it the run opens the home page and every later step looks for controls that are not there.
 - **Every step says what it should produce.** Name what must be true once the step has been carried out, not the action alone: "submit the support form and verify the confirmation page shows the ticket number", never "submit the support form". A step that names an action without an outcome asks the runner to judge whether something happened while looking at the page that succeeding produces \u2014 a submitted form comes back empty, a redirect moves the URL \u2014 and that is the shape behind several real failures.
 - **A step that enters a value writes the value.** "fill the subject field with Order not received", never "enter a subject". The runner is forbidden from inventing values, and enforces it: a fill whose value is in neither the step nor the page is refused, so a step that supplies none cannot be relied on to run.
-- If a step needs a credential or any secret, write it as a placeholder like {{env.TEST_PASSWORD}}. Never write a real or invented password, token or key.
+- If a step needs a credential, or any value that identifies an account or a person \u2014 an email address, a username, an account or customer number \u2014 write it as a placeholder like {{env.TEST_EMAIL}} or {{env.TEST_PASSWORD}}. Never write a real or invented one.
 - Keep the whole test to a handful of steps: one journey, not an exhaustive suite.`;
 }
 function plannerUserPrompt(input) {
@@ -1584,8 +1616,10 @@ function parseAgentAction(value) {
   return parsedAgentActionSchema.safeParse(value);
 }
 var assertJudgmentSchema = z2.object({
-  pass: z2.boolean().describe("Whether the snapshot satisfies the expectation."),
-  reason: z2.string().describe("One sentence explaining the judgment.")
+  reason: z2.string().describe("One sentence: what the STEP requires to be true now, and whether the snapshot shows it."),
+  pass: z2.boolean().describe(
+    "Whether the snapshot shows the STEP's own outcome. The expectation is only a claim offered in support; it never replaces the step. False if any part of the outcome the step asks for is not shown, or cannot be assessed from this snapshot."
+  )
 });
 var generatedTestSchema = z2.object({
   summary: z2.string().min(1).describe('One line naming the user journey this test covers, e.g. "Applying a discount updates the cart total".'),
@@ -1642,11 +1676,20 @@ function createBrain(model, generate = generateObject, budget) {
       return parsed.data;
     },
     async judge(step, expectation, snapshot, stepHistory) {
+      const asJudged = stepHistory?.map((entry) => ({
+        action: placeholdersAsLabels(entry.action),
+        result: placeholdersAsLabels(entry.result)
+      }));
       const result = await countedGenerate(generate, budget, {
         model,
         schema: assertJudgmentSchema,
         system: assertSystemPrompt(),
-        prompt: assertUserPrompt(step, expectation, snapshot, stepHistory),
+        prompt: assertUserPrompt(
+          placeholdersAsLabels(step),
+          placeholdersAsLabels(expectation),
+          snapshot,
+          asJudged
+        ),
         temperature: 0
       });
       const parsed = assertJudgmentSchema.safeParse(result.object);
@@ -1655,10 +1698,26 @@ function createBrain(model, generate = generateObject, budget) {
           `Model returned an invalid assert judgment: ${parsed.error.issues[0]?.message ?? "unknown"}`
         );
       }
-      return parsed.data;
+      return secretMismatch(placeholdersAsLabels(step), snapshot, asJudged, parsed.data);
     }
   };
 }
+function secretMismatch(judgedStep, snapshot, judgedRecord, judgment) {
+  if (!judgment.pass) return judgment;
+  const onPage = labelledVariables(snapshot);
+  const inRecord = labelledVariables((judgedRecord ?? []).map((e) => `${e.action}
+${e.result}`).join("\n"));
+  for (const name of labelledVariables(judgedStep)) {
+    const others = onPage.filter((other) => other !== name);
+    if (onPage.includes(name) || inRecord.includes(name) || others.length === 0) continue;
+    const shown = others.map((other) => redactionLabel(other)).join(", ");
+    return {
+      pass: false,
+      reason: `The step names ${redactionLabel(name)}, which appears neither on the page nor in this step's actions, while the page shows ${shown}: a different secret cannot satisfy it. (The judge had said: ${judgment.reason})`
+    };
+  }
+  return judgment;
+}
 function createPlanner(model, generate = generateObject, budget) {
   return {
     async planTest(input) {
@@ -1736,6 +1795,41 @@ import { mkdir as mkdir4, writeFile as writeFile3 } from "fs/promises";
 import path6 from "path";
 import { stringify } from "yaml";
 
+// src/runner/authoring.ts
+var VALUE_VERBS = ["fill", "enter", "type", "input", "set"];
+var LEADING_VALUE_VERB = new RegExp(`^\\s*(?:${VALUE_VERBS.join("|")})\\b`, "i");
+var CONNECTOR_WORDS = ["with", "to", "as", "using", "into", "from", "in"];
+var CONNECTOR_WORD = new RegExp(`\\b(?:${CONNECTOR_WORDS.join("|")})\\b`, "i");
+var CONNECTOR_SYMBOL = /["'`:=]|\{\{env\./i;
+var PHRASAL_IN = new RegExp(`^(\\s*(?:${VALUE_VERBS.join("|")}))\\s+in\\b`, "i");
+function namesAValue(step) {
+  const withoutPhrasal = step.replace(PHRASAL_IN, "$1");
+  return CONNECTOR_WORD.test(withoutPhrasal) || CONNECTOR_SYMBOL.test(withoutPhrasal);
+}
+function namesNoValue(step) {
+  return LEADING_VALUE_VERB.test(step) && !namesAValue(step);
+}
+function entersNamedValue(step) {
+  return LEADING_VALUE_VERB.test(step) && namesAValue(step);
+}
+function detectMissingValues(tests) {
+  const findings = [];
+  for (const test of tests) {
+    for (const origin of ["setup", "steps"]) {
+      const steps = origin === "setup" ? test.setup ?? [] : test.steps;
+      steps.forEach((step, position) => {
+        if (namesNoValue(step)) {
+          findings.push({ test, origin, index: position + 1, step });
+        }
+      });
+    }
+  }
+  return { findings };
+}
+function suggestValueClause(step) {
+  return `${step.trimEnd()} with <value>`;
+}
+
 // src/runner/testfile.ts
 import { readFile as readFile3, readdir } from "fs/promises";
 import path5 from "path";
@@ -1811,9 +1905,20 @@ var CREDENTIAL_WORD = /\b(password|passwd|api[ _-]?key|token|secret|credential)\
 var QUOTED_LITERAL = /["'][^"']+["']/;
 function findSecretLiterals(steps) {
   return steps.filter(
-    (step) => CREDENTIAL_WORD.test(step) && QUOTED_LITERAL.test(step) && !step.includes("{{env.")
+    (step) => CREDENTIAL_WORD.test(step) && !step.includes("{{env.") && (QUOTED_LITERAL.test(step) || entersNamedValue(step))
   );
 }
+var EMAIL_ADDRESS = /[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)+/g;
+function findUnsourcedEmails(steps, snapshot) {
+  const page = snapshot.toLowerCase();
+  const found = [];
+  steps.forEach((text, step) => {
+    for (const [address] of text.matchAll(EMAIL_ADDRESS)) {
+      if (!page.includes(address.toLowerCase())) found.push({ step, address });
+    }
+  });
+  return found;
+}
 function routeToSlug(route) {
   const slug = route.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "").slice(0, 60);
   return slug || "home";
@@ -1845,9 +1950,10 @@ async function generateForRoute(page, options) {
       `Cannot load ${url}: ${error instanceof Error ? error.message : String(error)}`
     );
   }
+  const pageSnapshot = mask(await takeSnapshot(page));
   const generated = await brain.planTest({
     route,
-    snapshot: mask(await takeSnapshot(page)),
+    snapshot: pageSnapshot,
     changedFiles
   });
   const leaked = findSecretLiterals(generated.steps);
@@ -1856,7 +1962,11 @@ async function generateForRoute(page, options) {
       `Generated steps contain literal secrets instead of {{env.VAR}} placeholders: ${leaked.join(" | ")}`
     );
   }
-  return { ...generated, routes: [route] };
+  return {
+    ...generated,
+    routes: [route],
+    unsourcedEmails: findUnsourcedEmails(generated.steps, pageSnapshot)
+  };
 }
 async function writeDraft(cwd, draft, meta) {
   const dir = path6.join(cwd, TESTS_RELATIVE_DIR);
@@ -2294,36 +2404,6 @@ async function writeJUnit(file, xml) {
   }
 }
 
-// src/runner/authoring.ts
-var VALUE_VERBS = ["fill", "enter", "type", "input", "set"];
-var LEADING_VALUE_VERB = new RegExp(`^\\s*(?:${VALUE_VERBS.join("|")})\\b`, "i");
-var CONNECTOR_WORDS = ["with", "to", "as", "using", "into", "from", "in"];
-var CONNECTOR_WORD = new RegExp(`\\b(?:${CONNECTOR_WORDS.join("|")})\\b`, "i");
-var CONNECTOR_SYMBOL = /["'`:=]|\{\{env\./i;
-var PHRASAL_IN = new RegExp(`^(\\s*(?:${VALUE_VERBS.join("|")}))\\s+in\\b`, "i");
-function namesNoValue(step) {
-  if (!LEADING_VALUE_VERB.test(step)) return false;
-  const withoutPhrasal = step.replace(PHRASAL_IN, "$1");
-  return !CONNECTOR_WORD.test(withoutPhrasal) && !CONNECTOR_SYMBOL.test(withoutPhrasal);
-}
-function detectMissingValues(tests) {
-  const findings = [];
-  for (const test of tests) {
-    for (const origin of ["setup", "steps"]) {
-      const steps = origin === "setup" ? test.setup ?? [] : test.steps;
-      steps.forEach((step, position) => {
-        if (namesNoValue(step)) {
-          findings.push({ test, origin, index: position + 1, step });
-        }
-      });
-    }
-  }
-  return { findings };
-}
-function suggestValueClause(step) {
-  return `${step.trimEnd()} with <value>`;
-}
-
 // src/runner/pool.ts
 async function runWithConcurrency(items, concurrency, run) {
   if (concurrency < 1) throw new RangeError(`concurrency must be at least 1, got ${concurrency}`);
@@ -3074,6 +3154,12 @@ async function planCommand(options) {
       } finally {
         await context.close();
       }
+      for (const { step, address } of draft.unsourcedEmails) {
+        console.error(
+          `  warning: step ${step + 1} writes an email address the page does not show (${address}).
+    If it is the account the test signs in as, use a placeholder like {{env.TEST_EMAIL}}.`
+        );
+      }
       generated.push(route);
       if (options.write) {
         try {
@@ -3206,7 +3292,7 @@ function parsePositiveNumber(flag) {
   };
 }
 var program = new Command();
-program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.20.0");
+program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.22.0");
 program.command("init").description("Scaffold .blastproof/ (config, tests, sample tests) in the current directory").action(async () => {
   try {
     const result = await initProject(process.cwd());