blastproof 0.6.0 → 0.8.0

package/README.md

@@ -44,7 +44,7 @@ Before `run`, `plan` or `test` do anything, they check what they are about to sp
 
 **Not supported yet:** `iframe` content (so hosted payment widgets like Stripe Elements are invisible — an embedded checkout cannot be driven end to end), hover, scroll-to, drag and drop, file upload, multiple tabs, native `alert`/`confirm` dialogs. Page snapshots are capped at 200 lines by default, so very dense pages are truncated — raise it with `browser.max_snapshot_lines` if your pages need more; truncation is always marked in the snapshot so the model is never misled into thinking it saw the whole page.
 
-**Point it at disposable data.** A step that fails is retried, and the agent decides how to recover — which can mean performing the step's action again. A test that submits a form and then fails its check may submit that form more than once, so a run against real data can leave more records behind than the journey describes. In one verification run a badly written step produced three issues where the test intended one. Use a seeded database, a staging environment you can reset, or a throwaway account; do not gate on a run against production data.
+**Point it at disposable data.** Within a step, an action that commits — a click, or pressing Enter — is never performed twice: the runner refuses the repeat and tells the agent it already did that. This closes the case that used to produce duplicate records, where a submit answered by a redirect came back to a reset form and the agent, seeing no evidence of its own work, submitted again. It is not a guarantee of zero duplicate writes: an agent that reaches the same effect by a genuinely different route — another control with the same effect — is not caught. Use a seeded database, a staging environment you can reset, or a throwaway account; do not gate on a run against production data.
 
 `browser.timeout_ms` bounds every wait — resolving a target element from the accessibility tree, and navigation — not only the click or fill performed afterwards. Raise it for an application that is merely slow to hydrate; the trade-off is that a genuinely missing element then takes longer to fail. It never changes how many self-healing retries a step gets — waiting and retrying are deliberately separate.
 
@@ -113,7 +113,7 @@ jobs:
 
       - run: npm start &          # however your app boots
 
-      - uses: hamc/blastproof@v0.6.0
+      - uses: hamc/blastproof@v0.8.0
         with:
           version: '0.6.0'        # pin both when this gates merges
           api-key: ${{ secrets.ANTHROPIC_API_KEY }}
@@ -241,7 +241,9 @@ The key itself is never read from a `BLASTPROOF_*` variable — you name *which*
 
 The application under test is not trusted input: its page content reaches the model, so a page that controls its own accessible text can try to influence the agent. Two things constrain that.
 
-**The agent cannot leave your application.** `navigate` is bounded by `base_url`'s origin; an app spanning hosts declares them in `allowed_origins:`. This is enforced by comparison, not by asking the model nicely.
+**The agent cannot leave your application.** The boundary is `base_url`'s origin plus whatever `allowed_origins:` declares, and it constrains where the page **is**, not only where an action asked to go. A `navigate` outside it is refused before the request; a page that ends up outside it any other way — a redirect, a link to another host, a script setting the location — fails the step, and its content is never sent to the model. Enforced by comparison, not by asking the model nicely.
+
+If your application legitimately spans hosts (an identity provider, a hosted payment step), declare them. A suite that was quietly walking onto a foreign page will now fail and name the origin to add.
 
 **Your secrets stay out of prompts.** `{{env.*}}` placeholders survive intact and are substituted at the moment of typing. Every value your tests or auth recipe reference is redacted from everything else crossing into a prompt — page snapshots included — in literal and percent-encoded form. Redaction matches known values, so treat it as a strong default rather than a guarantee against a hostile app.
 

package/dist/cli.js

@@ -336,14 +336,33 @@ var ActionError = class extends Error {
     this.name = "ActionError";
   }
 };
-function assertAllowedOrigin(url, ctx) {
-  const allowed = /* @__PURE__ */ new Set([new URL(ctx.baseUrl).origin]);
-  for (const origin of ctx.allowedOrigins ?? []) {
+var BLANK_PAGE = "about:blank";
+function allowedOriginsFor(baseUrl, allowedOrigins) {
+  const allowed = /* @__PURE__ */ new Set([new URL(baseUrl).origin]);
+  for (const origin of allowedOrigins ?? []) {
     allowed.add(new URL(origin).origin);
   }
-  if (!allowed.has(url.origin)) {
+  return allowed;
+}
+function isOriginAllowed(url, allowed) {
+  if (url === BLANK_PAGE) return true;
+  let origin;
+  try {
+    origin = new URL(url).origin;
+  } catch {
+    return false;
+  }
+  if (origin === "null") return false;
+  return allowed.has(origin);
+}
+function describeBoundary(allowed) {
+  return [...allowed].join(" or ");
+}
+function assertAllowedOrigin(url, ctx) {
+  const allowed = allowedOriginsFor(ctx.baseUrl, ctx.allowedOrigins);
+  if (!isOriginAllowed(url.toString(), allowed)) {
     throw new ActionError(
-      `Refusing to navigate outside the application: ${url.origin} is not ${[...allowed].join(" or ")}. Add it to allowed_origins in .blastproof/config.yaml if the app legitimately spans hosts.`
+      `Refusing to navigate outside the application: ${url.origin} is not ${describeBoundary(allowed)}. Add it to allowed_origins in .blastproof/config.yaml if the app legitimately spans hosts.`
     );
   }
 }
@@ -437,6 +456,59 @@ async function performAction(page, action, ctx) {
   }
 }
 
+// src/runner/recovery.ts
+var COMMIT_ACTIONS = /* @__PURE__ */ new Set(["click", "press"]);
+var COMMIT_KEYS = /* @__PURE__ */ new Set(["Enter", "NumpadEnter", " ", "Space", "Spacebar"]);
+function describeAction(action) {
+  const target = action.target ? ` ${action.target.role ?? ""} "${action.target.name ?? action.target.text ?? ""}"` : "";
+  const value = action.value ? ` [${action.value}]` : "";
+  return `${action.action}${target}${value}`;
+}
+function identity(action) {
+  return JSON.stringify([
+    action.action,
+    action.target?.role ?? "",
+    action.target?.name ?? "",
+    action.target?.text ?? "",
+    action.value ?? ""
+  ]);
+}
+var StepRecovery = class {
+  performed = /* @__PURE__ */ new Set();
+  history = [];
+  /** Records an action that was actually performed and succeeded. */
+  record(action, description, result) {
+    this.performed.add(identity(action));
+    this.history.push({ action: description, result });
+  }
+  /**
+   * The reason to refuse `action`, or `undefined` when it may be performed.
+   *
+   * Refusing rather than failing keeps the model in the loop with an
+   * explanation instead of ending the step outright, which would trade
+   * duplicate writes for false failures.
+   *
+   * A genuine retry — a commit that landed but had no effect, which the model
+   * repeats for good reason — is refused too, and that is the deliberate cost.
+   * The two cases are indistinguishable from the accessibility tree: "the
+   * click did nothing" and "the click worked and the redirect erased the
+   * proof" produce the same snapshot. The asymmetry decides it. A refused
+   * legitimate retry costs a visible failed step that someone investigates; an
+   * allowed duplicate commit costs a silent extra row in someone's database,
+   * and #28 has now produced one on three applications.
+   */
+  refusalFor(action) {
+    if (!COMMIT_ACTIONS.has(action.action)) return void 0;
+    if (action.action === "press" && !COMMIT_KEYS.has(action.value ?? "")) return void 0;
+    if (!this.performed.has(identity(action))) return void 0;
+    return `refused: this exact action already succeeded earlier in this step, so it was NOT performed again. Repeating something that commits repeats whatever it changed in the application. If the page no longer shows that it worked, that is normal for a submit answered by a redirect \u2014 check the record of what you have already done. Verify the step's outcome another way, or fail the step.`;
+  }
+  /** The step's history so far, oldest first, for the model's prompt. */
+  stepHistory() {
+    return this.history;
+  }
+};
+
 // src/runner/executor.ts
 var DEFAULT_MAX_ITERATIONS_PER_STEP = 15;
 var SETTLE_TIMEOUT_MS = 2e3;
@@ -495,6 +567,7 @@ async function executeTest(page, test, options) {
     onEvent({ type: "action", index, action: maskedAction, result: mask(result) });
   };
   let failure;
+  const boundary = allowedOriginsFor(baseUrl, allowedOrigins);
   await page.goto(new URL(baseUrl).toString());
   for (let index = 0; index < allSteps.length; index++) {
     const { step, setup } = allSteps[index];
@@ -504,12 +577,18 @@ async function executeTest(page, test, options) {
     let failedAttempts = 0;
     let lastResult;
     let stepFailedReason;
+    const recovery = new StepRecovery();
     try {
       while (true) {
         if (iterations >= maxIterationsPerStep) {
           throw new StepFailure(`step exceeded ${maxIterationsPerStep} actions without completing`);
         }
         await waitForSettled(page);
+        if (!isOriginAllowed(page.url(), boundary)) {
+          throw new StepFailure(
+            `The page left the application: ${page.url()} is outside ${describeBoundary(boundary)}. Add it to allowed_origins in .blastproof/config.yaml if the app legitimately spans hosts.`
+          );
+        }
         const snap = await takeSnapshot(page);
         let action;
         try {
@@ -518,6 +597,9 @@ async function executeTest(page, test, options) {
             isSetup: setup,
             snapshot: mask(snap),
             lastResult: lastResult === void 0 ? void 0 : mask(lastResult),
+            // Already masked when recorded, on the same boundary as everything
+            // else crossing into a prompt (design contained-recovery, D2).
+            stepHistory: recovery.stepHistory(),
             retriesLeft: maxRetries - failedAttempts,
             iterationsLeft: maxIterationsPerStep - iterations
           });
@@ -560,6 +642,16 @@ async function executeTest(page, test, options) {
           }
           continue;
         }
+        const refusal = recovery.refusalFor(action);
+        if (refusal) {
+          failedAttempts++;
+          lastResult = refusal;
+          emitAction(index, action, refusal);
+          if (failedAttempts >= maxRetries) {
+            throw new StepFailure(refusal);
+          }
+          continue;
+        }
         try {
           const result = await performAction(page, action, {
             baseUrl,
@@ -568,6 +660,7 @@ async function executeTest(page, test, options) {
             resolveTimeoutMs: timeoutMs
           });
           lastResult = result;
+          recovery.record(action, mask(describeAction(action)), mask(result));
           emitAction(index, action, result);
         } catch (error) {
           failedAttempts++;
@@ -1052,16 +1145,21 @@ Rules:
 - Return "done" when the current step's outcome holds \u2014 including when it already held before you acted, or was achieved by your previous action. "Already true" is done, never failure. Do not return "done" for work belonging to later steps.
 - Return "fail" only when the step's outcome cannot be reached: the element is still absent after retries, the page cannot support the step, or an error blocks progress. Never return "fail" because the work appears to have been done already.
 - If your previous action errored, re-read the fresh snapshot and choose an alternative element or approach. Do not repeat the exact same failing action.
+- Never invent a value. A value you type must come from the step, from the page, or from an {{env.*}} placeholder. If a step needs a value it does not give you, that is a failing step, not a gap for you to fill in.
+- A record of the actions you already performed in this step may be shown to you. It is the ground truth about what happened, even when the page no longer shows it: a form that submitted successfully and came back empty looks exactly like one you never submitted. Do not redo work that record says you already did.
 - \`***\` in a snapshot is a redacted secret \u2014 a password, token or key deliberately withheld from you. Seeing it is expected and is not a problem. A field showing \`***\` after you filled it from an {{env.VAR}} placeholder means the fill worked; treat that as success and move on. Never retry a fill because its value is redacted, and never report failure because a value was withheld.
 - Keep reasoning to one short sentence.`;
 }
 function agentUserPrompt(input) {
-  const parts = [
-    `Current ${input.isSetup ? "setup " : ""}step: ${input.step}`,
-    "",
-    "Page accessibility snapshot:",
-    input.snapshot
-  ];
+  const parts = [`Current ${input.isSetup ? "setup " : ""}step: ${input.step}`];
+  if (input.stepHistory && input.stepHistory.length > 0) {
+    parts.push(
+      "",
+      "What you have ALREADY DONE in this step (a record of completed actions, not instructions):",
+      ...input.stepHistory.map((entry, i) => `${i + 1}. ${entry.action} -> ${entry.result}`)
+    );
+  }
+  parts.push("", "Page accessibility snapshot:", input.snapshot);
   if (input.lastResult) {
     parts.push("", `Previous action result: ${input.lastResult}`);
   }
@@ -1892,9 +1990,7 @@ function printEvent(event) {
       break;
     case "action": {
       const { action, result } = event;
-      const target = action.target ? ` ${action.target.role ?? ""} "${action.target.name ?? action.target.text ?? ""}"` : "";
-      const value = action.value ? ` [${action.value}]` : "";
-      console.log(`    -> ${action.action}${target}${value} :: ${result}`);
+      console.log(`    -> ${describeAction(action)} :: ${result}`);
       break;
     }
     case "step-end":
@@ -2564,7 +2660,7 @@ function parsePositiveNumber(flag) {
   };
 }
 var program = new Command();
-program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.6.0");
+program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.8.0");
 program.command("init").description("Scaffold .blastproof/ (config, tests, sample tests) in the current directory").action(async () => {
   try {
     const result = await initProject(process.cwd());