blastproof 0.14.0 → 0.16.0

package/README.md

@@ -1,7 +1,7 @@
 <h1>
   <picture>
-    <source media="(prefers-color-scheme: dark)" srcset="./.github/logo-dark.svg">
-    <img src="./.github/logo-light.svg" alt="" width="30" height="30">
+    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/hamc/blastproof/main/.github/logo-dark.svg">
+    <img src="https://raw.githubusercontent.com/hamc/blastproof/main/.github/logo-light.svg" alt="" width="30" height="30">
   </picture>
   blastproof
 </h1>
@@ -18,10 +18,29 @@ git diff → impact mapping → test generation → agentic execution → report
 
 100% local. MIT. Bring your own LLM key.
 
+<img src="https://raw.githubusercontent.com/hamc/blastproof/main/.github/dogfood.gif" width="100%"
+     alt="A terminal running blastproof against this repo's demo app. It passes four steps, then fails
+          step 5: the page says the SAVE20 promo gave 20% off, but the discount shown is -$6.00 where
+          20% of $120.00 would be -$24.00. Score: 0.">
+
 ▶ **[Watch the introduction](https://www.youtube.com/shorts/miqN5FzMF_k)** — what it does, in a minute.
 
 **Documentation:** [Configuration](./docs/configuration.md) · [Testing behind a login](./docs/auth.md) · [Running in CI](./docs/ci.md) · [Contributing](./CONTRIBUTING.md) · [Architecture](./AGENTS.md)
 
+## Set it up with your coding agent
+
+Most of what the setup asks — which stack, which port, which journeys matter — is a question your coding agent can already answer by looking at your project. So let it answer them:
+
+```bash
+npx skills add hamc/blastproof
+```
+
+Then tell your agent: **"set up e2e tests"**. The skill installs for whichever agents you have — Claude Code, Cursor, Codex and others — and walks the whole path: check whether your markup is reachable at all, pick a provider, scaffold, generate drafts from your *running* app, curate them, run them, and write down the accessibility constraints that keep the suite alive as the project grows.
+
+Two things it deliberately will not do, so their absence does not read as a defect: it does not configure [authentication](./docs/auth.md), and it does not wire up [CI](./docs/ci.md). Both are worth doing after you have seen a green run on your own machine.
+
+The skill lives in [`skills/blastproof/`](./skills/blastproof/) and is worth reading even if you never install it. [`references/authoring.md`](./skills/blastproof/references/authoring.md) sets out what separates a test that detects something from one that reports Score 100 and detects nothing — including two drafts this tool generated that passed unedited and were worth nothing.
+
 ## Quick start
 
 ```bash
@@ -89,7 +108,7 @@ It is **not** a guarantee of zero duplicate writes. An agent that reaches the sa
 
 ## How it works
 
-<img src="./.github/coverage-flow.svg" width="100%"
+<img src="https://raw.githubusercontent.com/hamc/blastproof/main/.github/coverage-flow.svg" width="100%"
      alt="How a diff becomes a merge decision. In CI, unattended: changed files are matched against the routes: and ignore: globs; matched files contribute affected routes, files matching neither are reported as unclassified and fail the run only under --fail-on-unmapped. Tests declaring an affected route are executed and produce a weighted score, which --min-score gates on. An affected route no test declares is reported as a coverage gap and never fails the run. Separately and manually, outside CI: blastproof plan loads such a route in Chromium, makes one model call, and produces a YAML draft you review, edit and run before committing it.">
 
 **The boundary in the middle is the point.** Everything above it runs unattended on every pull request and ends in an exit code. Everything below it is something you choose to run, on your machine, and review before it lands.
@@ -145,6 +164,8 @@ The agent is **forbidden from inventing values** — one it types must come from
 
 The rule used to live only in the prompt, and a prompt instructs rather than enforces. Run against a real model, `fill the note field` did not fail — the agent made a value up, filled it, and the step **passed**, producing "This is a test note." on two runs and "This is a new note" on a third. A test going green over a value nobody wrote, differing between runs, is worse than a failure.
 
+A placeholder counts as a source **only when the step names that variable**. `fill the password field with {{env.TEST_PASSWORD}}` works; `fill the password field` does not become valid because the agent supplies `{{env.SOMETHING}}` itself. An agent cannot know the name of a variable nobody showed it, so one it produces is a guess — and a guessed variable would put a live credential into a field your test never pointed one at, in output that cannot redact a secret it was never told about.
+
 Two limits worth knowing. A value the page shows in one format and the field wants in another — `1234` in the step, `1,234.00` in the box — is refused, and the fix is to write the value the way it is typed. And a very short value (`3`) appears somewhere in almost any page, so it will pass; this closes fabricated content, not every fabricated character.
 
 `run` also warns about it first — the same rule caught earlier, from the test file, on every path, before launching a browser or asking for a key:
@@ -219,6 +240,10 @@ blastproof run --min-score 80    # one failing P2 is tolerated
 
 `--min-score` **replaces** the all-must-pass rule rather than adding to it. Only executed tests count: filtered and unrouted tests are neither numerator nor denominator, and a run that executed nothing scores 100, so a docs-only PR is never blocked. JUnit carries the score as a `<property name="score">`, and unrouted tests appear as `<skipped/>` so the coverage gap shows up in CI rather than vanishing.
 
+**The verdict behind that number is pinned, and that is not the same as reproducible.** The model call that decides whether a step passed runs at temperature 0, so the same page and the same expectation are not re-sampled into a different answer — a gate that flips on identical input teaches people to re-run until green, which is worse than no gate. What pinning does not survive: provider-side batching, floating point, and a gateway routing two calls to different providers or quantizations. Expect far less variance, not a guarantee of the same output twice.
+
+The calls that *choose* an action, and the one that drafts a test, are deliberately left free — that latitude is what re-resolves a control after a redesign instead of failing on it.
+
 Wiring this into a pipeline, with the gating patterns worth knowing: [Running in CI](./docs/ci.md).
 
 ## Without a browser or a key
@@ -244,17 +269,13 @@ If your application legitimately spans hosts (an identity provider, a hosted pay
 
 **Your secrets stay out of prompts.** `{{env.*}}` placeholders survive intact and are substituted at the moment of typing. Every value your tests or auth recipe reference is redacted from everything else crossing into a prompt — page snapshots included — in literal and percent-encoded form. Redaction matches known values, so treat it as a strong default rather than a guarantee against a hostile app.
 
+**A redacted value cannot be asserted on.** The masking is thorough by design, and page snapshots are not exempt — so a step that verifies text which happens to equal an `{{env.*}}` value can never pass, because the judge is shown `***` where the page shows the thing. The failure is the most misleading shape available: the test is right, the application is right, and the report blames the application. Put a value in `{{env.*}}` because it is a secret or because it varies by environment, but do not then write a step that asserts on it ([#87](https://github.com/hamc/blastproof/issues/87)).
+
 The system prompt also tells the model that page content is data, never instruction. That raises the cost of casual injection and is **not** a boundary — the origin constraint is. Do not point blastproof at an application you would not run locally.
 
 ## blastproof tests itself
 
-The **Dogfood** badge is blastproof running against the demo app in this repo — real Chromium, real model, scored and gated, with public logs. It catches real regressions rather than diffing strings: change the demo discount from 20% to 5% while the page still claims *"20% off"* and it reports
-
-```
-FAIL  P0  Promo code SAVE20 applies a 20% discount in the cart
-  reason: the discount is currently -$6.00, but a 20% discount on
-          $120.00 should be -$24.00
-```
+The **Dogfood** badge is blastproof running against the demo app in this repo — real Chromium, real model, scored and gated, with public logs. It catches real regressions rather than diffing strings: change the demo discount from 20% to 5% while the page still claims *"20% off"* and it fails the step — that is the run in the GIF at the top of this page, verbatim.
 
 No selector was updated to catch that. The agent read the value, did the arithmetic, and disagreed with the page.
 

package/dist/cli.js

@@ -458,7 +458,28 @@ function requireValue(action) {
   }
   return action.value;
 }
+var INTERCEPTS_POINTER = /(<[^>\n]{1,400}>)[^\n]{0,400}?intercepts pointer events/;
+var MAX_BLOCKER_LENGTH = 80;
+function truncateBlocker(tag) {
+  return tag.length <= MAX_BLOCKER_LENGTH ? tag : `${tag.slice(0, MAX_BLOCKER_LENGTH)}\u2026>`;
+}
+function obstructionFor(error, action) {
+  const message = error instanceof Error ? error.message : String(error);
+  const match = INTERCEPTS_POINTER.exec(message);
+  if (!match) return void 0;
+  const where = action.target ? ` on ${describeTarget(action.target)}` : "";
+  return new ActionError(
+    `blocked: the ${action.action}${where} was NOT performed. The target was found and is visible, enabled and stable \u2014 nothing about it is wrong. ${truncateBlocker(match[1])} is on top of it and received the pointer event instead. Something is covering the page: find it in the snapshot \u2014 a dialog, a cookie banner, an onboarding overlay \u2014 and dismiss it first, using its own close or accept control, or by pressing Escape with no target. Then act on this target again. Choosing a different name for the same target cannot help.`
+  );
+}
 async function performAction(page, action, ctx) {
+  try {
+    return await performResolvedAction(page, action, ctx);
+  } catch (error) {
+    throw obstructionFor(error, action) ?? error;
+  }
+}
+async function performResolvedAction(page, action, ctx) {
   switch (action.action) {
     case "navigate": {
       const value = resolve(requireValue(action), ctx);
@@ -509,7 +530,10 @@ async function performAction(page, action, ctx) {
 var COMMIT_ACTIONS = /* @__PURE__ */ new Set(["click", "press"]);
 var COMMIT_KEYS = /* @__PURE__ */ new Set(["Enter", "NumpadEnter", " ", "Space", "Spacebar"]);
 var SOURCED_VALUE_ACTIONS = /* @__PURE__ */ new Set(["fill", "select"]);
-var ENV_PLACEHOLDER2 = /^\s*\{\{env\.[A-Za-z_][A-Za-z0-9_]*\}\}\s*$/;
+function referencesOnlyNamedVars(value, step) {
+  const named = new Set(referencedEnvVars(step));
+  return referencedEnvVars(value).every((name) => named.has(name));
+}
 function normalise(text) {
   return text.replace(/\s+/g, " ").trim().toLowerCase();
 }
@@ -545,7 +569,16 @@ var StepRecovery = class {
    * step — which is also the whole of the "does not cross steps" rule.
    */
   readable;
+  /**
+   * The step exactly as written, for deciding which `{{env.*}}` variables it
+   * names (design env-placeholder-must-be-named, D3). Kept alongside the
+   * normalised copy rather than derived from it: `readable` is lowercased, and
+   * environment variable names are case-sensitive, so `TOKEN` and `token` must
+   * stay distinguishable in the one place that decides which secret gets typed.
+   */
+  step;
   constructor(step) {
+    this.step = step;
     this.readable = [normalise(step)];
   }
   /**
@@ -614,7 +647,11 @@ var StepRecovery = class {
     if (!SOURCED_VALUE_ACTIONS.has(action.action)) return void 0;
     const value = action.value;
     if (!value) return void 0;
-    if (ENV_PLACEHOLDER2.test(value)) return void 0;
+    if (!referencesOnlyNamedVars(value, this.step)) {
+      const named = referencedEnvVars(this.step);
+      return `refused: the value "${value}" was NOT typed, because this step does not reference that {{env.*}} variable. ${named.length > 0 ? `This step references ${named.map((n) => `{{env.${n}}}`).join(", ")}.` : "This step references no environment variable."} A placeholder is a source only when the step names it \u2014 otherwise the test never asked for that secret to be typed here. Use a value this step supplies, or fail the step and say it supplies none.`;
+    }
+    if (referencedEnvVars(value).length > 0) return void 0;
     const needle = normalise(value);
     if (needle === "") return void 0;
     if (this.readable.some((source) => source.includes(needle))) return void 0;
@@ -1305,6 +1342,7 @@ Rules:
 - Return "done" when the current step's outcome holds \u2014 including when it already held before you acted, or was achieved by your previous action. "Already true" is done, never failure. Do not return "done" for work belonging to later steps.
 - Return "fail" only when the step's outcome cannot be reached: the element is still absent after retries, the page cannot support the step, or an error blocks progress. Never return "fail" because the work appears to have been done already.
 - If your previous action errored, re-read the fresh snapshot and choose an alternative element or approach. Do not repeat the exact same failing action.
+- An action reported as "blocked" is the exception to that rule: it means another element is on top of your target, not that you picked the wrong target. Re-targeting cannot fix it. Whatever is covering the page is in the snapshot \u2014 a dialog, a cookie banner, an onboarding overlay \u2014 so dismiss that first, with its own close or accept control, or by pressing Escape with no target, and then act on your original target again. Overlays can be stacked: clearing one may reveal another, and that is progress, not failure.
 - Never invent a value. A value you type must come from the step, from the page, or from an {{env.*}} placeholder. This one is enforced, not merely asked: a fill or select whose value is in none of those is refused and not performed. If a step needs a value it does not give you, that is a failing step, not a gap for you to fill in.
 - A record of the actions you already performed in this step may be shown to you. It is the ground truth about what happened, even when the page no longer shows it: a form that submitted successfully and came back empty looks exactly like one you never submitted. Do not redo work that record says you already did.
 - \`***\` in a snapshot is a redacted secret \u2014 a password, token or key deliberately withheld from you. Seeing it is expected and is not a problem. A field showing \`***\` after you filled it from an {{env.VAR}} placeholder means the fill worked; treat that as success and move on. Never retry a fill because its value is redacted, and never report failure because a value was withheld.
@@ -1398,19 +1436,49 @@ function plannerUserPrompt(input) {
 
 // src/llm/schemas.ts
 import { z as z2 } from "zod";
+function absentAsNull(schema) {
+  return schema.nullable().transform((value) => value ?? void 0);
+}
+var ACTION_NAMES = ["navigate", "click", "fill", "press", "select", "assert", "done", "fail"];
 var agentActionSchema = z2.object({
-  action: z2.enum(["navigate", "click", "fill", "press", "select", "assert", "done", "fail"]).describe("The next browser action to perform for the current step."),
-  target: z2.object({
-    role: z2.string().optional().describe('ARIA role of the target element, e.g. "button", "link", "textbox".'),
-    name: z2.string().optional().describe("Accessible name of the target element, exactly as shown in the snapshot."),
-    text: z2.string().optional().describe("Visible text of the target element, used as a fallback when no role matches.")
-  }).optional().describe("Element to act on, resolved from the accessibility snapshot. Omit for navigate/done/fail."),
-  value: z2.string().optional().describe(
-    'Action payload: URL/path for navigate, text for fill, key for press (e.g. "Enter"), option label for select.'
+  action: z2.enum(ACTION_NAMES).describe("The next browser action to perform for the current step."),
+  target: absentAsNull(
+    z2.object({
+      role: absentAsNull(
+        z2.string().describe('ARIA role of the target element, e.g. "button", "link", "textbox".')
+      ),
+      name: absentAsNull(
+        z2.string().describe("Accessible name of the target element, exactly as shown in the snapshot.")
+      ),
+      text: absentAsNull(
+        z2.string().describe("Visible text of the target element, used as a fallback when no role matches.")
+      )
+    })
+  ).describe("Element to act on, resolved from the accessibility snapshot. Null for navigate/done/fail."),
+  value: absentAsNull(
+    z2.string().describe(
+      'Action payload: URL/path for navigate, text for fill, key for press (e.g. "Enter"), option label for select.'
+    )
   ),
   reasoning: z2.string().describe("One sentence explaining why this action moves the step forward."),
-  expectation: z2.string().optional().describe("For assert: the condition the current page snapshot must satisfy.")
+  expectation: absentAsNull(
+    z2.string().describe("For assert: the condition the current page snapshot must satisfy.")
+  )
+});
+var parsedAgentActionSchema = z2.object({
+  action: z2.enum(ACTION_NAMES),
+  target: z2.object({
+    role: z2.string().optional(),
+    name: z2.string().optional(),
+    text: z2.string().optional()
+  }).optional(),
+  value: z2.string().optional(),
+  reasoning: z2.string(),
+  expectation: z2.string().optional()
 });
+function parseAgentAction(value) {
+  return parsedAgentActionSchema.safeParse(value);
+}
 var assertJudgmentSchema = z2.object({
   pass: z2.boolean().describe("Whether the snapshot satisfies the expectation."),
   reason: z2.string().describe("One sentence explaining the judgment.")
@@ -1431,9 +1499,24 @@ var MalformedModelOutputError = class extends Error {
     this.name = "MalformedModelOutputError";
   }
 };
+var MAX_PROVIDER_DETAIL = 300;
+function withProviderDetail(error) {
+  if (!(error instanceof Error)) return error;
+  const body = error.responseBody;
+  if (typeof body !== "string" || body.length === 0) return error;
+  const detail = body.replace(/\s+/g, " ").trim().slice(0, MAX_PROVIDER_DETAIL);
+  if (error.message.includes(detail)) return error;
+  error.message = `${error.message} \u2014 provider said: ${detail}`;
+  return error;
+}
 async function countedGenerate(generate, budget, options) {
   budget.check();
-  const result = await generate(options);
+  let result;
+  try {
+    result = await generate(options);
+  } catch (error) {
+    throw withProviderDetail(error);
+  }
   budget.record(result.usage);
   return result;
 }
@@ -1446,7 +1529,7 @@ function createBrain(model, generate = generateObject, budget) {
         system: agentSystemPrompt(),
         prompt: agentUserPrompt(input)
       });
-      const parsed = agentActionSchema.safeParse(result.object);
+      const parsed = parseAgentAction(result.object);
       if (!parsed.success) {
         throw new MalformedModelOutputError(
           `Model returned an invalid action: ${parsed.error.issues[0]?.message ?? "unknown"}`
@@ -1459,7 +1542,8 @@ function createBrain(model, generate = generateObject, budget) {
         model,
         schema: assertJudgmentSchema,
         system: assertSystemPrompt(),
-        prompt: assertUserPrompt(step, expectation, snapshot, stepHistory)
+        prompt: assertUserPrompt(step, expectation, snapshot, stepHistory),
+        temperature: 0
       });
       const parsed = assertJudgmentSchema.safeParse(result.object);
       if (!parsed.success) {
@@ -1544,7 +1628,7 @@ function createModel(llm, env = process.env) {
 }
 
 // src/planner.ts
-import { writeFile as writeFile3 } from "fs/promises";
+import { mkdir as mkdir4, writeFile as writeFile3 } from "fs/promises";
 import path6 from "path";
 import { stringify } from "yaml";
 
@@ -1671,16 +1755,21 @@ async function generateForRoute(page, options) {
   return { ...generated, routes: [route] };
 }
 async function writeDraft(cwd, draft, meta) {
-  const file = path6.join(cwd, TESTS_RELATIVE_DIR, `${routeToSlug(meta.route)}.yaml`);
+  const dir = path6.join(cwd, TESTS_RELATIVE_DIR);
+  const file = path6.join(dir, `${routeToSlug(meta.route)}.yaml`);
   try {
+    await mkdir4(dir, { recursive: true });
     await writeFile3(file, renderTestYaml(draft, meta), { flag: "wx" });
   } catch (error) {
-    if (error.code === "EEXIST") {
+    const err = error;
+    if (err.code === "EEXIST" && err.syscall !== "mkdir") {
       throw new PlannerError(
         `Refusing to overwrite ${path6.relative(cwd, file)} (route ${meta.route}). Delete or rename it to regenerate.`
       );
     }
-    throw error;
+    throw new PlannerError(
+      `Cannot write draft to ${file}: ${fsReason(error)}. Check that ${path6.dirname(file)} is a directory you can write to, not a file.`
+    );
   }
   return file;
 }
@@ -1841,7 +1930,7 @@ function formatSpendLine(spend) {
 import path9 from "path";
 
 // src/report/html.ts
-import { mkdir as mkdir4, readFile as readFile4, writeFile as writeFile4 } from "fs/promises";
+import { mkdir as mkdir5, readFile as readFile4, writeFile as writeFile4 } from "fs/promises";
 import path7 from "path";
 var ESCAPES = {
   "&": "&amp;",
@@ -2006,7 +2095,7 @@ ${detail}
 }
 async function writeHtml(file, html) {
   try {
-    await mkdir4(path7.dirname(file), { recursive: true });
+    await mkdir5(path7.dirname(file), { recursive: true });
     await writeFile4(file, html, "utf8");
     return file;
   } catch (error) {
@@ -2017,7 +2106,7 @@ async function writeHtml(file, html) {
 }
 
 // src/report/junit.ts
-import { mkdir as mkdir5, writeFile as writeFile5 } from "fs/promises";
+import { mkdir as mkdir6, writeFile as writeFile5 } from "fs/promises";
 import path8 from "path";
 var ESCAPES2 = {
   "&": "&amp;",
@@ -2085,7 +2174,7 @@ ${reason}` : reason;
 }
 async function writeJUnit(file, xml) {
   try {
-    await mkdir5(path8.dirname(file), { recursive: true });
+    await mkdir6(path8.dirname(file), { recursive: true });
     await writeFile5(file, xml, "utf8");
     return file;
   } catch (error) {
@@ -2990,7 +3079,7 @@ function parsePositiveNumber(flag) {
   };
 }
 var program = new Command();
-program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.14.0");
+program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.16.0");
 program.command("init").description("Scaffold .blastproof/ (config, tests, sample tests) in the current directory").action(async () => {
   try {
     const result = await initProject(process.cwd());