blastproof 0.7.0 → 0.9.0

package/README.md

@@ -113,7 +113,7 @@ jobs:
 
       - run: npm start &          # however your app boots
 
-      - uses: hamc/blastproof@v0.7.0
+      - uses: hamc/blastproof@v0.9.0
         with:
           version: '0.6.0'        # pin both when this gates merges
           api-key: ${{ secrets.ANTHROPIC_API_KEY }}
@@ -167,6 +167,22 @@ blastproof plan --base main --dry-run                 # affected routes no test
 
 They report affected routes, files nobody has classified, and affected routes no test covers — a coverage-gap report with an exit code, useful even on a repo whose suite is Playwright or Cypress.
 
+## Running tests at once
+
+Tests run one at a time by default. Raise it when your tests can stand it:
+
+```yaml
+concurrency: 4
+```
+
+or `blastproof run --concurrency 4` for a single invocation. On this repository's own suite that takes a run from 156s to 68s — **2.3× faster, for the same 81 model calls.** Parallelism buys wall-clock, not spend.
+
+**The default is 1 on purpose, and raising it is your call to make.** Other test runners default to parallel because their tests are isolated by construction — separate processes, separate fixtures. These are journeys driven against **one running application**, so two tests can see each other's data. A suite is safe to parallelise when its tests do not write state that another test reads.
+
+The test in this repository's own suite that could not run beside itself is a good shape to recognise: it adds a note and then asserts *"one note on file"*. It writes shared server state, and it asserts on a global count. Either alone is a warning; together they mean the test's verdict depends on nothing else touching the application at that moment.
+
+Two practical notes. Four concurrent journeys are four times the traffic against whatever you pointed at — usually fine for a development instance, worth knowing for a shared one. And with several model calls in flight, a `budget:` limit can overshoot by up to the concurrency rather than by a single call, since the calls already sent are allowed to finish.
+
 ## Bounding a run
 
 Nothing stops a run by default. `budget:` puts a ceiling on `run`, `plan` and `test` alike — every model call any of them makes is counted:
@@ -182,7 +198,16 @@ Each limit is optional; with none set, nothing binds. They count **calls and tok
 
 Exhausting a budget **stops the run; it does not fail a test.** Running out of quota says nothing about the code under review. Unreached tests are reported as `not run`, a third state excluded from the score entirely, and the process exits 1 unconditionally — `--min-score` cannot rescue it, because the tests that finished are whichever ran first, not a representative sample.
 
-`--dry-run` reports the ceiling before you spend anything.
+**Every run reports what it spent**, so you can size a limit from experience rather than guesswork:
+
+```
+Spent: 82 model call(s), 115407 token(s)
+Score: 100
+```
+
+The figures also land in the JUnit report as `llm_calls` and `llm_tokens`, beside `score`, so a pipeline can trend cost without scraping output. A run stopped by its own budget reports the spend too — that is the case where the number is least guessable. Where a provider reports no token usage, the line says so rather than showing zero.
+
+`--dry-run` reports the ceiling before you spend anything. Read it as a maximum and nothing more: for this repository's own suite it says 735 calls where a real run spends 82. Size a budget from what your runs actually report, not from the ceiling.
 
 ## Testing behind a login
 
@@ -241,7 +266,9 @@ The key itself is never read from a `BLASTPROOF_*` variable — you name *which*
 
 The application under test is not trusted input: its page content reaches the model, so a page that controls its own accessible text can try to influence the agent. Two things constrain that.
 
-**The agent cannot leave your application.** `navigate` is bounded by `base_url`'s origin; an app spanning hosts declares them in `allowed_origins:`. This is enforced by comparison, not by asking the model nicely.
+**The agent cannot leave your application.** The boundary is `base_url`'s origin plus whatever `allowed_origins:` declares, and it constrains where the page **is**, not only where an action asked to go. A `navigate` outside it is refused before the request; a page that ends up outside it any other way — a redirect, a link to another host, a script setting the location — fails the step, and its content is never sent to the model. Enforced by comparison, not by asking the model nicely.
+
+If your application legitimately spans hosts (an identity provider, a hosted payment step), declare them. A suite that was quietly walking onto a foreign page will now fail and name the origin to add.
 
 **Your secrets stay out of prompts.** `{{env.*}}` placeholders survive intact and are substituted at the moment of typing. Every value your tests or auth recipe reference is redacted from everything else crossing into a prompt — page snapshots included — in literal and percent-encoded form. Redaction matches known values, so treat it as a strong default rather than a guarantee against a hostile app.
 

package/dist/cli.js

@@ -210,6 +210,7 @@ var RunBudget = class {
   startedAt;
   calls = 0;
   tokens = 0;
+  callsWithUsage = 0;
   constructor(options = {}) {
     this.maxCalls = options.maxCalls;
     this.maxTokens = options.maxTokens;
@@ -248,7 +249,25 @@ var RunBudget = class {
   /** Records what a completed model call spent. */
   record(usage) {
     this.calls += 1;
-    if (usage?.totalTokens !== void 0) this.tokens += usage.totalTokens;
+    if (usage?.totalTokens !== void 0) {
+      this.tokens += usage.totalTokens;
+      this.callsWithUsage += 1;
+    }
+  }
+  /**
+   * What this budget has been spent on, for every surface that reports it
+   * (design report-what-it-spent, D1). One method rather than four getters read
+   * separately: the console, the JUnit report and the HTML report take the same
+   * numbers from the same object, so they cannot disagree about what a run cost.
+   */
+  spend() {
+    return {
+      calls: this.calls,
+      maxCalls: this.maxCalls,
+      tokens: this.tokens,
+      callsWithUsage: this.callsWithUsage,
+      maxTokens: this.maxTokens
+    };
   }
 };
 function estimateMaxModelCalls(tests, maxIterationsPerStep, maxRetriesPerStep) {
@@ -336,14 +355,33 @@ var ActionError = class extends Error {
     this.name = "ActionError";
   }
 };
-function assertAllowedOrigin(url, ctx) {
-  const allowed = /* @__PURE__ */ new Set([new URL(ctx.baseUrl).origin]);
-  for (const origin of ctx.allowedOrigins ?? []) {
+var BLANK_PAGE = "about:blank";
+function allowedOriginsFor(baseUrl, allowedOrigins) {
+  const allowed = /* @__PURE__ */ new Set([new URL(baseUrl).origin]);
+  for (const origin of allowedOrigins ?? []) {
     allowed.add(new URL(origin).origin);
   }
-  if (!allowed.has(url.origin)) {
+  return allowed;
+}
+function isOriginAllowed(url, allowed) {
+  if (url === BLANK_PAGE) return true;
+  let origin;
+  try {
+    origin = new URL(url).origin;
+  } catch {
+    return false;
+  }
+  if (origin === "null") return false;
+  return allowed.has(origin);
+}
+function describeBoundary(allowed) {
+  return [...allowed].join(" or ");
+}
+function assertAllowedOrigin(url, ctx) {
+  const allowed = allowedOriginsFor(ctx.baseUrl, ctx.allowedOrigins);
+  if (!isOriginAllowed(url.toString(), allowed)) {
     throw new ActionError(
-      `Refusing to navigate outside the application: ${url.origin} is not ${[...allowed].join(" or ")}. Add it to allowed_origins in .blastproof/config.yaml if the app legitimately spans hosts.`
+      `Refusing to navigate outside the application: ${url.origin} is not ${describeBoundary(allowed)}. Add it to allowed_origins in .blastproof/config.yaml if the app legitimately spans hosts.`
     );
   }
 }
@@ -548,6 +586,7 @@ async function executeTest(page, test, options) {
     onEvent({ type: "action", index, action: maskedAction, result: mask(result) });
   };
   let failure;
+  const boundary = allowedOriginsFor(baseUrl, allowedOrigins);
   await page.goto(new URL(baseUrl).toString());
   for (let index = 0; index < allSteps.length; index++) {
     const { step, setup } = allSteps[index];
@@ -564,6 +603,11 @@ async function executeTest(page, test, options) {
           throw new StepFailure(`step exceeded ${maxIterationsPerStep} actions without completing`);
         }
         await waitForSettled(page);
+        if (!isOriginAllowed(page.url(), boundary)) {
+          throw new StepFailure(
+            `The page left the application: ${page.url()} is outside ${describeBoundary(boundary)}. Add it to allowed_origins in .blastproof/config.yaml if the app legitimately spans hosts.`
+          );
+        }
         const snap = await takeSnapshot(page);
         let action;
         try {
@@ -922,6 +966,13 @@ var configSchema = z.object({
   allowed_origins: z.array(z.string().url()).optional(),
   auth: authSchema.optional(),
   max_retries_per_step: z.number().int().min(1).default(3),
+  /**
+   * How many tests may run at once (design tests-in-parallel, D1). Defaults to
+   * 1 — tests are journeys driven against one running application, and whether
+   * two of them can run at the same time is a property of that application and
+   * those tests, not of the runner. Opted into by the person who knows.
+   */
+  concurrency: z.number().int().min(1, "concurrency must be at least 1").default(1),
   budget: budgetSchema.optional()
 });
 var ENV_OVERRIDES = {
@@ -1601,6 +1652,43 @@ function printPreflightFailures(failures) {
   for (const failure of failures) console.error(`  - ${failure}`);
 }
 
+// src/report/score.ts
+var WEIGHTS = { P0: 3, P1: 2, P2: 1 };
+var DEFAULT_WEIGHT = WEIGHTS.P1;
+function computeScore(results) {
+  let total = 0;
+  let passed = 0;
+  for (const result of results) {
+    if (result.status === "not-run") continue;
+    const weight = WEIGHTS[result.priority] ?? DEFAULT_WEIGHT;
+    total += weight;
+    if (result.status === "passed") passed += weight;
+  }
+  if (total === 0) return 100;
+  return Math.round(100 * passed / total);
+}
+function formatScoreLine(score, results, threshold) {
+  if (results.length === 0) {
+    const base = "Score: 100 (no tests executed)";
+    return threshold === void 0 ? base : `${base} \u2014 min-score ${threshold}: pass`;
+  }
+  if (threshold === void 0) return `Score: ${score}`;
+  return score >= threshold ? `Score: ${score} \u2014 min-score ${threshold}: pass` : `Score: ${score} \u2014 min-score ${threshold}: FAIL (below threshold)`;
+}
+function formatIncompleteLine(score, reason) {
+  return `Run incomplete: ${reason}
+Score over executed tests: ${score} (not a verdict \u2014 exit code 1 regardless of --min-score)`;
+}
+function formatSpendLine(spend) {
+  const calls = spend.maxCalls === void 0 ? `${spend.calls} model call(s)` : `${spend.calls} of ${spend.maxCalls} model call(s)`;
+  if (spend.callsWithUsage === 0) {
+    return `Spent: ${calls}; token usage not reported by the provider`;
+  }
+  const tokens = spend.maxTokens === void 0 ? `${spend.tokens} token(s)` : `${spend.tokens} of ${spend.maxTokens} token(s)`;
+  const coverage = spend.callsWithUsage < spend.calls ? ` (tokens reported by ${spend.callsWithUsage} of ${spend.calls} call(s))` : "";
+  return `Spent: ${calls}, ${tokens}${coverage}`;
+}
+
 // src/commands/run.ts
 import path9 from "path";
 
@@ -1745,7 +1833,7 @@ async function renderHtml(results, skipped, meta) {
 <body>
 <main>
   <h1>blastproof report</h1>
-  <p class="sub">${escapeHtml(generatedAt)} \xB7 ${seconds(meta.durationMs)}</p>
+  <p class="sub">${escapeHtml(generatedAt)} \xB7 ${seconds(meta.durationMs)}${meta.spend ? ` \xB7 ${escapeHtml(formatSpendLine(meta.spend))}` : ""}</p>
 
 ${banner}  <section class="score">
     <b>${meta.score}</b>
@@ -1799,6 +1887,11 @@ function renderJUnit(results, skipped, meta) {
     `<testsuite name="blastproof" tests="${results.length + skipped.length}" failures="${failures}" skipped="${skipped.length + notRun.length}" time="${seconds2(meta.durationMs)}">`,
     "  <properties>",
     `    <property name="score" value="${meta.score}"/>`,
+    ...meta.spend ? [`    <property name="llm_calls" value="${meta.spend.calls}"/>`] : [],
+    // Omitted rather than emitted as zero when no call reported usage: a
+    // property carrying 0 would be read by a pipeline as "this run spent no
+    // tokens", which is a different claim from "the provider did not say".
+    ...meta.spend && meta.spend.callsWithUsage > 0 ? [`    <property name="llm_tokens" value="${meta.spend.tokens}"/>`] : [],
     ...meta.incomplete !== void 0 ? [
       '    <property name="incomplete" value="true"/>',
       `    <property name="incomplete_reason" value="${escapeXml(meta.incomplete)}"/>`
@@ -1842,32 +1935,26 @@ async function writeJUnit(file, xml) {
   return file;
 }
 
-// src/report/score.ts
-var WEIGHTS = { P0: 3, P1: 2, P2: 1 };
-var DEFAULT_WEIGHT = WEIGHTS.P1;
-function computeScore(results) {
-  let total = 0;
-  let passed = 0;
-  for (const result of results) {
-    if (result.status === "not-run") continue;
-    const weight = WEIGHTS[result.priority] ?? DEFAULT_WEIGHT;
-    total += weight;
-    if (result.status === "passed") passed += weight;
-  }
-  if (total === 0) return 100;
-  return Math.round(100 * passed / total);
-}
-function formatScoreLine(score, results, threshold) {
-  if (results.length === 0) {
-    const base = "Score: 100 (no tests executed)";
-    return threshold === void 0 ? base : `${base} \u2014 min-score ${threshold}: pass`;
+// src/runner/pool.ts
+async function runWithConcurrency(items, concurrency, run) {
+  if (concurrency < 1) throw new RangeError(`concurrency must be at least 1, got ${concurrency}`);
+  const results = new Array(items.length);
+  let next = 0;
+  const worker = async () => {
+    while (true) {
+      const index = next++;
+      if (index >= items.length) return;
+      results[index] = await run(items[index], index);
+    }
+  };
+  const workers = [];
+  for (let i = 0; i < Math.min(concurrency, items.length); i++) {
+    workers.push(worker());
   }
-  if (threshold === void 0) return `Score: ${score}`;
-  return score >= threshold ? `Score: ${score} \u2014 min-score ${threshold}: pass` : `Score: ${score} \u2014 min-score ${threshold}: FAIL (below threshold)`;
-}
-function formatIncompleteLine(score, reason) {
-  return `Run incomplete: ${reason}
-Score over executed tests: ${score} (not a verdict \u2014 exit code 1 regardless of --min-score)`;
+  const settled = await Promise.allSettled(workers);
+  const failure = settled.find((outcome) => outcome.status === "rejected");
+  if (failure && failure.status === "rejected") throw failure.reason;
+  return results;
 }
 
 // src/runner/selection.ts
@@ -1958,19 +2045,19 @@ function notRunResults(tests, reason) {
     durationMs: 0
   }));
 }
-function printEvent(event) {
+function printEvent(event, write = console.log) {
   switch (event.type) {
     case "step-start":
-      console.log(`  ${event.setup ? "(setup) " : ""}step ${event.index + 1}/${event.total}: ${event.step}`);
+      write(`  ${event.setup ? "(setup) " : ""}step ${event.index + 1}/${event.total}: ${event.step}`);
       break;
     case "action": {
       const { action, result } = event;
-      console.log(`    -> ${describeAction(action)} :: ${result}`);
+      write(`    -> ${describeAction(action)} :: ${result}`);
       break;
     }
     case "step-end":
       if (event.status === "failed") {
-        console.log(`    X step failed: ${event.reason ?? "unknown reason"}`);
+        write(`    X step failed: ${event.reason ?? "unknown reason"}`);
       }
       break;
   }
@@ -2015,7 +2102,7 @@ ${notRun.length} test(s) not run (run stopped by its budget or deadline):`);
     for (const r of notRun) console.log(`  - ${r.summary} (${r.file})`);
   }
 }
-async function runOne(browser, test, config, sessionDir, session, runMask, budget) {
+async function runOne(browser, test, config, sessionDir, session, runMask, budget, write) {
   const brain = createBrain(createModel(config.llm).model, void 0, budget);
   let resolved;
   const mask = runMask;
@@ -2050,7 +2137,7 @@ async function runOne(browser, test, config, sessionDir, session, runMask, budge
       timeoutMs: config.browser.timeout_ms,
       maxSnapshotLines: config.browser.max_snapshot_lines,
       mask: (text) => mask.mask(text),
-      onEvent: printEvent
+      onEvent: (event) => printEvent(event, write)
     });
   } finally {
     await context.close();
@@ -2081,8 +2168,9 @@ function printImpactReport(impact, selection, cwd) {
   }
   console.log("---------------------------------------------------------------");
 }
-async function finalize(results, skipped, options, sessionDir, durationMs, impact, incomplete) {
+async function finalize(results, skipped, options, sessionDir, durationMs, impact, incomplete, spend) {
   if (results.length > 0) printSummary(results);
+  if (spend) console.log(formatSpendLine(spend));
   const score = computeScore(results);
   console.log(
     incomplete ? formatIncompleteLine(score, incomplete.message) : formatScoreLine(score, results, options.minScore)
@@ -2093,7 +2181,8 @@ async function finalize(results, skipped, options, sessionDir, durationMs, impac
       score,
       durationMs,
       cwd: options.cwd,
-      incomplete: incomplete?.message
+      incomplete: incomplete?.message,
+      spend
     });
     await writeJUnit(target, xml);
     console.log(`JUnit report: ${path9.relative(options.cwd, target)}`);
@@ -2105,7 +2194,8 @@ async function finalize(results, skipped, options, sessionDir, durationMs, impac
       durationMs,
       minScore: options.minScore,
       cwd: options.cwd,
-      incomplete: incomplete?.message
+      incomplete: incomplete?.message,
+      spend
     });
     await writeHtml(target, html);
     console.log(`HTML report: ${path9.relative(options.cwd, target)}`);
@@ -2292,21 +2382,38 @@ ${results.length} test file(s) could not be parsed:`);
     if (incomplete) {
       results.push(...notRunResults(selected, incomplete.message));
     } else {
-      for (let i = 0; i < selected.length; i++) {
-        const test = selected[i];
-        console.log(`
-> ${test.summary} [${test.priority}] (${path9.relative(options.cwd, test.path)})`);
+      const concurrency = options.concurrency ?? config.concurrency;
+      const streaming = concurrency === 1;
+      let stoppedBy;
+      const outcomes = await runWithConcurrency(selected, concurrency, async (test, index) => {
+        if (stoppedBy) return { index, stopped: stoppedBy };
+        const lines = [];
+        const header = `
+> ${test.summary} [${test.priority}] (${path9.relative(options.cwd, test.path)})`;
+        const write = streaming ? console.log : (line) => lines.push(line);
+        if (streaming) console.log(header);
+        else lines.push(header);
         try {
           budget.check();
-          results.push(await runOne(browser, test, config, sessionDir, session, runMask, budget));
+          const result = await runOne(browser, test, config, sessionDir, session, runMask, budget, write);
+          if (!streaming) for (const line of lines) console.log(line);
+          return { index, result };
         } catch (error) {
           if (error instanceof BudgetExhaustedError) {
-            incomplete = error;
-            results.push(...notRunResults(selected.slice(i), error.message));
-            break;
+            stoppedBy ??= error;
+            if (!streaming) for (const line of lines) console.log(line);
+            return { index, stopped: error };
           }
           throw error;
         }
+      });
+      for (const outcome of outcomes) {
+        if (outcome.result) {
+          results.push(outcome.result);
+          continue;
+        }
+        incomplete ??= outcome.stopped;
+        results.push(...notRunResults([selected[outcome.index]], outcome.stopped.message));
       }
     }
   } finally {
@@ -2319,7 +2426,13 @@ ${results.length} test file(s) could not be parsed:`);
     sessionDir,
     Date.now() - startedAt,
     impact,
-    incomplete
+    incomplete,
+    // The command that constructed the budget is the one that reports it
+    // (design report-what-it-spent, D2). `test` hands one budget to both of its
+    // phases deliberately, so reporting at the point of use rather than the
+    // point of ownership would print the running total twice for a single
+    // allowance, the second line silently including the first.
+    options.budget === void 0 ? budget.spend() : void 0
   );
 }
 
@@ -2549,6 +2662,7 @@ ${renderTestYaml(draft, { route, base })}`);
       for (const route of notAttempted) console.log(`  ${route}`);
     }
   }
+  if (options.budget === void 0) console.log(formatSpendLine(budget.spend()));
   console.log("---------------------------------------------------------------");
   return incomplete || failed.length > 0 ? EXIT_FAILED : EXIT_OK;
 }
@@ -2593,6 +2707,8 @@ async function testCommand(options) {
     budget
   });
   if (planCode === EXIT_USAGE) return EXIT_USAGE;
+  console.log(`
+${formatSpendLine(budget.spend())}`);
   console.log(
     "\nDrafts are not executed and do not affect the score \u2014 review them before they join the suite."
   );
@@ -2635,7 +2751,7 @@ function parsePositiveNumber(flag) {
   };
 }
 var program = new Command();
-program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.7.0");
+program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.9.0");
 program.command("init").description("Scaffold .blastproof/ (config, tests, sample tests) in the current directory").action(async () => {
   try {
     const result = await initProject(process.cwd());
@@ -2651,6 +2767,10 @@ program.command("run").description("Discover and run all tests under .blastproof
   "require a weighted score of at least n (0-100); replaces the all-must-pass rule",
   parseMinScore
 ).option("--junit [path]", "write a JUnit XML report (default: .blastproof/reports/<session>/junit.xml)").option("--html [path]", "write a self-contained HTML report (default: .blastproof/reports/<session>/report.html)").option("--fail-on-unmapped", "fail when a changed file matches no routes: or ignore: glob").option(
+  "--concurrency <n>",
+  "run this many tests at once (overrides config; default 1 \u2014 see the README on when this is safe)",
+  parsePositiveInt("--concurrency")
+).option(
   "--max-llm-calls <n>",
   "stop the run after this many model calls, reported as incomplete (overrides config)",
   parsePositiveInt("--max-llm-calls")
@@ -2678,6 +2798,7 @@ program.command("run").description("Discover and run all tests under .blastproof
         junit: options.junit,
         html: options.html,
         failOnUnmapped: options.failOnUnmapped,
+        concurrency: options.concurrency,
         maxLlmCalls: options.maxLlmCalls,
         maxTokens: options.maxTokens,
         maxDuration: options.maxDuration