blastproof 0.2.0 → 0.2.2

package/README.md

@@ -25,21 +25,26 @@ git diff → impact mapping → test generation → agentic execution → report
 ```bash
 npm install -g blastproof                    # requires Node.js >= 20.19
 npx playwright install --with-deps chromium  # one-time browser download
+
 cd your-project
-blastproof init
+blastproof init                              # scaffolds .blastproof/
+# start your app, then point base_url at it in .blastproof/config.yaml
 export ANTHROPIC_API_KEY=...                 # or OPENAI_API_KEY, or a local Ollama model
 blastproof run                               # runs .blastproof/tests/**/*.yaml agentically
 ```
 
+`init` scaffolds one smoke test that works against any app, plus a commented login template you rename and edit once it describes *your* login. Nothing is written that assumes anything about your application.
+
 Published with [npm provenance](https://docs.npmjs.com/generating-provenance-statements), so the tarball is verifiably built from this repository.
 
-Try it locally without your own app — this repo ships a demo shop:
+Try it without your own app — the demo shop lives in this repository, so clone it:
 
 ```bash
-node examples/demo-app/serve.mjs 4173 &   # home, login and cart + promo-code pages
-blastproof init
+git clone https://github.com/hamc/blastproof && cd blastproof
+npm install && npm run build
+node examples/demo-app/serve.mjs 4173 &   # home, login, cart, checkout, orders
 export ANTHROPIC_API_KEY=...
-blastproof run
+node dist/cli.js run
 ```
 
 > **Status:** the pipeline is complete and published. `init`, `run` (including `--impacted`), `plan` and `test`, with authentication, JUnit and HTML reports, a merge gate, and a GitHub Action. Pre-1.0: the command surface may still change.
@@ -116,7 +121,7 @@ Drafts are **previews by default** — nothing touches disk until `--write`, and
 
 Exit codes: 0 when every route generated (or nothing needed coverage), 1 when a route failed, 2 on usage/config/diff errors. A route that fails to load never aborts the others.
 
-**Known limitation:** a route behind authentication snapshots as the login wall, so its draft describes logging in rather than the feature. The `auth` config recipe is not applied by the planner yet — generate those routes after an auth session lands, or write them by hand.
+`plan` uses the same `auth:` recipe as `run`, so a route behind a login is drafted from the real page rather than from the login wall.
 
 ### Closing the coverage hole
 
@@ -205,7 +210,7 @@ jobs:
 
       - run: npm start &          # however your app boots
 
-      - uses: hamc/blastproof@v0.2.0
+      - uses: hamc/blastproof@v0.2.2
         with:
           api-key: ${{ secrets.ANTHROPIC_API_KEY }}
           base: ${{ github.event.pull_request.base.ref }}
@@ -217,8 +222,9 @@ Exit non-zero blocks the merge. Use the score in a later step:
 
 ```yaml
       - id: bp
-        uses: hamc/blastproof@v0.2.0
+        uses: hamc/blastproof@v0.2.2
         with:
+          version: '0.2.2'          # pin both when the result gates merges
           api-key: ${{ secrets.ANTHROPIC_API_KEY }}
           base: ${{ github.event.pull_request.base.ref }}
           min-score: '80'
@@ -267,7 +273,9 @@ allowed_origins:
 
 This is enforced by comparison, not by asking the model nicely, so it holds regardless of what the page says.
 
-**Your secrets never reach the model.** `{{env.*}}` placeholders stay intact all the way through the prompt and are substituted at the moment of typing. The model is told to pass them through unchanged. This matters because blastproof encourages pointing `llm.base_url` at a gateway you do not run — the credential now stays on your machine either way.
+**Your secrets stay out of the model's prompts.** `{{env.*}}` placeholders stay intact all the way through the prompt and are substituted at the moment of typing. Every value any test or the auth recipe references is also redacted from anything else crossing into a prompt — page snapshots included, since your app may render the credential itself — in both literal and percent-encoded form. This matters because blastproof encourages pointing `llm.base_url` at a gateway you do not run.
+
+Redaction matches known values, so it cannot anticipate every way a page might transform one before rendering it. Treat it as a strong default, not a guarantee against a hostile application.
 
 The system prompt also tells the model that page content is data under test and never an instruction to obey. That raises the cost of a casual injection and is **not** a security boundary — a determined one will get past prompt wording. The origin constraint is the boundary; treat the rest as hygiene, and do not point blastproof at an application you would not run locally.
 
@@ -356,6 +364,12 @@ Note the last one names *which variable* holds your key — the key itself is ne
 
 ## Development
 
+Built with AI assistance, using spec-driven development throughout: every change began as a written
+proposal with its design rationale, and those documents are kept rather than discarded — `openspec/`
+holds the reasoning behind each decision, including the alternatives that were rejected and why. The
+`.claude/`, `.cursor/` and `.opencode/` directories are agent configuration for the tools used to
+build it; ignore them unless you are contributing with one.
+
 This project uses **spec-driven development** via [OpenSpec](https://github.com/Fission-AI/OpenSpec). See [`AGENTS.md`](./AGENTS.md) for architecture, conventions and the contribution workflow — every change starts with an OpenSpec proposal.
 
 ```bash

package/dist/cli.js

@@ -93,21 +93,25 @@ sessions/
 .auth-state.json
 auth.json
 `;
-var SAMPLE_LOGIN_TEST = `summary: Login with valid credentials succeeds
+var SAMPLE_LOGIN_TEMPLATE = `# TEMPLATE \u2014 rename to login.yaml once these steps match your app.
+# Until then it is ignored: only .yaml/.yml files are discovered.
+#
+# Credentials come from the environment. Export them before running:
+#   export TEST_EMAIL=... TEST_PASSWORD=...
+# Substituted values are masked as *** everywhere and never reach the model.
+summary: Login with valid credentials succeeds
 priority: P0
 tags: [smoke, auth]
 routes: ["/login"]
-# This test exercises the login itself, so it must start signed out even when an
-# auth recipe is configured.
+# Exercises the login itself, so it must start signed out even when an auth
+# recipe is configured.
 auth: false
 steps:
   - navigate to /login
-  - fill the email field with demo@blastproof.dev
-  - fill the password field with demo123
+  - fill the email field with {{env.TEST_EMAIL}}
+  - fill the password field with {{env.TEST_PASSWORD}}
   - submit the login form
   - verify a welcome message is shown
-# Secrets tip: use {{env.VAR_NAME}} placeholders in steps (e.g. {{env.TEST_PASSWORD}})
-# and export the variable before running. Values are masked as *** in all output.
 `;
 async function exists(file) {
   try {
@@ -132,7 +136,7 @@ async function initProject(cwd = process.cwd()) {
   await writeIfAbsent(path.join(root, ".gitignore"), GITIGNORE, result);
   await writeIfAbsent(path.join(root, "config.yaml"), DEFAULT_CONFIG, result);
   await writeIfAbsent(path.join(root, "tests", "app-load.yaml"), SAMPLE_TEST, result);
-  await writeIfAbsent(path.join(root, "tests", "login.yaml"), SAMPLE_LOGIN_TEST, result);
+  await writeIfAbsent(path.join(root, "tests", "login.yaml.example"), SAMPLE_LOGIN_TEMPLATE, result);
   return result;
 }
 function formatInitGuidance(result) {
@@ -149,7 +153,7 @@ function formatInitGuidance(result) {
     "",
     "Next steps:",
     "  1. Set your LLM API key:  export ANTHROPIC_API_KEY=...   (or OPENAI_API_KEY; ollama needs no key)",
-    "  2. Point base_url at your app in .blastproof/config.yaml",
+    "  2. Start your app, then point base_url at it in .blastproof/config.yaml",
     "  3. Run your tests:        blastproof run",
     "",
     "First time with Playwright? Install the browser:  npx playwright install chromium"
@@ -211,9 +215,21 @@ var SecretsMask = class {
       if (value === void 0) {
         throw new MissingEnvError(name);
       }
-      this.secrets.add(value);
+      this.add(value);
     }
   }
+  /**
+   * Registers a value and the forms it takes on the way to a prompt. `navigate`
+   * reports a resolved URL, and `new URL()` percent-encodes — so a secret with a
+   * space stopped matching a literal search and passed through unmasked.
+   * This cannot cover every transform a page might apply; see the README.
+   */
+  add(value) {
+    if (!value) return;
+    this.secrets.add(value);
+    const encoded = encodeURIComponent(value);
+    if (encoded !== value) this.secrets.add(encoded);
+  }
   mask(text) {
     return maskSecrets(text, this.secrets);
   }
@@ -392,8 +408,8 @@ async function executeTest(page, test, options) {
           action = await brain.nextAction({
             step,
             isSetup: setup,
-            snapshot: snap,
-            lastResult,
+            snapshot: mask(snap),
+            lastResult: lastResult === void 0 ? void 0 : mask(lastResult),
             retriesLeft: maxRetries - failedAttempts,
             iterationsLeft: maxIterationsPerStep - iterations
           });
@@ -416,7 +432,7 @@ async function executeTest(page, test, options) {
         }
         if (action.action === "assert") {
           const expectation = action.expectation ?? action.reasoning;
-          const judgment = await brain.judge(expectation, snap);
+          const judgment = await brain.judge(mask(expectation), mask(snap));
           const result = judgment.pass ? `ok: assertion passed: ${judgment.reason}` : `assertion failed: ${judgment.reason}`;
           emitAction(index, action, result);
           if (judgment.pass) {
@@ -521,8 +537,7 @@ async function fromStorageState(file) {
   }
   return { storageState: parsed };
 }
-function fromStatic(auth) {
-  const mask = new SecretsMask();
+function fromStatic(auth, mask) {
   const session = {};
   if (auth.headers) {
     const headers = {};
@@ -539,7 +554,7 @@ function fromStatic(auth) {
     });
     session.storageState = { cookies, origins: [] };
   }
-  return { session, mask };
+  return session;
 }
 async function runJourney(...args) {
   try {
@@ -551,10 +566,8 @@ async function runJourney(...args) {
   }
 }
 async function fromSteps(options) {
-  const { auth, baseUrl, browser, brain, maxRetries, snapshot = defaultSnapshot, onEvent } = options;
+  const { auth, baseUrl, browser, brain, maxRetries, snapshot = defaultSnapshot, onEvent, mask } = options;
   const steps = auth.steps;
-  const mask = new SecretsMask();
-  for (const step of steps) mask.registerFrom(step);
   const context = await browser.newContext();
   try {
     const page = await context.newPage();
@@ -587,7 +600,7 @@ async function fromSteps(options) {
       );
     }
     if (auth.verify) {
-      const judgment = await brain.judge(auth.verify, await snapshot(page));
+      const judgment = await brain.judge(auth.verify, mask.mask(await snapshot(page)));
       if (!judgment.pass) {
         throw new AuthError(`Authentication could not be verified: ${mask.mask(judgment.reason)}`);
       }
@@ -620,7 +633,7 @@ async function authenticate(options) {
     return fromStorageState(path3.resolve(cwd, auth.storage_state));
   }
   if (auth.headers || auth.cookies) {
-    return fromStatic(auth).session;
+    return fromStatic(auth, options.mask);
   }
   const session = await fromSteps(options);
   if (auth.cache) await writeCached(cwd, session);
@@ -1164,7 +1177,7 @@ function renderTestYaml(draft, meta) {
   });
 }
 async function generateForRoute(page, options) {
-  const { route, baseUrl, changedFiles, brain, snapshot = defaultSnapshot } = options;
+  const { route, baseUrl, changedFiles, brain, mask, snapshot = defaultSnapshot } = options;
   const url = new URL(route, baseUrl).toString();
   try {
     await page.goto(url, { timeout: 3e4 });
@@ -1173,7 +1186,11 @@ async function generateForRoute(page, options) {
       `Cannot load ${url}: ${error instanceof Error ? error.message : String(error)}`
     );
   }
-  const generated = await brain.planTest({ route, snapshot: await snapshot(page), changedFiles });
+  const generated = await brain.planTest({
+    route,
+    snapshot: mask(await snapshot(page)),
+    changedFiles
+  });
   const leaked = findSecretLiterals(generated.steps);
   if (leaked.length > 0) {
     throw new PlannerError(
@@ -1489,12 +1506,21 @@ function applyUrlOverride(config, url) {
   }
   return { ...config, base_url: url };
 }
-function resolveSecretsAndSteps(test) {
+function buildRunMask(config, tests) {
   const mask = new SecretsMask();
+  for (const step of config.auth?.steps ?? []) mask.registerFrom(step);
+  for (const value of Object.values(config.auth?.headers ?? {})) mask.registerFrom(value);
+  for (const cookie of config.auth?.cookies ?? []) mask.registerFrom(cookie.value);
+  for (const test of tests) {
+    for (const step of [...test.setup ?? [], ...test.steps]) mask.registerFrom(step);
+  }
+  return mask;
+}
+function resolveSecretsAndSteps(test) {
   for (const step of [...test.setup ?? [], ...test.steps]) {
-    mask.registerFrom(step);
+    new SecretsMask().registerFrom(step);
   }
-  return { test, mask };
+  return { test };
 }
 function printEvent(event) {
   switch (event.type) {
@@ -1542,12 +1568,12 @@ X ${r.summary} (${r.file})`);
     if (r.screenshot) console.log(`  screenshot: ${r.screenshot}`);
   }
 }
-async function runOne(browser, test, config, sessionDir, session) {
+async function runOne(browser, test, config, sessionDir, session, runMask) {
   const brain = createBrain(createModel(config.llm).model);
   let resolved;
-  let mask;
+  const mask = runMask;
   try {
-    ({ test: resolved, mask } = resolveSecretsAndSteps(test));
+    ({ test: resolved } = resolveSecretsAndSteps(test));
   } catch (error) {
     if (error instanceof MissingEnvError) {
       return {
@@ -1627,22 +1653,24 @@ async function finalize(results, skipped, options, sessionDir, durationMs, impac
     await writeHtml(target, html);
     console.log(`HTML report: ${path9.relative(options.cwd, target)}`);
   }
-  const unclassified = options.failOnUnmapped ? impact?.unmappedFiles ?? [] : [];
-  if (unclassified.length > 0) {
-    console.error(
-      `error: ${unclassified.length} changed file(s) match no routes: or ignore: glob, so their blast radius is unknown:`
-    );
-    for (const file of unclassified) console.error(`  ${file}`);
-    console.error(
-      "Map them in routes: if they can affect a page, or list them in ignore: if they cannot."
-    );
-    return EXIT_FAILED;
-  }
+  if (reportUnclassified(options, impact)) return EXIT_FAILED;
   if (options.minScore !== void 0) {
     return score >= options.minScore ? EXIT_OK : EXIT_FAILED;
   }
   return results.some((result) => result.status === "failed") ? EXIT_FAILED : EXIT_OK;
 }
+function reportUnclassified(options, impact) {
+  const unclassified = options.failOnUnmapped ? impact?.unmappedFiles ?? [] : [];
+  if (unclassified.length === 0) return false;
+  console.error(
+    `error: ${unclassified.length} changed file(s) match no routes: or ignore: glob, so their blast radius is unknown:`
+  );
+  for (const file of unclassified) console.error(`  ${file}`);
+  console.error(
+    "Map them in routes: if they can affect a page, or list them in ignore: if they cannot."
+  );
+  return true;
+}
 function printDryRun(selected, baseUrl, cwd) {
   console.log(`
 Dry run: ${selected.length} test(s) selected, base_url=${baseUrl}`);
@@ -1664,6 +1692,12 @@ async function runCommand(options) {
   }
   const impacted = options.impacted ?? false;
   const base = options.base ?? "main";
+  if (options.failOnUnmapped && !impacted) {
+    console.error(
+      "error: --fail-on-unmapped classifies the files in a diff, so it requires --impacted."
+    );
+    return EXIT_USAGE;
+  }
   let impact;
   if (impacted) {
     let changedFiles;
@@ -1719,7 +1753,13 @@ async function runCommand(options) {
   }
   if (options.dryRun) {
     printDryRun(selected, config.base_url, options.cwd);
-    return EXIT_OK;
+    if (results.length > 0) {
+      console.error(`
+${results.length} test file(s) could not be parsed:`);
+      for (const broken of results) console.error(`  ${broken.file}: ${broken.reason ?? "invalid"}`);
+      return EXIT_FAILED;
+    }
+    return reportUnclassified(options, impact) ? EXIT_FAILED : EXIT_OK;
   }
   const sessionDir = path9.join(options.cwd, ".blastproof", "reports", sessionId());
   const startedAt = Date.now();
@@ -1742,6 +1782,7 @@ async function runCommand(options) {
   console.log(
     `blastproof run: ${selected.length} test(s), provider=${provider} model=${modelId}, base_url=${config.base_url}`
   );
+  const runMask = buildRunMask(config, parsed);
   const browser = await chromium.launch({ headless: config.browser.headless });
   try {
     let session;
@@ -1756,6 +1797,7 @@ async function runCommand(options) {
           browser,
           brain: createBrain(createModel(config.llm).model),
           maxRetries: config.max_retries_per_step,
+          mask: runMask,
           onEvent: printEvent
         });
       } catch (error) {
@@ -1769,7 +1811,7 @@ async function runCommand(options) {
     for (const test of selected) {
       console.log(`
 > ${test.summary} [${test.priority}] (${path9.relative(options.cwd, test.path)})`);
-      results.push(await runOne(browser, test, config, sessionDir, session));
+      results.push(await runOne(browser, test, config, sessionDir, session, runMask));
     }
   } finally {
     await browser.close();
@@ -1874,6 +1916,13 @@ async function planCommand(options) {
   console.log(
     `blastproof plan: ${work.length} route(s), provider=${provider} model=${modelId}, base_url=${config.base_url}`
   );
+  const mask = new SecretsMask();
+  for (const step of config.auth?.steps ?? []) mask.registerFrom(step);
+  for (const value of Object.values(config.auth?.headers ?? {})) mask.registerFrom(value);
+  for (const cookie of config.auth?.cookies ?? []) mask.registerFrom(cookie.value);
+  for (const test of suite) {
+    for (const step of [...test.setup ?? [], ...test.steps]) mask.registerFrom(step);
+  }
   const generated = [];
   const written = [];
   const failed = [];
@@ -1891,6 +1940,7 @@ async function planCommand(options) {
           browser,
           brain: createBrain(createModel(config.llm).model),
           maxRetries: config.max_retries_per_step,
+          mask,
           onEvent: printAuthEvent
         });
       } catch (error) {
@@ -1913,7 +1963,8 @@ async function planCommand(options) {
           route,
           baseUrl: config.base_url,
           changedFiles,
-          brain
+          brain,
+          mask: (text) => mask.mask(text)
         });
       } catch (error) {
         failed.push({ route, reason: error instanceof Error ? error.message : String(error) });
@@ -2013,7 +2064,7 @@ function parseMinScore(value) {
   return score;
 }
 var program = new Command();
-program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.2.0");
+program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.2.2");
 program.command("init").description("Scaffold .blastproof/ (config, tests, sample tests) in the current directory").action(async () => {
   try {
     const result = await initProject(process.cwd());