blastproof 0.13.0 → 0.15.0

package/README.md

@@ -141,11 +141,15 @@ Write steps that end in an observable result — text on the page, a count, a st
 - fill the note field                           # cannot run
 ```
 
-The agent is **forbidden from inventing values** — one it types must come from the step, from the page, or from an `{{env.*}}` placeholder. But that rule lives in a prompt, and a prompt instructs rather than enforces. Run against a real model, the second step does not fail: the agent makes a value up, fills it, and the step **passes**. `fill the note field` produced "This is a test note." on two runs and "This is a new note" on a third.
+The agent is **forbidden from inventing values** — one it types must come from the step, from the page, or from an `{{env.*}}` placeholder — and the runner enforces it rather than asking. A `fill` or `select` whose value appears in none of those is refused: it is not typed, the agent is told which sources it may draw from, and the step fails on the retry budget if it keeps insisting.
 
-That is worse than a failure. The test goes green having verified a value nobody wrote, differing between runs — a passing check over an unspecified input. Closing the gap in the runner is [#57](https://github.com/hamc/blastproof/issues/57); until then, this warning is what stands between you and a green test that means nothing.
+The rule used to live only in the prompt, and a prompt instructs rather than enforces. Run against a real model, `fill the note field` did not fail — the agent made a value up, filled it, and the step **passed**, producing "This is a test note." on two runs and "This is a new note" on a third. A test going green over a value nobody wrote, differing between runs, is worse than a failure.
 
-`run` warns about it first, on every path, before launching a browser or asking for a key:
+A placeholder counts as a source **only when the step names that variable**. `fill the password field with {{env.TEST_PASSWORD}}` works; `fill the password field` does not become valid because the agent supplies `{{env.SOMETHING}}` itself. An agent cannot know the name of a variable nobody showed it, so one it produces is a guess — and a guessed variable would put a live credential into a field your test never pointed one at, in output that cannot redact a secret it was never told about.
+
+Two limits worth knowing. A value the page shows in one format and the field wants in another — `1234` in the step, `1,234.00` in the box — is refused, and the fix is to write the value the way it is typed. And a very short value (`3`) appears somewhere in almost any page, so it will pass; this closes fabricated content, not every fabricated character.
+
+`run` also warns about it first — the same rule caught earlier, from the test file, on every path, before launching a browser or asking for a key:
 
 ```
 Authoring (a step enters a value but names none):
@@ -156,7 +160,7 @@ Authoring (a step enters a value but names none):
 
 Non-fatal by default — `--fail-on-authoring` turns it into exit 1 for teams enforcing it in CI. Taking the value from the page is fine and is not flagged: `fill the recipient field with the address shown on the confirmation page`.
 
-**The check reads English only.** A suite written in another language runs exactly as well but is not inspected, and prints no warning saying so — silence from this check means "nothing found in English", never "this suite is clean".
+**The warning reads English only.** A suite written in another language runs exactly as well but is not inspected, and prints no warning saying so — silence from this check means "nothing found in English", never "this suite is clean". The runner's refusal has no such limit: it compares text rather than parsing grammar, so a suite in any language is still held to the rule at run time.
 
 ## Commands
 

package/dist/cli.js

@@ -508,6 +508,14 @@ async function performAction(page, action, ctx) {
 // src/runner/recovery.ts
 var COMMIT_ACTIONS = /* @__PURE__ */ new Set(["click", "press"]);
 var COMMIT_KEYS = /* @__PURE__ */ new Set(["Enter", "NumpadEnter", " ", "Space", "Spacebar"]);
+var SOURCED_VALUE_ACTIONS = /* @__PURE__ */ new Set(["fill", "select"]);
+function referencesOnlyNamedVars(value, step) {
+  const named = new Set(referencedEnvVars(step));
+  return referencedEnvVars(value).every((name) => named.has(name));
+}
+function normalise(text) {
+  return text.replace(/\s+/g, " ").trim().toLowerCase();
+}
 function describeAction(action) {
   const target = action.target ? ` ${action.target.role ?? ""} "${action.target.name ?? action.target.text ?? ""}"` : "";
   const value = action.value ? ` [${action.value}]` : "";
@@ -525,10 +533,50 @@ function identity(action) {
 var StepRecovery = class {
   performed = /* @__PURE__ */ new Set();
   history = [];
+  /**
+   * Everything the model has been allowed to read this step, normalised: the
+   * step's own text, plus every snapshot it has been shown, plus every value it
+   * has already typed successfully.
+   *
+   * One accumulating haystack rather than the snapshot in hand (design D2). The
+   * model may read an order number on one page, navigate away, and type it into
+   * a box on another — at that moment the current snapshot does not contain it,
+   * and a check against only that snapshot would refuse a legitimate action.
+   *
+   * Bounded by construction: `max_snapshot_lines` caps each snapshot,
+   * `maxIterationsPerStep` caps how many arrive, and the instance dies with the
+   * step — which is also the whole of the "does not cross steps" rule.
+   */
+  readable;
+  /**
+   * The step exactly as written, for deciding which `{{env.*}}` variables it
+   * names (design env-placeholder-must-be-named, D3). Kept alongside the
+   * normalised copy rather than derived from it: `readable` is lowercased, and
+   * environment variable names are case-sensitive, so `TOKEN` and `token` must
+   * stay distinguishable in the one place that decides which secret gets typed.
+   */
+  step;
+  constructor(step) {
+    this.step = step;
+    this.readable = [normalise(step)];
+  }
+  /**
+   * Records a snapshot the model was shown, as a source it may quote from.
+   *
+   * Takes the **masked** text (design D4). `executor.ts` shows the model
+   * `mask(snap)` and never the raw tree, so a secret rendered on the page is
+   * `***` to the model and cannot have been copied by it. Accepting the raw
+   * snapshot here would credit the model with access it never had, and would
+   * leave the executor and the model disagreeing about what the page said.
+   */
+  observe(maskedSnapshot) {
+    this.readable.push(normalise(maskedSnapshot));
+  }
   /** Records an action that was actually performed and succeeded. */
   record(action, description, result) {
     this.performed.add(identity(action));
     this.history.push({ action: description, result });
+    if (action.value) this.readable.push(normalise(action.value));
   }
   /**
    * The reason to refuse `action`, or `undefined` when it may be performed.
@@ -547,11 +595,47 @@ var StepRecovery = class {
    * and #28 has now produced one on three applications.
    */
   refusalFor(action) {
+    return this.repeatedCommitRefusal(action) ?? this.unsourcedValueRefusal(action);
+  }
+  repeatedCommitRefusal(action) {
     if (!COMMIT_ACTIONS.has(action.action)) return void 0;
     if (action.action === "press" && !COMMIT_KEYS.has(action.value ?? "")) return void 0;
     if (!this.performed.has(identity(action))) return void 0;
     return `refused: this exact action already succeeded earlier in this step, so it was NOT performed again. Repeating something that commits repeats whatever it changed in the application. If the page no longer shows that it worked, that is normal for a submit answered by a redirect \u2014 check the record of what you have already done. Verify the step's outcome another way, or fail the step.`;
   }
+  /**
+   * Refuses a typed value that came from nowhere the model was entitled to read
+   * (design refuse-an-invented-value).
+   *
+   * `prompts.ts` has forbidden inventing a value since 0.7.0, and measured
+   * against a real model the rule simply does not hold: given `fill the note
+   * field`, the model supplied "This is a test note." twice and "This is a new
+   * note" once, and the step passed all three times. A prompt instructs; it does
+   * not enforce. The result is the false negative this project exists to remove
+   * — a green test over an input nobody wrote, differing run to run, with
+   * nothing in the report to say so because nothing knew.
+   *
+   * The comparison is `includes`, not equality: a value is legitimately a
+   * fragment of a step that also names the field, and of a snapshot that also
+   * holds the rest of the page.
+   *
+   * Unlike the authoring warning that predicts this before a run, nothing here
+   * parses English, so the guarantee holds for a suite written in any language.
+   */
+  unsourcedValueRefusal(action) {
+    if (!SOURCED_VALUE_ACTIONS.has(action.action)) return void 0;
+    const value = action.value;
+    if (!value) return void 0;
+    if (!referencesOnlyNamedVars(value, this.step)) {
+      const named = referencedEnvVars(this.step);
+      return `refused: the value "${value}" was NOT typed, because this step does not reference that {{env.*}} variable. ${named.length > 0 ? `This step references ${named.map((n) => `{{env.${n}}}`).join(", ")}.` : "This step references no environment variable."} A placeholder is a source only when the step names it \u2014 otherwise the test never asked for that secret to be typed here. Use a value this step supplies, or fail the step and say it supplies none.`;
+    }
+    if (referencedEnvVars(value).length > 0) return void 0;
+    const needle = normalise(value);
+    if (needle === "") return void 0;
+    if (this.readable.some((source) => source.includes(needle))) return void 0;
+    return `refused: the value "${value}" was NOT typed, because it appears neither in this step nor anywhere on the pages you have been shown. A value you type must come from the step, from the page, or from an {{env.*}} placeholder \u2014 one you make up would put the test's verdict on an input nobody wrote. Use a value the step or the page gives you, or fail the step and say it supplies none.`;
+  }
   /** The step's history so far, oldest first, for the model's prompt. */
   stepHistory() {
     return this.history;
@@ -626,7 +710,7 @@ async function executeTest(page, test, options) {
     let failedAttempts = 0;
     let lastResult;
     let stepFailedReason;
-    const recovery = new StepRecovery();
+    const recovery = new StepRecovery(step);
     try {
       while (true) {
         if (iterations >= maxIterationsPerStep) {
@@ -639,12 +723,14 @@ async function executeTest(page, test, options) {
           );
         }
         const snap = await takeSnapshot(page);
+        const maskedSnap = mask(snap);
+        recovery.observe(maskedSnap);
         let action;
         try {
           action = await brain.nextAction({
             step,
             isSetup: setup,
-            snapshot: mask(snap),
+            snapshot: maskedSnap,
             lastResult: lastResult === void 0 ? void 0 : mask(lastResult),
             // Already masked when recorded, on the same boundary as everything
             // else crossing into a prompt (design contained-recovery, D2).
@@ -675,16 +761,18 @@ async function executeTest(page, test, options) {
           let judgment = await brain.judge(
             mask(step),
             mask(expectation),
-            mask(snap),
+            maskedSnap,
             recovery.stepHistory()
           );
           if (!judgment.pass) {
             await waitForSettled(page);
             const freshSnap = await takeSnapshot(page);
+            const maskedFresh = mask(freshSnap);
+            recovery.observe(maskedFresh);
             judgment = await brain.judge(
               mask(step),
               mask(expectation),
-              mask(freshSnap),
+              maskedFresh,
               recovery.stepHistory()
             );
           }
@@ -1233,7 +1321,7 @@ Rules:
 - Return "done" when the current step's outcome holds \u2014 including when it already held before you acted, or was achieved by your previous action. "Already true" is done, never failure. Do not return "done" for work belonging to later steps.
 - Return "fail" only when the step's outcome cannot be reached: the element is still absent after retries, the page cannot support the step, or an error blocks progress. Never return "fail" because the work appears to have been done already.
 - If your previous action errored, re-read the fresh snapshot and choose an alternative element or approach. Do not repeat the exact same failing action.
-- Never invent a value. A value you type must come from the step, from the page, or from an {{env.*}} placeholder. If a step needs a value it does not give you, that is a failing step, not a gap for you to fill in.
+- Never invent a value. A value you type must come from the step, from the page, or from an {{env.*}} placeholder. This one is enforced, not merely asked: a fill or select whose value is in none of those is refused and not performed. If a step needs a value it does not give you, that is a failing step, not a gap for you to fill in.
 - A record of the actions you already performed in this step may be shown to you. It is the ground truth about what happened, even when the page no longer shows it: a form that submitted successfully and came back empty looks exactly like one you never submitted. Do not redo work that record says you already did.
 - \`***\` in a snapshot is a redacted secret \u2014 a password, token or key deliberately withheld from you. Seeing it is expected and is not a problem. A field showing \`***\` after you filled it from an {{env.VAR}} placeholder means the fill worked; treat that as success and move on. Never retry a fill because its value is redacted, and never report failure because a value was withheld.
 - Keep reasoning to one short sentence.`;
@@ -1305,7 +1393,7 @@ Rules:
 - Prefer the journey the changed files touch over a generic tour of the page. The changed files tell you which part of the page matters.
 - **The test starts at the application's base URL, not at this route.** Begin with a step that navigates to the route and says what should be visible once it loads \u2014 "navigate to /support and verify the heading "Contact support" is shown". Without it the run opens the home page and every later step looks for controls that are not there.
 - **Every step says what it should produce.** Name what must be true once the step has been carried out, not the action alone: "submit the support form and verify the confirmation page shows the ticket number", never "submit the support form". A step that names an action without an outcome asks the runner to judge whether something happened while looking at the page that succeeding produces \u2014 a submitted form comes back empty, a redirect moves the URL \u2014 and that is the shape behind several real failures.
-- **A step that enters a value writes the value.** "fill the subject field with Order not received", never "enter a subject". The runner is forbidden from inventing values, but a step that supplies none does not fail \u2014 the value gets made up, the step passes, and the test then verifies something nobody wrote.
+- **A step that enters a value writes the value.** "fill the subject field with Order not received", never "enter a subject". The runner is forbidden from inventing values, and enforces it: a fill whose value is in neither the step nor the page is refused, so a step that supplies none cannot be relied on to run.
 - If a step needs a credential or any secret, write it as a placeholder like {{env.TEST_PASSWORD}}. Never write a real or invented password, token or key.
 - Keep the whole test to a handful of steps: one journey, not an exhaustive suite.`;
 }
@@ -2371,7 +2459,7 @@ function printAuthoring(result, cwd) {
     console.error(`    \u2192 ${suggestValueClause(step)}`);
   }
   console.error(
-    "The runner is forbidden from inventing values, but only by instruction: in practice the model supplies one anyway, the step passes, and the test verifies a value nobody wrote \u2014 a different one on each run."
+    "The runner is forbidden from inventing values and enforces it: one it cannot trace to the step, the page or an {{env.*}} placeholder is refused, so a step supplying none fails at run time. Naming the value here is what makes it run."
   );
   console.error("Steps are inspected in English only; steps in other languages are not checked.");
 }
@@ -2918,7 +3006,7 @@ function parsePositiveNumber(flag) {
   };
 }
 var program = new Command();
-program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.13.0");
+program.name("blastproof").description("Open-source AI testing agent: plain-English YAML tests executed agentically on a real browser.").version("0.15.0");
 program.command("init").description("Scaffold .blastproof/ (config, tests, sample tests) in the current directory").action(async () => {
   try {
     const result = await initProject(process.cwd());