@hyperfixi/testing-framework 2.11.0 → 2.11.1

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hyperfixi/testing-framework",
-  "version": "2.11.0",
+  "version": "2.11.1",
   "description": "Cross-platform behavior testing suite for LokaScript applications",
   "main": "dist/index.js",
   "module": "dist/index.mjs",
@@ -57,11 +57,11 @@
   "author": "LokaScript Contributors",
   "license": "MIT",
   "dependencies": {
-    "@hyperfixi/core": "^2.11.0",
-    "@hyperfixi/patterns-reference": "^2.11.0",
-    "@lokascript/compilation-service": "^2.11.0",
-    "@lokascript/i18n": "^2.11.0",
-    "@lokascript/semantic": "^2.11.0",
+    "@hyperfixi/core": "^2.11.1",
+    "@hyperfixi/patterns-reference": "^2.11.1",
+    "@lokascript/compilation-service": "^2.11.1",
+    "@lokascript/i18n": "^2.11.1",
+    "@lokascript/semantic": "^2.11.1",
     "diff": "^8.0.3",
     "esbuild": "^0.28.0",
     "happy-dom": "^20.10.6",

package/src/agent-bench/README.md

@@ -71,13 +71,13 @@ Run file:
 }
 ```
 
-**No A/B run is committed here, deliberately.** The tasks and their reference
-implementations were authored in the same session that would have produced the
-candidates, so any one-shot number from that session measures recall of
-just-written answers, not generation. A meaningful run needs a generator that
-has not seen this directory — until one exists, the honest position is a harness
-with no number attached, not a flattering number with a caveat. `score` is fully
-implemented and ready for that run.
+**The committed run required an isolated generator, and waited for one.** The
+tasks and their references were authored in the same session that would have
+produced the candidates, so any one-shot number from that session would have
+measured recall of just-written answers, not generation. The run in
+[`runs/ab-2026-08-25.json`](../../runs/ab-2026-08-25.json) was generated by an
+agent that had never seen this directory — see [the A/B finding](#the-ab-run-2026-08-25)
+for the result and how isolation was enforced.
 
 Not a CI gate: LLM-in-the-loop is nondeterministic and this repo's gates stay
 deterministic. Half 1 **is** gated, because it has no generator in it.
@@ -160,3 +160,60 @@ reference, and the fixture markup it needs. Then run `verify-references` — a
 reference that does not parse, or produces no DOM effect, is rejected (same
 eligibility bar as R2's execution subset), because scoring against an empty
 signature would make wrong answers look right.
+
+### The A/B run, 2026-08-25
+
+The first run with a generator that had never seen this directory.
+[`runs/ab-2026-08-25.json`](../../runs/ab-2026-08-25.json).
+
+|                  | one-shot | loop     |
+| ---------------- | -------- | -------- |
+| parse rate       | 100%     | 100%     |
+| behavior rate    | **90%**  | **100%** |
+| parsed-but-wrong | 0        | 0        |
+
+**The loop closed the gap, and every point of it came through a diagnostic
+Arc 3b added.** Both one-shot failures were the same shape — `set <prop> of
+<target> to <value>`, which returns `ok: true` at confidence 1.0 with an
+**empty handler** (`parsed: "on()"`) and produces no DOM effect at all:
+
+```
+set @aria-expanded of #panel to 'true'   → on()   ⚠ UNCONSUMED_INPUT (6 tokens)
+set innerHTML of #output to 'Done'       → on()   ⚠ UNCONSUMED_INPUT (6 tokens)
+```
+
+Pre-3b these two rows would have been **silent** — `ok: true`, zero
+diagnostics — and the loop would have had nothing to react to, so the delta
+would have been 0. The `UNCONSUMED_INPUT` plumbing fix is what made them
+repairable. That is the benchmark's own claim about itself, measured rather
+than asserted: the loop is worth what the diagnostics are worth.
+
+The generator rewrote both to the possessive form (`set #panel's
+@aria-expanded to "true"`) after the warning, and reported a third repair the
+scoreboard cannot show — it rejected `set *innerHTML of #output to "Done"`,
+which parses **clean at confidence 1.0**, on IR inspection alone: `*` is the
+style sigil, so the IR read `destination=#output.*innerHTML`. That is the
+loop's step-2 discipline (_check the IR against your intent_) catching what no
+diagnostic could.
+
+Isolation, enforced rather than requested:
+
+- One-shot answers were written and `chmod 444` locked **before** the first
+  `feedback` call, so the loop phase could not retroactively revise them.
+- `feedback` is the only channel to the validator and returns only diagnostics
+  and the IR — never the reference, never the behavior verdict.
+- Audited from the generator's transcript afterward: **zero** Read/Glob/Grep
+  calls, all 48 Bash calls were the two permitted wrapper scripts, zero
+  commands referencing the repo path.
+- The generator was barred from the project docs too, not just this directory —
+  [AGENTS.md](../../../../AGENTS.md) quotes this benchmark's findings and names
+  the specific trap phrasings, so a doc-equipped generator would be reading
+  leaked answers.
+
+Read the number for what it is: **n=1 generator, 20 tasks, 2 tasks moved.** It
+is not a claim about models in general, and the one-shot arm is an
+_undocumented_ generator — a briefed one starts higher and the delta narrows.
+What is solid is the mechanism, which the probe half measures deterministically:
+the failures the loop can fix are exactly the ones something makes visible.
+Of the 7 answers the loop changed, 2 changed the outcome and 5 were cosmetic —
+iterating on already-correct answers cost nothing and broke nothing.