@adia-ai/a2ui-mcp 0.1.1 → 0.1.2

package/CHANGELOG.md

@@ -11,6 +11,81 @@ zettel strategies.
 
 ---
 
+## [0.1.2] - 2026-05-01
+
+Phase 2 of [`docs/specs/semantic-validator.md`](../../../docs/specs/semantic-validator.md)
+**promoted to default**. When `--semantic` is set, `eval-diff.mjs`
+now gates on the combined score at threshold 80 by default, instead
+of running in Phase 1 shadow-mode. **Behavior change** — see § Opt
+back into Phase 1 below.
+
+### Changed (default behavior when `--semantic` is set)
+
+- **`--gate-mode` default flipped from `structural` to `combined`
+  when `--semantic` is set.** `npm run eval:diff -- --engine zettel
+  --semantic` now produces a combined-gated `passRate` (gate on
+  `0.6 × validationScore + 0.4 × semanticScore`) instead of the
+  structural-only Phase 1 shadow output. The structural pass survives
+  as `row.passStructural` + `runObj.passRateStructural` for diagnostic
+  comparison.
+- **`--gate-threshold` default raised from 70 to 80.** Chosen via
+  the 2026-05-01 sweep — 70 produced zero pass-flips on the current
+  zettel engine output (cosmetic flip with no signal); 80 catches the
+  4 partial-verdict cases the spec was designed to surface; 85
+  over-aggressively rejects legit aligned items. See spec § Phase 2
+  status (2026-05-01) for the sweep table.
+- **Without `--semantic`, the default stays `structural`.** Semantic
+  work remains opt-in; the no-flag `npm run eval:diff` shape is
+  unchanged.
+
+### Opt back into Phase 1 shadow-mode
+
+- `--gate-mode structural` reverts the default flip — `row.pass`
+  gates on `validationScore` alone, semantic verdicts annotation-
+  only.
+- `--gate-threshold 70` reverts the threshold change.
+- Combine both for full v0.1.1 default behavior:
+  ```bash
+  node packages/a2ui/mcp/scripts/eval-diff.mjs --engine zettel \
+    --semantic --gate-mode structural
+  ```
+
+### Why the promotion is safe
+
+Per the Phase 2 exit criterion ("no unexplained regressions in first
+two full `eval-diff` runs"), zero pass-flips were observed at
+threshold=70 (the v0.1.1 default). At threshold=80, the four
+pass→fail flips are all `partial`-verdict items that fit the spec's
+motivating failure mode (structural=86–88 + semantic=54–69) — the
+intents the structural-only validator scored highly but the LLM
+judge correctly flagged as "wrong UI for this intent":
+
+- `user profile card with avatar` — structural 88, semantic 54
+- `calendar month view` — structural 86, semantic 65
+- `social media post card` — structural 86, semantic 69
+- `user onboarding checklist` — structural 87, semantic 67
+
+Pass rate drops from 83 → 79 (4 percentage points), all justified.
+
+### Updated baseline thresholds
+
+Per [`docs/specs/semantic-validator.md`](../../../docs/specs/semantic-validator.md)
+§ Phase 2 status (2026-05-01):
+
+- `avgScore ≥ 88` (structural — unchanged)
+- `avgSemanticScore ≥ 85` (new)
+- `avgCombined ≥ 87` (new)
+- `passRate ≥ 78%` at combined-gate threshold=80 (new; was 83%
+  structural-only)
+- `verdict aligned ≥ 84%` of judged items
+
+### Implementation references
+
+- [`scripts/eval-diff.mjs`](scripts/eval-diff.mjs) — default flip
+- [`scripts/semantic-stats.mjs`](scripts/semantic-stats.mjs) —
+  unchanged from v0.1.1; remains the validation tool for future
+  threshold revalidation passes
+
 ## [0.1.1] - 2026-05-01
 
 Phase 2 of [`docs/specs/semantic-validator.md`](../../../docs/specs/semantic-validator.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@adia-ai/a2ui-mcp",
-  "version": "0.1.1",
+  "version": "0.1.2",
   "description": "AdiaUI A2UI MCP server. Exposes the compose engine over MCP with an engine selector for monolithic + zettel strategies.",
   "type": "module",
   "bin": {

package/scripts/eval-diff.mjs

@@ -18,9 +18,9 @@
  *   node packages/a2ui/mcp/scripts/eval-diff.mjs --engine zettel  # fragment-graph only
  *   node packages/a2ui/mcp/scripts/eval-diff.mjs --limit 20
  *   node packages/a2ui/mcp/scripts/eval-diff.mjs --domain forms
- *   node packages/a2ui/mcp/scripts/eval-diff.mjs --semantic       # Phase 1: shadow-mode semantic annotations
- *   node packages/a2ui/mcp/scripts/eval-diff.mjs --semantic --gate-mode combined  # Phase 2: gate row.pass on combined score
- *   node packages/a2ui/mcp/scripts/eval-diff.mjs --semantic --gate-mode combined --gate-threshold 75
+ *   node packages/a2ui/mcp/scripts/eval-diff.mjs --semantic       # Phase 2 default: gates on combined score (>= 80)
+ *   node packages/a2ui/mcp/scripts/eval-diff.mjs --semantic --gate-mode structural  # Phase 1 shadow-mode (opt-back-in)
+ *   node packages/a2ui/mcp/scripts/eval-diff.mjs --semantic --gate-threshold 75    # Override default threshold
  */
 import '../../../../scripts/load-env.mjs';
 
@@ -48,13 +48,19 @@ const domain = opt('domain');
 // Shadow-mode semantic validator (Phase 1). Opt-in; zero effect on gating
 // when --gate-mode=structural (default).
 const semanticEnabled = args.includes('--semantic');
-// Phase 2 (gating mode):
-//   structural (default) — `row.pass` gated on validationScore alone (Phase 1 behavior)
-//   combined             — `row.pass` gated on (0.6 * validationScore + 0.4 * semanticScore)
-// Combined mode requires --semantic. Threshold defaults to 70 to match the
-// existing structural threshold; override with --gate-threshold N.
-const gateMode = opt('gate-mode') || 'structural';
-const gateThreshold = opt('gate-threshold') ? Number(opt('gate-threshold')) : 70;
+// Phase 2 (gating mode). Default depends on --semantic:
+//   without --semantic → 'structural' (no semantic work; Phase 1 baseline).
+//   with --semantic    → 'combined' since v0.1.2 (Phase 2 promotion); was
+//                        'structural' / shadow-mode in v0.1.1.
+//                        Opt-back-in to shadow with --gate-mode structural.
+// Combined mode gates on (0.6 * validationScore + 0.4 * semanticScore) at
+// threshold 80 — chosen via the 2026-05-01 sweep (70 produced zero
+// pass-flips; 80 catches the 4 partial-verdict cases the spec was designed
+// to surface; 85 over-aggressively rejects aligned items).
+// Override threshold with --gate-threshold N.
+const gateModeDefault = args.includes('--semantic') ? 'combined' : 'structural';
+const gateMode = opt('gate-mode') || gateModeDefault;
+const gateThreshold = opt('gate-threshold') ? Number(opt('gate-threshold')) : 80;
 if (!['structural', 'combined'].includes(gateMode)) {
   console.error(`[eval-diff] --gate-mode must be one of: structural | combined  (got: ${gateMode})`);
   process.exit(2);