@mieweb/ui 0.11.0-dev.52 → 0.11.0-dev.53

package/agent/evals/runs/2026-11-05.md

@@ -0,0 +1,92 @@
+# Manual eval run — 2026-11-05 (second run, live sandbox, #601)
+
+Second manual run of the golden-question sets — the first against the **live
+installed team** rather than emulation. Sandbox: a real Vite 8 (rolldown) +
+React 19 + TypeScript 7 + Tailwind 4 consumer on published
+`@mieweb/ui@0.11.0-dev.49`, team installed via
+`npx @mieweb/ui init-agent --experimental-agents` and driven through the VS
+Code agent picker (fresh chat per question, posed to the MIE UI Design Agent).
+Answers scored against **Expect**; every novel API/citation claim was verified
+against the library source before a PASS.
+
+Primary goal: regression-test the six corpus fixes merged after
+[run 1](2026-11-02.md). Secondary: broad spot-check coverage plus three
+scripted do-tasks exercising orchestration on real seeded-violation code.
+
+## Scores
+
+| Domain    | Sample                          | PASS | PARTIAL | FAIL |
+| --------- | ------------------------------- | ---- | ------- | ---- |
+| tasks     | T1 audit, T2 dark mode, T3 filter | 3  | 0       | 0    |
+| selection | SEL-6, SEL-12*, SEL-16*         | 3    | 0       | 0    |
+| styling   | STY-13*, STY-14*, STY-18        | 3    | 0       | 0    |
+| install   | INS-6, INS-14*, INS-19          | 3    | 0       | 0    |
+| a11y      | A11Y-8, A11Y-17*                | 2    | 0       | 0    |
+| i18n      | I18N-1, I18N-7, I18N-18         | 3    | 0       | 0    |
+| migration | MIG-1, MIG-9, MIG-17            | 3    | 0       | 0    |
+
+\* = retest of a run-1 PARTIAL/FAIL. **20/20 PASS** (run 1: 18/4/2).
+
+## Retests — all six run-1 fixes verified live
+
+1. **SEL-12** (run-1 FAIL): the wiring-headless routing row added to the
+   orchestrator works — question now routes to the Install Agent and answers
+   from the tailwind4 skill.
+2. **STY-13** (run-1 FAIL): the transition-stomp troubleshooting row works —
+   correct diagnosis (unlayered brand `*` shorthand beats layered
+   `transition-*` utilities), cites #460/#682 as open, prescribes layering
+   the transition rule while keeping tokens unlayered.
+3. **SEL-16** (eval defect): rephrased three-placement Ask now gets a clean
+   three-badge mapping.
+4. **STY-14** (run-1 PARTIAL): corpus-known-good diagnosis path followed,
+   #628 cited as open.
+5. **INS-14** (run-1 PARTIAL): git-source install note works — #489 cited,
+   published-package-only guidance given; the newer #709 ProseMirror
+   knowledge also flowed through.
+6. **A11Y-17** (run-1 PARTIAL): exit-animation note works — `aria-hidden` +
+   `inert` prescribed, #474 correctly cited as open.
+
+## Scripted tasks (orchestration on real code)
+
+- **T1 — compliance audit of seeded `App.tsx`:** caught all six seeded
+  violations (raw styled button, badge span, clickable div, physical `ml-2`,
+  unlocalized strings, class-only dark toggle) plus legitimate bonus findings;
+  orchestrated Migration → A11y → i18n with per-specialist attribution.
+- **T2 — dark-mode toggle:** routed to Styling; prescribed the real
+  `useTheme`/`ThemeProvider`/`ThemeToggle` API (verified against source).
+- **T3 — status filter dropdown:** correct Select-vs-Dropdown-vs-Autocomplete
+  selection reasoning; self-verified the `Select` API instead of guessing.
+
+## Findings
+
+1. **Eval-key/routing-table mismatch (filed as #720).** Four content-correct
+   answers were handled by a different agent than the eval key prescribes:
+   STY-14/STY-18 routed to Install (per the routing table's TW-config row),
+   MIG-1/MIG-17 answered by the orchestrator itself (icons policy, catalog
+   interpretation — README principle 3). In all four the shipped config looks
+   right and the keys look stale; #720 tracks the ownership decision and key
+   fixes.
+2. **Non-finding, recorded to prevent refiling:** the Migration Agent lacking
+   `editFiles` is **by design** — README principle 4 ("Auditors are
+   read-only") and the agent's own "Apply Fixes" handoff. Observed T1
+   behavior (audit report + handoff offer) is the design working.
+3. **Driver-sheet defect (local):** the run's T2 expectation named a
+   nonexistent `useDarkMode`; the real API is `useTheme`. The agent was
+   right, the sheet was wrong.
+
+## Observations (no action yet)
+
+- The skills don't document the `@next` dist-tag; asked about it, the agent
+  honestly refused to guess rather than inventing — acceptable until the
+  release-channel docs phase.
+- Do-tasks end with "want me to hand off?" rather than proactively invoking
+  the handoff; fine for audits, but worth considering ergonomics when the
+  user's ask is imperative ("fix", "add").
+- Honest-uncertainty behavior was consistently good: agents flagged
+  undocumented areas explicitly ("the skill doesn't document…") instead of
+  fabricating, across at least five answers.
+
+No new PARTIALs or FAILs. The run-1 fix loop (findings → corpus fixes →
+retest) is demonstrably effective; next run should sample the untested
+remainder of the question sets and any Phase 4 specialist domains as they
+land.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mieweb/ui",
-  "version": "0.11.0-dev.52",
+  "version": "0.11.0-dev.53",
   "description": "A themeable, accessible React component library built with Tailwind CSS",
   "author": "Medical Informatics Engineering, Inc.",
   "license": "SEE LICENSE IN LICENSE",