@ikon85/agent-workflow-kit 0.34.4 → 0.34.5

package/docs/research/consumer-owned-protocol-files.md

@@ -0,0 +1,238 @@
+# Consumer-owned protocol files: evolution without overwrite or starvation
+
+Verified on 2026-07-22 against `agent-workflow-kit` 0.33.0 in the issue #197
+worktree. This note researches how an updater should handle a locally adapted
+file that is also part of an evolving upstream protocol. It is decision input,
+not a product change.
+
+## Finding
+
+A protocol-critical file must not be both an immutable consumer-owned fork and
+the upstream kit's active protocol authority under the same path. Those two
+contracts are incompatible: permanent consumer ownership prevents protocol
+evolution, while ordinary upstream ownership cannot preserve local policy.
+
+The durable model is to split the mixed file into:
+
+1. an upstream-owned, immutable **protocol core** that ships and evolves with
+   the kit; and
+2. a consumer-owned, structured **extension overlay** containing only local
+   declarations and policy.
+
+The staged update candidate composes core plus overlay and validates the whole
+result before activation. A one-time migration extracts existing local
+extensions from the mixed file. Ambiguity blocks that migration rather than
+overwriting either side. A consumer that needs to change executable protocol
+behavior has made a real code fork and needs a separate artifact/version line,
+not `origin: consumer` on the upstream path.
+
+## Current local contract and failure mode
+
+The accepted divergence policy defines a consumer-owned path as a deliberate
+fork that receives no overwrite, conflict report, deletion prompt, or future
+kit improvement ([ADR 0001](../adr/0001-consumer-divergence-policy.md)). The
+reconciler implements that literally: as soon as an installed entry has
+`origin: 'consumer'`, it carries the old manifest record forward and skips all
+upstream comparison
+([`updateReconcile.mjs`](../../src/lib/updateReconcile.mjs)). This is coherent
+for an independent local file, but it is update starvation for a protocol
+authority.
+
+Issue #196 exposed the concrete mixed-file shape. The readiness harness and
+skills depend on `.claude/skills/skill-manifest.json`, but a real consumer owns
+and extends that same registry. The published 0.32.0 artifact was internally
+coherent; the consumer combined 15 local skill records and local notes with an
+older upstream registry, while the newer kit added five skills, 23 readiness
+capabilities, and 15 readiness declarations. Skipping the owned registry thus
+allowed a new executable harness to run against an old protocol declaration.
+
+The candidate transaction already has the right enforcement location: it
+checks candidate hashes and destination races before activation and rolls back
+all touched paths on failure
+([`updateCandidate.mjs`](../../src/lib/updateCandidate.mjs)). What is missing is
+a compositional protocol model and validator; copying more of the consumer or
+adding more path exclusions does not solve that ownership contradiction.
+
+## Comparable primary-source patterns
+
+| System | Primary-source behavior | Lesson for the kit |
+|---|---|---|
+| Git three-way file merge | `git merge-file` combines the changes from a recorded common base to the upstream side with the local side; overlapping edits become explicit conflicts and a clean merge has a distinct successful exit status. [Git documentation](https://git-scm.com/docs/git-merge-file) | The stored installed hash/base makes a one-time three-way migration possible. A merge is evidence to inspect, not permission to silently overwrite. Text merge is unsuitable as the permanent extension mechanism because protocol identity and invariants are semantic, not line based. |
+| GitHub forks | Syncing a fork explicitly fetches/merges upstream. Conflicts require a pull request or manual resolution; forcing sync overwrites the destination. [GitHub documentation](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork) | A true code fork has its own history and must deliberately integrate upstream. It cannot also promise automatic compatibility under the original identity. |
+| Kubernetes Server-Side Apply | Kubernetes tracks ownership per structured field. An attempted value change to another manager's field conflicts; force is an explicit ownership transfer. A manager can also relinquish a field or share ownership by agreeing on the same value. [Kubernetes documentation](https://kubernetes.io/docs/reference/using-api/server-side-apply/) | Ownership below the file level works for schema-defined data. Core fields and consumer extension fields can have separate managers, with collisions rejected and transfers explicit. This model should not be imitated for arbitrary source-code lines. |
+| Debian configuration files | Debian requires local configuration changes to survive upgrades. It distinguishes package conffiles from files generated and maintained by idempotent scripts, forbids mangling user configuration without asking, and warns that mixing the two handling styles causes repeated upgrade prompts. [Debian Policy](https://www.debian.org/doc/debian-policy/ch-files.html#behavior) `dpkg` separately exposes explicit keep-old/install-new/default choices for changed conffiles. [dpkg manual](https://manpages.debian.org/bookworm/dpkg/dpkg.1.en.html) | Choose one ownership model per artifact. Do not make one file simultaneously an upstream payload and a consumer-generated extension. Migrations must be idempotent, previewed, and preserve the old value on failure. |
+| Terraform provider state | Terraform stores a resource schema version, asks the provider to upgrade older state before planning, rejects unsupported prior versions, and retains prior state when an upgrader reports errors. Each upgrader must produce a complete current-version state. [Terraform Plugin Framework](https://developer.hashicorp.com/terraform/plugin/framework/resources/state-upgrade) | Give the overlay an explicit schema version and deterministic upgraders. Validate the fully upgraded/composed state, not only the changed records. Unsupported or ambiguous inputs block activation without destroying the prior state. |
+| Kubernetes API versioning | Kubernetes can serve old and new API versions concurrently, converts between representations, migrates stored objects separately, and removes an old version only after clients and stored objects have migrated. [Kubernetes CRD versioning](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/) | Compatibility needs an announced window: introduce/read both, migrate, count adoption, stop serving old, then remove conversion. A deprecation warning alone is not migration evidence. |
+| Homebrew taps | An extracted or alternative formula lives in a separate Git repository/tap; its owner is responsible for maintenance, deprecations, and security updates. Naming or qualification avoids collision with the core formula. [Homebrew tap documentation](https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap) | A real executable fork needs a distinct artifact namespace and maintenance owner. It should not silently shadow the kit's protocol core at the same path. |
+
+## Data/config extension versus code fork
+
+These cases need different contracts.
+
+### Structured consumer extension
+
+Use an overlay when the consumer is adding data the upstream runtime already
+knows how to interpret: local skill registrations, local capability evidence,
+board values, activation decisions, or project policy. The upstream owns the
+schema and executable semantics; the consumer owns values in a declared
+extension namespace.
+
+The overlay should have:
+
+- its own consumer-native path and `schemaVersion`;
+- stable record identities and an explicit allowed extension namespace;
+- no ability to redefine reserved upstream/core records;
+- deterministic, idempotent schema upgraders;
+- a compatibility declaration tying overlay schema versions to supported
+  protocol-core versions; and
+- a composed-state validator that checks references, uniqueness, required
+  declarations, and executable/declaration parity.
+
+This is the applicable model for the mixed skill/readiness manifest. The
+consumer's local skills are data extensions; the kit's shipped skill registry,
+readiness schema, and harness contract are protocol core.
+
+### Executable code fork
+
+Use a true fork when the consumer changes parsing, evaluation, migration, or
+other executable protocol behavior. The fork must then have a distinct package
+or path identity, its own version and release channel, an explicit upstream
+remote/base, and a deliberate sync/rebase/merge process. Compatibility with the
+upstream package is no longer automatic.
+
+Marking such code `consumer`-owned under the upstream path hides that reality:
+it keeps the name but abandons the update channel. The Homebrew and GitHub
+models make the trade-off honest: local history is preserved, upstream sync is
+possible, conflicts are explicit, and maintenance responsibility moves to the
+fork owner.
+
+## Recommended candidate and ownership contract
+
+### 1. Define protocol groups, not only independent files
+
+The package manifest should declare every protocol-critical group and its
+members, for example:
+
+```text
+readiness protocol
+  core registry:       kit-owned
+  runtime harness:     kit-owned
+  skill preflights:    kit-owned
+  consumer overlay:    consumer-owned structured data
+  validator/migrators: kit-owned
+```
+
+The candidate is valid only if every group is coherent as a whole. A
+consumer-owned member can therefore no longer make validation disappear.
+
+### 2. Keep upstream protocol core unownable under its canonical identity
+
+`own` should refuse a protocol-core path and point to the supported overlay or
+fork route. Existing owned core paths are legacy states to migrate, not states
+to perpetuate. This narrows ADR 0001 rather than weakening it: ordinary
+consumer-owned files remain immutable and skipped; protocol core is no longer
+an eligible ownership unit.
+
+### 3. Compose and validate before activation
+
+For every update, materialize the complete intended kit end state, load and
+upgrade the consumer overlay, compose both, and run protocol-specific
+invariants. Activation remains all-or-nothing. A bad overlay blocks the
+protocol group with a targeted diagnostic; it never causes the updater to
+overwrite local data or to activate a mixed-version group.
+
+Collision rules should be structural:
+
+- a consumer may add records only in its declared namespace;
+- a consumer record that collides with a core identity is rejected unless the
+  schema explicitly defines a non-overriding augmentation;
+- consumer values cannot replace core executable references or protocol
+  versions;
+- dangling references and unsupported overlay versions block activation; and
+- ownership transfer is an explicit user decision, never a `--yes` default.
+
+### 4. Migrate the legacy mixed file once
+
+For the existing combined registry, the staged candidate should retain three
+inputs: the last installed upstream base, the live consumer file, and the new
+upstream core. Migration should semantically classify records rather than
+blindly line-merge JSON:
+
+1. identify records unchanged from the recorded upstream base;
+2. extract clearly consumer-only records and allowed local policy into the new
+   overlay;
+3. install the complete new upstream core;
+4. compose and validate the result; and
+5. preview the extraction, backups, ownership changes, and any ambiguous rows.
+
+If provenance is ambiguous—for example, a consumer modified an upstream record
+instead of adding a namespaced one—the update stops at a user gate. The user
+chooses to port the change upstream, express it through a supported overlay
+field, or create a real fork. A raw three-way merge may help produce the review
+diff, but it must not decide semantic ownership.
+
+### 5. Version, negotiate, deprecate, then remove
+
+The protocol core and overlay schema need independent explicit versions and a
+supported compatibility matrix. Rollout follows the Kubernetes/Terraform
+pattern:
+
+1. ship readers and validators for both the legacy combined representation and
+   the new split representation;
+2. ship the idempotent migration and preview it in the candidate;
+3. activate core plus overlay only after full validation;
+4. report a fresh adoption count (`X of Y` consumers or known legacy states),
+   not an assumed completion;
+5. stop creating legacy state and make old-state diagnostics actionable; then
+6. remove the legacy reader only after all supported prior states have a
+   migration path and known consumers have migrated.
+
+Without remote telemetry, the evidence is local manifest/schema state plus
+consumer update/CI results. The kit should therefore use a release-bounded
+support window and a deterministic audit command, not claim global adoption.
+
+## Proposed rollout for issues #196/#197
+
+1. **Inventory and classify.** Derive all protocol-critical groups from shipped
+   executable references and manifests. Count every member and mark it core,
+   overlay, generated evidence, or independent file.
+2. **Add validation before enforcement.** Build the kit-only candidate and
+   protocol-group validator. Initially report legacy mixed ownership and prove
+   that a #196-style harness/registry mismatch is rejected.
+3. **Introduce the split readiness registry.** Ship an upstream-owned core
+   registry plus a consumer-owned extension overlay. The runtime reads the
+   composed view; new consumers start split.
+4. **Adopt existing consumers transactionally.** Extract Testreporter's 15
+   consumer-only skill records and local policy into the overlay, install the
+   current core, validate all core and local records, and change ownership only
+   as part of the verified candidate activation. Preserve a backup/recovery
+   receipt.
+5. **Guard the seam.** Reject future `own` operations on protocol-core paths;
+   point users to the overlay for data extensions or to a separately named
+   package/path for executable forks.
+6. **Sunset the combined format.** Keep the legacy reader for a declared
+   compatibility window, count locally known adoptions, then remove it in a
+   release that explicitly declares the minimum supported overlay/core schema.
+
+## Rejected rollout shapes
+
+- **Silently overwrite the owned file:** destroys legitimate local behavior
+  and violates the existing consumer contract.
+- **Keep skipping it forever:** preserves bytes but starves the protocol and
+  recreates #196 on every coupled evolution.
+- **Automatically disown it:** turns an ownership decision into an implicit
+  destructive migration.
+- **Blind three-way merge on every update:** reduces conflicts but leaves a
+  mixed authority and cannot validate semantic record identity.
+- **Field ownership for arbitrary code:** line/AST ownership is not a stable
+  extension API; local executable changes need an explicit fork.
+- **Fork the whole protocol for local records:** imposes permanent merge and
+  security-update work where a narrow data overlay is sufficient.
+
+## Decision implication
+
+The current definition “owning a path means forgoing all future kit
+improvements to it” remains valid for genuine independent forks. It should no
+longer be offered for canonical protocol-core paths. For those paths, the
+supported local customization mechanism is a versioned consumer overlay; the
+honest alternative is a separately identified and maintained code fork.

package/docs/research/frontend-agent-benchmarks.md

@@ -0,0 +1,282 @@
+# Frontend-agent benchmarks
+
+Research snapshot: 2026-07-22
+
+## Verdict
+
+Credible frontend benchmarks now exist, but **there is no single trustworthy
+"frontend capability" score**. The field measures four materially different
+things:
+
+1. visual generation and design preference;
+2. functional browser behaviour and interaction;
+3. implementation or repair inside an existing frontend repository;
+4. design taste, accessibility, and responsive behaviour.
+
+The strongest evidence stack for routing today is:
+
+- **Code Arena WebDev** as the current live signal for greenfield frontend and
+  human preference;
+- **Vision2Web** as the most promising controlled end-to-end benchmark once its
+  current season has results;
+- **SWE-bench Multimodal**, optionally through the OpenHands Index, for repair
+  in existing JavaScript/frontend repositories;
+- **DeepSWE** for model-effort curves, because the frontend sources generally
+  do not isolate effort;
+- local, dated outcomes for a project's actual stack, design system,
+  accessibility requirements, and definition of quality.
+
+The resolver must keep these evidence dimensions separate. Combining them is
+reasonable; pretending that one benchmark jointly proves model quality,
+effort, harness quality, frontend taste, and repository reliability is not.
+
+## Evidence map
+
+| Source | What it actually measures | Evaluation | Current routing value | Main limitation |
+|---|---|---|---|---|
+| [Code Arena WebDev](https://arena.ai/leaderboard/code/webdev) | Prompt-to-app generation under an agentic web-development harness | Blinded pairwise user preference over rendered, interactive outputs | High for current greenfield frontend model/harness preference | Preference is not a pass rate; model and harness are coupled; little comparable effort data |
+| [Vision2Web](https://vision2web-bench.github.io/) | Static responsive pages, interactive multi-page frontends, and full-stack sites from prototypes and requirements | VLM visual judge plus workflow-driven GUI agent | Potentially highest controlled end-to-end value | The active season's leaderboard is still empty; older-season scores are not directly comparable |
+| [SWE-bench Multimodal](https://www.swebench.com/multimodal) | Real issue resolution in visual JavaScript repositories | Repository tests, including visual tests for a subset | High for existing-repository repair | Does not measure greenfield design taste; historical public GitHub tasks create exposure risk |
+| [OpenHands Index](https://www.openhands.dev/blog/openhands-index) | A verified SWE-bench Multimodal subset under one OpenHands SDK, with cost/runtime | Executable benchmark plus standardized harness | Useful model comparison for repair | OpenHands performance is not automatically Claude Code or Codex performance |
+| [WebGen-Bench](https://proceedings.neurips.cc/paper_files/paper/2025/hash/6841eed8bb6a2ec49e49235c8115efee-Abstract-Datasets_and_Benchmarks_Track.html) | Multi-file websites generated from requirements, including interaction | 647 manually refined cases executed by a web-navigation agent; separate appearance judge | Good methodological reference for functionality | Published model set is old; automated verifier is fallible |
+| [DesignBench](https://github.com/WebPAI/DesignBench) | Generation, edit, repair, and compile repair in React, Vue, Angular, and vanilla HTML/CSS | Render similarity, compilation/code checks, and an MLLM judge | Good diagnostic taxonomy and reproducible local eval | Static public dataset and older model coverage; small isolated projects rather than mature repositories |
+| [ArtifactsBench](https://github.com/Tencent-Hunyuan/ArtifactsBenchmark) | Interactive visual artifacts, including components, SVGs, and games | Temporal screenshots plus checklist-guided MLLM judge | Useful automated secondary signal | Judge bias/circularity; artifact generation rather than repository maintenance |
+| [UI-Bench](https://arxiv.org/abs/2508.20410) | Holistic visual craft of text-to-app products | 4,000+ blinded expert pairwise judgments | Strong evidence about tool/product output quality | Ranks whole tools, not base models; explicitly excludes accessibility, load time, and code quality |
+| [Design2Code](https://arxiv.org/abs/2403.03163) | Screenshot-to-HTML reproduction | Automatic visual metrics validated against human rankings | Useful for visual-fidelity diagnostics | Static, mostly single-page reproduction; no repository or interaction evidence |
+
+## 1. The best current live signal: Code Arena WebDev
+
+Code Arena asks users to submit a web-development prompt, lets two anonymous
+models build deployable apps, and has users interact with both outputs before
+voting. The current implementation records the agent trajectory and aggregates
+pairwise preferences using a Bradley-Terry-style ranking. Voters are instructed
+to consider functionality, usability, fidelity, design, taste, and aesthetics
+([methodology](https://arena.ai/blog/code-arena/),
+[original WebDev methodology](https://arena.ai/blog/webdev-arena/)).
+
+At this snapshot the official leaderboard reports 506,528 votes:
+
+| Configuration | Score | Status |
+|---|---:|---|
+| Kimi K3 | 1678 ± 17 | preliminary |
+| Claude Fable 5 | 1634 ± 12 | established |
+| GPT-5.6 Sol `xhigh`, Codex harness | 1630 ± 11 | established |
+
+Fable and Sol have overlapping intervals. Kimi's apparent lead is relevant but
+must remain marked preliminary. These are **model-plus-harness observations**,
+not intrinsic model constants: only the Sol label exposes a comparable effort
+setting, and it explicitly names the Codex harness.
+
+The source is stronger than a static visual benchmark because prompts are live,
+outputs are interactive, identities are hidden during voting, and the pool
+changes with deployed models. It is also weaker than executable pass/fail tests:
+a preference vote blends correctness and taste, evaluator expertise varies, the
+prompt population is self-selected, and leaderboard position changes with the
+opponent pool.
+
+The seven current domains are also important for routing: reference-based
+design, brand/marketing, data/analytics, consumer products, gaming,
+simulations, and content-creation/editing tools. They were derived from more
+than 250,000 filtered prompts, and domain leaderboards use the same evaluation
+method ([category methodology](https://arena.ai/blog/new-categories-code-arena/)).
+Therefore the routing source should ingest category scores rather than collapse
+everything into one `frontend` number.
+
+## 2. The strongest prospective controlled benchmark: Vision2Web
+
+Vision2Web most closely matches the missing end-to-end contract. Its 193 tasks
+contain 918 prototype images and 1,255 test cases across 16 categories. It has
+three progressively harder levels:
+
+- static responsive webpages, evaluated separately on desktop, tablet, and
+  mobile;
+- interactive multi-page frontend applications;
+- long-horizon full-stack websites.
+
+It evaluates visual similarity with a VLM judge and functional behaviour with a
+GUI-agent verifier. Submissions include the model **and** agent framework, which
+is the right unit of evidence for an agent-routing system
+([project](https://vision2web-bench.github.io/),
+[paper](https://arxiv.org/abs/2603.26648),
+[submission/evaluation contract](https://huggingface.co/datasets/zai-org/Vision2Web-Leaderboard)).
+
+The leaderboard is seasonal: tasks and evaluators may change, old submissions
+are re-evaluated where possible, and scores from different seasons are
+explicitly not comparable. As of this snapshot, the **current season has no
+results**. Historical scores visible on the project page are useful for
+understanding the benchmark, but must not yet drive a current policy.
+
+This is the first source worth promoting to a primary routing adapter once the
+active season has enough submissions and reports judge versions, sample size,
+and confidence. Until then it should be represented as `candidate`, not as an
+empty score or inherited historical winner.
+
+## 3. Existing-repository frontend work: SWE-bench Multimodal
+
+SWE-bench Multimodal contains 517 test instances from 12 mainly JavaScript
+repositories. Issues include screenshots of bugs, mockups, diagrams, and visual
+error context. The underlying collection contains web frameworks, UI component
+libraries, mapping, charting, diagramming, and syntax-highlighting projects.
+Success requires the repository's fail-to-pass and pass-to-pass tests to pass
+([benchmark overview](https://www.swebench.com/multimodal),
+[ICLR paper](https://proceedings.iclr.cc/paper_files/paper/2025/file/07d6332ae36730707fddddba736d7b6c-Paper-Conference.pdf)).
+
+This makes it the best available answer to:
+
+> Can this agent-model-harness configuration understand a visual frontend issue,
+> navigate an established JavaScript repository, and land a test-passing fix?
+
+It does not answer whether the same configuration creates a tasteful new UI.
+Only 69 tasks use pixel-level visual testing; many other tasks are correctness
+or repository-navigation problems with visual context. The original baselines
+are stale, and the current official leaderboard still mixes different agents
+and models. Historical GitHub-derived tasks also remain exposed after release,
+so future leaderboard improvements require contamination caution.
+
+The OpenHands Index is useful as a normalized view: it runs a human-verified
+frontend subset through one OpenHands SDK and reports ability, cost, and runtime
+([Index methodology](https://www.openhands.dev/blog/openhands-index)). It should
+be stored as a separate harness observation, not generalized to Claude Code or
+Codex.
+
+## 4. Functional greenfield work: WebGen-Bench and newer diagnostics
+
+WebGen-Bench creates multi-file website codebases from 101 requirements and
+tests them with 647 operation/expected-outcome cases. Two PhD reviewers refined
+the cases. A WebVoyager-based agent executes them and returns `YES`, `NO`, or
+`PARTIAL`; reported agreement with manual testing ranged from 86.1% to 94.4%
+for the three evaluated model sets. Appearance was judged separately. In the
+published evaluation, Claude 3.5 Sonnet led appearance at 3.0/5 while
+DeepSeek-R1 led the general-model functional score at 27.8%
+([paper and evaluator validation](https://proceedings.neurips.cc/paper_files/paper/2025/file/6841eed8bb6a2ec49e49235c8115efee-Paper-Datasets_and_Benchmarks_Track.pdf)).
+
+That separation between functional and visual results is valuable. The scores
+are not a present-day routing table: models and harnesses are old, only 101
+projects are used, and an agent judging another agent adds a measurable error
+layer.
+
+DesignBench is a good reusable taxonomy for isolated frontend work. Its 900
+samples span React, Vue, Angular, and vanilla HTML/CSS across initial generation,
+edits, repair of visual defects, and compilation repair. Defect categories
+include occlusion, crowding, overlap, alignment, color/contrast, and overflow.
+It also found very low adoption of framework-native component structures and
+low UI-issue detection accuracy in the tested 2024/2025-era models
+([paper](https://arxiv.org/abs/2506.06251),
+[harness](https://github.com/WebPAI/DesignBench)). It is suitable for local
+diagnostic evaluation but not as a live winner feed.
+
+ArtifactsBench offers 1,825 component, visualization, and interactive-artifact
+tasks. It evaluates source plus three-step rendered screenshots against
+task-specific checklists with a multimodal judge and reports 94.4% ranking
+consistency with WebDev Arena. That makes it a useful automated corroborating
+source, but not a substitute for independent human judgment
+([official repository](https://github.com/Tencent-Hunyuan/ArtifactsBenchmark)).
+
+FrontendBench is conceptually attractive: its paper describes 148 prompt/test
+pairs across five component-complexity levels, browser execution, generated
+test scripts, and roughly 90.5% expert agreement
+([paper](https://arxiv.org/abs/2506.13832)). However, the paper's code/data
+release is still described as forthcoming. Until runnable artifacts and stable
+results exist, it is not practical as a source adapter.
+
+## 5. Taste, accessibility, and responsiveness remain distinct gaps
+
+UI-Bench is the strongest controlled evidence for **taste**. It uses 30 prompts,
+300 generated sites, more than 4,000 comparisons, and 194 invited professionals.
+Tool identities and left/right placement are hidden; the forced-choice question
+is which project the expert would be more likely to deliver to a client. Its
+authors deliberately avoid CLIP/FID-style automatic metrics as the primary
+endpoint because those proxies can mis-rank aesthetic preference
+([paper and protocol](https://ar5iv.labs.arxiv.org/html/2508.20410)).
+
+But UI-Bench ranks complete text-to-app products, not base models. Templates,
+asset pipelines, orchestration, repair passes, and post-processing all affect
+the result. It also explicitly excludes accessibility, load time, and code
+quality, and evaluates desktop layouts. It cannot justify a model route by
+itself.
+
+There are smaller accessibility studies. One tested eleven component patterns
+from WCAG 2.1 across ChatGPT 4o, Copilot Pro, Claude 3.7 Sonnet, and Grok 3
+([published study](https://doi.org/10.1007/s10209-025-01250-2)); another found
+308 WCAG 2.2 and cognitive-accessibility errors across six generated sites
+([ASSETS 2025 paper](https://doi.org/10.1145/3663547.3759755)). These establish
+that accessibility is not implied by visual quality. Their tiny task/model sets
+and stale versions make them diagnostics, not routing feeds.
+
+Vision2Web is the clearest emerging responsiveness measure because Level 1 has
+desktop, tablet, and mobile scores. With its current season empty, there is not
+yet a current model comparison that jointly and robustly measures
+responsiveness. DesignBench's overflow and contrast repairs are useful but do
+not amount to WCAG conformance or a viewport matrix.
+
+## 6. Benchmarks that do not qualify as frontend-building evidence
+
+- WebArena, VisualWebArena, BrowserGym, WorkArena, and WebVoyager primarily test
+  an agent **operating existing websites**. They are browser-use evidence, not
+  evidence that the agent can implement those websites
+  ([VisualWebArena](https://github.com/web-arena-x/visualwebarena),
+  [BrowserGym](https://github.com/ServiceNow/BrowserGym)).
+- WebSight is a synthetic screenshot/HTML training dataset, not a comparative
+  agent benchmark.
+- Design2Code is useful for screenshot fidelity but omits realistic
+  interaction and repository integration.
+- Raw JavaScript/TypeScript subsets of general coding benchmarks do not become
+  frontend benchmarks unless the tasks actually exercise rendering,
+  interaction, or visual requirements.
+
+## 7. Implications for a routing evidence catalog
+
+Every observation should preserve at least:
+
+```yaml
+workload:
+  lifecycle: greenfield | edit | repair
+  frontend_domain: reference-design | marketing | analytics | product | game | simulation | editor
+  repository_context: isolated | existing-repository
+  quality_axis: visual-preference | visual-fidelity | functional | accessibility | responsive
+
+configuration:
+  surface: codex | claude-code | openhands | other
+  harness: concrete-version
+  model: concrete-version
+  effort: low | medium | high | xhigh | max | unknown
+
+evidence:
+  source: code-arena | vision2web | swe-bench-multimodal | other
+  benchmark_revision: concrete-revision-or-season
+  observed_at: yyyy-mm-dd
+  score: value
+  uncertainty: value-or-unknown
+  sample_size: value-or-unknown
+  status: established | preliminary | candidate | stale
+```
+
+Routing rules should then follow these constraints:
+
+1. For greenfield frontend, prefer current Code Arena **domain-specific**
+   evidence, tempered by local outcomes.
+2. For existing-repository visual repair, prefer SWE-bench Multimodal or a
+   normalized OpenHands view.
+3. Promote Vision2Web when the active season has enough comparable submissions;
+   never carry a previous-season winner forward silently.
+4. Do not infer an effort curve from Code Arena. Combine its frontend evidence
+   with a separate effort benchmark such as DeepSWE and mark the inference.
+5. Treat model-plus-harness as the observed unit. A Claude model in OpenHands is
+   not evidence for the same model in Claude Code without corroboration.
+6. Accessibility and responsiveness require explicit constraints and local
+   verification until broader current leaderboards exist.
+7. Keep the maintainer's dated experience as legitimate local evidence. Public
+   benchmarks calibrate it; they do not automatically overwrite it.
+
+## Conclusion
+
+Frontend routing is no longer evidence-free. Code Arena already supports the
+claim that Kimi, Claude, and OpenAI configurations differ in real interactive
+frontend preference, and it currently places Kimi K3 first provisionally, with
+Fable 5 and Sol `xhigh` close behind. That result is not enough to derive a
+universal model-effort rule.
+
+The durable solution is a multi-source routing catalog: live human preference
+for greenfield work, controlled visual/functional evaluation when Vision2Web is
+populated, executable repository repair evidence, a separate effort curve, and
+local calibration for the exact product and design system.