@ikon85/agent-workflow-kit 0.43.0 → 0.44.0

package/docs/research/benchlm-routing-source.md

@@ -1,198 +0,0 @@
-# BenchLM as a routing-evidence source
-
-**Researched:** 2026-07-22  
-**Question:** What role, if any, should BenchLM play in the Kit's evidence-backed
-model and effort routing, and what does that imply for setup and kit-update?
-
-## Decision
-
-Use BenchLM as a **catalog, discovery, and corroboration source**, not as an
-authoritative routing source.
-
-BenchLM is unusually useful for finding new model families, benchmarks,
-pricing changes, supersession relationships, and gaps in current evidence. It
-must not directly choose a dispatch route, especially an effort level. A route
-observation still has to come from the benchmark owner and retain the complete
-configuration identity: model, effort, agent/harness, benchmark version,
-quality metric, uncertainty, and task cost.
-
-This gives the source hierarchy:
-
-1. **Benchmark owner artifacts** (DeepSWE, Artificial Analysis, OpenHands,
-   Code/Design Arena) provide decision evidence.
-2. **BenchLM** discovers and cross-checks those sources and model releases.
-3. **Local user evidence and preferences** calibrate the resulting policy.
-4. The approval-gated policy reconciler proposes, but never silently applies,
-   routing changes.
-
-## What BenchLM contributes
-
-BenchLM currently tracks 290 models and 323 benchmarks. Its useful coverage
-includes agentic work, coding, multimodal work, model pricing, context windows,
-runtime signals, release metadata, and supersession chains. Its methodology
-currently gives ranking weight to three agentic and five coding benchmarks;
-many more sources, including DeepSWE and most frontend sources, are display-only
-context ([methodology](https://benchlm.ai/methodology)).
-
-Frontend coverage is useful primarily as a source map. The current focused page
-combines sourced React Native Evals, Design2Code, Vision2Web, and related
-browser-task evidence, but calls itself a reporting family rather than a
-weighted category, contains only seven eligible models, and warns that models
-may have as few as two source rows
-([frontend/app-dev view](https://benchlm.ai/best/frontend-app-dev)). The owners
-remain more authoritative: [React Native Evals](https://rn-evals.vercel.app/)
-publishes agent success, token, and cost results, while
-[Vision2Web](https://vision2web-bench.github.io/) explicitly evaluates
-model-plus-framework submissions across visual, interactive, and full-stack
-tasks.
-
-BenchLM also exposes confidence separately from score, 90% score intervals,
-`Supported` versus `Estimated` states, source-coverage counts, and a dated
-BenchAlign method/frozen-input identifier. That is valuable for shortlisting
-and for detecting insufficient evidence, but it is confidence in BenchLM's
-composite inference rather than a pass-rate interval for our exact workload
-([confidence method](https://benchlm.ai/benchmark-confidence)).
-
-## Why it cannot drive routing directly
-
-### Effort and harness can disappear
-
-BenchLM's exported Sol model row records DeepSWE as `72.7`, but not the effort,
-harness, confidence interval, or cost
-([models export](https://benchlm.ai/data/models.json)). The owner artifact shows
-that this number is specifically `gpt-5-6-sol + mini-swe-agent + max`: 72.7%
-Pass@1 over four whole-benchmark runs, with a 95% run-to-run interval and about
-$8.39 mean cost per scored attempt
-([DeepSWE v1.1 artifact](https://deepswe.datacurve.ai/artifacts/v1.1/leaderboard-live.json)).
-
-This collapsing is fatal for model-plus-effort selection. Some older model
-families have explicit `high` or `max` variants in the BenchLM catalog, while
-current Sol, Terra, Luna, and Fable evidence can be folded into one base-model
-row. The overall/category scores therefore describe mixed published
-configurations, not a consistent effort curve.
-
-### Composite rankings are editorial inference
-
-BenchAlign normalizes unlike protocols, combines source families, and imputes
-missing evidence with wider uncertainty. BenchLM publishes the method version,
-frozen-input identifier, weighted benchmark list, and dated output, but its
-public downloads reference internal source files rather than a published,
-locally reproducible BenchAlign implementation. The composite is reproducible
-as a cached dated output, not independently rebuildable by the Kit from public
-code ([methodology](https://benchlm.ai/methodology)).
-
-Raw source rows are more useful than the composite, but the current
-machine-readable model export is a numeric benchmark map without per-row source
-URLs or complete execution settings. The Kit must follow the benchmark catalog
-to the owner rather than treating `models.json` as normalized routing evidence.
-
-### Current semantic inconsistencies require defensive ingestion
-
-The same-day exports currently contain examples that a production adapter must
-reject or downgrade:
-
-- BenchLM's DeepSWE Markdown page says that source metadata is preserved but
-  reports zero leaderboard models, while `models.json` contains six DeepSWE
-  scores.
-- `benchmarks.json` currently contains duplicate `sweMultimodal` definitions.
-- Its Design2Code and Vision2Web catalog entries currently point to a provider
-  documentation page rather than the benchmark owners. The official
-  [Design2Code repository](https://github.com/NoviScl/Design2Code) and
-  [Vision2Web site](https://vision2web-bench.github.io/) show the correct
-  provenance.
-
-These may be transient publishing defects, but they make last-known-good,
-semantic validation, and owner verification mandatory.
-
-## Machine-readable access, cadence, and legal posture
-
-BenchLM documents MIT-licensed JSON downloads for leaderboard, models,
-benchmarks, pricing, speed, and comparisons. Each export currently has
-`schemaVersion`, `generatedAt`, and `sourceLastUpdated`; `updates.json` provides
-a cheap change feed. BenchLM says it refreshes several times per week and often
-within hours of major launches ([dataset documentation](https://benchlm.ai/data)).
-
-The adapter should use the documented `/data/*.json` downloads and
-`updates.json`, not scrape rendered pages. Fetch at low cadence, send ordinary
-conditional requests where supported, retain attribution, and persist the raw
-snapshot hash and generation timestamp.
-
-There is one governance warning: the current Terms page still visibly contains
-launch placeholders for operator identity, jurisdiction, effective date, and
-contact details. It permits published downloads under their stated licenses
-and non-degrading automated access, but this unfinished legal surface reduces
-operational confidence. The MIT dataset declaration is clear; nonetheless the
-adapter should not depend on high-frequency or undocumented endpoints
-([dataset license](https://benchlm.ai/data),
-[terms](https://benchlm.ai/terms)). Underlying benchmark data remains subject
-to its original publisher's license.
-
-## Concrete adapter contract
-
-`BenchLmCatalogSource` should ingest only:
-
-- model/provider/family and release/supersession metadata;
-- current pricing and runtime hints;
-- benchmark definitions and owner links;
-- coverage, freshness, `Supported`/`Estimated`, and confidence metadata;
-- the update feed as an invalidation and discovery trigger.
-
-It should emit `candidate-discovered`, `source-changed`, `pricing-changed`, and
-`evidence-gap` events. It must not emit an authoritative
-`RouteObservation(model, effort, harness)` unless an owner adapter has verified
-the complete row.
-
-Every refresh must:
-
-1. validate the export schema and cross-file referential integrity;
-2. reject duplicate benchmark identities and implausible source changes;
-3. retain the last-known-good snapshot on failure;
-4. classify aggregate scores as `corroborating`, never `decisive`;
-5. resolve relevant rows through the original source adapter;
-6. show a human-readable diff before a personal policy changes.
-
-## Setup and kit-update implication
-
-BenchLM improves discovery, but it does not change ownership: setup asks which
-**surfaces/providers the user can and wants to use**, not which model currently
-tops a leaderboard. Availability, policy, and benchmark ability are separate.
-
-For a first setup or an existing installation without a routing profile:
-
-1. Detect installed/supported surfaces where possible and ask the user to
-   confirm the common choices (for example Claude Code, Codex, and explicitly
-   offered additional providers such as Google or Moonshot/Kimi).
-2. Record availability and dispatch capability separately: subscription/API,
-   allowed cross-provider execution, per-spawn model control, and per-spawn
-   effort control.
-3. Ask for the optimization preference and local overrides, or allow the safe
-   `inherit` mode.
-4. Use BenchLM to discover current candidate families, then owner sources to
-   construct the proposed model-plus-effort policy.
-5. Preview the policy and require approval before activation.
-
-The list of “common AIs” must be data-driven and versioned, not hard-coded in a
-skill prompt. BenchLM can nominate newly common candidates, while surface
-capability checks determine whether they are actually selectable.
-
-`kit-update` should install updated schemas, adapters, and provider-catalog
-logic. It should run a short routing preflight when the profile is absent,
-invalid, stale, references a removed model, or when a newly detected surface
-changes available choices. It should otherwise report `still valid` without
-re-running the whole interview. An update may refresh and cache evidence, but
-must never rewrite the user's provider choices, model routes, effort routes, or
-global configuration without a separately approved reconcile transaction.
-
-## Final classification
-
-| Use | Classification |
-|---|---|
-| Discover new models, providers, benchmarks, and releases | Recommended |
-| Detect evidence/pricing/freshness changes | Recommended with validation |
-| Corroborate owner-source observations | Useful |
-| Select a candidate shortlist | Advisory only |
-| Select model plus effort for dispatch | Rejected |
-| Treat BenchAlign overall/category score as capability truth | Rejected |
-
-BenchLM therefore adds real value to the planned routing wave, but as the
-system's **radar**, not its **autopilot**.

package/docs/research/consumer-owned-protocol-files.md

@@ -1,238 +0,0 @@
-# Consumer-owned protocol files: evolution without overwrite or starvation
-
-Verified on 2026-07-22 against `agent-workflow-kit` 0.33.0 in the issue #197
-worktree. This note researches how an updater should handle a locally adapted
-file that is also part of an evolving upstream protocol. It is decision input,
-not a product change.
-
-## Finding
-
-A protocol-critical file must not be both an immutable consumer-owned fork and
-the upstream kit's active protocol authority under the same path. Those two
-contracts are incompatible: permanent consumer ownership prevents protocol
-evolution, while ordinary upstream ownership cannot preserve local policy.
-
-The durable model is to split the mixed file into:
-
-1. an upstream-owned, immutable **protocol core** that ships and evolves with
-   the kit; and
-2. a consumer-owned, structured **extension overlay** containing only local
-   declarations and policy.
-
-The staged update candidate composes core plus overlay and validates the whole
-result before activation. A one-time migration extracts existing local
-extensions from the mixed file. Ambiguity blocks that migration rather than
-overwriting either side. A consumer that needs to change executable protocol
-behavior has made a real code fork and needs a separate artifact/version line,
-not `origin: consumer` on the upstream path.
-
-## Current local contract and failure mode
-
-The accepted divergence policy defines a consumer-owned path as a deliberate
-fork that receives no overwrite, conflict report, deletion prompt, or future
-kit improvement ([ADR 0001](../adr/0001-consumer-divergence-policy.md)). The
-reconciler implements that literally: as soon as an installed entry has
-`origin: 'consumer'`, it carries the old manifest record forward and skips all
-upstream comparison
-([`updateReconcile.mjs`](../../src/lib/updateReconcile.mjs)). This is coherent
-for an independent local file, but it is update starvation for a protocol
-authority.
-
-Issue #196 exposed the concrete mixed-file shape. The readiness harness and
-skills depend on `.claude/skills/skill-manifest.json`, but a real consumer owns
-and extends that same registry. The published 0.32.0 artifact was internally
-coherent; the consumer combined 15 local skill records and local notes with an
-older upstream registry, while the newer kit added five skills, 23 readiness
-capabilities, and 15 readiness declarations. Skipping the owned registry thus
-allowed a new executable harness to run against an old protocol declaration.
-
-The candidate transaction already has the right enforcement location: it
-checks candidate hashes and destination races before activation and rolls back
-all touched paths on failure
-([`updateCandidate.mjs`](../../src/lib/updateCandidate.mjs)). What is missing is
-a compositional protocol model and validator; copying more of the consumer or
-adding more path exclusions does not solve that ownership contradiction.
-
-## Comparable primary-source patterns
-
-| System | Primary-source behavior | Lesson for the kit |
-|---|---|---|
-| Git three-way file merge | `git merge-file` combines the changes from a recorded common base to the upstream side with the local side; overlapping edits become explicit conflicts and a clean merge has a distinct successful exit status. [Git documentation](https://git-scm.com/docs/git-merge-file) | The stored installed hash/base makes a one-time three-way migration possible. A merge is evidence to inspect, not permission to silently overwrite. Text merge is unsuitable as the permanent extension mechanism because protocol identity and invariants are semantic, not line based. |
-| GitHub forks | Syncing a fork explicitly fetches/merges upstream. Conflicts require a pull request or manual resolution; forcing sync overwrites the destination. [GitHub documentation](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork) | A true code fork has its own history and must deliberately integrate upstream. It cannot also promise automatic compatibility under the original identity. |
-| Kubernetes Server-Side Apply | Kubernetes tracks ownership per structured field. An attempted value change to another manager's field conflicts; force is an explicit ownership transfer. A manager can also relinquish a field or share ownership by agreeing on the same value. [Kubernetes documentation](https://kubernetes.io/docs/reference/using-api/server-side-apply/) | Ownership below the file level works for schema-defined data. Core fields and consumer extension fields can have separate managers, with collisions rejected and transfers explicit. This model should not be imitated for arbitrary source-code lines. |
-| Debian configuration files | Debian requires local configuration changes to survive upgrades. It distinguishes package conffiles from files generated and maintained by idempotent scripts, forbids mangling user configuration without asking, and warns that mixing the two handling styles causes repeated upgrade prompts. [Debian Policy](https://www.debian.org/doc/debian-policy/ch-files.html#behavior) `dpkg` separately exposes explicit keep-old/install-new/default choices for changed conffiles. [dpkg manual](https://manpages.debian.org/bookworm/dpkg/dpkg.1.en.html) | Choose one ownership model per artifact. Do not make one file simultaneously an upstream payload and a consumer-generated extension. Migrations must be idempotent, previewed, and preserve the old value on failure. |
-| Terraform provider state | Terraform stores a resource schema version, asks the provider to upgrade older state before planning, rejects unsupported prior versions, and retains prior state when an upgrader reports errors. Each upgrader must produce a complete current-version state. [Terraform Plugin Framework](https://developer.hashicorp.com/terraform/plugin/framework/resources/state-upgrade) | Give the overlay an explicit schema version and deterministic upgraders. Validate the fully upgraded/composed state, not only the changed records. Unsupported or ambiguous inputs block activation without destroying the prior state. |
-| Kubernetes API versioning | Kubernetes can serve old and new API versions concurrently, converts between representations, migrates stored objects separately, and removes an old version only after clients and stored objects have migrated. [Kubernetes CRD versioning](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/) | Compatibility needs an announced window: introduce/read both, migrate, count adoption, stop serving old, then remove conversion. A deprecation warning alone is not migration evidence. |
-| Homebrew taps | An extracted or alternative formula lives in a separate Git repository/tap; its owner is responsible for maintenance, deprecations, and security updates. Naming or qualification avoids collision with the core formula. [Homebrew tap documentation](https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap) | A real executable fork needs a distinct artifact namespace and maintenance owner. It should not silently shadow the kit's protocol core at the same path. |
-
-## Data/config extension versus code fork
-
-These cases need different contracts.
-
-### Structured consumer extension
-
-Use an overlay when the consumer is adding data the upstream runtime already
-knows how to interpret: local skill registrations, local capability evidence,
-board values, activation decisions, or project policy. The upstream owns the
-schema and executable semantics; the consumer owns values in a declared
-extension namespace.
-
-The overlay should have:
-
-- its own consumer-native path and `schemaVersion`;
-- stable record identities and an explicit allowed extension namespace;
-- no ability to redefine reserved upstream/core records;
-- deterministic, idempotent schema upgraders;
-- a compatibility declaration tying overlay schema versions to supported
-  protocol-core versions; and
-- a composed-state validator that checks references, uniqueness, required
-  declarations, and executable/declaration parity.
-
-This is the applicable model for the mixed skill/readiness manifest. The
-consumer's local skills are data extensions; the kit's shipped skill registry,
-readiness schema, and harness contract are protocol core.
-
-### Executable code fork
-
-Use a true fork when the consumer changes parsing, evaluation, migration, or
-other executable protocol behavior. The fork must then have a distinct package
-or path identity, its own version and release channel, an explicit upstream
-remote/base, and a deliberate sync/rebase/merge process. Compatibility with the
-upstream package is no longer automatic.
-
-Marking such code `consumer`-owned under the upstream path hides that reality:
-it keeps the name but abandons the update channel. The Homebrew and GitHub
-models make the trade-off honest: local history is preserved, upstream sync is
-possible, conflicts are explicit, and maintenance responsibility moves to the
-fork owner.
-
-## Recommended candidate and ownership contract
-
-### 1. Define protocol groups, not only independent files
-
-The package manifest should declare every protocol-critical group and its
-members, for example:
-
-```text
-readiness protocol
-  core registry:       kit-owned
-  runtime harness:     kit-owned
-  skill preflights:    kit-owned
-  consumer overlay:    consumer-owned structured data
-  validator/migrators: kit-owned
-```
-
-The candidate is valid only if every group is coherent as a whole. A
-consumer-owned member can therefore no longer make validation disappear.
-
-### 2. Keep upstream protocol core unownable under its canonical identity
-
-`own` should refuse a protocol-core path and point to the supported overlay or
-fork route. Existing owned core paths are legacy states to migrate, not states
-to perpetuate. This narrows ADR 0001 rather than weakening it: ordinary
-consumer-owned files remain immutable and skipped; protocol core is no longer
-an eligible ownership unit.
-
-### 3. Compose and validate before activation
-
-For every update, materialize the complete intended kit end state, load and
-upgrade the consumer overlay, compose both, and run protocol-specific
-invariants. Activation remains all-or-nothing. A bad overlay blocks the
-protocol group with a targeted diagnostic; it never causes the updater to
-overwrite local data or to activate a mixed-version group.
-
-Collision rules should be structural:
-
-- a consumer may add records only in its declared namespace;
-- a consumer record that collides with a core identity is rejected unless the
-  schema explicitly defines a non-overriding augmentation;
-- consumer values cannot replace core executable references or protocol
-  versions;
-- dangling references and unsupported overlay versions block activation; and
-- ownership transfer is an explicit user decision, never a `--yes` default.
-
-### 4. Migrate the legacy mixed file once
-
-For the existing combined registry, the staged candidate should retain three
-inputs: the last installed upstream base, the live consumer file, and the new
-upstream core. Migration should semantically classify records rather than
-blindly line-merge JSON:
-
-1. identify records unchanged from the recorded upstream base;
-2. extract clearly consumer-only records and allowed local policy into the new
-   overlay;
-3. install the complete new upstream core;
-4. compose and validate the result; and
-5. preview the extraction, backups, ownership changes, and any ambiguous rows.
-
-If provenance is ambiguous—for example, a consumer modified an upstream record
-instead of adding a namespaced one—the update stops at a user gate. The user
-chooses to port the change upstream, express it through a supported overlay
-field, or create a real fork. A raw three-way merge may help produce the review
-diff, but it must not decide semantic ownership.
-
-### 5. Version, negotiate, deprecate, then remove
-
-The protocol core and overlay schema need independent explicit versions and a
-supported compatibility matrix. Rollout follows the Kubernetes/Terraform
-pattern:
-
-1. ship readers and validators for both the legacy combined representation and
-   the new split representation;
-2. ship the idempotent migration and preview it in the candidate;
-3. activate core plus overlay only after full validation;
-4. report a fresh adoption count (`X of Y` consumers or known legacy states),
-   not an assumed completion;
-5. stop creating legacy state and make old-state diagnostics actionable; then
-6. remove the legacy reader only after all supported prior states have a
-   migration path and known consumers have migrated.
-
-Without remote telemetry, the evidence is local manifest/schema state plus
-consumer update/CI results. The kit should therefore use a release-bounded
-support window and a deterministic audit command, not claim global adoption.
-
-## Proposed rollout for issues #196/#197
-
-1. **Inventory and classify.** Derive all protocol-critical groups from shipped
-   executable references and manifests. Count every member and mark it core,
-   overlay, generated evidence, or independent file.
-2. **Add validation before enforcement.** Build the kit-only candidate and
-   protocol-group validator. Initially report legacy mixed ownership and prove
-   that a #196-style harness/registry mismatch is rejected.
-3. **Introduce the split readiness registry.** Ship an upstream-owned core
-   registry plus a consumer-owned extension overlay. The runtime reads the
-   composed view; new consumers start split.
-4. **Adopt existing consumers transactionally.** Extract Testreporter's 15
-   consumer-only skill records and local policy into the overlay, install the
-   current core, validate all core and local records, and change ownership only
-   as part of the verified candidate activation. Preserve a backup/recovery
-   receipt.
-5. **Guard the seam.** Reject future `own` operations on protocol-core paths;
-   point users to the overlay for data extensions or to a separately named
-   package/path for executable forks.
-6. **Sunset the combined format.** Keep the legacy reader for a declared
-   compatibility window, count locally known adoptions, then remove it in a
-   release that explicitly declares the minimum supported overlay/core schema.
-
-## Rejected rollout shapes
-
-- **Silently overwrite the owned file:** destroys legitimate local behavior
-  and violates the existing consumer contract.
-- **Keep skipping it forever:** preserves bytes but starves the protocol and
-  recreates #196 on every coupled evolution.
-- **Automatically disown it:** turns an ownership decision into an implicit
-  destructive migration.
-- **Blind three-way merge on every update:** reduces conflicts but leaves a
-  mixed authority and cannot validate semantic record identity.
-- **Field ownership for arbitrary code:** line/AST ownership is not a stable
-  extension API; local executable changes need an explicit fork.
-- **Fork the whole protocol for local records:** imposes permanent merge and
-  security-update work where a narrow data overlay is sufficient.
-
-## Decision implication
-
-The current definition “owning a path means forgoing all future kit
-improvements to it” remains valid for genuine independent forks. It should no
-longer be offered for canonical protocol-core paths. For those paths, the
-supported local customization mechanism is a versioned consumer overlay; the
-honest alternative is a separately identified and maintained code fork.