@ikon85/agent-workflow-kit 0.34.4 → 0.34.5

package/docs/research/model-effort-routing-benchmarks.md

@@ -0,0 +1,261 @@
+# Benchmark evidence for model and effort routing
+
+**Researched:** 2026-07-22  
+**Question:** How should current benchmark evidence inform a provider-neutral,
+user-owned routing policy without turning today's model and effort winners into
+permanent Kit defaults?
+
+## Conclusion
+
+The evidence supports a **mutable, workload-specific routing policy**, not a
+universal mapping such as `mechanical -> Luna/high`, `development ->
+Terra/max`, or `judgment -> Fable/high`.
+
+Three findings matter for the current design:
+
+1. **Effort scaling is model-specific.** On DeepSWE, moving from `xhigh` to
+   `max` adds about 9.4 pass-rate points for GPT-5.6 Terra and 10.3 points for
+   Luna. For Claude Fable 5, `high`, `xhigh`, and `max` overlap within the
+   reported uncertainty while average cost rises from $9.18 to $21.63 per
+   trial.
+2. **The best route depends on the workload and objective.** Sol leads the
+   current Artificial Analysis coding-agent aggregate and DataCurve DeepSWE;
+   Fable leads Sol on Artificial Analysis' repository-Q&A component and has
+   separate strengths in knowledge work. Luna is exceptionally cost-efficient.
+   Artificial Analysis' intelligence results place Luna and Sol, not Terra, on
+   the cross-family cost/quality frontier.
+3. **Benchmark rows are configurations, not timeless model facts.** Harness,
+   effort, fallback behavior, benchmark version, provider pricing, retries, and
+   access failures all affect the result. A model mapping inferred today can be
+   stale when a new family, alias, effort level, or benchmark revision lands.
+
+The Kit should therefore own a routing schema, evidence format, resolver, and
+reconciliation workflow. The user or organization should own the current
+model/effort choices. Benchmarks can generate a dated recommendation, but may
+not silently rewrite that policy.
+
+## Current evidence
+
+### DataCurve DeepSWE v1.1
+
+DeepSWE contains 113 original, long-horizon software-engineering tasks across
+91 repositories and five languages. The current leaderboard was updated on
+2026-07-21. Every row below uses the same `mini-swe-agent` harness and four
+whole-benchmark runs; pass-rate uncertainty is DataCurve's reported 95%
+run-to-run interval. Costs are average API cost per completed trial
+([leaderboard](https://deepswe.datacurve.ai/),
+[machine-readable leaderboard artifact](https://deepswe.datacurve.ai/artifacts/v1.1/leaderboard-live.json)).
+
+| Configuration | Pass@1 | 95% half-width | Avg cost/task | Attempts |
+|---|---:|---:|---:|---:|
+| GPT-5.6 Luna medium | 11.3% | ±0.8 | $0.22 | 452 |
+| GPT-5.6 Luna high | 44.2% | ±2.9 | $0.78 | 452 |
+| GPT-5.6 Luna xhigh | 56.9% | ±2.2 | $1.54 | 452 |
+| **GPT-5.6 Luna max** | **67.2%** | **±4.0** | **$3.03** | 448 |
+| GPT-5.6 Terra medium | 35.1% | ±3.4 | $0.58 | 450 |
+| GPT-5.6 Terra high | 53.8% | ±4.3 | $1.13 | 452 |
+| GPT-5.6 Terra xhigh | 60.2% | ±2.1 | $2.13 | 452 |
+| **GPT-5.6 Terra max** | **69.6%** | **±2.6** | **$4.95** | 451 |
+| GPT-5.6 Sol medium | 61.1% | ±1.6 | $1.86 | 452 |
+| **GPT-5.6 Sol high** | **69.4%** | **±1.4** | **$3.47** | 451 |
+| GPT-5.6 Sol xhigh | 70.7% | ±0.8 | $4.70 | 451 |
+| GPT-5.6 Sol max | 72.7% | ±2.8 | $8.39 | 450 |
+| Claude Fable 5 medium | 65.4% | ±4.4 | $6.09 | 436 |
+| **Claude Fable 5 high** | **68.6%** | **±1.1** | **$9.18** | 430 |
+| Claude Fable 5 xhigh | 69.9% | ±3.2 | $13.41 | 452 |
+| Claude Fable 5 max | 69.7% | ±4.0 | $21.63 | 436 |
+
+What this does and does not establish:
+
+- **Luna max is strongly efficient for this workload.** It reaches 67.2% for
+  $3.03, statistically overlaps Terra max and Fable's higher-effort rows, and
+  is cheaper than all of them.
+- **Terra max is a real within-family improvement.** It gains 9.4 points over
+  Terra xhigh and 15.9 over high. On point estimates across families, however,
+  Sol xhigh is both slightly cheaper and slightly higher-scoring, so Terra max
+  is not a universal Pareto winner.
+- **Sol high is a defensible efficiency point.** Sol max adds 3.3 points to
+  high for about 2.4 times the cost, and their confidence intervals overlap.
+  Sol xhigh adds 1.3 points for about 1.36 times the cost. This supports the
+  user's observation that `high` can be sufficient, but it does not prove the
+  higher efforts never matter.
+- **Fable high is the clearest “high is sufficient” result here.** High,
+  xhigh, and max are statistically overlapping; max has a slightly lower point
+  estimate than xhigh while costing 2.36 times high. The result argues against
+  defaulting Fable to max for implementation work.
+
+DeepSWE itself warns that adjacent frontier configurations often overlap in
+confidence intervals. Its v1.1 execution grades only committed patches in a
+fresh verifier container and removed dependency drift and flaky tests. It also
+reports that 73 of Fable's 2,260 trials did not complete after access was
+suspended during the sweep; Fable rates are calculated over completed trials
+([v1.1 report](https://deepswe.datacurve.ai/blog/deepswe-v1-1)).
+
+The standardized harness is both a strength and a limitation. It isolates the
+model comparison, but gives every model one Bash tool and a shared prompt
+instead of Codex's or Claude Code's native editing tools and prompts. DataCurve
+explicitly says the leaderboard does not directly represent those native
+products, and the corpus under-represents bug localization and refactoring
+([DeepSWE methodology and limitations](https://deepswe.datacurve.ai/blog/deepswe#limitations)).
+
+### Artificial Analysis
+
+#### Current coding-agent comparison
+
+Artificial Analysis' current Coding Agent Index v1.3 combines DeepSWE,
+Terminal-Bench, and SWE-Atlas-QnA. The currently exposed Codex/Claude Code rows
+use `max` effort. They are **agent + model + settings** measurements, not the
+same mini-swe-agent experiment as DataCurve
+([current comparison](https://artificialanalysis.ai/agents/coding-agents/comparisons/claude-code-vs-codex),
+[coding-agent methodology](https://artificialanalysis.ai/methodology/coding-agents-benchmarking)).
+
+| Agent configuration | Coding Agent Index | DeepSWE | Terminal-Bench | SWE-Atlas-QnA | Avg cost/task |
+|---|---:|---:|---:|---:|---:|
+| Codex + GPT-5.6 Sol max | 66.57 | 68.73% | 87.70% | 43.28% | $7.08 |
+| Codex + GPT-5.6 Terra max | 62.28 | 66.96% | 84.13% | 35.75% | $2.76 |
+| Codex + GPT-5.6 Luna max | 58.66 | 63.42% | 79.76% | 32.80% | $1.57 |
+| Claude Code + Fable 5 max | 65.85 | 66.08% | 82.54% | **48.92%** | $11.71 |
+
+Sol narrowly leads the aggregate and the implementation/terminal components;
+Fable leads repository Q&A. Luna gives up about 7.9 index points to Sol while
+costing about 78% less per task. That supports different routes for repository
+analysis, demanding implementation, and cost-sensitive execution rather than
+one global “best model.”
+
+Do not mix these figures with Artificial Analysis' July 9 launch chart without
+a version label. The launch chart used the prior scoring contract and reported
+80/77/77/75 for Sol/Terra/Fable/Luna. Coding Agent Index v1.2 changed
+SWE-Atlas-QnA from rubric reward to binary all-criteria success and v1.3 refined
+the alignment, so the current values are not a performance regression measured
+on an unchanged scale
+([launch analysis](https://artificialanalysis.ai/articles/gpt-5-6-has-landed),
+[versioned methodology](https://artificialanalysis.ai/methodology/coding-agents-benchmarking)).
+
+#### Intelligence-versus-cost by effort
+
+Artificial Analysis Intelligence Index v4.1 is a broader, English text-only
+composite: agents 34%, coding 24%, scientific reasoning 24%, and general
+capability 18%. Artificial Analysis estimates the composite's 95% confidence
+interval below ±1%, while warning that individual evaluations can be wider
+([Intelligence Index methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking)).
+
+Current model pages expose the following scores and weighted API cost per
+Intelligence Index task:
+
+| Model | Medium | High | Xhigh | Max |
+|---|---:|---:|---:|---:|
+| GPT-5.6 Sol | 54 / $0.314 | 56 / $0.453 | 58 / $0.682 | 59 / $1.037 |
+| GPT-5.6 Terra | 46 / $0.175 | 49 / $0.336 | 52 / $0.477 | 55 / $0.825 |
+| GPT-5.6 Luna | 38 / $0.050 | 46 / $0.095 | 49 / $0.139 | 51 / $0.209 |
+| Claude Fable 5 | — | — | — | 60 / $2.750 |
+
+Sources: Artificial Analysis model pages for
+[Sol max](https://artificialanalysis.ai/models/gpt-5-6-sol),
+[Sol high](https://artificialanalysis.ai/models/gpt-5-6-sol-high),
+[Terra max](https://artificialanalysis.ai/models/gpt-5-6-terra),
+[Terra high](https://artificialanalysis.ai/models/gpt-5-6-terra-high),
+[Luna max](https://artificialanalysis.ai/models/gpt-5-6-luna),
+[Luna high](https://artificialanalysis.ai/models/gpt-5-6-luna-high), and
+[Fable max](https://artificialanalysis.ai/models/claude-fable-5).
+
+Artificial Analysis' own cross-effort analysis concludes that Luna and Sol are
+ahead of Terra at every point on this composite's intelligence-versus-cost
+chart: for any Terra effort, a Luna or Sol configuration is at least as capable
+for no more cost, or equally capable for less. Luna is the standout
+cost-efficient family
+([GPT-5.6 intelligence-versus-cost analysis](https://artificialanalysis.ai/articles/gpt-5-6-intelligence-vs-cost-across-sol-terra-luna)).
+
+This result does **not** contradict Terra max's strong DeepSWE result. It shows
+why a single benchmark or a single global Pareto frontier cannot determine the
+route for every workload.
+
+## Provider guidance and availability
+
+OpenAI describes Sol, Terra, and Luna as durable capability tiers that can
+advance independently. Codex users with access can select each tier and an
+effort level; `max` is available across the family. OpenAI prices Sol at
+$5/$30, Terra at $2.50/$15, and Luna at $1/$6 per million input/output tokens
+([GPT-5.6 launch and availability](https://openai.com/index/gpt-5-6/)).
+
+Anthropic exposes `low`, `medium`, `high`, `xhigh`, and `max` where supported.
+It defines effort as a behavioral signal rather than a strict token budget and
+says the effect varies by workload. For Fable 5, Anthropic recommends starting
+at `high`, using `xhigh` only for the most capability-sensitive workloads, and
+lowering effort for routine work. Its general guidance is to use `max` only
+when the absolute highest capability justifies unconstrained token spending
+([Anthropic effort guidance](https://platform.claude.com/docs/en/build-with-claude/effort#recommended-effort-levels-for-claude-fable-5)).
+
+The Artificial Analysis Fable results also require a provenance warning:
+Fable's evaluated configuration uses adaptive reasoning at max effort with an
+Opus 4.8 fallback. It is a routing-system result, not a pure Fable-only result
+([Artificial Analysis Fable analysis](https://artificialanalysis.ai/articles/claude-fable-5-mythos)).
+
+## Measurement caveats
+
+- **Costs are estimates for the measured API traffic.** They include reported
+  input/output and supported cache prices, but exclude subscription economics,
+  developer supervision, CI, failed deployments, and the cost of rerunning an
+  incorrect patch. Provider or gateway pricing can differ.
+- **Retries differ by benchmark.** Artificial Analysis retries API failures up
+  to 30 times and withholds persistently broken results; this is reliability
+  handling, not multiple solution attempts. DeepSWE reports pass@1 over four
+  repeated whole-benchmark runs and exposes incomplete attempts separately.
+- **Pass rate is not expected project value.** A cheaper row can be worse if a
+  failure causes an expensive human recovery. Conversely, paying for max on a
+  highly constrained mechanical task can waste tokens without reducing risk.
+- **Statistical overlap matters.** Point estimates such as Sol high versus max
+  or Fable high versus max should not be treated as proven capability gaps when
+  their intervals overlap.
+- **Harness and task mix matter.** DataCurve's model-neutral Bash harness and
+  Artificial Analysis' native Codex/Claude Code rows answer different
+  questions. Neither directly measures this Kit's planning, grilling,
+  orchestration, review, or release workflows.
+- **Evidence expires.** Both leaderboards were updated during July 2026 and
+  their benchmark contracts have already changed. A recommendation without a
+  source date, benchmark version, and configuration identity is unsafe input
+  to automation.
+
+## Implication for the Kit routing policy
+
+The durable issue contract should continue to contain provider-neutral work
+intent. The mutable user or organization policy should be richer than a fixed
+`tier -> model + effort` lookup:
+
+```yaml
+routing-intent: implementation
+objective:
+  quality-floor: normal
+  optimize: expected-cost
+risk:
+  recovery-cost: medium
+```
+
+At reconciliation time, a surface adapter can evaluate the user's allowed
+models against a dated evidence catalog and propose a concrete route. The
+catalog entry must identify at least:
+
+- provider, model or alias, and effort;
+- benchmark name and version;
+- workload tags and harness;
+- score, uncertainty, average cost, and observation date;
+- fallback behavior and important missing data.
+
+The policy lifecycle should be:
+
+1. `setup-workflow` installs the schema and defaults to `inherit`; it may offer
+   an explicit user-local policy setup.
+2. Kit releases update schemas, adapters, and an optional evidence snapshot,
+   but never overwrite personal mappings.
+3. A separate `routing-policy reconcile` compares current choices with
+   available models and dated evidence, shows the relevant Pareto candidates,
+   and asks before changing the user-global policy.
+4. The resolver records the effective model and effort plus the policy/evidence
+   revision in run evidence.
+5. Local outcome telemetry and explicit user judgment may override public
+   benchmark recommendations. Public data supplies a prior; the user's real
+   workflow is the calibration set.
+
+This permits a current personal preference such as Fable high for planning,
+Sol or Terra for implementation, and Luna for cheap mechanical work without
+claiming that those mappings are correct for every consumer—or even for the
+same user after the next model or benchmark release.

package/docs/research/provider-neutral-agent-routing.md

@@ -0,0 +1,207 @@
+# Provider-neutral agent routing for shared work items
+
+**Researched:** 2026-07-22  
+**Question:** How should shared agent-workflow issues express model-routing intent across Claude Code and OpenAI Codex without embedding short-lived provider model names?
+
+## Recommendation
+
+Persist **task intent, not a model selection**, in shared issues. Use two small,
+provider-neutral fields:
+
+```yaml
+routing-intent: judgment | development | mechanical
+reasoning-intent: deep | balanced | light
+```
+
+The exact vocabulary is a Kit domain decision; the important constraint is that
+the values describe the work and desired trade-off, not Anthropic or OpenAI
+products. A surface-local resolver maps those values at dispatch time:
+
+```text
+shared issue intent
+        |
+        +-- Claude resolver -> current Claude alias/model + session effort
+        |
+        `-- Codex resolver  -> current Codex model + model_reasoning_effort
+```
+
+The default resolution must be `inherit` when no local policy exists or when a
+configured target cannot be proven usable. The orchestrator should report the
+effective resolution at session start, but it should not write that volatile
+result back into the issue.
+
+This separates three ownership layers:
+
+1. **Kit-owned schema and validation:** allowed intent values, inheritance,
+   diagnostics, and dispatch contract.
+2. **User/organization-owned surface policy:** how intent maps to models and
+   effort for that account, budget, provider, and current catalog.
+3. **Project-owned constraints, only when genuinely shared:** for example, a
+   prohibition on a model or a minimum capability required by every
+   collaborator. A project should not carry one maintainer's preferred model
+   mapping.
+
+## Verified platform facts
+
+### Claude Code
+
+Claude Code has Managed, User, Project, and Local scopes. For ordinary scalar
+settings, precedence is Managed, command-line, Local, Project, then User. A
+committed `.claude/settings.json` therefore affects all collaborators and
+overrides their user settings, while `.claude/settings.local.json` is personal
+and repository-specific ([Claude Code settings: scopes and precedence](https://code.claude.com/docs/en/settings#configuration-scopes)).
+
+Claude accepts provider aliases as well as full model IDs. Aliases such as
+`fable` and `sonnet` resolve to current models, while `default` returns to the
+runtime default for the account or organization. Aliases reduce version churn,
+but remain Claude-specific and do not encode a cross-provider task class
+([Claude Code model aliases](https://code.claude.com/docs/en/model-config#model-aliases)).
+
+The main-session model can be changed for a session with `/model` or at launch
+with `--model`; `ANTHROPIC_MODEL` and the persistent `model` setting are also
+supported. Organization restrictions and managed settings can replace or
+reject a requested selection, so a repository cannot assume that a named model
+is usable for every collaborator ([Claude Code model selection](https://code.claude.com/docs/en/model-config#setting-your-model),
+[Claude Code model restrictions](https://code.claude.com/docs/en/model-config#restrict-model-selection)).
+
+Claude subagent definitions support an alias, a full ID, or `inherit`; omission
+means `inherit`. Resolution currently considers
+`CLAUDE_CODE_SUBAGENT_MODEL`, a per-invocation value, subagent frontmatter, and
+finally the main conversation model. Excluded selections fall back to the
+inherited model. Extended-thinking configuration is inherited from the main
+conversation and has no per-subagent setting
+([Claude Code subagent model resolution](https://code.claude.com/docs/en/sub-agents#choose-a-model)).
+
+Claude effort levels are model-dependent. Unsupported levels are reduced to a
+supported level, organization policy can cap them, and some effort choices are
+session-only. Skill and subagent frontmatter can override the inherited session
+effort, subject to higher-precedence environment and organization constraints.
+Therefore a provider-neutral reasoning intent can be resolved on Claude, but a
+shared issue must not claim that one literal effort value has identical
+enforcement everywhere
+([Claude Code effort levels](https://code.claude.com/docs/en/model-config#adjust-effort-level),
+[Claude Code subagent effort](https://code.claude.com/docs/en/sub-agents#supported-frontmatter-fields)).
+
+Anthropic exposes `GET /v1/models`, whose result describes models available to
+that API credential ([Anthropic Models API](https://platform.claude.com/docs/en/api/models/list)).
+That is not a universal Claude Code discovery contract: Claude Code may run
+through a Claude subscription, Bedrock, Vertex, Foundry, or a gateway. Gateway
+discovery is optional, limited to compatible `/v1/models` gateways, cached,
+and falls back to built-in entries if discovery fails
+([Claude Code gateway model discovery](https://code.claude.com/docs/en/llm-gateway#model-selection)).
+
+### OpenAI Codex
+
+Codex reads personal defaults from `~/.codex/config.toml` and project overrides
+from trusted `.codex/config.toml` files. Its documented precedence is CLI,
+project config, selected user profile, user config, system config, then built-in
+defaults. Consequently, a committed project routing table can override a
+collaborator's personal defaults; untrusted projects skip project-local layers
+entirely ([Codex configuration precedence](https://learn.chatgpt.com/docs/config-file/config-basic#configuration-precedence)).
+
+Codex exposes `model` and `model_reasoning_effort` as separate configuration
+values ([Codex common configuration options](https://learn.chatgpt.com/docs/config-file/config-basic#common-configuration-options)).
+For custom subagents, a custom-agent file may override both. Otherwise each is
+resolved independently from an explicit spawn value, the `[agents]` default,
+and the parent session. Omitting a setting inherits it; changing the model
+without an explicit effort uses the selected model's default effort
+([Codex custom-agent resolution](https://learn.chatgpt.com/docs/agent-configuration/subagents#custom-agents)).
+
+Official Codex guidance explicitly allows leaving both values unpinned so the
+runtime can balance capability, speed, and price. It also documents that
+available reasoning levels depend on the selected model
+([Codex choosing models and reasoning](https://learn.chatgpt.com/docs/agent-configuration/subagents#choosing-models-and-reasoning)).
+
+OpenAI exposes `GET /v1/models` for models available to an API key
+([OpenAI Models API](https://platform.openai.com/docs/api-reference/models/list)).
+The Codex documentation does not provide a stable, public model-discovery API
+for the effective catalog of every Codex/ChatGPT account and local client.
+An API-key catalog therefore cannot safely populate routing for users whose
+Codex authentication, plan, organization policy, or client surface differs.
+
+## Options compared
+
+| Option | Survives catalog changes | Works across Claude and Codex | Multi-user behavior | Offline/update behavior | Verdict |
+|---|---:|---:|---|---|---|
+| Concrete model names in issues | No. Every rename, replacement, or access change makes existing issues stale. | No. Names and effort controls are provider-specific. | Assumes every collaborator has the same entitlement and budget. | Old issues remain wrong until rewritten. | Reject. Suitable only for an explicit reproducibility pin in a provider-specific experiment. |
+| Stable task intent resolved by user/org surface policy | Yes. Existing issues survive mapping changes. | Yes. Each surface resolves independently or inherits. | Preserves personal and organization choices. | Cached/local policy continues to work; unknown targets can inherit and warn. | **Adopt.** |
+| Project-local concrete routing table | Partly. One table can be updated, but every checkout still receives volatile provider data. | Only by maintaining multiple provider sections. | Project config outranks user config on both surfaces, so the maintainer's preferences can override collaborators. | Fresh clones can start stale; offline discovery cannot repair it. | Reject as the default. Permit only explicit team policy/constraints. |
+
+## Setup and update implications
+
+### `setup-workflow`
+
+`setup-workflow` should install or reconcile only the Kit-owned intent schema,
+templates, validation, and a documented inheritance fallback. It may **offer**
+to create a user-local routing policy for the detected surface, but only with
+explicit consent and never as a committed project default.
+
+Setup must not attempt to infer a complete routing table from `/v1/models`:
+
+- API discovery requires credentials and network access.
+- API availability is not equivalent to Claude Code or Codex subscription
+  availability across all supported providers and account types.
+- A discovered model ID says nothing about the user's desired cost/capability
+  trade-off.
+
+An interactive setup may validate explicitly selected targets using the active
+surface and show a preview. Failure or absence of discovery should leave the
+policy at `inherit`, not install guessed names.
+
+### `kit-update`
+
+`kit-update` should update the Kit-owned schema, resolver implementation, and
+provider adapter capabilities transactionally. It must preserve user-owned
+mappings and project-owned constraints. If the schema changes, migration must
+be semantic (for example, rename an intent key), previewed, and reversible;
+ordinary updates must not rewrite historical issues or replace personal model
+choices.
+
+At dispatch, the resolver should:
+
+1. read the issue's stable intent;
+2. apply organization/user policy and explicit session overrides according to
+   the active surface;
+3. resolve or inherit without network-dependent mutation;
+4. reject a malformed intent, but fall back to `inherit` with a visible warning
+   when a volatile target is unavailable;
+5. record the effective model and effort in run evidence, not in the durable
+   issue contract.
+
+This lets a later Kit release improve mappings for users who accept Kit
+defaults while allowing a maintainer's user-local policy to evolve independently.
+
+## Implication for #213
+
+#213 should not hardcode `sol`, `terra`, `luna`, Claude aliases, or full model
+IDs into the issue template or validator. It should instead establish the
+provider-neutral intent schema and make `to-issues` emit that schema. The
+execute-ready check should reject concrete provider model recommendations in
+shared handoffs, while orchestration resolves the intent on the active surface.
+
+The existing free-text `Recommended model: <Model [Effort]>` field conflates a
+durable work classification with a volatile local decision. Replace it with
+machine-readable intent and render the effective local recommendation only
+when a session is actually dispatched.
+
+## Open risks and decisions
+
+1. **Vocabulary quality:** `judgment/development/mechanical` and
+   `deep/balanced/light` need task-based definitions and contract tests so they
+   do not become disguised provider tiers.
+2. **Enforcement asymmetry:** Both surfaces can override model and effort for
+   subagents, but their precedence, supported values, organization controls,
+   and fallback behavior differ. The Kit must promise intent resolution, not
+   identical low-level enforcement.
+3. **Unavailable targets:** silent substitution hides policy drift. Inheritance
+   should be visible in run evidence, and strict team policies may choose to
+   fail instead.
+4. **Organization policy:** managed restrictions outrank personal/project
+   wishes. The resolver must treat the surface's effective selection as
+   authoritative.
+5. **Reproducibility exception:** a benchmark or regression reproduction may
+   legitimately pin a provider/model snapshot. That is a distinct,
+   provider-specific execution constraint, not the normal issue-routing field.
+6. **No universal discovery source:** model catalogs are useful validation
+   inputs where available, but cannot be the SSOT for setup or updates across
+   all supported account types and providers.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ikon85/agent-workflow-kit",
-  "version": "0.34.4",
+  "version": "0.34.5",
   "description": "Portable AI-agent workflow skills (plan → execute → land → learn) for Claude Code & Codex — grilling, TDD, diagnosis, two-axis code review, cross-model Codex review, design & domain-modeling, plus a skill router (ask-matt). npx init/update/diff/uninstall.",
   "type": "module",
   "bin": {

package/scripts/kit-update-pr.mjs

@@ -11,7 +11,7 @@ export const UPDATE_TITLE = 'chore: update agent workflow kit';
 const UPDATE_BODY = [
   'Automated, parity-verified update of `@ikon85/agent-workflow-kit`.',
   '',
-  'The transactional update candidate passed the consumer test suite before this branch was published.',
+  'The transactional update candidate passed the built-in Kit invariants before this branch was published.',
   '',
   'This pull request is never merged automatically.',
 ].join('\n');

package/scripts/kit-update-pr.test.mjs

@@ -46,6 +46,8 @@ test('automated update pull requests carry the behavior availability summary', a
   assert.match(created.body, /## Availability/);
   assert.match(created.body, /newly degraded: orchestrate-wave\.projectRecipe/);
   assert.match(created.body, /still unresolved: orchestrateWaveRecipe:invalid/);
+  assert.match(created.body, /built-in Kit invariants/);
+  assert.doesNotMatch(created.body, /consumer test/i);
   assert.match(created.body, /never merged automatically/);
 });
 

package/scripts/test_skill_readiness_contract.py

@@ -56,7 +56,12 @@ for (const skill of Object.keys(fixture.readinessFixture.skills)) {
 const kit = await makeKit({ '.claude/skills/to-prd/SKILL.md': 'fixture\n' });
 const consumer = await makeEmptyDir();
 async function setKitReadiness(manifest) {
-  const content = `${JSON.stringify(manifest, null, 2)}\n`;
+  const candidateManifest = structuredClone(manifest);
+  for (const [name, declaration] of Object.entries(candidateManifest.skills)) {
+    declaration.publish = name === 'to-prd';
+    if (name === 'to-prd') declaration.surfaces = ['claude'];
+  }
+  const content = `${JSON.stringify(candidateManifest, null, 2)}\n`;
   const path = join(kit, readinessPath);
   await mkdir(join(kit, '.claude/skills'), { recursive: true });
   await writeFile(path, content);

package/scripts/test_skill_setup_workflow_seeds.py

@@ -49,13 +49,12 @@ def classify(first_line, is_empty):
 
 
 def update_workflow_action(
-    provider, choice, destination_exists, prerequisites=True,
-    pull_requests_allowed=True,
+    provider, choice, destination_exists, pull_requests_allowed=True,
 ):
     """Reference decision table for the prompt-driven setup contract."""
     if provider != "github" or destination_exists or choice != "enable":
         return "skip"
-    return "create" if prerequisites and pull_requests_allowed else "skip"
+    return "create" if pull_requests_allowed else "skip"
 
 
 def load_census_setup_effects():
@@ -421,19 +420,18 @@ class SeedTemplatesValid(unittest.TestCase):
 
     def test_update_workflow_provider_and_choice_fixtures(self):
         fixtures = [
-            ("github", "enable", False, True, True, "create"),
-            ("github", "opt-out", False, True, True, "skip"),
-            ("github", "later", False, True, True, "skip"),
-            ("github", "enable", True, True, True, "skip"),
-            ("github", "enable", False, False, True, "skip"),
-            ("github", "enable", False, True, False, "skip"),
-            ("gitlab", "enable", False, True, True, "skip"),
-            ("local", "enable", False, True, True, "skip"),
+            ("github", "enable", False, True, "create"),
+            ("github", "opt-out", False, True, "skip"),
+            ("github", "later", False, True, "skip"),
+            ("github", "enable", True, True, "skip"),
+            ("github", "enable", False, False, "skip"),
+            ("gitlab", "enable", False, True, "skip"),
+            ("local", "enable", False, True, "skip"),
         ]
-        for provider, choice, exists, prerequisites, allowed, expected in fixtures:
+        for provider, choice, exists, allowed, expected in fixtures:
             self.assertEqual(
                 update_workflow_action(
-                    provider, choice, exists, prerequisites, allowed,
+                    provider, choice, exists, allowed,
                 ), expected,
             )
 
@@ -446,8 +444,8 @@ class SeedTemplatesValid(unittest.TestCase):
             "Ask later",
             "GitHub tracker",
             "skipped (already present)",
-            "package-lock.json",
-            "npm test",
+            "built-in Kit invariant validator",
+            "no Consumer package script",
             "can_approve_pull_request_reviews",
             "Allow GitHub Actions to create and approve pull requests",
             "explicit confirmation",
@@ -461,10 +459,13 @@ class SeedTemplatesValid(unittest.TestCase):
             "schedule:", "workflow_dispatch:", "contents: write",
             "pull-requests: write", "agent-workflow-kit-update-pr",
             "@ikon85/agent-workflow-kit@latest", "fetch-depth: 0",
-            "node-version: 22.14", "npm ci --ignore-scripts",
+            "node-version: 22.14", "Verify and upsert the Kit update pull request",
         ):
             self.assertIn(token, workflow)
-        for forbidden in ("npm_token", "NPM_TOKEN", "auto-merge", "gh pr merge"):
+        for forbidden in (
+            "npm_token", "NPM_TOKEN", "auto-merge", "gh pr merge",
+            "npm ci", "npm test", "package-lock.json",
+        ):
             self.assertNotIn(forbidden, workflow)
 
         mirror = (REPO / ".agents/skills/setup-workflow/assets/agent-workflow-kit-update.yml")